From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id B0AB9444B4 for ; Mon, 12 Sep 2022 23:49:36 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 952A668BADD; Tue, 13 Sep 2022 02:49:33 +0300 (EEST) Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 92C7468B9CE for ; Tue, 13 Sep 2022 02:49:26 +0300 (EEST) Received: by mail-lf1-f54.google.com with SMTP id k10so17417160lfm.4 for ; Mon, 12 Sep 2022 16:49:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=berkeley.edu; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=Py8MqhFEUvsx8OzDy7KGhRoI8DvkKI8OE/7A3hlI5Xc=; b=WFTqfo/VhvzlfnTYpWRAIo4gIe+tznbvxORzj0JDEm+tGP7T7T4FSSn+6GVH/9P5Cf ZBx2DIDP/djln31/cPKPuT0BuPGWw+lEeSNr9fkglDXtx3QhoPZv7Is4xI0N8STWAMwU VuvxgGjA3pJE6TS2SLBkLHjT5/pJPDLPzsMH4jGMyc1nh4XZoY5GvW2+2ZhOrA1HfF5e vZlsfXUQFHCK33cjagM0ldpcUWDhRp3QVugM50QaJsxvsQoBs3mEg6T++G+8eAkBDpYh F2GYoCCPeHfD3dXpuVi8vZpAB2MwYX+HggRCylz5mN4QHKzx0SvO06ZeVD51mAJLSGw8 YJCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=Py8MqhFEUvsx8OzDy7KGhRoI8DvkKI8OE/7A3hlI5Xc=; b=5t5DzSbGezvTHxF3qaFDP4TFQTpcMcAPmz1iQzTtaDnvRQrCADEHEHSKjf5LrLJom0 Q7svilvJhE7q6jFJpD/5Xu2J6qTHVXgdMttpmOmtPhlndVzOlmPPCplAEfolAPULLeZv XGkuvFUfRf3igLYAyK5IdE3vEUEJXQ2B8GfDp3BJXbF/xcl9YIezUCm9xVOXbksfdjMG LEbxVlCu1WcCMsEOLjNlNbSp3qOm9lsL7W2XEEDijc59K+ocUPFOZDCe4PLC3XW0Ix61 HO5p3qf8jQAgT1GEwtSL8w87dK0RM6tnZGdsBXt3zBAHBF7q+uwZVt5QrEeZoohUEUhh 0Emw== X-Gm-Message-State: ACgBeo2eg6P1iy4WyfbvHHJGlQU4tY3BMUM8OO4nll28oRWspv1/i7Dc en2l9MBC4J+WEQb3wq4X46RIXsVj2vy2R1c5I6I= X-Google-Smtp-Source: AA6agR4YI1WkGzcvW4irRSENyrWaVoKa52wphCTdsplbjb7NQ4nDapI84v+nkjzVCECUGrXZWtJvfBtF8AlWbmoMh0E= X-Received: by 2002:ac2:4f03:0:b0:496:e4:4d16 with SMTP id k3-20020ac24f03000000b0049600e44d16mr8807654lfr.250.1663026565451; Mon, 12 Sep 2022 16:49:25 -0700 (PDT) MIME-Version: 1.0 References: <20220831230918.1736362-1-chemag@gmail.com> <20220831230918.1736362-2-chemag@gmail.com> In-Reply-To: From: Chema Gonzalez Date: Mon, 12 Sep 2022 16:49:13 -0700 Message-ID: To: "Ronald S. Bultje" Subject: Re: [FFmpeg-devel] [PATCH 1/1] avcodec/mpegutils: add motion_vec debug mode X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: FFmpeg development discussions and patches Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Hi Paul, I tried to understand the per-frame side-data (and metadata) mechanism. Adding my notes here in case they help a future reader. Metadata and side-data seem like similar mechanisms to add auxiliary information to each frame. Main difference seems to be that metadata is simpler (a key/value dictionary), while side-data allows adding any data struct. Metadata operation is also simpler: There are filters that generate metadata (e.g. "signalstats" generates key-values such as `lavfi.signalstats.YMIN=3`, while "silencedetect" generates audio-related key-values like `lavfi.silence_start=0`). There are also 2x filters that print metadata ("vf_metadata" and "af_ametadata"). So e.g. to see what signalstats/silencedetect are generating, you can do: ``` $ ffmpeg -i in.264 -vf signalstats,metadata=mode=print -f null - ... [Parsed_metadata_1 @ ...] frame:0 pts:0 pts_time:0 [Parsed_metadata_1 @ ...] lavfi.signalstats.YMIN=3 ... ``` or: ``` $ ffmpeg -y -i in.wav -af "silencedetect=n=-10dB:d=1,ametadata=print" /tmp/out.wav ... [Parsed_ametadata_1 @ ...] frame:23 pts:47104 pts_time:0.981333 [Parsed_ametadata_1 @ ...] lavfi.silence_start=0 ... ``` Side-data operation is more complicated. There is some side-data information already generated (e.g. SEI_UNREGISTERED side-data). Some information requires explicitly asking for it. For example, the `MOTION_VECTORS` side-data, you need to enable AV_CODEC_FLAG2_EXPORT_MVS (which means calling ffmpeg/ffplay/ffprobe with "-flags2 +export_mvs"). The main filter to print side-data information is showinfo (`vf_showinfo` and `af_ashowinfo`). Now, the `vf_showinfo` filter only knows how to dump some of the side-data structs. In particular, it does not know how to dump MOTION_VECTORS side-data. So, if we add the motion vectors, and then ask showinfo to print it, we see: ``` $ ffmpeg -hide_banner -flags2 +export_mvs -export_side_data +mvs -export_side_data +prft -export_side_data +venc_params -export_side_data +film_grain -i /tmp/in.264 -vf showinfo -f null /dev/null ... -- frame 0 is a key frame: We can see SEI_UNREGISTERED and VIDEO_ENC_PARAMS info [Parsed_showinfo_0 @ 0x308fd40] config in time_base: 1/1200000, frame_rate: 25/1 [Parsed_showinfo_0 @ 0x308fd40] config out time_base: 0/0, frame_rate: 0/0 [Parsed_showinfo_0 @ 0x308fd40] n: 0 pts: 0 pts_time:0 duration: 48000 duration_time:0.04 pos: 0 fmt:yuv420p sar:0/1 s:1920x1080 i:P iskey:1 type:I checksum:F6BBEA9F plane_checksum:[AFB1432E 63F2F255 2887B50D] mean:[109 119 138] stdev:[43.0 12.7 13.3] [Parsed_showinfo_0 @ 0x308fd40] side data - User Data Unregistered: [Parsed_showinfo_0 @ 0x308fd40] UUID=47564adc-5c4c-433f-94ef-c5113cd143a8 [Parsed_showinfo_0 @ 0x308fd40] User Data=01ffccccff0200e4dd42 [Parsed_showinfo_0 @ 0x308fd40] [Parsed_showinfo_0 @ 0x308fd40] side data - video encoding parameters: type 1; qp=26; 8160 blocks; [Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709 color_primaries:bt709 color_trc:bt709 ... -- frame 1 is a P-frame: we can see VIDEO_ENC_PARAMS info, and a complain about "side-data type 8" (MOTION_VECTORS) [Parsed_showinfo_0 @ 0x308fd40] n: 1 pts: 48000 pts_time:0.04 duration: 48000 duration_time:0.04 pos: 259304 fmt:yuv420p sar:0/1 s:1920x1080 i:P iskey:0 type:B checksum:BC4E5C12 plane_checksum:[AEA8857A 34697DA4 805E58E5] mean:[109 119 138] stdev:[43.0 12.6 13.3] [Parsed_showinfo_0 @ 0x308fd40] side data - video encoding parameters: type 1; qp=26; 8160 blocks; -- showinfo does not dump MOTION_VECTORS side-data [Parsed_showinfo_0 @ 0x308fd40] side data - unknown side data type 8 (547280 bytes) [Parsed_showinfo_0 @ 0x308fd40] [Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709 color_primaries:bt709 color_trc:bt709 ... ``` So the best way right now to see the MVs is to use `doc/examples/extract_mvs`, which does exactly that: ``` $ make examples -j ... $ doc/examples/extract_mvs in.264 | head -40 | \ csvcut -C framenum,source,flags |csvlook | blockw | blockh | srcx | srcy | dstx | dsty | motion_x | motion_y | motion_scale | | ------ | ------ | ----- | ---- | ----- | ---- | -------- | -------- | ------------ | | 16 | 16 | 20 | 26 | 8 | 8 | 49 | 72 | 4 | | 16 | 16 | 152 | 15 | 136 | 8 | 65 | 28 | 4 | | 16 | 8 | 360 | 3 | 360 | 4 | 1 | -6 | 4 | | 16 | 8 | 360 | 13 | 360 | 12 | -1 | 4 | 4 | | 16 | 16 | 440 | 10 | 440 | 8 | 3 | 10 | 4 | | 8 | 16 | 829 | 7 | 836 | 8 | -31 | -6 | 4 | | 8 | 16 | 844 | 7 | 844 | 8 | -1 | -4 | 4 | ``` > Yes, it's called codecview. We can help understand how it works if you ask more specific questions, but something like "git grep EXPORT_DATA_MVS ../libavcodec/mpeg*.c" and checking the complement code in codecview should explain the basics. codecview will get the motion vectors (assuming you asked for them), and will overlay arrows showing the MV on top of the original video. ``` $ ffmpeg -flags2 +export_mvs -i input -vf codecview=mv=pf+bf+bb output.%04d.png ``` Adding one of the resulting frames in the next email. -Chema On Sat, Sep 3, 2022 at 5:08 AM Ronald S. Bultje wrote: > > Hi Chema, > > On Fri, Sep 2, 2022 at 11:12 AM Chema Gonzalez wrote: >> >> So is there a filter that already dumps this information? > > > Yes, it's called codecview. We can help understand how it works if you ask more specific questions, but something like "git grep EXPORT_DATA_MVS ../libavcodec/mpeg*.c" and checking the complement code in codecview should explain the basics. > > Ronald _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".