Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

["Ronald S. Bultje" <rsbultje@gmail.com>]

Hi,

On Mon, May 26, 2025 at 2:09 PM Timothée <
timothee.informatique@regaud-chapuy.fr> wrote:

> On 2025-05-23T13:55:57.000+02:00, Michael Niedermayer
> <michael@niedermayer.cc> wrote:
>
> > On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> >
> >>  On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
> >>  <michael@niedermayer.cc> wrote:
> >>
> >>>   On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer
> >>>   wrote:
> >>>
> >>>>    Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
> >>>>    Bultje wrote:
> >>>>
> >>>>>     Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>>>>     timothee.informatique@regaud-chapuy.fr> wrote:
> >>>>>
> >>>>>>      Hello, I am interested in expanding ffmpeg's capabilities
> >>>>>>      to extract low-level data from video codecs. Specifically,
> >>>>>>      I'd like to implement functionality that would allow
> >>>>>>      exporting frame data, macroblock information, quantization
> >>>>>>      tables, and similar codec-specific elements to binary
> >>>>>>      files for further analysis. After searching through the
> >>>>>>      documentation and existing features, I haven't found
> >>>>>>      similar functionality, though I may have missed something.
> >>>>>>      Has this been implemented before, or are there related
> >>>>>>      features I should examine?
> >>>>>
> >>>>>      Some older codecs implement minor variants for this, e.g.
> >>>>>     grep for AV_FRAME_DATA_MOTION_VECTORS, which attaches a
> >>>>>     frame's motion vectors to the picture data. I believe
> >>>>>     there's an example app and possibly a filter to overlay MVs
> >>>>>     on top of the video frame based on this concept. You could
> >>>>>     extend this to cover other (macro)block info. There used to
> >>>>>     be a variant of this for quant-tables also but I can't find
> >>>>>     it, maybe it was removed.
> >>>>
> >>>>     For motion vectors: ./ffplay -flags2 +export_mvs -i
> >>>>    matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
> >>>>    segmentation and type vissualization + also motion vectors:
> >>>>    ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
> >>>>    codecview=mv=pf+bf+bb For QP vissualization + also motion
> >>>>    vectors: ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
> >>>>    codecview=mv=pf+bf+bb For qp values dumped on the console
> >>>>    ./ffplay -debug qp -i matrixbench_mpeg2.mpg
> >>>
> >>>    And this can easily be extended to other codecs, ATM it should
> >>>   work with all 16x16 MB based codecs like
> >>>   msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization
> >>>   need codecview to be extended or versions around 3.4 which
> >>>   implemented it differently Implementing vissualization as done
> >>>   currently with sidedata and codecview is simple and efficient.
> >>>   It also would allow exporting the data to json by writing a
> >>>   codec2json filter in place of codecview Also all decoders
> >>>   already have all this data parsed and available so its simpler
> >>>   than trying to do it in a decoder independant way I would thus
> >>>   suggest implementations of this for modern codecs to follow the
> >>>   same path as the existing code. thx
> >>
> >>   Thanks for the helpful pointers! I will work on the codec2json
> >>  filter. Looking at the code, I see where I can access sidedata but
> >>  extracting qb table seems to fail. (in codecview.c l.233:
> >>  ff_qp_table_extract() return 0 and qp_table is empty) (I use
> >>  ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1
> >>  output.mp4 -y) Is is qp extraction not implemented yet? Or is it
> >>  because I have h264 video? If it's not implemented, I'm curious
> >>  why there’s already code that appears to handle it.
> >
> >  look at ff_print_debug_info2() theres probably something missing
> > The QP and MB type code was changed from being inside arrays of
> > pictures to sidedata. Something likely was lost/forgotten in the
> > process -debug qp works with h264 so likely teh export into sidedata
> > is not fully implemented
>
> Yes, this is exactly what is happening.
>
> After some research, I have found: `// FIXME qscale / qp ... stuff` in
> libavcodec/h264_slice.c, line 1871
>
> and also in libavfilter/qp_table.h:
> 34 /**
> 35 * Normalize the qscale factor
> 36 * FIXME Add support for other values of enum AVVideoEncParamsType
> 37 * besides AV_VIDEO_ENC_PARAMS_MPEG2.
> 38 */
> 39 static inline int ff_norm_qscale(int qscale, enum
> AVVideoEncParamsType type)
> 40 {
> 41     switch (type) {
> 42        case AV_VIDEO_ENC_PARAMS_MPEG2: return qscale >> 1;
> 43     }
> 44     return qscale;
> 45 }
>
> So if I understand correctly, QP tables are not attached to the frames
> except for MPEG2. This has been left as a FIXME for other codecs.
> Should I extend it to work for H.264, and maybe later for other
> codecs, thus removing the first FIXME and partially fixing the second?
> Or is this too specific and too complicated to maintain if it needs to
> be added for every codec?


I think it's fine to add it one codec at a time.

Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".