[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

Hello,

I am interested in expanding ffmpeg's capabilities to extract
low-level data from video codecs. Specifically, I'd like to implement
functionality that would allow exporting frame data, macroblock
information, quantization tables, and similar codec-specific elements
to binary files for further analysis.

After searching through the documentation and existing features, I
haven't found similar functionality, though I may have missed
something. Has this been implemented before, or are there related
features I should examine?

I'd appreciate your feedback on whether this feature would be
considered valuable for the project and if it aligns with ffmpeg's
development goals. If there's interest, I can provide a more detailed
technical proposal.

I'm new to both ffmpeg development and open source contributions in
general, so I welcome any guidance or information that might help me
contribute effectively.

Thank you for your time and consideration.

Timothée

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ronald S. Bultje]

Hi,

On Wed, May 21, 2025 at 9:34 AM Timothée <
timothee.informatique@regaud-chapuy.fr> wrote:

> Hello,
>
> I am interested in expanding ffmpeg's capabilities to extract
> low-level data from video codecs. Specifically, I'd like to implement
> functionality that would allow exporting frame data, macroblock
> information, quantization tables, and similar codec-specific elements
> to binary files for further analysis.
>
> After searching through the documentation and existing features, I
> haven't found similar functionality, though I may have missed
> something. Has this been implemented before, or are there related
> features I should examine?


Some older codecs implement minor variants for this, e.g. grep
for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
to the picture data. I believe there's an example app and possibly a filter
to overlay MVs on top of the video frame based on this concept. You could
extend this to cover other (macro)block info. There used to be a variant of
this for quant-tables also but I can't find it, maybe it was removed.

Placing the side-data in binary files is probably not really something that
libavcodec would do, but rather something that would exist at the
application level.

Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ramiro Polla]

Hi,

On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje@gmail.com> wrote:
> On Wed, May 21, 2025 at 9:34 AM Timothée <
> timothee.informatique@regaud-chapuy.fr> wrote:
> > I am interested in expanding ffmpeg's capabilities to extract
> > low-level data from video codecs. Specifically, I'd like to implement
> > functionality that would allow exporting frame data, macroblock
> > information, quantization tables, and similar codec-specific elements
> > to binary files for further analysis.
> >
> > After searching through the documentation and existing features, I
> > haven't found similar functionality, though I may have missed
> > something. Has this been implemented before, or are there related
> > features I should examine?
>
> Some older codecs implement minor variants for this, e.g. grep
> for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
> to the picture data. I believe there's an example app and possibly a filter
> to overlay MVs on top of the video frame based on this concept. You could
> extend this to cover other (macro)block info. There used to be a variant of
> this for quant-tables also but I can't find it, maybe it was removed.

If I recall correctly, we wanted to move away from exporting more of
this kind of codec-specific information as picture data.

Timothée, this kind of feature depends a lot on the codec that you
want to work with. It will also depend on which syntax elements you
need from the codec. In FFmpeg, there is the CBS code which describes
the bitstream for a few codecs, which might help you.

I have a separate project (called FFglitch), which lets you export
some elements (such as quantization tables, dct coefficients, and
motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON format,
and even modify them in the bitstream.

In the end, it will depend on what you want to do with the extracted
low-level data, and for which codecs.

Ramiro
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

The 2025-05-22T14:33:22.000+02:00, Ramiro Polla
<ramiro.polla@gmail.com> wrote :

> Hi,
> 
> On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje@gmail.com> wrote:
> 
>>  On Wed, May 21, 2025 at 9:34 AM Timothée <
>>  timothee.informatique@regaud-chapuy.fr> wrote: 
>>  
>>>   I am interested in expanding ffmpeg's capabilities to extract
>>>   low-level data from video codecs. Specifically, I'd like to
>>>   implement functionality that would allow exporting frame data,
>>>   macroblock information, quantization tables, and similar
>>>   codec-specific elements to binary files for further analysis.
>>>   After searching through the documentation and existing features,
>>>   I haven't found similar functionality, though I may have missed
>>>   something. Has this been implemented before, or are there
>>>   related features I should examine?
>>  
>>   Some older codecs implement minor variants for this, e.g. grep
>>  for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion
>>  vectors to the picture data. I believe there's an example app and
>>  possibly a filter to overlay MVs on top of the video frame based
>>  on this concept. You could extend this to cover other (macro)block
>>  info. There used to be a variant of this for quant-tables also but
>>  I can't find it, maybe it was removed.
> 
>  If I recall correctly, we wanted to move away from exporting more
> of this kind of codec-specific information as picture data.
> Timothée, this kind of feature depends a lot on the codec that you
> want to work with. It will also depend on which syntax elements you
> need from the codec. In FFmpeg, there is the CBS code which
> describes the bitstream for a few codecs, which might help you. I
> have a separate project (called FFglitch), which lets you export
> some elements (such as quantization tables, dct coefficients, and
> motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON
> format, and even modify them in the bitstream. In the end, it will
> depend on what you want to do with the extracted low-level data, and
> for which codecs. Ramiro

Thank you for your response and for mentioning FFglitch - it looks
like an excellent project that's very relevant to what I'm trying to
accomplish.

I have a question about FFglitch's capabilities: is it possible to
extract quantization parameter (QP) tables using ffedit? I've reviewed
the documentation but may have missed this functionality if it exists.

Ideally I would want to extract data for all existing codecs, but I
know this is impossible. My current focus is on implementing this
feature for H.264, with plans to extend support to H.265 and AVI
formats as well. I want to extract this data for scientific use.

Thank you again for your time and suggestions.

Timothée 

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ramiro Polla]

On Thu, May 22, 2025 at 7:03 PM Timothée
<timothee.informatique@regaud-chapuy.fr> wrote:
> The 2025-05-22T14:33:22.000+02:00, Ramiro Polla
> <ramiro.polla@gmail.com> wrote :
> > On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje@gmail.com> wrote:
> >>  On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>  timothee.informatique@regaud-chapuy.fr> wrote:
> >>>   I am interested in expanding ffmpeg's capabilities to extract
> >>>   low-level data from video codecs. Specifically, I'd like to
> >>>   implement functionality that would allow exporting frame data,
> >>>   macroblock information, quantization tables, and similar
> >>>   codec-specific elements to binary files for further analysis.
> >>>   After searching through the documentation and existing features,
> >>>   I haven't found similar functionality, though I may have missed
> >>>   something. Has this been implemented before, or are there
> >>>   related features I should examine?
> >>
> >>   Some older codecs implement minor variants for this, e.g. grep
> >>  for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion
> >>  vectors to the picture data. I believe there's an example app and
> >>  possibly a filter to overlay MVs on top of the video frame based
> >>  on this concept. You could extend this to cover other (macro)block
> >>  info. There used to be a variant of this for quant-tables also but
> >>  I can't find it, maybe it was removed.
> >
> >  If I recall correctly, we wanted to move away from exporting more
> > of this kind of codec-specific information as picture data.
> > Timothée, this kind of feature depends a lot on the codec that you
> > want to work with. It will also depend on which syntax elements you
> > need from the codec. In FFmpeg, there is the CBS code which
> > describes the bitstream for a few codecs, which might help you. I
> > have a separate project (called FFglitch), which lets you export
> > some elements (such as quantization tables, dct coefficients, and
> > motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON
> > format, and even modify them in the bitstream. In the end, it will
> > depend on what you want to do with the extracted low-level data, and
> > for which codecs. Ramiro
>
> Thank you for your response and for mentioning FFglitch - it looks
> like an excellent project that's very relevant to what I'm trying to
> accomplish.
>
> I have a question about FFglitch's capabilities: is it possible to
> extract quantization parameter (QP) tables using ffedit? I've reviewed
> the documentation but may have missed this functionality if it exists.

You can get the dqt for jpeg and qscale for mpeg-2, but that's about
it for the moment.

> Ideally I would want to extract data for all existing codecs, but I
> know this is impossible. My current focus is on implementing this
> feature for H.264, with plans to extend support to H.265 and AVI
> formats as well. I want to extract this data for scientific use.

I see a few qp fields in libavcodec/cbs_h264.h and
libavcodec/cbs_h265.h. I've never used this functionality before, but
it might help you.

OTOH, if you want to add h264 support to FFglitch I would be very happy :)

Ramiro
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ronald S. Bultje]

Hi,

On Thu, May 22, 2025 at 8:33 AM Ramiro Polla <ramiro.polla@gmail.com> wrote:

> On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje@gmail.com>
> wrote:
> > On Wed, May 21, 2025 at 9:34 AM Timothée <
> > timothee.informatique@regaud-chapuy.fr> wrote:
> > > I am interested in expanding ffmpeg's capabilities to extract
> > > low-level data from video codecs. Specifically, I'd like to implement
> > > functionality that would allow exporting frame data, macroblock
> > > information, quantization tables, and similar codec-specific elements
> > > to binary files for further analysis.
> > >
> > > After searching through the documentation and existing features, I
> > > haven't found similar functionality, though I may have missed
> > > something. Has this been implemented before, or are there related
> > > features I should examine?
> >
> > Some older codecs implement minor variants for this, e.g. grep
> > for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
> > to the picture data. I believe there's an example app and possibly a
> filter
> > to overlay MVs on top of the video frame based on this concept. You could
> > extend this to cover other (macro)block info. There used to be a variant
> of
> > this for quant-tables also but I can't find it, maybe it was removed.
>
> If I recall correctly, we wanted to move away from exporting more of
> this kind of codec-specific information as picture data.
>
> Timothée, this kind of feature depends a lot on the codec that you
> want to work with. It will also depend on which syntax elements you
> need from the codec. In FFmpeg, there is the CBS code which describes
> the bitstream for a few codecs, which might help you.


I believe CBS only describes headers, not block data?

Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2061 bytes --]

Hi Ronald

On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S. Bultje wrote:
> Hi,
> 
> On Wed, May 21, 2025 at 9:34 AM Timothée <
> timothee.informatique@regaud-chapuy.fr> wrote:
> 
> > Hello,
> >
> > I am interested in expanding ffmpeg's capabilities to extract
> > low-level data from video codecs. Specifically, I'd like to implement
> > functionality that would allow exporting frame data, macroblock
> > information, quantization tables, and similar codec-specific elements
> > to binary files for further analysis.
> >
> > After searching through the documentation and existing features, I
> > haven't found similar functionality, though I may have missed
> > something. Has this been implemented before, or are there related
> > features I should examine?
> 
> 
> Some older codecs implement minor variants for this, e.g. grep
> for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
> to the picture data. I believe there's an example app and possibly a filter
> to overlay MVs on top of the video frame based on this concept. You could
> extend this to cover other (macro)block info. There used to be a variant of
> this for quant-tables also but I can't find it, maybe it was removed.

For motion vectors:
./ffplay -flags2 +export_mvs -i matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb

For macroblock segmentation and type vissualization + also motion vectors:
ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb

For QP vissualization + also motion vectors:
ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb

For qp values dumped on the console
./ffplay  -debug qp  -i matrixbench_mpeg2.mpg

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The real ebay dictionary, page 1
"Used only once"    - "Some unspecified defect prevented a second use"
"In good condition" - "Can be repaird by experienced expert"
"As is" - "You wouldnt want it even if you were payed for it, if you knew ..."

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2930 bytes --]

On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote:
> Hi Ronald
> 
> On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S. Bultje wrote:
> > Hi,
> > 
> > On Wed, May 21, 2025 at 9:34 AM Timothée <
> > timothee.informatique@regaud-chapuy.fr> wrote:
> > 
> > > Hello,
> > >
> > > I am interested in expanding ffmpeg's capabilities to extract
> > > low-level data from video codecs. Specifically, I'd like to implement
> > > functionality that would allow exporting frame data, macroblock
> > > information, quantization tables, and similar codec-specific elements
> > > to binary files for further analysis.
> > >
> > > After searching through the documentation and existing features, I
> > > haven't found similar functionality, though I may have missed
> > > something. Has this been implemented before, or are there related
> > > features I should examine?
> > 
> > 
> > Some older codecs implement minor variants for this, e.g. grep
> > for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
> > to the picture data. I believe there's an example app and possibly a filter
> > to overlay MVs on top of the video frame based on this concept. You could
> > extend this to cover other (macro)block info. There used to be a variant of
> > this for quant-tables also but I can't find it, maybe it was removed.
> 
> For motion vectors:
> ./ffplay -flags2 +export_mvs -i matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb
> 
> For macroblock segmentation and type vissualization + also motion vectors:
> ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb
> 
> For QP vissualization + also motion vectors:
> ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg  -vf codecview=mv=pf+bf+bb
> 
> For qp values dumped on the console
> ./ffplay  -debug qp  -i matrixbench_mpeg2.mpg

And this can easily be extended to other codecs, ATM it should work with
all 16x16 MB based codecs like msmpeg4*/wmv*/mpeg1/2/4/h263/h264

mbtype and qp vissualization need codecview to be extended or versions around 3.4
which implemented it differently

Implementing vissualization as done currently with sidedata and codecview
is simple and efficient. It also would allow exporting the data to json
by writing a codec2json filter in place of codecview

Also all decoders already have all this data parsed and available so
its simpler than trying to do it in a decoder independant way

I would thus suggest implementations of this for modern codecs to
follow the same path as the existing code.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"You are 36 times more likely to die in a bathtub than at the hands of a
terrorist. Also, you are 2.5 times more likely to become a president and
2 times more likely to become an astronaut, than to die in a terrorist
attack." -- Thoughty2


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
<michael@niedermayer.cc> wrote:

> On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote:
> 
>>  Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
>>  Bultje wrote: 
>>  
>>>   Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
>>>   timothee.informatique@regaud-chapuy.fr> wrote: 
>>>   
>>>>    Hello, I am interested in expanding ffmpeg's capabilities to
>>>>    extract low-level data from video codecs. Specifically, I'd
>>>>    like to implement functionality that would allow exporting
>>>>    frame data, macroblock information, quantization tables, and
>>>>    similar codec-specific elements to binary files for further
>>>>    analysis. After searching through the documentation and
>>>>    existing features, I haven't found similar functionality,
>>>>    though I may have missed something. Has this been implemented
>>>>    before, or are there related features I should examine?
>>>   
>>>    Some older codecs implement minor variants for this, e.g. grep
>>>   for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's
>>>   motion vectors to the picture data. I believe there's an example
>>>   app and possibly a filter to overlay MVs on top of the video
>>>   frame based on this concept. You could extend this to cover
>>>   other (macro)block info. There used to be a variant of this for
>>>   quant-tables also but I can't find it, maybe it was removed.
>>  
>>   For motion vectors: ./ffplay -flags2 +export_mvs -i
>>  matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
>>  segmentation and type vissualization + also motion vectors:
>>  ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
>>  codecview=mv=pf+bf+bb For QP vissualization + also motion vectors:
>>  ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
>>  codecview=mv=pf+bf+bb For qp values dumped on the console ./ffplay
>>  -debug qp -i matrixbench_mpeg2.mpg
> 
>  And this can easily be extended to other codecs, ATM it should work
> with all 16x16 MB based codecs like
> msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization need
> codecview to be extended or versions around 3.4 which implemented it
> differently Implementing vissualization as done currently with
> sidedata and codecview is simple and efficient. It also would allow
> exporting the data to json by writing a codec2json filter in place
> of codecview Also all decoders already have all this data parsed and
> available so its simpler than trying to do it in a decoder
> independant way I would thus suggest implementations of this for
> modern codecs to follow the same path as the existing code. thx

Thanks for the helpful pointers!

I will work on the codec2json filter.

Looking at the code, I see where I can access sidedata but extracting
qb table seems to fail. (in codecview.c l.233: ff_qp_table_extract()
return 0 and qp_table is empty) (I use ./ffmpeg -flags2 +export_mvs -i
input.mp4 -vf codecview=qp=1 output.mp4 -y)

Is is qp extraction not implemented yet? Or is it because I have h264
video? If it's not implemented, I'm curious why there’s already code
that appears to handle it.

Timothée

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 4371 bytes --]

On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
> <michael@niedermayer.cc> wrote:
> 
> > On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote:
> > 
> >>  Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
> >>  Bultje wrote: 
> >>  
> >>>   Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>>   timothee.informatique@regaud-chapuy.fr> wrote: 
> >>>   
> >>>>    Hello, I am interested in expanding ffmpeg's capabilities to
> >>>>    extract low-level data from video codecs. Specifically, I'd
> >>>>    like to implement functionality that would allow exporting
> >>>>    frame data, macroblock information, quantization tables, and
> >>>>    similar codec-specific elements to binary files for further
> >>>>    analysis. After searching through the documentation and
> >>>>    existing features, I haven't found similar functionality,
> >>>>    though I may have missed something. Has this been implemented
> >>>>    before, or are there related features I should examine?
> >>>   
> >>>    Some older codecs implement minor variants for this, e.g. grep
> >>>   for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's
> >>>   motion vectors to the picture data. I believe there's an example
> >>>   app and possibly a filter to overlay MVs on top of the video
> >>>   frame based on this concept. You could extend this to cover
> >>>   other (macro)block info. There used to be a variant of this for
> >>>   quant-tables also but I can't find it, maybe it was removed.
> >>  
> >>   For motion vectors: ./ffplay -flags2 +export_mvs -i
> >>  matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
> >>  segmentation and type vissualization + also motion vectors:
> >>  ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
> >>  codecview=mv=pf+bf+bb For QP vissualization + also motion vectors:
> >>  ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
> >>  codecview=mv=pf+bf+bb For qp values dumped on the console ./ffplay
> >>  -debug qp -i matrixbench_mpeg2.mpg
> > 
> >  And this can easily be extended to other codecs, ATM it should work
> > with all 16x16 MB based codecs like
> > msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization need
> > codecview to be extended or versions around 3.4 which implemented it
> > differently Implementing vissualization as done currently with
> > sidedata and codecview is simple and efficient. It also would allow
> > exporting the data to json by writing a codec2json filter in place
> > of codecview Also all decoders already have all this data parsed and
> > available so its simpler than trying to do it in a decoder
> > independant way I would thus suggest implementations of this for
> > modern codecs to follow the same path as the existing code. thx
> 
> Thanks for the helpful pointers!
> 
> I will work on the codec2json filter.
> 
> Looking at the code, I see where I can access sidedata but extracting
> qb table seems to fail. (in codecview.c l.233: ff_qp_table_extract()
> return 0 and qp_table is empty) (I use ./ffmpeg -flags2 +export_mvs -i
> input.mp4 -vf codecview=qp=1 output.mp4 -y)
> 
> Is is qp extraction not implemented yet? Or is it because I have h264
> video? If it's not implemented, I'm curious why there’s already code
> that appears to handle it.

look at ff_print_debug_info2()
theres probably something missing

The QP and MB type code was changed from being inside arrays of pictures
to sidedata. Something likely was lost/forgotten in the process

-debug qp
works with h264 so likely teh export into sidedata is not fully
implemented

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"Nothing to hide" only works if the folks in power share the values of
you and everyone you know entirely and always will -- Tom Scott


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Timothée]

On 2025-05-23T13:55:57.000+02:00, Michael Niedermayer
<michael@niedermayer.cc> wrote:

> On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> 
>>  On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
>>  <michael@niedermayer.cc> wrote: 
>>  
>>>   On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer
>>>   wrote: 
>>>   
>>>>    Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
>>>>    Bultje wrote: 
>>>>    
>>>>>     Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
>>>>>     timothee.informatique@regaud-chapuy.fr> wrote: 
>>>>>     
>>>>>>      Hello, I am interested in expanding ffmpeg's capabilities
>>>>>>      to extract low-level data from video codecs. Specifically,
>>>>>>      I'd like to implement functionality that would allow
>>>>>>      exporting frame data, macroblock information, quantization
>>>>>>      tables, and similar codec-specific elements to binary
>>>>>>      files for further analysis. After searching through the
>>>>>>      documentation and existing features, I haven't found
>>>>>>      similar functionality, though I may have missed something.
>>>>>>      Has this been implemented before, or are there related
>>>>>>      features I should examine?
>>>>>     
>>>>>      Some older codecs implement minor variants for this, e.g.
>>>>>     grep for AV_FRAME_DATA_MOTION_VECTORS, which attaches a
>>>>>     frame's motion vectors to the picture data. I believe
>>>>>     there's an example app and possibly a filter to overlay MVs
>>>>>     on top of the video frame based on this concept. You could
>>>>>     extend this to cover other (macro)block info. There used to
>>>>>     be a variant of this for quant-tables also but I can't find
>>>>>     it, maybe it was removed.
>>>>    
>>>>     For motion vectors: ./ffplay -flags2 +export_mvs -i
>>>>    matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
>>>>    segmentation and type vissualization + also motion vectors:
>>>>    ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
>>>>    codecview=mv=pf+bf+bb For QP vissualization + also motion
>>>>    vectors: ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
>>>>    codecview=mv=pf+bf+bb For qp values dumped on the console
>>>>    ./ffplay -debug qp -i matrixbench_mpeg2.mpg
>>>   
>>>    And this can easily be extended to other codecs, ATM it should
>>>   work with all 16x16 MB based codecs like
>>>   msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization
>>>   need codecview to be extended or versions around 3.4 which
>>>   implemented it differently Implementing vissualization as done
>>>   currently with sidedata and codecview is simple and efficient.
>>>   It also would allow exporting the data to json by writing a
>>>   codec2json filter in place of codecview Also all decoders
>>>   already have all this data parsed and available so its simpler
>>>   than trying to do it in a decoder independant way I would thus
>>>   suggest implementations of this for modern codecs to follow the
>>>   same path as the existing code. thx
>>  
>>   Thanks for the helpful pointers! I will work on the codec2json
>>  filter. Looking at the code, I see where I can access sidedata but
>>  extracting qb table seems to fail. (in codecview.c l.233:
>>  ff_qp_table_extract() return 0 and qp_table is empty) (I use
>>  ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1
>>  output.mp4 -y) Is is qp extraction not implemented yet? Or is it
>>  because I have h264 video? If it's not implemented, I'm curious
>>  why there’s already code that appears to handle it.
> 
>  look at ff_print_debug_info2() theres probably something missing
> The QP and MB type code was changed from being inside arrays of
> pictures to sidedata. Something likely was lost/forgotten in the
> process -debug qp works with h264 so likely teh export into sidedata
> is not fully implemented

Yes, this is exactly what is happening.

After some research, I have found: `// FIXME qscale / qp ... stuff` in
libavcodec/h264_slice.c, line 1871

and also in libavfilter/qp_table.h: 
34 /** 
35 * Normalize the qscale factor 
36 * FIXME Add support for other values of enum AVVideoEncParamsType 
37 * besides AV_VIDEO_ENC_PARAMS_MPEG2. 
38 */ 
39 static inline int ff_norm_qscale(int qscale, enum
AVVideoEncParamsType type) 
40 { 
41     switch (type) { 
42        case AV_VIDEO_ENC_PARAMS_MPEG2: return qscale >> 1; 
43     } 
44     return qscale; 
45 } 

So if I understand correctly, QP tables are not attached to the frames
except for MPEG2. This has been left as a FIXME for other codecs. 
Should I extend it to work for H.264, and maybe later for other
codecs, thus removing the first FIXME and partially fixing the second?
Or is this too specific and too complicated to maintain if it needs to
be added for every codec?

Thx,

Timothée
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ronald S. Bultje]

Hi,

On Mon, May 26, 2025 at 2:09 PM Timothée <
timothee.informatique@regaud-chapuy.fr> wrote:

> On 2025-05-23T13:55:57.000+02:00, Michael Niedermayer
> <michael@niedermayer.cc> wrote:
>
> > On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote:
> >
> >>  On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer
> >>  <michael@niedermayer.cc> wrote:
> >>
> >>>   On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer
> >>>   wrote:
> >>>
> >>>>    Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S.
> >>>>    Bultje wrote:
> >>>>
> >>>>>     Hi, On Wed, May 21, 2025 at 9:34 AM Timothée <
> >>>>>     timothee.informatique@regaud-chapuy.fr> wrote:
> >>>>>
> >>>>>>      Hello, I am interested in expanding ffmpeg's capabilities
> >>>>>>      to extract low-level data from video codecs. Specifically,
> >>>>>>      I'd like to implement functionality that would allow
> >>>>>>      exporting frame data, macroblock information, quantization
> >>>>>>      tables, and similar codec-specific elements to binary
> >>>>>>      files for further analysis. After searching through the
> >>>>>>      documentation and existing features, I haven't found
> >>>>>>      similar functionality, though I may have missed something.
> >>>>>>      Has this been implemented before, or are there related
> >>>>>>      features I should examine?
> >>>>>
> >>>>>      Some older codecs implement minor variants for this, e.g.
> >>>>>     grep for AV_FRAME_DATA_MOTION_VECTORS, which attaches a
> >>>>>     frame's motion vectors to the picture data. I believe
> >>>>>     there's an example app and possibly a filter to overlay MVs
> >>>>>     on top of the video frame based on this concept. You could
> >>>>>     extend this to cover other (macro)block info. There used to
> >>>>>     be a variant of this for quant-tables also but I can't find
> >>>>>     it, maybe it was removed.
> >>>>
> >>>>     For motion vectors: ./ffplay -flags2 +export_mvs -i
> >>>>    matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock
> >>>>    segmentation and type vissualization + also motion vectors:
> >>>>    ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf
> >>>>    codecview=mv=pf+bf+bb For QP vissualization + also motion
> >>>>    vectors: ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf
> >>>>    codecview=mv=pf+bf+bb For qp values dumped on the console
> >>>>    ./ffplay -debug qp -i matrixbench_mpeg2.mpg
> >>>
> >>>    And this can easily be extended to other codecs, ATM it should
> >>>   work with all 16x16 MB based codecs like
> >>>   msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization
> >>>   need codecview to be extended or versions around 3.4 which
> >>>   implemented it differently Implementing vissualization as done
> >>>   currently with sidedata and codecview is simple and efficient.
> >>>   It also would allow exporting the data to json by writing a
> >>>   codec2json filter in place of codecview Also all decoders
> >>>   already have all this data parsed and available so its simpler
> >>>   than trying to do it in a decoder independant way I would thus
> >>>   suggest implementations of this for modern codecs to follow the
> >>>   same path as the existing code. thx
> >>
> >>   Thanks for the helpful pointers! I will work on the codec2json
> >>  filter. Looking at the code, I see where I can access sidedata but
> >>  extracting qb table seems to fail. (in codecview.c l.233:
> >>  ff_qp_table_extract() return 0 and qp_table is empty) (I use
> >>  ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1
> >>  output.mp4 -y) Is is qp extraction not implemented yet? Or is it
> >>  because I have h264 video? If it's not implemented, I'm curious
> >>  why there’s already code that appears to handle it.
> >
> >  look at ff_print_debug_info2() theres probably something missing
> > The QP and MB type code was changed from being inside arrays of
> > pictures to sidedata. Something likely was lost/forgotten in the
> > process -debug qp works with h264 so likely teh export into sidedata
> > is not fully implemented
>
> Yes, this is exactly what is happening.
>
> After some research, I have found: `// FIXME qscale / qp ... stuff` in
> libavcodec/h264_slice.c, line 1871
>
> and also in libavfilter/qp_table.h:
> 34 /**
> 35 * Normalize the qscale factor
> 36 * FIXME Add support for other values of enum AVVideoEncParamsType
> 37 * besides AV_VIDEO_ENC_PARAMS_MPEG2.
> 38 */
> 39 static inline int ff_norm_qscale(int qscale, enum
> AVVideoEncParamsType type)
> 40 {
> 41     switch (type) {
> 42        case AV_VIDEO_ENC_PARAMS_MPEG2: return qscale >> 1;
> 43     }
> 44     return qscale;
> 45 }
>
> So if I understand correctly, QP tables are not attached to the frames
> except for MPEG2. This has been left as a FIXME for other codecs.
> Should I extend it to work for H.264, and maybe later for other
> codecs, thus removing the first FIXME and partially fixing the second?
> Or is this too specific and too complicated to maintain if it needs to
> be added for every codec?


I think it's fine to add it one codec at a time.

Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Tomas Härdin]

ons 2025-05-21 klockan 15:34 +0200 skrev Timothée:
> Hello,
> 
> I am interested in expanding ffmpeg's capabilities to extract
> low-level data from video codecs. Specifically, I'd like to implement
> functionality that would allow exporting frame data, macroblock
> information, quantization tables, and similar codec-specific elements
> to binary files for further analysis.

So you want to convert data from one binary format to another? A text
format seems more sensible if you're going through that effort

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".