[FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Soft Works]

Hello everybody,


working on the subject of writing out filtergraph information obviously implies the goal of being able to visualize that data in some way. While I do have something for long, it's tailored for specific workflows and is hardly useful for most.
Anyway, it shouldn't be required to use once another software for visualize the output. This decimates the usability for any such feature by far.
Few days ago, I was thinking about ways for making this feature really useful for everybody without needing to jump through any extra hoops. We have that range of writers (now "text formatters"), and so I wondered whether there isn't some text format that is meant to directly represent a graph for visualization, which finally reminded me about


MERMAID

Had used it just twice or thrice on GitHub for simple things, and as I found out later, it's by far less capable than I would have expected (more than a decade of age, current version 11.x) --- but: it has a number of unique selling points:

- The format is simple and transparent
- It's hand-editable and easily readable
- There's quite an ecosystem around it with many integrations and tools available
  The #1 for me though is:
- It goes hand-in-hand with Markdown
  => almost everywhere Markdown is supported, Mermaid diagrams are supported as well

Practically, this means that you can paste a Mermaid diagram into any message field on GitHub (for example; forgejo supports it as well as many others) - and the diagram is rendered automatically with a kind of interactive viewer.
Enough reason to give it a try and I also had some quick success. Writing that format fits rather well into the logic of the existing text formatters without major changes.
That way I ended up with a new 


avtextformatter_mermaid ("mermaidwriter" in old terminology)

Most of the mentioned deficiencies could be remedied by CSS styling (even though it's been really tedious). 
Now well - when viewing filtergraphs, you often want to know details about the input and output streams. For that other tool I was parsing out this information from the ffmpeg log lines describing the streams on startup. But this work is happening right at the source of information, so I added input/output files and streams to the filtergraph printing as well (just as much detail as needed for graphing).
The unthankful effect: by that, it became apparent that something else is missing: encoders and decoders - added those as well and eventually got reasonable results.
Remained one last part:


Usability

Of course, it's still not a great experience when you need to copy/paste the generated diagram code in some web portal. To solve that, I've created once another textformatter(writer): mermaidhtml.
It's just an extension to the Mermaid formatter which wraps the diagram as an html file for local viewing (yet not fully offline).
The final idea would be to simplify usage as much as possible, so that all you need to do is adding a small option to the Ffmpeg command line, like -sg ('show graph') which would cause Ffmpeg to create such mermaid html file on exit and automatically launches the browser (if any) to view that file. 
While working or trying a range of commands, after another, this will also create a browser tab for each run, allowing for easy comparison between recent commands.


Examples

I've created a Gist with some examples of the output here:

https://gist.github.com/softworkz/a196b2d0e9e2df49f766abd92f508551

(also includes a zip with html file examples)



Questions

I'm curious what you think about it!

- What's good, what's bad, what should be changed/improved?

- What about the displayed information, should something be added?

- I'm folding out the buffer source/sink filters to simplify the view, is that ok?
  Should the TRIM filters be excluded as well?

- Since it's no longer just about filter graphs - what do you think about the term
  "Ffmpeg Execution Graph"?
  (other ideas welcome)

- Does anybody have some complex command lines for me to test?
  (no need to include media files, I can try to replicate 
  something similar)


Thanks,
sw







_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Diederick C. Niehorster]

On Tue, Mar 18, 2025 at 3:32 AM Soft Works
<softworkz-at-hotmail.com@ffmpeg.org> wrote:
>
> Examples
>
> I've created a Gist with some examples of the output here:
>
> https://gist.github.com/softworkz/a196b2d0e9e2df49f766abd92f508551
>
> (also includes a zip with html file examples)

Just speaking as a would be user of this feature: awesome!!
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Stefano Sabatini]

On date Tuesday 2025-03-18 02:32:13 +0000, Soft Works wrote:
> Hello everybody,
>  
> working on the subject of writing out filtergraph information
> obviously implies the goal of being able to visualize that data in
> some way. While I do have something for long, it's tailored for
> specific workflows and is hardly useful for most.
>
> Anyway, it shouldn't be required to use once another software for
> visualize the output. This decimates the usability for any such
> feature by far.  Few days ago, I was thinking about ways for making
> this feature really useful for everybody without needing to jump
> through any extra hoops. We have that range of writers (now "text
> formatters"), and so I wondered whether there isn't some text format
> that is meant to directly represent a graph for visualization
[...]
> Examples
> 
> I've created a Gist with some examples of the output here:
> 
> https://gist.github.com/softworkz/a196b2d0e9e2df49f766abd92f508551
> 
> (also includes a zip with html file examples) 
> Questions
> 
> I'm curious what you think about it!
> 
> - What's good, what's bad, what should be changed/improved?
> 
> - What about the displayed information, should something be added?
> 
> - I'm folding out the buffer source/sink filters to simplify the view, is that ok?
>   Should the TRIM filters be excluded as well?
> 
> - Since it's no longer just about filter graphs - what do you think about the term
>   "Ffmpeg Execution Graph"?
>   (other ideas welcome)
> 
> - Does anybody have some complex command lines for me to test?
>   (no need to include media files, I can try to replicate 
>   something similar)
> 

Thanks for working on this, from the examples it looks pretty amazing.

What it's not clear to me is how this builds up on top of text
formatters, since they are meant to render a tree structure in a
generic way. From this you can have a description of a graph, but then
you need specialized ad-hoc logic to convert it to a graph format.

What am I missing?
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Stefano Sabatini
> Sent: Freitag, 21. März 2025 19:27
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> 
> On date Tuesday 2025-03-18 02:32:13 +0000, Soft Works wrote:
> > Hello everybody,
> >
> > working on the subject of writing out filtergraph information
> > obviously implies the goal of being able to visualize that data in
> > some way. While I do have something for long, it's tailored for
> > specific workflows and is hardly useful for most.
> >
> > Anyway, it shouldn't be required to use once another software for
> > visualize the output. This decimates the usability for any such
> > feature by far.  Few days ago, I was thinking about ways for making
> > this feature really useful for everybody without needing to jump
> > through any extra hoops. We have that range of writers (now "text
> > formatters"), and so I wondered whether there isn't some text format
> > that is meant to directly represent a graph for visualization
> [...]
> > Examples
> >
> > I've created a Gist with some examples of the output here:
> >
> > https://gist.github.com/softworkz/a196b2d0e9e2df49f766abd92f508551
> >
> > (also includes a zip with html file examples)
> > Questions
> >
> > I'm curious what you think about it!
> >
> > - What's good, what's bad, what should be changed/improved?
> >
> > - What about the displayed information, should something be added?
> >
> > - I'm folding out the buffer source/sink filters to simplify the view,
> is that ok?
> >   Should the TRIM filters be excluded as well?
> >
> > - Since it's no longer just about filter graphs - what do you think
> about the term
> >   "Ffmpeg Execution Graph"?
> >   (other ideas welcome)
> >
> > - Does anybody have some complex command lines for me to test?
> >   (no need to include media files, I can try to replicate
> >   something similar)
> >
> 
> Thanks for working on this, from the examples it looks pretty amazing.
> 
> What it's not clear to me is how this builds up on top of text
> formatters, since they are meant to render a tree structure in a
> generic way. From this you can have a description of a graph, but then
> you need specialized ad-hoc logic to convert it to a graph format.
> 
> What am I missing?


Hehe, that's been a bit tricky indeed, but more in terms of figuring out that (and how) it can fit into the writer/textformatter patterns, the result is rather simple and it requires only some moderate extensions to the writer/formatter APIs:


It is using a Mermaid Flowchart graph with only 3 kinds of elements:

- Shapes
  i.e. "boxes", can't be nested
- Subgraphs
  Container for shapes, can be nested
- Links
  connection from one shape to another


A simple graph with 2 shapes in a subgraph looks like this:

...
flowchart TD
 subgraph s1["Subgraph 1"]
        A("Shape A") --> B("Shape B")
  end
...

There's a link from A to B - but: it can also be written in a different way:

...
flowchart TD
 subgraph s1["Subgraph 1"]
        A("Shape A")
        B("Shape B")
  end

  A --> B
...

This does the same thing. The link definitions can be at a different place in the document - which is one of the key points to make it work.

Then there are three new section flags to denote subgraph, shapes and links and two fields in the section struct (source_id_key, dest_id_key). For link-producing sections they indicate the names of the keys to use for building a link. A section can also produce both - shapes and links when the flags are set accordingly.
I also use another (existing) mechanism, which is the data object that can be provided on section start, only that it's a (publicly) defined "SectionContext" struct, which is a somewhat alternative direction. I kept it for the moment as suggestion because I didn't know which ideas you have for the API to evolve, so that's just something for discussion.

Best,
sw

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Stefano Sabatini]

On date Friday 2025-03-21 20:11:29 +0000, Soft Works wrote:
[...]
> > What it's not clear to me is how this builds up on top of text
> > formatters, since they are meant to render a tree structure in a
> > generic way. From this you can have a description of a graph, but then
> > you need specialized ad-hoc logic to convert it to a graph format.
> > 
> > What am I missing?
> 
> 
> Hehe, that's been a bit tricky indeed, but more in terms of figuring
> out that (and how) it can fit into the writer/textformatter
> patterns, the result is rather simple and it requires only some
> moderate extensions to the writer/formatter APIs:
> 
> 
> It is using a Mermaid Flowchart graph with only 3 kinds of elements:
> 
> - Shapes
>   i.e. "boxes", can't be nested
> - Subgraphs
>   Container for shapes, can be nested
> - Links
>   connection from one shape to another
> 
> 
> A simple graph with 2 shapes in a subgraph looks like this:
> 
> ...
> flowchart TD
>  subgraph s1["Subgraph 1"]
>         A("Shape A") --> B("Shape B")
>   end
> ...
> 
> There's a link from A to B - but: it can also be written in a different way:
> 
> ...
> flowchart TD
>  subgraph s1["Subgraph 1"]
>         A("Shape A")
>         B("Shape B")
>   end
> 
>   A --> B
> ...
> 
> This does the same thing. The link definitions can be at a different
> place in the document - which is one of the key points to make it
> work.
> 
> Then there are three new section flags to denote subgraph, shapes
> and links and two fields in the section struct (source_id_key,
> dest_id_key). For link-producing sections they indicate the names of
> the keys to use for building a link. A section can also produce both
> - shapes and links when the flags are set accordingly.  I also use
> another (existing) mechanism, which is the data object that can be
> provided on section start, only that it's a (publicly) defined
> "SectionContext" struct, which is a somewhat alternative
> direction.

So basically a graph can be represented as a collection of subraphs,
each subgraphs containing shapes, and links.

> I kept it for the moment as suggestion because I didn't
> know which ideas you have for the API to evolve, so that's just
> something for discussion.

One of the possible uses is to expose the data printed by
filters. E.g. detection filters are printing the information either in
the stderr using custom formats, this should really be converted to
something easier to consume (whatever formats for which you don't
need a custom parser).

The metadata muxer also might benefit from using a text writer, to
avoid again the need for a custom parser, and probably there are more
use cases easy to spot.

So at some point we want to make this API accessbile from the
libraries, that is to move them into libavutil. Again, it's fine to
expose this at the tools level first so we can experiment and refine
the interface before moving to a stable one.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[softworkz .]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Stefano Sabatini
> Sent: Mittwoch, 26. März 2025 23:53
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> 
> On date Friday 2025-03-21 20:11:29 +0000, Soft Works wrote:
> [...]
> > > What it's not clear to me is how this builds up on top of text
> > > formatters, since they are meant to render a tree structure in a
> > > generic way. From this you can have a description of a graph, but
> then
> > > you need specialized ad-hoc logic to convert it to a graph format.
> > >
> > > What am I missing?
> >
> >
> > Hehe, that's been a bit tricky indeed, but more in terms of figuring
> > out that (and how) it can fit into the writer/textformatter
> > patterns, the result is rather simple and it requires only some
> > moderate extensions to the writer/formatter APIs:
> >
> >
> > It is using a Mermaid Flowchart graph with only 3 kinds of elements:
> >
> > - Shapes
> >   i.e. "boxes", can't be nested
> > - Subgraphs
> >   Container for shapes, can be nested
> > - Links
> >   connection from one shape to another
> >
> >
> > A simple graph with 2 shapes in a subgraph looks like this:
> >
> > ...
> > flowchart TD
> >  subgraph s1["Subgraph 1"]
> >         A("Shape A") --> B("Shape B")
> >   end
> > ...
> >
> > There's a link from A to B - but: it can also be written in a
> different way:
> >
> > ...
> > flowchart TD
> >  subgraph s1["Subgraph 1"]
> >         A("Shape A")
> >         B("Shape B")
> >   end
> >
> >   A --> B
> > ...
> >
> > This does the same thing. The link definitions can be at a different
> > place in the document - which is one of the key points to make it
> > work.
> >
> > Then there are three new section flags to denote subgraph, shapes
> > and links and two fields in the section struct (source_id_key,
> > dest_id_key). For link-producing sections they indicate the names of
> > the keys to use for building a link. A section can also produce both
> > - shapes and links when the flags are set accordingly.  I also use
> > another (existing) mechanism, which is the data object that can be
> > provided on section start, only that it's a (publicly) defined
> > "SectionContext" struct, which is a somewhat alternative
> > direction.
> 
> So basically a graph can be represented as a collection of subraphs,
> each subgraphs containing shapes, and links.

Yes, and link can exist at the top level (of nesting) and otherwise at any position in the document.


> > I kept it for the moment as suggestion because I didn't
> > know which ideas you have for the API to evolve, so that's just
> > something for discussion.
> 
> One of the possible uses is to expose the data printed by
> filters. E.g. detection filters are printing the information either in
> the stderr using custom formats, this should really be converted to
> something easier to consume (whatever formats for which you don't
> need a custom parser).
> 
> The metadata muxer also might benefit from using a text writer, to
> avoid again the need for a custom parser, and probably there are more
> use cases easy to spot.
> 
> So at some point we want to make this API accessbile from the
> libraries, that is to move them into libavutil. Again, it's fine to
> expose this at the tools level first so we can experiment and refine
> the interface before moving to a stable one.
> _______________________________________________

Yes, I see quite a number use cases as well.
It's still a work in progress, but I've set up a GitHub repo for documentation and as a place of interaction. It's not for ffmpeg code, but there are tests, examples and a documentation Wiki:

https://github.com/softworkz/ffmpeg_output_apis/wiki

(a number of things are still incomplete)


I'm also exploring a way for self-printing the data schema for each use case. On the repo Wiki there are Entity-Relationship diagrams for the FFprobe and the Graphs schema. I'll follow up with more details..

sw






_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Nicolas George]

Stefano Sabatini (HE12025-03-26):
> One of the possible uses is to expose the data printed by
> filters. E.g. detection filters are printing the information either in
> the stderr using custom formats, this should really be converted to
> something easier to consume (whatever formats for which you don't
> need a custom parser).
> 
> The metadata muxer also might benefit from using a text writer, to
> avoid again the need for a custom parser, and probably there are more
> use cases easy to spot.

For the record, I have had detailed plans for all this for years. But
since it is very related to strings, it requires AVWriter, and it
requires the abstraction and type description layer that I want to
introduce that starts with AVWriter. Therefore, it cannot go forward
unless FFmpeg has once more proper leadership able to greenlight it.

> So at some point we want to make this API accessbile from the
> libraries, that is to move them into libavutil. Again, it's fine to
> expose this at the tools level first so we can experiment and refine
> the interface before moving to a stable one.

I wild recommend using the fftools as a staging area.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[softworkz .]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Nicolas George
> Sent: Donnerstag, 27. März 2025 20:28
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> 
> Stefano Sabatini (HE12025-03-26):
> > One of the possible uses is to expose the data printed by
> > filters. E.g. detection filters are printing the information either in
> > the stderr using custom formats, this should really be converted to
> > something easier to consume (whatever formats for which you don't
> > need a custom parser).
> >
> > The metadata muxer also might benefit from using a text writer, to
> > avoid again the need for a custom parser, and probably there are more
> > use cases easy to spot.
> 
> For the record, I have had detailed plans for all this for years. But
> since it is very related to strings, it requires AVWriter, and it
> requires the abstraction and type description layer that I want to
> introduce that starts with AVWriter. 

Hi Nicolas,

after doing all this, I came to the conclusion that a better string API would really be useful for these kinds of things (albeit not a necessity).

I thought I might take another look, could you point me at the latest version of your work in this regard?

> > So at some point we want to make this API accessbile from the
> > libraries, that is to move them into libavutil. Again, it's fine to
> > expose this at the tools level first so we can experiment and refine
> > the interface before moving to a stable one.
> 
> I wild recommend using the fftools as a staging area.

Yup, that's what we're doing. There's quite a range of improvements I'm seeing which should be done before making anything public.

sw



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1140 bytes --]

Hi

On Thu, Mar 27, 2025 at 08:27:45PM +0100, Nicolas George wrote:
> Stefano Sabatini (HE12025-03-26):
> > One of the possible uses is to expose the data printed by
> > filters. E.g. detection filters are printing the information either in
> > the stderr using custom formats, this should really be converted to
> > something easier to consume (whatever formats for which you don't
> > need a custom parser).
> > 
> > The metadata muxer also might benefit from using a text writer, to
> > avoid again the need for a custom parser, and probably there are more
> > use cases easy to spot.
> 
> For the record, I have had detailed plans for all this for years. But
> since it is very related to strings, it requires AVWriter, and it
> requires the abstraction and type description layer that I want to
> introduce that starts with AVWriter. Therefore, it cannot go forward

can you repost the AVWriter patches ?

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are too smart to engage in politics are punished by being
governed by those who are dumber. -- Plato 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[softworkz .]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> softworkz .
> Sent: Donnerstag, 27. März 2025 00:26
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> 
> 
> 
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Stefano Sabatini
> > Sent: Mittwoch, 26. März 2025 23:53
> > To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> >

[..]

> > One of the possible uses is to expose the data printed by
> > filters. E.g. detection filters are printing the information either in
> > the stderr using custom formats, this should really be converted to
> > something easier to consume (whatever formats for which you don't
> > need a custom parser).
> >
> > The metadata muxer also might benefit from using a text writer, to
> > avoid again the need for a custom parser, and probably there are more
> > use cases easy to spot.
> >
> > So at some point we want to make this API accessbile from the
> > libraries, that is to move them into libavutil. Again, it's fine to
> > expose this at the tools level first so we can experiment and refine
> > the interface before moving to a stable one.
> > _______________________________________________
> 
> Yes, I see quite a number use cases as well.
> It's still a work in progress, but I've set up a GitHub repo for
> documentation and as a place of interaction. It's not for ffmpeg code,
> but there are tests, examples and a documentation Wiki:
> 
> https://github.com/softworkz/ffmpeg_output_apis/wiki
> 
> (a number of things are still incomplete)
> 
> 
> I'm also exploring a way for self-printing the data schema for each use
> case. On the repo Wiki there are Entity-Relationship diagrams for the
> FFprobe and the Graphs schema. I'll follow up with more details..

Hi Stefano,

I've got everything in shape now in this repo I've also completed done the write-up on my experiment regarding schema self-printing: https://github.com/softworkz/ffmpeg_output_apis/wiki/Schema-Self-Printing

I'd be curious what you think about it (of course others as well).

Best,
sw




_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Nicolas George]

Michael Niedermayer (HE12025-03-29):
> can you repost the AVWriter patches ?

Sure, but I need to find the latest branch, and it is proving a cause of
procrastination for me. In the meantime, let me explain anew the
principles behind it, after a few years I might be able to express them
better.

(I hope it means you are considering re-asserting your authority to
approve it on principle pending review of the code. I will not invest
more coding into it unless I know nobody has the authority to put it to
waste with “it does not belong”.)


1. Why we need it

We use a lot of text for small things. As a command-line tool, we use
text to communicate with users. Even as a library, when GUIs do not have
the proper widget they will use text, and we need to provide the
necessary support functions.

We should have a rule that for every type we add, we must have a
corresponding to-text API function. Furthermore, these functions should
all have the same type (up to the type of the opaque pointer), so that
we can store them in a structure that describes the type along with
other information (from string, ref/unref, etc.).

We also need text for inter-process communication, like exporting
statistics from filters.

Some of these to-text conversions happen once per frame. That means they
should happen without dynamic allocation. Quite complex code is
warranted to prevent dynamic allocations once per frame: see the frame
and buffer pools too. But some of these to-text conversion have an
unbounded output. That means the API needs to be able to handle long
output.

BPrint meets the criteria: BPrint is as fast as a buffer on the stack
but can handle strings of arbitrary length.

Unfortunately, BPrint is ugly. The name is ugly, and more importantly,
it is inflexible, it works only with flat buffers in memory.


2. How AVWriter does it

AVWriter is an attempt — successful, in my opinion — at keeping what is
good in BPrint and fixing what is ugly, in particular by giving it
features similar to AVIO.

The design principle is the same as AVIO: a structure with callbacks
that perform the low-level operations.

The difference with AVIO is that effort were made to keep the structures
small and allocatable by the caller.

Also, the back-end is allowed to provide only a few methods: printf-like
or write-like or get_buffer-like, and the framework will make sure to
use one that is available. That means quite a bit of code to handle
doing a printf() on a back-end that only has get_buffer, but it is code
isolated within a single file and with ample FATE coverage.

One of the tricks I used to avoid dynamic allocations is to store the
structure of methods and the structure with an instance as a structure
with two pointer elements passed by value.

Another trick I used to avoid dynamic allocations is to store the size
of a structure in it. That way, if a caller is compiled with an old
version of the library, it stores a smaller size than the current one,
and the new version of the library can test and avoid accessing the new
fields. That lets the caller allocate the structure while keeping the
ability to add fields to the structure.

Apart from that, is is rather straightforward. The default AVWriter is a
simple wrapper around BPrint, but there are also quite a few other
built-in ones: wrapper around stdio and av_log (without storing the
whole string), wrapper around a fixed-size buffer. AVwriter can also act
as a filter: get an AVWriter, you can create a new one that will encode
to base64 or HTML entities on the fly.


3. Where it will go

The trick passing a struct of methods and the object as a structure with
two members passed by value is meant to become a very light-weight
object system. Thus if ctx is a pointer to a structure of type T, we can
define serialize_t as a structure with functions and consider
{ serialize_t, ctx } as an object of the class “serializable”. But we
can also make it an object of the class “refcounted” by pairing it with
another structure.

The object system, including casts from one class to another, can be
de-centralized, new classes ca be defined anywhere in the code without a
central registry. That will be useful to enhance several parts of our
code:

- Side data would not need to be added in libavutil. A pair of
  demuxer-muxer in libavformat can define a new type of side data by
  defining the methods to handle it.

- Filters with unusual parsing needs can define a new type of option and
  plug it into the standard options parsing system. It is extremely
  useful if the needs of the filter are too specific to add a type in
  libavutil but too far from something existing to be able to use it
  conveniently.

- Pluging new types into the options system will automatically fix our
  “escaping hell” issue where users sometimes need 42 levels of \ in
  front of a : to achieve their goals.

- We can implement a full-featured av_printf() function that serializes
  on the fly:

  av_printf(out, "Stream %d at %d Hz, %@ with layout %@\n",
      st->id, st->sample_rate,
      avany_sample_fmt(st->sample_fmt),
      avany_layout(st->channel_layout));

With this, it becomes possible to implement new features that rely a lot
on text. That includes:

- Better error reporting. Instead the error message going to the log,
  where nobody will see it if it is a GUI, instead of the application
  having to install a log handler, with the responsibility to assemble
  lines together and filter out unrelated messages that just happened to
  arrive at the same time, the error message is stored in the relevant
  context and can be retrieved cleanly later.

- Built-in comprehensive documentation. You can do
  av_get_documentation(ctx, …) and get explanations on how to use it.
  Depending on the flags given, the returned documentation can be terse
  and suitable for a tooltip or comprehensive including hyperlinks and
  reference for the syntax of the options.

- Serialization of high-level data structures into various standard
  formats: JSON, XML, etc., precisely what started this thread. That
  allows to factor the writers in ffprobe and that gives us means to
  exfiltrate statistics from filters.


4. Why it must be in FFmpeg

I acknowledge that nothing I described is even remotely specific to
FFmpeg. I could be arguing for AVWriter, a light-weight object system,
built-in documentation, etc., in a project that does something entirely
else.

That means in principle all this could go into a separate library, and
some people have argued for it. In principle it could, but in practice
it does not make sense for multiple reasons, but the main reason is
this:

As a matter of principle, we do not demand that users install
third-parties libraries in order to build FFmpeg. We rely on system
libraries, and we can rely on external libraries for optional features,
even very important ones (x264…), but if something is needed to run
FFmpeg at all then it must be in our own code base so that building from
sources will just work. Strings and error reporting cannot be optional,
they must be in FFmpeg.


I will come back later with the code as it was last time.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[softworkz .]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Nicolas George
> Sent: Freitag, 11. April 2025 19:09
> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> 
> Michael Niedermayer (HE12025-03-29):
> > can you repost the AVWriter patches ?
> 
> Sure, but I need to find the latest branch, and it is proving a cause of
> procrastination for me. In the meantime, let me explain anew the
> principles behind it, after a few years I might be able to express them
> better.
> 
> (I hope it means you are considering re-asserting your authority to
> approve it on principle pending review of the code. I will not invest
> more coding into it unless I know nobody has the authority to put it to
> waste with “it does not belong”.)
> 
> 
> 1. Why we need it
> 
> We use a lot of text for small things. As a command-line tool, we use
> text to communicate with users. Even as a library, when GUIs do not have
> the proper widget they will use text, and we need to provide the
> necessary support functions.
> 
> We should have a rule that for every type we add, we must have a
> corresponding to-text API function. Furthermore, these functions should
> all have the same type (up to the type of the opaque pointer), so that
> we can store them in a structure that describes the type along with
> other information (from string, ref/unref, etc.).
> 
> We also need text for inter-process communication, like exporting
> statistics from filters.
> 
> Some of these to-text conversions happen once per frame. That means they
> should happen without dynamic allocation. Quite complex code is
> warranted to prevent dynamic allocations once per frame: see the frame
> and buffer pools too. But some of these to-text conversion have an
> unbounded output. That means the API needs to be able to handle long
> output.
> 
> BPrint meets the criteria: BPrint is as fast as a buffer on the stack
> but can handle strings of arbitrary length.
> 
> Unfortunately, BPrint is ugly. The name is ugly, and more importantly,
> it is inflexible, it works only with flat buffers in memory.
> 
> 
> 2. How AVWriter does it
> 
> AVWriter is an attempt — successful, in my opinion — at keeping what is
> good in BPrint and fixing what is ugly, in particular by giving it
> features similar to AVIO.
> 
> The design principle is the same as AVIO: a structure with callbacks
> that perform the low-level operations.
> 
> The difference with AVIO is that effort were made to keep the structures
> small and allocatable by the caller.
> 
> Also, the back-end is allowed to provide only a few methods: printf-like
> or write-like or get_buffer-like, and the framework will make sure to
> use one that is available. That means quite a bit of code to handle
> doing a printf() on a back-end that only has get_buffer, but it is code
> isolated within a single file and with ample FATE coverage.
> 
> One of the tricks I used to avoid dynamic allocations is to store the
> structure of methods and the structure with an instance as a structure
> with two pointer elements passed by value.
> 
> Another trick I used to avoid dynamic allocations is to store the size
> of a structure in it. That way, if a caller is compiled with an old
> version of the library, it stores a smaller size than the current one,
> and the new version of the library can test and avoid accessing the new
> fields. That lets the caller allocate the structure while keeping the
> ability to add fields to the structure.
> 
> Apart from that, is is rather straightforward. The default AVWriter is a
> simple wrapper around BPrint, but there are also quite a few other
> built-in ones: wrapper around stdio and av_log (without storing the
> whole string), wrapper around a fixed-size buffer. AVwriter can also act
> as a filter: get an AVWriter, you can create a new one that will encode
> to base64 or HTML entities on the fly.
> 
> 
> 3. Where it will go
> 
> The trick passing a struct of methods and the object as a structure with
> two members passed by value is meant to become a very light-weight
> object system. Thus if ctx is a pointer to a structure of type T, we can
> define serialize_t as a structure with functions and consider
> { serialize_t, ctx } as an object of the class “serializable”. But we
> can also make it an object of the class “refcounted” by pairing it with
> another structure.
> 
> The object system, including casts from one class to another, can be
> de-centralized, new classes ca be defined anywhere in the code without a
> central registry. That will be useful to enhance several parts of our
> code:
> 
> - Side data would not need to be added in libavutil. A pair of
>   demuxer-muxer in libavformat can define a new type of side data by
>   defining the methods to handle it.
> 
> - Filters with unusual parsing needs can define a new type of option and
>   plug it into the standard options parsing system. It is extremely
>   useful if the needs of the filter are too specific to add a type in
>   libavutil but too far from something existing to be able to use it
>   conveniently.
> 
> - Pluging new types into the options system will automatically fix our
>   “escaping hell” issue where users sometimes need 42 levels of \ in
>   front of a : to achieve their goals.
> 
> - We can implement a full-featured av_printf() function that serializes
>   on the fly:
> 
>   av_printf(out, "Stream %d at %d Hz, %@ with layout %@\n",
>       st->id, st->sample_rate,
>       avany_sample_fmt(st->sample_fmt),
>       avany_layout(st->channel_layout));
> 
> With this, it becomes possible to implement new features that rely a lot
> on text. That includes:
> 
> - Better error reporting. Instead the error message going to the log,
>   where nobody will see it if it is a GUI, instead of the application
>   having to install a log handler, with the responsibility to assemble
>   lines together and filter out unrelated messages that just happened to
>   arrive at the same time, the error message is stored in the relevant
>   context and can be retrieved cleanly later.
> 
> - Built-in comprehensive documentation. You can do
>   av_get_documentation(ctx, …) and get explanations on how to use it.
>   Depending on the flags given, the returned documentation can be terse
>   and suitable for a tooltip or comprehensive including hyperlinks and
>   reference for the syntax of the options.
> 
> - Serialization of high-level data structures into various standard
>   formats: JSON, XML, etc., precisely what started this thread. That
>   allows to factor the writers in ffprobe and that gives us means to
>   exfiltrate statistics from filters.
> 
> 
> 4. Why it must be in FFmpeg
> 
> I acknowledge that nothing I described is even remotely specific to
> FFmpeg. I could be arguing for AVWriter, a light-weight object system,
> built-in documentation, etc., in a project that does something entirely
> else.
> 
> That means in principle all this could go into a separate library, and
> some people have argued for it. In principle it could, but in practice
> it does not make sense for multiple reasons, but the main reason is
> this:
> 
> As a matter of principle, we do not demand that users install
> third-parties libraries in order to build FFmpeg. We rely on system
> libraries, and we can rely on external libraries for optional features,
> even very important ones (x264…), but if something is needed to run
> FFmpeg at all then it must be in our own code base so that building from
> sources will just work. Strings and error reporting cannot be optional,
> they must be in FFmpeg.
> 
> 
> I will come back later with the code as it was last time.
> 
> Regards,
> 
> --
>   Nicolas George
> _______________________________________________


HI Nicolas,


> - Serialization of high-level data structures into various standard
>   formats: JSON, XML, etc., precisely what started this thread. That
>   allows to factor the writers in ffprobe and that gives us means to
>   exfiltrate statistics from filters.

just FYI - there aren't any writers in FFprobe anymore. Formatting and output are separate things now.

I had asked you about the AVWriter code - even before Michael did - out of interest, for checking out how much the implementation might benefit from AVWriter because I can remind just some fragments about it.
I don't even remember the "object system" aspect - which I find even more interesting, yet the presentation is a bit confusing. I believe that treating them as two separate things would make it a better and easier sell. 

I'm still not sure about the relation between AVWriter and "object-system". Is it that AVWriter is implemented according to that "object-system" proposal, i.e. being like all "objects" are supposed to be implemented in the future?
Or is AVWriter just using the same technique in a similar way but for a different purpose?


I notice how much frustration this subject appears to have caused for you, but you should not project that frustration on the wrong targets.
Like I said before - I don't want to cut you off from the work in the area of text formatting and writing and you are welcome to collaborate - if you want that.
Weirdly enough, I like many of the examples you are giving above. I can say that because I'm not cultivating my own frustrations. If you'd only manage to pretend being a friendly person, that should already suffice to make some good progress. Seriously.

Best,
sw








_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 8163 bytes --]

Hi Nicolas

On Fri, Apr 11, 2025 at 07:09:09PM +0200, Nicolas George wrote:
> Michael Niedermayer (HE12025-03-29):
> > can you repost the AVWriter patches ?
> 
> Sure, but I need to find the latest branch, and it is proving a cause of
> procrastination for me. In the meantime, let me explain anew the
> principles behind it, after a few years I might be able to express them
> better.
> 

> (I hope it means you are considering re-asserting your authority to
> approve it on principle pending review of the code. I will not invest
> more coding into it unless I know nobody has the authority to put it to
> waste with “it does not belong”.)

FFmpeg belongs to the People, to the Community. Its our project not just
my project.

This doesnt mean anyone should be able to block anything, this worked
when the Community was smaller.

In this specific case, I think the people who previously objected
are no longer active in FFmpeg. So simply reposting the patchset
seemed like a good idea to me.
Maybe noone has objections, maybe someone has valid and good suggestions
maybe someone has valid objections, that require technical changes.
Maybe we will have an objection by one person but a clear consensus
that most people are in favor of this being added to FFmpeg.

OTOH, if there are objections from multiple active developers who would
be affected by the patchset, then this is something we need to accept
and then the patchset would not be able to be applied as long as these
objections are standing.
And even if there is just one active developer objecting, we need to
very carefully listen and understand what he is saying before concluding
that theres a consensus with one objection. Because maybe we can learn
from the objection and improve the code for example


>
> 
> 1. Why we need it
> 
> We use a lot of text for small things. As a command-line tool, we use
> text to communicate with users. Even as a library, when GUIs do not have
> the proper widget they will use text, and we need to provide the
> necessary support functions.
> 
> We should have a rule that for every type we add, we must have a
> corresponding to-text API function. Furthermore, these functions should
> all have the same type (up to the type of the opaque pointer), so that
> we can store them in a structure that describes the type along with
> other information (from string, ref/unref, etc.).
> 
> We also need text for inter-process communication, like exporting
> statistics from filters.
> 
> Some of these to-text conversions happen once per frame. That means they
> should happen without dynamic allocation. Quite complex code is
> warranted to prevent dynamic allocations once per frame: see the frame
> and buffer pools too. But some of these to-text conversion have an
> unbounded output. That means the API needs to be able to handle long
> output.
> 
> BPrint meets the criteria: BPrint is as fast as a buffer on the stack
> but can handle strings of arbitrary length.
> 
> Unfortunately, BPrint is ugly. The name is ugly, and more importantly,
> it is inflexible, it works only with flat buffers in memory.
> 
> 
> 2. How AVWriter does it
> 
> AVWriter is an attempt — successful, in my opinion — at keeping what is
> good in BPrint and fixing what is ugly, in particular by giving it
> features similar to AVIO.
> 
> The design principle is the same as AVIO: a structure with callbacks
> that perform the low-level operations.
> 
> The difference with AVIO is that effort were made to keep the structures
> small and allocatable by the caller.
> 
> Also, the back-end is allowed to provide only a few methods: printf-like
> or write-like or get_buffer-like, and the framework will make sure to
> use one that is available. That means quite a bit of code to handle
> doing a printf() on a back-end that only has get_buffer, but it is code
> isolated within a single file and with ample FATE coverage.
> 
> One of the tricks I used to avoid dynamic allocations is to store the
> structure of methods and the structure with an instance as a structure
> with two pointer elements passed by value.
> 
> Another trick I used to avoid dynamic allocations is to store the size
> of a structure in it. That way, if a caller is compiled with an old
> version of the library, it stores a smaller size than the current one,
> and the new version of the library can test and avoid accessing the new
> fields. That lets the caller allocate the structure while keeping the
> ability to add fields to the structure.
> 
> Apart from that, is is rather straightforward. The default AVWriter is a
> simple wrapper around BPrint, but there are also quite a few other
> built-in ones: wrapper around stdio and av_log (without storing the
> whole string), wrapper around a fixed-size buffer. AVwriter can also act
> as a filter: get an AVWriter, you can create a new one that will encode
> to base64 or HTML entities on the fly.
> 
> 
> 3. Where it will go
> 
> The trick passing a struct of methods and the object as a structure with
> two members passed by value is meant to become a very light-weight
> object system. Thus if ctx is a pointer to a structure of type T, we can
> define serialize_t as a structure with functions and consider
> { serialize_t, ctx } as an object of the class “serializable”. But we
> can also make it an object of the class “refcounted” by pairing it with
> another structure.
> 
> The object system, including casts from one class to another, can be
> de-centralized, new classes ca be defined anywhere in the code without a
> central registry. That will be useful to enhance several parts of our
> code:
> 
> - Side data would not need to be added in libavutil. A pair of
>   demuxer-muxer in libavformat can define a new type of side data by
>   defining the methods to handle it.
> 
> - Filters with unusual parsing needs can define a new type of option and
>   plug it into the standard options parsing system. It is extremely
>   useful if the needs of the filter are too specific to add a type in
>   libavutil but too far from something existing to be able to use it
>   conveniently.
> 
> - Pluging new types into the options system will automatically fix our
>   “escaping hell” issue where users sometimes need 42 levels of \ in
>   front of a : to achieve their goals.
> 
> - We can implement a full-featured av_printf() function that serializes
>   on the fly:
> 
>   av_printf(out, "Stream %d at %d Hz, %@ with layout %@\n",
>       st->id, st->sample_rate,
>       avany_sample_fmt(st->sample_fmt),
>       avany_layout(st->channel_layout));
> 
> With this, it becomes possible to implement new features that rely a lot
> on text. That includes:
> 
> - Better error reporting. Instead the error message going to the log,
>   where nobody will see it if it is a GUI, instead of the application
>   having to install a log handler, with the responsibility to assemble
>   lines together and filter out unrelated messages that just happened to
>   arrive at the same time, the error message is stored in the relevant
>   context and can be retrieved cleanly later.
> 
> - Built-in comprehensive documentation. You can do
>   av_get_documentation(ctx, …) and get explanations on how to use it.
>   Depending on the flags given, the returned documentation can be terse
>   and suitable for a tooltip or comprehensive including hyperlinks and
>   reference for the syntax of the options.
> 
> - Serialization of high-level data structures into various standard
>   formats: JSON, XML, etc., precisely what started this thread. That
>   allows to factor the writers in ffprobe and that gives us means to
>   exfiltrate statistics from filters.

what you describe, really sounds cool

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[James Almer]

[-- Attachment #1.1.1: Type: text/plain, Size: 400 bytes --]

On 4/13/2025 12:13 PM, Michael Niedermayer wrote:
> FFmpeg belongs to the People, to the Community. Its our project not just
> my project.

> In this specific case, I think the people who previously objected
> are no longer active in FFmpeg.

These two statements clash with each other, for reasons you're well 
aware of. If you were serious about the first, the second would not be true.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".