[FFmpeg-devel] API enhancements / broken promises

[FFmpeg-devel] API enhancements / broken promises

[Nicolas George]

[-- Attachment #1.1: Type: text/plain, Size: 4081 bytes --]

Hi.

Over the years, I have promised quite a few enhancement to FFmpeg's API,
some of them connecting to user interface: stored error messages instead
of just logging, universal serialization of objects into various formats
(JSON, XML, etc.), a way to exfiltrate data from filters and other
components, better options parsing avoiding multiple levels of escaping,
asynchronous interface for protocols and later formats, avoiding global
state including a more reliable control of allowed components, and I
might be forgetting a few of them.

I will not be able to make good on these promises, mostly for no fault
of mine.

I have detailed plans on how to achieve any of these goals; I would not
have proposed otherwise. I can explain them if somebody is interested.

A lot of these projects require a good string API. Unfortunately, my
proposal for that is being blocked by a member of the tech committee
who, by their own admission, almost never deals with strings and never
used BPrint, and whose arguments amount to “I am not convinced we need
it”, and the others members of the committee do not care enough to
override them.

The other projects require other kinds of preliminary framework APIs
equally at risk of getting blocked for the same flimsy reasons, and I am
not so fond of wasting my time as to start implementing them in these
circumstances.

I have the hubris to believe that if I am not good at micro-optimizing
code and implementing compression standards, my skills as an architect
have proved useful to the project in the past and can be useful in the
future too, that they cover an area that complements the skills of
others.

(Regarding past architecture changes, the refcounted packets system and
the send/receive libavcodec interface are sound architecture, but they
are also quite obvious, not requiring a lot of creativity. Also, I
suspect I could have avoided one level of annoying indirection on the
refcounted buffers had I had my word to say at the time.)

The first steps of all these projects are not fancy, they will not make
the code spectacularly simpler. On the other hand, I have made sure they
are unobtrusive: a pair of files for the implementation, a few examples
of where they already help a little, not 100+ patch series that have to
absolutely be applied at once and/or steps on everybody's toes. If they
happen to completely fail, we just remove them, no harm done.

But they are necessary to move forward.

Yet they are blocked.

I will not install you a jacuzzi if you do not trust my judgement as an
architect that you need proper water drainage first.

This project is dying, for lack of leadership and vision.

In fact, this project has been dying ever since we wondered how we
should change to entice the forkers back instead of wondering how the
forkers needed to change to be accepted back. As a result, we changed in
the very direction that caused the fork to die.

And due to that change, I am more and more losing interest in FFmpeg. I
will probably continue to enhance the framework of libavfilter at a
snail's pace, because at least the battles I have to lead there are not
uphill, but do not expect much more from me.

Unless there is a surge of interest from the others core developers and
it is made clear that you want to move forward towards a better API and
you trust my planning as long as the code itself is good and either
unobtrusive or really useful.

But I am not holding my breath. Fortunately, I have others projects of
my own, and I will have more time to invest into them. A lot of the
thinking I have done for FFmpeg I will be able to reuse for my own
benefit.

Which leads me to my last point: I have confirmation that my appetence
as a teacher will be starved in the near future, so if somebody is
looking for tutoring in low-level Unix/C system and network programming,
a collaboration could be greatly appreciated and mutually beneficial.

I think it is all I have to say for now.

Regards,

-- 
  Nicolas George

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Stefano Sabatini]

On date Monday 2022-08-15 18:47:34 +0200, Nicolas George wrote:
> Hi.
> 
> Over the years, I have promised quite a few enhancement to FFmpeg's API,
> some of them connecting to user interface: stored error messages instead
> of just logging, universal serialization of objects into various formats
> (JSON, XML, etc.), a way to exfiltrate data from filters and other
> components, better options parsing avoiding multiple levels of escaping,
> asynchronous interface for protocols and later formats, avoiding global
> state including a more reliable control of allowed components, and I
> might be forgetting a few of them.
> 
> I will not be able to make good on these promises, mostly for no fault
> of mine.
> 
> I have detailed plans on how to achieve any of these goals; I would not
> have proposed otherwise. I can explain them if somebody is interested.
> 
> A lot of these projects require a good string API. Unfortunately, my
> proposal for that is being blocked by a member of the tech committee
> who, by their own admission, almost never deals with strings and never
> used BPrint, and whose arguments amount to “I am not convinced we need
> it”, and the others members of the committee do not care enough to
> override them.

Just my two cents since I've admittedly not being active for the past
several years, but I appreciate your contributions over the years and
I like the direction you put the project on.

I commented on the string API (in fact I did with the idea to
kickstart a proper review process).

About the tech committee, I don't know how much this changed but as
far as there is a review going on and no documented technical
objections during review, there should be no reason to block a new
API.

[...]
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3873 bytes --]

Hi

On Mon, Aug 15, 2022 at 06:47:34PM +0200, Nicolas George wrote:
> Hi.
> 
> Over the years, I have promised quite a few enhancement to FFmpeg's API,
> some of them connecting to user interface: stored error messages instead
> of just logging, universal serialization of objects into various formats
> (JSON, XML, etc.), a way to exfiltrate data from filters and other
> components, better options parsing avoiding multiple levels of escaping,
> asynchronous interface for protocols and later formats, avoiding global
> state including a more reliable control of allowed components, and I
> might be forgetting a few of them.
> 
> I will not be able to make good on these promises, mostly for no fault
> of mine.
> 
> I have detailed plans on how to achieve any of these goals; I would not
> have proposed otherwise. I can explain them if somebody is interested.
> 
> A lot of these projects require a good string API. Unfortunately, my
> proposal for that is being blocked by a member of the tech committee
> who, by their own admission, almost never deals with strings and never
> used BPrint, and whose arguments amount to “I am not convinced we need
> it”, and the others members of the committee do not care enough to
> override them.

As i said elsewhere i think replacing BPrint by AVWriter is a good idea.
and also that the TC does to the best of my knowledge have no power in
this case. We first need some patch and disagreement the TC could vote
on. ATM there is no public disagreement on a patch i think

But i didnt hit reply to repeat that. Rather i wanted to comment on
XML and JSON and FFmpeg.

I saw on IRC this (authors removed because their names are not important here,
iam really replying to the statments not the people)
A>       but I, IMVHO, dont think this project should get a XML parser
B>      "nice and limited" XML parser sounds like all sorts of "fun"
C>      Yeah, xml/json/... parser in ffmpeg is just... no

Whats the purpose of FFmpeg ?
"A complete, cross-platform solution to record, convert and stream audio and video."
first actually i think that should be chnaged to something like this:
"A complete, cross-platform solution to your multmedia needs"
Because there are subtitles and many other things we do care about that this
misses 

Now to achieve this do we need xml and json ?
grep tells me we have 500 matches (not counting docs) for xml and almost 100
for json
Also for streaming and some cases filtering being able to serialize objects
would be useful. xml and json seem better choices than some ad-hoc format
So i would awnser the question do we need XML and JSON, with yes we live
in a world that uses XML and JSON so if we give the option to use it too
that makes it easier for others to interact.

now do we need our own implementation of it ? I dont know but we have
in almost all cases favored our native implementations when someone wrote
one. And libxml2 has had so many security issues that i think we should
at least consider replacing it.


The 2nd thing that comes to mind with "purpose of FFmpeg" is FF stands for
"Fast Forward". To move fast forward we should not lose developers because of
rather minor disagreements.
IMO if Nicolas wants to write and maintain AVWriter and a simple xml parser
(which i presume has the goal to parse the XML we would generate for 
serialization and in other places). I really think telling Nicolas "no" is 
a unwise choice. But if someone is against very basic xml or json parsers
please speak up now and here because its still better to say "no" now than
after nicolas did the work.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I know you won't believe me, but the highest form of Human Excellence is
to question oneself and others. -- Socrates

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Nicolas George]

[-- Attachment #1.1: Type: text/plain, Size: 4831 bytes --]

Hi.

Michael Niedermayer (12022-08-17):
> As i said elsewhere i think replacing BPrint by AVWriter is a good idea.
> and also that the TC does to the best of my knowledge have no power in
> this case. We first need some patch and disagreement the TC could vote
> on. ATM there is no public disagreement on a patch i think
> 
> But i didnt hit reply to repeat that.

Thank you and thank Stefano for your encouraging replies. With them and
the comments I had in private and/or earlier, I am beginning to feel
confident I can override the argument-less objection.

Note that I am not asking for a blank check to push anything I want. I
will of course submit the code for review and adhere to the technical
comments made to it.

And I want to insist, not just for me: this is a mechanism we need.

Right now, the only way to know if a change will be accepted is to
submit a patch series in its almost final shape. For something
ambitious, that can mean a lot of work, and if the patches are
eventually rejected, that work goes to waste. The possibility that it
happens is discouraging.

And with that discouragement, we only get low-hanging fruits, changes
that are sure to be accepted. Or we get intimidation: submit a 100+
patch series, present the project with a fait accopli, and count on the
fact that nobody will dare object in the face of the sheer amount of
work, even if the proposal is flawed at its core and would need a
complete rewrite.

The lack of a procedure to engage in something ambitious without the
risk of outright rejection is limiting the progress of FFmpeg, and it is
sad.

Maybe this simple idea would be enough, if people like it: doc/plans.md
containing more-or-less detailed plans of ambitious plans. Patches to
doc/plans.md are reviewed on the mailing list like any other patches.
But once a plan is accepted, patches that implement it cannot be
objected on principle by naysayers and bikeshedders, only on technical
matters.

I will try to propose a small patch for that, to get the conversation
going.

> Rather i wanted to comment on
> XML and JSON and FFmpeg.

Thanks.

> I saw on IRC this (authors removed because their names are not important here,
> iam really replying to the statments not the people)
> A>       but I, IMVHO, dont think this project should get a XML parser
> B>      "nice and limited" XML parser sounds like all sorts of "fun"
> C>      Yeah, xml/json/... parser in ffmpeg is just... no
> 
> Whats the purpose of FFmpeg ?
> "A complete, cross-platform solution to record, convert and stream audio and video."
> first actually i think that should be chnaged to something like this:
> "A complete, cross-platform solution to your multmedia needs"
> Because there are subtitles and many other things we do care about that this
> misses 
> 
> Now to achieve this do we need xml and json ?
> grep tells me we have 500 matches (not counting docs) for xml and almost 100
> for json
> Also for streaming and some cases filtering being able to serialize objects
> would be useful. xml and json seem better choices than some ad-hoc format
> So i would awnser the question do we need XML and JSON, with yes we live
> in a world that uses XML and JSON so if we give the option to use it too
> that makes it easier for others to interact.
> 
> now do we need our own implementation of it ? I dont know but we have
> in almost all cases favored our native implementations when someone wrote
> one. And libxml2 has had so many security issues that i think we should
> at least consider replacing it.

Replacing our use of libxml2 was my reason for starting to think and
then write a XML parser, for exactly the reasons we describe, plus the
fact that libxml2 is not a system library. We need XML for at least
MPEG-DASH and IMF, which are unambiguously part of FFmpeg's purpose.

And the objective of the "limited" parser would be to be able to digest
DASH manifests found in the wild.

> The 2nd thing that comes to mind with "purpose of FFmpeg" is FF stands for
> "Fast Forward". To move fast forward we should not lose developers because of
> rather minor disagreements.
> IMO if Nicolas wants to write and maintain AVWriter and a simple xml parser
> (which i presume has the goal to parse the XML we would generate for 
> serialization and in other places). I really think telling Nicolas "no" is 
> a unwise choice. But if someone is against very basic xml or json parsers
> please speak up now and here because its still better to say "no" now than
> after nicolas did the work.

Thanks again. It is exactly my concern. If we collectively decide we do
not want some project, I can work on something else. But the uncertainty
is killing my motivation.

Regards,

-- 
  Nicolas George

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Tomas Härdin]

ons 2022-08-17 klockan 19:21 +0200 skrev Michael Niedermayer:
> 
> Now to achieve this do we need xml and json ?
> grep tells me we have 500 matches (not counting docs) for xml and
> almost 100
> for json
> Also for streaming and some cases filtering being able to serialize
> objects
> would be useful. xml and json seem better choices than some ad-hoc
> format
> So i would awnser the question do we need XML and JSON, with yes we
> live
> in a world that uses XML and JSON so if we give the option to use it
> too
> that makes it easier for others to interact.
> 
> now do we need our own implementation of it ? I dont know but we have
> in almost all cases favored our native implementations when someone
> wrote
> one. And libxml2 has had so many security issues that i think we
> should
> at least consider replacing it.

Absolutely not. The solution is to fix and improve libxml2, not to add
to the problem with our own XML parser which will inevitably have its
own set of bugs. NIH for its own sake does nothing but split developer
effort and increase the number of bugs.


Parsing is hard and the source of the vast majority of CVEs. This
project should take the advice of the langsec community to heart.
Resist the urge to write your own shotgun parsers because it is "fun".
Make your protocol context-free or regular!

/Tomas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Jean-Baptiste Kempf]

On Wed, 17 Aug 2022, at 19:21, Michael Niedermayer wrote:
> a unwise choice. But if someone is against very basic xml or json parsers
> please speak up now and here because its still better to say "no" now than
> after nicolas did the work.

Absolutely against this idea.

Both JSON and XML are very very very difficult to parse in a secure manner.

Doing a simple XML parser and a simple JSON parser might be simple tasks for any decent programmer, doing those parsers is extremely difficult because there are a lot of complex corners cases, even if you take a subset of XML. Unicode, encoding, entities decoding, binary data, languages are not something you can skip, even if you take a subset of XML.

Once you add document validation and DTD, namespaces, recursive XML or XPath/XQuery this makes it a project as big as the whole FFmpeg, and that's why libxml2 is so big.
If you just want DASH and TTML (and maybe fontconfig), you still have to do a large set of features.

And then you need to care about security. It's a difficult problem to fix, and seeing the track record of the security of open source multimedia projects, we should focus on our issues, not adding new ones.


If you believe that you can do a better job than thousands of people paid large amounts of money who spent decades on this problem, then, please do a separate project, host it on git.ffmpeg.org, git.videolan.org or github, and give us a fast streaming API. Please be sure that you validate most test-suites and cornercases too. And fuzz it.

Managing to do that would be an impact probably much bigger than FFmpeg, so don't hesitate. And then FFmpeg will be able to use it, and other projects too.

But for me, until this is ready and battle-tested, it's a hard no..

jb

-- 
Jean-Baptiste Kempf -  President
+33 672 704 734
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2449 bytes --]

On Thu, Aug 18, 2022 at 07:19:07PM +0200, Jean-Baptiste Kempf wrote:
> On Wed, 17 Aug 2022, at 19:21, Michael Niedermayer wrote:
> > a unwise choice. But if someone is against very basic xml or json parsers
> > please speak up now and here because its still better to say "no" now than
> > after nicolas did the work.
> 
> Absolutely against this idea.
> 
> Both JSON and XML are very very very difficult to parse in a secure manner.
> 
> Doing a simple XML parser and a simple JSON parser might be simple tasks for any decent programmer, doing those parsers is extremely difficult because there are a lot of complex corners cases, even if you take a subset of XML. Unicode, encoding, entities decoding, binary data, languages are not something you can skip, even if you take a subset of XML.
> 
> Once you add document validation and DTD, namespaces, recursive XML or XPath/XQuery this makes it a project as big as the whole FFmpeg, and that's why libxml2 is so big.
> If you just want DASH and TTML (and maybe fontconfig), you still have to do a large set of features.
> 
> And then you need to care about security. It's a difficult problem to fix, and seeing the track record of the security of open source multimedia projects, we should focus on our issues, not adding new ones.
> 
> 
> If you believe that you can do a better job than thousands of people paid large amounts of money who spent decades on this problem, then, please do a separate project, host it on git.ffmpeg.org, git.videolan.org or github, and give us a fast streaming API. Please be sure that you validate most test-suites and cornercases too. And fuzz it.
> 
> Managing to do that would be an impact probably much bigger than FFmpeg, so don't hesitate. And then FFmpeg will be able to use it, and other projects too.
> 
> But for me, until this is ready and battle-tested, it's a hard no..

ok
but just to clarify what i meant / was thinking of was a simple
key / value style parser no features beyond that. just enough so we can
use it to read our own generated xml from some object serialization.

thx


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The real ebay dictionary, page 1
"Used only once"    - "Some unspecified defect prevented a second use"
"In good condition" - "Can be repaird by experienced expert"
"As is" - "You wouldnt want it even if you were payed for it, if you knew ..."

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] API enhancements / broken promises

[Timo Rothenpieler]

On 19.08.2022 20:30, Michael Niedermayer wrote:
> ok
> but just to clarify what i meant / was thinking of was a simple
> key / value style parser no features beyond that. just enough so we can
> use it to read our own generated xml from some object serialization.

One can hardly call that XML then.
For something like that, it's much better to just use libxml2, if you 
really want to use XML, or just use another, super trivial format, like 
.ini or something.

XML, JSON, YAML and friends all have a massive amount of complexity 
attached to them.
Any kind of parser we add for them would almost inevitably grow over 
time, as people add more and more features to them. "Just one more 
simple patch".

I'd rather not open that can of worms.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".