I hesitated a long time before replying, but all considering, at this
point expressing what is weighing on my heart cannot make things worse.


Michael Niedermayer (12022-07-03):
> What is the timeline for the audio+video merge ?

I cannot give a timeline because I do not work on FFmpeg on a schedule,
I work on FFmpeg in my free time, for fun. And lately, working on FFmpeg
has been really un-fun. Recently, my contributions have been met with
indifference at best (see when I proposed a XML parser six months ago:
nobody cared), outright hostility at worst. Under these circumstances,
whenever I am considering working on something FFmpeg-related, I usually
find something else more fun to do.

I do not recognize the project I started contributing to more than
fifteen years ago. I do not even recognize the project that boasted the
clever optimization framework that made FFVP9 possible, it has become
increasingly hostile to trying new and more efficient ways of doing
things in favor of a corporate never-take-risks style of coding. I am
more and more often considering giving up and cutting my losses.

> IIUC this would resolve this deadlock (with extra work adapting the patchset
> so it would be work for SW adapting it and it would be work for you finishing
> the merge)
> Also can others help nicolas moving his work forward
> 
> What i suggest is to pick a time and then try to finish the merge before.
> If it succeeds this patchset needs updating and can move forward without
> this main objection
> OTOH if the time is not hit, we agree that the objection can no longer be
> used

My answer as maintainer of the framework of libavfilter is: no.

Of course, maintainers are not dictators. The members of the project can
collectively decide otherwise. But I have to warn you about the
consequences.

First, the issue about negotiation is not he only severe flaw in this
patch series. I can immediately quote another one: for text subtitles,
the approach of this proposal to synchronization is to feed everything
to libass as it comes and see what comes out. It will work on easy
cases, when the subtitles are interleaved with the video or come from a
separate file. But as soon as a filter will, for example, adjust the
timing to make the subtitles more early, it will just not work. Of
course, it was not tested because this patch series does not even offer
the feature to adjust the time of subtitles, which is frankly
ridiculous, it is one of the most obvious thing people might want to do.

Note that I did not have to perform a full review of the patch series to
find this flaw. I have been preparing to implement subtitles filtering
for years now, I know which aspects are tricky and hard to implement
properly. I only had to check precisely how it was done. And it turns
out it was not done at all.

I suspect that if I were to do a full review, I would find a few other
flaws. But the author has made painfully clear that they did not respect
my expertise in this area, I have no reason to waste my time doing it.

It illustrates what it means to be a maintainer. It does not only mean
the task of reviewing and applying bug fixes. The maintainer holds in
their head a knowledge of the code that cannot be fully shared in
writing. The maintainer also holds in their head plans to evolve and
extend the code.

I have plans for libavfilter. Not just vague ideas, but precise plans on
how to reach the goal. I have plans for subtitles filtering, of course.
But not only.

I have plans for filtering data packets, so that bistream filters do not
need to have a separate and redundant API and ffmpeg does not need a
separate code path for -c copy.

I have plans for threading, or more precisely integrating filters in a
global parallelized task and I/O scheduler.

I have plans for seeking, with the seek target going back from the
output to the input(s).

I have plans for partial graph reconfiguration, to accommodate format
changes mid-stream and let applications alter filtering in the middle.

All of this is exciting. I am eager to work on any of this.

Unfortunately, before it can happen, boring things need to be done.
Parts of the framework of libavfilter are very fragile. Working on any
of these is likely to break cases that were specifically fixed in the
past.

I can work on boring things if they are necessary to reach the exciting
parts.

What I cannot do is motivate myself to work on the boring things with
the threat that the exciting things will be snatched under me by an
inferior version from somebody who just refuses to engage with the
boring necessary things.

If this patch series gets applied, it will make the boring things a lot
harder to do and it will ruin some of the plans I mentioned above. Under
these circumstances, do not expect me to work on libavfilter again any
time soon, even if it is to apply an obviously valid fix for an
exploitable security issue.

So, the choice is:

- Apply this patch series, find a new maintainer for libavfilter and
  kiss goodbye to seeking, threading, reconfiguration (and subtitles
  filtering that work usefully).

- Make it clear that this patch series is rejected until the framework
  is robust enough and has enough test coverage.

- Let the situation continue to rot.

Note: The most urgent and boring task is adding FATE tests to the
heuristics of the negotiation process. It is actually rather easy. I
would be happy to offer advice and even tutoring if somebody wants to
contribute.

Regards,

-- 
  Nicolas George