Re: [FFmpeg-devel] [PATCH] lavfi: protection against premultiplied alpha (was: The patch series about premultiplied alpha)

[Niklas Haas via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

On Fri, 22 Aug 2025 15:47:04 +0200 Nicolas George via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> wrote:
> The filters that do not even touch the alpha plane are the most likely
> to produce invalid garbage with premultiplied alpha. They should NOT
> enable support.

Quod non sequitur.

Filters which do not touch the alpha channel, overwhelmingly either do not
even care about the pixel contents, or produce their desired effect
*regardless* of the colorspace.

Indeed, the choice of colorspace to do filtering in is often a delibarate
choice on the part of the user.

Do these filters also prevent you from feeding them frames in linear light?
What about frames in a perceptually uniform colorspace (like ICtCp)? Should
we block filters from accepting anything by default except linear light
floating point RGB?

> (The issue of gamma breaks all these assumptions, but the consensus on
> the issue of gamma seems to be to just ignore it and have the habit to
> compensate for it. If we were adding “float gamma” to AVFrame I would
> insist on the same caution.)

I mean, the fact that we already *have* nonlinear colorspaces (gamma trc) in
AVFrame, and that they are already handled the same way all other color space
properties are, is a pretty strong argument that we should do the same for
alpha mode.

Oh, and, to be clear, I do still plan on adding color trc and primaries to
the avfilter negotiation eventually, once we have more efficient native
primary conversion and tone mapping in libswscale.

(Of course, the default for those fields should also be the same as the status
quo, i.e. to support everything by default, for the exact same reasons)

> [...]
>
> This is not theoretical.

Dude, I understand the theory of colorspaces probably better than anybody
else in this project. A belabored explanation of why linear operations do
not commute over nonlinear functions is not necessary.

Your argument that filters somehow "want" or "expect" to perform linear
operations on linear colorspaces falls apart instantly when you realize that
the overwhelming majority of filters *already* accept colors in a number of
nonlinear colorspaces (and even nonlinear encodings of nonlinear colorspaces).

The matter of fact is, that filtering in straight alpha actually overwhelmingly
produces *wrong* results in practice. Why do you think all windowing systems,
all compositors etc. use premultiplied alpha internally? Of course, the math
being vastly simpler is a big part of it, but the main reason is that straight
alpha filtering has a tendency to bleed undesired color fringes from transparent
regions into colored ones; including in your example.

I could illustrate this in a number of ways, but the simplest would be the
following:

Ref: [0:v][1:v] overlay,avgblur=30
A: [0:v] avgblur=30 [a]; [1:v] avgblur=30 [b]; [a][b] overlay
B: [0:v] avgblur=30 [a]; [1:v] premultiply=inplace=1,avgblur=30 [b]; [a][b] overlay

Which one do you think will match the reference result, A or B?

> Do you really think drawutils was the only code that produces garbage
> with premultiplied? Drawutils is just sharing code between many filters
> that have needs similar enough. If a filter has needs that are not
> covered by drawutils, it will use its own code, but the same kind of
> code as drawutils. Code that will need fixing too.

I mean, if you see another filter that produces obviously broken results
with premultiplied alpha, feel free to point it out, and I will probably
get around to fixing it at some point. (Or fix it yourself - as you can see
from the example of drawutils, the fix is usually trivial)

The fact that the blacklist is nonempty, obviously does not on its own
constitute an argument for why a whitelist should therefore be the preferred
solution.

> Also, please remember this: with the work you did to make it properly
> part of the negotiation, automatic conversion filters are inserted
> automatically. If we forget to flag a filter that works, the annoyance
> will be minimal. And we can always flag it later, whereas garbage
> already written cannot be fixed.

True. I am not arguing from a point of mere practicality (practically, what
the default is does not matter much), but from a point of *consistency* and
API user expectation, which is obviously that things both 1) generally continue
working as they did before, and 2) new fields behave similar to existing fields.

Without a very clear overriding justification otherwise, I still see no reason
to delibarately break from the status quo of new negotiation properties being
include-all by default.

Arguably, everything else is also not backwards compatible.

> Regards,
>
> --
>   Nicolas George
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".