Re: [FFmpeg-devel] [PATCH] lavfi: protection against premultiplied alpha (was: The patch series about premultiplied alpha)

[Nicolas George via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

Hi.

Niklas Haas via ffmpeg-devel (HE12025-08-20):
> I accidentally deleted a line too many here, the text was supposed to read:
> 
> Do you expect filter authors to remember to enable support for
> premultiplied alpha, even if they don't even touch the alpha plane?

Hi. It is a nice coincidence that these two lines summarize perfectly
the mistake you make that is leading you to the wrong conclusion.
Therefore, I will only reply to this, addressing other points of your
mail as necessary.

The filters that do not even touch the alpha plane are the most likely
to produce invalid garbage with premultiplied alpha. They should NOT
enable support.

Let me explain.

First, by garbage, I mean: (1) not what the user expects + (2) not what
the documentation of the filters says + (3) not the same result as if
the processing were done in full range and conversion to premultiplied
alpha was done afterwards (and not just because of different rounding).
And by invalid garbage I mean cases where the value of a component ends
up greater than the value of alpha.

Processing frame data is written based on several assumptions of
linearity and independence: filters assume that the same (R,G,B) or
(Y,U,V) triplet corresponds to the same color, even if not taking A into
account; filters assume that averaging colors is done by averaging each
component, etc.

If we were talking about change in color primaries, then all would be
fine because these assumptions still hold for different primaries: it is
just a change of basis in the space of colors with the alpha direction a
stable subspace.

(The issue of gamma breaks all these assumptions, but the consensus on
the issue of gamma seems to be to just ignore it and have the habit to
compensate for it. If we were adding “float gamma” to AVFrame I would
insist on the same caution.)

But premultiplied alpha is not just a change of basis, it is not linear.

Let us just take an example: a filter will average two pixels, one very
dark red opaque and one bright red very transparent.

Full range input: (0.2, 0, 0, 1) and (1, 0, 0, 0.2)
Average component by component: (0.6, 0, 0, 0.6)

Conversion to premultiplied: (0.2, 0, 0, 1) and (0.2, 0, 0, 0.2)
Average component by component: (0.2, 0, 0, 0.6)

As you cans see, this not the same result. The same result would be
expressed as (0.36, 0, 0, 0.6) in premultiplied. This gave us 0.2/0.6 =~
0.33 red instead of 0.6 red.

That was for filters that do the average on all components, including
alpha. Since the space of valid components value with premultiplied
alpha is a pyramid, convex, the result is garbage but not invalid
garbage.

Now let us look at would produce a filter that does not take alpha into
account at all. The computations are the same, the filter will get
(0.2, 0, 0) for the average, and it will store it in a pixel, not
touching the alpha value for that pixel.

And this is where garbage becomes invalid: if the target pixel had alpha
0.1, then it now has (0.2, 0, 0, 0.1), which is invalid.

I took average, but the issue arises for all kinds of operations: every
time the values of multiple pixels are mixed together, the proper way to
compute it in premultiplied is not the usual way. It also applies to
filters that do not draw pixels but only compute statistics: they will
assume a pixel is dark when it is just transparent.


This is not theoretical.

Remember: you had to update drawutils, because you could see, with your
own eyes, that filters using drawutils produced garbage with
premultiplied.

Do you really think drawutils was the only code that produces garbage
with premultiplied? Drawutils is just sharing code between many filters
that have needs similar enough. If a filter has needs that are not
covered by drawutils, it will use its own code, but the same kind of
code as drawutils. Code that will need fixing too.

The only filters we can be sure will work with premultiplied are (1) the
filters that implement formulas specific to the premultiplied case, (2)
filters that only copy pixel values, all four components, and never do
any kind of computation on them and (3) filters that do not touch the
pixels at all, only the frame properties and metadata.

(Technically, (3) is a subset of (2).)

Most filters are in none of these cases. This is why I insist support
must be opt-in, white list in your own words.


Also, please remember this: with the work you did to make it properly
part of the negotiation, automatic conversion filters are inserted
automatically. If we forget to flag a filter that works, the annoyance
will be minimal. And we can always flag it later, whereas garbage
already written cannot be fixed.

(I want to emphasize that you did the work of making it part of the
negotiation not on my demand. It is useful, there is no doubt about, and
much better than the quick and dirty version I sent, and it is a good
thing you did it. But it was not on my demand and it does not change the
fact that support must be opt-in.)

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".