[FFmpeg-devel] Towards YUVJ removal

[FFmpeg-devel] Towards YUVJ removal

[Niklas Haas]

So, as was discussed at the last meeting, we should move towards
removing YUVJ. I want to gather feedback on what appears to be to the
major hurdle, and possible ways to solve it.

The basic major issue is how to handle the case of combining limited
range input with an encoder for a format that only accepts full range
data. The common case, for example, would be converting a frame from a
typical video file to a JPEG:

$ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg

Currently, this works because the JPEG encoder only advertises YUVJ
pixel formats, and therefore ffmpeg auto-inserts swscaler to convert
from limited range to full range. But this depends on conflating color
range and pixel formats, which is exactly what has been marked
deprecated for an eternity.

Now, there are some solutions I can see for how to handle this case in
a non-YUVJ world:

1. Simply output an error, and rely on the user to insert a conversion
   filter, e.g.:

   $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
   error: inputs to jpeg encoder must be full range

   $ ffmpeg -f lavfi -i smptebars -vframes 1 -vf scale=out_range=jpeg output.jpg
   ... works

2. Have the JPEG encoder take care of range conversion internally, by
   using sws to convert limited to full range.

3. Allow filters to start exposing color space metadata as part of
   filter negotiation, and then auto-insert swscaler whenever colorspace
   conversion needs to happen as a result of filters not accepting the
   corresponding color metadata. This would also allow more than just
   conversion between limited/full range.

4. Combine approach 1 with an encoder flag for "supports full range
   only", and have ffmpeg.c auto-insert a scale filter as the last step
   before the encoder if this flag is set (and input is limited range).

1 would be the most explicit but would break any existing pipeline that
includes conversion to jpeg, which is probably a very large number.

2 would be the least work, but violates abstraction boundaries.

3 would be the most work and is, IMO, of questionable gain.

4 would be both explicit and not break existing workflows.

Thoughts?
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Jean-Baptiste Kempf]

3. :)

On Fri, 9 Dec 2022, at 12:49, Niklas Haas wrote:
> So, as was discussed at the last meeting, we should move towards
> removing YUVJ. I want to gather feedback on what appears to be to the
> major hurdle, and possible ways to solve it.
>
> The basic major issue is how to handle the case of combining limited
> range input with an encoder for a format that only accepts full range
> data. The common case, for example, would be converting a frame from a
> typical video file to a JPEG:
>
> $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
>
> Currently, this works because the JPEG encoder only advertises YUVJ
> pixel formats, and therefore ffmpeg auto-inserts swscaler to convert
> from limited range to full range. But this depends on conflating color
> range and pixel formats, which is exactly what has been marked
> deprecated for an eternity.
>
> Now, there are some solutions I can see for how to handle this case in
> a non-YUVJ world:
>
> 1. Simply output an error, and rely on the user to insert a conversion
>    filter, e.g.:
>
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
>    error: inputs to jpeg encoder must be full range
>
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 -vf scale=out_range=jpeg output.jpg
>    ... works
>
> 2. Have the JPEG encoder take care of range conversion internally, by
>    using sws to convert limited to full range.
>
> 3. Allow filters to start exposing color space metadata as part of
>    filter negotiation, and then auto-insert swscaler whenever colorspace
>    conversion needs to happen as a result of filters not accepting the
>    corresponding color metadata. This would also allow more than just
>    conversion between limited/full range.
>
> 4. Combine approach 1 with an encoder flag for "supports full range
>    only", and have ffmpeg.c auto-insert a scale filter as the last step
>    before the encoder if this flag is set (and input is limited range).
>
> 1 would be the most explicit but would break any existing pipeline that
> includes conversion to jpeg, which is probably a very large number.
>
> 2 would be the least work, but violates abstraction boundaries.
>
> 3 would be the most work and is, IMO, of questionable gain.
>
> 4 would be both explicit and not break existing workflows.
>
> Thoughts?
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

-- 
Jean-Baptiste Kempf -  President
+33 672 704 734
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Andreas Rheinhardt]

Niklas Haas:
> So, as was discussed at the last meeting, we should move towards
> removing YUVJ. I want to gather feedback on what appears to be to the
> major hurdle, and possible ways to solve it.
> 
> The basic major issue is how to handle the case of combining limited
> range input with an encoder for a format that only accepts full range
> data. The common case, for example, would be converting a frame from a
> typical video file to a JPEG:
> 
> $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
> 
> Currently, this works because the JPEG encoder only advertises YUVJ
> pixel formats, and therefore ffmpeg auto-inserts swscaler to convert
> from limited range to full range. But this depends on conflating color
> range and pixel formats, which is exactly what has been marked
> deprecated for an eternity.
> 

This is incorrect: Here are the pixel formats advertised by the mjpeg
encoder:

    .p.pix_fmts     = (const enum AVPixelFormat[]) {
        AV_PIX_FMT_YUVJ420P, AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P,
        AV_PIX_FMT_YUV420P,  AV_PIX_FMT_YUV422P,  AV_PIX_FMT_YUV444P,
        AV_PIX_FMT_NONE
    },

ffmpeg only inserts the swscale filters, because said list is overridden
in get_compliance_normal_pix_fmts() in fftools/ffmpeg_filter.c. See
059fc2d9da5364627613fb3e6424079e14dbdfd3.
Also notice that the encoder errors out if fed with limited range data
depending upon strict_std_compliance (see
ff_mjpeg_encode_check_pix_fmt()), which is very similar to the first
approach below; the override being in fftools implies that the breakage
of the first approach would be confined to users of fftools.

> Now, there are some solutions I can see for how to handle this case in
> a non-YUVJ world:
> 
> 1. Simply output an error, and rely on the user to insert a conversion
>    filter, e.g.:
> 
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
>    error: inputs to jpeg encoder must be full range
> 
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 -vf scale=out_range=jpeg output.jpg
>    ... works
> 
> 2. Have the JPEG encoder take care of range conversion internally, by
>    using sws to convert limited to full range.
> 
> 3. Allow filters to start exposing color space metadata as part of
>    filter negotiation, and then auto-insert swscaler whenever colorspace
>    conversion needs to happen as a result of filters not accepting the
>    corresponding color metadata. This would also allow more than just
>    conversion between limited/full range.
> 
> 4. Combine approach 1 with an encoder flag for "supports full range
>    only", and have ffmpeg.c auto-insert a scale filter as the last step
>    before the encoder if this flag is set (and input is limited range).
> 
> 1 would be the most explicit but would break any existing pipeline that
> includes conversion to jpeg, which is probably a very large number.
> 
> 2 would be the least work, but violates abstraction boundaries.
> 
> 3 would be the most work and is, IMO, of questionable gain.
> 
> 4 would be both explicit and not break existing workflows.
> 
> Thoughts?

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Niklas Haas]

On Fri, 09 Dec 2022 13:10:02 +0100 Andreas Rheinhardt <andreas.rheinhardt@outlook.com> wrote:
> This is incorrect: Here are the pixel formats advertised by the mjpeg
> encoder:
> 
>     .p.pix_fmts     = (const enum AVPixelFormat[]) {
>         AV_PIX_FMT_YUVJ420P, AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P,
>         AV_PIX_FMT_YUV420P,  AV_PIX_FMT_YUV422P,  AV_PIX_FMT_YUV444P,
>         AV_PIX_FMT_NONE
>     },
> 
> ffmpeg only inserts the swscale filters, because said list is overridden
> in get_compliance_normal_pix_fmts() in fftools/ffmpeg_filter.c. See
> 059fc2d9da5364627613fb3e6424079e14dbdfd3.
> Also notice that the encoder errors out if fed with limited range data
> depending upon strict_std_compliance (see
> ff_mjpeg_encode_check_pix_fmt()), which is very similar to the first
> approach below; the override being in fftools implies that the breakage
> of the first approach would be confined to users of fftools.

Oh, you are right. So that presents an alternative to 4 - rather than an
encoder flag, we can tie the auto-conversion in fftools to be tied to
the strict_std_compliance check.

I will try implementing this approach, it may be the best compromise in
that it presents a clear upgrade path that doesn't break the common
case.

That said, there still is an encoder that has this problem: "amv".
Currently, this only advertises YUVJ, but after YUVJ removal, it would
be treated equivalently to mjpeg when `strict_std_compliance` is enabled.

But given that this is a less common format, I think, such a regression
would not be as big a concern. (And we can still special-case it in
fftools, if we want to)
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Paul B Mahol]

On 12/9/22, Niklas Haas <ffmpeg@haasn.xyz> wrote:
> On Fri, 09 Dec 2022 13:10:02 +0100 Andreas Rheinhardt
> <andreas.rheinhardt@outlook.com> wrote:
>> This is incorrect: Here are the pixel formats advertised by the mjpeg
>> encoder:
>>
>>     .p.pix_fmts     = (const enum AVPixelFormat[]) {
>>         AV_PIX_FMT_YUVJ420P, AV_PIX_FMT_YUVJ422P, AV_PIX_FMT_YUVJ444P,
>>         AV_PIX_FMT_YUV420P,  AV_PIX_FMT_YUV422P,  AV_PIX_FMT_YUV444P,
>>         AV_PIX_FMT_NONE
>>     },
>>
>> ffmpeg only inserts the swscale filters, because said list is overridden
>> in get_compliance_normal_pix_fmts() in fftools/ffmpeg_filter.c. See
>> 059fc2d9da5364627613fb3e6424079e14dbdfd3.
>> Also notice that the encoder errors out if fed with limited range data
>> depending upon strict_std_compliance (see
>> ff_mjpeg_encode_check_pix_fmt()), which is very similar to the first
>> approach below; the override being in fftools implies that the breakage
>> of the first approach would be confined to users of fftools.
>
> Oh, you are right. So that presents an alternative to 4 - rather than an
> encoder flag, we can tie the auto-conversion in fftools to be tied to
> the strict_std_compliance check.
>
> I will try implementing this approach, it may be the best compromise in
> that it presents a clear upgrade path that doesn't break the common
> case.
>
> That said, there still is an encoder that has this problem: "amv".
> Currently, this only advertises YUVJ, but after YUVJ removal, it would
> be treated equivalently to mjpeg when `strict_std_compliance` is enabled.
>
> But given that this is a less common format, I think, such a regression
> would not be as big a concern. (And we can still special-case it in
> fftools, if we want to)

Sounds too much hackish.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Hendrik Leppkes]

On Fri, Dec 9, 2022 at 12:49 PM Niklas Haas <ffmpeg@haasn.xyz> wrote:
>
> So, as was discussed at the last meeting, we should move towards
> removing YUVJ. I want to gather feedback on what appears to be to the
> major hurdle, and possible ways to solve it.
>
> The basic major issue is how to handle the case of combining limited
> range input with an encoder for a format that only accepts full range
> data. The common case, for example, would be converting a frame from a
> typical video file to a JPEG:
>
> $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
>
> Currently, this works because the JPEG encoder only advertises YUVJ
> pixel formats, and therefore ffmpeg auto-inserts swscaler to convert
> from limited range to full range. But this depends on conflating color
> range and pixel formats, which is exactly what has been marked
> deprecated for an eternity.
>
> Now, there are some solutions I can see for how to handle this case in
> a non-YUVJ world:
>
> 1. Simply output an error, and rely on the user to insert a conversion
>    filter, e.g.:
>
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
>    error: inputs to jpeg encoder must be full range
>
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 -vf scale=out_range=jpeg output.jpg
>    ... works
>
> 2. Have the JPEG encoder take care of range conversion internally, by
>    using sws to convert limited to full range.
>
> 3. Allow filters to start exposing color space metadata as part of
>    filter negotiation, and then auto-insert swscaler whenever colorspace
>    conversion needs to happen as a result of filters not accepting the
>    corresponding color metadata. This would also allow more than just
>    conversion between limited/full range.
>
> 4. Combine approach 1 with an encoder flag for "supports full range
>    only", and have ffmpeg.c auto-insert a scale filter as the last step
>    before the encoder if this flag is set (and input is limited range).
>
> 1 would be the most explicit but would break any existing pipeline that
> includes conversion to jpeg, which is probably a very large number.
>
> 2 would be the least work, but violates abstraction boundaries.
>
> 3 would be the most work and is, IMO, of questionable gain.
>
> 4 would be both explicit and not break existing workflows.
>

Some sort of metadata has to be present to indicate this. It is not
reasonable to assume users magically know what range a codec accepts,
and then go out of their way to hardcode a list that might need full
range.

So 1 alone is entirely not reasonable, especially if you think eg.
about an API user, who might create an app that may not show the
immediate error message, or even have a button to insert a scaler.
Instead the API should contain all the information for an application
to make the right choices if necessary.
Of course I already see people argue that JPEG accepts both depending
on standards compliance etc, but the metadata should allow me to make
the encoder work without needing to set standard compliance to some
value. Or in other words, it should just work the most
straight-forward way, without needing special knowledge about the
encoder or even format.

For mjpegenc this probably means it should only advertise full range.
Or this information needs to somehow be context-sensitive and allow me
to indicate the standards compliance value I care for. If the encoder
indicates both limited and full, but errors out on limited, thats not
useful.

- Hendrik
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Diederick C. Niehorster]

On Fri, Dec 9, 2022 at 1:45 PM Niklas Haas <ffmpeg@haasn.xyz> wrote:

> Oh, you are right. So that presents an alternative to 4 - rather than an
> encoder flag, we can tie the auto-conversion in fftools to be tied to
> the strict_std_compliance check.
>
> I will try implementing this approach, it may be the best compromise in
> that it presents a clear upgrade path that doesn't break the common
> case.
>
> That said, there still is an encoder that has this problem: "amv".
> Currently, this only advertises YUVJ, but after YUVJ removal, it would
> be treated equivalently to mjpeg when `strict_std_compliance` is enabled.
>
> But given that this is a less common format, I think, such a regression
> would not be as big a concern. (And we can still special-case it in
> fftools, if we want to)
>

As a user of the API, I do not think hacking fftools is the way to go. I
vote option 3, filter negotiation also takes color_range (etc) info into
account, and auto-inserts a scale where needed. But that is only half the
solution, and actually not the important part. If i have a frame source and
another sink that both have declared pixel formats, i can easily check
whether conversion needs to happen at all, and if so, just set up an empty
filter chain and have it auto-insert the required scale filter (super!).
Now i would have to also be able to:
1. have settings in the filter context (i guess) for color_range, etc, and
corresponding syntax for the args argument of
avfilter_graph_create_filter() and/or corresponding options to set, so that
filter input and output sides can be provided with correct info
2. query my sink for what it actually provides (a bunch more functions
ala av_buffersink_get_format() are needed to get color_range, etc)

I probably don't oversee the problem, so may be missing more that needs to
be done, or be off the mark here.

I also agree with Henrik that a way to query encoders for what they support
besides formats is critical, and that this should ideally be
context-senstive (so you can set certain options, and then query what
remains of supported pix_fmts, color_range, etc).

All the best,
Dee
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3598 bytes --]

On Fri, Dec 09, 2022 at 12:49:41PM +0100, Niklas Haas wrote:
> So, as was discussed at the last meeting, we should move towards
> removing YUVJ. I want to gather feedback on what appears to be to the
> major hurdle, and possible ways to solve it.
> 
> The basic major issue is how to handle the case of combining limited
> range input with an encoder for a format that only accepts full range
> data. The common case, for example, would be converting a frame from a
> typical video file to a JPEG:
> 
> $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
> 
> Currently, this works because the JPEG encoder only advertises YUVJ
> pixel formats, and therefore ffmpeg auto-inserts swscaler to convert
> from limited range to full range. But this depends on conflating color
> range and pixel formats, which is exactly what has been marked
> deprecated for an eternity.
> 
> Now, there are some solutions I can see for how to handle this case in
> a non-YUVJ world:
> 
> 1. Simply output an error, and rely on the user to insert a conversion
>    filter, e.g.:
> 
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
>    error: inputs to jpeg encoder must be full range
> 
>    $ ffmpeg -f lavfi -i smptebars -vframes 1 -vf scale=out_range=jpeg output.jpg
>    ... works
> 
> 2. Have the JPEG encoder take care of range conversion internally, by
>    using sws to convert limited to full range.
> 
> 3. Allow filters to start exposing color space metadata as part of
>    filter negotiation, and then auto-insert swscaler whenever colorspace
>    conversion needs to happen as a result of filters not accepting the
>    corresponding color metadata. This would also allow more than just
>    conversion between limited/full range.
> 
> 4. Combine approach 1 with an encoder flag for "supports full range
>    only", and have ffmpeg.c auto-insert a scale filter as the last step
>    before the encoder if this flag is set (and input is limited range).
> 
> 1 would be the most explicit but would break any existing pipeline that
> includes conversion to jpeg, which is probably a very large number.
> 
> 2 would be the least work, but violates abstraction boundaries.
> 
> 3 would be the most work and is, IMO, of questionable gain.
> 
> 4 would be both explicit and not break existing workflows.

3 is nice but is hard as it draws filter negotiation in and that has
to do something even for non trivial filter networks.

4 with the complexity in jpeg as disscussed elsewhere in this thread
also may or may not be as clean as it sounds here

but both 3 and 4 with some amendment sound reasonable to me
another option would be to either improve 

A. general "supported value" information
a encoder should be able to tell the caller what it supports depending
on some parameter, for example depending on profile and level
or std compliance. This would mean implementing AVClass.query_ranges()
and using av_opt_query_ranges() IIRC


B. error reporting so that 
failures become machiene readable. 
aka, "this failed because color range is not X || std complicance is > Y"
the lib user could then retry by applying the change if that is within what
the usecase wants

Both A and B obvioulsy have a much broader use than YUVJ removal

thx


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The real ebay dictionary, page 2
"100% positive feedback" - "All either got their money back or didnt complain"
"Best seller ever, very honest" - "Seller refunded buyer after failed scam"

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Niklas Haas]

On Fri, 09 Dec 2022 17:50:11 +0100 Michael Niedermayer <michael@niedermayer.cc> wrote:
> On Fri, Dec 09, 2022 at 12:49:41PM +0100, Niklas Haas wrote:
> > So, as was discussed at the last meeting, we should move towards
> > removing YUVJ. I want to gather feedback on what appears to be to the
> > major hurdle, and possible ways to solve it.
> > 
> > The basic major issue is how to handle the case of combining limited
> > range input with an encoder for a format that only accepts full range
> > data. The common case, for example, would be converting a frame from a
> > typical video file to a JPEG:
> > 
> > $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
> > 
> > Currently, this works because the JPEG encoder only advertises YUVJ
> > pixel formats, and therefore ffmpeg auto-inserts swscaler to convert
> > from limited range to full range. But this depends on conflating color
> > range and pixel formats, which is exactly what has been marked
> > deprecated for an eternity.
> > 
> > Now, there are some solutions I can see for how to handle this case in
> > a non-YUVJ world:
> > 
> > 1. Simply output an error, and rely on the user to insert a conversion
> >    filter, e.g.:
> > 
> >    $ ffmpeg -f lavfi -i smptebars -vframes 1 output.jpg
> >    error: inputs to jpeg encoder must be full range
> > 
> >    $ ffmpeg -f lavfi -i smptebars -vframes 1 -vf scale=out_range=jpeg output.jpg
> >    ... works
> > 
> > 2. Have the JPEG encoder take care of range conversion internally, by
> >    using sws to convert limited to full range.
> > 
> > 3. Allow filters to start exposing color space metadata as part of
> >    filter negotiation, and then auto-insert swscaler whenever colorspace
> >    conversion needs to happen as a result of filters not accepting the
> >    corresponding color metadata. This would also allow more than just
> >    conversion between limited/full range.
> > 
> > 4. Combine approach 1 with an encoder flag for "supports full range
> >    only", and have ffmpeg.c auto-insert a scale filter as the last step
> >    before the encoder if this flag is set (and input is limited range).
> > 
> > 1 would be the most explicit but would break any existing pipeline that
> > includes conversion to jpeg, which is probably a very large number.
> > 
> > 2 would be the least work, but violates abstraction boundaries.
> > 
> > 3 would be the most work and is, IMO, of questionable gain.
> > 
> > 4 would be both explicit and not break existing workflows.
> 
> 3 is nice but is hard as it draws filter negotiation in and that has
> to do something even for non trivial filter networks.
> 
> 4 with the complexity in jpeg as disscussed elsewhere in this thread
> also may or may not be as clean as it sounds here
> 
> but both 3 and 4 with some amendment sound reasonable to me
> another option would be to either improve 
> 
> A. general "supported value" information
> a encoder should be able to tell the caller what it supports depending
> on some parameter, for example depending on profile and level
> or std compliance. This would mean implementing AVClass.query_ranges()
> and using av_opt_query_ranges() IIRC
> 
> 
> B. error reporting so that 
> failures become machiene readable. 
> aka, "this failed because color range is not X || std complicance is > Y"
> the lib user could then retry by applying the change if that is within what
> the usecase wants
> 
> Both A and B obvioulsy have a much broader use than YUVJ removal
> 
> thx

So, the biggest roadblock I see with regards to making color range etc.
part of filter negotiation is that, currently, color range (and
colorspace and so on) are part of AVFrame, which can (in theory and
possibly also in practice) change suddenly mid-stream.

So we can't somehow *statically* negotiate the appropriate conversion
filter into the filter graph depending on a value that is only known
after filter negotiation is complete. Fortunately, there is not
currently any filter in ffmpeg that only supports one color range, so we
don't actually *need* filter negotiation for YUVJ removal.

It's straight-forward to add a list of supported color ranges (even
depending on compliance level) into `AVCodec`, but of course, this still
relies on API users knowing to check this new field in order to avoid
potential regressions.

How about having a special conversion layer exist only in front of
encoders (e.g. inside avcodec_send_frame) to deal with incoming frames
that have mismatched metadata relative to what the encoder supports?
With an option to turn it on/off in AVCodecContext.

This would NOT take care of pix_fmt conversion (since the filter chain
already handles that), but merely deal with mismatches between sent
AVFrames and AVCodec advertised capabilities.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Nicolas George]

Niklas Haas (12022-12-13):
> So, the biggest roadblock I see with regards to making color range etc.
> part of filter negotiation is that, currently, color range (and
> colorspace and so on) are part of AVFrame, which can (in theory and
> possibly also in practice) change suddenly mid-stream.

That is not a problem, a lot of properties are part of AVFrame yet are
not supposed to change mid-stream in libavfilter, starting with the
resolution.

The real issue is not a hard one, but an expensive one: the negotiation
code requires a lot of refactoring, and before that testing coverage,
before we can comfortably add new negotiated properties.

I am not familiar with color ranges. Are they orthogonal to pixel
formats or do they depend on them? In other words, should we negotiate
them both at once, like sample rate, sample format and channel layout,
or should we decide the pixel format first and then the color range? The
plan I have for refactoring the negotiation can do both, but the second
one will require more work.

> How about having a special conversion layer exist only in front of
> encoders (e.g. inside avcodec_send_frame) to deal with incoming frames
> that have mismatched metadata relative to what the encoder supports?

I do not like that idea. But since the real negotiation requires a lot
of work, I will not oppose. Just make sure it is unobtrusive and easily
removed later.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Towards YUVJ removal

[Martin Storsjö]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On Fri, 9 Dec 2022, Niklas Haas wrote:

> So, as was discussed at the last meeting, we should move towards
> removing YUVJ.

Libswscale still does have cases where it doesn't work correctly, for some 
conversions, unless differences in range is signalled with the YUVJ pixel 
formats (this, despite that swscale itself warns that the user is 
specifying the deprecated pixel formats).

Attached is a small testcase example, which uses swscale to convert from 
YUV to YUV, with various color spaces and ranges. When converting from 
limited to full range or vice versa, within the same colorspace, swscale 
fails to realize that this isn't a no-op operation, unless the YUVJ 
formats are used.

When running the attached test example, I get the following output 
(modulo swscale warnings):

red 601 limited -> 601 full YUV 49,109,184 -> 38,106,192
red 601 full -> 601 limited YUV 38,106,192 -> 49,109,184
red 709 limited -> 709 full YUV 39,115,184 -> 27,113,192
red 709 full -> 709 limited YUV 27,113,192 -> 39,115,184

If the example is modified to always set yuv_{in,out}->format to 
AV_PIX_FMT_YUV420P, then swscale no longer actually does any conversion:

red 601 limited -> 601 full YUV 49,109,184 -> 49,109,184
red 601 full -> 601 limited YUV 38,106,192 -> 38,106,192
red 709 limited -> 709 full YUV 39,115,184 -> 39,115,184
red 709 full -> 709 limited YUV 27,113,192 -> 27,113,192


So this is yet another small detail that needs to be fixed before we can 
proceed towards actually removing the YUVJ formats.


// Martin

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-csrc; name=yuv2yuv.c, Size: 2789 bytes --]

#include <libswscale/swscale.h>
#include <stdio.h>

static void yuv2yuv(int Y, int U, int V, int in_colorspace, int param_in_full,
                    int out_colorspace, int param_out_full, const char* name)
{
    AVFrame* yuv_in = av_frame_alloc();
    yuv_in->format = param_in_full ? AV_PIX_FMT_YUVJ420P : AV_PIX_FMT_YUV420P;
    yuv_in->width = 128;
    yuv_in->height = 128;
    yuv_in->color_range = param_in_full ? AVCOL_RANGE_JPEG : AVCOL_RANGE_MPEG;
    yuv_in->color_primaries = in_colorspace;
    yuv_in->color_trc = in_colorspace;
    yuv_in->colorspace = in_colorspace;
    av_frame_get_buffer(yuv_in, 0);

    AVFrame* yuv_out = av_frame_alloc();
    yuv_out->format = param_out_full ? AV_PIX_FMT_YUVJ420P : AV_PIX_FMT_YUV420P;
    yuv_out->width = 128;
    yuv_out->height = 128;
    yuv_out->color_range = param_out_full ? AVCOL_RANGE_JPEG : AVCOL_RANGE_MPEG;
    yuv_out->color_primaries = out_colorspace;
    yuv_out->color_trc = out_colorspace;
    yuv_out->colorspace = out_colorspace;
    av_frame_get_buffer(yuv_out, 0);

    memset(yuv_in->data[0], Y, yuv_in->linesize[0] * yuv_in->height);
    memset(yuv_in->data[1], U, yuv_in->linesize[1] * (yuv_in->height / 2));
    memset(yuv_in->data[2], V, yuv_in->linesize[2] * (yuv_in->height / 2));

    struct SwsContext *sws = sws_getContext(yuv_in->width, yuv_in->height, yuv_in->format,
                                            yuv_out->width, yuv_out->height, yuv_out->format,
                                            0, NULL, NULL, NULL);

    int in_full, out_full, brightness, contrast, saturation;
    const int *inv_table, *table;
    sws_getColorspaceDetails(sws, (int **)&inv_table, &in_full, (int **)&table,
                             &out_full, &brightness, &contrast, &saturation);
    in_full = param_in_full;
    out_full = param_out_full;
    inv_table = sws_getCoefficients(in_colorspace);
    table = sws_getCoefficients(out_colorspace);
    sws_setColorspaceDetails(sws, inv_table, in_full, table, out_full,
                             brightness, contrast, saturation);

    sws_scale_frame(sws, yuv_out, yuv_in);

    printf("%s YUV %d,%d,%d -> %d,%d,%d\n", name, Y, U, V,
           yuv_out->data[0][0], yuv_out->data[1][0], yuv_out->data[2][0]);
}

int main(int argc, char* argv[])
{
    yuv2yuv(49, 109, 184, SWS_CS_ITU601, 0, SWS_CS_ITU601, 1,
            "red 601 limited -> 601 full");
    yuv2yuv(38, 106, 192, SWS_CS_ITU601, 1, SWS_CS_ITU601, 0,
            "red 601 full -> 601 limited");
    yuv2yuv(39, 115, 184, SWS_CS_ITU709, 0, SWS_CS_ITU709, 1,
            "red 709 limited -> 709 full");
    yuv2yuv(27, 113, 192, SWS_CS_ITU709, 1, SWS_CS_ITU709, 0,
            "red 709 full -> 709 limited");
    return 0;
}

[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".