[FFmpeg-devel] PATCH: Force vaapi image formats to NV12-only

[FFmpeg-devel] PATCH: Force vaapi image formats to NV12-only

[Lluís Batlle i Rossell]

[-- Attachment #1: Type: text/plain, Size: 9 bytes --]

attached

[-- Attachment #2: 0001-Force-vaapi-image-formats-to-NV12-only.patch --]
[-- Type: text/plain, Size: 1513 bytes --]

From c6439f3a74529db25777029596791a62eb3c77d5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Llu=C3=ADs=20Batlle=20i=20Rossell?= <viric@viric.name>
Date: Thu, 8 Aug 2024 20:32:03 +0200
Subject: [PATCH] Force vaapi image formats to NV12-only

Vaapi drivers often lack proper image converesions and not all
situations allow vaGetImage or vaPutImage with the image formats
reported by the API. NV12 seems allowed in all circumstances.

With this change now one can use the hwaccel directly without
explicit conversions to nv12 for frame downloading to work.

gstreamer adopted a similar approach:
https://bugzilla.gnome.org/show_bug.cgi?id=752958
---
 libavutil/hwcontext_vaapi.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 12bc95119a..d678e58d07 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -418,7 +418,12 @@ static int vaapi_device_init(AVHWDeviceContext *hwdev)
     for (i = 0; i < image_count; i++) {
         fourcc  = image_list[i].fourcc;
         pix_fmt = vaapi_pix_fmt_from_fourcc(fourcc);
-        if (pix_fmt == AV_PIX_FMT_NONE) {
+        if (pix_fmt != AV_PIX_FMT_NV12) {
+            av_log(hwdev, AV_LOG_DEBUG, "Format %#x -> ignored.\n",
+                   fourcc);
+            continue;
+        }
+        else if (pix_fmt == AV_PIX_FMT_NONE) {
             av_log(hwdev, AV_LOG_DEBUG, "Format %#x -> unknown.\n",
                    fourcc);
         } else {
-- 
2.44.1


[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] PATCH: Force vaapi image formats to NV12-only

[Zhao Zhili]

> vaapi drivers often lack proper image converesions and not all
> situations allow vagetimage or vaputimage with the image formats
> reported by the api. nv12 seems allowed in all circumstances.
> 
> with this change now one can use the hwaccel directly without
> explicit conversions to nv12 for frame downloading to work.
> 
> gstreamer adopted a similar approach:
> https://bugzilla.gnome.org/show_bug.cgi?id=752958
> ---
>  libavutil/hwcontext_vaapi.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> index 12bc95119a..d678e58d07 100644
> --- a/libavutil/hwcontext_vaapi.c
> +++ b/libavutil/hwcontext_vaapi.c
> @@ -418,7 +418,12 @@ static int vaapi_device_init(avhwdevicecontext *hwdev)
>      for (i = 0; i < image_count; i++) {
>          fourcc  = image_list[i].fourcc;
>          pix_fmt = vaapi_pix_fmt_from_fourcc(fourcc);
> -        if (pix_fmt == av_pix_fmt_none) {
> +        if (pix_fmt != av_pix_fmt_nv12) {
> +            av_log(hwdev, av_log_debug, "format %#x -> ignored.\n",
> +                   fourcc);
> +            continue;
> +        }
> +        else if (pix_fmt == av_pix_fmt_none) {
>              av_log(hwdev, av_log_debug, "format %#x -> unknown.\n",
>                     fourcc);
>          } else {
> -- 
> 2.44.1

Isn’t it break all pixel formats with bit depth > 8?
I think we already have hwcontext API to select sw_format, this isn’t a bug
inside ffmpeg.

> On Aug 9, 2024, at 02:37, Lluís Batlle i Rossell <viric@viric.name> wrote:
> 
> attached
> <0001-Force-vaapi-image-formats-to-NV12-only.patch>_______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] PATCH: Force vaapi image formats to NV12-only

[Lluís Batlle i Rossell]

On Fri, Aug 09, 2024 at 11:49:54AM +0800, Zhao Zhili wrote:
> > vaapi drivers often lack proper image converesions and not all
> > situations allow vagetimage or vaputimage with the image formats
> > reported by the api. nv12 seems allowed in all circumstances.
> > 
> > with this change now one can use the hwaccel directly without
> > explicit conversions to nv12 for frame downloading to work.
> > 
> > gstreamer adopted a similar approach:
> > https://bugzilla.gnome.org/show_bug.cgi?id=752958
> 
> Isn’t it break all pixel formats with bit depth > 8?
> I think we already have hwcontext API to select sw_format, this isn’t a bug
> inside ffmpeg.

Correct... I didn't think of the need beyond NV12.

What if I redo the patch so I keep all formats, but I simply move NV12 to
the first place? That will make ffmpeg pick NV12 as default if NONE
specified.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] PATCH: Fallback to NV12 format in VAAPI drivers

[Lluís Batlle i Rossell]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

On Fri, Aug 09, 2024 at 09:43:53AM +0200, Lluís Batlle i Rossell wrote:
> On Fri, Aug 09, 2024 at 11:49:54AM +0800, Zhao Zhili wrote:
> > > vaapi drivers often lack proper image converesions and not all
> > > situations allow vagetimage or vaputimage with the image formats
> > > reported by the api. nv12 seems allowed in all circumstances.
> > > 
> > > with this change now one can use the hwaccel directly without
> > > explicit conversions to nv12 for frame downloading to work.
> > > 
> > > gstreamer adopted a similar approach:
> > > https://bugzilla.gnome.org/show_bug.cgi?id=752958
> > 
> > Isn’t it break all pixel formats with bit depth > 8?
> > I think we already have hwcontext API to select sw_format, this isn’t a bug
> > inside ffmpeg.
> 
> Correct... I didn't think of the need beyond NV12.
> 
> What if I redo the patch so I keep all formats, but I simply move NV12 to
> the first place? That will make ffmpeg pick NV12 as default if NONE
> specified.

I attach a different patch, so NV12 is only picked in case the dst format
is NONE.

[-- Attachment #2: 0001-Fallback-to-NV12-format-in-VAAPI-drivers.patch --]
[-- Type: text/plain, Size: 1897 bytes --]

From c633b6ba0975bd495df2479aa9a562958bb87e59 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Llu=C3=ADs=20Batlle=20i=20Rossell?= <viric@viric.name>
Date: Sat, 10 Aug 2024 10:45:14 +0200
Subject: [PATCH] Fallback to NV12 format in VAAPI drivers

Even if the hw format is listed in the supported ImageFormats. That's
because some drivers fail in vaGetImage even if requesting the transfer
in a format equal to the hw format.

NV12 is chosen only if the users don't specify any other particular
format.

gstreamer adopted a similar approach:
https://bugzilla.gnome.org/show_bug.cgi?id=752958
---
 libavutil/hwcontext_vaapi.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 12bc95119a..0396038017 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -720,20 +720,22 @@ static int vaapi_transfer_get_formats(AVHWFramesContext *hwfc,
 {
     VAAPIDeviceContext *ctx = hwfc->device_ctx->internal->priv;
     enum AVPixelFormat *pix_fmts;
-    int i, k, sw_format_available;
+    int i, k, nv12_format_available;
 
-    sw_format_available = 0;
+    /* Intel VAAPI drivers seem to support NV12 for vaGetImage,
+     * but fail for many formats announced in vaQueryImageFormats */
+    nv12_format_available = 0;
     for (i = 0; i < ctx->nb_formats; i++) {
-        if (ctx->formats[i].pix_fmt == hwfc->sw_format)
-            sw_format_available = 1;
+        if (ctx->formats[i].pix_fmt == AV_PIX_FMT_NV12)
+            nv12_format_available = 1;
     }
 
     pix_fmts = av_malloc((ctx->nb_formats + 1) * sizeof(*pix_fmts));
     if (!pix_fmts)
         return AVERROR(ENOMEM);
 
-    if (sw_format_available) {
-        pix_fmts[0] = hwfc->sw_format;
+    if (nv12_format_available) {
+        pix_fmts[0] = AV_PIX_FMT_NV12;
         k = 1;
     } else {
         k = 0;
-- 
2.44.1


[-- Attachment #3: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] PATCH: Fallback to NV12 format in VAAPI drivers

[Mark Thompson]

On 10/08/2024 09:51, Lluís Batlle i Rossell wrote:
> On Fri, Aug 09, 2024 at 09:43:53AM +0200, Lluís Batlle i Rossell wrote:
>> On Fri, Aug 09, 2024 at 11:49:54AM +0800, Zhao Zhili wrote:
>>>> vaapi drivers often lack proper image converesions and not all
>>>> situations allow vagetimage or vaputimage with the image formats
>>>> reported by the api. nv12 seems allowed in all circumstances.
>>>>
>>>> with this change now one can use the hwaccel directly without
>>>> explicit conversions to nv12 for frame downloading to work.
>>>>
>>>> gstreamer adopted a similar approach:
>>>> https://bugzilla.gnome.org/show_bug.cgi?id=752958
>>>
>>> Isn’t it break all pixel formats with bit depth > 8?
>>> I think we already have hwcontext API to select sw_format, this isn’t a bug
>>> inside ffmpeg.
>>
>> Correct... I didn't think of the need beyond NV12.
>>
>> What if I redo the patch so I keep all formats, but I simply move NV12 to
>> the first place? That will make ffmpeg pick NV12 as default if NONE
>> specified.
> 
> I attach a different patch, so NV12 is only picked in case the dst format
> is NONE.

What are the surface formats where this actually gets used, and on what hardware and driver?

It seems probably ok if this were restricted to 4:2:0 8-bit formats (as a surprise implicit downsample which can't be told anything about the source format seems very bad), but then what is it actually covering?

Thanks,

- Mark
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] PATCH: Fallback to NV12 format in VAAPI drivers

[Lluís Batlle i Rossell]

On Sun, Aug 11, 2024 at 06:57:49PM +0100, Mark Thompson wrote:
> On 10/08/2024 09:51, Lluís Batlle i Rossell wrote:
> > On Fri, Aug 09, 2024 at 09:43:53AM +0200, Lluís Batlle i Rossell wrote:
> >> On Fri, Aug 09, 2024 at 11:49:54AM +0800, Zhao Zhili wrote:
> >>>> vaapi drivers often lack proper image converesions and not all
> >>>> situations allow vagetimage or vaputimage with the image formats
> >>>> reported by the api. nv12 seems allowed in all circumstances.
> >>>>
> >>>> with this change now one can use the hwaccel directly without
> >>>> explicit conversions to nv12 for frame downloading to work.
> >>>>
> >>>> gstreamer adopted a similar approach:
> >>>> https://bugzilla.gnome.org/show_bug.cgi?id=752958
> >>>
> >>> Isn’t it break all pixel formats with bit depth > 8?
> >>> I think we already have hwcontext API to select sw_format, this isn’t a bug
> >>> inside ffmpeg.
> >>
> >> Correct... I didn't think of the need beyond NV12.
> >>
> >> What if I redo the patch so I keep all formats, but I simply move NV12 to
> >> the first place? That will make ffmpeg pick NV12 as default if NONE
> >> specified.
> > 
> > I attach a different patch, so NV12 is only picked in case the dst format
> > is NONE.
> 
> What are the surface formats where this actually gets used, and on what hardware and driver?
> 
> It seems probably ok if this were restricted to 4:2:0 8-bit formats (as a surprise implicit downsample which can't be told anything about the source format seems very bad), but then what is it actually covering?
> 

I have found that webcams doing mjpeg get the hw frames as yuv422, and
despite yuv422 is announced in ImageFormats, vaGetImage cannot download
it, with a pair of Intel cpus I tried. I looked at the driver code and
it looked broad.

So with the patches I'm exploring what can be a good approach to this
problem: va drivers report formats that maybe work through vaPutImage, but
not with vaGetImage, for what I've seen.

The immediate user problem of that is that operations using hwaccel vaapi
with mjpeg will end up failing, unless you specify an output format that
actually works. Otherwise, the vaGetImage operation says "not
implemented". So mjpeg hw decoding will not work unless
"-hwaccel_output_format" is given, or proper format is given to hwdownload
in filters.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".