[FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Le Bao Tin Ha]

Dear FFmpeg Community,

I hope this email finds you well. I am currently working on the Toradex Apalis iMX8QM board and I am trying to perform a hardware decode using the Amphion driver with ffmpeg h264_v4l2m2m codec. The decoder outputs a pixel format (NV12_8L128, a linear 8x128 NV12 based tiles) that is not supported yet by FFmpeg.

I attempted to add this new pixel format to FFmpeg by adding only the DEFINES which are similar to the classical NV12 (or NV12M), but it doesn't seem to be working as expected. I was wondering if you could help me with this issue. Could you provide me with some guidance on how to proceed and make sure that it works correctly? Or could you provide me an other alternative to simplify this?

Any help would be greatly appreciated. Thank you for your time.

Best regards,
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Kieran Kunhya]

On Mon, 6 Feb 2023 at 15:43, Le Bao Tin Ha <lebaotin.ha@savoirfairelinux.com>
wrote:

> I hope this email finds you well. I am currently working on the Toradex
> Apalis iMX8QM board and I am trying to perform a hardware decode using the
> Amphion driver with ffmpeg h264_v4l2m2m codec. The decoder outputs a pixel
> format (NV12_8L128, a linear 8x128 NV12 based tiles) that is not supported
> yet by FFmpeg.
>

Hello,

Are you able to provide more information about this nonstandard pixel
format?

Kieran
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Le Bao Tin Ha]

Hello, you can found the explanation here:

https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html

"V4L2_PIX_FMT_NV12M_8L128 is similar to V4L2_PIX_FMT_NV12M but stores pixels in 2D 8x128 tiles, and stores tiles linearly in memory. The image height must be aligned to a multiple of 128. The layouts of the luma and chroma planes are identical.

V4L2_PIX_FMT_NV12_8L128 is similar to V4L2_PIX_FMT_NV12M_8L128 but stores two planes in one memory."


----- Original Message -----
From: "Kieran Kunhya" <kierank@obe.tv>
To: "FFmpeg development discussions and patches" <ffmpeg-devel@ffmpeg.org>
Sent: Monday, February 6, 2023 7:15:22 PM
Subject: Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

On Mon, 6 Feb 2023 at 15:43, Le Bao Tin Ha <lebaotin.ha@savoirfairelinux.com>
wrote:

> I hope this email finds you well. I am currently working on the Toradex
> Apalis iMX8QM board and I am trying to perform a hardware decode using the
> Amphion driver with ffmpeg h264_v4l2m2m codec. The decoder outputs a pixel
> format (NV12_8L128, a linear 8x128 NV12 based tiles) that is not supported
> yet by FFmpeg.
>

Hello,

Are you able to provide more information about this nonstandard pixel
format?

Kieran
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Jean-Baptiste Kempf]

On Wed, 8 Feb 2023, at 10:47, Le Bao Tin Ha wrote:
> Hello, you can found the explanation here:
>
> https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html
>
> "V4L2_PIX_FMT_NV12M_8L128 is similar to V4L2_PIX_FMT_NV12M but stores 
> pixels in 2D 8x128 tiles, and stores tiles linearly in memory. The 
> image height must be aligned to a multiple of 128. The layouts of the 
> luma and chroma planes are identical.
>
> V4L2_PIX_FMT_NV12_8L128 is similar to V4L2_PIX_FMT_NV12M_8L128 but 
> stores two planes in one memory."

Do we really need those PIX_FMT in FFmpeg public API?

-- 
Jean-Baptiste Kempf -  President
+33 672 704 734
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Hendrik Leppkes]

On Wed, Feb 8, 2023 at 2:08 PM Jean-Baptiste Kempf <jb@videolan.org> wrote:
>
> On Wed, 8 Feb 2023, at 10:47, Le Bao Tin Ha wrote:
> > Hello, you can found the explanation here:
> >
> > https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html
> >
> > "V4L2_PIX_FMT_NV12M_8L128 is similar to V4L2_PIX_FMT_NV12M but stores
> > pixels in 2D 8x128 tiles, and stores tiles linearly in memory. The
> > image height must be aligned to a multiple of 128. The layouts of the
> > luma and chroma planes are identical.
> >
> > V4L2_PIX_FMT_NV12_8L128 is similar to V4L2_PIX_FMT_NV12M_8L128 but
> > stores two planes in one memory."
>
> Do we really need those PIX_FMT in FFmpeg public API?
>

Tiled formats also really don't fit into our format definitions, and
would not work with any generic pixel format handling code, as far as
I can tell.

- Hendrik
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Devin Heitmueller]

On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf <jb@videolan.org> wrote:
> Do we really need those PIX_FMT in FFmpeg public API?

Tiled formats are annoying, but not uncommon in popular embedded
platforms (e.g. NXP iMX, TI Davinci).  Of course ffmpeg could choose
to ignore them, but it would effectively prevent it from being used on
those platforms (and it's pretty essential to use the hardware blocks
to do any real video processing).

In the past there was an expectation that adding new pixel formats
required the contributor to also add corresponding routines to swscale
to convert to more common formats that ffmpeg works with.  You could
alternatively treat it as a packet format and require a decoder to do
the conversion to something more common to ffmpeg, but this would both
be expensive (in terms of CPU usage and memory copying) and
effectively prevent you from being able to capture or decode in the
native format and then hand off those buffers to other hardware blocks
like encoders, colorspace conversion, etc.

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller@ltnglobal.com
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Rémi Denis-Courmont]

Le keskiviikkona 8. helmikuuta 2023, 16.27.52 EET Devin Heitmueller a écrit :
> On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf <jb@videolan.org> wrote:
> > Do we really need those PIX_FMT in FFmpeg public API?
> 
> Tiled formats are annoying, but not uncommon in popular embedded
> platforms (e.g. NXP iMX, TI Davinci). 

Uh, not to deny that, but tiled formats are in no way an embedded thing, as 
you imply. They are also used by desktop GPUs. And that is not really 
relevant, I would argue, to the discussion, either way.

> Of course ffmpeg could choose
> to ignore them, but it would effectively prevent it from being used on
> those platforms (and it's pretty essential to use the hardware blocks
> to do any real video processing).

Insofar as the format is *only* used by a single module, I don't see the need 
to assign it a pixel format in libavutil. You could just as well define a 
generic V4L format that would be used for all weird V4L formats that nothing 
other than V4L code understands.

Ditto tiled DRM pixel formats for that matter. And while V4L is maybe not 
there yet, DRM has introduced format modifiers that anyway preclude any simple 
enumeration. My point being, there needs to be a way to further specify 
framework-specific formats outside of the avutil pixel format list, in any 
case.

-- 
レミ・デニ-クールモン
http://www.remlab.net/



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Devin Heitmueller]

Hi Remi,

On Wed, Feb 8, 2023 at 12:38 PM Rémi Denis-Courmont <remi@remlab.net> wrote:
> Uh, not to deny that, but tiled formats are in no way an embedded thing, as
> you imply. They are also used by desktop GPUs. And that is not really
> relevant, I would argue, to the discussion, either way.

I don't disagree that tiled formats can be found on desktop GPUs.  My
only point was that the importance of being able to pass through the
native format is much greater with embedded SOCs since in many cases
on desktops you have enough CPU power to get away with doing a
conversion.

Also, you're more likely to find that desktop GPUs which support tiled
formats also support formats that are likely to be supported by ffmpeg
today.  This is less likely to be the case with embedded SOCs.

> > Of course ffmpeg could choose
> > to ignore them, but it would effectively prevent it from being used on
> > those platforms (and it's pretty essential to use the hardware blocks
> > to do any real video processing).
>
> Insofar as the format is *only* used by a single module, I don't see the need
> to assign it a pixel format in libavutil. You could just as well define a
> generic V4L format that would be used for all weird V4L formats that nothing
> other than V4L code understands.

In many cases the lack of support for a pixel format may be the reason
that more platforms aren't supported.  Somebody has got to be the
first.  Suggesting that it's only used by a single module may be the
effect of not supporting the pixel format, not the other way around.

This is not to suggest that like most developers I don't hate the fact
that there are so many different pixel formats out there, and with
each new format proposed for addition to ffmpeg we have the same
arguments.

> Ditto tiled DRM pixel formats for that matter. And while V4L is maybe not
> there yet, DRM has introduced format modifiers that anyway preclude any simple
> enumeration. My point being, there needs to be a way to further specify
> framework-specific formats outside of the avutil pixel format list, in any
> case.

I can't really offer an opinion on the implementation detail regarding
the avutil pixel format list.  I suspect the OP is open to alternative
approaches as long as the original requirement is met.

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller@ltnglobal.com
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Request for assistance with adding new pixel format (NV12_8L128) in FFmpeg

[Philip Langdale]

On Wed, 08 Feb 2023 19:38:07 +0200
Rémi Denis-Courmont <remi@remlab.net> wrote:

> Le keskiviikkona 8. helmikuuta 2023, 16.27.52 EET Devin Heitmueller a
> écrit :
> > On Wed, Feb 8, 2023 at 8:08 AM Jean-Baptiste Kempf
> > <jb@videolan.org> wrote:  
> > > Do we really need those PIX_FMT in FFmpeg public API?  
> > 
> > Tiled formats are annoying, but not uncommon in popular embedded
> > platforms (e.g. NXP iMX, TI Davinci).   
> 
> Uh, not to deny that, but tiled formats are in no way an embedded
> thing, as you imply. They are also used by desktop GPUs. And that is
> not really relevant, I would argue, to the discussion, either way.
> 
> > Of course ffmpeg could choose
> > to ignore them, but it would effectively prevent it from being used
> > on those platforms (and it's pretty essential to use the hardware
> > blocks to do any real video processing).  
> 
> Insofar as the format is *only* used by a single module, I don't see
> the need to assign it a pixel format in libavutil. You could just as
> well define a generic V4L format that would be used for all weird V4L
> formats that nothing other than V4L code understands.
> 
> Ditto tiled DRM pixel formats for that matter. And while V4L is maybe
> not there yet, DRM has introduced format modifiers that anyway
> preclude any simple enumeration. My point being, there needs to be a
> way to further specify framework-specific formats outside of the
> avutil pixel format list, in any case.
> 

I never thought about it this way until you mentioned DRM but I think
that's the key insight.

If we want to take a leaf out of the DRM book (and wouldn't it be
wonderful if these m2m decoders used DRM formats?), then this format
would NV12 at the pix fmt layer with a modifier indicating the tiling
pattern. When we interact with these kinds of formats in VAAPI, for
example, VAAPI encapsulates the heavy lifting required to transfer
to/from system memory successfully, and when we are doing everything on
the GPU side, we pass the surface long with the modifier to whatever
eventually consumes it (eg: Display with GL or Vulkan, or encoding). In
none of these cases does ffmpeg code really need to care about the
modifier. Maybe we've been thinking about these m2m formats the wrong
way.

--phil
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".