[FFmpeg-devel] Re: [PATCH v2] avcodec/libvpxenc: fix sRGB colorspace for non-RGB pixel formats

[Guangyu Sun via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

On Tue, Feb 24, 2026 at 10:34 AM James Zern via ffmpeg-devel
<ffmpeg-devel@ffmpeg.org> wrote:
>
> On Mon, Feb 23, 2026 at 4:26 PM Guangyu Sun via ffmpeg-devel
> <ffmpeg-devel@ffmpeg.org> wrote:
> >
> > When encoding VP9 with a YUV pixel format (e.g. yuv420p) and
> > AVCOL_SPC_RGB colorspace metadata, libvpxenc unconditionally set
> > VPX_CS_SRGB. This produced a spec-violating bitstream: Profile 0
> > (4:2:0) with sRGB colorspace, which is only valid for Profile 1/3
> > (4:4:4). The resulting file is undecodable.
> >
> > Fix this by setting ctx->vpx_cs to VPX_CS_SRGB in set_pix_fmt()
> > for 4:4:4 YUV formats when AVCOL_SPC_RGB is set, matching the
> > existing GBRP path. This covers the legitimate case of RGB data in
> > YUV444 containers (e.g. H.264 High 4:4:4 with identity matrix).
> >
> > With this change, any AVCOL_SPC_RGB that reaches the switch in
> > set_colorspace() is guaranteed to be a subsampled format where
> > sRGB is invalid. Log a warning and fall back to VPX_CS_BT_709.
> >
> > To reproduce:
> >
> >   # generate a bad source
> >   ffmpeg -f lavfi -i testsrc=s=64x64:d=1:r=1 \
> >     -pix_fmt yuv420p -colorspace rgb bad.mp4
> >   # transcode with default parameters
> >   ffmpeg -i bad.mp4 bad.webm
> >   # check decoding
> >   ffmpeg -i bad.webm -f null -
> >   # -> 0 frames decoded, error
> >
> > Signed-off-by: Guangyu Sun <gsun@roblox.com>
> > ---
> > v2:
> >   - Only fall back to BT.709 if the pixel format is not 4:4:4,
> >     addressing feedback regarding Profile 1/3 support.
> >
> >  libavcodec/libvpxenc.c | 16 +++++++++++++++-
> >  1 file changed, 15 insertions(+), 1 deletion(-)
> >
> > [...]
> >          switch (avctx->colorspace) {
> > -        case AVCOL_SPC_RGB:         vpx_cs = VPX_CS_SRGB;      break;
> > +        case AVCOL_SPC_RGB:
> > +            // All sRGB-compatible formats (GBRP*, and YUV444* with
> > +            // AVCOL_SPC_RGB) set ctx->vpx_cs in set_pix_fmt() and
> > +            // take the branch above, so reaching here means a
> > +            // subsampled format incompatible with RGB colorspace.
> > +            av_log(avctx, AV_LOG_WARNING,
> > +                   "RGB colorspace is not compatible with pixel format %s, "
> > +                   "using BT.709 instead.\n",
> > +                   av_get_pix_fmt_name(avctx->pix_fmt));
> > +            vpx_cs = VPX_CS_BT_709;
>
> I don't think an arbitrary fallback is the best option here. The
> library should have rejected the configuration (which will be fixed),
> but in practice if the colorspace is RGB and there's no conversion
> being done, then this may not be what the user intended. I think it
> would be better to surface this as an error.

Thanks for the feedback. Will try to submit a patch to libvpx.
Meanwhile, do you think it is still worth returning an error here in
ffmpeg? I can send out a v3 if preferred.
>
> > +            break;
> >          case AVCOL_SPC_BT709:       vpx_cs = VPX_CS_BT_709;    break;
> >          case AVCOL_SPC_UNSPECIFIED: vpx_cs = VPX_CS_UNKNOWN;   break;
> >          case AVCOL_SPC_RESERVED:    vpx_cs = VPX_CS_RESERVED;  break;
> > --
> > 2.52.0
> >
> > _______________________________________________
> > ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
> > To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org
> _______________________________________________
> ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
> To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org
_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org