[FFmpeg-devel] [PATCH v2] avcodec/libvpxenc: fix sRGB colorspace for non-RGB pixel formats

[Guangyu Sun via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

When encoding VP9 with a YUV pixel format (e.g. yuv420p) and
AVCOL_SPC_RGB colorspace metadata, libvpxenc unconditionally set
VPX_CS_SRGB. This produced a spec-violating bitstream: Profile 0
(4:2:0) with sRGB colorspace, which is only valid for Profile 1/3
(4:4:4). The resulting file is undecodable.

Fix this by setting ctx->vpx_cs to VPX_CS_SRGB in set_pix_fmt()
for 4:4:4 YUV formats when AVCOL_SPC_RGB is set, matching the
existing GBRP path. This covers the legitimate case of RGB data in
YUV444 containers (e.g. H.264 High 4:4:4 with identity matrix).

With this change, any AVCOL_SPC_RGB that reaches the switch in
set_colorspace() is guaranteed to be a subsampled format where
sRGB is invalid. Log a warning and fall back to VPX_CS_BT_709.

To reproduce:

  # generate a bad source
  ffmpeg -f lavfi -i testsrc=s=64x64:d=1:r=1 \
    -pix_fmt yuv420p -colorspace rgb bad.mp4
  # transcode with default parameters
  ffmpeg -i bad.mp4 bad.webm
  # check decoding
  ffmpeg -i bad.webm -f null -
  # -> 0 frames decoded, error

Signed-off-by: Guangyu Sun <gsun@roblox.com>
---
v2:
  - Only fall back to BT.709 if the pixel format is not 4:4:4,
    addressing feedback regarding Profile 1/3 support.

 libavcodec/libvpxenc.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/libavcodec/libvpxenc.c b/libavcodec/libvpxenc.c
index 082709c41f..fd80d7ca04 100644
--- a/libavcodec/libvpxenc.c
+++ b/libavcodec/libvpxenc.c
@@ -835,6 +835,8 @@ static int set_pix_fmt(AVCodecContext *avctx, vpx_codec_caps_t codec_caps,
     case AV_PIX_FMT_GBRP:
         ctx->vpx_cs = VPX_CS_SRGB;
     case AV_PIX_FMT_YUV444P:
+        if (avctx->colorspace == AVCOL_SPC_RGB)
+            ctx->vpx_cs = VPX_CS_SRGB;
         enccfg->g_profile = 1;
         *img_fmt = VPX_IMG_FMT_I444;
         return 0;
@@ -870,6 +872,8 @@ static int set_pix_fmt(AVCodecContext *avctx, vpx_codec_caps_t codec_caps,
         ctx->vpx_cs = VPX_CS_SRGB;
     case AV_PIX_FMT_YUV444P10:
     case AV_PIX_FMT_YUV444P12:
+        if (avctx->colorspace == AVCOL_SPC_RGB)
+            ctx->vpx_cs = VPX_CS_SRGB;
         if (codec_caps & VPX_CODEC_CAP_HIGHBITDEPTH) {
             enccfg->g_profile = 3;
             *img_fmt = VPX_IMG_FMT_I44416;
@@ -893,7 +897,17 @@ static void set_colorspace(AVCodecContext *avctx)
         vpx_cs = ctx->vpx_cs;
     } else {
         switch (avctx->colorspace) {
-        case AVCOL_SPC_RGB:         vpx_cs = VPX_CS_SRGB;      break;
+        case AVCOL_SPC_RGB:
+            // All sRGB-compatible formats (GBRP*, and YUV444* with
+            // AVCOL_SPC_RGB) set ctx->vpx_cs in set_pix_fmt() and
+            // take the branch above, so reaching here means a
+            // subsampled format incompatible with RGB colorspace.
+            av_log(avctx, AV_LOG_WARNING,
+                   "RGB colorspace is not compatible with pixel format %s, "
+                   "using BT.709 instead.\n",
+                   av_get_pix_fmt_name(avctx->pix_fmt));
+            vpx_cs = VPX_CS_BT_709;
+            break;
         case AVCOL_SPC_BT709:       vpx_cs = VPX_CS_BT_709;    break;
         case AVCOL_SPC_UNSPECIFIED: vpx_cs = VPX_CS_UNKNOWN;   break;
         case AVCOL_SPC_RESERVED:    vpx_cs = VPX_CS_RESERVED;  break;
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org