[FFmpeg-devel] [PATCH] Bug fix attempt for #11691 - swscale: Fix heap-buffer-overflow in unscaled YUV-to-RGB conversion (PR #20864)

[FFmpeg-devel] [PATCH] Bug fix attempt for #11691 - swscale: Fix heap-buffer-overflow in unscaled YUV-to-RGB conversion (PR #20864)

[thomasdullien via ffmpeg-devel]

PR #20864 opened by thomasdullien
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20864
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20864.patch

For transparency: I am experimenting with an AI-assisted patch process, where the AI agent attempts to help
root-cause analyze a crash by means of reproducing the crash with ASAN, making a recording with 'rr', and
documenting the analysis at a granular level with verbatim quotes from the 'rr' trace. This root-cause analysis
is then iteratively reviewed (e.g. as a human reviewer I check it for accuracy and plausibility) before a patch is
generated. The process generates a detailed analysis report, an 'rr' trace that can be shared with other to help
with the verification, and a patch. Given that it is unlear how to best share the 'rr' trace, I have only attached the
detailed root-cause analysis document.

This patch attempts to address https://trac.ffmpeg.org/ticket/11691

It should be noted that I am very much not an expert in the ffmpeg codebase, so please bear with me if the proposed
patch isn't what is needed. I've re-run the test suite, and it looks very plausible to me.

--------- Description -----------------

Many YUV-to-RGB conversion functions in yuv2rgb.c process 2 scanlines
per iteration. When sws_scale() is called with insufficient source rows
available (e.g., srcSliceY=1 with only 2 total rows, leaving 1 row),
these functions would read beyond the allocated buffer bounds.

This occurred because the convert_unscaled path did not verify that at
least 2 rows were available before calling the conversion function. The
issue was particularly evident with formats like yuv422p when converting
to rgb4 or similar pixel formats.

Add bounds checking in scale_internal() before invoking convert_unscaled
to ensure at least 2 source rows remain after accounting for the slice
offset. If fewer than 2 rows are available, return 0 (no rows processed)
instead of attempting conversion, preventing the out-of-bounds access.

Fixes: https://trac.ffmpeg.org/ticket/11691

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
-------------------------------------------




From 6138573dd0364eee7051eaa1ba27cebee738bb5a Mon Sep 17 00:00:00 2001
From: "Thomas Dullien (AI assisted)" <thomas.dullien+ffmpeg@gmail.com>
Date: Sat, 8 Nov 2025 14:30:28 +0100
Subject: [PATCH] swscale: Fix heap-buffer-overflow in unscaled YUV-to-RGB
 conversion
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Many YUV-to-RGB conversion functions in yuv2rgb.c process 2 scanlines
per iteration. When sws_scale() is called with insufficient source rows
available (e.g., srcSliceY=1 with only 2 total rows, leaving 1 row),
these functions would read beyond the allocated buffer bounds.

This occurred because the convert_unscaled path did not verify that at
least 2 rows were available before calling the conversion function. The
issue was particularly evident with formats like yuv422p when converting
to rgb4 or similar pixel formats.

Add bounds checking in scale_internal() before invoking convert_unscaled
to ensure at least 2 source rows remain after accounting for the slice
offset. If fewer than 2 rows are available, return 0 (no rows processed)
instead of attempting conversion, preventing the out-of-bounds access.

Fixes: https://trac.ffmpeg.org/ticket/11691

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 libswscale/swscale.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/libswscale/swscale.c b/libswscale/swscale.c
index f4c7eccac4..7086b0f0e2 100644
--- a/libswscale/swscale.c
+++ b/libswscale/swscale.c
@@ -1140,6 +1140,23 @@ static int scale_internal(SwsContext *sws,
         int offset  = srcSliceY_internal;
         int slice_h = srcSliceH;
 
+        /* Many YUV-to-RGB conversion functions process 2 scanlines per iteration
+         * and require at least 2 rows to be available in the source buffer.
+         * Check if we have sufficient rows after considering the offset. */
+        int src_rows_remaining = sws->src_h - offset;
+        if (slice_h > src_rows_remaining) {
+            /* Clamp to available rows */
+            slice_h = src_rows_remaining;
+        }
+        if (slice_h < 2) {
+            /* Cannot safely process: conversion functions that process 2 rows
+             * per iteration would read beyond buffer bounds. Return 0 to indicate
+             * no rows were processed. This is a safety check to prevent
+             * heap-buffer-overflow in yuv2rgb.c and similar conversion functions. */
+            ret = 0;
+            goto unscaled_end;
+        }
+
         // for dst slice scaling, offset the pointers to match the unscaled API
         if (scale_dst) {
             av_assert0(offset == 0);
@@ -1160,6 +1177,7 @@ static int scale_internal(SwsContext *sws,
 
         ret = c->convert_unscaled(c, src2, srcStride2, offset, slice_h,
                                   dst2, dstStride2);
+unscaled_end:
         if (scale_dst)
             dst2[0] += dstSliceY * dstStride2[0];
     } else {
-- 
2.49.1

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org