[FFmpeg-devel] [PR] swscale/utils: sanity check scaling increments (PR #22305)

[Alex Teaca via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

PR #22305 opened by Alex Teaca (alex-teaca)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22305
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22305.patch

Large frame sizes produce scaling increments that overflow when stored
in the c->lumXInc and c->lumYInc int variables.

The calculation is performed in int64_t but truncated to int.
Later, the incorrect values are passed to initFilter which applies
arithmetic operations that produce undefined-behavior.

This patch rejects frame sizes where the scaling increments
don't fit into int (c->lumXInc and c->lumYInc).

Fixes: https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/21588
Signed-off-by: Alex Teaca <ionutalex.teaca@gmail.com>


>From 72ca1460b24eed0b043115d89a1ae24cda1fbcfa Mon Sep 17 00:00:00 2001
From: Alex Teaca <ionutalex.teaca@gmail.com>
Date: Fri, 27 Feb 2026 11:01:33 +0200
Subject: [PATCH] swscale/utils: sanity check scaling increments

Large frame sizes produce scaling increments that overflow when stored
in the c->lumXInc and c->lumYInc int variables.

The calculation is performed in int64_t but truncated to int.
Later, the incorrect values are passed to initFilter which applies
arithmetic operations that produce undefined-behavior.

This patch rejects frame sizes where the scaling increments
don't fit into int (c->lumXInc and c->lumYInc).

Fixes: https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/21588
Signed-off-by: Alex Teaca <ionutalex.teaca@gmail.com>
---
 libswscale/utils.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/libswscale/utils.c b/libswscale/utils.c
index 52095ab2c7..ced51c6147 100644
--- a/libswscale/utils.c
+++ b/libswscale/utils.c
@@ -1128,6 +1128,7 @@ av_cold int ff_sws_init_single_context(SwsContext *sws, SwsFilter *srcFilter,
     int dstH              = sws->dst_h;
     int dst_stride        = FFALIGN(dstW * sizeof(int16_t) + 66, 16);
     int flags, cpu_flags;
+    int64_t tmpLumXInc, tmpLumYInc;
     enum AVPixelFormat srcFormat, dstFormat;
     const AVPixFmtDescriptor *desc_src;
     const AVPixFmtDescriptor *desc_dst;
@@ -1219,8 +1220,18 @@ av_cold int ff_sws_init_single_context(SwsContext *sws, SwsFilter *srcFilter,
     if (!srcFilter)
         srcFilter = &dummyFilter;
 
-    c->lumXInc      = (((int64_t)srcW << 16) + (dstW >> 1)) / dstW;
-    c->lumYInc      = (((int64_t)srcH << 16) + (dstH >> 1)) / dstH;
+    tmpLumXInc      = (((int64_t)srcW << 16) + (dstW >> 1)) / dstW;
+    tmpLumYInc      = (((int64_t)srcH << 16) + (dstH >> 1)) / dstH;
+
+    if (tmpLumXInc > INT_MAX || tmpLumYInc > INT_MAX) {
+        av_log(c, AV_LOG_ERROR, "%dx%d -> %dx%d makes invalid scaling increments\n",
+               srcW, srcH, dstW, dstH);
+        return AVERROR(EINVAL);
+    }
+
+    c->lumXInc      = tmpLumXInc;
+    c->lumYInc      = tmpLumYInc;
+
     c->dstFormatBpp = av_get_bits_per_pixel(desc_dst);
     c->srcFormatBpp = av_get_bits_per_pixel(desc_src);
     c->vRounder     = 4 * 0x0001000100010001ULL;
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org