[FFmpeg-devel] [PATCHv3] swscale: prevent undefined behaviour in the PUTRGBA macro

From: Sean McGovern <gseanmcg@gmail.com>
To: ffmpeg-devel@ffmpeg.org
Subject: [FFmpeg-devel] [PATCHv3] swscale: prevent undefined behaviour in the PUTRGBA macro
Date: Tue,  9 Jul 2024 14:34:34 -0400
Message-ID: <20240709183434.10114-1-gseanmcg@gmail.com> (raw)
In-Reply-To: <20240708042517.5585-1-gseanmcg@gmail.com>

For even small values of 'asrc', shifting them by 24 bits or more
will cause arithmetic overflow and be caught by
GCC's undefined behaviour sanitizer.

Ensure the values do not overflow by up-casting the bracketed
expressions involving 'asrc' to int32_t.
---
 libswscale/yuv2rgb.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libswscale/yuv2rgb.c b/libswscale/yuv2rgb.c
index 977eb3a7dd..ac0b811f61 100644
--- a/libswscale/yuv2rgb.c
+++ b/libswscale/yuv2rgb.c
@@ -100,9 +100,9 @@ const int *sws_getCoefficients(int colorspace)
 
 #define PUTRGBA(dst, ysrc, asrc, i, abase)                              \
     Y              = ysrc[2 * i];                                       \
-    dst[2 * i]     = r[Y] + g[Y] + b[Y] + (asrc[2 * i]     << abase);   \
+    dst[2 * i]     = r[Y] + g[Y] + b[Y] + ((int32_t)(asrc[2 * i])     << abase);   \
     Y              = ysrc[2 * i + 1];                                   \
-    dst[2 * i + 1] = r[Y] + g[Y] + b[Y] + (asrc[2 * i + 1] << abase);
+    dst[2 * i + 1] = r[Y] + g[Y] + b[Y] + ((int32_t)(asrc[2 * i + 1]) << abase);
 
 #define PUTRGB48(dst, src, asrc, i, abase)          \
     Y                = src[ 2 * i];                 \
-- 
2.39.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
