[FFmpeg-devel] [PATCH 4/5] avcodec/apv_dsp: Avoid UB overflow in dequant

From: Michael Niedermayer <michael@niedermayer.cc>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: [FFmpeg-devel] [PATCH 4/5] avcodec/apv_dsp: Avoid UB overflow in dequant
Date: Thu,  3 Jul 2025 04:01:30 +0200
Message-ID: <20250703020131.2506428-4-michael@niedermayer.cc> (raw)
In-Reply-To: <20250703020131.2506428-1-michael@niedermayer.cc>

Fixes: signed integer overflow: 33632416 * 64 cannot be represented in type 'int'
Fixes: 421817631/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APV_fuzzer-4957386534354944

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/apv_dsp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/apv_dsp.c b/libavcodec/apv_dsp.c
index 07bb34ec0bf..8fbabcf63db 100644
--- a/libavcodec/apv_dsp.c
+++ b/libavcodec/apv_dsp.c
@@ -58,8 +58,8 @@ static void apv_decode_transquant_c(void *output,
 
         for (int y = 0; y < 8; y++) {
             for (int x = 0; x < 8; x++) {
-                int coeff = (input[y][x] * qmatrix[y][x] * (1 << qp_shift) +
-                             (1 << (bd_shift - 1))) >> bd_shift;
+                int coeff = ((int)(input[y][x] * qmatrix[y][x] * (1U << qp_shift) +
+                             (1 << (bd_shift - 1)))) >> bd_shift;
 
                 scaled_coeff[y][x] =
                     av_clip(coeff, APV_MIN_TRANS_COEFF,
-- 
2.49.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
