[FFmpeg-devel] [PATCH] avcodec/x86/mpegvideoencdsp_init: Fix left shift of negative number (PR #20733)

[FFmpeg-devel] [PATCH] avcodec/x86/mpegvideoencdsp_init: Fix left shift of negative number (PR #20733)

[mkver via ffmpeg-devel]

PR #20733 opened by mkver
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20733
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20733.patch

Uncovered by UBSan when running the mpegvideoencdsp checkasm
test.

Will apply this tonight unless there are objections.


>From 05b8608c76e76b7d8b4ce5e86e0940244fbb737e Mon Sep 17 00:00:00 2001
From: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Date: Tue, 21 Oct 2025 12:11:55 +0200
Subject: [PATCH] avcodec/x86/mpegvideoencdsp_init: Fix left shift of negative
 number

Uncovered by UBSan when running the mpegvideoencdsp checkasm
test.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
---
 libavcodec/x86/mpegvideoencdsp_init.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/x86/mpegvideoencdsp_init.c b/libavcodec/x86/mpegvideoencdsp_init.c
index 3cd16fefbf..bf5b722016 100644
--- a/libavcodec/x86/mpegvideoencdsp_init.c
+++ b/libavcodec/x86/mpegvideoencdsp_init.c
@@ -42,7 +42,7 @@ static int try_8x8basis_ssse3(const int16_t rem[64], const int16_t weight[64], c
     x86_reg i=0;
 
     av_assert2(FFABS(scale) < MAX_ABS);
-    scale <<= 16 + SCALE_OFFSET - BASIS_SHIFT + RECON_SHIFT;
+    scale *= 1 << (16 + SCALE_OFFSET - BASIS_SHIFT + RECON_SHIFT);
 
     __asm__ volatile(
         "pxor            %%xmm2, %%xmm2     \n\t"
@@ -87,7 +87,7 @@ static void add_8x8basis_ssse3(int16_t rem[64], const int16_t basis[64], int sca
     x86_reg i=0;
 
     if (FFABS(scale) < 1024) {
-        scale <<= 16 + SCALE_OFFSET - BASIS_SHIFT + RECON_SHIFT;
+        scale *= 1 << (16 + SCALE_OFFSET - BASIS_SHIFT + RECON_SHIFT);
         __asm__ volatile(
                 "movd                %3, %%xmm2     \n\t"
                 "punpcklwd       %%xmm2, %%xmm2     \n\t"
-- 
2.49.1

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org