[FFmpeg-devel] [PATCH] avcodec/snowenc: sign extend a variable before shifting

[FFmpeg-devel] [PATCH] avcodec/snowenc: sign extend a variable before shifting

[James Almer]

Fixes "libavcodec/snowenc.c:718:27: runtime error: left shift of 8509032 by 8 places cannot be represented in type 'int'"
as seen in fate-vsynth2-snow-hpel under ubsan.

Signed-off-by: James Almer <jamrial@gmail.com>
---
 libavcodec/snowenc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/snowenc.c b/libavcodec/snowenc.c
index 0b87d751b8..eac81d0d7c 100644
--- a/libavcodec/snowenc.c
+++ b/libavcodec/snowenc.c
@@ -715,7 +715,7 @@ static int get_dc(SnowEncContext *enc, int mb_x, int mb_y, int plane_index)
     }
     *b= backup;
 
-    return av_clip_uint8( ROUNDED_DIV(ab<<LOG2_OBMC_MAX, aa) ); //FIXME we should not need clipping
+    return av_clip_uint8( ROUNDED_DIV((int64_t)ab<<LOG2_OBMC_MAX, aa) ); //FIXME we should not need clipping
 }
 
 static inline int get_block_bits(SnowContext *s, int x, int y, int w){
-- 
2.46.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".