[FFmpeg-devel] [PATCH] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template

[FFmpeg-devel] [PATCH] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template

[Jeremy Dorfman]

Large rgb2yuv tables and high pixel values cause the intermediate
int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined
behavior. This causes libswscale built with LLVM -fsanitize=undefined to
assert. Using unsigned integers instead has defined behavior and
produces identical results, and makes rgb64ToUV_c_template match
rgb64ToY_c_template.

Fixes: signed integer overflow
---
 libswscale/input.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/libswscale/input.c b/libswscale/input.c
index 7ff7bfaa01..d7dbedd82f 100644
--- a/libswscale/input.c
+++ b/libswscale/input.c
@@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV,
     int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX];
     av_assert1(src1==src2);
     for (i = 0; i < width; i++) {
-        int r_b = input_pixel(&src1[i*4+0]);
-        int   g = input_pixel(&src1[i*4+1]);
-        int b_r = input_pixel(&src1[i*4+2]);
+        unsigned int r_b = input_pixel(&src1[i*4+0]);
+        unsigned int   g = input_pixel(&src1[i*4+1]);
+        unsigned int b_r = input_pixel(&src1[i*4+2]);
 
         dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;
         dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;
-- 
2.38.1.493.g58b659f92b-goog

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

On Tue, Nov 15, 2022 at 02:30:23PM -0500, Jeremy Dorfman wrote:
> Large rgb2yuv tables and high pixel values cause the intermediate
> int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined
> behavior. This causes libswscale built with LLVM -fsanitize=undefined to
> assert. Using unsigned integers instead has defined behavior and
> produces identical results, and makes rgb64ToUV_c_template match
> rgb64ToY_c_template.
> 
> Fixes: signed integer overflow
> ---
>  libswscale/input.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

There are more cases which should be affected by this too.
I sent a more complete patchset

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Observe your enemies, for they first find out your faults. -- Antisthenes

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".