[FFmpeg-devel] [PATCH] libswscale/output: fix signed integer overflow in yuv2rgba64

[Hankang Li via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

Clamp the intermediate alpha value within 19 bits, this prevents
overflow.

Note: This issue is only reproducible with --disable-x86asm. There is an
existing issue in fate-sws-unscaled with --disable-x86asm (ticket
 #21113). This change does not introduce any new regressions.

Signed-off-by: Hankang Li <hankang201222@gmail.com>
---
 libswscale/output.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/libswscale/output.c b/libswscale/output.c
index a3b8da54bf..afb3878170 100644
--- a/libswscale/output.c
+++ b/libswscale/output.c
@@ -1288,8 +1288,8 @@ yuv2rgba64_1_c_template(SwsInternal *c, const int32_t *buf0,
             Y2 += (1 << 13) - (1 << 29);
 
             if (hasAlpha) {
-                A1 = abuf0[i * 2    ] * (1 << 11);
-                A2 = abuf0[i * 2 + 1] * (1 << 11);
+                A1 = av_clip_uintp2(abuf0[i * 2    ], 19) * (1 << 11);
+                A2 = av_clip_uintp2(abuf0[i * 2 + 1], 19) * (1 << 11);
 
                 A1 += 1 << 13;
                 A2 += 1 << 13;
@@ -1337,8 +1337,8 @@ yuv2rgba64_1_c_template(SwsInternal *c, const int32_t *buf0,
             Y2 += (1 << 13) - (1 << 29);
 
             if (hasAlpha) {
-                A1 = abuf0[i * 2    ] * (1 << 11);
-                A2 = abuf0[i * 2 + 1] * (1 << 11);
+                A1 = av_clip_uintp2(abuf0[i * 2    ], 19) * (1 << 11);
+                A2 = av_clip_uintp2(abuf0[i * 2 + 1], 19) * (1 << 11);
 
                 A1 += 1 << 13;
                 A2 += 1 << 13;
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org