From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 216F44301C for ; Tue, 15 Nov 2022 22:17:08 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 432AF68BD45; Wed, 16 Nov 2022 00:17:06 +0200 (EET) Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6C37D68B695 for ; Wed, 16 Nov 2022 00:16:59 +0200 (EET) Received: (Authenticated sender: michael@niedermayer.cc) by mail.gandi.net (Postfix) with ESMTPSA id 69EC960003; Tue, 15 Nov 2022 22:16:58 +0000 (UTC) From: Michael Niedermayer To: FFmpeg development discussions and patches Date: Tue, 15 Nov 2022 23:16:56 +0100 Message-Id: <20221115221657.23290-1-michael@niedermayer.cc> X-Mailer: git-send-email 2.17.1 Subject: [FFmpeg-devel] [PATCH 1/2] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jeremy Dorfman MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: From: Jeremy Dorfman Large rgb2yuv tables and high pixel values cause the intermediate int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined behavior. This causes libswscale built with LLVM -fsanitize=undefined to assert. Using unsigned integers instead has defined behavior and produces identical results, and makes rgb64ToUV_c_template match rgb64ToY_c_template. Fixes: signed integer overflow Signed-off-by: Michael Niedermayer --- libswscale/input.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; -- 2.17.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".