From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTP id 2F8DF44CF3 for ; Tue, 15 Nov 2022 19:30:42 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id 2C79068BD14; Tue, 15 Nov 2022 21:30:40 +0200 (EET) Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 6C45568B0D3 for ; Tue, 15 Nov 2022 21:30:34 +0200 (EET) Received: by mail-yb1-f201.google.com with SMTP id 204-20020a2510d5000000b006be7970889cso14221602ybq.21 for ; Tue, 15 Nov 2022 11:30:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=ArO73+w4I/cijdCz5T5iRIfHM3ZQ9eyTXB/F7iAreyI=; b=UzwX8Aw4e1f+SSCb6Tx64FmMGYbHbzIak9vdnpE4yJYknBubV3JdWiOLlKSdnEXQpF DnfcNr1yDi9mMaH8WZdImwDbmpeHo7YSsH202c/sVCo4aDUx3n6QcuTK++UlIsJyFYth kyT1IhllOGsi4xe4bAI/KySm/ikP33P0oj7U/H5RTSa4pc5gwvFcZKa0NSPtuK1nnOJu UCZhMjJbLCIkyPcjOFdiqJ+cQKOn3JtLBfbWPaPDj2dYCyjvrP7m0uVvkN7R4q/471NJ cbBWheKEhOlChEykuIUcLhdo0Ir7meO92j8S27AL3fOVm6o2/Yhlzf4j4s5CiStDGhzn RmFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ArO73+w4I/cijdCz5T5iRIfHM3ZQ9eyTXB/F7iAreyI=; b=tDP8rHBsVgqV4l5ctnKSuzgSRRAfFRJSJpL5WQLUv0X4q2v+f944FJPCu3rIZem5JD Op69pFnjsKn+4Uqkb8G/3bBmAmo2gcWlfJZIE6W4H+hupG+0sqbd/B7rf2x9xx0G/LPM J61UCwPqltm6U/4b9c1E0THDxxPSYqFJaJK5jnF7sjMQo2HQ+L4EJWErAXUUePPOcgzZ xRVCZSvHn1b3IR1VFzrq++lr/Ro3Qa/LwV2cgwTAvo5vFG1zlA5upJQeDDpfgb72Q5VZ pT3uLR5Z25x2Ujl7kEYSnV0stUGQz+1OFlzTir36V5pokVkhqeMz56e0b9rxNIVqCzHo KA1g== X-Gm-Message-State: ACrzQf02ynTmicdBrK9v4wOaEvFD6Xdfd4iEvhqyyVCv7ftQl6mJQ+zu a+ym3HpafuJRwx1xAwJQNpYylXu29Po6UOCXpCz3SJzwpxl0IuxbqD9r1uO7CI67fH7Oi+4R7Ob c4tdAeb3oP49XPz1LXavJT0MVTgAolvBZ+/9cTUcMKuV8kJZ+eogR8I7ZlMlhrDRMEejB X-Google-Smtp-Source: AMsMyM4DiwEt/iOa1h1vcUq6YC8raNjHSsgS61GEvMacKerUhneM8jhiMznfE+Q+5jIseDt79szsKtxCdd6bEw== X-Received: from jdorfman01.cam.corp.google.com ([2620:15c:93:a:72da:426f:3662:b649]) (user=jdorfman job=sendgmr) by 2002:a25:1041:0:b0:6bd:6c15:b8f7 with SMTP id 62-20020a251041000000b006bd6c15b8f7mr66096569ybq.325.1668540631783; Tue, 15 Nov 2022 11:30:31 -0800 (PST) Date: Tue, 15 Nov 2022 14:30:23 -0500 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.493.g58b659f92b-goog Message-ID: <20221115193022.2362787-1-jdorfman@google.com> From: Jeremy Dorfman To: ffmpeg-devel@ffmpeg.org Subject: [FFmpeg-devel] [PATCH] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Jeremy Dorfman Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: Large rgb2yuv tables and high pixel values cause the intermediate int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined behavior. This causes libswscale built with LLVM -fsanitize=undefined to assert. Using unsigned integers instead has defined behavior and produces identical results, and makes rgb64ToUV_c_template match rgb64ToY_c_template. Fixes: signed integer overflow --- libswscale/input.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libswscale/input.c b/libswscale/input.c index 7ff7bfaa01..d7dbedd82f 100644 --- a/libswscale/input.c +++ b/libswscale/input.c @@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV, int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX]; av_assert1(src1==src2); for (i = 0; i < width; i++) { - int r_b = input_pixel(&src1[i*4+0]); - int g = input_pixel(&src1[i*4+1]); - int b_r = input_pixel(&src1[i*4+2]); + unsigned int r_b = input_pixel(&src1[i*4+0]); + unsigned int g = input_pixel(&src1[i*4+1]); + unsigned int b_r = input_pixel(&src1[i*4+2]); dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT; -- 2.38.1.493.g58b659f92b-goog _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".