From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ffmpeg-devel-bounces@ffmpeg.org>
Received: from ffbox0-bg.mplayerhq.hu (ffbox0-bg.ffmpeg.org [79.124.17.100])
	by master.gitmailbox.com (Postfix) with ESMTP id 4B20F40F19
	for <ffmpegdev@gitmailbox.com>; Thu, 17 Feb 2022 10:03:33 +0000 (UTC)
Received: from [127.0.1.1] (localhost [127.0.0.1])
	by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTP id CDB4F68B1CC;
	Thu, 17 Feb 2022 12:03:32 +0200 (EET)
Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com
 [209.85.221.73])
 by ffbox0-bg.mplayerhq.hu (Postfix) with ESMTPS id 1821E68B07E
 for <ffmpeg-devel@ffmpeg.org>; Thu, 17 Feb 2022 12:03:26 +0200 (EET)
Received: by mail-wr1-f73.google.com with SMTP id
 j8-20020adfc688000000b001e3322ced69so2080175wrg.13
 for <ffmpeg-devel@ffmpeg.org>; Thu, 17 Feb 2022 02:03:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112;
 h=date:message-id:mime-version:subject:from:to:cc;
 bh=z2bT2fPLvDXm891Tc56t27V5/fusQLjdKu3PLiPsC9I=;
 b=cD+B5V4Jlk8C0MotON8MDeKi3uPjVVU2yi2DKl0vaHkp/jis7o1dK0JYGGOLRJtvEv
 9XYag0y4igO6A5pCtdEX9yKfJ8mE45g23DsMvTBGyVVhvc4dQ0Bp+nBkhVJReXhzMYi1
 wzkeD0P+RW4NHIs1buG+MR4+wSfpWuiAf4PClF1a7eLV20z1iK21+1xLrXSiyg3u3xBr
 K5b3G2xdOwmza3QL0x/bla18BYVA+VCxm8bwYMe8XmqEfq/vYeqfWIKtaw+goRnj9W9N
 9GzLzO1BDE+JszVRf1G+dfbNBGhiUyD1x/XhnK+eAMUNqTlIX7CkKvbNfdqKvYbl8L5q
 f7pw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20210112;
 h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc;
 bh=z2bT2fPLvDXm891Tc56t27V5/fusQLjdKu3PLiPsC9I=;
 b=OCamkTELJti6DU7YRkPLdSX1hAoatVzARZpl/2mnHrXWRRnkPklNDt8E6QNJV87di2
 8T0mQPWneOAcUJmTB5IvY00tlTbRnmWA/Dghev+2cPT8wcCiquDObM8Uz+vMXrfW2O8j
 5fD/vuqV+FI2DMEoESt4KUrCrY5QRrEPM3W9l3B8E+peFQytvc/w6NHTXx8VMKeSh4B3
 QBB0K18+xWPGAJ08rMZHScbVX5Pjp/VXjiCPu7VTHsJSU8o5ocbqJvaMnj+hcrffZ2lk
 apFY0AN+lJZRfjW1vGwlVgKF7m8Wnh+woak+/bPR6VQ7tgbjVyjExDFZ8Ua/0N/Tye7v
 dodw==
X-Gm-Message-State: AOAM530RJAKz4kfiG598NpoLuSNPO8P2ejwY72uCdj6vZJ4bwo4zP3jn
 ne8V7J9cfvM2tmUgAnzhO8iyJI9qcTwvjMWQZJSg9FwiLA7ZEMa8AL7MHSj8tsGog5KPgKmYkRk
 605prZz5slcXEeyoyazaBSHeijGf9DII+vVST0f9s9yGMvJAIL7w3MbR4wyIvu0kwaeEDZgI=
X-Google-Smtp-Source: ABdhPJzepFGeioixOm4uXACs+hvX3KHEeaxBj3GvFJj7bTVjjbKPYYrAC+xc8Cq/JANOy/SR4DiLsV94g6MxJjg=
X-Received: from alankelly0.zrh.corp.google.com
 ([2a00:79e0:61:301:b159:808d:943e:13ba])
 (user=alankelly job=sendgmr) by 2002:a5d:6c68:0:b0:1e8:9827:b978 with SMTP id
 r8-20020a5d6c68000000b001e89827b978mr1568559wrz.633.1645092205341; Thu, 17
 Feb 2022 02:03:25 -0800 (PST)
Date: Thu, 17 Feb 2022 11:03:21 +0100
Message-Id: <20220217100321.1110443-1-alankelly@google.com>
Mime-Version: 1.0
X-Mailer: git-send-email 2.35.1.265.g69c8d7142f-goog
From: Alan Kelly <alankelly-at-google.com@ffmpeg.org>
To: ffmpeg-devel@ffmpeg.org
Subject: [FFmpeg-devel] [PATCH v2 1/5] libswscale: Check and propagate
 memory allocation errors from ff_shuffle_filter_coefficients.
X-BeenThere: ffmpeg-devel@ffmpeg.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: FFmpeg development discussions and patches <ffmpeg-devel.ffmpeg.org>
List-Unsubscribe: <https://ffmpeg.org/mailman/options/ffmpeg-devel>,
 <mailto:ffmpeg-devel-request@ffmpeg.org?subject=unsubscribe>
List-Archive: <https://ffmpeg.org/pipermail/ffmpeg-devel>
List-Post: <mailto:ffmpeg-devel@ffmpeg.org>
List-Help: <mailto:ffmpeg-devel-request@ffmpeg.org?subject=help>
List-Subscribe: <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>,
 <mailto:ffmpeg-devel-request@ffmpeg.org?subject=subscribe>
Reply-To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Cc: Alan Kelly <alankelly@google.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: ffmpeg-devel-bounces@ffmpeg.org
Sender: "ffmpeg-devel" <ffmpeg-devel-bounces@ffmpeg.org>
Archived-At: <https://master.gitmailbox.com/ffmpegdev/20220217100321.1110443-1-alankelly@google.com/>
List-Archive: <https://master.gitmailbox.com/ffmpegdev/>
List-Post: <mailto:ffmpegdev@gitmailbox.com>

---
 libswscale/swscale_internal.h |  2 +-
 libswscale/utils.c            | 11 ++++++++---
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/libswscale/swscale_internal.h b/libswscale/swscale_internal.h
index 3a78d95ba6..26d28d42e6 100644
--- a/libswscale/swscale_internal.h
+++ b/libswscale/swscale_internal.h
@@ -1144,5 +1144,5 @@ void ff_sws_slice_worker(void *priv, int jobnr, int threadnr,
 #define MAX_LINES_AHEAD 4
 
 //shuffle filter and filterPos for hyScale and hcScale filters in avx2
-void ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW);
+int ff_shuffle_filter_coefficients(SwsContext *c, int* filterPos, int filterSize, int16_t *filter, int dstW);
 #endif /* SWSCALE_SWSCALE_INTERNAL_H */
diff --git a/libswscale/utils.c b/libswscale/utils.c
index c5ea8853d5..344c87dfdf 100644
--- a/libswscale/utils.c
+++ b/libswscale/utils.c
@@ -278,7 +278,7 @@ static const FormatEntry format_entries[] = {
     [AV_PIX_FMT_P416LE]      = { 1, 1 },
 };
 
-void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){
+int ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSize, int16_t *filter, int dstW){
 #if ARCH_X86_64
     int i, j, k, l;
     int cpu_flags = av_get_cpu_flags();
@@ -292,6 +292,8 @@ void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSiz
                     }
                     if (filterSize > 4){
                         int16_t *tmp2 = av_malloc(dstW * filterSize * 2);
+                        if (!tmp2)
+                            return AVERROR(ENOMEM);
                         memcpy(tmp2, filter, dstW * filterSize * 2);
                         for (i = 0; i < dstW; i += 16){//pixel
                             for (k = 0; k < filterSize / 4; ++k){//fcoeff
@@ -310,6 +312,7 @@ void ff_shuffle_filter_coefficients(SwsContext *c, int *filterPos, int filterSiz
             }
         }
     }
+    return 0;
 #endif
 }
 
@@ -1836,7 +1839,8 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter,
                            get_local_pos(c, 0, 0, 0),
                            get_local_pos(c, 0, 0, 0))) < 0)
                 goto fail;
-            ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW);
+            if (ff_shuffle_filter_coefficients(c, c->hLumFilterPos, c->hLumFilterSize, c->hLumFilter, dstW) < 0)
+                goto nomem;
             if ((ret = initFilter(&c->hChrFilter, &c->hChrFilterPos,
                            &c->hChrFilterSize, c->chrXInc,
                            c->chrSrcW, c->chrDstW, filterAlign, 1 << 14,
@@ -1846,7 +1850,8 @@ av_cold int sws_init_context(SwsContext *c, SwsFilter *srcFilter,
                            get_local_pos(c, c->chrSrcHSubSample, c->src_h_chr_pos, 0),
                            get_local_pos(c, c->chrDstHSubSample, c->dst_h_chr_pos, 0))) < 0)
                 goto fail;
-            ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW);
+            if (ff_shuffle_filter_coefficients(c, c->hChrFilterPos, c->hChrFilterSize, c->hChrFilter, c->chrDstW) < 0)
+                goto nomem;
         }
     } // initialize horizontal stuff
 
-- 
2.35.1.265.g69c8d7142f-goog

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".