[FFmpeg-devel] [PR] avfilter/vf_convolution: various heap over/underflow fixes (PR #21368)

[FFmpeg-devel] [PR] avfilter/vf_convolution: various heap over/underflow fixes (PR #21368)

[Timo Rothenpieler via ffmpeg-devel]

PR #21368 opened by Timo Rothenpieler (BtbN)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21368
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21368.patch

Without all this extra clamping, the filter would happily over- and underflow the input and output image buffers.

Fixes #YWH-PGM40646-36


>From 8e2079c53e35a99a81382f2428862153f3a4d928 Mon Sep 17 00:00:00 2001
From: Timo Rothenpieler <timo@rothenpieler.org>
Date: Sat, 3 Jan 2026 21:23:02 +0100
Subject: [PATCH 1/3] avfilter/vf_convolution: clamp column and row offsets to
 actual width/height of buffer

Otherwise the buffer might be dramatically under or over-read.
---
 libavfilter/vf_convolution.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavfilter/vf_convolution.c b/libavfilter/vf_convolution.c
index ce42df2cde..2c4f706449 100644
--- a/libavfilter/vf_convolution.c
+++ b/libavfilter/vf_convolution.c
@@ -554,7 +554,7 @@ static void setup_row(int radius, const uint8_t *c[], const uint8_t *src, int st
     for (i = 0; i < radius * 2 + 1; i++) {
         int xoff = FFABS(x + i - radius);
 
-        xoff = xoff >= w ? 2 * w - 1 - xoff : xoff;
+        xoff = FFMIN(FFMAX(xoff >= w ? 2 * w - 1 - xoff : xoff, 0), w - 1);
 
         c[i] = src + xoff * bpc + y * stride;
     }
@@ -568,7 +568,7 @@ static void setup_column(int radius, const uint8_t *c[], const uint8_t *src, int
     for (i = 0; i < radius * 2 + 1; i++) {
         int xoff = FFABS(x + i - radius);
 
-        xoff = xoff >= h ? 2 * h - 1 - xoff : xoff;
+        xoff = FFMIN(FFMAX(xoff >= h ? 2 * h - 1 - xoff : xoff, 0), h - 1);
 
         c[i] = src + y * bpc + xoff * stride;
     }
-- 
2.49.1


>From a16db608a5246c82f660db4f2b80776887ecbc43 Mon Sep 17 00:00:00 2001
From: Timo Rothenpieler <timo@rothenpieler.org>
Date: Sat, 3 Jan 2026 21:45:41 +0100
Subject: [PATCH 2/3] avfilter/vf_convolution: don't over-read input stride in
 filter_column

---
 libavfilter/vf_convolution.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/libavfilter/vf_convolution.c b/libavfilter/vf_convolution.c
index 2c4f706449..81f66dcdac 100644
--- a/libavfilter/vf_convolution.c
+++ b/libavfilter/vf_convolution.c
@@ -502,11 +502,12 @@ static void filter_column(uint8_t *dst, int height,
         memset(sum, 0, sizeof(sum));
 
         for (int i = 0; i < 2 * radius + 1; i++) {
-            for (int off16 = 0; off16 < 16; off16++)
+            for (int off16 = 0; off16 < 16 && off16 < stride; off16++) {
                 sum[off16] += c[i][0 + y * stride + off16] * matrix[i];
+            }
         }
 
-        for (int off16 = 0; off16 < 16; off16++) {
+        for (int off16 = 0; off16 < 16 && off16 < stride && off16 < dstride; off16++) {
             sum[off16] = (int)(sum[off16] * rdiv + bias + 0.5f);
             dst[off16] = av_clip_uint8(sum[off16]);
         }
-- 
2.49.1


>From 9f5c6a8f6de328af4d577d9156d7acdb0b10f0ae Mon Sep 17 00:00:00 2001
From: Timo Rothenpieler <timo@rothenpieler.org>
Date: Sat, 3 Jan 2026 21:53:23 +0100
Subject: [PATCH 3/3] avfilter/vf_convolution: clamp x and y offsets to actual
 width/height of the image

Without this, if the input/output are sufficiently small enough, this
will over-write and read the buffers by however much the radius or
slice-size is.

Fixes #YWH-PGM40646-36
---
 libavfilter/vf_convolution.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/libavfilter/vf_convolution.c b/libavfilter/vf_convolution.c
index 81f66dcdac..2902d6117e 100644
--- a/libavfilter/vf_convolution.c
+++ b/libavfilter/vf_convolution.c
@@ -615,12 +615,16 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs)
             continue;
         }
         for (y = slice_start; y < slice_end; y += step) {
-            const int xoff = mode == MATRIX_COLUMN ? (y - slice_start) * bpc : radius * bpc;
-            const int yoff = mode == MATRIX_COLUMN ? radius * dstride : 0;
+            int xoff = mode == MATRIX_COLUMN ? (y - slice_start) * bpc : radius * bpc;
+            int yoff = mode == MATRIX_COLUMN ? radius * dstride : 0;
+            xoff = FFMIN(FFMAX(xoff, 0), width);
+            yoff = FFMIN(FFMAX(yoff, 0), height);
 
             for (x = 0; x < radius; x++) {
-                const int xoff = mode == MATRIX_COLUMN ? (y - slice_start) * bpc : x * bpc;
-                const int yoff = mode == MATRIX_COLUMN ? x * dstride : 0;
+                int xoff = mode == MATRIX_COLUMN ? (y - slice_start) * bpc : x * bpc;
+                int yoff = mode == MATRIX_COLUMN ? x * dstride : 0;
+                xoff = FFMIN(FFMAX(xoff, 0), width);
+                yoff = FFMIN(FFMAX(yoff, 0), height);
 
                 s->setup[plane](radius, c, src, stride, x, width, y, height, bpc);
                 s->filter[plane](dst + yoff + xoff, 1, rdiv,
@@ -632,8 +636,10 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs)
                              rdiv, bias, matrix, c, s->max, radius,
                              dstride, stride, slice_end - step);
             for (x = sizew - radius; x < sizew; x++) {
-                const int xoff = mode == MATRIX_COLUMN ? (y - slice_start) * bpc : x * bpc;
-                const int yoff = mode == MATRIX_COLUMN ? x * dstride : 0;
+                int xoff = mode == MATRIX_COLUMN ? (y - slice_start) * bpc : x * bpc;
+                int yoff = mode == MATRIX_COLUMN ? x * dstride : 0;
+                xoff = FFMIN(FFMAX(xoff, 0), width);
+                yoff = FFMIN(FFMAX(yoff, 0), height);
 
                 s->setup[plane](radius, c, src, stride, x, width, y, height, bpc);
                 s->filter[plane](dst + yoff + xoff, 1, rdiv,
-- 
2.49.1

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org