[FFmpeg-devel] [PATCH] avcodec/ppc/vp8dsp_altivec: Fix out-of-bounds access

[FFmpeg-devel] [PATCH] avcodec/ppc/vp8dsp_altivec: Fix out-of-bounds access

[Andreas Rheinhardt]

h_subpel_filters_inner[i] and h_subpel_filters_outer[i / 2]
belong together and the former allows the range 0..6,
so the latter needs to support 0..3. But it has only three
elements. Add another one.
The value for the last element has been guesstimated
from subpel_filters in libavcodec/vp8dsp.c.

This is also intended to fix FATE-failures with UBSan here:
https://fate.ffmpeg.org/report.cgi?time=20240312011016&slot=ppc-linux-gcc-13.2-ubsan-altivec-qemu

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
---
 libavcodec/ppc/vp8dsp_altivec.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/libavcodec/ppc/vp8dsp_altivec.c b/libavcodec/ppc/vp8dsp_altivec.c
index 12dac8b0a8..061914fc38 100644
--- a/libavcodec/ppc/vp8dsp_altivec.c
+++ b/libavcodec/ppc/vp8dsp_altivec.c
@@ -50,11 +50,12 @@ static const vec_s8 h_subpel_filters_inner[7] =
 // for 6tap filters, these are the outer two taps
 // The zeros mask off pixels 4-7 when filtering 0-3
 // and vice-versa
-static const vec_s8 h_subpel_filters_outer[3] =
+static const vec_s8 h_subpel_filters_outer[4] =
 {
     REPT4(0, 0, 2, 1),
     REPT4(0, 0, 3, 3),
     REPT4(0, 0, 1, 2),
+    REPT4(0, 0, 0, 0),
 };
 
 #define LOAD_H_SUBPEL_FILTER(i) \
-- 
2.40.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/ppc/vp8dsp_altivec: Fix out-of-bounds access

[Sean McGovern]

Andreas:

On Tue, Mar 12, 2024 at 9:14 PM Andreas Rheinhardt
<andreas.rheinhardt@outlook.com> wrote:
>
> h_subpel_filters_inner[i] and h_subpel_filters_outer[i / 2]
> belong together and the former allows the range 0..6,
> so the latter needs to support 0..3. But it has only three
> elements. Add another one.
> The value for the last element has been guesstimated
> from subpel_filters in libavcodec/vp8dsp.c.
>
> This is also intended to fix FATE-failures with UBSan here:
> https://fate.ffmpeg.org/report.cgi?time=20240312011016&slot=ppc-linux-gcc-13.2-ubsan-altivec-qemu
>
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
> ---
>  libavcodec/ppc/vp8dsp_altivec.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/libavcodec/ppc/vp8dsp_altivec.c b/libavcodec/ppc/vp8dsp_altivec.c
> index 12dac8b0a8..061914fc38 100644
> --- a/libavcodec/ppc/vp8dsp_altivec.c
> +++ b/libavcodec/ppc/vp8dsp_altivec.c
> @@ -50,11 +50,12 @@ static const vec_s8 h_subpel_filters_inner[7] =
>  // for 6tap filters, these are the outer two taps
>  // The zeros mask off pixels 4-7 when filtering 0-3
>  // and vice-versa
> -static const vec_s8 h_subpel_filters_outer[3] =
> +static const vec_s8 h_subpel_filters_outer[4] =
>  {
>      REPT4(0, 0, 2, 1),
>      REPT4(0, 0, 3, 3),
>      REPT4(0, 0, 1, 2),
> +    REPT4(0, 0, 0, 0),
>  };
>
>  #define LOAD_H_SUBPEL_FILTER(i) \
> --
> 2.40.1
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Confirming this patch fixes fate-checkasm-vp8dsp (and presumably the
other vp8 tests in GCC UBsan) on PowerPC QEMU, POWER7 (ppc64), and
POWER9 (ppc64le).

Thanks again,
Sean McGovern
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".