* [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check
@ 2022-09-28 16:33 Andreas Rheinhardt
2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 2/5] avcodec/rl2: Don't presume stride to be > 0 Andreas Rheinhardt
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Andreas Rheinhardt @ 2022-09-28 16:33 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Andreas Rheinhardt
This check is intended to be avoid buffer overflows,
yet there are four problems with it:
1. It has an in-built off-by-one error: len == out_end - out
is perfectly fine and nothing to worry about.
This off-by-one error led to the pixel in the lower-right corner
not being set properly for the back frame of the sample from
the rl2 FATE-test. This pixel is copied to every frame which
is the reason for the update to the reference file of said test.
With this patch, the output of the decoder matches the output
as captured from the reference decoder* (apart from the fact
that said reference somehow lacks the top part of the frame
(copied over from the background frame)).
2. Given that the stride of the buffer may be different
from the width of the video (despite one pixel taking one byte),
there is a second check lateron making the first check redundant
(if one returns immediately; a simple break at the second check
is not sufficient, because it only exits the inner loop).
3. The check is based around the assumption of the stride being
positive (it has this in common with the other check which
will be fixed in a future commit).
4. Even after fixing the off-by-one error, the check in
question is still triggered by all the non-background frames
in the FATE sample as well as by A1100100.RL2. In all these
cases, they use len == 255 and val == 128. For videos with
background frame this just means "copy from the background
frame", which would be done anyway lateron.* Yet for videos
without it copying it is necessary to avoid leaving
uninitialized parts in the video.
*: Available in https://samples.mplayerhq.hu/game-formats/voyeur-rl2/
**: Due to this, the code that copies the rest from the
back frame is no longer executed for any of the samples
available on the sample server. Given that these are only
the files from the demo version of this game, I don't know
whether this code is executed for any file in existence or not.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
---
libavcodec/rl2.c | 5 +-
tests/ref/fate/rl2 | 216 ++++++++++++++++++++++-----------------------
2 files changed, 109 insertions(+), 112 deletions(-)
diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c
index 5dedb96266..2464ad59ac 100644
--- a/libavcodec/rl2.c
+++ b/libavcodec/rl2.c
@@ -91,9 +91,6 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size,
break;
}
- if (len >= out_end - out)
- break;
-
if (s->back_frame)
val |= 0x80;
else
@@ -106,7 +103,7 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size,
out += stride_adj;
line_end += stride;
if (len >= out_end - out)
- break;
+ return;
}
}
}
diff --git a/tests/ref/fate/rl2 b/tests/ref/fate/rl2
index 9189822503..a3c6a75d05 100644
--- a/tests/ref/fate/rl2
+++ b/tests/ref/fate/rl2
@@ -3,111 +3,111 @@
#codec_id 0: rawvideo
#dimensions 0: 320x200
#sar 0: 0/1
-0, 0, 0, 1, 192000, 0x7112a667
-0, 1, 1, 1, 192000, 0x6936abf3
-0, 2, 2, 1, 192000, 0xb1f08981
-0, 3, 3, 1, 192000, 0x4ce7fece
-0, 4, 4, 1, 192000, 0xf04decde
-0, 5, 5, 1, 192000, 0x47fef74b
-0, 6, 6, 1, 192000, 0x99b42ac2
-0, 7, 7, 1, 192000, 0x3c4c419d
-0, 8, 8, 1, 192000, 0x66bf5588
-0, 9, 9, 1, 192000, 0xe1de4725
-0, 10, 10, 1, 192000, 0x348b2af9
-0, 11, 11, 1, 192000, 0x1ce73e83
-0, 12, 12, 1, 192000, 0xcdaa6e02
-0, 13, 13, 1, 192000, 0x370dc2ce
-0, 14, 14, 1, 192000, 0x1e1e40fe
-0, 15, 15, 1, 192000, 0x491470a4
-0, 16, 16, 1, 192000, 0x88c43e9a
-0, 17, 17, 1, 192000, 0x036f3f44
-0, 18, 18, 1, 192000, 0xc8be7e25
-0, 19, 19, 1, 192000, 0xbedb397d
-0, 20, 20, 1, 192000, 0x97c410f4
-0, 21, 21, 1, 192000, 0xc2c8225d
-0, 22, 22, 1, 192000, 0xe396bccb
-0, 23, 23, 1, 192000, 0x7e89c24c
-0, 24, 24, 1, 192000, 0xb044954c
-0, 25, 25, 1, 192000, 0x335890dd
-0, 26, 26, 1, 192000, 0x58a457c0
-0, 27, 27, 1, 192000, 0xeb0f4798
-0, 28, 28, 1, 192000, 0x0bfc39a1
-0, 29, 29, 1, 192000, 0x06a6905a
-0, 30, 30, 1, 192000, 0x5300c99b
-0, 31, 31, 1, 192000, 0x38f3f845
-0, 32, 32, 1, 192000, 0x6afa3543
-0, 33, 33, 1, 192000, 0x5106a8e0
-0, 34, 34, 1, 192000, 0xc76f0dab
-0, 35, 35, 1, 192000, 0x8efa6939
-0, 36, 36, 1, 192000, 0x64ea23d7
-0, 37, 37, 1, 192000, 0x421a2817
-0, 38, 38, 1, 192000, 0xc854fa18
-0, 39, 39, 1, 192000, 0x4a10d59c
-0, 40, 40, 1, 192000, 0x72637829
-0, 41, 41, 1, 192000, 0xdbbe2796
-0, 42, 42, 1, 192000, 0xff742e6b
-0, 43, 43, 1, 192000, 0xf94b9346
-0, 44, 44, 1, 192000, 0xc90ea53c
-0, 45, 45, 1, 192000, 0x177483bb
-0, 46, 46, 1, 192000, 0x3510369c
-0, 47, 47, 1, 192000, 0x501034bf
-0, 48, 48, 1, 192000, 0x315c744b
-0, 49, 49, 1, 192000, 0xdb5048ae
-0, 50, 50, 1, 192000, 0x09a86221
-0, 51, 51, 1, 192000, 0xb9c9568a
-0, 52, 52, 1, 192000, 0x5eee665b
-0, 53, 53, 1, 192000, 0xdef85517
-0, 54, 54, 1, 192000, 0x7896b5ad
-0, 55, 55, 1, 192000, 0x19fbad39
-0, 56, 56, 1, 192000, 0x63358748
-0, 57, 57, 1, 192000, 0xca0196a3
-0, 58, 58, 1, 192000, 0x0b321da3
-0, 59, 59, 1, 192000, 0xa07af07e
-0, 60, 60, 1, 192000, 0x21f9310c
-0, 61, 61, 1, 192000, 0x62d59874
-0, 62, 62, 1, 192000, 0x021227b4
-0, 63, 63, 1, 192000, 0x01dac0c2
-0, 64, 64, 1, 192000, 0xf3c33a74
-0, 65, 65, 1, 192000, 0xeebe83b7
-0, 66, 66, 1, 192000, 0x9ec77f97
-0, 67, 67, 1, 192000, 0xc91c2e37
-0, 68, 68, 1, 192000, 0x7b58751d
-0, 69, 69, 1, 192000, 0xb178dfbb
-0, 70, 70, 1, 192000, 0x2a63b5be
-0, 71, 71, 1, 192000, 0x44a407ac
-0, 72, 72, 1, 192000, 0x9a8e17dd
-0, 73, 73, 1, 192000, 0x5546f4c8
-0, 74, 74, 1, 192000, 0xeda94586
-0, 75, 75, 1, 192000, 0x941dfa6b
-0, 76, 76, 1, 192000, 0x9772301d
-0, 77, 77, 1, 192000, 0x8be16b16
-0, 78, 78, 1, 192000, 0x26dd4496
-0, 79, 79, 1, 192000, 0x27823797
-0, 80, 80, 1, 192000, 0xacc914f7
-0, 81, 81, 1, 192000, 0xcbb72c9b
-0, 82, 82, 1, 192000, 0x4bd3391c
-0, 83, 83, 1, 192000, 0x4e6adbfe
-0, 84, 84, 1, 192000, 0x7a791c75
-0, 85, 85, 1, 192000, 0xc4f59c94
-0, 86, 86, 1, 192000, 0xc4f59c94
-0, 87, 87, 1, 192000, 0x984a4a0b
-0, 88, 88, 1, 192000, 0x3353f31f
-0, 89, 89, 1, 192000, 0xa9d4dc5a
-0, 90, 90, 1, 192000, 0xb33425d0
-0, 91, 91, 1, 192000, 0x546d768a
-0, 92, 92, 1, 192000, 0xfefbe5c9
-0, 93, 93, 1, 192000, 0xbd6be61d
-0, 94, 94, 1, 192000, 0xf5792731
-0, 95, 95, 1, 192000, 0xccde0582
-0, 96, 96, 1, 192000, 0x857d58ee
-0, 97, 97, 1, 192000, 0xe914ce48
-0, 98, 98, 1, 192000, 0x1f736298
-0, 99, 99, 1, 192000, 0xec0b4846
-0, 100, 100, 1, 192000, 0xe1422624
-0, 101, 101, 1, 192000, 0x56e2b0e0
-0, 102, 102, 1, 192000, 0xc4190640
-0, 103, 103, 1, 192000, 0x7c461977
-0, 104, 104, 1, 192000, 0x34b1d5e8
-0, 105, 105, 1, 192000, 0xbdc70f7a
-0, 106, 106, 1, 192000, 0xb466cd8d
-0, 107, 107, 1, 192000, 0x0e86a04c
+0, 0, 0, 1, 192000, 0x7324a772
+0, 1, 1, 1, 192000, 0x6b48acfe
+0, 2, 2, 1, 192000, 0xb4028a8c
+0, 3, 3, 1, 192000, 0x4ef9ffd9
+0, 4, 4, 1, 192000, 0xf25fede9
+0, 5, 5, 1, 192000, 0x4a10f856
+0, 6, 6, 1, 192000, 0x9bc62bcd
+0, 7, 7, 1, 192000, 0x3e5e42a8
+0, 8, 8, 1, 192000, 0x68d15693
+0, 9, 9, 1, 192000, 0xe3f04830
+0, 10, 10, 1, 192000, 0x369d2c04
+0, 11, 11, 1, 192000, 0x1ef93f8e
+0, 12, 12, 1, 192000, 0xcfbc6f0d
+0, 13, 13, 1, 192000, 0x391fc3d9
+0, 14, 14, 1, 192000, 0x20304209
+0, 15, 15, 1, 192000, 0x4b2671af
+0, 16, 16, 1, 192000, 0x8ad63fa5
+0, 17, 17, 1, 192000, 0x0581404f
+0, 18, 18, 1, 192000, 0xcad07f30
+0, 19, 19, 1, 192000, 0xc0ed3a88
+0, 20, 20, 1, 192000, 0x99d611ff
+0, 21, 21, 1, 192000, 0xc4da2368
+0, 22, 22, 1, 192000, 0xe5a8bdd6
+0, 23, 23, 1, 192000, 0x809bc357
+0, 24, 24, 1, 192000, 0xb2569657
+0, 25, 25, 1, 192000, 0x356a91e8
+0, 26, 26, 1, 192000, 0x5ab658cb
+0, 27, 27, 1, 192000, 0xed2148a3
+0, 28, 28, 1, 192000, 0x0e0e3aac
+0, 29, 29, 1, 192000, 0x08b89165
+0, 30, 30, 1, 192000, 0x5512caa6
+0, 31, 31, 1, 192000, 0x3b05f950
+0, 32, 32, 1, 192000, 0x6d0c364e
+0, 33, 33, 1, 192000, 0x5318a9eb
+0, 34, 34, 1, 192000, 0xc9810eb6
+0, 35, 35, 1, 192000, 0x910c6a44
+0, 36, 36, 1, 192000, 0x66fc24e2
+0, 37, 37, 1, 192000, 0x442c2922
+0, 38, 38, 1, 192000, 0xca66fb23
+0, 39, 39, 1, 192000, 0x4c22d6a7
+0, 40, 40, 1, 192000, 0x74757934
+0, 41, 41, 1, 192000, 0xddd028a1
+0, 42, 42, 1, 192000, 0x01952f76
+0, 43, 43, 1, 192000, 0xfb5d9451
+0, 44, 44, 1, 192000, 0xcb20a647
+0, 45, 45, 1, 192000, 0x198684c6
+0, 46, 46, 1, 192000, 0x372237a7
+0, 47, 47, 1, 192000, 0x522235ca
+0, 48, 48, 1, 192000, 0x336e7556
+0, 49, 49, 1, 192000, 0xdd6249b9
+0, 50, 50, 1, 192000, 0x0bba632c
+0, 51, 51, 1, 192000, 0xbbdb5795
+0, 52, 52, 1, 192000, 0x61006766
+0, 53, 53, 1, 192000, 0xe10a5622
+0, 54, 54, 1, 192000, 0x7aa8b6b8
+0, 55, 55, 1, 192000, 0x1c0dae44
+0, 56, 56, 1, 192000, 0x65478853
+0, 57, 57, 1, 192000, 0xcc1397ae
+0, 58, 58, 1, 192000, 0x0d441eae
+0, 59, 59, 1, 192000, 0xa28cf189
+0, 60, 60, 1, 192000, 0x240b3217
+0, 61, 61, 1, 192000, 0x64e7997f
+0, 62, 62, 1, 192000, 0x042428bf
+0, 63, 63, 1, 192000, 0x03ecc1cd
+0, 64, 64, 1, 192000, 0xf5d53b7f
+0, 65, 65, 1, 192000, 0xf0d084c2
+0, 66, 66, 1, 192000, 0xa0d980a2
+0, 67, 67, 1, 192000, 0xcb2e2f42
+0, 68, 68, 1, 192000, 0x7d6a7628
+0, 69, 69, 1, 192000, 0xb38ae0c6
+0, 70, 70, 1, 192000, 0x2c75b6c9
+0, 71, 71, 1, 192000, 0x46b608b7
+0, 72, 72, 1, 192000, 0x9ca018e8
+0, 73, 73, 1, 192000, 0x5758f5d3
+0, 74, 74, 1, 192000, 0xefbb4691
+0, 75, 75, 1, 192000, 0x962ffb76
+0, 76, 76, 1, 192000, 0x99843128
+0, 77, 77, 1, 192000, 0x8df36c21
+0, 78, 78, 1, 192000, 0x28ef45a1
+0, 79, 79, 1, 192000, 0x299438a2
+0, 80, 80, 1, 192000, 0xaedb1602
+0, 81, 81, 1, 192000, 0xcdc92da6
+0, 82, 82, 1, 192000, 0x4de53a27
+0, 83, 83, 1, 192000, 0x507cdd09
+0, 84, 84, 1, 192000, 0x7c8b1d80
+0, 85, 85, 1, 192000, 0xc7079d9f
+0, 86, 86, 1, 192000, 0xc7079d9f
+0, 87, 87, 1, 192000, 0x9a5c4b16
+0, 88, 88, 1, 192000, 0x3565f42a
+0, 89, 89, 1, 192000, 0xabe6dd65
+0, 90, 90, 1, 192000, 0xb54626db
+0, 91, 91, 1, 192000, 0x567f7795
+0, 92, 92, 1, 192000, 0x011ce6d4
+0, 93, 93, 1, 192000, 0xbf7de728
+0, 94, 94, 1, 192000, 0xf78b283c
+0, 95, 95, 1, 192000, 0xcef0068d
+0, 96, 96, 1, 192000, 0x878f59f9
+0, 97, 97, 1, 192000, 0xeb26cf53
+0, 98, 98, 1, 192000, 0x218563a3
+0, 99, 99, 1, 192000, 0xee1d4951
+0, 100, 100, 1, 192000, 0xe354272f
+0, 101, 101, 1, 192000, 0x58f4b1eb
+0, 102, 102, 1, 192000, 0xc62b074b
+0, 103, 103, 1, 192000, 0x7e581a82
+0, 104, 104, 1, 192000, 0x36c3d6f3
+0, 105, 105, 1, 192000, 0xbfd91085
+0, 106, 106, 1, 192000, 0xb678ce98
+0, 107, 107, 1, 192000, 0x1098a157
--
2.34.1
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 6+ messages in thread* [FFmpeg-devel] [PATCH 2/5] avcodec/rl2: Don't presume stride to be > 0 2022-09-28 16:33 [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt @ 2022-09-28 16:35 ` Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 3/5] avcodec/rl2: Use ptrdiff_t for stride Andreas Rheinhardt ` (3 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Andreas Rheinhardt @ 2022-09-28 16:35 UTC (permalink / raw) To: ffmpeg-devel; +Cc: Andreas Rheinhardt Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com> --- Lots of fate tests fail if this assumption is not fulfilled. libavcodec/rl2.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c index 2464ad59ac..467c4913a4 100644 --- a/libavcodec/rl2.c +++ b/libavcodec/rl2.c @@ -65,7 +65,7 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, int i; const uint8_t *back_frame = s->back_frame; const uint8_t *in_end = in + size; - const uint8_t *out_end = out + stride * s->avctx->height; + const uint8_t *out_end = out + stride * s->avctx->height - stride_adj; uint8_t *line_end; /** copy start of the background frame */ @@ -100,18 +100,20 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, *out++ = (val == 0x80) ? *back_frame : val; back_frame++; if (out == line_end) { + if (out == out_end) + return; out += stride_adj; line_end += stride; - if (len >= out_end - out) - return; } } } /** copy the rest from the background frame */ if (s->back_frame) { - while (out < out_end) { + while (1) { memcpy(out, back_frame, line_end - out); + if (line_end == out_end) + break; back_frame += line_end - out; out = line_end + stride_adj; line_end += stride; -- 2.34.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 6+ messages in thread
* [FFmpeg-devel] [PATCH 3/5] avcodec/rl2: Use ptrdiff_t for stride 2022-09-28 16:33 [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 2/5] avcodec/rl2: Don't presume stride to be > 0 Andreas Rheinhardt @ 2022-09-28 16:35 ` Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 4/5] avcodec/rl2: Fix undefined pointer arithmetic Andreas Rheinhardt ` (2 subsequent siblings) 4 siblings, 0 replies; 6+ messages in thread From: Andreas Rheinhardt @ 2022-09-28 16:35 UTC (permalink / raw) To: ffmpeg-devel; +Cc: Andreas Rheinhardt Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com> --- libavcodec/rl2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c index 467c4913a4..76982f0426 100644 --- a/libavcodec/rl2.c +++ b/libavcodec/rl2.c @@ -57,11 +57,11 @@ typedef struct Rl2Context { * @param video_base offset of the rle data inside the frame */ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, - uint8_t *out, int stride, int video_base) + uint8_t *out, ptrdiff_t stride, int video_base) { int base_x = video_base % s->avctx->width; int base_y = video_base / s->avctx->width; - int stride_adj = stride - s->avctx->width; + ptrdiff_t stride_adj = stride - s->avctx->width; int i; const uint8_t *back_frame = s->back_frame; const uint8_t *in_end = in + size; -- 2.34.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 6+ messages in thread
* [FFmpeg-devel] [PATCH 4/5] avcodec/rl2: Fix undefined pointer arithmetic 2022-09-28 16:33 [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 2/5] avcodec/rl2: Don't presume stride to be > 0 Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 3/5] avcodec/rl2: Use ptrdiff_t for stride Andreas Rheinhardt @ 2022-09-28 16:35 ` Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 5/5] avcodec/rl2: Fix indentation Andreas Rheinhardt 2022-10-02 1:23 ` [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt 4 siblings, 0 replies; 6+ messages in thread From: Andreas Rheinhardt @ 2022-09-28 16:35 UTC (permalink / raw) To: ffmpeg-devel; +Cc: Andreas Rheinhardt Don't increment back_frame if it does not correspond to a real buffer. To do this, handle copying from the back frame separately from the "use coded value" codepath; also use memcpy for the former, as the chunks here are typically worth it. Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com> --- libavcodec/rl2.c | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c index 76982f0426..7938ef1d92 100644 --- a/libavcodec/rl2.c +++ b/libavcodec/rl2.c @@ -69,13 +69,16 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, uint8_t *line_end; /** copy start of the background frame */ + if (s->back_frame) { for (i = 0; i <= base_y; i++) { - if (s->back_frame) memcpy(out, back_frame, s->avctx->width); out += stride; back_frame += s->avctx->width; } back_frame += base_x - s->avctx->width; + } else { + out += stride * (base_y + 1); + } line_end = out - stride_adj; out += base_x - stride; @@ -89,16 +92,32 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, len = *in++; if (!len) break; + val &= 0x7F; } - if (s->back_frame) + if (back_frame) { + if (!val) { + do { + size_t copy = FFMIN(line_end - out, len); + memcpy(out, back_frame, copy); + out += copy; + back_frame += copy; + len -= copy; + if (out == line_end) { + if (out == out_end) + return; + out += stride_adj; + line_end += stride; + } + } while (len > 0); + continue; + } + back_frame += len; val |= 0x80; - else - val &= ~0x80; + } while (len--) { - *out++ = (val == 0x80) ? *back_frame : val; - back_frame++; + *out++ = val; if (out == line_end) { if (out == out_end) return; @@ -164,7 +183,9 @@ static av_cold int rl2_decode_init(AVCodecContext *avctx) back_size = avctx->extradata_size - EXTRADATA1_SIZE; if (back_size > 0) { - uint8_t *back_frame = av_mallocz(avctx->width*avctx->height); + /* The 254 are padding to ensure that pointer arithmetic stays within + * the buffer. */ + uint8_t *back_frame = av_mallocz(avctx->width * avctx->height + 254); if (!back_frame) return AVERROR(ENOMEM); rl2_rle_decode(s, avctx->extradata + EXTRADATA1_SIZE, back_size, -- 2.34.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 6+ messages in thread
* [FFmpeg-devel] [PATCH 5/5] avcodec/rl2: Fix indentation 2022-09-28 16:33 [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt ` (2 preceding siblings ...) 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 4/5] avcodec/rl2: Fix undefined pointer arithmetic Andreas Rheinhardt @ 2022-09-28 16:35 ` Andreas Rheinhardt 2022-10-02 1:23 ` [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt 4 siblings, 0 replies; 6+ messages in thread From: Andreas Rheinhardt @ 2022-09-28 16:35 UTC (permalink / raw) To: ffmpeg-devel; +Cc: Andreas Rheinhardt Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com> --- libavcodec/rl2.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c index 7938ef1d92..e427a27dce 100644 --- a/libavcodec/rl2.c +++ b/libavcodec/rl2.c @@ -62,7 +62,6 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, int base_x = video_base % s->avctx->width; int base_y = video_base / s->avctx->width; ptrdiff_t stride_adj = stride - s->avctx->width; - int i; const uint8_t *back_frame = s->back_frame; const uint8_t *in_end = in + size; const uint8_t *out_end = out + stride * s->avctx->height - stride_adj; @@ -70,12 +69,12 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, /** copy start of the background frame */ if (s->back_frame) { - for (i = 0; i <= base_y; i++) { + for (int i = 0; i <= base_y; i++) { memcpy(out, back_frame, s->avctx->width); - out += stride; - back_frame += s->avctx->width; - } - back_frame += base_x - s->avctx->width; + out += stride; + back_frame += s->avctx->width; + } + back_frame += base_x - s->avctx->width; } else { out += stride * (base_y + 1); } @@ -121,8 +120,8 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, if (out == line_end) { if (out == out_end) return; - out += stride_adj; - line_end += stride; + out += stride_adj; + line_end += stride; } } } -- 2.34.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check 2022-09-28 16:33 [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt ` (3 preceding siblings ...) 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 5/5] avcodec/rl2: Fix indentation Andreas Rheinhardt @ 2022-10-02 1:23 ` Andreas Rheinhardt 4 siblings, 0 replies; 6+ messages in thread From: Andreas Rheinhardt @ 2022-10-02 1:23 UTC (permalink / raw) To: ffmpeg-devel Andreas Rheinhardt: > This check is intended to be avoid buffer overflows, > yet there are four problems with it: > 1. It has an in-built off-by-one error: len == out_end - out > is perfectly fine and nothing to worry about. > This off-by-one error led to the pixel in the lower-right corner > not being set properly for the back frame of the sample from > the rl2 FATE-test. This pixel is copied to every frame which > is the reason for the update to the reference file of said test. > With this patch, the output of the decoder matches the output > as captured from the reference decoder* (apart from the fact > that said reference somehow lacks the top part of the frame > (copied over from the background frame)). > 2. Given that the stride of the buffer may be different > from the width of the video (despite one pixel taking one byte), > there is a second check lateron making the first check redundant > (if one returns immediately; a simple break at the second check > is not sufficient, because it only exits the inner loop). > 3. The check is based around the assumption of the stride being > positive (it has this in common with the other check which > will be fixed in a future commit). > 4. Even after fixing the off-by-one error, the check in > question is still triggered by all the non-background frames > in the FATE sample as well as by A1100100.RL2. In all these > cases, they use len == 255 and val == 128. For videos with > background frame this just means "copy from the background > frame", which would be done anyway lateron.* Yet for videos > without it copying it is necessary to avoid leaving > uninitialized parts in the video. > > *: Available in https://samples.mplayerhq.hu/game-formats/voyeur-rl2/ > **: Due to this, the code that copies the rest from the > back frame is no longer executed for any of the samples > available on the sample server. Given that these are only > the files from the demo version of this game, I don't know > whether this code is executed for any file in existence or not. > > Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com> > --- > libavcodec/rl2.c | 5 +- > tests/ref/fate/rl2 | 216 ++++++++++++++++++++++----------------------- > 2 files changed, 109 insertions(+), 112 deletions(-) > > diff --git a/libavcodec/rl2.c b/libavcodec/rl2.c > index 5dedb96266..2464ad59ac 100644 > --- a/libavcodec/rl2.c > +++ b/libavcodec/rl2.c > @@ -91,9 +91,6 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, > break; > } > > - if (len >= out_end - out) > - break; > - > if (s->back_frame) > val |= 0x80; > else > @@ -106,7 +103,7 @@ static void rl2_rle_decode(Rl2Context *s, const uint8_t *in, int size, > out += stride_adj; > line_end += stride; > if (len >= out_end - out) > - break; > + return; > } > } > } > diff --git a/tests/ref/fate/rl2 b/tests/ref/fate/rl2 > index 9189822503..a3c6a75d05 100644 > --- a/tests/ref/fate/rl2 > +++ b/tests/ref/fate/rl2 > @@ -3,111 +3,111 @@ > #codec_id 0: rawvideo > #dimensions 0: 320x200 > #sar 0: 0/1 > -0, 0, 0, 1, 192000, 0x7112a667 > -0, 1, 1, 1, 192000, 0x6936abf3 > -0, 2, 2, 1, 192000, 0xb1f08981 > -0, 3, 3, 1, 192000, 0x4ce7fece > -0, 4, 4, 1, 192000, 0xf04decde > -0, 5, 5, 1, 192000, 0x47fef74b > -0, 6, 6, 1, 192000, 0x99b42ac2 > -0, 7, 7, 1, 192000, 0x3c4c419d > -0, 8, 8, 1, 192000, 0x66bf5588 > -0, 9, 9, 1, 192000, 0xe1de4725 > -0, 10, 10, 1, 192000, 0x348b2af9 > -0, 11, 11, 1, 192000, 0x1ce73e83 > -0, 12, 12, 1, 192000, 0xcdaa6e02 > -0, 13, 13, 1, 192000, 0x370dc2ce > -0, 14, 14, 1, 192000, 0x1e1e40fe > -0, 15, 15, 1, 192000, 0x491470a4 > -0, 16, 16, 1, 192000, 0x88c43e9a > -0, 17, 17, 1, 192000, 0x036f3f44 > -0, 18, 18, 1, 192000, 0xc8be7e25 > -0, 19, 19, 1, 192000, 0xbedb397d > -0, 20, 20, 1, 192000, 0x97c410f4 > -0, 21, 21, 1, 192000, 0xc2c8225d > -0, 22, 22, 1, 192000, 0xe396bccb > -0, 23, 23, 1, 192000, 0x7e89c24c > -0, 24, 24, 1, 192000, 0xb044954c > -0, 25, 25, 1, 192000, 0x335890dd > -0, 26, 26, 1, 192000, 0x58a457c0 > -0, 27, 27, 1, 192000, 0xeb0f4798 > -0, 28, 28, 1, 192000, 0x0bfc39a1 > -0, 29, 29, 1, 192000, 0x06a6905a > -0, 30, 30, 1, 192000, 0x5300c99b > -0, 31, 31, 1, 192000, 0x38f3f845 > -0, 32, 32, 1, 192000, 0x6afa3543 > -0, 33, 33, 1, 192000, 0x5106a8e0 > -0, 34, 34, 1, 192000, 0xc76f0dab > -0, 35, 35, 1, 192000, 0x8efa6939 > -0, 36, 36, 1, 192000, 0x64ea23d7 > -0, 37, 37, 1, 192000, 0x421a2817 > -0, 38, 38, 1, 192000, 0xc854fa18 > -0, 39, 39, 1, 192000, 0x4a10d59c > -0, 40, 40, 1, 192000, 0x72637829 > -0, 41, 41, 1, 192000, 0xdbbe2796 > -0, 42, 42, 1, 192000, 0xff742e6b > -0, 43, 43, 1, 192000, 0xf94b9346 > -0, 44, 44, 1, 192000, 0xc90ea53c > -0, 45, 45, 1, 192000, 0x177483bb > -0, 46, 46, 1, 192000, 0x3510369c > -0, 47, 47, 1, 192000, 0x501034bf > -0, 48, 48, 1, 192000, 0x315c744b > -0, 49, 49, 1, 192000, 0xdb5048ae > -0, 50, 50, 1, 192000, 0x09a86221 > -0, 51, 51, 1, 192000, 0xb9c9568a > -0, 52, 52, 1, 192000, 0x5eee665b > -0, 53, 53, 1, 192000, 0xdef85517 > -0, 54, 54, 1, 192000, 0x7896b5ad > -0, 55, 55, 1, 192000, 0x19fbad39 > -0, 56, 56, 1, 192000, 0x63358748 > -0, 57, 57, 1, 192000, 0xca0196a3 > -0, 58, 58, 1, 192000, 0x0b321da3 > -0, 59, 59, 1, 192000, 0xa07af07e > -0, 60, 60, 1, 192000, 0x21f9310c > -0, 61, 61, 1, 192000, 0x62d59874 > -0, 62, 62, 1, 192000, 0x021227b4 > -0, 63, 63, 1, 192000, 0x01dac0c2 > -0, 64, 64, 1, 192000, 0xf3c33a74 > -0, 65, 65, 1, 192000, 0xeebe83b7 > -0, 66, 66, 1, 192000, 0x9ec77f97 > -0, 67, 67, 1, 192000, 0xc91c2e37 > -0, 68, 68, 1, 192000, 0x7b58751d > -0, 69, 69, 1, 192000, 0xb178dfbb > -0, 70, 70, 1, 192000, 0x2a63b5be > -0, 71, 71, 1, 192000, 0x44a407ac > -0, 72, 72, 1, 192000, 0x9a8e17dd > -0, 73, 73, 1, 192000, 0x5546f4c8 > -0, 74, 74, 1, 192000, 0xeda94586 > -0, 75, 75, 1, 192000, 0x941dfa6b > -0, 76, 76, 1, 192000, 0x9772301d > -0, 77, 77, 1, 192000, 0x8be16b16 > -0, 78, 78, 1, 192000, 0x26dd4496 > -0, 79, 79, 1, 192000, 0x27823797 > -0, 80, 80, 1, 192000, 0xacc914f7 > -0, 81, 81, 1, 192000, 0xcbb72c9b > -0, 82, 82, 1, 192000, 0x4bd3391c > -0, 83, 83, 1, 192000, 0x4e6adbfe > -0, 84, 84, 1, 192000, 0x7a791c75 > -0, 85, 85, 1, 192000, 0xc4f59c94 > -0, 86, 86, 1, 192000, 0xc4f59c94 > -0, 87, 87, 1, 192000, 0x984a4a0b > -0, 88, 88, 1, 192000, 0x3353f31f > -0, 89, 89, 1, 192000, 0xa9d4dc5a > -0, 90, 90, 1, 192000, 0xb33425d0 > -0, 91, 91, 1, 192000, 0x546d768a > -0, 92, 92, 1, 192000, 0xfefbe5c9 > -0, 93, 93, 1, 192000, 0xbd6be61d > -0, 94, 94, 1, 192000, 0xf5792731 > -0, 95, 95, 1, 192000, 0xccde0582 > -0, 96, 96, 1, 192000, 0x857d58ee > -0, 97, 97, 1, 192000, 0xe914ce48 > -0, 98, 98, 1, 192000, 0x1f736298 > -0, 99, 99, 1, 192000, 0xec0b4846 > -0, 100, 100, 1, 192000, 0xe1422624 > -0, 101, 101, 1, 192000, 0x56e2b0e0 > -0, 102, 102, 1, 192000, 0xc4190640 > -0, 103, 103, 1, 192000, 0x7c461977 > -0, 104, 104, 1, 192000, 0x34b1d5e8 > -0, 105, 105, 1, 192000, 0xbdc70f7a > -0, 106, 106, 1, 192000, 0xb466cd8d > -0, 107, 107, 1, 192000, 0x0e86a04c > +0, 0, 0, 1, 192000, 0x7324a772 > +0, 1, 1, 1, 192000, 0x6b48acfe > +0, 2, 2, 1, 192000, 0xb4028a8c > +0, 3, 3, 1, 192000, 0x4ef9ffd9 > +0, 4, 4, 1, 192000, 0xf25fede9 > +0, 5, 5, 1, 192000, 0x4a10f856 > +0, 6, 6, 1, 192000, 0x9bc62bcd > +0, 7, 7, 1, 192000, 0x3e5e42a8 > +0, 8, 8, 1, 192000, 0x68d15693 > +0, 9, 9, 1, 192000, 0xe3f04830 > +0, 10, 10, 1, 192000, 0x369d2c04 > +0, 11, 11, 1, 192000, 0x1ef93f8e > +0, 12, 12, 1, 192000, 0xcfbc6f0d > +0, 13, 13, 1, 192000, 0x391fc3d9 > +0, 14, 14, 1, 192000, 0x20304209 > +0, 15, 15, 1, 192000, 0x4b2671af > +0, 16, 16, 1, 192000, 0x8ad63fa5 > +0, 17, 17, 1, 192000, 0x0581404f > +0, 18, 18, 1, 192000, 0xcad07f30 > +0, 19, 19, 1, 192000, 0xc0ed3a88 > +0, 20, 20, 1, 192000, 0x99d611ff > +0, 21, 21, 1, 192000, 0xc4da2368 > +0, 22, 22, 1, 192000, 0xe5a8bdd6 > +0, 23, 23, 1, 192000, 0x809bc357 > +0, 24, 24, 1, 192000, 0xb2569657 > +0, 25, 25, 1, 192000, 0x356a91e8 > +0, 26, 26, 1, 192000, 0x5ab658cb > +0, 27, 27, 1, 192000, 0xed2148a3 > +0, 28, 28, 1, 192000, 0x0e0e3aac > +0, 29, 29, 1, 192000, 0x08b89165 > +0, 30, 30, 1, 192000, 0x5512caa6 > +0, 31, 31, 1, 192000, 0x3b05f950 > +0, 32, 32, 1, 192000, 0x6d0c364e > +0, 33, 33, 1, 192000, 0x5318a9eb > +0, 34, 34, 1, 192000, 0xc9810eb6 > +0, 35, 35, 1, 192000, 0x910c6a44 > +0, 36, 36, 1, 192000, 0x66fc24e2 > +0, 37, 37, 1, 192000, 0x442c2922 > +0, 38, 38, 1, 192000, 0xca66fb23 > +0, 39, 39, 1, 192000, 0x4c22d6a7 > +0, 40, 40, 1, 192000, 0x74757934 > +0, 41, 41, 1, 192000, 0xddd028a1 > +0, 42, 42, 1, 192000, 0x01952f76 > +0, 43, 43, 1, 192000, 0xfb5d9451 > +0, 44, 44, 1, 192000, 0xcb20a647 > +0, 45, 45, 1, 192000, 0x198684c6 > +0, 46, 46, 1, 192000, 0x372237a7 > +0, 47, 47, 1, 192000, 0x522235ca > +0, 48, 48, 1, 192000, 0x336e7556 > +0, 49, 49, 1, 192000, 0xdd6249b9 > +0, 50, 50, 1, 192000, 0x0bba632c > +0, 51, 51, 1, 192000, 0xbbdb5795 > +0, 52, 52, 1, 192000, 0x61006766 > +0, 53, 53, 1, 192000, 0xe10a5622 > +0, 54, 54, 1, 192000, 0x7aa8b6b8 > +0, 55, 55, 1, 192000, 0x1c0dae44 > +0, 56, 56, 1, 192000, 0x65478853 > +0, 57, 57, 1, 192000, 0xcc1397ae > +0, 58, 58, 1, 192000, 0x0d441eae > +0, 59, 59, 1, 192000, 0xa28cf189 > +0, 60, 60, 1, 192000, 0x240b3217 > +0, 61, 61, 1, 192000, 0x64e7997f > +0, 62, 62, 1, 192000, 0x042428bf > +0, 63, 63, 1, 192000, 0x03ecc1cd > +0, 64, 64, 1, 192000, 0xf5d53b7f > +0, 65, 65, 1, 192000, 0xf0d084c2 > +0, 66, 66, 1, 192000, 0xa0d980a2 > +0, 67, 67, 1, 192000, 0xcb2e2f42 > +0, 68, 68, 1, 192000, 0x7d6a7628 > +0, 69, 69, 1, 192000, 0xb38ae0c6 > +0, 70, 70, 1, 192000, 0x2c75b6c9 > +0, 71, 71, 1, 192000, 0x46b608b7 > +0, 72, 72, 1, 192000, 0x9ca018e8 > +0, 73, 73, 1, 192000, 0x5758f5d3 > +0, 74, 74, 1, 192000, 0xefbb4691 > +0, 75, 75, 1, 192000, 0x962ffb76 > +0, 76, 76, 1, 192000, 0x99843128 > +0, 77, 77, 1, 192000, 0x8df36c21 > +0, 78, 78, 1, 192000, 0x28ef45a1 > +0, 79, 79, 1, 192000, 0x299438a2 > +0, 80, 80, 1, 192000, 0xaedb1602 > +0, 81, 81, 1, 192000, 0xcdc92da6 > +0, 82, 82, 1, 192000, 0x4de53a27 > +0, 83, 83, 1, 192000, 0x507cdd09 > +0, 84, 84, 1, 192000, 0x7c8b1d80 > +0, 85, 85, 1, 192000, 0xc7079d9f > +0, 86, 86, 1, 192000, 0xc7079d9f > +0, 87, 87, 1, 192000, 0x9a5c4b16 > +0, 88, 88, 1, 192000, 0x3565f42a > +0, 89, 89, 1, 192000, 0xabe6dd65 > +0, 90, 90, 1, 192000, 0xb54626db > +0, 91, 91, 1, 192000, 0x567f7795 > +0, 92, 92, 1, 192000, 0x011ce6d4 > +0, 93, 93, 1, 192000, 0xbf7de728 > +0, 94, 94, 1, 192000, 0xf78b283c > +0, 95, 95, 1, 192000, 0xcef0068d > +0, 96, 96, 1, 192000, 0x878f59f9 > +0, 97, 97, 1, 192000, 0xeb26cf53 > +0, 98, 98, 1, 192000, 0x218563a3 > +0, 99, 99, 1, 192000, 0xee1d4951 > +0, 100, 100, 1, 192000, 0xe354272f > +0, 101, 101, 1, 192000, 0x58f4b1eb > +0, 102, 102, 1, 192000, 0xc62b074b > +0, 103, 103, 1, 192000, 0x7e581a82 > +0, 104, 104, 1, 192000, 0x36c3d6f3 > +0, 105, 105, 1, 192000, 0xbfd91085 > +0, 106, 106, 1, 192000, 0xb678ce98 > +0, 107, 107, 1, 192000, 0x1098a157 Will apply this patchset tonight unless there are objections. - Andreas _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-10-02 1:23 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-09-28 16:33 [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 2/5] avcodec/rl2: Don't presume stride to be > 0 Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 3/5] avcodec/rl2: Use ptrdiff_t for stride Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 4/5] avcodec/rl2: Fix undefined pointer arithmetic Andreas Rheinhardt 2022-09-28 16:35 ` [FFmpeg-devel] [PATCH 5/5] avcodec/rl2: Fix indentation Andreas Rheinhardt 2022-10-02 1:23 ` [FFmpeg-devel] [PATCH 1/5] avcodec/rl2: Remove wrong check Andreas Rheinhardt
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel This inbox may be cloned and mirrored by anyone: git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \ ffmpegdev@gitmailbox.com public-inbox-index ffmpegdev Example config snippet for mirrors. AGPL code for this site: git clone https://public-inbox.org/public-inbox.git