* [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour @ 2022-09-01 19:28 Carl Eugen Hoyos 2022-09-02 7:04 ` Anton Khirnov 0 siblings, 1 reply; 4+ messages in thread From: Carl Eugen Hoyos @ 2022-09-01 19:28 UTC (permalink / raw) To: FFmpeg development discussions and patches [-- Attachment #1: Type: text/plain, Size: 68 bytes --] Hi! Attached patch fixes ticket #9799. Please comment, Carl Eugen [-- Attachment #2: 0001-lavfi-rotate-Avoid-undefined-behaviour.patch --] [-- Type: text/x-patch, Size: 2031 bytes --] From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 From: Carl Eugen Hoyos <ceffmpeg@gmail.com> Date: Thu, 1 Sep 2022 20:55:54 +0200 Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. Fixes the following integer overflows: libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' Fixes ticket #9799, different output with different compilers. --- libavfilter/vf_rotate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c index 4429e3d543..d319dfe3d9 100644 --- a/libavfilter/vf_rotate.c +++ b/libavfilter/vf_rotate.c @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); AV_WL16(&dst_color[i], ((int64_t)((1<<16) - frac_y)*s0 + (int64_t)frac_y*s1) >> 32); } -- 2.30.1 [-- Attachment #3: Type: text/plain, Size: 251 bytes --] _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour 2022-09-01 19:28 [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour Carl Eugen Hoyos @ 2022-09-02 7:04 ` Anton Khirnov 2022-09-03 21:03 ` Carl Eugen Hoyos 0 siblings, 1 reply; 4+ messages in thread From: Anton Khirnov @ 2022-09-02 7:04 UTC (permalink / raw) To: FFmpeg development discussions and patches Quoting Carl Eugen Hoyos (2022-09-01 21:28:08) > Hi! > > Attached patch fixes ticket #9799. > > Please comment, Carl Eugen > > From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 > From: Carl Eugen Hoyos <ceffmpeg@gmail.com> > Date: Thu, 1 Sep 2022 20:55:54 +0200 > Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. > > Fixes the following integer overflows: > libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' > libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' > libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' > libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' > libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' > libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' > > Fixes ticket #9799, different output with different compilers. > --- > libavfilter/vf_rotate.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c > index 4429e3d543..d319dfe3d9 100644 > --- a/libavfilter/vf_rotate.c > +++ b/libavfilter/vf_rotate.c > @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, > int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); > int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); > int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); > - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); > - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); > + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); > + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); Given that the same casts also appear in the AV_WL16 below, it seems better to just change the types of frac_x/frac_y. Also, shouldn't the same change be done also to interpolate_bilinear8? -- Anton Khirnov _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour 2022-09-02 7:04 ` Anton Khirnov @ 2022-09-03 21:03 ` Carl Eugen Hoyos 2022-09-04 6:39 ` Michael Koch 0 siblings, 1 reply; 4+ messages in thread From: Carl Eugen Hoyos @ 2022-09-03 21:03 UTC (permalink / raw) To: FFmpeg development discussions and patches [-- Attachment #1: Type: text/plain, Size: 2552 bytes --] Am Fr., 2. Sept. 2022 um 09:04 Uhr schrieb Anton Khirnov <anton@khirnov.net>: > > Quoting Carl Eugen Hoyos (2022-09-01 21:28:08) > > Hi! > > > > Attached patch fixes ticket #9799. > > > > Please comment, Carl Eugen > > > > From 2cce687961c3b56a92d88184269bf9fa075ae297 Mon Sep 17 00:00:00 2001 > > From: Carl Eugen Hoyos <ceffmpeg@gmail.com> > > Date: Thu, 1 Sep 2022 20:55:54 +0200 > > Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. > > > > Fixes the following integer overflows: > > libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' > > libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' > > > > Fixes ticket #9799, different output with different compilers. > > --- > > libavfilter/vf_rotate.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c > > index 4429e3d543..d319dfe3d9 100644 > > --- a/libavfilter/vf_rotate.c > > +++ b/libavfilter/vf_rotate.c > > @@ -269,8 +269,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, > > int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); > > int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); > > int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); > > - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); > > - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); > > + int64_t s0 = (((int64_t)(1<<16) - frac_x)*s00 + (int64_t)frac_x*s01); > > + int64_t s1 = (((int64_t)(1<<16) - frac_x)*s10 + (int64_t)frac_x*s11); > > Given that the same casts also appear in the AV_WL16 below, it seems > better to just change the types of frac_x/frac_y. Definitely, new patch attached. > Also, shouldn't the same change be done also to interpolate_bilinear8? I was unable to reproduce with 8-bit input. Thank you, Carl Eugen [-- Attachment #2: 0001-lavfi-rotate-Avoid-undefined-behaviour.patch.txt --] [-- Type: text/plain, Size: 2487 bytes --] From 062b8a3a202f72125fa05582c2a3c0abc8305887 Mon Sep 17 00:00:00 2001 From: Carl Eugen Hoyos <ceffmpeg@gmail.com> Date: Sat, 3 Sep 2022 22:50:19 +0200 Subject: [PATCH] lavfi/rotate: Avoid undefined behaviour. Fixes the following integer overflows: libavfilter/vf_rotate.c:273:13: runtime error: signed integer overflow: 92951468 + 2058533568 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:37: runtime error: signed integer overflow: 39684 * 54149 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:13: runtime error: signed integer overflow: 247587320 + 1900985032 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:37: runtime error: signed integer overflow: 42584 * 50430 cannot be represented in type 'int' libavfilter/vf_rotate.c:272:50: runtime error: signed integer overflow: 65083 * 52912 cannot be represented in type 'int' libavfilter/vf_rotate.c:273:50: runtime error: signed integer overflow: 65286 * 38044 cannot be represented in type 'int' Fixes ticket #9799, different output with different compilers. --- libavfilter/vf_rotate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libavfilter/vf_rotate.c b/libavfilter/vf_rotate.c index 4429e3d543..96c250a459 100644 --- a/libavfilter/vf_rotate.c +++ b/libavfilter/vf_rotate.c @@ -258,8 +258,8 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, { int int_x = av_clip(x>>16, 0, max_x); int int_y = av_clip(y>>16, 0, max_y); - int frac_x = x&0xFFFF; - int frac_y = y&0xFFFF; + int64_t frac_x = x&0xFFFF; + int64_t frac_y = y&0xFFFF; int i; int int_x1 = FFMIN(int_x+1, max_x); int int_y1 = FFMIN(int_y+1, max_y); @@ -269,10 +269,10 @@ static uint8_t *interpolate_bilinear16(uint8_t *dst_color, int s01 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y ]); int s10 = AV_RL16(&src[src_linestep * int_x + i + src_linesize * int_y1]); int s11 = AV_RL16(&src[src_linestep * int_x1 + i + src_linesize * int_y1]); - int s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); - int s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); + int64_t s0 = (((1<<16) - frac_x)*s00 + frac_x*s01); + int64_t s1 = (((1<<16) - frac_x)*s10 + frac_x*s11); - AV_WL16(&dst_color[i], ((int64_t)((1<<16) - frac_y)*s0 + (int64_t)frac_y*s1) >> 32); + AV_WL16(&dst_color[i], (((1<<16) - frac_y)*s0 + frac_y*s1) >> 32); } return dst_color; -- 2.17.1 [-- Attachment #3: Type: text/plain, Size: 251 bytes --] _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour 2022-09-03 21:03 ` Carl Eugen Hoyos @ 2022-09-04 6:39 ` Michael Koch 0 siblings, 0 replies; 4+ messages in thread From: Michael Koch @ 2022-09-04 6:39 UTC (permalink / raw) To: ffmpeg-devel >>/Also, shouldn't the same change be done also to interpolate_bilinear8? / > I was unable to reproduce with 8-bit input. When I tested it, the issue was reproducible only with 14-bit and 16-bit input. 12-bit did work. _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-09-04 6:39 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-09-01 19:28 [FFmpeg-devel] [PATCH]lavfi/rotate: Fix undefined behaviour Carl Eugen Hoyos 2022-09-02 7:04 ` Anton Khirnov 2022-09-03 21:03 ` Carl Eugen Hoyos 2022-09-04 6:39 ` Michael Koch
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel This inbox may be cloned and mirrored by anyone: git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \ ffmpegdev@gitmailbox.com public-inbox-index ffmpegdev Example config snippet for mirrors. AGPL code for this site: git clone https://public-inbox.org/public-inbox.git