From: Kieran Kunhya via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Cc: Michael Niedermayer <michael@niedermayer.cc>,
Kieran Kunhya <kieran618@googlemail.com>
Subject: [FFmpeg-devel] Re: [PATCH] swscale/output: Fix unsigned cast position in yuv2* (PR #20698)
Date: Sat, 3 Jan 2026 05:31:21 +0000
Message-ID: <CABGuwEmRPV3fjBveKnRZjOgg49xhwTD35GS=nVEp3WzKxeq01g@mail.gmail.com> (raw)
In-Reply-To: <aO6gyYtrtKnF5Czk@neo>
I see that this has a CVE associated with it that Ubuntu rates "High"
Severity:
https://ubuntu.com/security/CVE-2025-63757
What is actually vulnerable here? A vulnerability that a pixel value might
theoretically be wrong.
Kieran
On Tue, 14 Oct 2025, 20:13 Michael Niedermayer via ffmpeg-devel, <
ffmpeg-devel@ffmpeg.org> wrote:
> On Mon, Oct 13, 2025 at 05:20:55PM +0100, Kieran Kunhya via ffmpeg-devel
> wrote:
> > On Mon, 13 Oct 2025, 14:28 michaelni via ffmpeg-devel, <
> > ffmpeg-devel@ffmpeg.org> wrote:
> >
> > > PR #20698 opened by michaelni
> > > URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20698
> > > Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20698.patch
> > >
> > >
> > > >From 93794583405909cfe8e7abffdd84c2454959ff18 Mon Sep 17 00:00:00 2001
> > > From: Michael Niedermayer <michael@niedermayer.cc>
> > > Date: Mon, 13 Oct 2025 14:32:45 +0200
> > > Subject: [PATCH 1/2] swscale/output: Fix integer overflow in
> > > yuv2ya16_X_c_template()
> > >
> > > Found-by: colod colod <colodcolod7@gmail.com>
> > >
> > > Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> > > ---
> > > libswscale/output.c | 4 ++--
> > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/libswscale/output.c b/libswscale/output.c
> > > index b873276c00..cb6630bd73 100644
> > > --- a/libswscale/output.c
> > > +++ b/libswscale/output.c
> > > @@ -1028,7 +1028,7 @@ yuv2ya16_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > int A = 0xffff;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > Y >>= 15;
> > > Y += (1<<3) + 0x8000;
> > > @@ -1037,7 +1037,7 @@ yuv2ya16_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > if (hasAlpha) {
> > > A = -0x40000000 + (1<<14);
> > > for (j = 0; j < lumFilterSize; j++)
> > > - A += (unsigned)(alpSrc[j][i] * lumFilter[j]);
> > > + A += alpSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > A >>= 15;
> > > A += 0x8000;
> > > --
> > > 2.49.1
> > >
> > >
> > > >From 9d0761453e5ecaa1344d63e2fcc9bd3cc6d4d0f6 Mon Sep 17 00:00:00 2001
> > > From: Michael Niedermayer <michael@niedermayer.cc>
> > > Date: Mon, 13 Oct 2025 14:46:16 +0200
> > > Subject: [PATCH 2/2] swscale/output: Fix unsigned cast position in
> yuv2*
> > >
> > > Fixes: signed overflow
> > >
> > > Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> > > ---
> > > libswscale/output.c | 90 ++++++++++++++++++++++-----------------------
> > > 1 file changed, 45 insertions(+), 45 deletions(-)
> > >
> > > diff --git a/libswscale/output.c b/libswscale/output.c
> > > index cb6630bd73..877db04f36 100644
> > > --- a/libswscale/output.c
> > > +++ b/libswscale/output.c
> > > @@ -503,8 +503,8 @@ static void yuv2nv12cX_c(enum AVPixelFormat
> dstFormat,
> > > const uint8_t *chrDither,
> > > int v = chrDither[(i + 3) & 7] << 12;
> > > int j;
> > > for (j=0; j<chrFilterSize; j++) {
> > > - u += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - v += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + u += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + v += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > >
> > > dest[2*i]= av_clip_uint8(u>>19);
> > > @@ -516,8 +516,8 @@ static void yuv2nv12cX_c(enum AVPixelFormat
> dstFormat,
> > > const uint8_t *chrDither,
> > > int v = chrDither[(i + 3) & 7] << 12;
> > > int j;
> > > for (j=0; j<chrFilterSize; j++) {
> > > - u += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - v += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + u += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + v += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > >
> > > dest[2*i]= av_clip_uint8(v>>19);
> > > @@ -577,8 +577,8 @@ static void yuv2p01xcX_c(int big_endian, const
> uint8_t
> > > *chrDither,
> > > int v = 1 << (shift - 1);
> > >
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - u += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - v += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + u += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + v += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > >
> > > output_pixel(&dest[2*i] , u);
> > > @@ -678,8 +678,8 @@ yuv2mono_X_c_template(SwsInternal *c, const int16_t
> > > *lumFilter,
> > > int Y2 = 1 << 18;
> > >
> > > for (j = 0; j < lumFilterSize; j++) {
> > > - Y1 += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > - Y2 += (unsigned)(lumSrc[j][i+1] * lumFilter[j]);
> > > + Y1 += lumSrc[j][i] * (unsigned)lumFilter[j];
> > > + Y2 += lumSrc[j][i+1] * (unsigned)lumFilter[j];
> > > }
> > > Y1 >>= 19;
> > > Y2 >>= 19;
> > > @@ -896,12 +896,12 @@ yuv2422_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > int V = 1 << 18;
> > >
> > > for (j = 0; j < lumFilterSize; j++) {
> > > - Y1 += (unsigned)(lumSrc[j][i * 2] * lumFilter[j]);
> > > - Y2 += (unsigned)(lumSrc[j][i * 2 + 1] * lumFilter[j]);
> > > + Y1 += lumSrc[j][i * 2] * (unsigned)lumFilter[j];
> > > + Y2 += lumSrc[j][i * 2 + 1] * (unsigned)lumFilter[j];
> > > }
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > > Y1 >>= 19;
> > > Y2 >>= 19;
> > > @@ -1802,12 +1802,12 @@ yuv2rgb_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > const void *r, *g, *b;
> > >
> > > for (j = 0; j < lumFilterSize; j++) {
> > > - Y1 += (unsigned)(lumSrc[j][i * 2] * lumFilter[j]);
> > > - Y2 += (unsigned)(lumSrc[j][i * 2 + 1] * lumFilter[j]);
> > > + Y1 += lumSrc[j][i * 2] * (unsigned)lumFilter[j];
> > > + Y2 += lumSrc[j][i * 2 + 1] * (unsigned)lumFilter[j];
> > > }
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > > Y1 >>= 19;
> > > Y2 >>= 19;
> > > @@ -1817,8 +1817,8 @@ yuv2rgb_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > A1 = 1 << 18;
> > > A2 = 1 << 18;
> > > for (j = 0; j < lumFilterSize; j++) {
> > > - A1 += (unsigned)(alpSrc[j][i * 2 ] * lumFilter[j]);
> > > - A2 += (unsigned)(alpSrc[j][i * 2 + 1] * lumFilter[j]);
> > > + A1 += alpSrc[j][i * 2 ] * (unsigned)lumFilter[j];
> > > + A2 += alpSrc[j][i * 2 + 1] * (unsigned)lumFilter[j];
> > > }
> > > A1 >>= 19;
> > > A2 >>= 19;
> > > @@ -2179,11 +2179,11 @@ yuv2rgb_full_X_c_template(SwsInternal *c, const
> > > int16_t *lumFilter,
> > > int V = (1<<9)-(128 << 19);
> > >
> > > for (j = 0; j < lumFilterSize; j++) {
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > > }
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > > Y >>= 10;
> > > U >>= 10;
> > > @@ -2191,7 +2191,7 @@ yuv2rgb_full_X_c_template(SwsInternal *c, const
> > > int16_t *lumFilter,
> > > if (hasAlpha) {
> > > A = 1 << 18;
> > > for (j = 0; j < lumFilterSize; j++) {
> > > - A += (unsigned)(alpSrc[j][i] * lumFilter[j]);
> > > + A += alpSrc[j][i] * (unsigned)lumFilter[j];
> > > }
> > > A >>= 19;
> > > if (A & 0x100)
> > > @@ -2360,11 +2360,11 @@ yuv2gbrp_full_X_c(SwsInternal *c, const int16_t
> > > *lumFilter,
> > > int R, G, B;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > >
> > > Y >>= 10;
> > > @@ -2375,7 +2375,7 @@ yuv2gbrp_full_X_c(SwsInternal *c, const int16_t
> > > *lumFilter,
> > > A = 1 << 18;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - A += (unsigned)(alpSrc[j][i] * lumFilter[j]);
> > > + A += alpSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > if (A & 0xF8000000)
> > > A = av_clip_uintp2(A, 27);
> > > @@ -2679,7 +2679,7 @@ yuv2ya8_X_c(SwsInternal *c, const int16_t
> *lumFilter,
> > > int Y = 1 << 18, A = 1 << 18;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > Y >>= 19;
> > > if (Y & 0x100)
> > > @@ -2687,7 +2687,7 @@ yuv2ya8_X_c(SwsInternal *c, const int16_t
> *lumFilter,
> > >
> > > if (hasAlpha) {
> > > for (j = 0; j < lumFilterSize; j++)
> > > - A += (unsigned)(alpSrc[j][i] * lumFilter[j]);
> > > + A += alpSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > A >>= 19;
> > >
> > > @@ -2793,11 +2793,11 @@ yuv2v30_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > int j;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > >
> > > Y = av_clip_uintp2(Y >> 17, 10);
> > > @@ -2847,11 +2847,11 @@ yuv2xv36_X_c(SwsInternal *c, const int16_t
> > > *lumFilter,
> > > int j;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++) {
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > }
> > >
> > > output_pixels(dest + 8 * i + 2, Y, 15, 12, 4)
> > > @@ -3011,13 +3011,13 @@ yuv2ayuv_X_c_template(SwsInternal *c, const
> > > int16_t *lumFilter,
> > > int V = 1 << 18, A = 255;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++)
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++)
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > >
> > > Y >>= 19;
> > > U >>= 19;
> > > @@ -3034,7 +3034,7 @@ yuv2ayuv_X_c_template(SwsInternal *c, const
> int16_t
> > > *lumFilter,
> > > A = 1 << 18;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - A += (unsigned)(alpSrc[j][i] * lumFilter[j]);
> > > + A += alpSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > A >>= 19;
> > >
> > > @@ -3105,13 +3105,13 @@ AYUVPACKEDWRAPPER(uyva, AV_PIX_FMT_UYVA)
> > > int U = 1 << (shift - 1), V = 1 << (shift - 1);
> \
> > >
> \
> > > for (j = 0; j < lumFilterSize; j++) {
> \
> > > - Y1 += (unsigned)(lumSrc[j][i * 2] * lumFilter[j]);
> > > \
> > > - Y2 += (unsigned)(lumSrc[j][i * 2 + 1] * lumFilter[j]);
> > > \
> > > + Y1 += lumSrc[j][i * 2] * (unsigned)lumFilter[j];
> > > \
> > > + Y2 += lumSrc[j][i * 2 + 1] * (unsigned)lumFilter[j];
> > > \
> > > }
> \
> > >
> \
> > > for (j = 0; j < chrFilterSize; j++) {
> \
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > \
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > \
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > > \
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > > \
> > > }
> \
> > >
> \
> > > output_pixel(dest + 8 * i + 0, Y1, bits);
> \
> > > @@ -3259,13 +3259,13 @@ yuv2vyu444_X_c(SwsInternal *c, const int16_t
> > > *lumFilter,
> > > int V = 1 << 18;
> > >
> > > for (j = 0; j < lumFilterSize; j++)
> > > - Y += (unsigned)(lumSrc[j][i] * lumFilter[j]);
> > > + Y += lumSrc[j][i] * (unsigned)lumFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++)
> > > - U += (unsigned)(chrUSrc[j][i] * chrFilter[j]);
> > > + U += chrUSrc[j][i] * (unsigned)chrFilter[j];
> > >
> > > for (j = 0; j < chrFilterSize; j++)
> > > - V += (unsigned)(chrVSrc[j][i] * chrFilter[j]);
> > > + V += chrVSrc[j][i] * (unsigned)chrFilter[j];
> > >
> > > Y >>= 19;
> > > U >>= 19;
> > > --
> > > 2.49.1
> > >
> >
> > Seems fine
>
> applied
>
> thx
>
> [...]
> --
> Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> Those who would give up essential Liberty, to purchase a little
> temporary Safety, deserve neither Liberty nor Safety -- Benjamin Franklin
> _______________________________________________
> ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
> To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org
>
_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org
prev parent reply other threads:[~2026-01-03 5:32 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-13 13:27 [FFmpeg-devel] " michaelni via ffmpeg-devel
2025-10-13 16:20 ` [FFmpeg-devel] " Kieran Kunhya via ffmpeg-devel
2025-10-14 19:13 ` Michael Niedermayer via ffmpeg-devel
2026-01-03 5:31 ` Kieran Kunhya via ffmpeg-devel [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CABGuwEmRPV3fjBveKnRZjOgg49xhwTD35GS=nVEp3WzKxeq01g@mail.gmail.com' \
--to=ffmpeg-devel@ffmpeg.org \
--cc=kieran618@googlemail.com \
--cc=michael@niedermayer.cc \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
This inbox may be cloned and mirrored by anyone:
git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
ffmpegdev@gitmailbox.com
public-inbox-index ffmpegdev
Example config snippet for mirrors.
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git