[FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[Michael Niedermayer]

This moves computations out of a loop

Fixes: signed integer overflow: 31665934879948800 * 9998 cannot be represented in type 'long'
Fixes: 69024/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-5949662967169024

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/j2kenc.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
index 8cf82f7216c..91e66d81048 100644
--- a/libavcodec/j2kenc.c
+++ b/libavcodec/j2kenc.c
@@ -1349,7 +1349,7 @@ static void makelayers(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
     }
 }
 
-static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
+static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda)
 {
     int passno, res = 0;
     for (passno = 0; passno < cblk->npasses; passno++){
@@ -1361,7 +1361,7 @@ static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
         dd = cblk->passes[passno].disto
            - (res ? cblk->passes[res-1].disto : 0);
 
-        if (((dd * dwt_norm) >> WMSEDEC_SHIFT) * dwt_norm >= dr * lambda)
+        if (dd  >= dr * lambda)
             res = passno+1;
     }
     return res;
@@ -1384,11 +1384,12 @@ static void truncpasses(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
                     Jpeg2000Band *band = reslevel->band + bandno;
                     Jpeg2000Prec *prec = band->prec + precno;
 
+                    int64_t dwt_norm = dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15;
+                    int64_t lambda_prime = av_rescale(s->lambda, 1 << WMSEDEC_SHIFT, dwt_norm * dwt_norm);
                     for (cblkno = 0; cblkno < prec->nb_codeblocks_height * prec->nb_codeblocks_width; cblkno++){
                         Jpeg2000Cblk *cblk = prec->cblk + cblkno;
 
-                        cblk->ninclpasses = getcut(cblk, s->lambda,
-                                (int64_t)dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15);
+                        cblk->ninclpasses = getcut(cblk, lambda_prime);
                         cblk->layers[0].data_start = cblk->data;
                         cblk->layers[0].cum_passes = cblk->ninclpasses;
                         cblk->layers[0].npasses = cblk->ninclpasses;
-- 
2.45.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 2/4] avcodec/utvideoenc: Use unsigned shift to build flags

[Michael Niedermayer]

Fixes: left shift of 255 by 24 places cannot be represented in type 'int'
Fixes: 69083/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_UTVIDEO_fuzzer-5608202363273216

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/utvideoenc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/utvideoenc.c b/libavcodec/utvideoenc.c
index 59e198458b0..b35b34d5a1b 100644
--- a/libavcodec/utvideoenc.c
+++ b/libavcodec/utvideoenc.c
@@ -239,7 +239,7 @@ static av_cold int utvideo_encode_init(AVCodecContext *avctx)
      * - Compression mode (none/huff)
      * And write the flags.
      */
-    c->flags  = (c->slices - 1) << 24;
+    c->flags  = (c->slices - 1U) << 24;
     c->flags |= 0 << 11; // bit field to signal interlaced encoding mode
     c->flags |= c->compression;
 
-- 
2.45.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 3/4] avcodec/mpeg12enc: Use av_rescale() in vbv_buffer_size computation

[Michael Niedermayer]

Fixes: signed integer overflow: 20 * 2314885530818453759 cannot be represented in type 'long'
Fixes: 69098/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MPEG2VIDEO_fuzzer-6107989688778752

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/mpeg12enc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/mpeg12enc.c b/libavcodec/mpeg12enc.c
index 140fda4bc23..f22ca8a19f7 100644
--- a/libavcodec/mpeg12enc.c
+++ b/libavcodec/mpeg12enc.c
@@ -332,7 +332,7 @@ static void mpeg1_encode_sequence_header(MpegEncContext *s)
     else
         /* VBV calculation: Scaled so that a VCD has the proper
          * VBV size of 40 kilobytes */
-        vbv_buffer_size = ((20 * s->bit_rate) / (1151929 / 2)) * 8 * 1024;
+        vbv_buffer_size = av_rescale_rnd(s->bit_rate, 20, 1151929 / 2, AV_ROUND_ZERO) * 8 * 1024;
     vbv_buffer_size = (vbv_buffer_size + 16383) / 16384;
 
     put_sbits(&s->pb, 18, v);
-- 
2.45.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH 4/4] avcodec/mpegvideo_enc: Do not duplicate pictures on shifting

[Michael Niedermayer]

Fixes: out of array access
Fixes: 69098/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MPEG2VIDEO_fuzzer-6107989688778752

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
---
 libavcodec/mpegvideo_enc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
index eda23056301..dc9a0009f53 100644
--- a/libavcodec/mpegvideo_enc.c
+++ b/libavcodec/mpegvideo_enc.c
@@ -1266,6 +1266,8 @@ static int load_input_picture(MpegEncContext *s, const AVFrame *pic_arg)
     /* shift buffer entries */
     for (int i = flush_offset; i <= MAX_B_FRAMES; i++)
         s->input_picture[i - flush_offset] = s->input_picture[i];
+    for (int i = MAX_B_FRAMES + 1 - flush_offset; i <= MAX_B_FRAMES; i++)
+        s->input_picture[i] = NULL;
 
     s->input_picture[encoding_delay] = pic;
 
-- 
2.45.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[Andreas Rheinhardt]

Michael Niedermayer:
> This moves computations out of a loop
> 
> Fixes: signed integer overflow: 31665934879948800 * 9998 cannot be represented in type 'long'
> Fixes: 69024/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-5949662967169024
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> ---
>  libavcodec/j2kenc.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
> index 8cf82f7216c..91e66d81048 100644
> --- a/libavcodec/j2kenc.c
> +++ b/libavcodec/j2kenc.c
> @@ -1349,7 +1349,7 @@ static void makelayers(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
>      }
>  }
>  
> -static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
> +static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda)
>  {
>      int passno, res = 0;
>      for (passno = 0; passno < cblk->npasses; passno++){
> @@ -1361,7 +1361,7 @@ static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
>          dd = cblk->passes[passno].disto
>             - (res ? cblk->passes[res-1].disto : 0);
>  
> -        if (((dd * dwt_norm) >> WMSEDEC_SHIFT) * dwt_norm >= dr * lambda)
> +        if (dd  >= dr * lambda)
>              res = passno+1;
>      }
>      return res;
> @@ -1384,11 +1384,12 @@ static void truncpasses(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
>                      Jpeg2000Band *band = reslevel->band + bandno;
>                      Jpeg2000Prec *prec = band->prec + precno;
>  
> +                    int64_t dwt_norm = dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15;
> +                    int64_t lambda_prime = av_rescale(s->lambda, 1 << WMSEDEC_SHIFT, dwt_norm * dwt_norm);
>                      for (cblkno = 0; cblkno < prec->nb_codeblocks_height * prec->nb_codeblocks_width; cblkno++){
>                          Jpeg2000Cblk *cblk = prec->cblk + cblkno;
>  
> -                        cblk->ninclpasses = getcut(cblk, s->lambda,
> -                                (int64_t)dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15);
> +                        cblk->ninclpasses = getcut(cblk, lambda_prime);
>                          cblk->layers[0].data_start = cblk->data;
>                          cblk->layers[0].cum_passes = cblk->ninclpasses;
>                          cblk->layers[0].npasses = cblk->ninclpasses;

Does this also fix the UB in the vsynth*-jpeg2000-yuva444p16 tests?

- Andreas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3180 bytes --]

On Fri, Jun 21, 2024 at 11:38:46AM +0200, Andreas Rheinhardt wrote:
> Michael Niedermayer:
> > This moves computations out of a loop
> > 
> > Fixes: signed integer overflow: 31665934879948800 * 9998 cannot be represented in type 'long'
> > Fixes: 69024/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-5949662967169024
> > 
> > Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> > ---
> >  libavcodec/j2kenc.c | 9 +++++----
> >  1 file changed, 5 insertions(+), 4 deletions(-)
> > 
> > diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
> > index 8cf82f7216c..91e66d81048 100644
> > --- a/libavcodec/j2kenc.c
> > +++ b/libavcodec/j2kenc.c
> > @@ -1349,7 +1349,7 @@ static void makelayers(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
> >      }
> >  }
> >  
> > -static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
> > +static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda)
> >  {
> >      int passno, res = 0;
> >      for (passno = 0; passno < cblk->npasses; passno++){
> > @@ -1361,7 +1361,7 @@ static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
> >          dd = cblk->passes[passno].disto
> >             - (res ? cblk->passes[res-1].disto : 0);
> >  
> > -        if (((dd * dwt_norm) >> WMSEDEC_SHIFT) * dwt_norm >= dr * lambda)
> > +        if (dd  >= dr * lambda)
> >              res = passno+1;
> >      }
> >      return res;
> > @@ -1384,11 +1384,12 @@ static void truncpasses(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
> >                      Jpeg2000Band *band = reslevel->band + bandno;
> >                      Jpeg2000Prec *prec = band->prec + precno;
> >  
> > +                    int64_t dwt_norm = dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15;
> > +                    int64_t lambda_prime = av_rescale(s->lambda, 1 << WMSEDEC_SHIFT, dwt_norm * dwt_norm);
> >                      for (cblkno = 0; cblkno < prec->nb_codeblocks_height * prec->nb_codeblocks_width; cblkno++){
> >                          Jpeg2000Cblk *cblk = prec->cblk + cblkno;
> >  
> > -                        cblk->ninclpasses = getcut(cblk, s->lambda,
> > -                                (int64_t)dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15);
> > +                        cblk->ninclpasses = getcut(cblk, lambda_prime);
> >                          cblk->layers[0].data_start = cblk->data;
> >                          cblk->layers[0].cum_passes = cblk->ninclpasses;
> >                          cblk->layers[0].npasses = cblk->ninclpasses;
> 
> Does this also fix the UB in the vsynth*-jpeg2000-yuva444p16 tests?

we will find out when this is applied, it looks like it might

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The real ebay dictionary, page 2
"100% positive feedback" - "All either got their money back or didnt complain"
"Best seller ever, very honest" - "Seller refunded buyer after failed scam"

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[Sean McGovern]

Hi,


On Fri, Jun 21, 2024, 05:39 Andreas Rheinhardt <
andreas.rheinhardt@outlook.com> wrote:

> Michael Niedermayer:
> > This moves computations out of a loop
> >
> > Fixes: signed integer overflow: 31665934879948800 * 9998 cannot be
> represented in type 'long'
> > Fixes:
> 69024/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-5949662967169024
> >
> > Found-by: continuous fuzzing process
> https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> > ---
> >  libavcodec/j2kenc.c | 9 +++++----
> >  1 file changed, 5 insertions(+), 4 deletions(-)
> >
> > diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
> > index 8cf82f7216c..91e66d81048 100644
> > --- a/libavcodec/j2kenc.c
> > +++ b/libavcodec/j2kenc.c
> > @@ -1349,7 +1349,7 @@ static void makelayers(Jpeg2000EncoderContext *s,
> Jpeg2000Tile *tile)
> >      }
> >  }
> >
> > -static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
> > +static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda)
> >  {
> >      int passno, res = 0;
> >      for (passno = 0; passno < cblk->npasses; passno++){
> > @@ -1361,7 +1361,7 @@ static int getcut(Jpeg2000Cblk *cblk, uint64_t
> lambda, int dwt_norm)
> >          dd = cblk->passes[passno].disto
> >             - (res ? cblk->passes[res-1].disto : 0);
> >
> > -        if (((dd * dwt_norm) >> WMSEDEC_SHIFT) * dwt_norm >= dr *
> lambda)
> > +        if (dd  >= dr * lambda)
> >              res = passno+1;
> >      }
> >      return res;
> > @@ -1384,11 +1384,12 @@ static void truncpasses(Jpeg2000EncoderContext
> *s, Jpeg2000Tile *tile)
> >                      Jpeg2000Band *band = reslevel->band + bandno;
> >                      Jpeg2000Prec *prec = band->prec + precno;
> >
> > +                    int64_t dwt_norm = dwt_norms[codsty->transform ==
> FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15;
> > +                    int64_t lambda_prime = av_rescale(s->lambda, 1 <<
> WMSEDEC_SHIFT, dwt_norm * dwt_norm);
> >                      for (cblkno = 0; cblkno <
> prec->nb_codeblocks_height * prec->nb_codeblocks_width; cblkno++){
> >                          Jpeg2000Cblk *cblk = prec->cblk + cblkno;
> >
> > -                        cblk->ninclpasses = getcut(cblk, s->lambda,
> > -                                (int64_t)dwt_norms[codsty->transform ==
> FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15);
> > +                        cblk->ninclpasses = getcut(cblk, lambda_prime);
> >                          cblk->layers[0].data_start = cblk->data;
> >                          cblk->layers[0].cum_passes = cblk->ninclpasses;
> >                          cblk->layers[0].npasses = cblk->ninclpasses;
>
> Does this also fix the UB in the vsynth*-jpeg2000-yuva444p16 tests?
>
> - Andreas
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>

I'm on staycation for my local jazz festival all week.
Was anyone able to test if this fixed the stuff Andreas mentioned?

-- Sean McGovern

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 4/4] avcodec/mpegvideo_enc: Do not duplicate pictures on shifting

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 748 bytes --]

On Thu, Jun 20, 2024 at 09:35:04PM +0200, Michael Niedermayer wrote:
> Fixes: out of array access
> Fixes: 69098/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MPEG2VIDEO_fuzzer-6107989688778752
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> ---
>  libavcodec/mpegvideo_enc.c | 2 ++
>  1 file changed, 2 insertions(+)

will apply


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The real ebay dictionary, page 2
"100% positive feedback" - "All either got their money back or didnt complain"
"Best seller ever, very honest" - "Seller refunded buyer after failed scam"

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3380 bytes --]

On Fri, Jun 21, 2024 at 11:09:45PM +0200, Michael Niedermayer wrote:
> On Fri, Jun 21, 2024 at 11:38:46AM +0200, Andreas Rheinhardt wrote:
> > Michael Niedermayer:
> > > This moves computations out of a loop
> > > 
> > > Fixes: signed integer overflow: 31665934879948800 * 9998 cannot be represented in type 'long'
> > > Fixes: 69024/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-5949662967169024
> > > 
> > > Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > > Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> > > ---
> > >  libavcodec/j2kenc.c | 9 +++++----
> > >  1 file changed, 5 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/libavcodec/j2kenc.c b/libavcodec/j2kenc.c
> > > index 8cf82f7216c..91e66d81048 100644
> > > --- a/libavcodec/j2kenc.c
> > > +++ b/libavcodec/j2kenc.c
> > > @@ -1349,7 +1349,7 @@ static void makelayers(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
> > >      }
> > >  }
> > >  
> > > -static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
> > > +static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda)
> > >  {
> > >      int passno, res = 0;
> > >      for (passno = 0; passno < cblk->npasses; passno++){
> > > @@ -1361,7 +1361,7 @@ static int getcut(Jpeg2000Cblk *cblk, uint64_t lambda, int dwt_norm)
> > >          dd = cblk->passes[passno].disto
> > >             - (res ? cblk->passes[res-1].disto : 0);
> > >  
> > > -        if (((dd * dwt_norm) >> WMSEDEC_SHIFT) * dwt_norm >= dr * lambda)
> > > +        if (dd  >= dr * lambda)
> > >              res = passno+1;
> > >      }
> > >      return res;
> > > @@ -1384,11 +1384,12 @@ static void truncpasses(Jpeg2000EncoderContext *s, Jpeg2000Tile *tile)
> > >                      Jpeg2000Band *band = reslevel->band + bandno;
> > >                      Jpeg2000Prec *prec = band->prec + precno;
> > >  
> > > +                    int64_t dwt_norm = dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15;
> > > +                    int64_t lambda_prime = av_rescale(s->lambda, 1 << WMSEDEC_SHIFT, dwt_norm * dwt_norm);
> > >                      for (cblkno = 0; cblkno < prec->nb_codeblocks_height * prec->nb_codeblocks_width; cblkno++){
> > >                          Jpeg2000Cblk *cblk = prec->cblk + cblkno;
> > >  
> > > -                        cblk->ninclpasses = getcut(cblk, s->lambda,
> > > -                                (int64_t)dwt_norms[codsty->transform == FF_DWT53][bandpos][lev] * (int64_t)band->i_stepsize >> 15);
> > > +                        cblk->ninclpasses = getcut(cblk, lambda_prime);
> > >                          cblk->layers[0].data_start = cblk->data;
> > >                          cblk->layers[0].cum_passes = cblk->ninclpasses;
> > >                          cblk->layers[0].npasses = cblk->ninclpasses;
> > 
> > Does this also fix the UB in the vsynth*-jpeg2000-yuva444p16 tests?
> 
> we will find out when this is applied, it looks like it might

added a note that this may fix UB in these tests to teh commit message and
will apply

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

it is not once nor twice but times without number that the same ideas make
their appearance in the world. -- Aristotle

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH 1/4] avcodec/j2kenc: Merge dwt_norm into lambda

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 741 bytes --]

On Thu, Jun 20, 2024 at 09:35:01PM +0200, Michael Niedermayer wrote:
> This moves computations out of a loop
> 
> Fixes: signed integer overflow: 31665934879948800 * 9998 cannot be represented in type 'long'
> Fixes: 69024/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_JPEG2000_fuzzer-5949662967169024
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
> ---
>  libavcodec/j2kenc.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)

will apply patchset

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

You can kill me, but you cannot change the truth.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".