[FFmpeg-devel] [PATCH] lavc/vvc: Fix derivation of inverse LMCS idx

[FFmpeg-devel] [PATCH] lavc/vvc: Fix derivation of inverse LMCS idx

[Frank Plowman]

The clamping of idxYInv from H.266(V3) section 8.8.2.3 was missing.
This could lead to OOB reads from lmcs->pivot or input_pivot.

I also changed the derivation of the forward LMCS idx to use a shift
rather than a division for speed and as this is actually how the
variable is declared in the specification (8.7.5.2).

Signed-off-by: Frank Plowman <post@frankplowman.com>
---
 libavcodec/vvc/ps.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/libavcodec/vvc/ps.c b/libavcodec/vvc/ps.c
index 01b4615eda..fae6655cc0 100644
--- a/libavcodec/vvc/ps.c
+++ b/libavcodec/vvc/ps.c
@@ -786,7 +786,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const H266RawAPS *rlmcs, const H266Raw
 
     //derive lmcs_fwd_lut
     for (uint16_t sample = 0; sample < max; sample++) {
-        const int idx_y = sample / org_cw;
+        const int idx_y = sample >> shift;
         const uint16_t fwd_sample = lmcs_derive_lut_sample(sample, lmcs->pivot,
             input_pivot, scale_coeff, idx_y, max);
         if (bit_depth > 8)
@@ -802,6 +802,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const H266RawAPS *rlmcs, const H266Raw
         uint16_t inv_sample;
         while (i <= lmcs->max_bin_idx && sample >= lmcs->pivot[i + 1])
             i++;
+        i = FFMIN(i, LMCS_MAX_BIN_SIZE - 1);
 
         inv_sample = lmcs_derive_lut_sample(sample, input_pivot, lmcs->pivot,
             inv_scale_coeff, i, max);
-- 
2.47.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] lavc/vvc: Fix derivation of inverse LMCS idx

[Frank Plowman]

Ping

On 02/02/2025 12:10, Frank Plowman wrote:
> The clamping of idxYInv from H.266(V3) section 8.8.2.3 was missing.
> This could lead to OOB reads from lmcs->pivot or input_pivot.
> 
> I also changed the derivation of the forward LMCS idx to use a shift
> rather than a division for speed and as this is actually how the
> variable is declared in the specification (8.7.5.2).
> 
> Signed-off-by: Frank Plowman <post@frankplowman.com>
> ---
>  libavcodec/vvc/ps.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/libavcodec/vvc/ps.c b/libavcodec/vvc/ps.c
> index 01b4615eda..fae6655cc0 100644
> --- a/libavcodec/vvc/ps.c
> +++ b/libavcodec/vvc/ps.c
> @@ -786,7 +786,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const H266RawAPS *rlmcs, const H266Raw
>  
>      //derive lmcs_fwd_lut
>      for (uint16_t sample = 0; sample < max; sample++) {
> -        const int idx_y = sample / org_cw;
> +        const int idx_y = sample >> shift;
>          const uint16_t fwd_sample = lmcs_derive_lut_sample(sample, lmcs->pivot,
>              input_pivot, scale_coeff, idx_y, max);
>          if (bit_depth > 8)
> @@ -802,6 +802,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const H266RawAPS *rlmcs, const H266Raw
>          uint16_t inv_sample;
>          while (i <= lmcs->max_bin_idx && sample >= lmcs->pivot[i + 1])
>              i++;
> +        i = FFMIN(i, LMCS_MAX_BIN_SIZE - 1);
>  
>          inv_sample = lmcs_derive_lut_sample(sample, input_pivot, lmcs->pivot,
>              inv_scale_coeff, i, max);

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] lavc/vvc: Fix derivation of inverse LMCS idx

[Nuo Mi]

On Thu, Feb 20, 2025 at 4:33 AM Frank Plowman <post@frankplowman.com> wrote:

> Ping
>
Sorry for missing this.
Thank you for the patch, applied

>
> On 02/02/2025 12:10, Frank Plowman wrote:
> > The clamping of idxYInv from H.266(V3) section 8.8.2.3 was missing.
> > This could lead to OOB reads from lmcs->pivot or input_pivot.
> >
> > I also changed the derivation of the forward LMCS idx to use a shift
> > rather than a division for speed and as this is actually how the
> > variable is declared in the specification (8.7.5.2).
> >
> > Signed-off-by: Frank Plowman <post@frankplowman.com>
> > ---
> >  libavcodec/vvc/ps.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavcodec/vvc/ps.c b/libavcodec/vvc/ps.c
> > index 01b4615eda..fae6655cc0 100644
> > --- a/libavcodec/vvc/ps.c
> > +++ b/libavcodec/vvc/ps.c
> > @@ -786,7 +786,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const
> H266RawAPS *rlmcs, const H266Raw
> >
> >      //derive lmcs_fwd_lut
> >      for (uint16_t sample = 0; sample < max; sample++) {
> > -        const int idx_y = sample / org_cw;
> > +        const int idx_y = sample >> shift;
> >          const uint16_t fwd_sample = lmcs_derive_lut_sample(sample,
> lmcs->pivot,
> >              input_pivot, scale_coeff, idx_y, max);
> >          if (bit_depth > 8)
> > @@ -802,6 +802,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const
> H266RawAPS *rlmcs, const H266Raw
> >          uint16_t inv_sample;
> >          while (i <= lmcs->max_bin_idx && sample >= lmcs->pivot[i + 1])
> >              i++;
> > +        i = FFMIN(i, LMCS_MAX_BIN_SIZE - 1);
> >
> >          inv_sample = lmcs_derive_lut_sample(sample, input_pivot,
> lmcs->pivot,
> >              inv_scale_coeff, i, max);
>
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".