Re: [FFmpeg-devel] [PATCH] avcodec/nvdec_mpeg2: fix order of quant matrix coefficients

From: Hendrik Leppkes <h.leppkes@gmail.com>
To: ffmpeg-devel@ffmpeg.org
Subject: Re: [FFmpeg-devel] [PATCH] avcodec/nvdec_mpeg2: fix order of quant matrix coefficients
Date: Mon, 15 May 2023 13:30:59 +0200
Message-ID: <CA+anqdzxX3FVyAsr6nJNfv3=DgjDST0E_DSwn-6G2PX2M4zG8Q@mail.gmail.com> (raw)
In-Reply-To: <20230515105104.1162-1-h.leppkes@gmail.com>

On Mon, May 15, 2023 at 12:51 PM Hendrik Leppkes <h.leppkes@gmail.com> wrote:
>
> mpeg2dec stores them permutated for the IDCT, nvdec expects them in
> plain raster order.
> ---
>  libavcodec/nvdec_mpeg12.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/libavcodec/nvdec_mpeg12.c b/libavcodec/nvdec_mpeg12.c
> index e10735587d..3b9ff60734 100644
> --- a/libavcodec/nvdec_mpeg12.c
> +++ b/libavcodec/nvdec_mpeg12.c
> @@ -83,8 +83,9 @@ static int nvdec_mpeg12_start_frame(AVCodecContext *avctx, const uint8_t *buffer
>      };
>
>      for (i = 0; i < 64; ++i) {
> -        ppc->QuantMatrixIntra[i] = s->intra_matrix[i];
> -        ppc->QuantMatrixInter[i] = s->inter_matrix[i];
> +        int n = s->idsp.idct_permutation[i];
> +        ppc->QuantMatrixIntra[i] = s->intra_matrix[n];
> +        ppc->QuantMatrixInter[i] = s->inter_matrix[n];
>      }
>
>      return 0;
> --
> 2.40.1.windows.1
>

Apparently mpeg4 and VDPAU have the same issue. I can test and fix
mpeg4, but do not have VDPAU setup, so .. untested commits incoming?

- Hendrik
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".