[FFmpeg-devel] [PATCH v2 1/3] lavc/qsv: allow to add more parameter buffers to QSV frame

[FFmpeg-devel] [PATCH v2 1/3] lavc/qsv: allow to add more parameter buffers to QSV frame

[Xiang, Haihao]

From: Haihao Xiang <haihao.xiang@intel.com>

Signed-off-by: Haihao Xiang <haihao.xiang@intel.com>
---
v2: rebased the patchset against the latest FFmpeg and added code to
make sure the corresponding extra parameter buffer is added for AV1
only.

 libavcodec/qsv.c          | 27 +++++++++++++++++++++++++++
 libavcodec/qsv_internal.h |  8 +++++++-
 libavcodec/qsvdec.c       |  8 +++++---
 3 files changed, 39 insertions(+), 4 deletions(-)

diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
index 9d08485c92..1a432dbd82 100644
--- a/libavcodec/qsv.c
+++ b/libavcodec/qsv.c
@@ -828,3 +828,30 @@ int ff_qsv_close_internal_session(QSVSession *qs)
 #endif
     return 0;
 }
+
+void ff_qsv_frame_add_ext_param (AVCodecContext *avctx, QSVFrame *frame,
+                                 mfxExtBuffer * param)
+{
+    int i;
+
+    for (i = 0; i < frame->num_ext_params; i++) {
+        mfxExtBuffer *ext_buffer = frame->ext_param[i];
+
+        if (ext_buffer->BufferId == param->BufferId) {
+            av_log(avctx, AV_LOG_WARNING, "A buffer with the same type has been "
+                   "added\n");
+            return;
+        }
+    }
+
+    if (frame->num_ext_params < QSV_MAX_FRAME_EXT_PARAMS) {
+        frame->ext_param[frame->num_ext_params] = param;
+        frame->num_ext_params++;
+        frame->surface.Data.NumExtParam = frame->num_ext_params;
+    } else {
+        av_log(avctx, AV_LOG_WARNING, "Ignore this extra buffer because do not "
+               "have enough space\n");
+    }
+
+
+}
diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
index fe9d5319c4..6a38e87d23 100644
--- a/libavcodec/qsv_internal.h
+++ b/libavcodec/qsv_internal.h
@@ -52,6 +52,8 @@
 
 #define QSV_MAX_ENC_PAYLOAD 2       // # of mfxEncodeCtrl payloads supported
 
+#define QSV_MAX_FRAME_EXT_PARAMS 4
+
 #define QSV_VERSION_ATLEAST(MAJOR, MINOR)   \
     (MFX_VERSION_MAJOR > (MAJOR) ||         \
      MFX_VERSION_MAJOR == (MAJOR) && MFX_VERSION_MINOR >= (MINOR))
@@ -74,7 +76,8 @@ typedef struct QSVFrame {
     mfxFrameSurface1 surface;
     mfxEncodeCtrl enc_ctrl;
     mfxExtDecodedFrameInfo dec_info;
-    mfxExtBuffer *ext_param;
+    mfxExtBuffer *ext_param[QSV_MAX_FRAME_EXT_PARAMS];
+    int num_ext_params;
 
     mfxPayload *payloads[QSV_MAX_ENC_PAYLOAD]; ///< used for enc_ctrl.Payload
 
@@ -138,4 +141,7 @@ int ff_qsv_init_session_frames(AVCodecContext *avctx, mfxSession *session,
 
 int ff_qsv_find_surface_idx(QSVFramesContext *ctx, QSVFrame *frame);
 
+void ff_qsv_frame_add_ext_param(AVCodecContext *avctx, QSVFrame *frame,
+                                mfxExtBuffer *param);
+
 #endif /* AVCODEC_QSV_INTERNAL_H */
diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
index d9e0fef1f1..783d252002 100644
--- a/libavcodec/qsvdec.c
+++ b/libavcodec/qsvdec.c
@@ -423,11 +423,13 @@ static int alloc_frame(AVCodecContext *avctx, QSVContext *q, QSVFrame *frame)
 
         frame->surface.Data.MemId = &q->frames_ctx.mids[ret];
     }
-    frame->surface.Data.ExtParam    = &frame->ext_param;
-    frame->surface.Data.NumExtParam = 1;
-    frame->ext_param                = (mfxExtBuffer*)&frame->dec_info;
+
+    frame->surface.Data.ExtParam    = frame->ext_param;
+    frame->surface.Data.NumExtParam = 0;
+    frame->num_ext_params           = 0;
     frame->dec_info.Header.BufferId = MFX_EXTBUFF_DECODED_FRAME_INFO;
     frame->dec_info.Header.BufferSz = sizeof(frame->dec_info);
+    ff_qsv_frame_add_ext_param(avctx, frame, (mfxExtBuffer *)&frame->dec_info);
 
     frame->used = 1;
 
-- 
2.17.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH v2 2/3] lavc/qsvdec: track the runtime session version

[Xiang, Haihao]

From: Haihao Xiang <haihao.xiang@intel.com>

We may check the runtime version for the given features

Signed-off-by: Haihao Xiang <haihao.xiang@intel.com>
---
 libavcodec/qsvdec.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
index 783d252002..8b83d5695f 100644
--- a/libavcodec/qsvdec.c
+++ b/libavcodec/qsvdec.c
@@ -59,6 +59,7 @@ static const AVRational mfx_tb = { 1, 90000 };
 typedef struct QSVContext {
     // the session used for decoding
     mfxSession session;
+    mfxVersion ver;
 
     // the session we allocated internally, in case the caller did not provide
     // one
@@ -202,6 +203,18 @@ static int qsv_init_session(AVCodecContext *avctx, QSVContext *q, mfxSession ses
         q->session = q->internal_qs.session;
     }
 
+    if (MFXQueryVersion(q->session, &q->ver) != MFX_ERR_NONE) {
+        av_log(avctx, AV_LOG_ERROR, "Error querying the session version. \n");
+        q->session = NULL;
+
+        if (q->internal_qs.session) {
+            MFXClose(q->internal_qs.session);
+            q->internal_qs.session = NULL;
+        }
+
+        return AVERROR_EXTERNAL;
+    }
+
     /* make sure the decoder is uninitialized */
     MFXVideoDECODE_Close(q->session);
 
-- 
2.17.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH v2 3/3] lavc/qsvdec: export AVFilmGrainParams side data

[Xiang, Haihao]

From: Haihao Xiang <haihao.xiang@intel.com>

When AV_CODEC_EXPORT_DATA_FILM_GRAIN is present, AV1 decoder should
disable film grain application and export the corresponding side data

Signed-off-by: Haihao Xiang <haihao.xiang@intel.com>
---
 libavcodec/qsv_internal.h |  3 ++
 libavcodec/qsvdec.c       | 91 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)

diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
index 6a38e87d23..58186ea7ca 100644
--- a/libavcodec/qsv_internal.h
+++ b/libavcodec/qsv_internal.h
@@ -76,6 +76,9 @@ typedef struct QSVFrame {
     mfxFrameSurface1 surface;
     mfxEncodeCtrl enc_ctrl;
     mfxExtDecodedFrameInfo dec_info;
+#if QSV_VERSION_ATLEAST(1, 34)
+    mfxExtAV1FilmGrainParam av1_film_grain_param;
+#endif
     mfxExtBuffer *ext_param[QSV_MAX_FRAME_EXT_PARAMS];
     int num_ext_params;
 
diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
index 8b83d5695f..32077ab31a 100644
--- a/libavcodec/qsvdec.c
+++ b/libavcodec/qsvdec.c
@@ -38,6 +38,7 @@
 #include "libavutil/pixfmt.h"
 #include "libavutil/time.h"
 #include "libavutil/imgutils.h"
+#include "libavutil/film_grain_params.h"
 
 #include "avcodec.h"
 #include "internal.h"
@@ -404,6 +405,11 @@ static int qsv_decode_header(AVCodecContext *avctx, QSVContext *q,
     param->ExtParam    = q->ext_buffers;
     param->NumExtParam = q->nb_ext_buffers;
 
+#if QSV_VERSION_ATLEAST(1, 34)
+    if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 1, 34) && avctx->codec_id == AV_CODEC_ID_AV1)
+        param->mfx.FilmGrain = (avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) ? 0 : param->mfx.FilmGrain;
+#endif
+
     return 0;
 }
 
@@ -443,6 +449,14 @@ static int alloc_frame(AVCodecContext *avctx, QSVContext *q, QSVFrame *frame)
     frame->dec_info.Header.BufferId = MFX_EXTBUFF_DECODED_FRAME_INFO;
     frame->dec_info.Header.BufferSz = sizeof(frame->dec_info);
     ff_qsv_frame_add_ext_param(avctx, frame, (mfxExtBuffer *)&frame->dec_info);
+#if QSV_VERSION_ATLEAST(1, 34)
+    if (QSV_RUNTIME_VERSION_ATLEAST(q->ver, 1, 34) && avctx->codec_id == AV_CODEC_ID_AV1) {
+        frame->av1_film_grain_param.Header.BufferId = MFX_EXTBUFF_AV1_FILM_GRAIN_PARAM;
+        frame->av1_film_grain_param.Header.BufferSz = sizeof(frame->av1_film_grain_param);
+        frame->av1_film_grain_param.FilmGrainFlags = 0;
+        ff_qsv_frame_add_ext_param(avctx, frame, (mfxExtBuffer *)&frame->av1_film_grain_param);
+    }
+#endif
 
     frame->used = 1;
 
@@ -513,6 +527,73 @@ static QSVFrame *find_frame(QSVContext *q, mfxFrameSurface1 *surf)
     return NULL;
 }
 
+#if QSV_VERSION_ATLEAST(1, 34)
+static int qsv_export_film_grain(AVCodecContext *avctx, mfxExtAV1FilmGrainParam *ext_param, AVFrame *frame)
+{
+    AVFilmGrainParams *fgp;
+    AVFilmGrainAOMParams *aom;
+    int i;
+
+    if (!(ext_param->FilmGrainFlags & MFX_FILM_GRAIN_APPLY))
+        return 0;
+
+    fgp = av_film_grain_params_create_side_data(frame);
+
+    if (!fgp)
+        return AVERROR(ENOMEM);
+
+    fgp->type = AV_FILM_GRAIN_PARAMS_AV1;
+    fgp->seed = ext_param->GrainSeed;
+    aom = &fgp->codec.aom;
+
+    aom->chroma_scaling_from_luma = !!(ext_param->FilmGrainFlags & MFX_FILM_GRAIN_CHROMA_SCALING_FROM_LUMA);
+    aom->scaling_shift = ext_param->GrainScalingMinus8 + 8;
+    aom->ar_coeff_lag = ext_param->ArCoeffLag;
+    aom->ar_coeff_shift = ext_param->ArCoeffShiftMinus6 + 6;
+    aom->grain_scale_shift = ext_param->GrainScaleShift;
+    aom->overlap_flag = !!(ext_param->FilmGrainFlags & MFX_FILM_GRAIN_OVERLAP);
+    aom->limit_output_range = !!(ext_param->FilmGrainFlags & MFX_FILM_GRAIN_CLIP_TO_RESTRICTED_RANGE);
+
+    aom->num_y_points = ext_param->NumYPoints;
+
+    for (i = 0; i < aom->num_y_points; i++) {
+        aom->y_points[i][0] = ext_param->PointY[i].Value;
+        aom->y_points[i][1] = ext_param->PointY[i].Scaling;
+    }
+
+    aom->num_uv_points[0] = ext_param->NumCbPoints;
+
+    for (i = 0; i < aom->num_uv_points[0]; i++) {
+        aom->uv_points[0][i][0] = ext_param->PointCb[i].Value;
+        aom->uv_points[0][i][1] = ext_param->PointCb[i].Scaling;
+    }
+
+    aom->num_uv_points[1] = ext_param->NumCrPoints;
+
+    for (i = 0; i < aom->num_uv_points[1]; i++) {
+        aom->uv_points[1][i][0] = ext_param->PointCr[i].Value;
+        aom->uv_points[1][i][1] = ext_param->PointCr[i].Scaling;
+    }
+
+    for (i = 0; i < 24; i++)
+        aom->ar_coeffs_y[i] = ext_param->ArCoeffsYPlus128[i] - 128;
+
+    for (i = 0; i < 25; i++) {
+        aom->ar_coeffs_uv[0][i] = ext_param->ArCoeffsCbPlus128[i] - 128;
+        aom->ar_coeffs_uv[1][i] = ext_param->ArCoeffsCrPlus128[i] - 128;
+    }
+
+    aom->uv_mult[0] = ext_param->CbMult;
+    aom->uv_mult[1] = ext_param->CrMult;
+    aom->uv_mult_luma[0] = ext_param->CbLumaMult;
+    aom->uv_mult_luma[1] = ext_param->CrLumaMult;
+    aom->uv_offset[0] = ext_param->CbOffset;
+    aom->uv_offset[1] = ext_param->CrOffset;
+
+    return 0;
+}
+#endif
+
 static int qsv_decode(AVCodecContext *avctx, QSVContext *q,
                       AVFrame *frame, int *got_frame,
                       const AVPacket *avpkt)
@@ -617,6 +698,16 @@ static int qsv_decode(AVCodecContext *avctx, QSVContext *q,
         outsurf = &out_frame->surface;
 
         frame->pts = MFX_PTS_TO_PTS(outsurf->Data.TimeStamp, avctx->pkt_timebase);
+#if QSV_VERSION_ATLEAST(1, 34)
+        if ((avctx->export_side_data & AV_CODEC_EXPORT_DATA_FILM_GRAIN) &&
+            QSV_RUNTIME_VERSION_ATLEAST(q->ver, 1, 34) &&
+            avctx->codec_id == AV_CODEC_ID_AV1) {
+            ret = qsv_export_film_grain(avctx, &out_frame->av1_film_grain_param, frame);
+
+            if (ret < 0)
+                return ret;
+        }
+#endif
 
         frame->repeat_pict =
             outsurf->Info.PicStruct & MFX_PICSTRUCT_FRAME_TRIPLING ? 4 :
-- 
2.17.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 1/3] lavc/qsv: allow to add more parameter buffers to QSV frame

[Xiang, Haihao]

On Mon, 2022-01-24 at 16:24 +0800, Xiang, Haihao wrote:
> From: Haihao Xiang <haihao.xiang@intel.com>
> 
> Signed-off-by: Haihao Xiang <haihao.xiang@intel.com>
> ---
> v2: rebased the patchset against the latest FFmpeg and added code to
> make sure the corresponding extra parameter buffer is added for AV1
> only.
> 
>  libavcodec/qsv.c          | 27 +++++++++++++++++++++++++++
>  libavcodec/qsv_internal.h |  8 +++++++-
>  libavcodec/qsvdec.c       |  8 +++++---
>  3 files changed, 39 insertions(+), 4 deletions(-)
> 
> diff --git a/libavcodec/qsv.c b/libavcodec/qsv.c
> index 9d08485c92..1a432dbd82 100644
> --- a/libavcodec/qsv.c
> +++ b/libavcodec/qsv.c
> @@ -828,3 +828,30 @@ int ff_qsv_close_internal_session(QSVSession *qs)
>  #endif
>      return 0;
>  }
> +
> +void ff_qsv_frame_add_ext_param (AVCodecContext *avctx, QSVFrame *frame,
> +                                 mfxExtBuffer * param)
> +{
> +    int i;
> +
> +    for (i = 0; i < frame->num_ext_params; i++) {
> +        mfxExtBuffer *ext_buffer = frame->ext_param[i];
> +
> +        if (ext_buffer->BufferId == param->BufferId) {
> +            av_log(avctx, AV_LOG_WARNING, "A buffer with the same type has
> been "
> +                   "added\n");
> +            return;
> +        }
> +    }
> +
> +    if (frame->num_ext_params < QSV_MAX_FRAME_EXT_PARAMS) {
> +        frame->ext_param[frame->num_ext_params] = param;
> +        frame->num_ext_params++;
> +        frame->surface.Data.NumExtParam = frame->num_ext_params;
> +    } else {
> +        av_log(avctx, AV_LOG_WARNING, "Ignore this extra buffer because do
> not "
> +               "have enough space\n");
> +    }
> +
> +
> +}
> diff --git a/libavcodec/qsv_internal.h b/libavcodec/qsv_internal.h
> index fe9d5319c4..6a38e87d23 100644
> --- a/libavcodec/qsv_internal.h
> +++ b/libavcodec/qsv_internal.h
> @@ -52,6 +52,8 @@
>  
>  #define QSV_MAX_ENC_PAYLOAD 2       // # of mfxEncodeCtrl payloads supported
>  
> +#define QSV_MAX_FRAME_EXT_PARAMS 4
> +
>  #define QSV_VERSION_ATLEAST(MAJOR, MINOR)   \
>      (MFX_VERSION_MAJOR > (MAJOR) ||         \
>       MFX_VERSION_MAJOR == (MAJOR) && MFX_VERSION_MINOR >= (MINOR))
> @@ -74,7 +76,8 @@ typedef struct QSVFrame {
>      mfxFrameSurface1 surface;
>      mfxEncodeCtrl enc_ctrl;
>      mfxExtDecodedFrameInfo dec_info;
> -    mfxExtBuffer *ext_param;
> +    mfxExtBuffer *ext_param[QSV_MAX_FRAME_EXT_PARAMS];
> +    int num_ext_params;
>  
>      mfxPayload *payloads[QSV_MAX_ENC_PAYLOAD]; ///< used for enc_ctrl.Payload
>  
> @@ -138,4 +141,7 @@ int ff_qsv_init_session_frames(AVCodecContext *avctx,
> mfxSession *session,
>  
>  int ff_qsv_find_surface_idx(QSVFramesContext *ctx, QSVFrame *frame);
>  
> +void ff_qsv_frame_add_ext_param(AVCodecContext *avctx, QSVFrame *frame,
> +                                mfxExtBuffer *param);
> +
>  #endif /* AVCODEC_QSV_INTERNAL_H */
> diff --git a/libavcodec/qsvdec.c b/libavcodec/qsvdec.c
> index d9e0fef1f1..783d252002 100644
> --- a/libavcodec/qsvdec.c
> +++ b/libavcodec/qsvdec.c
> @@ -423,11 +423,13 @@ static int alloc_frame(AVCodecContext *avctx, QSVContext
> *q, QSVFrame *frame)
>  
>          frame->surface.Data.MemId = &q->frames_ctx.mids[ret];
>      }
> -    frame->surface.Data.ExtParam    = &frame->ext_param;
> -    frame->surface.Data.NumExtParam = 1;
> -    frame->ext_param                = (mfxExtBuffer*)&frame->dec_info;
> +
> +    frame->surface.Data.ExtParam    = frame->ext_param;
> +    frame->surface.Data.NumExtParam = 0;
> +    frame->num_ext_params           = 0;
>      frame->dec_info.Header.BufferId = MFX_EXTBUFF_DECODED_FRAME_INFO;
>      frame->dec_info.Header.BufferSz = sizeof(frame->dec_info);
> +    ff_qsv_frame_add_ext_param(avctx, frame, (mfxExtBuffer *)&frame-
> >dec_info);
>  
>      frame->used = 1;

Will apply

-Haihao

>  
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 1/3] lavc/qsv: allow to add more parameter buffers to QSV frame

[Timo Rothenpieler]

[-- Attachment #1.1: Type: text/plain, Size: 189 bytes --]

On 27.01.2022 06:37, Xiang, Haihao wrote:
> Will apply
> 
> -Haihao
> 

Something in this patchset broke build on Windows:
https://github.com/BtbN/FFmpeg-Builds/runs/4991054208#step:4:9491

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4494 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 1/3] lavc/qsv: allow to add more parameter buffers to QSV frame

[Timo Rothenpieler]

[-- Attachment #1.1: Type: text/plain, Size: 315 bytes --]

On 29.01.2022 15:30, Timo Rothenpieler wrote:
> On 27.01.2022 06:37, Xiang, Haihao wrote:
>> Will apply
>>
>> -Haihao
>>
> 
> Something in this patchset broke build on Windows:
> https://github.com/BtbN/FFmpeg-Builds/runs/4991054208#step:4:9491

Seems like it really was the typo gcc suggests. Pushed a fix for it.

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4494 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 1/3] lavc/qsv: allow to add more parameter buffers to QSV frame

[Xiang, Haihao]

On Sat, 2022-01-29 at 15:41 +0100, Timo Rothenpieler wrote:
> On 29.01.2022 15:30, Timo Rothenpieler wrote:
> > On 27.01.2022 06:37, Xiang, Haihao wrote:
> > > Will apply
> > > 
> > > -Haihao
> > > 
> > 
> > Something in this patchset broke build on Windows:
> > https://github.com/BtbN/FFmpeg-Builds/runs/4991054208#step:4:9491
> 
> Seems like it really was the typo gcc suggests. Pushed a fix for it.

Sorry for this stupid error, and thanks for the patch, we'll check the ourinternal CI and review process. 

BRs
Haihao




_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".