From: "Wang, Fei W" <fei.w.wang-at-intel.com@ffmpeg.org> To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org> Subject: Re: [FFmpeg-devel] [PATCH v2 1/4] lavc/vaapi_encode_h265: Add P frame to GPB frame support for hevc_vaapi Date: Wed, 23 Feb 2022 13:33:23 +0000 Message-ID: <PH0PR11MB5030F9972163C8BA07438791CD3C9@PH0PR11MB5030.namprd11.prod.outlook.com> (raw) In-Reply-To: <757b53c74767bedb2f98a1373213feca3efd98b3.camel@intel.com> > -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Wang, > Fei W > Sent: Tuesday, February 22, 2022 4:49 PM > To: ffmpeg-devel@ffmpeg.org > Subject: Re: [FFmpeg-devel] [PATCH v2 1/4] lavc/vaapi_encode_h265: Add P > frame to GPB frame support for hevc_vaapi > > On Tue, 2022-02-22 at 05:46 +0000, Xiang, Haihao wrote: > > On Mon, 2022-02-21 at 12:06 +0000, Mark Thompson wrote: > > > On 21/02/2022 02:13, Fei Wang wrote: > > > > From: Linjie Fu <linjie.fu@intel.com> > > > > > > > > Use GPB frames to replace regular P frames if backend driver does > > > > not support it. > > > > > > > > - GPB: > > > > Generalized P and B picture. P frames replaced by B frames > > > > with > > > > forward-predict only, L0 == L1. Normal B frames still have 2 > > > > different ref_lists and allow bi-prediction > > > > > > > > Signed-off-by: Linjie Fu <linjie.fu@intel.com> > > > > Signed-off-by: Fei Wang <fei.w.wang@intel.com> > > > > --- > > > > libavcodec/vaapi_encode.c | 33 > > > > +++++++++++++++++++++++++++++++-- > > > > libavcodec/vaapi_encode.h | 1 + > > > > libavcodec/vaapi_encode_h265.c | 15 +++++++++++++++ > > > > 3 files changed, 47 insertions(+), 2 deletions(-) > > > > > > This always fails immediately on current iHD > > > (7e357b4bea76b2fe2522e6af41ae02ea69cec49e): > > > > > > $ ./ffmpeg_g -v 44 -y -hwaccel vaapi -hwaccel_output_format vaapi -i > > > in.mp4 -an -c:v hevc_vaapi -low_power 1 out.mp4 ... > > > [hevc_vaapi @ 0x560e81d45e80] Using input frames context (format > > > vaapi) with > > > hevc_vaapi encoder. > > > [hevc_vaapi @ 0x560e81d45e80] Input surface format is nv12. > > > [hevc_vaapi @ 0x560e81d45e80] Using VAAPI profile VAProfileHEVCMain > > > (17). > > > [hevc_vaapi @ 0x560e81d45e80] Using VAAPI entrypoint > > > VAEntrypointEncSliceLP (8). > > > [hevc_vaapi @ 0x560e81d45e80] Using VAAPI render target format > > > YUV420 (0x1). > > > [hevc_vaapi @ 0x560e81d45e80] Using CTU size 64x64, min CB size 8x8. > > > [hevc_vaapi @ 0x560e81d45e80] No quality level set; using default > > > (25). > > > [hevc_vaapi @ 0x560e81d45e80] RC mode: CQP. > > > [hevc_vaapi @ 0x560e81d45e80] RC quality: 25. > > > [hevc_vaapi @ 0x560e81d45e80] RC framerate: 30000/1001 (29.97 fps). > > > [hevc_vaapi @ 0x560e81d45e80] Use GPB B frames to replace regular P > > > frames. > > > [hevc_vaapi @ 0x560e81d45e80] Using intra, GPB-B-frames and B- > > > frames (supported references: 3 / 3). > > > [hevc_vaapi @ 0x560e81d45e80] All wanted packed headers available > > > (wanted 0xd, found 0x1f). > > > [hevc_vaapi @ 0x560e81d45e80] Using level 4. > > > ... > > > [hevc_vaapi @ 0x560e81d45e80] Failed to end picture encode issue: > > > 24 (internal > > > encoding error). > > > [hevc_vaapi @ 0x560e81d45e80] Encode failed: -5. > > > Video encoding failed > > > ... > > > $ cat /proc/cpuinfo | grep 'model name' | head -1 > > > model name : Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz > > > $ uname -v > > > #1 SMP PREEMPT Debian 5.16.7-2 (2022-02-09) > > > > > > Do you get this too, or is your setup different somehow? > > > > Hi Mark, > > > > I tested this patchset with iHD > > (7e357b4bea76b2fe2522e6af41ae02ea69cec49e) on CFL (low_power=0), RKL > > and DG1, i965 on SKL, and didn't see this issue before. > > This day I reproduced this issue on ICL. It seems iHD driver doesn't > > return right values for ICL. > > Thanks Mark to report this issue on ICL. > > I tested on TGL and CFL before submitted this patch set, all looks good. I will > check why this fail on ICL. > > And will fix your other comments together in next version. > > Fei > Thanks > > > > > > > > diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c > > > > index 3bf379b1a0..95eca7c288 100644 > > > > --- a/libavcodec/vaapi_encode.c > > > > +++ b/libavcodec/vaapi_encode.c > > > > @@ -1845,6 +1845,30 @@ static av_cold int > > > > vaapi_encode_init_gop_structure(AVCodecContext *avctx) > > > > ref_l1 = attr.value >> 16 & 0xffff; > > > > } > > > > > > > > + ctx->p_to_gpb = 0; > > > > + > > > > +#if VA_CHECK_VERSION(1, 9, 0) > > > > + attr = (VAConfigAttrib) { VAConfigAttribPredictionDirection > > > > }; > > > > + vas = vaGetConfigAttributes(ctx->hwctx->display, > > > > + ctx->va_profile, > > > > + ctx->va_entrypoint, > > > > + &attr, 1); > > > > > > This probably shouldn't be done at all if the user has selected a > > > codec without B-frames or a configuration which is intra-only, > > > because the log message is confusing: > > > > > > [mjpeg_vaapi @ 0x55b90d72ee00] Driver does not report whether > > > support GPB, use regular P frames. > > > [mjpeg_vaapi @ 0x55b90d72ee00] Using intra frames only. > > > > > > > + if (vas != VA_STATUS_SUCCESS) { > > > > + av_log(avctx, AV_LOG_WARNING, "Failed to query > > > > prediction direction > > > > " > > > > + "attribute: %d (%s).\n", vas, vaErrorStr(vas)); > > > > > > And fail? > > > > 4/4 also ignores the error. It would be better to handle the error in > > the same way and update 4/4 too. > > > > > > + } else if (attr.value == VA_ATTRIB_NOT_SUPPORTED) { > > > > + av_log(avctx, AV_LOG_VERBOSE, "Driver does not report > > > > whether " > > > > + "support GPB, use regular P frames.\n"); > > > > > > "support GPB" is a strange thing to say. It's a constraint - any > > > driver which supports B-frames will let you have the same thing in > > > both RefPicLists, but some require it because they don't support > > > P-frames. > > > > > > So maybe something like "Driver does not report any additional > > > prediction constraints, using P-frames."? > > > > > > > + } else { > > > > + if (attr.value & VA_PREDICTION_DIRECTION_BI_NOT_EMPTY) { > > > > + ctx->p_to_gpb = 1; > > > > + av_log(avctx, AV_LOG_VERBOSE, "Use GPB B frames to > > > > replace " > > > > + "regular P frames.\n"); > > > > > > Maybe "Driver does not support P-frames, replacing them with B- > > > frames."? > > > > > > > + } else > > > > + av_log(avctx, AV_LOG_VERBOSE, "Use regular P > > > > frames.\n"); > > > > + } > > > > +#endif > > > > + > > > > if (ctx->codec->flags & FLAG_INTRA_ONLY || > > > > avctx->gop_size <= 1) { > > > > av_log(avctx, AV_LOG_VERBOSE, "Using intra frames > > > > only.\n"); @@ -1861,8 +1885,13 @@ static av_cold int > > > > vaapi_encode_init_gop_structure(AVCodecContext *avctx) > > > > ctx->p_per_i = INT_MAX; > > > > ctx->b_per_p = 0; > > > > } else { > > > > - av_log(avctx, AV_LOG_VERBOSE, "Using intra, P- and B- > > > > frames " > > > > - "(supported references: %d / %d).\n", ref_l0, > > > > ref_l1); > > > > + if (ctx->p_to_gpb) > > > > + av_log(avctx, AV_LOG_VERBOSE, "Using intra, GPB-B- > > > > frames and " > > > > + "B-frames (supported references: %d / %d).\n", > > > > + ref_l0, ref_l1); > > > > > > Seems easier to just say intra and B-frames (though this isn't > > > really adding anything to the message above - it's still effectively > > > a P-frame referring to one previous frame, just using B-slices to do > > > it in a strange way). > > > > > > > + else > > > > + av_log(avctx, AV_LOG_VERBOSE, "Using intra, P- and B- > > > > frames " > > > > + "(supported references: %d / %d).\n", ref_l0, > > > > ref_l1); > > > > ctx->gop_size = avctx->gop_size; > > > > ctx->p_per_i = INT_MAX; > > > > ctx->b_per_p = avctx->max_b_frames; diff --git > > > > a/libavcodec/vaapi_encode.h b/libavcodec/vaapi_encode.h index > > > > b41604a883..61c5615eb8 100644 > > > > --- a/libavcodec/vaapi_encode.h > > > > +++ b/libavcodec/vaapi_encode.h > > > > @@ -328,6 +328,7 @@ typedef struct VAAPIEncodeContext { > > > > int idr_counter; > > > > int gop_counter; > > > > int end_of_stream; > > > > + int p_to_gpb; > > > > > > > > // Whether the driver supports ROI at all. > > > > int roi_allowed; > > > > diff --git a/libavcodec/vaapi_encode_h265.c > > > > b/libavcodec/vaapi_encode_h265.c index e1dc53dfa9..ea45893508 > > > > 100644 > > > > --- a/libavcodec/vaapi_encode_h265.c > > > > +++ b/libavcodec/vaapi_encode_h265.c > > > > @@ -886,6 +886,7 @@ static int > > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > > > > VAAPIEncodePictu > > > > re *pic, > > > > VAAPIEncodeSlice > > > > *slice) > > > > { > > > > + VAAPIEncodeContext *ctx = avctx->priv_data; > > > > VAAPIEncodeH265Context *priv = avctx->priv_data; > > > > VAAPIEncodeH265Picture *hpic = pic->priv_data; > > > > const H265RawSPS *sps = &priv->raw_sps; > > > > @@ -908,6 +909,9 @@ static int > > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > > > > > > > > sh->slice_type = hpic->slice_type; > > > > > > > > + if (sh->slice_type == HEVC_SLICE_P && ctx->p_to_gpb) > > > > + sh->slice_type = HEVC_SLICE_B; > > > > + > > > > sh->slice_pic_order_cnt_lsb = hpic->pic_order_cnt & > > > > (1 << (sps->log2_max_pic_order_cnt_lsb_minus4 + 4)) - 1; > > > > > > > > @@ -1066,6 +1070,9 @@ static int > > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > > > > av_assert0(pic->type == PICTURE_TYPE_P || > > > > pic->type == PICTURE_TYPE_B); > > > > vslice->ref_pic_list0[0] = vpic->reference_frames[0]; > > > > + if (ctx->p_to_gpb && pic->type == PICTURE_TYPE_P) > > > > + // Reference for GPB B-frame, L0 == L1 > > > > + vslice->ref_pic_list1[0] = vpic- > > > > >reference_frames[0]; > > > > } > > > > if (pic->nb_refs >= 2) { > > > > // Forward reference for B-frame. > > > > @@ -1073,6 +1080,14 @@ static int > > > > vaapi_encode_h265_init_slice_params(AVCodecContext *avctx, > > > > vslice->ref_pic_list1[0] = vpic->reference_frames[1]; > > > > } > > > > > > > > + if (pic->type == PICTURE_TYPE_P && ctx->p_to_gpb) { > > > > + vslice->slice_type = HEVC_SLICE_B; > > > > + for (i = 0; i < FF_ARRAY_ELEMS(vslice->ref_pic_list0); > > > > i++) { > > > > + vslice->ref_pic_list1[i].picture_id = vslice- > > > > > ref_pic_list0[i].picture_id; > > > > + vslice->ref_pic_list1[i].flags = vslice- > > > > > ref_pic_list0[i].flags; > > > > + } > > > > + } > > > > > > Is it sensible to implement this properly in the generic code rather > > > than having special ad-hoc code here? > > > > > > (Is there any chance this will also be a thing for e.g. H.264 or > > > H.266 in > > > future?) > > > > Considering that VAConfigAttribPredictionDirection is not a specific > > attribute for hevc in VA-API, I think it is acceptable to implement > > this in the generic code. I tried to add the same reference twice into VAAPIEncodePicture.refs, and h265_vaapi can get the correct L0/L1 and picture type without any change, test code: https://github.com/intel-media-ci/ffmpeg/pull/485/commits/f98fa04fe26333f09cdc9a1a11b0bc65783f9ae4 But this change breaks slice header's information get by VAAPIEncodePicture.refs/nb_refs: https://github.com/intel-media-ci/ffmpeg/blob/b14321e307fcd662f8d9aaa17c7f297cba2cbe4a/libavcodec/vaapi_encode_h265.c#L927 A tricky way is to reconstruct pic->refs/nb_refs here and remove the duplicated reference by using pic->refs[i]->priv_data. pic_order_cnt. Seems this way looks even worse compare to the current implementation in this patch. Not sure if you have any better idea? Thanks Fei > > > > Thanks > > Haihao > > > > > > + > > > > return 0; > > > > } > > > > > > > > > > - Mark > > > _______________________________________________ > > > ffmpeg-devel mailing list > > > ffmpeg-devel@ffmpeg.org > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > > > To unsubscribe, visit link above, or email > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe". > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org > with subject "unsubscribe". _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
next prev parent reply other threads:[~2022-02-23 13:33 UTC|newest] Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-02-21 2:13 Fei Wang 2022-02-21 2:13 ` [FFmpeg-devel] [PATCH v2 2/4] vaapi_encode: Move block size calculation after entrypoint selection Fei Wang 2022-02-21 2:13 ` [FFmpeg-devel] [PATCH v2 3/4] vaapi_encode_h265: Explicitly set and correct some flags Fei Wang 2022-02-21 2:13 ` [FFmpeg-devel] [PATCH v2 4/4] vaapi_encode_h265: Query encoding block sizes and features Fei Wang 2022-02-21 6:30 ` [FFmpeg-devel] [PATCH v2 1/4] lavc/vaapi_encode_h265: Add P frame to GPB frame support for hevc_vaapi Xiang, Haihao 2022-02-21 12:06 ` Mark Thompson 2022-02-22 5:46 ` Xiang, Haihao 2022-02-22 8:49 ` Wang, Fei W 2022-02-23 13:33 ` Wang, Fei W [this message] 2022-03-04 8:37 ` Wang, Fei W
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=PH0PR11MB5030F9972163C8BA07438791CD3C9@PH0PR11MB5030.namprd11.prod.outlook.com \ --to=fei.w.wang-at-intel.com@ffmpeg.org \ --cc=ffmpeg-devel@ffmpeg.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel This inbox may be cloned and mirrored by anyone: git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \ ffmpegdev@gitmailbox.com public-inbox-index ffmpegdev Example config snippet for mirrors. AGPL code for this site: git clone https://public-inbox.org/public-inbox.git