[FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Cameron Gutman]

Some scenarios (such as game streaming or videoconferencing) may use CBR
to strictly cap the maximum encoded bitrate, but they don't mind the
bitrate falling below the target if the encoder doesn't need the
additional headroom.

Allow users to opt-out of filler data in CBR mode for those usecases
where it is unwanted.

Signed-off-by: Cameron Gutman <aicommander@gmail.com>
---
v2: Rebased to resolve conflicts against master
---
 libavcodec/nvenc.c     | 2 +-
 libavcodec/nvenc.h     | 1 +
 libavcodec/nvenc_av1.c | 2 ++
 3 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 0f5e772b3e..0f2336b0ec 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
 
     av1->idrPeriod = cc->gopLength;
 
-    if (IS_CBR(cc->rcParams.rateControlMode)) {
+    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
         av1->enableBitstreamPadding = 1;
     }
 
diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
index e035e123c6..3431457422 100644
--- a/libavcodec/nvenc.h
+++ b/libavcodec/nvenc.h
@@ -309,6 +309,7 @@ typedef struct NvencContext
     int unidir_b;
     int split_encode_mode;
     int mdm, cll;
+    int filler_data;
 } NvencContext;
 
 int ff_nvenc_encode_init(AVCodecContext *avctx);
diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
index 01626113ab..c00817af7b 100644
--- a/libavcodec/nvenc_av1.c
+++ b/libavcodec/nvenc_av1.c
@@ -156,6 +156,8 @@ static const AVOption options[] = {
                                                             OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
     { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
     { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
+    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
+                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
 #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
     { "tf_level",     "Specifies the strength of the temporal filtering",
                                                             OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },
-- 
2.49.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Timo Rothenpieler]

On 29.03.2025 21:58, Cameron Gutman wrote:
> Some scenarios (such as game streaming or videoconferencing) may use CBR
> to strictly cap the maximum encoded bitrate, but they don't mind the
> bitrate falling below the target if the encoder doesn't need the
> additional headroom.

But why aren't they using vbr with a maxrate then?
I didn't add this conditionally since when in CBR mode, you actually 
really want a constant bitrate.
If you're okay with it dropping lower, why not use another mode with a 
maxrate set?

> Allow users to opt-out of filler data in CBR mode for those usecases
> where it is unwanted.
> 
> Signed-off-by: Cameron Gutman <aicommander@gmail.com>
> ---
> v2: Rebased to resolve conflicts against master
> ---
>   libavcodec/nvenc.c     | 2 +-
>   libavcodec/nvenc.h     | 1 +
>   libavcodec/nvenc_av1.c | 2 ++
>   3 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> index 0f5e772b3e..0f2336b0ec 100644
> --- a/libavcodec/nvenc.c
> +++ b/libavcodec/nvenc.c
> @@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
>   
>       av1->idrPeriod = cc->gopLength;
>   
> -    if (IS_CBR(cc->rcParams.rateControlMode)) {
> +    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
>           av1->enableBitstreamPadding = 1;
>       }
>   
> diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
> index e035e123c6..3431457422 100644
> --- a/libavcodec/nvenc.h
> +++ b/libavcodec/nvenc.h
> @@ -309,6 +309,7 @@ typedef struct NvencContext
>       int unidir_b;
>       int split_encode_mode;
>       int mdm, cll;
> +    int filler_data;
>   } NvencContext;
>   
>   int ff_nvenc_encode_init(AVCodecContext *avctx);
> diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
> index 01626113ab..c00817af7b 100644
> --- a/libavcodec/nvenc_av1.c
> +++ b/libavcodec/nvenc_av1.c
> @@ -156,6 +156,8 @@ static const AVOption options[] = {
>                                                               OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>       { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>       { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> +    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
> +                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>   #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
>       { "tf_level",     "Specifies the strength of the temporal filtering",
>                                                               OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Cameron Gutman]

On Sat, Mar 29, 2025 at 4:02 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>
> On 29.03.2025 21:58, Cameron Gutman wrote:
> > Some scenarios (such as game streaming or videoconferencing) may use CBR
> > to strictly cap the maximum encoded bitrate, but they don't mind the
> > bitrate falling below the target if the encoder doesn't need the
> > additional headroom.
>
> But why aren't they using vbr with a maxrate then?
> I didn't add this conditionally since when in CBR mode, you actually
> really want a constant bitrate.
> If you're okay with it dropping lower, why not use another mode with a
> maxrate set?
>

As I understand it, the rate control behavior of CBR without filler is
different from
just using VBR, particularly for low latency applications. Nvidia recommends use
of CBR [0] (with or without filler depending on application needs [1])
rather than
using VBR for low latency applications.

Also FWIW, we don't set enableFillerDataInsertion for CBR on H.264 and HEVC,
so those will dip below the target bitrate in CBR mode while AV1 will not.

[0]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#recommended-nvenc-settings__table-nvenc-settings
[1]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#rate-control

> > Allow users to opt-out of filler data in CBR mode for those usecases
> > where it is unwanted.
> >
> > Signed-off-by: Cameron Gutman <aicommander@gmail.com>
> > ---
> > v2: Rebased to resolve conflicts against master
> > ---
> >   libavcodec/nvenc.c     | 2 +-
> >   libavcodec/nvenc.h     | 1 +
> >   libavcodec/nvenc_av1.c | 2 ++
> >   3 files changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> > index 0f5e772b3e..0f2336b0ec 100644
> > --- a/libavcodec/nvenc.c
> > +++ b/libavcodec/nvenc.c
> > @@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
> >
> >       av1->idrPeriod = cc->gopLength;
> >
> > -    if (IS_CBR(cc->rcParams.rateControlMode)) {
> > +    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
> >           av1->enableBitstreamPadding = 1;
> >       }
> >
> > diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
> > index e035e123c6..3431457422 100644
> > --- a/libavcodec/nvenc.h
> > +++ b/libavcodec/nvenc.h
> > @@ -309,6 +309,7 @@ typedef struct NvencContext
> >       int unidir_b;
> >       int split_encode_mode;
> >       int mdm, cll;
> > +    int filler_data;
> >   } NvencContext;
> >
> >   int ff_nvenc_encode_init(AVCodecContext *avctx);
> > diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
> > index 01626113ab..c00817af7b 100644
> > --- a/libavcodec/nvenc_av1.c
> > +++ b/libavcodec/nvenc_av1.c
> > @@ -156,6 +156,8 @@ static const AVOption options[] = {
> >                                                               OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >       { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >       { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> > +    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
> > +                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >   #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
> >       { "tf_level",     "Specifies the strength of the temporal filtering",
> >                                                               OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Timo Rothenpieler]

On 29.03.2025 22:17, Cameron Gutman wrote:
> On Sat, Mar 29, 2025 at 4:02 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>
>> On 29.03.2025 21:58, Cameron Gutman wrote:
>>> Some scenarios (such as game streaming or videoconferencing) may use CBR
>>> to strictly cap the maximum encoded bitrate, but they don't mind the
>>> bitrate falling below the target if the encoder doesn't need the
>>> additional headroom.
>>
>> But why aren't they using vbr with a maxrate then?
>> I didn't add this conditionally since when in CBR mode, you actually
>> really want a constant bitrate.
>> If you're okay with it dropping lower, why not use another mode with a
>> maxrate set?
>>
> 
> As I understand it, the rate control behavior of CBR without filler is
> different from
> just using VBR, particularly for low latency applications. Nvidia recommends use
> of CBR [0] (with or without filler depending on application needs [1])
> rather than
> using VBR for low latency applications.
> 
> Also FWIW, we don't set enableFillerDataInsertion for CBR on H.264 and HEVC,
> so those will dip below the target bitrate in CBR mode while AV1 will not.

We probably should really. In the past outputBufferingPeriodSEI 
controlled this in some weird way.

The expectation of CBR by default should be to be actually constant no 
matter what.

I just don't know what actual role the buffering period SEI plays this 
day, with the new option now existing.

If you like you can add the same logic for that flag as well.

I'd also prefer this to be called strict_cbr or cbr_padding or 
something, to make it obvious what it relates to.

> [0]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#recommended-nvenc-settings__table-nvenc-settings
> [1]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#rate-control
> 
>>> Allow users to opt-out of filler data in CBR mode for those usecases
>>> where it is unwanted.
>>>
>>> Signed-off-by: Cameron Gutman <aicommander@gmail.com>
>>> ---
>>> v2: Rebased to resolve conflicts against master
>>> ---
>>>    libavcodec/nvenc.c     | 2 +-
>>>    libavcodec/nvenc.h     | 1 +
>>>    libavcodec/nvenc_av1.c | 2 ++
>>>    3 files changed, 4 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
>>> index 0f5e772b3e..0f2336b0ec 100644
>>> --- a/libavcodec/nvenc.c
>>> +++ b/libavcodec/nvenc.c
>>> @@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
>>>
>>>        av1->idrPeriod = cc->gopLength;
>>>
>>> -    if (IS_CBR(cc->rcParams.rateControlMode)) {
>>> +    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
>>>            av1->enableBitstreamPadding = 1;
>>>        }
>>>
>>> diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
>>> index e035e123c6..3431457422 100644
>>> --- a/libavcodec/nvenc.h
>>> +++ b/libavcodec/nvenc.h
>>> @@ -309,6 +309,7 @@ typedef struct NvencContext
>>>        int unidir_b;
>>>        int split_encode_mode;
>>>        int mdm, cll;
>>> +    int filler_data;
>>>    } NvencContext;
>>>
>>>    int ff_nvenc_encode_init(AVCodecContext *avctx);
>>> diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
>>> index 01626113ab..c00817af7b 100644
>>> --- a/libavcodec/nvenc_av1.c
>>> +++ b/libavcodec/nvenc_av1.c
>>> @@ -156,6 +156,8 @@ static const AVOption options[] = {
>>>                                                                OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>        { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>        { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>> +    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
>>> +                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>    #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
>>>        { "tf_level",     "Specifies the strength of the temporal filtering",
>>>                                                                OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },
>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Cameron Gutman]

On Sat, Mar 29, 2025 at 4:26 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>
> On 29.03.2025 22:17, Cameron Gutman wrote:
> > On Sat, Mar 29, 2025 at 4:02 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
> >>
> >> On 29.03.2025 21:58, Cameron Gutman wrote:
> >>> Some scenarios (such as game streaming or videoconferencing) may use CBR
> >>> to strictly cap the maximum encoded bitrate, but they don't mind the
> >>> bitrate falling below the target if the encoder doesn't need the
> >>> additional headroom.
> >>
> >> But why aren't they using vbr with a maxrate then?
> >> I didn't add this conditionally since when in CBR mode, you actually
> >> really want a constant bitrate.
> >> If you're okay with it dropping lower, why not use another mode with a
> >> maxrate set?
> >>
> >
> > As I understand it, the rate control behavior of CBR without filler is
> > different from
> > just using VBR, particularly for low latency applications. Nvidia recommends use
> > of CBR [0] (with or without filler depending on application needs [1])
> > rather than
> > using VBR for low latency applications.
> >
> > Also FWIW, we don't set enableFillerDataInsertion for CBR on H.264 and HEVC,
> > so those will dip below the target bitrate in CBR mode while AV1 will not.
>
> We probably should really. In the past outputBufferingPeriodSEI
> controlled this in some weird way.
>
> The expectation of CBR by default should be to be actually constant no
> matter what.
>
> I just don't know what actual role the buffering period SEI plays this
> day, with the new option now existing.
>
> If you like you can add the same logic for that flag as well.
>
> I'd also prefer this to be called strict_cbr or cbr_padding or
> something, to make it obvious what it relates to.

To ensure I understand, you want the following:
1. Rename the filler_data option in this patch to strict_cbr (keeping
the default value of 1)
2. Change H.264 and HEVC to set enableFillerDataInsertion for CBR by
default and add strict_cbr option to those too
3. Don't set outputBufferingPeriodSEI = 1 unless filler data is also enabled

Is that correct?

>
> > [0]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#recommended-nvenc-settings__table-nvenc-settings
> > [1]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#rate-control
> >
> >>> Allow users to opt-out of filler data in CBR mode for those usecases
> >>> where it is unwanted.
> >>>
> >>> Signed-off-by: Cameron Gutman <aicommander@gmail.com>
> >>> ---
> >>> v2: Rebased to resolve conflicts against master
> >>> ---
> >>>    libavcodec/nvenc.c     | 2 +-
> >>>    libavcodec/nvenc.h     | 1 +
> >>>    libavcodec/nvenc_av1.c | 2 ++
> >>>    3 files changed, 4 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> >>> index 0f5e772b3e..0f2336b0ec 100644
> >>> --- a/libavcodec/nvenc.c
> >>> +++ b/libavcodec/nvenc.c
> >>> @@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
> >>>
> >>>        av1->idrPeriod = cc->gopLength;
> >>>
> >>> -    if (IS_CBR(cc->rcParams.rateControlMode)) {
> >>> +    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
> >>>            av1->enableBitstreamPadding = 1;
> >>>        }
> >>>
> >>> diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
> >>> index e035e123c6..3431457422 100644
> >>> --- a/libavcodec/nvenc.h
> >>> +++ b/libavcodec/nvenc.h
> >>> @@ -309,6 +309,7 @@ typedef struct NvencContext
> >>>        int unidir_b;
> >>>        int split_encode_mode;
> >>>        int mdm, cll;
> >>> +    int filler_data;
> >>>    } NvencContext;
> >>>
> >>>    int ff_nvenc_encode_init(AVCodecContext *avctx);
> >>> diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
> >>> index 01626113ab..c00817af7b 100644
> >>> --- a/libavcodec/nvenc_av1.c
> >>> +++ b/libavcodec/nvenc_av1.c
> >>> @@ -156,6 +156,8 @@ static const AVOption options[] = {
> >>>                                                                OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>        { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>        { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>> +    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
> >>> +                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>    #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
> >>>        { "tf_level",     "Specifies the strength of the temporal filtering",
> >>>                                                                OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },
> >>
> >> _______________________________________________
> >> ffmpeg-devel mailing list
> >> ffmpeg-devel@ffmpeg.org
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>
> >> To unsubscribe, visit link above, or email
> >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Timo Rothenpieler]

On 29.03.2025 22:37, Cameron Gutman wrote:
> On Sat, Mar 29, 2025 at 4:26 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>
>> On 29.03.2025 22:17, Cameron Gutman wrote:
>>> On Sat, Mar 29, 2025 at 4:02 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>>>
>>>> On 29.03.2025 21:58, Cameron Gutman wrote:
>>>>> Some scenarios (such as game streaming or videoconferencing) may use CBR
>>>>> to strictly cap the maximum encoded bitrate, but they don't mind the
>>>>> bitrate falling below the target if the encoder doesn't need the
>>>>> additional headroom.
>>>>
>>>> But why aren't they using vbr with a maxrate then?
>>>> I didn't add this conditionally since when in CBR mode, you actually
>>>> really want a constant bitrate.
>>>> If you're okay with it dropping lower, why not use another mode with a
>>>> maxrate set?
>>>>
>>>
>>> As I understand it, the rate control behavior of CBR without filler is
>>> different from
>>> just using VBR, particularly for low latency applications. Nvidia recommends use
>>> of CBR [0] (with or without filler depending on application needs [1])
>>> rather than
>>> using VBR for low latency applications.
>>>
>>> Also FWIW, we don't set enableFillerDataInsertion for CBR on H.264 and HEVC,
>>> so those will dip below the target bitrate in CBR mode while AV1 will not.
>>
>> We probably should really. In the past outputBufferingPeriodSEI
>> controlled this in some weird way.
>>
>> The expectation of CBR by default should be to be actually constant no
>> matter what.
>>
>> I just don't know what actual role the buffering period SEI plays this
>> day, with the new option now existing.
>>
>> If you like you can add the same logic for that flag as well.
>>
>> I'd also prefer this to be called strict_cbr or cbr_padding or
>> something, to make it obvious what it relates to.
> 
> To ensure I understand, you want the following:
> 1. Rename the filler_data option in this patch to strict_cbr (keeping
> the default value of 1)

Yes, definitely keep it enabled by default.
I'm not too picky about the exact name, as long as it contains cbr.

Might even slightly prefer cbr_padding, since it's more obvious to the 
common user what it does.

> 2. Change H.264 and HEVC to set enableFillerDataInsertion for CBR by
> default and add strict_cbr option to those too

Yes, or whatever the option ends up named.

> 3. Don't set outputBufferingPeriodSEI = 1 unless filler data is also enabled

That's what I'm not sure about.
To my knowledge, and given that people stream to Twitch and other 
services with strict CBR requirements, h264 and hevc already produce 
strict CBR.

I'd want the new option to toggle the strict CBR behaviour, and would 
need to investigate first what that implies.

Might just be that filler data insertion is enabled by default there, 
but not for AV1.
Or it's actually somehow related to the buffering period SEI. Since it's 
the only original user of the IS_CBR check.

So for now, if you want to add the option for the other codecs to, just 
make it toggle the new flag.

(And keep in mind that the oldest SDK still supported does not have that 
flag, so it'll need a feature guard)

> Is that correct?
> 
>>
>>> [0]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#recommended-nvenc-settings__table-nvenc-settings
>>> [1]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#rate-control
>>>
>>>>> Allow users to opt-out of filler data in CBR mode for those usecases
>>>>> where it is unwanted.
>>>>>
>>>>> Signed-off-by: Cameron Gutman <aicommander@gmail.com>
>>>>> ---
>>>>> v2: Rebased to resolve conflicts against master
>>>>> ---
>>>>>     libavcodec/nvenc.c     | 2 +-
>>>>>     libavcodec/nvenc.h     | 1 +
>>>>>     libavcodec/nvenc_av1.c | 2 ++
>>>>>     3 files changed, 4 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
>>>>> index 0f5e772b3e..0f2336b0ec 100644
>>>>> --- a/libavcodec/nvenc.c
>>>>> +++ b/libavcodec/nvenc.c
>>>>> @@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
>>>>>
>>>>>         av1->idrPeriod = cc->gopLength;
>>>>>
>>>>> -    if (IS_CBR(cc->rcParams.rateControlMode)) {
>>>>> +    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
>>>>>             av1->enableBitstreamPadding = 1;
>>>>>         }
>>>>>
>>>>> diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
>>>>> index e035e123c6..3431457422 100644
>>>>> --- a/libavcodec/nvenc.h
>>>>> +++ b/libavcodec/nvenc.h
>>>>> @@ -309,6 +309,7 @@ typedef struct NvencContext
>>>>>         int unidir_b;
>>>>>         int split_encode_mode;
>>>>>         int mdm, cll;
>>>>> +    int filler_data;
>>>>>     } NvencContext;
>>>>>
>>>>>     int ff_nvenc_encode_init(AVCodecContext *avctx);
>>>>> diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
>>>>> index 01626113ab..c00817af7b 100644
>>>>> --- a/libavcodec/nvenc_av1.c
>>>>> +++ b/libavcodec/nvenc_av1.c
>>>>> @@ -156,6 +156,8 @@ static const AVOption options[] = {
>>>>>                                                                 OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>>>         { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>>>         { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>>> +    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
>>>>> +                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
>>>>>     #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
>>>>>         { "tf_level",     "Specifies the strength of the temporal filtering",
>>>>>                                                                 OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },
>>>>
>>>> _______________________________________________
>>>> ffmpeg-devel mailing list
>>>> ffmpeg-devel@ffmpeg.org
>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>
>>>> To unsubscribe, visit link above, or email
>>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>>

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Cameron Gutman]

On Sat, Mar 29, 2025 at 4:42 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>
> On 29.03.2025 22:37, Cameron Gutman wrote:
> > On Sat, Mar 29, 2025 at 4:26 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
> >>
> >> On 29.03.2025 22:17, Cameron Gutman wrote:
> >>> On Sat, Mar 29, 2025 at 4:02 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
> >>>>
> >>>> On 29.03.2025 21:58, Cameron Gutman wrote:
> >>>>> Some scenarios (such as game streaming or videoconferencing) may use CBR
> >>>>> to strictly cap the maximum encoded bitrate, but they don't mind the
> >>>>> bitrate falling below the target if the encoder doesn't need the
> >>>>> additional headroom.
> >>>>
> >>>> But why aren't they using vbr with a maxrate then?
> >>>> I didn't add this conditionally since when in CBR mode, you actually
> >>>> really want a constant bitrate.
> >>>> If you're okay with it dropping lower, why not use another mode with a
> >>>> maxrate set?
> >>>>
> >>>
> >>> As I understand it, the rate control behavior of CBR without filler is
> >>> different from
> >>> just using VBR, particularly for low latency applications. Nvidia recommends use
> >>> of CBR [0] (with or without filler depending on application needs [1])
> >>> rather than
> >>> using VBR for low latency applications.
> >>>
> >>> Also FWIW, we don't set enableFillerDataInsertion for CBR on H.264 and HEVC,
> >>> so those will dip below the target bitrate in CBR mode while AV1 will not.
> >>
> >> We probably should really. In the past outputBufferingPeriodSEI
> >> controlled this in some weird way.
> >>
> >> The expectation of CBR by default should be to be actually constant no
> >> matter what.
> >>
> >> I just don't know what actual role the buffering period SEI plays this
> >> day, with the new option now existing.
> >>
> >> If you like you can add the same logic for that flag as well.
> >>
> >> I'd also prefer this to be called strict_cbr or cbr_padding or
> >> something, to make it obvious what it relates to.
> >
> > To ensure I understand, you want the following:
> > 1. Rename the filler_data option in this patch to strict_cbr (keeping
> > the default value of 1)
>
> Yes, definitely keep it enabled by default.
> I'm not too picky about the exact name, as long as it contains cbr.
>
> Might even slightly prefer cbr_padding, since it's more obvious to the
> common user what it does.
>

Sounds good to me.

> > 2. Change H.264 and HEVC to set enableFillerDataInsertion for CBR by
> > default and add strict_cbr option to those too
>
> Yes, or whatever the option ends up named.
>
> > 3. Don't set outputBufferingPeriodSEI = 1 unless filler data is also enabled
>
> That's what I'm not sure about.
> To my knowledge, and given that people stream to Twitch and other
> services with strict CBR requirements, h264 and hevc already produce
> strict CBR.
>
> I'd want the new option to toggle the strict CBR behaviour, and would
> need to investigate first what that implies.
>
> Might just be that filler data insertion is enabled by default there,
> but not for AV1.
> Or it's actually somehow related to the buffering period SEI. Since it's
> the only original user of the IS_CBR check.
>

When looking through the older headers, I saw that
outputBufferingPeriodSEI used to have the following comment:

"When set for following rateControlMode : NV_ENC_PARAMS_RC_CBR,
NV_ENC_PARAMS_RC_CBR_LOWDELAY_HQ, NV_ENC_PARAMS_RC_CBR_HQ, filler data
is inserted if needed to achieve HRD bitrate"

However, this was removed starting in SDK 12.0. Since it's hard to say
what each driver version between SDK 9.1 and SDK 12.0 will do with a
mismatch between outputBufferingPeriodSEI and
enableFillerDataInsertion, I think it's probably safest to tie the
options together for now.

> So for now, if you want to add the option for the other codecs to, just
> make it toggle the new flag.
>
> (And keep in mind that the oldest SDK still supported does not have that
> flag, so it'll need a feature guard)

Thanks for the heads up. Looks like it was added in SDK 9.1.

>
> > Is that correct?
> >
> >>
> >>> [0]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#recommended-nvenc-settings__table-nvenc-settings
> >>> [1]: https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvenc-video-encoder-api-prog-guide/index.html#rate-control
> >>>
> >>>>> Allow users to opt-out of filler data in CBR mode for those usecases
> >>>>> where it is unwanted.
> >>>>>
> >>>>> Signed-off-by: Cameron Gutman <aicommander@gmail.com>
> >>>>> ---
> >>>>> v2: Rebased to resolve conflicts against master
> >>>>> ---
> >>>>>     libavcodec/nvenc.c     | 2 +-
> >>>>>     libavcodec/nvenc.h     | 1 +
> >>>>>     libavcodec/nvenc_av1.c | 2 ++
> >>>>>     3 files changed, 4 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> >>>>> index 0f5e772b3e..0f2336b0ec 100644
> >>>>> --- a/libavcodec/nvenc.c
> >>>>> +++ b/libavcodec/nvenc.c
> >>>>> @@ -1624,7 +1624,7 @@ static av_cold int nvenc_setup_av1_config(AVCodecContext *avctx)
> >>>>>
> >>>>>         av1->idrPeriod = cc->gopLength;
> >>>>>
> >>>>> -    if (IS_CBR(cc->rcParams.rateControlMode)) {
> >>>>> +    if (ctx->filler_data && IS_CBR(cc->rcParams.rateControlMode)) {
> >>>>>             av1->enableBitstreamPadding = 1;
> >>>>>         }
> >>>>>
> >>>>> diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
> >>>>> index e035e123c6..3431457422 100644
> >>>>> --- a/libavcodec/nvenc.h
> >>>>> +++ b/libavcodec/nvenc.h
> >>>>> @@ -309,6 +309,7 @@ typedef struct NvencContext
> >>>>>         int unidir_b;
> >>>>>         int split_encode_mode;
> >>>>>         int mdm, cll;
> >>>>> +    int filler_data;
> >>>>>     } NvencContext;
> >>>>>
> >>>>>     int ff_nvenc_encode_init(AVCodecContext *avctx);
> >>>>> diff --git a/libavcodec/nvenc_av1.c b/libavcodec/nvenc_av1.c
> >>>>> index 01626113ab..c00817af7b 100644
> >>>>> --- a/libavcodec/nvenc_av1.c
> >>>>> +++ b/libavcodec/nvenc_av1.c
> >>>>> @@ -156,6 +156,8 @@ static const AVOption options[] = {
> >>>>>                                                                 OFFSET(extra_sei),    AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>>>         { "a53cc",        "Use A53 Closed Captions (if available)", OFFSET(a53_cc),   AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>>>         { "s12m_tc",      "Use timecode (if available)",        OFFSET(s12m_tc),      AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>>> +    { "filler_data",  "Use filler data to ensure CBR bitrate is strictly adhered to",
> >>>>> +                                                            OFFSET(filler_data),  AV_OPT_TYPE_BOOL,  { .i64 = 1 }, 0, 1, VE },
> >>>>>     #ifdef NVENC_HAVE_H264_AND_AV1_TEMPORAL_FILTER
> >>>>>         { "tf_level",     "Specifies the strength of the temporal filtering",
> >>>>>                                                                 OFFSET(tf_level),     AV_OPT_TYPE_INT,   { .i64 = -1 }, -1, INT_MAX, VE, .unit = "tf_level" },
> >>>>
> >>>> _______________________________________________
> >>>> ffmpeg-devel mailing list
> >>>> ffmpeg-devel@ffmpeg.org
> >>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>>>
> >>>> To unsubscribe, visit link above, or email
> >>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >>
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2] avcodec/nvenc: add option to skip padding OBUs

[Timo Rothenpieler]

On 29.03.2025 23:28, Cameron Gutman wrote:
> On Sat, Mar 29, 2025 at 4:42 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>
>> On 29.03.2025 22:37, Cameron Gutman wrote:
>>> On Sat, Mar 29, 2025 at 4:26 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>>>
>>>> On 29.03.2025 22:17, Cameron Gutman wrote:
>>>>> On Sat, Mar 29, 2025 at 4:02 PM Timo Rothenpieler <timo@rothenpieler.org> wrote:
>>>>>>
>>>>>> On 29.03.2025 21:58, Cameron Gutman wrote:
>>>>>>> Some scenarios (such as game streaming or videoconferencing) may use CBR
>>>>>>> to strictly cap the maximum encoded bitrate, but they don't mind the
>>>>>>> bitrate falling below the target if the encoder doesn't need the
>>>>>>> additional headroom.
>>>>>>
>>>>>> But why aren't they using vbr with a maxrate then?
>>>>>> I didn't add this conditionally since when in CBR mode, you actually
>>>>>> really want a constant bitrate.
>>>>>> If you're okay with it dropping lower, why not use another mode with a
>>>>>> maxrate set?
>>>>>>
>>>>>
>>>>> As I understand it, the rate control behavior of CBR without filler is
>>>>> different from
>>>>> just using VBR, particularly for low latency applications. Nvidia recommends use
>>>>> of CBR [0] (with or without filler depending on application needs [1])
>>>>> rather than
>>>>> using VBR for low latency applications.
>>>>>
>>>>> Also FWIW, we don't set enableFillerDataInsertion for CBR on H.264 and HEVC,
>>>>> so those will dip below the target bitrate in CBR mode while AV1 will not.
>>>>
>>>> We probably should really. In the past outputBufferingPeriodSEI
>>>> controlled this in some weird way.
>>>>
>>>> The expectation of CBR by default should be to be actually constant no
>>>> matter what.
>>>>
>>>> I just don't know what actual role the buffering period SEI plays this
>>>> day, with the new option now existing.
>>>>
>>>> If you like you can add the same logic for that flag as well.
>>>>
>>>> I'd also prefer this to be called strict_cbr or cbr_padding or
>>>> something, to make it obvious what it relates to.
>>>
>>> To ensure I understand, you want the following:
>>> 1. Rename the filler_data option in this patch to strict_cbr (keeping
>>> the default value of 1)
>>
>> Yes, definitely keep it enabled by default.
>> I'm not too picky about the exact name, as long as it contains cbr.
>>
>> Might even slightly prefer cbr_padding, since it's more obvious to the
>> common user what it does.
>>
> 
> Sounds good to me.
> 
>>> 2. Change H.264 and HEVC to set enableFillerDataInsertion for CBR by
>>> default and add strict_cbr option to those too
>>
>> Yes, or whatever the option ends up named.
>>
>>> 3. Don't set outputBufferingPeriodSEI = 1 unless filler data is also enabled
>>
>> That's what I'm not sure about.
>> To my knowledge, and given that people stream to Twitch and other
>> services with strict CBR requirements, h264 and hevc already produce
>> strict CBR.
>>
>> I'd want the new option to toggle the strict CBR behaviour, and would
>> need to investigate first what that implies.
>>
>> Might just be that filler data insertion is enabled by default there,
>> but not for AV1.
>> Or it's actually somehow related to the buffering period SEI. Since it's
>> the only original user of the IS_CBR check.
>>
> 
> When looking through the older headers, I saw that
> outputBufferingPeriodSEI used to have the following comment:
> 
> "When set for following rateControlMode : NV_ENC_PARAMS_RC_CBR,
> NV_ENC_PARAMS_RC_CBR_LOWDELAY_HQ, NV_ENC_PARAMS_RC_CBR_HQ, filler data
> is inserted if needed to achieve HRD bitrate"
> 
> However, this was removed starting in SDK 12.0. Since it's hard to say
> what each driver version between SDK 9.1 and SDK 12.0 will do with a
> mismatch between outputBufferingPeriodSEI and
> enableFillerDataInsertion, I think it's probably safest to tie the
> options together for now.

Sounds good to me

>> So for now, if you want to add the option for the other codecs to, just
>> make it toggle the new flag.
>>
>> (And keep in mind that the oldest SDK still supported does not have that
>> flag, so it'll need a feature guard)
> 
> Thanks for the heads up. Looks like it was added in SDK 9.1.
> 
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".