Re: [FFmpeg-devel] [FFmpeg-cvslog] avfilter/vf_tpad: if there is no frame to clone return early

[Paul B Mahol <onemda@gmail.com>]

On 2/28/22, Thierry Foucu <tfoucu@gmail.com> wrote:
> On Sun, Feb 27, 2022 at 3:54 AM Paul B Mahol <onemda@gmail.com> wrote:
>
>> On Fri, Feb 25, 2022 at 10:08 PM Thierry Foucu <tfoucu@gmail.com> wrote:
>>
>> > On Thu, Feb 24, 2022 at 11:50 PM Paul B Mahol <onemda@gmail.com> wrote:
>> >
>> > > On Thu, Feb 24, 2022 at 11:21 PM Thierry Foucu <tfoucu@gmail.com>
>> wrote:
>> > >
>> > > > On Thu, Feb 24, 2022 at 2:19 PM Thierry Foucu <tfoucu@gmail.com>
>> > wrote:
>> > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Feb 24, 2022 at 1:50 PM Paul B Mahol <onemda@gmail.com>
>> > wrote:
>> > > > >
>> > > > >> On Thu, Feb 24, 2022 at 10:36 PM Thierry Foucu
>> > > > >> <tfoucu@gmail.com>
>> > > > wrote:
>> > > > >>
>> > > > >> > On Thu, Feb 24, 2022 at 1:28 PM Paul B Mahol
>> > > > >> > <onemda@gmail.com>
>> > > > wrote:
>> > > > >> >
>> > > > >> > > On Thu, Feb 24, 2022 at 10:12 PM Thierry Foucu <
>> > tfoucu@gmail.com>
>> > > > >> wrote:
>> > > > >> > >
>> > > > >> > > > On Thu, Feb 24, 2022 at 12:30 PM Paul B Mahol <
>> > git@videolan.org
>> > > >
>> > > > >> > wrote:
>> > > > >> > > >
>> > > > >> > > > > ffmpeg | branch: master | Paul B Mahol
>> > > > >> > > > > <onemda@gmail.com>
>> |
>> > > Thu
>> > > > >> Feb
>> > > > >> > 24
>> > > > >> > > > > 20:32:41 2022 +0100|
>> > > [3715f2f8643695940582ce040b7a052cccfb9db2]
>> > > > |
>> > > > >> > > > > committer: Paul B Mahol
>> > > > >> > > > >
>> > > > >> > > > > avfilter/vf_tpad: if there is no frame to clone return
>> early
>> > > > >> > > > >
>> > > > >> > > > > >
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=3715f2f8643695940582ce040b7a052cccfb9db2
>> > > > >> > > > > ---
>> > > > >> > > > >
>> > > > >> > > > >  libavfilter/vf_tpad.c | 5 +++++
>> > > > >> > > > >  1 file changed, 5 insertions(+)
>> > > > >> > > > >
>> > > > >> > > > > diff --git a/libavfilter/vf_tpad.c
>> > > > >> > > > > b/libavfilter/vf_tpad.c
>> > > > >> > > > > index e5acece1e4..120dbcb4d3 100644
>> > > > >> > > > > --- a/libavfilter/vf_tpad.c
>> > > > >> > > > > +++ b/libavfilter/vf_tpad.c
>> > > > >> > > > > @@ -148,6 +148,11 @@ static int activate(AVFilterContext
>> > *ctx)
>> > > > >> > > > >                                frame->data,
>> frame->linesize,
>> > > > >> > > > >                                0, 0, frame->width,
>> > > > frame->height);
>> > > > >> > > > >          } else if (s->stop_mode == 1) {
>> > > > >> > > > > +            if (!s->cache_stop) {
>> > > > >> > > > > +                s->pad_stop = 0;
>> > > > >> > > > > +                ff_outlink_set_status(outlink,
>> AVERROR_EOF,
>> > > > >> s->pts);
>> > > > >> > > > > +                return 0;
>> > > > >> > > > > +            }
>> > > > >> > > > >              frame = av_frame_clone(s->cache_stop);
>> > > > >> > > > >              if (!frame)
>> > > > >> > > > >                  return AVERROR(ENOMEM);
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > The problem with this solution is that the tpad will then
>> not
>> > do
>> > > > >> what
>> > > > >> > we
>> > > > >> > > > are expecting, which is padding video track, and the
>> > > > >> > > > output
>> > file
>> > > > >> will
>> > > > >> > not
>> > > > >> > > > have the desired duration.
>> > > > >> > > > Will it not be better to just output black frame (aka
>> > stop_mode
>> > > ==
>> > > > >> 0) ,
>> > > > >> > > > something like that?
>> > > > >> > > >
>> > > > >> > >
>> > > > >> > > I doubt that, clone is clone, there is no point in padding
>> > stream
>> > > > that
>> > > > >> > have
>> > > > >> > > no frames at all.
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> > The sample I forwarded to JB has video frames. The problem was
>> > that
>> > > > >> there
>> > > > >> > was a re-init of the filter chain after a resolution change
>> > > > >> > and
>> > > after
>> > > > >> the
>> > > > >> > re-init, there were no frames but got frames before the
>> > > > >> > re-init.
>> > > > >> > I will understand for a media file which has NEVER received a
>> > video
>> > > > >> frame
>> > > > >> > and in this case, this is the correct solution.
>> > > > >> >
>> > > > >>
>> > > > >> Hmm, so you encode output with resolution change, why not use
>> scale
>> > > > filter
>> > > > >> as first filter and then no resolution ever change for rest of
>> > graph?
>> > > > >> Adding support for resolution changes to all filters is very
>> > > > >> time
>> > > > >> consuming
>> > > > >> task and I see no real benefit in doing that now.
>> > > > >>
>> > > > >
>> > > > > We do have the scale filter in front of the tpad filter. Here is
>> the
>> > > > > filter chain we are using
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> idet=1.04:1.5,yadif=0:-1:1,scale=+528:+864:flags=bicubic,setsar=1,fps=fps=16.601,tpad=stop_mode=clone:stop_duration=2808ms
>> > > > >
>> > > >
>> > >
>> > > Is this just increasing size instead of setting constant one?
>> > >
>> >
>> > Nope. It will just scale the video to 528x864
>> >
>>
>> Then why filter is still reinited then?
>> Looks like serious bug to me.
>>
>
> Here is the verbose log when running ffmpeg with the filter chain. The
> input resolution went from 1280x692 then changed to 1280x720 followed by
> 1280x718
> And from the log, it is clear that the filter chain is being re-initialize.
>

All filters should not be reinitialized if only dimension change for
first filter.
This is current libavfilter limitation AFAIK.

> Input #0, matroska,webm, from '/tmp/crash_tpad':
>   Metadata:
>     ENCODER         : Lavf58.76.100
>   Duration: 00:08:07.44, start: 0.000000, bitrate: 78 kb/s
>   Stream #0:0(eng): Video: vp9 (Profile 0), 1 reference frame, yuv420p(tv),
> 1280x692, SAR 1:1 DAR 320:173, 59.94 fps, 59.94 tbr, 1k tbn (default)
>     Metadata:
>       ALPHA_MODE      : 1
>       DURATION        : 00:08:06.780000000
>   Stream #0:1(eng): Audio: opus, 48000 Hz, mono, fltp (default)
>     Metadata:
>       DURATION        : 00:08:07.440000000
> Stream mapping:
>   Stream #0:0 -> #0:0 (vp9 (native) -> wrapped_avframe (native))
>   Stream #0:1 -> #0:1 (opus (native) -> pcm_s16le (native))
> Press [q] to stop, [?] for help
> [graph_1_in_0_1 @ 0x55df4ff5f180] tb:1/48000 samplefmt:fltp
> samplerate:48000 chlayout:0x4
> [format_out_0_1 @ 0x55df4ff5fd40] auto-inserting filter 'auto_aresample_0'
> between the filter 'Parsed_anull_0' and the filter 'format_out_0_1'
> [auto_aresample_0 @ 0x55df4ff61340] ch:1 chl:mono fmt:fltp r:48000Hz ->
> ch:1 chl:mono fmt:s16 r:48000Hz
> [Parsed_scale_2 @ 0x55df4ffa7040] w:+528 h:+864 flags:'bicubic' interl:0
> [graph 0 input from stream 0:0 @ 0x55df4ffab580] w:1280 h:692
> pixfmt:yuv420p tb:1/1000 fr:19001/317 sar:1/1
> [Parsed_scale_2 @ 0x55df4ffa7040] w:1280 h:692 fmt:yuv420p sar:1/1 -> w:528
> h:864 fmt:yuv420p sar:5760/1903 flags:0x4
> [Parsed_setsar_3 @ 0x55df4ffa7e80] w:528 h:864 sar:5760/1903 dar:320/173 ->
> sar:1/1 dar:11/18
> [Parsed_fps_4 @ 0x55df4ffa8880] fps=16601/1000
> [Parsed_fps_4 @ 0x55df4ffa8880] Set first pts to 0
> Output #0, null, to 'pipe:':
>   Metadata:
>     encoder         : Lavf59.17.102
>   Stream #0:0(eng): Video: wrapped_avframe, 1 reference frame, yuv420p(tv,
> progressive), 528x864 (0x0) [SAR 1:1 DAR 11:18], q=2-31, 200 kb/s, 16.60
> fps, 16.60 tbn (default)
>     Metadata:
>       ALPHA_MODE      : 1
>       DURATION        : 00:08:06.780000000
>       encoder         : Lavc59.21.100 wrapped_avframe
>   Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
> (default)
>     Metadata:
>       DURATION        : 00:08:07.440000000
>       encoder         : Lavc59.21.100 pcm_s16le
> [Parsed_idet_0 @ 0x55df4ffa0700] Repeated Fields: Neither:    88 Top:     0
> Bottom:     0
> [Parsed_idet_0 @ 0x55df4ffa0700] Single frame detection: TFF:     0 BFF:
>   0 Progressive:     1 Undetermined:    87
> [Parsed_idet_0 @ 0x55df4ffa0700] Multi frame detection: TFF:     0 BFF:
> 0 Progressive:     1 Undetermined:    87
> [Parsed_fps_4 @ 0x55df4ffa8880] 87 frames in, 62 frames out; 27 frames
> dropped, 2 frames duplicated.
> [Parsed_scale_2 @ 0x55df4ffaa1c0] w:+528 h:+864 flags:'bicubic' interl:0
> [graph 0 input from stream 0:0 @ 0x55df4ff94140] w:1280 h:720
> pixfmt:yuv420p tb:1/1000 fr:19001/317 sar:1/1
> [scaler_out_0_0 @ 0x55df4ff9de00] w:528 h:864 flags:'' interl:0
> [Parsed_scale_2 @ 0x55df4ffaa1c0] w:1280 h:720 fmt:yuv420p sar:1/1 -> w:528
> h:864 fmt:yuv420p sar:32/11 flags:0x4
> [Parsed_setsar_3 @ 0x55df4ffaea40] w:528 h:864 sar:32/11 dar:16/9 ->
> sar:1/1 dar:11/18
> [Parsed_fps_4 @ 0x55df4ff91fc0] fps=16601/1000
> [scaler_out_0_0 @ 0x55df4ff9de00] w:528 h:864 fmt:yuv420p sar:1/1 -> w:528
> h:864 fmt:yuv420p sar:1/1 flags:0x0
> [Parsed_fps_4 @ 0x55df4ff91fc0] Set first pts to 64
> [Parsed_idet_0 @ 0x55df4ff9bd00] Repeated Fields: Neither:   998 Top:     0
> Bottom:     2
> [Parsed_idet_0 @ 0x55df4ff9bd00] Single frame detection: TFF:     0 BFF:
>   0 Progressive:    11 Undetermined:   989
> [Parsed_idet_0 @ 0x55df4ff9bd00] Multi frame detection: TFF:     0 BFF:
> 0 Progressive:  1000 Undetermined:     0
> [Parsed_fps_4 @ 0x55df4ff91fc0] 999 frames in, 8004 frames out; 39 frames
> dropped, 7044 frames duplicated.
> [Parsed_scale_2 @ 0x55df50130e80] w:+528 h:+864 flags:'bicubic' interl:0
> [graph 0 input from stream 0:0 @ 0x55df5029dd00] w:1280 h:718
> pixfmt:yuv420p tb:1/1000 fr:19001/317 sar:1/1
> [scaler_out_0_0 @ 0x55df4ff93640] w:528 h:864 flags:'' interl:0
> [Parsed_scale_2 @ 0x55df50130e80] w:1280 h:718 fmt:yuv420p sar:1/1 -> w:528
> h:864 fmt:yuv420p sar:11520/3949 flags:0x4
> [Parsed_setsar_3 @ 0x55df50413380] w:528 h:864 sar:11520/3949 dar:640/359
> -> sar:1/1 dar:11/18
> [Parsed_fps_4 @ 0x55df4ff7af80] fps=16601/1000
> [scaler_out_0_0 @ 0x55df4ff93640] w:528 h:864 fmt:yuv420p sar:1/1 -> w:528
> h:864 fmt:yuv420p sar:1/1 flags:0x0
> [Parsed_fps_4 @ 0x55df4ff7af80] Set first pts to 8081
> No more output streams to write to, finishing.
> frame= 8066 fps=1787 q=-0.0 Lsize=N/A time=00:08:07.44 bitrate=N/A speed=
> 108x
> video:3529kB audio:45698kB subtitle:0kB other streams:0kB global
> headers:0kB muxing overhead: unknown
> Input file #0 (/tmp/crash_tpad):
>   Input stream #0:0 (video): 1091 packets read (1667477 bytes); 1091 frames
> decoded;
>   Input stream #0:1 (audio): 8124 packets read (3026650 bytes); 8124 frames
> decoded (23397120 samples);
>   Total: 9215 packets (4694127 bytes) demuxed
> Output file #0 (pipe:):
>   Output stream #0:0 (video): 8066 frames encoded; 8066 packets muxed
> (3613568 bytes);
>   Output stream #0:1 (audio): 8124 frames encoded (23397120 samples); 8124
> packets muxed (46794240 bytes);
>   Total: 16190 packets (50407808 bytes) muxed
> [Parsed_idet_0 @ 0x55df4ff7f8c0] Repeated Fields: Neither:     1 Top:     0
> Bottom:     0
> [Parsed_idet_0 @ 0x55df4ff7f8c0] Single frame detection: TFF:     0 BFF:
>   0 Progressive:     0 Undetermined:     1
> [Parsed_idet_0 @ 0x55df4ff7f8c0] Multi frame detection: TFF:     0 BFF:
> 0 Progressive:     0 Undetermined:     1
> [Parsed_fps_4 @ 0x55df4ff7af80] 1 frames in, 0 frames out; 1 frames
> dropped, 0 frames duplicated.
> [AVIOContext @ 0x55df4fe98180] Statistics: 4759868 bytes read, 0 seeks
>
>
>
>>
>>
>> >
>> >
>> > >
>> > >
>> > > > > And it does crash.
>> > > > >
>> > > > > Even just
>> > > > > fps=fps=16.601,tpad=stop_mode=clone:stop_duration=2808ms
>> > > > > reproduce the problem.
>> > > > >
>> > > >
>> > > > Maybe an option to the tpad filter to decide what should be the
>> output
>> > > when
>> > > > the stop_mode == 1 and cache_stop is NULL?
>> > > > So, in some cases, people may want a color frame, and some other
>> > > > will
>> > > just
>> > > > want to exit right away?
>> > > >
>> > >
>> > > That is too much for a special option.
>> > >
>> >
>> > I understand.
>> > I'm just worried that due to some re-init of the filter chain and the
>> fact
>> > there are filters which may or may not output a frame after the
>> > re-init,
>> > the tpad filter will then have no cache frame to clone. And in this
>> > case,
>> > the file does not have the length expected.
>> >
>> > I guess if it is documented or av_log, people will be aware of the
>> > potential issue.
>> >
>> > Anyway, thanks a lot for the quick fix.
>> >
>> >
>> > > >
>> > > >
>> > > > >
>> > > > >
>> > > > >>
>> > > > >>
>> > > > >> >
>> > > > >> >
>> > > > >> > > >
>> > > > >> > > > iff --git a/libavfilter/vf_tpad.c b/libavfilter/vf_tpad.c
>> > > > >> > > > index e5acece1e4..5e4062e9f2 100644
>> > > > >> > > > --- a/libavfilter/vf_tpad.c
>> > > > >> > > > +++ b/libavfilter/vf_tpad.c
>> > > > >> > > > @@ -140,7 +140,7 @@ static int activate(AVFilterContext
>> *ctx)
>> > > > >> > > >              ff_outlink_set_status(outlink, AVERROR_EOF,
>> > > s->pts);
>> > > > >> > > >              return 0;
>> > > > >> > > >          }
>> > > > >> > > > -        if (s->stop_mode == 0) {
>> > > > >> > > > +        if (s->stop_mode == 0 || !s->cache_stop) {
>> > > > >> > > >              frame = ff_get_video_buffer(outlink,
>> outlink->w,
>> > > > >> > > outlink->h);
>> > > > >> > > >              if (!frame)
>> > > > >> > > >                  return AVERROR(ENOMEM);
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > > _______________________________________________
>> > > > >> > > > > ffmpeg-cvslog mailing list
>> > > > >> > > > > ffmpeg-cvslog@ffmpeg.org
>> > > > >> > > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-cvslog
>> > > > >> > > > >
>> > > > >> > > > > To unsubscribe, visit link above, or email
>> > > > >> > > > > ffmpeg-cvslog-request@ffmpeg.org with subject
>> > "unsubscribe".
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > --
>> > > > >> > > >
>> > > > >> > > > Thierry Foucu
>> > > > >> > > > _______________________________________________
>> > > > >> > > > ffmpeg-devel mailing list
>> > > > >> > > > ffmpeg-devel@ffmpeg.org
>> > > > >> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > > > >> > > >
>> > > > >> > > > To unsubscribe, visit link above, or email
>> > > > >> > > > ffmpeg-devel-request@ffmpeg.org with subject
>> > > > >> > > > "unsubscribe".
>> > > > >> > > >
>> > > > >> > > _______________________________________________
>> > > > >> > > ffmpeg-devel mailing list
>> > > > >> > > ffmpeg-devel@ffmpeg.org
>> > > > >> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > > > >> > >
>> > > > >> > > To unsubscribe, visit link above, or email
>> > > > >> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> > > > >> > >
>> > > > >> >
>> > > > >> >
>> > > > >> > --
>> > > > >> >
>> > > > >> > Thierry Foucu
>> > > > >> > _______________________________________________
>> > > > >> > ffmpeg-devel mailing list
>> > > > >> > ffmpeg-devel@ffmpeg.org
>> > > > >> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > > > >> >
>> > > > >> > To unsubscribe, visit link above, or email
>> > > > >> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> > > > >> >
>> > > > >> _______________________________________________
>> > > > >> ffmpeg-devel mailing list
>> > > > >> ffmpeg-devel@ffmpeg.org
>> > > > >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > > > >>
>> > > > >> To unsubscribe, visit link above, or email
>> > > > >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> > > > >>
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Thierry Foucu
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Thierry Foucu
>> > > > _______________________________________________
>> > > > ffmpeg-devel mailing list
>> > > > ffmpeg-devel@ffmpeg.org
>> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > > >
>> > > > To unsubscribe, visit link above, or email
>> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> > > >
>> > > _______________________________________________
>> > > ffmpeg-devel mailing list
>> > > ffmpeg-devel@ffmpeg.org
>> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > >
>> > > To unsubscribe, visit link above, or email
>> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> > >
>> >
>> >
>> > --
>> >
>> > Thierry Foucu
>> > _______________________________________________
>> > ffmpeg-devel mailing list
>> > ffmpeg-devel@ffmpeg.org
>> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> >
>> > To unsubscribe, visit link above, or email
>> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> >
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>>
>
>
> --
>
> Thierry Foucu
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".