Re: [FFmpeg-devel] [FFmpeg-cvslog] avfilter/vf_tpad: if there is no frame to clone return early

[Thierry Foucu <tfoucu@gmail.com>]

On Sun, Feb 27, 2022 at 3:54 AM Paul B Mahol <onemda@gmail.com> wrote:

> On Fri, Feb 25, 2022 at 10:08 PM Thierry Foucu <tfoucu@gmail.com> wrote:
>
> > On Thu, Feb 24, 2022 at 11:50 PM Paul B Mahol <onemda@gmail.com> wrote:
> >
> > > On Thu, Feb 24, 2022 at 11:21 PM Thierry Foucu <tfoucu@gmail.com>
> wrote:
> > >
> > > > On Thu, Feb 24, 2022 at 2:19 PM Thierry Foucu <tfoucu@gmail.com>
> > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Thu, Feb 24, 2022 at 1:50 PM Paul B Mahol <onemda@gmail.com>
> > wrote:
> > > > >
> > > > >> On Thu, Feb 24, 2022 at 10:36 PM Thierry Foucu <tfoucu@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> > On Thu, Feb 24, 2022 at 1:28 PM Paul B Mahol <onemda@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > On Thu, Feb 24, 2022 at 10:12 PM Thierry Foucu <
> > tfoucu@gmail.com>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > On Thu, Feb 24, 2022 at 12:30 PM Paul B Mahol <
> > git@videolan.org
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > ffmpeg | branch: master | Paul B Mahol <onemda@gmail.com>
> |
> > > Thu
> > > > >> Feb
> > > > >> > 24
> > > > >> > > > > 20:32:41 2022 +0100|
> > > [3715f2f8643695940582ce040b7a052cccfb9db2]
> > > > |
> > > > >> > > > > committer: Paul B Mahol
> > > > >> > > > >
> > > > >> > > > > avfilter/vf_tpad: if there is no frame to clone return
> early
> > > > >> > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=3715f2f8643695940582ce040b7a052cccfb9db2
> > > > >> > > > > ---
> > > > >> > > > >
> > > > >> > > > >  libavfilter/vf_tpad.c | 5 +++++
> > > > >> > > > >  1 file changed, 5 insertions(+)
> > > > >> > > > >
> > > > >> > > > > diff --git a/libavfilter/vf_tpad.c b/libavfilter/vf_tpad.c
> > > > >> > > > > index e5acece1e4..120dbcb4d3 100644
> > > > >> > > > > --- a/libavfilter/vf_tpad.c
> > > > >> > > > > +++ b/libavfilter/vf_tpad.c
> > > > >> > > > > @@ -148,6 +148,11 @@ static int activate(AVFilterContext
> > *ctx)
> > > > >> > > > >                                frame->data,
> frame->linesize,
> > > > >> > > > >                                0, 0, frame->width,
> > > > frame->height);
> > > > >> > > > >          } else if (s->stop_mode == 1) {
> > > > >> > > > > +            if (!s->cache_stop) {
> > > > >> > > > > +                s->pad_stop = 0;
> > > > >> > > > > +                ff_outlink_set_status(outlink,
> AVERROR_EOF,
> > > > >> s->pts);
> > > > >> > > > > +                return 0;
> > > > >> > > > > +            }
> > > > >> > > > >              frame = av_frame_clone(s->cache_stop);
> > > > >> > > > >              if (!frame)
> > > > >> > > > >                  return AVERROR(ENOMEM);
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > The problem with this solution is that the tpad will then
> not
> > do
> > > > >> what
> > > > >> > we
> > > > >> > > > are expecting, which is padding video track, and the output
> > file
> > > > >> will
> > > > >> > not
> > > > >> > > > have the desired duration.
> > > > >> > > > Will it not be better to just output black frame (aka
> > stop_mode
> > > ==
> > > > >> 0) ,
> > > > >> > > > something like that?
> > > > >> > > >
> > > > >> > >
> > > > >> > > I doubt that, clone is clone, there is no point in padding
> > stream
> > > > that
> > > > >> > have
> > > > >> > > no frames at all.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > The sample I forwarded to JB has video frames. The problem was
> > that
> > > > >> there
> > > > >> > was a re-init of the filter chain after a resolution change and
> > > after
> > > > >> the
> > > > >> > re-init, there were no frames but got frames before the re-init.
> > > > >> > I will understand for a media file which has NEVER received a
> > video
> > > > >> frame
> > > > >> > and in this case, this is the correct solution.
> > > > >> >
> > > > >>
> > > > >> Hmm, so you encode output with resolution change, why not use
> scale
> > > > filter
> > > > >> as first filter and then no resolution ever change for rest of
> > graph?
> > > > >> Adding support for resolution changes to all filters is very time
> > > > >> consuming
> > > > >> task and I see no real benefit in doing that now.
> > > > >>
> > > > >
> > > > > We do have the scale filter in front of the tpad filter. Here is
> the
> > > > > filter chain we are using
> > > > >
> > > > >
> > > >
> > >
> >
> idet=1.04:1.5,yadif=0:-1:1,scale=+528:+864:flags=bicubic,setsar=1,fps=fps=16.601,tpad=stop_mode=clone:stop_duration=2808ms
> > > > >
> > > >
> > >
> > > Is this just increasing size instead of setting constant one?
> > >
> >
> > Nope. It will just scale the video to 528x864
> >
>
> Then why filter is still reinited then?
> Looks like serious bug to me.
>

Here is the verbose log when running ffmpeg with the filter chain. The
input resolution went from 1280x692 then changed to 1280x720 followed by
1280x718
And from the log, it is clear that the filter chain is being re-initialize.

Input #0, matroska,webm, from '/tmp/crash_tpad':
  Metadata:
    ENCODER         : Lavf58.76.100
  Duration: 00:08:07.44, start: 0.000000, bitrate: 78 kb/s
  Stream #0:0(eng): Video: vp9 (Profile 0), 1 reference frame, yuv420p(tv),
1280x692, SAR 1:1 DAR 320:173, 59.94 fps, 59.94 tbr, 1k tbn (default)
    Metadata:
      ALPHA_MODE      : 1
      DURATION        : 00:08:06.780000000
  Stream #0:1(eng): Audio: opus, 48000 Hz, mono, fltp (default)
    Metadata:
      DURATION        : 00:08:07.440000000
Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (opus (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
[graph_1_in_0_1 @ 0x55df4ff5f180] tb:1/48000 samplefmt:fltp
samplerate:48000 chlayout:0x4
[format_out_0_1 @ 0x55df4ff5fd40] auto-inserting filter 'auto_aresample_0'
between the filter 'Parsed_anull_0' and the filter 'format_out_0_1'
[auto_aresample_0 @ 0x55df4ff61340] ch:1 chl:mono fmt:fltp r:48000Hz ->
ch:1 chl:mono fmt:s16 r:48000Hz
[Parsed_scale_2 @ 0x55df4ffa7040] w:+528 h:+864 flags:'bicubic' interl:0
[graph 0 input from stream 0:0 @ 0x55df4ffab580] w:1280 h:692
pixfmt:yuv420p tb:1/1000 fr:19001/317 sar:1/1
[Parsed_scale_2 @ 0x55df4ffa7040] w:1280 h:692 fmt:yuv420p sar:1/1 -> w:528
h:864 fmt:yuv420p sar:5760/1903 flags:0x4
[Parsed_setsar_3 @ 0x55df4ffa7e80] w:528 h:864 sar:5760/1903 dar:320/173 ->
sar:1/1 dar:11/18
[Parsed_fps_4 @ 0x55df4ffa8880] fps=16601/1000
[Parsed_fps_4 @ 0x55df4ffa8880] Set first pts to 0
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf59.17.102
  Stream #0:0(eng): Video: wrapped_avframe, 1 reference frame, yuv420p(tv,
progressive), 528x864 (0x0) [SAR 1:1 DAR 11:18], q=2-31, 200 kb/s, 16.60
fps, 16.60 tbn (default)
    Metadata:
      ALPHA_MODE      : 1
      DURATION        : 00:08:06.780000000
      encoder         : Lavc59.21.100 wrapped_avframe
  Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
(default)
    Metadata:
      DURATION        : 00:08:07.440000000
      encoder         : Lavc59.21.100 pcm_s16le
[Parsed_idet_0 @ 0x55df4ffa0700] Repeated Fields: Neither:    88 Top:     0
Bottom:     0
[Parsed_idet_0 @ 0x55df4ffa0700] Single frame detection: TFF:     0 BFF:
  0 Progressive:     1 Undetermined:    87
[Parsed_idet_0 @ 0x55df4ffa0700] Multi frame detection: TFF:     0 BFF:
0 Progressive:     1 Undetermined:    87
[Parsed_fps_4 @ 0x55df4ffa8880] 87 frames in, 62 frames out; 27 frames
dropped, 2 frames duplicated.
[Parsed_scale_2 @ 0x55df4ffaa1c0] w:+528 h:+864 flags:'bicubic' interl:0
[graph 0 input from stream 0:0 @ 0x55df4ff94140] w:1280 h:720
pixfmt:yuv420p tb:1/1000 fr:19001/317 sar:1/1
[scaler_out_0_0 @ 0x55df4ff9de00] w:528 h:864 flags:'' interl:0
[Parsed_scale_2 @ 0x55df4ffaa1c0] w:1280 h:720 fmt:yuv420p sar:1/1 -> w:528
h:864 fmt:yuv420p sar:32/11 flags:0x4
[Parsed_setsar_3 @ 0x55df4ffaea40] w:528 h:864 sar:32/11 dar:16/9 ->
sar:1/1 dar:11/18
[Parsed_fps_4 @ 0x55df4ff91fc0] fps=16601/1000
[scaler_out_0_0 @ 0x55df4ff9de00] w:528 h:864 fmt:yuv420p sar:1/1 -> w:528
h:864 fmt:yuv420p sar:1/1 flags:0x0
[Parsed_fps_4 @ 0x55df4ff91fc0] Set first pts to 64
[Parsed_idet_0 @ 0x55df4ff9bd00] Repeated Fields: Neither:   998 Top:     0
Bottom:     2
[Parsed_idet_0 @ 0x55df4ff9bd00] Single frame detection: TFF:     0 BFF:
  0 Progressive:    11 Undetermined:   989
[Parsed_idet_0 @ 0x55df4ff9bd00] Multi frame detection: TFF:     0 BFF:
0 Progressive:  1000 Undetermined:     0
[Parsed_fps_4 @ 0x55df4ff91fc0] 999 frames in, 8004 frames out; 39 frames
dropped, 7044 frames duplicated.
[Parsed_scale_2 @ 0x55df50130e80] w:+528 h:+864 flags:'bicubic' interl:0
[graph 0 input from stream 0:0 @ 0x55df5029dd00] w:1280 h:718
pixfmt:yuv420p tb:1/1000 fr:19001/317 sar:1/1
[scaler_out_0_0 @ 0x55df4ff93640] w:528 h:864 flags:'' interl:0
[Parsed_scale_2 @ 0x55df50130e80] w:1280 h:718 fmt:yuv420p sar:1/1 -> w:528
h:864 fmt:yuv420p sar:11520/3949 flags:0x4
[Parsed_setsar_3 @ 0x55df50413380] w:528 h:864 sar:11520/3949 dar:640/359
-> sar:1/1 dar:11/18
[Parsed_fps_4 @ 0x55df4ff7af80] fps=16601/1000
[scaler_out_0_0 @ 0x55df4ff93640] w:528 h:864 fmt:yuv420p sar:1/1 -> w:528
h:864 fmt:yuv420p sar:1/1 flags:0x0
[Parsed_fps_4 @ 0x55df4ff7af80] Set first pts to 8081
No more output streams to write to, finishing.
frame= 8066 fps=1787 q=-0.0 Lsize=N/A time=00:08:07.44 bitrate=N/A speed=
108x
video:3529kB audio:45698kB subtitle:0kB other streams:0kB global
headers:0kB muxing overhead: unknown
Input file #0 (/tmp/crash_tpad):
  Input stream #0:0 (video): 1091 packets read (1667477 bytes); 1091 frames
decoded;
  Input stream #0:1 (audio): 8124 packets read (3026650 bytes); 8124 frames
decoded (23397120 samples);
  Total: 9215 packets (4694127 bytes) demuxed
Output file #0 (pipe:):
  Output stream #0:0 (video): 8066 frames encoded; 8066 packets muxed
(3613568 bytes);
  Output stream #0:1 (audio): 8124 frames encoded (23397120 samples); 8124
packets muxed (46794240 bytes);
  Total: 16190 packets (50407808 bytes) muxed
[Parsed_idet_0 @ 0x55df4ff7f8c0] Repeated Fields: Neither:     1 Top:     0
Bottom:     0
[Parsed_idet_0 @ 0x55df4ff7f8c0] Single frame detection: TFF:     0 BFF:
  0 Progressive:     0 Undetermined:     1
[Parsed_idet_0 @ 0x55df4ff7f8c0] Multi frame detection: TFF:     0 BFF:
0 Progressive:     0 Undetermined:     1
[Parsed_fps_4 @ 0x55df4ff7af80] 1 frames in, 0 frames out; 1 frames
dropped, 0 frames duplicated.
[AVIOContext @ 0x55df4fe98180] Statistics: 4759868 bytes read, 0 seeks



>
>
> >
> >
> > >
> > >
> > > > > And it does crash.
> > > > >
> > > > > Even just fps=fps=16.601,tpad=stop_mode=clone:stop_duration=2808ms
> > > > > reproduce the problem.
> > > > >
> > > >
> > > > Maybe an option to the tpad filter to decide what should be the
> output
> > > when
> > > > the stop_mode == 1 and cache_stop is NULL?
> > > > So, in some cases, people may want a color frame, and some other will
> > > just
> > > > want to exit right away?
> > > >
> > >
> > > That is too much for a special option.
> > >
> >
> > I understand.
> > I'm just worried that due to some re-init of the filter chain and the
> fact
> > there are filters which may or may not output a frame after the re-init,
> > the tpad filter will then have no cache frame to clone. And in this case,
> > the file does not have the length expected.
> >
> > I guess if it is documented or av_log, people will be aware of the
> > potential issue.
> >
> > Anyway, thanks a lot for the quick fix.
> >
> >
> > > >
> > > >
> > > > >
> > > > >
> > > > >>
> > > > >>
> > > > >> >
> > > > >> >
> > > > >> > > >
> > > > >> > > > iff --git a/libavfilter/vf_tpad.c b/libavfilter/vf_tpad.c
> > > > >> > > > index e5acece1e4..5e4062e9f2 100644
> > > > >> > > > --- a/libavfilter/vf_tpad.c
> > > > >> > > > +++ b/libavfilter/vf_tpad.c
> > > > >> > > > @@ -140,7 +140,7 @@ static int activate(AVFilterContext
> *ctx)
> > > > >> > > >              ff_outlink_set_status(outlink, AVERROR_EOF,
> > > s->pts);
> > > > >> > > >              return 0;
> > > > >> > > >          }
> > > > >> > > > -        if (s->stop_mode == 0) {
> > > > >> > > > +        if (s->stop_mode == 0 || !s->cache_stop) {
> > > > >> > > >              frame = ff_get_video_buffer(outlink,
> outlink->w,
> > > > >> > > outlink->h);
> > > > >> > > >              if (!frame)
> > > > >> > > >                  return AVERROR(ENOMEM);
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > > _______________________________________________
> > > > >> > > > > ffmpeg-cvslog mailing list
> > > > >> > > > > ffmpeg-cvslog@ffmpeg.org
> > > > >> > > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-cvslog
> > > > >> > > > >
> > > > >> > > > > To unsubscribe, visit link above, or email
> > > > >> > > > > ffmpeg-cvslog-request@ffmpeg.org with subject
> > "unsubscribe".
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > >
> > > > >> > > > Thierry Foucu
> > > > >> > > > _______________________________________________
> > > > >> > > > ffmpeg-devel mailing list
> > > > >> > > > ffmpeg-devel@ffmpeg.org
> > > > >> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >> > > >
> > > > >> > > > To unsubscribe, visit link above, or email
> > > > >> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > > >> > > >
> > > > >> > > _______________________________________________
> > > > >> > > ffmpeg-devel mailing list
> > > > >> > > ffmpeg-devel@ffmpeg.org
> > > > >> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >> > >
> > > > >> > > To unsubscribe, visit link above, or email
> > > > >> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > Thierry Foucu
> > > > >> > _______________________________________________
> > > > >> > ffmpeg-devel mailing list
> > > > >> > ffmpeg-devel@ffmpeg.org
> > > > >> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >> >
> > > > >> > To unsubscribe, visit link above, or email
> > > > >> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > > >> >
> > > > >> _______________________________________________
> > > > >> ffmpeg-devel mailing list
> > > > >> ffmpeg-devel@ffmpeg.org
> > > > >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >>
> > > > >> To unsubscribe, visit link above, or email
> > > > >> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Thierry Foucu
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Thierry Foucu
> > > > _______________________________________________
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel@ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel@ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> > >
> >
> >
> > --
> >
> > Thierry Foucu
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>


-- 

Thierry Foucu
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".