[FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic

[FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic

[Steven Zhou]

fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic

Ever since commit 64f3feb added `avcodec_free_context(&enc->enc_ctx);` to
`enc_free()` the encoder context is closed before filtergraph output context. 
The filtergraph output context holds a reference to last AVFrame to pass through
it for FPS conversion purposes. This is problematic for HWFrameContexts which
may rely on encoder context to free HW AVFrames. This change unrefs the AVFrame
ref held by filtergraph output context during processing of EOF frame.

Signed-off-by: Steven Zhou <steven.zhou@netint.ca>
---
 fftools/ffmpeg_filter.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c
index a0dc4c745e..cf2d016b2d 100644
--- a/fftools/ffmpeg_filter.c
+++ b/fftools/ffmpeg_filter.c
@@ -2560,8 +2560,10 @@ static int fg_output_frame(OutputFilterPriv *ofp, FilterGraphThread *fgt,
         av_frame_move_ref(frame_prev, frame);
     }
 
-    if (!frame)
+    if (!frame) {
+        av_frame_unref(frame_prev);
         return close_output(ofp, fgt);
+    }
 
     return 0;
 }
-- 
2.25.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic

[Zhao Zhili]

> On Jun 24, 2025, at 11:09, Steven Zhou <steven.zhou@netint.ca> wrote:
> 
> fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic
> 
> Ever since commit 64f3feb added `avcodec_free_context(&enc->enc_ctx);` to
> `enc_free()` the encoder context is closed before filtergraph output context. 
> The filtergraph output context holds a reference to last AVFrame to pass through
> it for FPS conversion purposes. This is problematic for HWFrameContexts which
> may rely on encoder context to free HW AVFrames.

What’s the encoder? This sounds a defect in encoder implementation.

> This change unrefs the AVFrame
> ref held by filtergraph output context during processing of EOF frame.
> 
> Signed-off-by: Steven Zhou <steven.zhou@netint.ca>
> ---
> fftools/ffmpeg_filter.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c
> index a0dc4c745e..cf2d016b2d 100644
> --- a/fftools/ffmpeg_filter.c
> +++ b/fftools/ffmpeg_filter.c
> @@ -2560,8 +2560,10 @@ static int fg_output_frame(OutputFilterPriv *ofp, FilterGraphThread *fgt,
>         av_frame_move_ref(frame_prev, frame);
>     }
> 
> -    if (!frame)
> +    if (!frame) {
> +        av_frame_unref(frame_prev);
>         return close_output(ofp, fgt);
> +    }
> 
>     return 0;
> }
> -- 
> 2.25.1
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic

[Steven Zhou]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Zhao
> Zhili
> Sent: Monday, June 23, 2025 8:34 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame
> used for FPS conversion logic
> 
> 
> 
> > On Jun 24, 2025, at 11:09, Steven Zhou <steven.zhou@netint.ca> wrote:
> >
> > fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion
> > logic
> >
> > Ever since commit 64f3feb added `avcodec_free_context(&enc->enc_ctx);`
> > to `enc_free()` the encoder context is closed before filtergraph output
> context.
> > The filtergraph output context holds a reference to last AVFrame to
> > pass through it for FPS conversion purposes. This is problematic for
> > HWFrameContexts which may rely on encoder context to free HW
> AVFrames.
> 
> What’s the encoder? This sounds a defect in encoder implementation.

The encoder is the yet to be upstreamed Netint encoders. I will put them on Forgejo for review later this week.
Perhaps the encoder implementation does not conform to FFmpeg convention. If so, I can try to address this issue another way.

The Netint HW frame data held in `AVFrame->buf[0]` is tied 'sessions' of Netint HW dec/filt/enc in the Netint driver based on what Netint HW session most recently accessed it.
The AVBufferRef destructor for `AVFrame->buf[0]` in Netint HW frames will use the most recent session the HW frame passed through to communicate its destruction to the HW's driver.
This allows for the decoder session which generated a Netint HW AVframe to have its context and session closed whilst the AVFrame continues onto a Netint encoder.
Typically this occurs when decoder is closed for EOF whilst frames are still in flight to the encoder.

I noticed commit 64f3feb closes encoder before filtergraph; but filtergraph holds a ref to last AVFrame for FPS conversion usage.
This ref in filtergraph could be discarded earlier when filtergraph processes the EOF frame, hence this patch.

Alternatively, the Netint codecs can do without this patch. The frame's resources on HW would be free'd by other garbage-collection mechanisms.

> 
> > This change unrefs the AVFrame
> > ref held by filtergraph output context during processing of EOF frame.
> >
> > Signed-off-by: Steven Zhou <steven.zhou@netint.ca>
> > ---
> > fftools/ffmpeg_filter.c | 4 +++-
> > 1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index
> > a0dc4c745e..cf2d016b2d 100644
> > --- a/fftools/ffmpeg_filter.c
> > +++ b/fftools/ffmpeg_filter.c
> > @@ -2560,8 +2560,10 @@ static int fg_output_frame(OutputFilterPriv
> *ofp, FilterGraphThread *fgt,
> >         av_frame_move_ref(frame_prev, frame);
> >     }
> >
> > -    if (!frame)
> > +    if (!frame) {
> > +        av_frame_unref(frame_prev);
> >         return close_output(ofp, fgt);
> > +    }
> >
> >     return 0;
> > }
> > --
> > 2.25.1
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel@ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org
> with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion logic

[Zhao Zhili]

> On Jun 24, 2025, at 12:47, Steven Zhou <steven.zhou@netint.ca> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Zhao
>> Zhili
>> Sent: Monday, June 23, 2025 8:34 PM
>> To: FFmpeg development discussions and patches <ffmpeg-
>> devel@ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH] fftools/ffmpeg_filter: at EOF, unref frame
>> used for FPS conversion logic
>> 
>> 
>> 
>>> On Jun 24, 2025, at 11:09, Steven Zhou <steven.zhou@netint.ca> wrote:
>>> 
>>> fftools/ffmpeg_filter: at EOF, unref frame used for FPS conversion
>>> logic
>>> 
>>> Ever since commit 64f3feb added `avcodec_free_context(&enc->enc_ctx);`
>>> to `enc_free()` the encoder context is closed before filtergraph output
>> context.
>>> The filtergraph output context holds a reference to last AVFrame to
>>> pass through it for FPS conversion purposes. This is problematic for
>>> HWFrameContexts which may rely on encoder context to free HW
>> AVFrames.
>> 
>> What’s the encoder? This sounds a defect in encoder implementation.
> 
> The encoder is the yet to be upstreamed Netint encoders. I will put them on Forgejo for review later this week.
> Perhaps the encoder implementation does not conform to FFmpeg convention. If so, I can try to address this issue another way.
> 
> The Netint HW frame data held in `AVFrame->buf[0]` is tied 'sessions' of Netint HW dec/filt/enc in the Netint driver based on what Netint HW session most recently accessed it.
> The AVBufferRef destructor for `AVFrame->buf[0]` in Netint HW frames will use the most recent session the HW frame passed through to communicate its destruction to the HW's driver.
> This allows for the decoder session which generated a Netint HW AVframe to have its context and session closed whilst the AVFrame continues onto a Netint encoder.
> Typically this occurs when decoder is closed for EOF whilst frames are still in flight to the encoder.

Both v4l2_m2m and mediacodec decoders have similar issue, and they use reference count to delay destroy
decoder context when there are frames on the fly, e.g., mediacodec_decode_close. It’s a little complex, but more
reliable than fix on the user side of libavcodec.

> 
> I noticed commit 64f3feb closes encoder before filtergraph; but filtergraph holds a ref to last AVFrame for FPS conversion usage.
> This ref in filtergraph could be discarded earlier when filtergraph processes the EOF frame, hence this patch.
> 
> Alternatively, the Netint codecs can do without this patch. The frame's resources on HW would be free'd by other garbage-collection mechanisms.
> 
>> 
>>> This change unrefs the AVFrame
>>> ref held by filtergraph output context during processing of EOF frame.
>>> 
>>> Signed-off-by: Steven Zhou <steven.zhou@netint.ca>
>>> ---
>>> fftools/ffmpeg_filter.c | 4 +++-
>>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>> 
>>> diff --git a/fftools/ffmpeg_filter.c b/fftools/ffmpeg_filter.c index
>>> a0dc4c745e..cf2d016b2d 100644
>>> --- a/fftools/ffmpeg_filter.c
>>> +++ b/fftools/ffmpeg_filter.c
>>> @@ -2560,8 +2560,10 @@ static int fg_output_frame(OutputFilterPriv
>> *ofp, FilterGraphThread *fgt,
>>>        av_frame_move_ref(frame_prev, frame);
>>>    }
>>> 
>>> -    if (!frame)
>>> +    if (!frame) {
>>> +        av_frame_unref(frame_prev);
>>>        return close_output(ofp, fgt);
>>> +    }
>>> 
>>>    return 0;
>>> }
>>> --
>>> 2.25.1
>>> 
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel@ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>> 
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>> 
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org
>> with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".