[FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: fix vtctx reset condition

[FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: fix vtctx reset condition

[Hao Guan]

In vtenc_populate_extradata, the cleanup function vtenc_reset should not
be used when no error occurs, otherwise some color information is lost
(#11036).

This patch checks the status code and conducts the correct cleanup.

Signed-off-by: Hao Guan <hguandl@gmail.com>
---
 libavcodec/videotoolboxenc.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index 15c34d59c3..0aa4885232 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -2766,7 +2766,14 @@ static int vtenc_populate_extradata(AVCodecContext   *avctx,
 
 pe_cleanup:
     CVPixelBufferRelease(pix_buf);
-    vtenc_reset(vtctx);
+
+    if (status) {
+        vtenc_reset(vtctx);
+    } else if (vtctx->session) {
+        CFRelease(vtctx->session);
+        vtctx->session = NULL;
+    }
+
     vtctx->frame_ct_out = 0;
 
     av_assert0(status != 0 || (avctx->extradata && avctx->extradata_size > 0));
-- 
2.39.3 (Apple Git-146)

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: fix vtctx reset condition

[Zhao Zhili]

> On Jun 21, 2024, at 14:08, Hao Guan <hguandl@gmail.com> wrote:
> 
> In vtenc_populate_extradata, the cleanup function vtenc_reset should not
> be used when no error occurs, otherwise some color information is lost
> (#11036).
> 
> This patch checks the status code and conducts the correct cleanup.

LGTM, thanks!

> 
> Signed-off-by: Hao Guan <hguandl@gmail.com>
> ---
> libavcodec/videotoolboxenc.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
> index 15c34d59c3..0aa4885232 100644
> --- a/libavcodec/videotoolboxenc.c
> +++ b/libavcodec/videotoolboxenc.c
> @@ -2766,7 +2766,14 @@ static int vtenc_populate_extradata(AVCodecContext   *avctx,
> 
> pe_cleanup:
>     CVPixelBufferRelease(pix_buf);
> -    vtenc_reset(vtctx);
> +
> +    if (status) {
> +        vtenc_reset(vtctx);
> +    } else if (vtctx->session) {
> +        CFRelease(vtctx->session);
> +        vtctx->session = NULL;
> +    }
> +
>     vtctx->frame_ct_out = 0;
> 
>     av_assert0(status != 0 || (avctx->extradata && avctx->extradata_size > 0));
> -- 
> 2.39.3 (Apple Git-146)
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".