[FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Zhao Zhili]

From: Zhao Zhili <zhilizhao@tencent.com>

This can happen if encoder init failed before setup the queue, then
reorder_queue_flush() is called by close().

Fix ticket #10221

Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
---
 libavcodec/nvenc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 8a28454042..dda2dc3ba1 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -178,6 +178,8 @@ static void reorder_queue_flush(AVFifo *queue)
 {
     FrameData fd;
 
+    if (!queue)
+        return;
     while (av_fifo_read(queue, &fd, 1) >= 0)
         av_buffer_unref(&fd.frame_opaque_ref);
 }
-- 
2.25.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Steven Liu]

Zhao Zhili <quinkblack@foxmail.com> 于2023年2月28日周二 18:44写道：
>
> From: Zhao Zhili <zhilizhao@tencent.com>
>
> This can happen if encoder init failed before setup the queue, then
> reorder_queue_flush() is called by close().
>
> Fix ticket #10221
>
> Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
> ---
>  libavcodec/nvenc.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> index 8a28454042..dda2dc3ba1 100644
> --- a/libavcodec/nvenc.c
> +++ b/libavcodec/nvenc.c
> @@ -178,6 +178,8 @@ static void reorder_queue_flush(AVFifo *queue)
>  {
>      FrameData fd;
>
> +    if (!queue)
> +        return;
>      while (av_fifo_read(queue, &fd, 1) >= 0)
>          av_buffer_unref(&fd.frame_opaque_ref);
>  }
> --
> 2.25.1
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".


lgtm

Thanks
Steven
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Timo Rothenpieler]

On 28.02.2023 19:43, Zhao Zhili wrote:
> From: Zhao Zhili <zhilizhao@tencent.com>
> 
> This can happen if encoder init failed before setup the queue, then
> reorder_queue_flush() is called by close().
> 
> Fix ticket #10221
> 
> Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
> ---
>   libavcodec/nvenc.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> index 8a28454042..dda2dc3ba1 100644
> --- a/libavcodec/nvenc.c
> +++ b/libavcodec/nvenc.c
> @@ -178,6 +178,8 @@ static void reorder_queue_flush(AVFifo *queue)
>   {
>       FrameData fd;
>   
> +    if (!queue)
> +        return;
>       while (av_fifo_read(queue, &fd, 1) >= 0)
>           av_buffer_unref(&fd.frame_opaque_ref);
>   }

This should probably be an assert0, and the function should not be 
called to begin with if the queue is NULL.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Zhao Zhili]

> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Timo Rothenpieler
> Sent: 2023年2月28日 22:37
> To: ffmpeg-devel@ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference
> 
> On 28.02.2023 19:43, Zhao Zhili wrote:
> > From: Zhao Zhili <zhilizhao@tencent.com>
> >
> > This can happen if encoder init failed before setup the queue, then
> > reorder_queue_flush() is called by close().
> >
> > Fix ticket #10221
> >
> > Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>
> > ---
> >   libavcodec/nvenc.c | 2 ++
> >   1 file changed, 2 insertions(+)
> >
> > diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> > index 8a28454042..dda2dc3ba1 100644
> > --- a/libavcodec/nvenc.c
> > +++ b/libavcodec/nvenc.c
> > @@ -178,6 +178,8 @@ static void reorder_queue_flush(AVFifo *queue)
> >   {
> >       FrameData fd;
> >
> > +    if (!queue)
> > +        return;
> >       while (av_fifo_read(queue, &fd, 1) >= 0)
> >           av_buffer_unref(&fd.frame_opaque_ref);
> >   }
> 
> This should probably be an assert0, and the function should not be
> called to begin with if the queue is NULL.

I can move the check from here to close(). assert is helpful if it can catch
bugs during development. It doesn't help much here since this is a rare case 
which doesn't triggered during development.

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Timo Rothenpieler]

On 28.02.2023 16:03, Zhao Zhili wrote:
> I can move the check from here to close(). assert is helpful if it can catch
> bugs during development. It doesn't help much here since this is a rare case
> which doesn't triggered during development.

Crashing with an assertion is still much better than dereferencing a 
NULL pointer.

I'll have a look at the calling function in a bit, to see if it can be 
refactored to never run into the chance of calling this function when 
not needed.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".