Re: [FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

Re: [FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Andriy Gelman]

On Tue, 14. Dec 02:12, Cameron Gutman wrote:
> The V4L2M2M API operates asynchronously, so multiple packets can
> be enqueued before getting a batch of frames back. Since it was
> only possible to receive a frame by submitting another packet,
> there wasn't a way to drain those excess output frames from when
> avcodec_receive_frame() returned AVERROR(EAGAIN).
> 
> In my testing, this change reduced decode latency of a real-time
> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
> Raspberry Pi 4.
> 
> Signed-off-by: Cameron Gutman <aicommander@gmail.com>
> ---
>  libavcodec/v4l2_m2m_dec.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/v4l2_m2m_dec.c b/libavcodec/v4l2_m2m_dec.c
> index 224eb3d5e7..b0c3d30ac8 100644
> --- a/libavcodec/v4l2_m2m_dec.c
> +++ b/libavcodec/v4l2_m2m_dec.c
> @@ -142,8 +142,12 @@ static int v4l2_receive_frame(AVCodecContext *avctx, AVFrame *frame)
>  
>      if (!s->buf_pkt.size) {
>          ret = ff_decode_get_packet(avctx, &s->buf_pkt);
> -        if (ret < 0 && ret != AVERROR_EOF)
> -            return ret;
> +        if (ret < 0) {
> +            if (ret == AVERROR(EAGAIN))
> +                return ff_v4l2_context_dequeue_frame(capture, frame, 0);
> +            else if (ret != AVERROR_EOF)
> +                return ret;
> +        }
>      }

The change lgtm. I tested on Rpi4 and odroid xu4.
Will apply this on Sunday if no one else comments.

Thanks,
-- 
Andriy
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Andriy Gelman]

On Tue, 14. Dec 02:12, Cameron Gutman wrote:
> The V4L2M2M API operates asynchronously, so multiple packets can
> be enqueued before getting a batch of frames back. Since it was
> only possible to receive a frame by submitting another packet,
> there wasn't a way to drain those excess output frames from when
> avcodec_receive_frame() returned AVERROR(EAGAIN).
> 

> In my testing, this change reduced decode latency of a real-time
> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
> Raspberry Pi 4.

I was doing some more tests today, but didn't have any luck dequeuing a frame
if ff_decode_get_packet() returned EAGAIN.
Could you share the dataset?

Thanks,
-- 
Andriy
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Cameron Gutman]

On 12/27/21 00:17, Andriy Gelman wrote:
> On Tue, 14. Dec 02:12, Cameron Gutman wrote:
>> The V4L2M2M API operates asynchronously, so multiple packets can
>> be enqueued before getting a batch of frames back. Since it was
>> only possible to receive a frame by submitting another packet,
>> there wasn't a way to drain those excess output frames from when
>> avcodec_receive_frame() returned AVERROR(EAGAIN).
>>
> 
>> In my testing, this change reduced decode latency of a real-time
>> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
>> Raspberry Pi 4.
> 
> I was doing some more tests today, but didn't have any luck dequeuing a frame
> if ff_decode_get_packet() returned EAGAIN.

Hmm, maybe there is something different about your test harness?
I'm receiving 720p 60 FPS H.264 ES in real-time from the network.

For each H.264 encoded frame I receive off the network, my basic
approach is like this (simplified for brevity):

avcodec_send_packet(&pkt);
do {
    frame = av_frame_alloc();
    if ((err = avcodec_receive_frame(frame)) == 0) {
        render_frame(frame);
    }
} while (err == 0);

I'll usually get EAGAIN immediately for the first few frames I submit
(so no output frame yet), but then I'll get a batch of output frames
back after the first completed decode. That drains the excess latency
from the pipeline to avoid always being behind.

For cases where we want to prioritize latency over throughput, I've
had success with this approach too:

avcodec_send_packet(&pkt);
while (avcodec_receive_frame(frame) == AVERROR(EAGAIN)) {
    msleep(1);
}
render_frame(frame);

In this case, we can retry avcodec_receive_frame() until we get the
frame back that we just submitted for decoding.

The patch here enables both of these use-cases by allowing V4L2M2M
to retry getting a decoded frame without new input data. Both of
these also work with MMAL after the recent decoupled dataflow patch.

> Could you share the dataset?
> 

It is 720p60.h264 from here:
https://onedrive.live.com/?authkey=%21ALoKfcPfFeKyhzs&id=C15BF9770619F56%21165617&cid=0C15BF9770619F56

> Thanks,
> 

Thank you
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Andriy Gelman]

On Mon, 27. Dec 03:18, Cameron Gutman wrote:
> On 12/27/21 00:17, Andriy Gelman wrote:
> > On Tue, 14. Dec 02:12, Cameron Gutman wrote:
> >> The V4L2M2M API operates asynchronously, so multiple packets can
> >> be enqueued before getting a batch of frames back. Since it was
> >> only possible to receive a frame by submitting another packet,
> >> there wasn't a way to drain those excess output frames from when
> >> avcodec_receive_frame() returned AVERROR(EAGAIN).
> >>
> > 
> >> In my testing, this change reduced decode latency of a real-time
> >> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
> >> Raspberry Pi 4.
> > 
> > I was doing some more tests today, but didn't have any luck dequeuing a frame
> > if ff_decode_get_packet() returned EAGAIN.
> 
> Hmm, maybe there is something different about your test harness?
> I'm receiving 720p 60 FPS H.264 ES in real-time from the network.

I used ffmpeg, i.e.
./ffmpeg -codec:v h264_v4l2m2m -i 720p60.h264 -f null -

Anyway, I applied your patch but just removed the comment on latency
because I couldn't reproduce it.

> 
> For each H.264 encoded frame I receive off the network, my basic
> approach is like this (simplified for brevity):
> 
> avcodec_send_packet(&pkt);
> do {
>     frame = av_frame_alloc();
>     if ((err = avcodec_receive_frame(frame)) == 0) {
>         render_frame(frame);
>     }
> } while (err == 0);
> 
> I'll usually get EAGAIN immediately for the first few frames I submit
> (so no output frame yet), but then I'll get a batch of output frames
> back after the first completed decode. That drains the excess latency
> from the pipeline to avoid always being behind.
> 
> For cases where we want to prioritize latency over throughput, I've
> had success with this approach too:
> 
> avcodec_send_packet(&pkt);
> while (avcodec_receive_frame(frame) == AVERROR(EAGAIN)) {
>     msleep(1);
> }
> render_frame(frame);
> 
> In this case, we can retry avcodec_receive_frame() until we get the
> frame back that we just submitted for decoding.

> 
> The patch here enables both of these use-cases by allowing V4L2M2M
> to retry getting a decoded frame without new input data. Both of
> these also work with MMAL after the recent decoupled dataflow patch.
> 
> > Could you share the dataset?
> > 
> 

> It is 720p60.h264 from here:
> https://onedrive.live.com/?authkey=%21ALoKfcPfFeKyhzs&id=C15BF9770619F56%21165617&cid=0C15BF9770619F56

I could not decode this dataset on rpi4 (before or after the patch). Tried to upgrade kernel
to 5.16, but still same problem. Odroid xu4 seems to work fine.

I found a repo in github https://github.com/jc-kynesim/rpi-ffmpeg (I guess Rpi4
team), where the decoding is working. Doing a bisect points to this commit that fixes:
https://github.com/jc-kynesim/rpi-ffmpeg/commit/3e60c08552110d9da1b3b5d3637c6bd8b53eec69

There are lots of changes in the commit, so I'm not sure what exactly fixed the
problem or if it was by chance. It would be good to figure out.

-- 
Andriy
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Andriy Gelman]

On Tue, 28. Dec 18:41, Andriy Gelman wrote:
> On Mon, 27. Dec 03:18, Cameron Gutman wrote:
> > On 12/27/21 00:17, Andriy Gelman wrote:
> > > On Tue, 14. Dec 02:12, Cameron Gutman wrote:
> > >> The V4L2M2M API operates asynchronously, so multiple packets can
> > >> be enqueued before getting a batch of frames back. Since it was
> > >> only possible to receive a frame by submitting another packet,
> > >> there wasn't a way to drain those excess output frames from when
> > >> avcodec_receive_frame() returned AVERROR(EAGAIN).
> > >>
> > > 
> > >> In my testing, this change reduced decode latency of a real-time
> > >> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
> > >> Raspberry Pi 4.
> > > 
> > > I was doing some more tests today, but didn't have any luck dequeuing a frame
> > > if ff_decode_get_packet() returned EAGAIN.
> > 
> > Hmm, maybe there is something different about your test harness?
> > I'm receiving 720p 60 FPS H.264 ES in real-time from the network.
> 
> I used ffmpeg, i.e.
> ./ffmpeg -codec:v h264_v4l2m2m -i 720p60.h264 -f null -
> 
> Anyway, I applied your patch but just removed the comment on latency
> because I couldn't reproduce it.
> 
> > 
> > For each H.264 encoded frame I receive off the network, my basic
> > approach is like this (simplified for brevity):
> > 
> > avcodec_send_packet(&pkt);
> > do {
> >     frame = av_frame_alloc();
> >     if ((err = avcodec_receive_frame(frame)) == 0) {
> >         render_frame(frame);
> >     }
> > } while (err == 0);
> > 
> > I'll usually get EAGAIN immediately for the first few frames I submit
> > (so no output frame yet), but then I'll get a batch of output frames
> > back after the first completed decode. That drains the excess latency
> > from the pipeline to avoid always being behind.
> > 
> > For cases where we want to prioritize latency over throughput, I've
> > had success with this approach too:
> > 
> > avcodec_send_packet(&pkt);
> > while (avcodec_receive_frame(frame) == AVERROR(EAGAIN)) {
> >     msleep(1);
> > }
> > render_frame(frame);
> > 
> > In this case, we can retry avcodec_receive_frame() until we get the
> > frame back that we just submitted for decoding.
> 
> > 
> > The patch here enables both of these use-cases by allowing V4L2M2M
> > to retry getting a decoded frame without new input data. Both of
> > these also work with MMAL after the recent decoupled dataflow patch.
> > 
> > > Could you share the dataset?
> > > 
> > 
> 
> > It is 720p60.h264 from here:
> > https://onedrive.live.com/?authkey=%21ALoKfcPfFeKyhzs&id=C15BF9770619F56%21165617&cid=0C15BF9770619F56

> 
> I could not decode this dataset on rpi4 (before or after the patch). Tried to upgrade kernel
> to 5.16, but still same problem. Odroid xu4 seems to work fine.

This patch fixes decoding on the RPi4 for me:
https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210819085533.1174-2-ming.qian@nxp.com/

It would be nice to get the fix into the upcoming release.

-- 
Andriy
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".