[FFmpeg-devel] Flushing while decoding , but need already decoded frames

[FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Michael Henrik Bodenhoff via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2344 bytes --]

Hi ,

my team recently had to abandon switching to using FFmpeg from specific decoder implementations (NvDEC, Intel Media SDK , IPP and quite a few codec specific decoders) because of big performance issues because of the way FFmpeg works….. or at least we think it is (we’re FFmpeg noobs 😃  )

It's actually an issue we also had with Intel Media SDK,  leading us to pay Intel to extend Media SDK to do what we needed.

Our product is a video surveillance system, and that means we have to decode a LOT of video streams simultaneously.

For motion detection we want to only decode keyframes, and skip P and B frames , and that works fine with FFmpeg most of the time, except for when the video stream contains B frames.
Without B-Frames it’s really simple (simplified pseudocode) :

while(true)
{
  receiveStreamCompleteFrame();
  If(KeyFrame)
  {
    avcodec_send_packet();
    if(avcodec_receive_frame()==0)
    {
       // do motion detection
    }
  }
}

But! with B Frames FFmpeg doesn’t return keyframes when they are decoded, they are kept, and we can’t seem to flush them out. avcodec_flush_buffers allow us to continue to next keyframe, but it doesn’t seem to give us the keyframe we just gave to FFmpeg with avcodec_send_packet.

while(true)
{
  receiveStreamCompleteFrame();
  If(KeyFrame)
  {
    avcodec_send_packet();
    if(avcodec_receive_frame()==0)
    {
       // do motion detection
    }
    Else
    {
      avcodec_flush_buffers();
      if(avcodec_receive_frame()==0)
      {
         // do motion detection
      }
    }
  }
}

Calling avcodec_receive_frame after calling avcodec_flush_buffer results in -11 and no frame

is there anyway around this ? And if not, could FFmpeg be made to have this functionality ?

I tried contacting one of the FFmpeg consultants from https://ffmpeg.org/consulting.html but never got a response

Michael Bodenhoff
Principal Software Engineer
FT01
Email: mibo@milestone.dk

[Linkedin Logo]<https://linkedin.com/company/milestone-systems>[Twitter Logo]<https://twitter.com/milestonesys>[Youtube Logo]<https://www.youtube.com/user/Milestonesys>[Facebook Logo]<https://www.facebook.com/milestonesys/>[Instagram Logo]<https://www.instagram.com/milestonesystems/>

[Milestone Logo]<https://www.milestonesys.com/>


[-- Attachment #1.2: image001.png --]
[-- Type: image/png, Size: 183 bytes --]

[-- Attachment #1.3: image002.png --]
[-- Type: image/png, Size: 3730 bytes --]

[-- Attachment #1.4: image003.png --]
[-- Type: image/png, Size: 6222 bytes --]

[-- Attachment #1.5: image004.png --]
[-- Type: image/png, Size: 4532 bytes --]

[-- Attachment #1.6: image005.png --]
[-- Type: image/png, Size: 2547 bytes --]

[-- Attachment #1.7: image006.png --]
[-- Type: image/png, Size: 12009 bytes --]

[-- Attachment #1.8: image007.png --]
[-- Type: image/png, Size: 8722 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Andreas Rheinhardt]

Michael Henrik Bodenhoff via ffmpeg-devel:
> Hi ,
> 
> my team recently had to abandon switching to using FFmpeg from specific decoder implementations (NvDEC, Intel Media SDK , IPP and quite a few codec specific decoders) because of big performance issues because of the way FFmpeg works….. or at least we think it is (we’re FFmpeg noobs 😃  )
> 
> It's actually an issue we also had with Intel Media SDK,  leading us to pay Intel to extend Media SDK to do what we needed.
> 
> Our product is a video surveillance system, and that means we have to decode a LOT of video streams simultaneously.
> 
> For motion detection we want to only decode keyframes, and skip P and B frames , and that works fine with FFmpeg most of the time, except for when the video stream contains B frames.
> Without B-Frames it’s really simple (simplified pseudocode) :
> 
> while(true)
> {
>   receiveStreamCompleteFrame();
>   If(KeyFrame)
>   {
>     avcodec_send_packet();
>     if(avcodec_receive_frame()==0)
>     {
>        // do motion detection
>     }
>   }
> }
> 
> But! with B Frames FFmpeg doesn’t return keyframes when they are decoded, they are kept, and we can’t seem to flush them out. avcodec_flush_buffers allow us to continue to next keyframe, but it doesn’t seem to give us the keyframe we just gave to FFmpeg with avcodec_send_packet.
> 
> while(true)
> {
>   receiveStreamCompleteFrame();
>   If(KeyFrame)
>   {
>     avcodec_send_packet();
>     if(avcodec_receive_frame()==0)
>     {
>        // do motion detection
>     }
>     Else
>     {
>       avcodec_flush_buffers();
>       if(avcodec_receive_frame()==0)
>       {
>          // do motion detection
>       }
>     }
>   }
> }
> 
> Calling avcodec_receive_frame after calling avcodec_flush_buffer results in -11 and no frame
> 
> is there anyway around this ? And if not, could FFmpeg be made to have this functionality ?
> 
> I tried contacting one of the FFmpeg consultants from https://ffmpeg.org/consulting.html but never got a response
> 

Send your packet with the keyframe, send a NULL packet (to signal EOF),
then the internally stored frames should be output by
avcodec_receive_frame(). Then flush the decoder (to be able to send new
packets to it).

- Andreas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Andrey Turkin]

Have to say, this issue has been a long grievance of mine. There is no
reason to delay frames when the decoder is set up to ignore B frames
as there is no reordering to be done; ideally this should be
zero-delay case (packet goes in, frame goes out) yet the most common
decoders delay frames anyway, as if to decode B frames. Moreover, with
the "new" send/receive API I think there is no reason to delay frames
at all - a single send_packet could decode and queue multiple frames
to be received, so it makes sense to send frames as soon as possible -
yet that is not the case as well.

пт, 24 мая 2024 г. в 13:17, Andreas Rheinhardt <andreas.rheinhardt@outlook.com>:
>
> Michael Henrik Bodenhoff via ffmpeg-devel:
> > Hi ,
> >
> > my team recently had to abandon switching to using FFmpeg from specific decoder implementations (NvDEC, Intel Media SDK , IPP and quite a few codec specific decoders) because of big performance issues because of the way FFmpeg works….. or at least we think it is (we’re FFmpeg noobs 😃  )
> >
> > It's actually an issue we also had with Intel Media SDK,  leading us to pay Intel to extend Media SDK to do what we needed.
> >
> > Our product is a video surveillance system, and that means we have to decode a LOT of video streams simultaneously.
> >
> > For motion detection we want to only decode keyframes, and skip P and B frames , and that works fine with FFmpeg most of the time, except for when the video stream contains B frames.
> > Without B-Frames it’s really simple (simplified pseudocode) :
> >
> > while(true)
> > {
> >   receiveStreamCompleteFrame();
> >   If(KeyFrame)
> >   {
> >     avcodec_send_packet();
> >     if(avcodec_receive_frame()==0)
> >     {
> >        // do motion detection
> >     }
> >   }
> > }
> >
> > But! with B Frames FFmpeg doesn’t return keyframes when they are decoded, they are kept, and we can’t seem to flush them out. avcodec_flush_buffers allow us to continue to next keyframe, but it doesn’t seem to give us the keyframe we just gave to FFmpeg with avcodec_send_packet.
> >
> > while(true)
> > {
> >   receiveStreamCompleteFrame();
> >   If(KeyFrame)
> >   {
> >     avcodec_send_packet();
> >     if(avcodec_receive_frame()==0)
> >     {
> >        // do motion detection
> >     }
> >     Else
> >     {
> >       avcodec_flush_buffers();
> >       if(avcodec_receive_frame()==0)
> >       {
> >          // do motion detection
> >       }
> >     }
> >   }
> > }
> >
> > Calling avcodec_receive_frame after calling avcodec_flush_buffer results in -11 and no frame
> >
> > is there anyway around this ? And if not, could FFmpeg be made to have this functionality ?
> >
> > I tried contacting one of the FFmpeg consultants from https://ffmpeg.org/consulting.html but never got a response
> >
>
> Send your packet with the keyframe, send a NULL packet (to signal EOF),
> then the internally stored frames should be output by
> avcodec_receive_frame(). Then flush the decoder (to be able to send new
> packets to it).
>
> - Andreas
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Ronald S. Bultje]

Hi,

On Fri, May 24, 2024 at 7:39 AM Andrey Turkin <andrey.turkin@gmail.com>
wrote:

> Have to say, this issue has been a long grievance of mine. There is no
> reason to delay frames when the decoder is set up to ignore B frames
> as there is no reordering to be done
>

Frame threading?

Ronald
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Anton Khirnov]

Quoting Andrey Turkin (2024-05-24 13:39:15)
> Have to say, this issue has been a long grievance of mine. There is no
> reason to delay frames when the decoder is set up to ignore B frames
> as there is no reordering to be done;

This only holds for pre-h264 codecs.

-- 
Anton Khirnov
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Michael Henrik Bodenhoff via ffmpeg-devel]

Hi Andrey ,

"when the decoder is set up to ignore B frames"

How do you do that ? 😲

We must have missed something while we tried using FFmpeg, because we couldn't find a way of telling the decoder to ignore B frames, we actually got so desperate that we considered if it was possible to modify the incoming encoded stream to make FFmpeg believe that the stream doesn't contain B frames ☹

Michael Bodenhoff
Principal Software Engineer

FT01

Email: mibo@milestone.dk
-----Original Message-----
From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of Andrey Turkin
Sent: 24. maj 2024 13:39
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[You don't often get email from andrey.turkin@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Have to say, this issue has been a long grievance of mine. There is no reason to delay frames when the decoder is set up to ignore B frames as there is no reordering to be done; ideally this should be zero-delay case (packet goes in, frame goes out) yet the most common decoders delay frames anyway, as if to decode B frames. Moreover, with the "new" send/receive API I think there is no reason to delay frames at all - a single send_packet could decode and queue multiple frames to be received, so it makes sense to send frames as soon as possible - yet that is not the case as well.

пт, 24 мая 2024 г. в 13:17, Andreas Rheinhardt <andreas.rheinhardt@outlook.com>:
>
> Michael Henrik Bodenhoff via ffmpeg-devel:
> > Hi ,
> >
> > my team recently had to abandon switching to using FFmpeg from
> > specific decoder implementations (NvDEC, Intel Media SDK , IPP and
> > quite a few codec specific decoders) because of big performance
> > issues because of the way FFmpeg works….. or at least we think it is
> > (we’re FFmpeg noobs 😃  )
> >
> > It's actually an issue we also had with Intel Media SDK,  leading us to pay Intel to extend Media SDK to do what we needed.
> >
> > Our product is a video surveillance system, and that means we have to decode a LOT of video streams simultaneously.
> >
> > For motion detection we want to only decode keyframes, and skip P and B frames , and that works fine with FFmpeg most of the time, except for when the video stream contains B frames.
> > Without B-Frames it’s really simple (simplified pseudocode) :
> >
> > while(true)
> > {
> >   receiveStreamCompleteFrame();
> >   If(KeyFrame)
> >   {
> >     avcodec_send_packet();
> >     if(avcodec_receive_frame()==0)
> >     {
> >        // do motion detection
> >     }
> >   }
> > }
> >
> > But! with B Frames FFmpeg doesn’t return keyframes when they are decoded, they are kept, and we can’t seem to flush them out. avcodec_flush_buffers allow us to continue to next keyframe, but it doesn’t seem to give us the keyframe we just gave to FFmpeg with avcodec_send_packet.
> >
> > while(true)
> > {
> >   receiveStreamCompleteFrame();
> >   If(KeyFrame)
> >   {
> >     avcodec_send_packet();
> >     if(avcodec_receive_frame()==0)
> >     {
> >        // do motion detection
> >     }
> >     Else
> >     {
> >       avcodec_flush_buffers();
> >       if(avcodec_receive_frame()==0)
> >       {
> >          // do motion detection
> >       }
> >     }
> >   }
> > }
> >
> > Calling avcodec_receive_frame after calling avcodec_flush_buffer
> > results in -11 and no frame
> >
> > is there anyway around this ? And if not, could FFmpeg be made to have this functionality ?
> >
> > I tried contacting one of the FFmpeg consultants from
> > https://ff/
> > mpeg.org%2Fconsulting.html&data=05%7C02%7Cmibo%40milestone.dk%7C46bc
> > b557d2194eb8b76d08dc7be63528%7C4744e4b2072d4754be71250b45e049fe%7C0%
> > 7C0%7C638521475866379714%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=H
> > V8bPZyvDklo5iGs%2BnrW4uEJ6m%2B9tOExcyvt8W3VxyQ%3D&reserved=0 but
> > never got a response
> >
>
> Send your packet with the keyframe, send a NULL packet (to signal
> EOF), then the internally stored frames should be output by
> avcodec_receive_frame(). Then flush the decoder (to be able to send
> new packets to it).
>
> - Andreas
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmp/
> eg.org%2Fmailman%2Flistinfo%2Fffmpeg-devel&data=05%7C02%7Cmibo%40miles
> tone.dk%7C46bcb557d2194eb8b76d08dc7be63528%7C4744e4b2072d4754be71250b4
> 5e049fe%7C0%7C0%7C638521475866388265%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
> C4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
> &sdata=vxzy9fFGqhTghbquG0O9zUE1e%2FoPrkaJbsPAfkN7NtE%3D&reserved=0
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Flushing while decoding , but need already decoded frames

[Andreas Rheinhardt]

Michael Henrik Bodenhoff via ffmpeg-devel:
> Hi Andrey ,
> 
> "when the decoder is set up to ignore B frames"
> 
> How do you do that ? 😲
> 
> We must have missed something while we tried using FFmpeg, because we couldn't find a way of telling the decoder to ignore B frames, we actually got so desperate that we considered if it was possible to modify the incoming encoded stream to make FFmpeg believe that the stream doesn't contain B frames ☹
> 

There is AVCodecContext.skip_frame; some decoders (not all, but e.g.
H.264) will skip frames based upon this value. But it does not affect
the decoder delay.
(There is also AV_CODEC_FLAG_LOW_DELAY, but this is honored by even less
decoders (only some old ones).)

- Andreas

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".