[FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Michael Riedl]

This patch series adds support for WHIP and WHEP (WebRTC-HTTP ingestion protocol
and WebRTC-HTTP egress protocol). It supersedes the previous patch series. Thank
you all for your feedback and suggestions!

The WHIP and WHEP are defined in the following draft RFCs:
- WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip
- WHEP: https://datatracker.ietf.org/doc/draft-murillo-whep/

The implementation builds on FFmpeg's existing RTP, HTTP and SDP support as well
as libdatachannel for the rest. This library is much more lightweight than other
libraries (e.g. libwebrtc). At the same time, using this library avoids
reimplementing parts of WebRTC in FFmpeg.

This patch series was tested with WebRTC servers Dolby.io (formerly Millicast)
and SRS (https://github.com/ossrs/srs). Using a local server, an end-to-end
latency of 3 frames (50 ms at 60 fps) was measured with video-only output, and
12 frames (200 ms at 60 fps) with audio and video output. Using a DeckLink input
device, an end-to-end latency of about 4 frames (70 ms at 60 fps) was measured
with both, video-only and audio and video output.

Using a remote server, only the RTT is added to the end-to-end latency. For
example, using a server in Amsterdam (The Netherlands) from a location in
Salzburg (Austria), with RTT=18ms, an end-to-end latency of 50+18=68 ms was
measured for video-only output.


Michael Riedl (6):
  libavformat/http: expose actual Location header value
  libavformat/sdp: remove whitespaces in fmtp
  configure: add libdatachannel as external library
  libavformat/webrtc: add common code for WebRTC streaming
  libavformat/webrtc_demux: add WebRTC-HTTP egress protocol (WHEP)
    demuxer
  libavformat/webrtc_mux: add WebRTC-HTTP ingestion protocol (WHIP)
    muxer

 Changelog                        |   5 +
 MAINTAINERS                      |   1 +
 configure                        |   7 +
 doc/demuxers.texi                |  22 ++
 doc/muxers.texi                  |  21 ++
 libavformat/Makefile             |   2 +
 libavformat/allformats.c         |   2 +
 libavformat/http.c               |   1 +
 libavformat/sdp.c                |   4 +-
 libavformat/webrtc.c (new)       | 410 +++++++++++++++++++++++++++++++
 libavformat/webrtc.h (new)       |  70 ++++++
 libavformat/webrtc_demux.c (new) | 246 +++++++++++++++++++
 libavformat/webrtc_mux.c (new)   | 273 ++++++++++++++++++++
 13 files changed, 1062 insertions(+), 2 deletions(-)
 create mode 100644 libavformat/webrtc.c
 create mode 100644 libavformat/webrtc.h
 create mode 100644 libavformat/webrtc_demux.c
 create mode 100644 libavformat/webrtc_mux.c

--
2.39.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Tomas Härdin]

This patchset is missing tests. I know that we for some reason don't
really have tests for protocols, but I feel the issue is worthwhile to
bring up. I've worked a bit with WebRTC recently and it's fiddly, so
it'd be nice to have some automated thing that keeps track of which
WebRTC implementations this works with. Or maybe this is better handled
with an external project, testing which implementations interoperate?

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Michael Riedl]

On 11/14/23 11:05, Tomas Härdin wrote:
> This patchset is missing tests. I know that we for some reason don't
> really have tests for protocols, but I feel the issue is worthwhile to
> bring up. I've worked a bit with WebRTC recently and it's fiddly, so
> it'd be nice to have some automated thing that keeps track of which
> WebRTC implementations this works with. Or maybe this is better handled
> with an external project, testing which implementations interoperate?


I agree that having automated tests would be useful for stability in the future.
I tested the patchset with both SRS and Millicast, and it worked well.

For automated testing, we could use FATE, but it needs an extra server with
protection and someone to keep it updated. Testing with paid services like
Millicast is tricky.

Another option is an external project for WebRTC testing, but the challenge is
keeping it maintained and compatible with changes in implementations.

External services might update their software, causing issues with tests. We
would need a plan for dealing with that.

For paid services like Millicast, we need to figure out who pays for the tests.
Understanding the costs is essential if we want to use paid services for
testing.

I'd like to hear your thoughts on these points and how you would propose to
proceed with adding tests for protocols like WebRTC.


_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Tomas Härdin]

tis 2023-11-14 klockan 13:59 +0100 skrev Michael Riedl:
> Another option is an external project for WebRTC testing, but the
> challenge is
> keeping it maintained and compatible with changes in implementations.

Perhaps there are funds to raise for such an effort? I can imagine
Google et al have an interest in this.

> External services might update their software, causing issues with
> tests. We
> would need a plan for dealing with that.

Just making sure various free software implementations interoperate
would be more than enough work, and provides some confidence. Surveying
what features are supported by each implementation would also be of
interest I imagine. For example a client of mine uses aiortc which is
lacking FEC and rate control.

A local company that sometimes contributes to FFmpeg is also involved
in various streaming stuff (though not WebRTC so far, I think).

> For paid services like Millicast, we need to figure out who pays for
> the tests.
> Understanding the costs is essential if we want to use paid services
> for
> testing.
> 
> I'd like to hear your thoughts on these points and how you would
> propose to
> proceed with adding tests for protocols like WebRTC.

The way I've been testing this stuff so far is by hand via Firefox and
Chromium. Both can be automated, though frontend isn't really my area.

As it stands at the moment, we let users be guinea pigs for our
protocols. While not great, this at least means we get bug reports. But
there's no way to ensure old bugs don't resurface.

/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2229 bytes --]

On Tue, Nov 14, 2023 at 01:59:48PM +0100, Michael Riedl wrote:
> On 11/14/23 11:05, Tomas Härdin wrote:
> > This patchset is missing tests. I know that we for some reason don't
> > really have tests for protocols, but I feel the issue is worthwhile to
> > bring up. I've worked a bit with WebRTC recently and it's fiddly, so
> > it'd be nice to have some automated thing that keeps track of which
> > WebRTC implementations this works with. Or maybe this is better handled
> > with an external project, testing which implementations interoperate?
> 
> 
> I agree that having automated tests would be useful for stability in the future.
> I tested the patchset with both SRS and Millicast, and it worked well.
> 
> For automated testing, we could use FATE, but it needs an extra server with
> protection and someone to keep it updated. Testing with paid services like
> Millicast is tricky.
> 
> Another option is an external project for WebRTC testing, but the challenge is
> keeping it maintained and compatible with changes in implementations.
> 
> External services might update their software, causing issues with tests. We
> would need a plan for dealing with that.
> 
> For paid services like Millicast, we need to figure out who pays for the tests.
> Understanding the costs is essential if we want to use paid services for
> testing.
> 
> I'd like to hear your thoughts on these points and how you would propose to
> proceed with adding tests for protocols like WebRTC.

simple, add server support for this to FFmpeg.

FATE is run in cases without network access (for example googles ossfuzz setup
i belives does not permit the fuzzed code to access external things IIRC)

The practice of implementing only one end of a protocol is honestly wrong.
And if there is no usable free server, then even more so.

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you drop bombs on a foreign country and kill a hundred thousand
innocent people, expect your government to call the consequence
"unprovoked inhuman terrorist attacks" and use it to justify dropping
more bombs and killing more people. The technology changed, the idea is old.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Michael Riedl]

On 11/15/23 22:45, Michael Niedermayer wrote:
> On Tue, Nov 14, 2023 at 01:59:48PM +0100, Michael Riedl wrote:
>> On 11/14/23 11:05, Tomas Härdin wrote:
>>> This patchset is missing tests. I know that we for some reason don't
>>> really have tests for protocols, but I feel the issue is worthwhile to
>>> bring up. I've worked a bit with WebRTC recently and it's fiddly, so
>>> it'd be nice to have some automated thing that keeps track of which
>>> WebRTC implementations this works with. Or maybe this is better handled
>>> with an external project, testing which implementations interoperate?
>>
>> I agree that having automated tests would be useful for stability in the future.
>> I tested the patchset with both SRS and Millicast, and it worked well.
>>
>> For automated testing, we could use FATE, but it needs an extra server with
>> protection and someone to keep it updated. Testing with paid services like
>> Millicast is tricky.
>>
>> Another option is an external project for WebRTC testing, but the challenge is
>> keeping it maintained and compatible with changes in implementations.
>>
>> External services might update their software, causing issues with tests. We
>> would need a plan for dealing with that.
>>
>> For paid services like Millicast, we need to figure out who pays for the tests.
>> Understanding the costs is essential if we want to use paid services for
>> testing.
>>
>> I'd like to hear your thoughts on these points and how you would propose to
>> proceed with adding tests for protocols like WebRTC.
> simple, add server support for this to FFmpeg.
>
> FATE is run in cases without network access (for example googles ossfuzz setup
> i belives does not permit the fuzzed code to access external things IIRC)
>
> The practice of implementing only one end of a protocol is honestly wrong.
> And if there is no usable free server, then even more so.
>
> thx

Server support was planned for later, but maybe it's better to do it now.
Multiple solutions are possible, but one solution requires adding 2 more
(de)muxers for server support to this patch series. It would also be possible to
split the patch series into 2 parts, one for WHIP client and server support and
one for WHEP client and server support. But I'm not sure which solution is
better.

My idea is to add 2 more (de)muxers for server support. This would allow us to
test the following:
- WHIP muxer (client) --> WHIP demuxer (server)
- WHEP muxer (server) --> WHEP demuxer (client)

Regardless, this will only test the implementation against itself. If that makes
sense and sounds reasonable, I will add the server support to this patch series.

Please let me know what you think.

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH v2 0/6] WebRTC sub-second live streaming support

[Nicolas George]

Michael Riedl (12023-11-27):
> Regardless, this will only test the implementation against itself. If that makes
> sense and sounds reasonable, I will add the server support to this patch series.

Thanks. It will not be enough in a perfect world, but it still is a very
good start and much better than no testing at all.

Regards,

-- 
  Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".