From: Steven Liu <lingjiujianke@gmail.com>
To: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Cc: Jun Zhao <barryjzhao@tencent.com>, yangrtc <yangrtc@aliyun.com>,
Steven Liu <lq@chinaffmpeg.org>, Jack Lau <jacklau1222@qq.com>,
cloudwebrtc <duanweiwei1982@gmail.com>,
winlin <winlinvip@gmail.com>, Haibo Chen <495810242@qq.com>
Subject: Re: [FFmpeg-devel] [PATCH v3] avformat/whip: Add WHIP muxer support for subsecond latency streaming
Date: Wed, 4 Jun 2025 11:21:49 +0800
Message-ID: <CADxeRw=5o1NvZTvjr3buZ6fN0n-4i_QdAg41-Ne=MShEhA7XfQ@mail.gmail.com> (raw)
In-Reply-To: <tencent_9427342366B3287CD963D850915A242F2408@qq.com>
Jack Lau via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> 于2025年5月17日周六 17:25写道：
>
> 0. WHIP Version 3.
> 1. The WHIP muxer has been renamed and refined, with improved logging context and error messages for SSL, DTLS, and RTC.
> 2. Magic numbers have been replaced with macros and extracted to functions, and log levels have been altered for better clarity.
> 3. DTLS curve list has been updated, and SRTP profile names have been refined for FFmpeg and OpenSSL.
> 4. ICE STUN magic number has been refined, and RTP payload types have been updated based on Chrome's definition.
> 5. Fixed frame size has been refined to rtc->audio_par->frame_size, and h264_mp4toannexb is now used to convert MP4/ISOM to annexb.
> 6. OPUS timestamp issue has been addressed, and marker setting has been corrected after utilizing BSF.
> 7. DTLS handshake and ICE handling have been optimized for improved performance, with a single handshake timeout and server role to prevent ARQ.
> 8. Consolidated ICE request/response handling and DTLS handshake into a single function, and fixed OpenSSL build errors to work with Pion.
> 9. Merge TLS & DTLS implementation, shared BIO callbacks, read, write, print_ssl_error, openssl_init_ca_key_cert, init_bio_method function and shared same data structure
> 10. Modify configure that whip is enabled only dtls is enabled(just support openssl for now) to fix build error
>
> Co-authored-by: winlin <winlinvip@gmail.com>
> Co-authored-by: yangrtc <yangrtc@aliyun.com>
> Co-authored-by: cloudwebrtc <duanweiwei1982@gmail.com>
> Co-authored-by: Haibo Chen <495810242@qq.com>
> Co-authored-by: Steven Liu <lq@chinaffmpeg.org>
> Co-authored-by: Jun Zhao <barryjzhao@tencent.com>
> Signed-off-by: Jack Lau <jacklau1222@qq.com>
> ---
> configure | 12 +
> doc/muxers.texi | 47 +
> libavformat/Makefile | 1 +
> libavformat/allformats.c | 1 +
> libavformat/avio.c | 5 +-
> libavformat/http.c | 6 +
> libavformat/http.h | 2 +
> libavformat/protocols.c | 1 +
> libavformat/srtp.h | 4 +-
> libavformat/tls.c | 70 +-
> libavformat/tls.h | 58 +-
> libavformat/tls_openssl.c | 857 ++++++++++++++++-
> libavformat/whip.c | 1917 +++++++++++++++++++++++++++++++++++++
> 13 files changed, 2925 insertions(+), 56 deletions(-)
> create mode 100644 libavformat/whip.c
>
> diff --git a/configure b/configure
> index 0609dac4ab..2a3166962a 100755
> --- a/configure
> +++ b/configure
> @@ -3747,6 +3747,7 @@ wav_demuxer_select="riffdec"
> wav_muxer_select="riffenc"
> webm_chunk_muxer_select="webm_muxer"
> webm_dash_manifest_demuxer_select="matroska_demuxer"
> +whip_muxer_deps_any="dtls_protocol"
> wtv_demuxer_select="mpegts_demuxer riffdec"
> wtv_muxer_select="mpegts_muxer riffenc"
> xmv_demuxer_select="riffdec"
> @@ -3845,6 +3846,9 @@ srtp_protocol_select="rtp_protocol srtp"
> tcp_protocol_select="network"
> tls_protocol_deps_any="gnutls openssl schannel securetransport libtls mbedtls"
> tls_protocol_select="tcp_protocol"
> +# TODO: Support libtls, mbedtls, and gnutls.
> +dtls_protocol_deps_any="openssl"
> +dtls_protocol_select="udp_protocol"
> udp_protocol_select="network"
> udplite_protocol_select="network"
> unix_protocol_deps="sys_un_h"
> @@ -7192,6 +7196,14 @@ enabled rkmpp && { require_pkg_config rkmpp rockchip_mpp rockchip/r
> }
> enabled vapoursynth && require_headers "vapoursynth/VSScript4.h vapoursynth/VapourSynth4.h"
>
> +enabled openssl && {
> + enabled whip_muxer && {
> + $pkg_config --exists --print-errors "openssl >= 1.0.1k" ||
> + require_pkg_config openssl "openssl >= 1.0.1k" openssl/ssl.h SSL_library_init ||
> + require_pkg_config openssl "openssl >= 1.0.1k" openssl/ssl.h OPENSSL_init_ssl
> + }
> +}
> +
>
> if enabled gcrypt; then
> GCRYPT_CONFIG="${cross_prefix}libgcrypt-config"
> diff --git a/doc/muxers.texi b/doc/muxers.texi
> index 04b7f20b7e..30c95c3d34 100644
> --- a/doc/muxers.texi
> +++ b/doc/muxers.texi
> @@ -3879,4 +3879,51 @@ ffmpeg -f webm_dash_manifest -i video1.webm \
> manifest.xml
> @end example
>
> +@anchor{whip}
> +@section whip
> +
> +WebRTC (Real-Time Communication) muxer that supports sub-second latency streaming according to
> +the WHIP (WebRTC-HTTP ingestion protocol) specification.
> +
> +It uses HTTP as a signaling protocol to exchange SDP capabilities and ICE lite candidates. Then,
> +it uses STUN binding requests and responses to establish a session over UDP. Subsequently, it
> +initiates a DTLS handshake to exchange the SRTP encryption keys. Lastly, it splits video and
> +audio frames into RTP packets and encrypts them using SRTP.
> +
> +Ensure that you use H.264 without B frames and Opus for the audio codec. For example, to convert
> +an input file with @command{ffmpeg} to WebRTC:
> +@example
> +ffmpeg -re -i input.mp4 -acodec libopus -ar 48000 -ac 2 \
> + -vcodec libx264 -profile:v baseline -tune zerolatency -threads 1 -bf 0 \
> + -f whip "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream"
> +@end example
> +
> +For this example, we have employed low latency options, resulting in an end-to-end latency of
> +approximately 150ms.
> +
> +@subsection Options
> +
> +This muxer supports the following options:
> +
> +@table @option
> +
> +@item handshake_timeout @var{integer}
> +Set the timeout in milliseconds for ICE and DTLS handshake.
> +Default value is 5000.
> +
> +@item pkt_size @var{integer}
> +Set the maximum size, in bytes, of RTP packets that send out.
> +Default value is 1500.
> +
> +@item authorization @var{string}
> +The optional Bearer token for WHIP Authorization.
> +
> +@item cert_file @var{string}
> +The optional certificate file path for DTLS.
> +
> +@item key_file @var{string}
> +The optional private key file path for DTLS.
> +
> +@end table
> +
> @c man end MUXERS
> diff --git a/libavformat/Makefile b/libavformat/Makefile
> index 6c9992adab..96a6430158 100644
> --- a/libavformat/Makefile
> +++ b/libavformat/Makefile
> @@ -637,6 +637,7 @@ OBJS-$(CONFIG_WEBM_CHUNK_MUXER) += webm_chunk.o
> OBJS-$(CONFIG_WEBP_MUXER) += webpenc.o
> OBJS-$(CONFIG_WEBVTT_DEMUXER) += webvttdec.o subtitles.o
> OBJS-$(CONFIG_WEBVTT_MUXER) += webvttenc.o
> +OBJS-$(CONFIG_WHIP_MUXER) += whip.o avc.o http.o srtp.o tls_openssl.o
> OBJS-$(CONFIG_WSAUD_DEMUXER) += westwood_aud.o
> OBJS-$(CONFIG_WSAUD_MUXER) += westwood_audenc.o
> OBJS-$(CONFIG_WSD_DEMUXER) += wsddec.o rawdec.o
> diff --git a/libavformat/allformats.c b/libavformat/allformats.c
> index b5a23f9c17..3e5f127dc1 100644
> --- a/libavformat/allformats.c
> +++ b/libavformat/allformats.c
> @@ -517,6 +517,7 @@ extern const FFOutputFormat ff_webp_muxer;
> extern const FFInputFormat ff_webvtt_demuxer;
> extern const FFOutputFormat ff_webvtt_muxer;
> extern const FFInputFormat ff_wsaud_demuxer;
> +extern const FFOutputFormat ff_whip_muxer;
> extern const FFOutputFormat ff_wsaud_muxer;
> extern const FFInputFormat ff_wsd_demuxer;
> extern const FFInputFormat ff_wsvqa_demuxer;
> diff --git a/libavformat/avio.c b/libavformat/avio.c
> index d109f3adff..b146ac9f19 100644
> --- a/libavformat/avio.c
> +++ b/libavformat/avio.c
> @@ -339,8 +339,9 @@ static const struct URLProtocol *url_find_protocol(const char *filename)
> }
> }
> av_freep(&protocols);
> - if (av_strstart(filename, "https:", NULL) || av_strstart(filename, "tls:", NULL))
> - av_log(NULL, AV_LOG_WARNING, "https protocol not found, recompile FFmpeg with "
> + if (av_strstart(filename, "https:", NULL) || av_strstart(filename, "tls:", NULL) ||
> + av_strstart(filename, "dtls:", NULL))
> + av_log(NULL, AV_LOG_WARNING, "https or dtls protocol not found, recompile FFmpeg with "
> "openssl, gnutls or securetransport enabled.\n");
>
> return NULL;
> diff --git a/libavformat/http.c b/libavformat/http.c
> index f7b2a8a029..ff63c25969 100644
> --- a/libavformat/http.c
> +++ b/libavformat/http.c
> @@ -562,6 +562,12 @@ int ff_http_averror(int status_code, int default_averror)
> return default_averror;
> }
>
> +const char* ff_http_get_new_location(URLContext *h)
> +{
> + HTTPContext *s = h->priv_data;
> + return s->new_location;
> +}
> +
> static int http_write_reply(URLContext* h, int status_code)
> {
> int ret, body = 0, reply_code, message_len;
> diff --git a/libavformat/http.h b/libavformat/http.h
> index 5f650ef143..d1b691826b 100644
> --- a/libavformat/http.h
> +++ b/libavformat/http.h
> @@ -62,4 +62,6 @@ int ff_http_do_new_request2(URLContext *h, const char *uri, AVDictionary **optio
>
> int ff_http_averror(int status_code, int default_averror);
>
> +const char* ff_http_get_new_location(URLContext *h);
> +
> #endif /* AVFORMAT_HTTP_H */
> diff --git a/libavformat/protocols.c b/libavformat/protocols.c
> index 93a6d67261..d394454d41 100644
> --- a/libavformat/protocols.c
> +++ b/libavformat/protocols.c
> @@ -62,6 +62,7 @@ extern const URLProtocol ff_subfile_protocol;
> extern const URLProtocol ff_tee_protocol;
> extern const URLProtocol ff_tcp_protocol;
> extern const URLProtocol ff_tls_protocol;
> +extern const URLProtocol ff_dtls_protocol;
> extern const URLProtocol ff_udp_protocol;
> extern const URLProtocol ff_udplite_protocol;
> extern const URLProtocol ff_unix_protocol;
> diff --git a/libavformat/srtp.h b/libavformat/srtp.h
> index 3189f8f54b..35224cc9ba 100644
> --- a/libavformat/srtp.h
> +++ b/libavformat/srtp.h
> @@ -27,7 +27,7 @@
> struct AVAES;
> struct AVHMAC;
>
> -struct SRTPContext {
> +typedef struct SRTPContext {
> struct AVAES *aes;
> struct AVHMAC *hmac;
> int rtp_hmac_size, rtcp_hmac_size;
> @@ -40,7 +40,7 @@ struct SRTPContext {
> uint32_t roc;
>
> uint32_t rtcp_index;
> -};
> +} SRTPContext;
>
> int ff_srtp_set_crypto(struct SRTPContext *s, const char *suite,
> const char *params);
> diff --git a/libavformat/tls.c b/libavformat/tls.c
> index f96ff6215d..e06b7022bf 100644
> --- a/libavformat/tls.c
> +++ b/libavformat/tls.c
> @@ -1,6 +1,7 @@
> /*
> - * TLS/SSL Protocol
> + * TLS/DTLS/SSL Protocol
> * Copyright (c) 2011 Martin Storsjo
> + * Copyright (c) 2025 Jack Lau
> *
> * This file is part of FFmpeg.
> *
> @@ -20,6 +21,7 @@
> */
>
> #include "avformat.h"
> +#include "internal.h"
> #include "network.h"
> #include "os_support.h"
> #include "url.h"
> @@ -93,7 +95,7 @@ int ff_tls_open_underlying(TLSShared *c, URLContext *parent, const char *uri, AV
> c->listen = 1;
> }
>
> - ff_url_join(buf, sizeof(buf), "tcp", NULL, c->underlying_host, port, "%s", p);
> + ff_url_join(buf, sizeof(buf), c->is_dtls ? "udp" : "tcp", NULL, c->underlying_host, port, "%s", p);
>
> hints.ai_flags = AI_NUMERICHOST;
> if (!getaddrinfo(c->underlying_host, NULL, &hints, &ai)) {
> @@ -124,7 +126,65 @@ int ff_tls_open_underlying(TLSShared *c, URLContext *parent, const char *uri, AV
> }
>
> freeenv_utf8(env_http_proxy);
> - return ffurl_open_whitelist(&c->tcp, buf, AVIO_FLAG_READ_WRITE,
> - &parent->interrupt_callback, options,
> - parent->protocol_whitelist, parent->protocol_blacklist, parent);
> + if (c->is_dtls) {
> + av_dict_set_int(options, "connect", 1, 0);
> + av_dict_set_int(options, "fifo_size", 0, 0);
> + /* Set the max packet size to the buffer size. */
> + av_dict_set_int(options, "pkt_size", c->mtu, 0);
> + }
> + ret = ffurl_open_whitelist(c->is_dtls ? &c->udp : &c->tcp, buf, AVIO_FLAG_READ_WRITE,
> + &parent->interrupt_callback, options,
> + parent->protocol_whitelist, parent->protocol_blacklist, parent);
> + if (c->is_dtls) {
> + if (ret < 0) {
> + av_log(c, AV_LOG_ERROR, "WHIP: Failed to connect udp://%s:%d\n", c->underlying_host, port);
> + return ret;
> + }
> + /* Make the socket non-blocking, set to READ and WRITE mode after connected */
> + ff_socket_nonblock(ffurl_get_file_handle(c->udp), 1);
> + c->udp->flags |= AVIO_FLAG_READ | AVIO_FLAG_NONBLOCK;
> + }
> + return ret;
> }
> +
> +/**
> + * Read all data from the given URL url and store it in the given buffer bp.
> + */
> +int ff_url_read_all(const char *url, AVBPrint *bp)
> +{
> + int ret = 0;
> + AVDictionary *opts = NULL;
> + URLContext *uc = NULL;
> + char buf[MAX_URL_SIZE];
> +
> + ret = ffurl_open_whitelist(&uc, url, AVIO_FLAG_READ, NULL, &opts, NULL, NULL, NULL);
> + if (ret < 0) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to open url %s\n", url);
> + goto end;
> + }
> +
> + while (1) {
> + ret = ffurl_read(uc, buf, sizeof(buf));
> + if (ret == AVERROR_EOF) {
> + /* Reset the error because we read all response as answer util EOF. */
> + ret = 0;
> + break;
> + }
> + if (ret <= 0) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to read from url=%s, key is %s\n", url, bp->str);
> + goto end;
> + }
> +
> + av_bprintf(bp, "%.*s", ret, buf);
> + if (!av_bprint_is_complete(bp)) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Exceed max size %.*s, %s\n", ret, buf, bp->str);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> + }
> +
> +end:
> + ffurl_closep(&uc);
> + av_dict_free(&opts);
> + return ret;
> +}
> \ No newline at end of file
> diff --git a/libavformat/tls.h b/libavformat/tls.h
> index 6c6aa01a9a..cb626f1977 100644
> --- a/libavformat/tls.h
> +++ b/libavformat/tls.h
> @@ -1,6 +1,7 @@
> /*
> - * TLS/SSL Protocol
> + * TLS/DTLS/SSL Protocol
> * Copyright (c) 2011 Martin Storsjo
> + * Copyright (c) 2025 Jack Lau
> *
> * This file is part of FFmpeg.
> *
> @@ -22,10 +23,27 @@
> #ifndef AVFORMAT_TLS_H
> #define AVFORMAT_TLS_H
>
> +#include "libavutil/bprint.h"
> #include "libavutil/opt.h"
>
> #include "url.h"
>
> +/**
> + * Maximum size limit of a certificate and private key size.
> + */
> +#define MAX_CERTIFICATE_SIZE 8192
> +
> +enum DTLSState {
> + DTLS_STATE_NONE,
> +
> + /* Whether DTLS handshake is finished. */
> + DTLS_STATE_FINISHED,
> + /* Whether DTLS session is closed. */
> + DTLS_STATE_CLOSED,
> + /* Whether DTLS handshake is failed. */
> + DTLS_STATE_FAILED,
> +};
> +
> typedef struct TLSShared {
> char *ca_file;
> int verify;
> @@ -40,6 +58,25 @@ typedef struct TLSShared {
> int numerichost;
>
> URLContext *tcp;
> +
> + int is_dtls;
> +
> + enum DTLSState state;
> +
> + int use_external_udp;
> + URLContext *udp;
> +
> + /* The fingerprint of certificate, used in SDP offer. */
> + char *fingerprint;
> +
> + /* The certificate and private key content used for DTLS handshake */
> + char* cert_buf;
> + char* key_buf;
> + /**
> + * The size of RTP packet, should generally be set to MTU.
> + * Note that pion requires a smaller value, for example, 1200.
> + */
> + int mtu;
> } TLSShared;
>
> #define TLS_OPTFL (AV_OPT_FLAG_DECODING_PARAM | AV_OPT_FLAG_ENCODING_PARAM)
> @@ -51,10 +88,27 @@ typedef struct TLSShared {
> {"key_file", "Private key file", offsetof(pstruct, options_field . key_file), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL }, \
> {"listen", "Listen for incoming connections", offsetof(pstruct, options_field . listen), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, 1, .flags = TLS_OPTFL }, \
> {"verifyhost", "Verify against a specific hostname", offsetof(pstruct, options_field . host), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL }, \
> - {"http_proxy", "Set proxy to tunnel through", offsetof(pstruct, options_field . http_proxy), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL }
> + {"http_proxy", "Set proxy to tunnel through", offsetof(pstruct, options_field . http_proxy), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL }, \
> + {"use_external_udp", "Use external UDP from muxer or demuxer", offsetof(pstruct, options_field . use_external_udp), AV_OPT_TYPE_INT, { .i64 = 0}, 0, 1, .flags = TLS_OPTFL }, \
> + {"mtu", "Maximum Transmission Unit", offsetof(pstruct, options_field . mtu), AV_OPT_TYPE_INT, { .i64 = 0}, INT64_MIN, INT64_MAX, .flags = TLS_OPTFL}, \
> + {"fingerprint", "The optional fingerprint for DTLS", offsetof(pstruct, options_field . fingerprint), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL}, \
> + {"cert_buf", "The optional certificate buffer for DTLS", offsetof(pstruct, options_field . cert_buf), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL}, \
> + {"key_buf", "The optional private key buffer for DTLS", offsetof(pstruct, options_field . key_buf), AV_OPT_TYPE_STRING, .flags = TLS_OPTFL}
>
> int ff_tls_open_underlying(TLSShared *c, URLContext *parent, const char *uri, AVDictionary **options);
>
> +int ff_url_read_all(const char *url, AVBPrint *bp);
> +
> +int ff_dtls_set_udp(URLContext *h, URLContext *udp);
> +
> +int ff_dtls_export_materials(URLContext *h, char *dtls_srtp_materials, size_t materials_sz);
> +
> +int ff_dtls_state(URLContext *h);
> +
> +int ff_ssl_read_key_cert(char *key_url, char *cert_url, char *key_buf, size_t key_sz, char *cert_buf, size_t cert_sz, char **fingerprint);
> +
> +int ff_ssl_gen_key_cert(char *key_buf, size_t key_sz, char *cert_buf, size_t cert_sz, char **fingerprint);
> +
> void ff_gnutls_init(void);
> void ff_gnutls_deinit(void);
>
> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
> index 8b0cf9efb2..b589d5d90a 100644
> --- a/libavformat/tls_openssl.c
> +++ b/libavformat/tls_openssl.c
> @@ -1,6 +1,7 @@
> /*
> - * TLS/SSL Protocol
> + * TLS/DTLS/SSL Protocol
> * Copyright (c) 2011 Martin Storsjo
> + * Copyright (c) 2025 Jack Lau
> *
> * This file is part of FFmpeg.
> *
> @@ -19,8 +20,10 @@
> * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> */
>
> +#include "libavutil/mem.h"
> #include "network.h"
> #include "os_support.h"
> +#include "libavutil/random_seed.h"
> #include "url.h"
> #include "tls.h"
> #include "libavutil/opt.h"
> @@ -29,6 +32,436 @@
> #include <openssl/ssl.h>
> #include <openssl/err.h>
>
> +/**
> + * Returns a heap‐allocated null‐terminated string containing
> + * the PEM‐encoded public key. Caller must free.
> + */
> +static char *pkey_to_pem_string(EVP_PKEY *pkey) {
> + BIO *mem = NULL;
> + BUF_MEM *bptr = NULL;
> + char *pem_str = NULL;
> +
> + // Create a memory BIO
> + if (!(mem = BIO_new(BIO_s_mem())))
> + goto err;
> +
> + // Write public key in PEM form
> + if (!PEM_write_bio_PrivateKey(mem, pkey, NULL, NULL, 0, NULL, NULL))
> + goto err;
> +
> + // Extract pointer/length
> + BIO_get_mem_ptr(mem, &bptr);
> + if (!bptr || !bptr->length)
> + goto err;
> +
> + // Allocate string (+1 for NUL)
> + pem_str = av_malloc(bptr->length + 1);
> + if (!pem_str)
> + goto err;
> +
> + // Copy data & NUL‐terminate
> + memcpy(pem_str, bptr->data, bptr->length);
> + pem_str[bptr->length] = '\0';
> +
> +cleanup:
> + BIO_free(mem);
> + return pem_str;
> +
> +err:
> + // error path: free and return NULL
> + free(pem_str);
> + pem_str = NULL;
> + goto cleanup;
> +}
> +
> +/**
> + * Serialize an X509 certificate to a av_malloc’d PEM string.
> + * Caller must free the returned pointer.
> + */
> +static char *cert_to_pem_string(X509 *cert)
> +{
> + BIO *mem = BIO_new(BIO_s_mem());
> + BUF_MEM *bptr = NULL;
> + char *out = NULL;
> +
> + if (!mem) goto err;
> +
> + /* Write the PEM certificate */
> + if (!PEM_write_bio_X509(mem, cert))
> + goto err;
> +
> + BIO_get_mem_ptr(mem, &bptr);
> + if (!bptr || !bptr->length) goto err;
> +
> + out = av_malloc(bptr->length + 1);
> + if (!out) goto err;
> +
> + memcpy(out, bptr->data, bptr->length);
> + out[bptr->length] = '\0';
> +
> +cleanup:
> + BIO_free(mem);
> + return out;
> +
> +err:
> + free(out);
> + out = NULL;
> + goto cleanup;
> +}
> +
> +
> +/**
> + * Generate a SHA-256 fingerprint of an X.509 certificate.
> + *
> + * @param ctx AVFormatContext for logging (can be NULL)
> + * @param cert X509 certificate to fingerprint
> + * @return Newly allocated fingerprint string in "AA:BB:CC:…" format,
> + * or NULL on error (logs via av_log if ctx is not NULL).
> + * Caller must free() the returned string.
> + */
> +static char *generate_fingerprint(X509 *cert)
> +{
> + unsigned char md[EVP_MAX_MD_SIZE];
> + int n = 0;
> + AVBPrint fingerprint;
> + char *result = NULL;
> + int i;
> +
> + /* To prevent a crash during cleanup, always initialize it. */
> + av_bprint_init(&fingerprint, 0, AV_BPRINT_SIZE_UNLIMITED);
> +
> + if (X509_digest(cert, EVP_sha256(), md, &n) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to generate fingerprint, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto end;
> + }
> +
> + for (i = 0; i < n; i++) {
> + av_bprintf(&fingerprint, "%02X", md[i]);
> + if (i + 1 < n)
> + av_bprintf(&fingerprint, ":");
> + }
> +
> + if (!fingerprint.str || !strlen(fingerprint.str)) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Fingerprint is empty\n");
> + goto end;
> + }
> +
> + result = av_strdup(fingerprint.str);
> + if (!result) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Out of memory generating fingerprint\n");
> + }
> +
> +end:
> + av_bprint_finalize(&fingerprint, NULL);
> + return result;
> +}
> +
> +int ff_ssl_read_key_cert(char *key_url, char *cert_url, char *key_buf, size_t key_sz, char *cert_buf, size_t cert_sz, char **fingerprint)
> +{
> + int ret = 0;
> + BIO *key_b = NULL, *cert_b = NULL;
> + AVBPrint key_bp, cert_bp;
> + EVP_PKEY *pkey;
> + X509 *cert;
> + char *key_tem = NULL, *cert_tem = NULL;
> +
> + /* To prevent a crash during cleanup, always initialize it. */
> + av_bprint_init(&key_bp, 1, MAX_CERTIFICATE_SIZE);
> + av_bprint_init(&cert_bp, 1, MAX_CERTIFICATE_SIZE);
> +
> + /* Read key file. */
> + ret = ff_url_read_all(key_url, &key_bp);
> + if (ret < 0) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to open key file %s\n", key_url);
> + goto end;
> + }
> +
> + if (!(key_b = BIO_new(BIO_s_mem()))) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + BIO_write(key_b, key_bp.str, key_bp.len);
> + pkey = PEM_read_bio_PrivateKey(key_b, NULL, NULL, NULL);
> + if (!pkey) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to read private key from %s\n", key_url);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + /* Read certificate. */
> + ret = ff_url_read_all(cert_url, &cert_bp);
> + if (ret < 0) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to open cert file %s\n", cert_url);
> + goto end;
> + }
> +
> + if (!(cert_b = BIO_new(BIO_s_mem()))) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + BIO_write(cert_b, cert_bp.str, cert_bp.len);
> + cert = PEM_read_bio_X509(cert_b, NULL, NULL, NULL);
> + if (!cert) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to read certificate from %s\n", cert_url);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + key_tem = pkey_to_pem_string(pkey);
> + cert_tem = cert_to_pem_string(cert);
> +
> + snprintf(key_buf, key_sz, "%s", key_tem);
> + snprintf(cert_buf, cert_sz, "%s", cert_tem);
> +
> + /* Generate fingerprint. */
> + *fingerprint = generate_fingerprint(cert);
> + if (!*fingerprint) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to generate fingerprint from %s\n", cert_url);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> +end:
> + BIO_free(key_b);
> + av_bprint_finalize(&key_bp, NULL);
> + BIO_free(cert_b);
> + av_bprint_finalize(&cert_bp, NULL);
> + if (key_tem) av_free(key_tem);
> + if (cert_tem) av_free(cert_tem);
> + return ret;
> +}
> +
> +static int openssl_gen_private_key(EVP_PKEY **pkey, EC_KEY **eckey)
> +{
> + int ret = 0;
> +
> + /**
> + * Note that secp256r1 in openssl is called NID_X9_62_prime256v1 or prime256v1 in string,
> + * not NID_secp256k1 or secp256k1 in string.
> + *
> + * TODO: Should choose the curves in ClientHello.supported_groups, for example:
> + * Supported Group: x25519 (0x001d)
> + * Supported Group: secp256r1 (0x0017)
> + * Supported Group: secp384r1 (0x0018)
> + */
> +#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
> + EC_GROUP *ecgroup = NULL;
> + int curve = NID_X9_62_prime256v1;
> +#else
> + const char *curve = SN_X9_62_prime256v1;
> +#endif
> +
> +#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
> + *pkey = EVP_PKEY_new();
> + *eckey = EC_KEY_new();
> + ecgroup = EC_GROUP_new_by_curve_name(curve);
> + if (!ecgroup) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Create EC group by curve=%d failed, %s", curve, ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> +#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
> + /* For openssl 1.0, we must set the group parameters, so that cert is ok. */
> + EC_GROUP_set_asn1_flag(ecgroup, OPENSSL_EC_NAMED_CURVE);
> +#endif
> +
> + if (EC_KEY_set_group(*eckey, ecgroup) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Generate private key, EC_KEY_set_group failed, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (EC_KEY_generate_key(*eckey) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Generate private key, EC_KEY_generate_key failed, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (EVP_PKEY_set1_EC_KEY(*pkey, *eckey) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Generate private key, EVP_PKEY_set1_EC_KEY failed, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +#else
> + *pkey = EVP_EC_gen(curve);
> + if (!*pkey) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Generate private key, EVP_EC_gen curve=%s failed, %s\n", curve, ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +#endif
> + goto end;
> +
> +einval_end:
> + ret = AVERROR(EINVAL);
> +end:
> +#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
> + EC_GROUP_free(ecgroup);
> +#endif
> + return ret;
> +}
> +
> +static int openssl_gen_certificate(EVP_PKEY *pkey, X509 **cert, char **fingerprint)
> +{
> + int ret = 0, serial, expire_day;
> + const char *aor = "lavf";
> + X509_NAME* subject = NULL;
> +
> + *cert= X509_new();
> + if (!*cert) {
> + goto enomem_end;
> + }
> +
> + // TODO: Support non-self-signed certificate, for example, load from a file.
> + subject = X509_NAME_new();
> + if (!subject) {
> + goto enomem_end;
> + }
> +
> + serial = (int)av_get_random_seed();
> + if (ASN1_INTEGER_set(X509_get_serialNumber(*cert), serial) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set serial, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (X509_NAME_add_entry_by_txt(subject, "CN", MBSTRING_ASC, aor, strlen(aor), -1, 0) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set CN, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (X509_set_issuer_name(*cert, subject) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set issuer, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> + if (X509_set_subject_name(*cert, subject) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set subject name, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + expire_day = 365;
> + if (!X509_gmtime_adj(X509_get_notBefore(*cert), 0)) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set notBefore, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> + if (!X509_gmtime_adj(X509_get_notAfter(*cert), 60*60*24*expire_day)) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set notAfter, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (X509_set_version(*cert, 2) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set version, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (X509_set_pubkey(*cert, pkey) != 1) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set public key, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + if (!X509_sign(*cert, pkey, EVP_sha1())) {
> + av_log(NULL, AV_LOG_ERROR, "TLS: Failed to sign certificate, %s\n", ERR_error_string(ERR_get_error(), NULL));
> + goto einval_end;
> + }
> +
> + *fingerprint = generate_fingerprint(*cert);
> + if (!*fingerprint) {
> + goto enomem_end;
> + }
> +
> + goto end;
> +enomem_end:
> + ret = AVERROR(ENOMEM);
> + goto end;
> +einval_end:
> + ret = AVERROR(EINVAL);
> +end:
> + X509_NAME_free(subject);
> + //av_bprint_finalize(&fingerprint, NULL);
> + return ret;
> +}
> +
> +int ff_ssl_gen_key_cert(char *key_buf, size_t key_sz, char *cert_buf, size_t cert_sz, char **fingerprint)
> +{
> + int ret = 0;
> + EVP_PKEY *pkey = NULL;
> + EC_KEY *ec_key = NULL;
> + X509 *cert = NULL;
> + char *key_tem = NULL, *cert_tem = NULL;
> +
> + ret = openssl_gen_private_key(&pkey, &ec_key);
> + if (ret < 0) goto error;
> +
> + ret = openssl_gen_certificate(pkey, &cert, fingerprint);
> + if (ret < 0) goto error;
> +
> + key_tem = pkey_to_pem_string(pkey);
> + cert_tem = cert_to_pem_string(cert);
> +
> + snprintf(key_buf, key_sz, "%s", key_tem);
> + snprintf(cert_buf, cert_sz, "%s", cert_tem);
> +
> + if (key_tem) av_free(key_tem);
> + if (cert_tem) av_free(cert_tem);
> +error:
> + return ret;
> +}
> +
> +
> +/**
> + * Deserialize a PEM‐encoded private or public key from a NUL-terminated C string.
> + *
> + * @param pem_str The PEM text, e.g.
> + * "-----BEGIN PRIVATE KEY-----\n…\n-----END PRIVATE KEY-----\n"
> + * @param is_priv If non-zero, parse as a PRIVATE key; otherwise, parse as a PUBLIC key.
> + * @return EVP_PKEY* on success (must EVP_PKEY_free()), or NULL on error.
> + */
> +static EVP_PKEY *pkey_from_pem_string(const char *pem_str, int is_priv)
> +{
> + BIO *mem = BIO_new_mem_buf(pem_str, -1);
> + if (!mem) {
> + av_log(NULL, AV_LOG_ERROR, "BIO_new_mem_buf failed\n");
> + return NULL;
> + }
> +
> + EVP_PKEY *pkey = NULL;
> + if (is_priv) {
> + pkey = PEM_read_bio_PrivateKey(mem, NULL, NULL, NULL);
> + } else {
> + pkey = PEM_read_bio_PUBKEY(mem, NULL, NULL, NULL);
> + }
> +
> + if (!pkey)
> + av_log(NULL, AV_LOG_ERROR, "Failed to parse %s key from string\n",
> + is_priv ? "private" : "public");
> +
> + BIO_free(mem);
> + return pkey;
> +}
> +
> +/**
> + * Deserialize a PEM‐encoded certificate from a NUL-terminated C string.
> + *
> + * @param pem_str The PEM text, e.g.
> + * "-----BEGIN CERTIFICATE-----\n…\n-----END CERTIFICATE-----\n"
> + * @return X509* on success (must X509_free()), or NULL on error.
> + */
> +static X509 *cert_from_pem_string(const char *pem_str)
> +{
> + BIO *mem = BIO_new_mem_buf(pem_str, -1);
> + if (!mem) {
> + av_log(NULL, AV_LOG_ERROR, "BIO_new_mem_buf failed\n");
> + return NULL;
> + }
> +
> + X509 *cert = PEM_read_bio_X509(mem, NULL, NULL, NULL);
> + if (!cert) {
> + av_log(NULL, AV_LOG_ERROR, "Failed to parse certificate from string\n");
> + return NULL;
> + }
> +
> + BIO_free(mem);
> + return cert;
> +}
> +
> +
> typedef struct TLSContext {
> const AVClass *class;
> TLSShared tls_shared;
> @@ -38,8 +471,56 @@ typedef struct TLSContext {
> BIO_METHOD* url_bio_method;
> #endif
> int io_err;
> + char error_message[256];
> } TLSContext;
>
> +/**
> + * Retrieves the error message for the latest OpenSSL error.
> + *
> + * This function retrieves the error code from the thread's error queue, converts it
> + * to a human-readable string, and stores it in the TLSContext's error_message field.
> + * The error queue is then cleared using ERR_clear_error().
> + */
> +static const char* openssl_get_error(TLSContext *ctx)
> +{
> + int r2 = ERR_get_error();
> + if (r2) {
> + ERR_error_string_n(r2, ctx->error_message, sizeof(ctx->error_message));
> + } else
> + ctx->error_message[0] = '\0';
> +
> + ERR_clear_error();
> + return ctx->error_message;
> +}
> +
> +int ff_dtls_set_udp(URLContext *h, URLContext *udp)
> +{
> + TLSContext *c = h->priv_data;
> + c->tls_shared.udp = udp;
> + return 0;
> +}
> +
> +int ff_dtls_export_materials(URLContext *h, char *dtls_srtp_materials, size_t materials_sz)
> +{
> + int ret = 0;
> + const char* dst = "EXTRACTOR-dtls_srtp";
> + TLSContext *c = h->priv_data;
> +
> + ret = SSL_export_keying_material(c->ssl, dtls_srtp_materials, materials_sz,
> + dst, strlen(dst), NULL, 0, 0);
> + if (!ret) {
> + av_log(c, AV_LOG_ERROR, "TLS: Failed to export SRTP material, %s\n", openssl_get_error(c));
> + return -1;
> + }
> + return 0;
> +}
> +
> +int ff_dtls_state(URLContext *h)
> +{
> + TLSContext *c = h->priv_data;
> + return c->tls_shared.state;
> +}
> +
> /* OpenSSL 1.0.2 or below, then you would use SSL_library_init. If you are
> * using OpenSSL 1.1.0 or above, then the library will initialize
> * itself automatically.
> @@ -121,7 +602,7 @@ void ff_openssl_deinit(void)
> }
> #endif
>
> -static int print_tls_error(URLContext *h, int ret)
> +static int print_ssl_error(URLContext *h, int ret)
> {
> TLSContext *c = h->priv_data;
> int printed = 0, e, averr = AVERROR(EIO);
> @@ -193,7 +674,7 @@ static int url_bio_destroy(BIO *b)
> static int url_bio_bread(BIO *b, char *buf, int len)
> {
> TLSContext *c = GET_BIO_DATA(b);
> - int ret = ffurl_read(c->tls_shared.tcp, buf, len);
> + int ret = ffurl_read(c->tls_shared.is_dtls ? c->tls_shared.udp : c->tls_shared.tcp, buf, len);
> if (ret >= 0)
> return ret;
> BIO_clear_retry_flags(b);
> @@ -209,7 +690,7 @@ static int url_bio_bread(BIO *b, char *buf, int len)
> static int url_bio_bwrite(BIO *b, const char *buf, int len)
> {
> TLSContext *c = GET_BIO_DATA(b);
> - int ret = ffurl_write(c->tls_shared.tcp, buf, len);
> + int ret = ffurl_write(c->tls_shared.is_dtls ? c->tls_shared.udp : c->tls_shared.tcp, buf, len);
> if (ret >= 0)
> return ret;
> BIO_clear_retry_flags(b);
> @@ -250,11 +731,300 @@ static BIO_METHOD url_bio_method = {
> };
> #endif
>
> +static av_cold void init_bio_method(URLContext *h)
> +{
> + TLSContext *p = h->priv_data;
> + BIO *bio;
> +#if OPENSSL_VERSION_NUMBER >= 0x1010000fL
> + p->url_bio_method = BIO_meth_new(BIO_TYPE_SOURCE_SINK, "urlprotocol bio");
> + BIO_meth_set_write(p->url_bio_method, url_bio_bwrite);
> + BIO_meth_set_read(p->url_bio_method, url_bio_bread);
> + BIO_meth_set_puts(p->url_bio_method, url_bio_bputs);
> + BIO_meth_set_ctrl(p->url_bio_method, url_bio_ctrl);
> + BIO_meth_set_create(p->url_bio_method, url_bio_create);
> + BIO_meth_set_destroy(p->url_bio_method, url_bio_destroy);
> + bio = BIO_new(p->url_bio_method);
> + BIO_set_data(bio, p);
> +#else
> + bio = BIO_new(&url_bio_method);
> + bio->ptr = p;
> +#endif
> + SSL_set_bio(p->ssl, bio, bio);
> +}
> +
> +static void openssl_info_callback(const SSL *ssl, int where, int ret) {
> + const char *method = "undefined";
> + TLSContext *ctx = (TLSContext*)SSL_get_ex_data(ssl, 0);
> +
> + if (where & SSL_ST_CONNECT) {
> + method = "SSL_connect";
> + } else if (where & SSL_ST_ACCEPT)
> + method = "SSL_accept";
> +
> + if (where & SSL_CB_LOOP) {
> + av_log(ctx, AV_LOG_DEBUG, "Info method=%s state=%s(%s), where=%d, ret=%d\n",
> + method, SSL_state_string(ssl), SSL_state_string_long(ssl), where, ret);
> + } else if (where & SSL_CB_ALERT) {
> + method = (where & SSL_CB_READ) ? "read":"write";
> + av_log(ctx, AV_LOG_DEBUG, "Alert method=%s state=%s(%s), where=%d, ret=%d\n",
> + method, SSL_state_string(ssl), SSL_state_string_long(ssl), where, ret);
> + }
> +}
> +
> +/**
> + * Always return 1 to accept any certificate. This is because we allow the peer to
> + * use a temporary self-signed certificate for DTLS.
> + */
> +static int openssl_dtls_verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
> +{
> + return 1;
> +}
> +
> +static int dtls_handshake(URLContext *h)
> +{
> + int ret = 0, r0, r1;
> + TLSContext *p = h->priv_data;
> +
> + r0 = SSL_do_handshake(p->ssl);
> + r1 = SSL_get_error(p->ssl, r0);
> + if (r0 <= 0) {
> + if (r1 != SSL_ERROR_WANT_READ && r1 != SSL_ERROR_WANT_WRITE && r1 != SSL_ERROR_ZERO_RETURN) {
> + av_log(p, AV_LOG_ERROR, "TLS: Read failed, r0=%d, r1=%d %s\n", r0, r1, openssl_get_error(p));
> + ret = AVERROR(EIO);
> + goto end;
> + }
> + } else {
> + av_log(p, AV_LOG_TRACE, "TLS: Read %d bytes, r0=%d, r1=%d\n", r0, r0, r1);
> + }
> +
> + /* Check whether the DTLS is completed. */
> + if (SSL_is_init_finished(p->ssl) != 1)
> + goto end;
> +
> + p->tls_shared.state = DTLS_STATE_FINISHED;
> +end:
> + return ret;
> +}
> +
> +static av_cold int openssl_init_ca_key_cert(URLContext *h)
> +{
> + int ret;
> + TLSContext *p = h->priv_data;
> + TLSShared *c = &p->tls_shared;
> + EVP_PKEY *pkey = NULL;
> + X509 *cert = NULL;
> + /* setup ca, private key, certificate */
> + if (c->ca_file) {
> + if (!SSL_CTX_load_verify_locations(p->ctx, c->ca_file, NULL))
> + av_log(h, AV_LOG_ERROR, "SSL_CTX_load_verify_locations %s\n", openssl_get_error(p));
> + }
> +
> + if (c->cert_file) {
> + ret = SSL_CTX_use_certificate_chain_file(p->ctx, c->cert_file);
> + if (ret <= 0) {
> + av_log(h, AV_LOG_ERROR, "Unable to load cert file %s: %s\n",
> + c->cert_file, openssl_get_error(p));
> + ret = AVERROR(EIO);
> + goto fail;
> + }
> + } else if (p->tls_shared.cert_buf) {
> + cert = cert_from_pem_string(p->tls_shared.cert_buf);
> + if (SSL_CTX_use_certificate(p->ctx, cert) != 1) {
> + av_log(p, AV_LOG_ERROR, "SSL: Init SSL_CTX_use_certificate failed, %s\n", openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + return ret;
> + }
> + } else if (p->tls_shared.is_dtls){
> + av_log(p, AV_LOG_ERROR, "TLS: Init cert failed, %s\n", openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + goto fail;
> + }
> +
> + if (c->key_file) {
> + ret = SSL_CTX_use_PrivateKey_file(p->ctx, c->key_file, SSL_FILETYPE_PEM);
> + if (ret <= 0) {
> + av_log(h, AV_LOG_ERROR, "Unable to load key file %s: %s\n",
> + c->key_file, openssl_get_error(p));
> + ret = AVERROR(EIO);
> + goto fail;
> + }
> + } else if (p->tls_shared.key_buf) {
> + pkey = pkey_from_pem_string(p->tls_shared.key_buf, 1);
> + if (SSL_CTX_use_PrivateKey(p->ctx, pkey) != 1) {
> + av_log(p, AV_LOG_ERROR, "TLS: Init SSL_CTX_use_PrivateKey failed, %s\n", openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + return ret;
> + }
> + } else if (p->tls_shared.is_dtls){
> + av_log(p, AV_LOG_ERROR, "TLS: Init pkey failed, %s\n", openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + goto fail;
> + }
> + ret = 0;
> +fail:
> + return ret;
> +}
> +
> +/**
> + * Once the DTLS role has been negotiated - active for the DTLS client or passive for the
> + * DTLS server - we proceed to set up the DTLS state and initiate the handshake.
> + */
> +static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **options)
> +{
> + TLSContext *p = h->priv_data;
> + TLSShared *c = &p->tls_shared;
> + int ret = 0;
> + c->is_dtls = 1;
> + const char* ciphers = "ALL";
> + /**
> + * The profile for OpenSSL's SRTP is SRTP_AES128_CM_SHA1_80, see ssl/d1_srtp.c.
> + * The profile for FFmpeg's SRTP is SRTP_AES128_CM_HMAC_SHA1_80, see libavformat/srtp.c.
> + */
> + const char* profiles = "SRTP_AES128_CM_SHA1_80";
> + /* Refer to the test cases regarding these curves in the WebRTC code. */
> +#if OPENSSL_VERSION_NUMBER >= 0x10100000L /* OpenSSL 1.1.0 */
> + const char* curves = "X25519:P-256:P-384:P-521";
> +#elif OPENSSL_VERSION_NUMBER >= 0x10002000L /* OpenSSL 1.0.2 */
> + const char* curves = "P-256:P-384:P-521";
> +#endif
> +
> +#if OPENSSL_VERSION_NUMBER < 0x10002000L /* OpenSSL v1.0.2 */
> + p->ctx = SSL_CTX_new(DTLSv1_method());
> +#else
> + p->ctx = SSL_CTX_new(DTLS_method());
> +#endif
> + if (!p->ctx) {
> + ret = AVERROR(ENOMEM);
> + goto fail;
> + }
> +
> +#if OPENSSL_VERSION_NUMBER >= 0x10002000L /* OpenSSL 1.0.2 */
> + /* For ECDSA, we could set the curves list. */
> + if (SSL_CTX_set1_curves_list(p->ctx, curves) != 1) {
> + av_log(p, AV_LOG_ERROR, "TLS: Init SSL_CTX_set1_curves_list failed, curves=%s, %s\n",
> + curves, openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + return ret;
> + }
> +#endif
> +
> +#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
> +#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> + if (ctx->dtls_eckey)
> + SSL_CTX_set_tmp_ecdh(p->ctx, p->dtls_eckey);
> +#else
> + SSL_CTX_set_ecdh_auto(p->ctx, 1);
> +#endif
> +#endif
> +
> + /**
> + * We activate "ALL" cipher suites to align with the peer's capabilities,
> + * ensuring maximum compatibility.
> + */
> + if (SSL_CTX_set_cipher_list(p->ctx, ciphers) != 1) {
> + av_log(p, AV_LOG_ERROR, "TLS: Init SSL_CTX_set_cipher_list failed, ciphers=%s, %s\n",
> + ciphers, openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + return ret;
> + }
> + ret = openssl_init_ca_key_cert(h);
> + if (ret < 0) goto fail;
> +
> + /* Server will send Certificate Request. */
> + SSL_CTX_set_verify(p->ctx, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, openssl_dtls_verify_callback);
> + /* The depth count is "level 0:peer certificate", "level 1: CA certificate",
> + * "level 2: higher level CA certificate", and so on. */
> + SSL_CTX_set_verify_depth(p->ctx, 4);
> + /* Whether we should read as many input bytes as possible (for non-blocking reads) or not. */
> + SSL_CTX_set_read_ahead(p->ctx, 1);
> + /* Setup the SRTP context */
> + if (SSL_CTX_set_tlsext_use_srtp(p->ctx, profiles)) {
> + av_log(p, AV_LOG_ERROR, "TLS: Init SSL_CTX_set_tlsext_use_srtp failed, profiles=%s, %s\n",
> + profiles, openssl_get_error(p));
> + ret = AVERROR(EINVAL);
> + return ret;
> + }
> +
> + /* The ssl should not be created unless the ctx has been initialized. */
> + p->ssl = SSL_new(p->ctx);
> + if (!p->ssl) {
> + ret = AVERROR(ENOMEM);
> + goto fail;
> + }
> +
> + /* Setup the callback for logging. */
> + SSL_set_ex_data(p->ssl, 0, p);
> + SSL_set_info_callback(p->ssl, openssl_info_callback);
> + /**
> + * We have set the MTU to fragment the DTLS packet. It is important to note that the
> + * packet is split to ensure that each handshake packet is smaller than the MTU.
> + */
> + SSL_set_options(p->ssl, SSL_OP_NO_QUERY_MTU);
> + SSL_set_mtu(p->ssl, p->tls_shared.mtu);
> +#if OPENSSL_VERSION_NUMBER >= 0x100010b0L /* OpenSSL 1.0.1k */
> + DTLS_set_link_mtu(p->ssl, p->tls_shared.mtu);
> +#endif
> + init_bio_method(h);
> +
> + if (p->tls_shared.use_external_udp != 1) {
> + if ((ret = ff_tls_open_underlying(&p->tls_shared, h, url, options)) < 0) {
> + av_log(p, AV_LOG_ERROR, "Failed to connect %s\n", url);
> + return ret;
> + }
> + }
> +
> + /* Setup DTLS as passive, which is server role. */
> + c->listen ? SSL_set_accept_state(p->ssl) : SSL_set_connect_state(p->ssl);
> +
> + /**
> + * During initialization, we only need to call SSL_do_handshake once because SSL_read consumes
> + * the handshake message if the handshake is incomplete.
> + * To simplify maintenance, we initiate the handshake for both the DTLS server and client after
> + * sending out the ICE response in the start_active_handshake function. It's worth noting that
> + * although the DTLS server may receive the ClientHello immediately after sending out the ICE
> + * response, this shouldn't be an issue as the handshake function is called before any DTLS
> + * packets are received.
> + *
> + * The SSL_do_handshake can't be called if DTLS hasn't prepare for udp.
> + */
> + if (p->tls_shared.use_external_udp != 1) {
> + ret = dtls_handshake(h);
> + // Fatal SSL error, for example, no available suite when peer is DTLS 1.0 while we are DTLS 1.2.
> + if (ret < 0) {
> + av_log(p, AV_LOG_ERROR, "TLS: Failed to drive SSL context, ret=%d\n", ret);
> + return AVERROR(EIO);
> + }
> + }
> +
> + av_log(p, AV_LOG_VERBOSE, "TLS: Setup ok, MTU=%d, fingerprint %s\n",
> + p->tls_shared.mtu, p->tls_shared.fingerprint);
> +
> + ret = 0;
> +fail:
> + return ret;
> +}
> +
> +/**
> + * Cleanup the DTLS context.
> + */
> +static av_cold int dtls_close(URLContext *h)
> +{
> + TLSContext *ctx = h->priv_data;
> + SSL_free(ctx->ssl);
> + SSL_CTX_free(ctx->ctx);
> + av_freep(&ctx->tls_shared.fingerprint);
> + av_freep(&ctx->tls_shared.cert_buf);
> + av_freep(&ctx->tls_shared.key_buf);
> +#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
> + EC_KEY_free(ctx->dtls_eckey);
> +#endif
> + return 0;
> +}
> +
> static int tls_open(URLContext *h, const char *uri, int flags, AVDictionary **options)
> {
> TLSContext *p = h->priv_data;
> TLSShared *c = &p->tls_shared;
> - BIO *bio;
> int ret;
>
> #if OPENSSL_VERSION_NUMBER < 0x10100000L
> @@ -271,52 +1041,26 @@ static int tls_open(URLContext *h, const char *uri, int flags, AVDictionary **op
> // support for the old protocols immediately after creating the context.
> p->ctx = SSL_CTX_new(c->listen ? SSLv23_server_method() : SSLv23_client_method());
> if (!p->ctx) {
> - av_log(h, AV_LOG_ERROR, "%s\n", ERR_error_string(ERR_get_error(), NULL));
> + av_log(h, AV_LOG_ERROR, "%s\n", openssl_get_error(p));
> ret = AVERROR(EIO);
> goto fail;
> }
> SSL_CTX_set_options(p->ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
> - if (c->ca_file) {
> - if (!SSL_CTX_load_verify_locations(p->ctx, c->ca_file, NULL))
> - av_log(h, AV_LOG_ERROR, "SSL_CTX_load_verify_locations %s\n", ERR_error_string(ERR_get_error(), NULL));
> - }
> - if (c->cert_file && !SSL_CTX_use_certificate_chain_file(p->ctx, c->cert_file)) {
> - av_log(h, AV_LOG_ERROR, "Unable to load cert file %s: %s\n",
> - c->cert_file, ERR_error_string(ERR_get_error(), NULL));
> - ret = AVERROR(EIO);
> - goto fail;
> - }
> - if (c->key_file && !SSL_CTX_use_PrivateKey_file(p->ctx, c->key_file, SSL_FILETYPE_PEM)) {
> - av_log(h, AV_LOG_ERROR, "Unable to load key file %s: %s\n",
> - c->key_file, ERR_error_string(ERR_get_error(), NULL));
> - ret = AVERROR(EIO);
> - goto fail;
> - }
> + ret = openssl_init_ca_key_cert(h);
> + if (ret < 0) goto fail;
> // Note, this doesn't check that the peer certificate actually matches
> // the requested hostname.
> if (c->verify)
> SSL_CTX_set_verify(p->ctx, SSL_VERIFY_PEER|SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);
> p->ssl = SSL_new(p->ctx);
> if (!p->ssl) {
> - av_log(h, AV_LOG_ERROR, "%s\n", ERR_error_string(ERR_get_error(), NULL));
> + av_log(h, AV_LOG_ERROR, "%s\n", openssl_get_error(p));
> ret = AVERROR(EIO);
> goto fail;
> }
> -#if OPENSSL_VERSION_NUMBER >= 0x1010000fL
> - p->url_bio_method = BIO_meth_new(BIO_TYPE_SOURCE_SINK, "urlprotocol bio");
> - BIO_meth_set_write(p->url_bio_method, url_bio_bwrite);
> - BIO_meth_set_read(p->url_bio_method, url_bio_bread);
> - BIO_meth_set_puts(p->url_bio_method, url_bio_bputs);
> - BIO_meth_set_ctrl(p->url_bio_method, url_bio_ctrl);
> - BIO_meth_set_create(p->url_bio_method, url_bio_create);
> - BIO_meth_set_destroy(p->url_bio_method, url_bio_destroy);
> - bio = BIO_new(p->url_bio_method);
> - BIO_set_data(bio, p);
> -#else
> - bio = BIO_new(&url_bio_method);
> - bio->ptr = p;
> -#endif
> - SSL_set_bio(p->ssl, bio, bio);
> + SSL_set_ex_data(p->ssl, 0, p);
> + SSL_CTX_set_info_callback(p->ctx, openssl_info_callback);
> + init_bio_method(h);
> if (!c->listen && !c->numerichost)
> SSL_set_tlsext_host_name(p->ssl, c->host);
> ret = c->listen ? SSL_accept(p->ssl) : SSL_connect(p->ssl);
> @@ -325,7 +1069,7 @@ static int tls_open(URLContext *h, const char *uri, int flags, AVDictionary **op
> ret = AVERROR(EIO);
> goto fail;
> } else if (ret < 0) {
> - ret = print_tls_error(h, ret);
> + ret = print_ssl_error(h, ret);
> goto fail;
> }
>
> @@ -338,31 +1082,35 @@ fail:
> static int tls_read(URLContext *h, uint8_t *buf, int size)
> {
> TLSContext *c = h->priv_data;
> + URLContext *uc = c->tls_shared.is_dtls ? c->tls_shared.udp
> + : c->tls_shared.tcp;
> int ret;
> // Set or clear the AVIO_FLAG_NONBLOCK on c->tls_shared.tcp
> - c->tls_shared.tcp->flags &= ~AVIO_FLAG_NONBLOCK;
> - c->tls_shared.tcp->flags |= h->flags & AVIO_FLAG_NONBLOCK;
> + uc->flags &= ~AVIO_FLAG_NONBLOCK;
> + uc->flags |= h->flags & AVIO_FLAG_NONBLOCK;
> ret = SSL_read(c->ssl, buf, size);
> if (ret > 0)
> return ret;
> if (ret == 0)
> return AVERROR_EOF;
> - return print_tls_error(h, ret);
> + return print_ssl_error(h, ret);
> }
>
> static int tls_write(URLContext *h, const uint8_t *buf, int size)
> {
> TLSContext *c = h->priv_data;
> + URLContext *uc = c->tls_shared.is_dtls ? c->tls_shared.udp
> + : c->tls_shared.tcp;
> int ret;
> // Set or clear the AVIO_FLAG_NONBLOCK on c->tls_shared.tcp
> - c->tls_shared.tcp->flags &= ~AVIO_FLAG_NONBLOCK;
> - c->tls_shared.tcp->flags |= h->flags & AVIO_FLAG_NONBLOCK;
> + uc->flags &= ~AVIO_FLAG_NONBLOCK;
> + uc->flags |= h->flags & AVIO_FLAG_NONBLOCK;
> ret = SSL_write(c->ssl, buf, size);
> if (ret > 0)
> return ret;
> if (ret == 0)
> return AVERROR_EOF;
> - return print_tls_error(h, ret);
> + return print_ssl_error(h, ret);
> }
>
> static int tls_get_file_handle(URLContext *h)
> @@ -401,3 +1149,22 @@ const URLProtocol ff_tls_protocol = {
> .flags = URL_PROTOCOL_FLAG_NETWORK,
> .priv_data_class = &tls_class,
> };
> +
> +static const AVClass dtls_class = {
> + .class_name = "dtls",
> + .item_name = av_default_item_name,
> + .option = options,
> + .version = LIBAVUTIL_VERSION_INT,
> +};
> +
> +const URLProtocol ff_dtls_protocol = {
> + .name = "dtls",
> + .url_open2 = dtls_start,
> + .url_handshake = dtls_handshake,
> + .url_close = dtls_close,
> + .url_read = tls_read,
> + .url_write = tls_write,
> + .priv_data_size = sizeof(TLSContext),
> + .flags = URL_PROTOCOL_FLAG_NETWORK,
> + .priv_data_class = &dtls_class,
> +};
> diff --git a/libavformat/whip.c b/libavformat/whip.c
> new file mode 100644
> index 0000000000..0671e23635
> --- /dev/null
> +++ b/libavformat/whip.c
> @@ -0,0 +1,1917 @@
> +/*
> + * WebRTC-HTTP ingestion protocol (WHIP) muxer
> + * Copyright (c) 2023 The FFmpeg Project
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include "libavcodec/avcodec.h"
> +#include "libavcodec/codec_desc.h"
> +#include "libavcodec/h264.h"
> +#include "libavcodec/startcode.h"
> +#include "libavutil/base64.h"
> +#include "libavutil/bprint.h"
> +#include "libavutil/crc.h"
> +#include "libavutil/hmac.h"
> +#include "libavutil/intreadwrite.h"
> +#include "libavutil/lfg.h"
> +#include "libavutil/opt.h"
> +#include "libavutil/mem.h"
> +#include "libavutil/random_seed.h"
> +#include "libavutil/time.h"
> +#include "avc.h"
> +#include "nal.h"
> +#include "avio_internal.h"
> +#include "http.h"
> +#include "internal.h"
> +#include "mux.h"
> +#include "network.h"
> +#include "srtp.h"
> +#include "tls.h"
> +
> +/**
> + * Maximum size limit of a Session Description Protocol (SDP),
> + * be it an offer or answer.
> + */
> +#define MAX_SDP_SIZE 8192
> +
> +/**
> + * The size of the Secure Real-time Transport Protocol (SRTP) master key material
> + * that is exported by Secure Sockets Layer (SSL) after a successful Datagram
> + * Transport Layer Security (DTLS) handshake. This material consists of a key
> + * of 16 bytes and a salt of 14 bytes.
> + */
> +#define DTLS_SRTP_KEY_LEN 16
> +#define DTLS_SRTP_SALT_LEN 14
> +
> +/**
> + * The maximum size of the Secure Real-time Transport Protocol (SRTP) HMAC checksum
> + * and padding that is appended to the end of the packet. To calculate the maximum
> + * size of the User Datagram Protocol (UDP) packet that can be sent out, subtract
> + * this size from the `pkt_size`.
> + */
> +#define DTLS_SRTP_CHECKSUM_LEN 16
> +
> +/**
> + * When sending ICE or DTLS messages, responses are received via UDP. However, the peer
> + * may not be ready and return EAGAIN, in which case we should wait for a short duration
> + * and retry reading.
> + * For instance, if we try to read from UDP and get EAGAIN, we sleep for 5ms and retry.
> + * This macro is used to limit the total duration in milliseconds (e.g., 50ms), so we
> + * will try at most 5 times.
> + * Keep in mind that this macro should have a minimum duration of 5 ms.
> + */
> +#define ICE_DTLS_READ_INTERVAL 50
> +
> +/* The magic cookie for Session Traversal Utilities for NAT (STUN) messages. */
> +#define STUN_MAGIC_COOKIE 0x2112A442
> +
> +/**
> + * The DTLS content type.
> + * See https://tools.ietf.org/html/rfc2246#section-6.2.1
> + * change_cipher_spec(20), alert(21), handshake(22), application_data(23)
> + */
> +#define DTLS_CONTENT_TYPE_CHANGE_CIPHER_SPEC 20
> +
> +/**
> + * The DTLS record layer header has a total size of 13 bytes, consisting of
> + * ContentType (1 byte), ProtocolVersion (2 bytes), Epoch (2 bytes),
> + * SequenceNumber (6 bytes), and Length (2 bytes).
> + * See https://datatracker.ietf.org/doc/html/rfc9147#section-4
> + */
> +#define DTLS_RECORD_LAYER_HEADER_LEN 13
> +
> +/**
> + * The DTLS version number, which is 0xfeff for DTLS 1.0, or 0xfefd for DTLS 1.2.
> + * See https://datatracker.ietf.org/doc/html/rfc9147#name-the-dtls-record-layer
> + */
> +#define DTLS_VERSION_10 0xfeff
> +#define DTLS_VERSION_12 0xfefd
> +
> +/**
> + * Maximum size of the buffer for sending and receiving UDP packets.
> + * Please note that this size does not limit the size of the UDP packet that can be sent.
> + * To set the limit for packet size, modify the `pkt_size` parameter.
> + * For instance, it is possible to set the UDP buffer to 4096 to send or receive packets,
> + * but please keep in mind that the `pkt_size` option limits the packet size to 1400.
> + */
> +#define MAX_UDP_BUFFER_SIZE 4096
> +
> +/* Referring to Chrome's definition of RTP payload types. */
> +#define WHIP_RTP_PAYLOAD_TYPE_H264 106
> +#define WHIP_RTP_PAYLOAD_TYPE_OPUS 111
> +
> +/**
> + * The STUN message header, which is 20 bytes long, comprises the
> + * STUNMessageType (1B), MessageLength (2B), MagicCookie (4B),
> + * and TransactionID (12B).
> + * See https://datatracker.ietf.org/doc/html/rfc5389#section-6
> + */
> +#define ICE_STUN_HEADER_SIZE 20
> +
> +/**
> + * The RTP header is 12 bytes long, comprising the Version(1B), PT(1B),
> + * SequenceNumber(2B), Timestamp(4B), and SSRC(4B).
> + * See https://www.rfc-editor.org/rfc/rfc3550#section-5.1
> + */
> +#define WHIP_RTP_HEADER_SIZE 12
> +
> +/**
> + * For RTCP, PT is [128, 223] (or without marker [0, 95]). Literally, RTCP starts
> + * from 64 not 0, so PT is [192, 223] (or without marker [64, 95]), see "RTCP Control
> + * Packet Types (PT)" at
> + * https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-4
> + *
> + * For RTP, the PT is [96, 127], or [224, 255] with marker. See "RTP Payload Types (PT)
> + * for standard audio and video encodings" at
> + * https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-1
> + */
> +#define WHIP_RTCP_PT_START 192
> +#define WHIP_RTCP_PT_END 223
> +
> +/**
> + * In the case of ICE-LITE, these fields are not used; instead, they are defined
> + * as constant values.
> + */
> +#define WHIP_SDP_SESSION_ID "4489045141692799359"
> +#define WHIP_SDP_CREATOR_IP "127.0.0.1"
> +
> +/* Calculate the elapsed time from starttime to endtime in milliseconds. */
> +#define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000)
> +
> +/* STUN Attribute, comprehension-required range (0x0000-0x7FFF) */
> +enum STUNAttr {
> + STUN_ATTR_USERNAME = 0x0006, /// shared secret response/bind request
> + STUN_ATTR_USE_CANDIDATE = 0x0025, /// bind request
> + STUN_ATTR_MESSAGE_INTEGRITY = 0x0008, /// bind request/response
> + STUN_ATTR_FINGERPRINT = 0x8028, /// rfc5389
> +};
> +
> +enum WHIPState {
> + WHIP_STATE_NONE,
> +
> + /* The initial state. */
> + WHIP_STATE_INIT,
> + /* The muxer has sent the offer to the peer. */
> + WHIP_STATE_OFFER,
> + /* The muxer has received the answer from the peer. */
> + WHIP_STATE_ANSWER,
> + /**
> + * After parsing the answer received from the peer, the muxer negotiates the abilities
> + * in the offer that it generated.
> + */
> + WHIP_STATE_NEGOTIATED,
> + /* The muxer has connected to the peer via UDP. */
> + WHIP_STATE_UDP_CONNECTED,
> + /* The muxer has sent the ICE request to the peer. */
> + WHIP_STATE_ICE_CONNECTING,
> + /* The muxer has received the ICE response from the peer. */
> + WHIP_STATE_ICE_CONNECTED,
> + /* The muxer starts attempting the DTLS handshake. */
> + WHIP_STATE_DTLS_CONNECTING,
> + /* The muxer has finished the DTLS handshake with the peer. */
> + WHIP_STATE_DTLS_FINISHED,
> + /* The muxer has finished the SRTP setup. */
> + WHIP_STATE_SRTP_FINISHED,
> + /* The muxer is ready to send/receive media frames. */
> + WHIP_STATE_READY,
> + /* The muxer is failed. */
> + WHIP_STATE_FAILED,
> +};
> +
> +typedef struct WHIPContext {
> + AVClass *av_class;
> +
> + /* The state of the RTC connection. */
> + enum WHIPState state;
> + /* The callback return value for DTLS. */
> + int dtls_ret;
> + int dtls_closed;
> +
> + /* Parameters for the input audio and video codecs. */
> + AVCodecParameters *audio_par;
> + AVCodecParameters *video_par;
> +
> + /**
> + * The h264_mp4toannexb Bitstream Filter (BSF) bypasses the AnnexB packet;
> + * therefore, it is essential to insert the SPS and PPS before each IDR frame
> + * in such cases.
> + */
> + int h264_annexb_insert_sps_pps;
> +
> + /* The random number generator. */
> + AVLFG rnd;
> +
> + /* The ICE username and pwd fragment generated by the muxer. */
> + char ice_ufrag_local[9];
> + char ice_pwd_local[33];
> + /* The SSRC of the audio and video stream, generated by the muxer. */
> + uint32_t audio_ssrc;
> + uint32_t video_ssrc;
> + /* The PT(Payload Type) of stream, generated by the muxer. */
> + uint8_t audio_payload_type;
> + uint8_t video_payload_type;
> + /**
> + * This is the SDP offer generated by the muxer based on the codec parameters,
> + * DTLS, and ICE information.
> + */
> + char *sdp_offer;
> +
> + /* The ICE username and pwd from remote server. */
> + char *ice_ufrag_remote;
> + char *ice_pwd_remote;
> + /**
> + * This represents the ICE candidate protocol, priority, host and port.
> + * Currently, we only support one candidate and choose the first UDP candidate.
> + * However, we plan to support multiple candidates in the future.
> + */
> + char *ice_protocol;
> + char *ice_host;
> + int ice_port;
> +
> + /* The SDP answer received from the WebRTC server. */
> + char *sdp_answer;
> + /* The resource URL returned in the Location header of WHIP HTTP response. */
> + char *whip_resource_url;
> +
> + /* These variables represent timestamps used for calculating and tracking the cost. */
> + int64_t whip_starttime;
> + int64_t whip_init_time;
> + int64_t whip_offer_time;
> + int64_t whip_answer_time;
> + int64_t whip_udp_time;
> + int64_t whip_ice_time;
> + int64_t whip_dtls_time;
> + int64_t whip_srtp_time;
> +
> + /* The certificate and private key content used for DTLS hanshake */
> + char cert_buf[MAX_CERTIFICATE_SIZE];
> + char key_buf[MAX_CERTIFICATE_SIZE];
> + /* The fingerprint of certificate, used in SDP offer. */
> + char *dtls_fingerprint;
> + /**
> + * This represents the material used to build the SRTP master key. It is
> + * generated by DTLS and has the following layout:
> + * 16B 16B 14B 14B
> + * client_key | server_key | client_salt | server_salt
> + */
> + uint8_t dtls_srtp_materials[(DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN) * 2];
> +
> + char ssl_error_message[256];
> +
> + /* TODO: Use AVIOContext instead of URLContext */
> + URLContext *dtls_uc;
> +
> + /* The SRTP send context, to encrypt outgoing packets. */
> + SRTPContext srtp_audio_send;
> + SRTPContext srtp_video_send;
> + SRTPContext srtp_rtcp_send;
> + /* The SRTP receive context, to decrypt incoming packets. */
> + SRTPContext srtp_recv;
> +
> + /* The UDP transport is used for delivering ICE, DTLS and SRTP packets. */
> + URLContext *udp;
> + /* The buffer for UDP transmission. */
> + char buf[MAX_UDP_BUFFER_SIZE];
> +
> + /* The timeout in milliseconds for ICE and DTLS handshake. */
> + int handshake_timeout;
> + /**
> + * The size of RTP packet, should generally be set to MTU.
> + * Note that pion requires a smaller value, for example, 1200.
> + */
> + int pkt_size;
> + /**
> + * The optional Bearer token for WHIP Authorization.
> + * See https://www.ietf.org/archive/id/draft-ietf-wish-whip-08.html#name-authentication-and-authoriz
> + */
> + char* authorization;
> + /* The certificate and private key used for DTLS handshake. */
> + char* cert_file;
> + char* key_file;
> +} WHIPContext;
> +
> +/**
> + * Whether the packet is a DTLS packet.
> + */
> +static int is_dtls_packet(uint8_t *b, int size) {
> + uint16_t version = AV_RB16(&b[1]);
> + return size > DTLS_RECORD_LAYER_HEADER_LEN &&
> + b[0] >= DTLS_CONTENT_TYPE_CHANGE_CIPHER_SPEC &&
> + (version == DTLS_VERSION_10 || version == DTLS_VERSION_12);
> +}
> +
> +
> +/**
> + * Get or Generate a self-signed certificate and private key for DTLS,
> + * fingerprint for SDP
> + */
> +static av_cold int certificate_key_init(AVFormatContext *s)
> +{
> + int ret = 0;
> + WHIPContext *whip = s->priv_data;
> +
> + if (whip->cert_file && whip->key_file) {
> + /* Read the private key and certificate from the file. */
> + if ((ret = ff_ssl_read_key_cert(whip->key_file, whip->cert_file,
> + whip->key_buf, sizeof(whip->key_buf),
> + whip->cert_buf, sizeof(whip->cert_buf),
> + &whip->dtls_fingerprint)) < 0) {
> + av_log(s, AV_LOG_ERROR, "DTLS: Failed to read DTLS certificate from cert=%s, key=%s\n",
> + whip->cert_file, whip->key_file);
> + return ret;
> + }
> + } else {
> + /* Generate a private key to ctx->dtls_pkey and self-signed certificate. */
> + if ((ret = ff_ssl_gen_key_cert(whip->key_buf, sizeof(whip->key_buf),
> + whip->cert_buf, sizeof(whip->cert_buf),
> + &whip->dtls_fingerprint)) < 0) {
> + av_log(s, AV_LOG_ERROR, "DTLS: Failed to generate DTLS private key and certificate\n");
> + return ret;
> + }
> + }
> +
> + return ret;
> +}
> +
> +/**
> + * When DTLS state change.
> + */
> +static int dtls_context_on_state(AVFormatContext *s, const char* type, const char* desc)
> +{
> + int ret = 0;
> + WHIPContext *whip = s->priv_data;
> + int state = ff_dtls_state(whip->dtls_uc);
> +
> + if (state == DTLS_STATE_CLOSED) {
> + whip->dtls_closed = 1;
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: DTLS session closed, type=%s, desc=%s, elapsed=%dms\n",
> + type ? type : "", desc ? desc : "", ELAPSED(whip->whip_starttime, av_gettime()));
> + goto error;
> + }
> +
> + if (state == DTLS_STATE_FAILED) {
> + whip->state = WHIP_STATE_FAILED;
> + av_log(whip, AV_LOG_ERROR, "WHIP: DTLS session failed, type=%s, desc=%s\n",
> + type ? type : "", desc ? desc : "");
> + whip->dtls_ret = AVERROR(EIO);
> + goto error;
> + }
> +
> + if (state == DTLS_STATE_FINISHED && whip->state < WHIP_STATE_DTLS_FINISHED) {
> + whip->state = WHIP_STATE_DTLS_FINISHED;
> + whip->whip_dtls_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: DTLS handshake is done, elapsed=%dms\n",
> + ELAPSED(whip->whip_starttime, av_gettime()));
> + return ret;
> + }
> +error:
> + return -1;
> +}
> +
> +static av_cold int dtls_initialize(AVFormatContext *s)
> +{
> + WHIPContext *whip = s->priv_data;
> + /* reuse the udp created by whip */
> + ff_dtls_set_udp(whip->dtls_uc, whip->udp);
> + return 0;
> +}
> +
> +/**
> + * Initialize and check the options for the WebRTC muxer.
> + */
> +static av_cold int initialize(AVFormatContext *s)
> +{
> + int ret, ideal_pkt_size = 532;
> + WHIPContext *whip = s->priv_data;
> + uint32_t seed;
> +
> + whip->whip_starttime = av_gettime();
> +
> + ret = certificate_key_init(s);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to init certificate and key\n");
> + return ret;
> + }
> +
> + /* Initialize the random number generator. */
> + seed = av_get_random_seed();
> + av_lfg_init(&whip->rnd, seed);
> +
> + if (whip->pkt_size < ideal_pkt_size)
> + av_log(whip, AV_LOG_WARNING, "WHIP: pkt_size=%d(<%d) is too small, may cause packet loss\n",
> + whip->pkt_size, ideal_pkt_size);
> +
> + if (whip->state < WHIP_STATE_INIT)
> + whip->state = WHIP_STATE_INIT;
> + whip->whip_init_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: Init state=%d, handshake_timeout=%dms, pkt_size=%d, seed=%d, elapsed=%dms\n",
> + whip->state, whip->handshake_timeout, whip->pkt_size, seed, ELAPSED(whip->whip_starttime, av_gettime()));
> +
> + return 0;
> +}
> +
> +/**
> + * When duplicating a stream, the demuxer has already set the extradata, profile, and
> + * level of the par. Keep in mind that this function will not be invoked since the
> + * profile and level are set.
> + *
> + * When utilizing an encoder, such as libx264, to encode a stream, the extradata in
> + * par->extradata contains the SPS, which includes profile and level information.
> + * However, the profile and level of par remain unspecified. Therefore, it is necessary
> + * to extract the profile and level data from the extradata and assign it to the par's
> + * profile and level. Keep in mind that AVFMT_GLOBALHEADER must be enabled; otherwise,
> + * the extradata will remain empty.
> + */
> +static int parse_profile_level(AVFormatContext *s, AVCodecParameters *par)
> +{
> + int ret = 0;
> + const uint8_t *r = par->extradata, *r1, *end = par->extradata + par->extradata_size;
> + H264SPS seq, *const sps = &seq;
> + uint32_t state;
> + WHIPContext *whip = s->priv_data;
> +
> + if (par->codec_id != AV_CODEC_ID_H264)
> + return ret;
> +
> + if (par->profile != AV_PROFILE_UNKNOWN && par->level != AV_LEVEL_UNKNOWN)
> + return ret;
> +
> + if (!par->extradata || par->extradata_size <= 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Unable to parse profile from empty extradata=%p, size=%d\n",
> + par->extradata, par->extradata_size);
> + return AVERROR(EINVAL);
> + }
> +
> + while (1) {
> + r = avpriv_find_start_code(r, end, &state);
> + if (r >= end)
> + break;
> +
> + r1 = ff_nal_find_startcode(r, end);
> + if ((state & 0x1f) == H264_NAL_SPS) {
> + ret = ff_avc_decode_sps(sps, r, r1 - r);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to decode SPS, state=%x, size=%d\n",
> + state, (int)(r1 - r));
> + return ret;
> + }
> +
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: Parse profile=%d, level=%d from SPS\n",
> + sps->profile_idc, sps->level_idc);
> + par->profile = sps->profile_idc;
> + par->level = sps->level_idc;
> + }
> +
> + r = r1;
> + }
> +
> + return ret;
> +}
> +
> +/**
> + * Parses video SPS/PPS from the extradata of codecpar and checks the codec.
> + * Currently only supports video(h264) and audio(opus). Note that only baseline
> + * and constrained baseline profiles of h264 are supported.
> + *
> + * If the profile is less than 0, the function considers the profile as baseline.
> + * It may need to parse the profile from SPS/PPS. This situation occurs when ingesting
> + * desktop and transcoding.
> + *
> + * @param s Pointer to the AVFormatContext
> + * @returns Returns 0 if successful or AVERROR_xxx in case of an error.
> + *
> + * TODO: FIXME: There is an issue with the timestamp of OPUS audio, especially when
> + * the input is an MP4 file. The timestamp deviates from the expected value of 960,
> + * causing Chrome to play the audio stream with noise. This problem can be replicated
> + * by transcoding a specific file into MP4 format and publishing it using the WHIP
> + * muxer. However, when directly transcoding and publishing through the WHIP muxer,
> + * the issue is not present, and the audio timestamp remains consistent. The root
> + * cause is still unknown, and this comment has been added to address this issue
> + * in the future. Further research is needed to resolve the problem.
> + */
> +static int parse_codec(AVFormatContext *s)
> +{
> + int i, ret = 0;
> + WHIPContext *whip = s->priv_data;
> +
> + for (i = 0; i < s->nb_streams; i++) {
> + AVCodecParameters *par = s->streams[i]->codecpar;
> + const AVCodecDescriptor *desc = avcodec_descriptor_get(par->codec_id);
> + switch (par->codec_type) {
> + case AVMEDIA_TYPE_VIDEO:
> + if (whip->video_par) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Only one video stream is supported by RTC\n");
> + return AVERROR(EINVAL);
> + }
> + whip->video_par = par;
> +
> + if (par->codec_id != AV_CODEC_ID_H264) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported video codec %s by RTC, choose h264\n",
> + desc ? desc->name : "unknown");
> + return AVERROR_PATCHWELCOME;
> + }
> +
> + if (par->video_delay > 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported B frames by RTC\n");
> + return AVERROR_PATCHWELCOME;
> + }
> +
> + if ((ret = parse_profile_level(s, par)) < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to parse SPS/PPS from extradata\n");
> + return AVERROR(EINVAL);
> + }
> +
> + if (par->profile == AV_PROFILE_UNKNOWN) {
> + av_log(whip, AV_LOG_WARNING, "WHIP: No profile found in extradata, consider baseline\n");
> + return AVERROR(EINVAL);
> + }
> + if (par->level == AV_LEVEL_UNKNOWN) {
> + av_log(whip, AV_LOG_WARNING, "WHIP: No level found in extradata, consider 3.1\n");
> + return AVERROR(EINVAL);
> + }
> + break;
> + case AVMEDIA_TYPE_AUDIO:
> + if (whip->audio_par) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Only one audio stream is supported by RTC\n");
> + return AVERROR(EINVAL);
> + }
> + whip->audio_par = par;
> +
> + if (par->codec_id != AV_CODEC_ID_OPUS) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported audio codec %s by RTC, choose opus\n",
> + desc ? desc->name : "unknown");
> + return AVERROR_PATCHWELCOME;
> + }
> +
> + if (par->ch_layout.nb_channels != 2) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported audio channels %d by RTC, choose stereo\n",
> + par->ch_layout.nb_channels);
> + return AVERROR_PATCHWELCOME;
> + }
> +
> + if (par->sample_rate != 48000) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported audio sample rate %d by RTC, choose 48000\n", par->sample_rate);
> + return AVERROR_PATCHWELCOME;
> + }
> + break;
> + default:
> + av_log(whip, AV_LOG_ERROR, "WHIP: Codec type '%s' for stream %d is not supported by RTC\n",
> + av_get_media_type_string(par->codec_type), i);
> + return AVERROR_PATCHWELCOME;
> + }
> + }
> +
> + return ret;
> +}
> +
> +/**
> + * Generate SDP offer according to the codec parameters, DTLS and ICE information.
> + *
> + * Note that we don't use av_sdp_create to generate SDP offer because it doesn't
> + * support DTLS and ICE information.
> + *
> + * @return 0 if OK, AVERROR_xxx on error
> + */
> +static int generate_sdp_offer(AVFormatContext *s)
> +{
> + int ret = 0, profile, level, profile_iop;
> + const char *acodec_name = NULL, *vcodec_name = NULL;
> + AVBPrint bp;
> + WHIPContext *whip = s->priv_data;
> +
> + /* To prevent a crash during cleanup, always initialize it. */
> + av_bprint_init(&bp, 1, MAX_SDP_SIZE);
> +
> + if (whip->sdp_offer) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: SDP offer is already set\n");
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + snprintf(whip->ice_ufrag_local, sizeof(whip->ice_ufrag_local), "%08x",
> + av_lfg_get(&whip->rnd));
> + snprintf(whip->ice_pwd_local, sizeof(whip->ice_pwd_local), "%08x%08x%08x%08x",
> + av_lfg_get(&whip->rnd), av_lfg_get(&whip->rnd), av_lfg_get(&whip->rnd),
> + av_lfg_get(&whip->rnd));
> +
> + whip->audio_ssrc = av_lfg_get(&whip->rnd);
> + whip->video_ssrc = av_lfg_get(&whip->rnd);
> +
> + whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS;
> + whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264;
> +
> + av_bprintf(&bp, ""
> + "v=0\r\n"
> + "o=FFmpeg %s 2 IN IP4 %s\r\n"
> + "s=FFmpegPublishSession\r\n"
> + "t=0 0\r\n"
> + "a=group:BUNDLE 0 1\r\n"
> + "a=extmap-allow-mixed\r\n"
> + "a=msid-semantic: WMS\r\n",
> + WHIP_SDP_SESSION_ID,
> + WHIP_SDP_CREATOR_IP);
> +
> + if (whip->audio_par) {
> + if (whip->audio_par->codec_id == AV_CODEC_ID_OPUS)
> + acodec_name = "opus";
> +
> + av_bprintf(&bp, ""
> + "m=audio 9 UDP/TLS/RTP/SAVPF %u\r\n"
> + "c=IN IP4 0.0.0.0\r\n"
> + "a=ice-ufrag:%s\r\n"
> + "a=ice-pwd:%s\r\n"
> + "a=fingerprint:sha-256 %s\r\n"
> + "a=setup:passive\r\n"
> + "a=mid:0\r\n"
> + "a=sendonly\r\n"
> + "a=msid:FFmpeg audio\r\n"
> + "a=rtcp-mux\r\n"
> + "a=rtpmap:%u %s/%d/%d\r\n"
> + "a=ssrc:%u cname:FFmpeg\r\n"
> + "a=ssrc:%u msid:FFmpeg audio\r\n",
> + whip->audio_payload_type,
> + whip->ice_ufrag_local,
> + whip->ice_pwd_local,
> + whip->dtls_fingerprint,
> + whip->audio_payload_type,
> + acodec_name,
> + whip->audio_par->sample_rate,
> + whip->audio_par->ch_layout.nb_channels,
> + whip->audio_ssrc,
> + whip->audio_ssrc);
> + }
> +
> + if (whip->video_par) {
> + profile_iop = profile = whip->video_par->profile;
> + level = whip->video_par->level;
> + if (whip->video_par->codec_id == AV_CODEC_ID_H264) {
> + vcodec_name = "H264";
> + profile_iop &= AV_PROFILE_H264_CONSTRAINED;
> + profile &= (~AV_PROFILE_H264_CONSTRAINED);
> + }
> +
> + av_bprintf(&bp, ""
> + "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n"
> + "c=IN IP4 0.0.0.0\r\n"
> + "a=ice-ufrag:%s\r\n"
> + "a=ice-pwd:%s\r\n"
> + "a=fingerprint:sha-256 %s\r\n"
> + "a=setup:passive\r\n"
> + "a=mid:1\r\n"
> + "a=sendonly\r\n"
> + "a=msid:FFmpeg video\r\n"
> + "a=rtcp-mux\r\n"
> + "a=rtcp-rsize\r\n"
> + "a=rtpmap:%u %s/90000\r\n"
> + "a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n"
> + "a=ssrc:%u cname:FFmpeg\r\n"
> + "a=ssrc:%u msid:FFmpeg video\r\n",
> + whip->video_payload_type,
> + whip->ice_ufrag_local,
> + whip->ice_pwd_local,
> + whip->dtls_fingerprint,
> + whip->video_payload_type,
> + vcodec_name,
> + whip->video_payload_type,
> + profile,
> + profile_iop,
> + level,
> + whip->video_ssrc,
> + whip->video_ssrc);
> + }
> +
> + if (!av_bprint_is_complete(&bp)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Offer exceed max %d, %s\n", MAX_SDP_SIZE, bp.str);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + whip->sdp_offer = av_strdup(bp.str);
> + if (!whip->sdp_offer) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + if (whip->state < WHIP_STATE_OFFER)
> + whip->state = WHIP_STATE_OFFER;
> + whip->whip_offer_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: Generated state=%d, offer: %s\n", whip->state, whip->sdp_offer);
> +
> +end:
> + av_bprint_finalize(&bp, NULL);
> + return ret;
> +}
> +
> +/**
> + * Exchange SDP offer with WebRTC peer to get the answer.
> + *
> + * @return 0 if OK, AVERROR_xxx on error
> + */
> +static int exchange_sdp(AVFormatContext *s)
> +{
> + int ret;
> + char buf[MAX_URL_SIZE];
> + AVBPrint bp;
> + WHIPContext *whip = s->priv_data;
> + /* The URL context is an HTTP transport layer for the WHIP protocol. */
> + URLContext *whip_uc = NULL;
> + AVDictionary *opts = NULL;
> + char *hex_data = NULL;
> +
> + /* To prevent a crash during cleanup, always initialize it. */
> + av_bprint_init(&bp, 1, MAX_SDP_SIZE);
> +
> + if (!whip->sdp_offer || !strlen(whip->sdp_offer)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: No offer to exchange\n");
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + ret = snprintf(buf, sizeof(buf), "Cache-Control: no-cache\r\nContent-Type: application/sdp\r\n");
> + if (whip->authorization)
> + ret += snprintf(buf + ret, sizeof(buf) - ret, "Authorization: Bearer %s\r\n", whip->authorization);
> + if (ret <= 0 || ret >= sizeof(buf)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to generate headers, size=%d, %s\n", ret, buf);
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + av_dict_set(&opts, "headers", buf, 0);
> + av_dict_set_int(&opts, "chunked_post", 0, 0);
> +
> + hex_data = av_mallocz(2 * strlen(whip->sdp_offer) + 1);
> + if (!hex_data) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + ff_data_to_hex(hex_data, whip->sdp_offer, strlen(whip->sdp_offer), 0);
> + av_dict_set(&opts, "post_data", hex_data, 0);
> +
> + ret = ffurl_open_whitelist(&whip_uc, s->url, AVIO_FLAG_READ_WRITE, &s->interrupt_callback,
> + &opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to request url=%s, offer: %s\n", s->url, whip->sdp_offer);
> + goto end;
> + }
> +
> + if (ff_http_get_new_location(whip_uc)) {
> + whip->whip_resource_url = av_strdup(ff_http_get_new_location(whip_uc));
> + if (!whip->whip_resource_url) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + }
> +
> + while (1) {
> + ret = ffurl_read(whip_uc, buf, sizeof(buf));
> + if (ret == AVERROR_EOF) {
> + /* Reset the error because we read all response as answer util EOF. */
> + ret = 0;
> + break;
> + }
> + if (ret <= 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read response from url=%s, offer is %s, answer is %s\n",
> + s->url, whip->sdp_offer, whip->sdp_answer);
> + goto end;
> + }
> +
> + av_bprintf(&bp, "%.*s", ret, buf);
> + if (!av_bprint_is_complete(&bp)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Answer exceed max size %d, %.*s, %s\n", MAX_SDP_SIZE, ret, buf, bp.str);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> + }
> +
> + if (!av_strstart(bp.str, "v=", NULL)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Invalid answer: %s\n", bp.str);
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + whip->sdp_answer = av_strdup(bp.str);
> + if (!whip->sdp_answer) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + if (whip->state < WHIP_STATE_ANSWER)
> + whip->state = WHIP_STATE_ANSWER;
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: Got state=%d, answer: %s\n", whip->state, whip->sdp_answer);
> +
> +end:
> + ffurl_closep(&whip_uc);
> + av_bprint_finalize(&bp, NULL);
> + av_dict_free(&opts);
> + av_freep(&hex_data);
> + return ret;
> +}
> +
> +/**
> + * Parses the ICE ufrag, pwd, and candidates from the SDP answer.
> + *
> + * This function is used to extract the ICE ufrag, pwd, and candidates from the SDP answer.
> + * It returns an error if any of these fields is NULL. The function only uses the first
> + * candidate if there are multiple candidates. However, support for multiple candidates
> + * will be added in the future.
> + *
> + * @param s Pointer to the AVFormatContext
> + * @returns Returns 0 if successful or AVERROR_xxx if an error occurs.
> + */
> +static int parse_answer(AVFormatContext *s)
> +{
> + int ret = 0;
> + AVIOContext *pb;
> + char line[MAX_URL_SIZE];
> + const char *ptr;
> + int i;
> + WHIPContext *whip = s->priv_data;
> +
> + if (!whip->sdp_answer || !strlen(whip->sdp_answer)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: No answer to parse\n");
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + pb = avio_alloc_context(whip->sdp_answer, strlen(whip->sdp_answer), 0, NULL, NULL, NULL, NULL);
> + if (!pb)
> + return AVERROR(ENOMEM);
> +
> + for (i = 0; !avio_feof(pb); i++) {
> + ff_get_chomp_line(pb, line, sizeof(line));
> + if (av_strstart(line, "a=ice-ufrag:", &ptr) && !whip->ice_ufrag_remote) {
> + whip->ice_ufrag_remote = av_strdup(ptr);
> + if (!whip->ice_ufrag_remote) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + } else if (av_strstart(line, "a=ice-pwd:", &ptr) && !whip->ice_pwd_remote) {
> + whip->ice_pwd_remote = av_strdup(ptr);
> + if (!whip->ice_pwd_remote) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + } else if (av_strstart(line, "a=candidate:", &ptr) && !whip->ice_protocol) {
> + ptr = av_stristr(ptr, "udp");
> + if (ptr && av_stristr(ptr, "host")) {
> + char protocol[17], host[129];
> + int priority, port;
> + ret = sscanf(ptr, "%16s %d %128s %d typ host", protocol, &priority, host, &port);
> + if (ret != 4) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed %d to parse line %d %s from %s\n",
> + ret, i, line, whip->sdp_answer);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + if (av_strcasecmp(protocol, "udp")) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Protocol %s is not supported by RTC, choose udp, line %d %s of %s\n",
> + protocol, i, line, whip->sdp_answer);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + whip->ice_protocol = av_strdup(protocol);
> + whip->ice_host = av_strdup(host);
> + whip->ice_port = port;
> + if (!whip->ice_protocol || !whip->ice_host) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + }
> + }
> + }
> +
> + if (!whip->ice_pwd_remote || !strlen(whip->ice_pwd_remote)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: No remote ice pwd parsed from %s\n", whip->sdp_answer);
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + if (!whip->ice_ufrag_remote || !strlen(whip->ice_ufrag_remote)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: No remote ice ufrag parsed from %s\n", whip->sdp_answer);
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + if (!whip->ice_protocol || !whip->ice_host || !whip->ice_port) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: No ice candidate parsed from %s\n", whip->sdp_answer);
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + if (whip->state < WHIP_STATE_NEGOTIATED)
> + whip->state = WHIP_STATE_NEGOTIATED;
> + whip->whip_answer_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: SDP state=%d, offer=%luB, answer=%luB, ufrag=%s, pwd=%luB, transport=%s://%s:%d, elapsed=%dms\n",
> + whip->state, strlen(whip->sdp_offer), strlen(whip->sdp_answer), whip->ice_ufrag_remote, strlen(whip->ice_pwd_remote),
> + whip->ice_protocol, whip->ice_host, whip->ice_port, ELAPSED(whip->whip_starttime, av_gettime()));
> +
> +end:
> + avio_context_free(&pb);
> + return ret;
> +}
> +
> +/**
> + * Creates and marshals an ICE binding request packet.
> + *
> + * This function creates and marshals an ICE binding request packet. The function only
> + * generates the username attribute and does not include goog-network-info, ice-controlling,
> + * use-candidate, and priority. However, some of these attributes may be added in the future.
> + *
> + * @param s Pointer to the AVFormatContext
> + * @param buf Pointer to memory buffer to store the request packet
> + * @param buf_size Size of the memory buffer
> + * @param request_size Pointer to an integer that receives the size of the request packet
> + * @return Returns 0 if successful or AVERROR_xxx if an error occurs.
> + */
> +static int ice_create_request(AVFormatContext *s, uint8_t *buf, int buf_size, int *request_size)
> +{
> + int ret, size, crc32;
> + char username[128];
> + AVIOContext *pb = NULL;
> + AVHMAC *hmac = NULL;
> + WHIPContext *whip = s->priv_data;
> +
> + pb = avio_alloc_context(buf, buf_size, 1, NULL, NULL, NULL, NULL);
> + if (!pb)
> + return AVERROR(ENOMEM);
> +
> + hmac = av_hmac_alloc(AV_HMAC_SHA1);
> + if (!hmac) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + /* Write 20 bytes header */
> + avio_wb16(pb, 0x0001); /* STUN binding request */
> + avio_wb16(pb, 0); /* length */
> + avio_wb32(pb, STUN_MAGIC_COOKIE); /* magic cookie */
> + avio_wb32(pb, av_lfg_get(&whip->rnd)); /* transaction ID */
> + avio_wb32(pb, av_lfg_get(&whip->rnd)); /* transaction ID */
> + avio_wb32(pb, av_lfg_get(&whip->rnd)); /* transaction ID */
> +
> + /* The username is the concatenation of the two ICE ufrag */
> + ret = snprintf(username, sizeof(username), "%s:%s", whip->ice_ufrag_remote, whip->ice_ufrag_local);
> + if (ret <= 0 || ret >= sizeof(username)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to build username %s:%s, max=%lu, ret=%d\n",
> + whip->ice_ufrag_remote, whip->ice_ufrag_local, sizeof(username), ret);
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + /* Write the username attribute */
> + avio_wb16(pb, STUN_ATTR_USERNAME); /* attribute type username */
> + avio_wb16(pb, ret); /* size of username */
> + avio_write(pb, username, ret); /* bytes of username */
> + ffio_fill(pb, 0, (4 - (ret % 4)) % 4); /* padding */
> +
> + /* Write the use-candidate attribute */
> + avio_wb16(pb, STUN_ATTR_USE_CANDIDATE); /* attribute type use-candidate */
> + avio_wb16(pb, 0); /* size of use-candidate */
> +
> + /* Build and update message integrity */
> + avio_wb16(pb, STUN_ATTR_MESSAGE_INTEGRITY); /* attribute type message integrity */
> + avio_wb16(pb, 20); /* size of message integrity */
> + ffio_fill(pb, 0, 20); /* fill with zero to directly write and skip it */
> + size = avio_tell(pb);
> + buf[2] = (size - 20) >> 8;
> + buf[3] = (size - 20) & 0xFF;
> + av_hmac_init(hmac, whip->ice_pwd_remote, strlen(whip->ice_pwd_remote));
> + av_hmac_update(hmac, buf, size - 24);
> + av_hmac_final(hmac, buf + size - 20, 20);
> +
> + /* Write the fingerprint attribute */
> + avio_wb16(pb, STUN_ATTR_FINGERPRINT); /* attribute type fingerprint */
> + avio_wb16(pb, 4); /* size of fingerprint */
> + ffio_fill(pb, 0, 4); /* fill with zero to directly write and skip it */
> + size = avio_tell(pb);
> + buf[2] = (size - 20) >> 8;
> + buf[3] = (size - 20) & 0xFF;
> + /* Refer to the av_hash_alloc("CRC32"), av_hash_init and av_hash_final */
> + crc32 = av_crc(av_crc_get_table(AV_CRC_32_IEEE_LE), 0xFFFFFFFF, buf, size - 8) ^ 0xFFFFFFFF;
> + avio_skip(pb, -4);
> + avio_wb32(pb, crc32 ^ 0x5354554E); /* xor with "STUN" */
> +
> + *request_size = size;
> +
> +end:
> + avio_context_free(&pb);
> + av_hmac_free(hmac);
> + return ret;
> +}
> +
> +/**
> + * Create an ICE binding response.
> + *
> + * This function generates an ICE binding response and writes it to the provided
> + * buffer. The response is signed using the local password for message integrity.
> + *
> + * @param s Pointer to the AVFormatContext structure.
> + * @param tid Pointer to the transaction ID of the binding request. The tid_size should be 12.
> + * @param tid_size The size of the transaction ID, should be 12.
> + * @param buf Pointer to the buffer where the response will be written.
> + * @param buf_size The size of the buffer provided for the response.
> + * @param response_size Pointer to an integer that will store the size of the generated response.
> + * @return Returns 0 if successful or AVERROR_xxx if an error occurs.
> + */
> +static int ice_create_response(AVFormatContext *s, char *tid, int tid_size, uint8_t *buf, int buf_size, int *response_size)
> +{
> + int ret = 0, size, crc32;
> + AVIOContext *pb = NULL;
> + AVHMAC *hmac = NULL;
> + WHIPContext *whip = s->priv_data;
> +
> + if (tid_size != 12) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Invalid transaction ID size. Expected 12, got %d\n", tid_size);
> + return AVERROR(EINVAL);
> + }
> +
> + pb = avio_alloc_context(buf, buf_size, 1, NULL, NULL, NULL, NULL);
> + if (!pb)
> + return AVERROR(ENOMEM);
> +
> + hmac = av_hmac_alloc(AV_HMAC_SHA1);
> + if (!hmac) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + /* Write 20 bytes header */
> + avio_wb16(pb, 0x0101); /* STUN binding response */
> + avio_wb16(pb, 0); /* length */
> + avio_wb32(pb, STUN_MAGIC_COOKIE); /* magic cookie */
> + avio_write(pb, tid, tid_size); /* transaction ID */
> +
> + /* Build and update message integrity */
> + avio_wb16(pb, STUN_ATTR_MESSAGE_INTEGRITY); /* attribute type message integrity */
> + avio_wb16(pb, 20); /* size of message integrity */
> + ffio_fill(pb, 0, 20); /* fill with zero to directly write and skip it */
> + size = avio_tell(pb);
> + buf[2] = (size - 20) >> 8;
> + buf[3] = (size - 20) & 0xFF;
> + av_hmac_init(hmac, whip->ice_pwd_local, strlen(whip->ice_pwd_local));
> + av_hmac_update(hmac, buf, size - 24);
> + av_hmac_final(hmac, buf + size - 20, 20);
> +
> + /* Write the fingerprint attribute */
> + avio_wb16(pb, STUN_ATTR_FINGERPRINT); /* attribute type fingerprint */
> + avio_wb16(pb, 4); /* size of fingerprint */
> + ffio_fill(pb, 0, 4); /* fill with zero to directly write and skip it */
> + size = avio_tell(pb);
> + buf[2] = (size - 20) >> 8;
> + buf[3] = (size - 20) & 0xFF;
> + /* Refer to the av_hash_alloc("CRC32"), av_hash_init and av_hash_final */
> + crc32 = av_crc(av_crc_get_table(AV_CRC_32_IEEE_LE), 0xFFFFFFFF, buf, size - 8) ^ 0xFFFFFFFF;
> + avio_skip(pb, -4);
> + avio_wb32(pb, crc32 ^ 0x5354554E); /* xor with "STUN" */
> +
> + *response_size = size;
> +
> +end:
> + avio_context_free(&pb);
> + av_hmac_free(hmac);
> + return ret;
> +}
> +
> +/**
> + * A Binding request has class=0b00 (request) and method=0b000000000001 (Binding)
> + * and is encoded into the first 16 bits as 0x0001.
> + * See https://datatracker.ietf.org/doc/html/rfc5389#section-6
> + */
> +static int ice_is_binding_request(uint8_t *b, int size)
> +{
> + return size >= ICE_STUN_HEADER_SIZE && AV_RB16(&b[0]) == 0x0001;
> +}
> +
> +/**
> + * A Binding response has class=0b10 (success response) and method=0b000000000001,
> + * and is encoded into the first 16 bits as 0x0101.
> + */
> +static int ice_is_binding_response(uint8_t *b, int size)
> +{
> + return size >= ICE_STUN_HEADER_SIZE && AV_RB16(&b[0]) == 0x0101;
> +}
> +
> +/**
> + * In RTP packets, the first byte is represented as 0b10xxxxxx, where the initial
> + * two bits (0b10) indicate the RTP version,
> + * see https://www.rfc-editor.org/rfc/rfc3550#section-5.1
> + * The RTCP packet header is similar to RTP,
> + * see https://www.rfc-editor.org/rfc/rfc3550#section-6.4.1
> + */
> +static int media_is_rtp_rtcp(uint8_t *b, int size)
> +{
> + return size >= WHIP_RTP_HEADER_SIZE && (b[0] & 0xC0) == 0x80;
> +}
> +
> +/* Whether the packet is RTCP. */
> +static int media_is_rtcp(uint8_t *b, int size)
> +{
> + return size >= WHIP_RTP_HEADER_SIZE && b[1] >= WHIP_RTCP_PT_START && b[1] <= WHIP_RTCP_PT_END;
> +}
> +
> +/**
> + * This function handles incoming binding request messages by responding to them.
> + * If the message is not a binding request, it will be ignored.
> + */
> +static int ice_handle_binding_request(AVFormatContext *s, char *buf, int buf_size)
> +{
> + int ret = 0, size;
> + char tid[12];
> + WHIPContext *whip = s->priv_data;
> +
> + /* Ignore if not a binding request. */
> + if (!ice_is_binding_request(buf, buf_size))
> + return ret;
> +
> + if (buf_size < ICE_STUN_HEADER_SIZE) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Invalid STUN message, expected at least %d, got %d\n",
> + ICE_STUN_HEADER_SIZE, buf_size);
> + return AVERROR(EINVAL);
> + }
> +
> + /* Parse transaction id from binding request in buf. */
> + memcpy(tid, buf + 8, 12);
> +
> + /* Build the STUN binding response. */
> + ret = ice_create_response(s, tid, sizeof(tid), whip->buf, sizeof(whip->buf), &size);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to create STUN binding response, size=%d\n", size);
> + return ret;
> + }
> +
> + ret = ffurl_write(whip->udp, whip->buf, size);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send STUN binding response, size=%d\n", size);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +/**
> + * To establish a connection with the UDP server, we utilize ICE-LITE in a Client-Server
> + * mode. In this setup, FFmpeg acts as the UDP client, while the peer functions as the
> + * UDP server.
> + */
> +static int udp_connect(AVFormatContext *s)
> +{
> + int ret = 0;
> + char url[256];
> + AVDictionary *opts = NULL;
> + WHIPContext *whip = s->priv_data;
> +
> + /* Build UDP URL and create the UDP context as transport. */
> + ff_url_join(url, sizeof(url), "udp", NULL, whip->ice_host, whip->ice_port, NULL);
> +
> + av_dict_set_int(&opts, "connect", 1, 0);
> + av_dict_set_int(&opts, "fifo_size", 0, 0);
> + /* Set the max packet size to the buffer size. */
> + av_dict_set_int(&opts, "pkt_size", whip->pkt_size, 0);
> +
> + ret = ffurl_open_whitelist(&whip->udp, url, AVIO_FLAG_WRITE, &s->interrupt_callback,
> + &opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to connect udp://%s:%d\n", whip->ice_host, whip->ice_port);
> + goto end;
> + }
> +
> + /* Make the socket non-blocking, set to READ and WRITE mode after connected */
> + ff_socket_nonblock(ffurl_get_file_handle(whip->udp), 1);
> + whip->udp->flags |= AVIO_FLAG_READ | AVIO_FLAG_NONBLOCK;
> +
> + if (whip->state < WHIP_STATE_UDP_CONNECTED)
> + whip->state = WHIP_STATE_UDP_CONNECTED;
> + whip->whip_udp_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: UDP state=%d, elapsed=%dms, connected to udp://%s:%d\n",
> + whip->state, ELAPSED(whip->whip_starttime, av_gettime()), whip->ice_host, whip->ice_port);
> +
> +end:
> + av_dict_free(&opts);
> + return ret;
> +}
> +
> +static int ice_dtls_handshake(AVFormatContext *s)
> +{
> + int ret = 0, size, i;
> + int64_t starttime = av_gettime(), now;
> + WHIPContext *whip = s->priv_data;
> + AVDictionary *opts = NULL;
> + char str[8];
> + char buf[256], *cert_buf = NULL, *key_buf = NULL;
> +
> + if (whip->state < WHIP_STATE_UDP_CONNECTED || !whip->udp) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: UDP not connected, state=%d, udp=%p\n", whip->state, whip->udp);
> + return AVERROR(EINVAL);
> + }
> +
> + while (1) {
> + if (whip->state <= WHIP_STATE_ICE_CONNECTING) {
> + /* Build the STUN binding request. */
> + ret = ice_create_request(s, whip->buf, sizeof(whip->buf), &size);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to create STUN binding request, size=%d\n", size);
> + goto end;
> + }
> +
> + ret = ffurl_write(whip->udp, whip->buf, size);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send STUN binding request, size=%d\n", size);
> + goto end;
> + }
> +
> + if (whip->state < WHIP_STATE_ICE_CONNECTING)
> + whip->state = WHIP_STATE_ICE_CONNECTING;
> + }
> +
> +next_packet:
> + if (whip->state >= WHIP_STATE_DTLS_FINISHED)
> + /* DTLS handshake is done, exit the loop. */
> + break;
> +
> + now = av_gettime();
> + if (now - starttime >= whip->handshake_timeout * 1000) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: DTLS handshake timeout=%dms, cost=%dms, elapsed=%dms, state=%d\n",
> + whip->handshake_timeout, ELAPSED(starttime, now), ELAPSED(whip->whip_starttime, now), whip->state);
> + ret = AVERROR(ETIMEDOUT);
> + goto end;
> + }
> +
> + /* Read the STUN or DTLS messages from peer. */
> + for (i = 0; i < ICE_DTLS_READ_INTERVAL / 5 && whip->state < WHIP_STATE_DTLS_CONNECTING; i++) {
> + ret = ffurl_read(whip->udp, whip->buf, sizeof(whip->buf));
> + if (ret > 0)
> + break;
> + if (ret == AVERROR(EAGAIN)) {
> + av_usleep(5 * 1000);
> + continue;
> + }
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read message\n");
> + goto end;
> + }
> +
> + /* Got nothing, continue to process handshake. */
> + if (ret <= 0 && whip->state < WHIP_STATE_DTLS_CONNECTING)
> + continue;
> +
> + /* Handle the ICE binding response. */
> + if (ice_is_binding_response(whip->buf, ret)) {
> + if (whip->state < WHIP_STATE_ICE_CONNECTED) {
> + whip->state = WHIP_STATE_ICE_CONNECTED;
> + whip->whip_ice_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: ICE STUN ok, state=%d, url=udp://%s:%d, location=%s, username=%s:%s, res=%dB, elapsed=%dms\n",
> + whip->state, whip->ice_host, whip->ice_port, whip->whip_resource_url ? whip->whip_resource_url : "",
> + whip->ice_ufrag_remote, whip->ice_ufrag_local, ret, ELAPSED(whip->whip_starttime, av_gettime()));
> +
> + ff_url_join(buf, sizeof(buf), "dtls", NULL, whip->ice_host, whip->ice_port, NULL);
> + snprintf(str, sizeof(str), "%d", whip->pkt_size);
> + av_dict_set(&opts, "mtu", str, 0);
> + if (whip->cert_file) {
> + av_dict_set(&opts, "cert_file", whip->cert_file, 0);
> + } else
> + av_dict_set(&opts, "cert_buf", whip->cert_buf, 0);
> +
> + if (whip->key_file) {
> + av_dict_set(&opts, "key_file", whip->key_file, 0);
> + } else
> + av_dict_set(&opts, "key_buf", whip->key_buf, 0);
> +
> + av_dict_set(&opts, "fingerprint", whip->dtls_fingerprint, 0);
> + av_dict_set(&opts, "use_external_udp", "1", 0);
> + av_dict_set(&opts, "listen", "1", 0);
> + /* If got the first binding response, start DTLS handshake. */
> + ret = ffurl_open_whitelist(&whip->dtls_uc, buf, AVIO_FLAG_READ_WRITE, &s->interrupt_callback,
> + &opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
> + if (ret < 0)
> + goto end;
> + dtls_initialize(s);
> + }
> + goto next_packet;
> + }
> +
> + /* When a binding request is received, it is necessary to respond immediately. */
> + if (ice_is_binding_request(whip->buf, ret)) {
> + if ((ret = ice_handle_binding_request(s, whip->buf, ret)) < 0)
> + goto end;
> + goto next_packet;
> + }
> +
> + /* If got any DTLS messages, handle it. */
> + if (is_dtls_packet(whip->buf, ret) && whip->state >= WHIP_STATE_ICE_CONNECTED || whip->state == WHIP_STATE_DTLS_CONNECTING) {
> + whip->state = WHIP_STATE_DTLS_CONNECTING;
> + if ((ret = ffurl_handshake(whip->dtls_uc)) < 0)
> + goto end;
> + dtls_context_on_state(s, NULL, NULL);
> + goto next_packet;
> + }
> + }
> +
> +end:
> + if (cert_buf)
> + av_free(cert_buf);
> + if (key_buf)
> + av_free(key_buf);
> + return ret;
> +}
> +
> +/**
> + * Establish the SRTP context using the keying material exported from DTLS.
> + *
> + * Create separate SRTP contexts for sending video and audio, as their sequences differ
> + * and should not share a single context. Generate a single SRTP context for receiving
> + * RTCP only.
> + *
> + * @return 0 if OK, AVERROR_xxx on error
> + */
> +static int setup_srtp(AVFormatContext *s)
> +{
> + int ret;
> + char recv_key[DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN];
> + char send_key[DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN];
> + char buf[AV_BASE64_SIZE(DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN)];
> + /**
> + * The profile for OpenSSL's SRTP is SRTP_AES128_CM_SHA1_80, see ssl/d1_srtp.c.
> + * The profile for FFmpeg's SRTP is SRTP_AES128_CM_HMAC_SHA1_80, see libavformat/srtp.c.
> + */
> + const char* suite = "SRTP_AES128_CM_HMAC_SHA1_80";
> + WHIPContext *whip = s->priv_data;
> + ret = ff_dtls_export_materials(whip->dtls_uc, whip->dtls_srtp_materials, sizeof(whip->dtls_srtp_materials));
> + if (ret < 0)
> + goto end;
> + /**
> + * This represents the material used to build the SRTP master key. It is
> + * generated by DTLS and has the following layout:
> + * 16B 16B 14B 14B
> + * client_key | server_key | client_salt | server_salt
> + */
> + char *client_key = whip->dtls_srtp_materials;
> + char *server_key = whip->dtls_srtp_materials + DTLS_SRTP_KEY_LEN;
> + char *client_salt = server_key + DTLS_SRTP_KEY_LEN;
> + char *server_salt = client_salt + DTLS_SRTP_SALT_LEN;
> +
> + /* As DTLS server, the recv key is client master key plus salt. */
> + memcpy(recv_key, client_key, DTLS_SRTP_KEY_LEN);
> + memcpy(recv_key + DTLS_SRTP_KEY_LEN, client_salt, DTLS_SRTP_SALT_LEN);
> +
> + /* As DTLS server, the send key is server master key plus salt. */
> + memcpy(send_key, server_key, DTLS_SRTP_KEY_LEN);
> + memcpy(send_key + DTLS_SRTP_KEY_LEN, server_salt, DTLS_SRTP_SALT_LEN);
> +
> + /* Setup SRTP context for outgoing packets */
> + if (!av_base64_encode(buf, sizeof(buf), send_key, sizeof(send_key))) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to encode send key\n");
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + ret = ff_srtp_set_crypto(&whip->srtp_audio_send, suite, buf);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for audio send\n");
> + goto end;
> + }
> +
> + ret = ff_srtp_set_crypto(&whip->srtp_video_send, suite, buf);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for video send\n");
> + goto end;
> + }
> +
> + ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n");
> + goto end;
> + }
> +
> + /* Setup SRTP context for incoming packets */
> + if (!av_base64_encode(buf, sizeof(buf), recv_key, sizeof(recv_key))) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to encode recv key\n");
> + ret = AVERROR(EIO);
> + goto end;
> + }
> +
> + ret = ff_srtp_set_crypto(&whip->srtp_recv, suite, buf);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for recv\n");
> + goto end;
> + }
> +
> + if (whip->state < WHIP_STATE_SRTP_FINISHED)
> + whip->state = WHIP_STATE_SRTP_FINISHED;
> + whip->whip_srtp_time = av_gettime();
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: SRTP setup done, state=%d, suite=%s, key=%luB, elapsed=%dms\n",
> + whip->state, suite, sizeof(send_key), ELAPSED(whip->whip_starttime, av_gettime()));
> +
> +end:
> + return ret;
> +}
> +
> +/**
> + * Callback triggered by the RTP muxer when it creates and sends out an RTP packet.
> + *
> + * This function modifies the video STAP packet, removing the markers, and updating the
> + * NRI of the first NALU. Additionally, it uses the corresponding SRTP context to encrypt
> + * the RTP packet, where the video packet is handled by the video SRTP context.
> + */
> +static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
> +{
> + int ret, cipher_size, is_rtcp, is_video;
> + uint8_t payload_type;
> + AVFormatContext *s = opaque;
> + WHIPContext *whip = s->priv_data;
> + SRTPContext *srtp;
> +
> + /* Ignore if not RTP or RTCP packet. */
> + if (!media_is_rtp_rtcp(buf, buf_size))
> + return 0;
> +
> + /* Only support audio, video and rtcp. */
> + is_rtcp = media_is_rtcp(buf, buf_size);
> + payload_type = buf[1] & 0x7f;
> + is_video = payload_type == whip->video_payload_type;
> + if (!is_rtcp && payload_type != whip->video_payload_type && payload_type != whip->audio_payload_type)
> + return 0;
> +
> + /* Get the corresponding SRTP context. */
> + srtp = is_rtcp ? &whip->srtp_rtcp_send : (is_video? &whip->srtp_video_send : &whip->srtp_audio_send);
> +
> + /* Encrypt by SRTP and send out. */
> + cipher_size = ff_srtp_encrypt(srtp, buf, buf_size, whip->buf, sizeof(whip->buf));
> + if (cipher_size <= 0 || cipher_size < buf_size) {
> + av_log(whip, AV_LOG_WARNING, "WHIP: Failed to encrypt packet=%dB, cipher=%dB\n", buf_size, cipher_size);
> + return 0;
> + }
> +
> + ret = ffurl_write(whip->udp, whip->buf, cipher_size);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to write packet=%dB, ret=%d\n", cipher_size, ret);
> + return ret;
> + }
> +
> + return ret;
> +}
> +
> +/**
> + * Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP
> + * packets from the encoded frames.
> + *
> + * The corresponding SRTP context is utilized to encrypt each stream's RTP packets. For
> + * example, a video SRTP context is used for the video stream. Additionally, the
> + * "on_rtp_write_packet" callback function is set as the write function for each RTP
> + * muxer to send out encrypted RTP packets.
> + *
> + * @return 0 if OK, AVERROR_xxx on error
> + */
> +static int create_rtp_muxer(AVFormatContext *s)
> +{
> + int ret, i, is_video, buffer_size, max_packet_size;
> + AVFormatContext *rtp_ctx = NULL;
> + AVDictionary *opts = NULL;
> + uint8_t *buffer = NULL;
> + char buf[64];
> + WHIPContext *whip = s->priv_data;
> +
> + const AVOutputFormat *rtp_format = av_guess_format("rtp", NULL, NULL);
> + if (!rtp_format) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to guess rtp muxer\n");
> + ret = AVERROR(ENOSYS);
> + goto end;
> + }
> +
> + /* The UDP buffer size, may greater than MTU. */
> + buffer_size = MAX_UDP_BUFFER_SIZE;
> + /* The RTP payload max size. Reserved some bytes for SRTP checksum and padding. */
> + max_packet_size = whip->pkt_size - DTLS_SRTP_CHECKSUM_LEN;
> +
> + for (i = 0; i < s->nb_streams; i++) {
> + rtp_ctx = avformat_alloc_context();
> + if (!rtp_ctx) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + rtp_ctx->oformat = rtp_format;
> + if (!avformat_new_stream(rtp_ctx, NULL)) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + /* Pass the interrupt callback on */
> + rtp_ctx->interrupt_callback = s->interrupt_callback;
> + /* Copy the max delay setting; the rtp muxer reads this. */
> + rtp_ctx->max_delay = s->max_delay;
> + /* Copy other stream parameters. */
> + rtp_ctx->streams[0]->sample_aspect_ratio = s->streams[i]->sample_aspect_ratio;
> + rtp_ctx->flags |= s->flags & AVFMT_FLAG_BITEXACT;
> + rtp_ctx->strict_std_compliance = s->strict_std_compliance;
> +
> + /* Set the synchronized start time. */
> + rtp_ctx->start_time_realtime = s->start_time_realtime;
> +
> + avcodec_parameters_copy(rtp_ctx->streams[0]->codecpar, s->streams[i]->codecpar);
> + rtp_ctx->streams[0]->time_base = s->streams[i]->time_base;
> +
> + /**
> + * For H.264, consistently utilize the annexb format through the Bitstream Filter (BSF);
> + * therefore, we deactivate the extradata detection for the RTP muxer.
> + */
> + if (s->streams[i]->codecpar->codec_id == AV_CODEC_ID_H264) {
> + av_freep(&rtp_ctx->streams[i]->codecpar->extradata);
> + rtp_ctx->streams[i]->codecpar->extradata_size = 0;
> + }
> +
> + buffer = av_malloc(buffer_size);
> + if (!buffer) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> +
> + rtp_ctx->pb = avio_alloc_context(buffer, buffer_size, 1, s, NULL, on_rtp_write_packet, NULL);
> + if (!rtp_ctx->pb) {
> + ret = AVERROR(ENOMEM);
> + goto end;
> + }
> + rtp_ctx->pb->max_packet_size = max_packet_size;
> + rtp_ctx->pb->av_class = &ff_avio_class;
> +
> + is_video = s->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO;
> + snprintf(buf, sizeof(buf), "%d", is_video? whip->video_payload_type : whip->audio_payload_type);
> + av_dict_set(&opts, "payload_type", buf, 0);
> + snprintf(buf, sizeof(buf), "%d", is_video? whip->video_ssrc : whip->audio_ssrc);
> + av_dict_set(&opts, "ssrc", buf, 0);
> +
> + ret = avformat_write_header(rtp_ctx, &opts);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to write rtp header\n");
> + goto end;
> + }
> +
> + ff_format_set_url(rtp_ctx, av_strdup(s->url));
> + s->streams[i]->time_base = rtp_ctx->streams[0]->time_base;
> + s->streams[i]->priv_data = rtp_ctx;
> + rtp_ctx = NULL;
> + }
> +
> + if (whip->state < WHIP_STATE_READY)
> + whip->state = WHIP_STATE_READY;
> + av_log(whip, AV_LOG_INFO, "WHIP: Muxer state=%d, buffer_size=%d, max_packet_size=%d, "
> + "elapsed=%dms(init:%d,offer:%d,answer:%d,udp:%d,ice:%d,dtls:%d,srtp:%d)\n",
> + whip->state, buffer_size, max_packet_size, ELAPSED(whip->whip_starttime, av_gettime()),
> + ELAPSED(whip->whip_starttime, whip->whip_init_time),
> + ELAPSED(whip->whip_init_time, whip->whip_offer_time),
> + ELAPSED(whip->whip_offer_time, whip->whip_answer_time),
> + ELAPSED(whip->whip_answer_time, whip->whip_udp_time),
> + ELAPSED(whip->whip_udp_time, whip->whip_ice_time),
> + ELAPSED(whip->whip_ice_time, whip->whip_dtls_time),
> + ELAPSED(whip->whip_dtls_time, whip->whip_srtp_time));
> +
> +end:
> + if (rtp_ctx)
> + avio_context_free(&rtp_ctx->pb);
> + avformat_free_context(rtp_ctx);
> + av_dict_free(&opts);
> + return ret;
> +}
> +
> +/**
> + * RTC is connectionless, for it's based on UDP, so it check whether sesison is
> + * timeout. In such case, publishers can't republish the stream util the session
> + * is timeout.
> + * This function is called to notify the server that the stream is ended, server
> + * should expire and close the session immediately, so that publishers can republish
> + * the stream quickly.
> + */
> +static int dispose_session(AVFormatContext *s)
> +{
> + int ret;
> + char buf[MAX_URL_SIZE];
> + URLContext *whip_uc = NULL;
> + AVDictionary *opts = NULL;
> + WHIPContext *whip = s->priv_data;
> +
> + if (!whip->whip_resource_url)
> + return 0;
> +
> + ret = snprintf(buf, sizeof(buf), "Cache-Control: no-cache\r\n");
> + if (whip->authorization)
> + ret += snprintf(buf + ret, sizeof(buf) - ret, "Authorization: Bearer %s\r\n", whip->authorization);
> + if (ret <= 0 || ret >= sizeof(buf)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to generate headers, size=%d, %s\n", ret, buf);
> + ret = AVERROR(EINVAL);
> + goto end;
> + }
> +
> + av_dict_set(&opts, "headers", buf, 0);
> + av_dict_set_int(&opts, "chunked_post", 0, 0);
> + av_dict_set(&opts, "method", "DELETE", 0);
> + ret = ffurl_open_whitelist(&whip_uc, whip->whip_resource_url, AVIO_FLAG_READ_WRITE, &s->interrupt_callback,
> + &opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to DELETE url=%s\n", whip->whip_resource_url);
> + goto end;
> + }
> +
> + while (1) {
> + ret = ffurl_read(whip_uc, buf, sizeof(buf));
> + if (ret == AVERROR_EOF) {
> + ret = 0;
> + break;
> + }
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read response from DELETE url=%s\n", whip->whip_resource_url);
> + goto end;
> + }
> + }
> +
> + av_log(whip, AV_LOG_INFO, "WHIP: Dispose resource %s ok\n", whip->whip_resource_url);
> +
> +end:
> + ffurl_closep(&whip_uc);
> + av_dict_free(&opts);
> + return ret;
> +}
> +
> +/**
> + * Since the h264_mp4toannexb filter only processes the MP4 ISOM format and bypasses
> + * the annexb format, it is necessary to manually insert encoder metadata before each
> + * IDR when dealing with annexb format packets. For instance, in the case of H.264,
> + * we must insert SPS and PPS before the IDR frame.
> + */
> +static int h264_annexb_insert_sps_pps(AVFormatContext *s, AVPacket *pkt)
> +{
> + int ret = 0;
> + AVPacket *in = NULL;
> + AVCodecParameters *par = s->streams[pkt->stream_index]->codecpar;
> + uint32_t nal_size = 0, out_size = par ? par->extradata_size : 0;
> + uint8_t unit_type, sps_seen = 0, pps_seen = 0, idr_seen = 0, *out;
> + const uint8_t *buf, *buf_end, *r1;
> +
> + if (!pkt || !pkt->data || pkt->size <= 0)
> + return ret;
> + if (!par || !par->extradata || par->extradata_size <= 0)
> + return ret;
> +
> + /* Discover NALU type from packet. */
> + buf_end = pkt->data + pkt->size;
> + for (buf = ff_nal_find_startcode(pkt->data, buf_end); buf < buf_end; buf += nal_size) {
> + while (!*(buf++));
> + r1 = ff_nal_find_startcode(buf, buf_end);
> + if ((nal_size = r1 - buf) > 0) {
> + unit_type = *buf & 0x1f;
> + if (unit_type == H264_NAL_SPS) {
> + sps_seen = 1;
> + } else if (unit_type == H264_NAL_PPS) {
> + pps_seen = 1;
> + } else if (unit_type == H264_NAL_IDR_SLICE) {
> + idr_seen = 1;
> + }
> +
> + out_size += 3 + nal_size;
> + }
> + }
> +
> + if (!idr_seen || (sps_seen && pps_seen))
> + return ret;
> +
> + /* See av_bsf_send_packet */
> + in = av_packet_alloc();
> + if (!in)
> + return AVERROR(ENOMEM);
> +
> + ret = av_packet_make_refcounted(pkt);
> + if (ret < 0)
> + goto fail;
> +
> + av_packet_move_ref(in, pkt);
> +
> + /* Create a new packet with sps/pps inserted. */
> + ret = av_new_packet(pkt, out_size);
> + if (ret < 0)
> + goto fail;
> +
> + ret = av_packet_copy_props(pkt, in);
> + if (ret < 0)
> + goto fail;
> +
> + memcpy(pkt->data, par->extradata, par->extradata_size);
> + out = pkt->data + par->extradata_size;
> + buf_end = in->data + in->size;
> + for (buf = ff_nal_find_startcode(in->data, buf_end); buf < buf_end; buf += nal_size) {
> + while (!*(buf++));
> + r1 = ff_nal_find_startcode(buf, buf_end);
> + if ((nal_size = r1 - buf) > 0) {
> + AV_WB24(out, 0x00001);
> + memcpy(out + 3, buf, nal_size);
> + out += 3 + nal_size;
> + }
> + }
> +
> +fail:
> + if (ret < 0)
> + av_packet_unref(pkt);
> + av_packet_free(&in);
> +
> + return ret;
> +}
> +
> +static av_cold int whip_init(AVFormatContext *s)
> +{
> + int ret;
> + WHIPContext *whip = s->priv_data;
> +
> + if ((ret = initialize(s)) < 0)
> + goto end;
> +
> + if ((ret = parse_codec(s)) < 0)
> + goto end;
> +
> + if ((ret = generate_sdp_offer(s)) < 0)
> + goto end;
> +
> + if ((ret = exchange_sdp(s)) < 0)
> + goto end;
> +
> + if ((ret = parse_answer(s)) < 0)
> + goto end;
> +
> + if ((ret = udp_connect(s)) < 0)
> + goto end;
> +
> + if ((ret = ice_dtls_handshake(s)) < 0)
> + goto end;
> +
> + if ((ret = setup_srtp(s)) < 0)
> + goto end;
> +
> + if ((ret = create_rtp_muxer(s)) < 0)
> + goto end;
> +
> +end:
> + if (ret < 0 && whip->state < WHIP_STATE_FAILED)
> + whip->state = WHIP_STATE_FAILED;
> + if (ret >= 0 && whip->state >= WHIP_STATE_FAILED && whip->dtls_ret < 0)
> + ret = whip->dtls_ret;
> + return ret;
> +}
> +
> +static int whip_write_packet(AVFormatContext *s, AVPacket *pkt)
> +{
> + int ret;
> + WHIPContext *whip = s->priv_data;
> + AVStream *st = s->streams[pkt->stream_index];
> + AVFormatContext *rtp_ctx = st->priv_data;
> +
> + /* TODO: Send binding request every 1s as WebRTC heartbeat. */
> +
> + /**
> + * Receive packets from the server such as ICE binding requests, DTLS messages,
> + * and RTCP like PLI requests, then respond to them.
> + */
> + ret = ffurl_read(whip->udp, whip->buf, sizeof(whip->buf));
> + if (ret > 0) {
> + if (is_dtls_packet(whip->buf, ret)) {
> + if ((ret = ffurl_write(whip->dtls_uc, whip->buf, ret)) < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to handle DTLS message\n");
> + goto end;
> + }
> + }
> + } else if (ret != AVERROR(EAGAIN)) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read from UDP socket\n");
> + goto end;
> + }
> +
> + if (whip->h264_annexb_insert_sps_pps && st->codecpar->codec_id == AV_CODEC_ID_H264) {
> + if ((ret = h264_annexb_insert_sps_pps(s, pkt)) < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to insert SPS/PPS before IDR\n");
> + goto end;
> + }
> + }
> +
> + ret = ff_write_chained(rtp_ctx, 0, pkt, s, 0);
> + if (ret < 0) {
> + if (ret == AVERROR(EINVAL)) {
> + av_log(whip, AV_LOG_WARNING, "WHIP: Ignore failed to write packet=%dB, ret=%d\n", pkt->size, ret);
> + ret = 0;
> + } else
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to write packet, size=%d\n", pkt->size);
> + goto end;
> + }
> +
> +end:
> + if (ret < 0 && whip->state < WHIP_STATE_FAILED)
> + whip->state = WHIP_STATE_FAILED;
> + if (ret >= 0 && whip->state >= WHIP_STATE_FAILED && whip->dtls_ret < 0)
> + ret = whip->dtls_ret;
> + if (ret >= 0 && whip->dtls_closed)
> + ret = AVERROR(EIO);
> + return ret;
> +}
> +
> +static av_cold void whip_deinit(AVFormatContext *s)
> +{
> + int i, ret;
> + WHIPContext *whip = s->priv_data;
> +
> + ret = dispose_session(s);
> + if (ret < 0)
> + av_log(whip, AV_LOG_WARNING, "WHIP: Failed to dispose resource, ret=%d\n", ret);
> +
> + for (i = 0; i < s->nb_streams; i++) {
> + AVFormatContext* rtp_ctx = s->streams[i]->priv_data;
> + if (!rtp_ctx)
> + continue;
> +
> + av_write_trailer(rtp_ctx);
> + /**
> + * Keep in mind that it is necessary to free the buffer of pb since we allocate
> + * it and pass it to pb using avio_alloc_context, while avio_context_free does
> + * not perform this action.
> + */
> + av_freep(&rtp_ctx->pb->buffer);
> + avio_context_free(&rtp_ctx->pb);
> + avformat_free_context(rtp_ctx);
> + s->streams[i]->priv_data = NULL;
> + }
> +
> + av_freep(&whip->sdp_offer);
> + av_freep(&whip->sdp_answer);
> + av_freep(&whip->whip_resource_url);
> + av_freep(&whip->ice_ufrag_remote);
> + av_freep(&whip->ice_pwd_remote);
> + av_freep(&whip->ice_protocol);
> + av_freep(&whip->ice_host);
> + av_freep(&whip->authorization);
> + av_freep(&whip->cert_file);
> + av_freep(&whip->key_file);
> + ffurl_closep(&whip->udp);
> + ff_srtp_free(&whip->srtp_audio_send);
> + ff_srtp_free(&whip->srtp_video_send);
> + ff_srtp_free(&whip->srtp_rtcp_send);
> + ff_srtp_free(&whip->srtp_recv);
> + ffurl_close(whip->dtls_uc);
> +}
> +
> +static int whip_check_bitstream(AVFormatContext *s, AVStream *st, const AVPacket *pkt)
> +{
> + int ret = 1, extradata_isom = 0;
> + uint8_t *b = pkt->data;
> + WHIPContext *whip = s->priv_data;
> +
> + if (st->codecpar->codec_id == AV_CODEC_ID_H264) {
> + extradata_isom = st->codecpar->extradata_size > 0 && st->codecpar->extradata[0] == 1;
> + if (pkt->size >= 5 && AV_RB32(b) != 0x0000001 && (AV_RB24(b) != 0x000001 || extradata_isom)) {
> + ret = ff_stream_add_bitstream_filter(st, "h264_mp4toannexb", NULL);
> + av_log(whip, AV_LOG_VERBOSE, "WHIP: Enable BSF h264_mp4toannexb, packet=[%x %x %x %x %x ...], extradata_isom=%d\n",
> + b[0], b[1], b[2], b[3], b[4], extradata_isom);
> + } else
> + whip->h264_annexb_insert_sps_pps = 1;
> + }
> +
> + return ret;
> +}
> +
> +#define OFFSET(x) offsetof(WHIPContext, x)
> +#define DEC AV_OPT_FLAG_DECODING_PARAM
> +static const AVOption options[] = {
> + { "handshake_timeout", "Timeout in milliseconds for ICE and DTLS handshake.", OFFSET(handshake_timeout), AV_OPT_TYPE_INT, { .i64 = 5000 }, -1, INT_MAX, DEC },
> + { "pkt_size", "The maximum size, in bytes, of RTP packets that send out", OFFSET(pkt_size), AV_OPT_TYPE_INT, { .i64 = 1200 }, -1, INT_MAX, DEC },
> + { "authorization", "The optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, DEC },
> + { "cert_file", "The optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, DEC },
> + { "key_file", "The optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, DEC },
> + { NULL },
> +};
> +
> +static const AVClass whip_muxer_class = {
> + .class_name = "WHIP muxer",
> + .item_name = av_default_item_name,
> + .option = options,
> + .version = LIBAVUTIL_VERSION_INT,
> +};
> +
> +const FFOutputFormat ff_whip_muxer = {
> + .p.name = "whip",
> + .p.long_name = NULL_IF_CONFIG_SMALL("WHIP(WebRTC-HTTP ingestion protocol) muxer"),
> + .p.audio_codec = AV_CODEC_ID_OPUS,
> + .p.video_codec = AV_CODEC_ID_H264,
> + .p.flags = AVFMT_GLOBALHEADER | AVFMT_NOFILE,
> + .p.priv_class = &whip_muxer_class,
> + .priv_data_size = sizeof(WHIPContext),
> + .init = whip_init,
> + .write_packet = whip_write_packet,
> + .deinit = whip_deinit,
> + .check_bitstream = whip_check_bitstream,
> +};
> --
> 2.49.0
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
Pushed,
BTW, if you other developers want fix other problems, patchwelcome,
To Jack, you can continue submit more patches for whip in ffmpeg.
Thanks
Steven
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
next prev parent reply other threads:[~2025-06-04 3:22 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-17 9:24 Jack Lau via ffmpeg-devel
2025-05-20 11:37 ` Steven Liu
2025-06-04 3:21 ` Steven Liu [this message]
2025-06-04 3:38 ` Jack Lau
2025-06-04 11:34 ` Kieran Kunhya via ffmpeg-devel
2025-06-04 11:46 ` Steven Liu
2025-06-04 22:20 ` Kieran Kunhya via ffmpeg-devel
2025-06-04 23:03 ` Jack Lau
2025-06-05 13:15 ` Kieran Kunhya via ffmpeg-devel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CADxeRw=5o1NvZTvjr3buZ6fN0n-4i_QdAg41-Ne=MShEhA7XfQ@mail.gmail.com' \
--to=lingjiujianke@gmail.com \
--cc=495810242@qq.com \
--cc=barryjzhao@tencent.com \
--cc=duanweiwei1982@gmail.com \
--cc=ffmpeg-devel@ffmpeg.org \
--cc=jacklau1222@qq.com \
--cc=lq@chinaffmpeg.org \
--cc=winlinvip@gmail.com \
--cc=yangrtc@aliyun.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
This inbox may be cloned and mirrored by anyone:
git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
ffmpegdev@gitmailbox.com
public-inbox-index ffmpegdev
Example config snippet for mirrors.
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git