From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id DDDDA44AF7 for ; Sun, 12 Oct 2025 15:45:17 +0000 (UTC) Authentication-Results: ffbox; dkim=fail (body hash mismatch (got b'uTI71y/OMIZ2VQmpXxx9Uoc2OmL6C4yl0TVi05dfcao=', expected b'ZXDS6PTD0UllxWZbjggetNrM0veMJCrLACZMrC71l1g=')) header.d=qq.com header.a=rsa-sha256 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ffmpeg.org; i=@ffmpeg.org; q=dns/txt; s=mail; t=1760283813; h=message-id : to : date : in-reply-to : references : mime-version : reply-to : subject : list-id : list-archive : list-archive : list-help : list-owner : list-post : list-subscribe : list-unsubscribe : from : cc : content-type : content-transfer-encoding : from; bh=3rP/niw5yBA8utqZnb7lRA6b95hOe2MrQYdj8Y4jHEc=; b=d/lF05N3SZ3AlYqf2oOWPj/6KqbMVndmcXByIn7xcwTnsec0ebCLrkTJmCcY4SA1jm9gJ l4RPOn+kqrxAgB8zH3noPljANrBUPSGyoIG4pHDL3KgWK0Oa+uyh4eG++PA6dI4LyvLXGpc 1GsvW0a1gCaaTJsmbQ2ScwFdVAeBj1fOovpSILTQAqSAyeaPLPZnV65Hg/cXl16S6FDa6Pv 7NmY2cYzqDsLV6U+JVmI0WBRBKyACvJ/87BLKa9KPqVEbm7V1kG0DdpuGW7PxJrmSrvx1Vz dH+Qr9k9MFwKj6zc/mAF3gEmPEJ1FP3OWtNGXg2TBjq+oXRsKTDkFRF3eEqw== Received: from [172.19.0.2] (unknown [172.19.0.2]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 1DF4568F1DE; Sun, 12 Oct 2025 18:43:33 +0300 (EEST) ARC-Seal: i=1; cv=none; a=rsa-sha256; d=ffmpeg.org; s=arc; t=1760283806; b=dfuyCeVtEok2Zb4hlfLmeLKWROgUhjlZmUJ3mZww+spVv94JPgR9FFnFNJcxdscN1ymFJ kpgyIvn8XgFtwstiWN5WZcuuxc8zE0ks6Vg5kuRiHNOMuEpbOF2nacwpNI2ObIXzoNAjIYB xXrFtyWB6FHI9sXwH7hThhP6T5vsvKMVZGgHoCSwP3YIRqeyaNDUr+nHgi+9ZOIt6ii7r9u 31ZrH9A10faWrunZ6zBylgkB/z7KcQdK1/sxqv0YPb2Gk0zJ3dMljuy9OOBhTTgbc7187+j 42IhwYl3SgihEdyV2Imt+ved2bJ1VkV87InU7amG9TKhl9WDT79BnZHqmrqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=ffmpeg.org; s=arc; t=1760283806; h=from : sender : reply-to : subject : date : message-id : to : cc : mime-version : content-type : content-transfer-encoding : content-id : content-description : resent-date : resent-from : resent-sender : resent-to : resent-cc : resent-message-id : in-reply-to : references : list-id : list-help : list-unsubscribe : list-subscribe : list-post : list-owner : list-archive; bh=uTI71y/OMIZ2VQmpXxx9Uoc2OmL6C4yl0TVi05dfcao=; b=Abj/26Zmss4xzkUiP4y38TY5a0xI90waEWAMeQ07+lSr375ZmDYl67b+9whvPhbmjYLoc VBmJZlAjk9UEmXpgYGKxaRCThwFJ0mLXn9tQBmondNL6h5pZ0Ra8OoVZrewkVwE3kBaTiAp gH1gVb8FghcjQkYe7F8TduRF/MOxIkofsUkT/PXhTPsd0oTChwAXpJ7OTpR1vxXLKrcKafj dG/JaBqK6FsAZTUvEoIPKPdVSh5jdU1j7Q83uDU7lTNF4kELe3rJ7GAsgtDWwx/GXKz8OPZ t6u0tU9xbkdMpmn9Icg4/fxkHAkBrPE9Zp0LI3RGbRyMtKx/ALK/60nTpFwA== ARC-Authentication-Results: i=1; ffmpeg.org; dkim=pass header.d=qq.com; arc=none; dmarc=pass header.from=qq.com policy.dmarc=quarantine Authentication-Results: ffmpeg.org; dkim=pass header.d=qq.com; arc=none (Message is not ARC signed); dmarc=pass (Used From Domain Record) header.from=qq.com policy.dmarc=quarantine Received: from out162-62-57-87.mail.qq.com (out162-62-57-87.mail.qq.com [162.62.57.87]) by ffbox0-bg.ffmpeg.org (Postfix) with UTF8SMTP id 6653A68F2BD for ; Sun, 12 Oct 2025 18:42:55 +0300 (EEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201512; t=1760283758; bh=ZXDS6PTD0UllxWZbjggetNrM0veMJCrLACZMrC71l1g=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=TmGYXrTlJHThBt5W8Zi3E+wvucmpyytcQcubzJmQTnhkLWFFhBwh69UE8ZlcC8ex1 8fcqiWqaKFO26C7TeDpLY47M0sHPonxP+9MXvxMaFKA2fZp/r3b2W87S/91tSvbmgY VfPtEJJikxPVwtBpoWZ1pDy6LJCJUB+EDrm30phQ= Received: from Cave.localdomain ([3.25.94.32]) by newxmesmtplogicsvrszc13-0.qq.com (NewEsmtp) with SMTP id AA13B697; Sun, 12 Oct 2025 23:42:33 +0800 X-QQ-mid: xmsmtpt1760283753t5qdemcg1 Message-ID: X-QQ-XMAILINFO: NZYNQwNkdR3OzssqX0h58ladzwaJozyj4PJuBLtK1jxzTbyJDDDdBr/7OJbiG5 Gn/t2rt7Z+RflJtWrtx/Ck4/ACoIbHRub90Q/03xf9AQfAZ1V8IdD1NscnInwjVneiS5b2jT1vDo BVMjbLjrm9UMYtRrcGprsYIF6zZywDAbjbk0+NN8VbFAIhl+rtVOJr/R+TXHvC51mtU4507xJzOq y49HHXCn4JivyDiEe+o5yuehB0JLdFs8GQGtXq+57TYGURLX3ZaOznPiCAo4dykmu9Ddf9uj4LGa Fv4VVZZR2JmmNmrPukuNce5LKwvLVIfFAr4+Pz4yV/8F5x0N3lSGrwvFWX+Nbqy6JRM/+z96FKKk 8dkQqTVZi03lmgO7tvJm5TdaNhbZjejFGvphFh2FOczOkUkIezmkG++UfDGARjDCXxQT74QJHek8 t3IoSDbiGlXwqfNZey2WV9OnXs7o2J6isMYzSbTBwFuELU8BJphjqPHXPIzgNOs84enQK0VOyP6U i2+4hiESWLrBZx2qpl6ZIgi+OEArStbUsTACVONKZS0AChVWSpKkAFmlJEBDVeyCtSs2YnEYkg3r p3IIKkdYmWUAdXAauO2SoYcLLkC3BJ/qwSr4JZUxuN8CJpePi0NePcBbnVTjZh28xBT4BIVaFEFC UIkaRBQfDdBNloxlJWdYmsgCp6KAGNqR4KKbMAFTWrvnkSfPeurXhKfke10FzERAsgmDsBjAN1SC R1BdXcJOn/GHgZo1+tTz8EWv1OhHrWIQiQiGLPZHRCcDgrljNttBzEyZUivPd9AzP27oluxvJ25h eRVfDxV58n0qXVz0FshLiDfEICqhix21agWSoxzDDSLrKDeK7nlbsLBQ3vvSi3OGSzgROY4aOR17 DJvCbj9Y/f34+MsP1cDZAxKSYJE4UHZ/xTd8G0SAefgjmKHJWEwx6RYQoxuoFIeM8tBxUfTBeb/D /QYhqrmMrnG0UCCV3siMJCsTA5pani14EFVvBREl5KrJNTEQ1rsq1gVGqAzrDgtvKAD9vXCKEt8H 871Hlmx9GkqXVEsd5JDHVWn88ovrCYX9ZA8JRu2TPGGoseQ0LL X-QQ-XMRINFO: Mp0Kj//9VHAxr69bL5MkOOs= To: ffmpeg-devel@ffmpeg.org Date: Sun, 12 Oct 2025 23:42:28 +0800 X-OQ-MSGID: <20251012154228.1032135-1-1007668733@qq.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20251012152347.1022477-1-1007668733@qq.com> References: <20251012152347.1022477-1-1007668733@qq.com> MIME-Version: 1.0 Message-ID-Hash: BYSE4ATHWNZK7AE5M4UITCAACQK5JAAN X-Message-ID-Hash: BYSE4ATHWNZK7AE5M4UITCAACQK5JAAN X-MailFrom: SRS0=CIIc=4V=qq.com=1007668733@ffmpeg.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; header-match-ffmpeg-devel.ffmpeg.org-0; header-match-ffmpeg-devel.ffmpeg.org-1; header-match-ffmpeg-devel.ffmpeg.org-2; header-match-ffmpeg-devel.ffmpeg.org-3; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Reply-To: FFmpeg development discussions and patches Subject: [FFmpeg-devel] [PATCH 3/3] avformat/whip whep: add whep support List-Id: FFmpeg development discussions and patches Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: baigao via ffmpeg-devel Cc: baigao <1007668733@qq.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Archived-At: List-Archive: List-Post: --- libavformat/Makefile | 1 + libavformat/allformats.c | 1 + libavformat/rtc.c | 895 +++++++++++++++++++++++++++++++++++++-- libavformat/rtc.h | 38 +- libavformat/rtpdec.c | 6 +- libavformat/rtpdec.h | 11 + libavformat/whep.c | 457 ++++++++++++++++++++ libavformat/whip.c | 52 +-- 8 files changed, 1373 insertions(+), 88 deletions(-) create mode 100644 libavformat/whep.c diff --git a/libavformat/Makefile b/libavformat/Makefile index 9261245755..dadc1321b1 100644 --- a/libavformat/Makefile +++ b/libavformat/Makefile @@ -640,6 +640,7 @@ OBJS-$(CONFIG_WEBM_CHUNK_MUXER) += webm_chunk.o OBJS-$(CONFIG_WEBP_MUXER) += webpenc.o OBJS-$(CONFIG_WEBVTT_DEMUXER) += webvttdec.o subtitles.o OBJS-$(CONFIG_WEBVTT_MUXER) += webvttenc.o +OBJS-$(CONFIG_WHEP_DEMUXER) += whep.o rtc.o avc.o http.o srtp.o OBJS-$(CONFIG_WHIP_MUXER) += whip.o rtc.o avc.o http.o srtp.o OBJS-$(CONFIG_WSAUD_DEMUXER) += westwood_aud.o OBJS-$(CONFIG_WSAUD_MUXER) += westwood_audenc.o diff --git a/libavformat/allformats.c b/libavformat/allformats.c index 3a025da3db..cd7e3cc4c4 100644 --- a/libavformat/allformats.c +++ b/libavformat/allformats.c @@ -518,6 +518,7 @@ extern const FFOutputFormat ff_webp_muxer; extern const FFInputFormat ff_webvtt_demuxer; extern const FFOutputFormat ff_webvtt_muxer; extern const FFInputFormat ff_wsaud_demuxer; +extern const FFInputFormat ff_whep_demuxer; extern const FFOutputFormat ff_whip_muxer; extern const FFOutputFormat ff_wsaud_muxer; extern const FFInputFormat ff_wsd_demuxer; diff --git a/libavformat/rtc.c b/libavformat/rtc.c index 8c848b6026..57da5487b4 100644 --- a/libavformat/rtc.c +++ b/libavformat/rtc.c @@ -19,6 +19,11 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include "libavcodec/avcodec.h" +#include "libavcodec/codec_desc.h" +#include "libavcodec/defs.h" +#include "libavcodec/h264.h" +#include "libavcodec/h264_levels.h" #include "libavutil/time.h" #include "libavutil/intreadwrite.h" #include "libavutil/random_seed.h" @@ -26,18 +31,21 @@ #include "libavutil/hmac.h" #include "libavutil/mem.h" #include "libavutil/base64.h" +#include "libavutil/parseutils.h" #include "avio_internal.h" #include "internal.h" #include "network.h" #include "http.h" #include "rtc.h" +#include "rtp.h" +#include "rtpdec.h" /** * Maximum size limit of a Session Description Protocol (SDP), * be it an offer or answer. */ -#define MAX_SDP_SIZE 8192 +#define MAX_SDP_SIZE 16384 /** * The size of the Secure Real-time Transport Protocol (SRTP) master key material @@ -87,20 +95,31 @@ #define DTLS_VERSION_10 0xfeff #define DTLS_VERSION_12 0xfefd -/** - * Maximum size of the buffer for sending and receiving UDP packets. - * Please note that this size does not limit the size of the UDP packet that can be sent. - * To set the limit for packet size, modify the `pkt_size` parameter. - * For instance, it is possible to set the UDP buffer to 4096 to send or receive packets, - * but please keep in mind that the `pkt_size` option limits the packet size to 1400. - */ -#define MAX_UDP_BUFFER_SIZE 4096 - /* Referring to Chrome's definition of RTP payload types. */ #define RTC_RTP_PAYLOAD_TYPE_H264 106 #define RTC_RTP_PAYLOAD_TYPE_OPUS 111 #define RTC_RTP_PAYLOAD_TYPE_VIDEO_RTX 105 + /** + * The RTP header is 12 bytes long, comprising the Version(1B), PT(1B), + * SequenceNumber(2B), Timestamp(4B), and SSRC(4B). + * See https://www.rfc-editor.org/rfc/rfc3550#section-5.1 + */ +#define RTC_RTP_HEADER_SIZE 12 + +/** + * For RTCP, PT is [128, 223] (or without marker [0, 95]). Literally, RTCP starts + * from 64 not 0, so PT is [192, 223] (or without marker [64, 95]), see "RTCP Control + * Packet Types (PT)" at + * https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-4 + * + * For RTP, the PT is [96, 127], or [224, 255] with marker. See "RTP Payload Types (PT) + * for standard audio and video encodings" at + * https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-1 + */ +#define RTC_RTCP_PT_START 192 +#define RTC_RTCP_PT_END 223 + /** * The STUN message header, which is 20 bytes long, comprises the * STUNMessageType (1B), MessageLength (2B), MagicCookie (4B), @@ -129,6 +148,33 @@ enum STUNAttr { STUN_ATTR_ICE_CONTROLLING = 0x802A, /// ICE controlling role }; +#define OFFSET(x) offsetof(RTCContext, x) +#define ENC AV_OPT_FLAG_ENCODING_PARAM +#define DEC AV_OPT_FLAG_DECODING_PARAM +const AVOption ff_rtc_options[] = { + { "handshake_timeout", "Timeout in milliseconds for ICE and DTLS handshake.", OFFSET(handshake_timeout), AV_OPT_TYPE_INT, { .i64 = 5000 }, -1, INT_MAX, ENC|DEC }, + { "pkt_size", "The maximum size, in bytes, of RTP packets that send out", OFFSET(pkt_size), AV_OPT_TYPE_INT, { .i64 = 1200 }, -1, INT_MAX, ENC|DEC }, + { "buffer_size", "The buffer size, in bytes, of underlying protocol", OFFSET(buffer_size), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, ENC|DEC }, + { "authorization", "The optional Bearer token for RTC Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC|DEC }, + { "cert_file", "The optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC|DEC }, + { "key_file", "The optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC|DEC }, + { NULL }, +}; + +/** + * In RTP packets, the first byte is represented as 0b10xxxxxx + */ +int ff_rtc_media_is_rtp_rtcp(const uint8_t *b, int size) +{ + return size >= RTC_RTP_HEADER_SIZE && (b[0] & 0xC0) == 0x80; +} + +/* Whether the packet is RTCP. */ +int ff_rtc_media_is_rtcp(const uint8_t *b, int size) +{ + return size >= RTC_RTP_HEADER_SIZE && b[1] >= RTC_RTCP_PT_START && b[1] <= RTC_RTCP_PT_END; +} + /** * Whether the packet is a DTLS packet. */ @@ -139,6 +185,38 @@ int ff_rtc_is_dtls_packet(uint8_t *b, int size) { (version == DTLS_VERSION_10 || version == DTLS_VERSION_12); } +static void get_word_until_chars(char *buf, int buf_size, + const char *sep, const char **pp) +{ + const char *p; + char *q; + + p = *pp; + p += strspn(p, SPACE_CHARS); + q = buf; + while (!strchr(sep, *p) && *p != '\0') { + if ((q - buf) < buf_size - 1) + *q++ = *p; + p++; + } + if (buf_size > 0) + *q = '\0'; + *pp = p; +} + +static void get_word_sep(char *buf, int buf_size, const char *sep, + const char **pp) +{ + if (**pp == '/') (*pp)++; + get_word_until_chars(buf, buf_size, sep, pp); +} + + +static void get_word(char *buf, int buf_size, const char **pp) +{ + get_word_until_chars(buf, buf_size, SPACE_CHARS, pp); +} + /** * Get or Generate a self-signed certificate and private key for DTLS, * fingerprint for SDP @@ -215,6 +293,14 @@ av_cold int ff_rtc_initialize(AVFormatContext *s) rtc->audio_first_seq = av_lfg_get(&rtc->rnd) & 0x0fff; rtc->video_first_seq = rtc->audio_first_seq + 1; + /* Allocate UDP buffer */ + rtc->bufsize = MAX_UDP_BUFFER_SIZE; + rtc->buf = av_malloc(rtc->bufsize); + if (!rtc->buf) { + av_log(rtc, AV_LOG_ERROR, "Failed to allocate UDP buffer\n"); + return AVERROR(ENOMEM); + } + if (rtc->pkt_size < ideal_pkt_size) av_log(rtc, AV_LOG_WARNING, "pkt_size=%d(<%d) is too small, may cause packet loss\n", rtc->pkt_size, ideal_pkt_size); @@ -229,14 +315,14 @@ av_cold int ff_rtc_initialize(AVFormatContext *s) } /** - * Generate SDP offer according to the codec parameters, DTLS and ICE information. + * Generate demux SDP offer according to the codec parameters, DTLS and ICE information. * * Note that we don't use av_sdp_create to generate SDP offer because it doesn't * support DTLS and ICE information. * * @return 0 if OK, AVERROR_xxx on error */ -static int generate_sdp_offer(AVFormatContext *s) +static int generate_muxer_sdp_offer(AVFormatContext *s) { int ret = 0, profile_idc = 0, level, profile_iop = 0; const char *acodec_name = NULL, *vcodec_name = NULL; @@ -379,6 +465,478 @@ end: return ret; } +static char *generate_h264_fmtp(int profile_idc, int level_idc, int packetization_mode) +{ + char *fmtp; + int profile_level_id = (profile_idc << 16) | (level_idc & 0xFF); + fmtp = av_asprintf("level-asymmetry-allowed=1;packetization-mode=%d;profile-level-id=%06x", + packetization_mode, profile_level_id); + return fmtp; +} + +static char *generate_h265_fmtp(int profile_id, int tier_flag, int level_idc) +{ + char *fmtp; + fmtp = av_asprintf("profile-space=0;profile-id=%d;tier-flag=%d;level-id=%d", + profile_id, tier_flag, level_idc); + return fmtp; +} + +/** + * Structure to hold codec information for SDP generation + */ +typedef struct CodecInfo { + const char *enc_name; + enum AVCodecID codec_id; + int payload_type; + int clock_rate; + int channels; + char *fmtp; +} CodecInfo; + +/** + * Add multiple H264 codec configurations with different profiles. + */ +static int add_h264_codec_variants(CodecInfo **codecs, int *codec_count, int max_codecs, + const char *enc_name, int *pt) +{ + struct { + int profile_idc; + int level_idc; + } h264_profiles[] = { + {AV_PROFILE_H264_BASELINE, 0x1f}, // Baseline Profile, Level 3.1 + {AV_PROFILE_H264_MAIN, 0x1f}, // Main Profile, Level 3.1 + {AV_PROFILE_H264_HIGH, 0x1f}, // High Profile, Level 3.1 + }; + + for (int i = 0; i < FF_ARRAY_ELEMS(h264_profiles); i++) { + if (*codec_count >= max_codecs) + break; + + (*codecs)[*codec_count].enc_name = enc_name; + (*codecs)[*codec_count].codec_id = AV_CODEC_ID_H264; + (*codecs)[*codec_count].payload_type = (*pt)++; + (*codecs)[*codec_count].clock_rate = 90000; + (*codecs)[*codec_count].channels = 0; + (*codecs)[*codec_count].fmtp = generate_h264_fmtp(h264_profiles[i].profile_idc, + h264_profiles[i].level_idc, 1); + (*codec_count)++; + } + + return 0; +} + +/** + * Add multiple H265/HEVC codec configurations with different profiles. + */ +static int add_h265_codec_variants(CodecInfo **codecs, int *codec_count, int max_codecs, + const char *enc_name, int *pt) +{ + struct { + int profile_id; + int tier_flag; + int level_idc; + } hevc_profiles[] = { + {AV_PROFILE_HEVC_MAIN, 0, 93}, // Main Profile, Main Tier, Level 3.1 + {AV_PROFILE_HEVC_MAIN_10, 0, 93}, // Main 10 Profile, Main Tier, Level 3.1 + {AV_PROFILE_HEVC_SCC, 0, 93}, // Screen Content Coding, Main Tier, Level 3.1 + }; + + for (int i = 0; i < FF_ARRAY_ELEMS(hevc_profiles); i++) { + if (*codec_count >= max_codecs) + break; + + (*codecs)[*codec_count].enc_name = enc_name; + (*codecs)[*codec_count].codec_id = AV_CODEC_ID_HEVC; + (*codecs)[*codec_count].payload_type = (*pt)++; + (*codecs)[*codec_count].clock_rate = 90000; + (*codecs)[*codec_count].channels = 0; + (*codecs)[*codec_count].fmtp = generate_h265_fmtp(hevc_profiles[i].profile_id, + hevc_profiles[i].tier_flag, + hevc_profiles[i].level_idc); + (*codec_count)++; + } + + return 0; +} + +/** + * Get basic RTP parameters for non-H264/H265 codecs. + */ +static void get_basic_codec_info(enum AVCodecID codec_id, enum AVMediaType codec_type, + int *clock_rate, int *channels) +{ + *clock_rate = 0; + *channels = 0; + + if (codec_type == AVMEDIA_TYPE_VIDEO) { + *clock_rate = 90000; + } else if (codec_type == AVMEDIA_TYPE_AUDIO) { + switch (codec_id) { + case AV_CODEC_ID_OPUS: + *clock_rate = 48000; + *channels = 2; + break; + case AV_CODEC_ID_PCM_MULAW: + case AV_CODEC_ID_PCM_ALAW: + *clock_rate = 8000; + *channels = 1; + break; + case AV_CODEC_ID_ADPCM_G722: + *clock_rate = 8000; + *channels = 1; + break; + } + } +} + +/** + * Check if a codec is compatible with WebRTC/WHEP. + * only supports a subset of RTP codecs now. + */ +static int is_rtc_compatible_codec(enum AVCodecID codec_id, enum AVMediaType codec_type) +{ + if (codec_type == AVMEDIA_TYPE_VIDEO) { + switch (codec_id) { + case AV_CODEC_ID_VP8: + case AV_CODEC_ID_VP9: + case AV_CODEC_ID_H264: + case AV_CODEC_ID_AV1: + case AV_CODEC_ID_HEVC: + return 1; + default: + return 0; + } + } else if (codec_type == AVMEDIA_TYPE_AUDIO) { + switch (codec_id) { + case AV_CODEC_ID_OPUS: + case AV_CODEC_ID_PCM_MULAW: + case AV_CODEC_ID_PCM_ALAW: + case AV_CODEC_ID_ADPCM_G722: + return 1; + default: + return 0; + } + } + return 0; +} + +/** + * Collect all supported audio or video codecs that can be used in WHEP SDP offer. + */ +static int collect_supported_codecs(enum AVMediaType codec_type, CodecInfo **codec_list, int *count) +{ + CodecInfo *codecs = NULL; + int codec_count = 0; + int max_codecs = 100; + int pt = RTP_PT_PRIVATE; + void *opaque = NULL; + const RTPDynamicProtocolHandler *handler; + AVCodecParameters par; + int i; + int added_codecs[256] = {0}; + + codecs = av_mallocz(max_codecs * sizeof(CodecInfo)); + if (!codecs) + return AVERROR(ENOMEM); + + for (i = 0; i < RTP_PT_PRIVATE; i++) { + if (codec_count >= max_codecs) + break; + + memset(&par, 0, sizeof(par)); + if (ff_rtp_get_codec_info(&par, i) != 0) + continue; + + if (par.codec_type != codec_type) + continue; + + if (par.codec_id != AV_CODEC_ID_NONE) + continue; + + if (!is_rtc_compatible_codec(par.codec_id, codec_type)) + continue; + + if (added_codecs[par.codec_id]) + continue; + + codecs[codec_count].enc_name = ff_rtp_enc_name(i); + codecs[codec_count].codec_id = par.codec_id; + codecs[codec_count].payload_type = i; + + int clock_rate, channels; + get_basic_codec_info(par.codec_id, codec_type, &clock_rate, &channels); + codecs[codec_count].clock_rate = par.sample_rate > 0 ? par.sample_rate : clock_rate; + codecs[codec_count].channels = par.ch_layout.nb_channels > 0 ? par.ch_layout.nb_channels : channels; + codecs[codec_count].fmtp = NULL; + + added_codecs[par.codec_id] = 1; + codec_count++; + } + + opaque = NULL; + while ((handler = ff_rtp_handler_iterate(&opaque))) { + if (codec_count >= max_codecs) + break; + + if (handler->codec_type != codec_type) + continue; + + if (handler->codec_id == AV_CODEC_ID_NONE) + continue; + + if (!is_rtc_compatible_codec(handler->codec_id, codec_type)) + continue; + + if (added_codecs[handler->codec_id]) + continue; + + if (handler->codec_id == AV_CODEC_ID_H264) { + add_h264_codec_variants(&codecs, &codec_count, max_codecs, handler->enc_name, &pt); + } else if (handler->codec_id == AV_CODEC_ID_HEVC) { + add_h265_codec_variants(&codecs, &codec_count, max_codecs, handler->enc_name, &pt); + } else { + int payload_type = handler->static_payload_id > 0 ? + handler->static_payload_id : pt++; + + codecs[codec_count].enc_name = handler->enc_name; + codecs[codec_count].codec_id = handler->codec_id; + codecs[codec_count].payload_type = payload_type; + + int clock_rate, channels; + get_basic_codec_info(handler->codec_id, codec_type, &clock_rate, &channels); + codecs[codec_count].clock_rate = clock_rate; + codecs[codec_count].channels = channels; + codecs[codec_count].fmtp = NULL; + + codec_count++; + } + added_codecs[handler->codec_id] = 1; + } + + *codec_list = codecs; + *count = codec_count; + return 0; +} + +/** + * Generate demuxer SDP offer according to the codec parameters, DTLS and ICE information. + * + * Note that we don't use av_sdp_create to generate SDP offer because it doesn't + * support DTLS and ICE information. + * + * @return 0 if OK, AVERROR_xxx on error + */ +static int generate_demuxer_sdp_offer(AVFormatContext *s) +{ + int ret = 0, i; + AVBPrint bp; + RTCContext *rtc = s->priv_data; + CodecInfo *audio_codecs = NULL, *video_codecs = NULL; + int audio_count = 0, video_count = 0; + + av_bprint_init(&bp, 1, MAX_SDP_SIZE); + + if (rtc->sdp_offer) { + av_log(rtc, AV_LOG_ERROR, "SDP offer is already set\n"); + ret = AVERROR(EINVAL); + goto end; + } + + /* Generate SDP header */ + av_bprintf(&bp, "" + "v=0\r\n" + "o=FFmpeg %s 2 IN IP4 %s\r\n" + "s=FFmpegReceiveSession\r\n" + "t=0 0\r\n" + "a=group:BUNDLE 0 1\r\n" + "a=extmap-allow-mixed\r\n" + "a=msid-semantic: WMS\r\n", + RTC_SDP_SESSION_ID, + RTC_SDP_CREATOR_IP); + + snprintf(rtc->ice_ufrag_local, sizeof(rtc->ice_ufrag_local), "%08x", + av_lfg_get(&rtc->rnd)); + snprintf(rtc->ice_pwd_local, sizeof(rtc->ice_pwd_local), "%08x%08x%08x%08x", + av_lfg_get(&rtc->rnd), av_lfg_get(&rtc->rnd), av_lfg_get(&rtc->rnd), + av_lfg_get(&rtc->rnd)); + + ret = collect_supported_codecs(AVMEDIA_TYPE_VIDEO, &video_codecs, &video_count); + if (ret < 0) { + av_log(rtc, AV_LOG_ERROR, "Failed to collect video codecs\n"); + goto end; + } + + ret = collect_supported_codecs(AVMEDIA_TYPE_AUDIO, &audio_codecs, &audio_count); + if (ret < 0) { + av_log(rtc, AV_LOG_ERROR, "Failed to collect audio codecs\n"); + goto end; + } + + if (video_count > 0) { + int rtx_pt_start = RTP_PT_PRIVATE + 50; + + av_bprintf(&bp, "m=video 9 UDP/TLS/RTP/SAVPF"); + for (i = 0; i < video_count; i++) { + av_bprintf(&bp, " %u %u", video_codecs[i].payload_type, rtx_pt_start + i); + } + + av_bprintf(&bp, "\r\n" + "c=IN IP4 0.0.0.0\r\n" + "a=ice-ufrag:%s\r\n" + "a=ice-pwd:%s\r\n" + "a=fingerprint:sha-256 %s\r\n" + "a=setup:passive\r\n" + "a=mid:0\r\n", + rtc->ice_ufrag_local, + rtc->ice_pwd_local, + rtc->dtls_fingerprint); + + av_bprintf(&bp, + "a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\n" + "a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\n" + // "a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\n" + "a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\n" + // "a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\n" + // "a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\n" + // "a=extmap:10 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07\r\n" + // "a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\n" + // "a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\n" + // "a=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\n" + ); + + av_bprintf(&bp, + "a=recvonly\r\n" + "a=rtcp-mux\r\n" + "a=rtcp-rsize\r\n"); + + for (i = 0; i < video_count; i++) { + int rtx_pt = rtx_pt_start + i; + + av_bprintf(&bp, "a=rtpmap:%u %s/%d\r\n", + video_codecs[i].payload_type, + video_codecs[i].enc_name, + video_codecs[i].clock_rate); + + av_bprintf(&bp, "a=rtcp-fb:%u ccm fir\r\n", video_codecs[i].payload_type); + av_bprintf(&bp, "a=rtcp-fb:%u nack\r\n", video_codecs[i].payload_type); + av_bprintf(&bp, "a=rtcp-fb:%u nack pli\r\n", video_codecs[i].payload_type); + // av_bprintf(&bp, "a=rtcp-fb:%u goog-remb\r\n", video_codecs[i].payload_type); + // av_bprintf(&bp, "a=rtcp-fb:%u transport-cc\r\n", video_codecs[i].payload_type); + + if (video_codecs[i].fmtp) { + av_bprintf(&bp, "a=fmtp:%u %s\r\n", + video_codecs[i].payload_type, + video_codecs[i].fmtp); + } + + av_bprintf(&bp, "a=rtpmap:%u rtx/%d\r\n", rtx_pt, video_codecs[i].clock_rate); + av_bprintf(&bp, "a=fmtp:%u apt=%u\r\n", rtx_pt, video_codecs[i].payload_type); + } + } + + if (audio_count > 0) { + av_bprintf(&bp, "m=audio 9 UDP/TLS/RTP/SAVPF"); + for (i = 0; i < audio_count; i++) { + av_bprintf(&bp, " %u", audio_codecs[i].payload_type); + } + + av_bprintf(&bp, "\r\n" + "c=IN IP4 0.0.0.0\r\n" + "a=ice-ufrag:%s\r\n" + "a=ice-pwd:%s\r\n" + "a=fingerprint:sha-256 %s\r\n" + "a=setup:passive\r\n" + "a=mid:1\r\n", + rtc->ice_ufrag_local, + rtc->ice_pwd_local, + rtc->dtls_fingerprint); + + av_bprintf(&bp, + "a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\n" + "a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\n" + "a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\n" + // "a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\n" + "a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\n"); + + av_bprintf(&bp, + "a=recvonly\r\n" + "a=rtcp-mux\r\n"); + + for (i = 0; i < audio_count; i++) { + if (audio_codecs[i].channels > 0) { + av_bprintf(&bp, "a=rtpmap:%u %s/%d/%d\r\n", + audio_codecs[i].payload_type, + audio_codecs[i].enc_name, + audio_codecs[i].clock_rate, + audio_codecs[i].channels); + } else { + av_bprintf(&bp, "a=rtpmap:%u %s/%d\r\n", + audio_codecs[i].payload_type, + audio_codecs[i].enc_name, + audio_codecs[i].clock_rate); + } + + // av_bprintf(&bp, "a=rtcp-fb:%u goog-remb\r\n", audio_codecs[i].payload_type); + // av_bprintf(&bp, "a=rtcp-fb:%u transport-cc\r\n", audio_codecs[i].payload_type); + + if (audio_codecs[i].fmtp) { + av_bprintf(&bp, "a=fmtp:%u %s\r\n", + audio_codecs[i].payload_type, + audio_codecs[i].fmtp); + } + } + } + + if (!av_bprint_is_complete(&bp)) { + av_log(rtc, AV_LOG_ERROR, "Offer exceed max %d, %s\n", MAX_SDP_SIZE, bp.str); + ret = AVERROR(EIO); + goto end; + } + + rtc->sdp_offer = av_strdup(bp.str); + if (!rtc->sdp_offer) { + ret = AVERROR(ENOMEM); + goto end; + } + + if (rtc->state < RTC_STATE_OFFER) + rtc->state = RTC_STATE_OFFER; + rtc->rtc_offer_time = av_gettime_relative(); + av_log(rtc, AV_LOG_VERBOSE, "Generated demuxer state=%d, offer with %d audio and %d video codecs: %s\n", + rtc->state, audio_count, video_count, rtc->sdp_offer); + +end: + for (i = 0; i < audio_count; i++) { + if (audio_codecs) + av_freep(&audio_codecs[i].fmtp); + } + for (i = 0; i < video_count; i++) { + if (video_codecs) + av_freep(&video_codecs[i].fmtp); + } + av_freep(&audio_codecs); + av_freep(&video_codecs); + av_bprint_finalize(&bp, NULL); + return ret; +} + +/** + * Generate SDP offer according to the codec parameters, DTLS and ICE information. + * + * Note that we don't use av_sdp_create to generate SDP offer because it doesn't + * support DTLS and ICE information. + * + * @return 0 if OK, AVERROR_xxx on error + */ +static int generate_sdp_offer(AVFormatContext *s) { + if (s->iformat) { + return generate_demuxer_sdp_offer(s); + } else { + return generate_muxer_sdp_offer(s); + } +} + /** * Exchange SDP offer with WebRTC peer to get the answer. * @@ -503,7 +1061,7 @@ end: * @param s Pointer to the AVFormatContext * @returns Returns 0 if successful or AVERROR_xxx if an error occurs. */ -static int parse_answer(AVFormatContext *s) +static int parse_answer_ice(AVFormatContext *s) { int ret = 0; AVIOContext *pb; @@ -599,6 +1157,263 @@ end: return ret; } +/** + * SDP parsing state for demuxer + */ +typedef struct SDPParseState { + RTCStreamInfo *current_stream_info; // Current stream being parsed +} SDPParseState; + +static void sdp_parse_line(AVFormatContext *s, SDPParseState *state, + int letter, const char *buf) +{ + RTCContext *rtc = s->priv_data; + RTCStreamInfo *stream_info = state->current_stream_info; + const char *p = buf; + char word[256]; + + av_log(s, AV_LOG_TRACE, "sdp: %c='%s'\n", letter, buf); + + if (!stream_info && letter != 'm') + return; + + switch (letter) { + case 'm': { + /* m= */ + char media_type[64]; + enum AVMediaType codec_type; + int payload_type; + + get_word(media_type, sizeof(media_type), &p); + if (!strcmp(media_type, "audio")) { + codec_type = AVMEDIA_TYPE_AUDIO; + } else if (!strcmp(media_type, "video")) { + codec_type = AVMEDIA_TYPE_VIDEO; + } else { + state->current_stream_info = NULL; + return; + } + + get_word(word, sizeof(word), &p); + get_word(word, sizeof(word), &p); + get_word(word, sizeof(word), &p); + payload_type = atoi(word); + + stream_info = av_mallocz(sizeof(RTCStreamInfo)); + if (!stream_info) + return; + stream_info->payload_type = payload_type; + stream_info->codec_type = codec_type; + stream_info->codec_name = NULL; + stream_info->clock_rate = 0; + stream_info->channels = (codec_type == AVMEDIA_TYPE_AUDIO) ? 1 : 0; + stream_info->fmtp = NULL; + stream_info->direction = NULL; + stream_info->ssrc = 0; + stream_info->rtx_pt = -1; + stream_info->rtx_ssrc = 0; + + RTCStreamInfo **new_array = av_realloc_array(rtc->stream_infos, + rtc->nb_stream_infos + 1, + sizeof(RTCStreamInfo*)); + if (!new_array) { + av_freep(&stream_info); + return; + } + rtc->stream_infos = new_array; + rtc->stream_infos[rtc->nb_stream_infos] = stream_info; + rtc->nb_stream_infos++; + state->current_stream_info = stream_info; + av_log(s, AV_LOG_VERBOSE, "Parsed stream info %d: type=%s, pt=%d\n", + rtc->nb_stream_infos - 1, av_get_media_type_string(codec_type), payload_type); + break; + } + case 'a': { + if (av_strstart(buf, "rtpmap:", &p)) { + /* a=rtpmap: /[/] */ + char codec_name[256]; + int pt, clock_rate, channels = 1; + + get_word(word, sizeof(word), &p); + pt = atoi(word); + get_word_sep(codec_name, sizeof(codec_name), "/", &p); + get_word_sep(word, sizeof(word), "/", &p); + clock_rate = atoi(word); + if (*p == '/') { + p++; + get_word(word, sizeof(word), &p); + channels = atoi(word); + } + + if (!av_strcasecmp(codec_name, "rtx")) { + stream_info->rtx_pt = pt; + av_log(s, AV_LOG_VERBOSE, "Found RTX rtpmap: pt=%d for stream %d\n", + pt, rtc->nb_stream_infos - 1); + break; + } + + if (pt == stream_info->payload_type) { + stream_info->codec_name = av_strdup(codec_name); + stream_info->clock_rate = clock_rate; + + if (stream_info->codec_type == AVMEDIA_TYPE_AUDIO) { + stream_info->channels = channels; + } + av_log(s, AV_LOG_VERBOSE, "Parsed main stream rtpmap: type=%s, codec=%s, pt=%d, rate=%d\n", + av_get_media_type_string(stream_info->codec_type), codec_name, pt, clock_rate); + } + + } else if (av_strstart(buf, "fmtp:", &p)) { + /* a=fmtp: */ + int pt; + + get_word(word, sizeof(word), &p); + pt = atoi(word); + if (pt == stream_info->payload_type) { + stream_info->fmtp = av_strdup(p); + av_log(s, AV_LOG_VERBOSE, "Stored fmtp for stream info %d: %s\n", + rtc->nb_stream_infos - 1, p); + } + + } else if (av_strstart(buf, "ssrc-group:FID ", &p)) { + /* a=ssrc-group:FID */ + uint32_t ssrc, rtx_ssrc; + + get_word(word, sizeof(word), &p); + ssrc = strtoul(word, NULL, 10); + get_word(word, sizeof(word), &p); + rtx_ssrc = strtoul(word, NULL, 10); + stream_info->ssrc = ssrc; + stream_info->rtx_ssrc = rtx_ssrc; + av_log(s, AV_LOG_VERBOSE, "Stream info %d: ssrc-group FID %u, rtx=%u\n", + rtc->nb_stream_infos - 1, ssrc, rtx_ssrc); + + } else if (av_strstart(buf, "ssrc:", &p)) { + /* a=ssrc: [...] */ + uint32_t ssrc; + + get_word(word, sizeof(word), &p); + ssrc = strtoul(word, NULL, 10); + if (stream_info->ssrc == 0) { + stream_info->ssrc = ssrc; + av_log(s, AV_LOG_VERBOSE, "Stream info %d: main ssrc=%u\n", + rtc->nb_stream_infos - 1, ssrc); + } + } else if (!strcmp(buf, "sendrecv") || !strcmp(buf, "sendonly") || + !strcmp(buf, "recvonly") || !strcmp(buf, "inactive")) { + /* direction */ + /* a=sendrecv / a=sendonly / a=recvonly / a=inactive */ + av_freep(&stream_info->direction); + stream_info->direction = av_strdup(buf); + av_log(s, AV_LOG_VERBOSE, "Stream info %d: direction=%s\n", + rtc->nb_stream_infos - 1, buf); + } + break; + } + default: + break; + } +} + +/** + * Parse media information from SDP answer for demuxer. + */ +static int parse_answer_media(AVFormatContext *s) +{ + int ret = 0; + RTCContext *rtc = s->priv_data; + const char *p; + int letter; + char line[MAX_URL_SIZE], *q; + SDPParseState state = {0}; + + if (!rtc->sdp_answer || !strlen(rtc->sdp_answer)) { + av_log(rtc, AV_LOG_ERROR, "No answer to parse for media\n"); + return AVERROR(EINVAL); + } + + rtc->stream_infos = NULL; + rtc->nb_stream_infos = 0; + + p = rtc->sdp_answer; + for (;;) { + p += strspn(p, SPACE_CHARS); + letter = *p; + if (letter == '\0') + break; + p++; + if (*p != '=') + goto next_line; + p++; + /* get the content */ + q = line; + while (*p != '\n' && *p != '\r' && *p != '\0') { + if ((q - line) < sizeof(line) - 1) + *q++ = *p; + p++; + } + *q = '\0'; + sdp_parse_line(s, &state, letter, line); + next_line: + while (*p != '\n' && *p != '\0') + p++; + if (*p == '\n') + p++; + } + + if (rtc->nb_stream_infos == 0) { + av_log(rtc, AV_LOG_ERROR, "No valid media streams found in answer\n"); + ret = AVERROR(EINVAL); + goto end; + } + + av_log(rtc, AV_LOG_VERBOSE, "Parsed %d media stream infos from SDP answer\n", rtc->nb_stream_infos); + + /* Log RTX information for each stream */ + for (int i = 0; i < rtc->nb_stream_infos; i++) { + RTCStreamInfo *stream_info = rtc->stream_infos[i]; + if (stream_info && stream_info->rtx_pt >= 0) { + av_log(s, AV_LOG_INFO, "Stream info %d has RTX: pt=%d, rtx_ssrc=%u\n", + i, stream_info->rtx_pt, stream_info->rtx_ssrc); + } + } + +end: + if (ret < 0) { + /* Clean up on error */ + for (int i = 0; i < rtc->nb_stream_infos; i++) { + if (rtc->stream_infos[i]) { + av_freep(&rtc->stream_infos[i]->codec_name); + av_freep(&rtc->stream_infos[i]->fmtp); + av_freep(&rtc->stream_infos[i]->direction); + av_freep(&rtc->stream_infos[i]); + } + } + av_freep(&rtc->stream_infos); + rtc->nb_stream_infos = 0; + } + return ret; +} + +static int parse_answer(AVFormatContext *s) +{ + int ret; + + RTCContext *rtc = s->priv_data; + av_log(rtc, AV_LOG_VERBOSE, "answer:\r\n%s\r\n",rtc->sdp_answer); + + //demuxer need parse media + if (s->iformat) { + if ((ret = parse_answer_media(s)) < 0) + return ret; + } + + if ((ret = parse_answer_ice(s)) < 0) + return ret; + + return ret; +} + /** * Creates and marshals an ICE binding request packet. * @@ -812,7 +1627,7 @@ static int ice_handle_binding_request(AVFormatContext *s, char *buf, int buf_siz memcpy(tid, buf + 8, 12); /* Build the STUN binding response. */ - ret = ice_create_response(s, tid, sizeof(tid), rtc->buf, sizeof(rtc->buf), &size); + ret = ice_create_response(s, tid, sizeof(tid), rtc->buf, rtc->bufsize, &size); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to create STUN binding response, size=%d\n", size); return ret; @@ -886,7 +1701,7 @@ static int ice_dtls_handshake(AVFormatContext *s) while (1) { if (rtc->state <= RTC_STATE_ICE_CONNECTING) { /* Build the STUN binding request. */ - ret = ff_rtc_ice_create_request(s, rtc->buf, sizeof(rtc->buf), &size); + ret = ff_rtc_ice_create_request(s, rtc->buf, rtc->bufsize, &size); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to create STUN binding request, size=%d\n", size); goto end; @@ -919,7 +1734,7 @@ next_packet: for (i = 0; i < ICE_DTLS_READ_MAX_RETRY; i++) { if (rtc->state > RTC_STATE_ICE_CONNECTED) break; - ret = ffurl_read(rtc->udp, rtc->buf, sizeof(rtc->buf)); + ret = ffurl_read(rtc->udp, rtc->buf, rtc->bufsize); if (ret > 0) break; if (ret == AVERROR(EAGAIN)) { @@ -1015,13 +1830,12 @@ static int setup_srtp(AVFormatContext *s) int ret; char recv_key[DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN]; char send_key[DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN]; - char buf[AV_BASE64_SIZE(DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN)]; + RTCContext *rtc = s->priv_data; /** * The profile for OpenSSL's SRTP is SRTP_AES128_CM_SHA1_80, see ssl/d1_srtp.c. * The profile for FFmpeg's SRTP is SRTP_AES128_CM_HMAC_SHA1_80, see libavformat/srtp.c. */ - const char* suite = "SRTP_AES128_CM_HMAC_SHA1_80"; - RTCContext *rtc = s->priv_data; + av_strlcpy(rtc->suite, "SRTP_AES128_CM_HMAC_SHA1_80", sizeof(rtc->suite)); ret = ff_dtls_export_materials(rtc->dtls_uc, rtc->dtls_srtp_materials, sizeof(rtc->dtls_srtp_materials)); if (ret < 0) goto end; @@ -1045,44 +1859,44 @@ static int setup_srtp(AVFormatContext *s) memcpy(send_key + DTLS_SRTP_KEY_LEN, server_salt, DTLS_SRTP_SALT_LEN); /* Setup SRTP context for outgoing packets */ - if (!av_base64_encode(buf, sizeof(buf), send_key, sizeof(send_key))) { + if (!av_base64_encode(rtc->send_suite_param, sizeof(rtc->send_suite_param), send_key, sizeof(send_key))) { av_log(rtc, AV_LOG_ERROR, "Failed to encode send key\n"); ret = AVERROR(EIO); goto end; } - ret = ff_srtp_set_crypto(&rtc->srtp_audio_send, suite, buf); + ret = ff_srtp_set_crypto(&rtc->srtp_audio_send, rtc->suite, rtc->send_suite_param); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to set crypto for audio send\n"); goto end; } - ret = ff_srtp_set_crypto(&rtc->srtp_video_send, suite, buf); + ret = ff_srtp_set_crypto(&rtc->srtp_video_send, rtc->suite, rtc->send_suite_param); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to set crypto for video send\n"); goto end; } - ret = ff_srtp_set_crypto(&rtc->srtp_video_rtx_send, suite, buf); + ret = ff_srtp_set_crypto(&rtc->srtp_video_rtx_send, rtc->suite, rtc->send_suite_param); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to set crypto for video rtx send\n"); goto end; } - ret = ff_srtp_set_crypto(&rtc->srtp_rtcp_send, suite, buf); + ret = ff_srtp_set_crypto(&rtc->srtp_rtcp_send, rtc->suite, rtc->send_suite_param); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n"); goto end; } /* Setup SRTP context for incoming packets */ - if (!av_base64_encode(buf, sizeof(buf), recv_key, sizeof(recv_key))) { + if (!av_base64_encode(rtc->recv_suite_param, sizeof(rtc->recv_suite_param), recv_key, sizeof(recv_key))) { av_log(rtc, AV_LOG_ERROR, "Failed to encode recv key\n"); ret = AVERROR(EIO); goto end; } - ret = ff_srtp_set_crypto(&rtc->srtp_recv, suite, buf); + ret = ff_srtp_set_crypto(&rtc->srtp_recv, rtc->suite, rtc->recv_suite_param); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to set crypto for recv\n"); goto end; @@ -1092,7 +1906,7 @@ static int setup_srtp(AVFormatContext *s) rtc->state = RTC_STATE_SRTP_FINISHED; rtc->rtc_srtp_time = av_gettime_relative(); av_log(rtc, AV_LOG_VERBOSE, "SRTP setup done, state=%d, suite=%s, key=%zuB, elapsed=%.2fms\n", - rtc->state, suite, sizeof(send_key), ELAPSED(rtc->rtc_starttime, av_gettime_relative())); + rtc->state, rtc->suite, sizeof(send_key), ELAPSED(rtc->rtc_starttime, av_gettime_relative())); end: return ret; @@ -1158,6 +1972,7 @@ end: int ff_rtc_connect(AVFormatContext *s) { int ret = 0; + if ((ret = generate_sdp_offer(s)) < 0) goto end; @@ -1206,6 +2021,21 @@ void ff_rtc_close(AVFormatContext *s) s->streams[i]->priv_data = NULL; } + /* Free parsed stream info array (for demuxer) */ + if (rtc->stream_infos) { + for (i = 0; i < rtc->nb_stream_infos; i++) { + if (rtc->stream_infos[i]) { + av_freep(&rtc->stream_infos[i]->codec_name); + av_freep(&rtc->stream_infos[i]->fmtp); + av_freep(&rtc->stream_infos[i]->direction); + av_freep(&rtc->stream_infos[i]); + } + } + av_freep(&rtc->stream_infos); + rtc->nb_stream_infos = 0; + } + + av_freep(&rtc->buf); av_freep(&rtc->sdp_offer); av_freep(&rtc->sdp_answer); av_freep(&rtc->rtc_resource_url); @@ -1225,14 +2055,3 @@ void ff_rtc_close(AVFormatContext *s) ffurl_closep(&rtc->udp); } -#define OFFSET(x) offsetof(RTCContext, x) -#define ENC AV_OPT_FLAG_ENCODING_PARAM -const AVOption ff_rtc_options[] = { - { "handshake_timeout", "Timeout in milliseconds for ICE and DTLS handshake.", OFFSET(handshake_timeout), AV_OPT_TYPE_INT, { .i64 = 5000 }, -1, INT_MAX, ENC }, - { "pkt_size", "The maximum size, in bytes, of RTP packets that send out", OFFSET(pkt_size), AV_OPT_TYPE_INT, { .i64 = 1200 }, -1, INT_MAX, ENC }, - { "buffer_size", "The buffer size, in bytes, of underlying protocol", OFFSET(buffer_size), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, ENC }, - { "authorization", "The optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC }, - { "cert_file", "The optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC }, - { "key_file", "The optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC }, - { NULL }, -}; diff --git a/libavformat/rtc.h b/libavformat/rtc.h index 011e157b9f..d393c34950 100644 --- a/libavformat/rtc.h +++ b/libavformat/rtc.h @@ -27,7 +27,10 @@ #include "url.h" #include "tls.h" #include "srtp.h" +#include "rtpdec.h" +#include "network.h" +#include "libavutil/base64.h" #include "libavutil/lfg.h" #include "libavutil/log.h" #include "libavutil/opt.h" @@ -81,6 +84,25 @@ enum RTCState { */ #define MAX_UDP_BUFFER_SIZE 4096 +/** + * RTC stream information parsed from SDP + */ +typedef struct RTCStreamInfo { + int payload_type; + enum AVMediaType codec_type; + char *codec_name; + uint32_t ssrc; + int clock_rate; + char *fmtp; + int channels; + + char *direction; + + /* RTX information */ + int rtx_pt; + uint32_t rtx_ssrc; +} RTCStreamInfo; + typedef struct RTCContext { AVClass *av_class; @@ -178,10 +200,16 @@ typedef struct RTCContext { /* The SRTP receive context, to decrypt incoming packets. */ SRTPContext srtp_recv; + /* SRTP suite and parameters */ + char suite[64]; + char send_suite_param[AV_BASE64_SIZE(DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN)]; + char recv_suite_param[AV_BASE64_SIZE(DTLS_SRTP_KEY_LEN + DTLS_SRTP_SALT_LEN)]; + /* The UDP transport is used for delivering ICE, DTLS and SRTP packets. */ URLContext *udp; /* The buffer for UDP transmission. */ - char buf[MAX_UDP_BUFFER_SIZE]; + uint8_t* buf; + int bufsize; /* The timeout in milliseconds for ICE and DTLS handshake. */ int handshake_timeout; @@ -199,6 +227,10 @@ typedef struct RTCContext { /* The certificate and private key used for DTLS handshake. */ char* cert_file; char* key_file; + + /* for demuxer */ + RTCStreamInfo **stream_infos; + int nb_stream_infos; } RTCContext; int ff_rtc_initialize(AVFormatContext *s); @@ -215,6 +247,10 @@ int ff_rtc_ice_is_binding_response(uint8_t *b, int size); int ff_rtc_ice_create_request(AVFormatContext *s, uint8_t *buf, int buf_size, int *request_size); +int ff_rtc_media_is_rtp_rtcp(const uint8_t *b, int size); + +int ff_rtc_media_is_rtcp(const uint8_t *b, int size); + extern const AVOption ff_rtc_options[]; #endif /* AVFORMAT_RTC_H */ diff --git a/libavformat/rtpdec.c b/libavformat/rtpdec.c index 5872c0f59c..e9ab4477c5 100644 --- a/libavformat/rtpdec.c +++ b/libavformat/rtpdec.c @@ -140,7 +140,7 @@ static const RTPDynamicProtocolHandler *const rtp_dynamic_protocol_handler_list[ * @return the next registered rtp dynamic protocol handler * or NULL when the iteration is finished */ -static const RTPDynamicProtocolHandler *rtp_handler_iterate(void **opaque) +const RTPDynamicProtocolHandler *ff_rtp_handler_iterate(void **opaque) { uintptr_t i = (uintptr_t)*opaque; const RTPDynamicProtocolHandler *r = rtp_dynamic_protocol_handler_list[i]; @@ -156,7 +156,7 @@ const RTPDynamicProtocolHandler *ff_rtp_handler_find_by_name(const char *name, { void *i = 0; const RTPDynamicProtocolHandler *handler; - while (handler = rtp_handler_iterate(&i)) { + while ((handler = ff_rtp_handler_iterate(&i))) { if (handler->enc_name && !av_strcasecmp(name, handler->enc_name) && codec_type == handler->codec_type) @@ -170,7 +170,7 @@ const RTPDynamicProtocolHandler *ff_rtp_handler_find_by_id(int id, { void *i = 0; const RTPDynamicProtocolHandler *handler; - while (handler = rtp_handler_iterate(&i)) { + while ((handler = ff_rtp_handler_iterate(&i))) { if (handler->static_payload_id && handler->static_payload_id == id && codec_type == handler->codec_type) return handler; diff --git a/libavformat/rtpdec.h b/libavformat/rtpdec.h index c06f44b86c..327177a112 100644 --- a/libavformat/rtpdec.h +++ b/libavformat/rtpdec.h @@ -190,6 +190,17 @@ struct RTPDemuxContext { PayloadContext *dynamic_protocol_context; }; +/** + * Iterate over all registered rtp dynamic protocol handlers. + * + * @param opaque a pointer where libavformat will store the iteration state. + * Must point to NULL to start the iteration. + * + * @return the next registered rtp dynamic protocol handler + * or NULL when the iteration is finished + */ +const RTPDynamicProtocolHandler *ff_rtp_handler_iterate(void **opaque); + /** * Find a registered rtp dynamic protocol handler with the specified name. * diff --git a/libavformat/whep.c b/libavformat/whep.c new file mode 100644 index 0000000000..491f3e22df --- /dev/null +++ b/libavformat/whep.c @@ -0,0 +1,457 @@ +/* + * WebRTC-HTTP egress protocol (WHEP) demuxer + * Copyright (c) 2025 baigao + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/opt.h" +#include "libavutil/mem.h" +#include "libavutil/base64.h" +#include "libavutil/dict.h" +#include "libavutil/intreadwrite.h" +#include "libavutil/time.h" +#include "libavutil/mathematics.h" +#include "libavcodec/codec_desc.h" +#include "avio_internal.h" +#include "demux.h" +#include "internal.h" +#include "rtpdec.h" +#include "rtp.h" +#include "rtc.h" + +/** + * Initialize RTP dynamic protocol handler. + * + * Similar to init_rtp_handler and finalize_rtp_handler_init in rtsp.c + */ +static int init_rtp_handler(AVFormatContext *s, AVStream *st, + RTPDemuxContext *rtp_ctx, + const RTPDynamicProtocolHandler *handler, + PayloadContext **payload_ctx_out) +{ + PayloadContext *payload_ctx = NULL; + int ret; + + if (!handler) + return 0; + + if (handler->codec_id != AV_CODEC_ID_NONE) + st->codecpar->codec_id = handler->codec_id; + + if (handler->priv_data_size > 0) { + payload_ctx = av_mallocz(handler->priv_data_size); + if (!payload_ctx) + return AVERROR(ENOMEM); + } + + ff_rtp_parse_set_dynamic_protocol(rtp_ctx, payload_ctx, handler); + ffstream(st)->need_parsing = handler->need_parsing; + + if (handler->init) { + ret = handler->init(s, st->index, payload_ctx); + if (ret < 0) { + av_log(s, AV_LOG_ERROR, "Failed to initialize RTP handler '%s': %d\n", + handler->enc_name, ret); + if (payload_ctx) { + if (handler->close) + handler->close(payload_ctx); + av_free(payload_ctx); + } + return ret; + } + } + + *payload_ctx_out = payload_ctx; + return 0; +} + +/** + * Parse fmtp attributes for the stream. + */ +static int parse_fmtp(AVFormatContext *s, AVStream *st, + const RTPDynamicProtocolHandler *handler, + PayloadContext *payload_ctx, + int payload_type, const char *fmtp) +{ + char fmtp_line[1024]; + int ret; + + if (!fmtp || !handler || !handler->parse_sdp_a_line) + return 0; + + snprintf(fmtp_line, sizeof(fmtp_line), "fmtp:%d %s", payload_type, fmtp); + av_log(s, AV_LOG_INFO, "Processing fmtp for stream %d: %s\n", st->index, fmtp_line); + + ret = handler->parse_sdp_a_line(s, st->index, payload_ctx, fmtp_line); + if (ret < 0) { + av_log(s, AV_LOG_WARNING, "Failed to parse fmtp line for stream %d: %d\n", + st->index, ret); + } else { + av_log(s, AV_LOG_INFO, "Successfully processed fmtp for stream %d\n", st->index); + } + + return ret; +} + +/** + * Create RTP demuxer contexts for each stream. + */ +static int create_rtp_demuxer(AVFormatContext *s) +{ + int ret = 0, i; + RTCContext *rtc = s->priv_data; + + if (!rtc->stream_infos || rtc->nb_stream_infos == 0) { + av_log(rtc, AV_LOG_ERROR, "No stream info available for RTP demuxer\n"); + return AVERROR(EINVAL); + } + + for (i = 0; i < rtc->nb_stream_infos; i++) { + RTCStreamInfo *stream_info = rtc->stream_infos[i]; + AVStream *st; + RTPDemuxContext *rtp_ctx; + const RTPDynamicProtocolHandler *handler; + int payload_type; + + if (!stream_info) { + av_log(rtc, AV_LOG_ERROR, "Stream info %d is NULL\n", i); + ret = AVERROR(EINVAL); + goto fail; + } + + /* Skip inactive streams */ + if (stream_info->direction && strcmp(stream_info->direction, "inactive") == 0) { + av_log(rtc, AV_LOG_INFO, "Skipping inactive stream %d\n", i); + continue; + } + + st = avformat_new_stream(s, NULL); + if (!st) { + av_log(rtc, AV_LOG_ERROR, "Failed to create stream %d\n", i); + ret = AVERROR(ENOMEM); + goto fail; + } + + st->id = i; + st->codecpar->codec_type = stream_info->codec_type; + + payload_type = stream_info->payload_type; + if (payload_type < RTP_PT_PRIVATE) { + ff_rtp_get_codec_info(st->codecpar, payload_type); + } else if (stream_info->codec_name) { + st->codecpar->codec_id = ff_rtp_codec_id(stream_info->codec_name, stream_info->codec_type); + } else { + st->codecpar->codec_id = AV_CODEC_ID_NONE; + } + + if (stream_info->codec_type == AVMEDIA_TYPE_AUDIO) { + st->codecpar->sample_rate = stream_info->clock_rate; + if (stream_info->channels > 0) + av_channel_layout_default(&st->codecpar->ch_layout, stream_info->channels); + avpriv_set_pts_info(st, 32, 1, stream_info->clock_rate); + } else if (stream_info->codec_type == AVMEDIA_TYPE_VIDEO) { + avpriv_set_pts_info(st, 32, 1, stream_info->clock_rate); + } + + av_log(rtc, AV_LOG_VERBOSE, "Creating RTP demuxer for stream %d: type=%s, codec=%s, pt=%d, rate=%d\n", + i, av_get_media_type_string(stream_info->codec_type), + stream_info->codec_name ? stream_info->codec_name : avcodec_get_name(st->codecpar->codec_id), + payload_type, stream_info->clock_rate); + + rtp_ctx = ff_rtp_parse_open(s, st, payload_type, RTP_REORDER_QUEUE_DEFAULT_SIZE); + if (!rtp_ctx) { + av_log(rtc, AV_LOG_ERROR, "Failed to create RTP demuxer for stream %d\n", i); + ret = AVERROR(ENOMEM); + goto fail; + } + + handler = NULL; + if (payload_type < RTP_PT_PRIVATE) { + handler = ff_rtp_handler_find_by_id(payload_type, stream_info->codec_type); + } + if (!handler && stream_info->codec_name) { + handler = ff_rtp_handler_find_by_name(stream_info->codec_name, stream_info->codec_type); + } + + if (handler) { + PayloadContext *payload_ctx = NULL; + + av_log(rtc, AV_LOG_VERBOSE, "Found RTP handler '%s' for stream %d, codec=%s, pt=%d\n", + handler->enc_name, i, + stream_info->codec_name ? stream_info->codec_name : avcodec_get_name(st->codecpar->codec_id), + payload_type); + + ret = init_rtp_handler(s, st, rtp_ctx, handler, &payload_ctx); + if (ret < 0) { + av_log(rtc, AV_LOG_ERROR, "Failed to initialize RTP handler for stream %d\n", i); + ff_rtp_parse_close(rtp_ctx); + goto fail; + } + + parse_fmtp(s, st, handler, payload_ctx, payload_type, stream_info->fmtp); + } else { + av_log(rtc, AV_LOG_WARNING, "No RTP handler found for stream %d, codec=%s, pt=%d\n", + i, + stream_info->codec_name ? stream_info->codec_name : avcodec_get_name(st->codecpar->codec_id), + payload_type); + } + + rtp_ctx->ssrc = stream_info->ssrc; + av_log(rtc, AV_LOG_VERBOSE, "Set SSRC %u for stream %d\n", stream_info->ssrc, i); + + if (stream_info->rtx_pt >= 0) { + av_log(rtc, AV_LOG_INFO, "Stream %d has RTX support: rtx_pt=%d, rtx_ssrc=%u\n", + i, stream_info->rtx_pt, stream_info->rtx_ssrc); + /* TODO: Configure RTX support in RTPDemuxContext when RTX implementation is ready */ + } + + ff_rtp_parse_set_crypto(rtp_ctx, rtc->suite, rtc->recv_suite_param); + + st->priv_data = rtp_ctx; + av_log(rtc, AV_LOG_VERBOSE, "Created RTP demuxer for stream %d: type=%s, pt=%d\n", + i, av_get_media_type_string(st->codecpar->codec_type), payload_type); + } + + av_log(rtc, AV_LOG_VERBOSE, "Created %d RTP demuxer contexts\n", s->nb_streams); + return 0; + +fail: + for (i = 0; i < s->nb_streams; i++) { + if (s->streams[i]->priv_data) { + ff_rtp_parse_close(s->streams[i]->priv_data); + s->streams[i]->priv_data = NULL; + } + } + return ret; +} + +static av_cold int whep_read_header(AVFormatContext *s) +{ + int ret; + RTCContext *rtc = s->priv_data; + + if ((ret = ff_rtc_initialize(s)) < 0) + goto end; + + if ((ret = ff_rtc_connect(s)) < 0) + goto end; + + if ((ret = create_rtp_demuxer(s)) < 0) + goto end; + +end: + if (ret < 0) + rtc->state = RTC_STATE_FAILED; + return ret; +} + +/** + * Send encrypted RTCP packet using SRTP. + */ +static int send_encrypted_rtcp(AVFormatContext *s, const uint8_t *buf, int len) +{ + RTCContext *rtc = s->priv_data; + uint8_t encrypted_buf[MAX_UDP_BUFFER_SIZE]; + int cipher_size; + int ret; + + cipher_size = ff_srtp_encrypt(&rtc->srtp_rtcp_send, buf, len, + encrypted_buf, sizeof(encrypted_buf)); + if (cipher_size <= 0 || cipher_size < len) { + av_log(rtc, AV_LOG_WARNING, "Failed to encrypt RTCP packet=%dB, cipher=%dB\n", + len, cipher_size); + return AVERROR(EIO); + } + + ret = ffurl_write(rtc->udp, encrypted_buf, cipher_size); + if (ret < 0) { + av_log(rtc, AV_LOG_ERROR, "Failed to write encrypted RTCP packet=%dB, ret=%d\n", + cipher_size, ret); + return ret; + } + + av_log(rtc, AV_LOG_TRACE, "Sent encrypted RTCP packet: plain=%dB, cipher=%dB\n", + len, cipher_size); + return ret; +} + +static int send_rtcp_rr(AVFormatContext *s, RTPDemuxContext *rtp_ctx, int len) +{ + AVIOContext *rtcp_pb = NULL; + uint8_t *rtcp_buf = NULL; + int ret = 0; + + if (avio_open_dyn_buf(&rtcp_pb) >= 0) { + ff_rtp_check_and_send_back_rr(rtp_ctx, NULL, rtcp_pb, len); + int rtcp_len = avio_close_dyn_buf(rtcp_pb, &rtcp_buf); + if (rtcp_len > 0 && rtcp_buf) { + ret = send_encrypted_rtcp(s, rtcp_buf, rtcp_len); + av_free(rtcp_buf); + } + } + + return ret; +} + +static int send_rtcp_feedback(AVFormatContext *s, RTPDemuxContext *rtp_ctx) +{ + AVIOContext *rtcp_pb = NULL; + uint8_t *rtcp_buf = NULL; + int ret = 0; + + if (avio_open_dyn_buf(&rtcp_pb) >= 0) { + ff_rtp_send_rtcp_feedback(rtp_ctx, NULL, rtcp_pb); + int rtcp_len = avio_close_dyn_buf(rtcp_pb, &rtcp_buf); + if (rtcp_len > 0 && rtcp_buf) { + ret = send_encrypted_rtcp(s, rtcp_buf, rtcp_len); + av_free(rtcp_buf); + } + } + + return ret; +} + +static int whep_read_packet(AVFormatContext *s, AVPacket *pkt) +{ + int ret; + RTCContext *rtc = s->priv_data; + + while (1) { + /** + * Receive packets from the server suh as ICE binding requests, DTLS messages, + * and RTCP like PLI requests, then respond to them. + */ + ret = ffurl_read(rtc->udp, rtc->buf, rtc->bufsize); + if (ret < 0) { + if (ret == AVERROR(EAGAIN)) + return ret; + goto end; + } + + if (!ret) { + av_log(rtc, AV_LOG_ERROR, "Receive EOF from UDP socket\n"); + ret = AVERROR_EOF; + goto end; + } + + if (ff_rtc_is_dtls_packet(rtc->buf, ret)) { + if ((ret = ffurl_write(rtc->dtls_uc, rtc->buf, ret)) < 0) { + av_log(rtc, AV_LOG_ERROR, "Failed to handle DTLS message\n"); + goto end; + } + continue; + } else if (ff_rtc_media_is_rtp_rtcp(rtc->buf, ret)) { + int len = ret; + int is_rtcp = ff_rtc_media_is_rtcp(rtc->buf, ret); + + av_log(rtc, AV_LOG_TRACE, "Received %s packet, len %d\n", + is_rtcp ? "RTCP" : "RTP", ret); + + for (int i = 0; i < s->nb_streams; i++) { + AVStream *st = s->streams[i]; + RTPDemuxContext *rtp_ctx = st->priv_data; + if (!rtp_ctx) + continue; + + if (!is_rtcp) { + int pkt_payload_type = rtc->buf[1] & 0x7f; + int stream_id = st->id; + if (stream_id >= 0 && stream_id < rtc->nb_stream_infos && rtc->stream_infos[stream_id]) { + RTCStreamInfo *stream_info = rtc->stream_infos[stream_id]; + if (stream_info->rtx_pt >= 0 && pkt_payload_type == stream_info->rtx_pt) { + /* TODO: Implement RTX packet processing */ + av_log(rtc, AV_LOG_INFO, "Received RTX retransmission packet for stream %d (id=%d): " + "PT=%d, SSRC=%u, main_PT=%d\n", + i, stream_id, pkt_payload_type, stream_info->rtx_ssrc, rtp_ctx->payload_type); + continue; + } + } + + if (pkt_payload_type != rtp_ctx->payload_type) { + av_log(rtc, AV_LOG_INFO, "RTP packet PT=%d doesn't match stream %d PT=%d\n", + pkt_payload_type, i, rtp_ctx->payload_type); + continue; + } + } + + ret = ff_rtp_parse_packet(rtp_ctx, pkt, &rtc->buf, len); + if (!is_rtcp) { + if (ret == AVERROR(EAGAIN)) { + av_log(rtc, AV_LOG_DEBUG, "RTP packet buffered for stream %d\n", i); + continue; + } else if (ret >= 0 && pkt->size > 0) { + pkt->stream_index = i; + send_rtcp_rr(s, rtp_ctx, len); + send_rtcp_feedback(s, rtp_ctx); + goto end; + } else if (ret >= 0) { + av_log(rtc, AV_LOG_DEBUG, "RTP parsed but no output for stream %d\n", i); + } + } else { + /* TODO: Implement RTCP processing*/ + av_log(rtc, AV_LOG_DEBUG, "RECV RTCP, len=%d\n", len); + } + } + } else { + //TODO: Implement ICE processing + av_log(rtc, AV_LOG_TRACE, "Received other type data, len %d\n", ret); + } + } + +end: + if (ret < 0) + rtc->state = RTC_STATE_FAILED; + return ret; +} + +static av_cold int whep_read_close(AVFormatContext *s) +{ + int i; + + for (i = 0; i < s->nb_streams; i++) { + if (s->streams[i]->priv_data) { + ff_rtp_parse_close(s->streams[i]->priv_data); + s->streams[i]->priv_data = NULL; + } + } + + ff_rtc_close(s); + return 0; +} + +static const AVClass whep_demuxer_class = { + .class_name = "WHEP demuxer", + .item_name = av_default_item_name, + .option = ff_rtc_options, + .version = LIBAVUTIL_VERSION_INT, +}; + +const FFInputFormat ff_whep_demuxer = { + .p.name = "whep", + .p.long_name = NULL_IF_CONFIG_SMALL("WHEP(WebRTC-HTTP egress protocol) demuxer"), + .p.flags = AVFMT_GLOBALHEADER | AVFMT_NOFILE | AVFMT_EXPERIMENTAL, + .p.priv_class = &whep_demuxer_class, + .priv_data_size = sizeof(RTCContext), + .read_probe = NULL, + .read_header = whep_read_header, + .read_packet = whep_read_packet, + .read_close = whep_read_close, + .read_seek = NULL, + .read_play = NULL, + .read_pause = NULL, +}; diff --git a/libavformat/whip.c b/libavformat/whip.c index c73c8d5c26..542e4a2ac6 100644 --- a/libavformat/whip.c +++ b/libavformat/whip.c @@ -36,7 +36,6 @@ #include "rtp.h" #include "rtc.h" - /** * The maximum size of the Secure Real-time Transport Protocol (SRTP) HMAC checksum * and padding that is appended to the end of the packet. To calculate the maximum @@ -45,26 +44,6 @@ */ #define DTLS_SRTP_CHECKSUM_LEN 16 -/** - * The RTP header is 12 bytes long, comprising the Version(1B), PT(1B), - * SequenceNumber(2B), Timestamp(4B), and SSRC(4B). - * See https://www.rfc-editor.org/rfc/rfc3550#section-5.1 - */ -#define WHIP_RTP_HEADER_SIZE 12 - -/** - * For RTCP, PT is [128, 223] (or without marker [0, 95]). Literally, RTCP starts - * from 64 not 0, so PT is [192, 223] (or without marker [64, 95]), see "RTCP Control - * Packet Types (PT)" at - * https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-4 - * - * For RTP, the PT is [96, 127], or [224, 255] with marker. See "RTP Payload Types (PT) - * for standard audio and video encodings" at - * https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-1 - */ -#define WHIP_RTCP_PT_START 192 -#define WHIP_RTCP_PT_END 223 - /** * Refer to RFC 7675 5.1, * @@ -79,24 +58,6 @@ /* Calculate the elapsed time from starttime to endtime in milliseconds. */ #define ELAPSED(starttime, endtime) ((float)(endtime - starttime) / 1000) -/** - * In RTP packets, the first byte is represented as 0b10xxxxxx, where the initial - * two bits (0b10) indicate the RTP version, - * see https://www.rfc-editor.org/rfc/rfc3550#section-5.1 - * The RTCP packet header is similar to RTP, - * see https://www.rfc-editor.org/rfc/rfc3550#section-6.4.1 - */ -static int media_is_rtp_rtcp(const uint8_t *b, int size) -{ - return size >= WHIP_RTP_HEADER_SIZE && (b[0] & 0xC0) == 0x80; -} - -/* Whether the packet is RTCP. */ -static int media_is_rtcp(const uint8_t *b, int size) -{ - return size >= WHIP_RTP_HEADER_SIZE && b[1] >= WHIP_RTCP_PT_START && b[1] <= WHIP_RTCP_PT_END; -} - /** * When duplicating a stream, the demuxer has already set the extradata, profile, and * level of the par. Keep in mind that this function will not be invoked since the @@ -251,7 +212,6 @@ static int parse_codec(AVFormatContext *s) return ret; } - /** * Callback triggered by the RTP muxer when it creates and sends out an RTP packet. * @@ -268,11 +228,11 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) SRTPContext *srtp; /* Ignore if not RTP or RTCP packet. */ - if (!media_is_rtp_rtcp(buf, buf_size)) + if (!ff_rtc_media_is_rtp_rtcp(buf, buf_size)) return 0; /* Only support audio, video and rtcp. */ - is_rtcp = media_is_rtcp(buf, buf_size); + is_rtcp = ff_rtc_media_is_rtcp(buf, buf_size); payload_type = buf[1] & 0x7f; is_video = payload_type == rtc->video_payload_type; if (!is_rtcp && payload_type != rtc->video_payload_type && payload_type != rtc->audio_payload_type) @@ -282,7 +242,7 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) srtp = is_rtcp ? &rtc->srtp_rtcp_send : (is_video? &rtc->srtp_video_send : &rtc->srtp_audio_send); /* Encrypt by SRTP and send out. */ - cipher_size = ff_srtp_encrypt(srtp, buf, buf_size, rtc->buf, sizeof(rtc->buf)); + cipher_size = ff_srtp_encrypt(srtp, buf, buf_size, rtc->buf, rtc->bufsize); if (cipher_size <= 0 || cipher_size < buf_size) { av_log(rtc, AV_LOG_WARNING, "Failed to encrypt packet=%dB, cipher=%dB\n", buf_size, cipher_size); return 0; @@ -575,7 +535,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) */ if (now - rtc->rtc_last_consent_tx_time > WHIP_ICE_CONSENT_CHECK_INTERVAL * RTC_US_PER_MS) { int size; - ret = ff_rtc_ice_create_request(s, rtc->buf, sizeof(rtc->buf), &size); + ret = ff_rtc_ice_create_request(s, rtc->buf, rtc->bufsize, &size); if (ret < 0) { av_log(rtc, AV_LOG_ERROR, "Failed to create STUN binding request, size=%d\n", size); goto end; @@ -593,7 +553,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) * Receive packets from the server such as ICE binding requests, DTLS messages, * and RTCP like PLI requests, then respond to them. */ - ret = ffurl_read(rtc->udp, rtc->buf, sizeof(rtc->buf)); + ret = ffurl_read(rtc->udp, rtc->buf, rtc->bufsize); if (ret < 0) { if (ret == AVERROR(EAGAIN)) goto write_packet; @@ -616,7 +576,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) goto end; } } - if (media_is_rtcp(rtc->buf, ret)) { + if (ff_rtc_media_is_rtcp(rtc->buf, ret)) { uint8_t fmt = rtc->buf[0] & 0x1f; uint8_t pt = rtc->buf[1]; /** -- 2.51.0 _______________________________________________ ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org