From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 19A9A4B9D5 for ; Tue, 22 Jul 2025 12:37:34 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 1AB5F68CFDE; Tue, 22 Jul 2025 15:36:53 +0300 (EEST) Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTPS id 8E13468CF3C for ; Tue, 22 Jul 2025 15:36:51 +0300 (EEST) Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-235ea292956so49628205ad.1 for ; Tue, 22 Jul 2025 05:36:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1753187809; x=1753792609; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cjUfeR1Xh1WRVK5SiuUn7Y5at7gptJIp34log8GFZCo=; b=JR6arGTyWh/DTwj5oGaF9cJEM03NyXOEyJWMet8ynALUq5hwxCMdu6sft+zqApCFAS hBUQV1U9AENFmUwr5GUNiHnOMA+Fo6yabRj4zmDA/9xcOsjkjgoOH3jWWmLtzB6NbnSP TPI2rYJYkWTY9TNAeNJghJaYo/NjBEZsDvb+xm/AqDBradJ7z8jaUBKS5uIkV+yvogmH HQRUp6LrrE6lExKdL4ucLy8FCi2i7hYiVvATVUk4Cj7Iml+1hiNC+UUYmuCAy0Wuyuoj 5/wHISFslzNwISaKxbP0gcXhxFXn98FXet/t+BvGW2nrkb2wPcIr8K5ja8yFvHvyVm8/ qP5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753187809; x=1753792609; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cjUfeR1Xh1WRVK5SiuUn7Y5at7gptJIp34log8GFZCo=; b=vyeJVgRIMjcLeIQAjEH+Z/EkX+YNMyZghqKg7J7IUSdf9XLUkW3WsL3wEogHzHfKss ns4D+Kug/K7uyWf6vu2A5Y6453c7F44Pev82hrOm0/PN/PsJNpG70hF1GzS1RE/0/Pj2 jA94VCa/xa+Ife64UYRPWyNpISvV5C3Quz09jXl2EUYm7x0e67+KJBNHEK0tbj7npSCQ 4xoWwD+u3jSIu3slAPVKaSMieB4Jjhq5bWiTOhyrRmBR37sOelr/8YLyz2cdaA7ATML5 EJ1y4F6WwkdkY8Dn3twlhJsWmNJkiSeR3cnKUh9LOmWbQPBa19V9cIrBfkP+W67Yl4CK mTag== X-Gm-Message-State: AOJu0YyQENq7g13RRDjE78hIa5bMfMToLU0PbtfaTSTHJyJwr8PM2m8o K31MFhgHA6J5p90bT4ZEci+U89L2H6p+r8FRFgCXSzmM/FxUUGgfxwcr99Spb5H/TBY= X-Gm-Gg: ASbGncue7j02LFJNXb35Jy0yM85cLPuoax9FE28CGhhJM5pFGsGwabnQFLLGaWLW+4X dEMi1n+Npm2JBiXb5NCk+rTrPoD2W9KzSgKx1rvGv5e0bD1nTljzJRss6c7x/MAZEL886dKqR51 Np40k9Fnp7GQcZ+h8Fv657yBkKYBTgbhr/cg4h1jciLDCGWoC8I/tanjDKS/tjYgtjE0YSquBVc NAtgY5mUJXteuI33m9JGnVkrT2YML2Py/w/+pT9bJFFYSPq+tZ/nFlwtZ5df60b+kbehk+7tVuD rzuF0mFsueTjqne5CklA8QPSnek7dEF+Bze6DgdS8/nstnEQ3EM1zXzsx8EhKo3A8UEqw+LLWcp eP07gGYyyhp11Ox9hoEV+uuP0q8uuIYIto4uf1P4U64gvKw== X-Google-Smtp-Source: AGHT+IEmlttC1IGWlMaP/QySF6rqXIx4y7cnvpPi6fWPftqCfpG5H/rKzcK2ypgfSsJ8LU5ovGtC4Q== X-Received: by 2002:a17:903:fb0:b0:23c:7b65:9b08 with SMTP id d9443c01a7336-23e256844a0mr362269105ad.1.1753187809290; Tue, 22 Jul 2025 05:36:49 -0700 (PDT) Received: from localhost.localdomain ([182.126.128.169]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23e3b60edbasm75740715ad.70.2025.07.22.05.36.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Jul 2025 05:36:48 -0700 (PDT) From: Jack Lau X-Google-Original-From: Jack Lau To: ffmpeg-devel@ffmpeg.org Date: Tue, 22 Jul 2025 20:36:06 +0800 Message-ID: <20250722123616.53164-6-jacklau1222@qq.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250722123616.53164-1-jacklau1222@qq.com> References: <20250722123616.53164-1-jacklau1222@qq.com> MIME-Version: 1.0 X-Unsent: 1 Subject: [FFmpeg-devel] [PATCH v5 05/15] avformat/whip: implement NACK and RTX suppport X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Sergio Garcia Murillo , Jack Lau Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: RTP retransmission described in RFC4588 (RTX) is an effective packet loss recovery technique for real-time applications with relaxed delay bounds. This patch provides a minimal implementation for RTX and RTCP NACK (RFC3940) and its associated SDP signaling and negotiation. Co-authored-by: Sergio Garcia Murillo Signed-off-by: Jack Lau --- libavformat/whip.c | 206 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 202 insertions(+), 4 deletions(-) diff --git a/libavformat/whip.c b/libavformat/whip.c index 4de8eb2601..5907865d22 100644 --- a/libavformat/whip.c +++ b/libavformat/whip.c @@ -114,6 +114,7 @@ /* Referring to Chrome's definition of RTP payload types. */ #define WHIP_RTP_PAYLOAD_TYPE_H264 106 #define WHIP_RTP_PAYLOAD_TYPE_OPUS 111 +#define WHIP_RTP_PAYLOAD_TYPE_RTX 105 /** * The STUN message header, which is 20 bytes long, comprises the @@ -150,6 +151,11 @@ #define WHIP_SDP_SESSION_ID "4489045141692799359" #define WHIP_SDP_CREATOR_IP "127.0.0.1" +/** + * Retransmission / NACK support +*/ +#define HISTORY_SIZE_DEFAULT 512 + /* Calculate the elapsed time from starttime to endtime in milliseconds. */ #define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000) @@ -194,9 +200,16 @@ enum WHIPState { }; typedef enum WHIPFlags { - WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate + WHIP_FLAG_IGNORE_IPV6 = (1 << 0), // Ignore ipv6 candidate + WHIP_FLAG_DISABLE_RTX = (1 << 1) // Enable NACK and RTX } WHIPFlags; +typedef struct RtpHistoryItem { + uint16_t seq; // original RTP seq + int size; // length in bytes + uint8_t* buf; // malloc-ed copy +} RtpHistoryItem; + typedef struct WHIPContext { AVClass *av_class; @@ -285,6 +298,7 @@ typedef struct WHIPContext { /* The SRTP send context, to encrypt outgoing packets. */ SRTPContext srtp_audio_send; SRTPContext srtp_video_send; + SRTPContext srtp_video_rtx_send; SRTPContext srtp_rtcp_send; /* The SRTP receive context, to decrypt incoming packets. */ SRTPContext srtp_recv; @@ -309,6 +323,14 @@ typedef struct WHIPContext { /* The certificate and private key used for DTLS handshake. */ char* cert_file; char* key_file; + + /* RTX and NACK */ + uint8_t rtx_payload_type; + uint32_t video_rtx_ssrc; + uint16_t rtx_seq; + int history_size; + RtpHistoryItem *history; /* ring buffer */ + int hist_head; } WHIPContext; /** @@ -611,6 +633,16 @@ static int generate_sdp_offer(AVFormatContext *s) whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS; whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264; + /* RTX and NACK init */ + whip->rtx_payload_type = WHIP_RTP_PAYLOAD_TYPE_RTX; + whip->video_rtx_ssrc = av_lfg_get(&whip->rnd); + whip->rtx_seq = 0; + whip->hist_head = 0; + whip->history_size = FFMAX(64, whip->history_size); + whip->history = av_calloc(whip->history_size, sizeof(*whip->history)); + if (!whip->history) + return AVERROR(ENOMEM); + av_bprintf(&bp, "" "v=0\r\n" "o=FFmpeg %s 2 IN IP4 %s\r\n" @@ -661,7 +693,7 @@ static int generate_sdp_offer(AVFormatContext *s) } av_bprintf(&bp, "" - "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n" + "m=video 9 UDP/TLS/RTP/SAVPF %u %u\r\n" "c=IN IP4 0.0.0.0\r\n" "a=ice-ufrag:%s\r\n" "a=ice-pwd:%s\r\n" @@ -674,9 +706,16 @@ static int generate_sdp_offer(AVFormatContext *s) "a=rtcp-rsize\r\n" "a=rtpmap:%u %s/90000\r\n" "a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n" + "a=rtcp-fb:%u nack\r\n" + "a=rtpmap:%u rtx/90000\r\n" + "a=fmtp:%u apt=%u\r\n" + "a=ssrc-group:FID %u %u\r\n" + "a=ssrc:%u cname:FFmpeg\r\n" + "a=ssrc:%u msid:FFmpeg video\r\n" "a=ssrc:%u cname:FFmpeg\r\n" "a=ssrc:%u msid:FFmpeg video\r\n", whip->video_payload_type, + whip->rtx_payload_type, whip->ice_ufrag_local, whip->ice_pwd_local, whip->dtls_fingerprint, @@ -686,8 +725,16 @@ static int generate_sdp_offer(AVFormatContext *s) profile, whip->constraint_set_flags, level, + whip->video_payload_type, + whip->rtx_payload_type, + whip->rtx_payload_type, + whip->video_payload_type, + whip->video_ssrc, + whip->video_rtx_ssrc, whip->video_ssrc, - whip->video_ssrc); + whip->video_ssrc, + whip->video_rtx_ssrc, + whip->video_rtx_ssrc); } if (!av_bprint_is_complete(&bp)) { @@ -1400,6 +1447,12 @@ static int setup_srtp(AVFormatContext *s) goto end; } + ret = ff_srtp_set_crypto(&whip->srtp_video_rtx_send, suite, buf); + if (ret < 0) { + av_log(whip, AV_LOG_ERROR, "Failed to set crypto for video rtx send\n"); + goto end; + } + ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf); if (ret < 0) { av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n"); @@ -1429,6 +1482,38 @@ end: return ret; } + +/** + * RTX history helpers + */ + static int rtp_history_store(WHIPContext *whip, const uint8_t *buf, int size) +{ + int pos = whip->hist_head % whip->history_size; + RtpHistoryItem *it = &whip->history[pos]; + /* free older entry */ + av_free(it->buf); + it->buf = av_malloc(size); + if (!it->buf) + return AVERROR(ENOMEM); + + memcpy(it->buf, buf, size); + it->size = size; + it->seq = AV_RB16(buf + 2); + + whip->hist_head = ++pos; + return 0; +} + +static const RtpHistoryItem *rtp_history_find(const WHIPContext *whip, uint16_t seq) +{ + for (int i = 0; i < whip->history_size; i++) { + const RtpHistoryItem *it = &whip->history[i]; + if (it->buf && it->seq == seq) + return it; + } + return NULL; +} + /** * Callback triggered by the RTP muxer when it creates and sends out an RTP packet. * @@ -1465,6 +1550,12 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) return 0; } + /* Store only ORIGINAL video packets (non-RTX, non-RTCP) */ + if (!is_rtcp && is_video) { + ret = rtp_history_store(whip, buf, buf_size); + if (ret < 0) return ret; + } + ret = ffurl_write(whip->udp, whip->buf, cipher_size); if (ret < 0) { av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret); @@ -1473,6 +1564,48 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) return ret; } +/** + * See https://datatracker.ietf.org/doc/html/rfc4588 + * Build and send a single RTX packet + */ +static int send_rtx_packet(AVFormatContext *s, const uint8_t *orig_pkt_buf, int orig_size) +{ + int ret; + WHIPContext *whip = s->priv_data; + int new_size, cipher_size; + if (whip->flags & WHIP_FLAG_DISABLE_RTX) + return 0; + + /* allocate new buffer: header + 2 + payload */ + if (orig_size + 2 > sizeof(whip->buf)) + return 0; + + memcpy(whip->buf, orig_pkt_buf, orig_size); + + uint8_t *hdr = whip->buf; + uint16_t orig_seq = AV_RB16(hdr + 2); + + /* rewrite header */ + hdr[1] = (hdr[1] & 0x80) | whip->rtx_payload_type; /* keep M bit */ + AV_WB16(hdr + 2, whip->rtx_seq++); + AV_WB32(hdr + 8, whip->video_rtx_ssrc); + + /* shift payload 2 bytes */ + memmove(hdr + 12 + 2, hdr + 12, orig_size - 12); + AV_WB16(hdr + 12, orig_seq); + + new_size = orig_size + 2; + + /* Encrypt by SRTP and send out. */ + cipher_size = ff_srtp_encrypt(&whip->srtp_video_rtx_send, whip->buf, new_size, whip->buf, sizeof(whip->buf)); + if (cipher_size <= 0 || cipher_size < new_size) { + av_log(whip, AV_LOG_WARNING, "Failed to encrypt packet=%dB, cipher=%dB\n", new_size, cipher_size); + return 0; + } + ret = ffurl_write(whip->udp, whip->buf, cipher_size); + if (ret <= 0) av_log(whip, AV_LOG_ERROR, "Failed to send RTX packet\n"); + return ret; +} /** * Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP @@ -1781,6 +1914,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) WHIPContext *whip = s->priv_data; AVStream *st = s->streams[pkt->stream_index]; AVFormatContext *rtp_ctx = st->priv_data; + uint8_t *buf = NULL; /* TODO: Send binding request every 1s as WebRTC heartbeat. */ @@ -1796,11 +1930,72 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) goto end; } } + /** + * Handle RTCP NACK + * Refer to RFC 4585, Section 6.2.1 + * The Generic NACK message is identified by PT=RTPFB and FMT=1. + * TODO: disable retransmisstion when "-tune zerolatency" + */ + if (media_is_rtcp(whip->buf, ret)) { + int ptr = 0; + uint8_t pt = whip->buf[ptr + 1]; + uint8_t fmt = (whip->buf[ptr] & 0x1f); + if (ptr + 4 <= ret && pt == 205 && fmt == 1) { + /** + * Refer to RFC 3550, Section 6.4.1. + * The length of this RTCP packet in 32-bit words minus one, + * including the header and any padding. + */ + int rtcp_len = (AV_RB16(&whip->buf[ptr + 2]) + 1) * 4; + /* SRTCP index(4 bytes) + HMAC (SRTP_AES128_CM_SHA1_80 10bytes) */ + int srtcp_len = rtcp_len + 4 + 10; + if (srtcp_len == ret && rtcp_len >= 12) { + int i = 0; + buf = av_malloc(srtcp_len); + if (!buf) return AVERROR(ENOMEM); + memcpy(buf, whip->buf, srtcp_len); + int ret = ff_srtp_decrypt(&whip->srtp_recv, buf, &srtcp_len); + if (ret < 0) { + av_log(whip, AV_LOG_ERROR, "NACK packet(SRTCP) decrypt failed: %d, Can't send RTX packet\n", ret); + goto write_packet; + } + while (12 + i < rtcp_len) { + /** + * See https://datatracker.ietf.org/doc/html/rfc4585#section-6.1 + * Handle multi NACKs in bundled packet. + */ + uint16_t pid = AV_RB16(&buf[ptr + 12 + i]); + uint16_t blp = AV_RB16(&buf[ptr + 14 + i]); + + /* retransmit pid + any bit set in blp */ + for (int bit = -1; bit < 16; bit++) { + uint16_t seq = (bit < 0) ? pid : pid + bit + 1; + if (bit >= 0 && !(blp & (1 << bit))) + continue; + + const RtpHistoryItem *it = rtp_history_find(whip, seq); + if (it) { + av_log(whip, AV_LOG_VERBOSE, + "NACK, packet found: size: %d, seq=%d, rtx size=%d, lateset stored packet seq:%d\n", + it->size, seq, ret, whip->history[whip->hist_head-1].seq); + send_rtx_packet(s, it->buf, it->size); + } else { + av_log(whip, AV_LOG_VERBOSE, + "NACK, packet not found, seq=%d, latest stored packet seq: %d, latest rtx seq: %d\n", + seq, whip->history[whip->hist_head-1].seq, whip->rtx_seq); + } + } + i = i + 4; + } + av_freep(&buf); + } + } + } } else if (ret != AVERROR(EAGAIN)) { av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n"); goto end; } - +write_packet: if (whip->h264_annexb_insert_sps_pps && st->codecpar->codec_id == AV_CODEC_ID_H264) { if ((ret = h264_annexb_insert_sps_pps(s, pkt)) < 0) { av_log(whip, AV_LOG_ERROR, "Failed to insert SPS/PPS before IDR\n"); @@ -1819,6 +2014,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) } end: + if (buf) av_freep(&buf); if (ret < 0 && whip->state < WHIP_STATE_FAILED) whip->state = WHIP_STATE_FAILED; if (ret >= 0 && whip->state >= WHIP_STATE_FAILED && whip->dtls_ret < 0) @@ -1901,6 +2097,8 @@ static const AVOption options[] = { { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC }, { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" }, { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" }, + { "disable_rtx", "Disable RFC 4588 RTX", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" }, + { "rtx_history_size", "Packet history size", OFFSET(history_size), AV_OPT_TYPE_INT, { .i64 = HISTORY_SIZE_DEFAULT }, 64, 2048, ENC }, { NULL }, }; -- 2.49.0 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".