From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 5FF534C2B9 for ; Wed, 2 Jul 2025 04:26:53 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 2FE8868DCD8; Wed, 2 Jul 2025 07:25:41 +0300 (EEST) Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTPS id C392468DCF1 for ; Wed, 2 Jul 2025 07:25:37 +0300 (EEST) Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-7424ccbef4eso3654223b3a.2 for ; Tue, 01 Jul 2025 21:25:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1751430335; x=1752035135; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=366BCkOyFBshbnz/0Vu1pZ2Lb6lnK+POTLQfFI93ju8=; b=byrxjnln7Yuqv9qFaSEALCOkXW3GEEp+K8Y2vH/uHvvjeoP/KjY3M1pd4DxxuoUeTF 7Ws/k+azYZcTn73do0OyELk1uRoEBN9qsbXosOcvLcMyONSucG+JO1YuQsjTz4Qn4jQA dVRHuzYzoJaiMOUzyiGS1KnMyKzmwx2dmgaoLEfu20MzfcRHZbkOO9lQuIhEiVi3OoU/ uYeDAOWwyszAl/eksJvu7b0iZo38RFbucFmcRECL8aKgVtSjc+or4pdZUYAopn9eVwo0 98honrOABMG45A9H0BURJIs+wNt7Gobwog8gpJ24RhzQcx5ARkMlaPiMEFAIjR/Np45F vrPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751430335; x=1752035135; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=366BCkOyFBshbnz/0Vu1pZ2Lb6lnK+POTLQfFI93ju8=; b=DlbmEhSOr2aAxivVhHnz2SR7eoP653ajfqxFAyNwLvN6zMitmcdDZYSM41goWsHeQv Tgr7tw+2uUP6bihIcJtdcC1zpzvJ7LZEiAUfxTSQKTXK7y0knAY7HDjsDmj8WIW1OoHT bVCClMH1DB0Z53zSI+GhXSerr5w/+y1yMSKf8TEv2k/zcYaYAKVqTZanLIRAz4p6BF+a 6SNasMv+IekqnmMj+ETOrEP5X6WSSew78xWQhgOit9waNPYqVzXLMrLSf4TxTOcoJxC3 Pd58oDH5b7VlJg+6WQIQRj9r22jpA1s9vV5Io/3bgtm62xIyFysTbI+T/FZKfvPUtY6s dtFQ== X-Gm-Message-State: AOJu0YxO++mPPSuvLIYvREzHZ2JfbHyx1A+5XaFtcwWuAUALMbecREdi 4B23zklCD8i7IuQO8fUl4QpUzdU6WpbJHVOR7dKzvPEnumn/m2WdK69MW6/mFLavYNrhpg== X-Gm-Gg: ASbGnctrC5LCB+sVN7T0ICSFNjCxSKALoHMFCvMax0/wbzUdOPGMxxscuEsptOXRL9a 6CjeUbYLRoXcgwnoi3mLQfKL0R+v7fTYl9E1dVsuAFPTIxLoN+n92sYuQWmcKBBCCOra1d4tfFA WQ8qQ76USSjVtDmJPlCrmUOzbDk57yOeBbRI4Y8DIbdDV0+vZlvjbqxof2nJHHw+esj1N+bqe2i v+V4NWR8Jz4u0clGZq1/SHqr90am0NF0Z0AMZxWu/MxYc8Rq0UGnpGY+/lUWs9KMryhs+NpUN4d guZ2N5TT5OtKjLkoT704STtlVqBPbLNLZ/E8KfQx+m0TKb0gVG8pVp4+CFWmTo+KGA== X-Google-Smtp-Source: AGHT+IGFHSYCEOtX8NkUDryLR+GzjgRXR8DZCdQOm1LCIFNCagO89pogaDu7V5aU00De+n4U/d6y/g== X-Received: by 2002:a17:903:1b4f:b0:235:779:edfd with SMTP id d9443c01a7336-23c6e570636mr15857975ad.39.1751430334941; Tue, 01 Jul 2025 21:25:34 -0700 (PDT) Received: from localhost.localdomain ([5.34.218.76]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23acb2e258csm127670575ad.38.2025.07.01.21.25.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 01 Jul 2025 21:25:34 -0700 (PDT) From: Jack Lau X-Google-Original-From: Jack Lau To: ffmpeg-devel@ffmpeg.org Date: Wed, 2 Jul 2025 12:25:05 +0800 Message-ID: <20250702042505.72292-7-jacklau1222@qq.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250702042505.72292-6-jacklau1222@qq.com> References: <20250702042505.72292-1-jacklau1222@qq.com> <20250702042505.72292-2-jacklau1222@qq.com> <20250702042505.72292-3-jacklau1222@qq.com> <20250702042505.72292-4-jacklau1222@qq.com> <20250702042505.72292-5-jacklau1222@qq.com> <20250702042505.72292-6-jacklau1222@qq.com> MIME-Version: 1.0 X-Unsent: 1 Subject: [FFmpeg-devel] [PATCH 6/6] avformat/whip: implement NACK and RTX suppport X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Sergio Garcia Murillo , Jack Lau Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: RTP retransmission described in RFC4588 (RTX) is an effective packet loss recovery technique for real-time applications with relaxed delay bounds. This patch provides a minimal implementation for RTX and RTCP NACK (RFC3940) and its associated SDP signaling and negotiation. Co-authored-by: Sergio Garcia Murillo Signed-off-by: Jack Lau --- libavformat/whip.c | 198 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 195 insertions(+), 3 deletions(-) diff --git a/libavformat/whip.c b/libavformat/whip.c index 15d1de691e..a32177e0b4 100644 --- a/libavformat/whip.c +++ b/libavformat/whip.c @@ -114,6 +114,7 @@ /* Referring to Chrome's definition of RTP payload types. */ #define WHIP_RTP_PAYLOAD_TYPE_H264 106 #define WHIP_RTP_PAYLOAD_TYPE_OPUS 111 +#define WHIP_RTP_PAYLOAD_TYPE_RTX 105 /** * The STUN message header, which is 20 bytes long, comprises the @@ -150,6 +151,11 @@ #define WHIP_SDP_SESSION_ID "4489045141692799359" #define WHIP_SDP_CREATOR_IP "127.0.0.1" +/** + * Retransmission / NACK support +*/ +#define HISTORY_SIZE_DEFAULT 512 + /* Calculate the elapsed time from starttime to endtime in milliseconds. */ #define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000) @@ -194,9 +200,19 @@ enum WHIPState { }; typedef enum WHIPFlags { - WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate + WHIP_FLAG_IGNORE_IPV6 = (1 << 0), // Ignore ipv6 candidate + WHIP_FLAG_DISABLE_RTX = (1 << 1) // Enable NACK and RTX } WHIPFlags; +typedef struct RtpHistoryItem { + /* original RTP seq */ + uint16_t seq; + /* length in bytes */ + int size; + /* malloc-ed copy */ + uint8_t* pkt; +} RtpHistoryItem; + typedef struct WHIPContext { AVClass *av_class; @@ -285,6 +301,7 @@ typedef struct WHIPContext { /* The SRTP send context, to encrypt outgoing packets. */ SRTPContext srtp_audio_send; SRTPContext srtp_video_send; + SRTPContext srtp_video_rtx_send; SRTPContext srtp_rtcp_send; /* The SRTP receive context, to decrypt incoming packets. */ SRTPContext srtp_recv; @@ -309,6 +326,14 @@ typedef struct WHIPContext { /* The certificate and private key used for DTLS handshake. */ char* cert_file; char* key_file; + + /* RTX / NACK */ + uint8_t rtx_payload_type; + uint32_t video_rtx_ssrc; + uint16_t rtx_seq; + int history_size; + RtpHistoryItem *history; /* ring buffer */ + int hist_head; } WHIPContext; /** @@ -606,6 +631,16 @@ static int generate_sdp_offer(AVFormatContext *s) whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS; whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264; + /* RTX and NACK init */ + whip->rtx_payload_type = WHIP_RTP_PAYLOAD_TYPE_RTX; + whip->video_rtx_ssrc = av_lfg_get(&whip->rnd); + whip->rtx_seq = 0; + whip->hist_head = 0; + whip->history_size = FFMAX(64, whip->history_size); + whip->history = av_calloc(whip->history_size, sizeof(*whip->history)); + if (!whip->history) + return AVERROR(ENOMEM); + av_bprintf(&bp, "" "v=0\r\n" "o=FFmpeg %s 2 IN IP4 %s\r\n" @@ -656,7 +691,7 @@ static int generate_sdp_offer(AVFormatContext *s) } av_bprintf(&bp, "" - "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n" + "m=video 9 UDP/TLS/RTP/SAVPF %u %u\r\n" "c=IN IP4 0.0.0.0\r\n" "a=ice-ufrag:%s\r\n" "a=ice-pwd:%s\r\n" @@ -669,9 +704,16 @@ static int generate_sdp_offer(AVFormatContext *s) "a=rtcp-rsize\r\n" "a=rtpmap:%u %s/90000\r\n" "a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n" + "a=rtcp-fb:%u nack\r\n" + "a=rtpmap:%u rtx/90000\r\n" + "a=fmtp:%u apt=%u\r\n" + "a=ssrc-group:FID %u %u\r\n" + "a=ssrc:%u cname:FFmpeg\r\n" + "a=ssrc:%u msid:FFmpeg video\r\n" "a=ssrc:%u cname:FFmpeg\r\n" "a=ssrc:%u msid:FFmpeg video\r\n", whip->video_payload_type, + whip->rtx_payload_type, whip->ice_ufrag_local, whip->ice_pwd_local, whip->dtls_fingerprint, @@ -681,8 +723,16 @@ static int generate_sdp_offer(AVFormatContext *s) profile, whip->constraint_set_flags, level, + whip->video_payload_type, + whip->rtx_payload_type, + whip->rtx_payload_type, + whip->video_payload_type, + whip->video_ssrc, + whip->video_rtx_ssrc, whip->video_ssrc, - whip->video_ssrc); + whip->video_ssrc, + whip->video_rtx_ssrc, + whip->video_rtx_ssrc); } if (!av_bprint_is_complete(&bp)) { @@ -1396,6 +1446,12 @@ static int setup_srtp(AVFormatContext *s) goto end; } + ret = ff_srtp_set_crypto(&whip->srtp_video_rtx_send, suite, buf); + if (ret < 0) { + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for video rtx send\n"); + goto end; + } + ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf); if (ret < 0) { av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n"); @@ -1425,6 +1481,37 @@ end: return ret; } + +/** + * RTX history helpers + */ + static void rtp_history_store(WHIPContext *whip, const uint8_t *pkt, int size) +{ + int pos = whip->hist_head % whip->history_size; + RtpHistoryItem *it = &whip->history[pos]; + /* free older entry */ + av_free(it->pkt); + it->pkt = av_malloc(size); + if (!it->pkt) + return; + + memcpy(it->pkt, pkt, size); + it->size = size; + it->seq = AV_RB16(pkt + 2); + + whip->hist_head = ++pos; +} + +static const RtpHistoryItem *rtp_history_find(const WHIPContext *whip, uint16_t seq) +{ + for (int i = 0; i < whip->history_size; i++) { + const RtpHistoryItem *it = &whip->history[i]; + if (it->pkt && it->seq == seq) + return it; + } + return NULL; +} + /** * Callback triggered by the RTP muxer when it creates and sends out an RTP packet. * @@ -1461,6 +1548,10 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) return 0; } + /* Store only ORIGINAL video packets (non-RTX, non-RTCP) */ + if (!is_rtcp && is_video) + rtp_history_store(whip, buf, buf_size); + ret = ffurl_write(whip->udp, whip->buf, cipher_size); if (ret < 0) { av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret); @@ -1469,6 +1560,45 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) return ret; } +/** + * See https://datatracker.ietf.org/doc/html/rfc4588 + * Build and send a single RTX packet + */ +static int send_rtx_packet(AVFormatContext *s, const uint8_t *orig_pkt, int orig_size) +{ + WHIPContext *whip = s->priv_data; + int new_size, cipher_size; + if (whip->flags & WHIP_FLAG_DISABLE_RTX) + return 0; + + /* allocate new buffer: header + 2 + payload */ + if (orig_size + 2 > sizeof(whip->buf)) + return 0; + + memcpy(whip->buf, orig_pkt, orig_size); + + uint8_t *hdr = whip->buf; + uint16_t orig_seq = AV_RB16(hdr + 2); + + /* rewrite header */ + hdr[1] = (hdr[1] & 0x80) | whip->rtx_payload_type; /* keep M bit */ + AV_WB16(hdr + 2, whip->rtx_seq++); + AV_WB32(hdr + 8, whip->video_rtx_ssrc); + + /* shift payload 2 bytes */ + memmove(hdr + 12 + 2, hdr + 12, orig_size - 12); + AV_WB16(hdr + 12, orig_seq); + + new_size = orig_size + 2; + + /* Encrypt by SRTP and send out. */ + cipher_size = ff_srtp_encrypt(&whip->srtp_video_rtx_send, whip->buf, new_size, whip->buf, sizeof(whip->buf)); + if (cipher_size <= 0 || cipher_size < new_size) { + av_log(whip, AV_LOG_WARNING, "WHIP: Failed to encrypt packet=%dB, cipher=%dB\n", new_size, cipher_size); + return 0; + } + return ffurl_write(whip->udp, whip->buf, cipher_size); +} /** * Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP @@ -1791,6 +1921,66 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) goto end; } } + /** + * Handle RTCP NACK + * Refer to RFC 4585, Section 6.2.1 + * The Generic NACK message is identified by PT=RTPFB and FMT=1. + * TODO: disable retransmisstion when "-tune zerolatency" + */ + if (media_is_rtcp(whip->buf, ret)) { + int ptr = 0; + uint8_t pt = whip->buf[ptr + 1]; + uint8_t fmt = (whip->buf[ptr] & 0x1f); + if (ptr + 4 <= ret && pt == 205 && fmt == 1) { + /** + * Refer to RFC 3550, Section 6.4.1. + * The length of this RTCP packet in 32-bit words minus one, + * including the header and any padding. + */ + int rtcp_len = (AV_RB16(&whip->buf[ptr + 2]) + 1) * 4; + /* SRTCP index(4 bytes) + HMAC (SRTP_AES128_CM_SHA1_80 10bytes) */ + int srtcp_len = rtcp_len + 4 + 10; + if (srtcp_len == ret && rtcp_len >= 12) { + int i = 0; + uint8_t *pkt = av_malloc(srtcp_len); + memcpy(pkt, whip->buf, srtcp_len); + int ret = ff_srtp_decrypt(&whip->srtp_recv, pkt, &srtcp_len); + if (ret < 0) + av_log(whip, AV_LOG_ERROR, "WHIP: SRTCP decrypt failed: %d\n", ret); + while (12 + i < rtcp_len && ret == 0) { + /** + * See https://datatracker.ietf.org/doc/html/rfc4585#section-6.1 + * Handle multi NACKs in bundled packet. + */ + uint16_t pid = AV_RB16(&pkt[ptr + 12 + i]); + uint16_t blp = AV_RB16(&pkt[ptr + 14 + i]); + + /* retransmit pid + any bit set in blp */ + for (int bit = -1; bit < 16; bit++) { + uint16_t seq = (bit < 0) ? pid : pid + bit + 1; + if (bit >= 0 && !(blp & (1 << bit))) + continue; + + const RtpHistoryItem *it = rtp_history_find(whip, seq); + if (it) { + av_log(whip, AV_LOG_VERBOSE, + "WHIP: NACK, packet found: size: %d, seq=%d, rtx size=%d, lateset stored packet seq:%d\n", + it->size, seq, ret, whip->history[whip->hist_head-1].seq); + ret = send_rtx_packet(s, it->pkt, it->size); + if (ret <= 0 && !(whip->flags & WHIP_FLAG_DISABLE_RTX)) + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send RTX packet\n"); + } else { + av_log(whip, AV_LOG_VERBOSE, + "WHIP: NACK, packet not found, seq=%d, latest stored packet seq: %d, latest rtx seq: %d\n", + seq, whip->history[whip->hist_head-1].seq, whip->rtx_seq); + } + } + i = i + 4; + } + av_free(pkt); + } + } + } } else if (ret != AVERROR(EAGAIN)) { av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n"); goto end; @@ -1903,6 +2093,8 @@ static const AVOption options[] = { AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" }, { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" }, + { "disable_rtx", "Disable RFC 4588 RTX", 0, + AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" }, { NULL }, }; -- 2.49.0 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".