From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ffbox0-bg.ffmpeg.org (ffbox0-bg.ffmpeg.org [79.124.17.100]) by master.gitmailbox.com (Postfix) with ESMTPS id 8597844B3F for ; Mon, 21 Jul 2025 11:31:29 +0000 (UTC) Received: from [127.0.1.1] (localhost [127.0.0.1]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTP id 4E5F368BD48; Mon, 21 Jul 2025 14:30:54 +0300 (EEST) Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by ffbox0-bg.ffmpeg.org (Postfix) with ESMTPS id ED81568C112 for ; Mon, 21 Jul 2025 14:30:51 +0300 (EEST) Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-749068b9b63so2749634b3a.0 for ; Mon, 21 Jul 2025 04:30:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1753097450; x=1753702250; darn=ffmpeg.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cjUfeR1Xh1WRVK5SiuUn7Y5at7gptJIp34log8GFZCo=; b=B2KnLUouWKsSe+WpWePPUiN3Q+enEj2mCT6dxbsy67Xgm/25r1DiwFcil7yvh3G7l7 vO/JXHsCvYXJLTHz38FUUE9uvN5oP9dyqtJp9Ufk1M6HNezu1/cSlnreRKOJLgIjXTuw Pra0iGi/WQnzao7zeZHHK27ISlj19XInrvC8x8pzzzhn7XDle9ptJFtGaOaMfloGGaX0 V97n+RodtyQ1mgKjEY5XsB4V9dSS0onf/Db+SlkGUNyhZbgR4U7cXN6J0ji9kes4/A8M WgJJDGj3FDb0MWq8h5s5Rnon2LVatQaJuTCR7NDj6L1TsSBF6OeaGdLpScdoLJMl16pa nP2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753097450; x=1753702250; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cjUfeR1Xh1WRVK5SiuUn7Y5at7gptJIp34log8GFZCo=; b=X+ZOudk2YNLI1Ic/ePq49xdDc+Mjelb5D0zfv7moMpOKzCukEh5PclVAejByhkoN5C ymf5flKE4fYZLf3snqhwQtnCF0uG2UJc/bivAQf/TfP7zIOz0BOoo/JamDbCIrjthEDg m+oHiQ/VHUb0mtijgOEuVrFgKKCluPpaiMy85KTqhwXEiEj20kwhNN8CQvjaD/pq1M0U oH0JbdfXteOs3SqhmW5SQRT7ZnHrVUqeinXixEhHLAkupEul6iFBSLiaONxuW6psHSdk 9saaDzCdxxEEaOfUDgtq1C0SF99QixIQ7wWGsljb9cL8PJRtWv9Nflp0S+ORby7b3MD5 2Qig== X-Gm-Message-State: AOJu0YyfxtTouEbCnBLs5Yc+xM7ndT5MHtAaS98tafzoB4S0RQZQ7bjC G+sTc0rxzk8uQc4KQDibwk9b3Wo6iOgBSQsDVtzUOyoWdjps5ip1yk9Vl/oH4bx5 X-Gm-Gg: ASbGncuqoIwXvr7UKT91FeDoFqhgD0gldkjIKzVH4+XiuWhPEBGPU3sPa2C8FS34Osz WwRe5gw9MlNDQBJh/FGikfXCzUMwWKdhSBLYX+LO6f/A1POw75GSzvkm1H6dgFFUBuo4D2y1DYn MyfwwEw9Q6I8Z8308KcAPyEIHKdGu2Lp97ua1Zib8ykEhKNR5Jl2gRh/vdKmt+6OZRmcw2ASamv tFQamIU96qPdLbJd6ScRzE2JF8K3Vmrd0hrKpuiHG0qRPNMK6fCj7Qp3GG2DJPl6WoOINsDk32X /M6g60Oizh8LEb1cCcwfRYpZh9tiut9vBzMTdZ0cR46DBjJw9lTeaIhKOSTz8i8pghJQ2oaFpXf ISVO6KGJWSmrxIxsTJZyheBcMsIUgPGzW X-Google-Smtp-Source: AGHT+IEZhAq190HEvrhMoGKiVMiSAFxZvoYLMpUr3/Y8gdYcROLFKUqpUb/jQm6UkoW7SYePFoh6PQ== X-Received: by 2002:a05:6a00:189b:b0:748:edf5:95de with SMTP id d2e1a72fcca58-756e81a1279mr25363231b3a.10.1753097449600; Mon, 21 Jul 2025 04:30:49 -0700 (PDT) Received: from localhost.localdomain ([5.34.218.160]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-759cbd68350sm5630277b3a.148.2025.07.21.04.30.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 21 Jul 2025 04:30:49 -0700 (PDT) From: Jack Lau X-Google-Original-From: Jack Lau To: ffmpeg-devel@ffmpeg.org Date: Mon, 21 Jul 2025 19:30:16 +0800 Message-ID: <20250721113023.91931-6-jacklau1222@qq.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250721113023.91931-1-jacklau1222@qq.com> References: <20250721113023.91931-1-jacklau1222@qq.com> MIME-Version: 1.0 X-Unsent: 1 Subject: [FFmpeg-devel] [PATCH v4 05/11] avformat/whip: implement NACK and RTX suppport X-BeenThere: ffmpeg-devel@ffmpeg.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FFmpeg development discussions and patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: FFmpeg development discussions and patches Cc: Sergio Garcia Murillo , Jack Lau Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: ffmpeg-devel-bounces@ffmpeg.org Sender: "ffmpeg-devel" Archived-At: List-Archive: List-Post: RTP retransmission described in RFC4588 (RTX) is an effective packet loss recovery technique for real-time applications with relaxed delay bounds. This patch provides a minimal implementation for RTX and RTCP NACK (RFC3940) and its associated SDP signaling and negotiation. Co-authored-by: Sergio Garcia Murillo Signed-off-by: Jack Lau --- libavformat/whip.c | 206 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 202 insertions(+), 4 deletions(-) diff --git a/libavformat/whip.c b/libavformat/whip.c index 4de8eb2601..5907865d22 100644 --- a/libavformat/whip.c +++ b/libavformat/whip.c @@ -114,6 +114,7 @@ /* Referring to Chrome's definition of RTP payload types. */ #define WHIP_RTP_PAYLOAD_TYPE_H264 106 #define WHIP_RTP_PAYLOAD_TYPE_OPUS 111 +#define WHIP_RTP_PAYLOAD_TYPE_RTX 105 /** * The STUN message header, which is 20 bytes long, comprises the @@ -150,6 +151,11 @@ #define WHIP_SDP_SESSION_ID "4489045141692799359" #define WHIP_SDP_CREATOR_IP "127.0.0.1" +/** + * Retransmission / NACK support +*/ +#define HISTORY_SIZE_DEFAULT 512 + /* Calculate the elapsed time from starttime to endtime in milliseconds. */ #define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000) @@ -194,9 +200,16 @@ enum WHIPState { }; typedef enum WHIPFlags { - WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate + WHIP_FLAG_IGNORE_IPV6 = (1 << 0), // Ignore ipv6 candidate + WHIP_FLAG_DISABLE_RTX = (1 << 1) // Enable NACK and RTX } WHIPFlags; +typedef struct RtpHistoryItem { + uint16_t seq; // original RTP seq + int size; // length in bytes + uint8_t* buf; // malloc-ed copy +} RtpHistoryItem; + typedef struct WHIPContext { AVClass *av_class; @@ -285,6 +298,7 @@ typedef struct WHIPContext { /* The SRTP send context, to encrypt outgoing packets. */ SRTPContext srtp_audio_send; SRTPContext srtp_video_send; + SRTPContext srtp_video_rtx_send; SRTPContext srtp_rtcp_send; /* The SRTP receive context, to decrypt incoming packets. */ SRTPContext srtp_recv; @@ -309,6 +323,14 @@ typedef struct WHIPContext { /* The certificate and private key used for DTLS handshake. */ char* cert_file; char* key_file; + + /* RTX and NACK */ + uint8_t rtx_payload_type; + uint32_t video_rtx_ssrc; + uint16_t rtx_seq; + int history_size; + RtpHistoryItem *history; /* ring buffer */ + int hist_head; } WHIPContext; /** @@ -611,6 +633,16 @@ static int generate_sdp_offer(AVFormatContext *s) whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS; whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264; + /* RTX and NACK init */ + whip->rtx_payload_type = WHIP_RTP_PAYLOAD_TYPE_RTX; + whip->video_rtx_ssrc = av_lfg_get(&whip->rnd); + whip->rtx_seq = 0; + whip->hist_head = 0; + whip->history_size = FFMAX(64, whip->history_size); + whip->history = av_calloc(whip->history_size, sizeof(*whip->history)); + if (!whip->history) + return AVERROR(ENOMEM); + av_bprintf(&bp, "" "v=0\r\n" "o=FFmpeg %s 2 IN IP4 %s\r\n" @@ -661,7 +693,7 @@ static int generate_sdp_offer(AVFormatContext *s) } av_bprintf(&bp, "" - "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n" + "m=video 9 UDP/TLS/RTP/SAVPF %u %u\r\n" "c=IN IP4 0.0.0.0\r\n" "a=ice-ufrag:%s\r\n" "a=ice-pwd:%s\r\n" @@ -674,9 +706,16 @@ static int generate_sdp_offer(AVFormatContext *s) "a=rtcp-rsize\r\n" "a=rtpmap:%u %s/90000\r\n" "a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n" + "a=rtcp-fb:%u nack\r\n" + "a=rtpmap:%u rtx/90000\r\n" + "a=fmtp:%u apt=%u\r\n" + "a=ssrc-group:FID %u %u\r\n" + "a=ssrc:%u cname:FFmpeg\r\n" + "a=ssrc:%u msid:FFmpeg video\r\n" "a=ssrc:%u cname:FFmpeg\r\n" "a=ssrc:%u msid:FFmpeg video\r\n", whip->video_payload_type, + whip->rtx_payload_type, whip->ice_ufrag_local, whip->ice_pwd_local, whip->dtls_fingerprint, @@ -686,8 +725,16 @@ static int generate_sdp_offer(AVFormatContext *s) profile, whip->constraint_set_flags, level, + whip->video_payload_type, + whip->rtx_payload_type, + whip->rtx_payload_type, + whip->video_payload_type, + whip->video_ssrc, + whip->video_rtx_ssrc, whip->video_ssrc, - whip->video_ssrc); + whip->video_ssrc, + whip->video_rtx_ssrc, + whip->video_rtx_ssrc); } if (!av_bprint_is_complete(&bp)) { @@ -1400,6 +1447,12 @@ static int setup_srtp(AVFormatContext *s) goto end; } + ret = ff_srtp_set_crypto(&whip->srtp_video_rtx_send, suite, buf); + if (ret < 0) { + av_log(whip, AV_LOG_ERROR, "Failed to set crypto for video rtx send\n"); + goto end; + } + ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf); if (ret < 0) { av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n"); @@ -1429,6 +1482,38 @@ end: return ret; } + +/** + * RTX history helpers + */ + static int rtp_history_store(WHIPContext *whip, const uint8_t *buf, int size) +{ + int pos = whip->hist_head % whip->history_size; + RtpHistoryItem *it = &whip->history[pos]; + /* free older entry */ + av_free(it->buf); + it->buf = av_malloc(size); + if (!it->buf) + return AVERROR(ENOMEM); + + memcpy(it->buf, buf, size); + it->size = size; + it->seq = AV_RB16(buf + 2); + + whip->hist_head = ++pos; + return 0; +} + +static const RtpHistoryItem *rtp_history_find(const WHIPContext *whip, uint16_t seq) +{ + for (int i = 0; i < whip->history_size; i++) { + const RtpHistoryItem *it = &whip->history[i]; + if (it->buf && it->seq == seq) + return it; + } + return NULL; +} + /** * Callback triggered by the RTP muxer when it creates and sends out an RTP packet. * @@ -1465,6 +1550,12 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) return 0; } + /* Store only ORIGINAL video packets (non-RTX, non-RTCP) */ + if (!is_rtcp && is_video) { + ret = rtp_history_store(whip, buf, buf_size); + if (ret < 0) return ret; + } + ret = ffurl_write(whip->udp, whip->buf, cipher_size); if (ret < 0) { av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret); @@ -1473,6 +1564,48 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size) return ret; } +/** + * See https://datatracker.ietf.org/doc/html/rfc4588 + * Build and send a single RTX packet + */ +static int send_rtx_packet(AVFormatContext *s, const uint8_t *orig_pkt_buf, int orig_size) +{ + int ret; + WHIPContext *whip = s->priv_data; + int new_size, cipher_size; + if (whip->flags & WHIP_FLAG_DISABLE_RTX) + return 0; + + /* allocate new buffer: header + 2 + payload */ + if (orig_size + 2 > sizeof(whip->buf)) + return 0; + + memcpy(whip->buf, orig_pkt_buf, orig_size); + + uint8_t *hdr = whip->buf; + uint16_t orig_seq = AV_RB16(hdr + 2); + + /* rewrite header */ + hdr[1] = (hdr[1] & 0x80) | whip->rtx_payload_type; /* keep M bit */ + AV_WB16(hdr + 2, whip->rtx_seq++); + AV_WB32(hdr + 8, whip->video_rtx_ssrc); + + /* shift payload 2 bytes */ + memmove(hdr + 12 + 2, hdr + 12, orig_size - 12); + AV_WB16(hdr + 12, orig_seq); + + new_size = orig_size + 2; + + /* Encrypt by SRTP and send out. */ + cipher_size = ff_srtp_encrypt(&whip->srtp_video_rtx_send, whip->buf, new_size, whip->buf, sizeof(whip->buf)); + if (cipher_size <= 0 || cipher_size < new_size) { + av_log(whip, AV_LOG_WARNING, "Failed to encrypt packet=%dB, cipher=%dB\n", new_size, cipher_size); + return 0; + } + ret = ffurl_write(whip->udp, whip->buf, cipher_size); + if (ret <= 0) av_log(whip, AV_LOG_ERROR, "Failed to send RTX packet\n"); + return ret; +} /** * Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP @@ -1781,6 +1914,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) WHIPContext *whip = s->priv_data; AVStream *st = s->streams[pkt->stream_index]; AVFormatContext *rtp_ctx = st->priv_data; + uint8_t *buf = NULL; /* TODO: Send binding request every 1s as WebRTC heartbeat. */ @@ -1796,11 +1930,72 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) goto end; } } + /** + * Handle RTCP NACK + * Refer to RFC 4585, Section 6.2.1 + * The Generic NACK message is identified by PT=RTPFB and FMT=1. + * TODO: disable retransmisstion when "-tune zerolatency" + */ + if (media_is_rtcp(whip->buf, ret)) { + int ptr = 0; + uint8_t pt = whip->buf[ptr + 1]; + uint8_t fmt = (whip->buf[ptr] & 0x1f); + if (ptr + 4 <= ret && pt == 205 && fmt == 1) { + /** + * Refer to RFC 3550, Section 6.4.1. + * The length of this RTCP packet in 32-bit words minus one, + * including the header and any padding. + */ + int rtcp_len = (AV_RB16(&whip->buf[ptr + 2]) + 1) * 4; + /* SRTCP index(4 bytes) + HMAC (SRTP_AES128_CM_SHA1_80 10bytes) */ + int srtcp_len = rtcp_len + 4 + 10; + if (srtcp_len == ret && rtcp_len >= 12) { + int i = 0; + buf = av_malloc(srtcp_len); + if (!buf) return AVERROR(ENOMEM); + memcpy(buf, whip->buf, srtcp_len); + int ret = ff_srtp_decrypt(&whip->srtp_recv, buf, &srtcp_len); + if (ret < 0) { + av_log(whip, AV_LOG_ERROR, "NACK packet(SRTCP) decrypt failed: %d, Can't send RTX packet\n", ret); + goto write_packet; + } + while (12 + i < rtcp_len) { + /** + * See https://datatracker.ietf.org/doc/html/rfc4585#section-6.1 + * Handle multi NACKs in bundled packet. + */ + uint16_t pid = AV_RB16(&buf[ptr + 12 + i]); + uint16_t blp = AV_RB16(&buf[ptr + 14 + i]); + + /* retransmit pid + any bit set in blp */ + for (int bit = -1; bit < 16; bit++) { + uint16_t seq = (bit < 0) ? pid : pid + bit + 1; + if (bit >= 0 && !(blp & (1 << bit))) + continue; + + const RtpHistoryItem *it = rtp_history_find(whip, seq); + if (it) { + av_log(whip, AV_LOG_VERBOSE, + "NACK, packet found: size: %d, seq=%d, rtx size=%d, lateset stored packet seq:%d\n", + it->size, seq, ret, whip->history[whip->hist_head-1].seq); + send_rtx_packet(s, it->buf, it->size); + } else { + av_log(whip, AV_LOG_VERBOSE, + "NACK, packet not found, seq=%d, latest stored packet seq: %d, latest rtx seq: %d\n", + seq, whip->history[whip->hist_head-1].seq, whip->rtx_seq); + } + } + i = i + 4; + } + av_freep(&buf); + } + } + } } else if (ret != AVERROR(EAGAIN)) { av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n"); goto end; } - +write_packet: if (whip->h264_annexb_insert_sps_pps && st->codecpar->codec_id == AV_CODEC_ID_H264) { if ((ret = h264_annexb_insert_sps_pps(s, pkt)) < 0) { av_log(whip, AV_LOG_ERROR, "Failed to insert SPS/PPS before IDR\n"); @@ -1819,6 +2014,7 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt) } end: + if (buf) av_freep(&buf); if (ret < 0 && whip->state < WHIP_STATE_FAILED) whip->state = WHIP_STATE_FAILED; if (ret >= 0 && whip->state >= WHIP_STATE_FAILED && whip->dtls_ret < 0) @@ -1901,6 +2097,8 @@ static const AVOption options[] = { { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC }, { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" }, { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" }, + { "disable_rtx", "Disable RFC 4588 RTX", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" }, + { "rtx_history_size", "Packet history size", OFFSET(history_size), AV_OPT_TYPE_INT, { .i64 = HISTORY_SIZE_DEFAULT }, 64, 2048, ENC }, { NULL }, }; -- 2.49.0 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".