* [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset)
@ 2025-07-02 11:59 Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 1/6] avformat/whip: add whip_flags ignore_ipv6 to skip IPv6 ICE candidates Jack Lau
` (5 more replies)
0 siblings, 6 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Jack Lau
Version 2 of https://ffmpeg.org/pipermail/ffmpeg-devel/2025-July/346052.html
This patchset fix the following issues:
* every option break into two new lines (now every of them is only one line)
* lack check HAVE_STRUCT_SOCKADDR_IN6 (add this check)
* change the variable name (pkt -> buf) to avoid misunderstanding
* add rtx_history_size option which can set the size of rtp packet history buffer
Jack Lau (5):
avformat/whip: add whip_flags ignore_ipv6 to skip IPv6 ICE candidates
avformat/whip: fix typos
avformat/whip: fix H264 profile_iop bit map for SDP
avformat/whip: implement NACK and RTX suppport
avformat/whip: reindent whip options
winlin (1):
WHIP: X509 cert serial number should be positive.
libavformat/tls_openssl.c | 3 +-
libavformat/whip.c | 410 ++++++++++++++++++++++++++++----------
2 files changed, 303 insertions(+), 110 deletions(-)
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* [FFmpeg-devel] [PATCH v2 1/6] avformat/whip: add whip_flags ignore_ipv6 to skip IPv6 ICE candidates
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
@ 2025-07-02 11:59 ` Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 2/6] avformat/whip: fix typos Jack Lau
` (4 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Jack Lau
mark this ignore_ipv6 flag could ignore any IPv6 ICE candidate,
preventing “No route to host” errors on devices without IPv6 connectivity.
Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
libavformat/whip.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/libavformat/whip.c b/libavformat/whip.c
index 5fdbd6949d..42a6b2ec70 100644
--- a/libavformat/whip.c
+++ b/libavformat/whip.c
@@ -193,9 +193,14 @@ enum WHIPState {
WHIP_STATE_FAILED,
};
+typedef enum WHIPFlags {
+ WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate
+} WHIPFlags;
+
typedef struct WHIPContext {
AVClass *av_class;
+ uint32_t flags; // enum WHIPFlags
/* The state of the RTC connection. */
enum WHIPState state;
/* The callback return value for DTLS. */
@@ -879,6 +884,9 @@ static int parse_answer(AVFormatContext *s)
if (ptr && av_stristr(ptr, "host")) {
char protocol[17], host[129];
int priority, port;
+#if HAVE_STRUCT_SOCKADDR_IN6
+ struct in6_addr addr6;
+#endif
ret = sscanf(ptr, "%16s %d %128s %d typ host", protocol, &priority, host, &port);
if (ret != 4) {
av_log(whip, AV_LOG_ERROR, "WHIP: Failed %d to parse line %d %s from %s\n",
@@ -886,7 +894,12 @@ static int parse_answer(AVFormatContext *s)
ret = AVERROR(EIO);
goto end;
}
-
+#if HAVE_STRUCT_SOCKADDR_IN6
+ if (whip->flags & WHIP_FLAG_IGNORE_IPV6 && inet_pton(AF_INET6, host, &addr6) == 1) {
+ av_log(whip, AV_LOG_DEBUG, "Ignoring IPv6 ICE candidates %s, line %d %s \n", host, i, line);
+ continue;
+ }
+#endif
if (av_strcasecmp(protocol, "udp")) {
av_log(whip, AV_LOG_ERROR, "WHIP: Protocol %s is not supported by RTC, choose udp, line %d %s of %s\n",
protocol, i, line, whip->sdp_answer);
@@ -1898,6 +1911,8 @@ static const AVOption options[] = {
{ "authorization", "The optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
{ "cert_file", "The optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
{ "key_file", "The optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" },
+ { "ignore_ipv6", "(Optional) Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
{ NULL },
};
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* [FFmpeg-devel] [PATCH v2 2/6] avformat/whip: fix typos
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 1/6] avformat/whip: add whip_flags ignore_ipv6 to skip IPv6 ICE candidates Jack Lau
@ 2025-07-02 11:59 ` Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 3/6] avformat/whip: fix H264 profile_iop bit map for SDP Jack Lau
` (3 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Jack Lau
Remove redundant "WHIP: " prefix in log context
since it already add whip context.
Fix grammers in whip options descriptions
Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
libavformat/whip.c | 152 ++++++++++++++++++++++-----------------------
1 file changed, 76 insertions(+), 76 deletions(-)
diff --git a/libavformat/whip.c b/libavformat/whip.c
index 42a6b2ec70..932119a1c7 100644
--- a/libavformat/whip.c
+++ b/libavformat/whip.c
@@ -364,14 +364,14 @@ static int dtls_context_on_state(AVFormatContext *s, const char* type, const cha
if (state == DTLS_STATE_CLOSED) {
whip->dtls_closed = 1;
- av_log(whip, AV_LOG_VERBOSE, "WHIP: DTLS session closed, type=%s, desc=%s, elapsed=%dms\n",
+ av_log(whip, AV_LOG_VERBOSE, "DTLS session closed, type=%s, desc=%s, elapsed=%dms\n",
type ? type : "", desc ? desc : "", ELAPSED(whip->whip_starttime, av_gettime()));
goto error;
}
if (state == DTLS_STATE_FAILED) {
whip->state = WHIP_STATE_FAILED;
- av_log(whip, AV_LOG_ERROR, "WHIP: DTLS session failed, type=%s, desc=%s\n",
+ av_log(whip, AV_LOG_ERROR, "DTLS session failed, type=%s, desc=%s\n",
type ? type : "", desc ? desc : "");
whip->dtls_ret = AVERROR(EIO);
goto error;
@@ -380,7 +380,7 @@ static int dtls_context_on_state(AVFormatContext *s, const char* type, const cha
if (state == DTLS_STATE_FINISHED && whip->state < WHIP_STATE_DTLS_FINISHED) {
whip->state = WHIP_STATE_DTLS_FINISHED;
whip->whip_dtls_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: DTLS handshake is done, elapsed=%dms\n",
+ av_log(whip, AV_LOG_VERBOSE, "DTLS handshake is done, elapsed=%dms\n",
ELAPSED(whip->whip_starttime, av_gettime()));
return ret;
}
@@ -409,7 +409,7 @@ static av_cold int initialize(AVFormatContext *s)
ret = certificate_key_init(s);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to init certificate and key\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to init certificate and key\n");
return ret;
}
@@ -418,13 +418,13 @@ static av_cold int initialize(AVFormatContext *s)
av_lfg_init(&whip->rnd, seed);
if (whip->pkt_size < ideal_pkt_size)
- av_log(whip, AV_LOG_WARNING, "WHIP: pkt_size=%d(<%d) is too small, may cause packet loss\n",
+ av_log(whip, AV_LOG_WARNING, "pkt_size=%d(<%d) is too small, may cause packet loss\n",
whip->pkt_size, ideal_pkt_size);
if (whip->state < WHIP_STATE_INIT)
whip->state = WHIP_STATE_INIT;
whip->whip_init_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: Init state=%d, handshake_timeout=%dms, pkt_size=%d, seed=%d, elapsed=%dms\n",
+ av_log(whip, AV_LOG_VERBOSE, "Init state=%d, handshake_timeout=%dms, pkt_size=%d, seed=%d, elapsed=%dms\n",
whip->state, whip->handshake_timeout, whip->pkt_size, seed, ELAPSED(whip->whip_starttime, av_gettime()));
return 0;
@@ -457,7 +457,7 @@ static int parse_profile_level(AVFormatContext *s, AVCodecParameters *par)
return ret;
if (!par->extradata || par->extradata_size <= 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Unable to parse profile from empty extradata=%p, size=%d\n",
+ av_log(whip, AV_LOG_ERROR, "Unable to parse profile from empty extradata=%p, size=%d\n",
par->extradata, par->extradata_size);
return AVERROR(EINVAL);
}
@@ -471,12 +471,12 @@ static int parse_profile_level(AVFormatContext *s, AVCodecParameters *par)
if ((state & 0x1f) == H264_NAL_SPS) {
ret = ff_avc_decode_sps(sps, r, r1 - r);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to decode SPS, state=%x, size=%d\n",
+ av_log(whip, AV_LOG_ERROR, "Failed to decode SPS, state=%x, size=%d\n",
state, (int)(r1 - r));
return ret;
}
- av_log(whip, AV_LOG_VERBOSE, "WHIP: Parse profile=%d, level=%d from SPS\n",
+ av_log(whip, AV_LOG_VERBOSE, "Parse profile=%d, level=%d from SPS\n",
sps->profile_idc, sps->level_idc);
par->profile = sps->profile_idc;
par->level = sps->level_idc;
@@ -520,62 +520,62 @@ static int parse_codec(AVFormatContext *s)
switch (par->codec_type) {
case AVMEDIA_TYPE_VIDEO:
if (whip->video_par) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Only one video stream is supported by RTC\n");
+ av_log(whip, AV_LOG_ERROR, "Only one video stream is supported by RTC\n");
return AVERROR(EINVAL);
}
whip->video_par = par;
if (par->codec_id != AV_CODEC_ID_H264) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported video codec %s by RTC, choose h264\n",
+ av_log(whip, AV_LOG_ERROR, "Unsupported video codec %s by RTC, choose h264\n",
desc ? desc->name : "unknown");
return AVERROR_PATCHWELCOME;
}
if (par->video_delay > 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported B frames by RTC\n");
+ av_log(whip, AV_LOG_ERROR, "Unsupported B frames by RTC\n");
return AVERROR_PATCHWELCOME;
}
if ((ret = parse_profile_level(s, par)) < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to parse SPS/PPS from extradata\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to parse SPS/PPS from extradata\n");
return AVERROR(EINVAL);
}
if (par->profile == AV_PROFILE_UNKNOWN) {
- av_log(whip, AV_LOG_WARNING, "WHIP: No profile found in extradata, consider baseline\n");
+ av_log(whip, AV_LOG_WARNING, "No profile found in extradata, consider baseline\n");
return AVERROR(EINVAL);
}
if (par->level == AV_LEVEL_UNKNOWN) {
- av_log(whip, AV_LOG_WARNING, "WHIP: No level found in extradata, consider 3.1\n");
+ av_log(whip, AV_LOG_WARNING, "No level found in extradata, consider 3.1\n");
return AVERROR(EINVAL);
}
break;
case AVMEDIA_TYPE_AUDIO:
if (whip->audio_par) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Only one audio stream is supported by RTC\n");
+ av_log(whip, AV_LOG_ERROR, "Only one audio stream is supported by RTC\n");
return AVERROR(EINVAL);
}
whip->audio_par = par;
if (par->codec_id != AV_CODEC_ID_OPUS) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported audio codec %s by RTC, choose opus\n",
+ av_log(whip, AV_LOG_ERROR, "Unsupported audio codec %s by RTC, choose opus\n",
desc ? desc->name : "unknown");
return AVERROR_PATCHWELCOME;
}
if (par->ch_layout.nb_channels != 2) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported audio channels %d by RTC, choose stereo\n",
+ av_log(whip, AV_LOG_ERROR, "Unsupported audio channels %d by RTC, choose stereo\n",
par->ch_layout.nb_channels);
return AVERROR_PATCHWELCOME;
}
if (par->sample_rate != 48000) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Unsupported audio sample rate %d by RTC, choose 48000\n", par->sample_rate);
+ av_log(whip, AV_LOG_ERROR, "Unsupported audio sample rate %d by RTC, choose 48000\n", par->sample_rate);
return AVERROR_PATCHWELCOME;
}
break;
default:
- av_log(whip, AV_LOG_ERROR, "WHIP: Codec type '%s' for stream %d is not supported by RTC\n",
+ av_log(whip, AV_LOG_ERROR, "Codec type '%s' for stream %d is not supported by RTC\n",
av_get_media_type_string(par->codec_type), i);
return AVERROR_PATCHWELCOME;
}
@@ -603,7 +603,7 @@ static int generate_sdp_offer(AVFormatContext *s)
av_bprint_init(&bp, 1, MAX_SDP_SIZE);
if (whip->sdp_offer) {
- av_log(whip, AV_LOG_ERROR, "WHIP: SDP offer is already set\n");
+ av_log(whip, AV_LOG_ERROR, "SDP offer is already set\n");
ret = AVERROR(EINVAL);
goto end;
}
@@ -701,7 +701,7 @@ static int generate_sdp_offer(AVFormatContext *s)
}
if (!av_bprint_is_complete(&bp)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Offer exceed max %d, %s\n", MAX_SDP_SIZE, bp.str);
+ av_log(whip, AV_LOG_ERROR, "Offer exceed max %d, %s\n", MAX_SDP_SIZE, bp.str);
ret = AVERROR(EIO);
goto end;
}
@@ -715,7 +715,7 @@ static int generate_sdp_offer(AVFormatContext *s)
if (whip->state < WHIP_STATE_OFFER)
whip->state = WHIP_STATE_OFFER;
whip->whip_offer_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: Generated state=%d, offer: %s\n", whip->state, whip->sdp_offer);
+ av_log(whip, AV_LOG_VERBOSE, "Generated state=%d, offer: %s\n", whip->state, whip->sdp_offer);
end:
av_bprint_finalize(&bp, NULL);
@@ -750,7 +750,7 @@ static int exchange_sdp(AVFormatContext *s)
}
if (!whip->sdp_offer || !strlen(whip->sdp_offer)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: No offer to exchange\n");
+ av_log(whip, AV_LOG_ERROR, "No offer to exchange\n");
ret = AVERROR(EINVAL);
goto end;
}
@@ -759,7 +759,7 @@ static int exchange_sdp(AVFormatContext *s)
if (whip->authorization)
ret += snprintf(buf + ret, sizeof(buf) - ret, "Authorization: Bearer %s\r\n", whip->authorization);
if (ret <= 0 || ret >= sizeof(buf)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to generate headers, size=%d, %s\n", ret, buf);
+ av_log(whip, AV_LOG_ERROR, "Failed to generate headers, size=%d, %s\n", ret, buf);
ret = AVERROR(EINVAL);
goto end;
}
@@ -778,7 +778,7 @@ static int exchange_sdp(AVFormatContext *s)
ret = ffurl_open_whitelist(&whip_uc, s->url, AVIO_FLAG_READ_WRITE, &s->interrupt_callback,
&opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to request url=%s, offer: %s\n", s->url, whip->sdp_offer);
+ av_log(whip, AV_LOG_ERROR, "Failed to request url=%s, offer: %s\n", s->url, whip->sdp_offer);
goto end;
}
@@ -798,21 +798,21 @@ static int exchange_sdp(AVFormatContext *s)
break;
}
if (ret <= 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read response from url=%s, offer is %s, answer is %s\n",
+ av_log(whip, AV_LOG_ERROR, "Failed to read response from url=%s, offer is %s, answer is %s\n",
s->url, whip->sdp_offer, whip->sdp_answer);
goto end;
}
av_bprintf(&bp, "%.*s", ret, buf);
if (!av_bprint_is_complete(&bp)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Answer exceed max size %d, %.*s, %s\n", MAX_SDP_SIZE, ret, buf, bp.str);
+ av_log(whip, AV_LOG_ERROR, "Answer exceed max size %d, %.*s, %s\n", MAX_SDP_SIZE, ret, buf, bp.str);
ret = AVERROR(EIO);
goto end;
}
}
if (!av_strstart(bp.str, "v=", NULL)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Invalid answer: %s\n", bp.str);
+ av_log(whip, AV_LOG_ERROR, "Invalid answer: %s\n", bp.str);
ret = AVERROR(EINVAL);
goto end;
}
@@ -825,7 +825,7 @@ static int exchange_sdp(AVFormatContext *s)
if (whip->state < WHIP_STATE_ANSWER)
whip->state = WHIP_STATE_ANSWER;
- av_log(whip, AV_LOG_VERBOSE, "WHIP: Got state=%d, answer: %s\n", whip->state, whip->sdp_answer);
+ av_log(whip, AV_LOG_VERBOSE, "Got state=%d, answer: %s\n", whip->state, whip->sdp_answer);
end:
ffurl_closep(&whip_uc);
@@ -856,7 +856,7 @@ static int parse_answer(AVFormatContext *s)
WHIPContext *whip = s->priv_data;
if (!whip->sdp_answer || !strlen(whip->sdp_answer)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: No answer to parse\n");
+ av_log(whip, AV_LOG_ERROR, "No answer to parse\n");
ret = AVERROR(EINVAL);
goto end;
}
@@ -889,7 +889,7 @@ static int parse_answer(AVFormatContext *s)
#endif
ret = sscanf(ptr, "%16s %d %128s %d typ host", protocol, &priority, host, &port);
if (ret != 4) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed %d to parse line %d %s from %s\n",
+ av_log(whip, AV_LOG_ERROR, "Failed %d to parse line %d %s from %s\n",
ret, i, line, whip->sdp_answer);
ret = AVERROR(EIO);
goto end;
@@ -901,7 +901,7 @@ static int parse_answer(AVFormatContext *s)
}
#endif
if (av_strcasecmp(protocol, "udp")) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Protocol %s is not supported by RTC, choose udp, line %d %s of %s\n",
+ av_log(whip, AV_LOG_ERROR, "Protocol %s is not supported by RTC, choose udp, line %d %s of %s\n",
protocol, i, line, whip->sdp_answer);
ret = AVERROR(EIO);
goto end;
@@ -919,19 +919,19 @@ static int parse_answer(AVFormatContext *s)
}
if (!whip->ice_pwd_remote || !strlen(whip->ice_pwd_remote)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: No remote ice pwd parsed from %s\n", whip->sdp_answer);
+ av_log(whip, AV_LOG_ERROR, "No remote ice pwd parsed from %s\n", whip->sdp_answer);
ret = AVERROR(EINVAL);
goto end;
}
if (!whip->ice_ufrag_remote || !strlen(whip->ice_ufrag_remote)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: No remote ice ufrag parsed from %s\n", whip->sdp_answer);
+ av_log(whip, AV_LOG_ERROR, "No remote ice ufrag parsed from %s\n", whip->sdp_answer);
ret = AVERROR(EINVAL);
goto end;
}
if (!whip->ice_protocol || !whip->ice_host || !whip->ice_port) {
- av_log(whip, AV_LOG_ERROR, "WHIP: No ice candidate parsed from %s\n", whip->sdp_answer);
+ av_log(whip, AV_LOG_ERROR, "No ice candidate parsed from %s\n", whip->sdp_answer);
ret = AVERROR(EINVAL);
goto end;
}
@@ -939,7 +939,7 @@ static int parse_answer(AVFormatContext *s)
if (whip->state < WHIP_STATE_NEGOTIATED)
whip->state = WHIP_STATE_NEGOTIATED;
whip->whip_answer_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: SDP state=%d, offer=%luB, answer=%luB, ufrag=%s, pwd=%luB, transport=%s://%s:%d, elapsed=%dms\n",
+ av_log(whip, AV_LOG_VERBOSE, "SDP state=%d, offer=%luB, answer=%luB, ufrag=%s, pwd=%luB, transport=%s://%s:%d, elapsed=%dms\n",
whip->state, strlen(whip->sdp_offer), strlen(whip->sdp_answer), whip->ice_ufrag_remote, strlen(whip->ice_pwd_remote),
whip->ice_protocol, whip->ice_host, whip->ice_port, ELAPSED(whip->whip_starttime, av_gettime()));
@@ -990,7 +990,7 @@ static int ice_create_request(AVFormatContext *s, uint8_t *buf, int buf_size, in
/* The username is the concatenation of the two ICE ufrag */
ret = snprintf(username, sizeof(username), "%s:%s", whip->ice_ufrag_remote, whip->ice_ufrag_local);
if (ret <= 0 || ret >= sizeof(username)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to build username %s:%s, max=%lu, ret=%d\n",
+ av_log(whip, AV_LOG_ERROR, "Failed to build username %s:%s, max=%lu, ret=%d\n",
whip->ice_ufrag_remote, whip->ice_ufrag_local, sizeof(username), ret);
ret = AVERROR(EIO);
goto end;
@@ -1059,7 +1059,7 @@ static int ice_create_response(AVFormatContext *s, char *tid, int tid_size, uint
WHIPContext *whip = s->priv_data;
if (tid_size != 12) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Invalid transaction ID size. Expected 12, got %d\n", tid_size);
+ av_log(whip, AV_LOG_ERROR, "Invalid transaction ID size. Expected 12, got %d\n", tid_size);
return AVERROR(EINVAL);
}
@@ -1162,7 +1162,7 @@ static int ice_handle_binding_request(AVFormatContext *s, char *buf, int buf_siz
return ret;
if (buf_size < ICE_STUN_HEADER_SIZE) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Invalid STUN message, expected at least %d, got %d\n",
+ av_log(whip, AV_LOG_ERROR, "Invalid STUN message, expected at least %d, got %d\n",
ICE_STUN_HEADER_SIZE, buf_size);
return AVERROR(EINVAL);
}
@@ -1173,13 +1173,13 @@ static int ice_handle_binding_request(AVFormatContext *s, char *buf, int buf_siz
/* Build the STUN binding response. */
ret = ice_create_response(s, tid, sizeof(tid), whip->buf, sizeof(whip->buf), &size);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to create STUN binding response, size=%d\n", size);
+ av_log(whip, AV_LOG_ERROR, "Failed to create STUN binding response, size=%d\n", size);
return ret;
}
ret = ffurl_write(whip->udp, whip->buf, size);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send STUN binding response, size=%d\n", size);
+ av_log(whip, AV_LOG_ERROR, "Failed to send STUN binding response, size=%d\n", size);
return ret;
}
@@ -1209,7 +1209,7 @@ static int udp_connect(AVFormatContext *s)
ret = ffurl_open_whitelist(&whip->udp, url, AVIO_FLAG_WRITE, &s->interrupt_callback,
&opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to connect udp://%s:%d\n", whip->ice_host, whip->ice_port);
+ av_log(whip, AV_LOG_ERROR, "Failed to connect udp://%s:%d\n", whip->ice_host, whip->ice_port);
goto end;
}
@@ -1220,7 +1220,7 @@ static int udp_connect(AVFormatContext *s)
if (whip->state < WHIP_STATE_UDP_CONNECTED)
whip->state = WHIP_STATE_UDP_CONNECTED;
whip->whip_udp_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: UDP state=%d, elapsed=%dms, connected to udp://%s:%d\n",
+ av_log(whip, AV_LOG_VERBOSE, "UDP state=%d, elapsed=%dms, connected to udp://%s:%d\n",
whip->state, ELAPSED(whip->whip_starttime, av_gettime()), whip->ice_host, whip->ice_port);
end:
@@ -1238,7 +1238,7 @@ static int ice_dtls_handshake(AVFormatContext *s)
char buf[256], *cert_buf = NULL, *key_buf = NULL;
if (whip->state < WHIP_STATE_UDP_CONNECTED || !whip->udp) {
- av_log(whip, AV_LOG_ERROR, "WHIP: UDP not connected, state=%d, udp=%p\n", whip->state, whip->udp);
+ av_log(whip, AV_LOG_ERROR, "UDP not connected, state=%d, udp=%p\n", whip->state, whip->udp);
return AVERROR(EINVAL);
}
@@ -1247,13 +1247,13 @@ static int ice_dtls_handshake(AVFormatContext *s)
/* Build the STUN binding request. */
ret = ice_create_request(s, whip->buf, sizeof(whip->buf), &size);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to create STUN binding request, size=%d\n", size);
+ av_log(whip, AV_LOG_ERROR, "Failed to create STUN binding request, size=%d\n", size);
goto end;
}
ret = ffurl_write(whip->udp, whip->buf, size);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send STUN binding request, size=%d\n", size);
+ av_log(whip, AV_LOG_ERROR, "Failed to send STUN binding request, size=%d\n", size);
goto end;
}
@@ -1268,7 +1268,7 @@ next_packet:
now = av_gettime();
if (now - starttime >= whip->handshake_timeout * 1000) {
- av_log(whip, AV_LOG_ERROR, "WHIP: DTLS handshake timeout=%dms, cost=%dms, elapsed=%dms, state=%d\n",
+ av_log(whip, AV_LOG_ERROR, "DTLS handshake timeout=%dms, cost=%dms, elapsed=%dms, state=%d\n",
whip->handshake_timeout, ELAPSED(starttime, now), ELAPSED(whip->whip_starttime, now), whip->state);
ret = AVERROR(ETIMEDOUT);
goto end;
@@ -1283,7 +1283,7 @@ next_packet:
av_usleep(5 * 1000);
continue;
}
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read message\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to read message\n");
goto end;
}
@@ -1296,7 +1296,7 @@ next_packet:
if (whip->state < WHIP_STATE_ICE_CONNECTED) {
whip->state = WHIP_STATE_ICE_CONNECTED;
whip->whip_ice_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: ICE STUN ok, state=%d, url=udp://%s:%d, location=%s, username=%s:%s, res=%dB, elapsed=%dms\n",
+ av_log(whip, AV_LOG_VERBOSE, "ICE STUN ok, state=%d, url=udp://%s:%d, location=%s, username=%s:%s, res=%dB, elapsed=%dms\n",
whip->state, whip->ice_host, whip->ice_port, whip->whip_resource_url ? whip->whip_resource_url : "",
whip->ice_ufrag_remote, whip->ice_ufrag_local, ret, ELAPSED(whip->whip_starttime, av_gettime()));
@@ -1396,20 +1396,20 @@ static int setup_srtp(AVFormatContext *s)
/* Setup SRTP context for outgoing packets */
if (!av_base64_encode(buf, sizeof(buf), send_key, sizeof(send_key))) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to encode send key\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to encode send key\n");
ret = AVERROR(EIO);
goto end;
}
ret = ff_srtp_set_crypto(&whip->srtp_audio_send, suite, buf);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for audio send\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to set crypto for audio send\n");
goto end;
}
ret = ff_srtp_set_crypto(&whip->srtp_video_send, suite, buf);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for video send\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to set crypto for video send\n");
goto end;
}
@@ -1421,21 +1421,21 @@ static int setup_srtp(AVFormatContext *s)
/* Setup SRTP context for incoming packets */
if (!av_base64_encode(buf, sizeof(buf), recv_key, sizeof(recv_key))) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to encode recv key\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to encode recv key\n");
ret = AVERROR(EIO);
goto end;
}
ret = ff_srtp_set_crypto(&whip->srtp_recv, suite, buf);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for recv\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to set crypto for recv\n");
goto end;
}
if (whip->state < WHIP_STATE_SRTP_FINISHED)
whip->state = WHIP_STATE_SRTP_FINISHED;
whip->whip_srtp_time = av_gettime();
- av_log(whip, AV_LOG_VERBOSE, "WHIP: SRTP setup done, state=%d, suite=%s, key=%luB, elapsed=%dms\n",
+ av_log(whip, AV_LOG_VERBOSE, "SRTP setup done, state=%d, suite=%s, key=%luB, elapsed=%dms\n",
whip->state, suite, sizeof(send_key), ELAPSED(whip->whip_starttime, av_gettime()));
end:
@@ -1474,13 +1474,13 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
/* Encrypt by SRTP and send out. */
cipher_size = ff_srtp_encrypt(srtp, buf, buf_size, whip->buf, sizeof(whip->buf));
if (cipher_size <= 0 || cipher_size < buf_size) {
- av_log(whip, AV_LOG_WARNING, "WHIP: Failed to encrypt packet=%dB, cipher=%dB\n", buf_size, cipher_size);
+ av_log(whip, AV_LOG_WARNING, "Failed to encrypt packet=%dB, cipher=%dB\n", buf_size, cipher_size);
return 0;
}
ret = ffurl_write(whip->udp, whip->buf, cipher_size);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to write packet=%dB, ret=%d\n", cipher_size, ret);
+ av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret);
return ret;
}
@@ -1509,7 +1509,7 @@ static int create_rtp_muxer(AVFormatContext *s)
const AVOutputFormat *rtp_format = av_guess_format("rtp", NULL, NULL);
if (!rtp_format) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to guess rtp muxer\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to guess rtp muxer\n");
ret = AVERROR(ENOSYS);
goto end;
}
@@ -1577,7 +1577,7 @@ static int create_rtp_muxer(AVFormatContext *s)
ret = avformat_write_header(rtp_ctx, &opts);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to write rtp header\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to write rtp header\n");
goto end;
}
@@ -1589,7 +1589,7 @@ static int create_rtp_muxer(AVFormatContext *s)
if (whip->state < WHIP_STATE_READY)
whip->state = WHIP_STATE_READY;
- av_log(whip, AV_LOG_INFO, "WHIP: Muxer state=%d, buffer_size=%d, max_packet_size=%d, "
+ av_log(whip, AV_LOG_INFO, "Muxer state=%d, buffer_size=%d, max_packet_size=%d, "
"elapsed=%dms(init:%d,offer:%d,answer:%d,udp:%d,ice:%d,dtls:%d,srtp:%d)\n",
whip->state, buffer_size, max_packet_size, ELAPSED(whip->whip_starttime, av_gettime()),
ELAPSED(whip->whip_starttime, whip->whip_init_time),
@@ -1631,7 +1631,7 @@ static int dispose_session(AVFormatContext *s)
if (whip->authorization)
ret += snprintf(buf + ret, sizeof(buf) - ret, "Authorization: Bearer %s\r\n", whip->authorization);
if (ret <= 0 || ret >= sizeof(buf)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to generate headers, size=%d, %s\n", ret, buf);
+ av_log(whip, AV_LOG_ERROR, "Failed to generate headers, size=%d, %s\n", ret, buf);
ret = AVERROR(EINVAL);
goto end;
}
@@ -1642,7 +1642,7 @@ static int dispose_session(AVFormatContext *s)
ret = ffurl_open_whitelist(&whip_uc, whip->whip_resource_url, AVIO_FLAG_READ_WRITE, &s->interrupt_callback,
&opts, s->protocol_whitelist, s->protocol_blacklist, NULL);
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to DELETE url=%s\n", whip->whip_resource_url);
+ av_log(whip, AV_LOG_ERROR, "Failed to DELETE url=%s\n", whip->whip_resource_url);
goto end;
}
@@ -1653,12 +1653,12 @@ static int dispose_session(AVFormatContext *s)
break;
}
if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read response from DELETE url=%s\n", whip->whip_resource_url);
+ av_log(whip, AV_LOG_ERROR, "Failed to read response from DELETE url=%s\n", whip->whip_resource_url);
goto end;
}
}
- av_log(whip, AV_LOG_INFO, "WHIP: Dispose resource %s ok\n", whip->whip_resource_url);
+ av_log(whip, AV_LOG_INFO, "Dispose resource %s ok\n", whip->whip_resource_url);
end:
ffurl_closep(&whip_uc);
@@ -1804,18 +1804,18 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt)
if (ret > 0) {
if (is_dtls_packet(whip->buf, ret)) {
if ((ret = ffurl_write(whip->dtls_uc, whip->buf, ret)) < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to handle DTLS message\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to handle DTLS message\n");
goto end;
}
}
} else if (ret != AVERROR(EAGAIN)) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to read from UDP socket\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n");
goto end;
}
if (whip->h264_annexb_insert_sps_pps && st->codecpar->codec_id == AV_CODEC_ID_H264) {
if ((ret = h264_annexb_insert_sps_pps(s, pkt)) < 0) {
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to insert SPS/PPS before IDR\n");
+ av_log(whip, AV_LOG_ERROR, "Failed to insert SPS/PPS before IDR\n");
goto end;
}
}
@@ -1823,10 +1823,10 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt)
ret = ff_write_chained(rtp_ctx, 0, pkt, s, 0);
if (ret < 0) {
if (ret == AVERROR(EINVAL)) {
- av_log(whip, AV_LOG_WARNING, "WHIP: Ignore failed to write packet=%dB, ret=%d\n", pkt->size, ret);
+ av_log(whip, AV_LOG_WARNING, "Ignore failed to write packet=%dB, ret=%d\n", pkt->size, ret);
ret = 0;
} else
- av_log(whip, AV_LOG_ERROR, "WHIP: Failed to write packet, size=%d\n", pkt->size);
+ av_log(whip, AV_LOG_ERROR, "Failed to write packet, size=%d\n", pkt->size);
goto end;
}
@@ -1847,7 +1847,7 @@ static av_cold void whip_deinit(AVFormatContext *s)
ret = dispose_session(s);
if (ret < 0)
- av_log(whip, AV_LOG_WARNING, "WHIP: Failed to dispose resource, ret=%d\n", ret);
+ av_log(whip, AV_LOG_WARNING, "Failed to dispose resource, ret=%d\n", ret);
for (i = 0; i < s->nb_streams; i++) {
AVFormatContext* rtp_ctx = s->streams[i]->priv_data;
@@ -1894,7 +1894,7 @@ static int whip_check_bitstream(AVFormatContext *s, AVStream *st, const AVPacket
extradata_isom = st->codecpar->extradata_size > 0 && st->codecpar->extradata[0] == 1;
if (pkt->size >= 5 && AV_RB32(b) != 0x0000001 && (AV_RB24(b) != 0x000001 || extradata_isom)) {
ret = ff_stream_add_bitstream_filter(st, "h264_mp4toannexb", NULL);
- av_log(whip, AV_LOG_VERBOSE, "WHIP: Enable BSF h264_mp4toannexb, packet=[%x %x %x %x %x ...], extradata_isom=%d\n",
+ av_log(whip, AV_LOG_VERBOSE, "Enable BSF h264_mp4toannexb, packet=[%x %x %x %x %x ...], extradata_isom=%d\n",
b[0], b[1], b[2], b[3], b[4], extradata_isom);
} else
whip->h264_annexb_insert_sps_pps = 1;
@@ -1908,11 +1908,11 @@ static int whip_check_bitstream(AVFormatContext *s, AVStream *st, const AVPacket
static const AVOption options[] = {
{ "handshake_timeout", "Timeout in milliseconds for ICE and DTLS handshake.", OFFSET(handshake_timeout), AV_OPT_TYPE_INT, { .i64 = 5000 }, -1, INT_MAX, ENC },
{ "pkt_size", "The maximum size, in bytes, of RTP packets that send out", OFFSET(pkt_size), AV_OPT_TYPE_INT, { .i64 = 1200 }, -1, INT_MAX, ENC },
- { "authorization", "The optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
- { "cert_file", "The optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
- { "key_file", "The optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "authorization", "Optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "cert_file", "Optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
{ "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" },
- { "ignore_ipv6", "(Optional) Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
+ { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
{ NULL },
};
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* [FFmpeg-devel] [PATCH v2 3/6] avformat/whip: fix H264 profile_iop bit map for SDP
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 1/6] avformat/whip: add whip_flags ignore_ipv6 to skip IPv6 ICE candidates Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 2/6] avformat/whip: fix typos Jack Lau
@ 2025-07-02 11:59 ` Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 4/6] WHIP: X509 cert serial number should be positive Jack Lau
` (2 subsequent siblings)
5 siblings, 0 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Jack Lau
AVCodecParameters::profile only contains constraint_set1_flag
(AV_PROFILE_H264_CONSTRAINED 1<<9).
So add H264 constraints flag fully parse refer to hlsenc
write_codec_attr
Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
libavformat/whip.c | 47 ++++++++++++++++------------------------------
1 file changed, 16 insertions(+), 31 deletions(-)
diff --git a/libavformat/whip.c b/libavformat/whip.c
index 932119a1c7..e287a3062f 100644
--- a/libavformat/whip.c
+++ b/libavformat/whip.c
@@ -210,6 +210,7 @@ typedef struct WHIPContext {
/* Parameters for the input audio and video codecs. */
AVCodecParameters *audio_par;
AVCodecParameters *video_par;
+ uint8_t constraint_set_flags;
/**
* The h264_mp4toannexb Bitstream Filter (BSF) bypasses the AnnexB packet;
@@ -445,45 +446,30 @@ static av_cold int initialize(AVFormatContext *s)
static int parse_profile_level(AVFormatContext *s, AVCodecParameters *par)
{
int ret = 0;
- const uint8_t *r = par->extradata, *r1, *end = par->extradata + par->extradata_size;
- H264SPS seq, *const sps = &seq;
- uint32_t state;
+ const uint8_t *r = par->extradata;
WHIPContext *whip = s->priv_data;
if (par->codec_id != AV_CODEC_ID_H264)
return ret;
- if (par->profile != AV_PROFILE_UNKNOWN && par->level != AV_LEVEL_UNKNOWN)
- return ret;
-
if (!par->extradata || par->extradata_size <= 0) {
av_log(whip, AV_LOG_ERROR, "Unable to parse profile from empty extradata=%p, size=%d\n",
par->extradata, par->extradata_size);
return AVERROR(EINVAL);
}
- while (1) {
- r = avpriv_find_start_code(r, end, &state);
- if (r >= end)
- break;
-
- r1 = ff_nal_find_startcode(r, end);
- if ((state & 0x1f) == H264_NAL_SPS) {
- ret = ff_avc_decode_sps(sps, r, r1 - r);
- if (ret < 0) {
- av_log(whip, AV_LOG_ERROR, "Failed to decode SPS, state=%x, size=%d\n",
- state, (int)(r1 - r));
- return ret;
- }
-
- av_log(whip, AV_LOG_VERBOSE, "Parse profile=%d, level=%d from SPS\n",
- sps->profile_idc, sps->level_idc);
- par->profile = sps->profile_idc;
- par->level = sps->level_idc;
- }
+ if (AV_RB32(r) == 0x00000001 && (r[4] & 0x1F) == 7)
+ r = &r[5];
+ else if (AV_RB24(r) == 0x000001 && (r[3] & 0x1F) == 7)
+ r = &r[4];
+ else if (r[0] == 0x01) // avcC
+ r = &r[1];
+ else
+ return AVERROR(EINVAL);
- r = r1;
- }
+ if (par->profile == AV_PROFILE_UNKNOWN) par->profile = r[0];
+ whip->constraint_set_flags = r[1];
+ if (par->level == AV_LEVEL_UNKNOWN) par->level = r[2];
return ret;
}
@@ -594,7 +580,7 @@ static int parse_codec(AVFormatContext *s)
*/
static int generate_sdp_offer(AVFormatContext *s)
{
- int ret = 0, profile, level, profile_iop;
+ int ret = 0, profile, level;
const char *acodec_name = NULL, *vcodec_name = NULL;
AVBPrint bp;
WHIPContext *whip = s->priv_data;
@@ -662,11 +648,10 @@ static int generate_sdp_offer(AVFormatContext *s)
}
if (whip->video_par) {
- profile_iop = profile = whip->video_par->profile;
+ profile = whip->video_par->profile;
level = whip->video_par->level;
if (whip->video_par->codec_id == AV_CODEC_ID_H264) {
vcodec_name = "H264";
- profile_iop &= AV_PROFILE_H264_CONSTRAINED;
profile &= (~AV_PROFILE_H264_CONSTRAINED);
}
@@ -694,7 +679,7 @@ static int generate_sdp_offer(AVFormatContext *s)
vcodec_name,
whip->video_payload_type,
profile,
- profile_iop,
+ whip->constraint_set_flags,
level,
whip->video_ssrc,
whip->video_ssrc);
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* [FFmpeg-devel] [PATCH v2 4/6] WHIP: X509 cert serial number should be positive.
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
` (2 preceding siblings ...)
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 3/6] avformat/whip: fix H264 profile_iop bit map for SDP Jack Lau
@ 2025-07-02 11:59 ` Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 6/6] avformat/whip: reindent whip options Jack Lau
5 siblings, 0 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: winlin, Jack Lau
From: winlin <winlinvip@gmail.com>
See RFC5280 4.1.2.2
Co-authored-by: winlin <winlinvip@gmail.com>
Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
libavformat/tls_openssl.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
index 2a3905891d..4733faec9c 100644
--- a/libavformat/tls_openssl.c
+++ b/libavformat/tls_openssl.c
@@ -316,7 +316,8 @@ static int openssl_gen_certificate(EVP_PKEY *pkey, X509 **cert, char **fingerpri
goto enomem_end;
}
- serial = (int)av_get_random_seed();
+ // According to RFC5280 4.1.2.2, The serial number MUST be a positive integer
+ serial = (int)(av_get_random_seed() & 0x7FFFFFFF);
if (ASN1_INTEGER_set(X509_get_serialNumber(*cert), serial) != 1) {
av_log(NULL, AV_LOG_ERROR, "TLS: Failed to set serial, %s\n", ERR_error_string(ERR_get_error(), NULL));
goto einval_end;
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
` (3 preceding siblings ...)
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 4/6] WHIP: X509 cert serial number should be positive Jack Lau
@ 2025-07-02 11:59 ` Jack Lau
2025-07-02 13:46 ` Steven Liu
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 6/6] avformat/whip: reindent whip options Jack Lau
5 siblings, 1 reply; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Sergio Garcia Murillo, Jack Lau
RTP retransmission described in RFC4588 (RTX) is an effective packet
loss recovery technique for real-time applications with relaxed delay bounds.
This patch provides a minimal implementation for RTX and RTCP NACK (RFC3940)
and its associated SDP signaling and negotiation.
Co-authored-by: Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com>
Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
libavformat/whip.c | 198 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 195 insertions(+), 3 deletions(-)
diff --git a/libavformat/whip.c b/libavformat/whip.c
index e287a3062f..fa6e3855d3 100644
--- a/libavformat/whip.c
+++ b/libavformat/whip.c
@@ -114,6 +114,7 @@
/* Referring to Chrome's definition of RTP payload types. */
#define WHIP_RTP_PAYLOAD_TYPE_H264 106
#define WHIP_RTP_PAYLOAD_TYPE_OPUS 111
+#define WHIP_RTP_PAYLOAD_TYPE_RTX 105
/**
* The STUN message header, which is 20 bytes long, comprises the
@@ -150,6 +151,11 @@
#define WHIP_SDP_SESSION_ID "4489045141692799359"
#define WHIP_SDP_CREATOR_IP "127.0.0.1"
+/**
+ * Retransmission / NACK support
+*/
+#define HISTORY_SIZE_DEFAULT 512
+
/* Calculate the elapsed time from starttime to endtime in milliseconds. */
#define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000)
@@ -194,9 +200,19 @@ enum WHIPState {
};
typedef enum WHIPFlags {
- WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate
+ WHIP_FLAG_IGNORE_IPV6 = (1 << 0), // Ignore ipv6 candidate
+ WHIP_FLAG_DISABLE_RTX = (1 << 1) // Enable NACK and RTX
} WHIPFlags;
+typedef struct RtpHistoryItem {
+ /* original RTP seq */
+ uint16_t seq;
+ /* length in bytes */
+ int size;
+ /* malloc-ed copy */
+ uint8_t* buf;
+} RtpHistoryItem;
+
typedef struct WHIPContext {
AVClass *av_class;
@@ -285,6 +301,7 @@ typedef struct WHIPContext {
/* The SRTP send context, to encrypt outgoing packets. */
SRTPContext srtp_audio_send;
SRTPContext srtp_video_send;
+ SRTPContext srtp_video_rtx_send;
SRTPContext srtp_rtcp_send;
/* The SRTP receive context, to decrypt incoming packets. */
SRTPContext srtp_recv;
@@ -309,6 +326,14 @@ typedef struct WHIPContext {
/* The certificate and private key used for DTLS handshake. */
char* cert_file;
char* key_file;
+
+ /* RTX and NACK */
+ uint8_t rtx_payload_type;
+ uint32_t video_rtx_ssrc;
+ uint16_t rtx_seq;
+ int history_size;
+ RtpHistoryItem *history; /* ring buffer */
+ int hist_head;
} WHIPContext;
/**
@@ -606,6 +631,16 @@ static int generate_sdp_offer(AVFormatContext *s)
whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS;
whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264;
+ /* RTX and NACK init */
+ whip->rtx_payload_type = WHIP_RTP_PAYLOAD_TYPE_RTX;
+ whip->video_rtx_ssrc = av_lfg_get(&whip->rnd);
+ whip->rtx_seq = 0;
+ whip->hist_head = 0;
+ whip->history_size = FFMAX(64, whip->history_size);
+ whip->history = av_calloc(whip->history_size, sizeof(*whip->history));
+ if (!whip->history)
+ return AVERROR(ENOMEM);
+
av_bprintf(&bp, ""
"v=0\r\n"
"o=FFmpeg %s 2 IN IP4 %s\r\n"
@@ -656,7 +691,7 @@ static int generate_sdp_offer(AVFormatContext *s)
}
av_bprintf(&bp, ""
- "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n"
+ "m=video 9 UDP/TLS/RTP/SAVPF %u %u\r\n"
"c=IN IP4 0.0.0.0\r\n"
"a=ice-ufrag:%s\r\n"
"a=ice-pwd:%s\r\n"
@@ -669,9 +704,16 @@ static int generate_sdp_offer(AVFormatContext *s)
"a=rtcp-rsize\r\n"
"a=rtpmap:%u %s/90000\r\n"
"a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n"
+ "a=rtcp-fb:%u nack\r\n"
+ "a=rtpmap:%u rtx/90000\r\n"
+ "a=fmtp:%u apt=%u\r\n"
+ "a=ssrc-group:FID %u %u\r\n"
+ "a=ssrc:%u cname:FFmpeg\r\n"
+ "a=ssrc:%u msid:FFmpeg video\r\n"
"a=ssrc:%u cname:FFmpeg\r\n"
"a=ssrc:%u msid:FFmpeg video\r\n",
whip->video_payload_type,
+ whip->rtx_payload_type,
whip->ice_ufrag_local,
whip->ice_pwd_local,
whip->dtls_fingerprint,
@@ -681,8 +723,16 @@ static int generate_sdp_offer(AVFormatContext *s)
profile,
whip->constraint_set_flags,
level,
+ whip->video_payload_type,
+ whip->rtx_payload_type,
+ whip->rtx_payload_type,
+ whip->video_payload_type,
+ whip->video_ssrc,
+ whip->video_rtx_ssrc,
whip->video_ssrc,
- whip->video_ssrc);
+ whip->video_ssrc,
+ whip->video_rtx_ssrc,
+ whip->video_rtx_ssrc);
}
if (!av_bprint_is_complete(&bp)) {
@@ -1398,6 +1448,12 @@ static int setup_srtp(AVFormatContext *s)
goto end;
}
+ ret = ff_srtp_set_crypto(&whip->srtp_video_rtx_send, suite, buf);
+ if (ret < 0) {
+ av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for video rtx send\n");
+ goto end;
+ }
+
ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf);
if (ret < 0) {
av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n");
@@ -1427,6 +1483,37 @@ end:
return ret;
}
+
+/**
+ * RTX history helpers
+ */
+ static void rtp_history_store(WHIPContext *whip, const uint8_t *buf, int size)
+{
+ int pos = whip->hist_head % whip->history_size;
+ RtpHistoryItem *it = &whip->history[pos];
+ /* free older entry */
+ av_free(it->buf);
+ it->buf = av_malloc(size);
+ if (!it->buf)
+ return;
+
+ memcpy(it->buf, buf, size);
+ it->size = size;
+ it->seq = AV_RB16(buf + 2);
+
+ whip->hist_head = ++pos;
+}
+
+static const RtpHistoryItem *rtp_history_find(const WHIPContext *whip, uint16_t seq)
+{
+ for (int i = 0; i < whip->history_size; i++) {
+ const RtpHistoryItem *it = &whip->history[i];
+ if (it->buf && it->seq == seq)
+ return it;
+ }
+ return NULL;
+}
+
/**
* Callback triggered by the RTP muxer when it creates and sends out an RTP packet.
*
@@ -1463,6 +1550,10 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
return 0;
}
+ /* Store only ORIGINAL video packets (non-RTX, non-RTCP) */
+ if (!is_rtcp && is_video)
+ rtp_history_store(whip, buf, buf_size);
+
ret = ffurl_write(whip->udp, whip->buf, cipher_size);
if (ret < 0) {
av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret);
@@ -1471,6 +1562,45 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
return ret;
}
+/**
+ * See https://datatracker.ietf.org/doc/html/rfc4588
+ * Build and send a single RTX packet
+ */
+static int send_rtx_packet(AVFormatContext *s, const uint8_t *orig_pkt_buf, int orig_size)
+{
+ WHIPContext *whip = s->priv_data;
+ int new_size, cipher_size;
+ if (whip->flags & WHIP_FLAG_DISABLE_RTX)
+ return 0;
+
+ /* allocate new buffer: header + 2 + payload */
+ if (orig_size + 2 > sizeof(whip->buf))
+ return 0;
+
+ memcpy(whip->buf, orig_pkt_buf, orig_size);
+
+ uint8_t *hdr = whip->buf;
+ uint16_t orig_seq = AV_RB16(hdr + 2);
+
+ /* rewrite header */
+ hdr[1] = (hdr[1] & 0x80) | whip->rtx_payload_type; /* keep M bit */
+ AV_WB16(hdr + 2, whip->rtx_seq++);
+ AV_WB32(hdr + 8, whip->video_rtx_ssrc);
+
+ /* shift payload 2 bytes */
+ memmove(hdr + 12 + 2, hdr + 12, orig_size - 12);
+ AV_WB16(hdr + 12, orig_seq);
+
+ new_size = orig_size + 2;
+
+ /* Encrypt by SRTP and send out. */
+ cipher_size = ff_srtp_encrypt(&whip->srtp_video_rtx_send, whip->buf, new_size, whip->buf, sizeof(whip->buf));
+ if (cipher_size <= 0 || cipher_size < new_size) {
+ av_log(whip, AV_LOG_WARNING, "WHIP: Failed to encrypt packet=%dB, cipher=%dB\n", new_size, cipher_size);
+ return 0;
+ }
+ return ffurl_write(whip->udp, whip->buf, cipher_size);
+}
/**
* Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP
@@ -1793,6 +1923,66 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt)
goto end;
}
}
+ /**
+ * Handle RTCP NACK
+ * Refer to RFC 4585, Section 6.2.1
+ * The Generic NACK message is identified by PT=RTPFB and FMT=1.
+ * TODO: disable retransmisstion when "-tune zerolatency"
+ */
+ if (media_is_rtcp(whip->buf, ret)) {
+ int ptr = 0;
+ uint8_t pt = whip->buf[ptr + 1];
+ uint8_t fmt = (whip->buf[ptr] & 0x1f);
+ if (ptr + 4 <= ret && pt == 205 && fmt == 1) {
+ /**
+ * Refer to RFC 3550, Section 6.4.1.
+ * The length of this RTCP packet in 32-bit words minus one,
+ * including the header and any padding.
+ */
+ int rtcp_len = (AV_RB16(&whip->buf[ptr + 2]) + 1) * 4;
+ /* SRTCP index(4 bytes) + HMAC (SRTP_AES128_CM_SHA1_80 10bytes) */
+ int srtcp_len = rtcp_len + 4 + 10;
+ if (srtcp_len == ret && rtcp_len >= 12) {
+ int i = 0;
+ uint8_t *buf = av_malloc(srtcp_len);
+ memcpy(buf, whip->buf, srtcp_len);
+ int ret = ff_srtp_decrypt(&whip->srtp_recv, buf, &srtcp_len);
+ if (ret < 0)
+ av_log(whip, AV_LOG_ERROR, "WHIP: SRTCP decrypt failed: %d\n", ret);
+ while (12 + i < rtcp_len && !ret) {
+ /**
+ * See https://datatracker.ietf.org/doc/html/rfc4585#section-6.1
+ * Handle multi NACKs in bundled packet.
+ */
+ uint16_t pid = AV_RB16(&buf[ptr + 12 + i]);
+ uint16_t blp = AV_RB16(&buf[ptr + 14 + i]);
+
+ /* retransmit pid + any bit set in blp */
+ for (int bit = -1; bit < 16; bit++) {
+ uint16_t seq = (bit < 0) ? pid : pid + bit + 1;
+ if (bit >= 0 && !(blp & (1 << bit)))
+ continue;
+
+ const RtpHistoryItem *it = rtp_history_find(whip, seq);
+ if (it) {
+ av_log(whip, AV_LOG_VERBOSE,
+ "WHIP: NACK, packet found: size: %d, seq=%d, rtx size=%d, lateset stored packet seq:%d\n",
+ it->size, seq, ret, whip->history[whip->hist_head-1].seq);
+ ret = send_rtx_packet(s, it->buf, it->size);
+ if (ret <= 0 && !(whip->flags & WHIP_FLAG_DISABLE_RTX))
+ av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send RTX packet\n");
+ } else {
+ av_log(whip, AV_LOG_VERBOSE,
+ "WHIP: NACK, packet not found, seq=%d, latest stored packet seq: %d, latest rtx seq: %d\n",
+ seq, whip->history[whip->hist_head-1].seq, whip->rtx_seq);
+ }
+ }
+ i = i + 4;
+ }
+ av_free(buf);
+ }
+ }
+ }
} else if (ret != AVERROR(EAGAIN)) {
av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n");
goto end;
@@ -1898,6 +2088,8 @@ static const AVOption options[] = {
{ "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
{ "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" },
{ "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
+ { "disable_rtx", "Disable RFC 4588 RTX", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" },
+ { "rtx_history_size", "Packet history size", OFFSET(history_size), AV_OPT_TYPE_INT, { .i64 = HISTORY_SIZE_DEFAULT }, 64, 2048, ENC },
{ NULL },
};
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* [FFmpeg-devel] [PATCH v2 6/6] avformat/whip: reindent whip options
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
` (4 preceding siblings ...)
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport Jack Lau
@ 2025-07-02 11:59 ` Jack Lau
5 siblings, 0 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 11:59 UTC (permalink / raw)
To: ffmpeg-devel; +Cc: Jack Lau
Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
libavformat/whip.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/libavformat/whip.c b/libavformat/whip.c
index fa6e3855d3..0c44ef6c73 100644
--- a/libavformat/whip.c
+++ b/libavformat/whip.c
@@ -2081,13 +2081,13 @@ static int whip_check_bitstream(AVFormatContext *s, AVStream *st, const AVPacket
#define OFFSET(x) offsetof(WHIPContext, x)
#define ENC AV_OPT_FLAG_ENCODING_PARAM
static const AVOption options[] = {
- { "handshake_timeout", "Timeout in milliseconds for ICE and DTLS handshake.", OFFSET(handshake_timeout), AV_OPT_TYPE_INT, { .i64 = 5000 }, -1, INT_MAX, ENC },
- { "pkt_size", "The maximum size, in bytes, of RTP packets that send out", OFFSET(pkt_size), AV_OPT_TYPE_INT, { .i64 = 1200 }, -1, INT_MAX, ENC },
- { "authorization", "Optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
- { "cert_file", "Optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
- { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
- { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" },
- { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
+ { "handshake_timeout", "Timeout in milliseconds for ICE and DTLS handshake.", OFFSET(handshake_timeout), AV_OPT_TYPE_INT, { .i64 = 5000 }, -1, INT_MAX, ENC },
+ { "pkt_size", "The maximum size, in bytes, of RTP packets that send out", OFFSET(pkt_size), AV_OPT_TYPE_INT, { .i64 = 1200 }, -1, INT_MAX, ENC },
+ { "authorization", "Optional Bearer token for WHIP Authorization", OFFSET(authorization), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "cert_file", "Optional certificate file path for DTLS", OFFSET(cert_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
+ { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, 0, ENC, .unit = "flags" },
+ { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
{ "disable_rtx", "Disable RFC 4588 RTX", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" },
{ "rtx_history_size", "Packet history size", OFFSET(history_size), AV_OPT_TYPE_INT, { .i64 = HISTORY_SIZE_DEFAULT }, 64, 2048, ENC },
{ NULL },
--
2.49.0
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport Jack Lau
@ 2025-07-02 13:46 ` Steven Liu
2025-07-02 14:10 ` Jack Lau
0 siblings, 1 reply; 9+ messages in thread
From: Steven Liu @ 2025-07-02 13:46 UTC (permalink / raw)
To: FFmpeg development discussions and patches
Cc: Sergio Garcia Murillo, Jack Lau
Jack Lau <jacklau1222gm-at-gmail.com@ffmpeg.org> 于2025年7月2日周三 20:09写道：
>
> RTP retransmission described in RFC4588 (RTX) is an effective packet
> loss recovery technique for real-time applications with relaxed delay bounds.
>
> This patch provides a minimal implementation for RTX and RTCP NACK (RFC3940)
> and its associated SDP signaling and negotiation.
>
> Co-authored-by: Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com>
> Signed-off-by: Jack Lau <jacklau1222@qq.com>
> ---
> libavformat/whip.c | 198 ++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 195 insertions(+), 3 deletions(-)
>
> diff --git a/libavformat/whip.c b/libavformat/whip.c
> index e287a3062f..fa6e3855d3 100644
> --- a/libavformat/whip.c
> +++ b/libavformat/whip.c
> @@ -114,6 +114,7 @@
> /* Referring to Chrome's definition of RTP payload types. */
> #define WHIP_RTP_PAYLOAD_TYPE_H264 106
> #define WHIP_RTP_PAYLOAD_TYPE_OPUS 111
> +#define WHIP_RTP_PAYLOAD_TYPE_RTX 105
>
> /**
> * The STUN message header, which is 20 bytes long, comprises the
> @@ -150,6 +151,11 @@
> #define WHIP_SDP_SESSION_ID "4489045141692799359"
> #define WHIP_SDP_CREATOR_IP "127.0.0.1"
>
> +/**
> + * Retransmission / NACK support
> +*/
> +#define HISTORY_SIZE_DEFAULT 512
> +
> /* Calculate the elapsed time from starttime to endtime in milliseconds. */
> #define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000)
>
> @@ -194,9 +200,19 @@ enum WHIPState {
> };
>
> typedef enum WHIPFlags {
> - WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate
> + WHIP_FLAG_IGNORE_IPV6 = (1 << 0), // Ignore ipv6 candidate
> + WHIP_FLAG_DISABLE_RTX = (1 << 1) // Enable NACK and RTX
> } WHIPFlags;
>
> +typedef struct RtpHistoryItem {
> + /* original RTP seq */
> + uint16_t seq; /* original RTP seq */
> + /* length in bytes */
> + int size; /* length in bytes */
> + /* malloc-ed copy */
move the comments.
> + uint8_t* buf; /* malloc-ed copy */
> +} RtpHistoryItem;
for example:
171 typedef struct mkv_cuepoint {
172 uint64_t pts;
173 int stream_idx;
174 int64_t cluster_pos; ///< offset of the
cluster containing the block relative to the segment
175 int64_t relative_pos; ///< relative offset from
the position of the cluster containing the block
176 int64_t duration; ///< duration of the
block according to time base
177 } mkv_cuepoint;
> +
> typedef struct WHIPContext {
> AVClass *av_class;
>
> @@ -285,6 +301,7 @@ typedef struct WHIPContext {
> /* The SRTP send context, to encrypt outgoing packets. */
> SRTPContext srtp_audio_send;
> SRTPContext srtp_video_send;
> + SRTPContext srtp_video_rtx_send;
> SRTPContext srtp_rtcp_send;
> /* The SRTP receive context, to decrypt incoming packets. */
> SRTPContext srtp_recv;
> @@ -309,6 +326,14 @@ typedef struct WHIPContext {
> /* The certificate and private key used for DTLS handshake. */
> char* cert_file;
> char* key_file;
> +
> + /* RTX and NACK */
> + uint8_t rtx_payload_type;
> + uint32_t video_rtx_ssrc;
> + uint16_t rtx_seq;
> + int history_size;
> + RtpHistoryItem *history; /* ring buffer */
> + int hist_head;
> } WHIPContext;
>
> /**
> @@ -606,6 +631,16 @@ static int generate_sdp_offer(AVFormatContext *s)
> whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS;
> whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264;
>
> + /* RTX and NACK init */
> + whip->rtx_payload_type = WHIP_RTP_PAYLOAD_TYPE_RTX;
> + whip->video_rtx_ssrc = av_lfg_get(&whip->rnd);
> + whip->rtx_seq = 0;
> + whip->hist_head = 0;
> + whip->history_size = FFMAX(64, whip->history_size);
> + whip->history = av_calloc(whip->history_size, sizeof(*whip->history));
> + if (!whip->history)
> + return AVERROR(ENOMEM);
> +
> av_bprintf(&bp, ""
> "v=0\r\n"
> "o=FFmpeg %s 2 IN IP4 %s\r\n"
> @@ -656,7 +691,7 @@ static int generate_sdp_offer(AVFormatContext *s)
> }
>
> av_bprintf(&bp, ""
> - "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n"
> + "m=video 9 UDP/TLS/RTP/SAVPF %u %u\r\n"
> "c=IN IP4 0.0.0.0\r\n"
> "a=ice-ufrag:%s\r\n"
> "a=ice-pwd:%s\r\n"
> @@ -669,9 +704,16 @@ static int generate_sdp_offer(AVFormatContext *s)
> "a=rtcp-rsize\r\n"
> "a=rtpmap:%u %s/90000\r\n"
> "a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n"
> + "a=rtcp-fb:%u nack\r\n"
> + "a=rtpmap:%u rtx/90000\r\n"
> + "a=fmtp:%u apt=%u\r\n"
> + "a=ssrc-group:FID %u %u\r\n"
> + "a=ssrc:%u cname:FFmpeg\r\n"
> + "a=ssrc:%u msid:FFmpeg video\r\n"
> "a=ssrc:%u cname:FFmpeg\r\n"
> "a=ssrc:%u msid:FFmpeg video\r\n",
> whip->video_payload_type,
> + whip->rtx_payload_type,
> whip->ice_ufrag_local,
> whip->ice_pwd_local,
> whip->dtls_fingerprint,
> @@ -681,8 +723,16 @@ static int generate_sdp_offer(AVFormatContext *s)
> profile,
> whip->constraint_set_flags,
> level,
> + whip->video_payload_type,
> + whip->rtx_payload_type,
> + whip->rtx_payload_type,
> + whip->video_payload_type,
> + whip->video_ssrc,
> + whip->video_rtx_ssrc,
> whip->video_ssrc,
> - whip->video_ssrc);
> + whip->video_ssrc,
> + whip->video_rtx_ssrc,
> + whip->video_rtx_ssrc);
> }
>
> if (!av_bprint_is_complete(&bp)) {
> @@ -1398,6 +1448,12 @@ static int setup_srtp(AVFormatContext *s)
> goto end;
> }
>
> + ret = ff_srtp_set_crypto(&whip->srtp_video_rtx_send, suite, buf);
> + if (ret < 0) {
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for video rtx send\n");
Since av_log(whip, has been set, the "WHIP:" prefix might be
unnecessary. Same comment to bellow.
> + goto end;
> + }
> +
> ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf);
> if (ret < 0) {
> av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n");
> @@ -1427,6 +1483,37 @@ end:
> return ret;
> }
>
> +
> +/**
> + * RTX history helpers
> + */
> + static void rtp_history_store(WHIPContext *whip, const uint8_t *buf, int size)
return value.
> +{
> + int pos = whip->hist_head % whip->history_size;
> + RtpHistoryItem *it = &whip->history[pos];
> + /* free older entry */
> + av_free(it->buf);
> + it->buf = av_malloc(size);
> + if (!it->buf)
> + return;
return AVERROR(ENOMEM)
> +
> + memcpy(it->buf, buf, size);
> + it->size = size;
> + it->seq = AV_RB16(buf + 2);
> +
> + whip->hist_head = ++pos;
> +}
> +
> +static const RtpHistoryItem *rtp_history_find(const WHIPContext *whip, uint16_t seq)
> +{
> + for (int i = 0; i < whip->history_size; i++) {
> + const RtpHistoryItem *it = &whip->history[i];
> + if (it->buf && it->seq == seq)
> + return it;
> + }
> + return NULL;
> +}
> +
> /**
> * Callback triggered by the RTP muxer when it creates and sends out an RTP packet.
> *
> @@ -1463,6 +1550,10 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
> return 0;
> }
>
> + /* Store only ORIGINAL video packets (non-RTX, non-RTCP) */
> + if (!is_rtcp && is_video)
> + rtp_history_store(whip, buf, buf_size);
ret = rtp_history_store(whip, buf, buf_size);
if (ret < 0)
process failed;
> +
> ret = ffurl_write(whip->udp, whip->buf, cipher_size);
> if (ret < 0) {
> av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret);
> @@ -1471,6 +1562,45 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
>
> return ret;
> }
> +/**
> + * See https://datatracker.ietf.org/doc/html/rfc4588
> + * Build and send a single RTX packet
> + */
> +static int send_rtx_packet(AVFormatContext *s, const uint8_t *orig_pkt_buf, int orig_size)
> +{
> + WHIPContext *whip = s->priv_data;
> + int new_size, cipher_size;
> + if (whip->flags & WHIP_FLAG_DISABLE_RTX)
> + return 0;
> +
> + /* allocate new buffer: header + 2 + payload */
> + if (orig_size + 2 > sizeof(whip->buf))
> + return 0;
> +
> + memcpy(whip->buf, orig_pkt_buf, orig_size);
> +
> + uint8_t *hdr = whip->buf;
> + uint16_t orig_seq = AV_RB16(hdr + 2);
> +
> + /* rewrite header */
> + hdr[1] = (hdr[1] & 0x80) | whip->rtx_payload_type; /* keep M bit */
> + AV_WB16(hdr + 2, whip->rtx_seq++);
> + AV_WB32(hdr + 8, whip->video_rtx_ssrc);
> +
> + /* shift payload 2 bytes */
> + memmove(hdr + 12 + 2, hdr + 12, orig_size - 12);
> + AV_WB16(hdr + 12, orig_seq);
> +
> + new_size = orig_size + 2;
> +
> + /* Encrypt by SRTP and send out. */
> + cipher_size = ff_srtp_encrypt(&whip->srtp_video_rtx_send, whip->buf, new_size, whip->buf, sizeof(whip->buf));
> + if (cipher_size <= 0 || cipher_size < new_size) {
> + av_log(whip, AV_LOG_WARNING, "WHIP: Failed to encrypt packet=%dB, cipher=%dB\n", new_size, cipher_size);
> + return 0;
> + }
> + return ffurl_write(whip->udp, whip->buf, cipher_size);
> +}
>
> /**
> * Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP
> @@ -1793,6 +1923,66 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt)
> goto end;
> }
> }
> + /**
> + * Handle RTCP NACK
> + * Refer to RFC 4585, Section 6.2.1
> + * The Generic NACK message is identified by PT=RTPFB and FMT=1.
> + * TODO: disable retransmisstion when "-tune zerolatency"
> + */
> + if (media_is_rtcp(whip->buf, ret)) {
> + int ptr = 0;
> + uint8_t pt = whip->buf[ptr + 1];
> + uint8_t fmt = (whip->buf[ptr] & 0x1f);
> + if (ptr + 4 <= ret && pt == 205 && fmt == 1) {
> + /**
> + * Refer to RFC 3550, Section 6.4.1.
> + * The length of this RTCP packet in 32-bit words minus one,
> + * including the header and any padding.
> + */
> + int rtcp_len = (AV_RB16(&whip->buf[ptr + 2]) + 1) * 4;
> + /* SRTCP index(4 bytes) + HMAC (SRTP_AES128_CM_SHA1_80 10bytes) */
> + int srtcp_len = rtcp_len + 4 + 10;
> + if (srtcp_len == ret && rtcp_len >= 12) {
> + int i = 0;
> + uint8_t *buf = av_malloc(srtcp_len);
if (!buf)
process AVEROR(ENOMEM)
> + memcpy(buf, whip->buf, srtcp_len);
> + int ret = ff_srtp_decrypt(&whip->srtp_recv, buf, &srtcp_len);
> + if (ret < 0)
if error here, should free buf, otherwise will memleak.
> + av_log(whip, AV_LOG_ERROR, "WHIP: SRTCP decrypt failed: %d\n", ret);
> + while (12 + i < rtcp_len && !ret) {
> + /**
> + * See https://datatracker.ietf.org/doc/html/rfc4585#section-6.1
> + * Handle multi NACKs in bundled packet.
> + */
> + uint16_t pid = AV_RB16(&buf[ptr + 12 + i]);
> + uint16_t blp = AV_RB16(&buf[ptr + 14 + i]);
> +
> + /* retransmit pid + any bit set in blp */
> + for (int bit = -1; bit < 16; bit++) {
> + uint16_t seq = (bit < 0) ? pid : pid + bit + 1;
> + if (bit >= 0 && !(blp & (1 << bit)))
> + continue;
> +
> + const RtpHistoryItem *it = rtp_history_find(whip, seq);
> + if (it) {
> + av_log(whip, AV_LOG_VERBOSE,
> + "WHIP: NACK, packet found: size: %d, seq=%d, rtx size=%d, lateset stored packet seq:%d\n",
> + it->size, seq, ret, whip->history[whip->hist_head-1].seq);
> + ret = send_rtx_packet(s, it->buf, it->size);
> + if (ret <= 0 && !(whip->flags & WHIP_FLAG_DISABLE_RTX))
> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send RTX packet\n");
> + } else {
> + av_log(whip, AV_LOG_VERBOSE,
> + "WHIP: NACK, packet not found, seq=%d, latest stored packet seq: %d, latest rtx seq: %d\n",
> + seq, whip->history[whip->hist_head-1].seq, whip->rtx_seq);
> + }
> + }
> + i = i + 4;
> + }
> + av_free(buf);
> + }
> + }
> + }
> } else if (ret != AVERROR(EAGAIN)) {
> av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n");
> goto end;
> @@ -1898,6 +2088,8 @@ static const AVOption options[] = {
> { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
> { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" },
> { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
> + { "disable_rtx", "Disable RFC 4588 RTX", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" },
> + { "rtx_history_size", "Packet history size", OFFSET(history_size), AV_OPT_TYPE_INT, { .i64 = HISTORY_SIZE_DEFAULT }, 64, 2048, ENC },
> { NULL },
> };
>
> --
> 2.49.0
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport
2025-07-02 13:46 ` Steven Liu
@ 2025-07-02 14:10 ` Jack Lau
0 siblings, 0 replies; 9+ messages in thread
From: Jack Lau @ 2025-07-02 14:10 UTC (permalink / raw)
To: FFmpeg development discussions and patches
Hi
> On Jul 2, 2025, at 21:46, Steven Liu <lingjiujianke-at-gmail.com@ffmpeg.org> wrote:
>
> Jack Lau <jacklau1222gm-at-gmail.com@ffmpeg.org <mailto:jacklau1222gm-at-gmail.com@ffmpeg.org>> 于2025年7月2日周三 20:09写道：
>>
>> RTP retransmission described in RFC4588 (RTX) is an effective packet
>> loss recovery technique for real-time applications with relaxed delay bounds.
>>
>> This patch provides a minimal implementation for RTX and RTCP NACK (RFC3940)
>> and its associated SDP signaling and negotiation.
>>
>> Co-authored-by: Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com>
>> Signed-off-by: Jack Lau <jacklau1222@qq.com>
>> ---
>> libavformat/whip.c | 198 ++++++++++++++++++++++++++++++++++++++++++++-
>> 1 file changed, 195 insertions(+), 3 deletions(-)
>>
>> diff --git a/libavformat/whip.c b/libavformat/whip.c
>> index e287a3062f..fa6e3855d3 100644
>> --- a/libavformat/whip.c
>> +++ b/libavformat/whip.c
>> @@ -114,6 +114,7 @@
>> /* Referring to Chrome's definition of RTP payload types. */
>> #define WHIP_RTP_PAYLOAD_TYPE_H264 106
>> #define WHIP_RTP_PAYLOAD_TYPE_OPUS 111
>> +#define WHIP_RTP_PAYLOAD_TYPE_RTX 105
>>
>> /**
>> * The STUN message header, which is 20 bytes long, comprises the
>> @@ -150,6 +151,11 @@
>> #define WHIP_SDP_SESSION_ID "4489045141692799359"
>> #define WHIP_SDP_CREATOR_IP "127.0.0.1"
>>
>> +/**
>> + * Retransmission / NACK support
>> +*/
>> +#define HISTORY_SIZE_DEFAULT 512
>> +
>> /* Calculate the elapsed time from starttime to endtime in milliseconds. */
>> #define ELAPSED(starttime, endtime) ((int)(endtime - starttime) / 1000)
>>
>> @@ -194,9 +200,19 @@ enum WHIPState {
>> };
>>
>> typedef enum WHIPFlags {
>> - WHIP_FLAG_IGNORE_IPV6 = (1 << 0) // Ignore ipv6 candidate
>> + WHIP_FLAG_IGNORE_IPV6 = (1 << 0), // Ignore ipv6 candidate
>> + WHIP_FLAG_DISABLE_RTX = (1 << 1) // Enable NACK and RTX
>> } WHIPFlags;
>>
>> +typedef struct RtpHistoryItem {
>> + /* original RTP seq */
>> + uint16_t seq; /* original RTP seq */
>> + /* length in bytes */
>> + int size; /* length in bytes */
>> + /* malloc-ed copy */
> move the comments.
>> + uint8_t* buf; /* malloc-ed copy */
>> +} RtpHistoryItem;
>
> for example:
>
>
> 171 typedef struct mkv_cuepoint {
> 172 uint64_t pts;
> 173 int stream_idx;
> 174 int64_t cluster_pos; ///< offset of the
> cluster containing the block relative to the segment
> 175 int64_t relative_pos; ///< relative offset from
> the position of the cluster containing the block
> 176 int64_t duration; ///< duration of the
> block according to time base
> 177 } mkv_cuepoint;
>
>
>
>> +
>> typedef struct WHIPContext {
>> AVClass *av_class;
>>
>> @@ -285,6 +301,7 @@ typedef struct WHIPContext {
>> /* The SRTP send context, to encrypt outgoing packets. */
>> SRTPContext srtp_audio_send;
>> SRTPContext srtp_video_send;
>> + SRTPContext srtp_video_rtx_send;
>> SRTPContext srtp_rtcp_send;
>> /* The SRTP receive context, to decrypt incoming packets. */
>> SRTPContext srtp_recv;
>> @@ -309,6 +326,14 @@ typedef struct WHIPContext {
>> /* The certificate and private key used for DTLS handshake. */
>> char* cert_file;
>> char* key_file;
>> +
>> + /* RTX and NACK */
>> + uint8_t rtx_payload_type;
>> + uint32_t video_rtx_ssrc;
>> + uint16_t rtx_seq;
>> + int history_size;
>> + RtpHistoryItem *history; /* ring buffer */
>> + int hist_head;
>> } WHIPContext;
>>
>> /**
>> @@ -606,6 +631,16 @@ static int generate_sdp_offer(AVFormatContext *s)
>> whip->audio_payload_type = WHIP_RTP_PAYLOAD_TYPE_OPUS;
>> whip->video_payload_type = WHIP_RTP_PAYLOAD_TYPE_H264;
>>
>> + /* RTX and NACK init */
>> + whip->rtx_payload_type = WHIP_RTP_PAYLOAD_TYPE_RTX;
>> + whip->video_rtx_ssrc = av_lfg_get(&whip->rnd);
>> + whip->rtx_seq = 0;
>> + whip->hist_head = 0;
>> + whip->history_size = FFMAX(64, whip->history_size);
>> + whip->history = av_calloc(whip->history_size, sizeof(*whip->history));
>> + if (!whip->history)
>> + return AVERROR(ENOMEM);
>> +
>> av_bprintf(&bp, ""
>> "v=0\r\n"
>> "o=FFmpeg %s 2 IN IP4 %s\r\n"
>> @@ -656,7 +691,7 @@ static int generate_sdp_offer(AVFormatContext *s)
>> }
>>
>> av_bprintf(&bp, ""
>> - "m=video 9 UDP/TLS/RTP/SAVPF %u\r\n"
>> + "m=video 9 UDP/TLS/RTP/SAVPF %u %u\r\n"
>> "c=IN IP4 0.0.0.0\r\n"
>> "a=ice-ufrag:%s\r\n"
>> "a=ice-pwd:%s\r\n"
>> @@ -669,9 +704,16 @@ static int generate_sdp_offer(AVFormatContext *s)
>> "a=rtcp-rsize\r\n"
>> "a=rtpmap:%u %s/90000\r\n"
>> "a=fmtp:%u level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=%02x%02x%02x\r\n"
>> + "a=rtcp-fb:%u nack\r\n"
>> + "a=rtpmap:%u rtx/90000\r\n"
>> + "a=fmtp:%u apt=%u\r\n"
>> + "a=ssrc-group:FID %u %u\r\n"
>> + "a=ssrc:%u cname:FFmpeg\r\n"
>> + "a=ssrc:%u msid:FFmpeg video\r\n"
>> "a=ssrc:%u cname:FFmpeg\r\n"
>> "a=ssrc:%u msid:FFmpeg video\r\n",
>> whip->video_payload_type,
>> + whip->rtx_payload_type,
>> whip->ice_ufrag_local,
>> whip->ice_pwd_local,
>> whip->dtls_fingerprint,
>> @@ -681,8 +723,16 @@ static int generate_sdp_offer(AVFormatContext *s)
>> profile,
>> whip->constraint_set_flags,
>> level,
>> + whip->video_payload_type,
>> + whip->rtx_payload_type,
>> + whip->rtx_payload_type,
>> + whip->video_payload_type,
>> + whip->video_ssrc,
>> + whip->video_rtx_ssrc,
>> whip->video_ssrc,
>> - whip->video_ssrc);
>> + whip->video_ssrc,
>> + whip->video_rtx_ssrc,
>> + whip->video_rtx_ssrc);
>> }
>>
>> if (!av_bprint_is_complete(&bp)) {
>> @@ -1398,6 +1448,12 @@ static int setup_srtp(AVFormatContext *s)
>> goto end;
>> }
>>
>> + ret = ff_srtp_set_crypto(&whip->srtp_video_rtx_send, suite, buf);
>> + if (ret < 0) {
>> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to set crypto for video rtx send\n");
> Since av_log(whip, has been set, the "WHIP:" prefix might be
> unnecessary. Same comment to bellow.
>
>> + goto end;
>> + }
>> +
>> ret = ff_srtp_set_crypto(&whip->srtp_rtcp_send, suite, buf);
>> if (ret < 0) {
>> av_log(whip, AV_LOG_ERROR, "Failed to set crypto for rtcp send\n");
>> @@ -1427,6 +1483,37 @@ end:
>> return ret;
>> }
>>
>> +
>> +/**
>> + * RTX history helpers
>> + */
>> + static void rtp_history_store(WHIPContext *whip, const uint8_t *buf, int size)
> return value.
>> +{
>> + int pos = whip->hist_head % whip->history_size;
>> + RtpHistoryItem *it = &whip->history[pos];
>> + /* free older entry */
>> + av_free(it->buf);
>> + it->buf = av_malloc(size);
>> + if (!it->buf)
>> + return;
> return AVERROR(ENOMEM)
>> +
>> + memcpy(it->buf, buf, size);
>> + it->size = size;
>> + it->seq = AV_RB16(buf + 2);
>> +
>> + whip->hist_head = ++pos;
>> +}
>> +
>> +static const RtpHistoryItem *rtp_history_find(const WHIPContext *whip, uint16_t seq)
>> +{
>> + for (int i = 0; i < whip->history_size; i++) {
>> + const RtpHistoryItem *it = &whip->history[i];
>> + if (it->buf && it->seq == seq)
>> + return it;
>> + }
>> + return NULL;
>> +}
>> +
>> /**
>> * Callback triggered by the RTP muxer when it creates and sends out an RTP packet.
>> *
>> @@ -1463,6 +1550,10 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
>> return 0;
>> }
>>
>> + /* Store only ORIGINAL video packets (non-RTX, non-RTCP) */
>> + if (!is_rtcp && is_video)
>> + rtp_history_store(whip, buf, buf_size);
> ret = rtp_history_store(whip, buf, buf_size);
> if (ret < 0)
> process failed;
>
>> +
>> ret = ffurl_write(whip->udp, whip->buf, cipher_size);
>> if (ret < 0) {
>> av_log(whip, AV_LOG_ERROR, "Failed to write packet=%dB, ret=%d\n", cipher_size, ret);
>> @@ -1471,6 +1562,45 @@ static int on_rtp_write_packet(void *opaque, const uint8_t *buf, int buf_size)
>>
>> return ret;
>> }
>> +/**
>> + * See https://datatracker.ietf.org/doc/html/rfc4588
>> + * Build and send a single RTX packet
>> + */
>> +static int send_rtx_packet(AVFormatContext *s, const uint8_t *orig_pkt_buf, int orig_size)
>> +{
>> + WHIPContext *whip = s->priv_data;
>> + int new_size, cipher_size;
>> + if (whip->flags & WHIP_FLAG_DISABLE_RTX)
>> + return 0;
>> +
>> + /* allocate new buffer: header + 2 + payload */
>> + if (orig_size + 2 > sizeof(whip->buf))
>> + return 0;
>> +
>> + memcpy(whip->buf, orig_pkt_buf, orig_size);
>> +
>> + uint8_t *hdr = whip->buf;
>> + uint16_t orig_seq = AV_RB16(hdr + 2);
>> +
>> + /* rewrite header */
>> + hdr[1] = (hdr[1] & 0x80) | whip->rtx_payload_type; /* keep M bit */
>> + AV_WB16(hdr + 2, whip->rtx_seq++);
>> + AV_WB32(hdr + 8, whip->video_rtx_ssrc);
>> +
>> + /* shift payload 2 bytes */
>> + memmove(hdr + 12 + 2, hdr + 12, orig_size - 12);
>> + AV_WB16(hdr + 12, orig_seq);
>> +
>> + new_size = orig_size + 2;
>> +
>> + /* Encrypt by SRTP and send out. */
>> + cipher_size = ff_srtp_encrypt(&whip->srtp_video_rtx_send, whip->buf, new_size, whip->buf, sizeof(whip->buf));
>> + if (cipher_size <= 0 || cipher_size < new_size) {
>> + av_log(whip, AV_LOG_WARNING, "WHIP: Failed to encrypt packet=%dB, cipher=%dB\n", new_size, cipher_size);
>> + return 0;
>> + }
>> + return ffurl_write(whip->udp, whip->buf, cipher_size);
>> +}
>>
>> /**
>> * Creates dedicated RTP muxers for each stream in the AVFormatContext to build RTP
>> @@ -1793,6 +1923,66 @@ static int whip_write_packet(AVFormatContext *s, AVPacket *pkt)
>> goto end;
>> }
>> }
>> + /**
>> + * Handle RTCP NACK
>> + * Refer to RFC 4585, Section 6.2.1
>> + * The Generic NACK message is identified by PT=RTPFB and FMT=1.
>> + * TODO: disable retransmisstion when "-tune zerolatency"
>> + */
>> + if (media_is_rtcp(whip->buf, ret)) {
>> + int ptr = 0;
>> + uint8_t pt = whip->buf[ptr + 1];
>> + uint8_t fmt = (whip->buf[ptr] & 0x1f);
>> + if (ptr + 4 <= ret && pt == 205 && fmt == 1) {
>> + /**
>> + * Refer to RFC 3550, Section 6.4.1.
>> + * The length of this RTCP packet in 32-bit words minus one,
>> + * including the header and any padding.
>> + */
>> + int rtcp_len = (AV_RB16(&whip->buf[ptr + 2]) + 1) * 4;
>> + /* SRTCP index(4 bytes) + HMAC (SRTP_AES128_CM_SHA1_80 10bytes) */
>> + int srtcp_len = rtcp_len + 4 + 10;
>> + if (srtcp_len == ret && rtcp_len >= 12) {
>> + int i = 0;
>> + uint8_t *buf = av_malloc(srtcp_len);
> if (!buf)
> process AVEROR(ENOMEM)
Thanks for your detailed reviews! I totally agree the top of ideas.
>> + memcpy(buf, whip->buf, srtcp_len);
>> + int ret = ff_srtp_decrypt(&whip->srtp_recv, buf, &srtcp_len);
>> + if (ret < 0)
> if error here, should free buf, otherwise will memleak.
But I think this is not an issue,
>
>> + av_log(whip, AV_LOG_ERROR, "WHIP: SRTCP decrypt failed: %d\n", ret);
>> + while (12 + i < rtcp_len && !ret) {
>> + /**
>> + * See https://datatracker.ietf.org/doc/html/rfc4585#section-6.1
>> + * Handle multi NACKs in bundled packet.
>> + */
>> + uint16_t pid = AV_RB16(&buf[ptr + 12 + i]);
>> + uint16_t blp = AV_RB16(&buf[ptr + 14 + i]);
>> +
>> + /* retransmit pid + any bit set in blp */
>> + for (int bit = -1; bit < 16; bit++) {
>> + uint16_t seq = (bit < 0) ? pid : pid + bit + 1;
>> + if (bit >= 0 && !(blp & (1 << bit)))
>> + continue;
>> +
>> + const RtpHistoryItem *it = rtp_history_find(whip, seq);
>> + if (it) {
>> + av_log(whip, AV_LOG_VERBOSE,
>> + "WHIP: NACK, packet found: size: %d, seq=%d, rtx size=%d, lateset stored packet seq:%d\n",
>> + it->size, seq, ret, whip->history[whip->hist_head-1].seq);
>> + ret = send_rtx_packet(s, it->buf, it->size);
>> + if (ret <= 0 && !(whip->flags & WHIP_FLAG_DISABLE_RTX))
>> + av_log(whip, AV_LOG_ERROR, "WHIP: Failed to send RTX packet\n");
>> + } else {
>> + av_log(whip, AV_LOG_VERBOSE,
>> + "WHIP: NACK, packet not found, seq=%d, latest stored packet seq: %d, latest rtx seq: %d\n",
>> + seq, whip->history[whip->hist_head-1].seq, whip->rtx_seq);
>> + }
>> + }
>> + i = i + 4;
>> + }
>> + av_free(buf);
Because we have this free function after the loop so there’s no memory leak.
>> + }
>> + }
>> + }
>> } else if (ret != AVERROR(EAGAIN)) {
>> av_log(whip, AV_LOG_ERROR, "Failed to read from UDP socket\n");
>> goto end;
>> @@ -1898,6 +2088,8 @@ static const AVOption options[] = {
>> { "key_file", "Optional private key file path for DTLS", OFFSET(key_file), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, ENC },
>> { "whip_flags", "Set flags affecting WHIP connection behavior", OFFSET(flags), AV_OPT_TYPE_FLAGS, { .i64 = 0 }, 0, UINT_MAX, ENC, .unit = "flags" },
>> { "ignore_ipv6", "Ignore any IPv6 ICE candidate", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_IGNORE_IPV6 }, 0, UINT_MAX, ENC, .unit = "flags" },
>> + { "disable_rtx", "Disable RFC 4588 RTX", 0, AV_OPT_TYPE_CONST, { .i64 = WHIP_FLAG_DISABLE_RTX }, 0, UINT_MAX, ENC, .unit = "flags" },
>> + { "rtx_history_size", "Packet history size", OFFSET(history_size), AV_OPT_TYPE_INT, { .i64 = HISTORY_SIZE_DEFAULT }, 64, 2048, ENC },
>> { NULL },
>> };
>>
>> --
>> 2.49.0
>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request@ffmpeg.org <mailto:ffmpeg-devel-request@ffmpeg.org> with subject "unsubscribe”.
I’ll submit new version of this patch later.
Thanks
Jack
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org <mailto:ffmpeg-devel-request@ffmpeg.org> with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2025-07-02 14:10 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-07-02 11:59 [FFmpeg-devel] [PATCH v2 0/6] avformat/whip: Add NACK and RTX support (depends on ignore_ipv6 patchset) Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 1/6] avformat/whip: add whip_flags ignore_ipv6 to skip IPv6 ICE candidates Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 2/6] avformat/whip: fix typos Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 3/6] avformat/whip: fix H264 profile_iop bit map for SDP Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 4/6] WHIP: X509 cert serial number should be positive Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 5/6] avformat/whip: implement NACK and RTX suppport Jack Lau
2025-07-02 13:46 ` Steven Liu
2025-07-02 14:10 ` Jack Lau
2025-07-02 11:59 ` [FFmpeg-devel] [PATCH v2 6/6] avformat/whip: reindent whip options Jack Lau
Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
This inbox may be cloned and mirrored by anyone:
git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
ffmpegdev@gitmailbox.com
public-inbox-index ffmpegdev
Example config snippet for mirrors.
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git