[FFmpeg-devel] [PR] avformat/tls_openssl: two small cleanup (PR #21697)

[Jack Lau via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

PR #21697 opened by Jack Lau (JackLau)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21697
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21697.patch


>From db6e39b3a380b2635859bfca7864548c5626fb72 Mon Sep 17 00:00:00 2001
From: Jack Lau <jacklau1222gm@gmail.com>
Date: Mon, 9 Feb 2026 15:12:23 +0800
Subject: [PATCH 1/2] avformat/tls_openssl: use EINVAL when X509_digest failed

This function just calculate and copy the fingerprint
to the provided buf, will not allocate memory.

It fails when the input (such as cert) is invalid.

Signed-off-by: Jack Lau <jacklau1222gm@gmail.com>
---
 libavformat/tls_openssl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
index c7d6ff6cf4..821f9ffd04 100644
--- a/libavformat/tls_openssl.c
+++ b/libavformat/tls_openssl.c
@@ -100,7 +100,7 @@ static int x509_fingerprint(X509 *cert, char **fingerprint)
     if (X509_digest(cert, EVP_sha256(), md, &n) != 1) {
         av_log(NULL, AV_LOG_ERROR, "TLS: Failed to generate fingerprint, %s\n",
                ERR_error_string(ERR_get_error(), NULL));
-        return AVERROR(ENOMEM);
+        return AVERROR(EINVAL);
     }
 
     av_bprint_init(&buf, n*3, n*3);
-- 
2.52.0


>From 8dd94aa113c01a5c88c0205a6756638311b2ac9d Mon Sep 17 00:00:00 2001
From: Jack Lau <jacklau1222gm@gmail.com>
Date: Mon, 9 Feb 2026 17:27:03 +0800
Subject: [PATCH 2/2] avformat/tls_openssl: update the outdated comments

Loading CA certificate is supported.

Remove unrelated comments.

The underlying socket can be tcp or udp.

Signed-off-by: Jack Lau <jacklau1222gm@gmail.com>
---
 libavformat/tls_openssl.c | 15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)

diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
index 821f9ffd04..0ae0980cc3 100644
--- a/libavformat/tls_openssl.c
+++ b/libavformat/tls_openssl.c
@@ -267,7 +267,6 @@ static int openssl_gen_certificate(EVP_PKEY *pkey, X509 **cert, char **fingerpri
         goto enomem_end;
     }
 
-    // TODO: Support non-self-signed certificate, for example, load from a file.
     subject = X509_NAME_new();
     if (!subject) {
         goto enomem_end;
@@ -812,17 +811,7 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
     else
         SSL_set_connect_state(c->ssl);
 
-    /**
-     * During initialization, we only need to call SSL_do_handshake once because SSL_read consumes
-     * the handshake message if the handshake is incomplete.
-     * To simplify maintenance, we initiate the handshake for both the DTLS server and client after
-     * sending out the ICE response in the start_active_handshake function. It's worth noting that
-     * although the DTLS server may receive the ClientHello immediately after sending out the ICE
-     * response, this shouldn't be an issue as the handshake function is called before any DTLS
-     * packets are received.
-     *
-     * The SSL_do_handshake can't be called if DTLS hasn't prepare for udp.
-     */
+    /* The SSL_do_handshake can't be called if DTLS hasn't prepare for udp. */
     if (!c->tls_shared.external_sock) {
         ret = dtls_handshake(h);
         // Fatal SSL error, for example, no available suite when peer is DTLS 1.0 while we are DTLS 1.2.
@@ -933,7 +922,7 @@ static int tls_write(URLContext *h, const uint8_t *buf, int size)
     URLContext *uc = s->is_dtls ? s->udp : s->tcp;
     int ret;
 
-    // Set or clear the AVIO_FLAG_NONBLOCK on c->tls_shared.tcp
+    // Set or clear the AVIO_FLAG_NONBLOCK on the underlying socket
     uc->flags &= ~AVIO_FLAG_NONBLOCK;
     uc->flags |= h->flags & AVIO_FLAG_NONBLOCK;
 
-- 
2.52.0

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org