[FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Jack Lau via ffmpeg-devel]

fix the missing data structure pkey in the tls_context

Signed-off-by: Jack Lau <jacklau1222@qq.com>
---
 libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)

diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
index b589d5d90a..bddeee9af8 100644
--- a/libavformat/tls_openssl.c
+++ b/libavformat/tls_openssl.c
@@ -467,6 +467,7 @@ typedef struct TLSContext {
     TLSShared tls_shared;
     SSL_CTX *ctx;
     SSL *ssl;
+    EVP_PKEY *pkey;
 #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
     BIO_METHOD* url_bio_method;
 #endif
@@ -811,7 +812,7 @@ static av_cold int openssl_init_ca_key_cert(URLContext *h)
     int ret;
     TLSContext *p = h->priv_data;
     TLSShared *c = &p->tls_shared;
-    EVP_PKEY *pkey = NULL;
+    EVP_PKEY *pkey = p->pkey;
     X509 *cert = NULL;
     /* setup ca, private key, certificate */
     if (c->ca_file) {
@@ -876,6 +877,9 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
     int ret = 0;
     c->is_dtls = 1;
     const char* ciphers = "ALL";
+#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
+    EC_KEY *ec_key;
+#endif
     /**
      * The profile for OpenSSL's SRTP is SRTP_AES128_CM_SHA1_80, see ssl/d1_srtp.c.
      * The profile for FFmpeg's SRTP is SRTP_AES128_CM_HMAC_SHA1_80, see libavformat/srtp.c.
@@ -908,15 +912,6 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
     }
 #endif
 
-#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
-#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
-    if (ctx->dtls_eckey)
-        SSL_CTX_set_tmp_ecdh(p->ctx, p->dtls_eckey);
-#else
-    SSL_CTX_set_ecdh_auto(p->ctx, 1);
-#endif
-#endif
-
     /**
      * We activate "ALL" cipher suites to align with the peer's capabilities,
      * ensuring maximum compatibility.
@@ -930,6 +925,17 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
     ret = openssl_init_ca_key_cert(h);
     if (ret < 0) goto fail;
 
+#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
+#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
+    if (p->pkey)
+        ec_key = EVP_PKEY_get1_EC_KEY(p->pkey);
+    if (ec_key)
+        SSL_CTX_set_tmp_ecdh(p->ctx, ec_key);
+#else
+    SSL_CTX_set_ecdh_auto(p->ctx, 1);
+#endif
+#endif
+
     /* Server will send Certificate Request. */
     SSL_CTX_set_verify(p->ctx, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, openssl_dtls_verify_callback);
     /* The depth count is "level 0:peer certificate", "level 1: CA certificate",
@@ -1015,9 +1021,7 @@ static av_cold int dtls_close(URLContext *h)
     av_freep(&ctx->tls_shared.fingerprint);
     av_freep(&ctx->tls_shared.cert_buf);
     av_freep(&ctx->tls_shared.key_buf);
-#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
-    EC_KEY_free(ctx->dtls_eckey);
-#endif
+    EVP_PKEY_free(ctx->pkey);
     return 0;
 }
 
-- 
2.49.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Martin Storsjö]

On Thu, 5 Jun 2025, Jack Lau via ffmpeg-devel wrote:

> fix the missing data structure pkey in the tls_context
>
> Signed-off-by: Jack Lau <jacklau1222@qq.com>
> ---
> libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
> 1 file changed, 17 insertions(+), 13 deletions(-)

Thanks, this does fix the build break. However, I don't quite understand 
the fix...

> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
> index b589d5d90a..bddeee9af8 100644
> --- a/libavformat/tls_openssl.c
> +++ b/libavformat/tls_openssl.c
> @@ -467,6 +467,7 @@ typedef struct TLSContext {
>     TLSShared tls_shared;
>     SSL_CTX *ctx;
>     SSL *ssl;
> +    EVP_PKEY *pkey;
> #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
>     BIO_METHOD* url_bio_method;
> #endif

As far as I can see, nothing ever sets this new field, it is only used in 
a couple of places?

// Martin

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Jack Lau]

> On Jun 5, 2025, at 15:02, Martin Storsjö <martin@martin.st> wrote:
> 
> On Thu, 5 Jun 2025, Jack Lau via ffmpeg-devel wrote:
> 
>> fix the missing data structure pkey in the tls_context
>> 
>> Signed-off-by: Jack Lau <jacklau1222@qq.com>
>> ---
>> libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
>> 1 file changed, 17 insertions(+), 13 deletions(-)
> 
> Thanks, this does fix the build break. However, I don't quite understand the fix...
> 
>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>> index b589d5d90a..bddeee9af8 100644
>> --- a/libavformat/tls_openssl.c
>> +++ b/libavformat/tls_openssl.c
>> @@ -467,6 +467,7 @@ typedef struct TLSContext {
>>    TLSShared tls_shared;
>>    SSL_CTX *ctx;
>>    SSL *ssl;
>> +    EVP_PKEY *pkey;
>> #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
>>    BIO_METHOD* url_bio_method;
>> #endif
> 
> As far as I can see, nothing ever sets this new field, it is only used in a couple of places?
Thanks for the review.

The previous build error occurred because I forgot to properly set the EC_KEY when using OpenSSL versions earlier than 3.0.

In the current WHIP implementation, I initialize the key and certificate (either by reading from file or generating them) before the DTLS handshake, since the SDP requires fingerprints. The WHIP layer then passes the key and certificate content as strings into the DTLS context.

This fix ensures that the EVP_PKEY is loaded into the tls_context when DTLS starts. For OpenSSL versions below 1.0.2, we need to call SSL_CTX_set_tmp_ecdh, which requires an EC_KEY. So, i extract the EC_KEY from the EVP_PKEY.

I hope that explanation was clear—please feel free to reach out if you have any further questions.

> @@ -876,6 +877,9 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
>     int ret = 0;
>     c->is_dtls = 1;
>     const char* ciphers = "ALL";
> +#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> +    EC_KEY *ec_key;
> +#endif
>     /**
>      * The profile for OpenSSL's SRTP is SRTP_AES128_CM_SHA1_80, see ssl/d1_srtp.c.
>      * The profile for FFmpeg's SRTP is SRTP_AES128_CM_HMAC_SHA1_80, see libavformat/srtp.c.
> @@ -908,15 +912,6 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
>     }
> #endif
> 
> -#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
> -#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> -    if (ctx->dtls_eckey)
> -        SSL_CTX_set_tmp_ecdh(p->ctx, p->dtls_eckey);
> -#else
> -    SSL_CTX_set_ecdh_auto(p->ctx, 1);
> -#endif
> -#endif
> -
>     /**
>      * We activate "ALL" cipher suites to align with the peer's capabilities,
>      * ensuring maximum compatibility.
> @@ -930,6 +925,17 @@ static int dtls_start(URLContext *h, const char *url, int flags, AVDictionary **
>     ret = openssl_init_ca_key_cert(h);
>     if (ret < 0) goto fail;
> 
> +#if OPENSSL_VERSION_NUMBER < 0x10100000L // v1.1.x
> +#if OPENSSL_VERSION_NUMBER < 0x10002000L // v1.0.2
> +    if (p->pkey)
> +        ec_key = EVP_PKEY_get1_EC_KEY(p->pkey);
> +    if (ec_key)
> +        SSL_CTX_set_tmp_ecdh(p->ctx, ec_key);
> +#else
> +    SSL_CTX_set_ecdh_auto(p->ctx, 1);
> +#endif
> +#endif
> +
>     /* Server will send Certificate Request. */
>     SSL_CTX_set_verify(p->ctx, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, openssl_dtls_verify_callback);
>     /* The depth count is "level 0:peer certificate", "level 1: CA certificate",
> @@ -1015,9 +1021,7 @@ static av_cold int dtls_close(URLContext *h)
>     av_freep(&ctx->tls_shared.fingerprint);
>     av_freep(&ctx->tls_shared.cert_buf);
>     av_freep(&ctx->tls_shared.key_buf);
> -#if OPENSSL_VERSION_NUMBER < 0x30000000L /* OpenSSL 3.0 */
> -    EC_KEY_free(ctx->dtls_eckey);
> -#endif
> +    EVP_PKEY_free(ctx->pkey);
>     return 0;
> }
> 
> -- 
> 2.49.0

> 
> // Martin
Thanks
Jack
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Martin Storsjö]

On Thu, 5 Jun 2025, Jack Lau wrote:

>> On Jun 5, 2025, at 15:02, Martin Storsjö <martin@martin.st> wrote:
>> 
>> On Thu, 5 Jun 2025, Jack Lau via ffmpeg-devel wrote:
>> 
>>> fix the missing data structure pkey in the tls_context
>>> 
>>> Signed-off-by: Jack Lau <jacklau1222@qq.com>
>>> ---
>>> libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
>>> 1 file changed, 17 insertions(+), 13 deletions(-)
>> 
>> Thanks, this does fix the build break. However, I don't quite understand the fix...
>> 
>>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>>> index b589d5d90a..bddeee9af8 100644
>>> --- a/libavformat/tls_openssl.c
>>> +++ b/libavformat/tls_openssl.c
>>> @@ -467,6 +467,7 @@ typedef struct TLSContext {
>>>    TLSShared tls_shared;
>>>    SSL_CTX *ctx;
>>>    SSL *ssl;
>>> +    EVP_PKEY *pkey;
>>> #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
>>>    BIO_METHOD* url_bio_method;
>>> #endif
>> 
>> As far as I can see, nothing ever sets this new field, it is only used in a couple of places?
> Thanks for the review.
>
> The previous build error occurred because I forgot to properly set the 
> EC_KEY when using OpenSSL versions earlier than 3.0.
>
> In the current WHIP implementation, I initialize the key and certificate 
> (either by reading from file or generating them) before the DTLS 
> handshake, since the SDP requires fingerprints. The WHIP layer then 
> passes the key and certificate content as strings into the DTLS context.
>
> This fix ensures that the EVP_PKEY is loaded into the tls_context when 
> DTLS starts. For OpenSSL versions below 1.0.2, we need to call 
> SSL_CTX_set_tmp_ecdh, which requires an EC_KEY. So, i extract the EC_KEY 
> from the EVP_PKEY.
>
> I hope that explanation was clear—please feel free to reach out if you 
> have any further questions.

No that didn't answer my question.

As far as I can see, nothing sets the context variable p->pkey. It is used 
in openssl_init_ca_key_cert and later in dtls_start. But nothing ever sets 
p->key, so it will be NULL everywhere.

Did you test this code with openssl 1.0.2 (which those codepaths are for)?

It looks to me like this maybe should have an assignment in 
openssl_init_ca_key_cert, setting "p->key = pkey;" maybe?

// Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] avformat/tls_openssl: fix build error when openssl version < 3

[Jack Lau]

> On Jun 5, 2025, at 19:20, Martin Storsjö <martin@martin.st> wrote:
> 
> On Thu, 5 Jun 2025, Jack Lau wrote:
> 
>>> On Jun 5, 2025, at 15:02, Martin Storsjö <martin@martin.st> wrote:
>>> On Thu, 5 Jun 2025, Jack Lau via ffmpeg-devel wrote:
>>>> fix the missing data structure pkey in the tls_context
>>>> Signed-off-by: Jack Lau <jacklau1222@qq.com>
>>>> ---
>>>> libavformat/tls_openssl.c | 30 +++++++++++++++++-------------
>>>> 1 file changed, 17 insertions(+), 13 deletions(-)
>>> Thanks, this does fix the build break. However, I don't quite understand the fix...
>>>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>>>> index b589d5d90a..bddeee9af8 100644
>>>> --- a/libavformat/tls_openssl.c
>>>> +++ b/libavformat/tls_openssl.c
>>>> @@ -467,6 +467,7 @@ typedef struct TLSContext {
>>>>   TLSShared tls_shared;
>>>>   SSL_CTX *ctx;
>>>>   SSL *ssl;
>>>> +    EVP_PKEY *pkey;
>>>> #if OPENSSL_VERSION_NUMBER >= 0x1010000fL
>>>>   BIO_METHOD* url_bio_method;
>>>> #endif
>>> As far as I can see, nothing ever sets this new field, it is only used in a couple of places?
>> Thanks for the review.
>> 
>> The previous build error occurred because I forgot to properly set the EC_KEY when using OpenSSL versions earlier than 3.0.
>> 
>> In the current WHIP implementation, I initialize the key and certificate (either by reading from file or generating them) before the DTLS handshake, since the SDP requires fingerprints. The WHIP layer then passes the key and certificate content as strings into the DTLS context.
>> 
>> This fix ensures that the EVP_PKEY is loaded into the tls_context when DTLS starts. For OpenSSL versions below 1.0.2, we need to call SSL_CTX_set_tmp_ecdh, which requires an EC_KEY. So, i extract the EC_KEY from the EVP_PKEY.
>> 
>> I hope that explanation was clear—please feel free to reach out if you have any further questions.
> 
> No that didn't answer my question.
> 
> As far as I can see, nothing sets the context variable p->pkey. It is used in openssl_init_ca_key_cert and later in dtls_start. But nothing ever sets p->key, so it will be NULL everywhere.
> 
> Did you test this code with openssl 1.0.2 (which those codepaths are for)?
> 
> It looks to me like this maybe should have an assignment in openssl_init_ca_key_cert, setting "p->key = pkey;" maybe?
Thanks for your reminder! 
I’ve sent the patch v2 that fix this issue.

And I tested the major openssl versions (1.0.1, 1.0.2, 1.1.0, 3.0, latest) and it works well.

> 
> // Martin
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org <mailto:ffmpeg-devel-request@ffmpeg.org> with subject "unsubscribe”.
Thanks
Jack



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".