[FFmpeg-devel] [PATCH] libavcodec/dcadec: adjust the `ch_layout` when downmix is active

[FFmpeg-devel] [PATCH] libavcodec/dcadec: adjust the `ch_layout` when downmix is active

[Geoffrey McRae via ffmpeg-devel]

Applications making use of this codec with the `downmix` option are
segfaulting unless the `ch_layout` is overridden after `avcodec_open2`
as can be seen in projects like MythTV[1]

This patch fixes this by overriding the ch_layout as done in other
decoders such as AC3.

1: https://github.com/MythTV/mythtv/blob/af6f362a140cd59b9ed784a8c639fd456b5f6967/mythtv/libs/libmythtv/decoders/avformatdecoder.cpp#L4607

Signed-off-by: Geoffrey McRae <geoff@hostfission.com>
---
 libavcodec/dcadec.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/libavcodec/dcadec.c b/libavcodec/dcadec.c
index 1fee49cf4d..cc23731b90 100644
--- a/libavcodec/dcadec.c
+++ b/libavcodec/dcadec.c
@@ -366,12 +366,19 @@ FF_ENABLE_DEPRECATION_WARNINGS
 
     if (s->downmix_layout.nb_channels) {
         if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO) ||
-            !av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO_DOWNMIX))
+            !av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO_DOWNMIX)) {
             s->request_channel_layout = DCA_SPEAKER_LAYOUT_STEREO;
-        else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT0))
+            av_channel_layout_uninit(&avctx->ch_layout);
+            avctx->ch_layout = (AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO;
+        } else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT0)) {
             s->request_channel_layout = DCA_SPEAKER_LAYOUT_5POINT0;
-        else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT1))
+            av_channel_layout_uninit(&avctx->ch_layout);
+            avctx->ch_layout = (AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT0;
+        } else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT1)) {
             s->request_channel_layout = DCA_SPEAKER_LAYOUT_5POINT1;
+            av_channel_layout_uninit(&avctx->ch_layout);
+            avctx->ch_layout = (AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT1;
+        }
         else
             av_log(avctx, AV_LOG_WARNING, "Invalid downmix layout\n");
     }
-- 
2.39.2

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [PATCH] libavcodec/dcadec: adjust the `ch_layout` when downmix is active

[James Almer]

On 11/26/2023 1:33 AM, Geoffrey McRae via ffmpeg-devel wrote:
> Applications making use of this codec with the `downmix` option are
> segfaulting unless the `ch_layout` is overridden after `avcodec_open2`
> as can be seen in projects like MythTV[1]
> 
> This patch fixes this by overriding the ch_layout as done in other
> decoders such as AC3.
> 
> 1: https://github.com/MythTV/mythtv/blob/af6f362a140cd59b9ed784a8c639fd456b5f6967/mythtv/libs/libmythtv/decoders/avformatdecoder.cpp#L4607
> 
> Signed-off-by: Geoffrey McRae <geoff@hostfission.com>
> ---
>   libavcodec/dcadec.c | 13 ++++++++++---
>   1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/libavcodec/dcadec.c b/libavcodec/dcadec.c
> index 1fee49cf4d..cc23731b90 100644
> --- a/libavcodec/dcadec.c
> +++ b/libavcodec/dcadec.c
> @@ -366,12 +366,19 @@ FF_ENABLE_DEPRECATION_WARNINGS
>   
>       if (s->downmix_layout.nb_channels) {
>           if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO) ||
> -            !av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO_DOWNMIX))
> +            !av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO_DOWNMIX)) {
>               s->request_channel_layout = DCA_SPEAKER_LAYOUT_STEREO;
> -        else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT0))
> +            av_channel_layout_uninit(&avctx->ch_layout);
> +            avctx->ch_layout = (AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO;
> +        } else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT0)) {
>               s->request_channel_layout = DCA_SPEAKER_LAYOUT_5POINT0;
> -        else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT1))
> +            av_channel_layout_uninit(&avctx->ch_layout);
> +            avctx->ch_layout = (AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT0;
> +        } else if (!av_channel_layout_compare(&s->downmix_layout, &(AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT1)) {
>               s->request_channel_layout = DCA_SPEAKER_LAYOUT_5POINT1;
> +            av_channel_layout_uninit(&avctx->ch_layout);
> +            avctx->ch_layout = (AVChannelLayout)AV_CHANNEL_LAYOUT_5POINT1;
> +        }
>           else
>               av_log(avctx, AV_LOG_WARNING, "Invalid downmix layout\n");
>       }

Applied, thanks.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".