On 12/5/2025 2:33 AM, oliverchang via ffmpeg-devel wrote:
> PR #21107 opened by oliverchang
> URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21107
> Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/21107.patch
> 
> The ASAN heap-buffer-overflow in `scalable_channel_layout_config` at
> `libavformat/iamf_parse.c:435` was caused by an unchecked assumption
> that the channel layout of a scalable audio layer is a superset of the
> previous layer's channel layout.
> 
> `scalable_channel_layout_config` constructs a channel layout map by
> copying channels from the previous layer and adding new ones. The memory
> allocation is based on the target `loudspeaker_layout`. However, if the
> target layout doesn't encompass all previous channels (e.g., Mono to
> Stereo), copying previous channels followed by adding current ones could
> exceed the allocated size, causing a heap buffer overflow.
> 
> This commit adds a check to ensure the previous layer's channel layout
> is a subset of the current layer's layout by comparing their masks. If
> the condition isn't met, `AVERROR_INVALIDDATA` is returned.
> 
> Fixes: https://issues.oss-fuzz.com/issues/464965414
> 
> 
>>From 6b353995bea2f39dbb751ba868e156b4dd94e8a8 Mon Sep 17 00:00:00 2001
> From: Oliver Chang <ochang@google.com>
> Date: Fri, 5 Dec 2025 05:30:29 +0000
> Subject: [PATCH] avformat/iamf_parse: Fix heap-buffer-overflow
> 
> The ASAN heap-buffer-overflow in `scalable_channel_layout_config` at
> `libavformat/iamf_parse.c:435` was caused by an unchecked assumption
> that the channel layout of a scalable audio layer is a superset of the
> previous layer's channel layout.
> 
> `scalable_channel_layout_config` constructs a channel layout map by
> copying channels from the previous layer and adding new ones. The memory
> allocation is based on the target `loudspeaker_layout`. However, if the
> target layout doesn't encompass all previous channels (e.g., Mono to
> Stereo), copying previous channels followed by adding current ones could
> exceed the allocated size, causing a heap buffer overflow.
> 
> This commit adds a check to ensure the previous layer's channel layout
> is a subset of the current layer's layout by comparing their masks. If
> the condition isn't met, `AVERROR_INVALIDDATA` is returned.
> 
> Fixes: https://issues.oss-fuzz.com/issues/464965414
> ---
>   libavformat/iamf_parse.c | 8 ++++++--
>   1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/libavformat/iamf_parse.c b/libavformat/iamf_parse.c
> index 597d800be0..3bb55eb0e9 100644
> --- a/libavformat/iamf_parse.c
> +++ b/libavformat/iamf_parse.c
> @@ -399,8 +399,12 @@ static int scalable_channel_layout_config(void *s, AVIOContext *pb,
>               av_channel_layout_copy(&ch_layout, &ff_iamf_expanded_scalable_ch_layouts[expanded_loudspeaker_layout]);
>           } else if (loudspeaker_layout < 10) {
>               av_channel_layout_copy(&ch_layout, &ff_iamf_scalable_ch_layouts[loudspeaker_layout]);
> -            if (i)
> -                ch_layout.u.mask &= ~av_channel_layout_subset(&audio_element->element->layers[i-1]->ch_layout, UINT64_MAX);
> +            if (i) {
> +                uint64_t mask = av_channel_layout_subset(&audio_element->element->layers[i-1]->ch_layout, UINT64_MAX);
> +                if ((ch_layout.u.mask & mask) != mask)
> +                    return AVERROR_INVALIDDATA;
> +                ch_layout.u.mask &= ~mask;
> +            }
>           } else
>               ch_layout = (AVChannelLayout){ .order = AV_CHANNEL_ORDER_UNSPEC,
>                                                             .nb_channels = substream_count +

This breaks demuxing 
https://github.com/AOMediaCodec/libiamf/blob/main/tests/test_000230.iamf 
because the C channel is not in the second scalable layout (FL+FR).

The logic in the function is evidently flawed, so I'll have give it 
another look.