[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[Zhao Zhili via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

> On Dec 24, 2025, at 11:30, nyh163925 <code@ffmpeg.org> wrote:
> 
> 
> Hello FFmpeg community, I am from Xiaomi Vela, and we have built the Vela multimedia system using FFmpeg. Thank you for the invaluable work you do—FFmpeg is the cornerstone of many of our modules.
> 
> ISSUE: -fshort-enum ABI problem
> 
> In the process of application, we found that some embedded platforms (such as ARM Cortex-m) have enabled -fshort-enum optimization by default in their compilers, which leads to ABI in the cross passing of some enum * and int * in FFmpeg. For example, when passing enum AVSampleFormat * as int * in ff_det_common_fformats_for_list2(), it will cause check_fformat() error.
> 
> Context
> 
> C language Standard undefined behavior:
> The underlying type and size of an enum are implementation-defined. Many toolchains can use the smallest integer type that fits the values (e.g., with -fshort-enums), so sizeof(enum) can be less than sizeof(int), and alignment may differ.
> Casting enum* to int* and accessing through the int pointer runs into strict-aliasing and effective-type rules; it is not portable and may be undefined.
> 
> FFmpeg defaults to the equality of sizeof(enum) and sizeof(int):
> We have seen code patterns that implicitly assume “enum is a 4-byte int” and interchange enum* with int*. Typical examples include:
> 
> BufferSinkContext {
> enum AVSampleFormat *sample_formats
> ...
> }
> 
> int ff_set_common_formats_from_list2(const AVFilterContext *ctx,
>                                      AVFilterFormatsConfig **cfg_in,
>                                      AVFilterFormatsConfig **cfg_out,
>                                      const int *fmts)
> 
> static int asink_query_formats(const AVFilterContext *ctx,
>                                AVFilterFormatsConfig **cfg_in,
>                                AVFilterFormatsConfig **cfg_out)
>  {
>     const BufferSinkContext *buf = ctx->priv;
>     if (buf->nb_sample_formats) {
>        ret = ff_set_common_formats_from_list2(ctx, cfg_in, cfg_out, buf->sample_formats);
>        if (ret < 0)
>            return ret;
>     }
>  ...
>  }
> We are facing a conflict dilemma:
> We found that many ARM-based embedded compilers and third-party libraries default to -fshort-enums, which conflicts with FFmpeg's ABI (compiled with -fno-short-enums). However, if we enable the '-fshort-enums' in FFMPEG, the above ABI causes format negotiation and matching failures in our modules.
> Proposed paths forward
> 
> Change “assume 4 bytes” to “force 4 bytes”.
> Adopt a fixed 32-bit representation for enums at API/ABI boundaries
> vulkan for example, uses *_MAX_ENUM = 0x7FFFFFFF as last value for each enum, to avoid any and all problems with size
> 
> Stop assuming sizeof(enum) == 4.
> Gradually fix code to avoid enum* ↔ int* aliasing; use value-level conversions or memcpy where needed.
> Pros: preserves the size benefits where compilers choose smaller enums.
> Cons: requires auditing and incremental code changes.
> 
> We’re seeking the community’s guidance on which direction is preferable for FFmpeg: Should we standardize on a fixed 32-bit enum representation for ABI stability? Or should we remove the “enum == int” assumption and accept smaller enums, with code cleanup to avoid aliasing and stride errors?
> We’re ready to contribute patches in the direction the project prefers and to align with FFmpeg’s coding guidelines. Thanks again for your time and feedback.
> 

I think the issue is common and better be discussed on the mailing list. It's a big decision.

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org