[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[Michael Niedermayer via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

[-- Attachment #1.1: Type: text/plain, Size: 4270 bytes --]

Hi Zhao

On Tue, Dec 30, 2025 at 12:46:24AM +0800, Zhao Zhili via ffmpeg-devel wrote:
> 
> 
> > On Dec 24, 2025, at 11:30, nyh163925 <code@ffmpeg.org> wrote:
> > 
> > 
> > Hello FFmpeg community, I am from Xiaomi Vela, and we have built the Vela multimedia system using FFmpeg. Thank you for the invaluable work you do—FFmpeg is the cornerstone of many of our modules.
> > 
> > ISSUE: -fshort-enum ABI problem
> > 
> > In the process of application, we found that some embedded platforms (such as ARM Cortex-m) have enabled -fshort-enum optimization by default in their compilers, which leads to ABI in the cross passing of some enum * and int * in FFmpeg. For example, when passing enum AVSampleFormat * as int * in ff_det_common_fformats_for_list2(), it will cause check_fformat() error.
> > 
> > Context
> > 
> > C language Standard undefined behavior:
> > The underlying type and size of an enum are implementation-defined. Many toolchains can use the smallest integer type that fits the values (e.g., with -fshort-enums), so sizeof(enum) can be less than sizeof(int), and alignment may differ.
> > Casting enum* to int* and accessing through the int pointer runs into strict-aliasing and effective-type rules; it is not portable and may be undefined.
> > 
> > FFmpeg defaults to the equality of sizeof(enum) and sizeof(int):
> > We have seen code patterns that implicitly assume “enum is a 4-byte int” and interchange enum* with int*. Typical examples include:
> > 
> > BufferSinkContext {
> > enum AVSampleFormat *sample_formats
> > ...
> > }
> > 
> > int ff_set_common_formats_from_list2(const AVFilterContext *ctx,
> >                                      AVFilterFormatsConfig **cfg_in,
> >                                      AVFilterFormatsConfig **cfg_out,
> >                                      const int *fmts)
> > 
> > static int asink_query_formats(const AVFilterContext *ctx,
> >                                AVFilterFormatsConfig **cfg_in,
> >                                AVFilterFormatsConfig **cfg_out)
> >  {
> >     const BufferSinkContext *buf = ctx->priv;
> >     if (buf->nb_sample_formats) {
> >        ret = ff_set_common_formats_from_list2(ctx, cfg_in, cfg_out, buf->sample_formats);
> >        if (ret < 0)
> >            return ret;
> >     }
> >  ...
> >  }
> > We are facing a conflict dilemma:
> > We found that many ARM-based embedded compilers and third-party libraries default to -fshort-enums, which conflicts with FFmpeg's ABI (compiled with -fno-short-enums). However, if we enable the '-fshort-enums' in FFMPEG, the above ABI causes format negotiation and matching failures in our modules.
> > Proposed paths forward
> > 
> > Change “assume 4 bytes” to “force 4 bytes”.
> > Adopt a fixed 32-bit representation for enums at API/ABI boundaries
> > vulkan for example, uses *_MAX_ENUM = 0x7FFFFFFF as last value for each enum, to avoid any and all problems with size
> > 
> > Stop assuming sizeof(enum) == 4.
> > Gradually fix code to avoid enum* ↔ int* aliasing; use value-level conversions or memcpy where needed.
> > Pros: preserves the size benefits where compilers choose smaller enums.
> > Cons: requires auditing and incremental code changes.
> > 
> > We’re seeking the community’s guidance on which direction is preferable for FFmpeg: Should we standardize on a fixed 32-bit enum representation for ABI stability? Or should we remove the “enum == int” assumption and accept smaller enums, with code cleanup to avoid aliasing and stride errors?
> > We’re ready to contribute patches in the direction the project prefers and to align with FFmpeg’s coding guidelines. Thanks again for your time and feedback.
> > 
> 
> I think the issue is common and better be discussed on the mailing list. It's a big decision.

The way we mix enums and int currently is not good, so iam in favor of some
solution.
Which solution, i dont have a strong oppinion about.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Homeopathy is like voting while filling the ballot out with transparent ink.
Sometimes the outcome one wanted occurs. Rarely its worse than filling out
a ballot properly.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 163 bytes --]

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org