[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[Zhao Zhili via ffmpeg-devel]

> On Dec 24, 2025, at 11:30, nyh163925 <code@ffmpeg.org> wrote:
> 
> 
> Hello FFmpeg community, I am from Xiaomi Vela, and we have built the Vela multimedia system using FFmpeg. Thank you for the invaluable work you do—FFmpeg is the cornerstone of many of our modules.
> 
> ISSUE: -fshort-enum ABI problem
> 
> In the process of application, we found that some embedded platforms (such as ARM Cortex-m) have enabled -fshort-enum optimization by default in their compilers, which leads to ABI in the cross passing of some enum * and int * in FFmpeg. For example, when passing enum AVSampleFormat * as int * in ff_det_common_fformats_for_list2(), it will cause check_fformat() error.
> 
> Context
> 
> C language Standard undefined behavior:
> The underlying type and size of an enum are implementation-defined. Many toolchains can use the smallest integer type that fits the values (e.g., with -fshort-enums), so sizeof(enum) can be less than sizeof(int), and alignment may differ.
> Casting enum* to int* and accessing through the int pointer runs into strict-aliasing and effective-type rules; it is not portable and may be undefined.
> 
> FFmpeg defaults to the equality of sizeof(enum) and sizeof(int):
> We have seen code patterns that implicitly assume “enum is a 4-byte int” and interchange enum* with int*. Typical examples include:
> 
> BufferSinkContext {
> enum AVSampleFormat *sample_formats
> ...
> }
> 
> int ff_set_common_formats_from_list2(const AVFilterContext *ctx,
>                                      AVFilterFormatsConfig **cfg_in,
>                                      AVFilterFormatsConfig **cfg_out,
>                                      const int *fmts)
> 
> static int asink_query_formats(const AVFilterContext *ctx,
>                                AVFilterFormatsConfig **cfg_in,
>                                AVFilterFormatsConfig **cfg_out)
>  {
>     const BufferSinkContext *buf = ctx->priv;
>     if (buf->nb_sample_formats) {
>        ret = ff_set_common_formats_from_list2(ctx, cfg_in, cfg_out, buf->sample_formats);
>        if (ret < 0)
>            return ret;
>     }
>  ...
>  }
> We are facing a conflict dilemma:
> We found that many ARM-based embedded compilers and third-party libraries default to -fshort-enums, which conflicts with FFmpeg's ABI (compiled with -fno-short-enums). However, if we enable the '-fshort-enums' in FFMPEG, the above ABI causes format negotiation and matching failures in our modules.
> Proposed paths forward
> 
> Change “assume 4 bytes” to “force 4 bytes”.
> Adopt a fixed 32-bit representation for enums at API/ABI boundaries
> vulkan for example, uses *_MAX_ENUM = 0x7FFFFFFF as last value for each enum, to avoid any and all problems with size
> 
> Stop assuming sizeof(enum) == 4.
> Gradually fix code to avoid enum* ↔ int* aliasing; use value-level conversions or memcpy where needed.
> Pros: preserves the size benefits where compilers choose smaller enums.
> Cons: requires auditing and incremental code changes.
> 
> We’re seeking the community’s guidance on which direction is preferable for FFmpeg: Should we standardize on a fixed 32-bit enum representation for ABI stability? Or should we remove the “enum == int” assumption and accept smaller enums, with code cleanup to avoid aliasing and stride errors?
> We’re ready to contribute patches in the direction the project prefers and to align with FFmpeg’s coding guidelines. Thanks again for your time and feedback.
> 

I think the issue is common and better be discussed on the mailing list. It's a big decision.

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 4270 bytes --]

Hi Zhao

On Tue, Dec 30, 2025 at 12:46:24AM +0800, Zhao Zhili via ffmpeg-devel wrote:
> 
> 
> > On Dec 24, 2025, at 11:30, nyh163925 <code@ffmpeg.org> wrote:
> > 
> > 
> > Hello FFmpeg community, I am from Xiaomi Vela, and we have built the Vela multimedia system using FFmpeg. Thank you for the invaluable work you do—FFmpeg is the cornerstone of many of our modules.
> > 
> > ISSUE: -fshort-enum ABI problem
> > 
> > In the process of application, we found that some embedded platforms (such as ARM Cortex-m) have enabled -fshort-enum optimization by default in their compilers, which leads to ABI in the cross passing of some enum * and int * in FFmpeg. For example, when passing enum AVSampleFormat * as int * in ff_det_common_fformats_for_list2(), it will cause check_fformat() error.
> > 
> > Context
> > 
> > C language Standard undefined behavior:
> > The underlying type and size of an enum are implementation-defined. Many toolchains can use the smallest integer type that fits the values (e.g., with -fshort-enums), so sizeof(enum) can be less than sizeof(int), and alignment may differ.
> > Casting enum* to int* and accessing through the int pointer runs into strict-aliasing and effective-type rules; it is not portable and may be undefined.
> > 
> > FFmpeg defaults to the equality of sizeof(enum) and sizeof(int):
> > We have seen code patterns that implicitly assume “enum is a 4-byte int” and interchange enum* with int*. Typical examples include:
> > 
> > BufferSinkContext {
> > enum AVSampleFormat *sample_formats
> > ...
> > }
> > 
> > int ff_set_common_formats_from_list2(const AVFilterContext *ctx,
> >                                      AVFilterFormatsConfig **cfg_in,
> >                                      AVFilterFormatsConfig **cfg_out,
> >                                      const int *fmts)
> > 
> > static int asink_query_formats(const AVFilterContext *ctx,
> >                                AVFilterFormatsConfig **cfg_in,
> >                                AVFilterFormatsConfig **cfg_out)
> >  {
> >     const BufferSinkContext *buf = ctx->priv;
> >     if (buf->nb_sample_formats) {
> >        ret = ff_set_common_formats_from_list2(ctx, cfg_in, cfg_out, buf->sample_formats);
> >        if (ret < 0)
> >            return ret;
> >     }
> >  ...
> >  }
> > We are facing a conflict dilemma:
> > We found that many ARM-based embedded compilers and third-party libraries default to -fshort-enums, which conflicts with FFmpeg's ABI (compiled with -fno-short-enums). However, if we enable the '-fshort-enums' in FFMPEG, the above ABI causes format negotiation and matching failures in our modules.
> > Proposed paths forward
> > 
> > Change “assume 4 bytes” to “force 4 bytes”.
> > Adopt a fixed 32-bit representation for enums at API/ABI boundaries
> > vulkan for example, uses *_MAX_ENUM = 0x7FFFFFFF as last value for each enum, to avoid any and all problems with size
> > 
> > Stop assuming sizeof(enum) == 4.
> > Gradually fix code to avoid enum* ↔ int* aliasing; use value-level conversions or memcpy where needed.
> > Pros: preserves the size benefits where compilers choose smaller enums.
> > Cons: requires auditing and incremental code changes.
> > 
> > We’re seeking the community’s guidance on which direction is preferable for FFmpeg: Should we standardize on a fixed 32-bit enum representation for ABI stability? Or should we remove the “enum == int” assumption and accept smaller enums, with code cleanup to avoid aliasing and stride errors?
> > We’re ready to contribute patches in the direction the project prefers and to align with FFmpeg’s coding guidelines. Thanks again for your time and feedback.
> > 
> 
> I think the issue is common and better be discussed on the mailing list. It's a big decision.

The way we mix enums and int currently is not good, so iam in favor of some
solution.
Which solution, i dont have a strong oppinion about.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Homeopathy is like voting while filling the ballot out with transparent ink.
Sometimes the outcome one wanted occurs. Rarely its worse than filling out
a ballot properly.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 163 bytes --]

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[Zhao Zhili via ffmpeg-devel]

> On Dec 31, 2025, at 04:44, Michael Niedermayer via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> wrote:
> 
> Hi Zhao
> 
> On Tue, Dec 30, 2025 at 12:46:24AM +0800, Zhao Zhili via ffmpeg-devel wrote:
>> 
>> 
>>> On Dec 24, 2025, at 11:30, nyh163925 <code@ffmpeg.org> wrote:
>>> 
>>> 
>>> Hello FFmpeg community, I am from Xiaomi Vela, and we have built the Vela multimedia system using FFmpeg. Thank you for the invaluable work you do—FFmpeg is the cornerstone of many of our modules.
>>> 
>>> ISSUE: -fshort-enum ABI problem
>>> 
>>> In the process of application, we found that some embedded platforms (such as ARM Cortex-m) have enabled -fshort-enum optimization by default in their compilers, which leads to ABI in the cross passing of some enum * and int * in FFmpeg. For example, when passing enum AVSampleFormat * as int * in ff_det_common_fformats_for_list2(), it will cause check_fformat() error.
>>> 
>>> Context
>>> 
>>> C language Standard undefined behavior:
>>> The underlying type and size of an enum are implementation-defined. Many toolchains can use the smallest integer type that fits the values (e.g., with -fshort-enums), so sizeof(enum) can be less than sizeof(int), and alignment may differ.
>>> Casting enum* to int* and accessing through the int pointer runs into strict-aliasing and effective-type rules; it is not portable and may be undefined.
>>> 
>>> FFmpeg defaults to the equality of sizeof(enum) and sizeof(int):
>>> We have seen code patterns that implicitly assume “enum is a 4-byte int” and interchange enum* with int*. Typical examples include:
>>> 
>>> BufferSinkContext {
>>> enum AVSampleFormat *sample_formats
>>> ...
>>> }
>>> 
>>> int ff_set_common_formats_from_list2(const AVFilterContext *ctx,
>>>                                     AVFilterFormatsConfig **cfg_in,
>>>                                     AVFilterFormatsConfig **cfg_out,
>>>                                     const int *fmts)
>>> 
>>> static int asink_query_formats(const AVFilterContext *ctx,
>>>                               AVFilterFormatsConfig **cfg_in,
>>>                               AVFilterFormatsConfig **cfg_out)
>>> {
>>>    const BufferSinkContext *buf = ctx->priv;
>>>    if (buf->nb_sample_formats) {
>>>       ret = ff_set_common_formats_from_list2(ctx, cfg_in, cfg_out, buf->sample_formats);
>>>       if (ret < 0)
>>>           return ret;
>>>    }
>>> ...
>>> }
>>> We are facing a conflict dilemma:
>>> We found that many ARM-based embedded compilers and third-party libraries default to -fshort-enums, which conflicts with FFmpeg's ABI (compiled with -fno-short-enums). However, if we enable the '-fshort-enums' in FFMPEG, the above ABI causes format negotiation and matching failures in our modules.
>>> Proposed paths forward
>>> 
>>> Change “assume 4 bytes” to “force 4 bytes”.
>>> Adopt a fixed 32-bit representation for enums at API/ABI boundaries
>>> vulkan for example, uses *_MAX_ENUM = 0x7FFFFFFF as last value for each enum, to avoid any and all problems with size
>>> 
>>> Stop assuming sizeof(enum) == 4.
>>> Gradually fix code to avoid enum* ↔ int* aliasing; use value-level conversions or memcpy where needed.
>>> Pros: preserves the size benefits where compilers choose smaller enums.
>>> Cons: requires auditing and incremental code changes.
>>> 
>>> We’re seeking the community’s guidance on which direction is preferable for FFmpeg: Should we standardize on a fixed 32-bit enum representation for ABI stability? Or should we remove the “enum == int” assumption and accept smaller enums, with code cleanup to avoid aliasing and stride errors?
>>> We’re ready to contribute patches in the direction the project prefers and to align with FFmpeg’s coding guidelines. Thanks again for your time and feedback.
>>> 
>> 
>> I think the issue is common and better be discussed on the mailing list. It's a big decision.
> 
> The way we mix enums and int currently is not good, so iam in favor of some
> solution.
> Which solution, i dont have a strong oppinion about.

Once set AV_SAMPLE_FMT_MAX = 0x7FFFFFFF, it’s hard to revert and choose
another solution.

If we are unable to decide on which solution to adopt for the moment, can we start
by fixing some internal code issues caused by short-enum first?

> 
> thx
> 
> [...]
> 
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> Homeopathy is like voting while filling the ballot out with transparent ink.
> Sometimes the outcome one wanted occurs. Rarely its worse than filling out
> a ballot properly.
> _______________________________________________
> ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
> To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org

[FFmpeg-devel] Re: [FFFjo] [FFmpeg/FFmpeg] -fshort-enum breaks ABI compatibility and enum/int pointer conversions (Issue #21289)

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 5011 bytes --]

Hi

On Wed, Dec 31, 2025 at 10:31:30AM +0800, Zhao Zhili via ffmpeg-devel wrote:
> 
> 
> > On Dec 31, 2025, at 04:44, Michael Niedermayer via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> wrote:
> > 
> > Hi Zhao
> > 
> > On Tue, Dec 30, 2025 at 12:46:24AM +0800, Zhao Zhili via ffmpeg-devel wrote:
> >> 
> >> 
> >>> On Dec 24, 2025, at 11:30, nyh163925 <code@ffmpeg.org> wrote:
> >>> 
> >>> 
> >>> Hello FFmpeg community, I am from Xiaomi Vela, and we have built the Vela multimedia system using FFmpeg. Thank you for the invaluable work you do—FFmpeg is the cornerstone of many of our modules.
> >>> 
> >>> ISSUE: -fshort-enum ABI problem
> >>> 
> >>> In the process of application, we found that some embedded platforms (such as ARM Cortex-m) have enabled -fshort-enum optimization by default in their compilers, which leads to ABI in the cross passing of some enum * and int * in FFmpeg. For example, when passing enum AVSampleFormat * as int * in ff_det_common_fformats_for_list2(), it will cause check_fformat() error.
> >>> 
> >>> Context
> >>> 
> >>> C language Standard undefined behavior:
> >>> The underlying type and size of an enum are implementation-defined. Many toolchains can use the smallest integer type that fits the values (e.g., with -fshort-enums), so sizeof(enum) can be less than sizeof(int), and alignment may differ.
> >>> Casting enum* to int* and accessing through the int pointer runs into strict-aliasing and effective-type rules; it is not portable and may be undefined.
> >>> 
> >>> FFmpeg defaults to the equality of sizeof(enum) and sizeof(int):
> >>> We have seen code patterns that implicitly assume “enum is a 4-byte int” and interchange enum* with int*. Typical examples include:
> >>> 
> >>> BufferSinkContext {
> >>> enum AVSampleFormat *sample_formats
> >>> ...
> >>> }
> >>> 
> >>> int ff_set_common_formats_from_list2(const AVFilterContext *ctx,
> >>>                                     AVFilterFormatsConfig **cfg_in,
> >>>                                     AVFilterFormatsConfig **cfg_out,
> >>>                                     const int *fmts)
> >>> 
> >>> static int asink_query_formats(const AVFilterContext *ctx,
> >>>                               AVFilterFormatsConfig **cfg_in,
> >>>                               AVFilterFormatsConfig **cfg_out)
> >>> {
> >>>    const BufferSinkContext *buf = ctx->priv;
> >>>    if (buf->nb_sample_formats) {
> >>>       ret = ff_set_common_formats_from_list2(ctx, cfg_in, cfg_out, buf->sample_formats);
> >>>       if (ret < 0)
> >>>           return ret;
> >>>    }
> >>> ...
> >>> }
> >>> We are facing a conflict dilemma:
> >>> We found that many ARM-based embedded compilers and third-party libraries default to -fshort-enums, which conflicts with FFmpeg's ABI (compiled with -fno-short-enums). However, if we enable the '-fshort-enums' in FFMPEG, the above ABI causes format negotiation and matching failures in our modules.
> >>> Proposed paths forward
> >>> 
> >>> Change “assume 4 bytes” to “force 4 bytes”.
> >>> Adopt a fixed 32-bit representation for enums at API/ABI boundaries
> >>> vulkan for example, uses *_MAX_ENUM = 0x7FFFFFFF as last value for each enum, to avoid any and all problems with size
> >>> 
> >>> Stop assuming sizeof(enum) == 4.
> >>> Gradually fix code to avoid enum* ↔ int* aliasing; use value-level conversions or memcpy where needed.
> >>> Pros: preserves the size benefits where compilers choose smaller enums.
> >>> Cons: requires auditing and incremental code changes.
> >>> 
> >>> We’re seeking the community’s guidance on which direction is preferable for FFmpeg: Should we standardize on a fixed 32-bit enum representation for ABI stability? Or should we remove the “enum == int” assumption and accept smaller enums, with code cleanup to avoid aliasing and stride errors?
> >>> We’re ready to contribute patches in the direction the project prefers and to align with FFmpeg’s coding guidelines. Thanks again for your time and feedback.
> >>> 
> >> 
> >> I think the issue is common and better be discussed on the mailing list. It's a big decision.
> > 
> > The way we mix enums and int currently is not good, so iam in favor of some
> > solution.
> > Which solution, i dont have a strong oppinion about.
> 
> Once set AV_SAMPLE_FMT_MAX = 0x7FFFFFFF, it’s hard to revert and choose
> another solution.
> 
> If we are unable to decide on which solution to adopt for the moment, can we start
> by fixing some internal code issues caused by short-enum first?

yes
i suspect AVOption access to type enum will be the bulk of what needs fixing


thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The day soldiers stop bringing you their problems is the day you have stopped 
leading them. They have either lost confidence that you can help or concluded 
you do not care. Either case is a failure of leadership. - Colin Powell

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 163 bytes --]

_______________________________________________
ffmpeg-devel mailing list -- ffmpeg-devel@ffmpeg.org
To unsubscribe send an email to ffmpeg-devel-leave@ffmpeg.org