[FFmpeg-devel] [PATCH] aacencdsp: Improve consistency with assembly, for x87 math (PR #20322)

[FFmpeg-devel] [PATCH] aacencdsp: Improve consistency with assembly, for x87 math (PR #20322)

[Martin Storsjö via ffmpeg-devel]

PR #20322 opened by Martin Storsjö (mstorsjo)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20322
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20322.patch

Currently, the aacencdsp checkasm tests fails for many seeds,
if the C code has been built with x87 math. This happens because
the excess precision of x87 math can make it end up rounding
to a different integer, and the checkasm tests checks that the
output integers match exactly between C and assembly.

One such failing case is "tests/checkasm/checkasm --test=aacencdsp
41" when compiled with GCC. When compiled with Clang, the test
seed 21 produces a failure.

To avoid the issue, we need to limit the precision of intermediates
to their nominal float range, matching the assembly implementations.

This can be achieved when compiling with GCC, by just adding a single
cast.

To observe the effect of this cast, compile the following
snippet,

    int cast(float a, float b) {
        return (int)
    #ifdef CAST
            (float)
    #endif
            (a + b);
    }

with "gcc -m32 -std=c17 -O2", with/without -DCAST. For x86_64
cases (without the "-m32"), the cast doesn't make any difference
on the generated code.

This cast would seem to not have any effect, as a binary expression
with float inputs also would have the type float.

However, if compiling with GCC with -fexcess-precision=standard,
the cast forces limiting the precision according to the language
standard here - according to the GCC docs [1]:

> When compiling C or C++, if -fexcess-precision=standard is
> specified then excess precision follows the rules specified in
> ISO C99 or C++; in particular, both casts and assignments cause
> values to be rounded to their semantic types (whereas -ffloat-store
> only affects assignments). This option is enabled by default for
> C or C++ if a strict conformance option such as -std=c99 or
> -std=c++17 is used.

Ffmpeg's configure scripts enables -std=c17 by default.

This only helps with GCC though - the cast doesn't make any
difference for Clang. (Although, upstream Clang seems to default
to SSE math, while Ubuntu provided Clang defaults to x87 math.)
Limiting the precision with Clang would require casting to volatile
float for both intermediates here - and that does have a code
generation effect on all architectures.

[1] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html


From 95f123f746c8ba0b1eddfcafbf89ebc2493e69e1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Martin=20Storsj=C3=B6?= <martin@martin.st>
Date: Fri, 22 Aug 2025 23:26:11 +0300
Subject: [PATCH] aacencdsp: Improve consistency with assembly, for x87 math

Currently, the aacencdsp checkasm tests fails for many seeds,
if the C code has been built with x87 math. This happens because
the excess precision of x87 math can make it end up rounding
to a different integer, and the checkasm tests checks that the
output integers match exactly between C and assembly.

One such failing case is "tests/checkasm/checkasm --test=aacencdsp
41" when compiled with GCC. When compiled with Clang, the test
seed 21 produces a failure.

To avoid the issue, we need to limit the precision of intermediates
to their nominal float range, matching the assembly implementations.

This can be achieved when compiling with GCC, by just adding a single
cast.

To observe the effect of this cast, compile the following
snippet,

    int cast(float a, float b) {
        return (int)
    #ifdef CAST
            (float)
    #endif
            (a + b);
    }

with "gcc -m32 -std=c17 -O2", with/without -DCAST. For x86_64
cases (without the "-m32"), the cast doesn't make any difference
on the generated code.

This cast would seem to not have any effect, as a binary expression
with float inputs also would have the type float.

However, if compiling with GCC with -fexcess-precision=standard,
the cast forces limiting the precision according to the language
standard here - according to the GCC docs [1]:

> When compiling C or C++, if -fexcess-precision=standard is
> specified then excess precision follows the rules specified in
> ISO C99 or C++; in particular, both casts and assignments cause
> values to be rounded to their semantic types (whereas -ffloat-store
> only affects assignments). This option is enabled by default for
> C or C++ if a strict conformance option such as -std=c99 or
> -std=c++17 is used.

Ffmpeg's configure scripts enables -std=c17 by default.

This only helps with GCC though - the cast doesn't make any
difference for Clang. (Although, upstream Clang seems to default
to SSE math, while Ubuntu provided Clang defaults to x87 math.)
Limiting the precision with Clang would require casting to volatile
float for both intermediates here - and that does have a code
generation effect on all architectures.

[1] https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html
---
 libavcodec/aacencdsp.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/aacencdsp.h b/libavcodec/aacencdsp.h
index d0d86c3d70..684bbc254f 100644
--- a/libavcodec/aacencdsp.h
+++ b/libavcodec/aacencdsp.h
@@ -50,7 +50,7 @@ static inline void quantize_bands(int *out, const float *in, const float *scaled
 {
     for (int i = 0; i < size; i++) {
         float qc = scaled[i] * Q34;
-        int tmp = (int)FFMIN(qc + rounding, (float)maxval);
+        int tmp = (int)FFMIN((float)(qc + rounding), (float)maxval);
         if (is_signed && in[i] < 0.0f) {
             tmp = -tmp;
         }
-- 
2.49.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".