[FFmpeg-devel] [PATCH v2 1/2] x86: aacencdsp: Fix negating signed values in aac_quantize_bands

[FFmpeg-devel] [PATCH v2 1/2] x86: aacencdsp: Fix negating signed values in aac_quantize_bands

[Martin Storsjö]

Previously, we would do OR with the sign bit, forcing the output
to a negative value, while we want to negate it, by inverting the
sign bit.
---
 libavcodec/x86/aacencdsp.asm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/x86/aacencdsp.asm b/libavcodec/x86/aacencdsp.asm
index 86eaebcbe5..8e435b7d2a 100644
--- a/libavcodec/x86/aacencdsp.asm
+++ b/libavcodec/x86/aacencdsp.asm
@@ -96,7 +96,7 @@ cglobal aac_quantize_bands, 5, 5, 6, out, in, scaled, size, is_signed, maxval, Q
     addps     m2, m1
     minps     m2, m3
     andps     m5, m4, [inq+sizeq]
-    orps      m2, m5
+    xorps     m2, m5
     cvttps2dq m2, m2
     mova      [outq+sizeq], m2
     add       sizeq, mmsize
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] [PATCH v2 2/2] checkasm: aacencdsp: Actually test nonzero values in quant_bands

[Martin Storsjö]

Previously, we read elements from ff_aac_pow34sf_tab; however
that table is initialized to zero; one needs to call
ff_aac_float_common_init() to make sure that the table is
initialized.

However, given the range of the input values, a large number of
entries in ff_aac_pow34sf_tab would give results outside of the
range for signed 32 bit integers. As the largest aac_cb_maxval
entry is 16, it seems more reasonable to produce values within
an order of mangitude of that value.

(When hitting INT_MIN, implementations may end up with different
results depending on whether the value is negated as a float or
as an int. This corner case is irrelevant in practice as this
is way outside of the expected value range here.)

Coincidentally, this fixes linking checkasm with Apple's older
linker. (In Xcode 15, Apple switched to a new linker. The one in
older toolchains seems to have a bug where it won't figure out to
load object files from a static library, if the only symbol
referenced in the object file is a "common" symbol, i.e. one for
a zero-initialized variable. This issue can also be reproduced with
newer Apple toolchains by passing -Wl,-ld_classic to the linker.)
---
 tests/checkasm/aacencdsp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/checkasm/aacencdsp.c b/tests/checkasm/aacencdsp.c
index 5308a2ac03..713284211c 100644
--- a/tests/checkasm/aacencdsp.c
+++ b/tests/checkasm/aacencdsp.c
@@ -67,7 +67,7 @@ static void test_abs_pow34(AACEncDSPContext *s)
 static void test_quant_bands(AACEncDSPContext *s)
 {
     int maxval = randomize_elem(aac_cb_maxval);
-    float q34 = randomize_elem(ff_aac_pow34sf_tab);
+    float q34 = (float)rnd() / (UINT_MAX / 1024);
     float rounding = (rnd() & 1) ? ROUND_TO_ZERO : ROUND_STANDARD;
     LOCAL_ALIGNED_16(float, in, [BUF_SIZE]);
     LOCAL_ALIGNED_16(float, scaled, [BUF_SIZE]);
-- 
2.43.0

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".