Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 5856 bytes --]

On Thu, Dec 26, 2024 at 07:33:23PM +0000, Niklas Haas wrote:
> ffmpeg | branch: master | Niklas Haas <git@haasn.dev> | Mon Dec 16 14:49:39 2024 +0100| [af6d52eec66961f6a502b0f2f390c12226d087cd] | committer: Niklas Haas
> 
> swscale: use 16-bit intermediate precision for RGB/XYZ conversion
> 
> The current logic uses 12-bit linear light math, which is woefully insufficient
> and leads to nasty postarization artifacts. This patch simply switches the
> internal logic to 16-bit precision.
> 
> This raises the memory requirement of these tables from 32 kB to 272 kB.
> 
> All relevant FATE tests updated for improved accuracy.
> 
> Fixes: #4829
> Signed-off-by: Niklas Haas <git@haasn.dev>
> Sponsored-by: Sovereign Tech Fund
> 
> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=af6d52eec66961f6a502b0f2f390c12226d087cd
> ---

this breaks on x86-32

grep Err fatetest
make: *** [src/tests/Makefile:310: fate-filter-pixdesc-xyz12le] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixdesc-xyz12be] Error 1
make: *** [src/tests/Makefile:310: fate-pixfmt-xyz12le] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-fieldorder] Error 1
make: *** [src/tests/Makefile:310: fate-filter-pixfmts-crop] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-transpose] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-field] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-copy] Error 1
make: *** [src/tests/Makefile:310: fate-filter-pixfmts-hflip] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-il] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-null] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-vflip] Error 1
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-scale] Error 1

examples:
--- src/tests/ref/fate/filter-pixfmts-copy	2025-01-26 19:13:18.982388050 +0100
+++ tests/data/fate/filter-pixfmts-copy	2025-01-26 19:14:39.267034804 +0100
@@ -111,8 +111,8 @@
 xv36le              e08dcbde02f1c28a3554f372ad1278e2
 xv48be              ce34993b4b4411bba1d852b9b86aa39e
 xv48le              df913a7e61b162aa98303e5393e60c63
-xyz12be             f257f86373207af8aed0a1a05171df3b
-xyz12le             7922f99edc44a2c26a25becbea9914cc
+xyz12be             d4562ab725e3ab7e5573dd03218bb03c
+xyz12le             ce17d867c3fea4c42324e40353b72de4
 y210le              04e9487b6cce38e7531437e946cdd586
 y212le              825768be8fe92708ae80be84855066ed
 y216le              0e99aeddfee304e72d525d72998d9e9b
--- src/tests/ref/fate/filter-pixfmts-hflip	2025-01-26 19:13:18.982388050 +0100
+++ tests/data/fate/filter-pixfmts-hflip	2025-01-26 19:14:39.275034869 +0100
@@ -108,8 +108,8 @@
 xv36le              e478b4b54698beb3ce1b9a2dd691d544
 xv48be              e030a2c7b1b600cfacb691b6e90c2e3d
 xv48le              fbd7f8c65cd6fc9f9108dc9a1f977dc3
-xyz12be             3c50a51a3c486a0c6853e4bbbcf3f244
-xyz12le             e020897d826ea20ded16f30ea1eb018d
+xyz12be             c678c7f19c610063f352a0253121441f
+xyz12le             2320702cb1c8ccaad27d7004d4f3a195
 ya16be              70fa41c32ecaf3370edc38add6096db2
 ya16le              3b2c20f9e80717628ced6c6468507f63
 ya8                 4ad5920716de3d2fbbc49f95adb60345
Test filter-pixfmts-copy failed. Look at tests/data/fate/filter-pixfmts-copy.err for details.
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-copy] Error 1
Test filter-pixfmts-hflip failed. Look at tests/data/fate/filter-pixfmts-hflip.err for details.
make: *** [src/tests/Makefile:310: fate-filter-pixfmts-hflip] Error 1
TEST    vsynth2-mov-bgr24
TEST    vsynth2-mov-bpp15
--- src/tests/ref/fate/filter-pixfmts-il	2025-01-26 19:13:18.982388050 +0100
+++ tests/data/fate/filter-pixfmts-il	2025-01-26 19:14:39.315035190 +0100
@@ -110,8 +110,8 @@
 xv36le              102c0e817d375ddd6b2cfbb4262dec95
 xv48be              4d7376651fb7b3e84d00abad6c785aad
 xv48le              a1a8ff16d9a864568e5e557734bf3d6d
-xyz12be             b7d50e283360bf69fd661369110b26ef
-xyz12le             d5b1d45c3a136bb3d04f70a619c86c8d
+xyz12be             3d9ae42c65eb645bb8e99dc708716f2d
+xyz12le             b27ec91dcff02d2b94b17c136be55001
 y210le              d4cf9b53cd7ff22f087743d483e88480
 y212le              d5a2b4677ddb4a3bc3e5cd5cbb20f426
 y216le              9e44c6d76b09bcbe71738423b4b3d67a
Test filter-pixfmts-il failed. Look at tests/data/fate/filter-pixfmts-il.err for details.
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-il] Error 1
TEST    vsynth2-mov-bpp16
TEST    vsynth2-roqvideo
TEST    vsynth2-rpza
TEST    vsynth2-smc
--- src/tests/ref/fate/filter-pixfmts-null	2025-01-26 19:13:18.982388050 +0100
+++ tests/data/fate/filter-pixfmts-null	2025-01-26 19:14:39.451036285 +0100
@@ -111,8 +111,8 @@
 xv36le              e08dcbde02f1c28a3554f372ad1278e2
 xv48be              ce34993b4b4411bba1d852b9b86aa39e
 xv48le              df913a7e61b162aa98303e5393e60c63
-xyz12be             f257f86373207af8aed0a1a05171df3b
-xyz12le             7922f99edc44a2c26a25becbea9914cc
+xyz12be             d4562ab725e3ab7e5573dd03218bb03c
+xyz12le             ce17d867c3fea4c42324e40353b72de4
 y210le              04e9487b6cce38e7531437e946cdd586
 y212le              825768be8fe92708ae80be84855066ed
 y216le              0e99aeddfee304e72d525d72998d9e9b
TEST    vsynth2-snow
TEST    vsynth2-snow-hpel
Test filter-pixfmts-null failed. Look at tests/data/fate/filter-pixfmts-null.err for details.
make: *** [src/tests/Makefile:311: fate-filter-pixfmts-null] Error 1


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The difference between a dictatorship and a democracy is that every 4 years
the population together is allowed to provide 1 bit of input to the government.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2620 bytes --]

Hi

On Sun, Jan 26, 2025 at 07:26:59PM +0100, Michael Niedermayer wrote:
> On Thu, Dec 26, 2024 at 07:33:23PM +0000, Niklas Haas wrote:
> > ffmpeg | branch: master | Niklas Haas <git@haasn.dev> | Mon Dec 16 14:49:39 2024 +0100| [af6d52eec66961f6a502b0f2f390c12226d087cd] | committer: Niklas Haas
> > 
> > swscale: use 16-bit intermediate precision for RGB/XYZ conversion
> > 
> > The current logic uses 12-bit linear light math, which is woefully insufficient
> > and leads to nasty postarization artifacts. This patch simply switches the
> > internal logic to 16-bit precision.
> > 
> > This raises the memory requirement of these tables from 32 kB to 272 kB.
> > 
> > All relevant FATE tests updated for improved accuracy.
> > 
> > Fixes: #4829
> > Signed-off-by: Niklas Haas <git@haasn.dev>
> > Sponsored-by: Sovereign Tech Fund
> > 
> > > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=af6d52eec66961f6a502b0f2f390c12226d087cd
> > ---
> 
> this breaks on x86-32

ping

the fate tests for a major architecture are broken since this
This impacts future bisections over that range for example

you can trivially test this on x86-64 with something like:
--cross-prefix=/usr/i686-linux-gnu/bin/ --cc='ccache i686-linux-gnu-gcc-7' --arch=x86_32  --target-os=linux --enable-cross-compile
if you have the correponding packages installed

before af6d52eec66961f6a502b0f2f390c12226d087cd
TEST    filter-pixdesc-xyz12be

after af6d52eec66961f6a502b0f2f390c12226d087cd:
--- src/tests/ref/fate/filter-pixdesc-xyz12be	2025-03-14 00:46:40.445681223 +0100
+++ tests/data/fate/filter-pixdesc-xyz12be	2025-03-14 00:47:08.301903796 +0100
@@ -1 +1 @@
-pixdesc-xyz12be     1508a33dea936c45d9ee13f7743af00d
+pixdesc-xyz12be     198f43f452bc55f4ca1e0e0171de5c4c
Test filter-pixdesc-xyz12be failed. Look at tests/data/fate/filter-pixdesc-xyz12be.err for details.
make: *** [src/tests/Makefile:311: fate-filter-pixdesc-xyz12be] Error 1

in master:
--- src/tests/ref/fate/filter-pixdesc-xyz12be	2025-03-14 00:46:40.445681223 +0100
+++ tests/data/fate/filter-pixdesc-xyz12be	2025-03-14 00:48:43.042660972 +0100
@@ -1 +1 @@
-pixdesc-xyz12be     1508a33dea936c45d9ee13f7743af00d
+pixdesc-xyz12be     198f43f452bc55f4ca1e0e0171de5c4c
Test filter-pixdesc-xyz12be failed. Look at tests/data/fate/filter-pixdesc-xyz12be.err for details.
make: *** [src/tests/Makefile:315: fate-filter-pixdesc-xyz12be] Error 1


thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Never trust a computer, one day, it may think you are the virus. -- Compn

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1594 bytes --]

On Fri, Mar 14, 2025 at 12:54:25AM +0100, Michael Niedermayer wrote:
> Hi
> 
> On Sun, Jan 26, 2025 at 07:26:59PM +0100, Michael Niedermayer wrote:
> > On Thu, Dec 26, 2024 at 07:33:23PM +0000, Niklas Haas wrote:
> > > ffmpeg | branch: master | Niklas Haas <git@haasn.dev> | Mon Dec 16 14:49:39 2024 +0100| [af6d52eec66961f6a502b0f2f390c12226d087cd] | committer: Niklas Haas
> > > 
> > > swscale: use 16-bit intermediate precision for RGB/XYZ conversion
> > > 
> > > The current logic uses 12-bit linear light math, which is woefully insufficient
> > > and leads to nasty postarization artifacts. This patch simply switches the
> > > internal logic to 16-bit precision.
> > > 
> > > This raises the memory requirement of these tables from 32 kB to 272 kB.
> > > 
> > > All relevant FATE tests updated for improved accuracy.
> > > 
> > > Fixes: #4829
> > > Signed-off-by: Niklas Haas <git@haasn.dev>
> > > Sponsored-by: Sovereign Tech Fund
> > > 
> > > > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=af6d52eec66961f6a502b0f2f390c12226d087cd
> > > ---
> > 
> > this breaks on x86-32
> 
> ping
> 
> the fate tests for a major architecture are broken since this
> This impacts future bisections over that range for example

It also prevents others from easily seeing other failures as x86-32 is
already broken

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Freedom in capitalist society always remains about the same as it was in
ancient Greek republics: Freedom for slave owners. -- Vladimir Lenin

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

[Niklas Haas]

On Fri, 14 Mar 2025 00:54:25 +0100 Michael Niedermayer <michael@niedermayer.cc> wrote:
> Hi
> 
> On Sun, Jan 26, 2025 at 07:26:59PM +0100, Michael Niedermayer wrote:
> > On Thu, Dec 26, 2024 at 07:33:23PM +0000, Niklas Haas wrote:
> > > ffmpeg | branch: master | Niklas Haas <git@haasn.dev> | Mon Dec 16 14:49:39 2024 +0100| [af6d52eec66961f6a502b0f2f390c12226d087cd] | committer: Niklas Haas
> > > 
> > > swscale: use 16-bit intermediate precision for RGB/XYZ conversion
> > > 
> > > The current logic uses 12-bit linear light math, which is woefully insufficient
> > > and leads to nasty postarization artifacts. This patch simply switches the
> > > internal logic to 16-bit precision.
> > > 
> > > This raises the memory requirement of these tables from 32 kB to 272 kB.
> > > 
> > > All relevant FATE tests updated for improved accuracy.
> > > 
> > > Fixes: #4829
> > > Signed-off-by: Niklas Haas <git@haasn.dev>
> > > Sponsored-by: Sovereign Tech Fund
> > > 
> > > > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=af6d52eec66961f6a502b0f2f390c12226d087cd
> > > ---
> > 
> > this breaks on x86-32
> 
> ping
> 
> the fate tests for a major architecture are broken since this
> This impacts future bisections over that range for example

My bad, missed this email for some reason.

Will investigate and fix tomorrow.

> 
> you can trivially test this on x86-64 with something like:
> --cross-prefix=/usr/i686-linux-gnu/bin/ --cc='ccache i686-linux-gnu-gcc-7' --arch=x86_32  --target-os=linux --enable-cross-compile
> if you have the correponding packages installed
> 
> before af6d52eec66961f6a502b0f2f390c12226d087cd
> TEST    filter-pixdesc-xyz12be
> 
> after af6d52eec66961f6a502b0f2f390c12226d087cd:
> --- src/tests/ref/fate/filter-pixdesc-xyz12be	2025-03-14 00:46:40.445681223 +0100
> +++ tests/data/fate/filter-pixdesc-xyz12be	2025-03-14 00:47:08.301903796 +0100
> @@ -1 +1 @@
> -pixdesc-xyz12be     1508a33dea936c45d9ee13f7743af00d
> +pixdesc-xyz12be     198f43f452bc55f4ca1e0e0171de5c4c
> Test filter-pixdesc-xyz12be failed. Look at tests/data/fate/filter-pixdesc-xyz12be.err for details.
> make: *** [src/tests/Makefile:311: fate-filter-pixdesc-xyz12be] Error 1
> 
> in master:
> --- src/tests/ref/fate/filter-pixdesc-xyz12be	2025-03-14 00:46:40.445681223 +0100
> +++ tests/data/fate/filter-pixdesc-xyz12be	2025-03-14 00:48:43.042660972 +0100
> @@ -1 +1 @@
> -pixdesc-xyz12be     1508a33dea936c45d9ee13f7743af00d
> +pixdesc-xyz12be     198f43f452bc55f4ca1e0e0171de5c4c
> Test filter-pixdesc-xyz12be failed. Look at tests/data/fate/filter-pixdesc-xyz12be.err for details.
> make: *** [src/tests/Makefile:315: fate-filter-pixdesc-xyz12be] Error 1
> 
> 
> thx
> 
> [...]
> 
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> Never trust a computer, one day, it may think you are the virus. -- Compn
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

[Niklas Haas]

On Fri, 14 Mar 2025 01:09:30 +0100 Niklas Haas <ffmpeg@haasn.xyz> wrote:
> On Fri, 14 Mar 2025 00:54:25 +0100 Michael Niedermayer <michael@niedermayer.cc> wrote:
> > Hi
> >
> > On Sun, Jan 26, 2025 at 07:26:59PM +0100, Michael Niedermayer wrote:
> > > On Thu, Dec 26, 2024 at 07:33:23PM +0000, Niklas Haas wrote:
> > > > ffmpeg | branch: master | Niklas Haas <git@haasn.dev> | Mon Dec 16 14:49:39 2024 +0100| [af6d52eec66961f6a502b0f2f390c12226d087cd] | committer: Niklas Haas
> > > >
> > > > swscale: use 16-bit intermediate precision for RGB/XYZ conversion
> > > >
> > > > The current logic uses 12-bit linear light math, which is woefully insufficient
> > > > and leads to nasty postarization artifacts. This patch simply switches the
> > > > internal logic to 16-bit precision.
> > > >
> > > > This raises the memory requirement of these tables from 32 kB to 272 kB.
> > > >
> > > > All relevant FATE tests updated for improved accuracy.
> > > >
> > > > Fixes: #4829
> > > > Signed-off-by: Niklas Haas <git@haasn.dev>
> > > > Sponsored-by: Sovereign Tech Fund
> > > >
> > > > > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=af6d52eec66961f6a502b0f2f390c12226d087cd
> > > > ---
> > >
> > > this breaks on x86-32
> >
> > ping
> >
> > the fate tests for a major architecture are broken since this
> > This impacts future bisections over that range for example
>
> My bad, missed this email for some reason.
>
> Will investigate and fix tomorrow.

So, I found the bug, but I don't know how to fix it properly.

The problem is that tha XYZ/RGB conversion tables are calculated using excess
precision from the x87 FPU. This results in slightly different rounding behavior
in exactly one case (i == 3415).

Fixing this requires setting -mfpmath=sse at compile time. Strangely, I was
not able to force the compiler to discard the excess precision even with
forcing -fexcess-precision=standard and storing every single subexpression to
a `volatile double`.

How do we deal with such cases generally? I can think of several options, such
as hard-coding the table values (unappealing), or somehow using integer math
(very unappealing, how do we calculate an integer power without massive loss
of precision?)

>
> >
> > you can trivially test this on x86-64 with something like:
> > --cross-prefix=/usr/i686-linux-gnu/bin/ --cc='ccache i686-linux-gnu-gcc-7' --arch=x86_32  --target-os=linux --enable-cross-compile
> > if you have the correponding packages installed
> >
> > before af6d52eec66961f6a502b0f2f390c12226d087cd
> > TEST    filter-pixdesc-xyz12be
> >
> > after af6d52eec66961f6a502b0f2f390c12226d087cd:
> > --- src/tests/ref/fate/filter-pixdesc-xyz12be	2025-03-14 00:46:40.445681223 +0100
> > +++ tests/data/fate/filter-pixdesc-xyz12be	2025-03-14 00:47:08.301903796 +0100
> > @@ -1 +1 @@
> > -pixdesc-xyz12be     1508a33dea936c45d9ee13f7743af00d
> > +pixdesc-xyz12be     198f43f452bc55f4ca1e0e0171de5c4c
> > Test filter-pixdesc-xyz12be failed. Look at tests/data/fate/filter-pixdesc-xyz12be.err for details.
> > make: *** [src/tests/Makefile:311: fate-filter-pixdesc-xyz12be] Error 1
> >
> > in master:
> > --- src/tests/ref/fate/filter-pixdesc-xyz12be	2025-03-14 00:46:40.445681223 +0100
> > +++ tests/data/fate/filter-pixdesc-xyz12be	2025-03-14 00:48:43.042660972 +0100
> > @@ -1 +1 @@
> > -pixdesc-xyz12be     1508a33dea936c45d9ee13f7743af00d
> > +pixdesc-xyz12be     198f43f452bc55f4ca1e0e0171de5c4c
> > Test filter-pixdesc-xyz12be failed. Look at tests/data/fate/filter-pixdesc-xyz12be.err for details.
> > make: *** [src/tests/Makefile:315: fate-filter-pixdesc-xyz12be] Error 1
> >
> >
> > thx
> >
> > [...]
> >
> > --
> > Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> >
> > Never trust a computer, one day, it may think you are the virus. -- Compn
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [FFmpeg-cvslog] swscale: use 16-bit intermediate precision for RGB/XYZ conversion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3159 bytes --]

Hi

On Fri, Mar 14, 2025 at 09:44:22PM +0100, Niklas Haas wrote:
> On Fri, 14 Mar 2025 01:09:30 +0100 Niklas Haas <ffmpeg@haasn.xyz> wrote:
> > On Fri, 14 Mar 2025 00:54:25 +0100 Michael Niedermayer <michael@niedermayer.cc> wrote:
> > > Hi
> > >
> > > On Sun, Jan 26, 2025 at 07:26:59PM +0100, Michael Niedermayer wrote:
> > > > On Thu, Dec 26, 2024 at 07:33:23PM +0000, Niklas Haas wrote:
> > > > > ffmpeg | branch: master | Niklas Haas <git@haasn.dev> | Mon Dec 16 14:49:39 2024 +0100| [af6d52eec66961f6a502b0f2f390c12226d087cd] | committer: Niklas Haas
> > > > >
> > > > > swscale: use 16-bit intermediate precision for RGB/XYZ conversion
> > > > >
> > > > > The current logic uses 12-bit linear light math, which is woefully insufficient
> > > > > and leads to nasty postarization artifacts. This patch simply switches the
> > > > > internal logic to 16-bit precision.
> > > > >
> > > > > This raises the memory requirement of these tables from 32 kB to 272 kB.
> > > > >
> > > > > All relevant FATE tests updated for improved accuracy.
> > > > >
> > > > > Fixes: #4829
> > > > > Signed-off-by: Niklas Haas <git@haasn.dev>
> > > > > Sponsored-by: Sovereign Tech Fund
> > > > >
> > > > > > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=af6d52eec66961f6a502b0f2f390c12226d087cd
> > > > > ---
> > > >
> > > > this breaks on x86-32
> > >
> > > ping
> > >
> > > the fate tests for a major architecture are broken since this
> > > This impacts future bisections over that range for example
> >
> > My bad, missed this email for some reason.
> >
> > Will investigate and fix tomorrow.
> 
> So, I found the bug, but I don't know how to fix it properly.
> 
> The problem is that tha XYZ/RGB conversion tables are calculated using excess
> precision from the x87 FPU. This results in slightly different rounding behavior
> in exactly one case (i == 3415).
> 
> Fixing this requires setting -mfpmath=sse at compile time. Strangely, I was
> not able to force the compiler to discard the excess precision even with
> forcing -fexcess-precision=standard and storing every single subexpression to
> a `volatile double`.
> 
> How do we deal with such cases generally? I can think of several options, such
> as hard-coding the table values (unappealing), or somehow using integer math
> (very unappealing, how do we calculate an integer power without massive loss
> of precision?)

if its one case, would a
if(i == 3415)
    abcd= 12345.98765

fix it ?

do i understand correctly that you want to compute
pow(i / 4095.0, xyzgamma)
or something like that without floats ?

exp(log(i/4095) * xyzgamma)

is a classic one, here you just need a fixed point implemenattion of
log and exp
see log16 and exp16 from tests/tiny_psnr.c, maybe they are neough
or could serve as examples for higher precission ones

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The real ebay dictionary, page 2
"100% positive feedback" - "All either got their money back or didnt complain"
"Best seller ever, very honest" - "Seller refunded buyer after failed scam"

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".