[FFmpeg-devel] [RFC] FFV1 float support

[FFmpeg-devel] [RFC] FFV1 float support

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2748 bytes --]

Hi everyone

Current FFv1 code with my patchset supports 16bit floats. The implementation
is quite simple. Which is good

I have tried improving compression, the gains are incremental, but its not
large overall. For example 44% space gain on the remapping table is just
0.1% overall.

I have few float samples. Its mainly one high quality slideshow of unrelated
16bit float images. These excercise a wide range fo things including negative
color values.
I think I have only one single (non slideshow) video of float 16 samples,

It turns out the most efficient way to store floats that i found, is to remap
them to integers. Not to store tham as sign, exponent, mantisse, i tried that
for many hours.

Storing them using a remapping, has a very nice side effect though, and that is
it will be easy to add 32bit and 64bit float support (once there is some
sample data)
Because a image of 64bit floats, after its split into slices of up to 256x256
can always be mapped into 16bit integers within each slice.

What about the mapping itself, it uses a rather simple rle coder. ive spend
most of the day today tuning that and failing to really beat that.
Using context from the previous image didnt work with the slideshow material
i have nor that one video i have. I tried using sign and exponent as context,
tried previous run, relations of runs, low bits and many more, but no or
insignificant improvments of yesterdays implementation was achieved.

There was some paper, lynne pointed my too about float remapping but that seemed
about remapping scientific 1-dimensional data. And what that paper did was
bascially reverse mapping floats into decimal numbers like 1.23456*10^-2
No image or video i found has samples like that. Also the paper was messy

Considering the whole remap table is now bascially down to slightly more than
0.1% of teh whole file, its not worth to look at that more.

Beyond the remapping, the storage of float follows what we do with integers
split into slices, perform decorrelation between planes, encode pixel for
pixel and sample for sample with a preidctor, and context from the
surrounding qualitized pixel differences and range coder.


If people have more ideas that fit into FFv1 float support.
(that is, good compresssion vs complexity vs speed and an architecture that
 fits into FFv1) ...

If not, i intend to apply my current patchset, update the FFv1 specification
and move on to some other work

thx

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Homeopathy is like voting while filling the ballot out with transparent ink.
Sometimes the outcome one wanted occurs. Rarely its worse than filling out
a ballot properly.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Lynne]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 1903 bytes --]

On 06/03/2025 01:15, Michael Niedermayer wrote:
> Hi everyone
> 
> Current FFv1 code with my patchset supports 16bit floats. The implementation
> is quite simple. Which is good
> 
> I have tried improving compression, the gains are incremental, but its not
> large overall. For example 44% space gain on the remapping table is just
> 0.1% overall.
> 
> I have few float samples. Its mainly one high quality slideshow of unrelated
> 16bit float images. These excercise a wide range fo things including negative
> color values.
> I think I have only one single (non slideshow) video of float 16 samples,
> 
> It turns out the most efficient way to store floats that i found, is to remap
> them to integers. Not to store tham as sign, exponent, mantisse, i tried that
> for many hours.
> 
> Storing them using a remapping, has a very nice side effect though, and that is
> it will be easy to add 32bit and 64bit float support (once there is some
> sample data)
> Because a image of 64bit floats, after its split into slices of up to 256x256
> can always be mapped into 16bit integers within each slice.

*how*? Unlike ints, you cannot simply shift and extract bits from floats 
and treat them as floats.

> What about the mapping itself, it uses a rather simple rle coder. ive spend
> most of the day today tuning that and failing to really beat that.
> Using context from the previous image didnt work with the slideshow material
> i have nor that one video i have. I tried using sign and exponent as context,
> tried previous run, relations of runs, low bits and many more, but no or
> insignificant improvments of yesterdays implementation was achieved.

I did mention this once before, but you should look into the Daala/Opus 
way of storing rawbits into a separate chunk at the end of the 
bitstream. It avoids polluting the range context with equiprobable bits.

[-- Attachment #1.1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 637 bytes --]

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3867 bytes --]

Hi

On Thu, Mar 06, 2025 at 03:14:49AM +0100, Lynne wrote:
> On 06/03/2025 01:15, Michael Niedermayer wrote:
> > Hi everyone
> > 
> > Current FFv1 code with my patchset supports 16bit floats. The implementation
> > is quite simple. Which is good
> > 
> > I have tried improving compression, the gains are incremental, but its not
> > large overall. For example 44% space gain on the remapping table is just
> > 0.1% overall.
> > 
> > I have few float samples. Its mainly one high quality slideshow of unrelated
> > 16bit float images. These excercise a wide range fo things including negative
> > color values.
> > I think I have only one single (non slideshow) video of float 16 samples,
> > 
> > It turns out the most efficient way to store floats that i found, is to remap
> > them to integers. Not to store tham as sign, exponent, mantisse, i tried that
> > for many hours.
> > 
> > Storing them using a remapping, has a very nice side effect though, and that is
> > it will be easy to add 32bit and 64bit float support (once there is some
> > sample data)

> > Because a image of 64bit floats, after its split into slices of up to 256x256
> > can always be mapped into 16bit integers within each slice.
> 
> *how*? Unlike ints, you cannot simply shift and extract bits from floats and
> treat them as floats.

well, if the input is IEEE 754 32bit or 64bit floats or some 16bit floats.
they can be read with AV_RL16/32/64 and are then 16/32/64bit ints. So we have
slices of 256x256 or smaller of such integers.
A 256x256 slice has at maximum 64k distinct values, these values can
always be stored in 16bit, once they are remapped to a compact list.

There are no floats after the remapping and before its inverted in the decoder

directly doing anything with floats is not bit exactly defined in C so we
have to work with integers anyway in a lossless codec.


> 
> > What about the mapping itself, it uses a rather simple rle coder. ive spend
> > most of the day today tuning that and failing to really beat that.
> > Using context from the previous image didnt work with the slideshow material
> > i have nor that one video i have. I tried using sign and exponent as context,
> > tried previous run, relations of runs, low bits and many more, but no or
> > insignificant improvments of yesterdays implementation was achieved.
> 
> I did mention this once before, but you should look into the Daala/Opus way
> of storing rawbits into a separate chunk at the end of the bitstream. It
> avoids polluting the range context with equiprobable bits.

This may fit into my LSB work but that is not float specific and its not
needed for float support.

I originally thought we would do floats through RAW storing LSB, and special
handling the "always" 0 and 1 bits. It makes sense but just remapping only
the used values to a compact list eliminates all the constant bits for free
and we never have more than 16bits per sample

We still can implement storing LSB raw, i have written 2 implementations
the first without range coder was this:
https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-October/334718.html

I didnt like it, but it seems more popular

Also given the raw bits are a known and constant length. It could make
sense to put them first if they are stored non interleaved.

but IIRC, the implementation for LSB refered in the link decorrelates
after LSB is split, it should give more compression doing it the
other way around.
That said the LSB patch still isnt needed for float support, it could
improve speed at the expense of compression for 16 and 32bit samples (float or other)

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Republics decline into democracies and democracies degenerate into
despotisms. -- Aristotle

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Leo Izen]

On 3/6/25 11:37 AM, Michael Niedermayer wrote:
> Hi
>>
>> I did mention this once before, but you should look into the Daala/Opus way
>> of storing rawbits into a separate chunk at the end of the bitstream. It
>> avoids polluting the range context with equiprobable bits.
> 
> 
> I didnt like it, but it seems more popular
> 

May also be worth considering the JPEG XL way, where each entropy-coded 
symbol is a "hybrid integer" of a token read from the entropy stream 
(ANS) followed by zero or more "raw bits" that are spliced in.

- Leo Izen (Traneptora)

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Jerome Martinez]

Le 06/03/2025 à 19:17, Leo Izen a écrit :
> May also be worth considering the JPEG XL way, where each 
> entropy-coded symbol is a "hybrid integer" of a token read from the 
> entropy stream (ANS) followed by zero or more "raw bits" that are 
> spliced in.

I would keep FFV1 away from ANS in the next 15 years due to
https://en.wikipedia.org/wiki/Asymmetric_numeral_systems#Patent_controversy
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1309 bytes --]

Hi Traneptora

On Thu, Mar 06, 2025 at 01:17:30PM -0500, Leo Izen wrote:
> On 3/6/25 11:37 AM, Michael Niedermayer wrote:
> > Hi
> > > 
> > > I did mention this once before, but you should look into the Daala/Opus way
> > > of storing rawbits into a separate chunk at the end of the bitstream. It
> > > avoids polluting the range context with equiprobable bits.
> > 
> > 
> > I didnt like it, but it seems more popular
> > 
> 
> May also be worth considering the JPEG XL way, where each entropy-coded
> symbol is a "hybrid integer" of a token read from the entropy stream (ANS)
> followed by zero or more "raw bits" that are spliced in.

Adding a 3rd entropy coder is outside the scope of implementing float support

If such a encoder makes sense, that should be a seperate thing.

And if you belive it performs better than what we have, submit a
patch and benchmarks. Please make sure only to use material free of
patent issues, aka things that predate patents in a way that invalidates
the patents

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Let us carefully observe those good qualities wherein our enemies excel us
and endeavor to excel them, by avoiding what is faulty, and imitating what
is excellent in them. -- Plutarch

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Jerome Martinez]

Le 06/03/2025 à 17:37, Michael Niedermayer a écrit :
> On Thu, Mar 06, 2025 at 03:14:49AM +0100, Lynne wrote:
>> On 06/03/2025 01:15, Michael Niedermayer wrote:
>>> Hi everyone
>>>
>>> Current FFv1 code with my patchset supports 16bit floats. The implementation
>>> is quite simple. Which is good
>>>
>>> I have tried improving compression, the gains are incremental, but its not
>>> large overall. For example 44% space gain on the remapping table is just
>>> 0.1% overall.
>>>
>>> I have few float samples. Its mainly one high quality slideshow of unrelated
>>> 16bit float images. These excercise a wide range fo things including negative
>>> color values.
>>> I think I have only one single (non slideshow) video of float 16 samples,
>>>
>>> It turns out the most efficient way to store floats that i found, is to remap
>>> them to integers. Not to store tham as sign, exponent, mantisse, i tried that
>>> for many hours.

I also tried some stuff but IMO the remap to integer is what is 
currently the best for keeping the spec simple & with a good 
compression, and a more complex way is not needed until we find 
something which really makes a difference in term of compression ratio.
So let's keep it simple, just a remap to int, and we already get all the 
advantage of e.g. the incoming GPU implementation.


>>> [...]
>>> What about the mapping itself, it uses a rather simple rle coder. ive spend
>>> most of the day today tuning that and failing to really beat that.
>>> Using context from the previous image didnt work with the slideshow material
>>> i have nor that one video i have. I tried using sign and exponent as context,
>>> tried previous run, relations of runs, low bits and many more, but no or
>>> insignificant improvments of yesterdays implementation was achieved.
>> I did mention this once before, but you should look into the Daala/Opus way
>> of storing rawbits into a separate chunk at the end of the bitstream. It
>> avoids polluting the range context with equiprobable bits.
> This may fit into my LSB work but that is not float specific and its not
> needed for float support.
>
> I originally thought we would do floats through RAW storing LSB, and special
> handling the "always" 0 and 1 bits. It makes sense but just remapping only
> the used values to a compact list eliminates all the constant bits for free
> and we never have more than 16bits per sample
>
> We still can implement storing LSB raw, i have written 2 implementations
> the first without range coder was this:
> https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-October/334718.html
>
> I didnt like it, but it seems more popular

I am still in favor of this simple and independent way to store the LSB 
bits, the other tentatives seem to only add complexity and performance 
issues while not providing enough gain.


>
> Also given the raw bits are a known and constant length. It could make
> sense to put them first if they are stored non interleaved.

Putting them last is better IMO, we can compute the byte offset with the 
constant length while storing some config in the slice header.
Actually, there are many possibilities, I suggest:
- permitting both MSB & LSB, because we may have MSB 0 and we would no 
store them (see below)
- after the count of bits not compressed, we add a flag indicating if we 
store them or not (meaning they are 0)
- permitting a configuration per stream (Configuration Record) or per 
slice (Slice Header); if a decision is to keep only one place for 
keeping it simple, it would be in the slice header.

The rational is that we may have:
- 0 padding for the LSB is when we compress integer from DCP, we have 
input with 16-bit TIFF but only 12 relevant bits, and we need to keep 
the bit depth of 16-bit if we want to say that we do it lossless and 
that we can reverse, especially because 99.999% of frames are with 4 
bits of padding but you may have a frame for whatever reason (often a 
bug, but if we do lossless we need to store buggy frames) which is not 
0, so a performant storage of 0-filled LSB while permitting to store the 
actual bits if not 0.
- 0 padding for the MSB is when we compress float mapped to integer and 
float range is from 0.0 to 1.0 only, sign & most of mantissa are always 
0 so we can store them as a flag, but we prepare the stream to have 
values out of range (<0 or > 1).
- and we don't know that before we start to compress each frame/slice

Would you accept something like:
- start of the Range Coder
- current slice header
- flag indicating the usage of bits not compressed
- if this flag is 1,
   - count of MSB not compressed
   - flag indicating if the MSB bitfield is not stored i.e. 0-padding
   - count of LSB not compressed
   - flag indicating if the LSB bitfield is not stored i.e. 0-padding
- current slice content
- current slice footer
- end of the Range Coder
- MSB bitfield if relevant
- LSB bitfield if relevant


> [...]
> That said the LSB patch still isnt needed for float support, it could
> improve speed at the expense of compression for 16 and 32bit samples (float or other)

Advantage of remap of float to integer is that the features (float 
support & bitfields) are independent but they complement each other.

Jérôme
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] [RFC] FFV1 float support

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 5590 bytes --]

Hi Jerome

On Thu, Mar 06, 2025 at 10:20:58PM +0100, Jerome Martinez wrote:
> Le 06/03/2025 à 17:37, Michael Niedermayer a écrit :
> > On Thu, Mar 06, 2025 at 03:14:49AM +0100, Lynne wrote:
> > > On 06/03/2025 01:15, Michael Niedermayer wrote:
> > > > Hi everyone
> > > > 
> > > > Current FFv1 code with my patchset supports 16bit floats. The implementation
> > > > is quite simple. Which is good
> > > > 
> > > > I have tried improving compression, the gains are incremental, but its not
> > > > large overall. For example 44% space gain on the remapping table is just
> > > > 0.1% overall.
> > > > 
> > > > I have few float samples. Its mainly one high quality slideshow of unrelated
> > > > 16bit float images. These excercise a wide range fo things including negative
> > > > color values.
> > > > I think I have only one single (non slideshow) video of float 16 samples,
> > > > 
> > > > It turns out the most efficient way to store floats that i found, is to remap
> > > > them to integers. Not to store tham as sign, exponent, mantisse, i tried that
> > > > for many hours.
> 
> I also tried some stuff but IMO the remap to integer is what is currently
> the best for keeping the spec simple & with a good compression, and a more
> complex way is not needed until we find something which really makes a
> difference in term of compression ratio.
> So let's keep it simple, just a remap to int, and we already get all the
> advantage of e.g. the incoming GPU implementation.

yes, i fully agree


> 
> 
> > > > [...]
> > > > What about the mapping itself, it uses a rather simple rle coder. ive spend
> > > > most of the day today tuning that and failing to really beat that.
> > > > Using context from the previous image didnt work with the slideshow material
> > > > i have nor that one video i have. I tried using sign and exponent as context,
> > > > tried previous run, relations of runs, low bits and many more, but no or
> > > > insignificant improvments of yesterdays implementation was achieved.
> > > I did mention this once before, but you should look into the Daala/Opus way
> > > of storing rawbits into a separate chunk at the end of the bitstream. It
> > > avoids polluting the range context with equiprobable bits.
> > This may fit into my LSB work but that is not float specific and its not
> > needed for float support.
> > 
> > I originally thought we would do floats through RAW storing LSB, and special
> > handling the "always" 0 and 1 bits. It makes sense but just remapping only
> > the used values to a compact list eliminates all the constant bits for free
> > and we never have more than 16bits per sample
> > 
> > We still can implement storing LSB raw, i have written 2 implementations
> > the first without range coder was this:
> > https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2024-October/334718.html
> > 
> > I didnt like it, but it seems more popular
> 
> I am still in favor of this simple and independent way to store the LSB
> bits, the other tentatives seem to only add complexity and performance
> issues while not providing enough gain.
> 
> 
> > 
> > Also given the raw bits are a known and constant length. It could make
> > sense to put them first if they are stored non interleaved.
> 
> Putting them last is better IMO, we can compute the byte offset with the
> constant length while storing some config in the slice header.
> Actually, there are many possibilities, I suggest:
> - permitting both MSB & LSB, because we may have MSB 0 and we would no store
> them (see below)
> - after the count of bits not compressed, we add a flag indicating if we
> store them or not (meaning they are 0)
> - permitting a configuration per stream (Configuration Record) or per slice
> (Slice Header); if a decision is to keep only one place for keeping it
> simple, it would be in the slice header.
> 
> The rational is that we may have:
> - 0 padding for the LSB is when we compress integer from DCP, we have input
> with 16-bit TIFF but only 12 relevant bits, and we need to keep the bit
> depth of 16-bit if we want to say that we do it lossless and that we can
> reverse, especially because 99.999% of frames are with 4 bits of padding but
> you may have a frame for whatever reason (often a bug, but if we do lossless
> we need to store buggy frames) which is not 0, so a performant storage of
> 0-filled LSB while permitting to store the actual bits if not 0.
> - 0 padding for the MSB is when we compress float mapped to integer and
> float range is from 0.0 to 1.0 only, sign & most of mantissa are always 0 so
> we can store them as a flag, but we prepare the stream to have values out of
> range (<0 or > 1).
> - and we don't know that before we start to compress each frame/slice

We have a generic remap table with the float code.
1. This is not really float specific, it could be used with integers
2. Its per slice, so each slice can have a different table or none

Any bits that are always 0 or 1 will be perfectly optimized out using
that table. No mattter if LSB, MSB or in the middle
even if every color is a multiple of 13 the remap table will still perfectly
remove that and produce a continous range that is one thirteenth

If you disagree, please explain how thats not already solving every
variant and more ?

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are too smart to engage in politics are punished by being
governed by those who are dumber. -- Plato 

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".