[FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1274 bytes --]

Hi everyone

I propose using 15k$ from SPI for funding sws cleanup work.
this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
So it really is more a small price for a good deed and not proper payment.
This of course is only available to competent developers. (exact rules or how thats determined
would still need to be decided unless its a clear case)
Also the exact outcome and goal would need to be discussed by the community and whoever
does the work.
But some goals would probably be to make sws
* pleasent to work with
* similar speed or faster
* proper multithreading
* proper full colorspace convertion not ignoring gamma, primaries, ...
* clean / understandable modular design (maybe everything can be a "Filter" inside sws
  that get build into a chain)

Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)

Above all, this is just my oppinion, the actual SPI funding also would need to
be approved by the community. This can happen after a specific volunteer comes forth
or before, whichever way the community prefers.

thx

--
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

In a rich man's house there is no place to spit but his face.
-- Diogenes of Sinope

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Vittorio Giovara]

On Fri, Oct 13, 2023 at 3:19 PM Michael Niedermayer <michael@niedermayer.cc>
wrote:

> Hi everyone
>
> I propose using 15k$ from SPI for funding sws cleanup work.
> this is substantially less than what people belive this needs (see IRC
> logs from yesterday or so)
> So it really is more a small price for a good deed and not proper payment.
> This of course is only available to competent developers. (exact rules or
> how thats determined
> would still need to be decided unless its a clear case)
> Also the exact outcome and goal would need to be discussed by the
> community and whoever
> does the work.
> But some goals would probably be to make sws
> * pleasent to work with
> * similar speed or faster
> * proper multithreading
> * proper full colorspace convertion not ignoring gamma, primaries, ...
> * clean / understandable modular design (maybe everything can be a
> "Filter" inside sws
>   that get build into a chain)
>
> Proper payment (50k$ maybe) would be too much in relation to what SPI has
> ATM (150k$)
>
> Above all, this is just my oppinion, the actual SPI funding also would
> need to
> be approved by the community. This can happen after a specific volunteer
> comes forth
> or before, whichever way the community prefers.
>

Hi Michael,
I don't mean to rain on your parade, but I think there are better image
libraries that are more accurate, faster, and can implement proper
tonemapping than sws -- zimg, and libplacebo are the prime examples. I
believe it would make more sense to integrate one of them in sws as a
backend (and fallback) so that the api doesn't change, or, if we absolutely
need no external deps, then write an entirely new library, but 15k$ work
for "cleanup" on a library noone consciously wants to use seems wasteful
IMO.
-- 
Vittorio
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Lynne]

Oct 13, 2023, 22:31 by vittorio.giovara@gmail.com:

> On Fri, Oct 13, 2023 at 3:19 PM Michael Niedermayer <michael@niedermayer.cc>
> wrote:
>
>> Hi everyone
>>
>> I propose using 15k$ from SPI for funding sws cleanup work.
>> this is substantially less than what people belive this needs (see IRC
>> logs from yesterday or so)
>> So it really is more a small price for a good deed and not proper payment.
>> This of course is only available to competent developers. (exact rules or
>> how thats determined
>> would still need to be decided unless its a clear case)
>> Also the exact outcome and goal would need to be discussed by the
>> community and whoever
>> does the work.
>> But some goals would probably be to make sws
>> * pleasent to work with
>> * similar speed or faster
>> * proper multithreading
>> * proper full colorspace convertion not ignoring gamma, primaries, ...
>> * clean / understandable modular design (maybe everything can be a
>> "Filter" inside sws
>>  that get build into a chain)
>>
>> Proper payment (50k$ maybe) would be too much in relation to what SPI has
>> ATM (150k$)
>>
>> Above all, this is just my oppinion, the actual SPI funding also would
>> need to
>> be approved by the community. This can happen after a specific volunteer
>> comes forth
>> or before, whichever way the community prefers.
>>
>
> Hi Michael,
> I don't mean to rain on your parade, but I think there are better image
> libraries that are more accurate, faster, and can implement proper
> tonemapping than sws -- zimg, and libplacebo are the prime examples. I
> believe it would make more sense to integrate one of them in sws as a
> backend (and fallback) so that the api doesn't change, or, if we absolutely
> need no external deps, then write an entirely new library, but 15k$ work
> for "cleanup" on a library noone consciously wants to use seems wasteful
> IMO.
>

I agree, I'd rather see this money being spent to give libplacebo
a software path and having swscale link against it. We can
talk about importing the project as a whole separately - I don't
think it's necessary, though.
zimg would need far too much work, IMO. But I don't mind being wrong here.

I wouldn't object to writing a new library or improving swscale
either - but I do think this has to be done with a plan we could agree on.
As I said yesterday, it is easy to make a mistake.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 2372 bytes --]

On Fri, Oct 13, 2023 at 04:30:56PM -0400, Vittorio Giovara wrote:
> On Fri, Oct 13, 2023 at 3:19 PM Michael Niedermayer <michael@niedermayer.cc>
> wrote:
> 
> > Hi everyone
> >
> > I propose using 15k$ from SPI for funding sws cleanup work.
> > this is substantially less than what people belive this needs (see IRC
> > logs from yesterday or so)
> > So it really is more a small price for a good deed and not proper payment.
> > This of course is only available to competent developers. (exact rules or
> > how thats determined
> > would still need to be decided unless its a clear case)
> > Also the exact outcome and goal would need to be discussed by the
> > community and whoever
> > does the work.
> > But some goals would probably be to make sws
> > * pleasent to work with
> > * similar speed or faster
> > * proper multithreading
> > * proper full colorspace convertion not ignoring gamma, primaries, ...
> > * clean / understandable modular design (maybe everything can be a
> > "Filter" inside sws
> >   that get build into a chain)
> >
> > Proper payment (50k$ maybe) would be too much in relation to what SPI has
> > ATM (150k$)
> >
> > Above all, this is just my oppinion, the actual SPI funding also would
> > need to
> > be approved by the community. This can happen after a specific volunteer
> > comes forth
> > or before, whichever way the community prefers.
> >
> 
> Hi Michael,

Hi Vittorio

[...]

> if we absolutely
> need no external deps,

Yes, you are correct


> then write an entirely new library,

The path and the end result are 2 different things
The end result that this funding is for is a clean libswscale
aka a clean "image convertion lib inside FFmpeg".

I think the path should be decided by the developer doing the work, if she wants to
extend libplacebo and import that, sure ok too with me

Only the contents of the C, H and asm files affects the people using the code
And i expeted some reused code, so i call it cleanup.

Things called "rewrites" also often failed. So its IMHO a sligtly cursed term,
but they are just terms, what the rsult is matters

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I know you won't believe me, but the highest form of Human Excellence is
to question oneself and others. -- Socrates

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Niklas Haas]

On Fri, 13 Oct 2023 21:19:34 +0200 Michael Niedermayer <michael@niedermayer.cc> wrote:
> Hi everyone
> 
> I propose using 15k$ from SPI for funding sws cleanup work.
> this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
> So it really is more a small price for a good deed and not proper payment.
> This of course is only available to competent developers. (exact rules or how thats determined
> would still need to be decided unless its a clear case)
> Also the exact outcome and goal would need to be discussed by the community and whoever
> does the work.
> But some goals would probably be to make sws
> * pleasent to work with
> * similar speed or faster
> * proper multithreading
> * proper full colorspace convertion not ignoring gamma, primaries, ...
> * clean / understandable modular design (maybe everything can be a "Filter" inside sws
>   that get build into a chain)
> 
> Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)
> 
> Above all, this is just my oppinion, the actual SPI funding also would need to
> be approved by the community. This can happen after a specific volunteer comes forth
> or before, whichever way the community prefers.
> 
> thx

My gut instinct is that the correct path forwards is to first create a
new API which can internally dispatch to swscale, libplacebo or zimg
based on what's compiled/available/preferred (e.g. using GPU filters for
hwframes, CPU filters for swframes, zimg (or vf_colorspace's primitives)
for colorspace conversions ...).

Much of the pain points of my own recent experience with swscale
revolves around libswscale's very low level API, complete lack of
understanding of most AVFrame metadata, and relatively complex
configuration requirements. (vf_scale's comments even acknowledge that
libswscale should handle this stuff internally, so users just need to
point an AVFrame at both ends and let it do the right thing
automatically)

The second pain point of extending libswscale itself is that the
high-level configuration and low-level "scaling" primitives are deeply
entangled, reconfigured in various places, and hard to touch without
fear of breaking edge cases.

A new API would solve both problems. It would allow us to write new,
AVFrame-based, high-level "business logic", which could continue to call
into the low-level swscale implementation details and kernels for the
time being, and also tap into GPU filters (a la libplacebo) where
possible.

As time moves on we could replace those underlying primitives by cleaned
up rewrites where necessary, until the dependency on libswscale itself
is null.

Simultaneously, vf_scale can be rewritten on top of this API, which
would allow e.g. auto-scale filters to start doing scaling on GPU when
conversion between two hwformats is needed.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[James Almer]

On 10/13/2023 7:34 PM, Niklas Haas wrote:
> On Fri, 13 Oct 2023 21:19:34 +0200 Michael Niedermayer <michael@niedermayer.cc> wrote:
>> Hi everyone
>>
>> I propose using 15k$ from SPI for funding sws cleanup work.
>> this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
>> So it really is more a small price for a good deed and not proper payment.
>> This of course is only available to competent developers. (exact rules or how thats determined
>> would still need to be decided unless its a clear case)
>> Also the exact outcome and goal would need to be discussed by the community and whoever
>> does the work.
>> But some goals would probably be to make sws
>> * pleasent to work with
>> * similar speed or faster
>> * proper multithreading
>> * proper full colorspace convertion not ignoring gamma, primaries, ...
>> * clean / understandable modular design (maybe everything can be a "Filter" inside sws
>>    that get build into a chain)
>>
>> Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)
>>
>> Above all, this is just my oppinion, the actual SPI funding also would need to
>> be approved by the community. This can happen after a specific volunteer comes forth
>> or before, whichever way the community prefers.
>>
>> thx
> 
> My gut instinct is that the correct path forwards is to first create a
> new API which can internally dispatch to swscale, libplacebo or zimg
> based on what's compiled/available/preferred (e.g. using GPU filters for
> hwframes, CPU filters for swframes, zimg (or vf_colorspace's primitives)
> for colorspace conversions ...).
> 
> Much of the pain points of my own recent experience with swscale
> revolves around libswscale's very low level API, complete lack of
> understanding of most AVFrame metadata, and relatively complex

Anton wrote and pushed an AVFrame based API. It can surely be 
improved/extended to use AVFrame metadata.

> configuration requirements. (vf_scale's comments even acknowledge that
> libswscale should handle this stuff internally, so users just need to
> point an AVFrame at both ends and let it do the right thing
> automatically)
> 
> The second pain point of extending libswscale itself is that the
> high-level configuration and low-level "scaling" primitives are deeply
> entangled, reconfigured in various places, and hard to touch without
> fear of breaking edge cases.
> 
> A new API would solve both problems. It would allow us to write new,
> AVFrame-based, high-level "business logic", which could continue to call
> into the low-level swscale implementation details and kernels for the
> time being, and also tap into GPU filters (a la libplacebo) where
> possible.
> 
> As time moves on we could replace those underlying primitives by cleaned
> up rewrites where necessary, until the dependency on libswscale itself
> is null.
> 
> Simultaneously, vf_scale can be rewritten on top of this API, which
> would allow e.g. auto-scale filters to start doing scaling on GPU when
> conversion between two hwformats is needed.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Niklas Haas]

On Fri, 13 Oct 2023 19:42:44 -0300 James Almer <jamrial@gmail.com> wrote:
> Anton wrote and pushed an AVFrame based API. It can surely be 
> improved/extended to use AVFrame metadata.

Yes, this is actually a good idea. This API endpoint already has the
"correct" signature, so we could definitely re-use it (and SwsContext)
instead of introducing a new header file.

But to be clear, even with this sws_scale_frame API, you currently still
need to configure the SwsContext up-front - and that is the source of
problems IMO. I think the entire family of
sws_getContext/sws_init_context/sws_setColorspaceDetails are buggy,
unmaintainable nightmares.

Starting from scratch, this context would not exist at all. All required
metadata is available in the AVFrame itself, and it's trivial to
invalidate the internal state when something changes. The function
itself should be effectively stateless, with the SwsContext serving as a
mere cache.

So maybe a good path forwards is:

1. Make sws_scale_frame explicitly ignore the configured colorspace
   details in favor of AVFrame metadata.
2. Allow using sws_scale_frame with an SwsContext that has not been
   initialized, but merely allocated.
3. Deprecate sws_scale and the old configuration API

Then SwsContext would be serving double-duty between being the
configuration struct for the legacy API, while also being a cache for
the new API, until eventually being just the latter.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Vittorio Giovara]

On Fri, Oct 13, 2023 at 6:55 PM Niklas Haas <ffmpeg@haasn.xyz> wrote:

> On Fri, 13 Oct 2023 19:42:44 -0300 James Almer <jamrial@gmail.com> wrote:
> > Anton wrote and pushed an AVFrame based API. It can surely be
> > improved/extended to use AVFrame metadata.
>
> Yes, this is actually a good idea. This API endpoint already has the
> "correct" signature, so we could definitely re-use it (and SwsContext)
> instead of introducing a new header file.
>
> But to be clear, even with this sws_scale_frame API, you currently still
> need to configure the SwsContext up-front - and that is the source of
> problems IMO. I think the entire family of
> sws_getContext/sws_init_context/sws_setColorspaceDetails are buggy,
> unmaintainable nightmares.
>
> Starting from scratch, this context would not exist at all. All required
> metadata is available in the AVFrame itself, and it's trivial to
> invalidate the internal state when something changes. The function
> itself should be effectively stateless, with the SwsContext serving as a
> mere cache.
>
> So maybe a good path forwards is:
>
> 1. Make sws_scale_frame explicitly ignore the configured colorspace
>    details in favor of AVFrame metadata.
> 2. Allow using sws_scale_frame with an SwsContext that has not been
>    initialized, but merely allocated.
> 3. Deprecate sws_scale and the old configuration API
>
> Then SwsContext would be serving double-duty between being the
> configuration struct for the legacy API, while also being a cache for
> the new API, until eventually being just the latter.
>

TBF this is in part why i was suggesting a new library - I feel like sws is
affected by bad brading because of these caching issues and imprecise
conversion, and a new clean api in a new library would make a lot of sense
in my opinion.
-- 
Vittorio
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Cosmin Stejerean via ffmpeg-devel]

> On Oct 13, 2023, at 4:00 PM, Vittorio Giovara <vittorio.giovara@gmail.com> wrote:
> 
> TBF this is in part why i was suggesting a new library - I feel like sws is
> affected by bad brading because of these caching issues and imprecise
> conversion, and a new clean api in a new library would make a lot of sense
> in my opinion.

I think the branding issue would solve itself in short order if the actual implementation of swscale started to be good. My concern with adding a new library is that we'd end up in a situation where we have both swscale and a new library side by side for some extended period of time. 

By comparison adding cleaner APIs to swscale and then slowly strangling the old APIs (along the lines of Niklas' proposal) would allow for a more gradual transition that has a higher likelihood of success compared to a full rewrite IMO.

- Cosmin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Kieran Kunhya]

On Sat, 14 Oct 2023 at 00:17, Cosmin Stejerean via ffmpeg-devel <
ffmpeg-devel@ffmpeg.org> wrote:

>
>
> > On Oct 13, 2023, at 4:00 PM, Vittorio Giovara <
> vittorio.giovara@gmail.com> wrote:
> >
> > TBF this is in part why i was suggesting a new library - I feel like sws
> is
> > affected by bad brading because of these caching issues and imprecise
> > conversion, and a new clean api in a new library would make a lot of
> sense
> > in my opinion.
>
> I think the branding issue would solve itself in short order if the actual
> implementation of swscale started to be good. My concern with adding a new
> library is that we'd end up in a situation where we have both swscale and a
> new library side by side for some extended period of time.
>
> By comparison adding cleaner APIs to swscale and then slowly strangling
> the old APIs (along the lines of Niklas' proposal) would allow for a more
> gradual transition that has a higher likelihood of success compared to a
> full rewrite IMO.
>

The issue is not the API, the issue is that swscale is astonishingly
complex and difficult to understand internally, there are lots of different
codepaths and randomly you'll end up with a buggy or slow one and have no
idea how to fix it.

It's probably easier to start from scratch than to try and understand and
then fix swscale (years of work).

Regards,
Kieran Kunhya
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3142 bytes --]

On Sat, Oct 14, 2023 at 03:19:49PM +0100, Kieran Kunhya wrote:
> On Sat, 14 Oct 2023 at 00:17, Cosmin Stejerean via ffmpeg-devel <
> ffmpeg-devel@ffmpeg.org> wrote:
> 
> >
> >
> > > On Oct 13, 2023, at 4:00 PM, Vittorio Giovara <
> > vittorio.giovara@gmail.com> wrote:
> > >
> > > TBF this is in part why i was suggesting a new library - I feel like sws
> > is
> > > affected by bad brading because of these caching issues and imprecise
> > > conversion, and a new clean api in a new library would make a lot of
> > sense
> > > in my opinion.
> >
> > I think the branding issue would solve itself in short order if the actual
> > implementation of swscale started to be good. My concern with adding a new
> > library is that we'd end up in a situation where we have both swscale and a
> > new library side by side for some extended period of time.
> >
> > By comparison adding cleaner APIs to swscale and then slowly strangling
> > the old APIs (along the lines of Niklas' proposal) would allow for a more
> > gradual transition that has a higher likelihood of success compared to a
> > full rewrite IMO.
> >
> 
> The issue is not the API, the issue is that swscale is astonishingly
> complex and difficult to understand internally, there are lots of different
> codepaths

> and randomly you'll end up with a buggy or slow one

randomly ?
code in general doesnt give you randomly something very different.

So, why do i complain? because swscale has real issues and needs
to be improved. And these comments point in the wrong direction


> and have no
> idea how to fix it.

If you dont know how to fix it yourself, sending me a bug report is
probably a good start.


> 
> It's probably easier to start from scratch than to try and understand and
> then fix swscale (years of work).

Well there are 2 further aspects with that.

The first one is bluntly put. If you dont understand the old code, then
you probably are not qualified to write better code.
People tend not to successfully improve things they dont understand.

The 2nd issue is, ATM, i maintain swscale. If iam involved in the new
effort and understand it either because of that or because it has some
similarity then i can continue to maintain swscale. If its totally
different and i was totally not involded then i also will not maintain
it obviously.
This is something to be especially aware of in case the cleanup/new
code would be done by someone who comes, does it and leaves. you
could end up with nicer code thats then unmaintained.

PS: whats the real issue with sws ?
it evolved out of a piece yuv->rgb converter from a video player.
It evolved from that and stuff was added into it.
This is a similar situation to why ffmpeg.c needed cleanup

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Dictatorship: All citizens are under surveillance, all their steps and
actions recorded, for the politicians to enforce control.
Democracy: All politicians are under surveillance, all their steps and
actions recorded, for the citizens to enforce control.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Niklas Haas]

On Sat, 14 Oct 2023 19:00:36 +0200 Michael Niedermayer <michael@niedermayer.cc> wrote:
> Well there are 2 further aspects with that.
> 
> The first one is bluntly put. If you dont understand the old code, then
> you probably are not qualified to write better code.
> People tend not to successfully improve things they dont understand.

I have a deep understanding of colorspaces and the necessary conversion
steps between them, and am also in a good position to integrate
libplacebo as a possible backend.

However, I do not have a good understanding of CPU/SIMD code, nor the
various swscale internals, beyond the cursory investigation I needed for
some recent swscale bugs I encountered. So I'll definitely need some
help along the way to fully understand those swscale internals.

> The 2nd issue is, ATM, i maintain swscale. If iam involved in the new
> effort and understand it either because of that or because it has some
> similarity then i can continue to maintain swscale. If its totally
> different and i was totally not involded then i also will not maintain
> it obviously.

I think it would be possible to join forces to the extent needed to
arrive at a satisfactory result. At the very least, I have very limited
experience working with "irregular" packed formats.

Obviously, my intent is not to blanket discard the swscale internals. It
has years and years of optimized kernels for various platforms just
lying around, wanting to be used. Hence my proposal of redesigning the
high-level logic, rather than the low-level details.

> This is something to be especially aware of in case the cleanup/new
> code would be done by someone who comes, does it and leaves. you
> could end up with nicer code thats then unmaintained.
> 
> PS: whats the real issue with sws ?
> it evolved out of a piece yuv->rgb converter from a video player.
> It evolved from that and stuff was added into it.
> This is a similar situation to why ffmpeg.c needed cleanup

Yes, it amounts to the usual disentangling of various special cases and
branches into one top-down control flow that knows about all of these
special cases and fast/slow paths to begin with.

My goal is to arrive at a place where we have one single code flow that
looks something like:

1. Settle the complete descriptions of the source and destination format/csp
2. Establish a list of operations to get from A to B, taking into
   account user settings
3. Determine and dispatch the best available functions for each operation

With the necessary code separation and/or layers of abstraction in place
to make this design manageable. In particular, steps 1 and 2 should be
expanded to include things like conversion between primaries, conversion
between HDR and SDR, conversion between YUV/RGB, and so on.

In particular, I also want to eventually add the ability to plug "Apply
a 3DLUT" in as a possible operation type for colorspace conversion,
probably by sharing the code that is already written for vf_lut3d.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 5024 bytes --]

Hi

On Sat, Oct 14, 2023 at 07:24:51PM +0200, Niklas Haas wrote:
> On Sat, 14 Oct 2023 19:00:36 +0200 Michael Niedermayer <michael@niedermayer.cc> wrote:
> > Well there are 2 further aspects with that.
> > 
> > The first one is bluntly put. If you dont understand the old code, then
> > you probably are not qualified to write better code.
> > People tend not to successfully improve things they dont understand.
> 
> I have a deep understanding of colorspaces and the necessary conversion
> steps between them, and am also in a good position to integrate
> libplacebo as a possible backend.
> 
> However, I do not have a good understanding of CPU/SIMD code, nor the
> various swscale internals, beyond the cursory investigation I needed for
> some recent swscale bugs I encountered. So I'll definitely need some
> help along the way to fully understand those swscale internals.

iam happy to help., as far as i remember (and as far as i know) things.
Also ronald probably knows quite a bit from the older parts.
various people also added bits over the years, they can probably help
within these bits. And we had some GSoC contributions from Pedro Arthur,
he certainly is worth a try to ask if you have a question related to his
code.


> 
> > The 2nd issue is, ATM, i maintain swscale. If iam involved in the new
> > effort and understand it either because of that or because it has some
> > similarity then i can continue to maintain swscale. If its totally
> > different and i was totally not involded then i also will not maintain
> > it obviously.
> 
> I think it would be possible to join forces to the extent needed to
> arrive at a satisfactory result. At the very least, I have very limited
> experience working with "irregular" packed formats.
> 
> Obviously, my intent is not to blanket discard the swscale internals. It
> has years and years of optimized kernels for various platforms just
> lying around, wanting to be used. Hence my proposal of redesigning the
> high-level logic, rather than the low-level details.

yes, i expect that my role in this will probably be limited to awnsering
questions, reviewing code and maybe running some tests.
Not sure i will have the motivation to do much heavy restructuring of things
It might be that i suddenly see something i want to work on ;) but besides this
iam happy to leave it to you and others


> 
> > This is something to be especially aware of in case the cleanup/new
> > code would be done by someone who comes, does it and leaves. you
> > could end up with nicer code thats then unmaintained.
> > 
> > PS: whats the real issue with sws ?
> > it evolved out of a piece yuv->rgb converter from a video player.
> > It evolved from that and stuff was added into it.
> > This is a similar situation to why ffmpeg.c needed cleanup
> 
> Yes, it amounts to the usual disentangling of various special cases and
> branches into one top-down control flow that knows about all of these
> special cases and fast/slow paths to begin with.
> 
> My goal is to arrive at a place where we have one single code flow that
> looks something like:
> 
> 1. Settle the complete descriptions of the source and destination format/csp
> 2. Establish a list of operations to get from A to B, taking into
>    account user settings
> 3. Determine and dispatch the best available functions for each operation

I think this is the obvious logic way to design it, yes


> 
> With the necessary code separation and/or layers of abstraction in place
> to make this design manageable. In particular, steps 1 and 2 should be
> expanded to include things like conversion between primaries, conversion
> between HDR and SDR, conversion between YUV/RGB, and so on.

One thing that will be needed is to allow alternatives for seperation
for example an optimized codepath might do vertical scaling and convertion
to BGR16 in one piece. While both operations also should probably be available
individually.
This adds complexity but such combined operations are used in the current
codepathes so if the existing optimizations are supported then the current
way things are broken up into steps need to be supported


> 
> In particular, I also want to eventually add the ability to plug "Apply
> a 3DLUT" in as a possible operation type for colorspace conversion,
> probably by sharing the code that is already written for vf_lut3d.

Speaking of this, the horizontal and vertical scaling is basically a
FIR filter being applied. Which is also why swscale allows the user
to provide such filters, they can be applied by just using the already
needed codepathes with no extra complexity

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Breaking DRM is a little like attempting to break through a door even
though the window is wide open and the only thing in the house is a bunch
of things you dont want and which you would get tomorrow for free anyway

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Kieran Kunhya]

On Sat, 14 Oct 2023, 18:00 Michael Niedermayer, <michael@niedermayer.cc>
wrote:

> On Sat, Oct 14, 2023 at 03:19:49PM +0100, Kieran Kunhya wrote:
> > On Sat, 14 Oct 2023 at 00:17, Cosmin Stejerean via ffmpeg-devel <
> > ffmpeg-devel@ffmpeg.org> wrote:
> >
> > >
> > >
> > > > On Oct 13, 2023, at 4:00 PM, Vittorio Giovara <
> > > vittorio.giovara@gmail.com> wrote:
> > > >
> > > > TBF this is in part why i was suggesting a new library - I feel like
> sws
> > > is
> > > > affected by bad brading because of these caching issues and imprecise
> > > > conversion, and a new clean api in a new library would make a lot of
> > > sense
> > > > in my opinion.
> > >
> > > I think the branding issue would solve itself in short order if the
> actual
> > > implementation of swscale started to be good. My concern with adding a
> new
> > > library is that we'd end up in a situation where we have both swscale
> and a
> > > new library side by side for some extended period of time.
> > >
> > > By comparison adding cleaner APIs to swscale and then slowly strangling
> > > the old APIs (along the lines of Niklas' proposal) would allow for a
> more
> > > gradual transition that has a higher likelihood of success compared to
> a
> > > full rewrite IMO.
> > >
> >
> > The issue is not the API, the issue is that swscale is astonishingly
> > complex and difficult to understand internally, there are lots of
> different
> > codepaths
>
> > and randomly you'll end up with a buggy or slow one
>
> randomly ?
> code in general doesnt give you randomly something very different.
>

Come on, there's no need to be facetious. Change the PIX_FMT to a the same
sampling but a different packing and you a get totally different codepath,
sometimes just decides to use C. Likewise with any of the lightly
documented flags, you can have drastic changes in speed and quality unless
you pre-test all the code paths.


> So, why do i complain? because swscale has real issues and needs
> to be improved. And these comments point in the wrong direction
>
>
> > and have no
> > idea how to fix it.
>
> If you dont know how to fix it yourself, sending me a bug report is
> probably a good start.
>

Covered in here: https://trac.ffmpeg.org/wiki/swscale

Yuv422p10 to yuv420p10 has forced and useless and CPU costly scaling of the
luma channel with 32 bit intermediates last time I looked. All to be
shifted back to the original values.



>
>
> >
> > It's probably easier to start from scratch than to try and understand and
> > then fix swscale (years of work).
>
> Well there are 2 further aspects with that.
>
> The first one is bluntly put. If you dont understand the old code, then
> you probably are not qualified to write better code.
> People tend not to successfully improve things they dont understand.
>

I'm pretty sure you don't need to understand self-modifying code to write a
scaler.

The rest is covered here:
https://trac.ffmpeg.org/wiki/swscale


> The 2nd issue is, ATM, i maintain swscale. If iam involved in the new
> effort and understand it either because of that or because it has some
> similarity then i can continue to maintain swscale. If its totally
> different and i was totally not involded then i also will not maintain
> it obviously.
> This is something to be especially aware of in case the cleanup/new
> code would be done by someone who comes, does it and leaves. you
> could end up with nicer code thats then unmaintained.
>

Nicer code can be understood by more than one person.

Let's be honest you would block any attempt to even start removing cruft in
swscale like mmx, self-modifying code etc.


> PS: whats the real issue with sws ?
> it evolved out of a piece yuv->rgb converter from a video player.
> It evolved from that and stuff was added into it.
> This is a similar situation to why ffmpeg.c needed cleanup
>

Hmmm, building a simple thing for something and then "stuff being added",
that sounds like the arguments a few of us have been making in another
thread.

Regards,
Kieran Kunhya



> thx
>
> [...]
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> Dictatorship: All citizens are under surveillance, all their steps and
> actions recorded, for the politicians to enforce control.
> Democracy: All politicians are under surveillance, all their steps and
> actions recorded, for the citizens to enforce control.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Vittorio Giovara]

On Sat, Oct 14, 2023 at 1:00 PM Michael Niedermayer <michael@niedermayer.cc>
wrote:

>
> PS: whats the real issue with sws ?
> it evolved out of a piece yuv->rgb converter from a video player.
> It evolved from that and stuff was added into it.
> This is a similar situation to why ffmpeg.c needed cleanup
>

I'll give you two real issues:
* It's based on an archaic design that doesn't let people contribute to it
easily, and thus it's not very extensible. New code paths *can* be added,
but it's very difficult and it can lead easily astray, with often
unpredictable conversions and bugs subtly introduced.
* It's prefixed with sw- while the rest of libraries are prefixed with av-
(and to my understanding there is no real reason behind this) (yes i'm
aware of lswr).

I think the compromise of having a backend-based API backed by different
libraries is the way forward, no cleanups, no rewrites.
-- 
Vittorio
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Anton Khirnov]

Quoting Kieran Kunhya (2023-10-14 16:19:49)
> On Sat, 14 Oct 2023 at 00:17, Cosmin Stejerean via ffmpeg-devel <
> ffmpeg-devel@ffmpeg.org> wrote:
> 
> >
> >
> > > On Oct 13, 2023, at 4:00 PM, Vittorio Giovara <
> > vittorio.giovara@gmail.com> wrote:
> > >
> > > TBF this is in part why i was suggesting a new library - I feel like sws
> > is
> > > affected by bad brading because of these caching issues and imprecise
> > > conversion, and a new clean api in a new library would make a lot of
> > sense
> > > in my opinion.
> >
> > I think the branding issue would solve itself in short order if the actual
> > implementation of swscale started to be good. My concern with adding a new
> > library is that we'd end up in a situation where we have both swscale and a
> > new library side by side for some extended period of time.
> >
> > By comparison adding cleaner APIs to swscale and then slowly strangling
> > the old APIs (along the lines of Niklas' proposal) would allow for a more
> > gradual transition that has a higher likelihood of success compared to a
> > full rewrite IMO.
> >
> 
> The issue is not the API, the issue is that swscale is astonishingly
> complex and difficult to understand internally, there are lots of different
> codepaths and randomly you'll end up with a buggy or slow one and have no
> idea how to fix it.
> 
> It's probably easier to start from scratch than to try and understand and
> then fix swscale (years of work).

I've seen more than one attempt to do that over the years, all failed.
While I do agree that sws code is atrociously bad, I think that
* an attempt to reinvent it from scratch is quite likely to fail and
  produce nothing of value
* sws can be incrementally improved

-- 
Anton Khirnov
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1261 bytes --]

Hi

On Fri, Oct 13, 2023 at 11:16:50PM +0000, Cosmin Stejerean via ffmpeg-devel wrote:
> 
> 
> > On Oct 13, 2023, at 4:00 PM, Vittorio Giovara <vittorio.giovara@gmail.com> wrote:
> > 
> > TBF this is in part why i was suggesting a new library - I feel like sws is
> > affected by bad brading because of these caching issues and imprecise
> > conversion, and a new clean api in a new library would make a lot of sense
> > in my opinion.
> 
> I think the branding issue would solve itself in short order if the actual implementation of swscale started to be good. My concern with adding a new library is that we'd end up in a situation where we have both swscale and a new library side by side for some extended period of time. 
> 
> By comparison adding cleaner APIs to swscale and then slowly strangling the old APIs (along the lines of Niklas' proposal) would allow for a more gradual transition that has a higher likelihood of success compared to a full rewrite IMO.

+1

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Whats the most studid thing your enemy could do ? Blow himself up
Whats the most studid thing you could do ? Give up your rights and
freedom because your enemy blew himself up.


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Stefano Sabatini]

On date Friday 2023-10-13 21:19:34 +0200, Michael Niedermayer wrote:
> Hi everyone
> 
> I propose using 15k$ from SPI for funding sws cleanup work.
> this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
> So it really is more a small price for a good deed and not proper payment.
> This of course is only available to competent developers. (exact rules or how thats determined
> would still need to be decided unless its a clear case)
> Also the exact outcome and goal would need to be discussed by the community and whoever
> does the work.
> But some goals would probably be to make sws
> * pleasent to work with
> * similar speed or faster
> * proper multithreading
> * proper full colorspace convertion not ignoring gamma, primaries, ...
> * clean / understandable modular design (maybe everything can be a "Filter" inside sws
>   that get build into a chain)
> 
> Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)
> 
> Above all, this is just my oppinion, the actual SPI funding also would need to
> be approved by the community. This can happen after a specific volunteer comes forth
> or before, whichever way the community prefers.

Leaving apart the technical details about the implementation, this
should be feasible within the SPI framework (although this would
involve some paperwork and delays due to that).

It would be useful at this point to define the process to accept the
proposal and potential candidates. We have a technical committee which
might take the lead on that and probably have the last word on it,
since "approved by the community" is a bit vague and there is the risk
that there will be never an approval "from the community" because of
diverging views, or that we get stuck at the design level.

As a start, probably there should be a design doc somewhere, discussed
by the community and finally approved (by the technical committee??)
before we present the request and candidate to SPI. In fact probably
the design doc is the first thing a candidate might need to work on.

Also I'd avoid terms such as "rewrite" or "cleanup" since they have
bad connotations, maybe let's call it "review" or "refinement".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 3366 bytes --]

On Sat, Oct 14, 2023 at 07:53:04PM +0200, Stefano Sabatini wrote:
> On date Friday 2023-10-13 21:19:34 +0200, Michael Niedermayer wrote:
> > Hi everyone
> > 
> > I propose using 15k$ from SPI for funding sws cleanup work.
> > this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
> > So it really is more a small price for a good deed and not proper payment.
> > This of course is only available to competent developers. (exact rules or how thats determined
> > would still need to be decided unless its a clear case)
> > Also the exact outcome and goal would need to be discussed by the community and whoever
> > does the work.
> > But some goals would probably be to make sws
> > * pleasent to work with
> > * similar speed or faster
> > * proper multithreading
> > * proper full colorspace convertion not ignoring gamma, primaries, ...
> > * clean / understandable modular design (maybe everything can be a "Filter" inside sws
> >   that get build into a chain)
> > 
> > Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)
> > 
> > Above all, this is just my oppinion, the actual SPI funding also would need to
> > be approved by the community. This can happen after a specific volunteer comes forth
> > or before, whichever way the community prefers.
> 
> Leaving apart the technical details about the implementation, this
> should be feasible within the SPI framework (although this would
> involve some paperwork and delays due to that).
> 
> It would be useful at this point to define the process to accept the
> proposal and potential candidates. We have a technical committee which
> might take the lead on that and probably have the last word on it,
> since "approved by the community" is a bit vague and there is the risk
> that there will be never an approval "from the community" because of
> diverging views, or that we get stuck at the design level.

I think there are several shades of this

The community might simply have a consensus that X should be funded.
We achieved this both for traval and hw in all or nearly all cases.
And quite plausibly we will achieve this too for other cases

Hypothetically the community might have a consensus some work should
be funded but not agree on technical details.
Here honestly i think the developer doing the work should be the main
decission maker. She is the one doing the work, knowing the code best.
And most likely its one of the FFmpeg team doing the work.

The technical committee could be used if reviewer and author cannot agree.

I think we should stir clear of a "Planned economy" where some group thats
"distanced" from the actual work decides on fine details. These did not work
very well in real economies and i doubt it would work well for technical
design.
Furthermore the person doing specific work really should be, when she does
the work be the one with the best understanding of the code (otherwise we
"hired" the wrong person) so to me it makes sense she also would make
decissions on technical details. At the same time she also takes
responsibility for teh decission ...

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No great genius has ever existed without some touch of madness. -- Aristotle

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Cosmin Stejerean via ffmpeg-devel]

> On Oct 17, 2023, at 7:36 AM, Michael Niedermayer <michael@niedermayer.cc> wrote:
> 
> On Sat, Oct 14, 2023 at 07:53:04PM +0200, Stefano Sabatini wrote:
>> 
>> It would be useful at this point to define the process to accept the
>> proposal and potential candidates. We have a technical committee which
>> might take the lead on that and probably have the last word on it,
>> since "approved by the community" is a bit vague and there is the risk
>> that there will be never an approval "from the community" because of
>> diverging views, or that we get stuck at the design level.
> 
> I think there are several shades of this
> 
> The community might simply have a consensus that X should be funded.
> We achieved this both for traval and hw in all or nearly all cases.
> And quite plausibly we will achieve this too for other cases
> 
> Hypothetically the community might have a consensus some work should
> be funded but not agree on technical details.
> Here honestly i think the developer doing the work should be the main
> decission maker. She is the one doing the work, knowing the code best.
> And most likely its one of the FFmpeg team doing the work.

I think this makes sense for cases where there is easily reachable consensus. What happens when we can't easily reach consensus? For example it doesn't seem like we have consensus on funding improvements to swscale (compared to integrating a 3rd party library). Does that mean that work cannot get funded through SPI? 

This is where I think using the TC to make a decision where the community at large cannot reach consensus might be useful. It doesn't need to decide the fine technical points of how the work is done, but it can provide a useful mechanism to disagree and commit about whether the work should be done at all and provide the broad strokes (like improve swscale vs write a brand new library vs integrate some third party one).

- Cosmin



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Stefano Sabatini]

On date Tuesday 2023-10-17 16:58:41 +0000, ffmpeg-devel Mailing List wrote:
> 
> 
> > On Oct 17, 2023, at 7:36 AM, Michael Niedermayer <michael@niedermayer.cc> wrote:
> > 
> > On Sat, Oct 14, 2023 at 07:53:04PM +0200, Stefano Sabatini wrote:
> >> 
> >> It would be useful at this point to define the process to accept the
> >> proposal and potential candidates. We have a technical committee which
> >> might take the lead on that and probably have the last word on it,
> >> since "approved by the community" is a bit vague and there is the risk
> >> that there will be never an approval "from the community" because of
> >> diverging views, or that we get stuck at the design level.
> > 
> > I think there are several shades of this
> > 
> > The community might simply have a consensus that X should be funded.
> > We achieved this both for traval and hw in all or nearly all cases.
> > And quite plausibly we will achieve this too for other cases
> > 
> > Hypothetically the community might have a consensus some work should
> > be funded but not agree on technical details.
> > Here honestly i think the developer doing the work should be the main
> > decission maker. She is the one doing the work, knowing the code best.
> > And most likely its one of the FFmpeg team doing the work.
> 

> I think this makes sense for cases where there is easily reachable
> consensus. What happens when we can't easily reach consensus? For
> example it doesn't seem like we have consensus on funding
> improvements to swscale (compared to integrating a 3rd party
> library). Does that mean that work cannot get funded through SPI?
> 

> This is where I think using the TC to make a decision where the
> community at large cannot reach consensus might be useful. It
> doesn't need to decide the fine technical points of how the work is
> done, but it can provide a useful mechanism to disagree and commit
> about whether the work should be done at all and provide the broad
> strokes (like improve swscale vs write a brand new library vs
> integrate some third party one).

+1

And we should try to prevent both later complaints ("it was decided
against my will") or block development because a single or a minority
of developers is against it.

OTOH voting/decision making should only be seeked out in case there is
some disagreement which cannot be resolved during the preliminary
discussion on list/chat, or in case there is more than one candidate
for the task.

I cannot comment about what exact party should be called out (TC vs
GA).
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[James Almer]

On 10/14/2023 2:53 PM, Stefano Sabatini wrote:
> On date Friday 2023-10-13 21:19:34 +0200, Michael Niedermayer wrote:
>> Hi everyone
>>
>> I propose using 15k$ from SPI for funding sws cleanup work.
>> this is substantially less than what people belive this needs (see IRC logs from yesterday or so)
>> So it really is more a small price for a good deed and not proper payment.
>> This of course is only available to competent developers. (exact rules or how thats determined
>> would still need to be decided unless its a clear case)
>> Also the exact outcome and goal would need to be discussed by the community and whoever
>> does the work.
>> But some goals would probably be to make sws
>> * pleasent to work with
>> * similar speed or faster
>> * proper multithreading
>> * proper full colorspace convertion not ignoring gamma, primaries, ...
>> * clean / understandable modular design (maybe everything can be a "Filter" inside sws
>>    that get build into a chain)
>>
>> Proper payment (50k$ maybe) would be too much in relation to what SPI has ATM (150k$)
>>
>> Above all, this is just my oppinion, the actual SPI funding also would need to
>> be approved by the community. This can happen after a specific volunteer comes forth
>> or before, whichever way the community prefers.
> 
> Leaving apart the technical details about the implementation, this
> should be feasible within the SPI framework (although this would
> involve some paperwork and delays due to that).
> 
> It would be useful at this point to define the process to accept the
> proposal and potential candidates. We have a technical committee which
> might take the lead on that and probably have the last word on it,
> since "approved by the community" is a bit vague and there is the risk
> that there will be never an approval "from the community" because of
> diverging views, or that we get stuck at the design level.

"Approved by the community" would probably mean a vote by the GA if 
there's no clear consensus.

The technical committee should be left as a last resort, as usual, when 
there are two completely opposing views and no way to convince either 
party of the alternative.

> 
> As a start, probably there should be a design doc somewhere, discussed
> by the community and finally approved (by the technical committee??)
> before we present the request and candidate to SPI. In fact probably
> the design doc is the first thing a candidate might need to work on.
> 
> Also I'd avoid terms such as "rewrite" or "cleanup" since they have
> bad connotations, maybe let's call it "review" or "refinement".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Rémi Denis-Courmont]

Le perjantaina 13. lokakuuta 2023, 22.19.34 EEST Michael Niedermayer a écrit :
> But some goals would probably be to make sws
> * pleasent to work with
> * similar speed or faster
> * proper multithreading
> * proper full colorspace convertion not ignoring gamma, primaries, ...
> * clean / understandable modular design (maybe everything can be a "Filter"
> inside sws that get build into a chain)

It sounds very nice. But it also sounds very fuzzy and subjective. Unless you 
can put this in more objective terms such as would be expected of a statement 
of work, all the while not compromising the intent of the sponsorship, I would 
advise against using foundation funds.

But first...

> Proper payment (50k$ maybe) would be too much in relation to what SPI has
> ATM (150k$)

In my opinion, not "paying properly" is morally wrong, and sets a very bad 
example that it is acceptable not to pay developers properly.

Also it is really not that much money. I suppose that the yearly revenues are 
even only a fraction of that balance. If so, I would argue that it is not 
financially reasonable, doubly so when the overall world economy is in a rather 
iffy situation. If it were up to me, I would keep that money for infrastructure 
and other fees, purposeful hardware donations and travel.

Lastly using foundation funds to sponsor specific project may open a pandora 
box. What happens if more than one credible developer wants to take up the 
project? Or if other developers demand that their other reasonable clean-up 
projects be funded too? If the revenue and balance were much higher, you could 
hire all of them. But you can't, and then the simplest and safest way to avoid 
moral hazards is to hire nobody.

N.B.: I am not part of the GA, and I have neither the expertise and 
credibility nor the time and motivation to take up this project, so that's 
just my free advice.

-- 
Rémi Denis-Courmont
http://www.remlab.net/



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 4497 bytes --]

On Tue, Oct 17, 2023 at 09:50:41PM +0300, Rémi Denis-Courmont wrote:
> Le perjantaina 13. lokakuuta 2023, 22.19.34 EEST Michael Niedermayer a écrit :
> > But some goals would probably be to make sws
> > * pleasent to work with
> > * similar speed or faster
> > * proper multithreading
> > * proper full colorspace convertion not ignoring gamma, primaries, ...
> > * clean / understandable modular design (maybe everything can be a "Filter"
> > inside sws that get build into a chain)
> 
> It sounds very nice. But it also sounds very fuzzy and subjective. Unless you 
> can put this in more objective terms such as would be expected of a statement 
> of work, all the while not compromising the intent of the sponsorship, I would 
> advise against using foundation funds.

What i had in mind was that the developer who wants to work on this would
provide a offer that is more clearly defined.


> 
> But first...
> 
> > Proper payment (50k$ maybe) would be too much in relation to what SPI has
> > ATM (150k$)
> 
> In my opinion, not "paying properly" is morally wrong, and sets a very bad 
> example that it is acceptable not to pay developers properly.

It can be "proper" depending on where the developer is from, what she exactly
will do and how much time she will need. Also the developer may have other
sources of payment for this work or may benefit from a better swscale directly
or want to work on this volunteerly and just needs a little extra push.

I dont think its correct to view this in isolation as a
commercial US based software development contract. (i did make this mistake)


> 
> Also it is really not that much money. I suppose that the yearly revenues are 
> even only a fraction of that balance. If so, I would argue that it is not 
> financially reasonable, doubly so when the overall world economy is in a rather 
> iffy situation. If it were up to me, I would keep that money for infrastructure 
> and other fees, purposeful hardware donations and travel.

all the SPI money stuff is public.

But IMO the really important part is a different one. Not how much money we
have in SPI from donations today. But how much we could have if we USE the
money.
Why would someone donate to FFmpeg ? we have enough money for hw & travel.
If we use it to improve FFmpeg suddenly there is a reason to donate and also
theres a reason for us to ask for donations.
Only when we use the money and actively seek donations can we know what amount
of money would be available.

Also not using the money ever is almost certainly not what the people donating
wanted. And many different developers did over the last years indicate the need
for more solid financial sustainability. both SPI and FFlabs can be a parts in
this. Rejecting SPI seems not wise. Having multiple sources of funding seems a
good thing.


>
> Lastly using foundation funds to sponsor specific project may open a pandora 
> box.

FFlabs opened that already i think


> What happens if more than one credible developer wants to take up the
> project?

Then 15k$ was too much


> Or if other developers demand that their other reasonable clean-up
> projects be funded too?

they should ask IMHO


> If the revenue and balance were much higher, you could
> hire all of them. But you can't, and then the simplest and safest way to avoid 
> moral hazards is to hire nobody.

Let us all starve because if we start fishing and eating fish it might just be
that there are not enough fish

Whats the worst that can happen ?
We fund a few projects, we ask for donations a slight bit more actively,
maybe a news entry on the webpage pointing to what work was possible from
donations.
and it doesnt work out and our funds decrease from 120k with funding 5+ projects
to 60k. and at that point we return to hw and travel only use of the money


> 
> N.B.: I am not part of the GA, and I have neither the expertise and 
> credibility nor the time and motivation to take up this project, so that's 
> just my free advice.

your comments have always been interresting, even if i might disagree
with them !

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Frequently ignored answer#1 FFmpeg bugs should be sent to our bugtracker. User
questions about the command line tools should be sent to the ffmpeg-user ML.
And questions about how to use libav* should be sent to the libav-user ML.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

[-- Attachment #1.1: Type: text/plain, Size: 1335 bytes --]

On Tue, Oct 17, 2023 at 11:57:45PM +0200, Michael Niedermayer wrote:
> On Tue, Oct 17, 2023 at 09:50:41PM +0300, Rémi Denis-Courmont wrote:
[...]
> > If the revenue and balance were much higher, you could
> > hire all of them. But you can't, and then the simplest and safest way to avoid 
> > moral hazards is to hire nobody.
> 
> Let us all starve because if we start fishing and eating fish it might just be
> that there are not enough fish
> 
> Whats the worst that can happen ?
> We fund a few projects, we ask for donations a slight bit more actively,
> maybe a news entry on the webpage pointing to what work was possible from
> donations.
> and it doesnt work out and our funds decrease from 120k with funding 5+ projects
> to 60k. and at that point we return to hw and travel only use of the money

that should have been 150k (which is what we have curently) and 100k (where
i think we should rethink funding projects IF donations do not increase
to make us cashflow neutral)

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The day soldiers stop bringing you their problems is the day you have stopped 
leading them. They have either lost confidence that you can help or concluded 
you do not care. Either case is a failure of leadership. - Colin Powell

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Rémi Denis-Courmont]

Le keskiviikkona 18. lokakuuta 2023, 0.57.45 EEST Michael Niedermayer a écrit 
:
> On Tue, Oct 17, 2023 at 09:50:41PM +0300, Rémi Denis-Courmont wrote:
> > Le perjantaina 13. lokakuuta 2023, 22.19.34 EEST Michael Niedermayer a 
écrit :
> > > But some goals would probably be to make sws
> > > * pleasent to work with
> > > * similar speed or faster
> > > * proper multithreading
> > > * proper full colorspace convertion not ignoring gamma, primaries, ...
> > > * clean / understandable modular design (maybe everything can be a
> > > "Filter"
> > > inside sws that get build into a chain)
> > 
> > It sounds very nice. But it also sounds very fuzzy and subjective. Unless
> > you can put this in more objective terms such as would be expected of a
> > statement of work, all the while not compromising the intent of the
> > sponsorship, I would advise against using foundation funds.
> 
> What i had in mind was that the developer who wants to work on this would
> provide a offer that is more clearly defined.

That does not seem like a good idea, due to obvious moral hazards.

> > In my opinion, not "paying properly" is morally wrong, and sets a very bad
> > example that it is acceptable not to pay developers properly.
> 
> It can be "proper" depending on where the developer is from, what she
> exactly will do and how much time she will need. Also the developer may
> have other sources of payment for this work or may benefit from a better
> swscale directly or want to work on this volunteerly and just needs a
> little extra push.

You have to keep in mind that this working model is already intrinsically 
rather precarious (for the developer). It only gets worse if you make it 
underpaid and/or reliant on additional third party grants or motivation. Also 
if there are vested interest in getting the work done anyway, then it really 
should be funded by those vested interests: again, the foundation does not 
look like it can afford to spend much of anything in sponsored development at 
this time.

> But IMO the really important part is a different one. Not how much money we
> have in SPI from donations today. But how much we could have if we USE the
> money.

As long as the additional income does not become a legal liability, by all 
means, spend on stuff with good Return On Investment.

But while I can see the appeal of revamping swscale on a technical level as a 
professional software engineer myself, I am very Very VERY skeptic that it 
would have much if any ROI in terms of donations to FFmpeg.

> Why would someone donate to FFmpeg?

People will donate if they are willing, able and aware. Unfortunately, since 
FFmpeg is predominantly a library and almost exclusive back-end software, 
people do not know that they can even donate, even the few that are able and 
willing to do so.

> we have enough money for hw & travel.

Sure, but that is most certainly not at the top of the list of reasons for the 
dearth of donations.

> If we use it to improve FFmpeg suddenly there is a reason to donate and also
> theres a reason for us to ask for donations.

FFmpeg is nothing short of amazing as it is. Improving swscale is definitely 
not going to make a significant difference in that respect.

> Only when we use the money and actively seek donations can we know what
> amount of money would be available.

If you want to actively seek donations, you need to make people aware that you 
want donations. In other words, you need to do <s>marketing</s>public 
relations.

Improving swscale is NOT that. That's R&D, and it's only of immediate interest 
and notice to technically versed people.

> Also not using the money ever is almost certainly not what the people
> donating wanted.

This is true. A lot of people seem to think that open-source developers are 
permanently teenagers working in their parents' basements, and that donations 
are luxurious pocket money, when it really is small change.

But that does not change the fact that you simply don't have the right scale 
of revenue to sponsor development.

> And many different developers did over the last years indicate the need for
> more solid financial sustainability.

Spending what little money the foundation has on a highly technical and low 
external visibility development project is *not* sustainable by any reasonable 
common acceptance of the word.

When white collars utter "open-source sustainability", they mean is sustained 
and sustainable funding from commercial users of open-source, on a scale that 
is able to support ongoing development and maintenance activities.

A one-shot swscale improvement project is anything but.

> both SPI and FFlabs can be a parts in this. Rejecting SPI seems not wise.
> Having multiple sources of funding seems a good thing.

I am not rejecting SPI by principles. Certainly MediaWiki and Mozilla have at 
times shown that it is possible for foundation to pay their own developpers. I 
am rejecting SPI/FFmpeg because the revenues simply are not sufficient at this 
time.

Get a few hundreds of thousands € a year extra, and we can seriously talk 
about self-funding FFmpeg.

> > Lastly using foundation funds to sponsor specific project may open a
> > pandora box.
> 
> FFlabs opened that already i think

I don't know if FFlabs incited jealousy from developers for whatever real or 
perceived reasons. But FFlabs is (AFAIK) a for-profit business that gets money 
from its clients for specific purposes. Nothing like SPI.

> > What happens if more than one credible developer wants to take up the
> > project?
> 
> Then 15k$ was too much

So that's lowest bidder paradigm. Then you have to have a SoW first. See above.

(...)
> Whats the worst that can happen ?

The project fails because the terms were too vaguely defined. The GA become 
even more reluctant to spend what little money FFmpeg has. Developers are 
turned off from applying for sponsorship from FFmpeg. The contracted developer 
rage-quits.

> We fund a few projects, we ask for donations a slight bit more actively,
> maybe a news entry on the webpage pointing to what work was possible from
> donations.

Sigh. Ask for donations first and then if/when they actually materialise, write 
proper statement of words and *last* fund projects?


-- 
雷米‧德尼-库尔蒙
http://www.remlab.net/



_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Stefano Sabatini]

On date Wednesday 2023-10-18 19:30:11 +0300, Rémi Denis-Courmont wrote:
> Le keskiviikkona 18. lokakuuta 2023, 0.57.45 EEST Michael Niedermayer a écrit 
> :
> > On Tue, Oct 17, 2023 at 09:50:41PM +0300, Rémi Denis-Courmont wrote:
[...]
> > > What happens if more than one credible developer wants to take up the
> > > project?
> > 
> > Then 15k$ was too much
>
> So that's lowest bidder paradigm. Then you have to have a SoW first. See above.

This is similar to some of the issues with the GSOC program, the same
amount of money can mean something very different depending on the
contributors country/cost of life. So it is not really possible to
make a fixed price which is fair to everyone. Possibly it should be
computed depending on the country of the candidate developer depending
on some economic parameter.

> (...)
> > Whats the worst that can happen ?
> 
> The project fails because the terms were too vaguely defined. The GA become 
> even more reluctant to spend what little money FFmpeg has. Developers are 
> turned off from applying for sponsorship from FFmpeg. The contracted developer 
> rage-quits.

I can see this happening, but we cannot really predict without trying
- and we can learn from failure.

Another option would be to have something similar to bounties
(e.g. 100$ for each bug fixed - at least this might get low-hanging
fruits go away easily).

Or with some more organization we migth have the equivalent of FFSOC -
although this would be much more complicated to setup (since you have
to check with the student status and get the economical part straight,
deal with drops-out and so on).
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".