[FFmpeg-devel] CI

[FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 842 bytes --]

Hi

It seems the forgejo CI takes about
13min to do fate on aarch64 and x86-64 and build on win64

Locally i run
    fate + install on x86-64
    build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
    testprogs alltools examples build on x86-64, x86-32 and arm32
    in 2min 44sec

can we improve the speed vs amount of tests ratio ?
(its not a problem ATM, i did in fact not even notice as i never waited on it)

Iam just seeing the difference in time and i think there is potential for
optimization here

I dont think my box here is really special, just a
AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO

thx

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

While the State exists there can be no freedom; when there is freedom there
will be no State. -- Vladimir Lenin

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Timo Rothenpieler via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1245 bytes --]

On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> Hi
> 
> It seems the forgejo CI takes about
> 13min to do fate on aarch64 and x86-64 and build on win64
> 
> Locally i run
>      fate + install on x86-64
>      build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
>      testprogs alltools examples build on x86-64, x86-32 and arm32
>      in 2min 44sec
> 
> can we improve the speed vs amount of tests ratio ?
> (its not a problem ATM, i did in fact not even notice as i never waited on it)
> 
> Iam just seeing the difference in time and i think there is potential for
> optimization here
> 
> I dont think my box here is really special, just a
> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO

Well, the test runners are 4 cores and 8GB of RAM. So that'll be the 
primary difference in speed.
I think they're performing pretty good for being just that.

We could of course throw money at the problem and turn them into 16 core 
machines. That would up the hosting cost of the runners from currently 
3*7.5€ a month to 3*30€ a month. Just for the runners.

imo the current CI turnaround times are fine. 15-20 minutes per job is 
fine, as long as they can all run in parallel.

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2010 bytes --]

Hi

On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> > 
> > It seems the forgejo CI takes about
> > 13min to do fate on aarch64 and x86-64 and build on win64
> > 
> > Locally i run
> >      fate + install on x86-64
> >      build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> >      testprogs alltools examples build on x86-64, x86-32 and arm32
> >      in 2min 44sec
> > 
> > can we improve the speed vs amount of tests ratio ?
> > (its not a problem ATM, i did in fact not even notice as i never waited on it)
> > 
> > Iam just seeing the difference in time and i think there is potential for
> > optimization here
> > 
> > I dont think my box here is really special, just a
> > AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> 
> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> difference in speed.
> I think they're performing pretty good for being just that.
> 
> We could of course throw money at the problem and turn them into 16 core
> machines. That would up the hosting cost of the runners from currently
> 3*7.5€ a month to 3*30€ a month. Just for the runners.
> 
> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> as long as they can all run in parallel.

Option 1: 15-20 min CI turnaround,  270 € per year
Option 2:  4-5? min CI turnaround, 1080 € per year

we have over 150k $ it seems

Good use of capital can also lead to more donations

I think the main question is, "would we benefit from the faster trunaround"?
or not ?

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If the United States is serious about tackling the national security threats 
related to an insecure 5G network, it needs to rethink the extent to which it
values corporate profits and government espionage over security.-Bruce Schneier

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Timo Rothenpieler via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2531 bytes --]

On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> Hi
> 
> On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
>> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
>>> Hi
>>>
>>> It seems the forgejo CI takes about
>>> 13min to do fate on aarch64 and x86-64 and build on win64
>>>
>>> Locally i run
>>>       fate + install on x86-64
>>>       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
>>>       testprogs alltools examples build on x86-64, x86-32 and arm32
>>>       in 2min 44sec
>>>
>>> can we improve the speed vs amount of tests ratio ?
>>> (its not a problem ATM, i did in fact not even notice as i never waited on it)
>>>
>>> Iam just seeing the difference in time and i think there is potential for
>>> optimization here
>>>
>>> I dont think my box here is really special, just a
>>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
>>
>> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
>> difference in speed.
>> I think they're performing pretty good for being just that.
>>
>> We could of course throw money at the problem and turn them into 16 core
>> machines. That would up the hosting cost of the runners from currently
>> 3*7.5€ a month to 3*30€ a month. Just for the runners.
>>
>> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
>> as long as they can all run in parallel.
> 
> Option 1: 15-20 min CI turnaround,  270 € per year
> Option 2:  4-5? min CI turnaround, 1080 € per year
> 
> we have over 150k $ it seems
> 
> Good use of capital can also lead to more donations
> 
> I think the main question is, "would we benefit from the faster trunaround"?
> or not ?

You have to keep in mind, 4 Core 8GB is also the swarm of runners we get 
for free from Microsoft via GitHub.

So the choice is actually "Be able to process 20+ jobs in parallel that 
take 15-20 minutes each" vs. "Be able to process 3 or so at a time 
(roughly one PR/push) in 5 minutes".
So realistically, unless we also pay for an actual swarm of runners 
ourselves(which would cost 10k or more a year while being idle 95% of 
the time) the total turnaround time including wait for a free runner is 
probably still better with more of the smaller runners than less of the 
big ones.

It'd also make it a lot more pressing to think about every single CI job 
we add, vs. having a bit of leeway due to the over-abundance of runners.

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Kacper Michajlow via ffmpeg-devel]

On Thu, 21 Aug 2025 at 00:32, Timo Rothenpieler via ffmpeg-devel
<ffmpeg-devel@ffmpeg.org> wrote:
>
> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> >
> > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> >> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> >>> Hi
> >>>
> >>> It seems the forgejo CI takes about
> >>> 13min to do fate on aarch64 and x86-64 and build on win64
> >>>
> >>> Locally i run
> >>>       fate + install on x86-64
> >>>       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> >>>       testprogs alltools examples build on x86-64, x86-32 and arm32
> >>>       in 2min 44sec
> >>>
> >>> can we improve the speed vs amount of tests ratio ?
> >>> (its not a problem ATM, i did in fact not even notice as i never waited on it)
> >>>
> >>> Iam just seeing the difference in time and i think there is potential for
> >>> optimization here
> >>>
> >>> I dont think my box here is really special, just a
> >>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> >>
> >> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> >> difference in speed.
> >> I think they're performing pretty good for being just that.
> >>
> >> We could of course throw money at the problem and turn them into 16 core
> >> machines. That would up the hosting cost of the runners from currently
> >> 3*7.5€ a month to 3*30€ a month. Just for the runners.
> >>
> >> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> >> as long as they can all run in parallel.
> >
> > Option 1: 15-20 min CI turnaround,  270 € per year
> > Option 2:  4-5? min CI turnaround, 1080 € per year
> >
> > we have over 150k $ it seems
> >
> > Good use of capital can also lead to more donations
> >
> > I think the main question is, "would we benefit from the faster trunaround"?
> > or not ?
>
> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
> for free from Microsoft via GitHub.
>
> So the choice is actually "Be able to process 20+ jobs in parallel that
> take 15-20 minutes each" vs. "Be able to process 3 or so at a time
> (roughly one PR/push) in 5 minutes".
> So realistically, unless we also pay for an actual swarm of runners
> ourselves(which would cost 10k or more a year while being idle 95% of
> the time) the total turnaround time including wait for a free runner is
> probably still better with more of the smaller runners than less of the
> big ones.
>
> It'd also make it a lot more pressing to think about every single CI job
> we add, vs. having a bit of leeway due to the over-abundance of runners.

I agree with Timo, 15-20 minutes is not that bad for a CI job. Faster
is ofcourse better, but I would start worrying about it if we actually
start noticing the wait time. IMHO it is better to scale to more
parallel jobs instead of less faster ones.

Also on the topic of CI, I suspect we might add more jobs as needed.
Probably depends on priorities, to not overload existing runners
capacity.

I was wondering about integrating CIFuzz. Which basically runs fuzzers
based on coverage data that are affected by the PR changes. And runs
fuzzing for N minutes. This would allow it to catch trivial issues,
quickly and without the need for turnaround until the issue is
reported in the oss-fuzz infra. It uses existing corpus to seed the
fuzzing, so it should be good at catching things that are not "hidden"
too deep. Though this brings challenges, because it likely would need
a separate pool of workers to not starve and build jobs. And not sure
how well it would work in practice. For example if change touches too
many fuzzers, it wouldn't manage to run them in the time limit of say
20-30 minutes, which is probably the reasonable target. Though all
ffmpeg fuzers are specialized for codes, so codec specific code should
be fine to test. This is burning CI minutes, but if we have some to
spare, say from GH workers, it could help to weed out issues quickly.
Thoughts?

- Kacper
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1476 bytes --]

Hi

On Thu, Aug 21, 2025 at 06:03:12PM +0200, Kacper Michajlow via ffmpeg-devel wrote:
[...]
> Also on the topic of CI, I suspect we might add more jobs as needed.
> Probably depends on priorities, to not overload existing runners
> capacity.
> 
> I was wondering about integrating CIFuzz. Which basically runs fuzzers
> based on coverage data that are affected by the PR changes. And runs
> fuzzing for N minutes. This would allow it to catch trivial issues,
> quickly and without the need for turnaround until the issue is
> reported in the oss-fuzz infra. It uses existing corpus to seed the
> fuzzing, so it should be good at catching things that are not "hidden"
> too deep. Though this brings challenges, because it likely would need
> a separate pool of workers to not starve and build jobs. And not sure
> how well it would work in practice. For example if change touches too
> many fuzzers, it wouldn't manage to run them in the time limit of say
> 20-30 minutes, which is probably the reasonable target. Though all
> ffmpeg fuzers are specialized for codes, so codec specific code should
> be fine to test. This is burning CI minutes, but if we have some to
> spare, say from GH workers, it could help to weed out issues quickly.
> Thoughts?

immedeate fuzzing for pr, approved

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

It is what and why we do it that matters, not just one of them.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 3362 bytes --]

Hi

On Thu, Aug 21, 2025 at 12:31:46AM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> > 
> > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> > > On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> > > > Hi
> > > > 
> > > > It seems the forgejo CI takes about
> > > > 13min to do fate on aarch64 and x86-64 and build on win64
> > > > 
> > > > Locally i run
> > > >       fate + install on x86-64
> > > >       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> > > >       testprogs alltools examples build on x86-64, x86-32 and arm32
> > > >       in 2min 44sec
> > > > 
> > > > can we improve the speed vs amount of tests ratio ?
> > > > (its not a problem ATM, i did in fact not even notice as i never waited on it)
> > > > 
> > > > Iam just seeing the difference in time and i think there is potential for
> > > > optimization here
> > > > 
> > > > I dont think my box here is really special, just a
> > > > AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> > > 
> > > Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> > > difference in speed.
> > > I think they're performing pretty good for being just that.
> > > 
> > > We could of course throw money at the problem and turn them into 16 core
> > > machines. That would up the hosting cost of the runners from currently
> > > 3*7.5€ a month to 3*30€ a month. Just for the runners.
> > > 
> > > imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> > > as long as they can all run in parallel.
> > 
> > Option 1: 15-20 min CI turnaround,  270 € per year
> > Option 2:  4-5? min CI turnaround, 1080 € per year
> > 
> > we have over 150k $ it seems
> > 
> > Good use of capital can also lead to more donations
> > 
> > I think the main question is, "would we benefit from the faster trunaround"?
> > or not ?
> 
> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get for
> free from Microsoft via GitHub.
> 
> So the choice is actually "Be able to process 20+ jobs in parallel that take
> 15-20 minutes each" vs. "Be able to process 3 or so at a time (roughly one
> PR/push) in 5 minutes".
> So realistically, unless we also pay for an actual swarm of runners
> ourselves(which would cost 10k or more a year while being idle 95% of the
> time) the total turnaround time including wait for a free runner is probably
> still better with more of the smaller runners than less of the big ones.
> 
> It'd also make it a lot more pressing to think about every single CI job we
> add, vs. having a bit of leeway due to the over-abundance of runners.

for 1-2k$ you can buy a box that runs fate once and build on 6 times in
under 3minutes.

if one is not enough buy 3, use the extra capcity for fuzzing or rent out
to other projects

I must be stupid, because to me this looks cheaper, its also one time expense
these boxes can be used for 10 years

also no need to be reliable expensive servers, if you have 3.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"I am not trying to be anyone's saviour, I'm trying to think about the
 future and not be sad" - Elon Musk


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Kieran Kunhya via ffmpeg-devel]

On Thu, 21 Aug 2025, 11:33 Michael Niedermayer via ffmpeg-devel, <
ffmpeg-devel@ffmpeg.org> wrote:

> Hi
>
> On Thu, Aug 21, 2025 at 12:31:46AM +0200, Timo Rothenpieler via
> ffmpeg-devel wrote:
> > On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > > Hi
> > >
> > > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via
> ffmpeg-devel wrote:
> > > > On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> > > > > Hi
> > > > >
> > > > > It seems the forgejo CI takes about
> > > > > 13min to do fate on aarch64 and x86-64 and build on win64
> > > > >
> > > > > Locally i run
> > > > >       fate + install on x86-64
> > > > >       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared
> libs
> > > > >       testprogs alltools examples build on x86-64, x86-32 and arm32
> > > > >       in 2min 44sec
> > > > >
> > > > > can we improve the speed vs amount of tests ratio ?
> > > > > (its not a problem ATM, i did in fact not even notice as i never
> waited on it)
> > > > >
> > > > > Iam just seeing the difference in time and i think there is
> potential for
> > > > > optimization here
> > > > >
> > > > > I dont think my box here is really special, just a
> > > > > AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> > > >
> > > > Well, the test runners are 4 cores and 8GB of RAM. So that'll be the
> primary
> > > > difference in speed.
> > > > I think they're performing pretty good for being just that.
> > > >
> > > > We could of course throw money at the problem and turn them into 16
> core
> > > > machines. That would up the hosting cost of the runners from
> currently
> > > > 3*7.5€ a month to 3*30€ a month. Just for the runners.
> > > >
> > > > imo the current CI turnaround times are fine. 15-20 minutes per job
> is fine,
> > > > as long as they can all run in parallel.
> > >
> > > Option 1: 15-20 min CI turnaround,  270 € per year
> > > Option 2:  4-5? min CI turnaround, 1080 € per year
> > >
> > > we have over 150k $ it seems
> > >
> > > Good use of capital can also lead to more donations
> > >
> > > I think the main question is, "would we benefit from the faster
> trunaround"?
> > > or not ?
> >
> > You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
> for
> > free from Microsoft via GitHub.
> >
> > So the choice is actually "Be able to process 20+ jobs in parallel that
> take
> > 15-20 minutes each" vs. "Be able to process 3 or so at a time (roughly
> one
> > PR/push) in 5 minutes".
> > So realistically, unless we also pay for an actual swarm of runners
> > ourselves(which would cost 10k or more a year while being idle 95% of the
> > time) the total turnaround time including wait for a free runner is
> probably
> > still better with more of the smaller runners than less of the big ones.
> >
> > It'd also make it a lot more pressing to think about every single CI job
> we
> > add, vs. having a bit of leeway due to the over-abundance of runners.
>
> for 1-2k$ you can buy a box that runs fate once and build on 6 times in
> under 3minutes.
>
> if one is not enough buy 3, use the extra capcity for fuzzing or rent out
> to other projects
>
> I must be stupid, because to me this looks cheaper, its also one time
> expense
> these boxes can be used for 10 years
>
> also no need to be reliable expensive servers, if you have 3.
>
> thx
>

In the end you're paying for hosting, 24/7 electricity and not having to
worry about it.

I agree with both viewpoints. For "mission critical" stuff like CI we
should host at a proper hosting company.

Kieran

>
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Timo Rothenpieler via ffmpeg-devel]

On 22/08/2025 07:00, Kieran Kunhya via ffmpeg-devel wrote:
> On Thu, 21 Aug 2025, 11:33 Michael Niedermayer via ffmpeg-devel, <
> ffmpeg-devel@ffmpeg.org> wrote:
> 
>> Hi
>>
>> On Thu, Aug 21, 2025 at 12:31:46AM +0200, Timo Rothenpieler via
>> ffmpeg-devel wrote:
>>> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
>>>> Hi
>>>>
>>>> On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via
>> ffmpeg-devel wrote:
>>>>> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
>>>>>> Hi
>>>>>>
>>>>>> It seems the forgejo CI takes about
>>>>>> 13min to do fate on aarch64 and x86-64 and build on win64
>>>>>>
>>>>>> Locally i run
>>>>>>        fate + install on x86-64
>>>>>>        build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared
>> libs
>>>>>>        testprogs alltools examples build on x86-64, x86-32 and arm32
>>>>>>        in 2min 44sec
>>>>>>
>>>>>> can we improve the speed vs amount of tests ratio ?
>>>>>> (its not a problem ATM, i did in fact not even notice as i never
>> waited on it)
>>>>>>
>>>>>> Iam just seeing the difference in time and i think there is
>> potential for
>>>>>> optimization here
>>>>>>
>>>>>> I dont think my box here is really special, just a
>>>>>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
>>>>>
>>>>> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the
>> primary
>>>>> difference in speed.
>>>>> I think they're performing pretty good for being just that.
>>>>>
>>>>> We could of course throw money at the problem and turn them into 16
>> core
>>>>> machines. That would up the hosting cost of the runners from
>> currently
>>>>> 3*7.5€ a month to 3*30€ a month. Just for the runners.
>>>>>
>>>>> imo the current CI turnaround times are fine. 15-20 minutes per job
>> is fine,
>>>>> as long as they can all run in parallel.
>>>>
>>>> Option 1: 15-20 min CI turnaround,  270 € per year
>>>> Option 2:  4-5? min CI turnaround, 1080 € per year
>>>>
>>>> we have over 150k $ it seems
>>>>
>>>> Good use of capital can also lead to more donations
>>>>
>>>> I think the main question is, "would we benefit from the faster
>> trunaround"?
>>>> or not ?
>>>
>>> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
>> for
>>> free from Microsoft via GitHub.
>>>
>>> So the choice is actually "Be able to process 20+ jobs in parallel that
>> take
>>> 15-20 minutes each" vs. "Be able to process 3 or so at a time (roughly
>> one
>>> PR/push) in 5 minutes".
>>> So realistically, unless we also pay for an actual swarm of runners
>>> ourselves(which would cost 10k or more a year while being idle 95% of the
>>> time) the total turnaround time including wait for a free runner is
>> probably
>>> still better with more of the smaller runners than less of the big ones.
>>>
>>> It'd also make it a lot more pressing to think about every single CI job
>> we
>>> add, vs. having a bit of leeway due to the over-abundance of runners.
>>
>> for 1-2k$ you can buy a box that runs fate once and build on 6 times in
>> under 3minutes.
>>
>> if one is not enough buy 3, use the extra capcity for fuzzing or rent out
>> to other projects
>>
>> I must be stupid, because to me this looks cheaper, its also one time
>> expense
>> these boxes can be used for 10 years
>>
>> also no need to be reliable expensive servers, if you have 3.
>>
>> thx
>>
> 
> In the end you're paying for hosting, 24/7 electricity and not having to
> worry about it.

Yeah, in the end someone has to put these 3 servers somewhere, monitor 
them for hardware failures, and pay for their electricity.
Which isn't exactly cheap for 3 servers either.

> I agree with both viewpoints. For "mission critical" stuff like CI we
> should host at a proper hosting company.

There's also the option of renting a dedicated server from Hetzner or 
some other company, and put a bunch of runners on there.
Hetzner rents you an 8 core Zen4 Ryzen with 64GB of RAM for ~55€ a month.

But compared to the current virtual servers we rent from Hetzner, that's 
a pretty poor "performance per money" ratio, given we get one server 
with 4 cores and 8GB of RAM, which is plenty for our needs, for 7.5€ a 
month.
So we can rent up to 7 of those, while still paying less than for one 
dedicated server.
And dedicated servers are again more effort, because you do need to 
manually monitor them for any hardware faults, and then coordinate the 
replacement with Hetzner, and they do eventually die of old age.

There's also the option of writing some tool that automatically scales 
runners up/down at some cloud provider that bills minutely/secondly 
(Hetzner bills Servers per Hour, making this a lot less interesting).
That would require some serious cost calculation. But I feel like we 
have plenty of CI downtime to make it worthwhile.

But then again, GitHub/MS gives us 20 parallel runners for free, and we 
can freely pick if they're running on x86_64 or aarch64, Linux, Windows 
or even OSX.
As long as they do that, we only need to host a baseline of runners 
ourselves, and can scale out into that whenever there's a peak in usage.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 6161 bytes --]

On Fri, Aug 22, 2025 at 02:13:14PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> On 22/08/2025 07:00, Kieran Kunhya via ffmpeg-devel wrote:
> > On Thu, 21 Aug 2025, 11:33 Michael Niedermayer via ffmpeg-devel, <
> > ffmpeg-devel@ffmpeg.org> wrote:
> > 
> > > Hi
> > > 
> > > On Thu, Aug 21, 2025 at 12:31:46AM +0200, Timo Rothenpieler via
> > > ffmpeg-devel wrote:
> > > > On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > > > > Hi
> > > > > 
> > > > > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via
> > > ffmpeg-devel wrote:
> > > > > > On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> > > > > > > Hi
> > > > > > > 
> > > > > > > It seems the forgejo CI takes about
> > > > > > > 13min to do fate on aarch64 and x86-64 and build on win64
> > > > > > > 
> > > > > > > Locally i run
> > > > > > >        fate + install on x86-64
> > > > > > >        build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared
> > > libs
> > > > > > >        testprogs alltools examples build on x86-64, x86-32 and arm32
> > > > > > >        in 2min 44sec
> > > > > > > 
> > > > > > > can we improve the speed vs amount of tests ratio ?
> > > > > > > (its not a problem ATM, i did in fact not even notice as i never
> > > waited on it)
> > > > > > > 
> > > > > > > Iam just seeing the difference in time and i think there is
> > > potential for
> > > > > > > optimization here
> > > > > > > 
> > > > > > > I dont think my box here is really special, just a
> > > > > > > AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> > > > > > 
> > > > > > Well, the test runners are 4 cores and 8GB of RAM. So that'll be the
> > > primary
> > > > > > difference in speed.
> > > > > > I think they're performing pretty good for being just that.
> > > > > > 
> > > > > > We could of course throw money at the problem and turn them into 16
> > > core
> > > > > > machines. That would up the hosting cost of the runners from
> > > currently
> > > > > > 3*7.5€ a month to 3*30€ a month. Just for the runners.
> > > > > > 
> > > > > > imo the current CI turnaround times are fine. 15-20 minutes per job
> > > is fine,
> > > > > > as long as they can all run in parallel.
> > > > > 
> > > > > Option 1: 15-20 min CI turnaround,  270 € per year
> > > > > Option 2:  4-5? min CI turnaround, 1080 € per year
> > > > > 
> > > > > we have over 150k $ it seems
> > > > > 
> > > > > Good use of capital can also lead to more donations
> > > > > 
> > > > > I think the main question is, "would we benefit from the faster
> > > trunaround"?
> > > > > or not ?
> > > > 
> > > > You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
> > > for
> > > > free from Microsoft via GitHub.
> > > > 
> > > > So the choice is actually "Be able to process 20+ jobs in parallel that
> > > take
> > > > 15-20 minutes each" vs. "Be able to process 3 or so at a time (roughly
> > > one
> > > > PR/push) in 5 minutes".
> > > > So realistically, unless we also pay for an actual swarm of runners
> > > > ourselves(which would cost 10k or more a year while being idle 95% of the
> > > > time) the total turnaround time including wait for a free runner is
> > > probably
> > > > still better with more of the smaller runners than less of the big ones.
> > > > 
> > > > It'd also make it a lot more pressing to think about every single CI job
> > > we
> > > > add, vs. having a bit of leeway due to the over-abundance of runners.
> > > 
> > > for 1-2k$ you can buy a box that runs fate once and build on 6 times in
> > > under 3minutes.
> > > 
> > > if one is not enough buy 3, use the extra capcity for fuzzing or rent out
> > > to other projects
> > > 
> > > I must be stupid, because to me this looks cheaper, its also one time
> > > expense
> > > these boxes can be used for 10 years
> > > 
> > > also no need to be reliable expensive servers, if you have 3.
> > > 
> > > thx
> > > 
> > 
> > In the end you're paying for hosting, 24/7 electricity and not having to
> > worry about it.
> 
> Yeah, in the end someone has to put these 3 servers somewhere, monitor them
> for hardware failures, and pay for their electricity.
> Which isn't exactly cheap for 3 servers either.
> 
> > I agree with both viewpoints. For "mission critical" stuff like CI we
> > should host at a proper hosting company.
> 
> There's also the option of renting a dedicated server from Hetzner or some
> other company, and put a bunch of runners on there.
> Hetzner rents you an 8 core Zen4 Ryzen with 64GB of RAM for ~55€ a month.
> 
> But compared to the current virtual servers we rent from Hetzner, that's a
> pretty poor "performance per money" ratio, given we get one server with 4
> cores and 8GB of RAM, which is plenty for our needs, for 7.5€ a month.
> So we can rent up to 7 of those, while still paying less than for one
> dedicated server.
> And dedicated servers are again more effort, because you do need to manually
> monitor them for any hardware faults, and then coordinate the replacement
> with Hetzner, and they do eventually die of old age.
> 
> There's also the option of writing some tool that automatically scales
> runners up/down at some cloud provider that bills minutely/secondly (Hetzner
> bills Servers per Hour, making this a lot less interesting).
> That would require some serious cost calculation. But I feel like we have
> plenty of CI downtime to make it worthwhile.
> 
> But then again, GitHub/MS gives us 20 parallel runners for free, and we can
> freely pick if they're running on x86_64 or aarch64, Linux, Windows or even
> OSX.
> As long as they do that, we only need to host a baseline of runners
> ourselves, and can scale out into that whenever there's a peak in usage.

I guess next time i upgrade my box, i might just keep the old one running in a
corner.

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"Nothing to hide" only works if the folks in power share the values of
you and everyone you know entirely and always will -- Tom Scott


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 841 bytes --]

Hi Timo

On Fri, Aug 22, 2025 at 02:13:14PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
[...]
> But then again, GitHub/MS gives us 20 parallel runners for free, and we can
> freely pick if they're running on x86_64 or aarch64, Linux, Windows or even
> OSX.
> As long as they do that, we only need to host a baseline of runners
> ourselves, and can scale out into that whenever there's a peak in usage.

if they give us 20, can we test mips & arm with qemu in a way that it
does not block or delay merges.

I mean so that a pr is considered ok and mergeable before tzhe slow qemu
fate finishes but after the 2h or so it will display the result in the pr

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

He who knows, does not speak. He who speaks, does not know. -- Lao Tsu

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Timo Rothenpieler via ffmpeg-devel]

On 22/08/2025 16:54, Michael Niedermayer via ffmpeg-devel wrote:
> Hi Timo
> 
> On Fri, Aug 22, 2025 at 02:13:14PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> [...]
>> But then again, GitHub/MS gives us 20 parallel runners for free, and we can
>> freely pick if they're running on x86_64 or aarch64, Linux, Windows or even
>> OSX.
>> As long as they do that, we only need to host a baseline of runners
>> ourselves, and can scale out into that whenever there's a peak in usage.
> 
> if they give us 20, can we test mips & arm with qemu in a way that it
> does not block or delay merges.
> 
> I mean so that a pr is considered ok and mergeable before tzhe slow qemu
> fate finishes but after the 2h or so it will display the result in the pr
We could only run those tests on master, not on PRs.
Nobody is impacted by them then, and we still notice breakage reasonably 
fast.

For arm I'm not sure if we really need qemu? All it might take is a 
32bit arm chroot on aarch64? Not sure if it works like x86 though, where 
a 64bit CPU can also run 32bit code.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Martin Storsjö via ffmpeg-devel]

On Fri, 22 Aug 2025, Timo Rothenpieler via ffmpeg-devel wrote:

> For arm I'm not sure if we really need qemu? All it might take is a 
> 32bit arm chroot on aarch64? Not sure if it works like x86 though, where 
> a 64bit CPU can also run 32bit code.

It works pretty much like x86, yes - you don't need QEMU, you can run it 
natively on an aarch64 kernel.

And you don't need a separate chroot - on Debian/Ubuntu multiarch it's 
quite straightforward to install 32 bit arm as a separate architecture:

     dpkg --add-architecture armhf
     apt-get update && apt-get install g++-arm-linux-gnueabihf libc6:armhf libstdc++6:armhf

This is enough for being able to build things for it, with 
--cross-prefix=arm-linux-gnueabihf- and to run the binaries normally.

That said, the very latest generations of aarch64 chips have removed 
support for executing in 32 bit mode; luckily the recent github aarch64 
runners (which are quite cutting edge otherwise, and have features like 
SVE2) still do support 32 bit. But e.g. on Amazon, the Graviton 4 series 
no longer supports 32 bit, and the Apple M1 (and newer ones) also lack 32 
bit support entirely.

// Martin

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1658 bytes --]

Hi

On Fri, Aug 22, 2025 at 05:22:54PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> On 22/08/2025 16:54, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi Timo
> > 
> > On Fri, Aug 22, 2025 at 02:13:14PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> > [...]
> > > But then again, GitHub/MS gives us 20 parallel runners for free, and we can
> > > freely pick if they're running on x86_64 or aarch64, Linux, Windows or even
> > > OSX.
> > > As long as they do that, we only need to host a baseline of runners
> > > ourselves, and can scale out into that whenever there's a peak in usage.
> > 
> > if they give us 20, can we test mips & arm with qemu in a way that it
> > does not block or delay merges.
> > 
> > I mean so that a pr is considered ok and mergeable before tzhe slow qemu
> > fate finishes but after the 2h or so it will display the result in the pr

> We could only run those tests on master, not on PRs.
> Nobody is impacted by them then, and we still notice breakage reasonably
> fast.

yes, agree


> 
> For arm I'm not sure if we really need qemu? All it might take is a 32bit
> arm chroot on aarch64? Not sure if it works like x86 though, where a 64bit
> CPU can also run 32bit code.

yeah i thought the same a few moments after sending that mail, though i
have never tried

i guess riscv would then be the next on the list to have at least a build test

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If a bugfix only changes things apparently unrelated to the bug with no
further explanation, that is a good sign that the bugfix is wrong.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Niklas Haas via ffmpeg-devel]

On Fri, 22 Aug 2025 17:22:54 +0200 Timo Rothenpieler via ffmpeg-devel <ffmpeg-devel@ffmpeg.org> wrote:
> We could only run those tests on master, not on PRs.
> Nobody is impacted by them then, and we still notice breakage reasonably
> fast.

That seems reasonable to me.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".