[FFmpeg-devel] CI

[FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 842 bytes --]

Hi

It seems the forgejo CI takes about
13min to do fate on aarch64 and x86-64 and build on win64

Locally i run
    fate + install on x86-64
    build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
    testprogs alltools examples build on x86-64, x86-32 and arm32
    in 2min 44sec

can we improve the speed vs amount of tests ratio ?
(its not a problem ATM, i did in fact not even notice as i never waited on it)

Iam just seeing the difference in time and i think there is potential for
optimization here

I dont think my box here is really special, just a
AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO

thx

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

While the State exists there can be no freedom; when there is freedom there
will be no State. -- Vladimir Lenin

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Timo Rothenpieler via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1245 bytes --]

On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> Hi
> 
> It seems the forgejo CI takes about
> 13min to do fate on aarch64 and x86-64 and build on win64
> 
> Locally i run
>      fate + install on x86-64
>      build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
>      testprogs alltools examples build on x86-64, x86-32 and arm32
>      in 2min 44sec
> 
> can we improve the speed vs amount of tests ratio ?
> (its not a problem ATM, i did in fact not even notice as i never waited on it)
> 
> Iam just seeing the difference in time and i think there is potential for
> optimization here
> 
> I dont think my box here is really special, just a
> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO

Well, the test runners are 4 cores and 8GB of RAM. So that'll be the 
primary difference in speed.
I think they're performing pretty good for being just that.

We could of course throw money at the problem and turn them into 16 core 
machines. That would up the hosting cost of the runners from currently 
3*7.5€ a month to 3*30€ a month. Just for the runners.

imo the current CI turnaround times are fine. 15-20 minutes per job is 
fine, as long as they can all run in parallel.

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2010 bytes --]

Hi

On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> > 
> > It seems the forgejo CI takes about
> > 13min to do fate on aarch64 and x86-64 and build on win64
> > 
> > Locally i run
> >      fate + install on x86-64
> >      build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> >      testprogs alltools examples build on x86-64, x86-32 and arm32
> >      in 2min 44sec
> > 
> > can we improve the speed vs amount of tests ratio ?
> > (its not a problem ATM, i did in fact not even notice as i never waited on it)
> > 
> > Iam just seeing the difference in time and i think there is potential for
> > optimization here
> > 
> > I dont think my box here is really special, just a
> > AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> 
> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> difference in speed.
> I think they're performing pretty good for being just that.
> 
> We could of course throw money at the problem and turn them into 16 core
> machines. That would up the hosting cost of the runners from currently
> 3*7.5€ a month to 3*30€ a month. Just for the runners.
> 
> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> as long as they can all run in parallel.

Option 1: 15-20 min CI turnaround,  270 € per year
Option 2:  4-5? min CI turnaround, 1080 € per year

we have over 150k $ it seems

Good use of capital can also lead to more donations

I think the main question is, "would we benefit from the faster trunaround"?
or not ?

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If the United States is serious about tackling the national security threats 
related to an insecure 5G network, it needs to rethink the extent to which it
values corporate profits and government espionage over security.-Bruce Schneier

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Timo Rothenpieler via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 2531 bytes --]

On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> Hi
> 
> On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
>> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
>>> Hi
>>>
>>> It seems the forgejo CI takes about
>>> 13min to do fate on aarch64 and x86-64 and build on win64
>>>
>>> Locally i run
>>>       fate + install on x86-64
>>>       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
>>>       testprogs alltools examples build on x86-64, x86-32 and arm32
>>>       in 2min 44sec
>>>
>>> can we improve the speed vs amount of tests ratio ?
>>> (its not a problem ATM, i did in fact not even notice as i never waited on it)
>>>
>>> Iam just seeing the difference in time and i think there is potential for
>>> optimization here
>>>
>>> I dont think my box here is really special, just a
>>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
>>
>> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
>> difference in speed.
>> I think they're performing pretty good for being just that.
>>
>> We could of course throw money at the problem and turn them into 16 core
>> machines. That would up the hosting cost of the runners from currently
>> 3*7.5€ a month to 3*30€ a month. Just for the runners.
>>
>> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
>> as long as they can all run in parallel.
> 
> Option 1: 15-20 min CI turnaround,  270 € per year
> Option 2:  4-5? min CI turnaround, 1080 € per year
> 
> we have over 150k $ it seems
> 
> Good use of capital can also lead to more donations
> 
> I think the main question is, "would we benefit from the faster trunaround"?
> or not ?

You have to keep in mind, 4 Core 8GB is also the swarm of runners we get 
for free from Microsoft via GitHub.

So the choice is actually "Be able to process 20+ jobs in parallel that 
take 15-20 minutes each" vs. "Be able to process 3 or so at a time 
(roughly one PR/push) in 5 minutes".
So realistically, unless we also pay for an actual swarm of runners 
ourselves(which would cost 10k or more a year while being idle 95% of 
the time) the total turnaround time including wait for a free runner is 
probably still better with more of the smaller runners than less of the 
big ones.

It'd also make it a lot more pressing to think about every single CI job 
we add, vs. having a bit of leeway due to the over-abundance of runners.

[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Kacper Michajlow via ffmpeg-devel]

On Thu, 21 Aug 2025 at 00:32, Timo Rothenpieler via ffmpeg-devel
<ffmpeg-devel@ffmpeg.org> wrote:
>
> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> >
> > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> >> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> >>> Hi
> >>>
> >>> It seems the forgejo CI takes about
> >>> 13min to do fate on aarch64 and x86-64 and build on win64
> >>>
> >>> Locally i run
> >>>       fate + install on x86-64
> >>>       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> >>>       testprogs alltools examples build on x86-64, x86-32 and arm32
> >>>       in 2min 44sec
> >>>
> >>> can we improve the speed vs amount of tests ratio ?
> >>> (its not a problem ATM, i did in fact not even notice as i never waited on it)
> >>>
> >>> Iam just seeing the difference in time and i think there is potential for
> >>> optimization here
> >>>
> >>> I dont think my box here is really special, just a
> >>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> >>
> >> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> >> difference in speed.
> >> I think they're performing pretty good for being just that.
> >>
> >> We could of course throw money at the problem and turn them into 16 core
> >> machines. That would up the hosting cost of the runners from currently
> >> 3*7.5€ a month to 3*30€ a month. Just for the runners.
> >>
> >> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> >> as long as they can all run in parallel.
> >
> > Option 1: 15-20 min CI turnaround,  270 € per year
> > Option 2:  4-5? min CI turnaround, 1080 € per year
> >
> > we have over 150k $ it seems
> >
> > Good use of capital can also lead to more donations
> >
> > I think the main question is, "would we benefit from the faster trunaround"?
> > or not ?
>
> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
> for free from Microsoft via GitHub.
>
> So the choice is actually "Be able to process 20+ jobs in parallel that
> take 15-20 minutes each" vs. "Be able to process 3 or so at a time
> (roughly one PR/push) in 5 minutes".
> So realistically, unless we also pay for an actual swarm of runners
> ourselves(which would cost 10k or more a year while being idle 95% of
> the time) the total turnaround time including wait for a free runner is
> probably still better with more of the smaller runners than less of the
> big ones.
>
> It'd also make it a lot more pressing to think about every single CI job
> we add, vs. having a bit of leeway due to the over-abundance of runners.

I agree with Timo, 15-20 minutes is not that bad for a CI job. Faster
is ofcourse better, but I would start worrying about it if we actually
start noticing the wait time. IMHO it is better to scale to more
parallel jobs instead of less faster ones.

Also on the topic of CI, I suspect we might add more jobs as needed.
Probably depends on priorities, to not overload existing runners
capacity.

I was wondering about integrating CIFuzz. Which basically runs fuzzers
based on coverage data that are affected by the PR changes. And runs
fuzzing for N minutes. This would allow it to catch trivial issues,
quickly and without the need for turnaround until the issue is
reported in the oss-fuzz infra. It uses existing corpus to seed the
fuzzing, so it should be good at catching things that are not "hidden"
too deep. Though this brings challenges, because it likely would need
a separate pool of workers to not starve and build jobs. And not sure
how well it would work in practice. For example if change touches too
many fuzzers, it wouldn't manage to run them in the time limit of say
20-30 minutes, which is probably the reasonable target. Though all
ffmpeg fuzers are specialized for codes, so codec specific code should
be fine to test. This is burning CI minutes, but if we have some to
spare, say from GH workers, it could help to weed out issues quickly.
Thoughts?

- Kacper
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 1476 bytes --]

Hi

On Thu, Aug 21, 2025 at 06:03:12PM +0200, Kacper Michajlow via ffmpeg-devel wrote:
[...]
> Also on the topic of CI, I suspect we might add more jobs as needed.
> Probably depends on priorities, to not overload existing runners
> capacity.
> 
> I was wondering about integrating CIFuzz. Which basically runs fuzzers
> based on coverage data that are affected by the PR changes. And runs
> fuzzing for N minutes. This would allow it to catch trivial issues,
> quickly and without the need for turnaround until the issue is
> reported in the oss-fuzz infra. It uses existing corpus to seed the
> fuzzing, so it should be good at catching things that are not "hidden"
> too deep. Though this brings challenges, because it likely would need
> a separate pool of workers to not starve and build jobs. And not sure
> how well it would work in practice. For example if change touches too
> many fuzzers, it wouldn't manage to run them in the time limit of say
> 20-30 minutes, which is probably the reasonable target. Though all
> ffmpeg fuzers are specialized for codes, so codec specific code should
> be fine to test. This is burning CI minutes, but if we have some to
> spare, say from GH workers, it could help to weed out issues quickly.
> Thoughts?

immedeate fuzzing for pr, approved

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

It is what and why we do it that matters, not just one of them.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] CI

[Michael Niedermayer via ffmpeg-devel]

[-- Attachment #1.1: Type: text/plain, Size: 3362 bytes --]

Hi

On Thu, Aug 21, 2025 at 12:31:46AM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> > 
> > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> > > On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> > > > Hi
> > > > 
> > > > It seems the forgejo CI takes about
> > > > 13min to do fate on aarch64 and x86-64 and build on win64
> > > > 
> > > > Locally i run
> > > >       fate + install on x86-64
> > > >       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> > > >       testprogs alltools examples build on x86-64, x86-32 and arm32
> > > >       in 2min 44sec
> > > > 
> > > > can we improve the speed vs amount of tests ratio ?
> > > > (its not a problem ATM, i did in fact not even notice as i never waited on it)
> > > > 
> > > > Iam just seeing the difference in time and i think there is potential for
> > > > optimization here
> > > > 
> > > > I dont think my box here is really special, just a
> > > > AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> > > 
> > > Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> > > difference in speed.
> > > I think they're performing pretty good for being just that.
> > > 
> > > We could of course throw money at the problem and turn them into 16 core
> > > machines. That would up the hosting cost of the runners from currently
> > > 3*7.5€ a month to 3*30€ a month. Just for the runners.
> > > 
> > > imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> > > as long as they can all run in parallel.
> > 
> > Option 1: 15-20 min CI turnaround,  270 € per year
> > Option 2:  4-5? min CI turnaround, 1080 € per year
> > 
> > we have over 150k $ it seems
> > 
> > Good use of capital can also lead to more donations
> > 
> > I think the main question is, "would we benefit from the faster trunaround"?
> > or not ?
> 
> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get for
> free from Microsoft via GitHub.
> 
> So the choice is actually "Be able to process 20+ jobs in parallel that take
> 15-20 minutes each" vs. "Be able to process 3 or so at a time (roughly one
> PR/push) in 5 minutes".
> So realistically, unless we also pay for an actual swarm of runners
> ourselves(which would cost 10k or more a year while being idle 95% of the
> time) the total turnaround time including wait for a free runner is probably
> still better with more of the smaller runners than less of the big ones.
> 
> It'd also make it a lot more pressing to think about every single CI job we
> add, vs. having a bit of leeway due to the over-abundance of runners.

for 1-2k$ you can buy a box that runs fate once and build on 6 times in
under 3minutes.

if one is not enough buy 3, use the extra capcity for fuzzing or rent out
to other projects

I must be stupid, because to me this looks cheaper, its also one time expense
these boxes can be used for 10 years

also no need to be reliable expensive servers, if you have 3.

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"I am not trying to be anyone's saviour, I'm trying to think about the
 future and not be sad" - Elon Musk


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 251 bytes --]

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".