Re: [FFmpeg-devel] CI

[Kacper Michajlow via ffmpeg-devel <ffmpeg-devel@ffmpeg.org>]

On Thu, 21 Aug 2025 at 00:32, Timo Rothenpieler via ffmpeg-devel
<ffmpeg-devel@ffmpeg.org> wrote:
>
> On 8/20/2025 9:25 PM, Michael Niedermayer via ffmpeg-devel wrote:
> > Hi
> >
> > On Wed, Aug 20, 2025 at 05:56:27PM +0200, Timo Rothenpieler via ffmpeg-devel wrote:
> >> On 8/20/2025 1:26 AM, Michael Niedermayer via ffmpeg-devel wrote:
> >>> Hi
> >>>
> >>> It seems the forgejo CI takes about
> >>> 13min to do fate on aarch64 and x86-64 and build on win64
> >>>
> >>> Locally i run
> >>>       fate + install on x86-64
> >>>       build on x86-32, mingw64, arm32, mips, ppc, x86-64 + shared libs
> >>>       testprogs alltools examples build on x86-64, x86-32 and arm32
> >>>       in 2min 44sec
> >>>
> >>> can we improve the speed vs amount of tests ratio ?
> >>> (its not a problem ATM, i did in fact not even notice as i never waited on it)
> >>>
> >>> Iam just seeing the difference in time and i think there is potential for
> >>> optimization here
> >>>
> >>> I dont think my box here is really special, just a
> >>> AMD Ryzen 9 3950X 16-Core + Samsung SSD 970 PRO
> >>
> >> Well, the test runners are 4 cores and 8GB of RAM. So that'll be the primary
> >> difference in speed.
> >> I think they're performing pretty good for being just that.
> >>
> >> We could of course throw money at the problem and turn them into 16 core
> >> machines. That would up the hosting cost of the runners from currently
> >> 3*7.5€ a month to 3*30€ a month. Just for the runners.
> >>
> >> imo the current CI turnaround times are fine. 15-20 minutes per job is fine,
> >> as long as they can all run in parallel.
> >
> > Option 1: 15-20 min CI turnaround,  270 € per year
> > Option 2:  4-5? min CI turnaround, 1080 € per year
> >
> > we have over 150k $ it seems
> >
> > Good use of capital can also lead to more donations
> >
> > I think the main question is, "would we benefit from the faster trunaround"?
> > or not ?
>
> You have to keep in mind, 4 Core 8GB is also the swarm of runners we get
> for free from Microsoft via GitHub.
>
> So the choice is actually "Be able to process 20+ jobs in parallel that
> take 15-20 minutes each" vs. "Be able to process 3 or so at a time
> (roughly one PR/push) in 5 minutes".
> So realistically, unless we also pay for an actual swarm of runners
> ourselves(which would cost 10k or more a year while being idle 95% of
> the time) the total turnaround time including wait for a free runner is
> probably still better with more of the smaller runners than less of the
> big ones.
>
> It'd also make it a lot more pressing to think about every single CI job
> we add, vs. having a bit of leeway due to the over-abundance of runners.

I agree with Timo, 15-20 minutes is not that bad for a CI job. Faster
is ofcourse better, but I would start worrying about it if we actually
start noticing the wait time. IMHO it is better to scale to more
parallel jobs instead of less faster ones.

Also on the topic of CI, I suspect we might add more jobs as needed.
Probably depends on priorities, to not overload existing runners
capacity.

I was wondering about integrating CIFuzz. Which basically runs fuzzers
based on coverage data that are affected by the PR changes. And runs
fuzzing for N minutes. This would allow it to catch trivial issues,
quickly and without the need for turnaround until the issue is
reported in the oss-fuzz infra. It uses existing corpus to seed the
fuzzing, so it should be good at catching things that are not "hidden"
too deep. Though this brings challenges, because it likely would need
a separate pool of workers to not starve and build jobs. And not sure
how well it would work in practice. For example if change touches too
many fuzzers, it wouldn't manage to run them in the time limit of say
20-30 minutes, which is probably the reasonable target. Though all
ffmpeg fuzers are specialized for codes, so codec specific code should
be fine to test. This is burning CI minutes, but if we have some to
spare, say from GH workers, it could help to weed out issues quickly.
Thoughts?

- Kacper
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".