Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
 help / color / mirror / Atom feed
* [FFmpeg-devel] [PATCH] cpu: Limit the number of auto threads in 32 bit builds
@ 2022-09-05  9:30 Martin Storsjö
  2022-09-05 11:45 ` Andreas Rheinhardt
  0 siblings, 1 reply; 4+ messages in thread
From: Martin Storsjö @ 2022-09-05  9:30 UTC (permalink / raw)
  To: ffmpeg-devel

Limit the returned value from av_cpu_count to sensible amounts
in 32 bit builds.

This chosen limit, 64, is somewhat arbitrary - a 32 bit process
is capable of creating much more than 64 threads. But in many
cases, multiple parts of the encoding pipeline (decoder, filters,
encoders) all create a pool of threads, auto sized according to the
number of cores.

In one failing test, the process had managed to create 506 threads
before a pthread_create call failed.

In the current set of fate tests, the filter-lavd-scalenorm test
seems to be the limiting factor; in a 32 bit build (arm linux,
running on an aarch64 kernel), it starts failing with an auto
thread count somewhere around 85. Therefore, pick the maximum
with some margin below this.

This fixes running fate without any manually set number of threads
in 32 bit builds on machines with huge numbers of cores.
---
 libavutil/cpu.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/libavutil/cpu.c b/libavutil/cpu.c
index 0035e927a5..094bd71d3d 100644
--- a/libavutil/cpu.c
+++ b/libavutil/cpu.c
@@ -233,6 +233,12 @@ int av_cpu_count(void)
     nb_cpus = sysinfo.dwNumberOfProcessors;
 #endif
 
+#if SIZE_MAX <= UINT32_MAX
+    // Avoid running out of memory/address space in 32 bit builds, by
+    // limiting the number of auto threads.
+    nb_cpus = FFMIN(nb_cpus, 64);
+#endif
+
     if (!atomic_exchange_explicit(&printed, 1, memory_order_relaxed))
         av_log(NULL, AV_LOG_DEBUG, "detected %d logical cores\n", nb_cpus);
 
-- 
2.25.1

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [FFmpeg-devel] [PATCH] cpu: Limit the number of auto threads in 32 bit builds
  2022-09-05  9:30 [FFmpeg-devel] [PATCH] cpu: Limit the number of auto threads in 32 bit builds Martin Storsjö
@ 2022-09-05 11:45 ` Andreas Rheinhardt
  2022-09-05 11:50   ` Martin Storsjö
  0 siblings, 1 reply; 4+ messages in thread
From: Andreas Rheinhardt @ 2022-09-05 11:45 UTC (permalink / raw)
  To: ffmpeg-devel

Martin Storsjö:
> Limit the returned value from av_cpu_count to sensible amounts
> in 32 bit builds.
> 
> This chosen limit, 64, is somewhat arbitrary - a 32 bit process
> is capable of creating much more than 64 threads. But in many
> cases, multiple parts of the encoding pipeline (decoder, filters,
> encoders) all create a pool of threads, auto sized according to the
> number of cores.
> 
> In one failing test, the process had managed to create 506 threads
> before a pthread_create call failed.
> 
> In the current set of fate tests, the filter-lavd-scalenorm test
> seems to be the limiting factor; in a 32 bit build (arm linux,
> running on an aarch64 kernel), it starts failing with an auto
> thread count somewhere around 85. Therefore, pick the maximum
> with some margin below this.
> 
> This fixes running fate without any manually set number of threads
> in 32 bit builds on machines with huge numbers of cores.
> ---
>  libavutil/cpu.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/libavutil/cpu.c b/libavutil/cpu.c
> index 0035e927a5..094bd71d3d 100644
> --- a/libavutil/cpu.c
> +++ b/libavutil/cpu.c
> @@ -233,6 +233,12 @@ int av_cpu_count(void)
>      nb_cpus = sysinfo.dwNumberOfProcessors;
>  #endif
>  
> +#if SIZE_MAX <= UINT32_MAX
> +    // Avoid running out of memory/address space in 32 bit builds, by
> +    // limiting the number of auto threads.
> +    nb_cpus = FFMIN(nb_cpus, 64);
> +#endif
> +
>      if (!atomic_exchange_explicit(&printed, 1, memory_order_relaxed))
>          av_log(NULL, AV_LOG_DEBUG, "detected %d logical cores\n", nb_cpus);
>  

I don't think we should be lying in libavutil/cpu.c. We should instead
limit the number of threads in the functions that actually create
threads based upon the return value of av_cpu_count(); e.g. both
frame_thread_encoder.c (limit 64) as well as pthread_frame.c and
pthread_slice.c (limit 16) already limit these numbers.
lavu/slicethread.c doesn't seem to do so.

- Andreas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [FFmpeg-devel] [PATCH] cpu: Limit the number of auto threads in 32 bit builds
  2022-09-05 11:45 ` Andreas Rheinhardt
@ 2022-09-05 11:50   ` Martin Storsjö
  2022-09-05 11:54     ` Andreas Rheinhardt
  0 siblings, 1 reply; 4+ messages in thread
From: Martin Storsjö @ 2022-09-05 11:50 UTC (permalink / raw)
  To: FFmpeg development discussions and patches

On Mon, 5 Sep 2022, Andreas Rheinhardt wrote:

> Martin Storsjö:
>> Limit the returned value from av_cpu_count to sensible amounts
>> in 32 bit builds.
>> 
>> This chosen limit, 64, is somewhat arbitrary - a 32 bit process
>> is capable of creating much more than 64 threads. But in many
>> cases, multiple parts of the encoding pipeline (decoder, filters,
>> encoders) all create a pool of threads, auto sized according to the
>> number of cores.
>> 
>> In one failing test, the process had managed to create 506 threads
>> before a pthread_create call failed.
>> 
>> In the current set of fate tests, the filter-lavd-scalenorm test
>> seems to be the limiting factor; in a 32 bit build (arm linux,
>> running on an aarch64 kernel), it starts failing with an auto
>> thread count somewhere around 85. Therefore, pick the maximum
>> with some margin below this.
>> 
>> This fixes running fate without any manually set number of threads
>> in 32 bit builds on machines with huge numbers of cores.
>> ---
>>  libavutil/cpu.c | 6 ++++++
>>  1 file changed, 6 insertions(+)
>> 
>> diff --git a/libavutil/cpu.c b/libavutil/cpu.c
>> index 0035e927a5..094bd71d3d 100644
>> --- a/libavutil/cpu.c
>> +++ b/libavutil/cpu.c
>> @@ -233,6 +233,12 @@ int av_cpu_count(void)
>>      nb_cpus = sysinfo.dwNumberOfProcessors;
>>  #endif
>> 
>> +#if SIZE_MAX <= UINT32_MAX
>> +    // Avoid running out of memory/address space in 32 bit builds, by
>> +    // limiting the number of auto threads.
>> +    nb_cpus = FFMIN(nb_cpus, 64);
>> +#endif
>> +
>>      if (!atomic_exchange_explicit(&printed, 1, memory_order_relaxed))
>>          av_log(NULL, AV_LOG_DEBUG, "detected %d logical cores\n", nb_cpus);
>> 
>
> I don't think we should be lying in libavutil/cpu.c.

> We should instead limit the number of threads in the functions that 
> actually create threads based upon the return value of av_cpu_count(); 
> e.g. both frame_thread_encoder.c (limit 64) as well as pthread_frame.c 
> and pthread_slice.c (limit 16) already limit these numbers. 
> lavu/slicethread.c doesn't seem to do so.

Right, that's probably the more sensible thing to do.

For something like lavu/slicethread.c, it probably makes sense with a 
low-ish limit, like 16 or 32 - splitting up an individual frame in more 
slices than that probably doesn't make sense as automatic default, right?

// Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [FFmpeg-devel] [PATCH] cpu: Limit the number of auto threads in 32 bit builds
  2022-09-05 11:50   ` Martin Storsjö
@ 2022-09-05 11:54     ` Andreas Rheinhardt
  0 siblings, 0 replies; 4+ messages in thread
From: Andreas Rheinhardt @ 2022-09-05 11:54 UTC (permalink / raw)
  To: ffmpeg-devel

Martin Storsjö:
> On Mon, 5 Sep 2022, Andreas Rheinhardt wrote:
> 
>> Martin Storsjö:
>>> Limit the returned value from av_cpu_count to sensible amounts
>>> in 32 bit builds.
>>>
>>> This chosen limit, 64, is somewhat arbitrary - a 32 bit process
>>> is capable of creating much more than 64 threads. But in many
>>> cases, multiple parts of the encoding pipeline (decoder, filters,
>>> encoders) all create a pool of threads, auto sized according to the
>>> number of cores.
>>>
>>> In one failing test, the process had managed to create 506 threads
>>> before a pthread_create call failed.
>>>
>>> In the current set of fate tests, the filter-lavd-scalenorm test
>>> seems to be the limiting factor; in a 32 bit build (arm linux,
>>> running on an aarch64 kernel), it starts failing with an auto
>>> thread count somewhere around 85. Therefore, pick the maximum
>>> with some margin below this.
>>>
>>> This fixes running fate without any manually set number of threads
>>> in 32 bit builds on machines with huge numbers of cores.
>>> ---
>>>  libavutil/cpu.c | 6 ++++++
>>>  1 file changed, 6 insertions(+)
>>>
>>> diff --git a/libavutil/cpu.c b/libavutil/cpu.c
>>> index 0035e927a5..094bd71d3d 100644
>>> --- a/libavutil/cpu.c
>>> +++ b/libavutil/cpu.c
>>> @@ -233,6 +233,12 @@ int av_cpu_count(void)
>>>      nb_cpus = sysinfo.dwNumberOfProcessors;
>>>  #endif
>>>
>>> +#if SIZE_MAX <= UINT32_MAX
>>> +    // Avoid running out of memory/address space in 32 bit builds, by
>>> +    // limiting the number of auto threads.
>>> +    nb_cpus = FFMIN(nb_cpus, 64);
>>> +#endif
>>> +
>>>      if (!atomic_exchange_explicit(&printed, 1, memory_order_relaxed))
>>>          av_log(NULL, AV_LOG_DEBUG, "detected %d logical cores\n",
>>> nb_cpus);
>>>
>>
>> I don't think we should be lying in libavutil/cpu.c.
> 
>> We should instead limit the number of threads in the functions that
>> actually create threads based upon the return value of av_cpu_count();
>> e.g. both frame_thread_encoder.c (limit 64) as well as pthread_frame.c
>> and pthread_slice.c (limit 16) already limit these numbers.
>> lavu/slicethread.c doesn't seem to do so.
> 
> Right, that's probably the more sensible thing to do.
> 
> For something like lavu/slicethread.c, it probably makes sense with a
> low-ish limit, like 16 or 32 - splitting up an individual frame in more
> slices than that probably doesn't make sense as automatic default, right?
> 

Yes. E.g. MAX_AUTO_THREADS in pthread_slice.c is 16.

- Andreas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2022-09-05 11:54 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-05  9:30 [FFmpeg-devel] [PATCH] cpu: Limit the number of auto threads in 32 bit builds Martin Storsjö
2022-09-05 11:45 ` Andreas Rheinhardt
2022-09-05 11:50   ` Martin Storsjö
2022-09-05 11:54     ` Andreas Rheinhardt

Git Inbox Mirror of the ffmpeg-devel mailing list - see https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://master.gitmailbox.com/ffmpegdev/0 ffmpegdev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 ffmpegdev ffmpegdev/ https://master.gitmailbox.com/ffmpegdev \
		ffmpegdev@gitmailbox.com
	public-inbox-index ffmpegdev

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git