Searchlight: Default pool_size=None might not be a good idea for HPC systems

Hi,

I have been trying to track down reasons for large memory requirements of the searchlight code on research computing clusters at Princeton. I have done the testing on della but this would probably be a common problem on many HPC systems. It seems the default value for the argument pool_size in run_searchlight function is None. This will cause the worker threads to utilize all cores on a machine by default. On a personal machine this may be desirable but on a shared HPC resource this may not be a good idea since a user might not be provisioned all the cores on the node. When trying to run the searchlight example on della this was causing large memory requirements because the default behavior was to try and spawn a process for each core on the machine. In addition, if the code was run under srun with default arguments (a typical method for spawning MPI code under slurm schedulers), a process is spawned per rank, who in turn tries to spawn a processes for each core on the machine. This was the root cause of the large memory errors on della I believe.

I suggest a default value of 1 for pool_size or maybe clearer documentation if the default behavior is desirable on desktop\laptops. For a longer term solution, it may be preferable to avoid using multiprocessing at all and use MPI for the same node parallelization. If I am reading the code correctly then map is being used on each node to distribute work to processes. However, this is causing the blocks to be pickled and sent to each process since they are passed as arguments and not setup in shared memory. This seems basically equivalent to an MPI process. If it was rewritten using MPI this way it would have the added benefit of avoiding process fork which can cause trouble on certain HPC setups that use OpenMPI implementations. Della is one such case it seems but I am sure there are others.

Dave Turner

@davidt0x

@davidt0x, thanks for the effort to get BrainIAK going on Princeton clusters.

The current implementation gives good performance on laptops without extra configuration and without MPI. At the same time, it does not penalize cluster performance, if the properly configured. Having said that, I think there is room for discussion, so I am inviting people to share their thoughts.

Would it be possible for the system to test the value of cat /proc/$$/status | grep Cpus_allowed_list or similar at startup (if it exists) to see if they have any restrictions on cpu count?]
It seems like reading the Cpus_allowed value may yield better results, it returns a mask value that can be used to calculate how many CPUs are allowed to be used. Documentation of the mask value can be found here: https://linux.die.net/man/7/cpuset

Restrict multiprocessing to available CPUs

@Wildcarde

@Wildcarde, what do you think about PR #293?

Seems like it should cover it based on the docs.

Restrict multiprocessing to available CPUs (#293)

I looked at this patch and how it would work on della. It doesn't seem like it will work because apparently os.sched_getaffinity is not available on all systems. Della is apparently one of those systems.

>>> import os
>>> 'sched_getaffinity' in dir(os)
False

I did a little research and this seems like a commonly occurring issue. The library psutil and the following code snippet do seem to work on della at least:

>>> import psutil
>>> proc = psutil.Process()
>>> len(proc.cpu_affinity())
3

This is with three cores provisioned so it seems to be working. However, this requires another dependency which may be undesirable. What do you all think?

Maybe something like this is even better:

import os

 try:
     import psutil
 except ImportError:
     psutil = None

 def get_cpu_affinity():
     if hasattr(os, 'sched_getaffinity'):
         return len(os.sched_getaffinity(0))

     if psutil is not None:
         proc = psutil.Process()
         # cpu_affinity() is only available on Linux, Windows and FreeBSD
         if hasattr(proc, 'cpu_affinity'):
             return len(proc.cpu_affinity())

     return None

 print(get_cpu_affinity())

I adapted this from code I found in the Python perf module https://github.com/haypo/perf/blob/master/perf/_collect_metadata.py

Dave

Finally, maybe it could print a warning if it is defaulting to None.

The only problem I see with Psutil is that it does not offer wheels. This will be a problem when we start offering BrianIAK wheels (see issue #286).

What OS do you have on Della?

Della is a Centos 7 based system.

What Python installation are you using. I tried on a CentOS 7 machine and the following works on both Python from EPEL and Anaconda:

python3 -c "import os; print(len(os.sched_getaffinity(0)))"

Anaconda as well. Very strange, it is hard to find anything in the documentation regarding where and why it is missing.

You are using Python 3, not 2, right?

yeah, i tried with 3.6, it doesn't work in either 3 or 2.

utils: Add usable_cpu_count

See PR #296.

utils: Add usable_cpu_count (#296)

The original problem was solved (see #296 (comment)).

Navigation Menu

Repository navigation

Searchlight: Default pool_size=None might not be a good idea for HPC systems #291

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Searchlight: Default pool_size=None might not be a good idea for HPC systems #291

Description

Activity

mihaic commented on Nov 6, 2017

Wildcarde commented on Nov 7, 2017

mihaic commented on Nov 7, 2017

Wildcarde commented on Nov 7, 2017

davidt0x commented on Nov 10, 2017

davidt0x commented on Nov 10, 2017

davidt0x commented on Nov 10, 2017

mihaic commented on Nov 10, 2017

Wildcarde commented on Nov 10, 2017

mihaic commented on Nov 10, 2017

davidt0x commented on Nov 10, 2017

mihaic commented on Nov 10, 2017

davidt0x commented on Nov 10, 2017

mihaic commented on Nov 10, 2017

mihaic commented on Nov 13, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions