Repository navigation
utils: Add usable_cpu_count - #296
Conversation
We can remove this and rely only on `os.sched_getaffinity` when it supports the same OSes as `psutil`. See issue brainiak#291. See also https://bugs.python.org/issue26692.
| raw_data2=None, | ||
| voxel_unit=64, | ||
| process_num=4, | ||
| process_num=None, |
There was a problem hiding this comment.
The process_num was intentionally set to be 4, a small number, instead of the "usable cpus" in the system because on a cluster node, the "usable cpus" could be over 30, which will likely blow up the memory for having the process duplicate so many copies.
There was a problem hiding this comment.
The memory for forked processes is allocated copy-on-write. Unless _cross_validation_for_one_voxel allocates a lot of data, it should be fine to have lots of processes in the pool.
There was a problem hiding this comment.
I understand this. Previously we observed memory blowout when process_num was set to be "usable cpus". So I would say yes...
yidawang
left a comment
There was a problem hiding this comment.
Looks good to me for the FCMA part as well as utils.py
|
looks good. |
|
Just wanted to give a heads up that I gave it a test run on della and it seems resolve the issue. |
We can remove this and rely only on
os.sched_getaffinitywhen itsupports the same OSes as
psutil. See issue #291.See also https://bugs.python.org/issue26692.