I am using Python within a PBS-based system and I need to know the number of cpus available to me for use from inside Python. I know I can look it up from bash using qstat but that is not what I am looking for. In particular, I am using the recordlinkage library where I can specify the number of jobs to run in parallel but I am not sure whether I am being able to use all 5 CPUs when I request 5 CPUs and set n_jobs=-1. I face the same issue when running ML algorithms such as Random Forest where I set the n_jobs=-1 but don't know actually how many CPUs are being used.
The compute node I am working on has 64 cores. When I ask for 5 CPUs using the following command,
qsub -I -X -l ncpus=5;mem=10gb
and I open up IPython, I have tried the following:
multiprocessing.cpu_count()
which gives me 64. And also this:
os.cpu_count()
which also gives me 64. And this:
psutil.cpu_count()
and that too gives me 64. I even tried
psutil.cpu_count(logical=False)
That gives me 16. From this I am guessing there are 16 physical cores but due to multi-threading, it shows up as 64 with the other commands? Please correct me if I am wrong.
Finally I tried
len(os.sched_getaffinity(0))
Guess what? Still 64.
Since I requested 5 cpus and not all 64 on my compute node, len(os.sched_getaffinity(0)) should have given me 5. That is what I figured from this post. Any ideas?
