Is it possible to configure Linux OpenMPI v1.6.5 to use multiple working directories on a single node?

Question

I am running some quantum chemical calculations on a refurbished PowerEdge rack server, with dual quad-core Xeons, 16GB RAM, and a single 500GB SATA HDD. OS is vanilla Debian Wheezy. For one of the calculation types I'm running, I think I'm becoming disk-I/O-limited and so I installed a spare 160GB HDD I had on hand to try to ease the bottleneck.

In order to make use of the additional drive, though, I need to be able to tell OpenMPI to use multiple working directories. I know the --wdir commandline flag exists, but the documentation makes it look like it only accepts a single value. I unsuccessfully tried a hacky workaround:

1. Creating a new user scratch
2. Mounting the new 160GB drive as /home/scratch/
3. Setting up /home/bskinn/.ssh/config to default to user bskinn when SSH-ing to localhost and to default to user scratch when SSh-ing to the server's internal 192.168.x.x address
4. Creating appropriate RSA SSH keys, adding to /home/bskinn/.ssh/authorized_keys and /home/scratch/.ssh/authorized_keys, and SSH-ing into each to get the host information stored for auto-acceptance
5. Adding localhost and 192.168.x.x to my ompi_hosts file and passing to OMPI with --hostfile /home/bskinn/ompi_hosts

If I ssh localhost I get logged in as bskinn, and if I ssh 192.168.x.x I get logged in as scratch, just like I want. However, while OpenMPI doesn't throw any errors, it appears to just ignore the ompi_hosts file and just runs all processes locally.

So: Is there a way to get OpenMPI 1.6.5 to use multiple working directories?

ADDENDUM: I'm running ORCA computations. ORCA is distributed as pre-compiled, statically-linked binaries. In particular, the OMPI calls are hard-coded into said binaries and not accessible to me. So, I can't intercept what commands are passed to OMPI, except perhaps by some sleight-of-hand wherein I relocate the ORCA binaries and replace them with shell scripts. That seems like a lot more work than it's worth.

ORCA does provide the ability to pass through whatever OMPI environment variables I want; I'm only really interested in solutions on this level of complexity.

This is a cross-post from SuperUser -- I won the Tumbleweed badge for it there (yay?), so I'm guessing it's not a good place for it. I don't have rep to migrate it, so apologies if re-posting in this manner is frowned upon.

Answer 1

Instead of launching the executable, why don't you launch a script that changes directories into the one you want and then execs your program? OpenMPI sets $OMPI_COMM_WORLD_RANK in the environment equal to the rank that the task will get within MPI_COMM_WORLD inside the code, so you can do something like

foo.sh:

#!/bin/bash
total_tasks=100 # Or whatever
if [ $OMPI_COMM_WORLD -lt $($total_tasks/2) ]; then
  cd /home/scratch
fi
exec /path/to/my/chemistry/code arg1 arg2 arg3

Then if you make foo.sh executable (chmod +x foo.sh) and mpirun or mpiexec (or whatever) foo.sh, it should put the first half of the ranks with their working directory in /home/scratch and the latter half of the ranks wherever you were when you did the mpirun/exec.

You could also just patch this into your chemistry program if you have the source.

Finally, if these two drives are connected to the same controller, you may or may not get any additional bandwidth. Have you tried profiling your app? What code is it? Are you running it in an out-of-core mode?

Answer 2

If you don't have the source and you can't modify the mpirun/mpiexec command line at all, you're probably going to have a hard time doing things the way you tried. You should profile your runs if you can and verify that I/O is, in fact, your problem. If it is, just moving to a single SSD might be enough to make you some progress, but splitting across drives could be more challenging unless you have enough room (controller ports and physical space) to build a 2-3 drive striped RAID array (RAID 0). You are almost certainly better off this way than trying to explicitly stripe across different mount points. If you don't have the space for 3-4 total drives, but you do have room for two, you'll probably need to completely wipe and reinstall the system with a RAID 0 system partition to put everything on.

*Given the update and the request, I have added this extra answer.