85

I have a bash script that sets an environment variable an runs a command

LD_LIBRARY_PATH=my_path
sqsub -np $1 /homedir/anotherdir/executable

Now I want to use python instead of bash, because I want to compute some of the arguments that I am passing to the command.

I have tried

putenv("LD_LIBRARY_PATH", "my_path")

and

call("export LD_LIBRARY_PATH=my_path")

followed by

call("sqsub -np " + var1 + "/homedir/anotherdir/executable")

but always the program gives up because LD_LIBRARY_PATH is not set.

How can I fix this?

Thanks for help!

(if I export LD_LIBRARY_PATH before calling the python script everything works, but I would like python to determine the path and set the environment variable to the correct value)

Matthias 009
  • 1,444
  • 3
  • 15
  • 18
  • 1
    possible duplicate of [change current process environment](http://stackoverflow.com/questions/1178094/change-current-process-environment) – S.Lott Dec 03 '11 at 04:00
  • 1
    @S.Lott: can you please explain how I can apply that thread to my problem? (cause I do not understand it) – Matthias 009 Dec 03 '11 at 15:49
  • @S.Lott (addentum): in particular the excepted answer in that thread starts with "the reason os.environ["LD_LIBRARY_PATH"] does not work" and in my case it works – Matthias 009 Dec 03 '11 at 16:09

4 Answers4

88

bash:

LD_LIBRARY_PATH=my_path
sqsub -np $1 /path/to/executable

Similar, in Python:

import os
import subprocess
import sys

os.environ['LD_LIBRARY_PATH'] = "my_path" # visible in this process + all children
subprocess.check_call(['sqsub', '-np', sys.argv[1], '/path/to/executable'],
                      env=dict(os.environ, SQSUB_VAR="visible in this subprocess"))
jfs
  • 374,366
  • 172
  • 933
  • 1,594
  • subprocess.check_call(command, env=os.environ) 'module' object has no attribute 'check_call', I'll try using call for now – Matthias 009 Dec 03 '11 at 15:56
  • I would avoid a such solution, because it mutates the current process environment. Better pass a copy of it to the child process. – rdesgroppes Feb 21 '15 at 11:53
  • 1
    the explanation is critical here: visible in this process + all children – debuti Jun 04 '18 at 15:38
  • 1
    @rdesgroppes: I've added an option (`env`) where the updated environment is visible only in the child process (`$SQSUB_VAR` in the current process' environment remains unaffected) – jfs Apr 13 '22 at 13:21
  • Could this applied for the `HOME` path as well? – alper Apr 13 '22 at 21:58
19

You can add elements to your environment by using

os.environ['LD_LIBRARY_PATH'] = 'my_path'

and run subprocesses in a shell (that uses your os.environ) by using

subprocess.call('sqsub -np ' + var1 + '/homedir/anotherdir/executable', shell=True)
Manbeardo
  • 504
  • 4
  • 7
15

There are many good answers here but you should avoid at all cost to pass untrusted variables to subprocess using shell=True as this is a security risk. The variables can escape to the shell and run arbitrary commands! If you just can't avoid it at least use python3's shlex.quote() to escape the string (if you have multiple space-separated arguments, quote each split instead of the full string).

shell=False is always the default where you pass an argument array.

Now the safe solutions...

Method #1

Change your own process's environment - the new environment will apply to python itself and all subprocesses.

os.environ['LD_LIBRARY_PATH'] = 'my_path'
command = ['sqsub', '-np', var1, '/homedir/anotherdir/executable']
subprocess.check_call(command)

Method #2

Make a copy of the environment and pass is to the childen. You have total control over the children environment and won't affect python's own environment.

myenv = os.environ.copy()
myenv['LD_LIBRARY_PATH'] = 'my_path'
command = ['sqsub', '-np', var1, '/homedir/anotherdir/executable']
subprocess.check_call(command, env=myenv)

Method #3

Unix only: Execute env to set the environment variable. More cumbersome if you have many variables to modify and not portabe, but like #2 you retain full control over python and children environments.

command = ['env', 'LD_LIBRARY_PATH=my_path', 'sqsub', '-np', var1, '/homedir/anotherdir/executable']
subprocess.check_call(command)

Of course if var1 contain multiple space-separated argument they will now be passed as a single argument with spaces. To retain original behavior with shell=True you must compose a command array that contain the splitted string:

command = ['sqsub', '-np'] + var1.split() + ['/homedir/anotherdir/executable']
Thomas Guyot-Sionnest
  • 1,877
  • 19
  • 15
-2

Compact solution (provided you don't need other environment variables):

subprocess.check_call(
    'sqsub -np {} /homedir/anotherdir/executable'.format(var1).split(),
    env=dict(LD_LIBRARY_PATH=my_path))

Using the env command line tool:

subprocess.check_call('env LD_LIBRARY_PATH=my_path sqsub -np {} /homedir/anotherdir/executable'.format(var1).split())
rdesgroppes
  • 702
  • 9
  • 9
  • the first command won't work (you should pass a list, not string here and the wrong environment (sqsub won't be found)). The second command may break if `var1` contains shell metacharacters such as `'$\`!`. It is a bad practice to use `shell=True` if you don't need it e.g., if `var1` comes form an external source then the command injection is possible. – jfs Feb 19 '15 at 01:48
  • @J.F.Sebastian thank you for the very valid points. I addressed them in some way. Sure, splitting won't work if one has whitespaces in the path. – rdesgroppes Feb 20 '15 at 06:56
  • my comment says *nothing* about splitting. It is not an appropriate way to fix your answer: it doesn't fix the PATH issue. It introduces other issues. [My answer shows how to pass the list correctly](http://stackoverflow.com/a/8365493/4279). – jfs Feb 20 '15 at 07:06
  • @J.F.Sebastian Right, your comment says nothing about splitting... so what? Mine does, what's wrong with it? That said, both the compact and env-based solutions do work. Give them a try: they address the need for passing LD_LIBRARY_PATH. – rdesgroppes Feb 21 '15 at 12:00