18

I am using Popen function from the subprocess module to execute a command line tool:

subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)

The tool I am using takes a list of files that it then processes. In some cases, this list of files can be very long. Is there a way to find the max length that the args parameter can be? With a large number of files being passed to the tool, I am getting the following error:

Traceback (most recent call last):
  File "dump_output_sopuids.py", line 68, in <module>
    uid_map = create_sopuid_to_path_dict_dcmdump(dicom_files)
  File "dump_output_sopuids.py", line 41, in create_sopuid_to_path_dict_dcmdump
    dcmdump_output = subprocess.Popen(cmd,stdout=subprocess.PIPE).communicate(0)[0]
  File "c:\python26\lib\subprocess.py", line 621, in __init__
    errread, errwrite)
  File "c:\python26\lib\subprocess.py", line 830, in _execute_child
    startupinfo)
WindowsError: [Error 206] The filename or extension is too long

Is there a general way to find this max length? I found the following article on msdn: Command prompt (Cmd. exe) command-line string limitation but I don't want to hard code in the value. I would rather get the value at run time to break up the command into multiple calls.

I am using Python 2.6 on Windows XP 64.

Edit: adding code example

paths = ['file1.dat','file2.dat',...,'fileX.dat']
cmd = ['process_file.exe','+p'] + paths
cmd_output = subprocess.Popen(cmd,stdout=subprocess.PIPE).communicate(0)[0]

The problem occurs because each actual entry in the paths list is usually a very long file path AND there are several thousand of them.

I don't mind breaking up the command into multiple calls to process_file.exe. I am looking for a general way to get the max length that args can be so I know how many paths to send in for each run.

Jesse Vogt
  • 15,601
  • 14
  • 56
  • 71
  • could you provide an example value of what you provide for args? – gurney alex Mar 04 '10 at 17:56
  • I'm quite late to the party but I want to add that I got the same error due to my PATH environment variable becoming too long after adding many entries. – RedX Dec 21 '15 at 12:58

2 Answers2

12

If you're passing shell=False, then Cmd.exe does not come into play.

On windows, subprocess will use the CreateProcess function from Win32 API to create the new process. The documentation for this function states that the second argument (which is build by subprocess.list2cmdline) has a max length of 32,768 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.

Given your example, I suggest providing a value for executable (args[0]) and using args for the first parameter. If my reading of the CreateProcess documentation and of the subprocess module source code is correct, this should solve your problem.

[edit: removed the args[1:] bit after getting my hands on a windows machine and testing]

Jay Conrod
  • 27,875
  • 16
  • 93
  • 109
gurney alex
  • 12,461
  • 4
  • 40
  • 55
  • I am not sure if I follow your suggestion regarding using args[1:] for the first parameter. I have updated my question with a code example. +1 for the link and tip on CreateProcess – Jesse Vogt Mar 04 '10 at 18:55
  • I tried this but am still hitting a limit: subprocess.Popen(cmd[1:] + paths,executable=cmd[0],stdout=subprocess.PIPE). For now I am using 32000 as the limit for the command length and calling my command multiple times and collecting all the output. I would like to be able to not have 32000 hard coded in but get that value from the environment. – Jesse Vogt Mar 04 '10 at 20:09
  • Well as mentioned in the doc I quoted, the 32768 limit is hard coded in the CreateProcess primitive (thats the upper limit for 16bit signed integers, i.e. 2**15). As list2cmd will add quotes and spaces when building the command line, you will hit that limit before sum([len(a) for a in args]) reaches 2**15. Isn't there a way of using wildcards to pass your arguments to the executable? (wildcards are generally processed by the executable under windows) – gurney alex Mar 05 '10 at 08:11
  • Good point. I was taking spaces into account but had not thought of quotes. I was using 32000 as my hard coded limit which must have left enough room for the spaces. In practice I saw ~190 files fitting for each run (total of about 4k files). Some are in the same directory but most are spread across multiple directories. For now I am just leaving the hard coded limit in. – Jesse Vogt Mar 05 '10 at 12:35
  • Since there has not been too much activity on this question, I am going to accept your answer since you were able to find a good limit. I would have liked to have been able to find a way to more generally get the limit from the OS but this will work. Thanks for the help! – Jesse Vogt Mar 05 '10 at 12:36
  • 1
    Well the limit is hard coded on Windows to 2**15, and that's probably the case on 64bit versions of that OS. On posix system, no limits, except your RAM : Popen uses execvp or execvpe which uses a NULL terminated array of char* for arguments without size constraints. – gurney alex Mar 05 '10 at 14:15
  • @gurneyalex "No limits under POSIX" is not true; see my answer now. – tripleee Mar 02 '19 at 16:51
2

For Unix-like platforms, the kernel constant ARG_MAX is defined by POSIX. It is required to be at least 4096 bytes, though on modern systems, it's probably a megabyte or more.

On many systems, getconf ARG_MAX will reveal its value at the shell prompt.

The shell utility xargs conveniently allows you to break up a long command line. For example, if

python myscript.py *

fails in a large directory because the list of files expands to a value whose length in bytes exceeds ARG_MAX, you can work around it with something like

printf '%s\0' * |
xargs -0 python myscript.py

(The option -0 is a GNU extension, but really the only completely safe way to unambiguously pass a list of file names which could contain newlines, quoting characters, etc.) Maybe also explore

find . -maxdepth 1 -type f -exec python myscript.py {} +

Conversely, to pass a long list of arguments to subprocess.Popen() and friends, something like

p = subprocess.Popen(['xargs', '-0', 'command'],
    stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    stderr=subprocess.PIPE)
out, err = p.communicate('\0'.join(long_long_argument_list))

... where in most scenarios you should probably avoid raw Popen() and let a wrapper function like run() or check_call() do most of the work:

r = subprocess.run(['xargs', '-0', 'command'],
    input='\0'.join(long_long_argument_list),
    universal_newlines=True)
out = r.stdout

subprocess.run() supports text=True in 3.7+ as the new name of universal_newlines=True. Older Python versions than 3.5 didn't have run, so you need to fall back to the older legacy functions check_output, check_call, or (rarely) call.

tripleee
  • 158,107
  • 27
  • 234
  • 292
  • Perhaps see also https://stackoverflow.com/a/51950538/874188 for a number of issues and amplifications around `subprocess` on U*x platforms. – tripleee Mar 02 '19 at 16:46