-1

How to parallelly call OS.system() and wait them to finish their work? What's more, how to limit the max number of OS.system being parallelly called?

For example, to parallelly run Os.system('cd ~; ./run_tests ' + str(args_for_binary_program)) .

OS.system() is not the one that must be used.And if you have a better method, please let me know.

The CPU has 12 cores. And I am using Python3.8.

UPDATE: Thanks to all of the people who replied to this post. The similar posts mentioned in the comments do not clearly talk abot how to limit the max number of OS.system are parallely running.

For example, I need to run a specific binary program 100 times, but I must guarantee that never more than 5 programs are running at the same time. In other words, I need to know when each of the program finishes its work.

John
  • 1,927
  • 6
  • 18
  • What you ask is already provided by the [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) package. There's *much* more to parallelism than starting multiple background processes. – Panagiotis Kanavos May 09 '22 at 07:06
  • Does this answer your question? [how to run several executable using python?](https://stackoverflow.com/questions/9724499/how-to-run-several-executable-using-python) – MisterMiyagi May 09 '22 at 07:07
  • A similar question was asked earlier to which I provided this answer which you could easily adapt:- https://stackoverflow.com/questions/72167457/how-do-i-limit-the-number-of-processes-that-could-be-run-at-the-same-time-with-m/72167792#72167792 – Albert Winestein May 09 '22 at 07:08
  • Does this answer your question? [Python: execute cat subprocess in parallel](https://stackoverflow.com/questions/23611396/python-execute-cat-subprocess-in-parallel) – MisterMiyagi May 09 '22 at 07:09
  • @MisterMiyagi Parcially, indeed. How how to limit the max number of OS.system being parallelly called? – John May 09 '22 at 07:28
  • Does this answer your question? [Control the number of subprocesses using to call external commands in python](https://stackoverflow.com/questions/9808714/control-the-number-of-subprocesses-using-to-call-external-commands-in-python) – MisterMiyagi May 09 '22 at 07:54
  • Unrelated: Why use `cd ~` ? `cd` takes you to your home directory and is half the length. – Mark Setchell May 09 '22 at 08:03
  • If you run `cd` it changes to your home directory and works in all circumstances in all shells and takes 2 characters. If you run `cd ~` it takes 4 characters and may not work in all circumstances, so it seems a poorer option to me. I was inquiring if you knew of some benefit. – Mark Setchell May 09 '22 at 08:25
  • If you want to run no more than N child processes in parallel, just don't start more than N of them. – n. 1.8e9-where's-my-share m. May 09 '22 at 08:29
  • @MarkSetchell There is no special reason for why use `cd ~` other than `cd`. I just want to **explicitly** to inform the viewer that I need to change to my home directory. ***"may not work in all circumstances"***. I am a little interested in under what circumstances `cd ~` may does not work. :) – John May 09 '22 at 08:30
  • @n.1.8e9-where's-my-sharem. That's really a bad idea, indeed. For example, I need to run the program 100 times, but I must guarantee that never more than 5 programs are running at the same time. In other words, I need to know when each of the program finishes its work. – John May 09 '22 at 08:32
  • "I need to know when each of the program finishes its work" Yes, this is a basic functionality of running a job in background and you need to know how to do that regardless of whether you restrict the number of jobs or not. [See e.g.](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.sentinel) – n. 1.8e9-where's-my-share m. May 09 '22 at 09:11
  • @n.1.8e9-where's-my-sharem. I carefully read the document linked with your comment. As far as I can see, I still need to call `os.system()`( i.e. `multiprocessing.Process(Os.system('cd ~; ./run_tests ' + str(args_for_binary_program)))`). Am I right? – John May 09 '22 at 09:52
  • No, you are not. You don't need to call Os.system. There is no mention of Os.system on that page, and there are various examples of how to construct a process. – n. 1.8e9-where's-my-share m. May 09 '22 at 09:56
  • @n.1.8e9-where's-my-sharem. Which one is better for this case, `subprocess ` or `multiprocessing`? – John May 09 '22 at 12:03
  • I'm sorry, I now understand why you were asking about os.system inside multiprocessing. If you want to run other programs in parallel, this is indeed what you might do, because multiprocessing only lets you run pieces of *your own code* in parallel. [See this](https://stackoverflow.com/questions/13606867/what-is-the-difference-between-multiprocessing-and-subprocess) I would say in this context subprocess is better. – n. 1.8e9-where's-my-share m. May 09 '22 at 12:25

0 Answers0