I have multiple scripts (script1.py, script2.py, script3.py, ...) that ingest data into an elastic cluster. I created a file called main.py that will call each file on a subprocess in order to ingest data coming from each script. All python script generate fake data.
Content of script1.py:
#initialize some variables
while True:
# generate_random_data()
# insert_to_elk()
# time.sleep(1)
Content of script2.py:
#initialize some variables
while True:
# generate_random_data()
# insert_to_elk()
# time.sleep(1)
Content of script3.py:
#initialize some variables
while True:
# generate_random_data()
# insert_to_elk()
# time.sleep(1)
I tried the following with main.py:
from subprocess import *
p = Popen(['python', 'script1.py'], shell=True, stdin=PIPE, stdout=PIPE)
output = p.communicate()
print(output[0])
#run secondary script 2
p = Popen(['python', 'script2.py'], shell=True, stdin=PIPE, stdout=PIPE)
output = p.communicate()
print(output[0])
#run secondary script 3
p = Popen(['python', 'script3.py'], shell=True, stdin=PIPE, stdout=PIPE)
output = p.communicate()
print(output[0])
# run other scripts...
I tried the following too:
subprocess.run("python script1.py & python script2.py & python script3.py & ...", shell=True)
When I check indexes from elasticsearch I only see the index created by script1.py, same thing if I execute script2.py first. What I expect is getting all indexes created by each script, how can I do that please?
I am developing on Windows 10 OS using python 3.6.
EDIT 1:
In this question, I found that communicate() is a blocking method, commenting it result in executing each script independently but can't get the prints from each script, which result the following error:
File "script1.py", line 47, in <module>
print(json.dumps(json_source, indent=4, sort_keys=False))
OSError: [Errno 22] Invalid argument
Traceback (most recent call last):
File "script2.py", line 49, in <module>
print(json.dumps(json_source, indent=4, sort_keys=False))
OSError: [Errno 22] Invalid argument
Commenting both print(..) in each script and communicate() result in creating indexes from each script. How can I still print from each script in the current shell or is it impossible?
