How to convert output from subprocess to list of dictionaries

Question

For one of my task to call a python script and get the output from the script and do some operations on that. I have written a below code.

import subprocess
import json
chief_complaint = """ "Eernise, Randell is an 64-year-old male" """
command= """python predict_icd.py --cc """+str(chief_complaint)
command_proc=subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
print("response - ",command_proc.stdout.read())

The output that I am getting of above code is

command_proc.stdout.read() -  b"[{'text': 'CHA', 'start': 51, 'end': 54, 'code': ''}, {'text': 'CHA', 'start': 221, 'end': 224, 'code': ''}, {'text': 'dog', 'start': 286, 'end': 289, 'code': ''}, {'text': 'Schizophrenia', 'start': 351, 'end': 364, 'code': 'F20'}, {'text': 'chronic pain', 'start': 366, 'end': 378, 'code': 'G89.29'}, {'text': 'chest pain', 'start': 599, 'end': 609, 'code': 'R07.89'}, {'text': 'palpitations', 'start': 613, 'end': 625, 'code': 'R00.2'}, {'text': 'cough', 'start': 829, 'end': 834, 'code': 'R05'}, {'text': 'wheezing', 'start': 836, 'end': 844, 'code': 'R06.2'}, {'text': 'pain', 'start': 966, 'end': 970, 'code': ''}, {'text': 'schizophrenia', 'start': 1078, 'end': 1091, 'code': 'F20'}]\n"

I want to convert this output into list of dictionaries to perform some action. if there is any alternative way to call the script and get the output please suggest the better way.

tripleee · Accepted Answer · 2021-10-25T14:39:06.360

The absolutely best solution is to not run Python as a subprocess of itself. Instead, refactor predict_icd.py so you can import it and call its functions directly. See What is the best way to call a script from another script?

Your second best option is to change predict_icd.py so that it prints its results in a standard format like JSON. Then you can call it from any tool, not just Python.

The distant third option is to go with what you have and use ast.literal_eval() to parse out the results. I have also refactored to avoid the pesky and unnecessary shell=True and fix the orphaned Popen.

import subprocess
# import json  # unused import
from ast import literal_eval

chief_complaint = """ "Eernise, Randell is an 64-year-old male" """
command_proc = subprocess.run(
    ['python', 'predict_icd.py', '--cc', chief_complaint],
    capture_output=True, text=True, check=True)
response = literal_eval(command_proc.stdout)
print("response = ", response)

In some more detail, Popen will only start a process; you have to take care of it during its lifetime if you do that - but subprocess.run() already takes care of all this plumbing for you. Like its documentation says, avoid Popen when the higher-level functions from the standard library already do what you want.

Notice also the text=True which tells Python to send and receive text, not bytes to and from the subprocess.

Thanks for answering. I am using python3.6 and using your code, I got 2 errors 1st - TypeError: __init__() got an unexpected keyword argument 'capture_output'....2nd - TypeError: __init__() got an unexpected keyword argument 'text' — Kritik Alawat, Oct 25 '21 at 16:34
The keyword arguments `capture_output` and `text` are a bit newer than that. If you are really stuck on 3.6, you'll have to revert to using explicit `stdout=subprocess.PIPE` etc and switch to the legacy synonym `universal_newlines=True` instead of `text=True` ... still `run` should be a significant improvement over an incomplete ad-hoc reimplementation over `Popen`. You'll notice that 3.6 is slated to go EOL at the end of 2021. — tripleee, Oct 25 '21 at 16:36
And of course, my primary recommendation remains to not run Python as a subprocess of itself. — tripleee, Oct 25 '21 at 16:39

How to convert output from subprocess to list of dictionaries

1 Answers1