Using Argparse and Json together

Question

I am a beginner to Python.

I wanted to know if Argparse and JSON could be used together. Say, I have variables p,q,r

I could add them to argparse as -

parser.add_argument('-p','--param1',help='x variable', required=True)
parser.add_argument('-q','--param2',help='y variable', required=True)
parser.add_argument('-r','--param3',help='z variable', required=True)

Now suppose I wanted to read the same variables from JSON file, is it possible to do it? So I could input the values either from command line or a JSON file.

JSON input file -

{
    "testOwner": "my name",
    "tests": [
        "test1",
        "test2",
        "test3"
    ],

    "testParameters": {
        "test1": {
            "param1": "0",
            "param2": "20",
            "param3" : "True"
        },

        "test2": {
            "param1": "cc"
        }
    }   
}

Answer 1

The args Namespace from parse_args can be transformed into a dictionary with:

argparse_dict = vars(args)

The JSON values are also in a dictionary, say json_dict. You can copy selected values from one dictionary to the other, or do a whole scale update:

argparse_dict.update(json_dict)

This way the json_dict values over write the argparse ones.

If you want to preserve both, you either need to have different argument (key) names, or the values have to be lists, which you can append or extend. That takes a bit more work, starting with using the correct nargs value in argparse.

---

The revised parser produces, with a test input:

In [292]: args=parser.parse_args('-p one -q two -r three'.split())
In [293]: args
Out[293]: Namespace(param1='one', param2='two', param3='three')
In [295]: args_dict = vars(args)    
In [296]: args_dict
Out[296]: {'param1': 'one', 'param2': 'two', 'param3': 'three'}

The JSON string, when parsed (json.loads?) produces a dictionary like:

In [317]: json_dict
Out[317]: 
{'testOwner': 'my name',
 'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
  'test2': {'param1': 'cc'}},
 'tests': ['test1', 'test2', 'test3']}

I produced this by pasting your display into my Ipython session, but I think the JSON loader produces the same thing

The argparse values could be added with:

In [318]: json_dict['testParameters']['test3']=args_dict
In [319]: json_dict
Out[319]: 
{'testOwner': 'my name',
 'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
  'test2': {'param1': 'cc'},
  'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
 'tests': ['test1', 'test2', 'test3']}

Here I added it as a 3rd test set, taking (by conincidence) a name from the tests list. json_dict['testParameters']['test2']=args_dict would replace the values of test2.

One way to add the args values to the undefined values of 'test2' is:

In [320]: args_dict1=args_dict.copy()    
In [322]: args_dict1.update(json_dict['testParameters']['test2'])
In [324]: json_dict['testParameters']['test2']=args_dict1
In [325]: json_dict
Out[325]: 
{'testOwner': 'my name',
 'testParameters': {'test1': {'param1': '0', 'param2': '20', 'param3': 'True'},
  'test2': {'param1': 'cc', 'param2': 'two', 'param3': 'three'},
  'test3': {'param1': 'one', 'param2': 'two', 'param3': 'three'}},
 'tests': ['test1', 'test2', 'test3']}

I used this version of update to give priority to the 'cc' value in the JSON dictionary.

Answer 2

Turns out to be pretty easy with the following caveats

1. The setup overrides values in config files with values on the command line
2. It only uses default values if options have not been set on the command line nor the settings file
3. It does not check that the settings in the config file are valid

import argparse
import json

parser = argparse.ArgumentParser()

parser.add_argument('--save_json',
    help='Save settings to file in json format. Ignored in json file')
parser.add_argument('--load_json',
    help='Load settings from file in json format. Command line options override values in file.')

args = parser.parse_args()

if args.load_json:
    with open(args.load_json, 'rt') as f:
        t_args = argparse.Namespace()
        t_args.__dict__.update(json.load(f))
        args = parser.parse_args(namespace=t_args)

# Optional: support for saving settings into a json file
if args.save_json:
    with open(args.save_json, 'wt') as f:
        json.dump(vars(args), f, indent=4)

Answer 3

Given that your JSON file contains a dict of the form:

d = {"name": ["-x", "--xvar"], "help": "Help message", "required": True}

After creating the parser you could "unpack" the dict like so:

parser = argparse.ArgumentParser()
parser.add_argument(*(d.pop("name")), **d) 
# Put the 'name' as name/flag and then unpack the rest of
# the dict as the rest of the arguments
parser.parse_args("--xvar 12".split())
>>> Namespace(xvar='12')

However this forces you to maintain the dict keys to fit the arguments name of the method add_arguments. You also do not have a simple/straight forward way of using more advance behaviors like using the action, type, choices arguments.

Also you would have to change the form of your dict to contain the various arguments you want to use. One solution would be to have the name/flag as the key of the dict in a tuple and the arguments would be a dict:

d = {("-x", "--xvar"): {"help": "Help message for x", "required": True}, 
     ("-y", "--yvar"): {"help": "Help message for y", "required": True}}
for names, args in d.iteritems():
    parser.add_argument(*names, **args) # Use a similar unpacking 'magic' as the first example
parser.parse_args("-x 12 --yvar 42".split())
>>> Namespace(xvar='12', yvar='42')

EDIT Given the comments from the OP it looks like he wants to parse values taken from a JSON file.

d = {"-x": "12", "-y": "42"}
args = []
for item in d.items():
    args.extend(item)
parser.parse_args(args)
>>> Namespace(xvar='12', yvar='42')

EDIT 2

Looking at the argparse documentation this paragraph maybe somewhat relevant.