427

I have a command line script that I run with a lot of arguments. I have now come to a point where I have too many arguments, and I want to have some arguments in dictionary form too.

So in order to simplify things I would like to run the script with a settings file instead. I don't really know what libraries to use for the parsing of the file. What's the best practice for doing this? I could of course hammer something out myself, but if there is some library for this, I'm all ears.

A few 'demands':

  • Rather than using pickle I would like it to be a straight forward text file that can easily be read and edited.
  • I want to be able to add dictionary-like data in it, i.e., some form of nesting should be supported.

A simplified pseudo example file:

truck:
    color: blue
    brand: ford
city: new york
cabriolet:
    color: black
    engine:
        cylinders: 8
        placement: mid
    doors: 2
dreftymac
  • 29,742
  • 25
  • 114
  • 177
c00kiemonster
  • 20,805
  • 33
  • 89
  • 129

4 Answers4

269

You can have a regular Python module, say config.py, like this:

truck = dict(
    color = 'blue',
    brand = 'ford',
)
city = 'new york'
cabriolet = dict(
    color = 'black',
    engine = dict(
        cylinders = 8,
        placement = 'mid',
    ),
    doors = 2,
)

and use it like this:

import config
print(config.truck['color'])
oberbaum
  • 2,434
  • 7
  • 36
  • 52
dugres
  • 11,893
  • 8
  • 44
  • 50
  • 92
    This is a pretty bad idea as if you want to allow low-privileged users to be able to change configuration files only, this way you're essentially allowing them to sneak in privileged code. – nikolay Jun 27 '12 at 23:17
  • 199
    Allowing "low-privileged" users to change config for a more privileged program is probably a questionable setup anyway. – XTL Dec 12 '12 at 13:36
  • 2
    This gives no protection against "low-privileged" users changing the configuration. If you import the module at the beginning of a script, change the value of one of the variables and the import the rest of the modules you can modify the configuration values even if you don't have permission to write the configuration file. – Javier Castellanos Jan 22 '14 at 23:37
  • 1
    Perhaps it is not in the current working directory (cwd). In that case, you have to make it visible to Python by either changing the cwd with os.chdir (use os.getcwd() to know where you are) or adding the config file location to the PYTHONPATH (not recommended). Hope this helps. – Chris Apr 25 '14 at 07:06
  • Otherwise, if the config file is in a subdir, you can turn that subdir into a package by placing an empty `__init__.py` file in it. Then you could import your config file with `import subdir.config` – Chris Apr 25 '14 at 07:15
  • 25
    You may also run into issues packaging your project for deployment using a tool such as py2app. The user may not be able to edit the configuration file once it's distributed since it would invalidate the app's signature. – bschwagg Jan 17 '15 at 18:40
  • If using python as a config file, you can use exec() to load it into a dictionary or imp.new_module to turn it into a module. This way the configuration is not in the package and can be placed in a system-standard config location if you prefer (like /etc). For more advanced usage you can also prepopulate the dict you pass to exec with objects that your config file can use as a simple DSL. – Jeremy Jan 15 '16 at 17:46
  • 29
    The main disadvantage with this (otherwise very convenient option) is that ``.py`` files are executable, so any kind of code could be run while trying to load the configuration through ``import``. That's unacceptable from a security standpoint. – Apalala Feb 09 '16 at 23:18
  • 6
    Can't a version of this be done safely with `ast.literal_eval`? https://docs.python.org/3/library/ast.html#ast.literal_eval – André C. Andersen Sep 02 '16 at 21:49
  • 1
    (1) security is not always in issue, it really depends on the project. (2) A problem I found with config files like this is if you need to create them programaticlly. Then it is difficult – Jakobovski Jan 07 '19 at 13:04
  • Note, there is the [Python-like (Python dialect/subset) Starlark language](https://github.com/bazelbuild/starlark), which is intended for this use case of config files (and used by Bazel). The main feature is hermetic execution, i.e. execution cannot access the file system, network, system clock. It is safe to execute untrusted code. – Albert Mar 03 '21 at 13:00
  • Actually `import config` might not always work, see [python - How to import a module given the full path? - Stack Overflow](https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path) – user202729 Oct 24 '21 at 01:14
235

The sample config you provided is actually valid YAML. In fact, YAML meets all of your demands, is implemented in a large number of languages, and is extremely human friendly. I would highly recommend you use it. The PyYAML project provides a nice python module, that implements YAML.

To use the yaml module is extremely simple: 

import yaml
config = yaml.safe_load(open("path/to/config.yml"))
Alan W. Smith
  • 23,261
  • 4
  • 66
  • 92
Benson
  • 22,091
  • 2
  • 39
  • 49
  • 6
    yaml is always something I turn to; the format can be from dead simple to supporting embedded python code, and the standard library does the heavy lifting of parsing and sanitation for you. – Todor Minakov Oct 10 '15 at 05:56
  • 21
    Agreed. For you or users writing YAML, [here is the best YAML reference that I know of](http://camel.readthedocs.org/en/latest/yamlref.html). The [official documentation](http://yaml.org/) is unfortunately a spec aimed at implementers, and nothing else, but Eevee's guide is fantastic. – Esteis Jan 29 '16 at 13:33
  • 9
    For us uninitiated, that's ```pip3 install pyyaml``` to get it ready to import into python scripts. – user8675309 Mar 11 '19 at 04:36
  • 5
    Beware, yaml is only friendly if you keep it very simple, it by default has tons of problematic, bordering on unsafe features. Try https://hitchdev.com/strictyaml/ instead as a safe-by-default lite alternative. – Gringo Suave Oct 30 '19 at 23:45
  • 2
    See [Munch](https://pypi.org/project/munch/), https://stackoverflow.com/questions/52570869/load-yaml-as-nested-objects-instead-of-dictionary-in-python `import yaml; from munch import munchify; f = munchify(yaml.load(…)); print(fo.d.try)` – Hans Ginzel Jun 21 '20 at 20:35
  • @HansGinzel You should make an answer on its own, as what you are suggesting is way easier to use – Begoodpy Oct 29 '20 at 13:57
151

I Found this the most useful and easy to use https://wiki.python.org/moin/ConfigParserExamples

You just create a "myfile.ini" like:

[SectionOne]
Status: Single
Name: Derek
Value: Yes
Age: 30
Single: True

[SectionTwo]
FavoriteColor=Green
[SectionThree]
FamilyName: Johnson

[Others]
Route: 66

And retrieve the data like:

>>> import ConfigParser
>>> Config = ConfigParser.ConfigParser()
>>> Config
<ConfigParser.ConfigParser instance at 0x00BA9B20>
>>> Config.read("myfile.ini")
['c:\\tomorrow.ini']
>>> Config.sections()
['Others', 'SectionThree', 'SectionOne', 'SectionTwo']
>>> Config.options('SectionOne')
['Status', 'Name', 'Value', 'Age', 'Single']
>>> Config.get('SectionOne', 'Status')
'Single'
Maviles
  • 2,831
  • 2
  • 23
  • 35
66

Yaml and Json are the simplest and most commonly used file formats to store settings/config. PyYaml can be used to parse yaml. Json is already part of python from 2.5. Yaml is a superset of Json. Json will solve most uses cases except multi line strings where escaping is required. Yaml takes care of these cases too.

>>> import json
>>> config = {'handler' : 'adminhandler.py', 'timeoutsec' : 5 }
>>> json.dump(config, open('/tmp/config.json', 'w'))
>>> json.load(open('/tmp/config.json'))   
{u'handler': u'adminhandler.py', u'timeoutsec': 5}
Anoop
  • 1,677
  • 18
  • 24
  • 18
    While more or less equivalent, json isn't nearly as human readable as yaml. Since his sample config is actually valid yaml, I'd stress that instead of json. – Benson Feb 20 '11 at 22:23
  • 28
    Using "json.dump(config, fp, sort_keys=True, indent=4)" improves readability. – phobie Jul 01 '13 at 16:55
  • 4
    [A simple python class loading its data members from a JSON file in RAII manner](https://gist.github.com/nadya-p/b25519cf3a74d1bed86ed9b1d8c71692) – Hope May 16 '18 at 10:16