25

Package managers for JavaScript like npm and yarn use a package.json to specify 'top-level' dependencies, and create a lock-file to keep track of the specific versions of all packages (i.e. top-level and sub-level dependencies) that are installed as a result.

In addition, the package.json allows us to make a distinction between types of top-level dependencies, such as production and development.

For Python, on the other hand, we have pip. I suppose the pip equivalent of a lock-file would be the result of pip freeze > requirements.txt.

However, if you maintain only this single requirements.txt file, it is difficult to distinguish between top-level and sub-level dependencies (you would need for e.g. pipdeptree -r to figure those out). This can be a real pain if you want to remove or change top-level dependencies, as it is easy to be left with orphaned packages (as far as I know, pip does not remove sub-dependencies when you pip uninstall a package).

Now, I wonder: Is there some convention for dealing with different types of these requirements files and distinguishing between top-level and sub-level dependencies with pip?

For example, I can imagine having a requirements-prod.txt which contains only the top-level requirements for the production environment, as the (simplified) equivalent of package.json, and a requirements-prod.lock, which contains the output of pip freeze, and acts as my lock-file. In addition I could have a requirements-dev.txt for development dependencies, and so on and so forth.

I would like to know if this is the way to go, or if there is a better approach.

p.s. The same question could be asked for conda's environment.yml.

djvg
  • 8,264
  • 3
  • 53
  • 83

1 Answers1

28

There are at least three good options available today:

  1. Poetry uses pyproject.toml and poetry.lock files, much in the same way that package.json and lock files work in the JavaScript world.

    This is now my preferred solution.

  2. Pipenv uses Pipfile and Pipfile.lock, also much like you describe the JavaScript files.

Both Poetry and Pipenv do more than just dependency management. Out of the box, they also create and maintain virtual environments for your projects.

  1. pip-tools provides pip-compile and pip-sync commands. Here, requirements.in lists your direct dependencies, often with loose version constraints and pip-compile generates locked down requirements.txt files from your .in files.

    This used to be my preferred solution. It's backwards-compatible (the generated requirements.txt can be processed by pip) and the pip-sync tool ensures that the virtualenv exactly matches the locked versions, removing things that aren't in your "lock" file.

Chris
  • 112,704
  • 77
  • 249
  • 231
  • 1
    Thanks for the great answer, which pointed me to [this interesting post](https://nvie.com/posts/better-package-management/). However, I am hesitant in adopting `pipenv`, with its use of `virtualenv` instead of `conda`, because I really like (and rely on) `conda`'s ability to manage `Python` versions. – djvg Oct 05 '18 at 12:51
  • That's another point in favour of `pip-tools`, IMO. It doesn't try to do too much for you. – Chris Oct 05 '18 at 13:10
  • And `pip-tools` also takes care of "it is easy to be left with orphaned packages" since it removes anything that's not in the requirements file. – Chris Oct 05 '18 at 13:34
  • Sounds good, I'll have a look at it. Does introduce another dependency though. ;-) – djvg Oct 05 '18 at 13:49
  • Yes, to work around limitations of `pip` itself. There are manual workarounds using just `pip` but then you're a lot more likely to make a mistake. The fact that `pip-compile` outputs a `pip`-compatible `requirements.txt` file means you can just `pip install -r requirements.txt` on new machines and then work with `pip-tools` moving forward. I usually install `pip-tools` into new environments on creation. – Chris Oct 05 '18 at 13:58
  • What I need is, I'd like to run `pip install` and generate the lock file automatically and commit the lockfile in source code as well. So next time, in a new system, I can install the exact same package versions by reference lockfile more than to install some newer version on-fly. – BMW Jun 24 '19 at 01:36
  • @BMW, are you asking for clarification about something? Both options listed here do what you want, just with slightly different syntax. – Chris Jun 24 '19 at 01:41
  • Jumping in here to add a +1 to `pip-tools`, I like how lightweight it is and how compatible it is with any other tool out there, since it's output is just a plain old requirements.txt. – Nick Crews Apr 07 '22 at 20:49