PiptipsFriday, November 04, 2016
The package manager for python,
pip has some shortcomings, and we will look at solving the biggest two: not being able to record only the top level dependencies in a project (the dependencies of the dependencies also get lumped together), and finding a way to record only dev dependencies.
Create a requirements file with only top level dependencies
You can create a list of your dependencies with
pip freeze. However, if you look at the output, it's going to be obvious right away that something is not right:
Whoa. Yeah, I wanted the dependencies of flask to be installed for sure, but why would I want that in my requirements? Now I have to keep track of my depedencies' dependencies (and so on all the way down), because when I decide that I no longer need flask, I have to remove the dependencies of it manually. The requirements output actually looks like what you might find in a lockfile (like composer.lock for Composer, or shrinkwrap.json for NPM), not what you might find in your declared dependencies.
To solve the above problem, we have to use a separate tool, for god knows what reason, because this should be the package manager's job, but it's not included in pip. So if you are also wondering about this, don't worry, that tool thankfully exists, and it's called pip-tools, you install it with the usual
pip install pip-tools. It introduces a real dependency file called
requirements.in and makes
requirements.txt the lock file, AS IT SHOULD HAVE BEEN ALL ALONG.
There are two commands it provides:
This will read your
requirements.in file, and create a
requirements.txt, with the output that you would expect:
$ echo "flask" >> requirements.in $ pip-compile # # This file is autogenerated by pip-compile # To update, run: # # pip-compile --output-file requirements.txt requirements.in # click==6.6 # via flask flask==0.11.1 itsdangerous==0.24 # via flask Jinja2==2.8 # via flask MarkupSafe==0.23 # via jinja2 Werkzeug==0.11.11 # via flask
And boom, you are done,
requirements.in only contains your top level dependencies, and as a bonus your
requirements.txt now shows which package is required by another package.
This command will install all the packages in your requirements.txt, and remove any superflous packages that you have installed, but not recorded in your requirements. The difference between this, and the usual
pip install -r requirements.txt is the latter part. Since
pip install -r does not remove superflous packages, if you have forgotten to record something, you will not find out about it, until it blows up in some other environment. Instead of using
pip install X, you put X in your requirements.in file, and then do
pip-compile && pip-sync to actually install the package, avoiding the aforementioned problem.
As a bonus, updating your packages can be done with
pip-compile -U. This will take a look at dependencies, and look for updates for any package that has a flexible pinning (eg. >=1.0)
Both composer and npm can record "dev" requirements, that is, when you need packages for development, but not for production. Pip doesn't have this feature explicitly, and it doesn't really advertise it, but it can actually do this. You can put a line with
-r otherfile.txt on the top of your requirements, and it will include another file with a list of packages. So to do a separation of dev libraries, you can have two files:
And then you use the requirements file that you need for the current environment. Pip-tools also supports this in the