Of course I have a backup!

Random blobs of wisdom about software development

Piptips

Friday, November 04, 2016

The package manager for python, pip has some shortcomings, and we will look at solving the biggest two: not being able to record only the top level dependencies in a project (the dependencies of the dependencies also get lumped together), and finding a way to record only dev dependencies.

Create a requirements file with only top level dependencies

You can create a list of your dependencies with pip freeze. However, if you look at the output, it's going to be obvious right away that something is not right:

$ pip install flask

Collecting flask
...

$ pip freeze
click==6.6
Flask==0.11.1
itsdangerous==0.24
Jinja2==2.8
MarkupSafe==0.23
Werkzeug==0.11.11

Whoa. Yeah, I wanted the dependencies of flask to be installed for sure, but why would I want that in my requirements? Now I have to keep track of my depedencies' dependencies (and so on all the way down), because when I decide that I no longer need flask, I have to remove the dependencies of it manually. The requirements output actually looks like what you might find in a lockfile (like composer.lock for Composer, or shrinkwrap.json for NPM), not what you might find in your declared dependencies.

Pip-tools

To solve the above problem, we have to use a separate tool, for god knows what reason, because this should be the package manager's job, but it's not included in pip. So if you are also wondering about this, don't worry, that tool thankfully exists, and it's called pip-tools, you install it with the usual pip install pip-tools. It introduces a real dependency file called requirements.in and makes requirements.txt the lock file, AS IT SHOULD HAVE BEEN ALL ALONG.

There are two commands it provides:

pip-compile [requirements-in]

This will read your requirements.in file, and create a requirements.txt, with the output that you would expect:

$ echo "flask" >> requirements.in
$ pip-compile

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
click==6.6                # via flask
flask==0.11.1
itsdangerous==0.24        # via flask
Jinja2==2.8               # via flask
MarkupSafe==0.23          # via jinja2
Werkzeug==0.11.11         # via flask

And boom, you are done, requirements.in only contains your top level dependencies, and as a bonus your requirements.txt now shows which package is required by another package.

pip-sync

This command will install all the packages in your requirements.txt, and remove any superflous packages that you have installed, but not recorded in your requirements. The difference between this, and the usual pip install -r requirements.txt is the latter part. Since pip install -r does not remove superflous packages, if you have forgotten to record something, you will not find out about it, until it blows up in some other environment. Instead of using pip install X, you put X in your requirements.in file, and then do pip-compile && pip-sync to actually install the package, avoiding the aforementioned problem.

As a bonus, updating your packages can be done with pip-compile -U. This will take a look at dependencies, and look for updates for any package that has a flexible pinning (eg. >=1.0)

Dev requirements

Both composer and npm can record "dev" requirements, that is, when you need packages for development, but not for production. Pip doesn't have this feature explicitly, and it doesn't really advertise it, but it can actually do this. You can put a line with -r otherfile.txt on the top of your requirements, and it will include another file with a list of packages. So to do a separation of dev libraries, you can have two files:

requirements.dev.txt
-r requirements.txt
nose
requirements.txt
flask

And then you use the requirements file that you need for the current environment. Pip-tools also supports this in the *.in files.

This was written by Norbert Kéri, posted on Friday, November 04, 2016, at 15:02

Tagged as:

Post a comment

Providing your email is optional, it is never published or shared, it is only used for auto approval purposes. If you already have at least 1 approved comment(s) tied to your email, you don't have to wait for moderation, otherwise the author must approve your comment.

Please solve this totally random captcha