How to split project requirements with pip

The following is a fairly basic proposal to manage completely independent isolated environments, simple because it only requires:

1. Python (the corresponding version)
2. pip, unless you have Python 2.7.9+ or Python 3.4+, where it is already included
3. virtualenv

In addition to being simple, it is fully compatible with both Linux, Windows and MAC. It is based on Virtualenv , this tool basically "creates" isolated local Python environments in specific folders and the idea is that one "activates" these environments according to their own needs, we could maintain an environment for development and another for testing, or create one new development to test new versions of a certain package or even a new version of Python, all this in the framework of a local folder without the need for administration privileges or to modify the system.

Let's see your situation. We already have the version of python that we are going to use in the project installed and running on the system, we have it pipinstalled and updated. Well, the first thing is going to be to install Virtualenv , simple: pip install virtualenv, and we are ready to start.

Typically, these "virtualenv" environments are created as subfolders of our project, but they can be anywhere on the system. Suppose our project is in ~/Proyectos/Mi-proyecto, we are going to create an environment for development in Python 3x in that folder and we will call it venv-desa(the name is the least important), we will do

# cd ~/Proyectos/Mi-proyecto/
# virtualenv -p python3 venv-desa

With this we have created a folder ~/Proyectos/Mi-proyecto/venv-desaand in it we have installed: Python (3x) and some basic packages setuptools, pkg_resources, pip, wheel, nothing more and nothing less. Now we simply have to "activate" it, this process simply modifies the pathand some other environment variables to point to our venv-desa. Activation on Linux is done, standing in the root folder of the project:

source venv-desa/bin/activate
# o de la forma más compacta
. venv-desa/bin/activate

In Windows it changes a little nothing else

venv-desa\Scripts\activate.bat

Regardless of the system the effects are the same:

(venv-desa) # which python
/home/usuario/Proyectos/Mi-proyecto/venv-desa/bin/python
(venv-desa) # which pip
/home/usuario/Proyectos/Mi-proyecto/venv-desa/bin/pip
(venv-desa) # echo $PATH
/home/usuario/Proyectos/Mi-proyecto/venv-desa/bin:/home/usuario/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

On the one hand, it modified the "prompt" indicating the name of the virtual environment and on the other it modified the path, so that when we run any of the python tools and python itself, we do it from this new folder. Now we only have to install the libraries or packages that we are going to use for our development environment, simply with the command pip install [paquete], everything that we install will end up in this new folder. Finally if we do we pip freeze > desa-requirements.txtwill have our list of requirements for our development environment, which will help us if we have to regenerate the environment. If we want to disable our environment, we just do # deactivate.

Now, we see that setting up a test environment venv-testwill only vary in the packages that we install and finally: pip freeze > test-requirements.txt. In the end we will have our two environments venv-desawith venv-testtheir own totally isolated packages that we can use alternately or even at the same time in different terminals. If we need to "regenerate" either of the two environments, on another computer or even OS, the steps are almost the same as what we have seen:

# cd ~/Proyectos/Mi-proyecto/
# virtualenv -p python3 venv-desa
# . venv-desa/bin/activate
# pip install -r desa-requirements.txt

O well

# cd ~/Proyectos/Mi-proyecto/
# virtualenv -p python3 venv-test
# . venv-test/bin/activate
# pip install -r test-requirements.txt

As I was saying, this is a very simple and above all clear alternative to understand, surely there are better and more automated tools and procedures, but this scheme, at least for me, has been very practical.

I found this answer on SOen but but lost the reference. Unfortunately, this functionality is not properly documented so it could go unnoticed.

As documented in Patricio Moracho 's excellent answer , using can create requirements files, but in this alternative, we avoid repeating packages in the files by making three different requirements files:pip

• base.txt- which contains the packages common to the environments used
• local.txt- which in this example contain the packages that are only used in local development
• prod.txt- contains the packages that are only used in production.

base.txt file

This is a normal requirements file

Django==1.11.3
django-admin-tools==0.8.1
django-authtools==1.6.0
django-classy-tags==0.8.0
django-environ==0.4.3
django-taggit==0.22.1
django-taggit-templatetags2==1.6.1
Markdown==2.6.8
psycopg2==2.7.3
Pygments==2.2.0
pytz==2017.2
requests==2.18.2
rollbar==0.13.12
Unipath==1.1

local.txt file

The requirements file can have several commands, which are half explained in the documentationpip , including references to other requirements files, and this is where the magic happens:

-r base.txt
django-debug-toolbar

This file tells you to install the packages listed in firstbase.txt and then the ones below (in this case, just one).

prod.txt file

Packages used in production are listed here.

-r base.txt
gunicorn
django-redis-cache

Use

So, to install the packages in the development environment, do this:

pip install -r local.txt

And on the production server:

pip install -r prod.txt

Note

Version references are optional, but were initially generated with the pip freeze base.txt.