In this post we will show how to set up an advanced directory structure for painless maintenance of version-specific and version-independent Python packages.
In Python, reusable code is distributed in so-called modules and packages. A Python package is a folder containing several Python modules and possibly sub-packages.
When a module X is imported, Python searches its module search path and loads the first module named X that it finds. This is a very powerful and far-reaching concept, because it facilitates and proliferates reuse much more than approaches where module dependencies are resolved explicitly (using some kind of central configuration file or such).
Python expands on this concept further with the notion of site-packages directories. A site-packages directory is where you install additional python packages, including your own reusable code.1 There can be more than one site-packages directory.
When python installs, a default site-packages folder is set up. On Windows, the folder is created inside the Python installation directory tree. (For example as C:Program FilesPython2.6Libsite-packages.) Hence, every installed Python version has a different site-packages directory. We call this the version-specific site-packages directory.
This is not where you want to install pure Python modules and packages!
Python differentiates between “pure-python” modules and “native” or “built-in” modules. The latter are modules compiled for a specific python version. They won’t work with any other Python version.
These are the modules that belong into the version-specific Python site-packages directory.
But pure-python modules are usually inter-operable across Python versions. There is no sense maintaining redundant copies of them.
Hence, we set up a version-independent site-packages directory as well.
On Windows Vista, a good place is either C:UsersPublicLibsPythonsite-packages or C:UsersLibsPythonsite-packages. (We prefer the former.) If you installed Python into C:UsersPublicPythonX.Y you should use C:UsersPublicPythonLibssite-packages instead. On Windows XP, you may also consider C:Program FilesPythonLibssite-packages or C:Program FilesPythonAnyLibssite-packages (to contrast with C:Program FilesPythonX.YLibssite-packages).
Where ever you put that folder, there are two important steps to take:
In each version-specific site-packages directory, create a PTH file (this is just an ordinary text file; you may call it xsites.pth or whatever you want as long it has the suffix “.pth”), and enter in it the location of your version-independent site-packages directory.
In the version-independent site-packages directory, create a file named sitecustomize.py with the following content:
import os, site ; site.addsitedir( os.path.dirname(__file__) )
Now when you start Python, both the version-specific and the common, version-independent modules and packages will be accessible.
This way, you have to maintain mostly just a single collection of pure-python modules and packages. And when you switch to a newer Python version, you only need to re-install version-specific stuff.
A real time-saver, that much we can tell.