Import Path¶

site is automatically imported each time the interpreter starts up. On import, it extends sys.path with site-specific names constructed by combining the prefix values sys.prefix and sys.exec_prefix with several suffixes. The prefix values used are saved in the module-level variable PREFIXES for reference later. Under Windows, the suffixes are an empty string and lib/site-packages. For Unix-like platforms, the values are lib/python$version/site-packages and lib/site-python.

import sys
import os
import platform
import site

if 'Windows' in platform.platform():
    SUFFIXES = [
        '',
        'lib/site-packages',
        ]
else:
    SUFFIXES = [
        'lib/python%s/site-packages' % sys.version[:3],
        'lib/site-python',
        ]

print 'Path prefixes:'
for p in site.PREFIXES:
    print '  ', p

for prefix in sorted(set(site.PREFIXES)):
    print
    for suffix in SUFFIXES:
        path = os.path.join(prefix, suffix).rstrip(os.sep)
        print path
        print '   exists:', os.path.exists(path)
        print '  in path:', path in sys.path

Each of the paths resulting from the combinations is tested, and those that exist are added to sys.path.

$ python site_import_path.py
Path prefixes:
   /Library/Frameworks/Python.framework/Versions/2.7
   /Library/Frameworks/Python.framework/Versions/2.7

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
   exists: True
  in path: True
/Library/Frameworks/Python.framework/Versions/2.7/lib/site-python
   exists: False
  in path: False

User Directories¶

In addition to the global site-packages paths, site is responsible for adding the user-specific locations to the import path. The user-specific paths are all based on the USER_BASE directory, which usually located in a part of the filesystem owned (and writable) by the current user. Inside the USER_BASE is a site-packages directory, with the path accessible as USER_SITE.

import site

print 'Base:', site.USER_BASE
print 'Site:', site.USER_SITE

The USER_SITE path name is created using the same platform-specific values described above.

$ python site_user_base.py
Base: /Users/dhellmann/.local
Site: /Users/dhellmann/.local/lib/python2.7/site-packages

The user base directory can be set through the PYTHONUSERBASE environment variable, and has platform-specific defaults (~/Python$version/site-packages for Windows and ~/.local for non-Windows).

You can check the USER_BASE value from outside of your Python program by running site from the command line. site will give you the name of the directory whether or not it exists, but it is only added to the import path when it does.

$ python -m site --user-base


$ python -m site --user-site


$ PYTHONUSERBASE=/tmp/$USER python -m site --user-base


$ PYTHONUSERBASE=/tmp/$USER python -m site --user-site

The user directory is disabled under some circumstances that would pose security issues. For example, if the process is running with a different effective user or group id than the actual user that started it. Your application can check the setting by examining ENABLE_USER_SITE.

import site

status = {
    None:'Disabled for security',
    True:'Enabled',
    False:'Disabled by command-line option',
    }

print 'Flag   :', site.ENABLE_USER_SITE
print 'Meaning:', status[site.ENABLE_USER_SITE]

The user directory can also be explicitly disabled on the command line with -s.

$ python site_enable_user_site.py
Flag   : True
Meaning: Enabled

$ python -s site_enable_user_site.py
Flag   : False
Meaning: Disabled by command-line option

Path Configuration Files¶

As paths are added to the import path, they are also scanned for path configuration files. A path configuration file is a plain text file with the extension .pth. Each line in the file can take one of four forms:

1. A full or relative path to another location that should be added to the import path.
2. A Python statement to be executed. All such lines must begin with an import statement.
3. Blank lines are ignored.
4. A line starting with # is treated as a comment and ignored.

Path configuration files can be used to extend the import path to look in locations that would not have been added automatically. For example, Distribute adds a path to easy-install.pth when it installs a package in “develop” mode using python setup.py develop.

The function for extending sys.path is public, so we can use it in example programs to show how the path configuration files work. Given a directory with_modules containing the file mymodule.py with this print statement showing how the module was imported:

import os
print 'Loaded', __name__, 'from', __file__[len(os.getcwd())+1:]

This script shows how addsitedir() extends the import path so the interpreter can find the desired module.

import site
import os
import sys

script_directory = os.path.dirname(__file__)
module_directory = os.path.join(script_directory, sys.argv[1])

try:
    import mymodule
except ImportError, err:
    print 'Could not import mymodule:', err

print
before_len = len(sys.path)
site.addsitedir(module_directory)
print 'New paths:'
for p in sys.path[before_len:]:
    print '  ', p

print
import mymodule

After the directory containing the module is added to sys.path, the script can import mymodule without issue.

$ python site_addsitedir.py with_modules
Could not import mymodule: No module named mymodule

New paths:
   /Users/dhellmann/Devel/pymotw-ja/t2y/PyMOTW/site/with_modules

Loaded mymodule from with_modules/mymodule.py

If the directory given to addsitedir() includes any files matching the pattern *.pth, they are loaded as path configuration files. For example, if we create with_pth/pymotw.pth containing:

# Add a single subdirectory to the path.
./subdir

and copy mymodule.py to with_pth/subdir/mymodule.py, then we can import it by adding with_pth as a site directory, even though the module is not in that directory.

$ python site_addsitedir.py with_pth
Could not import mymodule: No module named mymodule

New paths:
   /Users/dhellmann/Devel/pymotw-ja/t2y/PyMOTW/site/with_pth
   /Users/dhellmann/Devel/pymotw-ja/t2y/PyMOTW/site/with_pth/subdir

Loaded mymodule from with_pth/subdir/mymodule.py

If a site directory contains multiple .pth files, they are processed in alphabetical order.

$ ls -F with_multiple_pth
a.pth
b.pth
from_a/
from_b/

$ cat with_multiple_pth/a.pth
./from_a

$ cat with_multiple_pth/b.pth
./from_b

In this case, the module is found in with_multiple_pth/from_a because a.pth is read before b.pth.

$ python site_addsitedir.py with_multiple_pth
Could not import mymodule: No module named mymodule

New paths:
   /Users/dhellmann/Devel/pymotw-ja/t2y/PyMOTW/site/with_multiple_pth
   /Users/dhellmann/Devel/pymotw-ja/t2y/PyMOTW/site/with_multiple_pth/from_a
   /Users/dhellmann/Devel/pymotw-ja/t2y/PyMOTW/site/with_multiple_pth/from_b

Loaded mymodule from with_multiple_pth/from_a/mymodule.py

sitecustomize¶

The site module is also responsible for loading site-wide customization defined by the local site owner in a sitecustomize module. Uses for sitecustomize include extending the import path and enabling coverage, profiling, or other development tools.

For example, this sitecustomize.py script extends the import path with a directory based on the current platform. The platform-specific path in /opt/python is added to the import path, so any packages installed there can be imported. A system like this is useful for sharing packages containing compiled extension modules between hosts on a network via a shared filesystem. Only the sitecustomize.py script needs to be installed on each host, and the other packages can be accessed from the file server.

print 'Loading sitecustomize.py'

import site
import platform
import os
import sys

path = os.path.join('/opt', 'python', sys.version[:3], platform.platform())
print 'Adding new path', path
                    
site.addsitedir(path)
    

A simple script can be used to show that sitecustomize.py is imported before Python starts running your own code.

import sys

print 'Running main program'

print 'End of path:', sys.path[-1]

Since sitecustomize is meant for system-wide configuration, it should be installed somewere in the default path (usally in the site-packages directory). This example sets PYTHONPATH explicitly to ensure the module is picked up.

$ PYTHONPATH=with_sitecustomize python with_sitecustomize/site_sitecustomize.py
Loading sitecustomize.py
Adding new path /opt/python/2.7/Darwin-11.4.2-x86_64-i386-64bit
Running main program
End of path: /opt/python/2.7/Darwin-11.4.2-x86_64-i386-64bit

usercustomize¶

Similar to sitecustomize, the usercustomize module can be used to set up user-specific settings each time the interpreter starts up. usercustomize is loaded after sitecustomize, so site-wide customizations can be overridden.

In environments where a user’s home directory is shared on several servers running different operating systems or versions, the standard user directory mechanism may not work for user-specific installations of packages. In these cases, platform-specific directory tree can be used instead.

print 'Loading usercustomize.py'

import site
import platform
import os
import sys

path = os.path.expanduser(os.path.join('~', 'python', sys.version[:3], platform.platform()))
print 'Adding new path', path
                    
site.addsitedir(path)
    

Another simple script, similar to the one used for sitecustomize, can be used to show that usercustomize.py is imported before Python starts running your own code.

import sys

print 'Running main program'

print 'End of path:', sys.path[-1]

Since usercustomize is meant for user-specific configuration for a user, it should be installed somewhere in the user’s default path, but not on the site-wide path. The default USER_BASE directory is a good location. This example sets PYTHONPATH explicitly to ensure the module is picked up.

$ PYTHONPATH=with_usercustomize python with_usercustomize/site_usercustomize.py
Loading usercustomize.py
Adding new path /Users/dhellmann/python/2.7/Darwin-11.4.2-x86_64-i386-64bit
Running main program
End of path: /Users/dhellmann/python/2.7/Darwin-11.4.2-x86_64-i386-64bit

When the user site directory feature is disabled, usercustomize is not imported, whether it is located in the user site directory or elsewhere.

$ PYTHONPATH=with_usercustomize python -s with_usercustomize/site_usercustomize.py
Running main program
End of path: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages

Disabling site¶

To maintain backwards-compatibility with versions of Python from before the automatic import was added, the interpreter accepts an -S option.

$ python -S site_import_path.py
Path prefixes:
   sys.prefix     : /Library/Frameworks/Python.framework/Versions/2.7
   sys.exec_prefix: /Library/Frameworks/Python.framework/Versions/2.7

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
   exists: True
  in path: False
/Library/Frameworks/Python.framework/Versions/2.7/lib/site-python
   exists: False
  in path: False