Python Module Search Path

Introduction

The lesson on creating Python modules highlighted that a module you create is accessible to files within the same directory, which can be convenient but may not align well with good organizational practices. For this reason, it’s possible to designate other directories on disk in which Python searches for modules. Those directories and the sequence in which they are searched are called the Python module search path. When Python executes an import statement to load a module, it follows the search path to locate the module file.

Python Module Search Path Sources

The search path consists of several directories where Python looks for modules. By default, Python includes the following locations in its module search path:

  1. The directory containing the script being executed.
  2. The directories listed in the PYTHONPATH environment variable.
  3. The installation-dependent default directories.

Customizing the Module Search Path With PYTHONPATH

An environment variable is an easily-set value that an operating system keeps and makes available to executing programs. Consequently, adding more directories to the PYTHONPATH environment variable (see #2 in the previous section) is a convenient way to customize the module search path.

If you are developing in a Unix-based operating system, you can add a directory to the PYTHONPATH with the following export command:

export PYTHONPATH=/path/to/custom/modules:$PYTHONPATH

Replace /path/to/custom/modules with the actual path to the directory containing your custom modules. By adding $PYTHONPATH at the end, this command prepends your directory to any previous ones that were already there.

Other operating systems, such as Windows, may have different ways of setting environment variables.

The PYTHONPATH variable is a convenient place to include project-specific directories, facilitating the import of custom modules used by a project. However, if you are using an integrated development environment (IDE), it may already be adding the project directories into the module search path for you, and you may not need to change PYTHONPATH on your own.

Understanding sys.path

When a Python script is run, the Python interpreter loads the module search path it compiled from the various sources into a Python variable called sys.path. (It’s a variable named path that is accessible from the sys module.) Once sys.path is loaded, the Python interpreter will use what’s there as its module search path.

sys.path is a list of directory strings. Let’s print it out:

import sys

for dir in sys.path:
    print(dir)

Output:

/Applications/Programming/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev
/Applications/Programming/PyCharm CE.app/Contents/plugins/python-ce/helpers/third_party/thriftpy
/Applications/Programming/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev
/opt/anaconda3/envs/pythonProject/lib/python310.zip
/opt/anaconda3/envs/pythonProject/lib/python3.10
/opt/anaconda3/envs/pythonProject/lib/python3.10/lib-dynload
/opt/anaconda3/envs/pythonProject/lib/python3.10/site-packages
/Users/python/Documents/Projects/Learn Python

The code first imports the sys module (line 1), then loops through all the directories (line 3) and prints them out (line 4). As you can see in the output, there are a few directories related to my IDE (PyCharm), then some related to the Python installation I’m using (Anaconda 3), and the final directory belongs to the project.

Changing sys.path at Run Time

Since sys.path is a variable, and Python ultimately draws its search path from there, it’s possible to change the search path at run time by modifying sys.path.

To create the next example, I placed a small module file called my_greeints in the /Library/my_super_modules directory on my computer. It contains the following code:

def say_hello():
    print("Hello")

The example will use sys.path to use it.

Example:

import sys

sys.path.append('/Library/my_super_modules')

import my_greeting

my_greeting.say_hello()

Output:

Hello

The example above adds the /Library/my_super_modules directory to the module search path by appending it to sys.path (line 3). It is now available for module import and can be used to import the my_greeting module (line 5). That module is then used to print “Hello” (line 7).

Summary & Reference for the Python Module Search Path

The Python module search path is a list of directories in which Python will search for modules.

By default, Python includes the following locations in its module search path:

  1. The directory containing the script being executed.
  2. The directories listed in the PYTHONPATH environment variable.
  3. The installation-dependent default directories.

If you are developing in a Unix-based operating system, you can add a directory to the PYTHONPATH with an export command.

export PYTHONPATH=/path/to/custom/modules:$PYTHONPATH

When a Python script is run, the Python interpreter loads the module search path into a list variable called sys.path.

import sys

for dir in sys.path:
    print(dir)

Python ultimately uses the directories in sys.path to search for modules. Consequently, the search path can be changed at run time by assigning to sys.path.

import sys

sys.path.append('/Library/my_super_modules')
Tags