Use Virtualenv to customize Python

On this page:

Overview
Set up your own Python space
Customize your Python space
Advanced use
Get help

Overview

Python is flexible in the ways you can set it up and extend its functionality. Virtualenv is a package that simplifies the process of installing other packages. It shares similarities with conda in that it creates a customized Python environment. However, it is typically much smaller than a conda environment. Additionally, it can either share specific aspects of the current Python build or, like a conda environment, remain completely isolated from it. The choice between using conda or virtual environments often comes down to personal preference. However, conda environments tend to consume significantly more disk space than virtual environments. Also, while pip, the package installer for virtual environments, is limited to installing Python packages, conda can install non-Python packages as well.

At Indiana University, most versions of Python available on the research supercomputers include the virtualenv package. Following are instructions for using Virtualenv to set up and customize personal Python spaces on IU's research supercomputers.

For more about Virtualenv in general, see the Virtualenv User Guide.

Set up your own Python space

To get started using Virtualenv:

Navigate to the directory where you want to create your new Python build. Your home directory is a good option, because it is backed up and persistent; your Slate directory also is a good option, because it is optimized for fast access from the compute nodes of the IU research supercomputers.
Make sure the desired Python version is added to your user environment; for instructions, see the Set up your user environment section of Use Python on IU research supercomputers.
Use the python -m venv command to create your new virtual environment. Make sure to give it a name that differentiates it from other Python builds. For example:
```
python -m venv genomicsPython
```
The venv command will:
- Create a new subdirectory (for example, genomicsPython) within the current directory.
- Create a new Python build (based on the currently loaded version) in the new subdirectory.
- Install pip and setuptools, which you can use to install additional packages within the virtual environment.
To switch to your virtual Python environment (for example, genomicsPython), enter:
```
source genomicsPython/bin/activate
```
The bin/activate script will:
- Modify your shell prompt to indicate which environment is currently active; for example:
```
bkyloren@login1:~> source genomicsPython/bin/activate
(genomicsPython)bkyloren@login1:~>
```
- Change your $PATH environment variable so its first entry is your virtual environment's bin directory. Consequently, entering which python on the modified shell prompt should return the path to your virtual build; for example, on Big Red 200:
```
(genomicsPython)bkyloren@login1:~> which python
/geode2/home/u010/bkyloren/Big Red 200/genomicsPython/bin/python
```
To undo these changes and switch back to the original Python build, at the virtual environment shell prompt, enter deactivate; for example:
```
(genomicsPython)bkyloren@login1:~> deactivate
bkyloren@login1:~>
```

Customize your Python space

Once your virtual environment is activated, you can use pip or setup.py to install additional Python packages to its site-packages directory (for example, following the above examples, ~/genomicsPython/lib/python3.8/site-packages).

Note:

When using pip to add packages to your virtual environment, you do not need the --user option; because Virtualenv creates your own Python build, with its own site-packages directory, you can install whatever packages you want into it.

Alternatively, you can give your virtual environment access to the system-wide site-packages directory by using virtualenv with the --system-site-packages option; for example:

python -m venv --system-site-packages genomicsPython

For more about adding Python packages, see Install Python packages on the research supercomputers at IU.

Advanced use

Various use cases may require further modifications to the Python environment. These could include:

Conflicts exist between packages in your virtual environment and packages in the system-wide Python build.
Packages in your virtual environment require system libraries that are absent.
A new version of Python is needed.

Any currently loaded modules, as well as the PYTHONPATH and LD_LIBRARY_PATH environment variables, will affect your virtual environment.

To build your own Python from scratch:

Download the desired version; for example:

wget https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz

Extract the archived files; for example:
```
tar -xzf Python-3.9.6.tgz
```
Change to the new Python directory; for example:
```
cd Python-3.9.6
```

Create the makefile; for example:

./configure --prefix=~/pythonbuild --enable-shared

Compile and create executables; for example:
```
make
```
Copy executables and other files to their final directories; for example:
```
make install
```
Change your LD_LIBRARY_PATH environment variable (to make the new Python available); for example, add the following line to the bottom of your ~/.bashrc file:
```
export LD_LIBRARY_PATH=~/pythonbuild/lib:$LD_LIBRARY_PATH
```
Change your PATH environment variable (to make the new Python available); for example, add the following line to the bottom of your ~/.bashrc file:
```
export PATH=~/pythonbuild/bin:$PATH
```
Create your virtual environment using the new Python interpreter; for example:
```
python3 -m venv genomicsPython
```
Note:
The Python that you build will be called python3; if you want to call it using the python command, you will need to create a link by hand:
```
ln -s ~/pythonbuild/bin/python3 ~/pythonbuild/bin/python
```
You will also need to do the same for pip:
```
ln -s ~/pythonbuild/bin/pip3 ~/pythonbuild/bin/pip
```

Get help

If you have questions or need help using Python on IU's research supercomputers, contact the UITS Research Applications and Deep Learning team.