Use Virtualenv to customize Python

On this page:


Python is flexible in the ways you can set it up and extend its functionality. Virtualenv is a package that simplifies the process of installing other packages. It is similar to Anaconda in that it creates a small, personalized Python environment that can either share certain aspects of the current Python build or be completely isolated from it. However, it differs from Anaconda in the way it implements this personalized environment. The decision to use one over the other is purely one of personal preference.

At Indiana University, most versions of Python available on the research supercomputers include the virtualenv package. Following are instructions for using Virtualenv to set up and customize personal Python spaces on IU's research supercomputers.

For more about Virtualenv in general, see the Virtualenv User Guide.

Set up your own Python space

To get started using Virtualenv:

  1. Navigate to the directory where you want to create your new Python build. Your home directory is a good option, because it is backed up and persistent.
  2. Make sure the desired Python version is added to your user environment; for instructions, see the Set up your user environment section of Use Python on IU research supercomputers.
  3. Use the python -m venv command to create your new virtual environment. Make sure to give it a name that differentiates it from other Python builds; for example, to create a virtual environment for genomics applications on Carbonate:
      [bkyloren@h2 ~]$ python -m venv genomicsPython
      [bkyloren@h2 ~]$

    The venv command will:

    • Create a new subdirectory (for example, genomicsPython) within the current directory.
    • Create a new Python build (based on the currently loaded version) in the new subdirectory.
    • Install pip and setuptools, which you can use to install additional packages within the virtual environment.
  4. To switch to your virtual Python environment (for example, genomicsPython), enter:
      source genomicsPython/bin/activate

    The bin/activate script will:

    • Modify your shell prompt to indicate which environment is currently active; for example, on Carbonate:
        [bkyloren@h2 ~]$ source genomicsPython/bin/activate
        (genomicsPython)[bkyloren@h2 ~]$
    • Change your $PATH environment variable so its first entry is your virtual environment's bin directory. Consequently, entering which python on the modified shell prompt should return the path to your virtual build; for example:
      (genomicsPython)[bkyloren@h2 ~]$ which python
  5. To undo these changes and switch back to the original Python build, at the virtual environment shell prompt, enter deactivate; for example:
      (genomicsPython)[bkyloren@h2 ~]$ deactivate
      [bkyloren@h2 ~]$

Customize your Python space

Once your virtual environment is activated, you can use pip or to install additional Python packages to its site-packages directory (for example, following the above examples, ~/genomicsPython/lib/python3.8/site-packages).

When using pip to add packages to your virtual environment, you do not need the --user option; because Virtualenv creates your own Python build, with its own site-packages directory, you can install whatever packages you want into it.

Alternatively, you can give your virtual environment access to the system-wide site-packages directory by using virtualenv with the --system-site-packages option; for example:

  python -m venv --system-site-packages genomicsPython

For more about adding Python packages, see Install Python packages on the research supercomputers at IU.

Advanced use

Various use cases may require further modifications to the Python environment. These could include:

  • Conflicts exist between packages in your virtual environment and packages in the system-wide Python build.
  • Packages in your virtual environment require system libraries that are absent.
  • A new version of Python is needed.

Any currently loaded modules, as well as the PYTHONPATH and LD_LIBRARY_PATH environment variables, will affect your virtual environment.

To build your own Python from scratch:

  1. Download the desired version; for example:
  2. Extract the archived files; for example:
      tar -xzf Python-3.9.6.tgz
  3. Change to the new Python directory; for example:
      cd Python-3.9.6
  4. Create the makefile; for example:
      ./configure --prefix=~/pythonbuild --enable-shared
  5. Compile and create executables; for example:
  6. Copy executables and other files to their final directories; for example:
      make install
  7. Change your LD_LIBRARY_PATH environment variable (to make the new Python available); for example, add the following line to the bottom of your ~/.bashrc file:
      export LD_LIBRARY_PATH=~/pythonbuild/lib:$LD_LIBRARY_PATH
  8. Change your PATH environment variable (to make the new Python available); for example, add the following line to the bottom of your ~/.bashrc file:
      export PATH=~/pythonbuild/bin:$PATH
  9. Create your virtual environment using the new Python interpreter; for example:
      python3 -m venv genomicsPython

    The Python that you build will be called python3; if you want to call it using the python command, you will need to create a link by hand:

      ln -s ~/pythonbuild/bin/python3 ~/pythonbuild/bin/python

    You will also need to do the same for pip:

      ln -s ~/pythonbuild/bin/pip3 ~/pythonbuild/bin/pip

Get help

If you have questions or need help using Python on IU's research supercomputers, contact the UITS Research Applications and Deep Learning team.

This is document aonm in the Knowledge Base.
Last modified on 2021-08-05 13:15:21.