Use Virtualenv to customize Python

Python is flexible in the ways you can set it up and extend its functionality. Virtualenv is a package that simplifies the process of installing other packages. It is similar to Anaconda in that it creates a small, personalized Python environment that can either share certain aspects of the current Python build or be completely isolated from it. However, it differs from Anaconda in the way it implements this personalized environment. The decision to use one over the other is purely one of personal preference.

At Indiana University, most versions of Python available on the research computing systems include the virtualenv package. Following are instructions for using Virtualenv to set up and customize personal Python spaces on IU's research computing systems.

For more about Virtualenv in general, see the Virtualenv User Guide.

On this page:


Set up your own Python space

To get started using Virtualenv:

  1. Navigate to the directory where you want to create your new Python build. Your home directory is a good option, because it is backed up and persistent.
  2. Make sure the desired Python version is added to your user environment; for instructions, see the Set up your user environment section of Use Python on IU research computing systems.
  3. Use the virtualenv command to create your new virtual environment. Make sure to give it a name that differentiates it from other Python builds; for example, to create a virtual environment for genomics applications on Karst:
      [bkyloren@h2 ~]$ virtualenv genomicsPython
      New python executable in genomicsPython/bin/python
      Installing setuptools, pip...done.
    

    The virtualenv command will:

    • Create a new subdirectory (e.g., genomicsPython) within the current directory.
    • Create a new Python build (based on the currently loaded version) in the new subdirectory.
    • Install pip and setuptools, which you can use to install additional packages within the virtual environment.
  4. To switch to your virtual Python environment (e.g., genomicsPython), enter:
      source genomicsPython/bin/activate
    

    The bin/activate script will:

    • Modify your shell prompt to indicate which environment is currently active; for example, on Karst:
        [bkyloren@h2 ~]$ source genomicsPython/bin/activate
        (genomicsPython)[bkyloren@h2 ~]$
      
    • Change your $PATH environment variable so its first entry is your virtual environment's bin directory. Consequently, entering which python on the modified shell prompt should return the path to your virtual build; for example:
        (genomicsPython)[bkyloren@h2 ~]$ which python
        /gpfs/home/b/k/bkyloren/Karst/genomicsPython/bin/python
      
  5. To undo these changes and switch back to the original Python build, at the virtual environment shell prompt, enter deactivate; for example:
      (genomicsPython)[bkyloren@h2 ~]$ deactivate
      [bkyloren@h2 ~]$
    

Customize your Python space

Once your virtual environment is activated, you can use pip or setup.py to install additional Python packages to its site-packages directory (e.g., following the above examples, ~/genomicsPython/lib/python2.7/site-packages).

Note:
When using pip to add packages to your virtual environment, you do not need the --user option; because Virtualenv creates your own Python build, with its own site-packages directory, you can install whatever packages you want into it.

Alternatively, you can give your virtual environment access to the system-wide site-packages directory by using virtualenv with the --system-site-packages option; for example:

  virtualenv --system-site-packages genomicsPython

For more about adding Python packages, see Install Python packages on the research computing systems at IU.

Advanced use

Various use cases may require further modifications to the Python environment. These could include:

  • Conflicts exist between packages in your virtual environment and packages in the system-wide Python build.
  • Packages in your virtual environment require system libraries that are absent.
  • A new version of Python is needed.

Any currently loaded modules, as well as the PYTHONPATH and LD_LIBRARY_PATH environment variables, will affect your virtual environment.

To build your own Python from scratch:

  1. Download the desired version; for example:
      wget https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz
    
  2. Extract the archived files; for example:
      tar -xzf Python-2.7.13.tgz
    
  3. Change to the new Python directory; for example:
      cd Python-2.7.13
    
  4. Create the makefile; for example:
      ./configure --prefix=~/pythonbuild --enable-shared
    
  5. Compile and create executables; for example:
      make
    
  6. Copy executables and other files to their final directories; for example:
      make install
    
  7. Change your LD_LIBRARY_PATH environment variable (to make the new Python available); for example, add the following line to the bottom of your ~/.bashrc file:
      export LD_LIBRARY_PATH=~/pythonbuild/lib:$LD_LIBRARY_PATH
    
  8. Create your virtual environment using the new Python interpreter; for example:
      virtualenv -p ~/pythonbuild/bin/python genomicsPython
    
    Note:
    This may fail if the new Python is very different from the Python build in your user environment (the one with which virtualenv is installed). If this happens, you may need to either start with a different Python module in your user environment or add virtualenv to your new Python build.

Get help

If you have questions or need help using Python on IU's research computing systems, contact the UITS Scientific Applications and Performance Tuning (SciAPT) team.

Support for IU research computing systems, software, and services is provided by the Research Technologies division of UITS. To ask a question or get help, contact UITS Research Technologies.

This is document aonm in the Knowledge Base.
Last modified on 2018-07-16 13:28:04.

Contact us

For help or to comment, email the UITS Support Center.