Use Jupyter Notebook on Quartz

On this page:

Overview
Use Jupyter Notebook on Quartz via RED
- X11 forwarding
Use Jupyter Notebook on Quartz with your local web browser as the interface
- Port forwarding
Access Slate, Slate-Project, or Slate-Scratch directories from Jupyter Notebook
Get help

Overview

At Indiana University, UITS Research Applications and Deep Learning (RADL) makes Jupyter Notebook available for use on Research Desktop (RED) via the Applications menu. Jupyter Lab is available on RED also; to launch Jupyter Lab on RED, open a Terminal window, load the python module, and then enter jupyter lab. (In the instructions below, if you want to use Jupyter Lab instead of Jupyter Notebook, replace the command jupyter notebook with jupyter lab.)

When running Python code on compute nodes, RADL recommends using the python command to run your code as a Python script file rather than using Jupyter Notebook to run Python code. This is because Jupyter's interactive nature does not allow for efficient use of compute node resources.

If you want to convert an existing notebook to a Python script, use the following command (replace name_of_notebook.ipynb with the name of your notebook):

jupyter nbconvert --to python name_of_notebook.ipynb

The above command produces a Python script (in this example, named name_of_notebook.py) that can be executed with the following command:

python name_of_notebook.py

When circumstances require you to use Jupyter Notebook on a compute node, RADL recommends that you:

Use an interactive job on Quartz to connect to a compute node.
In most cases, use one of the interactive, debug, or gpu-debug partitions.
Use one of the following two methods.

Use Jupyter Notebook on Quartz via RED

X11 forwarding

This method uses X11 forwarding to launch the Firefox web browser on the compute node, the interface for which is displayed on your RED desktop.

Use the ThinLinc Client to connect to RED; for help, see Connect to RED. Alternatively, use a web browser to access https://red.uits.iu.edu.
In RED, double-click the Interactive Job icon to launch an interactive job on Quartz.
Alternatively, if you need more time, cores, or memory, or need a GPU node, open a Terminal window and use srun with the --x11 option to request the resources that you need; for example:
```
srun -A slurm-account-name -p interactive --nodes=1 --ntasks-per-node=8 --time=1:00:00 --mem=20G --x11 --pty bash
```
Replace slurm-account-name with your allocation's Slurm Account Name.

For more about submitting interactive jobs, see Use Slurm to submit and manage jobs on IU's research computing systems.
Once the interactive job is launched, load the python module; on the command line, enter:
```
module load python
```
Note:

If you are connecting to a GPU node, load the python/gpu module instead.
Launch Jupyter Notebook; on the command line, enter:
```
jupyter notebook
```

Use Jupyter Notebook on Quartz with your local web browser as the interface

Port forwarding

On your workstation, use any SSH2 client to connect to quartz.uits.iu.edu with port forwarding enabled; for example, on the command line, enter (replace username with your IU username):
```
ssh -L 8000:localhost:8001 username@quartz.uits.iu.edu
```
Launch an interactive job; on the command line, enter:
```
salloc -A slurm-account-name --nodes=1 --ntasks-per-node=8 --time=1:00:00 --mem=20G
```
Replace slurm-account-name with your allocation's Slurm Account Name.

For more about submitting interactive jobs, see Use Slurm to submit and manage jobs on IU's research computing systems.
Once the job has started, your command prompt should include the name of the node allocated to the job (for example, c3). If you don't see the node name, at the command prompt, enter squeue -u username (replace username with your IU username) to see the names of all nodes currently allocated to any of your running jobs.
Using port forwarding, connect to the node on which your job is running; on the command line, enter (replace node-name with the name of the node on which your job is running):
```
ssh -L 8001:localhost:8888 node-name
```
Load the Python module; at your interactive job's command prompt, enter:
```
module load python
```
Note:

If you are connecting to a GPU node, use module load python/gpu instead.
Launch Jupyter Notebook without a browser connection; at your interactive job's command prompt, enter:
```
jupyter notebook --no-browser
```
A URL will be generated for connecting to the server; it should look something like this:
```
http://localhost:8888/tree?token=d280240d46a1c5aa493f093283f4c6035beb546e7bc218a2
```
Paste the URL into your web browser, but replace the 8888 in the URL with 8000; for example:
```
http://localhost:8000/tree?token=d280240d46a1c5aa493f093283f4c6035beb546e7bc218a2
```

If everything is working correctly, your local web browser should display Jupyter Notebook running on the compute node.

Note:

When connecting with port forwarding, ports such as 8000, 8001, etc., may be in use. If this is the case, exit to RED or to your local machine either by closing your Terminal window and opening a new one, or by entering exit until the command prompt matches your base terminal's command prompt. Instead of trying 8001, you may try 8002, 8003, and so on; for example:

ssh -L 8000:localhost:8003 username@quartz.uits.iu.edu

salloc -A slurm-account-name --nodes=1 --ntasks-per-node=8 --time=1:00:00 --mem=20G

ssh -L 8003:localhost:8888 username@node-name

Also, the port number for Jupyter may be different from 8888 (for example, 8889). If this is the case, exit the connection from the login node to the compute node, and reconnect with SSH using the port number specified in the output of the jupyter notebook command. Alternatively, you can specify a port number when launching Jupyter Notebook; for example:

jupyter notebook --no-browser --port=8080

In this case, you would need to change 8888 in the steps above to port number you specified.

Access Slate, Slate-Project, or Slate-Scratch directories from Jupyter Notebook

When you launch Jupyter Notebook via RED from a Terminal window or the Applications menu, the Files tab shows your Quartz home directory (for example, /N/u/username/Quartz) by default and does not provide a way to navigate to directories higher in the file system (to /N/slate/username, for example). Following are some methods for accessing your Slate, Slate-Project, or Slate-Scratch directories from within Jupyter Notebook.

Change directories before launching Jupyter Notebook in a Terminal window

Before launching Jupyter Notebook from the command line, navigate to your Slate, Slate-Project, or Slate-Scratch directory. When Jupyter Notebook launches, the Files tab will show your Slate, Slate-Project, or Slate-Scratch directory, depending on the path you specify in the first step.

On the command line, enter the following, depending on the resource you want to access:
- Slate:
```
cd /N/slate/username
```
  Replace username with your IU username.
- Slate-Project:
```
cd /N/project/project_name
```
  Replace project_name with the project name associated with the Slate-Project space you are accessing.
- Slate-Scratch:
```
cd /N/scratch/username
```
  Replace username with your IU username.
Load the python module; on the command line, enter:
```
module load python
```
Note:

If you are connecting to a GPU node, use module load python/gpu instead.
Launch Jupyter Notebook; on the command line, enter:
```
jupyter notebook
```

Create a symbolic link in your home directory for your Slate, Slate-Project, or Slate-Scratch space

You can create a symbolic link in your Quartz home directory for your Slate, Slate-Project, or Slate-Scratch space. After creating the symbolic link, you will see it displayed on the Files tab in Jupyter Notebook as a folder in your home directory. To create a symbolic link:

On RED, open a Terminal window.
On the command line, enter the following, depending on the resource you want to access in Jupyter Notebook:
- Slate:
```
ln -s /N/slate/username ${HOME}/SlateDir
```
  Replace username with your IU username.
- Slate-Project:
```
ln -s /N/project/project_name ${HOME}/SlateProjectDir
```
  Replace project_name with the project name associated with the Slate-Project space you are accessing.
- Slate-Scratch:
```
ln -s /N/scratch/username ${HOME}/SlateScratchDir
```
  Replace username with your IU username.

Change the default working directory for Jupyter

You can configure Jupyter Notebook to use your Slate, Slate-Project, or Slate-Scratch directory as the default working directory shown on the Files tab:

On RED, open a Terminal Window.
Load the python module; on the command line, enter:
```
module load python
```
Generate a configuration file; on the command line, enter:
```
jupyter notebook --generate-config
```
The configuration file will be saved to the .jupyter subdirectory in your Quartz home directory (for example, at /N/u/username/Quartz/.jupyter/jupyter_notebook_config.py). Open the configuration file with an editor, and then find the line similar to this:
```
# c.NotebookApp.notebook_dir = ''
```
Replace the above line with the following, depending on the resource you want to access:
- Slate:
```
c.NotebookApp.notebook_dir = '/N/slate/username'
```
  Replace username with your IU username.
- Slate-Project:
```
c.NotebookApp.notebook_dir = '/N/project/project_name'
```
  Replace project_name with the project name associated with the Slate-Project space you are accessing.
- Slate-Scratch:
```
c.NotebookApp.notebook_dir = '/N/scratch/username'
```
  Replace username with your IU username.

Get help

If you need help or have a question about using Jupyter Notebook on IU's research supercomputers, contact the UITS Research Applications and Deep Learning team.

If you are using Research Desktop (RED) and need help or have questions, use this form to contact the Research Desktop development team directly.