README.md 6.95 KB
Newer Older
Willi Rath's avatar
Willi Rath committed
1
# How to work with JupyterLab everywhere
Willi Rath's avatar
Willi Rath committed
2

3 4 5 6 7
> 
> _This guide **will** walk you through the complete setup of JupyterLab and
> one or more scientific Python computing environments on your local computer
> or on a remote machine._
> 
Willi Rath's avatar
Willi Rath committed
8

9
> 
Christina Roth's avatar
Christina Roth committed
10 11
> _This guide **will not** explain how to use the shell, how to connect to 
> remote machines via SSH, or how to make sure you have access to the relevant
12 13
> networks, e.g., via VPN._
> 
14

15 16 17
> 
> _Currently, **only Linux / Unix** and **MacOSX** are covered._
> 
Willi Rath's avatar
Willi Rath committed
18

Willi Rath's avatar
Willi Rath committed
19

Willi Rath's avatar
Willi Rath committed
20
## Install the base environment and JupyterLab
Willi Rath's avatar
Willi Rath committed
21

22 23 24 25 26 27
> 
> _This will show how to download the installer, install a minimal Python
> environment, and start JupyerLab.  Execute the steps below on the computer
> where the calculation should be performed (e.g., your local computer or a
> computer in a remote computing centre)._
> 
28

Willi Rath's avatar
Willi Rath committed
29

Willi Rath's avatar
Willi Rath committed
30 31
### Download and install Miniconda3

32 33 34 35
> 
> _(No matter if you want to work with Python 3, which you probably should, or
> Python 2, the **base environment should be Python 3**.)_
> 
Willi Rath's avatar
Willi Rath committed
36

Willi Rath's avatar
Willi Rath committed
37
To download the latest installer
Willi Rath's avatar
Willi Rath committed
38

Willi Rath's avatar
Willi Rath committed
39
- on Linux, run:
Willi Rath's avatar
Willi Rath committed
40 41 42 43
  ```bash
  curl https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3.sh
  ```

Willi Rath's avatar
Willi Rath committed
44
- on MaxOSX, run:
Willi Rath's avatar
Willi Rath committed
45 46 47 48 49 50 51 52 53 54 55
  ```bash
  curl https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o Miniconda3.sh
  ```

To install, run (on both, Linux and MacOSX):
```bash
bash Miniconda3.sh -b -p ${HOME}/miniconda3
```

(If this fails, check
<https://conda.io/docs/user-guide/install/#regular-installation>, make sure to
Willi Rath's avatar
Willi Rath committed
56
follow the instructions for Miniconda**3** and do _**not**_ make this
Willi Rath's avatar
Willi Rath committed
57 58 59
installation your standard Python.)


Willi Rath's avatar
Willi Rath committed
60
### Activate the base environment and install JupyterLab
61 62 63 64
Before continuing make sure you are in a bash shell by typing
```bash
echo $SHELL | grep bash || bash
```
Willi Rath's avatar
Willi Rath committed
65 66 67 68 69 70 71 72 73 74 75 76 77

To activate the base environment, run:
```bash
source ${HOME}/miniconda3/bin/activate base
```

Then (and in the same terminal), install `jupyterlab` and `nb_conda_kernels`,
by running:
```bash
conda install jupyterlab nb_conda_kernels
```


Willi Rath's avatar
Willi Rath committed
78
### Start JupyterLab
Willi Rath's avatar
Willi Rath committed
79

80 81 82 83 84
After activating the base environment with
```bash
source ${HOME}/miniconda3/bin/activate base
```
run:
Willi Rath's avatar
Willi Rath committed
85
```bash
86
jupyter lab --no-browser --ip 127.0.0.1
Willi Rath's avatar
Willi Rath committed
87 88 89 90 91 92
```

This will print a few lines to your screen and finally show something like:
```
Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
93
        http://127.0.0.1:8888/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Willi Rath's avatar
Willi Rath committed
94 95

```
Willi Rath's avatar
Willi Rath committed
96
Copy the URL (from your shell!) to your web browser.  You should see JupyterLab.
Willi Rath's avatar
Willi Rath committed
97

98 99 100 101 102
>
> _**Note** that at this point, you only have a minimal Python 3 installation with
> a functional JupyterLab frontend.  To perform scientific analyses, you need to add one
> or more environments with a selection of scientific software._
>
103

Willi Rath's avatar
Willi Rath committed
104 105 106

## Add a scientific Python environment and use it

107 108 109 110
> 
> _This will show how to add a Python environment that can be used for scientific
> analyses._
> 
Willi Rath's avatar
Willi Rath committed
111

112 113 114 115 116 117 118 119 120 121 122 123 124
### Add a scientific environment

To add a new environment, first activate the `base` environment:
```bash
source ${HOME}/miniconda3/bin/activate base
```

Then, create an environment (called `py3_std` in this example, and containing
Python 3, `numpy`, `matplotlib`, `scipy`):
```bash
conda create -n py3_std python=3 numpy matplotlib scipy ipykernel
```

125 126 127 128
> 
> _(Do **always** install `ipykernel` into the environment, to make sure that
> JupyterLab recognizes the new environment as a kernel.)_
> 
129 130


Willi Rath's avatar
Willi Rath committed
131
### Use the environment in JupyterLab
132 133

Above, we created a new environment `py3_std`.  To use it, activate the `base`
Willi Rath's avatar
Willi Rath committed
134
environment and start JupyterLab.  There, you should be able to create a new
135 136
notebook and choose the `py3_std` environment as its kernel.

137 138 139 140 141 142 143 144
> 
> _At this point, you know how to set up a full Python installation including a 
> JupyterLab front end in the `base` environment and one or more scientific
> computing environments that can be used in real scientific analyses. The
> instructions above work on any local or remote  Linux / Unix or MacOSX machine.
> Below, you'll learn how to connect to JupyterLab if you installed Python on a
> remote machine that you can only connect to via SSH._
> 
145

146

Willi Rath's avatar
Willi Rath committed
147
## Connect to JupyterLab running on a remote machine
Willi Rath's avatar
Willi Rath committed
148

149 150 151 152
> 
> _This will show how to connect to Jupyterlab if your web browser cannot
> directly access its address._
> 
Willi Rath's avatar
Willi Rath committed
153 154


Willi Rath's avatar
Willi Rath committed
155 156
### Start Jupyterlab on a remote machine

157
After installing Jupyter and your Python environments on a remote machine,
158
follow the instructions on how to [Start JupyterLab](#start-jupyterlab) above.
159 160
But before pasting the URL provided by Jupyter into your browser, make sure to
follow the next steps.
Willi Rath's avatar
Willi Rath committed
161 162 163 164 165 166 167 168 169 170 171


### The essential steps

To let your browser see the network like it looks from a remote machine
(called `host.example.com` here), run the following:
```bash
ssh -f -D localhost:54321 user@host.example.com sleep 15
chromium-browser --proxy-server="socks5://localhost:54321"
```

Willi Rath's avatar
Willi Rath committed
172 173
This will open an SSH session that provides a local network socket listening
on port `54321` that tunnels all traffic through `host.example.com` and then tell
Christina Roth's avatar
Christina Roth committed
174
chromium to use this socket as a proxy server.  (The next step provides a script
Willi Rath's avatar
Willi Rath committed
175 176 177
that bundles this and handles a few caveats like isolating the tunneled browser
session from your other activities on the internet.)

178 179 180 181
> 
> _Note that instead of `54321`, we could have used any free non-privileged (number
> between 1024 and 65535) port._
> 
Willi Rath's avatar
Willi Rath committed
182 183 184 185 186 187 188 189 190 191 192 193 194 195 196


### Wrapped in a script

Download the script (only needed once):
```bash
curl https://git.geomar.de/python/jupyter_on_HPC_setup_guide/raw/master/scripts/run_chromium_through_ssh_tunnel.sh -O
chmod a+x run_chromium_through_ssh_tunnel.sh
```

And run it to connect to `host.example.com`:
```bash
./run_chromium_through_ssh_tunnel.sh user@host.example.com
```

197 198 199 200 201
Note that these steps will only work, if `host.example.com` is a known host in 
`.ssh/known_hosts`. (To test this, simply create a ssh connection to 
`host.example.com`, if it is not yet a known host, you will be asked whether to 
continue and if you do, `host.example.com` will be added to `.ssh/known_hosts`.)

Willi Rath's avatar
Willi Rath committed
202

Willi Rath's avatar
Willi Rath committed
203
### Connect your browser to the remote JupyterLab
204

205 206 207
Now, you can paste the URL (usually of the form
`http://localhost:8888/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`) 
provided by JupyterLab into the tunneled browser.
208 209


210 211 212 213 214 215 216 217 218
## Appendix: Trouble Shooting

- If you get `ImportError: [...]: version GLIBC_[...] not found` errors on
  relatively old machines, make sure to priorize the `defaults` channel over
  `conda-forge`.  This is done by specifying `-c defaults` **before**
  `-c conda-forge` in `conda install` or `conda create` commands.



Willi Rath's avatar
Willi Rath committed
219 220 221 222 223
## Appendix: Jargon

- `conda` is the package manager of the [Anaconda Python
  distribution](https://en.wikipedia.org/wiki/Anaconda_(Python_distribution))

Willi Rath's avatar
Willi Rath committed
224 225 226 227
- [conda environments](https://conda.io/docs/user-guide/tasks/manage-environments.html)
  are collections of packages / versions that make it possible to switch
  between different sets of software.

Willi Rath's avatar
Willi Rath committed
228 229
- [JupyterLab](https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906) is
  an integrated web-based computing environment