Commit 27df292c authored by Willi Rath's avatar Willi Rath Committed by Katharina Höflich

Add jobscripts for Jupyter on HPC nodes

parent ff528744
......@@ -85,7 +85,7 @@ conda activate base
jupyter lab --no-browser --ip 127.0.0.1
```
If you installed Python to the computer you're sitting in front of, this will automatically start a browser and connect to JypyterLab. If this does not happen, copy the URL given at the end of the output of the above commands into your browser:
If you installed Python to the computer you're sitting in front of, this will automatically start a browser and connect to JupyterLab. If this does not happen, copy the URL given at the end of the output of the above commands into your browser:
```
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
......@@ -102,7 +102,7 @@ In JupyterLab, you should be able to create a new notebook and choose the `py3_s
At this point, you know how to set up a full Python installation including a JupyterLab front end in the `base` environment and one or more scientific computing environments that can be used in real scientific analyses. The instructions above work on any local or remote Linux / Unix or MacOSX machine. Below, you'll learn how to connect to JupyterLab if you installed Python on a remote machine that you can only connect to via SSH.
### Start Jupyterlab on a remote machine
### Start JupyterLab on a remote machine
After installing Jupyter and your Python environments on a remote machine, follow the instructions on how to [Use the environment in JupyterLab](#use-the-environment-in-jupyterlab) above. But before pasting the URL provided by Jupyter into your browser, make sure to follow the next steps.
......@@ -142,6 +142,47 @@ First, make sure to have `Git bash` installed. You can obtain it by installing
Then, follow the steps above but replace `run_chromium_through_ssh_tunnel.sh` by `run_chromium_through_ssh_tunnel_WINDOWS.sh`.
## Start JupyterLab on a compute node of an HPC centre
In [job-scripts/](job-scripts/), there's example scripts showing how to start a job that runs JupyterLab on a compute node.
### On Nesh
With [`nesh-linux-cluser-jupyterlab.sh`](job-scripts/nesh-linux-cluser-jupyterlab.sh), you can submit a job as follows:
```shell
qsub nesh-linux-cluster-jupyterlab.sh \
-l elapstim_req=<hh:mm:ss> \
-b <node-no> \
-l cpunum_job=<cpu-no> \
-l memsz_job=<mem-size> \
-q <batch-class>
```
Checking the status of the job and retrieving the URL on which to connect to JupyterLab is done with:
```shell
bash nesh-linux-cluster-jupyterlab.sh <jobid>
```
### At HLRN Berlin
With [`hlrn-goettingen-jupyterlab.sh`](job-scripts/hlrn-goettingen-jupyterlab.sh), you can submit a job with
```shell
sbatch -t <hh:mm:ss> hlrn-goettingen-jupyterlab.sh
```
where `<hh:mm:ss>` specifies the desired walltime.
Checking the status of the job and retrieving the URL on which to connect to JupyterLab is done with:
```shell
bash hlrn-goettingen-jupyterlab.sh <jobid>
```
You can also just run
```shell
bash hlrn-goettingen-jupyterlab.sh
```
to start jupyterlab where you are (e.g. on a login node).
## Appendix: Trouble Shooting
- If you get `ImportError: [...]: version GLIBC_[...] not found` errors on relatively old machines, make sure to priorize the `defaults` channel over `conda-forge`. This is done by specifying `-c defaults` **before** `-c conda-forge` in `conda install` or `conda create` commands.
......
#!/usr/bin/env bash
#SBATCH --job-name=jupyterlab
#SBATCH --partition=large:shared
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --output=/path/to/logs/jlab.o%j
#SBATCH --error=/path/to/logs/jlab.o%j
#SBATCH --mail-user=usermailaddress
#SBATCH --mail-type=END,FAIL
#SBATCH --account=username
#
# to use the script, modify the user-specific entries to the SBATCH commands
# and then run sbatch -t hh:mm:ss ./hlrn-goettingen-jupyterlab.sh
# -t hh:mm:ss specifies the walltime
# you can also just run ./hlrn-goettingen-jupyterlab.sh to start jupyterlab
# where you are (e.g. on a login node)
#
export XDG_RUNTIME_DIR=$LOCAL_TMPDIR
if [ -n "${1}" ]; then
scontrol show job ${1} &> /dev/null || { echo "Job does not exist..."; exit; }
status=$(scontrol show job ${1} | grep 'JobState' | awk -F ' ' '{print $1}')
if [ "$status" == 'JobState=PENDING' ]; then
squeue --start -j ${1} | awk -F ' ' '{print $6}'
else
scontrol show job ${1} | grep "RunTime" | awk -F ' ' '{print $1}'
ofile=$(scontrol show job ${1} | grep 'StdErr' | awk -F ' ' '{print $1}'); ofile=${ofile##*=}
cat ${ofile} | grep -m1 -e 'token' || echo 'Jupyter Lab not yet properly running... please wait.'
fi
else
cd ${HOME}
echo "host $(hostname)"
conda activate base && \
jupyter lab --ip=$(hostname) --no-browser
# for conda versions below 4.6 use
# source $HOME/miniconda3/bin/activate base && \
# jupyter lab --ip=$(hostname) --no-browser
fi
#!/bin/bash
#PBS -N jupyterlab -o jupyterlab.out -j o
#
# NEC frontends
#
# ./nesh-linux-cluster-jupyterlab.sh
#
# workflow for NEC Linux cluster:
# ```
# $ qsub nesh-linux-cluster-jupyterlab.sh -l elapstim_req=<hh:mm:ss> -b <node-no> -l cpunum_job=<cpu-no> -l memsz_job=<mem-size> -q <batch-class>
# $ ./nesh-linux-cluster-jupyterlab.sh <jobid>
# $ qdel <jobid> # free resources after you are done!
# ```
#
# short example:
# ```
# $ qsub nesh-linux-cluster-jupyterlab.sh -l elapstim_req=<00:05:00> -b 1 -l cpunum_job=4 -l memsz_job=8gb -q clmedium
# Request 134408.nesh-batch submitted to queue: clmedium.
# $ ./nesh-linux-cluster-jupyterlab.sh 134408
# 4.53 minutes remaining elapse time at host neshcl305
# [I 16:38:35.025 LabApp] http://neshcl305:8889/?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# $ qdel 134408
# Request 134408.nesh-batch was deleted.
# ```
#
if [ -n "${1}" ]; then
qstat ${1} | grep -q "does not exist" && { echo "Job does not exist..."; exit; }
status=$(qstat ${1} -f | echo $(string=$(grep 'Current State'); echo ${string##* }))
if [ "$status" == 'Queued' ]; then
echo $(qstat ${1} -f | echo "$(grep 'Planned Start Time')")
else
qstat ${1} -f | echo "scale=2; $(string=$(grep 'Remaining Elapse'); echo ${string//[!0-9]})/60" | \
echo "$(bc) minutes remaining elapse time at" $(qcat -o -b -n25 ${1} | grep -m1 -e 'host') &&
qcat -o -b -n25 ${1} | grep -m1 -e 'token' || echo 'Jupyter Lab not yet properly running... please wait.'
fi
else
cd $PBS_O_WORKDIR
echo "host $(hostname)"
conda activate base && \
jupyter lab --ip=$(hostname) --no-browser
# for conda versions below 4.6 use
# source $HOME/miniconda3/bin/activate base && \
# jupyter lab --ip=$(hostname) --no-browser
fi
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment