Skip to content

Outline

1. Current status

Splitting computation into independent tasks:

for year in {1958..2017}; do
    ./calculate_for_year.sh ${year}
done

Or (naively) parallelizing over these independent tasks:

msub <<EOF
#!/bin/bash
#PBS -l feature=smp2
#PBS -l nodes=1:ppn=40
#PBS -l walltime=12:00:00

echo {1958..2017} | tr ' ' '\n' | xargs -n1 -P${PBS_NP} ./calculate_for_year.sh ${year}

EOF

2. Problems with current status

  • User has to explicitly care for parallelization
    • choose "axis" along which do parallelize
    • set up the logics and logistics of the parallelizatoin
  • does not scale well beyond limits of a CPU socket / node / data center / ...

3. Towards distributed systems

image

4. Demo

...

Edited by Willi Rath