Commit 2bfca7e1 authored by Willi Rath's avatar Willi Rath

Rewrite using local mirrors and templates

parent f41b1e77
Pipeline #548 passed with stage
in 11 minutes and 16 seconds
......@@ -5,5 +5,5 @@ test:
- env
- source activate root
- target_dir=${CONDA_PREFIX}
- ./update_anaconda.sh ${target_dir}
- shopt -s extglob
- ./update_anaconda.sh ${target_dir} envs/!(monitoring_gallery).yml
......@@ -6,37 +6,23 @@ Collection of conda environments and info on where and how to use them.
## Contribute
**Recommended:** [File an issue here][issues] and describe what you need.
**Recommended:** [File an issue here][issues] and describe what you need.
**Or:** [Fork this repository here][forks], add an evrionment, and create a
merge request.
merge request. (Adding / editing the respective `_env_templates/*yml` and
running `python create_env_files.py` would be ideal.)
[issues]: https://git.geomar.de/python/conda_environments/issues
[forks]: https://git.geomar.de/python/conda_environments/forks
[issues]: https://git.geomar.de/python/conda_environments/issues
[forks]: https://git.geomar.de/python/conda_environments/forks
## See available environments
Have a look at
[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files).
Currently, there are
- [`ffmpeg`](https://git.geomar.de/python/conda_environments/tree/master/env_files/ffmpeg.yml)
: Provides nothing but `ffmpeg`.
- [`matlab_kernel`](https://git.geomar.de/python/conda_environments/tree/master/env_files/matlab_kernel.yml)
: Provides Jupyter with a Matlab kernel. See <https://git.geomar.de/python/nb_kernels/issues/4> for details.
- [`monitoring`](https://git.geomar.de/python/conda_environments/tree/master/env_files/monitoring.yml)
: Provides an environment used by
[TM/Monitoring](https://git.geomar.de/TM/Monitoring).
- [`py2_std`](https://git.geomar.de/python/conda_environments/tree/master/env_files/py2_std.yml)
: Provides a comprehensive (still growing) Python2 stack.
- [`py3_std`](https://git.geomar.de/python/conda_environments/tree/master/env_files/py3_std.yml)
: Provides a comprehensive (still growing) Python3 stack.
- [`pyferret`](https://git.geomar.de/python/conda_environments/tree/master/env_files/pyferret.yml)
: Provides nothing but `pyferret`.
There are also a few non-standard environments in
[opt_env_files/](https://git.geomar.de/python/conda_environments/tree/master/opt_env_files).
[envs/](https://git.geomar.de/python/conda_environments/tree/master/_envs/).
There, e.g., is an environment `py3_std` described in the file `py2_std.yml`.
As with this example, all other env names also match the file name before the
`.yml`.
## Using the environments
......@@ -71,14 +57,17 @@ There is a setup script doing a full setup:
Run it without arguments to get a help message.
To install all environments defined in
[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files)
[envs/](https://git.geomar.de/python/conda_environments/tree/master/envs/)
to a path `<target_dir>`, run:
```bash
git clone git@git.geomar.de:python/conda_environments.git
cd conda_environments
./setup_anaconda.sh <target dir>
./setup_anaconda.sh <target dir> envs/*
```
> **Note** that you might not want to install all environments. If you're
> unsure, use only `envs/py3_std.yml` (and maybe `py2_std.yml`).
Then, follow the steps above to activate the respective enviroment.
### Complete setup on MacOSX
......@@ -89,14 +78,17 @@ There is a setup script doing a full setup:
Run it without arguments to get a help message.
To install all environments defined in
[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files)
[envs/](https://git.geomar.de/python/conda_environments/tree/master/envs/)
to a path `<target_dir>`, run:
```bash
git clone git@git.geomar.de:python/conda_environments.git
cd conda_environments
./setup_anaconda_MacOSX.sh <target dir>
./setup_anaconda_MacOSX.sh <target dir> envs/*
```
> **Note** that you might not want to install all environments. If you're
> unsure, use only `envs/py3_std.yml` (and maybe `py2_std.yml`).
Then, follow the steps above to activate the respective enviroment.
### Adding an environment to an existing setup
......@@ -108,7 +100,7 @@ source <path-to-miniconda-bin>/activate root
```
and then use
```bash
conda env create -f env_files/<envname>.yml
conda env create -f envs/<envname>.yml
```
### Updating an environment
......@@ -118,7 +110,7 @@ There is an update script:
In principle, you can run:
```bash
conda env update -f env_files/<envname>.yml
conda env update -f envs/<envname>.yml
```
There may, however, be envs using `pip` to download source code directly.
These may fail to really update.
......@@ -129,6 +121,15 @@ conda env remove -n <envname>
```
and then
```bash
conda env create -f env_files/<envname>.yml
conda env create -f envs/<envname>.yml
```
## Advanced topics
### Install from an offline mirror
In `_envs_*`, there are env files matching those in the main directory which
point to local mirrors instead of the channels. Use these if you're behind a
firewall. (But note that on the machines where local mirrors are available,
there should be a working installation providing these envs anyway.)
name: atom_env
channels:
- {anaconda}
- {conda-forge}
dependencies:
- python=3
- anaconda-client
- autopep8
- flake8
- flake8-docstrings
- flake8-polyfill
- pep8
- pip
- pycodestyle
- pydocstyle
- pyflakes
- pylama
- pytest
# Python 3 standard environment
name: ffmpeg
channels:
- conda-forge
- {anaconda}
- {conda-forge}
dependencies:
- python=3
- anaconda-client
......
name: matlab_kernel
channels:
- conda-forge
- {anaconda}
- {conda-forge}
dependencies:
- python=3.4
- jupyter
......
name: monitoring
channels:
- willirath
- conda-forge
- {anaconda}
- {willirath}
- {conda-forge}
dependencies:
- python=2.7.12
- anaconda-client
......
name: monitoring_gallery
channels:
- conda-forge
- {conda-forge}
- {anaconda}
dependencies:
- python=2.7
- anaconda-client
......
name: nco
channels:
- conda-forge
- {anaconda}
- {conda-forge}
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- willirath
- conda-forge
- {anaconda}
- {willirath}
- {conda-forge}
dependencies:
- python=2.7.12
- anaconda-client
......
# Python 3 standard environment
name: py3_std
channels:
- willirath
- conda-forge
- {anaconda}
- {willirath}
- {conda-forge}
dependencies:
- python=3
- anaconda-client
......
name: pyferret
channels:
- conda-forge
- {anaconda}
- {conda-forge}
dependencies:
- matplotlib=1.5.3
- pyferret
......
name: tm_deploy
channels:
- conda-forge
- {anaconda}
- {conda-forge}
dependencies:
- python=3
- cartopy
# Python 3 standard environment
name: ffmpeg
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- ffmpeg
name: matlab_kernel
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3.4
- jupyter
- pip:
- matlab_kernel
name: monitoring
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- ipython
- jupyter
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- seaborn
- pip:
- "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"
name: monitoring_gallery
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
dependencies:
- python=2.7
- anaconda-client
- ConfigParser
- ffmpeg
- pip:
- sigal
- "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"
name: nco
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pyngl
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
- pip:
- pymatlab
# Python 3 standard environment
name: py3_std
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
name: pyferret
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- matplotlib=1.5.3
- pyferret
- ferret_datasets
name: tm_deploy
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- cartopy
# Python 3 standard environment
name: ffmpeg
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- ffmpeg
name: matlab_kernel
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3.4
- jupyter
- pip:
- matlab_kernel
name: monitoring
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- ipython
- jupyter
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- seaborn
- pip:
- "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"
name: monitoring_gallery
channels:
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
dependencies:
- python=2.7
- anaconda-client
- ConfigParser
- ffmpeg
- pip:
- sigal
- "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"
name: nco
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pyngl
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
- pip:
- pymatlab
# Python 3 standard environment
name: py3_std
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
name: pyferret
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- matplotlib=1.5.3
- pyferret
- ferret_datasets
name: tm_deploy
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- cartopy
import configparser
import logging
from pathlib import Path
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s : %(levelname)-8s %(message)s')
logger = logging.getLogger(__name__)
# Open config file detailing the paths / URLs to the channels for different
# machines.
config_file_name = 'locations.ini'
config = configparser.ConfigParser()
config.read(config_file_name)
# For each section (indicating the location or kind of network environment),
# read all env templates, expand the channel placeholders, and write them to
# the respective yml-files.
for section_name in config.sections():
for input_file_name in Path('_env_templates').glob('*.yml'):
env_dir = Path(section_name)
env_dir.mkdir(exist_ok=True)
output_file_name = env_dir / input_file_name.name
with input_file_name.open() as input_file, \
output_file_name.open(mode='w') as output_file:
logger.info('Creating {}'.format(output_file.name))
template = input_file.read()
output_file.write(template.format(**config[section_name]))
# Python 3 standard environment
name: ffmpeg
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- python=3
- anaconda-client
- ffmpeg
name: matlab_kernel
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- python=3.4
- jupyter
- pip:
- matlab_kernel
name: monitoring
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/willirath
- https://conda.anaconda.org/conda-forge
dependencies:
- python=2.7.12
- anaconda-client
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- ipython
- jupyter
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- seaborn
- pip:
- "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"
name: monitoring_gallery
channels:
- https://conda.anaconda.org/conda-forge
- https://repo.continuum.io/pkgs/free
dependencies:
- python=2.7
- anaconda-client
- ConfigParser
- ffmpeg
- pip:
- sigal
- "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"
name: nco
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/willirath
- https://conda.anaconda.org/conda-forge
dependencies:
- python=2.7.12
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pyngl
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
- pip:
- pymatlab
# Python 3 standard environment
name: py3_std
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/willirath
- https://conda.anaconda.org/conda-forge
dependencies:
- python=3
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
name: pyferret
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- matplotlib=1.5.3
- pyferret
- ferret_datasets
name: tm_deploy
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- python=3
- cartopy
[envs]
anaconda = https://repo.continuum.io/pkgs/free
willirath = https://conda.anaconda.org/willirath
conda-forge = https://conda.anaconda.org/conda-forge