Commit 2bfca7e1 authored by Willi Rath's avatar Willi Rath

Rewrite using local mirrors and templates

parent f41b1e77
Pipeline #548 passed with stage
in 11 minutes and 16 seconds
...@@ -5,5 +5,5 @@ test: ...@@ -5,5 +5,5 @@ test:
- env - env
- source activate root - source activate root
- target_dir=${CONDA_PREFIX} - target_dir=${CONDA_PREFIX}
- ./update_anaconda.sh ${target_dir} - shopt -s extglob
- ./update_anaconda.sh ${target_dir} envs/!(monitoring_gallery).yml
...@@ -6,37 +6,23 @@ Collection of conda environments and info on where and how to use them. ...@@ -6,37 +6,23 @@ Collection of conda environments and info on where and how to use them.
## Contribute ## Contribute
**Recommended:** [File an issue here][issues] and describe what you need. **Recommended:** [File an issue here][issues] and describe what you need.
**Or:** [Fork this repository here][forks], add an evrionment, and create a **Or:** [Fork this repository here][forks], add an evrionment, and create a
merge request. merge request. (Adding / editing the respective `_env_templates/*yml` and
running `python create_env_files.py` would be ideal.)
[issues]: https://git.geomar.de/python/conda_environments/issues [issues]: https://git.geomar.de/python/conda_environments/issues
[forks]: https://git.geomar.de/python/conda_environments/forks [forks]: https://git.geomar.de/python/conda_environments/forks
## See available environments ## See available environments
Have a look at Have a look at
[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files). [envs/](https://git.geomar.de/python/conda_environments/tree/master/_envs/).
Currently, there are
- [`ffmpeg`](https://git.geomar.de/python/conda_environments/tree/master/env_files/ffmpeg.yml)
: Provides nothing but `ffmpeg`.
- [`matlab_kernel`](https://git.geomar.de/python/conda_environments/tree/master/env_files/matlab_kernel.yml)
: Provides Jupyter with a Matlab kernel. See <https://git.geomar.de/python/nb_kernels/issues/4> for details.
- [`monitoring`](https://git.geomar.de/python/conda_environments/tree/master/env_files/monitoring.yml)
: Provides an environment used by
[TM/Monitoring](https://git.geomar.de/TM/Monitoring).
- [`py2_std`](https://git.geomar.de/python/conda_environments/tree/master/env_files/py2_std.yml)
: Provides a comprehensive (still growing) Python2 stack.
- [`py3_std`](https://git.geomar.de/python/conda_environments/tree/master/env_files/py3_std.yml)
: Provides a comprehensive (still growing) Python3 stack.
- [`pyferret`](https://git.geomar.de/python/conda_environments/tree/master/env_files/pyferret.yml)
: Provides nothing but `pyferret`.
There are also a few non-standard environments in
[opt_env_files/](https://git.geomar.de/python/conda_environments/tree/master/opt_env_files).
There, e.g., is an environment `py3_std` described in the file `py2_std.yml`.
As with this example, all other env names also match the file name before the
`.yml`.
## Using the environments ## Using the environments
...@@ -71,14 +57,17 @@ There is a setup script doing a full setup: ...@@ -71,14 +57,17 @@ There is a setup script doing a full setup:
Run it without arguments to get a help message. Run it without arguments to get a help message.
To install all environments defined in To install all environments defined in
[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files) [envs/](https://git.geomar.de/python/conda_environments/tree/master/envs/)
to a path `<target_dir>`, run: to a path `<target_dir>`, run:
```bash ```bash
git clone git@git.geomar.de:python/conda_environments.git git clone git@git.geomar.de:python/conda_environments.git
cd conda_environments cd conda_environments
./setup_anaconda.sh <target dir> ./setup_anaconda.sh <target dir> envs/*
``` ```
> **Note** that you might not want to install all environments. If you're
> unsure, use only `envs/py3_std.yml` (and maybe `py2_std.yml`).
Then, follow the steps above to activate the respective enviroment. Then, follow the steps above to activate the respective enviroment.
### Complete setup on MacOSX ### Complete setup on MacOSX
...@@ -89,14 +78,17 @@ There is a setup script doing a full setup: ...@@ -89,14 +78,17 @@ There is a setup script doing a full setup:
Run it without arguments to get a help message. Run it without arguments to get a help message.
To install all environments defined in To install all environments defined in
[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files) [envs/](https://git.geomar.de/python/conda_environments/tree/master/envs/)
to a path `<target_dir>`, run: to a path `<target_dir>`, run:
```bash ```bash
git clone git@git.geomar.de:python/conda_environments.git git clone git@git.geomar.de:python/conda_environments.git
cd conda_environments cd conda_environments
./setup_anaconda_MacOSX.sh <target dir> ./setup_anaconda_MacOSX.sh <target dir> envs/*
``` ```
> **Note** that you might not want to install all environments. If you're
> unsure, use only `envs/py3_std.yml` (and maybe `py2_std.yml`).
Then, follow the steps above to activate the respective enviroment. Then, follow the steps above to activate the respective enviroment.
### Adding an environment to an existing setup ### Adding an environment to an existing setup
...@@ -108,7 +100,7 @@ source <path-to-miniconda-bin>/activate root ...@@ -108,7 +100,7 @@ source <path-to-miniconda-bin>/activate root
``` ```
and then use and then use
```bash ```bash
conda env create -f env_files/<envname>.yml conda env create -f envs/<envname>.yml
``` ```
### Updating an environment ### Updating an environment
...@@ -118,7 +110,7 @@ There is an update script: ...@@ -118,7 +110,7 @@ There is an update script:
In principle, you can run: In principle, you can run:
```bash ```bash
conda env update -f env_files/<envname>.yml conda env update -f envs/<envname>.yml
``` ```
There may, however, be envs using `pip` to download source code directly. There may, however, be envs using `pip` to download source code directly.
These may fail to really update. These may fail to really update.
...@@ -129,6 +121,15 @@ conda env remove -n <envname> ...@@ -129,6 +121,15 @@ conda env remove -n <envname>
``` ```
and then and then
```bash ```bash
conda env create -f env_files/<envname>.yml conda env create -f envs/<envname>.yml
``` ```
## Advanced topics
### Install from an offline mirror
In `_envs_*`, there are env files matching those in the main directory which
point to local mirrors instead of the channels. Use these if you're behind a
firewall. (But note that on the machines where local mirrors are available,
there should be a working installation providing these envs anyway.)
name: atom_env
channels:
- {anaconda}
- {conda-forge}
dependencies:
- python=3
- anaconda-client
- autopep8
- flake8
- flake8-docstrings
- flake8-polyfill
- pep8
- pip
- pycodestyle
- pydocstyle
- pyflakes
- pylama
- pytest
# Python 3 standard environment
name: ffmpeg name: ffmpeg
channels: channels:
- conda-forge - {anaconda}
- {conda-forge}
dependencies: dependencies:
- python=3 - python=3
- anaconda-client - anaconda-client
......
name: matlab_kernel name: matlab_kernel
channels: channels:
- conda-forge - {anaconda}
- {conda-forge}
dependencies: dependencies:
- python=3.4 - python=3.4
- jupyter - jupyter
......
name: monitoring name: monitoring
channels: channels:
- willirath - {anaconda}
- conda-forge - {willirath}
- {conda-forge}
dependencies: dependencies:
- python=2.7.12 - python=2.7.12
- anaconda-client - anaconda-client
......
name: monitoring_gallery name: monitoring_gallery
channels: channels:
- conda-forge - {conda-forge}
- {anaconda}
dependencies: dependencies:
- python=2.7 - python=2.7
- anaconda-client - anaconda-client
......
name: nco name: nco
channels: channels:
- conda-forge - {anaconda}
- {conda-forge}
dependencies: dependencies:
- anaconda-client - anaconda-client
- nco - nco
# Python 2.7 standard environment
name: py2_std name: py2_std
channels: channels:
- willirath - {anaconda}
- conda-forge - {willirath}
- {conda-forge}
dependencies: dependencies:
- python=2.7.12 - python=2.7.12
- anaconda-client - anaconda-client
......
# Python 3 standard environment
name: py3_std name: py3_std
channels: channels:
- willirath - {anaconda}
- conda-forge - {willirath}
- {conda-forge}
dependencies: dependencies:
- python=3 - python=3
- anaconda-client - anaconda-client
......
name: pyferret name: pyferret
channels: channels:
- conda-forge - {anaconda}
- {conda-forge}
dependencies: dependencies:
- matplotlib=1.5.3 - matplotlib=1.5.3
- pyferret - pyferret
......
name: tm_deploy name: tm_deploy
channels: channels:
- conda-forge - {anaconda}
- {conda-forge}
dependencies: dependencies:
- python=3 - python=3
- cartopy - cartopy
# Python 3 standard environment
name: ffmpeg
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- ffmpeg
name: matlab_kernel
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3.4
- jupyter
- pip:
- matlab_kernel
name: monitoring
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- ipython
- jupyter
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- seaborn
- pip:
- "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"
name: monitoring_gallery
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
dependencies:
- python=2.7
- anaconda-client
- ConfigParser
- ffmpeg
- pip:
- sigal
- "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"
name: nco
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pyngl
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
- pip:
- pymatlab
# Python 3 standard environment
name: py3_std
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
name: pyferret
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- matplotlib=1.5.3
- pyferret
- ferret_datasets
name: tm_deploy
channels:
- file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
- file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- cartopy
# Python 3 standard environment
name: ffmpeg
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- ffmpeg
name: matlab_kernel
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3.4
- jupyter
- pip:
- matlab_kernel
name: monitoring
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- ipython
- jupyter
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- seaborn
- pip:
- "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"
name: monitoring_gallery
channels:
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
dependencies:
- python=2.7
- anaconda-client
- ConfigParser
- ffmpeg
- pip:
- sigal
- "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"
name: nco
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=2.7.12
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pyngl
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
- pip:
- pymatlab
# Python 3 standard environment
name: py3_std
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- pytables
- seaborn
- seawater
- spyder
- texlive-core
- xarray
name: pyferret
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- matplotlib=1.5.3
- pyferret
- ferret_datasets
name: tm_deploy
channels:
- file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
- file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
dependencies:
- python=3
- cartopy
import configparser
import logging
from pathlib import Path
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s : %(levelname)-8s %(message)s')
logger = logging.getLogger(__name__)
# Open config file detailing the paths / URLs to the channels for different
# machines.
config_file_name = 'locations.ini'
config = configparser.ConfigParser()
config.read(config_file_name)
# For each section (indicating the location or kind of network environment),
# read all env templates, expand the channel placeholders, and write them to
# the respective yml-files.
for section_name in config.sections():
for input_file_name in Path('_env_templates').glob('*.yml'):
env_dir = Path(section_name)
env_dir.mkdir(exist_ok=True)
output_file_name = env_dir / input_file_name.name
with input_file_name.open() as input_file, \
output_file_name.open(mode='w') as output_file:
logger.info('Creating {}'.format(output_file.name))
template = input_file.read()
output_file.write(template.format(**config[section_name]))
# Python 3 standard environment
name: ffmpeg
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- python=3
- anaconda-client
- ffmpeg
name: matlab_kernel
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- python=3.4
- jupyter
- pip:
- matlab_kernel
name: monitoring
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/willirath
- https://conda.anaconda.org/conda-forge
dependencies:
- python=2.7.12
- anaconda-client
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- ipython
- jupyter
- matplotlib=1.5.3
- nco
- netCDF4
- numpy
- pandas
- seaborn
- pip:
- "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"
name: monitoring_gallery
channels:
- https://conda.anaconda.org/conda-forge
- https://repo.continuum.io/pkgs/free
dependencies:
- python=2.7
- anaconda-client
- ConfigParser
- ffmpeg
- pip:
- sigal
- "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"
name: nco
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/conda-forge
dependencies:
- anaconda-client
- nco
# Python 2.7 standard environment
name: py2_std
channels:
- https://repo.continuum.io/pkgs/free
- https://conda.anaconda.org/willirath
- https://conda.anaconda.org/conda-forge
dependencies:
- python=2.7.12
- anaconda-client
- aospy
- basemap
- cartopy
- cartopy_offlinedata
- cf_units
- cmocean
- dask
- eofs
- haversine
- hdf5
- ipython
- jupyter
- line_profiler
- matplotlib=1.5.3
- nco
- netCDF4
-