Rewrite using local mirrors and templates

.gitlab-ci.yml

@@ -5,5 +5,5 @@ test:
     - env
     - source activate root
     - target_dir=${CONDA_PREFIX}
-    - ./update_anaconda.sh ${target_dir}
- 
+    - shopt -s extglob
+    - ./update_anaconda.sh ${target_dir} envs/!(monitoring_gallery).yml

README.md

@@ -9,7 +9,8 @@ Collection of conda environments and info on where and how to use them.
 **Recommended:** [File an issue here][issues] and describe what you need.
 
 **Or:** [Fork this repository here][forks], add an evrionment, and create a
-merge request.
+merge request.  (Adding / editing the respective `_env_templates/*yml` and
+running `python create_env_files.py` would be ideal.)
 
 [issues]: https://git.geomar.de/python/conda_environments/issues
 [forks]: https://git.geomar.de/python/conda_environments/forks
@@ -17,26 +18,11 @@ merge request.
 ## See available environments
 
 Have a look at
-[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files).
-Currently, there are
-
-- [`ffmpeg`](https://git.geomar.de/python/conda_environments/tree/master/env_files/ffmpeg.yml)
-  : Provides nothing but `ffmpeg`.
-- [`matlab_kernel`](https://git.geomar.de/python/conda_environments/tree/master/env_files/matlab_kernel.yml)
-  : Provides Jupyter with a Matlab kernel.  See <https://git.geomar.de/python/nb_kernels/issues/4> for details.
-- [`monitoring`](https://git.geomar.de/python/conda_environments/tree/master/env_files/monitoring.yml)
-  : Provides an environment used by 
-  [TM/Monitoring](https://git.geomar.de/TM/Monitoring).
-- [`py2_std`](https://git.geomar.de/python/conda_environments/tree/master/env_files/py2_std.yml)
-  : Provides a comprehensive (still growing) Python2 stack.
-- [`py3_std`](https://git.geomar.de/python/conda_environments/tree/master/env_files/py3_std.yml)
-  : Provides a comprehensive (still growing) Python3 stack.
-- [`pyferret`](https://git.geomar.de/python/conda_environments/tree/master/env_files/pyferret.yml)
-  : Provides nothing but `pyferret`.
-
-There are also a few non-standard environments in
-[opt_env_files/](https://git.geomar.de/python/conda_environments/tree/master/opt_env_files).
+[envs/](https://git.geomar.de/python/conda_environments/tree/master/_envs/).
 
+There, e.g., is an environment `py3_std` described in the file `py2_std.yml`.
+As with this example, all other env names also match the file name before the
+`.yml`.
 
 ## Using the environments
 
@@ -71,14 +57,17 @@ There is a setup script doing a full setup:
 Run it without arguments to get a help message.
 
 To install all environments defined in
-[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files)
+[envs/](https://git.geomar.de/python/conda_environments/tree/master/envs/)
 to a path `<target_dir>`, run:
 ```bash
 git clone git@git.geomar.de:python/conda_environments.git
 cd conda_environments
-./setup_anaconda.sh <target dir>
+./setup_anaconda.sh <target dir> envs/*
 ```
 
+> **Note** that you might not want to install all environments.  If you're
+> unsure, use only `envs/py3_std.yml` (and maybe `py2_std.yml`).
+
 Then, follow the steps above to activate the respective enviroment.
 
 ### Complete setup on MacOSX
@@ -89,14 +78,17 @@ There is a setup script doing a full setup:
 Run it without arguments to get a help message.
 
 To install all environments defined in
-[env_files/](https://git.geomar.de/python/conda_environments/tree/master/env_files)
+[envs/](https://git.geomar.de/python/conda_environments/tree/master/envs/)
 to a path `<target_dir>`, run:
 ```bash
 git clone git@git.geomar.de:python/conda_environments.git
 cd conda_environments
-./setup_anaconda_MacOSX.sh <target dir>
+./setup_anaconda_MacOSX.sh <target dir> envs/*
 ```
 
+> **Note** that you might not want to install all environments.  If you're
+> unsure, use only `envs/py3_std.yml` (and maybe `py2_std.yml`).
+
 Then, follow the steps above to activate the respective enviroment.
 
 ### Adding an environment to an existing setup
@@ -108,7 +100,7 @@ source <path-to-miniconda-bin>/activate root
 ```
 and then use
 ```bash
-conda env create -f env_files/<envname>.yml
+conda env create -f envs/<envname>.yml
 ```
 
 ### Updating an environment
@@ -118,7 +110,7 @@ There is an update script:
 
 In principle, you can run:
 ```bash
-conda env update -f env_files/<envname>.yml
+conda env update -f envs/<envname>.yml
 ```
 There may, however, be envs using `pip` to download source code directly.
 These may fail to really update.
@@ -129,6 +121,15 @@ conda env remove -n <envname>
 ```
 and then
 ```bash
-conda env create -f env_files/<envname>.yml
+conda env create -f envs/<envname>.yml
 ```
 
+
+## Advanced topics
+
+### Install from an offline mirror
+
+In `_envs_*`, there are env files matching those in the main directory which
+point to local mirrors instead of the channels.  Use these if you're behind a
+firewall.  (But note that on the machines where local mirrors are available,
+there should be a working installation providing these envs anyway.)

_env_templates/atom_env.yml

+
+name: atom_env
+channels:
+  - {anaconda}
+  - {conda-forge}
+dependencies:
+  - python=3
+  - anaconda-client
+  - autopep8
+  - flake8
+  - flake8-docstrings
+  - flake8-polyfill
+  - pep8
+  - pip
+  - pycodestyle
+  - pydocstyle
+  - pyflakes
+  - pylama
+  - pytest

env_files/ffmpeg.yml → _env_templates/ffmpeg.yml

-# Python 3 standard environment
 name: ffmpeg
 channels:
-  - conda-forge
+  - {anaconda}
+  - {conda-forge}
 dependencies:
   - python=3
   - anaconda-client

env_files/matlab_kernel.yml → _env_templates/matlab_kernel.yml

 name: matlab_kernel
 channels:
-  - conda-forge
+  - {anaconda}
+  - {conda-forge}
 dependencies:
   - python=3.4
   - jupyter

env_files/monitoring.yml → _env_templates/monitoring.yml

 name: monitoring
 channels:
-  - willirath
-  - conda-forge
+  - {anaconda}
+  - {willirath}
+  - {conda-forge}
 dependencies:
   - python=2.7.12
   - anaconda-client

opt_env_files/monitoring_gallery.yml → _env_templates/monitoring_gallery.yml

 name: monitoring_gallery
 channels:
-  - conda-forge
+  - {conda-forge}
+  - {anaconda}
 dependencies:
   - python=2.7
   - anaconda-client

opt_env_files/nco.yml → _env_templates/nco.yml

 name: nco
 channels:
-  - conda-forge
+  - {anaconda}
+  - {conda-forge}
 dependencies:
   - anaconda-client
   - nco

env_files/py2_std.yml → _env_templates/py2_std.yml

-# Python 2.7 standard environment
 name: py2_std
 channels:
-  - willirath
-  - conda-forge
+  - {anaconda}
+  - {willirath}
+  - {conda-forge}
 dependencies:
   - python=2.7.12
   - anaconda-client

env_files/py3_std.yml → _env_templates/py3_std.yml

-# Python 3 standard environment
 name: py3_std
 channels:
-  - willirath
-  - conda-forge
+  - {anaconda}
+  - {willirath}
+  - {conda-forge}
 dependencies:
   - python=3
   - anaconda-client

env_files/pyferret.yml → _env_templates/pyferret.yml

 name: pyferret
 channels:
-  - conda-forge
+  - {anaconda}
+  - {conda-forge}
 dependencies:
   - matplotlib=1.5.3
   - pyferret

opt_env_files/tm_deploy.yml → _env_templates/tm_deploy.yml

 name: tm_deploy
 channels:
-  - conda-forge
+  - {anaconda}
+  - {conda-forge}
 dependencies:
   - python=3
   - cartopy

_envs_hlrn/ffmpeg.yml

+# Python 3 standard environment
+name: ffmpeg
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3
+  - anaconda-client
+  - ffmpeg

_envs_hlrn/matlab_kernel.yml

+name: matlab_kernel
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3.4
+  - jupyter
+  - pip:
+     - matlab_kernel

_envs_hlrn/monitoring.yml

+name: monitoring
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=2.7.12
+  - anaconda-client
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - ipython
+  - jupyter
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - seaborn
+  - pip:
+     - "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"

_envs_hlrn/monitoring_gallery.yml

+name: monitoring_gallery
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+dependencies:
+  - python=2.7
+  - anaconda-client
+  - ConfigParser
+  - ffmpeg
+  - pip:
+    - sigal
+    - "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"

_envs_hlrn/nco.yml

+name: nco
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - anaconda-client
+  - nco

_envs_hlrn/py2_std.yml

+# Python 2.7 standard environment
+name: py2_std
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=2.7.12
+  - anaconda-client
+  - aospy
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - eofs
+  - haversine
+  - hdf5
+  - ipython
+  - jupyter
+  - line_profiler
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - pyngl
+  - pytables
+  - seaborn
+  - seawater
+  - spyder
+  - texlive-core
+  - xarray
+  - pip:
+      - pymatlab

_envs_hlrn/py3_std.yml

+# Python 3 standard environment
+name: py3_std
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/willirath_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3
+  - anaconda-client
+  - aospy
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - eofs
+  - haversine
+  - hdf5
+  - ipython
+  - jupyter
+  - line_profiler
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - pytables
+  - seaborn
+  - seawater
+  - spyder
+  - texlive-core
+  - xarray

_envs_hlrn/pyferret.yml

+name: pyferret
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - matplotlib=1.5.3
+  - pyferret
+  - ferret_datasets

_envs_hlrn/tm_deploy.yml

+name: tm_deploy
+channels:
+  - file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/
+  - file://home/b/shkifmwr/TM/data/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3
+  - cartopy

_envs_vulcanus/ffmpeg.yml

+# Python 3 standard environment
+name: ffmpeg
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3
+  - anaconda-client
+  - ffmpeg

_envs_vulcanus/matlab_kernel.yml

+name: matlab_kernel
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3.4
+  - jupyter
+  - pip:
+     - matlab_kernel

_envs_vulcanus/monitoring.yml

+name: monitoring
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=2.7.12
+  - anaconda-client
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - ipython
+  - jupyter
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - seaborn
+  - pip:
+     - "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"

_envs_vulcanus/monitoring_gallery.yml

+name: monitoring_gallery
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+dependencies:
+  - python=2.7
+  - anaconda-client
+  - ConfigParser
+  - ffmpeg
+  - pip:
+    - sigal
+    - "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"

_envs_vulcanus/nco.yml

+name: nco
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - anaconda-client
+  - nco

_envs_vulcanus/py2_std.yml

+# Python 2.7 standard environment
+name: py2_std
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=2.7.12
+  - anaconda-client
+  - aospy
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - eofs
+  - haversine
+  - hdf5
+  - ipython
+  - jupyter
+  - line_profiler
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - pyngl
+  - pytables
+  - seaborn
+  - seawater
+  - spyder
+  - texlive-core
+  - xarray
+  - pip:
+      - pymatlab

_envs_vulcanus/py3_std.yml

+# Python 3 standard environment
+name: py3_std
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3
+  - anaconda-client
+  - aospy
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - eofs
+  - haversine
+  - hdf5
+  - ipython
+  - jupyter
+  - line_profiler
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - pytables
+  - seaborn
+  - seawater
+  - spyder
+  - texlive-core
+  - xarray

_envs_vulcanus/pyferret.yml

+name: pyferret
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - matplotlib=1.5.3
+  - pyferret
+  - ferret_datasets

_envs_vulcanus/tm_deploy.yml

+name: tm_deploy
+channels:
+  - file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+  - file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+dependencies:
+  - python=3
+  - cartopy

create_env_files.py

+import configparser
+import logging
+from pathlib import Path
+
+logging.basicConfig(
+        level=logging.DEBUG,
+        format='%(asctime)s : %(levelname)-8s %(message)s')
+logger = logging.getLogger(__name__)
+
+# Open config file detailing the paths / URLs to the channels for different
+# machines.
+config_file_name = 'locations.ini'
+config = configparser.ConfigParser()
+config.read(config_file_name)
+
+# For each section (indicating the location or kind of network environment),
+# read all env templates, expand the channel placeholders, and write them to
+# the respective yml-files.
+for section_name in config.sections():
+    for input_file_name in Path('_env_templates').glob('*.yml'):
+        env_dir = Path(section_name)
+        env_dir.mkdir(exist_ok=True)
+        output_file_name = env_dir / input_file_name.name
+        with input_file_name.open() as input_file, \
+                output_file_name.open(mode='w') as output_file:
+            logger.info('Creating {}'.format(output_file.name))
+            template = input_file.read()
+            output_file.write(template.format(**config[section_name]))

envs/ffmpeg.yml

+# Python 3 standard environment
+name: ffmpeg
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - python=3
+  - anaconda-client
+  - ffmpeg

envs/matlab_kernel.yml

+name: matlab_kernel
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - python=3.4
+  - jupyter
+  - pip:
+     - matlab_kernel

envs/monitoring.yml

+name: monitoring
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/willirath
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - python=2.7.12
+  - anaconda-client
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - ipython
+  - jupyter
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - seaborn
+  - pip:
+     - "git+https://github.com/willirath/xarray.git@6f5e30f3a2f8ca907f0fd91bcd571e7d98027896#egg=xarray_wrath_allow_ploting_datetime"

envs/monitoring_gallery.yml

+name: monitoring_gallery
+channels:
+  - https://conda.anaconda.org/conda-forge
+  - https://repo.continuum.io/pkgs/free
+dependencies:
+  - python=2.7
+  - anaconda-client
+  - ConfigParser
+  - ffmpeg
+  - pip:
+    - sigal
+    - "git+ssh://git@git.geomar.de/TM/MonitoringGallery.git@v0.2"

envs/nco.yml

+name: nco
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - anaconda-client
+  - nco

envs/py2_std.yml

+# Python 2.7 standard environment
+name: py2_std
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/willirath
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - python=2.7.12
+  - anaconda-client
+  - aospy
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - eofs
+  - haversine
+  - hdf5
+  - ipython
+  - jupyter
+  - line_profiler
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - pyngl
+  - pytables
+  - seaborn
+  - seawater
+  - spyder
+  - texlive-core
+  - xarray
+  - pip:
+      - pymatlab

envs/py3_std.yml

+# Python 3 standard environment
+name: py3_std
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/willirath
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - python=3
+  - anaconda-client
+  - aospy
+  - basemap
+  - cartopy
+  - cartopy_offlinedata
+  - cf_units
+  - cmocean
+  - dask
+  - eofs
+  - haversine
+  - hdf5
+  - ipython
+  - jupyter
+  - line_profiler
+  - matplotlib=1.5.3
+  - nco
+  - netCDF4
+  - numpy
+  - pandas
+  - pytables
+  - seaborn
+  - seawater
+  - spyder
+  - texlive-core
+  - xarray

envs/pyferret.yml

+name: pyferret
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - matplotlib=1.5.3
+  - pyferret
+  - ferret_datasets

envs/tm_deploy.yml

+name: tm_deploy
+channels:
+  - https://repo.continuum.io/pkgs/free
+  - https://conda.anaconda.org/conda-forge
+dependencies:
+  - python=3
+  - cartopy

locations.ini

+[envs]
+anaconda = https://repo.continuum.io/pkgs/free
+willirath = https://conda.anaconda.org/willirath
+conda-forge = https://conda.anaconda.org/conda-forge
+
+[_envs_vulcanus]
+anaconda = file://media/wrath/wrath_003/conda_mirror/anaconda_local_mirror/
+willirath = file://media/wrath/wrath_003/conda_mirror/willirath_local_mirror/
+conda-forge = file://media/wrath/wrath_003/conda_mirror/conda-forge_local_mirror/
+
+[_envs_hlrn]
+anaconda = file://home/b/shkifmwr/TM/data/conda_mirror/anaconda_local_mirror/