deploy_data issueshttps://git.geomar.de/data/tools/deploy_data/-/issues2020-06-11T11:15:39Zhttps://git.geomar.de/data/tools/deploy_data/-/issues/9Add plain-text overview of version details2020-06-11T11:15:39ZWilli RathAdd plain-text overview of version detailsOne of our usage patterns is: "Download whole directories from a centrally deployed copy." This breaks the git-references, because we use worktree. To help these users to at least be able to know the exact version of the data (even if t...One of our usage patterns is: "Download whole directories from a centrally deployed copy." This breaks the git-references, because we use worktree. To help these users to at least be able to know the exact version of the data (even if their copy is not a valid Git repo anymore), we should add a file with the output of something like:
```shell
$ git describe --tags
$ git remote -v
$ # what else?
```https://git.geomar.de/data/tools/deploy_data/-/issues/8"Error updating the git index:" when deploying with moved LFS files2017-11-30T11:09:22ZWilli Rath"Error updating the git index:" when deploying with moved LFS filesWhen I tried to deploy data/ERAIN_SFC12_MM@9e7c31aba58befc9d02d8a2945e95ad5977820f8 (which moves files around) into the existing `.../v1.x.x/`, the following happened:
```
Error updating the git index:
error: data/grb/ERAIN_SFC12_MM_10S...When I tried to deploy data/ERAIN_SFC12_MM@9e7c31aba58befc9d02d8a2945e95ad5977820f8 (which moves files around) into the existing `.../v1.x.x/`, the following happened:
```
Error updating the git index:
error: data/grb/ERAIN_SFC12_MM_10SI_207_1-456.grb: cannot add to the index - missing --add option?
fatal: Unable to process path data/grb/ERAIN_SFC12_MM_10SI_207_1-456.grb
```
Removing the worktree, leaving `.bare/ERAIN_SFC12_MM/` untouched, and re-deploying worked fine.https://git.geomar.de/data/tools/deploy_data/-/issues/7API returns a truncated list of projects2017-09-07T18:07:40ZWilli RathAPI returns a truncated list of projectsCurrently, <https://git.geomar.de/data/tools/deploy_data/blob/master/generate_calls_for_projects_and_tags.sh#L20> only returns the first approx. 20 lines of the listing shown in <https://git.geomar.de/data>.
How to make sure to get all ...Currently, <https://git.geomar.de/data/tools/deploy_data/blob/master/generate_calls_for_projects_and_tags.sh#L20> only returns the first approx. 20 lines of the listing shown in <https://git.geomar.de/data>.
How to make sure to get all projects?
(@mscheinert: Did you see truncated API responses before?)Willi RathWilli Rathhttps://git.geomar.de/data/tools/deploy_data/-/issues/6Hanging git clone with exploding ssh window sizes2017-11-28T11:21:05ZWilli RathHanging git clone with exploding ssh window sizesOn Nesh, cloning the large repo <https://git.geomar.de/data/ARGO/> quite persistently failed around `Receiving objects: 81%`.
## Problem
Following <https://stackoverflow.com/a/13274606>, I tried `ping -i 0.7 git.geomar.de` while clonin...On Nesh, cloning the large repo <https://git.geomar.de/data/ARGO/> quite persistently failed around `Receiving objects: 81%`.
## Problem
Following <https://stackoverflow.com/a/13274606>, I tried `ping -i 0.7 git.geomar.de` while cloning and it seemed to help.
SSH debug-level 2 (`-vv`) output showed that on the attempts that hung, the window size was constantly increased till SSH stopped reacting.
## Reproduce
Steps to reproduce this:
```bash
# create destiation for logging
log_dir=$HOME/_log
mkdir -p $log_dir
# Clone ARGO repo, skip Git LFS, and modify ssh command to use
# higher debug level and time-stamped log file.
GIT_SSH_COMMAND="ssh -vv -E $log_dir/ssh_\`date -Ins\`.log" \
GIT_LFS_SKIP_SMUDGE=1 \
git clone git@git.geomar.de:data/ARGO.git
```
Stop / repeat until it hangs (`S+` for the respective SSH command in `ps haux | grep $USER`).
Then, the `$HOME/_log/ssh_*.log` contain the level-2 ssh logs. Those Git sessions that hang will show an exploding window size.
## Visualize
With the different `_log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log`, look at:
```bash
grep adjust _log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log | cut -d ' ' -f 5 | grep -v adj | gnuplot -p -e 'plot "/dev/stdin"'
```
### sucessfull `git clone`:
![ssh_window_sizes_successful_git_clone](/uploads/19be1ecca26c5c21f4b5bb74c9f7110c/ssh_window_sizes_successful_git_clone.png)
### hanging `git clone`:
![ssh_window_size_hanging_git_clone](/uploads/9ea51442ae1f3e37b3bfd95ded7660af/ssh_window_size_hanging_git_clone.png)Willi RathWilli Rathhttps://git.geomar.de/data/tools/deploy_data/-/issues/5Facilitate moving whole checkouts by using relative links2017-08-21T10:48:14ZWilli RathFacilitate moving whole checkouts by using relative linksSwitching https://git.geomar.de/data/tools/deploy_data/blob/master/deploy_data.sh#L42 to a relative link, may make us able to move the whole structure.Switching https://git.geomar.de/data/tools/deploy_data/blob/master/deploy_data.sh#L42 to a relative link, may make us able to move the whole structure.Willi RathWilli Rathhttps://git.geomar.de/data/tools/deploy_data/-/issues/4Automatic distribution2017-08-18T17:17:17ZWilli RathAutomatic distributionAdd possibility to automatically distribute the data sets.
Essential info:
- `$base_path` to data repos
- pairs of `$name` and `$reference` to be checked out
Then, cycle through a bunch of:
```bash
deploy_data.sh -n ${name} -p ${base_...Add possibility to automatically distribute the data sets.
Essential info:
- `$base_path` to data repos
- pairs of `$name` and `$reference` to be checked out
Then, cycle through a bunch of:
```bash
deploy_data.sh -n ${name} -p ${base_path} -u git@git.geomar.de:data/${name}.git -r ${reference}
```https://git.geomar.de/data/tools/deploy_data/-/issues/3How to clearly define dependencies?2017-08-18T16:09:27ZWilli RathHow to clearly define dependencies?As, e.g., Git LFS is quickly evolving, we need to clearly define the dependencies.
I did all my tests with conda-forge/git and conda-forge/git-lfs.
Using a conda env for deployment and all / many of the tasks in the repositories th...As, e.g., Git LFS is quickly evolving, we need to clearly define the dependencies.
I did all my tests with conda-forge/git and conda-forge/git-lfs.
Using a conda env for deployment and all / many of the tasks in the repositories themselves would also benefit from the usually up-to date conda-forge packages for netCDF, CDO, NCO etc.
@all: What do you think?Rolling updateshttps://git.geomar.de/data/tools/deploy_data/-/issues/2Bare repos don't fetch new tags2017-06-22T07:03:51ZWilli RathBare repos don't fetch new tagsWe might want to go for `git clone --mirror` and then `git remote update`.
It seems that (with Git `2.12.2` and Git LFS `2.1.1`) mirror does not download LFS files.We might want to go for `git clone --mirror` and then `git remote update`.
It seems that (with Git `2.12.2` and Git LFS `2.1.1`) mirror does not download LFS files.https://git.geomar.de/data/tools/deploy_data/-/issues/1Make sure to only download LFS files if necessary2017-06-22T07:03:32ZWilli RathMake sure to only download LFS files if necessaryIt might be sufficient to prefix the git clone command in [deploy_data.sh](deploy_data.sh) with `GIT_LFS_SKIP_SMUDGE=1`. But let's test if this really checks out all files first.It might be sufficient to prefix the git clone command in [deploy_data.sh](deploy_data.sh) with `GIT_LFS_SKIP_SMUDGE=1`. But let's test if this really checks out all files first.Willi RathWilli Rath