Hanging git clone with exploding ssh window sizes (#6) · Issues · data / tools / deploy_data

Hanging git clone with exploding ssh window sizes

On Nesh, cloning the large repo https://git.geomar.de/data/ARGO/ quite persistently failed around Receiving objects: 81%.

Problem

Following https://stackoverflow.com/a/13274606, I tried ping -i 0.7 git.geomar.de while cloning and it seemed to help.

SSH debug-level 2 (-vv) output showed that on the attempts that hung, the window size was constantly increased till SSH stopped reacting.

Reproduce

Steps to reproduce this:

# create destiation for logging
log_dir=$HOME/_log
mkdir -p $log_dir

# Clone ARGO repo, skip Git LFS, and modify ssh command to use
# higher debug level and time-stamped log file.
GIT_SSH_COMMAND="ssh -vv -E $log_dir/ssh_\`date -Ins\`.log" \
    GIT_LFS_SKIP_SMUDGE=1 \
    git clone git@git.geomar.de:data/ARGO.git

Stop / repeat until it hangs (S+ for the respective SSH command in ps haux | grep $USER).

Then, the $HOME/_log/ssh_*.log contain the level-2 ssh logs. Those Git sessions that hang will show an exploding window size.

Visualize

With the different _log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log, look at:

grep adjust _log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log | cut -d ' ' -f 5 | grep -v adj | gnuplot -p -e 'plot "/dev/stdin"'

sucessfull `git clone`:

hanging `git clone`:

On Nesh, cloning the large repo <https://git.geomar.de/data/ARGO/> quite persistently failed around `Receiving objects: 81%`.

## Problem

Following <https://stackoverflow.com/a/13274606>, I tried `ping -i 0.7 git.geomar.de` while cloning and it seemed to help.

SSH debug-level 2 (`-vv`) output showed that on the attempts that hung, the window size was constantly increased till SSH stopped reacting.

## Reproduce

Steps to reproduce this:

```bash
# create destiation for logging
log_dir=$HOME/_log
mkdir -p $log_dir

# Clone ARGO repo, skip Git LFS, and modify ssh command to use
# higher debug level and time-stamped log file.
GIT_SSH_COMMAND="ssh -vv -E $log_dir/ssh_\`date -Ins\`.log" \
    GIT_LFS_SKIP_SMUDGE=1 \
    git clone git@git.geomar.de:data/ARGO.git
```

Stop / repeat until it hangs (`S+` for the respective SSH command in `ps haux | grep $USER`).

Then, the `$HOME/_log/ssh_*.log` contain the level-2 ssh logs.  Those Git sessions that hang will show an exploding window size.

## Visualize

With the different `_log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log`, look at:

```bash
grep adjust _log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log | cut -d ' ' -f 5 | grep -v adj | gnuplot -p -e 'plot "/dev/stdin"'
```

### sucessfull `git clone`:

![ssh_window_sizes_successful_git_clone](/uploads/19be1ecca26c5c21f4b5bb74c9f7110c/ssh_window_sizes_successful_git_clone.png)

### hanging `git clone`:

![ssh_window_size_hanging_git_clone](/uploads/9ea51442ae1f3e37b3bfd95ded7660af/ssh_window_size_hanging_git_clone.png)