Skip to content

Hanging git clone with exploding ssh window sizes

On Nesh, cloning the large repo https://git.geomar.de/data/ARGO/ quite persistently failed around Receiving objects: 81%.

Problem

Following https://stackoverflow.com/a/13274606, I tried ping -i 0.7 git.geomar.de while cloning and it seemed to help.

SSH debug-level 2 (-vv) output showed that on the attempts that hung, the window size was constantly increased till SSH stopped reacting.

Reproduce

Steps to reproduce this:

# create destiation for logging
log_dir=$HOME/_log
mkdir -p $log_dir

# Clone ARGO repo, skip Git LFS, and modify ssh command to use
# higher debug level and time-stamped log file.
GIT_SSH_COMMAND="ssh -vv -E $log_dir/ssh_\`date -Ins\`.log" \
    GIT_LFS_SKIP_SMUDGE=1 \
    git clone git@git.geomar.de:data/ARGO.git

Stop / repeat until it hangs (S+ for the respective SSH command in ps haux | grep $USER).

Then, the $HOME/_log/ssh_*.log contain the level-2 ssh logs. Those Git sessions that hang will show an exploding window size.

Visualize

With the different _log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log, look at:

grep adjust _log/ssh_2017-^C-04T19\:47\:59\,285709796+0200.log | cut -d ' ' -f 5 | grep -v adj | gnuplot -p -e 'plot "/dev/stdin"'

sucessfull git clone:

ssh_window_sizes_successful_git_clone

hanging git clone:

ssh_window_size_hanging_git_clone