Skip to content
Hands-on IT IT Drills

Day 14: Several Servers with ssh and rsync

Goal: Run commands and scripts on another server while quoting correctly, synchronize files with rsync, open a tunnel and distribute the toolbox to several servers with notes-deploy. · Duration: approx. 90 min · Prerequisite: Day 13

Briefing

  • ssh host command runs a command on another machine. The exit code comes back, and stdin and stdout flow through the connection.
  • A command line is read twice: first by your shell, then by the shell on the target machine. Single quotes protect against yours, double quotes only partially.
  • rsync synchronizes directories and transfers only what has changed. --delete removes whatever no longer exists in the original. -n shows beforehand what would happen.
  • An entry in ~/.ssh/config gives a server a short name with all its settings – scripts then only write ssh ziel.
 VM sh ──ssh/rsync──► VM ziel: ~/sh-werkzeug/{bin,lib}
   notes-deploy ziel ─► rsync ─► ssh ziel 'bash -n …'
   ssh -R 18080:127.0.0.1:8080 ziel ─► notes-check on ziel checks notes on sh

Drill 1: Keys and target server

On the laptop, the second VM:

multipass launch 26.04 --name ziel --cpus 1 --memory 1G --disk 10G
multipass info ziel | grep IPv4

On the VM sh, a dedicated key for the lab:

source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag14 && cd ~/sh-lab/tag14
ssh-keygen -q -t ed25519 -N '' -C sh-lab -f ~/.ssh/sh-lab_ed25519
cat ~/.ssh/sh-lab_ed25519.pub | cut -c1-30

Expected:

ssh-ed25519 AAAAC3NzaC1lZDI1

You add the public key on ziel, from the laptop:

multipass exec ziel -- bash -c "echo '<contents of sh-lab_ed25519.pub>' \
  >> ~/.ssh/authorized_keys"

Back on sh: an entry in the SSH configuration (fill in the IP of ziel):

ZIEL_IP=192.168.64.9
cat >> ~/.ssh/config <<EOF

Host ziel
  HostName $ZIEL_IP
  User ubuntu
  IdentityFile ~/.ssh/sh-lab_ed25519
  StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config
ssh ziel 'hostname; bash --version | head -1' 2>&1 | grep -v '^Warning: Permanently'

Expected:

ziel
GNU bash, version 5.3.9(1)-release (x86_64-pc-linux-gnu)

accept-new accepts the key of a new server on first contact but rejects a changed one – a sensible middle ground for scripts. The dedicated key sh-lab_ed25519 leaves an existing id_ed25519 untouched.

Save your progress? Sign up for free

Drill 2: Remote commands

ssh ziel "echo Zuhause ist $HOSTNAME"
ssh ziel 'echo Zuhause ist $HOSTNAME'
ssh ziel 'exit 3'; echo "Exit: $?"
ssh ziel 'ls /gibtsnicht' 2>&1; echo "Exit: $?"
datei="Notiz mit Leerzeichen.txt"
ssh ziel "touch /tmp/$datei; ls -1 /tmp/Notiz* mit Leerzeichen.txt"
ssh ziel "rm -f /tmp/Notiz mit Leerzeichen.txt; touch /tmp/$(printf '%q' "$datei")"
ssh ziel 'ls -1 /tmp/Notiz*'

Expected:

Zuhause ist sh
Zuhause ist ziel
Exit: 3
ls: cannot access '/gibtsnicht': No such file or directory
Exit: 2
/tmp/Notiz
Leerzeichen.txt
mit
/tmp/Notiz mit Leerzeichen.txt
  • In double quotes, your shell replaces $HOSTNAME before ssh starts. In single quotes, $HOSTNAME arrives unchanged and is replaced over there.
  • touch /tmp/$datei arrives over there as touch /tmp/Notiz mit Leerzeichen.txt – and the shell there turns it into three files: /tmp/Notiz as well as mit and Leerzeichen.txt in the home directory. Your shell has already used up the quotes.
  • printf '%q' quotes for the second shell. For more than one value, a script via stdin is cleaner:
cat > remote-info <<'EOF'
set -euo pipefail
echo "Host:   $(hostname)"
echo "Platte: $(df -h / | awk 'NR == 2 {print $5}') belegt"
echo "Argument: [$1]"
EOF
ssh ziel bash -s -- "'zwei Wörter'" < remote-info | sed -E 's/[0-9]+%/…%/'

Expected:

Host:   ziel
Platte: …% belegt
Argument: [zwei Wörter]

bash -s reads the script from stdin, -- separates the arguments. The argument needs two layers of quotes: the outer ones for your shell, the inner ones for the one over there.

Save your progress? Sign up for free

Drill 3: rsync

opts=(-az --delete --exclude .git --exclude tests)
rsync "${opts[@]}" ~/sh-lab/werkzeug/ ziel:sh-werkzeug/
ssh ziel 'ls sh-werkzeug; ls sh-werkzeug/bin | wc -l'
rsync "${opts[@]}" -n -i ~/sh-lab/werkzeug/ ziel:sh-werkzeug/ | wc -l
touch ~/sh-lab/werkzeug/bin/hallo
rsync "${opts[@]}" -n -i ~/sh-lab/werkzeug/ ziel:sh-werkzeug/

Expected:

bin
ci.sh
lib
14
0
<f..t...... bin/hallo
  • The trailing slash on werkzeug/ means “transfer the contents”. Without it, a folder werkzeug ends up inside sh-werkzeug.
  • -a preserves permissions and times, -z compresses, -n is the dry run, -i shows per file what changes. <f..t...... means: the file would be sent, the time is different. The options are in an array – as on Day 5.
  • The second dry run is empty – rsync only transfers differences.

Save your progress? Sign up for free

Drill 4: notes-deploy

cat > ~/sh-lab/werkzeug/bin/notes-deploy <<'EOF'
#!/usr/bin/env bash
# notes-deploy – distributes the toolbox: notes-deploy [-n] host…
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need rsync ssh
probe=()
if [[ ${1:-} == -n ]]; then
  probe=(-n)
  shift
fi
(($# > 0)) || die "Aufruf: $PROG [-n] host…" 2
quelle="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/.."

verteile() {
  local host=$1
  rsync -az "${probe[@]}" --delete --exclude .git --exclude tests \
    "$quelle/" "$host:sh-werkzeug/"
  ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" \
    'for f in sh-werkzeug/bin/* sh-werkzeug/lib/*.sh; do bash -n "$f" || exit 1; done'
}

declare -A pid
for host in "$@"; do
  verteile "$host" >"/tmp/notes-deploy.$host.log" 2>&1 &
  pid[$host]=$!
done
fehler=0
for host in "$@"; do
  if wait "${pid[$host]}"; then
    info "$host: ok"
  else
    warn "$host: gescheitert – siehe /tmp/notes-deploy.$host.log"
    fehler=1
  fi
done
exit "$fehler"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-deploy
~/sh-lab/werkzeug/ci.sh | tail -1
notes-deploy ziel 2>&1 | sed -E 's/^[0-9:]{8}/…/'
notes-deploy ziel gibtsnicht.invalid 2>&1 | sed -E 's/^[0-9:]{8}/…/'
echo "Exit: ${PIPESTATUS[0]}"

Expected:

ci: alles grün
… notes-deploy: INFO ziel: ok
… notes-deploy: INFO ziel: ok
… notes-deploy: WARN gibtsnicht.invalid: gescheitert – siehe /tmp/notes-deploy.gibtsnic…
Exit: 1
  • All servers at the same time, each with its own log – and an honest overall exit code. $? after the pipe would be that of sed; ${PIPESTATUS[0]} is that of notes-deploy.
  • verteile runs with & in a subshell in which set -e applies: if rsync fails, ssh never even gets its turn.
  • bash -n checks the syntax without executing – a quick test of whether the files have arrived completely.
  • BatchMode=yes prevents ssh from asking for a password and leaving a script hanging.

Save your progress? Sign up for free

Drill 5: A tunnel

On the VM sh, notes only listens on 127.0.0.1 – it is not reachable from ziel. A reverse tunnel brings the port along:

ssh ziel 'curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 \
  http://127.0.0.1:18080/healthz'
ssh -R 18080:127.0.0.1:8080 ziel \
  'NOTES_URL=http://127.0.0.1:18080 sh-werkzeug/bin/notes-check -v'

Expected:

000
200 /healthz
200 /notes
200 /version
3 ok, 0 Fehler

-R 18080:127.0.0.1:8080 opens port 18080 on ziel and forwards it through the SSH connection to 127.0.0.1:8080 on sh. The tunnel exists as long as the command runs. Conversely, -L brings a port from over there to you.

Save your progress? Sign up for free

Checkpoint

ssh ziel 'ls sh-werkzeug/bin | grep -c notes-'

Expected: all notes tools on ziel.

13
ssh -o BatchMode=yes ziel true && echo "ssh ohne Passwort"

Expected:

ssh ohne Passwort
rsync -azn -i --delete --exclude .git --exclude tests \
  ~/sh-lab/werkzeug/ ziel:sh-werkzeug/ | wc -l

Expected: nothing left to transfer.

0

Challenge

Write notes-remote host command…: it runs a command from the toolbox on the host and passes notes from sh through a tunnel. notes-remote ziel notes-check -q; echo $? should return 0.

Note: ssh -R, set NOTES_URL over there, and prepare the arguments with printf '%q ' for the second shell.

Solution

cat > ~/sh-lab/werkzeug/bin/notes-remote <<'EOF'
#!/usr/bin/env bash
# notes-remote – run a tool on a host: notes-remote host tool [args…]
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
(($# >= 2)) || die "Aufruf: $PROG host werkzeug [args…]" 2
host=$1
shift
befehl=$(printf '%q ' "$@")
port=$((18000 + RANDOM % 1000))
ssh -o BatchMode=yes -o ExitOnForwardFailure=yes \
  -R "$port:127.0.0.1:${NOTES_URL##*:}" "$host" \
  "NOTES_URL=http://127.0.0.1:$port PATH=\$HOME/sh-werkzeug/bin:\$PATH $befehl"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-remote
~/sh-lab/werkzeug/ci.sh | tail -1
notes-deploy ziel 2>/dev/null
notes-remote ziel notes-check -q; echo "Exit: $?"
notes-remote ziel notes-check -v /healthz '/ad*min'; echo "Exit: $?"

Expected:

ci: alles grün
Exit: 0
200 /healthz
404 /ad*min
1 ok, 1 Fehler
Exit: 1
  • ${NOTES_URL##*:} takes the port from the URL.
  • A random port avoids collisions when two runs tunnel at the same time; ExitOnForwardFailure aborts instead of carrying on without a tunnel.
  • Thanks to printf '%q', /ad*min arrives over there unchanged. Without it, the shell there would interpret the asterisk as a file name pattern.

Save your progress? Sign up for free

Enterprise Note

  • Keys with a purpose: A dedicated key per automation, restricted in authorized_keys with command=, from= and no-port-forwarding – not an admin’s personal key.
  • Many servers: Beyond a handful of servers, tools like Ansible take over the distribution, with inventory, idempotence and reports. The patterns here – parallel, one log per host, honest exit code – stay the same.
  • ControlMaster: ControlMaster auto and ControlPersist 60s in ~/.ssh/config keep a connection open. That makes many short ssh calls significantly faster.