Day 14: Several Servers with ssh and rsync
Goal: Run commands and scripts on another server while quoting correctly, synchronize files with rsync, open a tunnel and distribute the toolbox to several servers with
notes-deploy. · Duration: approx. 90 min · Prerequisite: Day 13
Briefing
ssh host commandruns a command on another machine. The exit code comes back, and stdin and stdout flow through the connection.- A command line is read twice: first by your shell, then by the shell on the target machine. Single quotes protect against yours, double quotes only partially.
- rsync synchronizes directories and transfers only
what has changed.
--deleteremoves whatever no longer exists in the original.-nshows beforehand what would happen. - An entry in
~/.ssh/configgives a server a short name with all its settings – scripts then only writessh ziel.
VM sh ──ssh/rsync──► VM ziel: ~/sh-werkzeug/{bin,lib}
notes-deploy ziel ─► rsync ─► ssh ziel 'bash -n …'
ssh -R 18080:127.0.0.1:8080 ziel ─► notes-check on ziel checks notes on sh
Drill 1: Keys and target server
On the laptop, the second VM:
multipass launch 26.04 --name ziel --cpus 1 --memory 1G --disk 10G
multipass info ziel | grep IPv4
On the VM sh, a dedicated key for the lab:
source ~/sh-lab/setup/env.sh
mkdir -p ~/sh-lab/tag14 && cd ~/sh-lab/tag14
ssh-keygen -q -t ed25519 -N '' -C sh-lab -f ~/.ssh/sh-lab_ed25519
cat ~/.ssh/sh-lab_ed25519.pub | cut -c1-30Expected:
ssh-ed25519 AAAAC3NzaC1lZDI1
You add the public key on ziel, from the laptop:
multipass exec ziel -- bash -c "echo '<contents of sh-lab_ed25519.pub>' \
>> ~/.ssh/authorized_keys"
Back on sh: an entry in the SSH configuration (fill
in the IP of ziel):
ZIEL_IP=192.168.64.9
cat >> ~/.ssh/config <<EOF
Host ziel
HostName $ZIEL_IP
User ubuntu
IdentityFile ~/.ssh/sh-lab_ed25519
StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config
ssh ziel 'hostname; bash --version | head -1' 2>&1 | grep -v '^Warning: Permanently'Expected:
ziel
GNU bash, version 5.3.9(1)-release (x86_64-pc-linux-gnu)
accept-new accepts the key of a new
server on first contact but rejects a changed one – a sensible middle
ground for scripts. The dedicated key sh-lab_ed25519 leaves
an existing id_ed25519 untouched.
Save your progress? Sign up for free
Drill 2: Remote commands
ssh ziel "echo Zuhause ist $HOSTNAME"
ssh ziel 'echo Zuhause ist $HOSTNAME'
ssh ziel 'exit 3'; echo "Exit: $?"
ssh ziel 'ls /gibtsnicht' 2>&1; echo "Exit: $?"
datei="Notiz mit Leerzeichen.txt"
ssh ziel "touch /tmp/$datei; ls -1 /tmp/Notiz* mit Leerzeichen.txt"
ssh ziel "rm -f /tmp/Notiz mit Leerzeichen.txt; touch /tmp/$(printf '%q' "$datei")"
ssh ziel 'ls -1 /tmp/Notiz*'Expected:
Zuhause ist sh
Zuhause ist ziel
Exit: 3
ls: cannot access '/gibtsnicht': No such file or directory
Exit: 2
/tmp/Notiz
Leerzeichen.txt
mit
/tmp/Notiz mit Leerzeichen.txt
- In double quotes, your shell replaces
$HOSTNAMEbeforesshstarts. In single quotes,$HOSTNAMEarrives unchanged and is replaced over there. touch /tmp/$dateiarrives over there astouch /tmp/Notiz mit Leerzeichen.txt– and the shell there turns it into three files:/tmp/Notizas well asmitandLeerzeichen.txtin the home directory. Your shell has already used up the quotes.printf '%q'quotes for the second shell. For more than one value, a script via stdin is cleaner:
cat > remote-info <<'EOF'
set -euo pipefail
echo "Host: $(hostname)"
echo "Platte: $(df -h / | awk 'NR == 2 {print $5}') belegt"
echo "Argument: [$1]"
EOF
ssh ziel bash -s -- "'zwei Wörter'" < remote-info | sed -E 's/[0-9]+%/…%/'Expected:
Host: ziel
Platte: …% belegt
Argument: [zwei Wörter]
bash -s reads the script from stdin, --
separates the arguments. The argument needs two layers of quotes: the
outer ones for your shell, the inner ones for the one over there.
Save your progress? Sign up for free
Drill 3: rsync
opts=(-az --delete --exclude .git --exclude tests)
rsync "${opts[@]}" ~/sh-lab/werkzeug/ ziel:sh-werkzeug/
ssh ziel 'ls sh-werkzeug; ls sh-werkzeug/bin | wc -l'
rsync "${opts[@]}" -n -i ~/sh-lab/werkzeug/ ziel:sh-werkzeug/ | wc -l
touch ~/sh-lab/werkzeug/bin/hallo
rsync "${opts[@]}" -n -i ~/sh-lab/werkzeug/ ziel:sh-werkzeug/Expected:
bin
ci.sh
lib
14
0
<f..t...... bin/hallo
- The trailing slash on
werkzeug/means “transfer the contents”. Without it, a folderwerkzeugends up insidesh-werkzeug. -apreserves permissions and times,-zcompresses,-nis the dry run,-ishows per file what changes.<f..t......means: the file would be sent, the time is different. The options are in an array – as on Day 5.- The second dry run is empty – rsync only transfers differences.
Save your progress? Sign up for free
Drill 4: notes-deploy
cat > ~/sh-lab/werkzeug/bin/notes-deploy <<'EOF'
#!/usr/bin/env bash
# notes-deploy – distributes the toolbox: notes-deploy [-n] host…
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
need rsync ssh
probe=()
if [[ ${1:-} == -n ]]; then
probe=(-n)
shift
fi
(($# > 0)) || die "Aufruf: $PROG [-n] host…" 2
quelle="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/.."
verteile() {
local host=$1
rsync -az "${probe[@]}" --delete --exclude .git --exclude tests \
"$quelle/" "$host:sh-werkzeug/"
ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" \
'for f in sh-werkzeug/bin/* sh-werkzeug/lib/*.sh; do bash -n "$f" || exit 1; done'
}
declare -A pid
for host in "$@"; do
verteile "$host" >"/tmp/notes-deploy.$host.log" 2>&1 &
pid[$host]=$!
done
fehler=0
for host in "$@"; do
if wait "${pid[$host]}"; then
info "$host: ok"
else
warn "$host: gescheitert – siehe /tmp/notes-deploy.$host.log"
fehler=1
fi
done
exit "$fehler"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-deploy
~/sh-lab/werkzeug/ci.sh | tail -1
notes-deploy ziel 2>&1 | sed -E 's/^[0-9:]{8}/…/'
notes-deploy ziel gibtsnicht.invalid 2>&1 | sed -E 's/^[0-9:]{8}/…/'
echo "Exit: ${PIPESTATUS[0]}"Expected:
ci: alles grün
… notes-deploy: INFO ziel: ok
… notes-deploy: INFO ziel: ok
… notes-deploy: WARN gibtsnicht.invalid: gescheitert – siehe /tmp/notes-deploy.gibtsnic…
Exit: 1
- All servers at the same time, each with its own log – and an honest
overall exit code.
$?after the pipe would be that ofsed;${PIPESTATUS[0]}is that ofnotes-deploy. verteileruns with&in a subshell in whichset -eapplies: ifrsyncfails,sshnever even gets its turn.bash -nchecks the syntax without executing – a quick test of whether the files have arrived completely.BatchMode=yespreventssshfrom asking for a password and leaving a script hanging.
Save your progress? Sign up for free
Drill 5: A tunnel
On the VM sh, notes only listens on 127.0.0.1 – it is
not reachable from ziel. A reverse tunnel brings the
port along:
ssh ziel 'curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 \
http://127.0.0.1:18080/healthz'
ssh -R 18080:127.0.0.1:8080 ziel \
'NOTES_URL=http://127.0.0.1:18080 sh-werkzeug/bin/notes-check -v'Expected:
000
200 /healthz
200 /notes
200 /version
3 ok, 0 Fehler
-R 18080:127.0.0.1:8080 opens port 18080 on ziel and
forwards it through the SSH connection to 127.0.0.1:8080 on
sh. The tunnel exists as long as the command runs. Conversely,
-L brings a port from over there to you.
Save your progress? Sign up for free
Checkpoint
ssh ziel 'ls sh-werkzeug/bin | grep -c notes-'Expected: all notes tools on ziel.
13
ssh -o BatchMode=yes ziel true && echo "ssh ohne Passwort"Expected:
ssh ohne Passwort
rsync -azn -i --delete --exclude .git --exclude tests \
~/sh-lab/werkzeug/ ziel:sh-werkzeug/ | wc -lExpected: nothing left to transfer.
0
Challenge
Write notes-remote host command…: it runs a command from
the toolbox on the host and passes notes from sh through a tunnel.
notes-remote ziel notes-check -q; echo $? should return
0.
Note: ssh -R, set
NOTES_URL over there, and prepare the arguments with
printf '%q ' for the second shell.
Solution
cat > ~/sh-lab/werkzeug/bin/notes-remote <<'EOF'
#!/usr/bin/env bash
# notes-remote – run a tool on a host: notes-remote host tool [args…]
set -euo pipefail
source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../lib/lib.sh"
(($# >= 2)) || die "Aufruf: $PROG host werkzeug [args…]" 2
host=$1
shift
befehl=$(printf '%q ' "$@")
port=$((18000 + RANDOM % 1000))
ssh -o BatchMode=yes -o ExitOnForwardFailure=yes \
-R "$port:127.0.0.1:${NOTES_URL##*:}" "$host" \
"NOTES_URL=http://127.0.0.1:$port PATH=\$HOME/sh-werkzeug/bin:\$PATH $befehl"
EOF
chmod +x ~/sh-lab/werkzeug/bin/notes-remote
~/sh-lab/werkzeug/ci.sh | tail -1
notes-deploy ziel 2>/dev/null
notes-remote ziel notes-check -q; echo "Exit: $?"
notes-remote ziel notes-check -v /healthz '/ad*min'; echo "Exit: $?"Expected:
ci: alles grün
Exit: 0
200 /healthz
404 /ad*min
1 ok, 1 Fehler
Exit: 1
${NOTES_URL##*:}takes the port from the URL.- A random port avoids collisions when two runs tunnel at the same
time;
ExitOnForwardFailureaborts instead of carrying on without a tunnel. - Thanks to
printf '%q',/ad*minarrives over there unchanged. Without it, the shell there would interpret the asterisk as a file name pattern.
Save your progress? Sign up for free
Enterprise Note
- Keys with a purpose: A dedicated key per
automation, restricted in
authorized_keyswithcommand=,from=andno-port-forwarding– not an admin’s personal key. - Many servers: Beyond a handful of servers, tools like Ansible take over the distribution, with inventory, idempotence and reports. The patterns here – parallel, one log per host, honest exit code – stay the same.
- ControlMaster:
ControlMaster autoandControlPersist 60sin~/.ssh/configkeep a connection open. That makes many shortsshcalls significantly faster.