Skip to content
Hands-on IT IT Drills

Day 12: SSH and macOS commands

Goal: Control SSH connections through a clean configuration and use macOS specifics from the terminal. · Duration: approx. 90 min · Prerequisite: Day 11

Briefing

  • Everything runs with ssh -F ~/term-lab/ssh/config. The lab neither reads nor writes your real ~/.ssh.
  • A dedicated sshd on 127.0.0.1:2222 plays bastion and target server. The macOS “Remote Login” sharing setting stays off.
  • ssh -G <host> shows the effective configuration without connecting. That’s your debugger.
  • For macOS: reading is free, writing (defaults write, launchctl) only deliberately and with a backup (Day 13).
Mac ──ssh──► lab-bastion (127.0.0.1:2222) ──ProxyJump──► lab-app (127.0.0.1:2222)
 │  ControlMaster: one TCP connection, socket ~/term-lab/ssh/cm-<hash>
 └─ ssh-agent (lab socket) with lab key id_lab

Drill 1: Lab key and sample configuration

mkdir -p ~/term-lab/ssh && chmod 700 ~/term-lab/ssh && cd ~/term-lab/ssh
ssh-keygen -q -t ed25519 -f id_lab -N '' -C 'lab-key (Lab-Wert)'
cp $BUCH/ssh/config config
ssh -G -F config app-web01 | grep -E '^(hostname|user|proxyjump|controlpersist) '

Expected:

user deploy
hostname app-web01.internal.example.com
controlpersist 600
proxyjump bastion

-N '' means: no passphrase. That’s only okay for the lab key. Real keys get a passphrase that the keychain remembers (Drill 5).

Why %h.internal.example.com? Host app-* matches any name with this prefix, %h inserts it. That way one block serves a hundred servers.

Warning: With SSH, the first value found wins. Specific Host blocks belong at the top, Host * at the end.

Save your progress? Sign up for free

Drill 2: A practice sshd on localhost

An sshd runs without root too, if it uses a high port and its own keys:

ssh-keygen -q -t ed25519 -f hostkey -N '' -C lab-host
cp id_lab.pub authorized_keys && chmod 600 authorized_keys
cat > sshd_config <<EOF
Port 2222
ListenAddress 127.0.0.1
HostKey $HOME/term-lab/ssh/hostkey
AuthorizedKeysFile $HOME/term-lab/ssh/authorized_keys
PasswordAuthentication no
KbdInteractiveAuthentication no
UsePAM no
StrictModes no
PidFile $HOME/term-lab/ssh/sshd.pid
EOF
/usr/sbin/sshd -t -f ~/term-lab/ssh/sshd_config && echo "sshd_config ok"
/usr/sbin/sshd -f ~/term-lab/ssh/sshd_config -E ~/term-lab/ssh/sshd.log
tail -1 sshd.log

Expected:

sshd_config ok
Server listening on 127.0.0.1 port 2222.

The heredoc without quotes (<<EOF) substitutes $HOME while writing, because sshd_config doesn’t know the tilde. -f also needs an absolute path: sshd changes to / and re-executes itself for each connection – with a relative path, every login fails. Now append the lab hosts to the configuration:

cat >> config <<'EOF'

Host lab-bastion
  HostName 127.0.0.1
  Port 2222
  IdentityFile ~/term-lab/ssh/id_lab

Host lab-app
  HostName 127.0.0.1
  Port 2222
  IdentityFile ~/term-lab/ssh/id_lab
  ProxyJump lab-bastion
EOF
alias lssh='ssh -F ~/term-lab/ssh/config'
lssh -o StrictHostKeyChecking=accept-new lab-bastion 'echo hallo von $(hostname -s)'

Expected:

Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.
hallo von <dein-mac>

The blocks come after Host *. That’s okay here because Host * sets neither HostName nor Port. ControlMaster and known_hosts from Host * still apply.

Save your progress? Sign up for free

Drill 3: ControlMaster

The first connection opens a master socket, all further ones share it:

ls ~/term-lab/ssh/cm-*
lssh -O check lab-bastion
time (lssh lab-bastion true)
time (lssh -o ControlPath=none lab-bastion true)

Expected (shortened):

/Users/<du>/term-lab/ssh/cm-3138ceb4c1d01333e4071a34291d27c4ce972f7a
Master running (pid=6855)
( lssh lab-bastion true; )  0,00s user 0,00s system 30% cpu 0,007 total
( lssh -o ControlPath=none lab-bastion true; )  0,01s user ... 0,051 total

Locally it’s 7 instead of 51 milliseconds. With a real server behind a bastion, a second of handshake turns into a few milliseconds. You notice that immediately with git fetch, scp, rsync and Ansible.

%C in ControlPath is a hash of host, port and user. That keeps the path short, because Unix sockets only tolerate about 100 characters.

lssh -O exit lab-bastion

Expected:

Exit request sent.

Save your progress? Sign up for free

Drill 4: ProxyJump and port forwarding

lab-app is only reachable via the bastion – at least according to the configuration:

lssh lab-app 'echo via jump: $SSH_CONNECTION'
lssh -G lab-app | grep -E '^(proxyjump|hostname|port) '

Expected:

via jump: 127.0.0.1 52383 127.0.0.1 2222
hostname 127.0.0.1
port 2222
proxyjump lab-bastion

Ad hoc, without configuration, the same works with -J:

lssh -J lab-bastion -p 2222 -i ~/term-lab/ssh/id_lab 127.0.0.1 'echo J ok'

Expected:

J ok

Security: Use ProxyJump instead of ForwardAgent yes. With agent forwarding, root on the bastion can abuse your agent for its own logins. ProxyJump only passes encrypted bytes through.

Port forwarding (high ports, so nothing collides with your services on 8000/8080):

mkdir -p ~/term-lab/www && echo "hallo aus dem Lab" > ~/term-lab/www/index.html
python3 -m http.server 18080 --bind 127.0.0.1 -d ~/term-lab/www >/dev/null 2>&1 &
WEB=$!
lssh -f -N -L 19080:127.0.0.1:18080 lab-bastion
curl -s http://127.0.0.1:19080/

Expected:

hallo aus dem Lab

-f -N sends ssh to the background without a remote command. Because ControlMaster auto applies, this connection becomes the master. lssh -O forward -L 19081:127.0.0.1:18080 lab-bastion adds more tunnels later without reconnecting.

lssh -O exit lab-bastion
kill $WEB

Save your progress? Sign up for free

Drill 5: ssh-agent and keychain

macOS starts an agent for you via launchd. The lab uses its own so that your real keys don’t show up:

eval "$(ssh-agent -a ~/term-lab/ssh/agent.sock)"
ssh-add -l
ssh-add -t 1h ~/term-lab/ssh/id_lab
ssh-add -l

Expected (fingerprint shortened):

Agent pid 13021
The agent has no identities.
Identity added: /Users/<du>/term-lab/ssh/id_lab (lab-key (Lab-Wert))
Lifetime set to 3600 seconds
256 SHA256:… lab-key (Lab-Wert) (ED25519)

-t 1h makes the key expire after an hour. At the end, stop the lab agent:

ssh-agent -k

Expected:

unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 13021 killed;

For real keys with a passphrase, Apple’s OpenSSH uses the keychain. That writes to your login keychain, so here only for reading:

man ssh-add | col -b | grep -A2 -- '--apple-use-keychain'

Expected:

     --apple-use-keychain
         When adding identities, each passphrase will also be stored in
         the user's keychain.  When removing identities with -d, each

In the real ~/.ssh/config this goes together with AddKeysToAgent yes and UseKeychain yes (already in the sample config). Then macOS asks for the passphrase once and never again.

Tip: With the 1Password SSH agent, private keys never leave the vault. It’s hooked in via IdentityAgent "~/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"; you confirm every use with Touch ID. The block is commented out in $BUCH/ssh/config.

Save your progress? Sign up for free

Drill 6: open, pbcopy/pbpaste, mdfind, caffeinate

cd ~/term-lab/projekt
git log --oneline -3 | pbcopy
pbpaste | wc -l
pbpaste | tr a-z A-Z | pbcopy
pbpaste | head -1
open .                       # Finder in the current folder
open -R README.md            # reveal file in the Finder
open -a TextEdit README.md   # open with a specific app

Expected:

       3
2E4E6AA CHORE: CONFIG, MANIFESTE, REPORT

pbcopy/pbpaste connect pipes to the clipboard. Together with y in Neovim (Day 11) and tmux copy mode (Day 8), there’s only one clipboard left.

Spotlight in the terminal:

mdfind -count 'kMDItemFSName == "*.md"'
mdfind -onlyin ~/term-lab 'kMDItemFSName == "*.go"'
mdls -name kMDItemKind -name kMDItemContentType README.md

Expected (shortened, may differ depending on the index state):

9546
/Users/<du>/term-lab/projekt/cmd/shop/main.go
/Users/<du>/term-lab/projekt/internal/cart/cart.go
kMDItemContentType = "dyn.ah62d4rv4ge8043a"
kMDItemKind        = "Markdown document"

Spotlight only indexes freshly created files after a while. Until then, mdfind returns nothing and mdls only (null). For immediate results, there’s still fd (Day 6).

caffeinate keeps the Mac awake as long as a command runs:

caffeinate -i make test      # no sleep during the tests
caffeinate -d -t 3600 &      # display on for one hour
time caffeinate -i -t 1

Expected (shortened):

caffeinate -i -t 1  0,00s user 0,00s system 0% cpu 1,051 total

Save your progress? Sign up for free

Drill 7: defaults, networkQuality, say, osascript, security

defaults reads and writes the apps’ settings. The script from the book only reads and suggests the matching write commands:

$BUCH/macos/defaults-lesen.sh | head -5
defaults read -g AppleLocale
defaults read-type -g AppleLocale

Expected (values depend on your Mac):

Domain                   Key                      Ist-Wert
com.apple.dock           autohide                 (Standard)
  Setzen mit: defaults write com.apple.dock autohide -bool true
com.apple.dock           show-recents             (Standard)
  Setzen mit: defaults write com.apple.dock show-recents -bool false
de_DE
Type is string

You practice writing on your own plist file instead of a real domain:

defaults write ~/term-lab/demo.plist Greeting -string "Hallo Lab"
defaults read ~/term-lab/demo.plist
plutil -p ~/term-lab/demo.plist

Expected:

{
    Greeting = "Hallo Lab";
}
{
  "Greeting" => "Hallo Lab"
}

Warning: defaults write on real domains often only takes effect after killall Dock/killall Finder – and without a backup there’s no undo. First defaults export com.apple.dock dock-backup.plist, then change (Day 13).

More helpers, all read-only or local:

sw_vers --productVersion
sysctl -n machdep.cpu.brand_string
networkQuality -M 8 | tail -4
say -v '?' | grep de_DE | head -2
say -v Anna "Build fertig"
osascript -e 'return (1 + 1) as text'
osascript -l JavaScript -e 'Math.max(3, 7)'

Expected (shortened, on the test machine):

15.6.1
Apple M4
Uplink capacity: 15.146 Mbps
Downlink capacity: 52.026 Mbps
Responsiveness: Medium (227.622 milliseconds | 263 RPM)
Idle Latency: 30.415 milliseconds | 1972 RPM
Anna                de_DE    # Hallo! Ich heiße Anna.
Eddy (Deutsch (Deutschland)) de_DE    # Hallo! Ich heiße Eddy.
2
7

Responsiveness is the latency under load. It explains choppy calls better than the raw bandwidth. And make test; say fertig calls you back when a long run is over.

security talks to the keychain. In the lab, with a keychain of its own that doesn’t end up in the search list:

KC=~/term-lab/lab.keychain-db
security create-keychain -p lab-kc-pass-1234 $KC
security add-generic-password -a lab -s term-lab-api -w 'sk-lab-master-1234' $KC
security find-generic-password -a lab -s term-lab-api -w $KC
security delete-keychain $KC

Expected:

sk-lab-master-1234

In real life, that’s how you load tokens into variables instead of writing them into dotfiles: export GITHUB_TOKEN=$(security find-generic-password -s github-cli -w).

Save your progress? Sign up for free

Drill 8: A launchd agent

On the Mac, launchd replaces cron. A user agent is a plist file. The example writes a timestamp every 5 minutes:

mkdir -p ~/term-lab/launchd && cd ~/term-lab/launchd
sed "s/USERNAME/$(id -un)/" $BUCH/macos/de.term-lab.heartbeat.plist \
  > de.term-lab.heartbeat.plist
plutil -lint de.term-lab.heartbeat.plist
plutil -extract Label raw de.term-lab.heartbeat.plist
plutil -extract StartInterval raw de.term-lab.heartbeat.plist

Expected:

de.term-lab.heartbeat.plist: OK
de.term-lab.heartbeat
300

plutil -lint finds typos before launchd silently ignores them:

sed 's|</dict>||' de.term-lab.heartbeat.plist > kaputt.plist
plutil -lint kaputt.plist

Expected:

kaputt.plist: Close tag on line 28 does not match open tag dict

Activating it is a deliberate step outside the lab. The commands for reference (not run in the book):

cp de.term-lab.heartbeat.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/de.term-lab.heartbeat.plist
launchctl print gui/$(id -u)/de.term-lab.heartbeat | head
launchctl kickstart -k gui/$(id -u)/de.term-lab.heartbeat   # run immediately
launchctl bootout gui/$(id -u)/de.term-lab.heartbeat        # unload again

You can check at any time, read-only, whether the service is loaded:

launchctl print gui/$(id -u)/de.term-lab.heartbeat 2>&1 | head -2

Expected (as long as it’s not activated):

Bad request.
Could not find service "de.term-lab.heartbeat" in domain for user gui: 501

Tip: load/unload are deprecated. bootstrap/bootout give clear error messages. Since macOS 13, “Login Items” in System Settings also reports every new background service.

Save your progress? Sign up for free

Shortcut table

Key/command Effect Mnemonic
Enter ~ . terminate a hung SSH session tilde-dot = the end
Enter ~ # show forwarded connections # = number
Enter ~ ? all escape sequences ? = help
ssh -G host effective config without connecting G = Get
ssh -O check/exit host check/exit the master O = Operation
⌘⇧. in the Finder show hidden files dot = dotfiles
cmd \| pbcopy output to the clipboard pasteboard
open -R datei show file in the Finder R = Reveal

Checkpoint

cd ~/term-lab/ssh
alias lssh='ssh -F ~/term-lab/ssh/config'
ok() { if eval "$1" >/dev/null 2>&1; then echo "✔ $2"; else echo "✘ $2"; fi; }
ok '[ "$(stat -f %Lp id_lab)" = 600 ]' "Lab-Key hat 0600"
ok 'kill -0 "$(cat sshd.pid)"' "Übungs-sshd läuft"
ok 'lssh -o BatchMode=yes lab-app true' "Login über ProxyJump"
ok 'lssh -G lab-app | grep -qx "proxyjump lab-bastion"' "ProxyJump konfiguriert"
ok 'lssh -G lab-bastion | grep -q "^controlpath .*/ssh/cm-"' "ControlPath im Lab"
ok 'plutil -lint ~/term-lab/launchd/de.term-lab.heartbeat.plist' "plist gültig"
ok '! launchctl print gui/$(id -u)/de.term-lab.heartbeat' "Agent nicht geladen"

Expected:

✔ Lab-Key hat 0600
✔ Übungs-sshd läuft
✔ Login über ProxyJump
✔ ProxyJump konfiguriert
✔ ControlPath im Lab
✔ plist gültig
✔ Agent nicht geladen

Challenge

Create a host lab-web that, on connecting, automatically forwards local port 19080 to the web server from Drill 4 (port 18080). curl localhost:19080 should return “hallo aus dem Lab” (“hello from the lab”) without you typing -L.

Hint: What’s -L on the command line is called LocalForward in the configuration.

Solution

cat >> ~/term-lab/ssh/config <<'EOF'

Host lab-web
  HostName 127.0.0.1
  Port 2222
  IdentityFile ~/term-lab/ssh/id_lab
  LocalForward 19080 127.0.0.1:18080
  ControlPath none
EOF
python3 -m http.server 18080 --bind 127.0.0.1 -d ~/term-lab/www >/dev/null 2>&1 &
WEB=$!
lssh -f -N lab-web
curl -s http://127.0.0.1:19080/
lssh -G lab-web | grep localforward

Expected:

hallo aus dem Lab
localforward 19080 [127.0.0.1]:18080

ControlPath none gives the tunnel its own connection. Otherwise it would attach to the master of lab-bastion, since both have the same host, port and user.

Clean up – stop the tunnel, web server and practice sshd:

pkill -f 'ssh -F .*-N lab-web'
kill $WEB
kill "$(cat ~/term-lab/ssh/sshd.pid)"

Save your progress? Sign up for free

Enterprise Note

  • Keys with a passphrase plus keychain or hardware (1Password, Secretive, YubiKey). No keys without a passphrase on laptops.
  • Instead of maintaining known_hosts: SSH certificates (@cert-authority) – for example signed by Vault/OpenBao with a short TTL.
  • In many companies, bastion access goes through Teleport, AWS SSM or Boundary; ProxyCommand hooks them into the same ~/.ssh/config.
  • No ControlPersist on shared machines: anyone who gets to your socket uses your session without a key.
  • launchd agents and defaults belong in the dotfiles repo (Day 13). MDM profiles can overwrite local defaults at any time.