Day 15: Competition
Goal: Find and fix five incidents, each of which actually happened while testing this book. · Duration: approx. 120 min · Prerequisite: Day 0–14
Rules of Play
- Every round starts with a situation: What happened, what is the team reporting?
- You find the cause with the tools from the book:
onevm show,onehost show, the logs under/var/log/one, SSH into the VMs. Only then do you fix it. - The files for the rounds are in
~/one-lab/code/wettkampf/. - There is a hint for every round. Only read it if you have been stuck for ten minutes.
- Count the rounds you complete without a hint.
Round 1: No htop
Situation: A colleague “cleaned up” and removed a line he did not recognize:
sudo sed -i '/IGNORE_RESOLVCONF=yes/d' /etc/default/dnsmasq
sudo systemctl restart dnsmasqSince then, the team reports: the VMs from alpine-start
no longer have htop, even though the startup script runs
through.
onetemplate instantiate alpine-start --name wk1 --user-inputs GRUSS=wkAfter a minute and a half:
IP=$(onevm show wk1 | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP "cat /root/start.log; which htop"Expected:
fertig
The script writes “fertig”, but htop is missing.
Note: The startup script does not abort on errors. Check inside the VM whether it reaches the internet, once by IP and once by name.
Solution
ssh $IP "ping -c1 -W2 1.1.1.1 | tail -1; nslookup dl-cdn.alpinelinux.org 2>&1 | tail -1"Expected:
round-trip min/avg/max = 5.424/5.424/5.424 ms
** server can't find dl-cdn.alpinelinux.org: REFUSED
By IP it works, by name it does not: that is the DNS error from Day
1. apk add failed, and echo fertig ran
anyway.
echo "IGNORE_RESOLVCONF=yes" | sudo tee -a /etc/default/dnsmasq
sudo systemctl restart dnsmasq
onevm terminate --hard wk1
onetemplate instantiate alpine-start --name wk1b --user-inputs GRUSS=wkAfter a minute and a half:
IP=$(onevm show wk1b | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP which htop
onevm terminate --hard wk1bExpected (shortened):
/usr/bin/htop
Prevention: Start startup scripts with
set -e; then a failed apk add aborts the
script instead of letting it report “fertig”. And the fix from Day 1
belongs in the file with a comment, so that nobody mistakes it for
junk.
Save your progress? Sign up for free
Round 2: The second database
Situation: A team built its own Postgres appliance, using an older script of yours. The first VM ran beautifully. After a rebuild with the same data disk, Postgres no longer starts.
mkdir -p ~/one-lab/wk2
cd ~/one-lab/wk2
cp ~/one-lab/code/wettkampf/runde2/db.sh .
sed -e 's/^NAME = "notes-db"/NAME = "wk-db"/' -e 's/IMAGE = "notes-daten"/IMAGE = "wk-daten"/' \
-e 's/10.10.10.20/10.10.10.28/' \
-e 's|START_SCRIPT_BASE64 = "[^"]*"|START_SCRIPT_BASE64 = "SKRIPT"|' \
../tag06/db.tpl > wk-db.tpl
sed -i "s|SKRIPT|$(base64 -w0 db.sh)|" wk-db.tpl
oneimage create --name wk-daten --type DATABLOCK --size 512 --persistent --datastore default
onetemplate create wk-db.tpl
onetemplate instantiate wk-db --name wk --user-inputs DB_PASSWORD=wk-passAfter two minutes, Postgres is running:
IP=$(onevm show wk | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP rc-service postgresql statusExpected:
* status: started
Rebuild:
onevm terminate wkWhen onevm list no longer shows wk:
onetemplate instantiate wk-db --name wk --user-inputs DB_PASSWORD=wk-passAfter two minutes:
IP=$(onevm show wk | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP "rc-service postgresql status; tail -1 /var/log/postgresql/postmaster.log"Expected:
* status: stopped
postgres: could not access the server configuration file "/var/lib/postgresql/16/data/postgresql.conf": No such file or directory
Note: ls -la in the data directory. And
compare the script with the one from Day 6.
Solution
ssh $IP "ls -la /var/lib/postgresql/16/data/*.conf"
diff ~/one-lab/tag06/db.sh db.shExpected (shortened):
lrwxrwxrwx 1 root root 29 … pg_hba.conf -> /etc/postgresql16/pg_hba.conf
lrwxrwxrwx 1 root root 31 … pg_ident.conf -> /etc/postgresql16/pg_ident.conf
lrwxrwxrwx 1 root root 33 … postgresql.conf -> /etc/postgresql16/postgresql.conf
10,11d9
< # Konfiguration bei den Daten lassen, nicht auf der flüchtigen Systemplatte
< echo conf_dir=/var/lib/postgresql/16/data >> /etc/conf.d/postgresql
The case from Day 6: without conf_dir, the configuration
files were on the system disk of the first VM, and that no longer
exists. The data is intact; only the three files are missing. You
rebuild them, this time on the data disk:
ssh $IP 'cd /var/lib/postgresql/16/data && \
rm postgresql.conf pg_hba.conf pg_ident.conf && \
install -o postgres -g postgres -m 600 /usr/share/postgresql16/postgresql.conf.sample postgresql.conf && \
install -o postgres -g postgres -m 600 /usr/share/postgresql16/pg_ident.conf.sample pg_ident.conf && \
echo "listen_addresses = '"'"'*'"'"'" >> postgresql.conf && \
printf "local all all peer\nhost notes notes 10.10.10.0/24 scram-sha-256\n" > pg_hba.conf && \
echo conf_dir=/var/lib/postgresql/16/data >> /etc/conf.d/postgresql && \
rc-service postgresql start'Expected (shortened):
* Starting PostgreSQL 16 ... [ ok ]
pg_hba.conf is not copied from the sample: it contains
placeholders such as @remove-line-for-nolocal@ that
Postgres chokes on. Afterwards, the script needs to be fixed, otherwise
this happens again at the next rebuild. For the lab, you tear down:
onevm terminate --hard wk
onetemplate delete wk-dbWhen onevm list no longer shows wk:
oneimage delete wk-datenSave your progress? Sign up for free
Round 3: Shared, but not quite
Situation: Team blau should use
notes-schnell from Day 10. A colleague shared the
template:
onetemplate chmod notes-schnell 604
A="--user blau-dev --password lab-blau-pass-123"
onetemplate instantiate notes-schnell --name wk3 \
--user-inputs DB_PASSWORD=x,API_KEY=y,NOTES_VERSION=0.7.1 $AExpected:
[one.template.instantiate] User [2] : Not authorized to perform USE IMAGE [9].
Note: What does a VM from this template consist of?
Solution
The message names the object: image 9, that is,
notes-basis. On Day 8, it was the image
Alpine Linux 3.20 and the security group. Starting a VM
means using every object it consists of. The sharing has to be
complete:
oneimage show notes-basis | grep -A3 PERMISSIONSExpected:
PERMISSIONS
OWNER : um-
GROUP : ---
OTHER : ---
With oneimage chmod notes-basis 604, it would work. But
remember the Challenge from Day 10: the VM would then end up in the
intern network. For the lab, you revoke the sharing:
onetemplate chmod notes-schnell 600Save your progress? Sign up for free
Round 4: The nameless hook
Situation: A colleague built a “simpler” hook, without any script at all:
onehook create ~/one-lab/code/wettkampf/runde4/kurz.hook
onetemplate instantiate alpine --name wk4 --user-inputs GRUSS=wAfter a minute:
sudo journalctl -t opennebula-hook --since "-2min" --no-pager | tail -2Expected (times will differ):
Nov 12 20:58:21 one opennebula-hook[1338534]: VM ist gestartet
Nov 12 20:58:21 one opennebula-hook[1338542]: VM wk4 läuft
The second line comes from your hook from Day 11. The first is supposed to name the VM ID, but it does not.
Note:
cat ~/one-lab/code/wettkampf/runde4/kurz.hook
Solution
ARGUMENTS = "-t opennebula-hook VM $ID ist gestartet"
In ARGUMENTS, OpenNebula only replaces certain
placeholders, above all $TEMPLATE. $ID is not
one of them and stays empty. The right way is the one from Day 11: pass
the template and read it in the script. That makes the hook
superfluous:
onehook delete vm-kurz
onevm terminate --hard wk4Save your progress? Sign up for free
Round 5: The reservation that is none
Situation: A colleague wanted to raise the reservation from Day 14 to 2 GB:
echo 'RESERVED_MEM="2G"' > /tmp/r.txt
onehost update localhost --append /tmp/r.txt
onehost listExpected:
ID NAME CLUSTER TVM ALLOCATED_CPU ALLOCATED_MEM STAT
0 localhost default 3 150 / 200 (75%) 1.3G / 3.8G (32%) on
Instead of less, suddenly 3.8 GB are available for VMs again.
Note: Day 14, Drill 1.
Solution
RESERVED_MEM expects kilobytes. 2G is not a
number, so no reservation applies at all anymore; the old one is gone
too. That is worse than before. 2 GB are 2097152 KB:
echo 'RESERVED_MEM="2097152"' > /tmp/r.txt
onehost update localhost --append /tmp/r.txt
onehost listExpected:
0 localhost default 3 150 / 200 (75%) 1.3G / 1.8G (68%) on
For the lab, you restore the value from Day 14:
echo 'RESERVED_MEM="1572864"' > /tmp/r.txt
onehost update localhost --append /tmp/r.txtPrevention: Unit errors only show up if you check
after every change. onehost list right afterwards is a
must.
Save your progress? Sign up for free
Scoring
| Rounds without a hint | Verdict |
|---|---|
| 5 | You can run OpenNebula in production. |
| 3–4 | Solid. Reread the days for the rounds you botched. |
| 0–2 | Do Day 1, 6, 8, 11, and 14 again. |
OpenNebula did not report four of the five faults as errors: a script that says “fertig”, a hook with an empty placeholder, an ignored unit, a configuration on the wrong disk. The lesson of this day: after every change, check whether the result is right, not just whether the command ran through.