Skip to content
Hands-on IT IT Drills

Day 15: Competition

Goal: Find and fix five incidents, each of which actually happened while testing this book. · Duration: approx. 120 min · Prerequisite: Day 0–14

Rules of Play

  • Every round starts with a situation: What happened, what is the team reporting?
  • You find the cause with the tools from the book: onevm show, onehost show, the logs under /var/log/one, SSH into the VMs. Only then do you fix it.
  • The files for the rounds are in ~/one-lab/code/wettkampf/.
  • There is a hint for every round. Only read it if you have been stuck for ten minutes.
  • Count the rounds you complete without a hint.

Round 1: No htop

Situation: A colleague “cleaned up” and removed a line he did not recognize:

sudo sed -i '/IGNORE_RESOLVCONF=yes/d' /etc/default/dnsmasq
sudo systemctl restart dnsmasq

Since then, the team reports: the VMs from alpine-start no longer have htop, even though the startup script runs through.

onetemplate instantiate alpine-start --name wk1 --user-inputs GRUSS=wk

After a minute and a half:

IP=$(onevm show wk1 | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP "cat /root/start.log; which htop"

Expected:

fertig

The script writes “fertig”, but htop is missing.

Note: The startup script does not abort on errors. Check inside the VM whether it reaches the internet, once by IP and once by name.

Solution

ssh $IP "ping -c1 -W2 1.1.1.1 | tail -1; nslookup dl-cdn.alpinelinux.org 2>&1 | tail -1"

Expected:

round-trip min/avg/max = 5.424/5.424/5.424 ms
** server can't find dl-cdn.alpinelinux.org: REFUSED

By IP it works, by name it does not: that is the DNS error from Day 1. apk add failed, and echo fertig ran anyway.

echo "IGNORE_RESOLVCONF=yes" | sudo tee -a /etc/default/dnsmasq
sudo systemctl restart dnsmasq
onevm terminate --hard wk1
onetemplate instantiate alpine-start --name wk1b --user-inputs GRUSS=wk

After a minute and a half:

IP=$(onevm show wk1b | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP which htop
onevm terminate --hard wk1b

Expected (shortened):

/usr/bin/htop

Prevention: Start startup scripts with set -e; then a failed apk add aborts the script instead of letting it report “fertig”. And the fix from Day 1 belongs in the file with a comment, so that nobody mistakes it for junk.

Save your progress? Sign up for free

Round 2: The second database

Situation: A team built its own Postgres appliance, using an older script of yours. The first VM ran beautifully. After a rebuild with the same data disk, Postgres no longer starts.

mkdir -p ~/one-lab/wk2
cd ~/one-lab/wk2
cp ~/one-lab/code/wettkampf/runde2/db.sh .
sed -e 's/^NAME = "notes-db"/NAME = "wk-db"/' -e 's/IMAGE = "notes-daten"/IMAGE = "wk-daten"/' \
  -e 's/10.10.10.20/10.10.10.28/' \
  -e 's|START_SCRIPT_BASE64 = "[^"]*"|START_SCRIPT_BASE64 = "SKRIPT"|' \
  ../tag06/db.tpl > wk-db.tpl
sed -i "s|SKRIPT|$(base64 -w0 db.sh)|" wk-db.tpl
oneimage create --name wk-daten --type DATABLOCK --size 512 --persistent --datastore default
onetemplate create wk-db.tpl
onetemplate instantiate wk-db --name wk --user-inputs DB_PASSWORD=wk-pass

After two minutes, Postgres is running:

IP=$(onevm show wk | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP rc-service postgresql status

Expected:

 * status: started

Rebuild:

onevm terminate wk

When onevm list no longer shows wk:

onetemplate instantiate wk-db --name wk --user-inputs DB_PASSWORD=wk-pass

After two minutes:

IP=$(onevm show wk | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh $IP "rc-service postgresql status; tail -1 /var/log/postgresql/postmaster.log"

Expected:

 * status: stopped
postgres: could not access the server configuration file "/var/lib/postgresql/16/data/postgresql.conf": No such file or directory

Note: ls -la in the data directory. And compare the script with the one from Day 6.

Solution

ssh $IP "ls -la /var/lib/postgresql/16/data/*.conf"
diff ~/one-lab/tag06/db.sh db.sh

Expected (shortened):

lrwxrwxrwx 1 root root 29 … pg_hba.conf -> /etc/postgresql16/pg_hba.conf
lrwxrwxrwx 1 root root 31 … pg_ident.conf -> /etc/postgresql16/pg_ident.conf
lrwxrwxrwx 1 root root 33 … postgresql.conf -> /etc/postgresql16/postgresql.conf
10,11d9
< # Konfiguration bei den Daten lassen, nicht auf der flüchtigen Systemplatte
< echo conf_dir=/var/lib/postgresql/16/data >> /etc/conf.d/postgresql

The case from Day 6: without conf_dir, the configuration files were on the system disk of the first VM, and that no longer exists. The data is intact; only the three files are missing. You rebuild them, this time on the data disk:

ssh $IP 'cd /var/lib/postgresql/16/data && \
  rm postgresql.conf pg_hba.conf pg_ident.conf && \
  install -o postgres -g postgres -m 600 /usr/share/postgresql16/postgresql.conf.sample postgresql.conf && \
  install -o postgres -g postgres -m 600 /usr/share/postgresql16/pg_ident.conf.sample pg_ident.conf && \
  echo "listen_addresses = '"'"'*'"'"'" >> postgresql.conf && \
  printf "local all all peer\nhost notes notes 10.10.10.0/24 scram-sha-256\n" > pg_hba.conf && \
  echo conf_dir=/var/lib/postgresql/16/data >> /etc/conf.d/postgresql && \
  rc-service postgresql start'

Expected (shortened):

 * Starting PostgreSQL 16 ... [ ok ]

pg_hba.conf is not copied from the sample: it contains placeholders such as @remove-line-for-nolocal@ that Postgres chokes on. Afterwards, the script needs to be fixed, otherwise this happens again at the next rebuild. For the lab, you tear down:

onevm terminate --hard wk
onetemplate delete wk-db

When onevm list no longer shows wk:

oneimage delete wk-daten

Save your progress? Sign up for free

Round 3: Shared, but not quite

Situation: Team blau should use notes-schnell from Day 10. A colleague shared the template:

onetemplate chmod notes-schnell 604
A="--user blau-dev --password lab-blau-pass-123"
onetemplate instantiate notes-schnell --name wk3 \
  --user-inputs DB_PASSWORD=x,API_KEY=y,NOTES_VERSION=0.7.1 $A

Expected:

[one.template.instantiate] User [2] : Not authorized to perform USE IMAGE [9].

Note: What does a VM from this template consist of?

Solution

The message names the object: image 9, that is, notes-basis. On Day 8, it was the image Alpine Linux 3.20 and the security group. Starting a VM means using every object it consists of. The sharing has to be complete:

oneimage show notes-basis | grep -A3 PERMISSIONS

Expected:

PERMISSIONS
OWNER          : um-
GROUP          : ---
OTHER          : ---

With oneimage chmod notes-basis 604, it would work. But remember the Challenge from Day 10: the VM would then end up in the intern network. For the lab, you revoke the sharing:

onetemplate chmod notes-schnell 600

Save your progress? Sign up for free

Round 4: The nameless hook

Situation: A colleague built a “simpler” hook, without any script at all:

onehook create ~/one-lab/code/wettkampf/runde4/kurz.hook
onetemplate instantiate alpine --name wk4 --user-inputs GRUSS=w

After a minute:

sudo journalctl -t opennebula-hook --since "-2min" --no-pager | tail -2

Expected (times will differ):

Nov 12 20:58:21 one opennebula-hook[1338534]: VM ist gestartet
Nov 12 20:58:21 one opennebula-hook[1338542]: VM wk4 läuft

The second line comes from your hook from Day 11. The first is supposed to name the VM ID, but it does not.

Note: cat ~/one-lab/code/wettkampf/runde4/kurz.hook

Solution

ARGUMENTS = "-t opennebula-hook VM $ID ist gestartet"

In ARGUMENTS, OpenNebula only replaces certain placeholders, above all $TEMPLATE. $ID is not one of them and stays empty. The right way is the one from Day 11: pass the template and read it in the script. That makes the hook superfluous:

onehook delete vm-kurz
onevm terminate --hard wk4

Save your progress? Sign up for free

Round 5: The reservation that is none

Situation: A colleague wanted to raise the reservation from Day 14 to 2 GB:

echo 'RESERVED_MEM="2G"' > /tmp/r.txt
onehost update localhost --append /tmp/r.txt
onehost list

Expected:

  ID NAME            CLUSTER    TVM      ALLOCATED_CPU      ALLOCATED_MEM STAT
   0 localhost       default      3    150 / 200 (75%)  1.3G / 3.8G (32%) on

Instead of less, suddenly 3.8 GB are available for VMs again.

Note: Day 14, Drill 1.

Solution

RESERVED_MEM expects kilobytes. 2G is not a number, so no reservation applies at all anymore; the old one is gone too. That is worse than before. 2 GB are 2097152 KB:

echo 'RESERVED_MEM="2097152"' > /tmp/r.txt
onehost update localhost --append /tmp/r.txt
onehost list

Expected:

   0 localhost       default      3    150 / 200 (75%)  1.3G / 1.8G (68%) on

For the lab, you restore the value from Day 14:

echo 'RESERVED_MEM="1572864"' > /tmp/r.txt
onehost update localhost --append /tmp/r.txt

Prevention: Unit errors only show up if you check after every change. onehost list right afterwards is a must.

Save your progress? Sign up for free

Scoring

Rounds without a hint Verdict
5 You can run OpenNebula in production.
3–4 Solid. Reread the days for the rounds you botched.
0–2 Do Day 1, 6, 8, 11, and 14 again.

OpenNebula did not report four of the five faults as errors: a script that says “fertig”, a hook with an empty placeholder, an ignored unit, a configuration on the wrong disk. The lesson of this day: after every change, check whether the result is right, not just whether the command ran through.