Day 16: The Same with Ansible
Goal: The manual work from Days 2, 3, 4 and 6 lives in two Ansible roles (
linstorfor all hosts,linstor-controller-hafor the controller candidates) that describe the lab without deviation: a dry run and two real runs end withchanged=0, and an outdated satellite is brought up to date by playbook. · Duration: approx. 45 min · Prerequisite: Day 15
Briefing
- Every task in the roles corresponds to a command you ran by hand. Whoever changes something in production reads roles, not a guide.
- The goal is not a rebuild: the roles describe the current state. A dry run without changes proves that role and manual work match.
- What happens only once on the controller does not
belong in the roles: registering nodes, pools, creating
linstor_db, moving the database, starting the controller.
site.yml
├─ linstor_satellites (ctrl, w1, w2, w3) → role linstor
│ PPA, DRBD, LINSTOR debs, hold, module, LVM filter, client, logind, services
└─ linstor_controllers (w1, w2, w3) → role linstor-controller-ha
OCF-Agent, Controller disabled, Drop-in, Mount-Unit, Promoter
Drill 1: Ansible and the inventory
Ansible comes from the Ubuntu packages. The ansible
package brings the collections along, including
community.general for modprobe:
# on k0s-ctrl
mkdir -p ~/k0s-lab/tag16 && cd ~/k0s-lab/tag16
sudo apt-get install -y ansible
ansible --version | head -1
ansible-galaxy collection list community.general | tail -1Expected (abridged):
ansible [core 2.16.3]
community.general 8.3.0
The collections take up a good 500 MB under
/usr/lib/python3/dist-packages. If you do not want that,
install only the ansible-core package and fetch
community.general with
ansible-galaxy collection install. Now the configuration
and the inventory:
# on k0s-ctrl
cat > ansible.cfg <<'EOF'
[defaults]
inventory = inventory.ini
interpreter_python = /usr/bin/python3
[ssh_connection]
pipelining = True
EOF
cat > inventory.ini <<'EOF'
# Alle Hosts mit DRBD. Ohne storage_ip = Witness ohne Storage-Netz.
[linstor_satellites]
k0s-ctrl mgmt_ip=172.16.0.176
k0s-w1 mgmt_ip=172.16.0.186 storage_ip=10.10.0.11 zone=a
k0s-w2 mgmt_ip=172.16.0.36 storage_ip=10.10.0.12 zone=b
k0s-w3 mgmt_ip=172.16.0.72 storage_ip=10.10.0.13 zone=a
# candidates for the LINSTOR controller
[linstor_controllers]
k0s-w1
k0s-w2
k0s-w3
# k0s-Worker (kubelet)
[k0s_workers]
k0s-w1
k0s-w2
k0s-w3
EOF
ansible-inventory --graph
ansible all -m ping -oExpected (abridged):
@all:
|--@ungrouped:
|--@linstor_satellites:
| |--k0s-ctrl
...
|--@k0s_workers:
| |--k0s-w1
| |--k0s-w2
| |--k0s-w3
k0s-w2 | SUCCESS => {"changed": false,"ping": "pong"}
k0s-ctrl | SUCCESS => {"changed": false,"ping": "pong"}
k0s-w3 | SUCCESS => {"changed": false,"ping": "pong"}
k0s-w1 | SUCCESS => {"changed": false,"ping": "pong"}
Ansible resolves the host names via ~/.ssh/config from
Day 0; user and key come from there. storage_ip is missing
for k0s-ctrl – that is how the role recognizes the witness.
The role does not use zone; it documents the zones
alongside Aux/zone and the Kubernetes label.
Save your progress? Sign up for free
Drill 2: Version and checksums in one place
You checked the checksums against the package index on Day 15. They
are still sitting next to the packages on k0s-w1:
# on k0s-ctrl
ssh k0s-w1 'cd ~/debs && sha256sum linstor-*_1.35.3-1_all.deb'Expected:
73ebdbe27243f5d95742c8fb18bccb7a803bd63f7d83493dbef05be68a50cb5a linstor-common_1...
e926a9b9520bc1693134ddfaaaf200e2764d38bb916d7c5e3c16fe42b6ebd495 linstor-controll...
14a89eda8647bd7e4ff6547f5a5f82b2739487698eaa0308f08ffca3545d192d linstor-satellit...
# on k0s-ctrl
mkdir -p group_vars
cat > group_vars/all.yml <<'EOF'
# version and checksums in exactly one place (Day 3, Day 15)
linstor_version: "1.35.3-1"
linstor_sha256:
linstor-common: 73ebdbe27243f5d95742c8fb18bccb7a803bd63f7d83493dbef05be68a50cb5a
linstor-satellite: 14a89eda8647bd7e4ff6547f5a5f82b2739487698eaa0308f08ffca3545d192d
linstor-controller: e926a9b9520bc1693134ddfaaaf200e2764d38bb916d7c5e3c16fe42b6ebd495
linstor_pool: https://packages.linbit.com/public/dists/trixie/misc/pool
linstor_deb_dir: /home/ubuntu/debs
linstor_pakete: [linstor-common, linstor-satellite]
linstor_vip: 10.10.0.100
linstor_vip_nic: ens19
linstor_kandidaten: >-
{{ groups['linstor_controllers'] | map('extract', hostvars, 'mgmt_ip')
| join(',') }}
EOF
cat > group_vars/linstor_controllers.yml <<'EOF'
# linstor-satellite und -controller verlangen linstor-common in genau derselben
# Version: alle drei Pakete in einem Schritt tauschen.
linstor_pakete: [linstor-common, linstor-satellite, linstor-controller]
linstor_vip_agent: >-
ocf:heartbeat:IPaddr2 linstor_vip ip={{ linstor_vip }}
cidr_netmask=24 nic={{ linstor_vip_nic }}
EOF
ansible k0s-ctrl -m debug -a var=linstor_kandidaten -oExpected:
k0s-ctrl | SUCCESS => { "changed": false, "linstor_kandidaten": "172.16.0.186,...
>- folds the lines – only so that they fit into the
book. Where a line is indented further, the line break stays; it does no
harm in the Jinja expression. The second file is the important one:
linstor-satellite and linstor-controller
depend on linstor-common in exactly the
same version. If you swap only common and satellite on a candidate, you
break the controller. That is why the same task installs all three
packages there.
Save your progress? Sign up for free
Drill 3: The role linstor
# on k0s-ctrl
mkdir -p roles/linstor/{tasks,handlers,templates,files}
cat > roles/linstor/tasks/main.yml <<'EOF'
- name: LINBIT-PPA (wie Tag 3, deb822-Datei)
ansible.builtin.command:
cmd: add-apt-repository -y ppa:linbit/linbit-drbd9-stack
creates: /etc/apt/sources.list.d/linbit-ubuntu-linbit-drbd9-stack-noble.sources
- name: DRBD, Werkzeuge, Java und Kernel-Header
ansible.builtin.apt:
name:
- "linux-headers-{{ ansible_facts.kernel }}"
- drbd-dkms
- drbd-utils
- drbd-reactor
- linstor-client
- lvm2
- thin-provisioning-tools
- default-jre-headless
state: present
- name: Ordner für die LINSTOR-Pakete
ansible.builtin.file:
path: "{{ linstor_deb_dir }}"
state: directory
- name: LINSTOR-Pakete laden und Prüfsumme vergleichen
ansible.builtin.get_url:
url: "{{ linstor_pool }}/{{ item }}_{{ linstor_version }}_all.deb"
dest: "{{ linstor_deb_dir }}/{{ item }}_{{ linstor_version }}_all.deb"
checksum: "sha256:{{ linstor_sha256[item] }}"
loop: "{{ linstor_pakete }}"
- name: Installierte Pakete erfassen
ansible.builtin.package_facts:
# Gemessen wird am Satellite: weicht er ab, kommen alle Pakete neu.
- name: LINSTOR-Pakete in einem Schritt installieren
ansible.builtin.command:
argv: "{{ ['apt-get', 'install', '-y', '--allow-downgrades',
'--allow-change-held-packages'] + debs }}"
chdir: "{{ linstor_deb_dir }}"
vars:
debs: "{{ linstor_pakete | map('regex_replace', '^(.+)$',
'./\\1_' ~ linstor_version ~ '_all.deb') | list }}"
ist: "{{ ansible_facts.packages['linstor-satellite'] | default([{}]) }}"
when: ist[0].version | default('') != linstor_version
- name: Versionen festhalten
ansible.builtin.dpkg_selections:
name: "{{ item }}"
selection: hold
loop: "{{ linstor_pakete + ['drbd-dkms', 'drbd-utils'] }}"
- name: DRBD beim Start laden
ansible.builtin.copy:
dest: /etc/modules-load.d/drbd.conf
content: "drbd\n"
mode: "0644"
- name: DRBD jetzt laden
community.general.modprobe:
name: drbd
state: present
- name: LVM soll DRBD-Geräte nicht scannen (Datei vom handgebauten Host)
ansible.builtin.copy:
src: lvmlocal.conf
dest: /etc/lvm/lvmlocal.conf
mode: "0644"
- name: Client-Konfiguration (VIP oder Kandidatenliste)
ansible.builtin.template:
src: linstor-client.conf.j2
dest: /etc/linstor/linstor-client.conf
mode: "0644"
- name: logind darf kubelet 60 s Zeit geben (Tag 2)
ansible.builtin.copy:
dest: /etc/systemd/logind.conf.d/zz-kubelet.conf
content: "[Login]\nInhibitDelayMaxSec=60\n"
mode: "0644"
when: inventory_hostname in groups['k0s_workers']
notify: [logind neu starten, k0sworker neu starten]
- name: Satellite und drbd-reactor laufen
ansible.builtin.systemd_service:
name: "{{ item }}"
enabled: true
state: started
loop: [linstor-satellite, drbd-reactor]
EOFThe client template distinguishes workers and the witness; the handlers belong to the logind drop-in:
# on k0s-ctrl
cat > roles/linstor/templates/linstor-client.conf.j2 <<'EOF'
[global]
{% if storage_ip is defined %}
controllers={{ linstor_vip }}
{% else %}
controllers={{ linstor_kandidaten }}
{% endif %}
EOF
cat > roles/linstor/handlers/main.yml <<'EOF'
- name: logind neu starten
ansible.builtin.systemd_service:
name: systemd-logind
state: restarted
- name: k0sworker neu starten
ansible.builtin.systemd_service:
name: k0sworker
state: restarted
throttle: 1
EOF
scp k0s-w1:/etc/lvm/lvmlocal.conf roles/linstor/files/
grep -A2 '^devices' roles/linstor/files/lvmlocal.confExpected:
devices {
global_filter = [ "r|^/dev/drbd|" ]
}
| Task | done by hand in |
|---|---|
| PPA, DRBD, headers, Java | Day 3, Drills 1–3 |
Debs with checksum, hold |
Day 3, Drill 3; Day 15, Drill 2 |
Module, LVM filter in lvmlocal.conf |
Day 3, Drill 5 |
| Client configuration | Day 4, Drill 1; Day 6, Drill 4 |
zz-kubelet.conf for logind |
Day 2 |
Three decisions are built into this. A single
apt-get install with all files installs the packages from
packages.linbit.com – one at a time, it fails because of
the fixed dependency on linstor-common. You take
lvmlocal.conf in full from the hand-built host instead of
patching lines: blockinfile would add a second
devices section next to the hand-written block. And
add-apt-repository adds the PPA as on Day 3;
apt_repository would create a second source as a
.list file next to it.
Save your progress? Sign up for free
Drill 4: The role linstor-controller-ha
# on k0s-ctrl
mkdir -p roles/linstor-controller-ha/{tasks,handlers,templates,files}
cat > roles/linstor-controller-ha/tasks/main.yml <<'EOF'
# Das Paket linstor-controller installiert die Rolle linstor (linstor_pakete).
- name: OCF-Agent für die VIP
ansible.builtin.apt:
name: resource-agents-base
state: present
# Nur enabled, kein state: Die Rolle startet und stoppt den Controller nie.
- name: Controller nicht von systemd starten lassen
ansible.builtin.systemd_service:
name: linstor-controller
enabled: false
- name: Ordner für den Drop-in
ansible.builtin.file:
path: /etc/systemd/system/linstor-controller.service.d
state: directory
- name: Controller startet nur auf dem Mount (Tag 6)
ansible.builtin.copy:
src: nur-mit-db.conf
dest: /etc/systemd/system/linstor-controller.service.d/nur-mit-db.conf
mode: "0644"
notify: systemd neu laden
- name: Mount-Unit für die Controller-Datenbank
ansible.builtin.copy:
src: var-lib-linstor.mount
dest: /etc/systemd/system/var-lib-linstor.mount
mode: "0644"
notify: systemd neu laden
- name: Promoter für linstor_db
ansible.builtin.template:
src: linstor_db.toml.j2
dest: /etc/drbd-reactor.d/linstor_db.toml
mode: "0644"
notify: drbd-reactor neu laden
EOF
cat > roles/linstor-controller-ha/handlers/main.yml <<'EOF'
- name: systemd neu laden
ansible.builtin.systemd_service:
daemon_reload: true
# reload, nie restart: Ein Neustart stoppt auf dem aktiven Node die ganze Kette.
- name: drbd-reactor neu laden
ansible.builtin.systemd_service:
name: drbd-reactor
state: reloaded
EOFThe three files are the same as on Day 6; the VIP comes from the variables:
# on k0s-ctrl
cat > roles/linstor-controller-ha/files/nur-mit-db.conf <<'EOF'
[Unit]
ConditionPathIsMountPoint=/var/lib/linstor
EOF
cat > roles/linstor-controller-ha/files/var-lib-linstor.mount <<'EOF'
[Unit]
Description=Filesystem for the LINSTOR controller
[Mount]
What=/dev/drbd/by-res/linstor_db/0
Where=/var/lib/linstor
EOF
cat > roles/linstor-controller-ha/templates/linstor_db.toml.j2 <<'EOF'
[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = [
"var-lib-linstor.mount",
"{{ linstor_vip_agent }}",
"linstor-controller.service",
]
on-drbd-demote-failure = "reboot"
stop-services-on-exit = true
EOF
cat > site.yml <<'EOF'
- name: DRBD und LINSTOR auf allen Hosts
hosts: linstor_satellites
become: true
roles: [linstor]
- name: Controller-HA auf den Kandidaten
hosts: linstor_controllers
become: true
roles: [linstor-controller-ha]
EOF
ansible-playbook --syntax-check site.ymlExpected:
playbook: site.yml
Two details decide how operations go. linstor-controller
only gets enabled: false, no
state: with state: stopped, every run would
stop the active controller. Because it is part of the target
drbd-services@linstor_db, the whole chain goes down with
it: VIP gone, unmount, DRBD becomes Secondary, and the promoter restarts
it on whichever node promotes first – 21 s without API in the lab, on
every run. And the handler does reloaded, never
restarted: when drbd-reactor restarts on the active node,
stop-services-on-exit = true shuts down mount, VIP and
controller.
Deliberately not in the roles:
| Step | Why not | Day |
|---|---|---|
Nodes, interfaces, paths, Aux/zone |
one-time, via the LINSTOR API | 4 |
Pools, rg-k8s |
one-time, the StorageClass overrides rg-k8s |
5, 8 |
linstor_db, DB move |
one-time, destructive when repeated | 6 |
| Starting the controller | drbd-reactor’s job | 6 |
Save your progress? Sign up for free
Drill 5: The dry run
# on k0s-ctrl
ansible-playbook site.yml --check --diff | tail -5Expected (abridged, approx. 1 min):
PLAY RECAP *********************************************************************
k0s-ctrl : ok=12 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ...
k0s-w1 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
k0s-w2 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
k0s-w3 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
changed=0 on all four: the roles describe exactly the
lab you built by hand. Skipped are the logind drop-in on
k0s-ctrl (not a worker) and the package update on all hosts
(the version matches). Had you worked differently somewhere,
--diff would show the lines Ansible would change.
Save your progress? Sign up for free
Drill 6: Two real runs
# on k0s-ctrl
time ansible-playbook site.yml | tail -6
ansible-playbook site.yml | tail -6Expected (abridged, twice):
PLAY RECAP *********************************************************************
k0s-ctrl : ok=12 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ...
k0s-w1 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
k0s-w2 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
k0s-w3 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
real 0m54.703s
changed=0 twice is the proof of idempotence: a run only
changes what deviates from the desired state. No service was restarted,
pg-client noticed nothing.
Tip: What is tested here is that the roles match the manual installation. A run against an empty host would additionally install packages and build the DRBD module (Day 3, approx. 3.5 min). Test that on a fresh VM before using the roles for real.
Save your progress? Sign up for free
Drill 7: An update with Ansible
A host with an old version: install LINSTOR 1.34.2 by hand on the
witness; the debs have been in ~/debs since Day 3:
# on k0s-ctrl
cd ~/debs
sudo apt-get install -y --allow-downgrades --allow-change-held-packages \
./linstor-common_1.34.2-1_all.deb ./linstor-satellite_1.34.2-1_all.deb
cd ~/k0s-lab/tag16
sleep 15
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6 | grep ctrl
apt-mark showholdExpected (abridged):
dpkg: warning: downgrading linstor-satellite from 1.35.3-1 to 1.34.2-1
...
k0s-ctrl | OFFLINE(VERSION MISMATCH)
drbd-dkms
drbd-utils
Along the way, the hold marks of the two packages are
gone. What does the dry run say?
# on k0s-ctrl
ansible-playbook site.yml --limit k0s-ctrl --check | grep -A1 'in einem Schritt'Expected:
TASK [linstor : LINSTOR-Pakete in einem Schritt installieren] ******************
skipping: [k0s-ctrl]
Warning:
--checkdoes not runcommandtasks and therefore does not show the missing update as a change. A green dry run does not mean here that the versions match – check them withdpkg-query(Speed Drill 1).
The real run:
# on k0s-ctrl
ansible-playbook site.yml --limit k0s-ctrl | grep -E -A3 'in einem Schritt|RECAP'
sleep 15
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6 | grep ctrlExpected (abridged):
TASK [linstor : LINSTOR-Pakete in einem Schritt installieren] ******************
changed: [k0s-ctrl]
...
PLAY RECAP *********************************************************************
k0s-ctrl : ok=13 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ...
k0s-ctrl | Online
Two changes: packages swapped, hold set again. For a
real update, you change linstor_version and the three
checksums in group_vars/all.yml and follow the order from
Day 15: --limit first on the candidates without the active
controller, then on the witness, then drbd-reactorctl evict
on the active node and only after that --limit on it. A run
over all hosts at once would also swap the active controller.
Save your progress? Sign up for free
Exercise: Speed Drills
- Show the satellite version on all four hosts with an ad-hoc command. Target time: 20 s
- Show the tasks of the playbook without running it. Target time: 15 s
- Show the zones of the workers according to the inventory. Target time: 15 s
- Do a dry run for
k0s-w2only. Target time: 30 s
Solution
# on k0s-ctrl
ansible linstor_satellites -m command -a 'dpkg-query -W linstor-satellite' -o
ansible-playbook site.yml --list-tasks | head -8
ansible k0s_workers -m debug -a var=zone -o
ansible-playbook site.yml --check --limit k0s-w2 | tail -2Expected (abridged):
k0s-w3 | CHANGED | rc=0 | (stdout) linstor-satellite 1.35.3-1
k0s-ctrl | CHANGED | rc=0 | (stdout) linstor-satellite 1.35.3-1
...
play #1 (linstor_satellites): DRBD und LINSTOR auf allen Hosts TAGS: []
tasks:
linstor : LINBIT-PPA (wie Tag 3, deb822-Datei) TAGS: []
...
k0s-w1 | SUCCESS => { "changed": false, "zone": "a"}
k0s-w2 | SUCCESS => { "changed": false, "zone": "b"}
k0s-w3 | SUCCESS => { "changed": false, "zone": "a"}
k0s-w2 : ok=20 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ...
CHANGED for the ad-hoc command only means
that Ansible cannot know whether a command changed anything.
Save your progress? Sign up for free
Checkpoint
# on k0s-ctrl
ansible-playbook site.yml | tail -5Expected: changed=0 and
failed=0 on all four hosts.
# on k0s-ctrl
apt-mark showholdExpected: drbd-dkms,
drbd-utils, linstor-common,
linstor-satellite.
# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6Expected: four nodes Online.
Challenge
A colleague deleted the drop-in nur-mit-db.conf on
k0s-w3 and enabled linstor-controller with
systemctl enable. Reproduce this, find both deviations with
Ansible without changing anything, and then repair them.
Hint: For files, --diff shows the
content; for services, only changed.
Solution
# on k0s-w3
sudo rm /etc/systemd/system/linstor-controller.service.d/nur-mit-db.conf
sudo systemctl daemon-reload
sudo systemctl enable linstor-controller# on k0s-ctrl
ansible-playbook site.yml --limit k0s-w3 --check --diff \
| grep -E -A6 'starten lassen|nur auf dem Mount'Expected (abridged):
TASK [linstor-controller-ha : Controller nicht von systemd starten lassen] *****
changed: [k0s-w3]
...
TASK [linstor-controller-ha : Controller startet nur auf dem Mount (Tag 6)] ****
--- before
+++ after: /home/ubuntu/k0s-lab/tag16/roles/linstor-controller-ha/files/nur-mit-db.conf
@@ -0,0 +1,2 @@
+[Unit]
+ConditionPathIsMountPoint=/var/lib/linstor
# on k0s-ctrl
ansible-playbook site.yml --limit k0s-w3 | grep -B1 '^changed'
ssh k0s-w3 'systemctl is-enabled linstor-controller; systemctl is-active \
linstor-controller'
ssh k0s-w3 'systemctl cat linstor-controller | grep Condition'Expected (abridged):
TASK [linstor-controller-ha : Controller nicht von systemd starten lassen] *****
changed: [k0s-w3]
--
TASK [linstor-controller-ha : Controller startet nur auf dem Mount (Tag 6)] ****
changed: [k0s-w3]
disabled
inactive
ConditionPathIsMountPoint=/var/lib/linstor
The controller did not start in the process: enabled
only takes effect on the next boot. That is exactly the moment when,
without the drop-in, a second controller with an empty database would
have started.
Save your progress? Sign up for free
Interview Note
- How do you tell that a role matches the running system?
--check --diffwithout changes, then two real runs withchanged=0. - Why does the role not start the LINSTOR controller? The
controller belongs to drbd-reactor. The role only sets
enabled: false;state: stoppedwould stop the active controller and with it the whole chain until the promoter restarts it; arestartof drbd-reactor would do the same. - What do you not automate with a role? One-time steps on the
controller: registration, pools,
linstor_db. They belong in a runbook or in a separate playbook that is triggered deliberately.
Enterprise Note
- Roles in Git with review, runs from a pipeline or AWX/Automation Controller with logging, not from a laptop.
- Mirror the package source (aptly, Pulp) instead of fetching from the
internet on every run; version and checksum per environment
(
group_vars/prod,group_vars/staging). - Rolling updates as a separate playbook with
serial: 1, pre- and post-checks (--faultyempty, nodeOnline) andevictbefore the active controller – not as a manual--limitsequence. - Test roles with Molecule against fresh VMs, including the first-installation path with the DKMS build.
- With LINBIT SDS, ready-made Ansible roles and packages with support are available.