Skip to content
Hands-on IT IT Drills

Day 16: The Same with Ansible

Goal: The manual work from Days 2, 3, 4 and 6 lives in two Ansible roles (linstor for all hosts, linstor-controller-ha for the controller candidates) that describe the lab without deviation: a dry run and two real runs end with changed=0, and an outdated satellite is brought up to date by playbook. · Duration: approx. 45 min · Prerequisite: Day 15

Briefing

  • Every task in the roles corresponds to a command you ran by hand. Whoever changes something in production reads roles, not a guide.
  • The goal is not a rebuild: the roles describe the current state. A dry run without changes proves that role and manual work match.
  • What happens only once on the controller does not belong in the roles: registering nodes, pools, creating linstor_db, moving the database, starting the controller.
 site.yml
  ├─ linstor_satellites (ctrl, w1, w2, w3) → role linstor
  │     PPA, DRBD, LINSTOR debs, hold, module, LVM filter, client, logind, services
  └─ linstor_controllers (w1, w2, w3)      → role linstor-controller-ha
        OCF-Agent, Controller disabled, Drop-in, Mount-Unit, Promoter

Drill 1: Ansible and the inventory

Ansible comes from the Ubuntu packages. The ansible package brings the collections along, including community.general for modprobe:

# on k0s-ctrl
mkdir -p ~/k0s-lab/tag16 && cd ~/k0s-lab/tag16
sudo apt-get install -y ansible
ansible --version | head -1
ansible-galaxy collection list community.general | tail -1

Expected (abridged):

ansible [core 2.16.3]
community.general 8.3.0

The collections take up a good 500 MB under /usr/lib/python3/dist-packages. If you do not want that, install only the ansible-core package and fetch community.general with ansible-galaxy collection install. Now the configuration and the inventory:

# on k0s-ctrl
cat > ansible.cfg <<'EOF'
[defaults]
inventory = inventory.ini
interpreter_python = /usr/bin/python3

[ssh_connection]
pipelining = True
EOF
cat > inventory.ini <<'EOF'
# Alle Hosts mit DRBD. Ohne storage_ip = Witness ohne Storage-Netz.
[linstor_satellites]
k0s-ctrl mgmt_ip=172.16.0.176
k0s-w1   mgmt_ip=172.16.0.186 storage_ip=10.10.0.11 zone=a
k0s-w2   mgmt_ip=172.16.0.36  storage_ip=10.10.0.12 zone=b
k0s-w3   mgmt_ip=172.16.0.72  storage_ip=10.10.0.13 zone=a

# candidates for the LINSTOR controller
[linstor_controllers]
k0s-w1
k0s-w2
k0s-w3

# k0s-Worker (kubelet)
[k0s_workers]
k0s-w1
k0s-w2
k0s-w3
EOF
ansible-inventory --graph
ansible all -m ping -o

Expected (abridged):

@all:
  |--@ungrouped:
  |--@linstor_satellites:
  |  |--k0s-ctrl
...
  |--@k0s_workers:
  |  |--k0s-w1
  |  |--k0s-w2
  |  |--k0s-w3
k0s-w2 | SUCCESS => {"changed": false,"ping": "pong"}
k0s-ctrl | SUCCESS => {"changed": false,"ping": "pong"}
k0s-w3 | SUCCESS => {"changed": false,"ping": "pong"}
k0s-w1 | SUCCESS => {"changed": false,"ping": "pong"}

Ansible resolves the host names via ~/.ssh/config from Day 0; user and key come from there. storage_ip is missing for k0s-ctrl – that is how the role recognizes the witness. The role does not use zone; it documents the zones alongside Aux/zone and the Kubernetes label.

Save your progress? Sign up for free

Drill 2: Version and checksums in one place

You checked the checksums against the package index on Day 15. They are still sitting next to the packages on k0s-w1:

# on k0s-ctrl
ssh k0s-w1 'cd ~/debs && sha256sum linstor-*_1.35.3-1_all.deb'

Expected:

73ebdbe27243f5d95742c8fb18bccb7a803bd63f7d83493dbef05be68a50cb5a  linstor-common_1...
e926a9b9520bc1693134ddfaaaf200e2764d38bb916d7c5e3c16fe42b6ebd495  linstor-controll...
14a89eda8647bd7e4ff6547f5a5f82b2739487698eaa0308f08ffca3545d192d  linstor-satellit...
# on k0s-ctrl
mkdir -p group_vars
cat > group_vars/all.yml <<'EOF'
# version and checksums in exactly one place (Day 3, Day 15)
linstor_version: "1.35.3-1"
linstor_sha256:
  linstor-common: 73ebdbe27243f5d95742c8fb18bccb7a803bd63f7d83493dbef05be68a50cb5a
  linstor-satellite: 14a89eda8647bd7e4ff6547f5a5f82b2739487698eaa0308f08ffca3545d192d
  linstor-controller: e926a9b9520bc1693134ddfaaaf200e2764d38bb916d7c5e3c16fe42b6ebd495
linstor_pool: https://packages.linbit.com/public/dists/trixie/misc/pool
linstor_deb_dir: /home/ubuntu/debs
linstor_pakete: [linstor-common, linstor-satellite]
linstor_vip: 10.10.0.100
linstor_vip_nic: ens19
linstor_kandidaten: >-
  {{ groups['linstor_controllers'] | map('extract', hostvars, 'mgmt_ip')
     | join(',') }}
EOF
cat > group_vars/linstor_controllers.yml <<'EOF'
# linstor-satellite und -controller verlangen linstor-common in genau derselben
# Version: alle drei Pakete in einem Schritt tauschen.
linstor_pakete: [linstor-common, linstor-satellite, linstor-controller]
linstor_vip_agent: >-
  ocf:heartbeat:IPaddr2 linstor_vip ip={{ linstor_vip }}
  cidr_netmask=24 nic={{ linstor_vip_nic }}
EOF
ansible k0s-ctrl -m debug -a var=linstor_kandidaten -o

Expected:

k0s-ctrl | SUCCESS => {    "changed": false,    "linstor_kandidaten": "172.16.0.186,...

>- folds the lines – only so that they fit into the book. Where a line is indented further, the line break stays; it does no harm in the Jinja expression. The second file is the important one: linstor-satellite and linstor-controller depend on linstor-common in exactly the same version. If you swap only common and satellite on a candidate, you break the controller. That is why the same task installs all three packages there.

Save your progress? Sign up for free

Drill 3: The role linstor

# on k0s-ctrl
mkdir -p roles/linstor/{tasks,handlers,templates,files}
cat > roles/linstor/tasks/main.yml <<'EOF'
- name: LINBIT-PPA (wie Tag 3, deb822-Datei)
  ansible.builtin.command:
    cmd: add-apt-repository -y ppa:linbit/linbit-drbd9-stack
    creates: /etc/apt/sources.list.d/linbit-ubuntu-linbit-drbd9-stack-noble.sources

- name: DRBD, Werkzeuge, Java und Kernel-Header
  ansible.builtin.apt:
    name:
      - "linux-headers-{{ ansible_facts.kernel }}"
      - drbd-dkms
      - drbd-utils
      - drbd-reactor
      - linstor-client
      - lvm2
      - thin-provisioning-tools
      - default-jre-headless
    state: present

- name: Ordner für die LINSTOR-Pakete
  ansible.builtin.file:
    path: "{{ linstor_deb_dir }}"
    state: directory

- name: LINSTOR-Pakete laden und Prüfsumme vergleichen
  ansible.builtin.get_url:
    url: "{{ linstor_pool }}/{{ item }}_{{ linstor_version }}_all.deb"
    dest: "{{ linstor_deb_dir }}/{{ item }}_{{ linstor_version }}_all.deb"
    checksum: "sha256:{{ linstor_sha256[item] }}"
  loop: "{{ linstor_pakete }}"

- name: Installierte Pakete erfassen
  ansible.builtin.package_facts:

# Gemessen wird am Satellite: weicht er ab, kommen alle Pakete neu.
- name: LINSTOR-Pakete in einem Schritt installieren
  ansible.builtin.command:
    argv: "{{ ['apt-get', 'install', '-y', '--allow-downgrades',
               '--allow-change-held-packages'] + debs }}"
    chdir: "{{ linstor_deb_dir }}"
  vars:
    debs: "{{ linstor_pakete | map('regex_replace', '^(.+)$',
               './\\1_' ~ linstor_version ~ '_all.deb') | list }}"
    ist: "{{ ansible_facts.packages['linstor-satellite'] | default([{}]) }}"
  when: ist[0].version | default('') != linstor_version

- name: Versionen festhalten
  ansible.builtin.dpkg_selections:
    name: "{{ item }}"
    selection: hold
  loop: "{{ linstor_pakete + ['drbd-dkms', 'drbd-utils'] }}"

- name: DRBD beim Start laden
  ansible.builtin.copy:
    dest: /etc/modules-load.d/drbd.conf
    content: "drbd\n"
    mode: "0644"

- name: DRBD jetzt laden
  community.general.modprobe:
    name: drbd
    state: present

- name: LVM soll DRBD-Geräte nicht scannen (Datei vom handgebauten Host)
  ansible.builtin.copy:
    src: lvmlocal.conf
    dest: /etc/lvm/lvmlocal.conf
    mode: "0644"

- name: Client-Konfiguration (VIP oder Kandidatenliste)
  ansible.builtin.template:
    src: linstor-client.conf.j2
    dest: /etc/linstor/linstor-client.conf
    mode: "0644"

- name: logind darf kubelet 60 s Zeit geben (Tag 2)
  ansible.builtin.copy:
    dest: /etc/systemd/logind.conf.d/zz-kubelet.conf
    content: "[Login]\nInhibitDelayMaxSec=60\n"
    mode: "0644"
  when: inventory_hostname in groups['k0s_workers']
  notify: [logind neu starten, k0sworker neu starten]

- name: Satellite und drbd-reactor laufen
  ansible.builtin.systemd_service:
    name: "{{ item }}"
    enabled: true
    state: started
  loop: [linstor-satellite, drbd-reactor]
EOF

The client template distinguishes workers and the witness; the handlers belong to the logind drop-in:

# on k0s-ctrl
cat > roles/linstor/templates/linstor-client.conf.j2 <<'EOF'
[global]
{% if storage_ip is defined %}
controllers={{ linstor_vip }}
{% else %}
controllers={{ linstor_kandidaten }}
{% endif %}
EOF
cat > roles/linstor/handlers/main.yml <<'EOF'
- name: logind neu starten
  ansible.builtin.systemd_service:
    name: systemd-logind
    state: restarted

- name: k0sworker neu starten
  ansible.builtin.systemd_service:
    name: k0sworker
    state: restarted
  throttle: 1
EOF
scp k0s-w1:/etc/lvm/lvmlocal.conf roles/linstor/files/
grep -A2 '^devices' roles/linstor/files/lvmlocal.conf

Expected:

devices {
    global_filter = [ "r|^/dev/drbd|" ]
}
Task done by hand in
PPA, DRBD, headers, Java Day 3, Drills 1–3
Debs with checksum, hold Day 3, Drill 3; Day 15, Drill 2
Module, LVM filter in lvmlocal.conf Day 3, Drill 5
Client configuration Day 4, Drill 1; Day 6, Drill 4
zz-kubelet.conf for logind Day 2

Three decisions are built into this. A single apt-get install with all files installs the packages from packages.linbit.com – one at a time, it fails because of the fixed dependency on linstor-common. You take lvmlocal.conf in full from the hand-built host instead of patching lines: blockinfile would add a second devices section next to the hand-written block. And add-apt-repository adds the PPA as on Day 3; apt_repository would create a second source as a .list file next to it.

Save your progress? Sign up for free

Drill 4: The role linstor-controller-ha

# on k0s-ctrl
mkdir -p roles/linstor-controller-ha/{tasks,handlers,templates,files}
cat > roles/linstor-controller-ha/tasks/main.yml <<'EOF'
# Das Paket linstor-controller installiert die Rolle linstor (linstor_pakete).
- name: OCF-Agent für die VIP
  ansible.builtin.apt:
    name: resource-agents-base
    state: present

# Nur enabled, kein state: Die Rolle startet und stoppt den Controller nie.
- name: Controller nicht von systemd starten lassen
  ansible.builtin.systemd_service:
    name: linstor-controller
    enabled: false

- name: Ordner für den Drop-in
  ansible.builtin.file:
    path: /etc/systemd/system/linstor-controller.service.d
    state: directory

- name: Controller startet nur auf dem Mount (Tag 6)
  ansible.builtin.copy:
    src: nur-mit-db.conf
    dest: /etc/systemd/system/linstor-controller.service.d/nur-mit-db.conf
    mode: "0644"
  notify: systemd neu laden

- name: Mount-Unit für die Controller-Datenbank
  ansible.builtin.copy:
    src: var-lib-linstor.mount
    dest: /etc/systemd/system/var-lib-linstor.mount
    mode: "0644"
  notify: systemd neu laden

- name: Promoter für linstor_db
  ansible.builtin.template:
    src: linstor_db.toml.j2
    dest: /etc/drbd-reactor.d/linstor_db.toml
    mode: "0644"
  notify: drbd-reactor neu laden
EOF
cat > roles/linstor-controller-ha/handlers/main.yml <<'EOF'
- name: systemd neu laden
  ansible.builtin.systemd_service:
    daemon_reload: true

# reload, nie restart: Ein Neustart stoppt auf dem aktiven Node die ganze Kette.
- name: drbd-reactor neu laden
  ansible.builtin.systemd_service:
    name: drbd-reactor
    state: reloaded
EOF

The three files are the same as on Day 6; the VIP comes from the variables:

# on k0s-ctrl
cat > roles/linstor-controller-ha/files/nur-mit-db.conf <<'EOF'
[Unit]
ConditionPathIsMountPoint=/var/lib/linstor
EOF
cat > roles/linstor-controller-ha/files/var-lib-linstor.mount <<'EOF'
[Unit]
Description=Filesystem for the LINSTOR controller

[Mount]
What=/dev/drbd/by-res/linstor_db/0
Where=/var/lib/linstor
EOF
cat > roles/linstor-controller-ha/templates/linstor_db.toml.j2 <<'EOF'
[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = [
  "var-lib-linstor.mount",
  "{{ linstor_vip_agent }}",
  "linstor-controller.service",
]
on-drbd-demote-failure = "reboot"
stop-services-on-exit = true
EOF
cat > site.yml <<'EOF'
- name: DRBD und LINSTOR auf allen Hosts
  hosts: linstor_satellites
  become: true
  roles: [linstor]

- name: Controller-HA auf den Kandidaten
  hosts: linstor_controllers
  become: true
  roles: [linstor-controller-ha]
EOF
ansible-playbook --syntax-check site.yml

Expected:


playbook: site.yml

Two details decide how operations go. linstor-controller only gets enabled: false, no state: with state: stopped, every run would stop the active controller. Because it is part of the target drbd-services@linstor_db, the whole chain goes down with it: VIP gone, unmount, DRBD becomes Secondary, and the promoter restarts it on whichever node promotes first – 21 s without API in the lab, on every run. And the handler does reloaded, never restarted: when drbd-reactor restarts on the active node, stop-services-on-exit = true shuts down mount, VIP and controller.

Deliberately not in the roles:

Step Why not Day
Nodes, interfaces, paths, Aux/zone one-time, via the LINSTOR API 4
Pools, rg-k8s one-time, the StorageClass overrides rg-k8s 5, 8
linstor_db, DB move one-time, destructive when repeated 6
Starting the controller drbd-reactor’s job 6

Save your progress? Sign up for free

Drill 5: The dry run

# on k0s-ctrl
ansible-playbook site.yml --check --diff | tail -5

Expected (abridged, approx. 1 min):

PLAY RECAP *********************************************************************
k0s-ctrl  : ok=12   changed=0    unreachable=0    failed=0    skipped=2    rescued=0 ...
k0s-w1    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...
k0s-w2    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...
k0s-w3    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...

changed=0 on all four: the roles describe exactly the lab you built by hand. Skipped are the logind drop-in on k0s-ctrl (not a worker) and the package update on all hosts (the version matches). Had you worked differently somewhere, --diff would show the lines Ansible would change.

Save your progress? Sign up for free

Drill 6: Two real runs

# on k0s-ctrl
time ansible-playbook site.yml | tail -6
ansible-playbook site.yml | tail -6

Expected (abridged, twice):

PLAY RECAP *********************************************************************
k0s-ctrl  : ok=12   changed=0    unreachable=0    failed=0    skipped=2    rescued=0 ...
k0s-w1    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...
k0s-w2    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...
k0s-w3    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...

real    0m54.703s

changed=0 twice is the proof of idempotence: a run only changes what deviates from the desired state. No service was restarted, pg-client noticed nothing.

Tip: What is tested here is that the roles match the manual installation. A run against an empty host would additionally install packages and build the DRBD module (Day 3, approx. 3.5 min). Test that on a fresh VM before using the roles for real.

Save your progress? Sign up for free

Drill 7: An update with Ansible

A host with an old version: install LINSTOR 1.34.2 by hand on the witness; the debs have been in ~/debs since Day 3:

# on k0s-ctrl
cd ~/debs
sudo apt-get install -y --allow-downgrades --allow-change-held-packages \
  ./linstor-common_1.34.2-1_all.deb ./linstor-satellite_1.34.2-1_all.deb
cd ~/k0s-lab/tag16
sleep 15
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6 | grep ctrl
apt-mark showhold

Expected (abridged):

dpkg: warning: downgrading linstor-satellite from 1.35.3-1 to 1.34.2-1
...
 k0s-ctrl | OFFLINE(VERSION MISMATCH)
drbd-dkms
drbd-utils

Along the way, the hold marks of the two packages are gone. What does the dry run say?

# on k0s-ctrl
ansible-playbook site.yml --limit k0s-ctrl --check | grep -A1 'in einem Schritt'

Expected:

TASK [linstor : LINSTOR-Pakete in einem Schritt installieren] ******************
skipping: [k0s-ctrl]

Warning: --check does not run command tasks and therefore does not show the missing update as a change. A green dry run does not mean here that the versions match – check them with dpkg-query (Speed Drill 1).

The real run:

# on k0s-ctrl
ansible-playbook site.yml --limit k0s-ctrl | grep -E -A3 'in einem Schritt|RECAP'
sleep 15
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6 | grep ctrl

Expected (abridged):

TASK [linstor : LINSTOR-Pakete in einem Schritt installieren] ******************
changed: [k0s-ctrl]
...
PLAY RECAP *********************************************************************
k0s-ctrl  : ok=13   changed=2    unreachable=0    failed=0    skipped=1    rescued=0 ...
 k0s-ctrl | Online

Two changes: packages swapped, hold set again. For a real update, you change linstor_version and the three checksums in group_vars/all.yml and follow the order from Day 15: --limit first on the candidates without the active controller, then on the witness, then drbd-reactorctl evict on the active node and only after that --limit on it. A run over all hosts at once would also swap the active controller.

Save your progress? Sign up for free

Exercise: Speed Drills

  1. Show the satellite version on all four hosts with an ad-hoc command. Target time: 20 s
  2. Show the tasks of the playbook without running it. Target time: 15 s
  3. Show the zones of the workers according to the inventory. Target time: 15 s
  4. Do a dry run for k0s-w2 only. Target time: 30 s

Solution

# on k0s-ctrl
ansible linstor_satellites -m command -a 'dpkg-query -W linstor-satellite' -o
ansible-playbook site.yml --list-tasks | head -8
ansible k0s_workers -m debug -a var=zone -o
ansible-playbook site.yml --check --limit k0s-w2 | tail -2

Expected (abridged):

k0s-w3 | CHANGED | rc=0 | (stdout) linstor-satellite    1.35.3-1
k0s-ctrl | CHANGED | rc=0 | (stdout) linstor-satellite  1.35.3-1
...
  play #1 (linstor_satellites): DRBD und LINSTOR auf allen Hosts    TAGS: []
    tasks:
      linstor : LINBIT-PPA (wie Tag 3, deb822-Datei)    TAGS: []
...
k0s-w1 | SUCCESS => {    "changed": false,    "zone": "a"}
k0s-w2 | SUCCESS => {    "changed": false,    "zone": "b"}
k0s-w3 | SUCCESS => {    "changed": false,    "zone": "a"}
k0s-w2    : ok=20   changed=0    unreachable=0    failed=0    skipped=1    rescued=0 ...

CHANGED for the ad-hoc command only means that Ansible cannot know whether a command changed anything.

Save your progress? Sign up for free

Checkpoint

# on k0s-ctrl
ansible-playbook site.yml | tail -5

Expected: changed=0 and failed=0 on all four hosts.

# on k0s-ctrl
apt-mark showhold

Expected: drbd-dkms, drbd-utils, linstor-common, linstor-satellite.

# on k0s-ctrl
linstor --no-utf8 --no-color node list | cut -d'|' -f2,6

Expected: four nodes Online.

Challenge

A colleague deleted the drop-in nur-mit-db.conf on k0s-w3 and enabled linstor-controller with systemctl enable. Reproduce this, find both deviations with Ansible without changing anything, and then repair them.

Hint: For files, --diff shows the content; for services, only changed.

Solution

# on k0s-w3
sudo rm /etc/systemd/system/linstor-controller.service.d/nur-mit-db.conf
sudo systemctl daemon-reload
sudo systemctl enable linstor-controller
# on k0s-ctrl
ansible-playbook site.yml --limit k0s-w3 --check --diff \
  | grep -E -A6 'starten lassen|nur auf dem Mount'

Expected (abridged):

TASK [linstor-controller-ha : Controller nicht von systemd starten lassen] *****
changed: [k0s-w3]
...
TASK [linstor-controller-ha : Controller startet nur auf dem Mount (Tag 6)] ****
--- before
+++ after: /home/ubuntu/k0s-lab/tag16/roles/linstor-controller-ha/files/nur-mit-db.conf
@@ -0,0 +1,2 @@
+[Unit]
+ConditionPathIsMountPoint=/var/lib/linstor
# on k0s-ctrl
ansible-playbook site.yml --limit k0s-w3 | grep -B1 '^changed'
ssh k0s-w3 'systemctl is-enabled linstor-controller; systemctl is-active \
  linstor-controller'
ssh k0s-w3 'systemctl cat linstor-controller | grep Condition'

Expected (abridged):

TASK [linstor-controller-ha : Controller nicht von systemd starten lassen] *****
changed: [k0s-w3]
--
TASK [linstor-controller-ha : Controller startet nur auf dem Mount (Tag 6)] ****
changed: [k0s-w3]
disabled
inactive
ConditionPathIsMountPoint=/var/lib/linstor

The controller did not start in the process: enabled only takes effect on the next boot. That is exactly the moment when, without the drop-in, a second controller with an empty database would have started.

Save your progress? Sign up for free

Interview Note

  • How do you tell that a role matches the running system? --check --diff without changes, then two real runs with changed=0.
  • Why does the role not start the LINSTOR controller? The controller belongs to drbd-reactor. The role only sets enabled: false; state: stopped would stop the active controller and with it the whole chain until the promoter restarts it; a restart of drbd-reactor would do the same.
  • What do you not automate with a role? One-time steps on the controller: registration, pools, linstor_db. They belong in a runbook or in a separate playbook that is triggered deliberately.

Enterprise Note

  • Roles in Git with review, runs from a pipeline or AWX/Automation Controller with logging, not from a laptop.
  • Mirror the package source (aptly, Pulp) instead of fetching from the internet on every run; version and checksum per environment (group_vars/prod, group_vars/staging).
  • Rolling updates as a separate playbook with serial: 1, pre- and post-checks (--faulty empty, node Online) and evict before the active controller – not as a manual --limit sequence.
  • Test roles with Molecule against fresh VMs, including the first-installation path with the DKMS build.
  • With LINBIT SDS, ready-made Ansible roles and packages with support are available.