Skip to content
Hands-on IT IT Drills

Day 14: Operations

Goal: Reserve memory for the host itself, put a host into maintenance, back up OpenNebula’s database, and know how an upgrade works and what comes after it. · Duration: approx. 60 min · Prerequisite: Day 13

Briefing

  • The scheduler calculates with allocated values (Day 2). So that the front-end, database, and hypervisor themselves have breathing room, RESERVED_MEM and RESERVED_CPU on the host subtract a fixed share.
  • onehost disable takes a host out of scheduling: running VMs keep running, new ones no longer arrive. This is maintenance mode.
  • The most important thing OpenNebula owns is its database. Without it, nobody knows which VM belongs to whom. onedb backup backs it up.
  • An upgrade consists of new packages and onedb upgrade, which raises the database schema.

Drill 1: Reserve memory

The host has 3.8 GB, and OpenNebula including MariaDB takes a good 1.5 GB of it. These 1.5 GB should not be available to VMs in the first place:

mkdir -p ~/one-lab/tag14
cd ~/one-lab/tag14
echo 'RESERVED_MEM="1536M"' > reserve.txt
onehost update localhost --append reserve.txt
onehost show localhost | grep -E 'TOTAL \+/- RESERVED|RESERVED_MEM'

Expected (shortened):

  TOTAL +/- RESERVED  : 3.8G
RESERVED_MEM="1536M"

The attribute is there, but nothing has changed. Not even after a minute.

Tried it: RESERVED_MEM expects a number in kilobytes. OpenNebula accepts a value like 1536M without an error message and then ignores it.

echo 'RESERVED_MEM="1572864"' > reserve.txt
onehost update localhost --append reserve.txt
onehost list

Expected (values will differ):

  ID NAME            CLUSTER    TVM      ALLOCATED_CPU      ALLOCATED_MEM STAT
   0 localhost       default      3    150 / 200 (75%)  1.3G / 2.3G (53%) on

1572864 KB are 1.5 GB. The scheduler now sees 2.3 GB for VMs instead of 3.8 GB. A positive value subtracts; a negative one would add memory, that is, overcommit.

Save your progress? Sign up for free

Drill 2: Maintenance

onehost disable localhost
onehost list

Expected (shortened):

   0 localhost       default      3    150 / 200 (75%)  1.3G / 2.3G (53%) dsbl

The service’s three VMs keep running. A new VM:

onetemplate instantiate alpine --name wartung --user-inputs GRUSS=w

Half a minute later:

onevm show wartung | grep -E '^STATE|SCHED_MESSAGE'

Expected:

STATE               : PENDING
SCHED_MESSAGE="Thu Nov 12 20:38:49 2026: Cannot dispatch VM: No hosts enabled to run VMs"

Maintenance over:

onehost enable localhost

After the scheduler’s next run, wartung starts on its own. With multiple hosts, you move the running VMs away before maintenance: onehost flush distributes them to other hosts, with shared storage even via live migration.

onevm terminate --hard wartung

Save your progress? Sign up for free

Drill 3: Back up the database

onedb works as oneadmin and reads the credentials from OpenNebula’s configuration:

sudo -u oneadmin onedb version

Expected:

Shared: 7.4.0
Local:  7.4.0
Required shared version: 7.4.0
Required local version:  7.4.0

This is the version of the schema, not of the software (7.4.1). It only changes when a new version touches the data model.

sudo -u oneadmin bash -c 'cd /tmp && onedb backup /var/lib/one/one-backup.sql' | tail -2
sudo ls -la /var/lib/one/one-backup.sql

Expected (size and time will differ):

Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file
-rw-rw-r-- 1 oneadmin oneadmin 2612082 Nov 12 20:38 /var/lib/one/one-backup.sql

A plain SQL file. It contains all objects, but no disks. Day 7 backs up images and VMs; the database backs up the knowledge about them. For a complete rebuild, you also need /etc/one with the configuration and /var/lib/one/.one with the services’ keys.

Save your progress? Sign up for free

Drill 4: An upgrade, on paper

Your lab runs the latest version, so a real upgrade is not possible. The procedure for a minor upgrade such as 7.4 → 7.6 on a front-end like this one:

Step Command
1. Read the release notes docs.opennebula.io, “Upgrade” section
2. Stop services sudo systemctl stop opennebula opennebula-flow opennebula-gate opennebula-fireedge
3. Back up the database onedb backup as in Drill 3
4. Point the repository to the new version change the line in /etc/apt/sources.list.d/opennebula.list
5. Packages sudo apt update && sudo apt upgrade
6. Raise the schema sudo -u oneadmin onedb upgrade
7. Start services sudo systemctl start opennebula …
8. Update hosts onehost sync --force, distributes new drivers to all hosts

The VMs keep running during the upgrade; only management and the API are briefly gone.

Save your progress? Sign up for free

Drill 5: What you cannot try out

Two things need more than a 4 GB VM:

  • OneKE: The “OneKS Appliance” template from Day 1 builds a Kubernetes cluster from multiple VMs, with its own router, load balancer, and storage. 16 GB of RAM is the minimum for that.
  • High availability: Three front-ends with Raft consensus (onezone server-add). If the leader fails, another front-end takes over within seconds.

Save your progress? Sign up for free

Checkpoint

onehost show localhost | grep RESERVED_MEM

Expected: RESERVED_MEM="1572864".

sudo ls /var/lib/one/one-backup.sql

Expected: the file is listed.

onehost list

Expected: STAT on.

Challenge

What does ALLOCATED_MEM show if, after reserving, you plan the host fuller than it is? Try a VM with 1 GB while the three service VMs are running. Does it start?

Solution

onetemplate instantiate alpine --name gross --user-inputs GRUSS=g --memory 1024

Half a minute later:

onevm show gross | grep -E '^STATE|SCHED_MESSAGE'

Expected:

STATE               : ACTIVE

1.3 GB allocated plus 1 GB fits into 2.3 GB. A second such VM would stay in PENDING with Not enough MEMORY, even if the host actually still had room. The reservation protects the host but costs space for VMs. You find the right value by watching USED (REAL) over a few days.

onevm terminate --hard gross

Save your progress? Sign up for free

Enterprise Note

  • Monitoring: OpenNebula comes with a Prometheus integration with exporters for hosts, VMs, and datastores, plus ready-made Grafana dashboards. Alerts on ERROR states of hosts and VMs are the minimum.
  • Database in production: MariaDB or MySQL with replication, and onedb backup every night via cron, stored outside the front-end.
  • Test upgrades: Raise a copy of the database with onedb upgrade on a test front-end before it happens in production. A failed schema upgrade is the only case in which OpenNebula does not keep running on its own.