Day 14: Operations
Goal: Reserve memory for the host itself, put a host into maintenance, back up OpenNebula’s database, and know how an upgrade works and what comes after it. · Duration: approx. 60 min · Prerequisite: Day 13
Briefing
- The scheduler calculates with allocated values (Day 2). So that the
front-end, database, and hypervisor themselves have breathing room,
RESERVED_MEMandRESERVED_CPUon the host subtract a fixed share. onehost disabletakes a host out of scheduling: running VMs keep running, new ones no longer arrive. This is maintenance mode.- The most important thing OpenNebula owns is its
database. Without it, nobody knows which VM belongs to
whom.
onedb backupbacks it up. - An upgrade consists of new packages and
onedb upgrade, which raises the database schema.
Drill 1: Reserve memory
The host has 3.8 GB, and OpenNebula including MariaDB takes a good 1.5 GB of it. These 1.5 GB should not be available to VMs in the first place:
mkdir -p ~/one-lab/tag14
cd ~/one-lab/tag14
echo 'RESERVED_MEM="1536M"' > reserve.txt
onehost update localhost --append reserve.txt
onehost show localhost | grep -E 'TOTAL \+/- RESERVED|RESERVED_MEM'Expected (shortened):
TOTAL +/- RESERVED : 3.8G
RESERVED_MEM="1536M"
The attribute is there, but nothing has changed. Not even after a minute.
Tried it:
RESERVED_MEMexpects a number in kilobytes. OpenNebula accepts a value like1536Mwithout an error message and then ignores it.
echo 'RESERVED_MEM="1572864"' > reserve.txt
onehost update localhost --append reserve.txt
onehost listExpected (values will differ):
ID NAME CLUSTER TVM ALLOCATED_CPU ALLOCATED_MEM STAT
0 localhost default 3 150 / 200 (75%) 1.3G / 2.3G (53%) on
1572864 KB are 1.5 GB. The scheduler now sees 2.3 GB for VMs instead of 3.8 GB. A positive value subtracts; a negative one would add memory, that is, overcommit.
Save your progress? Sign up for free
Drill 2: Maintenance
onehost disable localhost
onehost listExpected (shortened):
0 localhost default 3 150 / 200 (75%) 1.3G / 2.3G (53%) dsbl
The service’s three VMs keep running. A new VM:
onetemplate instantiate alpine --name wartung --user-inputs GRUSS=wHalf a minute later:
onevm show wartung | grep -E '^STATE|SCHED_MESSAGE'Expected:
STATE : PENDING
SCHED_MESSAGE="Thu Nov 12 20:38:49 2026: Cannot dispatch VM: No hosts enabled to run VMs"
Maintenance over:
onehost enable localhostAfter the scheduler’s next run, wartung starts on its
own. With multiple hosts, you move the running VMs away before
maintenance: onehost flush distributes them to other hosts,
with shared storage even via live migration.
onevm terminate --hard wartungSave your progress? Sign up for free
Drill 3: Back up the database
onedb works as oneadmin and reads the
credentials from OpenNebula’s configuration:
sudo -u oneadmin onedb versionExpected:
Shared: 7.4.0
Local: 7.4.0
Required shared version: 7.4.0
Required local version: 7.4.0
This is the version of the schema, not of the software (7.4.1). It only changes when a new version touches the data model.
sudo -u oneadmin bash -c 'cd /tmp && onedb backup /var/lib/one/one-backup.sql' | tail -2
sudo ls -la /var/lib/one/one-backup.sqlExpected (size and time will differ):
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file
-rw-rw-r-- 1 oneadmin oneadmin 2612082 Nov 12 20:38 /var/lib/one/one-backup.sql
A plain SQL file. It contains all objects, but no disks. Day 7 backs
up images and VMs; the database backs up the knowledge about
them. For a complete rebuild, you also need /etc/one with
the configuration and /var/lib/one/.one with the services’
keys.
Save your progress? Sign up for free
Drill 4: An upgrade, on paper
Your lab runs the latest version, so a real upgrade is not possible. The procedure for a minor upgrade such as 7.4 → 7.6 on a front-end like this one:
| Step | Command |
|---|---|
| 1. Read the release notes | docs.opennebula.io, “Upgrade” section |
| 2. Stop services | sudo systemctl stop opennebula opennebula-flow opennebula-gate opennebula-fireedge |
| 3. Back up the database | onedb backup as in Drill 3 |
| 4. Point the repository to the new version | change the line in
/etc/apt/sources.list.d/opennebula.list |
| 5. Packages | sudo apt update && sudo apt upgrade |
| 6. Raise the schema | sudo -u oneadmin onedb upgrade |
| 7. Start services | sudo systemctl start opennebula … |
| 8. Update hosts | onehost sync --force, distributes new drivers to all
hosts |
The VMs keep running during the upgrade; only management and the API are briefly gone.
Save your progress? Sign up for free
Drill 5: What you cannot try out
Two things need more than a 4 GB VM:
- OneKE: The “OneKS Appliance” template from Day 1 builds a Kubernetes cluster from multiple VMs, with its own router, load balancer, and storage. 16 GB of RAM is the minimum for that.
- High availability: Three front-ends with Raft
consensus (
onezone server-add). If the leader fails, another front-end takes over within seconds.
Save your progress? Sign up for free
Checkpoint
onehost show localhost | grep RESERVED_MEMExpected: RESERVED_MEM="1572864".
sudo ls /var/lib/one/one-backup.sqlExpected: the file is listed.
onehost listExpected: STAT on.
Challenge
What does ALLOCATED_MEM show if, after
reserving, you plan the host fuller than it is? Try a VM with 1 GB while
the three service VMs are running. Does it start?
Solution
onetemplate instantiate alpine --name gross --user-inputs GRUSS=g --memory 1024Half a minute later:
onevm show gross | grep -E '^STATE|SCHED_MESSAGE'Expected:
STATE : ACTIVE
1.3 GB allocated plus 1 GB fits into 2.3 GB. A second such VM would
stay in PENDING with Not enough MEMORY, even
if the host actually still had room. The reservation protects the host
but costs space for VMs. You find the right value by watching
USED (REAL) over a few days.
onevm terminate --hard grossSave your progress? Sign up for free
Enterprise Note
- Monitoring: OpenNebula comes with a Prometheus
integration with exporters for hosts, VMs, and datastores, plus
ready-made Grafana dashboards. Alerts on
ERRORstates of hosts and VMs are the minimum. - Database in production: MariaDB or MySQL with
replication, and
onedb backupevery night via cron, stored outside the front-end. - Test upgrades: Raise a copy of the database with
onedb upgradeon a test front-end before it happens in production. A failed schema upgrade is the only case in which OpenNebula does not keep running on its own.