Skip to content
Hands-on IT IT Drills

Day 2: Architecture and CLI

Goal: Know the parts of OpenNebula, read their logs, and understand how the scheduler finds a host for a VM, and why it sometimes finds none. · Duration: approx. 60 min · Prerequisite: Day 1

Briefing

  • oned is the core. It holds the state of all objects in the database and offers an XML-RPC API on port 2633. CLI, Sunstone, Terraform, everything talks to this API (Day 13).
  • Drivers do the actual work: the VMM driver starts VMs via libvirt, the Transfer Manager (TM) copies disks, the IM driver monitors hosts. These are scripts under /var/lib/one/remotes.
  • The scheduler distributes waiting VMs to hosts. It calculates with what the templates request (allocated), not with what is actually consumed (real).
 CLI / Sunstone / Terraform ── XML-RPC :2633 ──► oned ── MariaDB
                                                   │
                     ┌──────────────┬──────────────┼──────────────┐
                 Scheduler      VMM (kvm)      TM (qcow2)      IM (kvm)
                 wählt Host     libvirt        kopiert Disks   überwacht Host
Object CLI In short
Host onehost a hypervisor
Datastore onedatastore storage location for images or running VMs
Image oneimage a disk
Template onetemplate blueprint of a VM
VM onevm a running or stopped instance
Virtual Network onevnet a network with addresses
User, Group, ACL oneuser, onegroup, oneacl permissions (Day 8)

Drill 1: Services and version

systemctl list-units 'opennebula*' --no-legend | cut -c1-80

Expected (shortened):

  opennebula-fireedge.service        loaded active running OpenNebula FireEdge Ser
  opennebula-flow.service            loaded active running OpenNebula Flow Service
  opennebula-gate.service            loaded active running OpenNebula Gate Service
  opennebula-hem.service             loaded active running OpenNebula Hook Executi
...
  opennebula.service                 loaded active running OpenNebula Cloud Contro
Service Job
opennebula oned with scheduler
opennebula-fireedge Sunstone, the web UI
opennebula-flow OneFlow, services made of multiple VMs (Day 9)
opennebula-gate OneGate, back channel from the VMs (Day 11)
opennebula-hem Hook Execution Manager, runs hooks (Day 11)
cat /var/lib/one/remotes/VERSION
onezone list

Expected:

7.4.1
C   ID NAME            ENDPOINT                                      FED_INDEX  STAT
*    0 OpenNebula      http://localhost:2633/RPC2                    -1         on

The zone is one OpenNebula installation. Multiple zones can be joined into a federation. The endpoint shows the XML-RPC API.

Save your progress? Sign up for free

Drill 2: Where the logs are

sudo ls /var/log/one/

Expected (shortened):

0.log 1.log 2.log 3.log fireedge.error fireedge.log monitor.log oned.log oneflow.log
onegate.log onehem.log rank_sched.log …
  • oned.log: everything the core does.
  • <ID>.log: one file per VM, with every driver step (Day 1, Challenge).
  • rank_sched.log: the scheduler’s decisions.
  • monitor.log: host monitoring.

Save your progress? Sign up for free

Drill 3: Allocated and real

onehost show localhost | sed -n '/HOST SHARES/,/USED (ALLOCATED)/p'

Expected (values will differ):

HOST SHARES
RUNNING VMS           : 0
MEMORY
  TOTAL               : 3.8G
  TOTAL +/- RESERVED  : 3.8G
  USED (REAL)         : 1.6G
  USED (ALLOCATED)    : 0K

Below it follows the same block for the CPU.

  • REAL measures what the host actually consumes, including OpenNebula itself and MariaDB.
  • ALLOCATED only counts what the VMs get according to their templates. The scheduler decides based on this.

So to the scheduler, a host with 3.8 GB and zero allocated VMs looks empty, even though 1.6 GB is already in use. RESERVED (Day 14) brings order here.

Save your progress? Sign up for free

Drill 4: A VM that does not fit

instantiate can override individual values of the template. Ask for 4 GB:

onetemplate instantiate 1 --name gross --memory 4096

Expected (your ID will differ):

VM ID: 4

Half a minute later:

onevm list
onevm show gross | grep -E '^STATE|SCHED_MESSAGE'

Expected (shortened):

  ID USER     GROUP    NAME            STAT  CPU     MEM HOST             TIME
   4 oneadmin oneadmin gross           pend    1      4G              0d 00h00
STATE               : PENDING
SCHED_MESSAGE="Thu Nov 12 19:04:01 2026: [HOST] Not enough MEMORY on hosts: 0"

The VM waits in PENDING, and the scheduler writes the reason into the VM itself. The host has 3.8 GB, the VM wants 4 GB. It waits until a host becomes free, even for days.

Save your progress? Sign up for free

Drill 5: Output for scripts

The CLI tables are for humans. For scripts there are two ways: selected columns or JSON.

onevm list --list ID,NAME,STAT,MEM

Expected:

  ID NAME            STAT     MEM
   4 gross           pend      4G
onevm show gross --json | jq -r '.VM.USER_TEMPLATE.SCHED_MESSAGE'

Expected:

Thu Nov 12 19:04:01 2026: [HOST] Not enough MEMORY on hosts: 0

miniONE installed jq along with everything else. With -x instead of --json, you get the same data as XML, the way the API delivers it.

Save your progress? Sign up for free

Checkpoint

onevm show gross | grep '^STATE'

Expected: STATE : PENDING.

sudo grep -c 'Deploying VM' /var/log/one/oned.log

Expected: a number greater than 0, one line per started VM.

Challenge

Get gross running without deleting and recreating it.

Note: onevm has a command for changing sizes. Look in onevm --help.

Solution

onevm resize gross --memory 512

Half a minute later:

onevm list

Expected:

  ID USER     GROUP    NAME            STAT  CPU     MEM HOST             TIME
   4 oneadmin oneadmin gross           runn    1    512M localhost    0d 00h01

resize works on waiting and powered-off VMs. On running VMs only if the template allows hotplug (Day 7). The oned log shows when the scheduler deployed the VM:

sudo grep 'VM 4' /var/log/one/oned.log | tail -2

Expected (shortened):

… [Z0][DiM][D]: Deploying VM 4

Clean up:

onevm terminate --hard gross

--hard switches the VM off immediately without giving the operating system time to shut down. For a throwaway VM, that is fine.

Save your progress? Sign up for free

Enterprise Note

  • Overcommitting: In templates, CPU (share) and VCPU (visible cores) let you overcommit CPU deliberately, for example CPU=0.25 with VCPU=2. Better not to overcommit RAM, otherwise the host’s OOM killer steps in.
  • Scheduler rules: SCHED_REQUIREMENTS and SCHED_RANK in the template control where a VM may go and which host is preferred. This lets you separate production and test onto different hosts, for example.
  • Central logs: In production, oned.log and the VM logs go to a central system. The VM logs contain every driver step and are the first source during incidents.