Skip to content
Hands-on IT IT Drills

Day 9: OneFlow

Goal: Describe notes-api and Postgres as one service with two roles, start them in the right order, scale the API, and rescue a service that gets stuck while scaling. · Duration: approx. 75 min · Prerequisite: Day 8

Briefing

  • OneFlow groups multiple VMs into a service. A service template (JSON) describes roles: which VM template, how many VMs (cardinality), who starts before whom (parents).
  • "deployment": "straight" means: a role only starts once all its parents are running. Without this setting, all roles start at the same time, as on Day 6.
  • A service has its own states: DEPLOYING, RUNNING, SCALING, COOLDOWN (pause after scaling), FAILED_….
 Service notes-service (straight)
   Rolle db   (notes-db, 1 VM, fest: min 1 / max 1)
     └─► Rolle api (notes, 1..n VMs, parents: db)

The individual VMs from Day 6 have to go, because the data disk notes-daten can only be used by one VM at a time:

onevm terminate notes
onevm terminate db

Drill 1: The service template

Roles point to VM templates by ID. Your IDs will differ, so you insert them with awk:

mkdir -p ~/one-lab/tag09
cd ~/one-lab/tag09
cat > notes-service.json <<'EOF'
{
  "name": "notes-service",
  "deployment": "straight",
  "roles": [
    {
      "name": "db",
      "type": "vm",
      "template_id": DB_ID,
      "cardinality": 1,
      "user_inputs_values": {
        "DB_PASSWORD": "lab-db-pass-123"
      }
    },
    {
      "name": "api",
      "type": "vm",
      "template_id": API_ID,
      "cardinality": 1,
      "parents": [ "db" ],
      "user_inputs_values": {
        "DB_PASSWORD": "lab-db-pass-123",
        "API_KEY": "lab-api-key-123",
        "NOTES_VERSION": "0.7.1"
      }
    }
  ]
}
EOF
sed -i "s/DB_ID/$(onetemplate show notes-db | awk '/^ID/ {print $3}')/" notes-service.json
sed -i "s/API_ID/$(onetemplate show notes | awk '/^ID/ {print $3}')/" notes-service.json
grep template_id notes-service.json

Expected (your IDs):

      "template_id": 4,
      "template_id": 5,

user_inputs_values fills in the USER_INPUTS of the VM templates from Day 6. Without them, OneFlow would not start, because the passwords are mandatory.

oneflow-template create notes-service.json

Expected:

ID: 1

Save your progress? Sign up for free

Drill 2: Start

oneflow-template instantiate notes-service

Expected:

ID: 2

Watch, every 15 seconds:

oneflow list
onevm list

Expected: at first, only db_0_(service_2) is created. The API VM only comes once the database is running. After a little over a minute:

  ID USER     GROUP    NAME                  STARTTIME STAT
   2 oneadmin oneadmin notes-service    11/12 19:55:10 RUNNING
  ID USER     GROUP    NAME            STAT  CPU     MEM HOST             TIME
  27 oneadmin oneadmin api_0_(service_ runn  0.5    384M localhost    0d 00h01
  26 oneadmin oneadmin db_0_(service_2 runn  0.5    512M localhost    0d 00h02
oneflow show notes-service | sed -n '/ROLE api/,$p'

Expected (shortened):

ROLE api
ROLE STATE          : RUNNING
PARENTS             : db
CARDINALITY         : 1
...
LOG MESSAGES
11/12/26 19:55 [I] New state: DEPLOYING_NETS
11/12/26 19:55 [I] New state: DEPLOYING
11/12/26 19:56 [I] New state: RUNNING

And yesterday’s note? The data disk is the same:

V=$(onevm list --no-header | grep api_0 | awk '{print $1}')
IP=$(onevm show $V | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
curl -s $IP:8080/notes; echo

Expected:

[{"id":"602722bb-05ab-4559-ac6e-47609ba9ab2b","text":"Hallo OpenNebula"}]

Save your progress? Sign up for free

Drill 3: Scale

Two API VMs instead of one:

oneflow scale notes-service api 2

After two minutes:

oneflow show notes-service | sed -n '/ROLE api/,/LOG MESSAGES/p'

Expected (shortened):

ROLE api
ROLE STATE          : COOLDOWN
PARENTS             : db
CARDINALITY         : 2
...
    27 api_0_(service_2)        oneadmin        oneadmin
    28 api_1_(service_2)        oneadmin        oneadmin

The second API VM talks to the same database and returns the same note. Then comes COOLDOWN: a five-minute pause during which the service does not scale further. This protects against flapping when an automatic rule (Day 11) reacts too quickly.

Save your progress? Sign up for free

Drill 4: The role that must not scale

After the cooldown, you try the same with the database:

oneflow scale notes-service db 2

Shortly afterwards:

oneflow list
oneflow show notes-service | sed -n '/LOG MESSAGES/,$p' | tail -3

Expected (shortened):

   2 oneadmin oneadmin notes-service    11/12 19:55:10 FAILED_SCALING
11/12/26 20:03 [I] Role db scaling up from 1 to 2 nodes
11/12/26 20:03 [E] Role db : Instantiate failed for template 4; [one.template.instantiate] Error allocating a new virtual machine template. Cannot get IP/MAC lease from virtual network 1.
11/12/26 20:03 [I] New state: FAILED_SCALING

The second database VM does not get the fixed IP 10.10.10.20; the first one has it. Had the IP been available, the persistent disk would have failed at the latest. And now?

oneflow scale notes-service db 1
oneflow update notes-service

Both end with Service cannot be … in state: FAILED_SCALING. oneflow recover notes-service only tries once more, with the same error. The way out in the lab: delete the service. The data lives on the persistent disk and survives that.

oneflow recover --delete notes-service

Half a minute later:

oneflow list
onevm list

Expected (shortened):

  ID USER     GROUP    NAME                  STARTTIME STAT
  ID USER     GROUP    NAME            STAT  CPU     MEM HOST             TIME
  26 oneadmin oneadmin db_0_(service_2 runn  0.5    512M localhost    0d 00h12

Tried it: The service is gone, the API VMs too, but the database VM keeps running, as an orphan without a service. It holds on to the data disk and the IP. After a recover --delete, always check onevm list and clean up leftovers by hand.

onevm terminate 26

The ID is yours from onevm list.

Save your progress? Sign up for free

Drill 5: Prevention

A stateful role should not be able to scale at all. min_vms and max_vms set the limits:

jq '(.roles[] | select(.name=="db")) += {"min_vms": 1, "max_vms": 1}' notes-service.json \
  > neu.json && mv neu.json notes-service.json
oneflow-template update notes-service notes-service.json
oneflow-template instantiate notes-service

Expected:

ID: 3

When the service reports RUNNING:

oneflow scale notes-service db 2

Expected:

Can't scale service 3: Maximum cardinality is 1

Rejected before anything happens.

Save your progress? Sign up for free

Checkpoint

oneflow list

Expected: notes-service with RUNNING.

onevm list

Expected: exactly db_0_(service_3) and api_0_(service_3), no orphans.

diff ~/one-lab/tag09/notes-service.json ~/one-lab/code/loesung/tag09/notes-service.json

Expected: no output if your template IDs are 4 and 5. Otherwise only these two lines differ.

Challenge

The API role should get limits too: at least 1, at most 3 VMs. Change the running service without recreating it, and check that scale … api 4 is rejected.

Note: oneflow update takes a file with the entire service document. The current document is in oneflow show … --json.

Solution

oneflow show notes-service --json | jq '.DOCUMENT.TEMPLATE.BODY' > body.json
jq '(.roles[] | select(.name=="api")) += {"min_vms": 1, "max_vms": 3}' body.json > body2.json
oneflow update notes-service body2.json
oneflow scale notes-service api 4

Expected (shortened):

Can't scale service 3: Maximum cardinality is 3

The service template remains untouched by this. If you want the limit for future services too, it also belongs in notes-service.json.

Save your progress? Sign up for free

Enterprise Note

  • Elasticity: Roles can have rules, such as “one more VM when the average CPU load is above 80 percent” (elasticity_policies). The VMs report the values via OneGate (Day 11).
  • Readiness: With "ready_status_gate": true, straight does not just wait for RUNNING but until the VM reports READY=YES via OneGate. Then the API really only starts once Postgres accepts connections.
  • State does not belong in scaling roles: Databases belong in their own roles with max_vms: 1 or entirely outside the service. Only what has no state gets scaled.