Skip to content
Hands-on IT IT Drills

Day 5: Virtual Networks

Goal: Build an internal network without internet, start VMs with two network cards, assign fixed addresses, and block ports with security groups. · Duration: approx. 75 min · Prerequisite: Day 4

Briefing

  • A Virtual Network (VNet) describes a network: which bridge on the host, which driver, and which Address Ranges (AR) VMs get their addresses from.
  • OpenNebula assigns addresses itself, without DHCP. The IP is in the context (Day 4), and one-context configures it inside the VM. Every assigned address is a lease.
  • The fw driver attaches VMs to a Linux bridge and filters with iptables. The rules for this are security groups: lists of allowed connections. By default, every network gets the default group, and it allows everything.
 Host one
 ├─ minionebr 192.168.150.1 ── NAT ── Internet      VNet vnet   (192.168.150.2–51)
 │     ├─ VM a eth0
 │     └─ VM c eth0
 └─ intern0   (keine IP, kein NAT)                  VNet intern (10.10.10.10–29)
       ├─ VM a eth1
       └─ VM b eth0   ← nur über a erreichbar, kein Internet

Drill 1: The existing network

onevnet show vnet | sed -n '/VIRTUAL NETWORK TEMPLATE/,/^$/p'

Expected:

VIRTUAL NETWORK TEMPLATE
BRIDGE="minionebr"
BRIDGE_TYPE="linux"
DNS="192.168.150.1"
GATEWAY="192.168.150.1"
OUTER_VLAN_ID=""
PHYDEV=""
SECURITY_GROUPS="0"
VLAN_ID=""
VN_MAD="fw"

GATEWAY and DNS end up in the context of every VM. SECURITY_GROUPS="0" is the default group.

Save your progress? Sign up for free

Drill 2: An internal network

No gateway, no DNS, no IP on the host. VMs in it only see each other:

mkdir -p ~/one-lab/tag05
cd ~/one-lab/tag05
cat > intern.net <<'EOF'
NAME = "intern"
VN_MAD = "fw"
BRIDGE = "intern0"
AR = [ TYPE = "IP4", IP = "10.10.10.10", SIZE = "20" ]
EOF
onevnet create intern.net
onevnet list

Expected:

ID: 1
  ID USER     GROUP    NAME            CLUSTERS   BRIDGE   STATE  LEASES OUTD ERRO
   1 oneadmin oneadmin intern          0          intern0  rdy         0    0    0
   0 oneadmin oneadmin vnet            0          minioneb rdy         0    0    0

OpenNebula creates the bridge intern0 as soon as the first VM needs it.

Two VMs: a in both networks, b only in the internal one. --nic replaces the template’s networks; you separate multiple networks with a comma:

onetemplate instantiate alpine --name a --user-inputs GRUSS=a --nic vnet,intern
onetemplate instantiate alpine --name b --user-inputs GRUSS=b --nic intern

After a minute:

onevm show a | grep -E 'ETH[01]_IP='
onevm show b | grep -E 'ETH[01]_IP='

Expected (addresses will differ):

  ETH0_IP="192.168.150.13",
  ETH1_IP="10.10.10.12",
  ETH0_IP="10.10.10.11",

b cannot be reached from the host; the host is not attached to intern0:

ping -c1 -W2 10.10.10.11 | tail -2

Expected:

1 packets transmitted, 0 received, 100% packet loss, time 0ms

The way leads through a. Alpine does not allow forwarding connections via SSH (AllowTcpForwarding no), but you may bring along the SSH agent. Then a logs in to b with your key, without the key ever leaving a:

eval $(ssh-agent -s)
ssh-add ~/.ssh/id_ed25519
IP_A=$(onevm show a | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh -A root@$IP_A "ssh -o StrictHostKeyChecking=accept-new [email protected] \
  'hostname; ip route; ping -c1 -W2 1.1.1.1 2>&1 | tail -1'"

Expected (shortened):

b
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.11
ping: sendto: Network unreachable

b only knows its own network. Exactly right for a database (Day 6).

Save your progress? Sign up for free

Drill 3: Fixed addresses

Some VMs always need the same IP. The NIC gets it passed along:

onetemplate instantiate alpine --name fest --user-inputs GRUSS=f --nic "vnet:IP=192.168.150.40"
onevm show fest | grep ETH0_IP=

Expected (shortened):

  ETH0_IP="192.168.150.40",

Addresses that OpenNebula should never assign, for example because a foreign device sits there, you block with hold:

onevnet hold vnet 192.168.150.50
onevnet show vnet | sed -n '/^LEASES/,/^VIRTUAL ROUTERS/p' | grep -E 'OWNER|150.40|150.50'

Expected:

AR  OWNER                    MAC             IP PORT_FORWARD   IP6
0   V:16       02:00:c0:a8:96:28 192.168.150.40     -     -
0   V:-1       02:00:c0:a8:96:32 192.168.150.50     -     -

V:16 belongs to VM 16, V:-1 is on hold. Anyone requesting the held address gets Cannot get IP/MAC lease from virtual network 0. onevnet release vnet 192.168.150.50 frees it again.

Save your progress? Sign up for free

Drill 4: Security groups

A group that only lets in SSH, port 8080, and ping, but allows everything outbound:

cat > web.sg <<'EOF'
NAME = "web"
RULE = [ PROTOCOL = "TCP", RULE_TYPE = "inbound", RANGE = "22,8080" ]
RULE = [ PROTOCOL = "ICMP", RULE_TYPE = "inbound" ]
RULE = [ PROTOCOL = "ALL", RULE_TYPE = "outbound" ]
EOF
onesecgroup create web.sg

Expected:

ID: 100

Rules are allow lists, and the lists of all groups on a NIC are added together. As long as default (everything allowed) is among them, web is useless. So web replaces the default group in the vnet network:

echo 'SECURITY_GROUPS="100"' > /tmp/sg.txt
onevnet update vnet --append /tmp/sg.txt
onevnet show vnet | grep SECURITY_GROUPS

Expected:

SECURITY_GROUPS="100"

This applies to NICs created from now on. A new VM c with two small services on port 8080 and 9000:

onetemplate instantiate alpine --name c --user-inputs GRUSS=c

After a minute:

IP_C=$(onevm show c | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh root@$IP_C \
  "(nc -lk -p 9000 -e echo neun >/dev/null 2>&1 &); (nc -lk -p 8080 -e echo acht >/dev/null 2>&1 &)"

Probe from a, another VM:

ssh root@$IP_A "for p in 22 8080 9000; do \
  if nc -w2 $IP_C \$p </dev/null >/dev/null 2>&1; then echo \"\$p offen\"; else echo \"\$p zu\"; fi; done"

Expected:

22 offen
8080 offen
9000 zu

Tried it: From the host itself, all three ports were open. The host is the network’s gateway, its packets do not go through the bridge, and the security group rules only apply there. So always test security groups from another VM or from a machine outside.

Save your progress? Sign up for free

Checkpoint

onevnet list

Expected: intern and vnet.

onesecgroup list | grep web

Expected: 100 with the name web.

diff ~/one-lab/tag05/web.sg ~/one-lab/code/loesung/tag05/web.sg

Expected: no output.

Challenge

Clean up: all of today’s VMs should be gone, and the held address free again. What happens if you try to delete a VM with terminate --hard right after creating it?

Solution

onevnet release vnet 192.168.150.50
for v in a b c fest; do onevm terminate --hard $v; done
onevm list

If you delete a VM that is just starting, OpenNebula reports:

[one.vm.action] Error performing action "terminate-hard": Could not terminate VM 16, wrong state BOOT.

In PROLOG and BOOT, drivers are working on the VM, and OpenNebula does not interrupt them. Wait a few seconds, then it works. In the end:

  ID USER     GROUP    NAME            STAT  CPU     MEM HOST             TIME

The networks and the security group stay; Day 6 needs them.

Save your progress? Sign up for free

Enterprise Note

  • VLANs and VXLAN: In production, networks are not separated with local bridges but with VN_MAD="802.1Q" (VLAN tags on a physical card) or vxlan (tunnels between hosts). Then one internal network spans all hosts.
  • Virtual Router: For networks without an exit, OpenNebula offers a router appliance from the Marketplace with NAT, DHCP, and a load balancer. With it, b gets to the internet in a controlled way, without the host acting as a router.
  • Maintain security groups: The default group allows everything. In a production cloud, you replace it in every network with stricter groups, as in Drill 4. If a group changes, onesecgroup commit pushes the rules to running VMs.