Day 5: Virtual Networks
Goal: Build an internal network without internet, start VMs with two network cards, assign fixed addresses, and block ports with security groups. · Duration: approx. 75 min · Prerequisite: Day 4
Briefing
- A Virtual Network (VNet) describes a network: which bridge on the host, which driver, and which Address Ranges (AR) VMs get their addresses from.
- OpenNebula assigns addresses itself, without DHCP. The IP is in the context (Day 4), and one-context configures it inside the VM. Every assigned address is a lease.
- The
fwdriver attaches VMs to a Linux bridge and filters with iptables. The rules for this are security groups: lists of allowed connections. By default, every network gets thedefaultgroup, and it allows everything.
Host one
├─ minionebr 192.168.150.1 ── NAT ── Internet VNet vnet (192.168.150.2–51)
│ ├─ VM a eth0
│ └─ VM c eth0
└─ intern0 (keine IP, kein NAT) VNet intern (10.10.10.10–29)
├─ VM a eth1
└─ VM b eth0 ← nur über a erreichbar, kein Internet
Drill 1: The existing network
onevnet show vnet | sed -n '/VIRTUAL NETWORK TEMPLATE/,/^$/p'Expected:
VIRTUAL NETWORK TEMPLATE
BRIDGE="minionebr"
BRIDGE_TYPE="linux"
DNS="192.168.150.1"
GATEWAY="192.168.150.1"
OUTER_VLAN_ID=""
PHYDEV=""
SECURITY_GROUPS="0"
VLAN_ID=""
VN_MAD="fw"
GATEWAY and DNS end up in the context of
every VM. SECURITY_GROUPS="0" is the default
group.
Save your progress? Sign up for free
Drill 2: An internal network
No gateway, no DNS, no IP on the host. VMs in it only see each other:
mkdir -p ~/one-lab/tag05
cd ~/one-lab/tag05
cat > intern.net <<'EOF'
NAME = "intern"
VN_MAD = "fw"
BRIDGE = "intern0"
AR = [ TYPE = "IP4", IP = "10.10.10.10", SIZE = "20" ]
EOF
onevnet create intern.net
onevnet listExpected:
ID: 1
ID USER GROUP NAME CLUSTERS BRIDGE STATE LEASES OUTD ERRO
1 oneadmin oneadmin intern 0 intern0 rdy 0 0 0
0 oneadmin oneadmin vnet 0 minioneb rdy 0 0 0
OpenNebula creates the bridge intern0 as soon as the
first VM needs it.
Two VMs: a in both networks, b only in the
internal one. --nic replaces the template’s networks; you
separate multiple networks with a comma:
onetemplate instantiate alpine --name a --user-inputs GRUSS=a --nic vnet,intern
onetemplate instantiate alpine --name b --user-inputs GRUSS=b --nic internAfter a minute:
onevm show a | grep -E 'ETH[01]_IP='
onevm show b | grep -E 'ETH[01]_IP='Expected (addresses will differ):
ETH0_IP="192.168.150.13",
ETH1_IP="10.10.10.12",
ETH0_IP="10.10.10.11",
b cannot be reached from the host; the host is not
attached to intern0:
ping -c1 -W2 10.10.10.11 | tail -2Expected:
1 packets transmitted, 0 received, 100% packet loss, time 0ms
The way leads through a. Alpine does not allow
forwarding connections via SSH (AllowTcpForwarding no), but
you may bring along the SSH agent. Then a logs in to
b with your key, without the key ever leaving
a:
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_ed25519
IP_A=$(onevm show a | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh -A root@$IP_A "ssh -o StrictHostKeyChecking=accept-new [email protected] \
'hostname; ip route; ping -c1 -W2 1.1.1.1 2>&1 | tail -1'"Expected (shortened):
b
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.11
ping: sendto: Network unreachable
b only knows its own network. Exactly right for a
database (Day 6).
Save your progress? Sign up for free
Drill 3: Fixed addresses
Some VMs always need the same IP. The NIC gets it passed along:
onetemplate instantiate alpine --name fest --user-inputs GRUSS=f --nic "vnet:IP=192.168.150.40"
onevm show fest | grep ETH0_IP=Expected (shortened):
ETH0_IP="192.168.150.40",
Addresses that OpenNebula should never assign, for example because a
foreign device sits there, you block with hold:
onevnet hold vnet 192.168.150.50
onevnet show vnet | sed -n '/^LEASES/,/^VIRTUAL ROUTERS/p' | grep -E 'OWNER|150.40|150.50'Expected:
AR OWNER MAC IP PORT_FORWARD IP6
0 V:16 02:00:c0:a8:96:28 192.168.150.40 - -
0 V:-1 02:00:c0:a8:96:32 192.168.150.50 - -
V:16 belongs to VM 16, V:-1 is on hold.
Anyone requesting the held address gets
Cannot get IP/MAC lease from virtual network 0.
onevnet release vnet 192.168.150.50 frees it again.
Save your progress? Sign up for free
Drill 4: Security groups
A group that only lets in SSH, port 8080, and ping, but allows everything outbound:
cat > web.sg <<'EOF'
NAME = "web"
RULE = [ PROTOCOL = "TCP", RULE_TYPE = "inbound", RANGE = "22,8080" ]
RULE = [ PROTOCOL = "ICMP", RULE_TYPE = "inbound" ]
RULE = [ PROTOCOL = "ALL", RULE_TYPE = "outbound" ]
EOF
onesecgroup create web.sgExpected:
ID: 100
Rules are allow lists, and the lists of all groups
on a NIC are added together. As long as default (everything
allowed) is among them, web is useless. So web
replaces the default group in the vnet
network:
echo 'SECURITY_GROUPS="100"' > /tmp/sg.txt
onevnet update vnet --append /tmp/sg.txt
onevnet show vnet | grep SECURITY_GROUPSExpected:
SECURITY_GROUPS="100"
This applies to NICs created from now on. A new VM c
with two small services on port 8080 and 9000:
onetemplate instantiate alpine --name c --user-inputs GRUSS=cAfter a minute:
IP_C=$(onevm show c | grep -o 'ETH0_IP="[0-9.]*' | cut -d'"' -f2)
ssh root@$IP_C \
"(nc -lk -p 9000 -e echo neun >/dev/null 2>&1 &); (nc -lk -p 8080 -e echo acht >/dev/null 2>&1 &)"Probe from a, another VM:
ssh root@$IP_A "for p in 22 8080 9000; do \
if nc -w2 $IP_C \$p </dev/null >/dev/null 2>&1; then echo \"\$p offen\"; else echo \"\$p zu\"; fi; done"Expected:
22 offen
8080 offen
9000 zu
Tried it: From the host itself, all three ports were open. The host is the network’s gateway, its packets do not go through the bridge, and the security group rules only apply there. So always test security groups from another VM or from a machine outside.
Save your progress? Sign up for free
Checkpoint
onevnet listExpected: intern and vnet.
onesecgroup list | grep webExpected: 100 with the name web.
diff ~/one-lab/tag05/web.sg ~/one-lab/code/loesung/tag05/web.sgExpected: no output.
Challenge
Clean up: all of today’s VMs should be gone, and the held address
free again. What happens if you try to delete a VM with
terminate --hard right after creating it?
Solution
onevnet release vnet 192.168.150.50
for v in a b c fest; do onevm terminate --hard $v; done
onevm listIf you delete a VM that is just starting, OpenNebula reports:
[one.vm.action] Error performing action "terminate-hard": Could not terminate VM 16, wrong state BOOT.
In PROLOG and BOOT, drivers are working on
the VM, and OpenNebula does not interrupt them. Wait a few seconds, then
it works. In the end:
ID USER GROUP NAME STAT CPU MEM HOST TIME
The networks and the security group stay; Day 6 needs them.
Save your progress? Sign up for free
Enterprise Note
- VLANs and VXLAN: In production, networks are not
separated with local bridges but with
VN_MAD="802.1Q"(VLAN tags on a physical card) orvxlan(tunnels between hosts). Then one internal network spans all hosts. - Virtual Router: For networks without an exit,
OpenNebula offers a router appliance from the Marketplace with NAT,
DHCP, and a load balancer. With it,
bgets to the internet in a controlled way, without the host acting as a router. - Maintain security groups: The
defaultgroup allows everything. In a production cloud, you replace it in every network with stricter groups, as in Drill 4. If a group changes,onesecgroup commitpushes the rules to running VMs.