Xen and the Art of Consolidation
Tom
Eastep
2006-03-14
2006
Thomas M. Eastep
Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
GNU Free Documentation
License
.
Before Xen
Prior to adopting Xen, I had a home
office crowded with 5 systems, three monitors a scanner and a printer. The
systems were:
Firewall
Public Server in a DMZ (mail)
Private Server (wookie)
My personal Linux Desktop (ursa)
My work system (docked laptop running Windows XP).
The result was a very crowded and noisy room.
After Xen
Xen has allowed me to reduce the noise and clutter considerably. I
now have three systems with two monitors. I've also replaced the
individual printer and scanner with a Multifunction
FAX/Scanner/Printer.
The systems now include:
Combination Firewall/Public Server/Private Server using Xen
(created by building out my Linux desktop system).
My work system.
My Linux desktop (wookie, which is actually the old public
server box)
All of the Linux systems run SuSE 10.0.
If you are unfamiliar with Xen networking, I recommend that you read
the first section of the companion Xen and
Shorewall article.
Here is a high-level diagram of our network.
![]()
As shown in this diagram, the Xen system has three physical network
interfaces. These are:
eth0 -- conntected to
the switch in my office. That switch is cabled to a second switch in
my wife's office where there is my wife's desktop and her networked
printer.
eth1 -- connected to our
DSL "Modem".
eth2 -- connected to a
Wireless Access Point (WAP) that interfaces to our wireless
network.
There are three Xen domains. Dom0 (ursa) is used as a file server.
One DomU (which is usually Domain 1) is used as a firewall and the other
(lists, normally Domain 2) is used as a public Web/FTP/Mail/DNS server.
Because Xen 3 only supports three virtual interfaces per DomU, I also use
ursa as a gateway for our wireless network rather than placing that
function in the firewall DomU (that domain already has three interfaces).
Shorewall runs in both Dom0 and in the firewall domain.
The system has 1.5GB of RAM so I allocate 512MB to each server and
448MB to the firewall (the remaining 64MB is used by Xen).
Below are the relevant configuration files for the three domains.
The "loopback.nloopbacks=..." entries are used to restrict the number of
"vethn" devices that the Xen kernel creates. I use
partitions on my hard drives for DomU storage devices.
/boot/grub/menu.lst — here is the entry that
boots Xen in Dom0.
title XEN
root (hd0,1)
kernel /boot/xen.gz dom0_mem=524288 sched=bvt
module /boot/vmlinuz-xen root=/dev/hda2 vga=0x31a selinux=0 resume=/dev/hda1 splash=silent showopts loopback.nloopbacks=1
module /boot/initrd-xen
/etc/xen/auto/01-gateway — configuration file
for the firewall domain
# -*- mode: python; -*-
# configuration name:
name = "gateway"
# usable ram:
memory = 448
# kernel and initrd:
kernel = "/boot/vmlinuz-xen"
ramdisk = "/boot/initrd-xen"
# boot device:
root = "/dev/hdb2"
# boot to run level:
extra = "loopback.nloopbacks=0 3"
# network interface:
vif = [ 'mac=aa:cc:00:00:00:02, bridge=xenbr0', 'mac=aa:cc:00:00:00:03, bridge=xenbr1' , 'mac=00:a0:cc:63:66:89, bridge=xenbr2' ]
hostname = name
# storage devices:
disk = [ 'phy:hdb2,hdb2,w' ]
/etc/xen/auto/02-server — configuration file
for the lists domain
# -*- mode: python; -*-
# configuration name:
name = "server"
# usable ram:
memory = 512
# kernel and initrd:
kernel = "/boot/vmlinuz-xen"
ramdisk = "/boot/initrd-xen"
# boot device:
root = "/dev/hda3"
# boot to run level:
extra = "loopback.nloopbacks=0 3"
# network interface:
vif = [ 'mac=aa:cc:00:00:00:01, bridge=xenbr1' ]
hostname = name
# storage devices:
disk = [ 'phy:hda3,hda3,w' ]
With all three Xen domains up and running, the system looks as shown
in the following diagram.
![]()
The zones correspond to the Shorewall zones in the Dom0
configuration.
SuSE 10.0 includes Xen 3.0 which does not support PCI delegation; I
therefore use a bridged configuration with three briges (one for each
network interface). When Shorewall starts during boot, it creates the
three bridges and the tap device tap0 and adds tap0 to xenbr0. tap0 is used by OpenVPN to secure the Wifi zone.
Here is /etc/shorewall/init in Dom0:
for bridge in xenbr0 xenbr1 xenbr2; do
if [ -z "$(/sbin/brctl show 2> /dev/null | fgrep $bridge)" ]; then
/sbin/brctl addbr $bridge
/sbin/ip link set dev $bridge up
case $bridge in
xenbr2)
mac=`ip link show eth1 | grep 'link\/ether' | sed -e 's/.*ether \(..:..:..:..:..:..\).*/\1/'`
[ "$mac" = "fe:ff:ff:ff:ff:ff" ] || /sbin/ip link set dev eth1 addr fe:ff:ff:ff:ff:ff
/sbin/ip link set dev eth1 up
/sbin/brctl addif xenbr2 eth1
;;
xenbr0)
if ! qt /sbin/ip link ls dev tap0; then
/usr/sbin/openvpn --mktun --dev tap0
/sbin/ip link set dev tap0 up
/sbin/brctl addif xenbr0 tap0
fi
;;
esac
fi
done
The goals for the Shorewall configuration in Dom0 are as
follows:
Isolate the Wireless Network so that only VPN access to the
local lan is allowed.
Allow traffic to flow unrestricted through the three bridges.
This is done by configuring the hosts connected to each bridge as a
separate zone and relying on the implicit intra-zone ACCEPT policy to
permit traffic through the bridge.
Ensure that there is no stray traffic between the zones. This is
a "belt+suspenders" measure since there should be no routing between
the bridges (because they don't have IP addresses).
The configuration is a simple one:
/etc/shorewall/zones:
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
fw firewall
Wifi ipv4
loc ipv4
dmz ipv4
net ipv4
#LAST LINE - ADD YOUR ENTRIES ABOVE THIS ONE - DO NOT REMOVE
/etc/shorewall/policy:
#SOURCE DEST POLICY LOG LIMIT:BURST
# LEVEL
Wifi all REJECT info
dmz all REJECT info
all dmz REJECT info
net all REJECT info
all net REJECT info
all all ACCEPT
#LAST LINE -- DO NOT REMOVE
/etc/shorewall/interfaces:
#ZONE INTERFACE BROADCAST OPTIONS
Wifi eth2 192.168.3.255 dhcp,maclist
loc xenbr0 192.168.1.255 dhcp,routeback
dmz xenbr1 - routeback
net xenbr2 - routeback
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/rules:
#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/
# PORT PORT(S) DEST LIMIT GROUP
#SECTION ESTABLISHED
#SECTION RELATED
SECTION NEW
#############################################################################################################
#
# BS Address rules
#
DROP Wifi net:15.0.0.0/8
DROP Wifi net:16.0.0.0/8
#
# Insecure Wireless to local network
#
ACCEPT Wifi loc udp 500
ACCEPT Wifi loc udp 53
ACCEPT Wifi loc udp 4500
Ping/ACCEPT Wifi loc
#
# Insecure Wireless to firewall
#
ACCEPT Wifi fw udp 123
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
In the firewall DomU, I run a conventional three-interface firewall
with Proxy ARP DMZ -- it is very similar to the firewall described in the
Shorewall Setup Guide. The
firewall runs a routed OpenVPN server to
provide roadwarrior access for our two laptops. Here is the firewall's
view of the network:
![]()
The Shorewall configuration files are shown below. All routing and
secondary IP addresses are handled in the SuSE network
configuration.
/etc/shorewall/shorewall.conf:
TARTUP_ENABLED=Yes
VERBOSITY=0
LOGFILE=/var/log/firewall
LOGFORMAT="Shorewall:%s:%s:"
LOGTAGONLY=No
LOGRATE=
LOGBURST=
LOGALLNEW=
BLACKLIST_LOGLEVEL=
MACLIST_LOG_LEVEL=$LOG
TCP_FLAGS_LOG_LEVEL=$LOG
RFC1918_LOG_LEVEL=$LOG
SMURF_LOG_LEVEL=$LOG
LOG_MARTIANS=No
IPTABLES=/usr/sbin/iptables
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
SHOREWALL_SHELL=/bin/bash
SUBSYSLOCK=
MODULESDIR=
CONFIG_PATH=/etc/shorewall:/usr/share/shorewall
RESTOREFILE=standard
IPSECFILE=zones
IP_FORWARDING=On
ADD_IP_ALIASES=No
ADD_SNAT_ALIASES=No
RETAIN_ALIASES=No
TC_ENABLED=Internal
CLEAR_TC=Yes
MARK_IN_FORWARD_CHAIN=Yes
CLAMPMSS=Yes
ROUTE_FILTER=No
DETECT_DNAT_IPADDRS=Yes
MUTEX_TIMEOUT=60
ADMINISABSENTMINDED=Yes
BLACKLISTNEWONLY=Yes
DELAYBLACKLISTLOAD=No
MODULE_SUFFIX=
DISABLE_IPV6=Yes
BRIDGING=No
DYNAMIC_ZONES=No
PKTTYPE=No
RFC1918_STRICT=Yes
MACLIST_TTL=60
SAVE_IPSETS=No
MAPOLDACTIONS=No
FASTACCEPT=Yes
BLACKLIST_DISPOSITION=DROP
MACLIST_TABLE=mangle
MACLIST_DISPOSITION=DROP
TCP_FLAGS_DISPOSITION=DROP
/etc/shorewall/zones:
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
fw firewall
net ipv4 #Internet
loc ipv4 #Local wired Zone
dmz:loc ipv4 #DMZ -- server running in virtual machine at 206.124.146.177
vpn ipv4 #Open VPN clients
#LAST LINE - ADD YOUR ENTRIES ABOVE THIS ONE - DO NOT REMOVE
/etc/shorewall/policy:
#SOURCE DEST POLICY LOG LIMIT:BURST
# LEVEL
$FW $FW ACCEPT
$FW net ACCEPT
loc net ACCEPT
$FW vpn ACCEPT
vpn net ACCEPT
vpn loc ACCEPT
loc vpn ACCEPT
$FW loc ACCEPT
loc $FW REJECT $LOG
net all DROP $LOG 10/sec:40
all all REJECT $LOG
#LAST LINE -- DO NOT REMOVE
/etc/shorewall/params (edited):
MIRRORS=<comma-separated list of Shorewall mirrors>
NTPSERVERS=<comma-separated list of NTP servers I sync with>
POPSERVERS=<comma-separated list of server IP addresses>
LOG=info
INT_IF=eth0
DMZ_IF=eth1
EXT_IF=eth3
OMAK=<IP address at our second home>
#LAST LINE - ADD YOUR ENTRIES ABOVE THIS ONE - DO NOT REMOVE
/etc/shorewall/init:
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
/
/etc/shorewall/interfaces:
#ZONE INTERFACE BROADCAST OPTIONS
net $EXT_IF 206.124.146.255 dhcp,norfc1918,logmartians,blacklist,tcpflags,nosmurfs
dmz $DMZ_IF 192.168.0.255 logmartians
loc $INT_IF 192.168.1.255 dhcp,routeback,logmartians
vpn tun+ -
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/nat:
#EXTERNAL INTERFACE INTERNAL ALL LOCAL
# INTERFACES
206.124.146.178 $EXT_IF 192.168.1.5 No No
206.124.146.180 $EXT_IF 192.168.1.6 No No
#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE
/etc/shorewall/masq:
#INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC
+$EXT_IF:192.168.1.1 0.0.0.0/0 192.168.1.254
$EXT_IF 192.168.0.0/22 206.124.146.179
#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE
/etc/shorewall/proxyarp:
#ADDRESS INTERFACE EXTERNAL HAVEROUTE PERSISTENT
192.168.1.1 $EXT_IF $INT_IF yes
206.124.146.177 $DMZ_IF $EXT_IF yes
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/tunnels:
#TYPE ZONE GATEWAY GATEWAY
# ZONE
openvpnserver:udp net 0.0.0.0/0
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/actions:
#ACTION
Mirrors # Accept traffic from Shorewall Mirrors
#LAST LINE - ADD YOUR ENTRIES ABOVE THIS ONE - DO NOT REMOVE
/etc/shorewall/action.Mirrors:
#TARGET SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE
# PORT PORT(S) DEST LIMIT
ACCEPT $MIRRORS
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
/etc/shorewall/rules:
SECTION NEW
###############################################################################################################################################################################
#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/
# PORT PORT(S) DEST LIMIT GROUP
###############################################################################################################################################################################
REJECT:$LOG loc net tcp 25
REJECT:$LOG loc net udp 1025:1031
#
# Stop NETBIOS crap
#
REJECT loc net tcp 137,445
REJECT loc net udp 137:139
#
# Stop my idiotic work laptop from sending to the net with an HP source/dest IP address
#
DROP loc:!192.168.0.0/22 net
###############################################################################################################################################################################
# Local Network to Firewall
#
DROP loc:!192.168.0.0/22 fw # Silently drop traffic with an HP source IP from my XP box
ACCEPT loc fw tcp 22
ACCEPT loc fw tcp time,631,8080
ACCEPT loc fw udp 161,ntp,631
ACCEPT loc:192.168.1.5 fw udp 111
DROP loc fw tcp 3185 #SuSE Meta pppd
Ping/ACCEPT loc fw
REDIRECT loc 3128 tcp 80 - !206.124.146.177
###############################################################################################################################################################################
# Road Warriors to Firewall
#
ACCEPT vpn fw tcp ssh,time,631,8080
ACCEPT vpn fw udp 161,ntp,631
Ping/ACCEPT vpn fw
###############################################################################################################################################################################
# Road Warriors to DMZ
#
ACCEPT vpn dmz udp domain
ACCEPT vpn dmz tcp www,smtp,smtps,domain,ssh,imap,https,imaps,ftp,10023,pop3 -
Ping/ACCEPT vpn dmz
###############################################################################################################################################################################
# Local network to DMZ
#
ACCEPT loc dmz udp domain
ACCEPT loc dmz tcp ssh,smtps,www,ftp,imaps,domain,https -
ACCEPT loc dmz tcp smtp
Trcrt/ACCEPT loc dmz
###############################################################################################################################################################################
# Internet to ALL -- drop NewNotSyn packets
#
dropNotSyn net fw tcp
dropNotSyn net loc tcp
dropNotSyn net dmz tcp
###############################################################################################################################################################################
# Internet to DMZ
#
ACCEPT net dmz udp domain
LOG:$LOG net:64.126.128.0/18 dmz tcp smtp
ACCEPT net dmz tcp smtps,www,ftp,imaps,domain,https -
ACCEPT net dmz tcp smtp - 206.124.146.177,206.124.146.178
ACCEPT net dmz udp 33434:33454
Mirrors net dmz tcp rsync
Limit:$LOG:SSHA,3,60\
net dmz tcp 22
Trcrt/ACCEPT net dmz
##############################################################################################################################################################################
#
# Net to Local
#
# When I'm "on the road", the following two rules allow me VPN access back home using PPTP.
#
DNAT net loc:192.168.1.4 tcp 1729
DNAT net loc:192.168.1.4 gre
#
# Roadwarrior access to Wookie
#
ACCEPT net:$OMAK loc tcp 22
Limit:$LOG:SSHA,3,60\
net loc tcp 22
#
# ICQ
#
ACCEPT net loc:192.168.1.3 tcp 113,4000:4100
#
# Bittorrent
#
ACCEPT net loc:192.168.1.3 tcp 6881:6889,6969
ACCEPT net loc:192.168.1.3 udp 6881:6889,6969
#
# Real Audio
#
ACCEPT net loc:192.168.1.3 udp 6970:7170
#
# Skype
#
ACCEPT net loc:192.168.1.6 tcp 1194
#
# Traceroute
#
Trcrt/ACCEPT net loc:192.168.1.3
#
# Silently Handle common probes
#
REJECT net loc tcp www,ftp,https
DROP net loc icmp 8
###############################################################################################################################################################################
# DMZ to Internet
#
ACCEPT dmz net udp domain,ntp
ACCEPT dmz net tcp echo,ftp,ssh,smtp,whois,domain,www,81,https,cvspserver,2702,2703,8080
ACCEPT dmz net:$POPSERVERS tcp pop3
Ping/ACCEPT dmz net
#
# Some FTP clients seem prone to sending the PORT command split over two packets. This prevents the FTP connection tracking
# code from processing the command and setting up the proper expectation. The following rule allows active FTP to work in these cases
# but logs the connection so I can keep an eye on this potential security hole.
#
ACCEPT:$LOG dmz net tcp 1024: 20
###############################################################################################################################################################################
# Local to DMZ
#
ACCEPT loc dmz udp domain,xdmcp
ACCEPT loc dmz tcp www,smtp,smtps,domain,ssh,imap,rsync,https,imaps,ftp,10023,pop3,3128
Trcrt/ACCEPT loc dmz
###############################################################################################################################################################################
# DMZ to Local
#
ACCEPT dmz net:192.168.1.254 udp 123
ACCEPT dmz loc:192.168.1.5 tcp 21
Ping/ACCEPT dmz loc
###############################################################################################################################################################################
# DMZ to Firewall -- ntp & snmp, Silently reject Auth
#
ACCEPT dmz fw tcp 161,ssh
ACCEPT dmz fw udp 161
REJECT dmz fw tcp auth
Ping/ACCEPT dmz fw
###############################################################################################################################################################################
# Internet to Firewall
#
REJECT net fw tcp www,ftp,https
DROP net fw icmp 8
ACCEPT net fw udp 33434:33454
ACCEPT net:$OMAK fw udp ntp
ACCEPT net fw tcp auth
ACCEPT net:$OMAK fw tcp 22
Limit:$LOG:SSHA,3,60\
net fw tcp 22
Trcrt/ACCEPT net fw
###############################################################################################################################################################################
# Firewall to DMZ
#
ACCEPT fw dmz tcp domain,www,ftp,ssh,smtp,https,993,465
ACCEPT fw dmz udp domain
REJECT fw dmz udp 137:139
Ping/ACCEPT fw dmz
##############################################################################################################################################################################
# Avoid logging Freenode.net probes
#
DROP net:82.96.96.3 all
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE