shorewall_code/docs/OpenVZ.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
  <!--$Id$-->

  <articleinfo>
    <title>Shorewall and OpenVZ</title>

    <authorgroup>
      <author>
        <firstname>Tom</firstname>

        <surname>Eastep</surname>
      </author>
    </authorgroup>

    <pubdate><?dbtimestamp format="Y/m/d"?></pubdate>

    <copyright>
      <year>2009</year>

      <holder>Thomas M. Eastep</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <section>
    <title>Introduction</title>

    <para><ulink url="http://wiki.openvz.org/">Open Virtuoso (OpenVZ)</ulink>
    is an open source kernel-based virtualization solution from
    <trademark><ulink
    url="http://www.parallels.com">Parallels</ulink></trademark> (formerly
    <trademark>SWSoft</trademark>). Virtual servers take the form of
    <firstterm>containers</firstterm> (the OpenVZ documentation calls these
    <firstterm>Virtual Environments</firstterm> or <firstterm>VEs</firstterm>)
    which are created via <firstterm>templates</firstterm>. Templates are
    available for a wide variety of distributions and architectures.</para>

    <para>OpenVZ requires a patched kernel. Beginning with Lenny,
    <trademark>Debian</trademark> supplies OpenVZ kernels through the standard
    stable repository.</para>
  </section>

  <section>
    <title>Shorewall on an OpenVZ Host</title>

    <para>As with any Shorewall installation involving other software, we
    suggest that you first install OpenVZ and get it working before attempting
    to add Shorewall. Alternatively, execute <command>shorewall
    clear</command> while <ulink
    url="http://wiki.openvz.org/Installation_on_Debian">installing and
    configuring OpenVZ</ulink>.</para>

    <section>
      <title>Networking</title>

      <para>The default OpenVZ networking configuration uses Proxy ARP. You
      assign containers IP addresses in the IP network from one of your
      interfaces and you are expected to set the proxy_arp flag on that
      interface
      (<filename>/proc/sys/net/ipv4/conf/<replaceable>interface</replaceable>/proxy_arp</filename>).</para>

      <para>OpenVZ creates a point-to-point virtual interface in the host with
      a rather odd configuration.</para>

      <para>Example (Single VE with IP address 206.124.146.178):</para>

      <programlisting>gateway:~# <command>ip addr ls dev venet0</command>
10: venet0: &lt;BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UNKNOWN 
    link/void 
gateway:~# <command>ip route ls dev venet0</command>
206.124.146.178  scope link 
gateway:~# </programlisting>

      <para>The interface has no IP configuration yet it has a route to
      206.124.146.178!</para>

      <para>From within the VE with IP address 206.124.146.178, we have the
      following:</para>

      <programlisting>server:~ # <command>ip addr ls</command>
1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
    inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: venet0: &lt;BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UNKNOWN 
    link/void 
    inet 127.0.0.1/32 scope host venet0
    inet 206.124.146.178/32 scope global venet0:0
server:~ # <command>ip route ls</command>
192.0.2.0/24 dev venet0  scope link 
127.0.0.0/8 dev lo  scope link 
default via 192.0.2.1 dev venet0 
server:~ # </programlisting>

      <para>There are a couple of unique features of this
      configuration:</para>

      <itemizedlist>
        <listitem>
          <para>127.0.0.1/32 is configured on venet0 although the main routing
          table routes loopback traffic through the <filename
          class="devicefile">lo</filename> interface as normal.</para>
        </listitem>

        <listitem>
          <para>There is a route to 192.0.2.0/24 through venet0 even though
          the interface has no IP address in that network. Note: 192.0.2.0/24
          is reserved for use in documentation and for testing.</para>
        </listitem>

        <listitem>
          <para>The default route is via 192.0.2.1 yet there is no interface
          on the host with that IP address.</para>
        </listitem>
      </itemizedlist>

      <para>All of this doesn't really affect the Shorewall configuration but
      it is interesting none the less.</para>
    </section>

    <section>
      <title>Shorewall Configuration</title>

      <para>We recommend handling the strange OpenVZ configuration in
      Shorewall as follows:</para>

      <para><filename>/etc/shorewall/zones</filename>:</para>

      <programlisting>###############################################################################
#ZONE    TYPE       OPTIONS                IN                       OUT
#                                          OPTIONS                  OPTIONS
net      ipv4
vz       ipv4</programlisting>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>###############################################################################
#ZONE    INTERFACE          BROADCAST      OPTIONS
net      eth0               -              proxyarp=1
vz       venet0             -              routeback,rp_filter=0</programlisting>
    </section>

    <section>
      <title>Multi-ISP</title>

      <para>If you run Shorewall Multi-ISP support on the host, you should
      arrange for traffic to your containers to use the main routing table. In
      the configuration shown here, this entry in /etc/shorewall/route_rules
      is appropriate:</para>

      <programlisting>#SOURCE            DEST              PROVIDER          PRIORITY
-                  206.124.146.178   main              1000</programlisting>
    </section>

    <section>
      <title>RFC 1918 Addresses in a Container</title>

      <para>You can assign an RFC 1918 address to a VE and use masquerade/SNAT
      to provide Internet access to the container. This is just a normal
      simple Shorewall configuration as shown in the <ulink
      url="two-interface.htm">Two-interface Quick Start Guide</ulink>. In this
      configuration the firewall's internal interface is <filename
      class="devicefile">venet0</filename>. Be sure to include the options
      shown above.</para>
    </section>
  </section>

  <section>
    <title>Shorewall in an OpenVZ Virtual Environment</title>

    <para>If you have obtained an OpenVZ VE from a hosting service provider,
    you may find it difficult to configure any type of firewall within your
    VE. There are two VE parameters that control iptables behavior within the
    container:</para>

    <variablelist>
      <varlistentry>
        <term>--iptables <replaceable>name </replaceable></term>

        <listitem>
          <para>Restrict access to iptables modules inside a container (The
          OpenVZ claims that by default all iptables modules that are loaded
          in the host system are accessible inside a container; I haven't
          tried that).</para>

          <para>You can use the following values for
          <replaceable>name</replaceable>: <option>iptable_filter</option>,
          <option>iptable_mangle</option>, <option>ipt_limit</option>,
          <option>ipt_multiport</option>, <option>ipt_tos</option>,
          <option>ipt_TOS</option>, <option>ipt_REJECT</option>,
          <option>ipt_TCPMSS</option>, <option>ipt_tcpmss</option>,
          <option>ipt_ttl</option>, <option>ipt_LOG</option>,
          <option>ipt_length</option>, <option>ip_conntrack</option>,
          <option>ip_conntrack_ftp</option>,
          <option>ip_conntrack_irc</option>, <option>ipt_conntrack</option>,
          <option>ipt_state</option>, <option>ipt_helper</option>,
          <option>iptable_nat</option>, <option>ip_nat_ftp</option>,
          <option>ip_nat_irc</option>, <option>ipt_REDIRECT</option>,
          <option>xt_mac</option>, <option>ipt_owner</option>.</para>

          <para>If your provider is using this option, you may be in deep
          trouble trying to use Shorewall in your container. Look at the
          output of <command>shorewall show capabilities</command> and weep.
          Then try to get your provider to remove this restriction on your
          container.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>--numiptent <replaceable>num</replaceable></term>

        <listitem>
          <para>This parameter limits the number of iptables rules that are
          allowed within the container. The default is 100 which is too small
          for a Shorewall configuration. We recommend setting this to at least
          200.</para>
        </listitem>
      </varlistentry>
    </variablelist>

    <para>if you see annoying error messages as shown below during
    start/restart, remove the module-init-tools package from the VE.</para>

    <programlisting>server:/etc/shorewall # shorewall restart
Compiling...
Compiling /etc/shorewall/zones...
Compiling /etc/shorewall/interfaces...
Determining Hosts in Zones...
Preprocessing Action Files...
   Pre-processing /usr/share/shorewall/action.Drop...
   Pre-processing /usr/share/shorewall/action.Reject...
Compiling /etc/shorewall/policy...
Adding Anti-smurf Rules
Adding rules for DHCP
Compiling TCP Flags filtering...
Compiling Kernel Route Filtering...
Compiling Martian Logging...
Compiling MAC Filtration -- Phase 1...
Compiling /etc/shorewall/rules...
Generating Transitive Closure of Used-action List...
Processing /usr/share/shorewall/action.Reject for chain Reject...
Processing /usr/share/shorewall/action.Drop for chain Drop...
Compiling MAC Filtration -- Phase 2...
Applying Policies...
Generating Rule Matrix...
Creating iptables-restore input...
Compiling iptables-restore input for chain mangle:...
Compiling /etc/shorewall/routestopped...
Shorewall configuration compiled to /var/lib/shorewall/.restart
Restarting Shorewall....
Initializing...
Processing /etc/shorewall/init ...
Processing /etc/shorewall/tcclear ...
Setting up Route Filtering...
Setting up Martian Logging...
Setting up Proxy ARP...
Setting up Traffic Control...
Preparing iptables-restore input...
Running /usr/sbin/iptables-restore...
<emphasis role="bold">FATAL: Could not load /lib/modules/2.6.26-2-openvz-amd64/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.26-2-openvz-amd64/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.26-2-openvz-amd64/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.26-2-openvz-amd64/modules.dep: No such file or directory</emphasis>
IPv4 Forwarding Enabled
Processing /etc/shorewall/start ...
Processing /etc/shorewall/started ...
done.
</programlisting>
  </section>

  <section>
    <title>Working Example</title>

    <para>This section presents a working example. This is the configuration
    at shorewall.net during the summer of 2009.</para>

    <para>The network diagram is shown below.</para>

    <graphic fileref="images/Network2009c.png" />

    <para>The two systems shown in the green box are OpenVZ Virtual
    Environments (containers).</para>

    <section>
      <title>OpenVZ Configuration</title>

      <para>In the files below, items in <emphasis role="bold">bold
      font</emphasis> are relevant to the networking/Shorewall
      configuration.</para>

      <para><filename>/etc/vz/conf</filename> (long lines folded for
      clarity).</para>

      <programlisting>## Global parameters
VIRTUOZZO=yes
LOCKDIR=/var/lib/vz/lock
DUMPDIR=/var/lib/vz/dump
VE0CPUUNITS=1000

## Logging parameters
LOGGING=yes
LOGFILE=/var/log/vzctl.log
LOG_LEVEL=0
VERBOSE=0

## Disk quota parameters
DISK_QUOTA=no
VZFASTBOOT=no

# The name of the device whose ip address will be used as source ip for VE.
# By default automatically assigned.
<emphasis role="bold">VE_ROUTE_SRC_DEV="eth3"</emphasis>

# Controls which interfaces to send ARP requests and modify APR tables on.
NEIGHBOUR_DEVS=detect

## Template parameters
TEMPLATE=/var/lib/vz/template

## Defaults for VEs
VE_ROOT=/home/vz/root/$VEID
VE_PRIVATE=/home/vz/private/$VEID
CONFIGFILE="vps.basic"
#DEF_OSTEMPLATE="fedora-core-4"
DEF_OSTEMPLATE="debian"

## Load vzwdog module
VZWDOG="no"

## IPv4 iptables kernel modules
<emphasis role="bold">IPTABLES="iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos
          ipt_TOS ipt_REJECT ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_LOG ipt_length
          ip_conntrack ip_conntrack_ftp ip_conntrack_irc ipt_conntrack
          ipt_state ipt_helper iptable_nat  ip_nat_ftp  ip_nat_irc ipt_REDIRECT
          xt_mac ipt_owner"
</emphasis>
## Enable IPv6
IPV6="no"</programlisting>

      <para><filename>/etc/vz/conf/101.conf</filename>:</para>

      <programlisting>ONBOOT="yes"

# UBC parameters (in form of barrier:limit)
KMEMSIZE="574890800:589781600"
LOCKEDPAGES="256:256"
PRIVVMPAGES="1073741824:2137483648"
SHMPAGES="21504:21504"
NUMPROC="240:240"
PHYSPAGES="0:9223372036854775807"
VMGUARPAGES="262144:9223372036854775807"
OOMGUARPAGES="26112:9223372036854775807"
NUMTCPSOCK="360:360"
NUMFLOCK="188:206"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="1720320:2703360"
TCPRCVBUF="1720320:2703360"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="360:360"
DCACHESIZE="3409920:3624960"
NUMFILE="9312:9312"
AVNUMPROC="180:180"
<emphasis role="bold">NUMIPTENT="200:200"</emphasis>

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="1048576:1153024"
DISKINODES="200000:220000"
QUOTATIME="0"

# CPU fair sheduler parameter
CPUUNITS="1000"

VE_ROOT="/home/vz/root/$VEID"
VE_PRIVATE="/home/vz/private/$VEID"
OSTEMPLATE="suse-11.1-x86_64"
ORIGIN_SAMPLE="vps.basic"
<emphasis role="bold">HOSTNAME="lists.shorewall.net"
IP_ADDRESS="206.124.146.177"
NAMESERVER="127.0.0.1"
NAME="lists"
SEARCHDOMAIN="shorewall.net"</emphasis></programlisting>

      <para>This VE is the main server at shorewall.net. Note that some of the
      memory parameters are set ridiculously large -- I got tired of
      out-of-memory issues.</para>

      <para><filename>/etc/vz/conf/102.conf</filename> (nearly default
      configuration on Debian):</para>

      <programlisting>ONBOOT="yes"

# UBC parameters (in form of barrier:limit)
KMEMSIZE="14372700:14790164"
LOCKEDPAGES="256:256"
PRIVVMPAGES="65536:69632"
SHMPAGES="21504:21504"
NUMPROC="240:240"
PHYSPAGES="0:9223372036854775807"
VMGUARPAGES="33792:9223372036854775807"
OOMGUARPAGES="26112:9223372036854775807"
NUMTCPSOCK="360:360"
NUMFLOCK="188:206"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="1720320:2703360"
TCPRCVBUF="1720320:2703360"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="360:360"
DCACHESIZE="3409920:3624960"
NUMFILE="9312:9312"
AVNUMPROC="180:180"
<emphasis role="bold">NUMIPTENT="200:200"</emphasis>

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="1048576:1153024"
DISKINODES="200000:220000"
QUOTATIME="0"

# CPU fair sheduler parameter
CPUUNITS="1000"

VE_ROOT="/home/vz/root/$VEID"
VE_PRIVATE="/home/vz/private/$VEID"
OSTEMPLATE="debian-5.0-amd64-minimal"
ORIGIN_SAMPLE="vps.basic"
<emphasis role="bold">HOSTNAME="server.shorewall.net"
IP_ADDRESS="206.124.146.178"
NAMESERVER="206.124.146.177"
NAME="server"</emphasis></programlisting>

      <para>I really don't use this server for anything currently but I'm
      planning to eventually splt the services between the two VEs.</para>
    </section>

    <section>
      <title>Shorewall Configuration on the Host</title>

      <para>Below are exerpts from the configuration files as they pertain to
      the OpenVZ environment.</para>

      <para><filename>/etc/shorewall/zones</filename>:</para>

      <programlisting>#ZONE           TYPE            OPTIONS         IN                      OUT
#                                               OPTIONS                 OPTIONS
fw              firewall
net             ipv4            #Internet
loc             ipv4            #Local wired Zone
<emphasis role="bold">dmz             ipv4            #DMZ</emphasis>
...</programlisting>

      <para><filename>/etc/shorewall/params</filename>:</para>

      <programlisting>NET_IF=eth3
INT_IF=eth1
<emphasis role="bold">VPS_IF=venet0</emphasis>
...</programlisting>

      <para><filename>/etc/shorewall/interfaces</filename>:<programlisting>#ZONE   INTERFACE       BROADCAST               OPTIONS
net     $NET_IF         detect                  dhcp,blacklist,tcpflags,optional,routefilter=0,nosmurfs,logmartions=0,<emphasis
            role="bold">proxyarp=1</emphasis>
loc     $INT_IF         detect                  dhcp,logmartians=1,routefilter=1,nets=(172.20.1.0/24),tcpflags
<emphasis role="bold">dmz     $VPS_IF         detect                  logmartians=0,routefilter=0,nets=(206.124.146.177,206.124.146.178),routeback</emphasis>
...</programlisting>This is a multi-ISP configuration so entries are required
      in <filename>/etc/shorewall/route_rules</filename>:</para>

      <programlisting>#SOURCE   DEST                 PROVIDER  PRIORITY
-         172.20.0.0/24        main      1000
<emphasis role="bold">-         206.124.146.177      main      1001
-         206.124.146.178      main      1001</emphasis></programlisting>
    </section>

    <section>
      <title>Shorewall Configuration on Server</title>

      <para>I have set up Shorewall on Server (206.124.146.178) just to have
      an environment to test with. It is a quite vanilla one-interface
      configuration.</para>

      <para>/etc/shorewall/zones:</para>

      <programlisting>#ZONE       TYPE         OPTIONS           IN                OUT
#                                          OPTIONS           OPTIONS
fw          firewall
net         ipv4</programlisting>

      <para>/etc/shorewall/interfaces:</para>

      <programlisting>#ZONE   INTERFACE       BROADCAST       OPTIONS
net     <emphasis role="bold">venet0 </emphasis>         detect          dhcp,tcpflags,logmartians,nosmurfs</programlisting>
    </section>
  </section>
</article>