shorewall_code/docs/MultiISP.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<article>
  <!--$Id$-->

  <articleinfo>
    <title>Shorewall and Multiple Internet Connections</title>

    <authorgroup>
      <author>
        <firstname>Tom</firstname>

        <surname>Eastep</surname>
      </author>
    </authorgroup>

    <pubdate>2006-05-06</pubdate>

    <copyright>
      <year>2005</year>

      <year>2006</year>

      <holder>Thomas M. Eastep</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <warning>
    <para>Make sure you are<emphasis role="bold"> running a current,
    vendor-supported distribution</emphasis>, before attempting to perform
    this setup, older distributions do not meet the minimum requirements,and
    you will need to recompile iptables, kernel and other software on your
    system. If you don't follow this advice,<emphasis role="bold">we will
    <emphasis role="bold">not</emphasis> be able to help
    you</emphasis>.</para>
  </warning>

  <warning>
    <para>Reading just Shorewall documentation is probably not going to give
    you enough background to use this material. Shorewall may make iptables
    easy but the Shorewall team doesn't have the resources to be able to
    spoon-feed Linux policy routing to you (please remember that the user's
    manual for a tractor doesn't teach you to grow corn either). You will
    likely need to refer to the following additional information:</para>

    <itemizedlist>
      <listitem>
        <para>The LARTC HOWTO: <ulink
        url="http://www.lartc.org">http://www.lartc.org</ulink></para>
      </listitem>

      <listitem>
        <para>Output of <command>man ip</command></para>
      </listitem>

      <listitem>
        <para>Output of <command>ip route help</command> and <command>ip rule
        help</command></para>
      </listitem>
    </itemizedlist>
  </warning>

  <section>
    <title>Multiple Internet Connection Support</title>

    <para>Beginning with Shorewall 2.3.2, limited support is included for
    multiple internet connections. Limitations of this support are as
    follows:</para>

    <itemizedlist>
      <listitem>
        <para>It utilizes static routing configuration. As such, there is no
        provision for reacting to the failure of either of the uplinks.</para>
      </listitem>

      <listitem>
        <para>The routing changes are made and the route cache is purged when
        Shorewall is started <emphasis role="bold">and when Shorewall is
        restarted</emphasis> (unless you specify the "-n" option to
        <command>shorewall restart</command>). Ideally, restarting the packet
        filter should have no effect on routing.</para>
      </listitem>
    </itemizedlist>

    <section>
      <title>Overview</title>

      <para>Let's assume that a firewall is connected via two separate
      ethernet interfaces to two different ISPs as in the following
      diagram.</para>

      <graphic align="center" fileref="images/TwoISPs.png" valign="middle" />

      <itemizedlist>
        <listitem>
          <para>eth0 connects to ISP1. The IP address of eth0 is
          206.124.146.176 and the ISP's gateway router has IP address
          206.124.146.254.</para>
        </listitem>

        <listitem>
          <para>eth1 connects to ISP 2. The IP address of eth1 is
          130.252.99.27 and the ISP's gateway router has IP address
          130.252.99.254.</para>
        </listitem>

        <listitem>
          <para>eth2 connects to the local LAN. Its IP configuration is not
          relevant to this discussion.</para>
        </listitem>
      </itemizedlist>

      <para>Each of these <firstterm>providers</firstterm> is described in an
      entry in the file <filename>/etc/shorewall/providers</filename>.</para>

      <para>Entries in <filename>/etc/shorewall/providers</filename> can
      specify that outgoing connections are to be load-balanced between the
      two ISPs. Entries in <filename>/etc/shorewall/tcrules</filename> can be
      used to direct particular outgoing connections to one ISP or the other.
      Use of <filename>/etc/shorewall/tcrules</filename> is not required for
      <filename>/etc/shorewall/providers</filename> to work, but you must
      select a unique MARK value for each provider so Shorewall can set up the
      correct marking rules for you.</para>

      <para>When you use the <emphasis role="bold">track</emphasis> option in
      <filename>/etc/shorewall/providers</filename>, connections from the
      internet are automatically routed back out of the correct interface and
      through the correct ISP gateway. This works whether the connection is
      handled by the firewall itself or if it is routed or port-forwarded to a
      system behind the firewall.</para>

      <para>Shorewall will set up the routing and will update the
      <filename>/etc/iproute2/rt_tables</filename> to include the table names
      and number of the tables that it adds.</para>

      <caution>
        <para>This feature uses <ulink url="traffic_shaping.htm">packet
        marking</ulink> to control the routing. As a consequence, there are
        some restrictions concerning entries in
        <filename>/etc/shorewall/tcrules</filename>:</para>

        <itemizedlist>
          <listitem>
            <para>Packet marking for traffic control purposes may not be done
            in the PREROUTING table for connections involving providers with
            'track' specified (see below).</para>
          </listitem>

          <listitem>
            <para>You may not use the SAVE or RESTORE options.</para>
          </listitem>

          <listitem>
            <para>You may not use connection marking.</para>
          </listitem>
        </itemizedlist>
      </caution>

      <para>The <filename>/etc/shorewall/providers</filename> file can also be
      used in other routing scenarios. See the <ulink
      url="Shorewall_Squid_Usage.html">Squid documentation</ulink> for an
      example.</para>
    </section>

    <section>
      <title>/etc/shorewall/providers File</title>

      <para>Entries in this file have the following columns. As in all
      Shorewall configuration files, enter "-" in a column if you don't want
      to enter any value.</para>

      <variablelist>
        <varlistentry>
          <term>NAME</term>

          <listitem>
            <para>The provider name. Must begin with a letter and consist of
            letters and digits. The provider name becomes the name of the
            generated routing table for this provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>NUMBER</term>

          <listitem>
            <para>A number between 1 and 252. This becomes the routing table
            number for the generated table for this provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>MARK</term>

          <listitem>
            <para>A mark value used in your /etc/shorewall/tcrules file to
            direct packets to this provider. Shorewall will also mark
            connections that have seen input from this provider with this
            value and will restore the packet mark in the PREROUTING
            CHAIN.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>DUPLICATE</term>

          <listitem>
            <para>Gives the name or number of a routing table to duplicate.
            May be 'main' or the name or number of a previously declared
            provider. For most applications, you want to specify 'main'
            here.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>INTERFACE</term>

          <listitem>
            <para>The name of the interface to the provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>GATEWAY</term>

          <listitem>
            <para>The IP address of the provider's Gateway router.</para>

            <para>You can enter <emphasis role="bold">detect</emphasis> here
            and Shorewall will attempt to automatically determine the gateway
            IP address.</para>

            <para><emphasis role="bold">Hint:</emphasis> <emphasis
            role="bold">"detect"</emphasis> is appropriate for use in cases
            where the interface named in the INTERFACE column is dynamically
            configured via DHCP etc.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>OPTIONS</term>

          <listitem>
            <para>A comma-separated list from the following:</para>

            <variablelist>
              <varlistentry>
                <term>track</term>

                <listitem>
                  <para>If specified, connections FROM this interface are to
                  be tracked so that responses may be routed back out this
                  same interface.</para>

                  <para>You want to specify 'track' if internet hosts will be
                  connecting to local servers through this provider. Any time
                  that you specify 'track', you will also want to specify
                  'balance' (see below).</para>

                  <para>Use of this feature requires that your kernel and
                  iptables support CONNMARK target and connmark match support.
                  It does not require the ROUTE target extension.</para>

                  <warning>
                    <para>iptables 1.3.1 is broken with respect to CONNMARK
                    and iptables-save/iptables-restore. This means that if you
                    configure multiple ISPs, <command>shorewall
                    restore</command> may fail. If it does, you may patch your
                    iptables using the patch at <ulink
                    url="http://shorewall.net/pub/shorewall/contrib/iptables/CONNMARK.diff">http://shorewall.net/pub/shorewall/contrib/iptables/CONNMARK.diff</ulink>.</para>
                  </warning>

                  <important>
                    <para>If you are using
                    <filename>/etc/shorewall/providers</filename> because you
                    have multiple internet connections, we recommend that you
                    specify 'track' even if you don't need it. It helps
                    maintain long-term connections in which there are
                    significant periods with no traffic.</para>
                  </important>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>balance</term>

                <listitem>
                  <para>The providers that have 'balance' specified will get
                  outbound traffic load-balanced among them. Balancing will
                  not be perfect, as it is route based, and routes are cached.
                  This means that routes to often-used sites will always be
                  over the same provider.</para>

                  <para>By default, each provider is given the same weight (1)
                  . You can change the weight of a given provider by following
                  <emphasis>balance</emphasis> with "=" and the desired weight
                  (e.g., balance=2). The weights reflect the relative
                  bandwidth of the providers connections and should be small
                  numbers since the kernel actually creates additional default
                  routes for each weight increment.</para>

                  <important>
                    <para>If you are using
                    <filename>/etc/shorewall/providers</filename> because you
                    have multiple internet connections, we recommend that you
                    specify 'balance' even if you don't need it. You can still
                    use entries in <filename>/etc/shorewall/tcrules</filename>
                    to force traffic to one provider or another.</para>
                  </important>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>loose</term>

                <listitem>
                  <para>Do not include routing rules that force traffic whose
                  source IP is an address of the INTERFACE to be routed to
                  this provider. Useful for defining providers that are to be
                  used only when the appropriate packet mark is
                  applied.</para>
                </listitem>
              </varlistentry>
            </variablelist>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>COPY</term>

          <listitem>
            <para>When you specify an existing table in the DUPLICATE column,
            Shorewall copies all routes through the interface specified in the
            INTERFACE column plus the interfaces listed in this column. At a
            minumum, you should list all interfaces on your firewall in this
            column except those internet interfaces specified in the INTERFACE
            column of entries in this file.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </section>

    <section>
      <title>What an entry in the Providers File Does</title>

      <para>Adding another entry in the providers file simply creates an
      alternate routing table for you. In addition:</para>

      <orderedlist>
        <listitem>
          <para>Unless <emphasis role="bold">loose</emphasis> is specified, an
          ip rule is generated for each IP address on the INTERFACE that
          routes traffic from that address through the associated routing
          table.</para>
        </listitem>

        <listitem>
          <para>If you specify <emphasis role="bold">track</emphasis>, then
          connections which have had at least one packet arrive on the
          interface listed in the INTERFACE column have their connection mark
          set to the value in the MARK column. In the PREROUTING chain,
          packets with a connection mark have their packet mark set to the
          value of the associated connection mark; packets marked in this way
          bypass any prerouting rules that you create in
          <filename>/etc/shorewall/tcrules</filename>. This ensures that
          packets associated with connections from outside are always routed
          out of the correct interface.</para>
        </listitem>

        <listitem>
          <para>If you specify <emphasis role="bold">balance</emphasis>, then
          Shorewall will replace the 'default' route with weight 100 in the
          'main' routing table with a load-balancing route among those
          gateways where <emphasis role="bold">balance</emphasis> was
          specified. So if you configure default routes, be sure that their
          weight is less than 100 or the route added by Shorewall will not be
          used.</para>
        </listitem>
      </orderedlist>

      <para>That's <emphasis role="bold">all</emphasis> that these entries do.
      You still have to follow the principle stated in the <ulink
      url="Shorewall_and_Routing.html">Shorewall Routing
      documentation</ulink>:</para>

      <orderedlist>
        <listitem>
          <para>Routing determines where packets are to be sent.</para>
        </listitem>

        <listitem>
          <para>Once routing determines where the packet is to go, the
          firewall (Shorewall) determines if the packet is allowed to go
          there.</para>
        </listitem>
      </orderedlist>

      <para>The bottom line is that if you want traffic to go out through a
      particular provider then you <emphasis>must </emphasis>mark that traffic
      with the provider's MARK value in
      <filename>/etc/shorewall/tcrules</filename> and you must do that marking
      in the PREROUTING chain.</para>

      <warning>
        <para>Entries in <filename>/etc/shorewall/providers</filename>
        permanently alter your firewall/gateway's routing; that is, the effect
        of these changes is not reversed by <command>shorewall stop</command>
        or <command>shorewall clear</command>. To restore routing to its
        original state, you may have to restart your network. This can usually
        be done by <command>/etc/init.d/network restart</command> or
        <command>/etc/init.d/networking restart</command>. Check your
        distribution's networking documentation.</para>

        <para>Here are some additional things to consider:</para>

        <itemizedlist>
          <listitem>
            <para>You can mitigate the effect of the Shorewall-generated
            changes to your routing table by specifying a
            <emphasis>metric</emphasis> for each default route that you
            configure. Shorewall will generate a load-balancing default route
            (assuming that <emphasis role="bold">balance</emphasis> has been
            specified for some of the providers) that does not include a
            metric and that will therefore not replace any existing route that
            has a non-zero metric.</para>
          </listitem>

          <listitem>
            <para>The <command>-n</command> option to <command>shorewall
            restart</command> and <command>shorewall restore</command> can be
            used to prevent the command from changing your routing.</para>
          </listitem>

          <listitem>
            <para>The <filename>/etc/shorewall/stopped</filename> file can
            also be used to restore routing when you stop Shorewall. With your
            firewall in it's normal (single-table) routing configuration, you
            can capture the contents as follows:</para>

            <programlisting>ip route ls &gt; routes</programlisting>

            <para>Here's what the <filename>routes</filename> file looked like
            after I did that on my firewall:</para>

            <programlisting>192.168.1.1 dev eth3  scope link
206.124.146.177 dev eth1  scope link
192.168.2.2 dev tun0  proto kernel  scope link  src 192.168.2.1
192.168.2.0/24 via 192.168.2.2 dev tun0
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.254
206.124.146.0/24 dev eth3  proto kernel  scope link  src 206.124.146.176
169.254.0.0/16 dev eth0  scope link
127.0.0.0/8 dev lo  scope link
default via 206.124.146.254 dev eth3</programlisting>

            <para>Now edit the file as shown below:</para>

            <programlisting><command>ip route flush table main
ip route add</command> 192.168.1.1 dev eth3  scope link
<command>ip route add </command>206.124.146.177 dev eth1  scope link
<command>ip route add </command>192.168.2.2 dev tun0  proto kernel  scope link  src 192.168.2.1
<command>ip route add </command>192.168.2.0/24 via 192.168.2.2 dev tun0
<command>ip route add </command>192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.254
<command>ip route add </command>206.124.146.0/24 dev eth3  proto kernel  scope link  src 206.124.146.176
<command>ip route add </command>169.254.0.0/16 dev eth0  scope link
<command>ip route add </command>127.0.0.0/8 dev lo  scope link
<command>ip route add </command>default via 206.124.146.254 dev eth3
<command>ip route flush cache</command></programlisting>

            <para>Now paste the contents of that file into
            <filename>/etc/shorewall/stopped</filename>.</para>

            <para>You might also want to consider adding the following to the
            file:</para>

            <programlisting><command>ip rule ls</command> | while read priority rule; do
    case ${priority%:} in
        0|3276[67])
            ;;
        *) ip rule del $rule
            ;;
    esac
done</programlisting>

            <para>That will delete all but the default routing rules.</para>
          </listitem>
        </itemizedlist>
      </warning>
    </section>

    <section>
      <title>What an entry in the Providers File Does NOT Do</title>

      <para>Given that Shorewall is simply a tool to configure Netfilter and
      does not run continuously in your system, entries in the providers file
      <emphasis role="bold">do not provide any automatic failover in the event
      of failure of one of your Internet connections</emphasis>.</para>
    </section>

    <section>
      <title>Example</title>

      <para>The configuration in the figure at the top of this section would
      be specified in <filename>/etc/shorewall/providers</filename> as
      follows.</para>

      <programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2</programlisting>

      <para>Other configuration files go something like this:</para>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …</programlisting>

      <para><filename>/etc/shorewall/policy</filename>:</para>

      <programlisting>#SOURCE    DESTINATION    POLICY     LIMIT:BURST
net        net            DROP</programlisting>

      <para>Regardless of whether you have masqueraded hosts or not, <emphasis
      role="bold">YOU MUST ADD THESE TWO ENTRIES TO
      <filename>/etc/shorewall/masq</filename></emphasis>:</para>

      <programlisting>#INTERFACE       SUBNET            ADDRESS
eth0             130.252.99.27     206.124.146.176
eth1             206.124.146.176   130.252.99.27</programlisting>

      <note>
        <para>If you have a Dynamic IP address on either of the interfaces,
        you can use shell variables to construct the above rules. For example,
        if <filename class="devicefile">eth0</filename> had a dynamic IP
        address, then:</para>

        <para><filename>/etc/shorewall/params</filename>:</para>

        <programlisting>ETH0_IP=$(find_first_interface_address eth0)</programlisting>

        <para>/etc/shorewall/masq:</para>

        <programlisting>#INTERFACE       SUBNET            ADDRESS
eth0             130.252.99.27     $ETH0_IP
eth1             $ETH0_IP          130.252.99.27</programlisting>
      </note>

      <para>If you have masqueraded hosts, be sure to update
      <filename>/etc/shorewall/masq</filename> to masquerade to both ISPs. For
      example, if you masquerade all hosts connected to <filename
      class="devicefile">eth2</filename> then:</para>

      <programlisting>#INTERFACE       SUBNET            ADDRESS
eth0             eth2              206.124.146.176
eth1             eth2              130.252.99.27</programlisting>

      <warning>
        <para>Entries in <filename>/etc/shorewall/masq</filename> have no
        effect on which ISP a particular connection will be sent through. That
        is rather the purpose of entries in
        <filename>/etc/shorewall/tcrules</filename>.</para>
      </warning>

      <para>Now suppose that you want to route all outgoing SMTP traffic from
      your local network through ISP 2. You would make this entry in <ulink
      url="traffic_shaping.htm">/etc/shorewall/tcrules</ulink> (and if you are
      running a version of Shorewall earlier than 3.0.0, you would set
      TC_ENABLED=Yes in <ulink
      url="???">/etc/shorewall/shorewall.conf</ulink>).</para>

      <programlisting>#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2:P             &lt;local network&gt; 0.0.0.0/0       tcp     25</programlisting>
    </section>

    <section>
      <title>/etc/shorewall/route_rules</title>

      <para>The <filename>/etc/shorewall/route_rules</filename> file was added
      in Shorewall version 3.2.0. The <filename>route_rules</filename> file
      allows assigning certain traffic to a particular provider just as
      entries in the <filename>tcrules</filename> file. The difference between
      the two files is that entries in <filename>route_rules</filename> are
      independent of Netfilter.</para>

      <section>
        <title>Routing Rules</title>

        <para>Routing rules are maintained by the Linux kernel and can be
        displayed using the <command>ip rule ls</command> command. When
        routing a packet, the rules are processed in turn until the packet is
        successfully routed.</para>

        <programlisting>gateway:~ # <command>ip rule ls</command>
0:      from all lookup local                &lt;=== Local (to the firewall) IP addresses
10001:  from all fwmark 0x1 lookup Blarg     &lt;=== This and the next rule are generated by the
10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
20000:  from 206.124.146.176 lookup Blarg    &lt;=== This and the next rule are generated unless
20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
32766:  from all lookup main                 &lt;=== This is the routing table shown by 'iproute -n'
32767:  from all lookup default              &lt;=== This table is usually empty
gateway:~ #</programlisting>

        <para>In the above example, there are two providers: Blarg and Comcast
        with MARK 1 going to Blarg and mark 2 going to Comcast.</para>
      </section>

      <section>
        <title>Columns in the route_rules file</title>

        <para>Columns in the file are:</para>

        <variablelist>
          <varlistentry>
            <term>SOURCE (Optional)</term>

            <listitem>
              <para>An ip address (network or host) that matches the source IP
              address in a packet. May also be specified as an interface name
              optionally followed by ":" and an address. If the device 'lo' is
              specified, the packet must originate from the firewall
              itself.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>DEST (Optional)</term>

            <listitem>
              <para>An ip address (network or host) that matches the
              destination IP address in a packet.</para>

              <para>If you choose to omit either SOURCE or DEST, place "-" in
              that column. Note that you may not omit both SOURCE and
              DEST.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>PROVIDER</term>

            <listitem>
              <para>The provider to route the traffic through. May be
              expressed either as the provider name or the provider
              number.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>PRIORITY</term>

            <listitem>
              <para>The rule's priority which determines the order in which
              the rules are processed.</para>

              <para>1000-1999 Before Shorewall-generated 'MARK' rules</para>

              <para>11000- 11999 After 'MARK' rules but before
              Shorewall-generated rules for ISP interfaces.</para>

              <para>26000-26999 After ISP interface rules but before 'default'
              rule.</para>

              <para>Rules with equal priority are applied in the order in
              which they appear in the file.</para>
            </listitem>
          </varlistentry>
        </variablelist>

        <para>Example 1: You want all traffic entering the firewall on eth1 to
        be routed through Comcast.</para>

        <programlisting>#SOURCE            DEST      PROVIDER        PRIORITY
eth1               -         Comcast         1000</programlisting>

        <para>With this entry, the output of <command>ip rule ls</command>
        would be as follows.</para>

        <para><programlisting>gateway:~ # <command>ip rule ls</command>
0:      from all lookup local
1000:   from all iif eth1 lookup Comcast
10001:  from all fwmark 0x1 lookup Blarg
10002:  from all fwmark 0x2 lookup Comcast
20000:  from 206.124.146.176 lookup Blarg
20256:  from 24.12.22.33 lookup Comcast
32766:  from all lookup main
32767:  from all lookup default
gateway:~ #</programlisting>Note that because we used a priority of 1000, the
        test for <filename class="devicefile">eth1</filename> is inserted
        before the fwmark tests.</para>

        <para>Example 2: You use OpenVPN (routed setup /tunX) in combination
        with multiple providers. In this case you have to set up a rule to
        ensure that the OpenVPN traffic is routed back through the tunX
        interface(s) rather than through any of the providers. 10.8.0.0/24 is
        the subnet choosen in your OpenVPN configuration (server 10.8.0.0
        255.255.255.0).</para>

        <programlisting>#SOURCE                 DEST            PROVIDER        PRIORITY
-                       10.8.0.0/24     main            1000</programlisting>
      </section>
    </section>
  </section>
</article>