shorewall_code/docs/MultiISP.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
  <!--ble$Id$-->

  <articleinfo>
    <title>Shorewall and Multiple Internet Connections</title>

    <authorgroup>
      <author>
        <firstname>Tom</firstname>

        <surname>Eastep</surname>
      </author>
    </authorgroup>

    <pubdate><?dbtimestamp format="Y/m/d"?></pubdate>

    <copyright>
      <year>2005</year>

      <year>2006</year>

      <year>2007</year>

      <year>2008</year>

      <year>2009</year>

      <holder>Thomas M. Eastep</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <warning>
    <para>This document describes the Multi-ISP facility in Shorewall 4.3.5
    and later. If you are running an earlier release, please see the
    documentation for that release.</para>
  </warning>

  <warning>
    <para>Reading just Shorewall documentation is probably not going to give
    you enough background to use this material. Shorewall may make iptables
    easy but the Shorewall team doesn't have the resources to be able to
    spoon-feed Linux policy routing to you (please remember that the user's
    manual for a tractor doesn't teach you to grow corn either). You will
    likely need to refer to the following additional information:</para>

    <itemizedlist>
      <listitem>
        <para>The LARTC HOWTO: <ulink
        url="http://www.lartc.org">http://www.lartc.org</ulink></para>
      </listitem>

      <listitem>
        <para>Output of <command>man ip</command></para>
      </listitem>

      <listitem>
        <para>Output of <command>ip route help</command> and <command>ip rule
        help</command></para>
      </listitem>
    </itemizedlist>
  </warning>

  <section id="Support">
    <title>Multiple Internet Connection Support</title>

    <para>Shorewall includes limited support for multiple Internet
    connections. Limitations of this support are as follows:</para>

    <itemizedlist>
      <listitem>
        <para>It utilizes static routing configuration. If there is a change
        in the routing topopogy, Shorewall must be restarted.</para>
      </listitem>

      <listitem>
        <para>The routing changes are made and the route cache is purged when
        Shorewall is started <emphasis role="bold">and when Shorewall is
        restarted</emphasis> (unless you specify the "-n" option to
        <command>shorewall restart</command>). Ideally, restarting the packet
        filter should have no effect on routing.</para>
      </listitem>

      <listitem>
        <para>For most routing applications, <ulink
        url="http://www.quagga.net/">Quagga</ulink> is a better solution
        although it requires that your ISPs offer routing protocol
        support.</para>
      </listitem>
    </itemizedlist>

    <section id="Overview">
      <title>Overview</title>

      <para>Let's assume that a firewall is connected via two separate
      Ethernet interfaces to two different ISPs as in the following
      diagram.</para>

      <graphic align="center" fileref="images/TwoISPs.png" valign="middle" />

      <itemizedlist>
        <listitem>
          <para>eth0 connects to ISP1. The IP address of eth0 is
          206.124.146.176 and the ISP's gateway router has IP address
          206.124.146.254.</para>
        </listitem>

        <listitem>
          <para>eth1 connects to ISP 2. The IP address of eth1 is
          130.252.99.27 and the ISP's gateway router has IP address
          130.252.99.254.</para>
        </listitem>

        <listitem>
          <para>eth2 connects to the local LAN. Its IP configuration is not
          relevant to this discussion.</para>
        </listitem>
      </itemizedlist>

      <para>Each of these <firstterm>providers</firstterm> is described in an
      entry in the file <filename>/etc/shorewall/providers</filename>.</para>

      <para>Entries in <filename>/etc/shorewall/providers</filename> can
      specify that outgoing connections are to be load-balanced between the
      two ISPs. Entries in <filename>/etc/shorewall/tcrules</filename> and
      <filename>/etc/shorewall/route_rules</filename> can be used to direct
      particular outgoing connections to one ISP or the other. Use of
      <filename>/etc/shorewall/tcrules</filename> is not required for
      <filename>/etc/shorewall/providers</filename> to work, but in most
      cases, you must select a unique MARK value for each provider so
      Shorewall can set up the correct marking rules for you.</para>

      <para>When you use the <emphasis role="bold">track</emphasis> option in
      <filename>/etc/shorewall/providers</filename>, connections from the
      Internet are automatically routed back out of the correct interface and
      through the correct ISP gateway. This works whether the connection is
      handled by the firewall itself or if it is routed or port-forwarded to a
      system behind the firewall.</para>

      <para>Shorewall will set up the routing and will update the
      <filename>/etc/iproute2/rt_tables</filename> to include the table names
      and numbers of the tables that it adds.</para>

      <caution>
        <para>This feature uses <ulink url="traffic_shaping.htm">packet
        marking</ulink> to control the routing. As a consequence, there are
        some restrictions concerning entries in
        <filename>/etc/shorewall/tcrules</filename>:</para>

        <itemizedlist>
          <listitem>
            <para>Packet marking for traffic control purposes may not be done
            in the PREROUTING table for connections involving providers with
            'track' specified (see below).</para>
          </listitem>

          <listitem>
            <para>You may not use the SAVE or RESTORE options unless you also
            set HIGH_ROUTE_MARKS=Yes in
            <filename>/etc/shorewall/shorewall.conf</filename>.</para>
          </listitem>

          <listitem>
            <para>You may not use connection marking unless you also set
            HIGH_ROUTE_MARKS=Yes in
            <filename>/etc/shorewall/shorewall.conf</filename>.</para>
          </listitem>
        </itemizedlist>
      </caution>

      <para>The <filename>/etc/shorewall/providers</filename> file can also be
      used in other routing scenarios. See the <ulink
      url="Shorewall_Squid_Usage.html">Squid documentation</ulink> for an
      example.</para>
    </section>

    <section id="providers">
      <title>/etc/shorewall/providers File</title>

      <para>Entries in this file have the following columns. As in all
      Shorewall configuration files, enter "-" in a column if you don't want
      to enter any value.</para>

      <variablelist>
        <varlistentry>
          <term>NAME</term>

          <listitem>
            <para>The provider name. Must begin with a letter and consist of
            letters and digits. The provider name becomes the name of the
            generated routing table for this provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>NUMBER</term>

          <listitem>
            <para>A number between 1 and 252. This becomes the routing table
            number for the generated table for this provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>MARK</term>

          <listitem>
            <para>A mark value used in your /etc/shorewall/tcrules file to
            direct packets to this provider. Shorewall will also mark
            connections that have seen input from this provider with this
            value and will restore the packet mark in the PREROUTING CHAIN.
            Mark values must be in the range 1-255.</para>

            <para>Alternatively, you may set HIGH_ROUTE_MARKS=Yes in
            <filename>/etc/shorewall/shorewall.conf</filename>. This allows
            you to:</para>

            <itemizedlist>
              <listitem>
                <para>Use connection marks for traffic shaping, provided that
                you assign those marks in the FORWARD chain.</para>
              </listitem>

              <listitem>
                <para>Use mark values &gt; 255 for provider marks in this
                column. These mark values must be a multiple of 256 in the
                range 256-65280 (hex equivalent 0x100 - 0xFF00 with the
                low-order 8 bits being zero).</para>
              </listitem>
            </itemizedlist>

            <para>This column may be omitted if you don´t use packet marking
            to direct connections to a particular provider and you don´t
            specify <emphasis role="bold">track</emphasis> in the OPTIONS
            column.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>DUPLICATE</term>

          <listitem>
            <para>Gives the name or number of a routing table to duplicate.
            May be 'main' or the name or number of a previously declared
            provider. For most applications, you want to specify 'main' here.
            This field should be be specified as '-' when USE_DEFAULT_RT=Yes
            in <filename>shorewall.conf</filename></para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>INTERFACE</term>

          <listitem>
            <para>The name of the interface to the provider. Where multiple
            providers share the same interface (which is not recommended), you
            must follow the name of the interface by a colon (":") and the IP
            address assigned by this provider (e.g., eth0:206.124.146.176).
            See <link linkend="Shared">below</link> for additional
            considerations.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>GATEWAY</term>

          <listitem>
            <para>The IP address of the provider's Gateway router.</para>

            <para>You can enter <emphasis role="bold">detect</emphasis> here
            and Shorewall will attempt to automatically determine the gateway
            IP address.</para>

            <para><emphasis role="bold">Hint:</emphasis> <emphasis
            role="bold">"detect"</emphasis> is appropriate for use in cases
            where the interface named in the INTERFACE column is dynamically
            configured via DHCP etc. Be sure, however, that you don't have
            stale dhcp client state files in <filename
            class="directory">/var/lib/dhcpcd </filename>or
            <filename>/var/lib/dhclient-*.lease</filename> because Shorewall
            may try to use those stale files to determine the gateway
            address.</para>

            <para>The GATEWAY may be omitted (enter '-') for point-to-point
            links.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>OPTIONS</term>

          <listitem>
            <para>A comma-separated list from the following:</para>

            <variablelist>
              <varlistentry>
                <term>track</term>

                <listitem>
                  <para>If specified, connections FROM this interface are to
                  be tracked so that responses may be routed back out this
                  same interface.</para>

                  <para>You want to specify 'track' if Internet hosts will be
                  connecting to local servers through this provider. Any time
                  that you specify 'track', you will normally want to also
                  specify 'balance' (see below). 'track' will also ensure that
                  outgoing connections remain stay anchored to a single
                  provider and don't try to switch providers when route cache
                  entries expire.</para>

                  <para>Use of this feature requires that your kernel and
                  iptables include CONNMARK target and connmark match support
                  (<emphasis role="bold">Warning</emphasis>: Standard
                  <trademark>Debian</trademark> and
                  <trademark>Ubuntu</trademark> kernels may lack that
                  support!).</para>

                  <important>
                    <para>If you are using
                    <filename>/etc/shorewall/providers</filename> because you
                    have multiple Internet connections, we recommend that you
                    specify <emphasis role="bold">track</emphasis> even if you
                    don't need it. It helps maintain long-term connections in
                    which there are significant periods with no
                    traffic.</para>
                  </important>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>balance</term>

                <listitem>
                  <para>The providers that have <emphasis
                  role="bold">balance</emphasis> specified will get outbound
                  traffic load-balanced among them. Balancing will not be
                  perfect, as it is route based, and routes are cached. This
                  means that routes to often-used sites will always be over
                  the same provider.</para>

                  <para>By default, each provider is given the same weight (1)
                  . You can change the weight of a given provider by following
                  <emphasis role="bold">balance</emphasis> with "=" and the
                  desired weight (e.g., balance=2). The weights reflect the
                  relative bandwidth of the providers connections and should
                  be small numbers since the kernel actually creates
                  additional default routes for each weight increment.</para>

                  <important>
                    <para>If you are using
                    <filename>/etc/shorewall/providers</filename> because you
                    have multiple Internet connections, we recommend that you
                    specify <emphasis role="bold">balance</emphasis> even if
                    you don't need it. You can still use entries in
                    <filename>/etc/shorewall/tcrules</filename> and
                    <filename>/etc/shorewall/route_rules</filename> to force
                    all traffic to one provider or another.<note>
                        <para>If you don't heed this advice then please read
                        and follow the advice in <ulink
                        url="FAQ.htm#faq57">FAQ 57</ulink> and <ulink
                        url="FAQ.htm#faq58">FAQ 58</ulink>.</para>
                      </note></para>
                  </important>

                  <important>
                    <para>If you specify <emphasis
                    role="bold">balance</emphasis> and still find that all
                    traffic is going out through only one provider, you may
                    need to install a kernel built with
                    CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have
                    reported that this change has corrected similar
                    problems.</para>

                    <para>The SuSE 10.0 kernel is subject to this problem, and
                    <ulink
                    url="https://bugzilla.novell.com/show_bug.cgi?id=190908">
                    a kernel oops may result in this circumstance.</ulink>
                    SUSE 10.1 and SLES 10 have
                    CONFIG_IP_ROUTE_MULTIPATH_CACHED=n set by default. The
                    source of the problem seems to be <ulink
                    url="http://news.gmane.org/find-root.php?message_id=%3c00da01c5b35a%24b12b9860%241b00a8c0%40cruncher%3e">an
                    incompatibility between the LARTC patches and
                    CONFIG_IP_ROUTE_MULTIPATH_CACHED.</ulink></para>
                  </important>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>loose</term>

                <listitem>
                  <para>Do not include routing rules that force traffic whose
                  source IP is an address of the INTERFACE to be routed to
                  this provider. Useful for defining providers that are to be
                  used only when the appropriate packet mark is
                  applied.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>optional</term>

                <listitem>
                  <para>Shorewall will determine if this interface is up and
                  has a configured IP address. If it is not, a warning is
                  issued and this provider is not configured.</para>

                  <note>
                    <para><emphasis role="bold">optional</emphasis> is
                    designed to detect interface states that will cause
                    <command>shorewall start</command> or <command>shorewall
                    restart</command> to fail; just because an interface is in
                    a state that Shorewall can [re]start without error doesn't
                    mean that traffic can actually be sent through the
                    interface.</para>

                    <para>You can supply an 'isusable' <ulink
                    url="shorewall_extension_scripts.htm">extension
                    script</ulink> to extend Shorewall's interface state
                    detection. See also the <link
                    linkend="LinkMonitor">Gateway Monitoring and
                    Failover</link> section below.</para>
                  </note>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>src=<replaceable>source-address</replaceable></term>

                <listitem>
                  <para>Specifies the source address to use when routing to
                  this provider and none is known (the local client has bound
                  to the 0 address). May not be specified when an
                  <replaceable>address</replaceable> is given in the INTERFACE
                  column. If this option is not used, Shorewall substitutes
                  the primary IP address on the interface named in the
                  INTERFACE column.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>mtu=<replaceable>number</replaceable></term>

                <listitem>
                  <para>Specifies the MTU when forwarding through this
                  provider. If not given, the MTU of the interface named in
                  the INTERFACE column is assumed.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term><emphasis
                role="bold">fallback[=<replaceable>weight</replaceable>]</emphasis></term>

                <listitem>
                  <para>Indicates that a default route through the provider
                  should be added to the <firstterm>default</firstterm>
                  routing table (table 253). If a
                  <replaceable>weight</replaceable> is given, a balanced route
                  is added with the weight of this provider equal to the
                  specified <replaceable>weight</replaceable>. If the option
                  is given without a <replaceable>weight</replaceable>, a
                  separate default route is added through the provider's
                  gateway; the route has a metric equal to the provider's
                  NUMBER. The option is ignored with a warning message if
                  USE_DEFAULT_RT=Yes in
                  <filename>shorewall.conf</filename>.</para>
                </listitem>
              </varlistentry>
            </variablelist>

            <para>For those of you who are confused between<emphasis
            role="bold"> track</emphasis> and <emphasis
            role="bold">balance</emphasis>:</para>

            <itemizedlist>
              <listitem>
                <para><emphasis role="bold">track</emphasis> governs incoming
                connections (but is also useful for binding long-running
                connections to the same interface).</para>
              </listitem>

              <listitem>
                <para><emphasis role="bold">balance</emphasis> governs
                outgoing connections.</para>
              </listitem>
            </itemizedlist>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>COPY</term>

          <listitem>
            <para>A comma-separated list of interface names. Wildcards
            specified using an asterisk ("*") are permitted (e.g., tun*
            ).</para>

            <para>When you specify an existing table in the DUPLICATE column,
            Shorewall copies all routes through the interface specified in the
            INTERFACE column plus the interfaces listed in this column.
            Normally, you will list all interfaces on your firewall in this
            column except those Internet interfaces specified in the INTERFACE
            column of entries in this file.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </section>

    <section id="Providers">
      <title>What an entry in the Providers File Does</title>

      <para>Adding another entry in the providers file simply creates an
      alternate routing table for you (see the<ulink
      url="http://www.lartc.org"> LARTC Howto</ulink>). The table will usually
      contain two routes:</para>

      <orderedlist>
        <listitem>
          <para>A host route to the specified GATEWAY through the specified
          INTERFACE.</para>
        </listitem>

        <listitem>
          <para>A default route through the GATEWAY.</para>
        </listitem>
      </orderedlist>

      <para>Note that the first route is omitted if "-" is specified as the
      GATEWAY; in that case, the default route does not specify a gateway
      (point-to-point link).</para>

      <para>If the DUPLICATE column is non-empty, then routes from the table
      named in that column are copied into the new table. By default, all
      routes (except default routes) are copied. The set of routes copied can
      be restricted using the COPY column which lists the interfaces whose
      routes you want copied. You will generally want to include all local
      interfaces in this list. You should exclude the loopback interface (lo)
      and any interfaces that do not have an IP configuration. You should also
      omit interfaces like <emphasis role="bold">tun</emphasis> interfaces
      that are created dynamically. Traffic to networks handled by those
      interfaces should be routed through the main table using entries in
      <filename>/etc/shorewall/route_rules</filename> (see Example 2 <link
      linkend="Examples">below</link>) or by using <link
      linkend="USE_DEFAULT_RT">USE_DEFAULT_RT=Yes</link>.</para>

      <para>In addition:</para>

      <orderedlist>
        <listitem>
          <para>Unless <emphasis role="bold">loose</emphasis> is specified, an
          ip rule is generated for each IP address on the INTERFACE that
          routes traffic from that address through the associated routing
          table.</para>
        </listitem>

        <listitem>
          <para>If you specify <emphasis role="bold">track</emphasis>, then
          connections which have had at least one packet arrive on the
          interface listed in the INTERFACE column have their connection mark
          set to the value in the MARK column. In the PREROUTING chain,
          packets with a connection mark have their packet mark set to the
          value of the associated connection mark; packets marked in this way
          bypass any prerouting rules that you create in
          <filename>/etc/shorewall/tcrules</filename>. This ensures that
          packets associated with connections from outside are always routed
          out of the correct interface.</para>
        </listitem>

        <listitem>
          <para>If you specify <emphasis role="bold">balance</emphasis>, then
          Shorewall will replace the 'default' route with weight 100 in the
          'main' routing table with a load-balancing route among those
          gateways where <emphasis role="bold">balance</emphasis> was
          specified. So if you configure default routes, be sure that their
          weight is less than 100 or the route added by Shorewall will not be
          used.</para>
        </listitem>
      </orderedlist>

      <para>That's <emphasis role="bold">all</emphasis> that these entries do.
      You still have to follow the principle stated in the <ulink
      url="Shorewall_and_Routing.html">Shorewall Routing
      documentation</ulink>:</para>

      <orderedlist>
        <listitem>
          <para>Routing determines where packets are to be sent.</para>
        </listitem>

        <listitem>
          <para>Once routing determines where the packet is to go, the
          firewall (Shorewall) determines if the packet is allowed to go
          there.</para>
        </listitem>
      </orderedlist>

      <para>The bottom line is that if you want traffic to go out through a
      particular provider then you <emphasis>must </emphasis>mark that traffic
      with the provider's MARK value in
      <filename>/etc/shorewall/tcrules</filename> and you must do that marking
      in the PREROUTING chain; or, you must provide the appropriate rules in
      <filename>/etc/shorewall/route_rules</filename>.</para>
    </section>

    <section>
      <title>/etc/shorewall/masq and Multi-ISP</title>

      <para>If you masquerade a local network, you will need to add masquerade
      rules for both external interfaces. Referring to the diagram above, if
      each of the interfaces has only a single IP address and you have no
      systems with public IP addresses behind your firewall, then I suggest
      the following simple entries:</para>

      <programlisting>#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27</programlisting>

      <para>If you have a public subnet (for example 206.124.146.176/30)
      behind your firewall, then use exclusion:</para>

      <programlisting>#INTERFACE       SOURCE               ADDRESS
eth0             !206.124.146.176/29  206.124.146.176
eth1             0.0.0.0/0            130.252.99.27</programlisting>

      <para>Note that exclusion is only used on the interface corresponding to
      internal subnetwork.</para>

      <para>If you have multiple IP addresses on one of your interfaces, you
      can use a similar technique -- simple exclude the smallest network that
      contains all of those addresses from being masqueraded.</para>

      <warning>
        <para>Entries in <filename>/etc/shorewall/masq</filename> have no
        effect on which ISP a particular connection will be sent through. That
        is rather the purpose of entries in
        <filename>/etc/shorewall/tcrules</filename> and
        <filename>/etc/shorewall/route_rules</filename>.</para>
      </warning>
    </section>

    <section id="Martians">
      <title>Martians</title>

      <para>One problem that often arises with Multi-ISP configuration is
      'Martians'. If your Internet interfaces are configured with the
      <emphasis role="bold">routefilter</emphasis> option in
      <filename>/etc/shorewall/interfaces</filename> (remember that if you set
      that option, you should also select <emphasis
      role="bold">logmartians</emphasis>), then things may not work correctly
      and you will see messages like this:</para>

      <programlisting>Feb  9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 
Feb  9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00</programlisting>

      <para>The above message is somewhat awkwardly phrased. The source IP in
      this incoming packet was 64.86.88.116 and the destination IP address was
      206.124.146.176. Another gotcha is that the incoming packet has already
      had the destination IP address changed for DNAT or because the original
      outgoing connection was altered by an entry in
      <filename>/etc/shorewall/masq</filename> (SNAT or Masquerade). So the
      destination IP address (206.124.146.176) may not have been the
      destination IP address in the packet as it was initially
      received.</para>

      <para>There a couple of common causes for these problems:</para>

      <orderedlist>
        <listitem>
          <para>You have connected both of your external interfaces to the
          same hub/switch. Connecting multiple firewall interfaces to a common
          hub or switch is always a bad idea that will result in
          hard-to-diagnose problems.</para>
        </listitem>

        <listitem>
          <para>You are specifying both the <emphasis
          role="bold">loose</emphasis> and <emphasis
          role="bold">balance</emphasis> options on your provider(s). This can
          cause individual connections to ping-pong back and forth between the
          interfaces which is almost guaranteed to cause problems.</para>
        </listitem>

        <listitem>
          <para>You are redirecting traffic from the firewall system out of
          one interface or the other using packet marking in your
          <filename>/etc/shorewall/tcrules</filename> file. A better approach
          is to configure the application to use the appropriate local IP
          address (the IP address of the interface that you want the
          application to use). See <link linkend="Local">below</link>.</para>
        </listitem>
      </orderedlist>

      <para>If all else fails, remove the <emphasis
      role="bold">routefilter</emphasis> option from your external interfaces.
      If you do this, you may wish to add rules to log and drop packets from
      the Internet that have source addresses in your local networks. For
      example, if the local LAN in the above diagram is 192.168.1.0/24, then
      you would add this rule:</para>

      <programlisting>#ACTION          SOURCE                     DEST
DROP:info        net:192.168.1.0/24         all</programlisting>

      <para>Be sure the above rule is added before any other rules with
      <emphasis>net</emphasis> in the SOURCE column.</para>
    </section>

    <section id="Example1">
      <title id="Example">Example</title>

      <para>The configuration in the figure at the top of this section would
      be specified in <filename>/etc/shorewall/providers</filename> as
      follows.</para>

      <programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2</programlisting>

      <para>Other configuration files go something like this:</para>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …</programlisting>

      <para><filename>/etc/shorewall/policy</filename>:</para>

      <programlisting>#SOURCE    DESTINATION    POLICY     LIMIT:BURST
net        net            DROP</programlisting>

      <para><filename>/etc/shorewall/masq</filename>:</para>

      <programlisting>#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27</programlisting>
    </section>

    <section id="Applications">
      <title>Routing a Particular Application Through a Specific
      Interface</title>

      <para>This continues the example in the preceding section.</para>

      <para>Now suppose that you want to route all outgoing SMTP traffic from
      your local network through ISP 2. You would make this entry in <ulink
      url="traffic_shaping.htm">/etc/shorewall/tcrules</ulink> (and if you are
      running a version of Shorewall earlier than 3.0.0, you would set
      TC_ENABLED=Yes in <ulink
      url="???">/etc/shorewall/shorewall.conf</ulink>).</para>

      <programlisting>#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2:P             &lt;local network&gt; 0.0.0.0/0       tcp     25</programlisting>

      <para>Note that traffic from the firewall itself must be handled in a
      different rule:</para>

      <programlisting>#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                               PORT(S)
2               $FW             0.0.0.0/0       tcp     25</programlisting>
    </section>

    <section id="morethan2">
      <title>More than 2 Providers</title>

      <para>When there are more than two providers, you need to extend the
      two-provider case in the expected way:</para>

      <orderedlist>
        <listitem>
          <para>For each external address, you need an entry in
          <filename>/etc/shorewall/masq</filename> to handle the case where a
          connection using that address as the SOURCE is sent out of the
          interfaces other than the one that the address is configured
          on.</para>
        </listitem>

        <listitem>
          <para>For each external interface, you need to add an entry to
          <filename>/etc/shorewall/masq</filename>.</para>
        </listitem>
      </orderedlist>

      <para>If we extend the above example to add eth3 with IP address
      16.105.78.4 with gateway 16.105.78.254, then:</para>

      <para><filename>/etc/shorewall/providers</filename>:<programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2
ISP3    3       3       main            eth3            16.105.78.254   track,balance    eth2</programlisting></para>

      <para><filename>/etc/shorewall/masq</filename>:<programlisting>#INTERFACE       SUBNET            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27
eth3             0.0.0.0/0         16.105.78.4</programlisting></para>
    </section>

    <section id="Local">
      <title>Applications running on the Firewall</title>

      <para>As <link linkend="Applications">noted above</link>, separate
      entries in <filename>/etc/shorewall/tcrules</filename> are required for
      traffic originating from the firewall.</para>

      <para>Experience has shown that in some cases, problems occur with
      applications running on the firewall itself. This is especially true
      when you have specified <emphasis role="bold">routefilter</emphasis> on
      your external interfaces in /etc/shorewall/interfaces (see <link
      linkend="Martians">above</link>). When this happens, it is suggested
      that you have the application use specific local IP addresses rather
      than 0.</para>

      <para>Examples:</para>

      <itemizedlist>
        <listitem>
          <para>Squid: In <filename>squid.conf</filename>, set <emphasis
          role="bold">tcp_outgoing_address</emphasis> to the IP address of the
          interface that you want Squid to use.</para>
        </listitem>

        <listitem>
          <para>In OpenVPN, set <emphasis role="bold">local
          </emphasis>(<emphasis role="bold">--local</emphasis> on the command
          line) to the IP address that you want the server to receive
          connections on.</para>
        </listitem>
      </itemizedlist>

      <para>Note that some traffic originating on the firewall doesn't have a
      SOURCE IP address before routing. At least one Shorewall user reports
      that an entry in <filename>/etc/shorewall/route_rules</filename> with
      'lo' in the SOURCE column seems to be the most reliable way to direct
      such traffic to a particular ISP.</para>
    </section>

    <section id="route_rules">
      <title>/etc/shorewall/route_rules</title>

      <para>The <filename>route_rules</filename> file allows assigning certain
      traffic to a particular provider just as entries in the
      <filename>tcrules</filename> file. The difference between the two files
      is that entries in <filename>route_rules</filename> are independent of
      Netfilter.</para>

      <section id="Routing_rules">
        <title>Routing Rules</title>

        <para>Routing rules are maintained by the Linux kernel and can be
        displayed using the <command>ip rule ls</command> command. When
        routing a packet, the rules are processed in turn until the packet is
        successfully routed.</para>

        <programlisting>gateway:~ # <command>ip rule ls</command>
0:      from all lookup local                &lt;=== Local (to the firewall) IP addresses
10001:  from all fwmark 0x1 lookup Blarg     &lt;=== This and the next rule are generated by the
10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
20000:  from 206.124.146.176 lookup Blarg    &lt;=== This and the next rule are generated unless
20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
32766:  from all lookup main                 &lt;=== This is the routing table shown by 'iproute -n'
32767:  from all lookup default              &lt;=== This table is usually empty
gateway:~ #</programlisting>

        <para>In the above example, there are two providers: Blarg and Comcast
        with MARK 1 going to Blarg and mark 2 going to Comcast.</para>
      </section>

      <section id="route_rules_columns">
        <title>Columns in the route_rules file</title>

        <para>Columns in the file are:</para>

        <variablelist>
          <varlistentry>
            <term>SOURCE (Optional)</term>

            <listitem>
              <para>An ip address (network or host) that matches the source IP
              address in a packet. May also be specified as an interface name
              optionally followed by ":" and an address. If the device 'lo' is
              specified, the packet must originate from the firewall
              itself.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>DEST (Optional)</term>

            <listitem>
              <para>An ip address (network or host) that matches the
              destination IP address in a packet.</para>

              <para>If you choose to omit either SOURCE or DEST, place "-" in
              that column. Note that you may not omit both SOURCE and
              DEST.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>PROVIDER</term>

            <listitem>
              <para>The provider to route the traffic through. May be
              expressed either as the provider name or the provider
              number.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>PRIORITY</term>

            <listitem>
              <para>The rule's priority which determines the order in which
              the rules are processed.</para>

              <para>1000-1999 Before Shorewall-generated 'MARK' rules</para>

              <para>11000- 11999 After 'MARK' rules but before
              Shorewall-generated rules for ISP interfaces.</para>

              <para>26000-26999 After ISP interface rules but before 'default'
              rule.</para>

              <para>Rules with equal priority are applied in the order in
              which they appear in the file.</para>
            </listitem>
          </varlistentry>
        </variablelist>
      </section>

      <section>
        <title>Multi-ISP and VPN</title>

        <para>For those VPN types that use routing to direct traffic to remote
        VPN clients (including but not limited to OpenVPN in routed mode and
        PPTP), the VPN software adds a host route to the <emphasis
        role="bold">main</emphasis> table for each VPN client. The best
        approach is to use USE_DEFAULT_RT=Yes as described <link
        linkend="USE_DEFAULT_RT">below</link>. If that isn't possible, you
        must add a routing rule in the 1000-1999 range to specify the
        <emphasis role="bold">main</emphasis> table for traffic addressed to
        those clients. See<link linkend="Openvpn"> Example 2</link>
        below.</para>

        <para>If you have an IPSEC gateway on your firewall, be sure to
        arrange for ESP packets to be routed out of the same interface that
        you have configured your keying daemon to use.</para>
      </section>

      <section id="Examples">
        <title>Examples</title>

        <para><emphasis role="bold">Example 1:</emphasis> You want all traffic
        entering the firewall on eth1 to be routed through Comcast.</para>

        <programlisting>#SOURCE            DEST      PROVIDER        PRIORITY
eth1               -         Comcast         1000</programlisting>

        <para>With this entry, the output of <command>ip rule ls</command>
        would be as follows.</para>

        <para><programlisting>gateway:~ # <command>ip rule ls</command>
0:      from all lookup local
<emphasis role="bold">1000:   from all iif eth1 lookup Comcast</emphasis>
10001:  from all fwmark 0x1 lookup Blarg
10002:  from all fwmark 0x2 lookup Comcast
20000:  from 206.124.146.176 lookup Blarg
20256:  from 24.12.22.33 lookup Comcast
32766:  from all lookup main
32767:  from all lookup default
gateway:~ #</programlisting>Note that because we used a priority of 1000, the
        test for <filename class="devicefile">eth1</filename> is inserted
        before the fwmark tests.</para>

        <para id="Openvpn"><emphasis role="bold">Example 2:</emphasis> You use
        OpenVPN (routed setup w/tunX) in combination with multiple providers.
        In this case you have to set up a rule to ensure that the OpenVPN
        traffic is routed back through the tunX interface(s) rather than
        through any of the providers. 10.8.0.0/24 is the subnet chosen in your
        OpenVPN configuration (server 10.8.0.0 255.255.255.0).</para>

        <programlisting>#SOURCE                 DEST            PROVIDER        PRIORITY
-                       10.8.0.0/24     main            1000</programlisting>
      </section>
    </section>

    <section id="USE_DEFAULT_RT">
      <title>USE_DEFAULT_RT</title>

      <para>USE_DEFAULT_RT is an option in <ulink
      url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5).</para>

      <para>One of the drawbacks of the Multi-ISP support as described in the
      preceding sections is that changes to the main table made by
      applications are not added to the individual provider tables. This makes
      route rules such as described in <link linkend="Openvpn">one of the
      examples above</link> necessary.</para>

      <para>USE_DEFAULT_RT=Yes works around that problem by passing packets
      through the main table first rather than last. This has a number of
      implications:</para>

      <orderedlist>
        <listitem>
          <para>Both the DUPLICATE and the COPY columns in the providers file
          must remain empty or contain "-". The individual provider routing
          tables generated when USE_DEFAULT_RT=Yes contain only a host route
          to the gateway and a default route via the gateway.</para>
        </listitem>

        <listitem>
          <para>The <emphasis role="bold">balance</emphasis> option is assumed
          for all interfaces that do not have the <emphasis
          role="bold">loose</emphasis> option. When you want both <emphasis
          role="bold">balance</emphasis> and <emphasis
          role="bold">loose</emphasis>, both must be specified.</para>
        </listitem>

        <listitem>
          <para>The default route generated by Shorewall is added to the
          <emphasis>default</emphasis> routing table (253) rather than to the
          main routing table (254).</para>
        </listitem>

        <listitem>
          <para>Packets are sent through the main routing table by a routing
          rule with priority 999. In ), the priority range 1-998 may be used
          for inserting rules that bypass the main table.</para>
        </listitem>

        <listitem>
          <para>You should disable all default route management outside of
          Shorewall. If a default route is inadvertently added to the main
          table while Shorewall is started, then all policy routing will stop
          working except for those routing rules in the priority range
          1-998.</para>
        </listitem>

        <listitem>
          <para>For ppp interfaces, the GATEWAY may remain unspecified ("-").
          For those interfaces managed by dhcpcd or dhclient, you may specify
          'detect' in the GATEWAY column; Shorewall will use the dhcp client's
          database to determine the gateway IP address. All other interfaces
          must have a GATEWAY specified explicitly.</para>
        </listitem>
      </orderedlist>

      <para>Although 'balance' is automatically assumed when
      USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider
      except when you explicitly direct it to use the other provider via
      <ulink
      url="manpages/shorewall-route_rules.html">shorewall-route_rules</ulink>
      (5) or <ulink
      url="manpages/shorewall-tcrules.html">shorewall-tcrules</ulink>
      (5).</para>

      <para>Example (send all traffic through the 'shorewall' provider unless
      otherwise directed).</para>

      <para>/etc/shorewall/providers:<programlisting>#NAME    NUMBER MARK DUPLICATE INTERFACE GATEWAY       OPTIONS
linksys       1    1    -        wlan0   172.20.1.1    track,balance=1,optional
shorewall     2    2    -        eth0    192.168.1.254 track,balance=2,optional</programlisting>/etc/shorewall/route_rules:<programlisting>#SOURCE     DEST      PROVIDER        PRIORITY
-           -         shorewall       11999</programlisting></para>
    </section>

    <section id="LinkMonitor">
      <title>Gateway Monitoring and Failover</title>

      <para>There are a couple of options available for monitoring the status
      of provider links and taking action when a failure occurs.</para>

      <section id="swping">
        <title>SWPING</title>

        <para>Shorewall includes a sample monitoring script
        <filename>swping</filename>. The <filename>swping</filename> file is
        available in the main directory contained in the Shorewall-common
        tarball and is included in the Shorewall-common documentation
        directory on the Shorewall-common RPM. The script is inspired by
        Angsuman Chakraborty's <ulink
        url="http://blog.taragana.com/index.php/archive/how-to-load-balancing-failover-with-dual-multi-wan-adsl-cable-connections-on-linux/">gwping</ulink>
        script.</para>

        <important>
          <para>These samples are offered <emphasis>as is</emphasis> — they
          work for me but I don't make any claim that they will work for
          anyone else. But if you have a need for automated link monitoring,
          they offer you a place to start.</para>
        </important>

        <para>The script should be copied to a directory on root's PATH such
        as <filename>/usr/local/sbin/</filename>.</para>

        <para>The script works by sending pings to <emphasis>target</emphasis>
        IP addresses through each external interface. These targets must not
        depend on any routes other than those that are present in the main
        routing table. That ensures that a route is available to the target
        even when the target's interface is not working and Shorewall has
        omitted it from the routing configuration. An interface is assumed to
        be <firstterm>up</firstterm> when a specified number (UP_COUNT) of
        consecutive ping operations succeed. Similarly, an interface is
        assumed to be <firstterm>down</firstterm> when a specified number
        (DOWN_COUNT) of consecutive ping operations fail. You can specify the
        interval between pings (PING_INTERVAL).</para>

        <para>The script monitors two interfaces but it is a trivial exercise
        to extend it to more than two. At the top are a number of variables to
        set:</para>

        <programlisting>#
# IP family -- 4 or 6
#
FAMILY=4
#
# The commands to run when the status of a line changes. Multiple commands may be specified 
# when separated by semicolons (";")
#
COMMAND=

...
#
# Interfaces to monitor -- you may use shell variables from your params file
#
IF1=eth0
IF2=eth1
#
# Sites to Ping. Must depend only on routes in the 'main' routing table. If not specified,
# the interface is assumed to be managed by dhcpcd and the script uses the gateway address
# from /var/lib/dhcpcd/dhcpcd-${IFx}.info
#
TARGET1=
TARGET2=
#
# How often to ping
#
PING_INTERVAL=5
#
# Value for ping's -W option
#
PING_TIMEOUT=2
#
# This many successive pings must succeed for the interface to be marked up when it is down
#
UP_COUNT=5
#
# This many successive pings must fail for the interface to be marked down when it is up
#
DOWN_COUNT=2</programlisting>

        <para>If you leave COMMAND empty, the script sets its value
        automatically depending on whether Shorewall-lite is installed.</para>

        <para>When the status of an interface changes:</para>

        <itemizedlist>
          <listitem>
            <para>For each interface, a file is placed in /etc/shorewall to
            record the status of the interface: either 0 (UP) or 1 (DOWN). The
            name of the file is
            <filename><replaceable>interface</replaceable>.status</filename>
            where <replaceable>interface</replaceable> is the interface (e.g.,
            <filename>eth0.status</filename>).</para>
          </listitem>

          <listitem>
            <para>A <command>shorewall -f restart</command> command is
            executed (<command>shorewall-lite restart</command>, if
            Shorewall-lite is installed).</para>
          </listitem>

          <listitem>
            <para>The contents of the main routing table are displayed.</para>
          </listitem>
        </itemizedlist>

        <para>The .status files are intended to be used with the following
        <filename>/etc/shorewall/isusable</filename> script.<programlisting>local status=0

case $1 in
    <emphasis role="bold">eth0|eth1</emphasis>)
        [ -f /etc/shorewall/${1}.status ] &amp;&amp; status=$(cat /etc/shorewall/${1}.status)
        ;;
esac

return $status</programlisting></para>

        <para>Be sure that you modify the interface names to match your
        configuration.</para>

        <para>Also included is a sample init script
        (<filename>swping.init</filename>) to start the monitoring daemon.
        Copy it to<filename> /etc/init.d/swping</filename> and use your
        distribution's SysV init tools to cause it to be run at boot. It works
        on <trademark>OpenSuSE</trademark> 11.0 -- YMMV. Modify the PROG and
        STATEDIR variables as needed.</para>

        <para>As an alternative to using the init script, you can add the
        following to <filename>/etc/shorewall/started</filename>:</para>

        <programlisting>if [ "$COMMAND" = start ]; then
   killall -9 swping 2&gt; /dev/null #be sure that there are none left running
   /usr/local/sbin/swping &amp;
fi</programlisting>

        <para>and add this to
        <filename>/etc/shorewall/stopped</filename>.</para>

        <para><programlisting>if [ "$COMMAND" = stop -o "$COMMAND" = clear ]; then
   killall -9 swping 2&gt; /dev/null
fi</programlisting></para>

        <para>This simple script has a number of limitations:</para>

        <orderedlist>
          <listitem>
            <para>It only works on IPv4 or IPv6 but not both at once. So if
            you want to monitor both IPv4 and IPv6, you need to clone the
            script are run two copies; one for IPv4 and one for IPv6.</para>
          </listitem>

          <listitem>
            <para>It can only detect the gateway for interfaces managed by
            dhcpcd.</para>
          </listitem>

          <listitem>
            <para>It's method of determining whether an interface is up or
            down is crude. You will normally specify the default gateway for
            each provider as the sites to ping and being able to ping the
            default gateway is not a surefire indication that the provider is
            usable. The method of determining whether a site is up or down is
            also crude.</para>
          </listitem>

          <listitem>
            <para>Because of the crudeness of the algorithm, hysteresis may
            occur.</para>
          </listitem>

          <listitem>
            <para>It is tricky to configure a system such that the system
            works correctly when one of its providers is down unless you
            largely don't care which interface is used.</para>
          </listitem>
        </orderedlist>
      </section>

      <section id="lsm">
        <title>Link Status Monitor (LSM)</title>

        <para><ulink url="http://lsm.foobar.fi/">Link Status Monitor</ulink>
        was written by Mika Ilmaranta &lt;ilmis at nullnet.fi&gt; and performs
        more sophisticated monitoring than the simple swping script described
        in the preceding section.</para>

        <para>I personally use LSM here at shorewall.net (configuration is
        described <link linkend="Complete">below</link>). Here are my relevant
        configuration files:</para>

        <para><filename>/etc/shorewall/isusable</filename>:</para>

        <programlisting>local status
status=0

case $1 in
    eth0|eth3)
        [ -f /etc/shorewall/${1}.status ] &amp;&amp; status=$(cat /etc/shorewall/${1}.status)
        ;;
esac

return $status</programlisting>

        <para><filename>/etc/shorewall/started</filename>:</para>

        <programlisting>###############################################################################
# My 'restored' script calls this one if there is no lsm process running
###############################################################################
if [ "$COMMAND" = start -o "$COMMAND" = restore ]; then
   killproc lsm 2&gt; /dev/null
   cat &lt;&lt;EOF &gt; /etc/lsm/shorewall.conf
connection {
    name=Avvanta
    checkip=206.124.146.254
    device=eth0
    ttl=2
}

connection {
    name=Comcast
    checkip=$ETH3_GATEWAY
    device=eth3
    ttl=1
}
EOF
   rm -f /etc/shorewall/*.status
   /usr/sbin/lsm /etc/lsm/lsm.conf &gt;&gt; /var/log/lsm
fi</programlisting>

        <para>eth3 has a dynamic IP address so I need to use the
        Shorewall-detected gateway address ($ETH3_GATEWAY).</para>

        <para><filename>/etc/shorewall/restored</filename>:</para>

        <programlisting>if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
   run_started_exit
fi</programlisting>

        <para><filename>/etc/lsm/lsm.conf</filename>:</para>

        <programlisting>#
# Defaults for the connection entries
#
defaults {
  name=defaults
  checkip=127.0.0.1
  eventscript=/etc/lsm/script
  max_packet_loss=20
  max_successive_pkts_lost=7
  min_packet_loss=5
  min_successive_pkts_rcvd=10
  interval_ms=2000
  timeout_ms=2000
  warn_email=teastep@shorewall.net
  check_arp=0
  sourceip=
  ttl=64
}

include /etc/lsm/shorewall.conf</programlisting>

        <para><filename>/etc/lsm/script</filename><programlisting>#!/bin/sh
#
# (C) 2009 Mika Ilmaranta &lt;ilmis@nullnet.fi&gt;
# (C) 2009 Tom Eastep &lt;teastep@shorewall.net&gt;
#
# License: GPLv2
#

STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}

cat &lt;&lt;EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}

Hi,

Connection ${NAME} is now ${STATE}.

Following parameters were passed:
newstate     = ${STATE}
name         = ${NAME}
checkip      = ${CHECKIP}
device       = ${DEVICE}
warn_email   = ${WARN_EMAIL}

Packet counters:
replied      = ${REPLIED} packets replied
waiting      = ${WAITING} packets waiting for reply
timeout      = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late   = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd    = ${CONS_RCVD} consecutively received replies in sequence
cons_wait    = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss    = ${CONS_MISS} consecutive packets that have timed out
avg_rtt      = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this

Your LSM Daemon

EOM

[ ${STATE} = up ] &amp;&amp; state=0 || state=1

echo $state &gt; /etc/shorewall/${DEVICE}.status

/sbin/shorewall -f restart &gt;&gt; /var/log/lsm 2&gt;&amp;1

/sbin/shorewall show routing &gt;&gt; /var/log/lsm

exit 0;

#EOF</programlisting>:</para>
      </section>
    </section>

    <section id="Shared">
      <title>Two Providers Sharing an Interface</title>

      <para>Shared interface support has the following characteristics"</para>

      <orderedlist numeration="loweralpha">
        <listitem>
          <para>Only Ethernet (or Ethernet-like) interfaces can be used. For
          inbound traffic, the MAC addresses of the gateway routers are used
          to determine which provider a packet was received through. Note that
          only routed traffic can be categorized using this technique.</para>
        </listitem>

        <listitem>
          <para>You must specify the address on the interface that corresponds
          to a particular provider in the INTERFACE column by following the
          interface name with a colon (":") and the address.</para>
        </listitem>

        <listitem>
          <para>Entries in <filename>/etc/shorewall/masq</filename> must be
          qualified by the provider name (or number).</para>
        </listitem>

        <listitem>
          <para>This feature requires Realm Match support in your kernel and
          iptables.</para>
        </listitem>

        <listitem>
          <para>You must add route_rules entries for networks that are
          accessed through a particular provider.</para>
        </listitem>

        <listitem>
          <para>If you have additional IP addresses through either provider,
          you must add <filename>route_rules</filename> to direct traffic FROM
          each of those addresses through the appropriate provider.</para>
        </listitem>

        <listitem>
          <para>You must manually add MARK rules for traffic known to come
          from each provider.</para>
        </listitem>

        <listitem>
          <para>You must specify a gateway IP address in the GATEWAY column
          of<filename> /etc/shorewall/providers</filename>; <emphasis
          role="bold">detect</emphasis> is not permitted.</para>
        </listitem>
      </orderedlist>

      <para>Taken together, b. and h. effectively preclude using this
      technique with dynamic IP addresses.</para>

      <para>Example:</para>

      <para>This is our home network circa fall 2008. We have two Internet
      providers:</para>

      <orderedlist>
        <listitem>
          <para>Comcast -- Cable modem with one dynamic IP address.</para>
        </listitem>

        <listitem>
          <para>Avvanta -- ADSL with 5 static IP addresses.</para>
        </listitem>
      </orderedlist>

      <para>Because the old <trademark>Compaq</trademark>
      <trademark>Presario</trademark> that I use for a firewall only has three
      PCI slots and no onboard Ethernet, it doesn't have enough Ethernet
      controllers to support both providers. So I use a Linksys WRT300n pre-N
      router as a gateway to Comcast. Note that because the Comcast IP address
      is dynamic, I could not share a single firewall interface between the
      two providers directly.</para>

      <para>On my personal laptop (ursa), I have 9 virtual machines running
      various Linux distributions. <emphasis>It is the Shorewall configuration
      on ursa that I will describe here</emphasis>.</para>

      <para>Below is a diagram of our network:<graphic align="center"
      fileref="images/Network2008a.png" /></para>

      <para>The local wired network in my office is connected to both gateways
      and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast
      gateway has local IP address 172.20.1.1 while the Avvanta gateway has
      local IP address 172.20.1.254. Ursa's eth0 interface has a single IP
      address (172.20.1.130).</para>

      <para>This configuration uses USE_DEFAULT_RT=Yes in
      <filename>shorewall.conf </filename>(see <link
      linkend="USE_DEFAULT_RT">above</link>).</para>

      <para>Here is the <filename>providers</filename> file:</para>

      <programlisting>#NAME          NUMBER   MARK DUPLICATE INTERFACE            GATEWAY      OPTIONS                            COPY
comcast        1        1    -         eth0:172.20.1.130    172.20.1.1   track,loose,balance,optional
avvanta        2        2    -         eth0:172.20.1.130    172.20.1.254 track,optional,loose
wireless       3        3    -         wlan0                172.20.1.1   track,optional</programlisting>

      <para>Several things to note:</para>

      <orderedlist>
        <listitem>
          <para>172.20.1.130 is specified as the <filename
          class="devicefile">eth0</filename> IP address for both
          providers.</para>
        </listitem>

        <listitem>
          <para>Both wired providers have the <emphasis
          role="bold">loose</emphasis> option. This prevents Shorewall from
          automatically generating routing rules based on the source IP
          address.</para>
        </listitem>

        <listitem>
          <para>Only <emphasis role="bold">comcast</emphasis> has the
          <emphasis role="bold">balance</emphasis> option. With
          USE_DEFAULT_RT=yes, that means that <emphasis
          role="bold">comcast</emphasis> will be the default provider. While
          <emphasis role="bold">balance</emphasis> is the default, with
          USE_DEFAULT_RT=Yes, it must be specified explicitly when <emphasis
          role="bold">loose</emphasis> is also specified.</para>
        </listitem>

        <listitem>
          <para>I always disable the <emphasis role="bold">wireless</emphasis>
          interface when the laptop is connected to the wired network.</para>
        </listitem>

        <listitem>
          <para>I use a different Shorewall configuration when I take the
          laptop on the road.</para>
        </listitem>
      </orderedlist>

      <para>Here is the route_rules file:<programlisting>#SOURCE                 DEST                    PROVIDER        PRIORITY
-                       206.124.146.176/31      avvanta         1000
-                       206.124.146.178/31      avvanta         1000
-                       206.124.146.180/32      avvanta         1000</programlisting></para>

      <para>Those rules direct traffic to the five static Avvanta IP addresses
      (only two are currently used) through the <emphasis
      role="bold">avvanta</emphasis> provider.</para>

      <para>Here is the tcrules file (MARK_IN_FORWARD_CHAIN=No in
      <filename>shorewall.conf</filename>):<programlisting>#MARK           SOURCE          DEST            PROTO   PORT(S)         CLIENT  USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
#                                                                       PORT(S) 
2               $FW             0.0.0.0/0       tcp     21
2               $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
2               $FW             0.0.0.0/0       tcp     119</programlisting></para>

      <para>These rules:</para>

      <itemizedlist>
        <listitem>
          <para>Use <emphasis role="bold">avvanta</emphasis> for FTP.</para>
        </listitem>

        <listitem>
          <para>Use <emphasis role="bold">avvanta</emphasis> for NTTP</para>
        </listitem>
      </itemizedlist>

      <para>The remaining files are for a rather standard two-interface config
      with a bridge as the local interface.</para>

      <para><filename>zones</filename>:<programlisting>#ZONE   IPSEC   OPTIONS                 IN                      OUT
#       ONLY                            OPTIONS                 OPTIONS
fw      firewall
net     ipv4
kvm     ipv4</programlisting><filename>policy</filename>:<programlisting>net             net             NONE
fw              net             ACCEPT
fw              kvm             ACCEPT
kvm             all             ACCEPT
net             all             DROP            info
all             all             REJECT          info</programlisting></para>

      <para>interfaces:<programlisting>#ZONE    INTERFACE      BROADCAST       OPTIONS                 GATEWAY
#
net     eth0            detect          dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore
net     wlan0           detect          dhcp,tcpflags,routefilter,blacklist,logmartians,optional
kvm     br0             detect          routeback       #Virtual Machines</programlisting><note>
          <para><filename class="devicefile">wlan0</filename> is the wireless
          adapter in the notebook. Used when the laptop is in our home but not
          connected to the wired network.</para>
        </note></para>

      <para>masq:<programlisting>#INTERFACE              SUBNET          ADDRESS         PROTO   PORT(S) IPSEC
eth0                    192.168.0.0/24
wlan0                   192.168.0.0/24</programlisting><note>
          <para>Because the firewall has only a single external IP address, I
          don't need to specify the providers in the masq rules.</para>
        </note></para>
    </section>
  </section>

  <section id="Complete">
    <title>A Complete Working Example</title>

    <para>This section describes the network at shorewall.net early in 2009.
    The configuration is as follows:</para>

    <itemizedlist>
      <listitem>
        <para>Two providers:</para>

        <itemizedlist>
          <listitem>
            <para>Avvanta -- A slow (1.5mb/384kb) DSL service with 5 static IP
            addresses.</para>
          </listitem>

          <listitem>
            <para>Comcast -- A fast (20mb/10mb) Cable circuit with a single
            <emphasis>dynamic</emphasis> address.</para>
          </listitem>
        </itemizedlist>
      </listitem>

      <listitem>
        <para>A local network consisting of wired and wireless client systems.
        A Linksys WRT300N wireless router is used as an access point for the
        wireless hosts.</para>
      </listitem>

      <listitem>
        <para>A DMZ hosting a single server (lists.shorewall.net aka
        www1.shorewall.net, ftp1.shorewall.net,etc.)</para>
      </listitem>
    </itemizedlist>

    <para>The network is pictured in the following diagram:</para>

    <graphic align="center" fileref="images/Network2009.png" />

    <para>Because of the speed of the cable provider, all traffic uses that
    provider unless there is a specific need for the traffic to use the DSL
    line.</para>

    <itemizedlist>
      <listitem>
        <para>Responses to connections from the Internet to one of the DSL IP
        addresses -- the <emphasis role="bold">track</emphasis> option takes
        care of that.</para>
      </listitem>

      <listitem>
        <para>Connections initiated by the server and connections requested by
        clients on the firewall that have bound their local socket to one of
        the DSL IP addresses. Two entries in
        <filename>/etc/shorewall/route_rules</filename> take care of that
        traffic.</para>
      </listitem>
    </itemizedlist>

    <para>As a consequence, I have disabled all route filtering on the
    firewall and only use the <emphasis role="bold">balance</emphasis> option
    in <filename>/etc/shorewall/providers</filename> on the Comcast provider
    whose default route in the main table is established by DHCP. By
    specifying the <emphasis role="bold">fallback</emphasis> option on
    Avvanta, I ensure that there is still a default route if Comcast is down.
    <link linkend="lsm">lsm</link> is used to monitor the links.</para>

    <para><filename>/etc/sysctl.conf</filename>:</para>

    <programlisting>net.ipv4.conf.all.rp_filter = 0</programlisting>

    <para><filename>/etc/shorewall/shorewall.conf</filename>:</para>

    <programlisting>ROUTE_FILTER=No
RESTORE_DEFAULT_ROUTE=No</programlisting>

    <para>RESTORE_DEFAULT_ROUTE=No causes the default route in the main table
    to be deleted when the Comcast link is unavailable. That way, the default
    route in the default table will be used until Comcast is available
    again.</para>

    <para><filename>/etc/shorewall/providers</filename>:</para>

    <programlisting>#NAME  NUMBER MARK  DUPLICATE INTERFACE GATEWAY         OPTIONS                 COPY
Avvanta     1 0x100 main      eth0      206.124.146.254 track,loose,fallback    eth2,eth4,tun*
Comcast     2 0x200 main      eth3      detect          track,balance           eth2,eth4,tun*
#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE</programlisting>

    <para>The <emphasis role="bold">loose</emphasis> option on Avvanta results
    in fewer routing rules. The first two routing rules below insure that all
    traffic from Avvanta-assigned IP addresses is sent via the Avvanta
    provider. The 'tun*' included in the COPY column is there because I run a
    routed OpenVPN server on the firewall.</para>

    <para><filename>/etc/shorewall/route_rules</filename>:</para>

    <programlisting>#SOURCE            DEST           PROVIDER PRIORITY
-                  172.20.0.0/24  main     1000    # Addresses assigned by routed OpenVPN server
206.124.146.176/30 -              Avvanta  26000
206.124.146.180    -              Avvanta  26000
-                  216.168.3.44   Avvanta  26000   # Avvanta NNTP Server -- verifies source IP address
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE</programlisting>

    <para>The <filename>/etc/shorewall/route_rules </filename>entries provide
    all of the provider selection necessary so my
    <filename>/etc/shorewall/tcrules</filename> file is used exclusively for
    traffic shaping of the Avvanta line. Note that I still need to provide
    values in the MARK colum of <filename>/etc/shorewall/providers</filename>
    because I specify <emphasis role="bold">track</emphasis> on both
    providers.</para>

    <para>Here is the output of <command>shorewall show
    routing</command>:</para>

    <programlisting>Routing Rules

0:     from all lookup local
1000:  from all to 172.20.0.0/24 lookup main 
10000: from all fwmark 0x100 lookup Avvanta 
10001: from all fwmark 0x200 lookup Comcast 
20256: from 71.227.156.229 lookup Comcast 
26000: from 206.124.146.176/30 lookup Avvanta 
26000: from 206.124.146.180 lookup Avvanta 
26000: from all to 216.168.3.44 lookup Avvanta 
32766: from all lookup main 
32767: from all lookup default 

Table Avvanta:

206.124.146.254 dev eth0  scope link  src 206.124.146.176 
206.124.146.177 dev eth4  scope link 
172.20.1.0/24 dev eth2  proto kernel  scope link  src 172.20.1.254 
206.124.146.0/24 dev eth0  proto kernel  scope link  src 206.124.146.176 
169.254.0.0/16 dev eth0  scope link 
default via 206.124.146.254 dev eth0  src 206.124.146.176 

Table Comcast:

206.124.146.177 dev eth4  scope link 
71.227.156.1 dev eth3  scope link  src 71.227.156.229 
172.20.1.0/24 dev eth2  proto kernel  scope link  src 172.20.1.254 
71.227.156.0/23 dev eth3  proto kernel  scope link  src 71.227.156.229 
default via 71.227.156.1 dev eth3  src 71.227.156.229 

Table default:

default via 206.124.146.254 dev eth0 metric 1

Table local:

broadcast 127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1 
broadcast 172.20.1.0 dev eth2  proto kernel  scope link  src 172.20.1.254 
broadcast 206.124.146.255 dev eth0  proto kernel  scope link  src 206.124.146.176 
local 206.124.146.179 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 206.124.146.178 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 206.124.146.176 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 206.124.146.176 dev eth4  proto kernel  scope host  src 206.124.146.176 
broadcast 71.227.157.255 dev eth3  proto kernel  scope link  src 71.227.156.229 
broadcast 71.227.156.0 dev eth3  proto kernel  scope link  src 71.227.156.229 
local 172.20.1.254 dev eth2  proto kernel  scope host  src 172.20.1.254 
local 127.0.0.2 dev lo  proto kernel  scope host  src 127.0.0.1 
broadcast 172.20.1.255 dev eth2  proto kernel  scope link  src 172.20.1.254 
local 71.227.156.229 dev eth3  proto kernel  scope host  src 71.227.156.229 
broadcast 206.124.146.0 dev eth0  proto kernel  scope link  src 206.124.146.176 
broadcast 127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1 
local 206.124.146.180 dev eth0  proto kernel  scope host  src 206.124.146.176 
local 127.0.0.1 dev lo  proto kernel  scope host  src 127.0.0.1 
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1 

Table main:

206.124.146.177 dev eth4  scope link 
172.20.1.0/24 dev eth2  proto kernel  scope link  src 172.20.1.254 
206.124.146.0/24 dev eth0  proto kernel  scope link  src 206.124.146.176 
71.227.156.0/23 dev eth3  proto kernel  scope link  src 71.227.156.229 
169.254.0.0/16 dev eth0  scope link 
127.0.0.0/8 dev lo  scope link 
default via 71.227.156.1 dev eth3 </programlisting>

    <para><filename>/etc/shorewall/interfaces</filename>:</para>

    <programlisting>#ZONE    INTERFACE         BROADCAST           OPTIONS
loc      eth2              detect              dhcp,routeback
dmz      eth4              detect
net      eth0              detect              dhcp,blacklist,tcpflags,optional
net      eth3              detect              dhcp,blacklist,tcpflags,optional
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE</programlisting>

    <para><filename>/etc/shorewall/masq</filename>:</para>

    <programlisting>#INTERFACE   SOURCE           ADDRESS      PROTO      PORT(S)      IPSEC

COMMENT Masquerade Local Network
eth3         0.0.0.0/0
eth0        !206.124.146.0/24 206.124.146.179

#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE</programlisting>

    <para>All traffic leaving eth3 must use the dynamic IP address assigned to
    that interface as the SOURCE address. All traffic leaving eth0 that does
    not have a SOURCE address falling within the Avvanta subnet
    (206.124.146.0/24) must have its SOURCE address changed to
    206.124.146.179.</para>
  </section>
</article>