shorewall_code/docs/MultiISP.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
  <!--ble$Id$-->

  <articleinfo>
    <title>Shorewall and Multiple Internet Connections</title>

    <authorgroup>
      <author>
        <firstname>Tom</firstname>

        <surname>Eastep and Mr Dash Four</surname>
      </author>
    </authorgroup>

    <pubdate><?dbtimestamp format="Y/m/d"?></pubdate>

    <copyright>
      <year>2005</year>

      <year>2006</year>

      <year>2007</year>

      <year>2008</year>

      <year>2009</year>

      <year>2010</year>

      <year>2011</year>

      <year>2012 2013</year>

      <holder>Thomas M. Eastep,</holder>

      <holder>2013 Mr Dash Four</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <warning>
    <para>This document describes the Multi-ISP facility in <emphasis
    role="bold">Shorewall 4.4.26 and later</emphasis>. If you are running an
    earlier release, please see the documentation for that release.</para>
  </warning>

  <warning>
    <para>Reading just Shorewall documentation is probably not going to give
    you enough background to use this material. Shorewall may make iptables
    easy but the Shorewall team doesn't have the resources to be able to
    spoon-feed Linux policy routing to you (please remember that the user's
    manual for a tractor doesn't teach you to grow corn either). You will
    likely need to refer to the following additional information:</para>

    <itemizedlist>
      <listitem>
        <para>The LARTC HOWTO: <ulink
        url="http://www.lartc.org">http://www.lartc.org</ulink></para>
      </listitem>

      <listitem>
        <para>Output of <command>man ip</command></para>
      </listitem>

      <listitem>
        <para>Output of <command>ip route help</command> and <command>ip rule
        help</command></para>
      </listitem>
    </itemizedlist>
  </warning>

  <section id="Support">
    <title>Multiple Internet Connection Support</title>

    <para>Shorewall includes limited support for multiple Internet
    connections. Limitations of this support are as follows:</para>

    <itemizedlist>
      <listitem>
        <para>It utilizes static routing configuration. If there is a change
        in the routing topopogy, Shorewall must be restarted.</para>
      </listitem>

      <listitem>
        <para>The routing changes are made and the route cache is purged when
        Shorewall is started <emphasis role="bold">and when Shorewall is
        restarted</emphasis> (unless you specify the "-n" option to
        <command>shorewall restart</command>). Ideally, restarting the packet
        filter should have no effect on routing.</para>
      </listitem>

      <listitem>
        <para>For most routing applications, <ulink
        url="http://www.quagga.net/">Quagga</ulink> is a better solution
        although it requires that your ISPs offer routing protocol
        support.</para>
      </listitem>
    </itemizedlist>

    <section id="Overview">
      <title>Overview</title>

      <para>Let's assume that a firewall is connected via two separate
      Ethernet interfaces to two different ISPs.<footnote>
          <para>While we describe a setup using different ISPs in this
          article, the facility also works with two uplinks from the same
          ISP.</para>
        </footnote> as in the following diagram.</para>

      <graphic align="center" fileref="images/TwoISPs.png" valign="middle"/>

      <itemizedlist>
        <listitem>
          <para>eth0 connects to ISP1. The IP address of eth0 is
          206.124.146.176 and the ISP's gateway router has IP address
          206.124.146.254.</para>
        </listitem>

        <listitem>
          <para>eth1 connects to ISP 2. The IP address of eth1 is
          130.252.99.27 and the ISP's gateway router has IP address
          130.252.99.254.</para>
        </listitem>

        <listitem>
          <para>eth2 connects to the local LAN. Its IP configuration is not
          relevant to this discussion.</para>
        </listitem>
      </itemizedlist>

      <para>Each of these <firstterm>providers</firstterm> is described in an
      entry in the file <filename>/etc/shorewall/providers</filename>.</para>

      <para>Entries in <filename>/etc/shorewall/providers</filename> can
      specify that outgoing connections are to be load-balanced between the
      two ISPs. Entries in <filename>/etc/shorewall/mangle</filename> and
      <filename>/etc/shorewall/rtrules</filename> can be used to direct
      particular outgoing connections to one ISP or the other. Use of
      <filename>/etc/shorewall/mangle</filename> (or
      <filename>/etc/shorewall/tcrules</filename>) is not required for
      <filename>/etc/shorewall/providers</filename> to work, but in most
      cases, you must select a unique MARK value for each provider so
      Shorewall can set up the correct marking rules for you.</para>

      <important>
        <para><filename>/etc/shorewall/mangle</filename> superseded
        <filename>/etc/shorewall/tcrules</filename> in Shorewall 4.6.0.</para>
      </important>

      <para>When you use the <emphasis role="bold">track</emphasis> option in
      <filename>/etc/shorewall/providers</filename>, connections from the
      Internet are automatically routed back out of the correct interface and
      through the correct ISP gateway. This works whether the connection is
      handled by the firewall itself or if it is routed or port-forwarded to a
      system behind the firewall.</para>

      <para>Shorewall will set up the routing and will update the
      <filename>/etc/iproute2/rt_tables</filename> to include the table names
      and numbers of the tables that it adds.</para>

      <caution>
        <para>This feature uses <ulink url="traffic_shaping.htm">packet
        marking</ulink> to control the routing. As a consequence, there are
        some restrictions concerning entries in
        <filename>/etc/shorewall/mangle</filename>:</para>

        <itemizedlist>
          <listitem>
            <para>Packet marking for traffic control purposes may not be done
            in the PREROUTING table for connections involving providers with
            'track' specified (see below).</para>
          </listitem>

          <listitem>
            <para>You may not use the SAVE or RESTORE options unless you also
            set HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET &gt; 0 with Shorewall
            4.4.26 and later) in
            <filename>/etc/shorewall/shorewall.conf</filename>.</para>

            <note>
              <para>In Shorewall 4.4.26, the HIGH_ROUTE_MARKS and
              WIDE_TC_MARKS options in
              <filename>/etc/shorewall/shorewall.conf</filename> were replaced
              by the PROVIDER_OFFSET and TC_BITS options. Look <ulink
              url="PacketMarking.html#Values">here</ulink> for details.</para>
            </note>
          </listitem>

          <listitem>
            <para>You may not use connection marking unless you also set
            HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET &gt; 0 with Shorewall 4.4.26
            and later) in
            <filename>/etc/shorewall/shorewall.conf</filename>.</para>
          </listitem>
        </itemizedlist>
      </caution>

      <para>The <filename>/etc/shorewall/providers</filename> file can also be
      used in other routing scenarios. See the <ulink
      url="Shorewall_Squid_Usage.html">Squid documentation</ulink> for an
      example.</para>
    </section>

    <section>
      <title>USE_DEFAULT_RT</title>

      <para>The behavior and configuration of Multiple ISP support is
      dependent on the setting of USE_DEFAULT_RT in shorewall[6].conf.</para>

      <para>When USE_DEFAULT_RT=Yes, packets are first routed through the main
      routing table <emphasis>which does not contain a default
      route</emphasis>. Packets which fail to be routed by an entry in the
      main table are then passed to shorewall-defined routing tables based on
      your Multi-ISP configuration. The advantage of this approach is that
      dynamic changes to the ip configuration, such as VPNs going up and down,
      do not require notificaiton of Shorewall. USE_DEFAULT_RT is now the
      default and use of USE_DEFAULT_RT=No is deprecated.</para>

      <para>When USE_DEFAULT_RT=No, packets are routed via Shorewall-generated
      routing tables. As a consequence, the main routing table must be copied
      into each of those tables and must be recopied when there is a change to
      the main table. This can only be accomplished via a
      <command>shorewall[6] reload</command> or <command>restart</command>
      command.</para>
    </section>

    <section id="providers">
      <title>/etc/shorewall/providers File</title>

      <para>Entries in this file have the following columns. As in all
      Shorewall configuration files, enter "-" in a column if you don't want
      to enter any value.</para>

      <variablelist>
        <varlistentry>
          <term>NAME</term>

          <listitem>
            <para>The provider name. Must begin with a letter and consist of
            letters and digits. The provider name becomes the name of the
            generated routing table for this provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>NUMBER</term>

          <listitem>
            <para>A number between 1 and 252. This becomes the routing table
            number for the generated table for this provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>MARK</term>

          <listitem>
            <para>A mark value used in your<filename> /etc/shorewall/mangle
            </filename>file to direct packets to this provider. Shorewall will
            also mark connections that have seen input from this provider with
            this value and will restore the packet mark in the PREROUTING
            CHAIN. Mark values must be in the range 1-255.</para>

            <para>Alternatively, you may set HIGH_ROUTE_MARKS=Yes
            (PROVIDER_OFFSET &gt; 0 with Shorewall 4.4.26 and later) in
            <filename>/etc/shorewall/shorewall.conf</filename>. This allows
            you to:</para>

            <itemizedlist>
              <listitem>
                <para>Use connection marks for traffic shaping, provided that
                you assign those marks in the FORWARD chain.</para>
              </listitem>

              <listitem>
                <para>Use mark values &gt; 255 for provider marks in this
                column.</para>

                <itemizedlist>
                  <listitem>
                    <para>With HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET=8), these
                    mark values must be a multiple of 256 in the range
                    256-65280 (hex equivalent 0x100 - 0xFF00 with the
                    low-order 8 bits being zero); or</para>
                  </listitem>

                  <listitem>
                    <para>Set WIDE_TC_MARKS=Yes in <ulink
                    url="manpages/shorewall.conf.html">shorewall.conf
                    </ulink>(5) (PROVIDER_OFFSET=16), and use mark values in
                    the range 0x10000 - 0xFF0000 with the low-order 16 bits
                    being zero.</para>
                  </listitem>
                </itemizedlist>
              </listitem>
            </itemizedlist>

            <para>This column may be omitted if you don´t use packet marking
            to direct connections to a particular provider.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>DUPLICATE</term>

          <listitem>
            <para>Gives the name or number of a routing table to duplicate.
            May be 'main' or the name or number of a previously declared
            provider. This field should be be specified as '-' when
            USE_DEFAULT_RT=Yes in <filename>shorewall.conf. When
            USE_DEFAULT_RT=No (not recommended), this column is normally
            specified as <option>main</option>.</filename></para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>INTERFACE</term>

          <listitem>
            <para>The name of the interface to the provider. Where multiple
            providers share the same interface, you must follow the name of
            the interface by a colon (":") and the IP address assigned by this
            provider (e.g., eth0:206.124.146.176). See <link
            linkend="Shared">below</link> for additional
            considerations.</para>

            <para>The interface must have been previously defined in <ulink
            url="manpages/shorewall-interfaces.html">shorewall-interfaces</ulink>
            (5). In general, that interface should not have the
            <option>proxyarp</option> option specified unless
            <option>loose</option> is given in the OPTIONS column of this
            entry.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>GATEWAY</term>

          <listitem>
            <para>The IP address of the provider's Gateway router.</para>

            <para>You can enter <emphasis role="bold">detect</emphasis> here
            and Shorewall will attempt to automatically determine the gateway
            IP address.</para>

            <para><emphasis role="bold">Hint:</emphasis> <emphasis
            role="bold">"detect"</emphasis> is appropriate for use in cases
            where the interface named in the INTERFACE column is dynamically
            configured via DHCP etc. Be sure, however, that you don't have
            stale dhcp client state files in <filename
            class="directory">/var/lib/dhcpcd </filename>or
            <filename>/var/lib/dhclient-*.lease</filename> because Shorewall
            may try to use those stale files to determine the gateway
            address.</para>

            <para>If Shorewall is unable to detect the gateway, it is likely
            because you are using a DHCP client that Shorewall doesn't
            natively support. You can work around that issue by using the
            <emphasis role="bold">findgw</emphasis> <ulink
            url="shorewall_extension_scripts.htm">extension
            script.</ulink></para>

            <para>For example, these examples from Mika Ilmaranta, work with
            RHEL7-based systems with nmcli:</para>

            <programlisting>nmcli --terse --fields IP6.GATEWAY device show ${1} | cut -f2- -d':' # IPv6

nmcli --terse --fields IP4.GATEWAY device show ${1} | cut -f2- -d':' #IPv4
</programlisting>

            <para>This one from PGNd works on OpenSuSE running wicked:</para>

            <programlisting>svc_status=$( systemctl is-active wickedd-dhcp4.service )

if [ $svc_status == 'active' ]; then
  data="/var/lib/wicked/lease-${1}-dhcp-ipv4.xml"
  if [ -f $data ]; then
    gateway=$( xml_grep 'gateway' $data --text_only )
    echo $gateway
  fi
fi</programlisting>

            <para>The GATEWAY may be omitted (enter '-') for point-to-point
            links.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>OPTIONS</term>

          <listitem>
            <para>A comma-separated list from the following:</para>

            <variablelist>
              <varlistentry>
                <term>track</term>

                <listitem>
                  <para><important>
                      <para>Beginning with Shorwall 4.3.3, <emphasis
                      role="bold">track</emphasis> defaults to the setting of
                      the <option>TRACK_PROVIDERS</option> option in <ulink
                      url="manpages/shorewall.conf">shorewall.conf
                      </ulink>(5). To disable this option when you have
                      specified TRACK_PROVIDERS=Yes, you must specify
                      <emphasis role="bold">notrack</emphasis> (see
                      below).</para>
                    </important>If specified, connections FROM this interface
                  are to be tracked so that responses may be routed back out
                  this same interface.</para>

                  <para>You want to specify 'track' if Internet hosts will be
                  connecting to local servers through this provider. Any time
                  that you specify 'track', you will normally want to also
                  specify 'balance' (see below). 'track' will also ensure that
                  outgoing connections remain stay anchored to a single
                  provider and don't try to switch providers when route cache
                  entries expire.</para>

                  <para>Use of this feature requires that your kernel and
                  iptables include CONNMARK target and connmark match support
                  (<emphasis role="bold">Warning</emphasis>: Until recently,
                  standard <trademark>Debian</trademark> and
                  <trademark>Ubuntu</trademark> kernels lacked that support.
                  <emphasis>Both Lenny and Jaunty do have the proper
                  support</emphasis>).</para>

                  <important>
                    <para>If you are running a version of Shorewall earlier
                    than 4.4.3 and are using
                    <filename>/etc/shorewall/providers</filename> because you
                    have multiple Internet connections, we recommend that you
                    specify <emphasis role="bold">track</emphasis> even if you
                    don't need it. It helps maintain long-term connections in
                    which there are significant periods with no
                    traffic.</para>
                  </important>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>balance</term>

                <listitem>
                  <para>The providers that have <emphasis
                  role="bold">balance</emphasis> specified will get outbound
                  traffic load-balanced among them. Balancing will not be
                  perfect, as it is route based, and routes are cached. This
                  means that routes to often-used sites will always be over
                  the same provider.</para>

                  <para>By default, each provider is given the same weight (1)
                  . You can change the weight of a given provider by following
                  <emphasis role="bold">balance</emphasis> with "=" and the
                  desired weight (e.g., balance=2). The weights reflect the
                  relative bandwidth of the providers connections and should
                  be small numbers since the kernel actually creates
                  additional default routes for each weight increment.</para>

                  <important>
                    <para>If you are using
                    <filename>/etc/shorewall/providers</filename> because you
                    have multiple Internet connections, we recommend that you
                    specify <emphasis role="bold">balance</emphasis> even if
                    you don't need it. You can still use entries in
                    <filename>/etc/shorewall/mangle</filename> and
                    <filename>/etc/shorewall/rtrules</filename> to force all
                    traffic to one provider or another.<note>
                        <para>If you don't heed this advice then please read
                        and follow the advice in <ulink
                        url="FAQ.htm#faq57">FAQ 57</ulink> and <ulink
                        url="FAQ.htm#faq58">FAQ 58</ulink>.</para>
                      </note></para>
                  </important>

                  <para>Prior to Shorewall 5.1.1, <emphasis
                  role="bold">balance=1</emphasis> is the default when
                  USE_DEFAULT_RT=Yes and neither the
                  <option>fallback</option>, <option>loose</option>,
                  <option>load</option> or <option>tproxy</option> option is
                  specified. Beginning with Shorewall 5.1.1, <emphasis
                  role="bold">balance=1</emphasis> is the default when both
                  USE_DEFAULT_RT=Yes and BALANCE_PROVIDERS=Yes and neither the
                  <option>fallback</option>, <option>loose</option>,
                  <option>load</option> nor <option>tproxy</option> option is
                  specified.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>loose</term>

                <listitem>
                  <para>Do not generate routing rules that force traffic whose
                  source IP is an address of the INTERFACE to be routed to
                  this provider. Useful for defining providers that are to be
                  used only when the appropriate packet mark is
                  applied.</para>

                  <para>Shorewall makes no attempt to consolidate the routing
                  rules added when <emphasis role="bold">loose</emphasis> is
                  not specified. So, if you have multiple IP addresses on a
                  provider interface, you may be able to replace the rules
                  that Shorewall generates with one or two rules in
                  <filename>/etc/shorewall/rtrules</filename>. In that case,
                  you can specify <emphasis role="bold">loose</emphasis> to
                  suppress Shorewall's rule generation. See the <link
                  linkend="Complete">example</link> below.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>notrack</term>

                <listitem>
                  <para>Added in Shorewall 4.4.3. This option turns off the
                  <emphasis role="bold">track</emphasis> option.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>optional</term>

                <listitem>
                  <note>
                    <para>This option is deprecated in favor of the
                    <option>optional</option> <ulink
                    url="manpages/shorewall-interfaces.html">interface
                    option</ulink>. That option performs the same
                    function.</para>
                  </note>

                  <para>Shorewall will determine if this interface is up and
                  has a configured IP address. If it is not, a warning is
                  issued and this provider is not configured.</para>

                  <note>
                    <para><emphasis role="bold">optional</emphasis> is
                    designed to detect interface states that will cause
                    <command>shorewall start</command> or <command>shorewall
                    restart</command> to fail; just because an interface is in
                    a state that Shorewall can [re]start without error doesn't
                    mean that traffic can actually be sent through the
                    interface.</para>

                    <para>You can supply an 'isusable' <ulink
                    url="shorewall_extension_scripts.htm">extension
                    script</ulink> to extend Shorewall's interface state
                    detection. See also the <link
                    linkend="LinkMonitor">Gateway Monitoring and
                    Failover</link> section below.</para>
                  </note>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>primary</term>

                <listitem>
                  <para>Added in Shorewall 4.6.6, <emphasis
                  role="bold">primary</emphasis> is a synonym for <emphasis
                  role="bold">balance=1</emphasis> and is preferred when the
                  remaining providers specify <emphasis
                  role="bold">fallback</emphasis> or <emphasis
                  role="bold">tproxy</emphasis>.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>src=<replaceable>source-address</replaceable></term>

                <listitem>
                  <para>Specifies the source address to use when routing to
                  this provider and none is known (the local client has bound
                  to the 0 address). May not be specified when an
                  <replaceable>address</replaceable> is given in the INTERFACE
                  column. If this option is not used, Shorewall substitutes
                  the primary IP address on the interface named in the
                  INTERFACE column.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>mtu=<replaceable>number</replaceable></term>

                <listitem>
                  <para>Specifies the MTU when forwarding through this
                  provider. If not given, the MTU of the interface named in
                  the INTERFACE column is assumed.</para>
                </listitem>
              </varlistentry>

              <varlistentry>
                <term>fallback[=<replaceable>weight</replaceable>]</term>

                <listitem>
                  <para>Indicates that a default route through the provider
                  should be added to the <firstterm>default</firstterm>
                  routing table (table 253). If a
                  <replaceable>weight</replaceable> is given, a balanced route
                  is added with the weight of this provider equal to the
                  specified <replaceable>weight</replaceable>. If the option
                  is given without a <replaceable>weight</replaceable>, a
                  separate default route is added through the provider's
                  gateway; the route has a metric equal to the provider's
                  NUMBER.</para>

                  <para>Prior to Shorewall 4.4.24, the option is ignored with
                  a warning message if USE_DEFAULT_RT=Yes in
                  <filename>shorewall.conf</filename>.</para>

                  <warning>
                    <para>If you set this option on an interface, you must
                    disable route filtering on the interface. Include
                    'routefilter=0,logmartions=0' in the OPTIONS column of
                    <ulink
                    url="manpages/shorewall-interfaces.html">shorewall-interfaces</ulink>(5).</para>
                  </warning>
                </listitem>
              </varlistentry>
            </variablelist>

            <para>For those of you who are confused between<emphasis
            role="bold"> track</emphasis> and <emphasis
            role="bold">balance</emphasis>:</para>

            <itemizedlist>
              <listitem>
                <para><emphasis role="bold">track</emphasis> governs incoming
                connections (but is also useful for binding long-running
                connections to the same interface).</para>
              </listitem>

              <listitem>
                <para><emphasis role="bold">balance</emphasis> governs
                outgoing connections.</para>
              </listitem>
            </itemizedlist>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>COPY</term>

          <listitem>
            <para>A comma-separated list of other interfaces on your firewall.
            Wildcards specified using an asterisk ("*") are permitted (e.g.,
            tun* ). Usually used only when DUPLICATE is <option>main</option>.
            Only copy routes through INTERFACE and through interfaces listed
            here. If you only wish to copy routes through INTERFACE, enter
            <option>none</option> in this column.</para>

            <note>
              <para>Beginning with Shorewall 4.4.15, provider routing tables
              can be augmeted with additional routes through use of the <link
              linkend="routes">/etc/shorewall/routes</link> file.</para>
            </note>
          </listitem>
        </varlistentry>
      </variablelist>
    </section>

    <section id="Providers">
      <title>What an entry in the Providers File Does</title>

      <para>Adding another entry in the providers file simply creates an
      alternate routing table for you (see the<ulink
      url="http://www.lartc.org"> LARTC Howto</ulink>). The table will usually
      contain two routes:</para>

      <orderedlist>
        <listitem>
          <para>A host route to the specified GATEWAY through the specified
          INTERFACE.</para>
        </listitem>

        <listitem>
          <para>A default route through the GATEWAY.</para>
        </listitem>
      </orderedlist>

      <para>Note that the first route is omitted if "-" is specified as the
      GATEWAY; in that case, the default route does not specify a gateway
      (point-to-point link).</para>

      <para>If the DUPLICATE column is non-empty, then routes from the table
      named in that column are copied into the new table. By default, all
      routes (except default routes) are copied. The set of routes copied can
      be restricted using the COPY column which lists the interfaces whose
      routes you want copied. You will generally want to include all local
      interfaces in this list. You should exclude the loopback interface (lo)
      and any interfaces that do not have an IP configuration. You should also
      omit interfaces like <emphasis role="bold">tun</emphasis> interfaces
      that are created dynamically. Traffic to networks handled by those
      interfaces should be routed through the main table using entries in
      <filename>/etc/shorewall/rtrules</filename> (see Example 2 <link
      linkend="Examples">below</link>) or by using <link
      linkend="USE_DEFAULT_RT">USE_DEFAULT_RT=Yes</link> (recommended)</para>

      <para>In addition:</para>

      <orderedlist>
        <listitem>
          <para>Unless <emphasis role="bold">loose</emphasis> is specified, an
          ip rule is generated for each IP address on the INTERFACE that
          routes traffic from that address through the associated routing
          table.</para>
        </listitem>

        <listitem>
          <para>If you specify <emphasis role="bold">track</emphasis>, then
          connections which have had at least one packet arrive on the
          interface listed in the INTERFACE column have their connection mark
          set to the value in the MARK column. In the PREROUTING chain,
          packets with a connection mark have their packet mark set to the
          value of the associated connection mark; packets marked in this way
          bypass any prerouting rules that you create in
          <filename>/etc/shorewall/mangle</filename>. This ensures that
          packets associated with connections from outside are always routed
          out of the correct interface.</para>
        </listitem>

        <listitem>
          <para>If you specify <emphasis role="bold">balance</emphasis>, then
          Shorewall will replace the 'default' route with weight 100 in the
          'main' routing table with a load-balancing route among those
          gateways where <emphasis role="bold">balance</emphasis> was
          specified. So if you configure default routes, be sure that their
          weight is less than 100 or the route added by Shorewall will not be
          used.</para>
        </listitem>
      </orderedlist>

      <para>That's <emphasis role="bold">all</emphasis> that these entries do.
      You still have to follow the principle stated in the <ulink
      url="Shorewall_and_Routing.html">Shorewall Routing
      documentation</ulink>:</para>

      <orderedlist>
        <listitem>
          <para>Routing determines where packets are to be sent.</para>
        </listitem>

        <listitem>
          <para>Once routing determines where the packet is to go, the
          firewall (Shorewall) determines if the packet is allowed to go there
          and controls rewriting of the SOURCE IP address
          (SNAT/MASQUERADE).</para>
        </listitem>
      </orderedlist>

      <para>The bottom line is that if you want traffic to go out through a
      particular provider then you <emphasis>must </emphasis>mark that traffic
      with the provider's MARK value in
      <filename>/etc/shorewall/mangle</filename> and you must do that marking
      in the PREROUTING chain; or, you must provide the appropriate rules in
      <filename>/etc/shorewall/rtrules</filename>.</para>
    </section>

    <section>
      <title>What an entry in the Providers File Does Not Do</title>

      <para>Shorewall itself provides no mechanism for dealing with provider
      links that are in the up state but not responsive. If you want
      transparent failover when a link is unresponsive, you must configure all
      provider interfaces as <emphasis role="bold">optional</emphasis> (<ulink
      url="manpages/shorewall-interfaces.html">shorewall-interfaces(5)</ulink>)
      then <link linkend="LinkMonitor">install and configure
      FOOLSM</link>.</para>

      <para><ulink url="Shorewall-init.html">Shorewall-init</ulink> provides
      for handling links that go hard down and are later brought back
      up.</para>
    </section>

    <section id="masq">
      <title>./etc/shorewall/masq (/etc/shorewall/snat) and Multi-ISP</title>

      <para>If you masquerade a local network, you will need to add masquerade
      rules for both external interfaces. Referring to the diagram above, if
      each of the interfaces has only a single IP address and you have no
      systems with public IP addresses behind your firewall, then I suggest
      the following simple entries:</para>

      <programlisting>#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27</programlisting>

      <para>When running Shorewall 5.0.14 or later, the equivalent
      <filename>/etc/shorewall/snat</filename> is:</para>

      <programlisting>#ACTION                SOURCE          DEST                PROTO   PORT
SNAT(206.124.146.176)  0.0.0.0/0       eth0
SNAT(130252.99.27)     0.0.0.0/0       eth1</programlisting>

      <para>If you have a public subnet (for example 206.124.146.176/30)
      behind your firewall, then use exclusion:</para>

      <programlisting>#INTERFACE       SOURCE               ADDRESS
eth0             !206.124.146.176/29  206.124.146.176
eth1             0.0.0.0/0            130.252.99.27</programlisting>

      <para>The equivalent <filename>/etc/shorewall/snat</filename> is:</para>

      <programlisting>#ACTION                SOURCE              DEST                PROTO   PORT
SNAT(206.124.146.176)  !206.124.146.176/29 eth0
SNAT(130.252.99.27)    0.0.0.0/0           eth1</programlisting>

      <para>Note that exclusion is only used on the interface corresponding to
      internal subnetwork.</para>

      <para>If you have multiple IP addresses on one of your interfaces, you
      can use a similar technique -- simplY exclude the smallest network that
      contains all of those addresses from being masqueraded.</para>

      <warning>
        <para>Entries in <filename>/etc/shorewall/masq</filename>
        (<filename>/etc/shorewall/snat</filename>) have no effect on which ISP
        a particular connection will be sent through. That is rather the
        purpose of entries in <filename>/etc/shorewall/mangle</filename> and
        <filename>/etc/shorewall/rtrules</filename>.</para>
      </warning>
    </section>

    <section id="Martians">
      <title>Martians</title>

      <para>One problem that often arises with Multi-ISP configuration is
      'Martians'. If you set ROUTE_FILTER=Yes in
      <filename>/etc/shorewall/shorewall.conf</filename> or if your Internet
      interfaces are configured with the <emphasis
      role="bold">routefilter</emphasis> option in
      <filename>/etc/shorewall/interfaces</filename> (remember that if you set
      that option, you should also select <emphasis
      role="bold">logmartians</emphasis>), then things may not work correctly
      and you will see messages like this:</para>

      <programlisting>Feb  9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 
Feb  9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00</programlisting>

      <para>The above message is somewhat awkwardly phrased. The source IP in
      this incoming packet was 64.86.88.116 and the destination IP address was
      206.124.146.176. Another gotcha is that the incoming packet has already
      had the destination IP address changed for DNAT or because the original
      outgoing connection was altered by an entry in
      <filename>/etc/shorewall/masq</filename> or
      <filename>/etc/shorewall/snat</filename> (SNAT or Masquerade). So the
      destination IP address (206.124.146.176) may not have been the
      destination IP address in the packet as it was initially
      received.</para>

      <para>There a couple of common causes for these problems:</para>

      <orderedlist>
        <listitem>
          <para>You have connected both of your external interfaces to the
          same hub/switch. Connecting multiple firewall interfaces to a common
          hub or switch is always a bad idea that will result in
          hard-to-diagnose problems.</para>
        </listitem>

        <listitem>
          <para>You are specifying both the <emphasis
          role="bold">loose</emphasis> and <emphasis
          role="bold">balance</emphasis> options on your provider(s). This can
          cause individual connections to ping-pong back and forth between the
          interfaces which is almost guaranteed to cause problems.</para>
        </listitem>

        <listitem>
          <para>You are redirecting traffic from the firewall system out of
          one interface or the other using packet marking in your
          <filename>/etc/shorewall/mangle</filename> file. A better approach
          is to configure the application to use the appropriate local IP
          address (the IP address of the interface that you want the
          application to use). See <link linkend="Local">below</link>.</para>
        </listitem>
      </orderedlist>

      <para>If all else fails, remove the <emphasis
      role="bold">routefilter</emphasis> option from your external interfaces.
      If you do this, you may wish to add rules to log and drop packets from
      the Internet that have source addresses in your local networks. For
      example, if the local LAN in the above diagram is 192.168.1.0/24, then
      you would add this rule:</para>

      <programlisting>#ACTION          SOURCE                     DEST
DROP:info        net:192.168.1.0/24         all</programlisting>

      <para>Be sure the above rule is added before any other rules with
      <emphasis>net</emphasis> in the SOURCE column.</para>

      <important>
        <para>If you set ROUTE_FILTER=Yes in
        <filename>/etc/shorewall/shorewall.conf</filename>, then setting
        <emphasis role="bold">routefilter</emphasis>=0 in <ulink
        url="manpages/shorewall-interfaces.html">shorewall-interfaces</ulink>
        (5) will not disable route filtering on a given interface. You must
        set ROUTE_FILTER=No in <ulink
        url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5), then
        set the <emphasis role="bold">routefilter</emphasis> option on those
        interfaces on which you want route filtering.</para>
      </important>
    </section>

    <section id="Example1">
      <title id="Example">Legacy Example</title>

      <para>This section describes the legacy method of configuring multiple
      uplinks. It is deprecated in favor of the USE_DEFAULT_RT=Yes
      configuration described <link
      linkend="USE_DEFAULT_RT">below</link>.</para>

      <para>The configuration in the figure at the top of this section would
      be specified in <filename>/etc/shorewall/providers</filename> as
      follows.</para>

      <programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2</programlisting>

      <para>Other configuration files go something like this:</para>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …</programlisting>

      <para><filename>/etc/shorewall/policy</filename>:</para>

      <programlisting>#SOURCE    DESTINATION    POLICY     LOGLEVEL     LIMIT
net        net            DROP</programlisting>

      <para><filename>/etc/shorewall/masq</filename>:</para>

      <programlisting>#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27</programlisting>
    </section>

    <section id="Example2">
      <title id="Example99">Example using USE_DEFAULT_RT=Yes</title>

      <para>This section shows the differences in configuring the above
      example with USE_DEFAULT_RT=Yes. The changes are confined to the
      DUPLICATE and COPY columns of the providers file.</para>

      <para>The configuration in the figure at the top of this section would
      be specified in <filename>/etc/shorewall/providers</filename> as
      follows.</para>

      <programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       <emphasis role="bold">- </emphasis>              eth0            206.124.146.254 track,balance    <emphasis
          role="bold">-</emphasis>
ISP2    2       2       <emphasis role="bold">-</emphasis>               eth1            130.252.99.254  track,balance    <emphasis
          role="bold">-</emphasis></programlisting>

      <para>Other configuration files go something like this:</para>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          …          
net      eth1         detect          …</programlisting>

      <para><filename>/etc/shorewall/policy</filename>:</para>

      <programlisting>#SOURCE    DESTINATION    POLICY     LOGLEVEL     LIMIT
net        net            DROP</programlisting>

      <para><filename>/etc/shorewall/masq</filename>:</para>

      <programlisting>#INTERFACE       SOURCE            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27</programlisting>

      <para>When running Shorewall 5.0.14 or later, the equivalent
      <filename>/etc/shorewall/snat</filename> is:</para>

      <programlisting>#ACTION                SOURCE          DEST                PROTO   PORT
SNAT(206.124.146.176)  0.0.0.0/0       eth0
SNAT(130.252.99.27)    0.0.0.0/0       eth1</programlisting>
    </section>

    <section id="Applications">
      <title>Routing a Particular Application Through a Specific
      Interface</title>

      <para>This continues the example in the preceding section.</para>

      <para>Now suppose that you want to route all outgoing SMTP traffic from
      your local network through ISP 2. If you are running Shorewall 4.6.0 or
      later, you would make this entry in <ulink
      url="manpages/shorewall-mangle.html">/etc/shorewall/mangle</ulink>.</para>

      <programlisting>#ACTION         SOURCE          DEST            PROTO   DPORT   SPORT   USER    TEST
MARK(2):P       &lt;local network&gt; 0.0.0.0/0       tcp     25</programlisting>

      <para>Note that traffic from the firewall itself must be handled in a
      different rule:</para>

      <programlisting>#ACTION         SOURCE          DEST            PROTO   DPORT   SPORT   USER    TEST
MARK(2)         $FW             0.0.0.0/0       tcp     25</programlisting>

      <para>If you are running a Shorewall version earlier than 4.6.0, the
      above rules in <ulink
      url="manpages4/manpages/shorewall-tcrules.html">/etc/shorewall/tcrules</ulink>
      would be:</para>

      <programlisting>#ACTION         SOURCE          DEST            PROTO   DPORT   SPORT   USER    TEST
2:P             &lt;local network&gt; 0.0.0.0/0       tcp     25</programlisting>

      <para>And for traffic from the firewall:</para>

      <programlisting>#ACTION         SOURCE          DEST            PROTO   DPORT   SPORT   USER    TEST
2               $FW             0.0.0.0/0       tcp     25</programlisting>
    </section>

    <section id="PortForwarding">
      <title>Port Forwarding</title>

      <para>Shorewall provides considerable flexibility for port forwarding in
      a multi-ISP environment.</para>

      <para>Normal port forwarding rules such as the following will forward
      from both providers.</para>

      <para><filename>/etc/shorewall/rules</filename>:</para>

      <programlisting>#ACTION        SOURCE             DEST              PROTO     DPORT            SPORT        ORIGDEST
DNAT           net                loc:192.168.1.3   tcp       25</programlisting>

      <para>Continuing the above example, to forward only connection requests
      from ISP 1, you can either:</para>

      <orderedlist>
        <listitem>
          <para>Qualify the SOURCE by ISP 1's interface:</para>

          <programlisting>#ACTION        SOURCE             DEST              PROTO     DPORT            SPORT        ORIGDEST
DNAT           net<emphasis role="bold">:eth0</emphasis>           loc:192.168.1.3   tcp       25</programlisting>

          <para>or</para>
        </listitem>

        <listitem>
          <para>Specify the IP address of ISP 1 in the ORIGDEST column:</para>

          <programlisting>#ACTION        SOURCE             DEST              PROTO     DPORT            SPORT        ORIGDEST
DNAT           net                loc:192.168.1.3   tcp       25               <emphasis
              role="bold">-            206.124.146.176</emphasis></programlisting>
        </listitem>
      </orderedlist>
    </section>

    <section id="morethan2">
      <title>More than 2 Providers</title>

      <para>When there are more than two providers, you need to extend the
      two-provider case in the expected way:</para>

      <orderedlist>
        <listitem>
          <para>For each external address, you need an entry in
          <filename>/etc/shorewall/masq</filename> to handle the case where a
          connection using that address as the SOURCE is sent out of the
          interfaces other than the one that the address is configured
          on.</para>
        </listitem>

        <listitem>
          <para>For each external interface, you need to add an entry to
          <filename>/etc/shorewall/masq</filename>
          (<filename>/etc/shorewall/snat</filename>).</para>
        </listitem>
      </orderedlist>

      <para>If we extend the above example to add eth3 with IP address
      16.105.78.4 with gateway 16.105.78.254, then:</para>

      <para><filename>/etc/shorewall/providers</filename>:<programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2
ISP3    3       3       main            eth3            16.105.78.254   track,balance    eth2</programlisting></para>

      <para><filename>/etc/shorewall/masq</filename>:<programlisting>#INTERFACE       SUBNET            ADDRESS
eth0             0.0.0.0/0         206.124.146.176
eth1             0.0.0.0/0         130.252.99.27
eth3             0.0.0.0/0         16.105.78.4</programlisting></para>

      <para>When running Shorewall 5.0.14 or later, the equivalent
      <filename>/etc/shorewall/snat</filename> is:</para>

      <programlisting>#ACTION                SOURCE          DEST                PROTO   PORT
SNAT(206.124.146.176)  0.0.0.0/0       eth0
SNAT(130.252.99.27)    0.0.0.0/0       eth1
SNAT(16.105.78.4)      0.0.0.0/0       eth2</programlisting>
    </section>

    <section id="rtrules">
      <title>/etc/shorewall/rtrules (formerly
      /etc/shorewall/route_rules)</title>

      <para>The <filename>rtrules</filename> file allows assigning certain
      traffic to a particular provider just as entries in the
      <filename>mangle</filename> file. The difference between the two files
      is that entries in <filename>rtrules</filename> are independent of
      Netfilter.</para>

      <section id="Routing_rules">
        <title>Routing Rules</title>

        <para>Routing rules are maintained by the Linux kernel and can be
        displayed using the <command>ip rule ls</command> command. When
        routing a packet, the rules are processed in turn until the packet is
        successfully routed.</para>

        <programlisting>gateway:~ # <command>ip rule ls</command>
0:      from all lookup local                &lt;=== Local (to the firewall) IP addresses
10001:  from all fwmark 0x1 lookup Blarg     &lt;=== This and the next rule are generated by the
10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
20000:  from 206.124.146.176 lookup Blarg    &lt;=== This and the next rule are generated unless
20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
32766:  from all lookup main                 &lt;=== This is the routing table shown by 'iproute -n'
32767:  from all lookup default              &lt;=== This table is usually empty
gateway:~ #</programlisting>

        <para>In the above example, there are two providers: Blarg and Comcast
        with MARK 1 going to Blarg and mark 2 going to Comcast.</para>
      </section>

      <section id="rtrules_columns">
        <title>Columns in the rtrules file</title>

        <para>Columns in the file are:</para>

        <variablelist>
          <varlistentry>
            <term>SOURCE (Optional)</term>

            <listitem>
              <para>An ip address (network or host) that matches the source IP
              address in a packet. May also be specified as an interface name
              optionally followed by ":" and an address. If the device 'lo' is
              specified, the packet must originate from the firewall
              itself.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>DEST (Optional)</term>

            <listitem>
              <para>An ip address (network or host) that matches the
              destination IP address in a packet.</para>

              <para>If you choose to omit either SOURCE or DEST, place "-" in
              that column. Note that you may not omit both SOURCE and
              DEST.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>PROVIDER</term>

            <listitem>
              <para>The provider to route the traffic through. May be
              expressed either as the provider name or the provider
              number.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>PRIORITY</term>

            <listitem>
              <para>The rule's priority which determines the order in which
              the rules are processed.</para>

              <para>1000-1999 Before Shorewall-generated 'MARK' rules</para>

              <para>11000- 11999 After 'MARK' rules but before
              Shorewall-generated rules for ISP interfaces.</para>

              <para>26000-26999 After ISP interface rules but before 'default'
              rule.</para>

              <para>Rules with equal priority are applied in the order in
              which they appear in the file.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>MARK (Optional - added in Shorewall 4.4.25)</term>

            <listitem>
              <para>Mark and optional mask in the form
              <replaceable>mark</replaceable>[/<replaceable>mask</replaceable>].
              For this rule to be applied to a packet, the packet's mark value
              must match the <replaceable>mark</replaceable> when logically
              anded with the <replaceable>mask</replaceable>. If a
              <replaceable>mask</replaceable> is not supplied, Shorewall
              supplies a suitable provider mask.</para>
            </listitem>
          </varlistentry>
        </variablelist>
      </section>

      <section>
        <title>Multi-ISP and VPN</title>

        <para>For those VPN types that use routing to direct traffic to remote
        VPN clients (including but not limited to OpenVPN in routed mode and
        PPTP), the VPN software adds a host route to the <emphasis
        role="bold">main</emphasis> table for each VPN client. The best
        approach is to use USE_DEFAULT_RT=Yes as described <link
        linkend="USE_DEFAULT_RT">below</link>. If that isn't possible, you
        must add a routing rule in the 1000-1999 range to specify the
        <emphasis role="bold">main</emphasis> table for traffic addressed to
        those clients. See<link linkend="Openvpn"> Example 2</link>
        below.</para>

        <para>If you have an IPSEC gateway on your firewall, be sure to
        arrange for ESP packets to be routed out of the same interface that
        you have configured your keying daemon to use.</para>
      </section>

      <section id="Examples">
        <title>Examples</title>

        <para><emphasis role="bold">Example 1:</emphasis> You want all traffic
        entering the firewall on eth1 to be routed through Comcast.</para>

        <programlisting>#SOURCE            DEST      PROVIDER        PRIORITY
eth1               -         Comcast         1000</programlisting>

        <para>With this entry, the output of <command>ip rule ls</command>
        would be as follows.</para>

        <para><programlisting>gateway:~ # <command>ip rule ls</command>
0:      from all lookup local
<emphasis role="bold">1000:   from all iif eth1 lookup Comcast</emphasis>
10001:  from all fwmark 0x1 lookup Blarg
10002:  from all fwmark 0x2 lookup Comcast
20000:  from 206.124.146.176 lookup Blarg
20256:  from 24.12.22.33 lookup Comcast
32766:  from all lookup main
32767:  from all lookup default
gateway:~ #</programlisting>Note that because we used a priority of 1000, the
        test for <filename class="devicefile">eth1</filename> is inserted
        before the fwmark tests.</para>

        <para id="Openvpn"><emphasis role="bold">Example 2:</emphasis> You use
        OpenVPN (routed setup w/tunX) in combination with multiple providers.
        In this case you have to set up a rule to ensure that the OpenVPN
        traffic is routed back through the tunX interface(s) rather than
        through any of the providers. 10.8.0.0/24 is the subnet chosen in your
        OpenVPN configuration (server 10.8.0.0 255.255.255.0).</para>

        <programlisting>#SOURCE                 DEST            PROVIDER        PRIORITY
-                       10.8.0.0/24     main            1000</programlisting>
      </section>
    </section>

    <section id="Local">
      <title>Applications running on the Firewall - making them use a
      particular provider</title>

      <para>As <link linkend="Applications">noted above</link>, separate
      entries in <filename>/etc/shorewall/mangle</filename> are required for
      traffic originating from the firewall.</para>

      <para>Experience has shown that in some cases, problems occur with
      applications running on the firewall itself. This is especially true
      when you have specified <emphasis role="bold">routefilter</emphasis> on
      your external interfaces in /etc/shorewall/interfaces (see <link
      linkend="Martians">above</link>). When this happens, it is suggested
      that you have the application use specific local IP addresses rather
      than 0.</para>

      <para>Examples:</para>

      <itemizedlist>
        <listitem>
          <para>Squid: In <filename>squid.conf</filename>, set <emphasis
          role="bold">tcp_outgoing_address</emphasis> to the IP address of the
          interface that you want Squid to use.</para>
        </listitem>

        <listitem>
          <para>In OpenVPN, set <emphasis role="bold">local
          </emphasis>(<emphasis role="bold">--local</emphasis> on the command
          line) to the IP address that you want the server to receive
          connections on.</para>
        </listitem>
      </itemizedlist>

      <para>Note that some traffic originating on the firewall doesn't have a
      SOURCE IP address before routing. At least one Shorewall user reports
      that an entry in <filename>/etc/shorewall/rtrules</filename> with 'lo'
      in the SOURCE column seems to be the most reliable way to direct such
      traffic to a particular ISP.</para>

      <para>Example:</para>

      <programlisting>#SOURCE     DEST      PROVIDER        PRIORITY
lo          -         shorewall       1000</programlisting>
    </section>

    <section id="routes">
      <title>/etc/shorewall/routes File</title>

      <para>Beginning with Shorewall 4.4.15, additional routes can be added to
      the provider routing tables using the /etc/shorewall/routes file.</para>

      <para>The columns in the file are as follows.</para>

      <variablelist>
        <varlistentry>
          <term><emphasis role="bold">PROVIDER</emphasis></term>

          <listitem>
            <para>The name or number of a provider defined in <ulink
            url="shorewall-providers.html">shorewall-providers</ulink>
            (5).</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><emphasis role="bold">DEST</emphasis></term>

          <listitem>
            <para>Destination host address or network address.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><emphasis role="bold">GATEWAY</emphasis> (Optional)</term>

          <listitem>
            <para>If specified, gives the IP address of the gateway to the
            DEST.</para>

            <para>Beginning with Shorewall 4.5.14, you may specify
            <option>blackhole</option> in this column to create a
            <firstterm>blackhole</firstterm> route. When
            <option>blackhole</option> is specified, the DEVICE column must be
            empty.</para>

            <para>Beginning with Shorewall 4.5.15, you may specify
            <option>prohibit</option> or <option>unreachable</option> to
            create a <firstterm>prohibit</firstterm> or
            <firstterm>unreachable</firstterm> route respectively. Again, the
            DEVICE column must be empty.</para>

            <para>See the next section for additional information.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><emphasis role="bold">DEVICE</emphasis> (Optional)</term>

          <listitem>
            <para>Specifies the device route. If neither DEVICE nor GATEWAY is
            given, then the INTERFACE specified for the PROVIDER in <ulink
            url="manpages/shorewall-providers.html">shorewall-providers</ulink>
            (5).</para>
          </listitem>
        </varlistentry>
      </variablelist>

      <para>Assume the following entry in
      <filename>/etc/shorewall/providers</filename>:</para>

      <programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
Comcast      1     -    xxx             eth2            ....     </programlisting>

      <para>The following table gives some example entries in the file and the
      <command>ip route</command> command which results.</para>

      <programlisting><emphasis role="bold">#PROVIDER     DEST             GATEWAY         DEVICE</emphasis>        | <emphasis
          role="bold">             Generated Command</emphasis>
Comcast       172.20.1.0/24    -               eth0          | ip -4 route add 172.20.1.0/24 dev eth0 table 1
Comcast       192.168.4.0/24   172.20.1.1                    | ip -4 route add 192.168.1.0/24 via 172.20.1.1 table 1
Comcast       192.168.4.0/24                                 | ip -4 route add 192.168.4.0/24 dev eth2 table 1 </programlisting>
    </section>

    <section id="null_routing">
      <title>Null Routing</title>

      <para>Null routing is a type of routing which discards a given packet
      instead of directing it through a specific predefined route. Generally
      speaking, there are 3 different types of Null routing as indicated
      below:</para>

      <orderedlist>
        <listitem>
          <para>Unreachable routes</para>

          <para>When used, a request for a routing decision returns a
          destination with an unreachable route type, an ICMP unreachable is
          generated (icmp type 3) and returned to the source address.</para>

          <para>Example:</para>

          <programlisting>ip route add unreachable 10.22.0.12
ip route add unreachable 192.168.14.0/26
ip route add unreachable 82.32.0.0/12
</programlisting>

          <para>Unreachable routes are usually indicated by a dash ("-") in
          the "Iface" column when "route -n" is executed:</para>

          <programlisting>~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.22.0.12      -               255.255.255.255 !H    0      -        0 -
192.168.14.0    -               255.255.255.192 !     0      -        0 -
82.32.0.0       -               255.240.0.0     !     0      -        0 -
</programlisting>
        </listitem>

        <listitem>
          <para>Prohibit routes</para>

          <para>Similar to "unreachable" routes above, when a request for a
          routing decision returns a destination with a prohibit route type,
          the kernel generates an ICMP prohibited to return to the source
          address.</para>

          <para>Example:</para>

          <programlisting>ip route add prohibit 10.22.0.12
ip route add prohibit 192.168.14.0/26
ip route add prohibit 82.32.0.0/12</programlisting>

          <para>"Prohibit" type routes are also indicated by a dash in the
          "Iface" column as shown above.</para>
        </listitem>

        <listitem>
          <para>Blackhole routes</para>

          <para>The difference between this type of routing and the previous
          two listed above is that a packet matching a route with the route
          type blackhole is simply discarded (DROPed). No ICMP is sent and no
          packet is forwarded.</para>

          <para>Example:</para>

          <programlisting>ip route add blackhole 10.22.0.12
ip route add blackhole 192.168.14.0/26
ip route add blackhole 82.32.0.0/12</programlisting>

          <para>Blackhole routes are usually indicated with a star ("*") in
          the "Iface" column:</para>

          <programlisting>~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.22.0.12      0.0.0.0         255.255.255.255 UH    0      0        0 *
192.168.14.0    0.0.0.0         255.255.255.192 U     0      0        0 *
82.32.0.0       0.0.0.0         255.240.0.0     U     0      0        0 *</programlisting>
        </listitem>
      </orderedlist>

      <section>
        <title>Null Routing Implementation in Shorewall</title>

        <para>As of Shorewall 4.5.14, the only type of null routing
        implemented in Shorewall is "blackhole" routing. This can be specified
        in two different ways as described below.</para>

        <orderedlist>
          <listitem>
            <para>Null Routing with NULL_ROUTE_RFC1918 shorewall.conf
            configuration option.</para>

            <para>When NULL_ROUTE_RFC1918 is set to Yes, it causes Shorewall
            to null-route the IPv4 address ranges reserved by RFC1918 (private
            networks).</para>

            <para>When combined with route filtering (ROUTE_FILTER=Yes or
            routefilter in <ulink
            url="manpages/shorewall-interfaces.html">shorewall-interfaces</ulink>(5)),
            this option ensures that packets with an RFC1918 source address
            are only accepted from interfaces having known routes to networks
            using such addresses.</para>

            <para>When this option is used, the blackhole routes for all
            RFC1918 subnets are defined for the "main" routing table only.
            These, however, can be copied over to different routing tables or
            further customised and fine-tuned to suit individual needs by
            using the "routes" file (see below).</para>

            <para>For example, by specifying NULL_ROUTE_RFC1918=Yes in
            shorewall.conf, Shorewall generates 3 different route statements
            to be executed at Shorewall startup:</para>

            <programlisting>ip route replace blackhole 10.0.0.0/8
ip route replace blackhole 172.16.0.0/12
ip replace blackhole 192.168.0.0/16
</programlisting>

            <important>
              <para>When NULL_ROUTE_RFC1918=Yes is used, Shorewall creates a
              shell script file in ${VARDIR}/undo_rfc1918_routing to undo the
              null routing, if needed (see below as to some instances when
              this may be necessary).</para>
            </important>
          </listitem>

          <listitem>
            <para>Null Routing Using Shorewall "routes" (added in Shorewall
            4.5.14)</para>

            <para>By definition, entries in this file are used to define
            routes to be added to provider routing tables, including the
            default routing table (main).</para>

            <para>This option allows for a better control over what is defined
            as a null route in Shorewall and also allows for custom-defined
            subnets (in addition to RFC1918 type networks) to be added.
            Blackhole routes defined in this way need to include the word
            "blackhole" in the GATEWAY column and the DEVICE column must also
            be ommitted (see example below).</para>

            <para>Example of use
            (<filename>/etc/shorewall/routes</filename>):</para>

            <programlisting>#PROVIDER       DEST            GATEWAY         DEVICE
main            10.0.0.0/8      blackhole
dmz             82.32.0.0/12    blackhole
dmz             192.168.14.0/26 blackhole
</programlisting>

            <para>The above generates the following 3 statements for execution
            upon Shorewall startup:</para>

            <programlisting>ip route add blackhole 10.0.0.0/8 table main
ip route add blackhole 82.32.0.0/12 table dmz
ip route add blackhole 192.168.14.0/26 table dmz</programlisting>

            <important>
              <para>When blackhole routes are added to a
              <replaceable>provider</replaceable> (including 'main'),
              Shorewall creates a shell script file in
              ${VARDIR}/undo_<replaceable>provider</replaceable>_routing to
              undo the routing, if needed (see below as to some instances when
              this may be necessary).</para>
            </important>
          </listitem>
        </orderedlist>

        <para>Beginning with Shorewall 4.5.15, Shorewall also supports
        "unreachable" and "prohibit" routing.</para>

        <orderedlist>
          <listitem>
            <para>The NULL_ROUTE_RFC1918 option may be set to "blackhole",
            "prohibit" or "unreachable" in addition to "Yes" and "No".</para>

            <para>Shorewall will create the three route statements using the
            specified type type. For compatibility with earlier releases,
            "Yes" is equivalent to "blackhole".</para>

            <para>For example, if NULL_ROUTE_RFC1918=prohibit, then the
            following three route statements will be executed at Shorewall
            startup:</para>

            <programlisting>ip route replace prohibit 10.0.0.0/8
ip route replace prohibit 172.16.0.0/12
ip replace prohibit 192.168.0.0/16
</programlisting>
          </listitem>

          <listitem>
            <para>The words "prohibit" and "unreachable" may be placed in the
            GATEWAY column of
            <filename>/etc/shorewall/routes</filename>.</para>

            <para>The DEVICE column must be omitted.</para>

            <para>Example of use
            (<filename>/etc/shorewall/routes</filename>):</para>

            <programlisting>#PROVIDER       DEST            GATEWAY         DEVICE
main            10.0.0.0/8      unreachable
dmz             82.32.0.0/12    unreachable
dmz             192.168.14.0/26 unreachable
</programlisting>

            <para>The above generates the following 3 statements for execution
            upon Shorewall startup:</para>

            <programlisting>ip route add unreachable 10.0.0.0/8 table main
ip route add unreachable 82.32.0.0/12 table dmz
ip route add unreachable 192.168.14.0/26 table dmz</programlisting>

            <important>
              <para>When prohibit or unreachable routes are added to a
              <replaceable>provider</replaceable> (including 'main'),
              Shorewall creates a shell script file in
              ${VARDIR}/undo_<replaceable>provider</replaceable>_routing to
              undo the routing, if needed (see below as to some instances when
              this may be necessary).</para>
            </important>
          </listitem>
        </orderedlist>
      </section>

      <section>
        <title>Important Points To Remember When Using Null Routing in
        Shorewall</title>

        <orderedlist>
          <listitem>
            <para>In order to create "pinhole" in a particular blackhole
            route, at least one route needs to be defined in addition to the
            null route.</para>

            <para>Lets take the following example: We need to null-route all
            addresses from the 10.0.0.0/8 range, <emphasis
            role="bold">except</emphasis> 10.1.0.0/24. In such a case we need
            to define two routes in our "routes" file (assuming the default
            "main" routing table is used and also assuming that 10.1.0.0/24 is
            routed via the default gateway on eth0 and we need to use
            'blackhole' type null routing).</para>

            <para><filename>/etc/shorewall/routes</filename>:</para>

            <programlisting>#PROVIDER       DEST            GATEWAY         DEVICE
main            10.0.0.0/8      blackhole
main            10.1.0.0/24     -               eth0</programlisting>

            <para>The above will generate 2 statements for execution when
            Shorewall starts:</para>

            <programlisting>ip route replace blackhole 10.0.0.0/8 table main
ip route replace 10.1.0.0/24 table main
</programlisting>

            <para>The order in which the two routes above are defined in
            "routes" is not important, simply because, by definition, routes
            with lower mask value are always traversed first. In that way,
            packets originating from or destined to 10.1.0.0/24 will always be
            processed before the 10.0.0.0/8 blackhole route.</para>
          </listitem>

          <listitem>
            <para>Null routes, by their definition, are not attached to any
            network device. What this means in reality is that when the status
            of a particular device changes (either going up or down), that has
            absolutely <emphasis role="bold">no</emphasis> effect on the null
            routes defined (as already indicated, these are "static" and can
            only be removed by executing "ip route del" or by executing the
            relevant ${VARDIR}/undo_*_routing shell script).</para>

            <important>
              <para>The ${VARDIR}/undo_*_routing scripts generated by
              Shorewall 4.5.14 and earlier cannot be executed directly from
              the shell without first sourcing
              ${SHAREDIR}/shorewall/functions. Example:</para>

              <programlisting>. /usr/share/shorewall/functions
. /var/lib/shorewall/undo_x_routing</programlisting>
            </important>

            <para>This sometimes may lead to undesirable side effect: when a
            network interface goes down (even temporarily), then <emphasis
            role="bold">all</emphasis> routes defined or attached to that
            interface are simply deleted from the routing table by the kernel,
            while the blackhole routes are untouched.</para>

            <para>Lets take our example above: when eth0 goes down, then the
            route we defined in "routes" for our private subnet (10.1.0.0/24)
            will be deleted from the routing table. As soon as eth0 goes back
            up again, unless the route for our private 10.1.0.0/24 subnet is
            defined again, all packets originating from or destined to
            10.1.0.0/24 will simply be dropped by the kernel!</para>

            <para>An indication of this type of behaviour is getting endless
            "martian" packets reported in the system log, like so:</para>

            <programlisting>IPv4: martian source 10.1.0.7 from 10.1.0.1, on dev eth0</programlisting>

            <para>There are currently two possible solutions to this
            particular problem:</para>

            <orderedlist numeration="upperalpha">
              <listitem>
                <para>Add all network-interface dependent routes (the ones
                which are deleted when that interface goes down) to your
                distribution's network configuration system. On Redhat and
                derivatives, that would be
                <filename>/etc/sysconfig/network-scripts/route-X</filename>
                (where "X" is the name of the interface in question). On
                Debian and derivatives, it is
                <filename>/etc/network/interfaces</filename>.</para>

                <para>That way, when the network device goes back up again,
                the Linux OS will add these routes "automatically". Using our
                example above - to add a route to 10.1.0.0/24 using the
                default gateway on eth0 and also using the main routing table,
                the following needs to be added to
                <filename>/etc/sysconfig/network-scripts/route-eth0</filename>
                (Redhat and derivatives):</para>

                <programlisting>10.1.0.0/24 dev eth0 table main
</programlisting>

                <para>On Debian and derivatives (in the eth0 stanza of
                <filename>/etc/network/interfaces</filename>):</para>

                <programlisting>iface eth0 ...
       ... 
       post-up ip route add 10.1.0.0/24 dev eth0 table main</programlisting>
              </listitem>

              <listitem>
                <para>A more elegant solution is, in addition to the
                "standard" shorewall package (shorewall-lite, shorewall, etc),
                to add <ulink url="Shorewall-init.html">shorewall-init</ulink>
                to take care of this automatically.</para>

                <para>With this approach, when the network interface is
                brought back up, the OS passes control to /sbin/ifup-local,
                which forms part of the shorewall-init package, and that
                script, in turn, executes the appropriate command to reload
                the network device settings in the already-compiled
                ${VARDIR}/firewall file.</para>

                <para>When shorewall-init is used, all configuration settings
                (routes, interface options etc) are kept in one place and do
                not have to be defined separately (via
                /etc/sysconfig/network-scripts/route-X for example), which
                eases maintenance efforts quite considerably.</para>
              </listitem>
            </orderedlist>
          </listitem>
        </orderedlist>
      </section>
    </section>

    <section>
      <title>Looking at the routing tables</title>

      <para>To look at the various routing tables, you must use the <emphasis
      role="bold">ip</emphasis> utility. To see the entire routing
      configuration (including rules), the command is <command>shorewall show
      routing</command>. To look at an individual provider's table use
      <command>ip route ls table <replaceable>provider</replaceable></command>
      where <replaceable>provider</replaceable> can be either the provider
      name or number.</para>

      <para>Example:</para>

      <programlisting>lillycat:- #<command>ip route ls</command>
144.77.167.142 dev ppp0  proto kernel  scope link  src 144.177.121.199
71.190.227.208 dev ppp1  proto kernel  scope link  src 71.24.88.151
192.168.7.254 dev eth1  scope link  src 192.168.7.1
192.168.7.253 dev eth1  scope link  src 192.168.7.1
192.168.7.0/24 dev eth1  proto kernel  scope link  src 192.168.7.1
192.168.5.0/24 via 192.168.4.2 dev eth0
192.168.4.0/24 dev eth0  proto kernel  scope link  src 192.168.4.223
192.168.1.0/24 via 192.168.4.222 dev eth0
default
        nexthop dev ppp1 weight 2
        nexthop dev ppp0 weight 1
lillycat: #ip <command>route ls table 1</command>
144.77.167.142 dev ppp0  proto kernel  scope link  src 144.177.121.199 
192.168.5.0/24 via 192.168.4.2 dev eth0 
192.168.4.0/24 dev eth0  proto kernel  scope link  src 192.168.4.223 
192.168.1.0/24 via 192.168.4.222 dev eth0 
default dev ppp0  scope link 
lillycat: #</programlisting>
    </section>

    <section id="USE_DEFAULT_RT">
      <title>USE_DEFAULT_RT</title>

      <para>USE_DEFAULT_RT is an option in <ulink
      url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5).</para>

      <para>One of the drawbacks of the Multi-ISP support as described in the
      preceding sections is that changes to the main table made by
      applications are not added to the individual provider tables. This makes
      route rules such as described in <link linkend="Openvpn">one of the
      examples above</link> necessary.</para>

      <para>USE_DEFAULT_RT=Yes works around that problem by passing packets
      through the main table first rather than last. This has a number of
      implications:</para>

      <orderedlist>
        <listitem>
          <para>Both the DUPLICATE and the COPY columns in the providers file
          must remain empty or contain "-". The individual provider routing
          tables generated when USE_DEFAULT_RT=Yes contain only a host route
          to the gateway and a default route via the gateway.</para>
        </listitem>

        <listitem>
          <para>The <emphasis role="bold">balance</emphasis> option is assumed
          for all interfaces that do not have the <emphasis
          role="bold">loose</emphasis> option. When you want both <emphasis
          role="bold">balance</emphasis> and <emphasis
          role="bold">loose</emphasis>, both must be specified.</para>
        </listitem>

        <listitem>
          <para>The default route generated by Shorewall is added to the
          <emphasis>default</emphasis> routing table (253) rather than to the
          main routing table (254).</para>
        </listitem>

        <listitem>
          <para>Packets are sent through the main routing table by a routing
          rule with priority 999. The priority range 1-998 may be used for
          inserting rules that bypass the main table.</para>
        </listitem>

        <listitem>
          <para>You should disable all default route management outside of
          Shorewall. If a default route is inadvertently added to the main
          table while Shorewall is started, then all policy routing will stop
          working except for those routing rules in the priority range
          1-998.</para>
        </listitem>

        <listitem>
          <para>For ppp interfaces, the GATEWAY may remain unspecified ("-").
          For those interfaces managed by dhcpcd or dhclient, you may specify
          'detect' in the GATEWAY column; Shorewall will use the dhcp client's
          database to determine the gateway IP address. All other interfaces
          must have a GATEWAY specified explicitly.</para>
        </listitem>
      </orderedlist>

      <para>The configuration in the figure at the top of this section would
      be specified in <filename>/etc/shorewall/providers</filename> as
      follows.</para>

      <programlisting>#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
ISP1    1       1       -               eth0            206.124.146.254 track            -
ISP2    2       2       -               eth1            130.252.99.254  track            -  </programlisting>

      <para>The remainder of the example is the same.</para>

      <para>Although 'balance' is automatically assumed when
      USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider
      except when you explicitly direct it to use the other provider via
      <ulink url="manpages/shorewall-rtrules.html">shorewall-rtrules</ulink>
      (5) or <ulink
      url="manpages4/manpages/shorewall-tcrules.html">shorewall-mangle</ulink>
      (5).</para>

      <para>Example (send all traffic through the 'shorewall' provider unless
      otherwise directed).</para>

      <para>/etc/shorewall/providers:<programlisting>#NAME    NUMBER MARK DUPLICATE INTERFACE GATEWAY       OPTIONS
linksys       1    1    -        wlan0   172.20.1.1    track,balance=1,optional
shorewall     2    2    -        eth0    192.168.1.254 track,balance=2,optional</programlisting>/etc/shorewall/rtrules:<programlisting>#SOURCE     DEST      PROVIDER        PRIORITY
-           -         shorewall       11999</programlisting></para>

      <para>Tuomo Soini describes the following issue when using
      USE_DEFAULT_RT=Yes.</para>

      <para>He has a /27 network (let.s call it 70.90.191.0/27) from his
      primary ISP and his secondary ISP supplies him with a dynamic IP address
      on the 91.156.0.0/19 network. From the output of <command>shorewall show
      routing</command>:</para>

      <programlisting>999:    from all lookup main 
10000:  from all fwmark 0x100 lookup ISP1 
10001:  from all fwmark 0x200 lookup ISP2</programlisting>

      <para>Note that the main routing table is consulted prior to the marks
      for his two provlders. When clients in the large /19 network connected
      to his /27 (through ISP1), the responses were routed out of the ISP2
      interface because the main routing table included a route to the
      /19.</para>

      <para>The solution was to add an additional entry to rtrules:</para>

      <programlisting>#SOURCE             DEST           PROVIDER       PRIORITY
70.90.191.0/27      91.156.0.0/19  ISP1           900</programlisting>

      <para>With this additional entry, the routing rules are as below and
      traffic from the /27 is returned via ISP1.</para>

      <programlisting>900:    from 70.90.191.0/27 to 91.156.0.0/19 lookup ISP1 
999:    from all lookup main 
10000:  from all fwmark 0x100 lookup ISP1 
10001:  from all fwmark 0x200 lookup ISP2</programlisting>

      <section>
        <title>DHCP with USE_DEFAULT_RT</title>

        <para>When USE_DEFAULT_RT=Yes, you don't want your DHCP client
        inserting a default route into the main routing table.</para>

        <section>
          <title>Debian</title>

          <para>In this Debian-specific example, eth0 is managed by
          dhcpcd.</para>

          <para><filename>/etc/default/dhcpcd</filename>:</para>

          <programlisting># Config file for dhcpcd. Note that you have to edit the interface
# name below, or duplicate the configuration for different interfaces.
# If you are editing this file just to get DNS servers set by DHCP,
# then you should consider installing the resolvconf package instead.

case ${INTERFACE} in
<emphasis role="bold">eth0</emphasis>) 

# Uncomment this to allow dhcpcd to set the DNS servers in /etc/resolv.conf
# If you are using resolvconf then you can leave this commented out.
#SET_DNS='yes'

# Uncomment this to allow dhcpcd to set hostname of the host to the
# hostname option supplied by DHCP server.
#SET_HOSTNAME='yes'

# Uncomment this to allow dhcpcd to set the NTP servers in /etc/ntp.conf
#SET_NTP='yes'

# Uncomment this to allow dhcpcd to set the YP servers in /etc/yp.conf
#SET_YP='yes'

# Add other options here, see man 8 dhcpcd-bin for details.
OPTIONS=(<emphasis role="bold">--nogateway</emphasis> --nodns --nontp <emphasis
              role="bold">--script /etc/shorewall/dhcpcd.sh</emphasis>)
;;

# Add other interfaces here
*)
;;

esac
</programlisting>

          <para><filename>/etc/shorewall/start</filename>:</para>

          <programlisting>cat &lt;&lt;EOF &gt; /var/lib/shorewall/eth0.info
ETH0_GATEWAY=$SW_ETH0_GATEWAY
ETH0_ADDRESS=$SW_ETH0_ADDRESS
EOF</programlisting>

          <para><filename>/etc/shorewall/dhcpd.sh</filename>:</para>

          <programlisting>#!/bin/sh

if [ $2 != down ]; then
    if [ -f /var/lib/dhcpcd/dhcpcd-eth0.info ]; then
        . /var/lib/dhcpcd/dhcpcd-eth0.info
    else
        logger -p daemon.err "/var/lib/dhcpcd/dhcpcd-eth0.info does not exist!"
        exit 1
    fi

    logger -p daemon.info "DHCP-assigned address/gateway for eth0 is $IPADDR/$GATEWAYS"

    [ -f /var/lib/shorewall/eth0.info ] &amp;&amp; . /var/lib/shorewall/eth0.info
    
    if [ "$GATEWAYS" != "$ETH0_GATEWAY" -o "$IPADDR" != "$ETH0_ADDRESS" ]; then
        logger -p daemon.info "eth0 IP configuration changed - restarting foolsm and Shorewall"
        killall foolsm
        /sbin/shorewall restart
    fi
fi
</programlisting>

          <para>A couple of things to notice about
          <filename>/etc/shorewall/dhcpcd.sh</filename>:</para>

          <itemizedlist>
            <listitem>
              <para>It is hard-coded for eth0</para>
            </listitem>

            <listitem>
              <para>It assumes the use of <link linkend="lsm">FOOLSM</link>;
              If you aren't using foolsm, you can change the log message and
              remove the 'killall foolsm'</para>
            </listitem>

            <listitem>
              <para>It restarts Shorewall if the current IPv4 address of eth0
              and the gateway through eth0 are not the same as they were when
              Shorewall was last started.</para>
            </listitem>
          </itemizedlist>
        </section>

        <section>
          <title>RedHat and Derivatives</title>

          <para>On Redhat-based systems, specify DEFROUTE=No in the device's
          ifcfg file.</para>

          <para><filename>/etc/sysconfig/networking/network-scripts/ifcfg-eth2</filename>:</para>

          <programlisting>BOOTPROTO=dhcp
<emphasis role="bold">PERSISTENT_DHCLIENT=yes</emphasis>
PEERDNS=no
PEERNTP=no
<emphasis role="bold">DEFROUTE=no</emphasis>
DHCLIENTARGS="-nc"
DEVICE=eth2
ONBOOT=yes</programlisting>
        </section>

        <section>
          <title>SuSE and Derivatives</title>

          <para>On these systems, set DHCLIENT_SET_DEFAULT_ROUTE=No in the
          device's ifcfg file.</para>
        </section>
      </section>
    </section>

    <section id="load">
      <title>An alternative form of balancing</title>

      <para>Beginning with Shorewall 4.5.0, an alternative to the
      <option>balance</option>=<replaceable>weight</replaceable> option in
      <ulink
      url="manpages/shorewall-providers.html">shorewall-providers</ulink> (5)
      is available in the form of a PROBABILITY column in <ulink
      url="manpages/shorewall-mangle.html">shorewall-mangle</ulink>(5) (<ulink
      url="manpages4/manpages/shorewall-tcrules.html">shorewall-tcrules</ulink>)
      (5). This feature requires the <firstterm>Statistic Match</firstterm>
      capability in your iptables and kernel.</para>

      <para>This method works when there are multiple links to the same ISP
      where both links have the same default gateway.</para>

      <para>The key features of this method are:</para>

      <orderedlist>
        <listitem>
          <para>Providers to be balanced are given a <replaceable>load
          factor</replaceable> using the <option>load</option>= option in
          <ulink
          url="manpages/shorewall-providers.html">shorewall-providers</ulink>
          (5).</para>
        </listitem>

        <listitem>
          <para>A load factor is a number in the range 0 &lt; number &lt;= 1
          and specifies the probability that any particular new connection
          will be assigned to the associated provider.</para>
        </listitem>

        <listitem>
          <para>When one of the interfaces is disabled or enabled, the load
          factors of the currently-available interfaces are adjusted so that
          the sum of these remaining load factors totals to the sum of all
          interfaces that specify <option>load</option>=.</para>
        </listitem>
      </orderedlist>

      <para>Here's an example that sends 1/3 of the connections through
      provider ComcastC and the rest through ComastB.</para>

      <para><filename>/etc/shorewall/shorewall.conf</filename>:</para>

      <programlisting>MARK_IN_FORWARD_CHAIN=No
...
USE_DEFAULT_RT=Yes
...
TC_BITS=0
PROVIDER_BITS=2
PROVIDER_OFFSET=16
MASK_BITS=8
ZONE_BITS=4
</programlisting>

      <note>
        <para>PROVIDER_OFFSET=16 and ZONE_BITS=4 means that the provider mask
        will be 0xf0000.</para>
      </note>

      <para><filename>/etc/shorewall/providers</filename>:</para>

      <programlisting>#NAME    NUMBER MARK DUPLICATE  INTERFACE GATEWAY       OPTIONS
ComcastB 1      -    -          eth1      70.90.191.126 loose,balance,load=0.66666667
ComcastC 2      -    -          eth0      detect        loose,fallback,load=0.33333333</programlisting>

      <note>
        <para>The <option>loose</option> option is specified so that the
        compiler will not generate and rules based on interface IP addresses.
        That way we have complete control over the priority of such rules
        through entries in the rtrules file.</para>
      </note>

      <para><filename>/etc/shorewall/rtrules</filename>:</para>

      <programlisting>#SOURCE             DEST  PROVIDER  PRIORITY
70.90.191.120/29    -     ComcastB  1000
&amp;eth0               -     ComcastC  1000</programlisting>

      <note>
        <para>This example assumes that eth0 has a dynamic address, so
        <emphasis role="bold">&amp;eth0</emphasis> is used in the SOURCE
        column. That will cause the first IP address of eth0 to be substituted
        when the firewall is started/restarted.</para>
      </note>

      <note>
        <para>Priority = 1000 means that these rules will come before rules
        that select a provider based on marks.</para>
      </note>
    </section>

    <section id="LinkMonitor">
      <title>Gateway Monitoring and Failover</title>

      <para>There is an option (FOOLSM) available for monitoring the status of
      provider links and taking action when a failure occurs. FOOLSM assumes
      that each provider has a unique nexthop gateway.</para>

      <para>You specify the <option>optional</option> option in
      <filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE    INTERFACE    BROADCAST       OPTIONS
net      eth0         detect          <emphasis role="bold">optional</emphasis>         
net      eth1         detect          <emphasis role="bold">optional</emphasis></programlisting>

      <section id="lsm">
        <title>Link Status Monitor (FOOLSM)</title>

        <para><ulink url="http://lsm.foobar.fi/">Link Status Monitor</ulink>
        was written by Mika Ilmaranta &lt;ilmis at nullnet.fi&gt; and performs
        more sophisticated monitoring than the simple SWPING script that
        preceded it.</para>

        <important>
          <para>If you have installed Shorewall-init, you should disable its
          ifup/ifdown/NetworkManager integration (set IFUPDOWN=0 in the <ulink
          url="Manpages/shorewall-init.html">Shorewall-init configuration
          file</ulink>) before installing LSM.</para>
        </important>

        <important>
          <para>To avoid an achronym clash with <emphasis>Linux Security
          Module</emphasis>, the Link Status Monitor is now called
          <emphasis>foolsm</emphasis>.</para>
        </important>

        <para>Like many Open Source products, FOOLSM is poorly documented.
        It's main configuration file is normally kept in
        <filename>/etc/foolsm/foolsm.conf</filename>, but the file's name is
        passed as an argument to the foolsm program so you can name it
        anything you want.</para>

        <para>The sample <filename>foolsm.conf</filename> included with the
        product shows some of the possibilities for configuration. One feature
        that is not mentioned in the sample is that an "include" directive is
        supported. This allows additional files to be sourced in from the main
        configuration file.</para>

        <para>FOOLSM monitors the status of the links defined in its
        configuration file and runs a user-provided script when the status of
        a link changes. The script name is specified in the
        <firstterm>eventscript</firstterm> option in the configuration file.
        Key arguments to the script are as follows:</para>

        <variablelist>
          <varlistentry>
            <term>$1</term>

            <listitem>
              <para>The state of the link ('up' or 'down')</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>$2</term>

            <listitem>
              <para>The name of the connection as specified in the
              configuration file.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>$4</term>

            <listitem>
              <para>The name of the network interface associated with the
              connection.</para>
            </listitem>
          </varlistentry>

          <varlistentry>
            <term>$5</term>

            <listitem>
              <para>The email address of the person specified to receive
              notifications. Specified in the
              <firstterm>warn_email</firstterm> option in the configuration
              file.</para>
            </listitem>
          </varlistentry>
        </variablelist>

        <para>It is the responsibility of the script to perform any action
        needed in reaction to the connection state change. The default script
        supplied with FOOLSM composes an email and sends it to $5.</para>

        <para>I personally use FOOLSM here at shorewall.net (configuration is
        described <link linkend="Complete">below</link>). I have set things up
        so that:</para>

        <itemizedlist>
          <listitem>
            <para>Shorewall [re]starts foolsm during processing of the
            <command>start</command> and <command>restore</command> commands.
            I don't have Shorewall restart foolsm during Shorewall
            <command>restart</command> because I restart Shorewall much more
            often than the average user is likely to do.</para>
          </listitem>

          <listitem>
            <para>Shorewall starts foolsm because I have a dynamic IP address
            from one of my providers (Comcast); Shorewall detects the default
            gateway to that provider and creates a secondary configuration
            file (<filename>/etc/foolsm/shorewall.conf</filename>) that
            contains the link configurations. That file is included by
            <filename>/etc/foolsm/foolsm.conf</filename>.</para>
          </listitem>

          <listitem>
            <para>The script run by FOOLSM during state change
            (<filename>/etc/foolsm/script) </filename>writes a<filename>
            ${VARDIR}/xxx.status</filename> file when the status of an
            interface changes. Those files are read by the
            <filename>isusable</filename> extension script (see below).</para>
          </listitem>
        </itemizedlist>

        <para>Below are my relevant configuration files.</para>

        <warning>
          <para>These files only work with Shorewall-perl 4.4 Beta 2 and
          later.</para>
        </warning>

        <para><filename>/etc/shorewall/params:</filename></para>

        <programlisting>EXT_IF=eth0
COM_IF=eth1</programlisting>

        <para><filename>/etc/shorewall/isusable</filename>:</para>

        <programlisting>local status=0
#
# Read the status file (if any) created by /etc/foolsm/script
#
[ -f ${VARDIR}/${1}.status ] &amp;&amp; status=$(cat ${VARDIR}/${1}.status)

return $status</programlisting>

        <para><filename>/etc/shorewall/lib.private</filename>:</para>

        <programlisting>###############################################################################
# Create /etc/foolsm/shorewall.conf
# Remove the current interface status files
# Start foolsm
###############################################################################
start_foolsm() {
   #
   # Kill any existing foolsm process(es)
   #
   killall foolsm 2&gt; /dev/null
   #
   # Create the Shorewall-specific part of the FOOLSM configuration. This file is
   # included by /etc/foolsm/foolsm.conf
   #
   # Avvanta has a static gateway while Comcast's is dynamic
   #
   cat &lt;&lt;EOF &gt; /etc/foolsm/shorewall.conf
connection {
    name=Avvanta
    checkip=206.124.146.254
    device=$EXT_IF
    ttl=2
}

connection {
    name=Comcast
    checkip=${SW_ETH0_GATEWAY:-71.231.152.1}
    device=$COM_IF
    ttl=1
}
EOF
   #
   # Run FOOLSM -- by default, it forks into the background
   #
   /usr/sbin/foolsm -c /etc/foolsm/foolsm.conf &gt;&gt; /var/log/foolsm
}</programlisting>

        <para>eth0 has a dynamic IP address so I need to use the
        Shorewall-detected gateway address ($SW_ETH1_GATEWAY). I supply a
        default value to be used in the event that detection fails.</para>

        <note>
          <para>In Shorewall 4.4.7 and earlier, the variable name is
          ETH1_GATEWAY.</para>
        </note>

        <para><filename>/etc/shorewall/started</filename>:</para>

        <programlisting>##################################################################################
# [re]start foolsm if this is a 'start' command or if foolsm isn't running
##################################################################################
if [ "$COMMAND" = start -o -z "$(ps ax | grep 'foolsm ' | grep -v 'grep ' )" ]; then
    start_foolsm
fi</programlisting>

        <para><filename>/etc/shorewall/restored</filename>:</para>

        <programlisting>##################################################################################
# Start foolsm if it isn't running
##################################################################################
if [ -z "$(ps ax | grep 'foolsm ' | grep -v 'grep ' )" ]; then
   start_foolsm
fi</programlisting>

        <para><filename>/etc/foolsm/foolsm.conf</filename>:</para>

        <programlisting>#
# Defaults for the connection entries
#
defaults {
  name=defaults
  checkip=127.0.0.1
  eventscript=/etc/foolsm/script
  max_packet_loss=20
  max_successive_pkts_lost=7
  min_packet_loss=5
  min_successive_pkts_rcvd=10
  interval_ms=2000
  timeout_ms=2000
  warn_email=you@yourdomain.com
  check_arp=0
  sourceip=
  ttl=0
}

include /etc/foolsm/shorewall.conf</programlisting>

        <para><filename>/etc/foolsm/script</filename> (Shorewall 4.4.23 and
        later - note that this script must be executable by
        root)<programlisting>#!/bin/sh
#
# (C) 2009 Mika Ilmaranta &lt;ilmis@nullnet.fi&gt;
# (C) 2009 Tom Eastep &lt;teastep@shorewall.net&gt;
#
# License: GPLv2
#

STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}

if [ -f /usr/share/shorewall-lite/lib.base ]; then
    VARDIR=/var/lib/shorewall-lite
    STATEDIR=/etc/shorewall-lite
    TOOL=/sbin/shorewall-lite
else
    VARDIR=/var/lib/shorewall
    STATEDIR=/etc/shorewall
    TOOL=/sbin/shorewall
fi

[ -f ${STATEDIR}/vardir ] &amp;&amp; . ${STATEDIR}/vardir

cat &lt;&lt;EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}

Hi,

Connection ${NAME} is now ${STATE}.

Following parameters were passed:
newstate     = ${STATE}
name         = ${NAME}
checkip      = ${CHECKIP}
device       = ${DEVICE}
warn_email   = ${WARN_EMAIL}

Packet counters:
replied      = ${REPLIED} packets replied
waiting      = ${WAITING} packets waiting for reply
timeout      = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late   = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd    = ${CONS_RCVD} consecutively received replies in sequence
cons_wait    = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss    = ${CONS_MISS} consecutive packets that have timed out
avg_rtt      = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this

Your FOOLSM Daemon

EOM

if [ ${STATE} = up ]; then
# echo 0 &gt; ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier
  ${VARDIR}/firewall enable ${DEVICE}
else
#  echo 1 &gt; ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier
   ${VARDIR}/firewall disable ${DEVICE}
fi

$TOOL show routing &gt;&gt; /var/log/foolsm

exit 0

#EOF</programlisting>Prior to Shorewall 4.4.23, it was necessary to restart
        the firewall when an interface transitions between the usable and
        unusable states.<programlisting>#!/bin/sh
#
# (C) 2009 Mika Ilmaranta &lt;ilmis@nullnet.fi&gt;
# (C) 2009 Tom Eastep &lt;teastep@shorewall.net&gt;
#
# License: GPLv2
#

STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}

if [ -f /usr/share/shorewall-lite/lib.base ]; then
    VARDIR=/var/lib/shorewall-lite
    STATEDIR=/etc/shorewall-lite
    TOOL=/sbin/shorewall-lite
else
    VARDIR=/var/lib/shorewall
    STATEDIR=/etc/shorewall
    TOOL=/sbin/shorewall
fi

[ -f ${STATEDIR}/vardir ] &amp;&amp; . ${STATEDIR}/vardir

cat &lt;&lt;EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}

Hi,

Connection ${NAME} is now ${STATE}.

Following parameters were passed:
newstate     = ${STATE}
name         = ${NAME}
checkip      = ${CHECKIP}
device       = ${DEVICE}
warn_email   = ${WARN_EMAIL}

Packet counters:
replied      = ${REPLIED} packets replied
waiting      = ${WAITING} packets waiting for reply
timeout      = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late   = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd    = ${CONS_RCVD} consecutively received replies in sequence
cons_wait    = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss    = ${CONS_MISS} consecutive packets that have timed out
avg_rtt      = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this

Your FOOLSM Daemon

EOM

# Uncomment the next two lines if you are running Shorewall 4.4.x or earlier

# [ ${STATE} = up ] &amp;&amp; state=0 || state=1
# echo $state &gt; ${VARDIR}/${DEVICE}.status

<emphasis role="bold">$TOOL restart -f &gt;&gt; /var/log/foolsm 2&gt;&amp;1</emphasis>

$TOOL show routing &gt;&gt; /var/log/foolsm

exit 0

#EOF</programlisting></para>
      </section>
    </section>

    <section id="Shared">
      <title>Two Providers Sharing an Interface</title>

      <para>Shared interface support has the following characteristics:</para>

      <orderedlist numeration="loweralpha">
        <listitem>
          <para>Only Ethernet (or Ethernet-like) interfaces can be used. For
          inbound traffic, the MAC addresses of the gateway routers are used
          to determine which provider a packet was received through. Note that
          only routed traffic can be categorized using this technique.</para>
        </listitem>

        <listitem>
          <para>You must specify the address on the interface that corresponds
          to a particular provider in the INTERFACE column by following the
          interface name with a colon (":") and the address.</para>
        </listitem>

        <listitem>
          <para>Entries in <filename>/etc/shorewall/masq</filename> and
          <filename>/etc/shorewall/snat</filename> must be qualified by the
          provider name (or number).</para>
        </listitem>

        <listitem>
          <para>This feature requires Realm Match support in your kernel and
          iptables.</para>
        </listitem>

        <listitem>
          <para>You must add rtrules entries for networks that are accessed
          through a particular provider.</para>
        </listitem>

        <listitem>
          <para>If you have additional IP addresses through either provider,
          you must add <filename>rtrules</filename> to direct traffic FROM
          each of those addresses through the appropriate provider.</para>
        </listitem>

        <listitem>
          <para>You must manually add MARK rules for traffic known to come
          from each provider.</para>
        </listitem>

        <listitem>
          <para>You must specify a gateway IP address in the GATEWAY column
          of<filename> /etc/shorewall/providers</filename>; <emphasis
          role="bold">detect</emphasis> is not permitted.</para>
        </listitem>
      </orderedlist>

      <para>Taken together, b. and h. effectively preclude using this
      technique with dynamic IP addresses.</para>

      <para>Example:</para>

      <para>This is our home network circa fall 2008. We have two Internet
      providers:</para>

      <orderedlist>
        <listitem>
          <para>Comcast -- Cable modem with one dynamic IP address.</para>
        </listitem>

        <listitem>
          <para>Avvanta -- ADSL with 5 static IP addresses.</para>
        </listitem>
      </orderedlist>

      <para>Because the old <trademark>Compaq</trademark>
      <trademark>Presario</trademark> that I use for a firewall only has three
      PCI slots and no onboard Ethernet, it doesn't have enough Ethernet
      controllers to support both providers. So I use a Linksys WRT300n pre-N
      router as a gateway to Comcast. Note that because the Comcast IP address
      is dynamic, I could not share a single firewall interface between the
      two providers directly.</para>

      <para>On my personal laptop (ursa), I have 9 virtual machines running
      various Linux distributions. <emphasis>It is the Shorewall configuration
      on ursa that I will describe here</emphasis>.</para>

      <para>Below is a diagram of our network:<graphic align="center"
      fileref="images/Network2008a.png"/></para>

      <para>The local wired network in my office is connected to both gateways
      and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast
      gateway has local IP address 172.20.1.1 while the Avvanta gateway has
      local IP address 172.20.1.254. Ursa's eth0 interface has a single IP
      address (172.20.1.130).</para>

      <para>This configuration uses USE_DEFAULT_RT=Yes in
      <filename>shorewall.conf </filename>(see <link
      linkend="USE_DEFAULT_RT">above</link>).</para>

      <para>Here is the <filename>providers</filename> file:</para>

      <programlisting>#NAME          NUMBER   MARK DUPLICATE INTERFACE            GATEWAY      OPTIONS                            COPY
comcast        1        1    -         eth0:172.20.1.130    172.20.1.1   track,loose,balance,optional
avvanta        2        2    -         eth0:172.20.1.130    172.20.1.254 track,optional,loose
wireless       3        3    -         wlan0                172.20.1.1   track,optional</programlisting>

      <para>Several things to note:</para>

      <orderedlist>
        <listitem>
          <para>172.20.1.130 is specified as the <filename
          class="devicefile">eth0</filename> IP address for both
          providers.</para>
        </listitem>

        <listitem>
          <para>Both wired providers have the <emphasis
          role="bold">loose</emphasis> option. This prevents Shorewall from
          automatically generating routing rules based on the source IP
          address.</para>
        </listitem>

        <listitem>
          <para>Only <emphasis role="bold">comcast</emphasis> has the
          <emphasis role="bold">balance</emphasis> option. With
          USE_DEFAULT_RT=yes, that means that <emphasis
          role="bold">comcast</emphasis> will be the default provider. While
          <emphasis role="bold">balance</emphasis> is the default, with
          USE_DEFAULT_RT=Yes, it must be specified explicitly when <emphasis
          role="bold">loose</emphasis> is also specified.</para>
        </listitem>

        <listitem>
          <para>I always disable the <emphasis role="bold">wireless</emphasis>
          interface when the laptop is connected to the wired network.</para>
        </listitem>

        <listitem>
          <para>I use a different Shorewall configuration when I take the
          laptop on the road.</para>
        </listitem>
      </orderedlist>

      <para>Here is the rtrules file:<programlisting>#SOURCE                 DEST                    PROVIDER        PRIORITY
-                       206.124.146.176/31      avvanta         1000
-                       206.124.146.178/31      avvanta         1000
-                       206.124.146.180/32      avvanta         1000</programlisting></para>

      <para>Those rules direct traffic to the five static Avvanta IP addresses
      (only two are currently used) through the <emphasis
      role="bold">avvanta</emphasis> provider.</para>

      <para>Here is the mangle file (MARK_IN_FORWARD_CHAIN=No in
      <filename>shorewall.conf</filename>):<programlisting>#ACTION               SOURCE          DEST            PROTO   DPORT           SPORT  USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
MARK(2)               $FW             0.0.0.0/0       tcp     21
MARK(2)               $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
MARK(2)               $FW             0.0.0.0/0       tcp     119</programlisting></para>

      <para>If you are still using a tcrules file, you should consider
      switching to using a mangle file (<command>shorewall update -t</command>
      will do that for you). Here are the equivalent tcrules entries:</para>

      <programlisting>#MARK           SOURCE          DEST            PROTO   DPORT           SPORT   USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
2               $FW             0.0.0.0/0       tcp     21
2               $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
2               $FW             0.0.0.0/0       tcp     119</programlisting>

      <para>These rules:</para>

      <itemizedlist>
        <listitem>
          <para>Use <emphasis role="bold">avvanta</emphasis> for FTP.</para>
        </listitem>

        <listitem>
          <para>Use <emphasis role="bold">avvanta</emphasis> for NTTP</para>
        </listitem>
      </itemizedlist>

      <para>The same rules converted to use the mangle file are:</para>

      <programlisting>#MARK           SOURCE          DEST            PROTO   DPORT           SPORT   USER    TEST    LENGTH  TOS     CONNBYTES       HELPER
MARK(2)         $FW             0.0.0.0/0       tcp     21
MARK(2)         $FW             0.0.0.0/0       tcp     -               -       -       -       -       -       -               ftp
MARK(2)         $FW             0.0.0.0/0       tcp     119</programlisting>

      <para>The remaining files are for a rather standard two-interface config
      with a bridge as the local interface.</para>

      <para><filename>zones</filename>:<programlisting>#ZONE   IPSEC   OPTIONS                 IN_OPTIONS              OUT_OPTIONS
fw      firewall
net     ipv4
kvm     ipv4</programlisting><filename>policy</filename>:<programlisting>net             net             NONE
fw              net             ACCEPT
fw              kvm             ACCEPT
kvm             all             ACCEPT
net             all             DROP            info
all             all             REJECT          info</programlisting></para>

      <para>interfaces:<programlisting>#ZONE    INTERFACE      OPTIONS
#
net     eth0            dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore
net     wlan0           dhcp,tcpflags,routefilter,blacklist,logmartians,optional
kvm     br0             routeback       #Virtual Machines</programlisting><note>
          <para><filename class="devicefile">wlan0</filename> is the wireless
          adapter in the notebook. Used when the laptop is in our home but not
          connected to the wired network.</para>
        </note></para>

      <para>masq:<programlisting>#INTERFACE              SUBNET          ADDRESS         PROTO   DPORT   IPSEC
eth0                    192.168.0.0/24
wlan0                   192.168.0.0/24</programlisting><note>
          <para>Because the firewall has only a single external IP address, I
          don't need to specify the providers in the masq rules.</para>
        </note></para>
    </section>
  </section>

  <section id="Complete">
    <title>A Complete Working Example</title>

    <para>This section describes the network at shorewall.net in late 2013.
    The configuration is as follows:</para>

    <itemizedlist>
      <listitem>
        <para>Two providers:</para>

        <itemizedlist>
          <listitem>
            <para>ComcastC -- A consumer-grade Comcast cable line with a
            dynamic IP address.</para>
          </listitem>

          <listitem>
            <para>ComcastB -- A Comcast Business-class line with 5 static IP
            addresses.</para>
          </listitem>
        </itemizedlist>
      </listitem>

      <listitem>
        <para>A local network consisting of wired and wireless client systems.
        A wireless-N router is used as an access point for the wireless
        hosts.</para>
      </listitem>

      <listitem>
        <para>A DMZ hosting a two servers (one has two public IP addresses -
        one for receiving email and one for sending) and a system dedicaed to
        running irssi (usually via IPv6)</para>
      </listitem>
    </itemizedlist>

    <para>The network is pictured in the following diagram:</para>

    <graphic fileref="images/Network2013.png"/>

    <section>
      <title>IPv4 Configuration</title>

      <para>The Business Gateway manages a gigabit local network with address
      10.0.1.1/24. So The firewall is given address 10.0.1.11/24 and the
      gateway is configured to route the public IP block via that address. The
      gateway's firewall is only enabled for the 10.0.1.0/24 network.</para>

      <para>Because the business network is faster and more reliable, the
      configuration favors sending local network traffic via that uplink
      rather than the consumer line.</para>

      <para>Here are the key entries in
      <filename>/etc/shorewall/params</filename>:</para>

      <programlisting>LOG=NFLOG

INT_IF=eth2
TUN_IF=tun+
COMB_IF=eth1
COMC_IF=eth0

STATISTICAL=
PROXY=
FALLBACK=
PROXYDMZ=
SQUID2=</programlisting>

      <para>The last five variables are used to configure the firewall
      differently to exercise various Shorewall features. Their use requires
      Shorewall 4.5.2 or later.</para>

      <para>Here are the key entries in
      <filename>/etc/shorewall/shorewall.conf</filename>:</para>

      <programlisting>###############################################################################
#                       F I R E W A L L   O P T I O N S
###############################################################################

...

ACCOUNTING_TABLE=mangle

...

AUTOMAKE=Yes

BLACKLISTNEWONLY=Yes

...

EXPAND_POLICIES=No

EXPORTMODULES=Yes

FASTACCEPT=No

..

<emphasis role="bold">KEEP_RT_TABLES=Yes</emphasis> #This is necessary when both IPv4 and IPv6 Multi-ISP are used

LEGACY_FASTSTART=Yes

LOAD_HELPERS_ONLY=Yes

...

MARK_IN_FORWARD_CHAIN=No

MODULE_SUFFIX=ko

MULTICAST=No

MUTEX_TIMEOUT=60

NULL_ROUTE_RFC1918=Yes

OPTIMIZE=31

OPTIMIZE_ACCOUNTING=No

REQUIRE_INTERFACE=No

<emphasis role="bold">RESTORE_DEFAULT_ROUTE=No</emphasis>

RETAIN_ALIASES=No

<emphasis role="bold">ROUTE_FILTER=No</emphasis>

SAVE_IPSETS=

TC_ENABLED=No

TC_EXPERT=No

TC_PRIOMAP="2 3 3 3 2 3 1 1 2 2 2 2 2 2 2 2"

<emphasis role="bold">TRACK_PROVIDERS=Yes</emphasis>

<emphasis role="bold">USE_DEFAULT_RT=Yes</emphasis>

<emphasis role="bold">USE_PHYSICAL_NAMES=Yes</emphasis>

ZONE2ZONE=-

################################################################################
#                       P A C K E T  M A R K  L A Y O U T
################################################################################

TC_BITS=8

<emphasis role="bold">PROVIDER_BITS=2</emphasis>

<emphasis role="bold">PROVIDER_OFFSET=16</emphasis>

MASK_BITS=8

ZONE_BITS=0</programlisting>

      <para>I use USE_DEFAULT_RT=Yes and since there are only two providers,
      two provider bits are all that are required.</para>

      <para>Here is /etc/shorewall/zones:</para>

      <programlisting>fw              firewall
loc             ip           #Local Zone
net             ip           #Internet
smc:net         ip           #10.0.1.0/24
vpn             ip           #OpenVPN clients
dmz             ip           #LXC Containers</programlisting>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE  INTERFACE        OPTIONS
loc    INT_IF           dhcp,physical=$INT_IF,ignore=1,wait=5,routefilter,nets=172.20.1.0/24,routeback
net    COMB_IF          optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMB_IF,upnp,nosmurfs,tcpflags
net    COMC_IF          optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMC_IF,upnp,nosmurfs,tcpflags,dhcp
vpn    TUN_IF+          physical=tun+,ignore=1
dmz    br0              routeback,proxyarp=1,required,wait=30
</programlisting>

      <para><filename>/etc/shorewall/hosts:</filename></para>

      <programlisting>#ZONE   HOST(S)                                 OPTIONS
smc     COMB_IF:10.1.10.0/24
smc     COMC_IF:10.0.0.0/24</programlisting>

      <para><filename>/etc/shorewall/providers</filename>:</para>

      <programlisting>#NAME             NUMBER   MARK    DUPLICATE  INTERFACE	GATEWAY         OPTIONS               COPY
?if $FALLBACK
ComcastB          1        0x10000 -          COMB_IF     70.90.191.126 loose,fallback
ComcastC          2        0x20000 -          COMC_IF     detect        loose,fallback
?elsif $STATISTICAL
ComcastB          1        0x10000 -          COMB_IF     70.90.191.126 loose,load=0.66666667
ComcastC          2        0x20000 -          COMC_IF     detect        loose,load=0.33333333
?else
<emphasis role="bold">ComcastB          1        0x10000 -          COMB_IF     70.90.191.126 loose,balance=2
ComcastC          2        0x20000 -          COMC_IF     detect        loose,balance</emphasis>
?endif
?if $PROXY &amp;&amp; ! $SQUID2
Squid             3        -       -          lo          -             tproxy 
?endif
</programlisting>

      <para>Notice that in the current balance mode, as in the STATISTICAL
      mode, the business line is favored 2:1 over the consumer line.</para>

      <para>Here is <filename>/etc/shorewall/rtrules</filename>:</para>

      <programlisting>#SOURCE             DEST             PROVIDER  PRIORITY
70.90.191.121       -                ComcastB  1000
70.90.191.123       -                ComcastB  1000
&amp;COMC_IF            -                ComcastC  1000
br0                 -                ComcastB  11000
172.20.1.191        -                ComcastB  1000
</programlisting>

      <para>For reference, this configuration generates these routing
      rules:</para>

      <programlisting>root@gateway:~# ip rule ls
0:      from all lookup local 
1:      from all fwmark 0x80000/0x80000 lookup TProxy 
999:    from all lookup main 
1000:   from 70.90.191.121 lookup ComcastB 
1000:   from 70.90.191.123 lookup ComcastB 
1000:   from 172.20.1.191 lookup ComcastB 
1000:   from 10.0.0.4 lookup ComcastC 
10000:  from all fwmark 0x10000/0x30000 lookup ComcastB 
10001:  from all fwmark 0x20000/0x30000 lookup ComcastC 
11000:  from all iif br0 lookup ComcastB 
32765:  from all lookup balance 
32767:  from all lookup default 
root@gateway:~# </programlisting>

      <para><filename>/etc/shorewall/mangle</filename> is not used to support
      Multi-ISP:</para>

      <programlisting>#MARK                           SOURCE        DEST          PROTO  DPORT   SPORT
TTL(+1):P                       INT_IF        -
SAME:P                          INT_IF        -             tcp    80,443
?if $PROXY &amp;&amp; ! $SQUID2
   DIVERT                       COMB_IF       -             tcp    -       80
   DIVERT                       COMC_IF       -             tcp    -       80
   DIVERT                       br0           172.20.1.0/24 tcp    -       80
   TPROXY(3129,172.20.1.254)    INT_IF        -             tcp    80
   ?if $PROXYDMZ
      TPROXY(3129,172.20.1.254) br0           -             tcp    80
   ?endif
?endif
</programlisting>
    </section>

    <section>
      <title>IPv6 Configuration</title>

      <para>The IPv6 configuration has two separate sub-nets, both services
      through 6in4 tunnels from <ulink
      url="http://tunnelbroker.he.net">Hurricane Electric</ulink>. They are
      both configured through the Business IPv4 uplink. I originally had the
      sit2 tunnel configured through the consumer uplink but Comcast (Xfinity)
      decided to start blocking HE IPv6 tunnels on their consumer network,
      preferring their own 6to4 IPv6 solution.</para>

      <para>One HE tunnel handles the servers and one tunnel handles the local
      network.</para>

      <para>Here are the key entries in
      <filename>/etc/shorewall6/shorewall6.conf</filename>:</para>

      <programlisting>###############################################################################
#                      F I R E W A L L  O P T I O N S
###############################################################################

...

FASTACCEPT=No

FORWARD_CLEAR_MARK=Yes

IMPLICIT_CONTINUE=No

<emphasis role="bold">IP_FORWARDING=Keep</emphasis>

<emphasis role="bold">KEEP_RT_TABLES=Yes</emphasis> #Required when both IPv4 and IPv6 Multi-ISP are used

...

TRACK_PROVIDERS=No

<emphasis role="bold">USE_DEFAULT_RT=Yes</emphasis>

ZONE2ZONE=-

...

################################################################################
#                      P A C K E T  M A R K  L A Y O U T
################################################################################

TC_BITS=8

PROVIDER_BITS=8

PROVIDER_OFFSET=8

MASK_BITS=8

ZONE_BITS=0
</programlisting>

      <para>Here is <filename>/etc/shorewall6/zones</filename>:</para>

      <programlisting>#ZONE   TYPE    OPTIONS
fw      firewall
net     ipv6
loc     ipv6
dmz     ipv6</programlisting>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE   INTERFACE       OPTIONS
net     sit1            forward=1,sfilter=2001:470:b:227::40/124,optional
net     sit2            forward=1,sfilter=2001:470:b:227::40/124,optional
net     sit3            forward=1,sfilter=2001:470:b:227::40/124,optional
loc     eth2            forward=1
dmz     br0             routeback,forward=1,required</programlisting>

      <para><filename>/etc/shorewall/providers</filename>:</para>

      <programlisting>#NAME   NUMBER    MARK    DUPLICATE   INTERFACE GATEWAY                OPTIONS            COPY
LOC     4        0x100    -           sit2      -                      track,balance,loose
DMZ     5        0x200    -           sit1      -                      track,fallback,loose
6to4    6        0x300    -           sit3      ::192.88.99.1          track,fallback,loose</programlisting>

      <para>Notice that the provider numbers are disjoint from those in the
      IPv4 configuration. This allows for unique provider names in
      <filename>/etc/iproute2/rt_tables</filename>:</para>

      <programlisting>#
# reserved values
#
255     local
254     main
253     default
250     balance
0       unspec
#
# local
#
1       ComcastB
2       ComcastC
3       TProxy
4       LOC
5       DMZ
6       6to4</programlisting>

      <para>The <filename>/etc/shorewall6/rtrules</filename> file is
      straight-forward:</para>

      <programlisting>#SOURCE                DEST                     PROVIDER             PRIORITY
2001:470:B:227::1/64   ::/0                     DMZ                  11000
2001:470:B:787::1/64   ::/0                     LOC                  11000
2002:465a:bf79::1/64   ::/0                     6to4                 11000</programlisting>

      <para>This results in the following routing rules:</para>

      <programlisting>root@gateway:~# <command>ip -6 rule ls</command>
0:     from all lookup local 
999:   from all lookup main 
10003: from all fwmark 0x100/0xff00 lookup LOC 
10004: from all fwmark 0x200/0xff00 lookup DMZ 
10005: from all fwmark 0x300/0xff00 lookup 6to4 
11000: from 2001:470:b:787::1/64 lookup LOC 
11000: from 2001:470:b:227::1/64 lookup DMZ 
11000: from 2002:465a:bf79::1/64 lookup 6to4 
32765: from all lookup balance 
32767: from all lookup default 
root@gateway:~# </programlisting>
    </section>
  </section>
</article>