extern/shorewall_code
1
0
Fork 1
mirror of https://gitlab.com/shorewall/code.git synced 2024-12-21 13:50:59 +01:00
Code Issues Packages Projects Releases Wiki Activity
9e984b10f6
shorewall_code/docs/MultiISP.xml
Tom Eastep 9e984b10f6 More updates to the Multi-ISP doc 
Signed-off-by: Tom Eastep <teastep@shorewall.net>
2012-10-28 12:10:06 -07:00

2717 lines
106 KiB
XML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
<!--ble$Id$-->
<articleinfo>
<title>Shorewall and Multiple Internet Connections</title>
<authorgroup>
<author>
<firstname>Tom</firstname>
<surname>Eastep</surname>
</author>
</authorgroup>
<pubdate><?dbtimestamp format="Y/m/d"?></pubdate>
<copyright>
<year>2005</year>
<year>2006</year>
<year>2007</year>
<year>2008</year>
<year>2009</year>
<year>2010</year>
<year>2011</year>
<year>2012</year>
<holder>Thomas M. Eastep</holder>
</copyright>
<legalnotice>
<para>Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
<quote><ulink url="GnuCopyright.htm">GNU Free Documentation
License</ulink></quote>.</para>
</legalnotice>
</articleinfo>
<warning>
<para>This document describes the Multi-ISP facility in <emphasis
role="bold">Shorewall 4.4.26 and later</emphasis>. If you are running an
earlier release, please see the documentation for that release.</para>
</warning>
<warning>
<para>Reading just Shorewall documentation is probably not going to give
you enough background to use this material. Shorewall may make iptables
easy but the Shorewall team doesn't have the resources to be able to
spoon-feed Linux policy routing to you (please remember that the user's
manual for a tractor doesn't teach you to grow corn either). You will
likely need to refer to the following additional information:</para>
<itemizedlist>
<listitem>
<para>The LARTC HOWTO: <ulink
url="http://www.lartc.org">http://www.lartc.org</ulink></para>
</listitem>
<listitem>
<para>Output of <command>man ip</command></para>
</listitem>
<listitem>
<para>Output of <command>ip route help</command> and <command>ip rule
help</command></para>
</listitem>
</itemizedlist>
</warning>
<section id="Support">
<title>Multiple Internet Connection Support</title>
<para>Shorewall includes limited support for multiple Internet
connections. Limitations of this support are as follows:</para>
<itemizedlist>
<listitem>
<para>It utilizes static routing configuration. If there is a change
in the routing topopogy, Shorewall must be restarted.</para>
</listitem>
<listitem>
<para>The routing changes are made and the route cache is purged when
Shorewall is started <emphasis role="bold">and when Shorewall is
restarted</emphasis> (unless you specify the "-n" option to
<command>shorewall restart</command>). Ideally, restarting the packet
filter should have no effect on routing.</para>
</listitem>
<listitem>
<para>For most routing applications, <ulink
url="http://www.quagga.net/">Quagga</ulink> is a better solution
although it requires that your ISPs offer routing protocol
support.</para>
</listitem>
</itemizedlist>
<section id="Overview">
<title>Overview</title>
<para>Let's assume that a firewall is connected via two separate
Ethernet interfaces to two different ISPs.<footnote>
<para>While we describe a setup using different ISPs in this
article, the facility also works with two uplinks from the same
ISP.</para>
</footnote> as in the following diagram.</para>
<graphic align="center" fileref="images/TwoISPs.png" valign="middle"/>
<itemizedlist>
<listitem>
<para>eth0 connects to ISP1. The IP address of eth0 is
206.124.146.176 and the ISP's gateway router has IP address
206.124.146.254.</para>
</listitem>
<listitem>
<para>eth1 connects to ISP 2. The IP address of eth1 is
130.252.99.27 and the ISP's gateway router has IP address
130.252.99.254.</para>
</listitem>
<listitem>
<para>eth2 connects to the local LAN. Its IP configuration is not
relevant to this discussion.</para>
</listitem>
</itemizedlist>
<para>Each of these <firstterm>providers</firstterm> is described in an
entry in the file <filename>/etc/shorewall/providers</filename>.</para>
<para>Entries in <filename>/etc/shorewall/providers</filename> can
specify that outgoing connections are to be load-balanced between the
two ISPs. Entries in <filename>/etc/shorewall/tcrules</filename> and
<filename>/etc/shorewall/rtrules</filename> can be used to direct
particular outgoing connections to one ISP or the other. Use of
<filename>/etc/shorewall/tcrules</filename> is not required for
<filename>/etc/shorewall/providers</filename> to work, but in most
cases, you must select a unique MARK value for each provider so
Shorewall can set up the correct marking rules for you.</para>
<para>When you use the <emphasis role="bold">track</emphasis> option in
<filename>/etc/shorewall/providers</filename>, connections from the
Internet are automatically routed back out of the correct interface and
through the correct ISP gateway. This works whether the connection is
handled by the firewall itself or if it is routed or port-forwarded to a
system behind the firewall.</para>
<para>Shorewall will set up the routing and will update the
<filename>/etc/iproute2/rt_tables</filename> to include the table names
and numbers of the tables that it adds.</para>
<caution>
<para>This feature uses <ulink url="traffic_shaping.htm">packet
marking</ulink> to control the routing. As a consequence, there are
some restrictions concerning entries in
<filename>/etc/shorewall/tcrules</filename>:</para>
<itemizedlist>
<listitem>
<para>Packet marking for traffic control purposes may not be done
in the PREROUTING table for connections involving providers with
'track' specified (see below).</para>
</listitem>
<listitem>
<para>You may not use the SAVE or RESTORE options unless you also
set HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET &gt; 0 with Shorewall
4.4.26 and later) in
<filename>/etc/shorewall/shorewall.conf</filename>.</para>
</listitem>
<listitem>
<para>You may not use connection marking unless you also set
HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET &gt; 0 with Shorewall 4.4.26
and later) in
<filename>/etc/shorewall/shorewall.conf</filename>.</para>
</listitem>
</itemizedlist>
</caution>
<para>The <filename>/etc/shorewall/providers</filename> file can also be
used in other routing scenarios. See the <ulink
url="Shorewall_Squid_Usage.html">Squid documentation</ulink> for an
example.</para>
</section>
<section id="providers">
<title>/etc/shorewall/providers File</title>
<para>Entries in this file have the following columns. As in all
Shorewall configuration files, enter "-" in a column if you don't want
to enter any value.</para>
<variablelist>
<varlistentry>
<term>NAME</term>
<listitem>
<para>The provider name. Must begin with a letter and consist of
letters and digits. The provider name becomes the name of the
generated routing table for this provider.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>NUMBER</term>
<listitem>
<para>A number between 1 and 252. This becomes the routing table
number for the generated table for this provider.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>MARK</term>
<listitem>
<para>A mark value used in your /etc/shorewall/tcrules file to
direct packets to this provider. Shorewall will also mark
connections that have seen input from this provider with this
value and will restore the packet mark in the PREROUTING CHAIN.
Mark values must be in the range 1-255.</para>
<para>Alternatively, you may set HIGH_ROUTE_MARKS=Yes
(PROVIDER_OFFSET &gt; 0 with Shorewall 4.4.26 and later) in
<filename>/etc/shorewall/shorewall.conf</filename>. This allows
you to:</para>
<itemizedlist>
<listitem>
<para>Use connection marks for traffic shaping, provided that
you assign those marks in the FORWARD chain.</para>
</listitem>
<listitem>
<para>Use mark values &gt; 255 for provider marks in this
column.</para>
<itemizedlist>
<listitem>
<para>With HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET=8), these
mark values must be a multiple of 256 in the range
256-65280 (hex equivalent 0x100 - 0xFF00 with the
low-order 8 bits being zero); or</para>
</listitem>
<listitem>
<para>Set WIDE_TC_MARKS=Yes in <ulink
url="manpages/shorewall.conf.html">shorewall.conf
</ulink>(5) (PROVIDER_OFFSET=16), and use mark values in
the range 0x10000 - 0xFF0000 with the low-order 16 bits
being zero.</para>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
<para>This column may be omitted if you don´t use packet marking
to direct connections to a particular provider.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>DUPLICATE</term>
<listitem>
<para>Gives the name or number of a routing table to duplicate.
May be 'main' or the name or number of a previously declared
provider. For most applications, you want to specify 'main' here.
This field should be be specified as '-' when USE_DEFAULT_RT=Yes
in <filename>shorewall.conf</filename></para>
</listitem>
</varlistentry>
<varlistentry>
<term>INTERFACE</term>
<listitem>
<para>The name of the interface to the provider. Where multiple
providers share the same interface, you must follow the name of
the interface by a colon (":") and the IP address assigned by this
provider (e.g., eth0:206.124.146.176). See <link
linkend="Shared">below</link> for additional
considerations.</para>
<para>The interface must have been previously defined in <ulink
url="manpages/shorewall-interfaces.html">shorewall-interfaces</ulink>
(5). In general, that interface should not have the
<option>proxyarp</option> option specified unless
<option>loose</option> is given in the OPTIONS column of this
entry.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>GATEWAY</term>
<listitem>
<para>The IP address of the provider's Gateway router.</para>
<para>You can enter <emphasis role="bold">detect</emphasis> here
and Shorewall will attempt to automatically determine the gateway
IP address.</para>
<para><emphasis role="bold">Hint:</emphasis> <emphasis
role="bold">"detect"</emphasis> is appropriate for use in cases
where the interface named in the INTERFACE column is dynamically
configured via DHCP etc. Be sure, however, that you don't have
stale dhcp client state files in <filename
class="directory">/var/lib/dhcpcd </filename>or
<filename>/var/lib/dhclient-*.lease</filename> because Shorewall
may try to use those stale files to determine the gateway
address.</para>
<para>The GATEWAY may be omitted (enter '-') for point-to-point
links.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>OPTIONS</term>
<listitem>
<para>A comma-separated list from the following:</para>
<variablelist>
<varlistentry>
<term>track</term>
<listitem>
<para><important>
<para>Beginning with Shorwall 4.3.3, <emphasis
role="bold">track</emphasis> defaults to the setting of
the <option>TRACK_PROVIDERS</option> option in <ulink
url="manpages/shorewall.conf">shorewall.conf
</ulink>(5). To disable this option when you have
specified TRACK_PROVIDERS=Yes, you must specify
<emphasis role="bold">notrack</emphasis> (see
below).</para>
</important>If specified, connections FROM this interface
are to be tracked so that responses may be routed back out
this same interface.</para>
<para>You want to specify 'track' if Internet hosts will be
connecting to local servers through this provider. Any time
that you specify 'track', you will normally want to also
specify 'balance' (see below). 'track' will also ensure that
outgoing connections remain stay anchored to a single
provider and don't try to switch providers when route cache
entries expire.</para>
<para>Use of this feature requires that your kernel and
iptables include CONNMARK target and connmark match support
(<emphasis role="bold">Warning</emphasis>: Until recently,
standard <trademark>Debian</trademark> and
<trademark>Ubuntu</trademark> kernels lacked that support.
<emphasis>Both Lenny and Jaunty do have the proper
support</emphasis>).</para>
<important>
<para>If you are running a version of Shorewall earlier
than 4.4.3 and are using
<filename>/etc/shorewall/providers</filename> because you
have multiple Internet connections, we recommend that you
specify <emphasis role="bold">track</emphasis> even if you
don't need it. It helps maintain long-term connections in
which there are significant periods with no
traffic.</para>
</important>
</listitem>
</varlistentry>
<varlistentry>
<term>balance</term>
<listitem>
<para>The providers that have <emphasis
role="bold">balance</emphasis> specified will get outbound
traffic load-balanced among them. Balancing will not be
perfect, as it is route based, and routes are cached. This
means that routes to often-used sites will always be over
the same provider.</para>
<para>By default, each provider is given the same weight (1)
. You can change the weight of a given provider by following
<emphasis role="bold">balance</emphasis> with "=" and the
desired weight (e.g., balance=2). The weights reflect the
relative bandwidth of the providers connections and should
be small numbers since the kernel actually creates
additional default routes for each weight increment.</para>
<important>
<para>If you are using
<filename>/etc/shorewall/providers</filename> because you
have multiple Internet connections, we recommend that you
specify <emphasis role="bold">balance</emphasis> even if
you don't need it. You can still use entries in
<filename>/etc/shorewall/tcrules</filename> and
<filename>/etc/shorewall/rtrules</filename> to force all
traffic to one provider or another.<note>
<para>If you don't heed this advice then please read
and follow the advice in <ulink
url="FAQ.htm#faq57">FAQ 57</ulink> and <ulink
url="FAQ.htm#faq58">FAQ 58</ulink>.</para>
</note></para>
</important>
</listitem>
</varlistentry>
<varlistentry>
<term>loose</term>
<listitem>
<para>Do not generate routing rules that force traffic whose
source IP is an address of the INTERFACE to be routed to
this provider. Useful for defining providers that are to be
used only when the appropriate packet mark is
applied.</para>
<para>Shorewall makes no attempt to consolidate the routing
rules added when <emphasis role="bold">loose</emphasis> is
not specified. So, if you have multiple IP addresses on a
provider interface, you may be able to replace the rules
that Shorewall generates with one or two rules in
<filename>/etc/shorewall/rtrules</filename>. In that case,
you can specify <emphasis role="bold">loose</emphasis> to
suppress Shorewall's rule generation. See the <link
linkend="Complete">example</link> below.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>notrack</term>
<listitem>
<para>Added in Shorewall 4.4.3. This option turns off the
<emphasis role="bold">track</emphasis> option.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>optional</term>
<listitem>
<note>
<para>This option is deprecated in favor of the
<option>optional</option> <ulink
url="manpages/shorewall-interfaces.html">interface
option</ulink>. That option performs the same
function.</para>
</note>
<para>Shorewall will determine if this interface is up and
has a configured IP address. If it is not, a warning is
issued and this provider is not configured.</para>
<note>
<para><emphasis role="bold">optional</emphasis> is
designed to detect interface states that will cause
<command>shorewall start</command> or <command>shorewall
restart</command> to fail; just because an interface is in
a state that Shorewall can [re]start without error doesn't
mean that traffic can actually be sent through the
interface.</para>
<para>You can supply an 'isusable' <ulink
url="shorewall_extension_scripts.htm">extension
script</ulink> to extend Shorewall's interface state
detection. See also the <link
linkend="LinkMonitor">Gateway Monitoring and
Failover</link> section below.</para>
</note>
</listitem>
</varlistentry>
<varlistentry>
<term>src=<replaceable>source-address</replaceable></term>
<listitem>
<para>Specifies the source address to use when routing to
this provider and none is known (the local client has bound
to the 0 address). May not be specified when an
<replaceable>address</replaceable> is given in the INTERFACE
column. If this option is not used, Shorewall substitutes
the primary IP address on the interface named in the
INTERFACE column.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>mtu=<replaceable>number</replaceable></term>
<listitem>
<para>Specifies the MTU when forwarding through this
provider. If not given, the MTU of the interface named in
the INTERFACE column is assumed.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>fallback[=<replaceable>weight</replaceable>]</term>
<listitem>
<para>Indicates that a default route through the provider
should be added to the <firstterm>default</firstterm>
routing table (table 253). If a
<replaceable>weight</replaceable> is given, a balanced route
is added with the weight of this provider equal to the
specified <replaceable>weight</replaceable>. If the option
is given without a <replaceable>weight</replaceable>, a
separate default route is added through the provider's
gateway; the route has a metric equal to the provider's
NUMBER.</para>
<para>Prior to Shorewall 4.4.24, the option is ignored with
a warning message if USE_DEFAULT_RT=Yes in
<filename>shorewall.conf</filename>.</para>
</listitem>
</varlistentry>
</variablelist>
<para>For those of you who are confused between<emphasis
role="bold"> track</emphasis> and <emphasis
role="bold">balance</emphasis>:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">track</emphasis> governs incoming
connections (but is also useful for binding long-running
connections to the same interface).</para>
</listitem>
<listitem>
<para><emphasis role="bold">balance</emphasis> governs
outgoing connections.</para>
</listitem>
</itemizedlist>
</listitem>
</varlistentry>
<varlistentry>
<term>COPY</term>
<listitem>
<para>A comma-separated list of interface names. Wildcards
specified using an asterisk ("*") are permitted (e.g., tun*
).</para>
<para>When you specify an existing table in the DUPLICATE column,
Shorewall copies all routes through the interface specified in the
INTERFACE column plus the interfaces listed in this column.
Normally, you will list all interfaces on your firewall in this
column except those Internet interfaces specified in the INTERFACE
column of entries in this file.</para>
<note>
<para>Beginning with Shorewall 4.4.15, provider routing tables
can be augmeted with additional routes through use of the <link
linkend="routes">/etc/shorewall/routes</link> file.</para>
</note>
</listitem>
</varlistentry>
</variablelist>
</section>
<section id="Providers">
<title>What an entry in the Providers File Does</title>
<para>Adding another entry in the providers file simply creates an
alternate routing table for you (see the<ulink
url="http://www.lartc.org"> LARTC Howto</ulink>). The table will usually
contain two routes:</para>
<orderedlist>
<listitem>
<para>A host route to the specified GATEWAY through the specified
INTERFACE.</para>
</listitem>
<listitem>
<para>A default route through the GATEWAY.</para>
</listitem>
</orderedlist>
<para>Note that the first route is omitted if "-" is specified as the
GATEWAY; in that case, the default route does not specify a gateway
(point-to-point link).</para>
<para>If the DUPLICATE column is non-empty, then routes from the table
named in that column are copied into the new table. By default, all
routes (except default routes) are copied. The set of routes copied can
be restricted using the COPY column which lists the interfaces whose
routes you want copied. You will generally want to include all local
interfaces in this list. You should exclude the loopback interface (lo)
and any interfaces that do not have an IP configuration. You should also
omit interfaces like <emphasis role="bold">tun</emphasis> interfaces
that are created dynamically. Traffic to networks handled by those
interfaces should be routed through the main table using entries in
<filename>/etc/shorewall/rtrules</filename> (see Example 2 <link
linkend="Examples">below</link>) or by using <link
linkend="USE_DEFAULT_RT">USE_DEFAULT_RT=Yes</link>.</para>
<para>In addition:</para>
<orderedlist>
<listitem>
<para>Unless <emphasis role="bold">loose</emphasis> is specified, an
ip rule is generated for each IP address on the INTERFACE that
routes traffic from that address through the associated routing
table.</para>
</listitem>
<listitem>
<para>If you specify <emphasis role="bold">track</emphasis>, then
connections which have had at least one packet arrive on the
interface listed in the INTERFACE column have their connection mark
set to the value in the MARK column. In the PREROUTING chain,
packets with a connection mark have their packet mark set to the
value of the associated connection mark; packets marked in this way
bypass any prerouting rules that you create in
<filename>/etc/shorewall/tcrules</filename>. This ensures that
packets associated with connections from outside are always routed
out of the correct interface.</para>
</listitem>
<listitem>
<para>If you specify <emphasis role="bold">balance</emphasis>, then
Shorewall will replace the 'default' route with weight 100 in the
'main' routing table with a load-balancing route among those
gateways where <emphasis role="bold">balance</emphasis> was
specified. So if you configure default routes, be sure that their
weight is less than 100 or the route added by Shorewall will not be
used.</para>
</listitem>
</orderedlist>
<para>That's <emphasis role="bold">all</emphasis> that these entries do.
You still have to follow the principle stated in the <ulink
url="Shorewall_and_Routing.html">Shorewall Routing
documentation</ulink>:</para>
<orderedlist>
<listitem>
<para>Routing determines where packets are to be sent.</para>
</listitem>
<listitem>
<para>Once routing determines where the packet is to go, the
firewall (Shorewall) determines if the packet is allowed to go there
and controls rewriting of the SOURCE IP address
(SNAT/MASQUERADE).</para>
</listitem>
</orderedlist>
<para>The bottom line is that if you want traffic to go out through a
particular provider then you <emphasis>must </emphasis>mark that traffic
with the provider's MARK value in
<filename>/etc/shorewall/tcrules</filename> and you must do that marking
in the PREROUTING chain; or, you must provide the appropriate rules in
<filename>/etc/shorewall/rtrules</filename>.</para>
</section>
<section>
<title>/etc/shorewall/masq and Multi-ISP</title>
<para>If you masquerade a local network, you will need to add masquerade
rules for both external interfaces. Referring to the diagram above, if
each of the interfaces has only a single IP address and you have no
systems with public IP addresses behind your firewall, then I suggest
the following simple entries:</para>
<programlisting>#INTERFACE SOURCE ADDRESS
eth0 0.0.0.0/0 206.124.146.176
eth1 0.0.0.0/0 130.252.99.27</programlisting>
<para>If you have a public subnet (for example 206.124.146.176/30)
behind your firewall, then use exclusion:</para>
<programlisting>#INTERFACE SOURCE ADDRESS
eth0 !206.124.146.176/29 206.124.146.176
eth1 0.0.0.0/0 130.252.99.27</programlisting>
<para>Note that exclusion is only used on the interface corresponding to
internal subnetwork.</para>
<para>If you have multiple IP addresses on one of your interfaces, you
can use a similar technique -- simplY exclude the smallest network that
contains all of those addresses from being masqueraded.</para>
<warning>
<para>Entries in <filename>/etc/shorewall/masq</filename> have no
effect on which ISP a particular connection will be sent through. That
is rather the purpose of entries in
<filename>/etc/shorewall/tcrules</filename> and
<filename>/etc/shorewall/rtrules</filename>.</para>
</warning>
</section>
<section id="Martians">
<title>Martians</title>
<para>One problem that often arises with Multi-ISP configuration is
'Martians'. If your Internet interfaces are configured with the
<emphasis role="bold">routefilter</emphasis> option in
<filename>/etc/shorewall/interfaces</filename> (remember that if you set
that option, you should also select <emphasis
role="bold">logmartians</emphasis>), then things may not work correctly
and you will see messages like this:</para>
<programlisting>Feb 9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1
Feb 9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00</programlisting>
<para>The above message is somewhat awkwardly phrased. The source IP in
this incoming packet was 64.86.88.116 and the destination IP address was
206.124.146.176. Another gotcha is that the incoming packet has already
had the destination IP address changed for DNAT or because the original
outgoing connection was altered by an entry in
<filename>/etc/shorewall/masq</filename> (SNAT or Masquerade). So the
destination IP address (206.124.146.176) may not have been the
destination IP address in the packet as it was initially
received.</para>
<para>There a couple of common causes for these problems:</para>
<orderedlist>
<listitem>
<para>You have connected both of your external interfaces to the
same hub/switch. Connecting multiple firewall interfaces to a common
hub or switch is always a bad idea that will result in
hard-to-diagnose problems.</para>
</listitem>
<listitem>
<para>You are specifying both the <emphasis
role="bold">loose</emphasis> and <emphasis
role="bold">balance</emphasis> options on your provider(s). This can
cause individual connections to ping-pong back and forth between the
interfaces which is almost guaranteed to cause problems.</para>
</listitem>
<listitem>
<para>You are redirecting traffic from the firewall system out of
one interface or the other using packet marking in your
<filename>/etc/shorewall/tcrules</filename> file. A better approach
is to configure the application to use the appropriate local IP
address (the IP address of the interface that you want the
application to use). See <link linkend="Local">below</link>.</para>
</listitem>
</orderedlist>
<para>If all else fails, remove the <emphasis
role="bold">routefilter</emphasis> option from your external interfaces.
If you do this, you may wish to add rules to log and drop packets from
the Internet that have source addresses in your local networks. For
example, if the local LAN in the above diagram is 192.168.1.0/24, then
you would add this rule:</para>
<programlisting>#ACTION SOURCE DEST
DROP:info net:192.168.1.0/24 all</programlisting>
<para>Be sure the above rule is added before any other rules with
<emphasis>net</emphasis> in the SOURCE column.</para>
</section>
<section id="Example1">
<title id="Example">Legacy Example</title>
<para>This section describes the legacy method of configuring multiple
uplinks. It is deprecated in favor of the USE_DEFAULT_RT=Yes
configuration described <link
linkend="USE_DEFAULT_RT">below</link>.</para>
<para>The configuration in the figure at the top of this section would
be specified in <filename>/etc/shorewall/providers</filename> as
follows.</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
ISP1 1 1 main eth0 206.124.146.254 track,balance eth2
ISP2 2 2 main eth1 130.252.99.254 track,balance eth2</programlisting>
<para>Other configuration files go something like this:</para>
<para><filename>/etc/shorewall/interfaces</filename>:</para>
<programlisting>#ZONE INTERFACE BROADCAST OPTIONS
net eth0 detect …
net eth1 detect …</programlisting>
<para><filename>/etc/shorewall/policy</filename>:</para>
<programlisting>#SOURCE DESTINATION POLICY LIMIT:BURST
net net DROP</programlisting>
<para><filename>/etc/shorewall/masq</filename>:</para>
<programlisting>#INTERFACE SOURCE ADDRESS
eth0 0.0.0.0/0 206.124.146.176
eth1 0.0.0.0/0 130.252.99.27</programlisting>
</section>
<section id="Applications">
<title>Routing a Particular Application Through a Specific
Interface</title>
<para>This continues the example in the preceding section.</para>
<para>Now suppose that you want to route all outgoing SMTP traffic from
your local network through ISP 2. You would make this entry in <ulink
url="traffic_shaping.htm">/etc/shorewall/tcrules</ulink> (and if you are
running a version of Shorewall earlier than 3.0.0, you would set
TC_ENABLED=Yes in <ulink
url="???">/etc/shorewall/shorewall.conf</ulink>).</para>
<programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST
# PORT(S)
2:P &lt;local network&gt; 0.0.0.0/0 tcp 25</programlisting>
<para>Note that traffic from the firewall itself must be handled in a
different rule:</para>
<programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST
# PORT(S)
2 $FW 0.0.0.0/0 tcp 25</programlisting>
</section>
<section id="PortForwarding">
<title>Port Forwarding</title>
<para>Shorewall provides considerable flexibility for port forwarding in
a multi-ISP environment.</para>
<para>Normal port forwarding rules such as the following will forward
from both providers.</para>
<para><filename>/etc/shorewall/rules</filename>:</para>
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL
# PORTS(S) DEST
DNAT net loc:192.168.1.3 tcp 25</programlisting>
<para>Continuing the above example, to forward only connection requests
from ISP 1, you can either:</para>
<orderedlist>
<listitem>
<para>Qualify the SOURCE by ISP 1's interface:</para>
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL
# PORTS(S) DEST
DNAT net<emphasis role="bold">:eth0</emphasis> loc:192.168.1.3 tcp 25</programlisting>
<para>or</para>
</listitem>
<listitem>
<para>Specify the IP address of ISP 1 in the ORIGINAL DEST
column:</para>
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL
# PORTS(S) DEST
DNAT net loc:192.168.1.3 tcp 25 <emphasis
role="bold">- 206.124.146.176</emphasis></programlisting>
</listitem>
</orderedlist>
</section>
<section id="morethan2">
<title>More than 2 Providers</title>
<para>When there are more than two providers, you need to extend the
two-provider case in the expected way:</para>
<orderedlist>
<listitem>
<para>For each external address, you need an entry in
<filename>/etc/shorewall/masq</filename> to handle the case where a
connection using that address as the SOURCE is sent out of the
interfaces other than the one that the address is configured
on.</para>
</listitem>
<listitem>
<para>For each external interface, you need to add an entry to
<filename>/etc/shorewall/masq</filename>.</para>
</listitem>
</orderedlist>
<para>If we extend the above example to add eth3 with IP address
16.105.78.4 with gateway 16.105.78.254, then:</para>
<para><filename>/etc/shorewall/providers</filename>:<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
ISP1 1 1 main eth0 206.124.146.254 track,balance eth2
ISP2 2 2 main eth1 130.252.99.254 track,balance eth2
ISP3 3 3 main eth3 16.105.78.254 track,balance eth2</programlisting></para>
<para><filename>/etc/shorewall/masq</filename>:<programlisting>#INTERFACE SUBNET ADDRESS
eth0 0.0.0.0/0 206.124.146.176
eth1 0.0.0.0/0 130.252.99.27
eth3 0.0.0.0/0 16.105.78.4</programlisting></para>
</section>
<section id="Local">
<title>Applications running on the Firewall -making them use a
particular provider</title>
<para>As <link linkend="Applications">noted above</link>, separate
entries in <filename>/etc/shorewall/tcrules</filename> are required for
traffic originating from the firewall.</para>
<para>Experience has shown that in some cases, problems occur with
applications running on the firewall itself. This is especially true
when you have specified <emphasis role="bold">routefilter</emphasis> on
your external interfaces in /etc/shorewall/interfaces (see <link
linkend="Martians">above</link>). When this happens, it is suggested
that you have the application use specific local IP addresses rather
than 0.</para>
<para>Examples:</para>
<itemizedlist>
<listitem>
<para>Squid: In <filename>squid.conf</filename>, set <emphasis
role="bold">tcp_outgoing_address</emphasis> to the IP address of the
interface that you want Squid to use.</para>
</listitem>
<listitem>
<para>In OpenVPN, set <emphasis role="bold">local
</emphasis>(<emphasis role="bold">--local</emphasis> on the command
line) to the IP address that you want the server to receive
connections on.</para>
</listitem>
</itemizedlist>
<para>Note that some traffic originating on the firewall doesn't have a
SOURCE IP address before routing. At least one Shorewall user reports
that an entry in <filename>/etc/shorewall/rtrules</filename> with 'lo'
in the SOURCE column seems to be the most reliable way to direct such
traffic to a particular ISP.</para>
<para>Example:</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
lo - shorewall 1000</programlisting>
</section>
<section id="rtrules">
<title>/etc/shorewall/rtrules (formerly
/etc/shorewall/route_rules)</title>
<para>The <filename>rtrules</filename> file allows assigning certain
traffic to a particular provider just as entries in the
<filename>tcrules</filename> file. The difference between the two files
is that entries in <filename>rtrules</filename> are independent of
Netfilter.</para>
<section id="Routing_rules">
<title>Routing Rules</title>
<para>Routing rules are maintained by the Linux kernel and can be
displayed using the <command>ip rule ls</command> command. When
routing a packet, the rules are processed in turn until the packet is
successfully routed.</para>
<programlisting>gateway:~ # <command>ip rule ls</command>
0: from all lookup local &lt;=== Local (to the firewall) IP addresses
10001: from all fwmark 0x1 lookup Blarg &lt;=== This and the next rule are generated by the
10002: from all fwmark 0x2 lookup Comcast 'MARK' values in /etc/shorewall/providers.
20000: from 206.124.146.176 lookup Blarg &lt;=== This and the next rule are generated unless
20256: from 24.12.22.33 lookup Comcast 'loose' is specified; based in the output of 'ip addr ls'
32766: from all lookup main &lt;=== This is the routing table shown by 'iproute -n'
32767: from all lookup default &lt;=== This table is usually empty
gateway:~ #</programlisting>
<para>In the above example, there are two providers: Blarg and Comcast
with MARK 1 going to Blarg and mark 2 going to Comcast.</para>
</section>
<section id="rtrules_columns">
<title>Columns in the rtrules file</title>
<para>Columns in the file are:</para>
<variablelist>
<varlistentry>
<term>SOURCE (Optional)</term>
<listitem>
<para>An ip address (network or host) that matches the source IP
address in a packet. May also be specified as an interface name
optionally followed by ":" and an address. If the device 'lo' is
specified, the packet must originate from the firewall
itself.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>DEST (Optional)</term>
<listitem>
<para>An ip address (network or host) that matches the
destination IP address in a packet.</para>
<para>If you choose to omit either SOURCE or DEST, place "-" in
that column. Note that you may not omit both SOURCE and
DEST.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>PROVIDER</term>
<listitem>
<para>The provider to route the traffic through. May be
expressed either as the provider name or the provider
number.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>PRIORITY</term>
<listitem>
<para>The rule's priority which determines the order in which
the rules are processed.</para>
<para>1000-1999 Before Shorewall-generated 'MARK' rules</para>
<para>11000- 11999 After 'MARK' rules but before
Shorewall-generated rules for ISP interfaces.</para>
<para>26000-26999 After ISP interface rules but before 'default'
rule.</para>
<para>Rules with equal priority are applied in the order in
which they appear in the file.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>MARK (Optional - added in Shorewall 4.4.25)</term>
<listitem>
<para>Mark and optional mask in the form
<replaceable>mark</replaceable>[/<replaceable>mask</replaceable>].
For this rule to be applied to a packet, the packet's mark value
must match the <replaceable>mark</replaceable> when logically
anded with the <replaceable>mask</replaceable>. If a
<replaceable>mask</replaceable> is not supplied, Shorewall
supplies a suitable provider mask.</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section>
<title>Multi-ISP and VPN</title>
<para>For those VPN types that use routing to direct traffic to remote
VPN clients (including but not limited to OpenVPN in routed mode and
PPTP), the VPN software adds a host route to the <emphasis
role="bold">main</emphasis> table for each VPN client. The best
approach is to use USE_DEFAULT_RT=Yes as described <link
linkend="USE_DEFAULT_RT">below</link>. If that isn't possible, you
must add a routing rule in the 1000-1999 range to specify the
<emphasis role="bold">main</emphasis> table for traffic addressed to
those clients. See<link linkend="Openvpn"> Example 2</link>
below.</para>
<para>If you have an IPSEC gateway on your firewall, be sure to
arrange for ESP packets to be routed out of the same interface that
you have configured your keying daemon to use.</para>
</section>
<section id="Examples">
<title>Examples</title>
<para><emphasis role="bold">Example 1:</emphasis> You want all traffic
entering the firewall on eth1 to be routed through Comcast.</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
eth1 - Comcast 1000</programlisting>
<para>With this entry, the output of <command>ip rule ls</command>
would be as follows.</para>
<para><programlisting>gateway:~ # <command>ip rule ls</command>
0: from all lookup local
<emphasis role="bold">1000: from all iif eth1 lookup Comcast</emphasis>
10001: from all fwmark 0x1 lookup Blarg
10002: from all fwmark 0x2 lookup Comcast
20000: from 206.124.146.176 lookup Blarg
20256: from 24.12.22.33 lookup Comcast
32766: from all lookup main
32767: from all lookup default
gateway:~ #</programlisting>Note that because we used a priority of 1000, the
test for <filename class="devicefile">eth1</filename> is inserted
before the fwmark tests.</para>
<para id="Openvpn"><emphasis role="bold">Example 2:</emphasis> You use
OpenVPN (routed setup w/tunX) in combination with multiple providers.
In this case you have to set up a rule to ensure that the OpenVPN
traffic is routed back through the tunX interface(s) rather than
through any of the providers. 10.8.0.0/24 is the subnet chosen in your
OpenVPN configuration (server 10.8.0.0 255.255.255.0).</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
- 10.8.0.0/24 main 1000</programlisting>
</section>
</section>
<section id="routes">
<title>/etc/shorewall/routes File</title>
<para>Beginning with Shorewall 4.4.15, additional routes can be added to
the provider routing tables using the /etc/shorewall/routes file.</para>
<para>The columns in the file are as follows.</para>
<variablelist>
<varlistentry>
<term><emphasis role="bold">PROVIDER</emphasis></term>
<listitem>
<para>The name or number of a provider defined in <ulink
url="shorewall-providers.html">shorewall-providers</ulink>
(5).</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">DEST</emphasis></term>
<listitem>
<para>Destination host address or network address.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">GATEWAY</emphasis> (Optional)</term>
<listitem>
<para>If specified, gives the IP address of the gateway to the
DEST.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><emphasis role="bold">DEVICE</emphasis> (Optional)</term>
<listitem>
<para>Specifies the device route. If neither DEVICE nor GATEWAY is
given, then the INTERFACE specified for the PROVIDER in <ulink
url="manpages/shorewall-providers.html">shorewall-providers</ulink>
(5).</para>
</listitem>
</varlistentry>
</variablelist>
<para>Assume the following entry in
<filename>/etc/shorewall/providers</filename>:</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
Comcast 1 - xxx eth2 .... </programlisting>
<para>The following table gives some example entries in the file and the
<command>ip route</command> command which results.</para>
<programlisting><emphasis role="bold">#PROVIDER DEST GATEWAY DEVICE</emphasis> | <emphasis
role="bold"> Generated Command</emphasis>
Comcast 172.20.1.0/24 - eth0 | ip -4 route add 172.20.1.0/24 dev eth0 table 1
Comcast 192.168.4.0/24 172.20.1.1 | ip -4 route add 192.168.1.0/24 via 172.20.1.1 table 1
Comcast 192.168.4.0/24 | ip -4 route add 192.168.4.0/24 dev eth2 table 1 </programlisting>
</section>
<section>
<title>Looking at the routing tables</title>
<para>To look at the various routing tables, you must use the <emphasis
role="bold">ip</emphasis> utility. To see the entire routing
configuration (including rules), the command is <command>shorewall show
routing</command>. To look at an individual provider's table use
<command>ip route ls table <replaceable>provider</replaceable></command>
where <replaceable>provider</replaceable> can be either the provider
name or number.</para>
<para>Example:</para>
<programlisting>lillycat:- #<command>ip route ls</command>
144.77.167.142 dev ppp0 proto kernel scope link src 144.177.121.199
71.190.227.208 dev ppp1 proto kernel scope link src 71.24.88.151
192.168.7.254 dev eth1 scope link src 192.168.7.1
192.168.7.253 dev eth1 scope link src 192.168.7.1
192.168.7.0/24 dev eth1 proto kernel scope link src 192.168.7.1
192.168.5.0/24 via 192.168.4.2 dev eth0
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.223
192.168.1.0/24 via 192.168.4.222 dev eth0
default
nexthop dev ppp1 weight 2
nexthop dev ppp0 weight 1
lillycat: #ip <command>route ls table 1</command>
144.77.167.142 dev ppp0 proto kernel scope link src 144.177.121.199
192.168.5.0/24 via 192.168.4.2 dev eth0
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.223
192.168.1.0/24 via 192.168.4.222 dev eth0
default dev ppp0 scope link
lillycat: #</programlisting>
</section>
<section id="USE_DEFAULT_RT">
<title>USE_DEFAULT_RT</title>
<para>USE_DEFAULT_RT is an option in <ulink
url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5).</para>
<para>One of the drawbacks of the Multi-ISP support as described in the
preceding sections is that changes to the main table made by
applications are not added to the individual provider tables. This makes
route rules such as described in <link linkend="Openvpn">one of the
examples above</link> necessary.</para>
<para>USE_DEFAULT_RT=Yes works around that problem by passing packets
through the main table first rather than last. This has a number of
implications:</para>
<orderedlist>
<listitem>
<para>Both the DUPLICATE and the COPY columns in the providers file
must remain empty or contain "-". The individual provider routing
tables generated when USE_DEFAULT_RT=Yes contain only a host route
to the gateway and a default route via the gateway.</para>
</listitem>
<listitem>
<para>The <emphasis role="bold">balance</emphasis> option is assumed
for all interfaces that do not have the <emphasis
role="bold">loose</emphasis> option. When you want both <emphasis
role="bold">balance</emphasis> and <emphasis
role="bold">loose</emphasis>, both must be specified.</para>
</listitem>
<listitem>
<para>The default route generated by Shorewall is added to the
<emphasis>default</emphasis> routing table (253) rather than to the
main routing table (254).</para>
</listitem>
<listitem>
<para>Packets are sent through the main routing table by a routing
rule with priority 999. The priority range 1-998 may be used for
inserting rules that bypass the main table.</para>
</listitem>
<listitem>
<para>You should disable all default route management outside of
Shorewall. If a default route is inadvertently added to the main
table while Shorewall is started, then all policy routing will stop
working except for those routing rules in the priority range
1-998.</para>
</listitem>
<listitem>
<para>For ppp interfaces, the GATEWAY may remain unspecified ("-").
For those interfaces managed by dhcpcd or dhclient, you may specify
'detect' in the GATEWAY column; Shorewall will use the dhcp client's
database to determine the gateway IP address. All other interfaces
must have a GATEWAY specified explicitly.</para>
</listitem>
</orderedlist>
<para>The configuration in the figure at the top of this section would
be specified in <filename>/etc/shorewall/providers</filename> as
follows.</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
ISP1 1 1 - eth0 206.124.146.254 track -
ISP2 2 2 - eth1 130.252.99.254 track - </programlisting>
<para>The remainder of the example is the same.</para>
<para>Although 'balance' is automatically assumed when
USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider
except when you explicitly direct it to use the other provider via
<ulink url="manpages/shorewall-rtrules.html">shorewall-rtrules</ulink>
(5) or <ulink
url="manpages/shorewall-tcrules.html">shorewall-tcrules</ulink>
(5).</para>
<para>Example (send all traffic through the 'shorewall' provider unless
otherwise directed).</para>
<para>/etc/shorewall/providers:<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS
linksys 1 1 - wlan0 172.20.1.1 track,balance=1,optional
shorewall 2 2 - eth0 192.168.1.254 track,balance=2,optional</programlisting>/etc/shorewall/rtrules:<programlisting>#SOURCE DEST PROVIDER PRIORITY
- - shorewall 11999</programlisting></para>
<para>Tuomo Soini describes the following issue when using
USE_DEFAULT_RT=Yes.</para>
<para>He has a /27 network (let.s call it 70.90.191.0/27) from his
primary ISP and his secondary ISP supplies him with a dynamic IP address
on the 91.156.0.0/19 network. From the output of <command>shorewall show
routing</command>:</para>
<programlisting>999: from all lookup main
10000: from all fwmark 0x100 lookup ISP1
10001: from all fwmark 0x200 lookup ISP2</programlisting>
<para>Note that the main routing table is consulted prior to the marks
for his two provlders. When clients in the large /19 network connected
to his /27 (through ISP1), the responses were routed out of the ISP2
interface because the main routing table included a route to the
/19.</para>
<para>The solution was to add an additional entry to rtrules:</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
70.90.191.0/27 91.156.0.0/19 ISP1 900</programlisting>
<para>With this additional entry, the routing rules are as below and
traffic from the /27 is returned via ISP1.</para>
<programlisting>900: from 70.90.191.0/27 to 91.156.0.0/19 lookup ISP1
999: from all lookup main
10000: from all fwmark 0x100 lookup ISP1
10001: from all fwmark 0x200 lookup ISP2</programlisting>
<section>
<title>DHCP with USE_DEFAULT_RT</title>
<para>When USE_DEFAULT_RT=Yes, you don't want your DHCP client
inserting a default route into the main routing table.</para>
<section>
<title>Debian</title>
<para>In this Debian-specific example, eth0 is managed by
dhcpcd.</para>
<para><filename>/etc/default/dhcpcd</filename>:</para>
<programlisting># Config file for dhcpcd. Note that you have to edit the interface
# name below, or duplicate the configuration for different interfaces.
# If you are editing this file just to get DNS servers set by DHCP,
# then you should consider installing the resolvconf package instead.
case ${INTERFACE} in
<emphasis role="bold">eth0</emphasis>)
# Uncomment this to allow dhcpcd to set the DNS servers in /etc/resolv.conf
# If you are using resolvconf then you can leave this commented out.
#SET_DNS='yes'
# Uncomment this to allow dhcpcd to set hostname of the host to the
# hostname option supplied by DHCP server.
#SET_HOSTNAME='yes'
# Uncomment this to allow dhcpcd to set the NTP servers in /etc/ntp.conf
#SET_NTP='yes'
# Uncomment this to allow dhcpcd to set the YP servers in /etc/yp.conf
#SET_YP='yes'
# Add other options here, see man 8 dhcpcd-bin for details.
OPTIONS=(<emphasis role="bold">--nogateway</emphasis> --nodns --nontp <emphasis
role="bold">--script /etc/shorewall/dhcpcd.sh</emphasis>)
;;
# Add other interfaces here
*)
;;
esac
</programlisting>
<para><filename>/etc/shorewall/start</filename>:</para>
<programlisting>cat &lt;&lt;EOF &gt; /var/lib/shorewall/eth0.info
ETH0_GATEWAY=$SW_ETH0_GATEWAY
ETH0_ADDRESS=$SW_ETH0_ADDRESS
EOF</programlisting>
<para><filename>/etc/shorewall/dhcpd.sh</filename>:</para>
<programlisting>#!/bin/sh
if [ $2 != down ]; then
if [ -f /var/lib/dhcpcd/dhcpcd-eth0.info ]; then
. /var/lib/dhcpcd/dhcpcd-eth0.info
else
logger -p daemon.err "/var/lib/dhcpcd/dhcpcd-eth0.info does not exist!"
exit 1
fi
logger -p daemon.info "DHCP-assigned address/gateway for eth0 is $IPADDR/$GATEWAYS"
[ -f /var/lib/shorewall/eth0.info ] &amp;&amp; . /var/lib/shorewall/eth0.info
if [ "$GATEWAYS" != "$ETH0_GATEWAY" -o "$IPADDR" != "$ETH0_ADDRESS" ]; then
logger -p daemon.info "eth0 IP configuration changed - restarting lsm and Shorewall"
killall lsm
/sbin/shorewall restart
fi
fi
</programlisting>
<para>A couple of things to notice about
<filename>/etc/shorewall/dhcpcd.sh</filename>:</para>
<itemizedlist>
<listitem>
<para>It is hard-coded for eth0</para>
</listitem>
<listitem>
<para>It assumes the use of <link linkend="lsm">LSM</link>; If
you aren't using lSM, you can change the log message and remove
the 'killall lsm'</para>
</listitem>
<listitem>
<para>It restarts Shorewall if the current IPv4 address of eth0
and the gateway through eth0 are not the same as they were when
Shorewall was last started.</para>
</listitem>
</itemizedlist>
</section>
<section>
<title>RedHat and Derivatives</title>
<para>On Redhat-based systems, specify DEFROUTE=No in the device's
ifcfg file.</para>
<para><filename>/etc/sysconfig/networking/network-scripts/ifcfg-eth2</filename>:</para>
<programlisting>BOOTPROTO=dhcp
<emphasis role="bold">PERSISTENT_DHCLIENT=yes</emphasis>
PEERDNS=no
PEERNTP=no
<emphasis role="bold">DEFROUTE=no</emphasis>
DHCLIENTARGS="-nc"
DEVICE=eth2
ONBOOT=yes</programlisting>
</section>
<section>
<title>SuSE and Derivatives</title>
<para>On these systems, set DHCLIENT_SET_DEFAULT_ROUTE=No in the
device's ifcfg file.</para>
</section>
</section>
</section>
<section id="load">
<title>An alternative form of balancing</title>
<para>Beginning with Shorewall 4.5.0, an alternative to the
<option>balance</option>=<replaceable>weight</replaceable> option in
<ulink
url="manpages/shorewall-providers.html">shorewall-providers</ulink> (5)
is available in the form of a PROBABILITY column in <ulink
url="???">shorewall-tcrules</ulink> (5). This feature requires the
<firstterm>Statistic Match</firstterm> capability in your iptables and
kernel.</para>
<para>This method works when there are multiple links to the same ISP
where both links have the same default gateway.</para>
<para>The key features of this method are:</para>
<orderedlist>
<listitem>
<para>Providers to be balanced are given a <replaceable>load
factor</replaceable> using the <option>load</option>= option in
<ulink
url="manpages/shorewall-providers.html">shorewall-providers</ulink>
(5).</para>
</listitem>
<listitem>
<para>A load factor is a number in the range 0 &lt; number &lt;= 1
and specifies the probability that any particular new connection
will be assigned to the associated provider.</para>
</listitem>
<listitem>
<para>When one of the interfaces is disabled or enabled, the load
factors of the currently-available interfaces are adjusted so that
the sum of these remaining load factors totals to the sum of all
interfaces that specify <option>load</option>=.</para>
</listitem>
</orderedlist>
<para>Here's an example that sends 1/3 of the connections through
provider ComcastC and the rest through ComastB.</para>
<para><filename>/etc/shorewall/shorewall.conf</filename>:</para>
<programlisting>MARK_IN_FORWARD_CHAIN=No
...
USE_DEFAULT_RT=Yes
...
TC_BITS=0
PROVIDER_BITS=2
PROVIDER_OFFSET=16
MASK_BITS=8
ZONE_BITS=4
</programlisting>
<note>
<para>PROVIDER_OFFSET=16 and ZONE_BITS=4 means that the provider mask
will be 0xf0000.</para>
</note>
<para><filename>/etc/shorewall/providers</filename>:</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS
ComcastB 1 - - eth1 70.90.191.126 loose,balance,load=0.66666667
ComcastC 2 - - eth0 detect loose,fallback,load=0.33333333</programlisting>
<note>
<para>The <option>loose</option> option is specified so that the
compiler will not generate and rules based on interface IP addresses.
That way we have complete control over the priority of such rules
through entries in the rtrules file.</para>
</note>
<para><filename>/etc/shorewall/rtrules</filename>:</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
70.90.191.120/29 - ComcastB 1000
&amp;eth0 - ComcastC 1000</programlisting>
<note>
<para>This example assumes that eth0 has a dynamic address, so
<emphasis role="bold">&amp;eth0</emphasis> is used in the SOURCE
column. That will cause the first IP address of eth0 to be substituted
when the firewall is started/restarted.</para>
</note>
<note>
<para>Priority = 1000 means that these rules will come before rules
that select a provider based on marks.</para>
</note>
</section>
<section id="LinkMonitor">
<title>Gateway Monitoring and Failover</title>
<para>There are a couple of options available for monitoring the status
of provider links and taking action when a failure occurs. Both of these
options assume that each provider has a unique nexthop gateway; if two
or more providers use the same gateway router then neither option is
suitable.</para>
<para>You specify the <option>optional</option> option in
<filename>/etc/shorewall/interfaces</filename>:</para>
<programlisting>#ZONE INTERFACE BROADCAST OPTIONS
net eth0 detect <emphasis role="bold">optional</emphasis>
net eth1 detect <emphasis role="bold">optional</emphasis></programlisting>
<section id="swping">
<title>SWPING</title>
<para>Shorewall includes a sample monitoring script
<filename>swping</filename>. The <filename>swping</filename> file is
available in the main directory contained in the Shorewall-common
tarball and is included in the Shorewall-common documentation
directory in the Shorewall-common RPM. The script is inspired by
Angsuman Chakraborty's <ulink
url="http://blog.taragana.com/index.php/archive/how-to-load-balancing-failover-with-dual-multi-wan-adsl-cable-connections-on-linux/">gwping</ulink>
script.</para>
<important>
<para>These samples are offered <emphasis>as is</emphasis> — they
work for me but I don't make any claim that they will work for
anyone else. But if you have a need for automated link monitoring,
they offer you a place to start.</para>
</important>
<important>
<para>If you have installed Shorewall-init, you should disable its
ifup/ifdown/NetworkManager integration (set IFUPDOWN=0 in the <ulink
url="Manpages/shorewall-init.html">Shorewall-init configuration
file</ulink>).</para>
</important>
<para>The script should be copied to a directory on root's PATH such
as <filename>/usr/local/sbin/</filename>.</para>
<para>The script works by sending pings to <emphasis>target</emphasis>
IP addresses through each external interface. These targets must not
depend on any routes other than those that are present in the main
routing table. That ensures that a route is available to the target
even when the target's interface is not working and Shorewall has
omitted it from the routing configuration. An interface is assumed to
be <firstterm>up</firstterm> when a specified number (UP_COUNT) of
consecutive ping operations succeed. Similarly, an interface is
assumed to be <firstterm>down</firstterm> when a specified number
(DOWN_COUNT) of consecutive ping operations fail. You can specify the
interval between pings (PING_INTERVAL).</para>
<para>The script monitors two interfaces but it is a trivial exercise
to extend it to more than two. At the top are a number of variables to
set:</para>
<programlisting>#
# IP family -- 4 or 6
#
FAMILY=4
#
# The commands to run when the status of a line changes. Multiple commands may be specified
# when separated by semicolons (";")
#
COMMAND=
...
#
# Interfaces to monitor -- you may use shell variables from your params file
#
IF1=eth0
IF2=eth1
#
# Sites to Ping. Must depend only on routes in the 'main' routing table. If not specified,
# the interface is assumed to be managed by dhcpcd and the script uses the gateway address
# from /var/lib/dhcpcd/dhcpcd-${IFx}.info
#
TARGET1=
TARGET2=
#
# How often to ping
#
PING_INTERVAL=5
#
# Value for ping's -W option
#
PING_TIMEOUT=2
#
# This many successive pings must succeed for the interface to be marked up when it is down
#
UP_COUNT=5
#
# This many successive pings must fail for the interface to be marked down when it is up
#
DOWN_COUNT=2</programlisting>
<para>If you leave COMMAND empty, the script sets its value
automatically depending on whether Shorewall-lite is installed.</para>
<para>When the status of an interface changes:</para>
<itemizedlist>
<listitem>
<para>For each interface, a file is placed in ${VARDIR} (normally
/var/lib/shorewall) to record the status of the interface: either
0 (UP) or 1 (DOWN). The name of the file is
<filename><replaceable>interface</replaceable>.status</filename>
where <replaceable>interface</replaceable> is the interface (e.g.,
<filename>eth0.status</filename>).</para>
<important>
<para>Beginning with Shorewall 4.5.0, the generated script
automatically maintains this .status file.</para>
</important>
</listitem>
<listitem>
<para>A <command>shorewall -f restart</command> command is
executed (<command>shorewall-lite restart</command>, if
Shorewall-lite is installed).</para>
</listitem>
<listitem>
<para>The contents of the main routing table are displayed.</para>
</listitem>
</itemizedlist>
<para>The .status files are intended to be used with the following
<filename>/etc/shorewall/isusable</filename> script.<programlisting>local status=0
[ -f ${VARDIR}/${1}.status ] &amp;&amp; status=$(cat ${VARDIR}/${1}.status)
return $status</programlisting></para>
<para>The above script is installed in <filename
class="directory">/etc/shorewall</filename> in Shorewall releases
4.3.11 - 4.5.0. Beginning with Shorewall 4.5.1, it is no longer
installed in <filename class="directory">/etc/shorewall</filename>,
but may be copied there from <filename
class="directory">/usr/share/shorewall/configfiles</filename>.</para>
<para>Also included is a sample init script
(<filename>swping.init</filename>) to start the monitoring daemon.
Copy it to<filename> /etc/init.d/swping</filename> and use your
distribution's SysV init tools to cause it to be run at boot. It works
on <trademark>OpenSuSE</trademark> 11.0 -- YMMV. Modify the PROG and
STATEDIR variables as needed.</para>
<para>As an alternative to using the init script, you can add the
following to <filename>/etc/shorewall/started</filename>:</para>
<programlisting>if [ "$COMMAND" = start ]; then
killall -9 swping 2&gt; /dev/null #be sure that there are none left running
/usr/local/sbin/swping &amp;
fi</programlisting>
<para>and add this to
<filename>/etc/shorewall/stopped</filename>.</para>
<para><programlisting>if [ "$COMMAND" = stop -o "$COMMAND" = clear ]; then
killall -9 swping 2&gt; /dev/null
fi</programlisting></para>
<para>This simple script has a number of limitations:</para>
<orderedlist>
<listitem>
<para>It only works on IPv4 or IPv6 but not both at once. So if
you want to monitor both IPv4 and IPv6, you need to clone the
script are run two copies; one for IPv4 and one for IPv6.</para>
</listitem>
<listitem>
<para>It can only detect the gateway for interfaces managed by
dhcpcd.</para>
</listitem>
<listitem>
<para>It's method of determining whether an interface is up or
down is crude. You will normally specify the default gateway for
each provider as the sites to ping and being able to ping the
default gateway is not a surefire indication that the provider is
usable. The method of determining whether a site is up or down is
also crude.</para>
</listitem>
<listitem>
<para>Because of the crudeness of the algorithm, hysteresis may
occur.</para>
</listitem>
<listitem>
<para>It is tricky to configure a system such that the system
works correctly when one of its providers is down unless you
largely don't care which interface is used.</para>
</listitem>
</orderedlist>
</section>
<section id="lsm">
<title>Link Status Monitor (LSM)</title>
<para><ulink url="http://lsm.foobar.fi/">Link Status Monitor</ulink>
was written by Mika Ilmaranta &lt;ilmis at nullnet.fi&gt; and performs
more sophisticated monitoring than the simple swping script described
in the preceding section.</para>
<important>
<para>If you have installed Shorewall-init, you should disable its
ifup/ifdown/NetworkManager integration (set IFUPDOWN=0 in the <ulink
url="Manpages/shorewall-init.html">Shorewall-init configuration
file</ulink>) before installing LSM.</para>
</important>
<para>Like many Open Source products, LSM is poorly documented. It's
main configuration file is normally kept in
<filename>/etc/lsm/lsm.conf</filename>, but the file's name is passed
as an argument to the lsm program so you can name it anything you
want.</para>
<para>The sample <filename>lsm.conf</filename> included with the
product shows some of the possibilities for configuration. One feature
that is not mentioned in the sample is that an "include" directive is
supported. This allows additional files to be sourced in from the main
configuration file.</para>
<para>LSM monitors the status of the links defined in its
configuration file and runs a user-provided script when the status of
a link changes. The script name is specified in the
<firstterm>eventscript</firstterm> option in the configuration file.
Key arguments to the script are as follows:</para>
<variablelist>
<varlistentry>
<term>$1</term>
<listitem>
<para>The state of the link ('up' or 'down')</para>
</listitem>
</varlistentry>
<varlistentry>
<term>$2</term>
<listitem>
<para>The name of the connection as specified in the
configuration file.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>$4</term>
<listitem>
<para>The name of the network interface associated with the
connection.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>$5</term>
<listitem>
<para>The email address of the person specified to receive
notifications. Specified in the
<firstterm>warn_email</firstterm> option in the configuration
file.</para>
</listitem>
</varlistentry>
</variablelist>
<para>It is the responsibility of the script to perform any action
needed in reaction to the connection state change. The default script
supplied with LSM composes an email and sends it to $5.</para>
<para>I personally use LSM here at shorewall.net (configuration is
described <link linkend="Complete">below</link>). I have set things up
so that:</para>
<itemizedlist>
<listitem>
<para>Shorewall [re]starts lsm during processing of the
<command>start</command> and <command>restore</command> commands.
I don't have Shorewall restart lsm during Shorewall
<command>restart</command> because I restart Shorewall much more
often than the average user is likely to do.</para>
</listitem>
<listitem>
<para>Shorewall starts lsm because I have a dynamic IP address
from one of my providers (Comcast); Shorewall detects the default
gateway to that provider and creates a secondary configuration
file (<filename>/etc/lsm/shorewall.conf</filename>) that contains
the link configurations. That file is included by
<filename>/etc/lsm/lsm.conf</filename>.</para>
</listitem>
<listitem>
<para>The script run by LSM during state change
(<filename>/etc/lsm/script) </filename>writes a<filename>
${VARDIR}/xxx.status</filename> file when the status of an
interface changes. Those files are read by the
<filename>isusable</filename> extension script (see below).</para>
</listitem>
</itemizedlist>
<para>Below are my relevant configuration files.</para>
<warning>
<para>These files only work with Shorewall-perl 4.4 Beta 2 and
later.</para>
</warning>
<para><filename>/etc/shorewall/isusable</filename>:</para>
<programlisting>local status=0
#
# Read the status file (if any) created by /etc/lsm/script
#
[ -f ${VARDIR}/${1}.status ] &amp;&amp; status=$(cat ${VARDIR}/${1}.status)
return $status</programlisting>
<para><filename>/etc/shorewall/lib.private</filename>:</para>
<programlisting>###############################################################################
# Create /etc/lsm/shorewall.conf
# Remove the current interface status files
# Start lsm
###############################################################################
start_lsm() {
#
# Kill any existing lsm process(es)
#
killall lsm 2&gt; /dev/null
#
# Create the Shorewall-specific part of the LSM configuration. This file is
# included by /etc/lsm/lsm.conf
#
# Avvanta has a static gateway while Comcast's is dynamic
#
cat &lt;&lt;EOF &gt; /etc/lsm/shorewall.conf
connection {
name=Avvanta
checkip=206.124.146.254
device=$EXT_IF
ttl=2
}
connection {
name=Comcast
checkip=${SW_ETH0_GATEWAY:-71.231.152.1}
device=$COM_IF
ttl=1
}
EOF
#
# Since LSM assumes that interfaces start in the 'up' state, remove any
# existing status files that might have an interface in the down state
#
rm -f /var/lib/shorewall/*.status
#
# Run LSM -- by default, it forks into the background
#
/usr/sbin/lsm /etc/lsm/lsm.conf &gt;&gt; /var/log/lsm
}</programlisting>
<para>eth0 has a dynamic IP address so I need to use the
Shorewall-detected gateway address ($SW_ETH1_GATEWAY). I supply a
default value to be used in the event that detection fails.</para>
<note>
<para>In Shorewall 4.4.7 and earlier, the variable name is
ETH1_GATEWAY.</para>
</note>
<para><filename>/etc/shorewall/started</filename>:</para>
<programlisting>##################################################################################
# [re]start lsm if this is a 'start' command or if lsm isn't running
##################################################################################
if [ "$COMMAND" = start -o -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
start_lsm
fi</programlisting>
<para><filename>/etc/shorewall/restored</filename>:</para>
<programlisting>##################################################################################
# Start lsm if it isn't running
##################################################################################
if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then
start_lsm
fi</programlisting>
<para><filename>/etc/lsm/lsm.conf</filename>:</para>
<programlisting>#
# Defaults for the connection entries
#
defaults {
name=defaults
checkip=127.0.0.1
eventscript=/etc/lsm/script
max_packet_loss=20
max_successive_pkts_lost=7
min_packet_loss=5
min_successive_pkts_rcvd=10
interval_ms=2000
timeout_ms=2000
warn_email=you@yourdomain.com
check_arp=0
sourceip=
ttl=0
}
include /etc/lsm/shorewall.conf</programlisting>
<para><filename>/etc/lsm/script</filename><programlisting>#!/bin/sh
#
# (C) 2009 Mika Ilmaranta &lt;ilmis@nullnet.fi&gt;
# (C) 2009 Tom Eastep &lt;teastep@shorewall.net&gt;
#
# License: GPLv2
#
STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}
if [ -f /usr/share/shorewall-lite/lib.base ]; then
VARDIR=/var/lib/shorewall-lite
STATEDIR=/etc/shorewall-lite
else
VARDIR=/var/lib/shorewall
STATEDIR=/etc/shorewall
fi
[ -f ${STATEDIR}/vardir ] &amp;&amp; . ${STATEDIR}/vardir
cat &lt;&lt;EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}
Hi,
Connection ${NAME} is now ${STATE}.
Following parameters were passed:
newstate = ${STATE}
name = ${NAME}
checkip = ${CHECKIP}
device = ${DEVICE}
warn_email = ${WARN_EMAIL}
Packet counters:
replied = ${REPLIED} packets replied
waiting = ${WAITING} packets waiting for reply
timeout = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd = ${CONS_RCVD} consecutively received replies in sequence
cons_wait = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss = ${CONS_MISS} consecutive packets that have timed out
avg_rtt = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this
Your LSM Daemon
EOM
# Uncomment the next two lines if you are running Shorewall 4.4.x or earlier
# [ ${STATE} = up ] &amp;&amp; state=0 || state=1
# echo $state &gt; ${VARDIR}/${DEVICE}.status
/sbin/shorewall restart -f &gt;&gt; /var/log/lsm 2&gt;&amp;1
/sbin/shorewall show routing &gt;&gt; /var/log/lsm
exit 0
#EOF</programlisting>Beginning with Shorewall 4.4.23, it is not necessary to
restart the firewall when an interface transitions between the usable
and unusable
states.<filename>/etc/lsm/script</filename><programlisting>#!/bin/sh
#
# (C) 2009 Mika Ilmaranta &lt;ilmis@nullnet.fi&gt;
# (C) 2009 Tom Eastep &lt;teastep@shorewall.net&gt;
#
# License: GPLv2
#
STATE=${1}
NAME=${2}
CHECKIP=${3}
DEVICE=${4}
WARN_EMAIL=${5}
REPLIED=${6}
WAITING=${7}
TIMEOUT=${8}
REPLY_LATE=${9}
CONS_RCVD=${10}
CONS_WAIT=${11}
CONS_MISS=${12}
AVG_RTT=${13}
if [ -f /usr/share/shorewall-lite/lib.base ]; then
VARDIR=/var/lib/shorewall-lite
STATEDIR=/etc/shorewall-lite
else
VARDIR=/var/lib/shorewall
STATEDIR=/etc/shorewall
fi
[ -f ${STATEDIR}/vardir ] &amp;&amp; . ${STATEDIR}/vardir
cat &lt;&lt;EOM | mail -s "${NAME} ${STATE}, DEV ${DEVICE}" ${WARN_EMAIL}
Hi,
Connection ${NAME} is now ${STATE}.
Following parameters were passed:
newstate = ${STATE}
name = ${NAME}
checkip = ${CHECKIP}
device = ${DEVICE}
warn_email = ${WARN_EMAIL}
Packet counters:
replied = ${REPLIED} packets replied
waiting = ${WAITING} packets waiting for reply
timeout = ${TIMEOUT} packets that have timed out (= packet loss)
reply_late = ${REPLY_LATE} packets that received a reply after timeout
cons_rcvd = ${CONS_RCVD} consecutively received replies in sequence
cons_wait = ${CONS_WAIT} consecutive packets waiting for reply
cons_miss = ${CONS_MISS} consecutive packets that have timed out
avg_rtt = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating this
Your LSM Daemon
EOM
<emphasis role="bold">if [ ${STATE} = up ]; then
# echo 0 &gt; ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier
${VARDIR}/firewall enable ${DEVICE}
else
# echo 1 &gt; ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier
${VARDIR}/firewall disable ${DEVICE}
fi
</emphasis>
/sbin/shorewall show routing &gt;&gt; /var/log/lsm
exit 0
#EOF</programlisting></para>
</section>
</section>
<section id="Shared">
<title>Two Providers Sharing an Interface</title>
<para>Shared interface support has the following characteristics:</para>
<orderedlist numeration="loweralpha">
<listitem>
<para>Only Ethernet (or Ethernet-like) interfaces can be used. For
inbound traffic, the MAC addresses of the gateway routers are used
to determine which provider a packet was received through. Note that
only routed traffic can be categorized using this technique.</para>
</listitem>
<listitem>
<para>You must specify the address on the interface that corresponds
to a particular provider in the INTERFACE column by following the
interface name with a colon (":") and the address.</para>
</listitem>
<listitem>
<para>Entries in <filename>/etc/shorewall/masq</filename> must be
qualified by the provider name (or number).</para>
</listitem>
<listitem>
<para>This feature requires Realm Match support in your kernel and
iptables.</para>
</listitem>
<listitem>
<para>You must add rtrules entries for networks that are accessed
through a particular provider.</para>
</listitem>
<listitem>
<para>If you have additional IP addresses through either provider,
you must add <filename>rtrules</filename> to direct traffic FROM
each of those addresses through the appropriate provider.</para>
</listitem>
<listitem>
<para>You must manually add MARK rules for traffic known to come
from each provider.</para>
</listitem>
<listitem>
<para>You must specify a gateway IP address in the GATEWAY column
of<filename> /etc/shorewall/providers</filename>; <emphasis
role="bold">detect</emphasis> is not permitted.</para>
</listitem>
</orderedlist>
<para>Taken together, b. and h. effectively preclude using this
technique with dynamic IP addresses.</para>
<para>Example:</para>
<para>This is our home network circa fall 2008. We have two Internet
providers:</para>
<orderedlist>
<listitem>
<para>Comcast -- Cable modem with one dynamic IP address.</para>
</listitem>
<listitem>
<para>Avvanta -- ADSL with 5 static IP addresses.</para>
</listitem>
</orderedlist>
<para>Because the old <trademark>Compaq</trademark>
<trademark>Presario</trademark> that I use for a firewall only has three
PCI slots and no onboard Ethernet, it doesn't have enough Ethernet
controllers to support both providers. So I use a Linksys WRT300n pre-N
router as a gateway to Comcast. Note that because the Comcast IP address
is dynamic, I could not share a single firewall interface between the
two providers directly.</para>
<para>On my personal laptop (ursa), I have 9 virtual machines running
various Linux distributions. <emphasis>It is the Shorewall configuration
on ursa that I will describe here</emphasis>.</para>
<para>Below is a diagram of our network:<graphic align="center"
fileref="images/Network2008a.png"/></para>
<para>The local wired network in my office is connected to both gateways
and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast
gateway has local IP address 172.20.1.1 while the Avvanta gateway has
local IP address 172.20.1.254. Ursa's eth0 interface has a single IP
address (172.20.1.130).</para>
<para>This configuration uses USE_DEFAULT_RT=Yes in
<filename>shorewall.conf </filename>(see <link
linkend="USE_DEFAULT_RT">above</link>).</para>
<para>Here is the <filename>providers</filename> file:</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
comcast 1 1 - eth0:172.20.1.130 172.20.1.1 track,loose,balance,optional
avvanta 2 2 - eth0:172.20.1.130 172.20.1.254 track,optional,loose
wireless 3 3 - wlan0 172.20.1.1 track,optional</programlisting>
<para>Several things to note:</para>
<orderedlist>
<listitem>
<para>172.20.1.130 is specified as the <filename
class="devicefile">eth0</filename> IP address for both
providers.</para>
</listitem>
<listitem>
<para>Both wired providers have the <emphasis
role="bold">loose</emphasis> option. This prevents Shorewall from
automatically generating routing rules based on the source IP
address.</para>
</listitem>
<listitem>
<para>Only <emphasis role="bold">comcast</emphasis> has the
<emphasis role="bold">balance</emphasis> option. With
USE_DEFAULT_RT=yes, that means that <emphasis
role="bold">comcast</emphasis> will be the default provider. While
<emphasis role="bold">balance</emphasis> is the default, with
USE_DEFAULT_RT=Yes, it must be specified explicitly when <emphasis
role="bold">loose</emphasis> is also specified.</para>
</listitem>
<listitem>
<para>I always disable the <emphasis role="bold">wireless</emphasis>
interface when the laptop is connected to the wired network.</para>
</listitem>
<listitem>
<para>I use a different Shorewall configuration when I take the
laptop on the road.</para>
</listitem>
</orderedlist>
<para>Here is the rtrules file:<programlisting>#SOURCE DEST PROVIDER PRIORITY
- 206.124.146.176/31 avvanta 1000
- 206.124.146.178/31 avvanta 1000
- 206.124.146.180/32 avvanta 1000</programlisting></para>
<para>Those rules direct traffic to the five static Avvanta IP addresses
(only two are currently used) through the <emphasis
role="bold">avvanta</emphasis> provider.</para>
<para>Here is the tcrules file (MARK_IN_FORWARD_CHAIN=No in
<filename>shorewall.conf</filename>):<programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS CONNBYTES HELPER
# PORT(S)
2 $FW 0.0.0.0/0 tcp 21
2 $FW 0.0.0.0/0 tcp - - - - - - - ftp
2 $FW 0.0.0.0/0 tcp 119</programlisting></para>
<para>These rules:</para>
<itemizedlist>
<listitem>
<para>Use <emphasis role="bold">avvanta</emphasis> for FTP.</para>
</listitem>
<listitem>
<para>Use <emphasis role="bold">avvanta</emphasis> for NTTP</para>
</listitem>
</itemizedlist>
<para>The remaining files are for a rather standard two-interface config
with a bridge as the local interface.</para>
<para><filename>zones</filename>:<programlisting>#ZONE IPSEC OPTIONS IN OUT
# ONLY OPTIONS OPTIONS
fw firewall
net ipv4
kvm ipv4</programlisting><filename>policy</filename>:<programlisting>net net NONE
fw net ACCEPT
fw kvm ACCEPT
kvm all ACCEPT
net all DROP info
all all REJECT info</programlisting></para>
<para>interfaces:<programlisting>#ZONE INTERFACE BROADCAST OPTIONS GATEWAY
#
net eth0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore
net wlan0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional
kvm br0 detect routeback #Virtual Machines</programlisting><note>
<para><filename class="devicefile">wlan0</filename> is the wireless
adapter in the notebook. Used when the laptop is in our home but not
connected to the wired network.</para>
</note></para>
<para>masq:<programlisting>#INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC
eth0 192.168.0.0/24
wlan0 192.168.0.0/24</programlisting><note>
<para>Because the firewall has only a single external IP address, I
don't need to specify the providers in the masq rules.</para>
</note></para>
</section>
</section>
<section id="Complete">
<title>A Complete Working Example</title>
<para>This section describes the network at shorewall.net in late 2012.
The configuration is as follows:</para>
<itemizedlist>
<listitem>
<para>Two providers:</para>
<itemizedlist>
<listitem>
<para>ComcastC -- A consumer-grade Comcast cable line with a
dynamic IP address.</para>
</listitem>
<listitem>
<para>ComcastB -- A Comcast Business-class line with 5 static IP
addresses.</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>A local network consisting of wired and wireless client systems.
A wireless-N router is used as an access point for the wireless
hosts.</para>
</listitem>
<listitem>
<para>A DMZ hosting a two servers (one has two public IP addresses -
one for receiving email and one for sending) and a system dedicaed to
running irssi (usually via IPv6)</para>
</listitem>
</itemizedlist>
<para>The network is pictured in the following diagram:</para>
<graphic fileref="images/Network2012a.png"/>
<section>
<title>IPv4 Configuration</title>
<para>The Business Gateway manages a gigabit local network with address
10.0.1.1/24. So The firewall is given address 10.0.1.11/24 and the
gateway is configured to route the public IP block via that address. The
gateway's firewall is only enabled for the 10.0.1.0/24 network.</para>
<para>Because the business network is faster and more reliable, the
configuration favors sending local network traffic via that uplink
rather than the consumer line.</para>
<para>Here are the key entries in
<filename>/etc/shorewall/params</filename>:</para>
<programlisting>LOG=NFLOG
INT_IF=eth2
TUN_IF=tun+
COMB_IF=eth1
COMC_IF=eth0
STATISTICAL=
PROXY=
FALLBACK=
PROXYDMZ=
SQUID2=</programlisting>
<para>The last three variables are used to configure the firewall
differently to exercise various Shorewall features.</para>
<para>Here are the key entries in
<filename>/etc/shorewall/shorewall.conf</filename>:</para>
<programlisting>###############################################################################
# F I R E W A L L O P T I O N S
###############################################################################
...
ACCOUNTING_TABLE=mangle
...
AUTOMAKE=Yes
BLACKLISTNEWONLY=Yes
...
EXPAND_POLICIES=No
EXPORTMODULES=Yes
FASTACCEPT=No
..
<emphasis role="bold">KEEP_RT_TABLES=Yes</emphasis> #This is necessary when both IPv4 and IPv6 Multi-ISP are used
LEGACY_FASTSTART=Yes
LOAD_HELPERS_ONLY=Yes
...
MARK_IN_FORWARD_CHAIN=No
MODULE_SUFFIX=ko
MULTICAST=No
MUTEX_TIMEOUT=60
NULL_ROUTE_RFC1918=Yes
OPTIMIZE=31
OPTIMIZE_ACCOUNTING=No
REQUIRE_INTERFACE=No
<emphasis role="bold">RESTORE_DEFAULT_ROUTE=No</emphasis>
RETAIN_ALIASES=No
<emphasis role="bold">ROUTE_FILTER=No</emphasis>
SAVE_IPSETS=
TC_ENABLED=No
TC_EXPERT=No
TC_PRIOMAP="2 3 3 3 2 3 1 1 2 2 2 2 2 2 2 2"
<emphasis role="bold">TRACK_PROVIDERS=Yes</emphasis>
<emphasis role="bold">USE_DEFAULT_RT=Yes</emphasis>
<emphasis role="bold">USE_PHYSICAL_NAMES=Yes</emphasis>
ZONE2ZONE=-
################################################################################
# P A C K E T M A R K L A Y O U T
################################################################################
TC_BITS=8
<emphasis role="bold">PROVIDER_BITS=2</emphasis>
<emphasis role="bold">PROVIDER_OFFSET=16</emphasis>
MASK_BITS=8
ZONE_BITS=0</programlisting>
<para>I use USE_DEFAULT_RT=Yes and since there are only two providers,
two provider bits are all that are required.</para>
<para>Here is /etc/shorewall/zones:</para>
<programlisting>fw firewall
loc ip #Local Zone
net ip #Internet
smc:net ip #10.0.1.0/24
vpn ip #OpenVPN clients
dmz ip #LXC Containers</programlisting>
<para><filename>/etc/shorewall/interfaces</filename>:</para>
<programlisting>#ZONE INTERFACE OPTIONS
loc INT_IF dhcp,physical=$INT_IF,required,wait=5,routefilter,nets=172.20.1.0/24
net COMB_IF optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMB_IF,upnp,nosmurfs,tcpflags
net COMC_IF optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMC_IF,upnp,nosmurfs,tcpflags,dhcp
vpn TUN_IF+ physical=tun+,ignore=1
dmz br0 routeback,proxyarp=1
- lo ignore</programlisting>
<para><filename>/etc/shorewall/hosts:</filename></para>
<programlisting>#ZONE HOST(S) OPTIONS
smc COMB_IF:10.1.10.0/24</programlisting>
<para><filename>/etc/shorewall/providers</filename>:</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
?if $FALLBACK
ComcastB 1 0x10000 - COMB_IF 70.90.191.126 loose,fallback
ComcastC 2 0x20000 - COMC_IF detect loose,fallback
?elsif $STATISTICAL
ComcastB 1 0x10000 - COMB_IF 70.90.191.126 loose,load=0.66666667
ComcastC 2 0x20000 - COMC_IF detect loose,load=0.33333333
?else
<emphasis role="bold">ComcastB 1 0x10000 - COMB_IF 70.90.191.126 loose,balance=2
ComcastC 2 0x20000 - COMC_IF detect loose,balance</emphasis>
?endif
?if $PROXY &amp;&amp; ! $SQUID2
Squid 3 - - lo - tproxy
?endif
</programlisting>
<para>Notice that in the current balance mode, as in the STATISTICAL
mode, the business line is favored 2:1 over the consumer line.</para>
<para>Here is <filename>/etc/shorewall/rtrules</filename>:</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
70.90.191.121 - ComcastB 1000
70.90.191.123 - ComcastB 1000
&amp;COMC_IF - ComcastC 1000
172.20.1.145 - ComcastC 1000
172.20.1.146 - ComcastC 1000
br0 - ComcastB 11000</programlisting>
<para>For reference, this configuration generates these routing
rules:</para>
<programlisting>root@gateway:~# ip rule ls
0: from all lookup local
999: from all lookup main
1000: from 70.90.191.121 lookup ComcastB
1000: from 70.90.191.123 lookup ComcastB
1000: from 67.170.121.6 lookup ComcastC
1000: from 172.20.1.145 lookup ComcastC
1000: from 172.20.1.146 lookup ComcastC
10000: from all fwmark 0x10000/0x30000 lookup ComcastB
10001: from all fwmark 0x20000/0x30000 lookup ComcastC
11000: from all iif br0 lookup ComcastB
32765: from all lookup balance
32767: from all lookup default
root@gateway:~# </programlisting>
<para><filename>/etc/shorewall/tcrules</filename> is not used to support
Multi-ISP:</para>
<programlisting>#MARK SOURCE DEST PROTO DEST SOURCE
# PORT(S) PORT(S)
FORMAT 2
TTL(+1):P INT_IF -
SAME:P INT_IF - tcp 80,443
?if $PROXY &amp;&amp; ! $SQUID2
DIVERT COMB_IF - tcp - 80
DIVERT COMC_IF - tcp - 80
DIVERT br0 172.20.1.0/24 tcp - 80
TPROXY(3129,172.20.1.254) INT_IF - tcp 80
?if $PROXYDMZ
TPROXY(3129,172.20.1.254) br0 - tcp 80
?endif
?endof
</programlisting>
</section>
<section>
<title>IPv6 Configuration</title>
<para>The IPv6 configuration has two separate sub-nets, both services
through 6in4 tunnels from <ulink
url="http://tunnelbroker.he.net">Hurricane Electric</ulink>. They are
both configured through the Business IPv4 uplink. I originally had the
sit2 tunnel configured through the consumer uplink but Comcast (Xfinity)
decided to start blocking HE IPv6 tunnels on their consumer network,
preferring their own 6to4 IPv6 solution.</para>
<para>One HE tunnel handles the servers and one tunnel handles the local
network.</para>
<para>Here are the key entries in
<filename>/etc/shorewall6/shorewall6.conf</filename>:</para>
<programlisting>###############################################################################
# F I R E W A L L O P T I O N S
###############################################################################
...
FASTACCEPT=No
FORWARD_CLEAR_MARK=Yes
IMPLICIT_CONTINUE=No
<emphasis role="bold">IP_FORWARDING=Keep</emphasis>
<emphasis role="bold">KEEP_RT_TABLES=Yes</emphasis> #Required when both IPv4 and IPv6 Multi-ISP are used
...
TRACK_PROVIDERS=No
<emphasis role="bold">USE_DEFAULT_RT=Yes</emphasis>
ZONE2ZONE=-
...
################################################################################
# P A C K E T M A R K L A Y O U T
################################################################################
TC_BITS=8
PROVIDER_BITS=8
PROVIDER_OFFSET=8
MASK_BITS=8
ZONE_BITS=0
</programlisting>
<para>Here is <filename>/etc/shorewall6/zones</filename>:</para>
<programlisting/>
<programlisting>#ZONE TYPE OPTIONS
fw firewall
net ipv6
loc ipv6
dmz ipv6</programlisting>
<para><filename>/etc/shorewall/interfaces</filename>:</para>
<programlisting>#ZONE INTERFACE OPTIONS
net sit1 forward=1,sfilter=2001:470:b:227::40/124,optional
net sit2 forward=1,sfilter=2001:470:b:227::40/124,optional
net sit3 forward=1,sfilter=2001:470:b:227::40/124,optional
loc eth2 forward=1
dmz br0 routeback,forward=1,required</programlisting>
<para><filename>/etc/shorewall/providers</filename>:</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
LOC 4 0x100 - sit2 - track,balance,loose
DMZ 5 0x200 - sit1 - track,fallback,loose
6to4 6 0x300 - sit3 ::192.88.99.1 track,fallback,loose</programlisting>
<para>Notice that the provider numbers are disjoint from those in the
IPv4 configuration. This allows for unique provider names in
<filename>/etc/iproute2/rt_tables</filename>:</para>
<programlisting>#
# reserved values
#
255 local
254 main
253 default
250 balance
0 unspec
#
# local
#
1 ComcastB
2 ComcastC
3 TProxy
4 LOC
5 DMZ
6 6to4</programlisting>
<para>The <filename>/etc/shorewall6/rtrules</filename> file is
straight-forward:</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
2001:470:B:227::1/64 ::/0 DMZ 11000
2001:470:B:787::1/64 ::/0 LOC 11000
2002:465a:bf79::1/64 ::/0 6to4 11000</programlisting>
<para>This results in the following routing rules:</para>
<programlisting>root@gateway:~# <command>ip -6 rule ls</command>
0: from all lookup local
999: from all lookup main
10003: from all fwmark 0x100/0xff00 lookup LOC
10004: from all fwmark 0x200/0xff00 lookup DMZ
10005: from all fwmark 0x300/0xff00 lookup 6to4
11000: from 2001:470:b:787::1/64 lookup LOC
11000: from 2001:470:b:227::1/64 lookup DMZ
11000: from 2002:465a:bf79::1/64 lookup 6to4
32765: from all lookup balance
32767: from all lookup default
root@gateway:~# </programlisting>
</section>
</section>
</article>
Reference in New Issue View Git Blame Copy Permalink