extern/shorewall_code
1
0
Fork 1
mirror of https://gitlab.com/shorewall/code.git synced 2024-11-26 17:43:15 +01:00
Code Issues Packages Projects Releases Wiki Activity
37b8fca94e
shorewall_code/docs/MultiISP.xml

1259 lines
54 KiB
XML
Raw Blame History

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
<!--ble$Id$-->
<articleinfo>
<title>Shorewall and Multiple Internet Connections</title>
<authorgroup>
<author>
<firstname>Tom</firstname>
<surname>Eastep</surname>
</author>
</authorgroup>
<pubdate><?dbtimestamp format="Y/m/d"?></pubdate>
<copyright>
<year>2005</year>
<year>2006</year>
<year>2007</year>
<year>2008</year>
<holder>Thomas M. Eastep</holder>
</copyright>
<legalnotice>
<para>Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
<quote><ulink url="GnuCopyright.htm">GNU Free Documentation
License</ulink></quote>.</para>
</legalnotice>
</articleinfo>
<warning>
<para>This document describes the Multi-ISP facility in Shorewall 4.0 and
later. If you are running an earlier release, please see the documentation
for that release.</para>
</warning>
<warning>
<para>Reading just Shorewall documentation is probably not going to give
you enough background to use this material. Shorewall may make iptables
easy but the Shorewall team doesn't have the resources to be able to
spoon-feed Linux policy routing to you (please remember that the user's
manual for a tractor doesn't teach you to grow corn either). You will
likely need to refer to the following additional information:</para>
<itemizedlist>
<listitem>
<para>The LARTC HOWTO: <ulink
url="http://www.lartc.org">http://www.lartc.org</ulink></para>
</listitem>
<listitem>
<para>Output of <command>man ip</command></para>
</listitem>
<listitem>
<para>Output of <command>ip route help</command> and <command>ip rule
help</command></para>
</listitem>
</itemizedlist>
</warning>
<section id="Support">
<title>Multiple Internet Connection Support</title>
<para>Beginning with Shorewall 2.3.2, limited support is included for
multiple Internet connections. Limitations of this support are as
follows:</para>
<itemizedlist>
<listitem>
<para>It utilizes static routing configuration. As such, there is no
provision for reacting to the failure of any of the uplinks.</para>
</listitem>
<listitem>
<para>The routing changes are made and the route cache is purged when
Shorewall is started <emphasis role="bold">and when Shorewall is
restarted</emphasis> (unless you specify the "-n" option to
<command>shorewall restart</command>). Ideally, restarting the packet
filter should have no effect on routing.</para>
</listitem>
<listitem>
<para>Prior to Shorewall 3.4.0, the routes and route rules added by
this support were not completely removed during <command>shorewall
stop</command>, <command>shorewall clear</command> or
<command>shorewall restart</command>.</para>
</listitem>
<listitem>
<para>For most routing applications, <ulink
url="http://www.quagga.net/">Quagga</ulink> is a better
solution.</para>
</listitem>
</itemizedlist>
<section id="Overview">
<title>Overview</title>
<para>Let's assume that a firewall is connected via two separate
Ethernet interfaces to two different ISPs as in the following
diagram.</para>
<graphic align="center" fileref="images/TwoISPs.png" valign="middle" />
<itemizedlist>
<listitem>
<para>eth0 connects to ISP1. The IP address of eth0 is
206.124.146.176 and the ISP's gateway router has IP address
206.124.146.254.</para>
</listitem>
<listitem>
<para>eth1 connects to ISP 2. The IP address of eth1 is
130.252.99.27 and the ISP's gateway router has IP address
130.252.99.254.</para>
</listitem>
<listitem>
<para>eth2 connects to the local LAN. Its IP configuration is not
relevant to this discussion.</para>
</listitem>
</itemizedlist>
<para>Each of these <firstterm>providers</firstterm> is described in an
entry in the file <filename>/etc/shorewall/providers</filename>.</para>
<para>Entries in <filename>/etc/shorewall/providers</filename> can
specify that outgoing connections are to be load-balanced between the
two ISPs. Entries in <filename>/etc/shorewall/tcrules</filename> can be
used to direct particular outgoing connections to one ISP or the other.
Use of <filename>/etc/shorewall/tcrules</filename> is not required for
<filename>/etc/shorewall/providers</filename> to work, but you must
select a unique MARK value for each provider so Shorewall can set up the
correct marking rules for you.</para>
<para>When you use the <emphasis role="bold">track</emphasis> option in
<filename>/etc/shorewall/providers</filename>, connections from the
Internet are automatically routed back out of the correct interface and
through the correct ISP gateway. This works whether the connection is
handled by the firewall itself or if it is routed or port-forwarded to a
system behind the firewall.</para>
<para>Shorewall will set up the routing and will update the
<filename>/etc/iproute2/rt_tables</filename> to include the table names
and numbers of the tables that it adds.</para>
<caution>
<para>This feature uses <ulink url="traffic_shaping.htm">packet
marking</ulink> to control the routing. As a consequence, there are
some restrictions concerning entries in
<filename>/etc/shorewall/tcrules</filename>:</para>
<itemizedlist>
<listitem>
<para>Packet marking for traffic control purposes may not be done
in the PREROUTING table for connections involving providers with
'track' specified (see below).</para>
</listitem>
<listitem>
<para>You may not use the SAVE or RESTORE options.</para>
</listitem>
<listitem>
<para>You may not use connection marking.</para>
</listitem>
</itemizedlist>
</caution>
<para>The <filename>/etc/shorewall/providers</filename> file can also be
used in other routing scenarios. See the <ulink
url="Shorewall_Squid_Usage.html">Squid documentation</ulink> for an
example.</para>
</section>
<section id="providers">
<title>/etc/shorewall/providers File</title>
<para>Entries in this file have the following columns. As in all
Shorewall configuration files, enter "-" in a column if you don't want
to enter any value.</para>
<variablelist>
<varlistentry>
<term>NAME</term>
<listitem>
<para>The provider name. Must begin with a letter and consist of
letters and digits. The provider name becomes the name of the
generated routing table for this provider.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>NUMBER</term>
<listitem>
<para>A number between 1 and 252. This becomes the routing table
number for the generated table for this provider.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>MARK</term>
<listitem>
<para>A mark value used in your /etc/shorewall/tcrules file to
direct packets to this provider. Shorewall will also mark
connections that have seen input from this provider with this
value and will restore the packet mark in the PREROUTING CHAIN.
Mark values must be in the range 1-255.</para>
<para>Beginning with Shorewall version 3.2.0 Beta 6, you may use
may set HIGH_ROUTE_MARKS=Yes in
<filename>/etc/shorewall/shorewall.conf</filename>. This allows
you to:</para>
<itemizedlist>
<listitem>
<para>Use connection marks for traffic shaping, provided that
you assign those marks in the FORWARD chain.</para>
</listitem>
<listitem>
<para>Use mark values &gt; 255 for provider marks in this
column. These mark values must be a multiple of 256 in the
range 256-65280 (hex equivalent 0x100 - 0xFF00 with the
low-order 8 bits being zero).</para>
</listitem>
</itemizedlist>
</listitem>
</varlistentry>
<varlistentry>
<term>DUPLICATE</term>
<listitem>
<para>Gives the name or number of a routing table to duplicate.
May be 'main' or the name or number of a previously declared
provider. For most applications, you want to specify 'main' here.
This field should be be specified as '-' when USE_DEFAULT_RT=Yes
in <filename>shorewall.conf</filename></para>
</listitem>
</varlistentry>
<varlistentry>
<term>INTERFACE</term>
<listitem>
<para>The name of the interface to the provider. Where multiple
providers share the same interface (which is not recommended), you
must follow the name of the interface by a colon (":") and the IP
address assigned by this provider (e.g., eth0:206.124.146.176).
See <link linkend="Shared">below</link> for additional
considerations.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>GATEWAY</term>
<listitem>
<para>The IP address of the provider's Gateway router.</para>
<para>You can enter <emphasis role="bold">detect</emphasis> here
and Shorewall will attempt to automatically determine the gateway
IP address.</para>
<para><emphasis role="bold">Hint:</emphasis> <emphasis
role="bold">"detect"</emphasis> is appropriate for use in cases
where the interface named in the INTERFACE column is dynamically
configured via DHCP etc.</para>
<para>The GATEWAY may be omitted (enter '-') for point-to-point
links.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>OPTIONS</term>
<listitem>
<para>A comma-separated list from the following:</para>
<variablelist>
<varlistentry>
<term>track</term>
<listitem>
<para>If specified, connections FROM this interface are to
be tracked so that responses may be routed back out this
same interface.</para>
<para>You want to specify 'track' if Internet hosts will be
connecting to local servers through this provider. Any time
that you specify 'track', you will also want to specify
'balance' (see below).</para>
<para>Use of this feature requires that your kernel and
iptables include CONNMARK target and connmark match support
(<emphasis role="bold">Warning</emphasis>: Standard
<trademark>Debian</trademark> and
<trademark>Ubuntu</trademark> kernels are lacking that
support!).</para>
<warning>
<para>A <emphasis role="bold">bug</emphasis> in Shorewall
versions 3.2.0-3.2.10, 3.4.0-3.4.6 and 4.0.0-4.0.2
prevents proper handling of PREROUTING marks when
HIGH_ROUTE_MARKS=No and the <emphasis
role="bold">track</emphasis> option is specified. Patches
are available to correct this problem:</para>
<para>Shorewall version 3.2.0-3.4.3: <ulink
url="http://www1.shorewall.net/pub/shorewall/3.2/shorewall-3.2.10/errata/patches/Shorewall/patch-3.2.10-2.diff">http://www1.shorewall.net/pub/shorewall/3.2/shorewall-3.2.10/errata/patches/Shorewall/patch-3.2.10-2.diff
</ulink></para>
<para>Shorewall version 3.4.4-3.4.6: <ulink
url="http://www1.shorewall.net/pub/shorewall/3.4/shorewall-3.4.6/errata/patches/Shorewall/patch-3.4.6-1.diff">http://www1.shorewall.net/pub/shorewall/3.4/shorewall-3.4.6/errata/patches/Shorewall/patch-3.4.6-1.diff</ulink></para>
<para>Shorewall-shell version 4.0.0-4.0.2: <ulink
url="http://www1.shorewall.net/pub/shorewall/4.0/shorewall-4.0.2/errata/patches/Shorewall-shell/patch-shell-4.0.2-2.diff">http://www1.shorewall.net/pub/shorewall/4.0/shorewall-4.0.2/errata/patches/Shorewall-shell/patch-shell-4.0.2-2.diff</ulink></para>
</warning>
<important>
<para>If you are using
<filename>/etc/shorewall/providers</filename> because you
have multiple Internet connections, we recommend that you
specify 'track' even if you don't need it. It helps
maintain long-term connections in which there are
significant periods with no traffic.</para>
</important>
</listitem>
</varlistentry>
<varlistentry>
<term>balance</term>
<listitem>
<para>The providers that have 'balance' specified will get
outbound traffic load-balanced among them. Balancing will
not be perfect, as it is route based, and routes are cached.
This means that routes to often-used sites will always be
over the same provider.</para>
<para>By default, each provider is given the same weight (1)
. You can change the weight of a given provider by following
<emphasis>balance</emphasis> with "=" and the desired weight
(e.g., balance=2). The weights reflect the relative
bandwidth of the providers connections and should be small
numbers since the kernel actually creates additional default
routes for each weight increment.</para>
<important>
<para>If you are using
<filename>/etc/shorewall/providers</filename> because you
have multiple Internet connections, we recommend that you
specify 'balance' even if you don't need it. You can still
use entries in <filename>/etc/shorewall/tcrules</filename>
to force all traffic to one provider or another.<note>
<para>If you don't heed this advice then be prepared
to read <ulink url="FAQ.htm#faq57">FAQ 57</ulink> and
<ulink url="FAQ.htm#faq58">FAQ 58</ulink>.</para>
</note></para>
</important>
<important>
<para>If you specify 'balance' and still find that all
traffic is going out through only one provider, you may
need to install a kernel built with
CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have
reported that this change has corrected similar
problems.</para>
<para>The SuSE 10.0 kernel is subject to this problem, and
<ulink
url="https://bugzilla.novell.com/show_bug.cgi?id=190908">
a kernel oops may result in this circumstance.</ulink>
SUSE 10.1 and SLES 10 have
CONFIG_IP_ROUTE_MULTIPATH_CACHED=n set by default. The
source of the problem seems to be <ulink
url="http://news.gmane.org/find-root.php?message_id=%3c00da01c5b35a%24b12b9860%241b00a8c0%40cruncher%3e">an
incompatibility between the LARTC patches and
CONFIG_IP_ROUTE_MULTIPATH_CACHED.</ulink></para>
</important>
</listitem>
</varlistentry>
<varlistentry>
<term>loose</term>
<listitem>
<para>Do not include routing rules that force traffic whose
source IP is an address of the INTERFACE to be routed to
this provider. Useful for defining providers that are to be
used only when the appropriate packet mark is applied.
Should not be specified together with <emphasis
role="bold">balance</emphasis>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>optional (added in Shorewall 3.2.2)</term>
<listitem>
<para>Shorewall will determine of this interface is up and
has a configured IPv4 address. If it is not, a warning is
issued and this provider is not configured.</para>
<note>
<para>'optional' is designed to detect interface states
that will cause <command>shorewall start</command> or
<command>shorewall restart</command> to fail; just because
an interface is in a state that Shorewall can [re]start
without error doesn't mean that traffic can actually be
sent through the interface.</para>
<para>Beginning with Shorewall-perl 4.0.3, you can supply
an 'isusable' <ulink
url="shorewall_extension_scripts.htm">extension
script</ulink> to extend Shorewall's interface state
detection.</para>
</note>
</listitem>
</varlistentry>
<varlistentry>
<term>src=<replaceable>source-address</replaceable> (Added in
Shorewall-perl 4.1.5)</term>
<listitem>
<para>Specifies the source address to use when routing to
this provider and none is known (the local client has bound
to the 0 address). May not be specified when an
<replaceable>address</replaceable> is given in the INTERFACE
column. If this option is not used, Shorewall substitutes
the primary IP address on the interface named in the
INTERFACE column.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>mtu=<replaceable>number</replaceable> (Added in
Shorewall-perl 4.1.5)</term>
<listitem>
<para>Specifies the MTU when forwarding through this
provider. If not given, the MTU of the interface named in
the INTERFACE column is assumed.</para>
</listitem>
</varlistentry>
</variablelist>
<para>For those of you who are terminally confused
between<emphasis role="bold"> track</emphasis> and <emphasis
role="bold">balance</emphasis>:</para>
<itemizedlist>
<listitem>
<para><emphasis role="bold">track</emphasis> governs incoming
connections.</para>
</listitem>
<listitem>
<para><emphasis role="bold">balance</emphasis> governs
outgoing connections.</para>
</listitem>
</itemizedlist>
</listitem>
</varlistentry>
<varlistentry>
<term>COPY</term>
<listitem>
<para>A comma-separated list if interface names. Wildcards
specified using an asterisk ("*") are permitted (e.g., tun*
).</para>
<para>When you specify an existing table in the DUPLICATE column,
Shorewall copies all routes through the interface specified in the
INTERFACE column plus the interfaces listed in this column.
Normally, you will list all interfaces on your firewall in this
column except those Internet interfaces specified in the INTERFACE
column of entries in this file.</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section id="Providers">
<title>What an entry in the Providers File Does</title>
<para>Adding another entry in the providers file simply creates an
alternate routing table for you. The table will usually contain two
routes:</para>
<orderedlist>
<listitem>
<para>A host route to the specified GATEWAY through the specified
INTERFACE.</para>
</listitem>
<listitem>
<para>A default route through the GATEWAY.</para>
</listitem>
</orderedlist>
<para>Note that the first route is omitted if "-" is specified as the
GATEWAY; in that case, the default route does not specify a gateway
(point-to-point link).</para>
<para>If the DUPLICATE column is non-empty, then routes from the table
named in that column are copied into the new table. By default, all
routes (except default routes) are copied. The set of routes copied can
be restricted using the COPY column which lists the interfaces whose
routes you want copied. You will generally want to include all local
interfaces in this list. You should exclude the loopback interface (lo)
and any interfaces that do not have an IPv4 configuration. You should
also omit interfaces like <emphasis role="bold">tun</emphasis>
interfaces that are created dynamically. Traffic to networks handled by
those interfaces should be routed through the main table using entries
in <filename>/etc/shorewall/route_rules</filename> (see Example 2 <link
linkend="Examples">below</link>).</para>
<para>In addition:</para>
<orderedlist>
<listitem>
<para>Unless <emphasis role="bold">loose</emphasis> is specified, an
ip rule is generated for each IP address on the INTERFACE that
routes traffic from that address through the associated routing
table.</para>
</listitem>
<listitem>
<para>If you specify <emphasis role="bold">track</emphasis>, then
connections which have had at least one packet arrive on the
interface listed in the INTERFACE column have their connection mark
set to the value in the MARK column. In the PREROUTING chain,
packets with a connection mark have their packet mark set to the
value of the associated connection mark; packets marked in this way
bypass any prerouting rules that you create in
<filename>/etc/shorewall/tcrules</filename>. This ensures that
packets associated with connections from outside are always routed
out of the correct interface.</para>
</listitem>
<listitem>
<para>If you specify <emphasis role="bold">balance</emphasis>, then
Shorewall will replace the 'default' route with weight 100 in the
'main' routing table with a load-balancing route among those
gateways where <emphasis role="bold">balance</emphasis> was
specified. So if you configure default routes, be sure that their
weight is less than 100 or the route added by Shorewall will not be
used.</para>
</listitem>
</orderedlist>
<para>That's <emphasis role="bold">all</emphasis> that these entries do.
You still have to follow the principle stated in the <ulink
url="Shorewall_and_Routing.html">Shorewall Routing
documentation</ulink>:</para>
<orderedlist>
<listitem>
<para>Routing determines where packets are to be sent.</para>
</listitem>
<listitem>
<para>Once routing determines where the packet is to go, the
firewall (Shorewall) determines if the packet is allowed to go
there.</para>
</listitem>
</orderedlist>
<para>The bottom line is that if you want traffic to go out through a
particular provider then you <emphasis>must </emphasis>mark that traffic
with the provider's MARK value in
<filename>/etc/shorewall/tcrules</filename> and you must do that marking
in the PREROUTING chain; or, you must provide the appropriate rules in
<filename>/etc/shorewall/route_rules</filename>.</para>
</section>
<section id="Provider_Doesnt">
<title>What an entry in the Providers File Does NOT Do</title>
<para>Given that Shorewall is simply a tool to configure Netfilter and
does not run continuously in your system, entries in the providers file
<emphasis role="bold">do not provide any automatic failover in the event
of failure of one of your Internet connections</emphasis>.</para>
</section>
<section id="Martians">
<title>Martians</title>
<para>One problem that often arises with Multi-ISP configuration is
'Martians'. If your Internet interfaces are configured with the
<emphasis role="bold">routefilter</emphasis> option in
<filename>/etc/shorewall/interfaces</filename> (remember that if you set
that option, you should also select <emphasis
role="bold">logmartians</emphasis>), then things may not work correctly
and you will see messages like this:</para>
<programlisting>Feb 9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1
Feb 9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00</programlisting>
<para>The above message is somewhat awkwardly phrased. The source IP in
this incoming packet was 64.86.88.116 and the destination IP address is
206.124.146.176. Another gotcha is that the incoming packet has already
had the destination IP address changed for DNAT or because the original
outgoing connection was altered by an entry in
<filename>/etc/shorewall/masq</filename> (SNAT or Masquerade). So the
destination IP address (206.124.146.176) may not have been the
destination IP address in the packet as it was initially
received.</para>
<para>There a couple of common causes for these problems:</para>
<orderedlist>
<listitem>
<para>You have connected both of your external interfaces to the
same hub/switch. Connecting multiple firewall interfaces to a common
hub or switch is always a bad idea that will result in
hard-to-diagnose problems.</para>
</listitem>
<listitem>
<para>You are specifying both the <emphasis
role="bold">loose</emphasis> and <emphasis
role="bold">balance</emphasis> options on your provider(s). This
causes individual connections to ping-pong back and forth between
the interfaces which is guaranteed to cause problems.</para>
</listitem>
<listitem>
<para>You are redirecting traffic from the local system out of one
interface or the other using packet marking in your
<filename>/etc/shorewall/tcrules</filename> file. A better approach
is to configure the application to use the appropriate local IP
address (the IP address of the interface that you want the
application to use). See <link linkend="Local">below</link>.</para>
</listitem>
</orderedlist>
<para>If all else fails, remove the <emphasis
role="bold">routefilter</emphasis> option from your external interfaces.
If you do this, you may wish to add rules to log and drop packets from
the Internet that have source addresses in your local networks. For
example, if the local LAN in the above diagram is 192.168.1.0/24, then
you would add this rule:</para>
<programlisting>#ACTION SOURCE DEST
DROP:info net:192.168.1.0/24 all</programlisting>
<para>Be sure the above rule is added before any other rules with
<emphasis>net</emphasis> in the SOURCE column.</para>
</section>
<section id="Example1">
<title id="Example">Example</title>
<para>The configuration in the figure at the top of this section would
be specified in <filename>/etc/shorewall/providers</filename> as
follows.</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
ISP1 1 1 main eth0 206.124.146.254 track,balance eth2
ISP2 2 2 main eth1 130.252.99.254 track,balance eth2</programlisting>
<para>Other configuration files go something like this:</para>
<para><filename>/etc/shorewall/interfaces</filename>:</para>
<programlisting>#ZONE INTERFACE BROADCAST OPTIONS
net eth0 detect …
net eth1 detect …</programlisting>
<para><filename>/etc/shorewall/policy</filename>:</para>
<programlisting>#SOURCE DESTINATION POLICY LIMIT:BURST
net net DROP</programlisting>
<para>Regardless of whether you have masqueraded hosts or not, the
following entries are required in
<filename>/etc/shorewall/masq</filename> if you plan to redirect
connections from the firewall using entries in
<filename>/etc/shorewall/tcrules</filename> or if you specify <emphasis
role="bold">balance</emphasis> on your providers.</para>
<programlisting>#INTERFACE SUBNET ADDRESS
eth0 130.252.99.27 206.124.146.176
eth1 206.124.146.176 130.252.99.27</programlisting>
<para>Those entries ensure that traffic originating on the firewall and
redirected via packet marks always has the source IP address
corresponding to the interface that it is routed out of.</para>
<note>
<para>If you have a Dynamic IP address on either of the interfaces,
you can use shell variables to construct the above rules. For example,
if <filename class="devicefile">eth0</filename> had a dynamic IP
address, then:</para>
<para><filename>/etc/shorewall/params</filename>:</para>
<programlisting>ETH0_IP=$(find_first_interface_address eth0)</programlisting>
<para>For optional interfaces, use
<emphasis>find_first_interface_address_if_any</emphasis> in place of
<emphasis>find_first_interface_address</emphasis>.</para>
<para>/etc/shorewall/masq:</para>
<programlisting>#INTERFACE SUBNET ADDRESS
eth0 130.252.99.27 $ETH0_IP
eth1 $ETH0_IP 130.252.99.27</programlisting>
</note>
<para>If you have masqueraded hosts, be sure to update
<filename>/etc/shorewall/masq</filename> to masquerade to both ISPs. For
example, if you masquerade all hosts connected to <filename
class="devicefile">eth2</filename> then:</para>
<programlisting>#INTERFACE SUBNET ADDRESS
eth0 eth2 206.124.146.176
eth1 eth2 130.252.99.27</programlisting>
<warning>
<para>Entries in <filename>/etc/shorewall/masq</filename> have no
effect on which ISP a particular connection will be sent through. That
is rather the purpose of entries in
<filename>/etc/shorewall/tcrules</filename> or
<filename>/etc/shorewall/route_rules</filename>.</para>
</warning>
<para>Now suppose that you want to route all outgoing SMTP traffic from
your local network through ISP 2. You would make this entry in <ulink
url="traffic_shaping.htm">/etc/shorewall/tcrules</ulink> (and if you are
running a version of Shorewall earlier than 3.0.0, you would set
TC_ENABLED=Yes in <ulink
url="???">/etc/shorewall/shorewall.conf</ulink>).</para>
<programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST
# PORT(S)
2:P &lt;local network&gt; 0.0.0.0/0 tcp 25</programlisting>
</section>
<section id="morethan2">
<title>More than 2 Providers</title>
<para>When there are more than two providers, you need to extend the
two-provider case in the expected way:</para>
<orderedlist>
<listitem>
<para>For each external address, you need an entry in
<filename>/etc/shorewall/masq</filename> to handle the case where a
connection using that address as the SOURCE is sent out of the
interfaces other than the one that the address is configured
on.</para>
</listitem>
<listitem>
<para>For each external interface, you need to add an entry to
<filename>/etc/shorewall/masq</filename> for each internal network
that needs to be masqueraded (or use SNAT) through that
interface.</para>
</listitem>
</orderedlist>
<para>If we extend the above example to add eth3 with IP address
16.105.78.4 with gateway 16.105.78.254, then:</para>
<para><filename>/etc/shorewall/providers</filename>:<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
ISP1 1 1 main eth0 206.124.146.254 track,balance eth2
ISP2 2 2 main eth1 130.252.99.254 track,balance eth2
ISP3 3 3 main eth3 16.105.78.254 track,balance eth2</programlisting></para>
<para><filename>/etc/shorewall/masq</filename>:<programlisting>#INTERFACE SUBNET ADDRESS
eth0 130.252.99.27 206.124.146.176
eth3 130.252.99.27 16.105.78.4
eth1 206.124.146.176 130.252.99.27
eth3 206.124.146.176 16.105.78.4
eth0 16.106.78.4 206.124.146.176
eth1 16.106.78.4 130.252.99.27
eth0 eth2 206.124.146.176
eth1 eth2 130.252.99.27
eth3 eth2 16.105.78.4</programlisting></para>
</section>
<section id="Local">
<title>Applications running on the Firewall</title>
<para>Experience has shown that in some cases, problems occur with
applications running on the firewall itself. This is especially true
when you have specified <emphasis role="bold">routefilter</emphasis> on
your external interfaces in /etc/shorewall/interfaces (see <link
linkend="Martians">above</link>). When this happens, it is suggested
that you have the application use specific local IP addresses rather
than 0.</para>
<para>Examples:</para>
<itemizedlist>
<listitem>
<para>Squid: In <filename>squid.conf</filename>, set <emphasis
role="bold">tcp_outgoing_address</emphasis> to the IP address of the
interface that you want Squid to use.</para>
</listitem>
<listitem>
<para>In OpenVPN, set <emphasis role="bold">local
</emphasis>(<emphasis role="bold">--local</emphasis> on the command
line) to the IP address that you want the server to receive
connections on.</para>
</listitem>
</itemizedlist>
<para>Note that some traffic originating on the firewall doesn't have a
SOURCE IP address before routing. At least one Shorewall user reports
that an entry in <filename>/etc/shorewall/route_rules</filename> with
'lo' in the SOURCE column seems to be the most reliable way to direct
such traffic to a particular ISP.</para>
</section>
<section id="IPSEC">
<title>IPSEC</title>
<para>If you have an IPSEC gateway on your firewall, be sure to arrange
for ESP packets to be routed out of the same interface that you have
configured your keying daemon to use.</para>
</section>
<section id="route_rules">
<title>/etc/shorewall/route_rules</title>
<para>The <filename>/etc/shorewall/route_rules</filename> file was added
in Shorewall version 3.2.0. The <filename>route_rules</filename> file
allows assigning certain traffic to a particular provider just as
entries in the <filename>tcrules</filename> file. The difference between
the two files is that entries in <filename>route_rules</filename> are
independent of Netfilter.</para>
<section id="Routing_rules">
<title>Routing Rules</title>
<para>Routing rules are maintained by the Linux kernel and can be
displayed using the <command>ip rule ls</command> command. When
routing a packet, the rules are processed in turn until the packet is
successfully routed.</para>
<programlisting>gateway:~ # <command>ip rule ls</command>
0: from all lookup local &lt;=== Local (to the firewall) IP addresses
10001: from all fwmark 0x1 lookup Blarg &lt;=== This and the next rule are generated by the
10002: from all fwmark 0x2 lookup Comcast 'MARK' values in /etc/shorewall/providers.
20000: from 206.124.146.176 lookup Blarg &lt;=== This and the next rule are generated unless
20256: from 24.12.22.33 lookup Comcast 'loose' is specified; based in the output of 'ip addr ls'
32766: from all lookup main &lt;=== This is the routing table shown by 'iproute -n'
32767: from all lookup default &lt;=== This table is usually empty
gateway:~ #</programlisting>
<para>In the above example, there are two providers: Blarg and Comcast
with MARK 1 going to Blarg and mark 2 going to Comcast.</para>
</section>
<section id="route_rules_columns">
<title>Columns in the route_rules file</title>
<para>Columns in the file are:</para>
<variablelist>
<varlistentry>
<term>SOURCE (Optional)</term>
<listitem>
<para>An ip address (network or host) that matches the source IP
address in a packet. May also be specified as an interface name
optionally followed by ":" and an address. If the device 'lo' is
specified, the packet must originate from the firewall
itself.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>DEST (Optional)</term>
<listitem>
<para>An ip address (network or host) that matches the
destination IP address in a packet.</para>
<para>If you choose to omit either SOURCE or DEST, place "-" in
that column. Note that you may not omit both SOURCE and
DEST.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>PROVIDER</term>
<listitem>
<para>The provider to route the traffic through. May be
expressed either as the provider name or the provider
number.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>PRIORITY</term>
<listitem>
<para>The rule's priority which determines the order in which
the rules are processed.</para>
<para>1000-1999 Before Shorewall-generated 'MARK' rules</para>
<para>11000- 11999 After 'MARK' rules but before
Shorewall-generated rules for ISP interfaces.</para>
<para>26000-26999 After ISP interface rules but before 'default'
rule.</para>
<para>Rules with equal priority are applied in the order in
which they appear in the file.</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section id="Examples">
<title>Examples</title>
<para><emphasis role="bold">Example 1:</emphasis> You want all traffic
entering the firewall on eth1 to be routed through Comcast.</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
eth1 - Comcast 1000</programlisting>
<para>With this entry, the output of <command>ip rule ls</command>
would be as follows.</para>
<para><programlisting>gateway:~ # <command>ip rule ls</command>
0: from all lookup local
1000: from all iif eth1 lookup Comcast
10001: from all fwmark 0x1 lookup Blarg
10002: from all fwmark 0x2 lookup Comcast
20000: from 206.124.146.176 lookup Blarg
20256: from 24.12.22.33 lookup Comcast
32766: from all lookup main
32767: from all lookup default
gateway:~ #</programlisting>Note that because we used a priority of 1000, the
test for <filename class="devicefile">eth1</filename> is inserted
before the fwmark tests.</para>
<para id="Openvpn"><emphasis role="bold">Example 2:</emphasis> You use
OpenVPN (routed setup w/tunX) in combination with multiple providers.
In this case you have to set up a rule to ensure that the OpenVPN
traffic is routed back through the tunX interface(s) rather than
through any of the providers. 10.8.0.0/24 is the subnet chosen in your
OpenVPN configuration (server 10.8.0.0 255.255.255.0).</para>
<programlisting>#SOURCE DEST PROVIDER PRIORITY
- 10.8.0.0/24 main 1000</programlisting>
</section>
</section>
<section id="USE_DEFAULT_RT">
<title>USE_DEFAULT_RT</title>
<para>Beginning with Shorewall 4.2.0 Beta3, Shorewall-perl has supported
a USE_DEFAULT_RT option in <ulink
url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5).</para>
<para>One of the drawbacks of the Mulit-ISP support as described in the
preceding sections is that changes to the main table made by
applications are not added to the individual provider tables. This makes
route rules such as described in <link linkend="Openvpn">one of the
examples above</link> necessary.</para>
<para>USE_DEFAULT_RT=Yes works around that problem by passing packets
through the main table first rather than last. This has a number of
implications:</para>
<orderedlist>
<listitem>
<para>Both the DUPLICATE and the COPY columns in the providers file
must remain empty or contain "-". The individual provider routing
tables generated when USE_DEFAULT_RT=Yes contain only a host route
to the gateway and a default route via the gateway.</para>
</listitem>
<listitem>
<para>The <emphasis role="bold">balance</emphasis> option is assumed
for all interfaces that do not have the <emphasis
role="bold">loose</emphasis> option. When you want both <emphasis
role="bold">balance</emphasis> and <emphasis
role="bold">loose</emphasis>, both must be specified.</para>
</listitem>
<listitem>
<para>The default route generated by Shorewall is added to the
<emphasis>default</emphasis> routing table (253) rather than to the
main routing table (254).</para>
</listitem>
<listitem>
<para>Packets are sent through the main routing table by a routing
rule with priority 999. In ), the priority range 1-998 may be used
for inserting rules that bypass the main table.</para>
</listitem>
<listitem>
<para>All provider gateways must be specified explicitly in the
GATEWAY column. 'detect' may not be specified. Note that for ppp
interfaces, the GATEWAY may remain unspecified ("-").</para>
</listitem>
<listitem>
<para>You should disable all default route management outside of
Shorewall. If a default route is inadvertently added to the main
table while Shorewall is started, then all policy routing will stop
working except for those routing rules in the priority range
1-998.</para>
</listitem>
</orderedlist>
<para>Although 'balance' is automatically assumed when
USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider
except when you explicitly direct it to use the other provider via
<ulink
url="manpages/shorewall-route_rules.html">shorewall-route_rules</ulink>
(5) or <ulink
url="manpages/shorewall-tcrules.html">shorewall-tcrules</ulink>
(5).</para>
<para>Example (send all traffic through the 'shorewall' provider unless
otherwise directed).</para>
<para>/etc/shorewall/providers:<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS
linksys 1 1 - wlan0 172.20.1.1 track,balance=1,optional
shorewall 2 2 - eth0 192.168.1.254 track,balance=2,optional</programlisting>/etc/shorewall/route_rules:<programlisting>#SOURCE DEST PROVIDER PRIORITY
- - shorewall 11999</programlisting></para>
</section>
<section id="Shared">
<title>Two Providers Sharing an Interface</title>
<para>Shared interface support is available only in Shorewall-perl 4.2.0
and later.</para>
<orderedlist numeration="loweralpha">
<listitem>
<para>Only Ethernet (or Ethernet-like) interfaces can be used. For
inbound traffic, the MAC addresses of the gateway routers are used
to determine which provider a packet was received through. Note that
only routed traffic can be categorized using this technique.</para>
</listitem>
<listitem>
<para>You must specify the address on the interface that corresponds
to a particular provider in the INTERFACE column by following the
interface name with a colon (":") and the address.</para>
</listitem>
<listitem>
<para>Entries in <filename>/etc/shorewall/masq</filename> must be
qualified by the provider name (or number).</para>
</listitem>
<listitem>
<para>This feature requires Realm Match support in your kernel and
iptables.</para>
</listitem>
<listitem>
<para>You must add route_rules entries for networks that are
accessed through a particular provider.</para>
</listitem>
<listitem>
<para>If you have additional IP addresses through either provider,
you must add <filename>route_rules</filename> to direct traffic FROM
each of those addresses through the appropriate provider.</para>
</listitem>
<listitem>
<para>You must manually add MARK rules for traffic known to come
from each provider.</para>
</listitem>
<listitem>
<para>You must specify a gateway IP address in the GATEWAY column
of<filename> /etc/shorewall/providers</filename>; <emphasis
role="bold">detect</emphasis> is not permitted.</para>
</listitem>
</orderedlist>
<para>Taken together, b. and h. effectively preclude using this
technique with dynamic IP addresses.</para>
<para>Example:</para>
<para>This is our home network circa fall 2008. We have two internet
providers:</para>
<orderedlist>
<listitem>
<para>Comcast -- Cable modem with one dynamic IP address.</para>
</listitem>
<listitem>
<para>Avvanta -- ADSL with 5 static IP addresses.</para>
</listitem>
</orderedlist>
<para>Because the old <trademark>Compaq</trademark>
<trademark>Presario</trademark> that I use for a firewall only has three
PCI slots and no onboard Ethernet, it doesn't have enough Ethernet
controllers to support both providers. So I use a Linksys WRT300n pre-N
router as a gateway to Comcast. Note that because the Comcast IP address
is dynamic, I could not share a single firewall interface between the
two providers directly.</para>
<para>On my personal laptop (ursa), I have 9 virtual machines running
various Linux distributions. <emphasis>It is the Shorewall configuration
on ursa that I will describe here</emphasis>.</para>
<para>Below is a diagram of our network:<graphic align="center"
fileref="images/Network2008a.png" /></para>
<para>The local wired network in my office is connected to both gateways
and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast
gateway has local IP address 172.20.1.1 while the Avvanta gateway has
local IP address 172.20.1.254. Ursa's eth0 interface has a single IP
address (172.20.1.130).</para>
<para>This configuration uses USE_DEFAULT_RT=Yes in
<filename>shorewall.conf </filename>(see <link
linkend="USE_DEFAULT_RT">above</link>).</para>
<para>Here is the <filename>providers</filename> file:</para>
<programlisting>#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
comcast 1 1 - eth0:172.20.1.130 172.20.1.1 track,loose,balance,optional
avvanta 2 2 - eth0:172.20.1.130 172.20.1.254 track,optional,loose
wireless 3 3 - wlan0 172.20.1.1 track,optional</programlisting>
<para>Several things to note:</para>
<orderedlist>
<listitem>
<para>172.20.1.130 is specified as the <filename
class="devicefile">eth0</filename> IP address for both
providers.</para>
</listitem>
<listitem>
<para>Both wired providers have the <emphasis
role="bold">loose</emphasis> option. This prevents Shorewall from
automatically generating routing rules based on the source IP
address.</para>
</listitem>
<listitem>
<para>Only <emphasis role="bold">comcast</emphasis> has the
<emphasis role="bold">balance</emphasis> option. With
USE_DEFAULT_RT=yes, that means that <emphasis
role="bold">comcast</emphasis> will be the default provider. While
<emphasis role="bold">balance</emphasis> is the default, with
USE_DEFAULT_RT=Yes, it must be specified explicitly when <emphasis
role="bold">loose</emphasis> is also specified.</para>
</listitem>
<listitem>
<para>I always disable the <emphasis role="bold">wireless</emphasis>
interface when the laptop is connected to the wired network.</para>
</listitem>
<listitem>
<para>I use a different Shorewall configuration when I take the
laptop on the road.</para>
</listitem>
</orderedlist>
<para>Here is the route_rules file:<programlisting>#SOURCE DEST PROVIDER PRIORITY
- 206.124.146.176/31 avvanta 1000
- 206.124.146.178/31 avvanta 1000
- 206.124.146.180/32 avvanta 1000</programlisting></para>
<para>Those rules direct traffic to the five static Avvanta IP addresses
(only two are currently used) through the <emphasis
role="bold">avvanta</emphasis> provider.</para>
<para>Here is the tcrules file (MARK_IN_FORWARD_CHAIN=No in
<filename>shorewall.conf</filename>):<programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS CONNBYTES HELPER
# PORT(S)
2 $FW 0.0.0.0/0 tcp 21
2 $FW 0.0.0.0/0 tcp - - - - - - - ftp
2 $FW 0.0.0.0/0 tcp 119</programlisting></para>
<para>These rules:</para>
<itemizedlist>
<listitem>
<para>Use <emphasis role="bold">avvanta</emphasis> for FTP.</para>
</listitem>
<listitem>
<para>Use <emphasis role="bold">avvanta</emphasis> for NTTP</para>
</listitem>
</itemizedlist>
<para>The remaining files are for a rather standard two-interface config
with a bridge as the local interface.</para>
<para><filename>zones</filename>:<programlisting>#ZONE IPSEC OPTIONS IN OUT
# ONLY OPTIONS OPTIONS
fw firewall
net ipv4
kvm ipv4</programlisting><filename>policy</filename>:<programlisting>net net NONE
fw net ACCEPT
fw kvm ACCEPT
kvm all ACCEPT
net all DROP info
all all REJECT info</programlisting></para>
<para>interfaces:<programlisting>#ZONE INTERFACE BROADCAST OPTIONS GATEWAY
#
net eth0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore
net wlan0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional
kvm br0 detect routeback #Virtual Machines</programlisting><note>
<para><filename class="devicefile">wlan0</filename> is the wireless
adapter in the notebook. Used when the laptop is in our home but not
connected to the wired network.</para>
</note></para>
<para>masq:<programlisting>#INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC
eth0 192.168.0.0/24
wlan0 192.168.0.0/24</programlisting><note>
<para>Because the firewall has only a single external IP address, I
don't need to specify the providers in the masq rules.</para>
</note></para>
</section>
</section>
</article>
Reference in New Issue View Git Blame Copy Permalink