Shorewall and Multiple Internet ConnectionsTomEastep20052006Thomas M. EastepPermission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
GNU Free Documentation
License.Make sure you are running a current,
vendor-supported distribution, before attempting to perform
this setup, older distributions do not meet the minimum requirements,and
you will need to recompile iptables, kernel and other software on your
system. If you don't follow this advice,we will
not be able to help
you.Reading just Shorewall documentation is probably not going to give
you enough background to use this material. Shorewall may make iptables
easy but the Shorewall team doesn't have the resources to be able to
spoon-feed Linux policy routing to you (please remember that the user's
manual for a tractor doesn't teach you to grow corn either). You will
likely need to refer to the following additional information:The LARTC HOWTO: http://www.lartc.orgOutput of man ipOutput of ip route help and ip rule
helpMultiple Internet Connection SupportBeginning with Shorewall 2.3.2, limited support is included for
multiple internet connections. Limitations of this support are as
follows:It utilizes static routing configuration. As such, there is no
provision for reacting to the failure of either of the uplinks.The routing changes are made and the route cache is purged when
Shorewall is started and when Shorewall is
restarted (unless you specify the "-n" option to
shorewall restart). Ideally, restarting the packet
filter should have no effect on routing.OverviewLet's assume that a firewall is connected via two separate
ethernet interfaces to two different ISPs as in the following
diagram.eth0 connects to ISP1. The IP address of eth0 is
206.124.146.176 and the ISP's gateway router has IP address
206.124.146.254.eth1 connects to ISP 2. The IP address of eth1 is
130.252.99.27 and the ISP's gateway router has IP address
130.252.99.254.eth2 connects to the local LAN. Its IP configuration is not
relevant to this discussion.Each of these providers is described in an
entry in the file /etc/shorewall/providers.Entries in /etc/shorewall/providers can
specify that outgoing connections are to be load-balanced between the
two ISPs. Entries in /etc/shorewall/tcrules can be
used to direct particular outgoing connections to one ISP or the other.
Use of /etc/shorewall/tcrules is not required for
/etc/shorewall/providers to work, but you must
select a unique MARK value for each provider so Shorewall can set up the
correct marking rules for you.When you use the track option in
/etc/shorewall/providers, connections from the
internet are automatically routed back out of the correct interface and
through the correct ISP gateway. This works whether the connection is
handled by the firewall itself or if it is routed or port-forwarded to a
system behind the firewall.Shorewall will set up the routing and will update the
/etc/iproute2/rt_tables to include the table names
and number of the tables that it adds.This feature uses packet
marking to control the routing. As a consequence, there are
some restrictions concerning entries in
/etc/shorewall/tcrules:Packet marking for traffic control purposes may not be done
in the PREROUTING table for connections involving providers with
'track' specified (see below).You may not use the SAVE or RESTORE options.You may not use connection marking.The /etc/shorewall/providers file can also be
used in other routing scenarios. See the Squid documentation for an
example./etc/shorewall/providers FileEntries in this file have the following columns. As in all
Shorewall configuration files, enter "-" in a column if you don't want
to enter any value.NAMEThe provider name. Must begin with a letter and consist of
letters and digits. The provider name becomes the name of the
generated routing table for this provider.NUMBERA number between 1 and 252. This becomes the routing table
number for the generated table for this provider.MARKA mark value used in your /etc/shorewall/tcrules file to
direct packets to this provider. Shorewall will also mark
connections that have seen input from this provider with this
value and will restore the packet mark in the PREROUTING CHAIN.
Mark values must be in the range 1-255.Beginning with Shorewall version 3.2.0 Beta 6, you may use
may set HIGH_ROUTE_MARKS=Yes in
/etc/shorewall/shorewall.conf. This allows
you to:Use connection marks for traffic shaping, provided that
you assign those marks in the FORWARD table.Use mark values > 255 for provider marks in this
column. These mark values must be a multiple of 256 in the
range 256-65280 (hex equivalent 0x100 - 0xFF00 with the
low-order 8 bits being zero).DUPLICATEGives the name or number of a routing table to duplicate.
May be 'main' or the name or number of a previously declared
provider. For most applications, you want to specify 'main'
here.INTERFACEThe name of the interface to the provider.GATEWAYThe IP address of the provider's Gateway router.You can enter detect here
and Shorewall will attempt to automatically determine the gateway
IP address.Hint:"detect" is appropriate for use in cases
where the interface named in the INTERFACE column is dynamically
configured via DHCP etc.OPTIONSA comma-separated list from the following:trackIf specified, connections FROM this interface are to
be tracked so that responses may be routed back out this
same interface.You want to specify 'track' if internet hosts will be
connecting to local servers through this provider. Any time
that you specify 'track', you will also want to specify
'balance' (see below).Use of this feature requires that your kernel and
iptables support CONNMARK target and connmark match support.
It does not require the ROUTE target extension.iptables 1.3.1 is broken with respect to CONNMARK
and iptables-save/iptables-restore. This means that if you
configure multiple ISPs, shorewall
restore may fail. If it does, you may patch your
iptables using the patch at http://shorewall.net/pub/shorewall/contrib/iptables/CONNMARK.diff.If you are using
/etc/shorewall/providers because you
have multiple internet connections, we recommend that you
specify 'track' even if you don't need it. It helps
maintain long-term connections in which there are
significant periods with no traffic.balanceThe providers that have 'balance' specified will get
outbound traffic load-balanced among them. Balancing will
not be perfect, as it is route based, and routes are cached.
This means that routes to often-used sites will always be
over the same provider.By default, each provider is given the same weight (1)
. You can change the weight of a given provider by following
balance with "=" and the desired weight
(e.g., balance=2). The weights reflect the relative
bandwidth of the providers connections and should be small
numbers since the kernel actually creates additional default
routes for each weight increment.If you are using
/etc/shorewall/providers because you
have multiple internet connections, we recommend that you
specify 'balance' even if you don't need it. You can still
use entries in /etc/shorewall/tcrules
to force traffic to one provider or another.If you don't heed this advice then be prepared
to read FAQ 57 and
FAQ 58.If you specify 'balance' and still find that all
traffic is going out through only one provider, you may
need to install a kernel built with
CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have
reported that this change has corrected similar
problems.The SuSE 10.0 kernel is subject to this problem, and
a kernel oops may result in this circumstance.
SUSE 10.1 and SLES 10 have
CONFIG_IP_ROUTE_MULTIPATH_CACHED=n set by default. The
source of the problem seems to be an
incompatibility between the LARTC patches and
CONFIG_IP_ROUTE_MULTIPATH_CACHED.looseDo not include routing rules that force traffic whose
source IP is an address of the INTERFACE to be routed to
this provider. Useful for defining providers that are to be
used only when the appropriate packet mark is
applied.optional (added in Shorewall 3.2.2)Shorewall will determine of this interface is up and
has a configured IPv4 address. If it is not, a warning is
issued and this provider is not configured.'optional' is designed to detect interface states
that will cause shorewall start or
shorewall restart to fail; just because
an interface is in a state that Shorewall can [re]start
without error doesn't mean that traffic can actually be
sent through the interface.COPYWhen you specify an existing table in the DUPLICATE column,
Shorewall copies all routes through the interface specified in the
INTERFACE column plus the interfaces listed in this column. At a
minumum, you should list all interfaces on your firewall in this
column except those internet interfaces specified in the INTERFACE
column of entries in this file.What an entry in the Providers File DoesAdding another entry in the providers file simply creates an
alternate routing table for you. In addition:Unless loose is specified, an
ip rule is generated for each IP address on the INTERFACE that
routes traffic from that address through the associated routing
table.If you specify track, then
connections which have had at least one packet arrive on the
interface listed in the INTERFACE column have their connection mark
set to the value in the MARK column. In the PREROUTING chain,
packets with a connection mark have their packet mark set to the
value of the associated connection mark; packets marked in this way
bypass any prerouting rules that you create in
/etc/shorewall/tcrules. This ensures that
packets associated with connections from outside are always routed
out of the correct interface.If you specify balance, then
Shorewall will replace the 'default' route with weight 100 in the
'main' routing table with a load-balancing route among those
gateways where balance was
specified. So if you configure default routes, be sure that their
weight is less than 100 or the route added by Shorewall will not be
used.That's all that these entries do.
You still have to follow the principle stated in the Shorewall Routing
documentation:Routing determines where packets are to be sent.Once routing determines where the packet is to go, the
firewall (Shorewall) determines if the packet is allowed to go
there.The bottom line is that if you want traffic to go out through a
particular provider then you must mark that traffic
with the provider's MARK value in
/etc/shorewall/tcrules and you must do that marking
in the PREROUTING chain.Entries in /etc/shorewall/providers
permanently alter your firewall/gateway's routing; that is, the effect
of these changes is not reversed by shorewall stop
or shorewall clear. To restore routing to its
original state, you may have to restart your network. This can usually
be done by /etc/init.d/network restart or
/etc/init.d/networking restart. Check your
distribution's networking documentation.Here are some additional things to consider:You can mitigate the effect of the Shorewall-generated
changes to your routing table by specifying a
metric for each default route that you
configure. Shorewall will generate a load-balancing default route
(assuming that balance has been
specified for some of the providers) that does not include a
metric and that will therefore not replace any existing route that
has a non-zero metric.The -n option to shorewall
restart and shorewall restore can be
used to prevent the command from changing your routing.The /etc/shorewall/stopped file can
also be used to restore routing when you stop Shorewall. With your
firewall in its normal (single-table) routing configuration, you
can capture the contents as follows:ip route ls > routesHere's what the routes file looked like
after I did that on my firewall:192.168.1.1 dev eth3 scope link
206.124.146.177 dev eth1 scope link
192.168.2.2 dev tun0 proto kernel scope link src 192.168.2.1
192.168.2.0/24 via 192.168.2.2 dev tun0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.254
206.124.146.0/24 dev eth3 proto kernel scope link src 206.124.146.176
169.254.0.0/16 dev eth0 scope link
127.0.0.0/8 dev lo scope link
default via 206.124.146.254 dev eth3Now edit the file as shown below:ip route flush table main
ip route add 192.168.1.1 dev eth3 scope link
ip route add 206.124.146.177 dev eth1 scope link
ip route add 192.168.2.2 dev tun0 proto kernel scope link src 192.168.2.1
ip route add 192.168.2.0/24 via 192.168.2.2 dev tun0
ip route add 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.254
ip route add 206.124.146.0/24 dev eth3 proto kernel scope link src 206.124.146.176
ip route add 169.254.0.0/16 dev eth0 scope link
ip route add 127.0.0.0/8 dev lo scope link
ip route add default via 206.124.146.254 dev eth3
ip route flush cacheNow paste the contents of that file into
/etc/shorewall/stopped.You might also want to consider adding the following to the
file:ip rule ls | while read priority rule; do
case ${priority%:} in
0|3276[67])
;;
*)
ip rule del $rule
;;
esac
doneThat code will delete all but the default routing
rules.What an entry in the Providers File Does NOT DoGiven that Shorewall is simply a tool to configure Netfilter and
does not run continuously in your system, entries in the providers file
do not provide any automatic failover in the event
of failure of one of your Internet connections.ExampleThe configuration in the figure at the top of this section would
be specified in /etc/shorewall/providers as
follows.#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
ISP1 1 1 main eth0 206.124.146.254 track,balance eth2
ISP2 2 2 main eth1 130.252.99.254 track,balance eth2Other configuration files go something like this:/etc/shorewall/interfaces:#ZONE INTERFACE BROADCAST OPTIONS
net eth0 detect …
net eth1 detect …/etc/shorewall/policy:#SOURCE DESTINATION POLICY LIMIT:BURST
net net DROPRegardless of whether you have masqueraded hosts or not, YOU MUST ADD THESE TWO ENTRIES TO
/etc/shorewall/masq:#INTERFACE SUBNET ADDRESS
eth0 130.252.99.27 206.124.146.176
eth1 206.124.146.176 130.252.99.27Those entries ensure that traffic originating on the firewall
always has the source IP address corresponding to the interface that it
is routed out of.If you have a Dynamic IP address on either of the interfaces,
you can use shell variables to construct the above rules. For example,
if eth0 had a dynamic IP
address, then:/etc/shorewall/params:ETH0_IP=$(find_first_interface_address eth0)/etc/shorewall/masq:#INTERFACE SUBNET ADDRESS
eth0 130.252.99.27 $ETH0_IP
eth1 $ETH0_IP 130.252.99.27If you have masqueraded hosts, be sure to update
/etc/shorewall/masq to masquerade to both ISPs. For
example, if you masquerade all hosts connected to eth2 then:#INTERFACE SUBNET ADDRESS
eth0 eth2 206.124.146.176
eth1 eth2 130.252.99.27Entries in /etc/shorewall/masq have no
effect on which ISP a particular connection will be sent through. That
is rather the purpose of entries in
/etc/shorewall/tcrules.Now suppose that you want to route all outgoing SMTP traffic from
your local network through ISP 2. You would make this entry in /etc/shorewall/tcrules (and if you are
running a version of Shorewall earlier than 3.0.0, you would set
TC_ENABLED=Yes in /etc/shorewall/shorewall.conf).#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST
# PORT(S)
2:P <local network> 0.0.0.0/0 tcp 25/etc/shorewall/route_rulesThe /etc/shorewall/route_rules file was added
in Shorewall version 3.2.0. The route_rules file
allows assigning certain traffic to a particular provider just as
entries in the tcrules file. The difference between
the two files is that entries in route_rules are
independent of Netfilter.Entries in /etc/shorewall/route_rules
permanently alter your firewall/gateway's routing; that is, the effect
of these changes is not reversed by shorewall stop
or shorewall clear. To restore routing to its
original state, you may have to restart your network. This can usually
be done by /etc/init.d/network restart or
/etc/init.d/networking restart. Check your
distribution's networking documentation.Please also see the warning in the section
above entitled What an Entry in the Providers File
Does.Routing RulesRouting rules are maintained by the Linux kernel and can be
displayed using the ip rule ls command. When
routing a packet, the rules are processed in turn until the packet is
successfully routed.gateway:~ # ip rule ls
0: from all lookup local <=== Local (to the firewall) IP addresses
10001: from all fwmark 0x1 lookup Blarg <=== This and the next rule are generated by the
10002: from all fwmark 0x2 lookup Comcast 'MARK' values in /etc/shorewall/providers.
20000: from 206.124.146.176 lookup Blarg <=== This and the next rule are generated unless
20256: from 24.12.22.33 lookup Comcast 'loose' is specified; based in the output of 'ip addr ls'
32766: from all lookup main <=== This is the routing table shown by 'iproute -n'
32767: from all lookup default <=== This table is usually empty
gateway:~ #In the above example, there are two providers: Blarg and Comcast
with MARK 1 going to Blarg and mark 2 going to Comcast.Columns in the route_rules fileColumns in the file are:SOURCE (Optional)An ip address (network or host) that matches the source IP
address in a packet. May also be specified as an interface name
optionally followed by ":" and an address. If the device 'lo' is
specified, the packet must originate from the firewall
itself.DEST (Optional)An ip address (network or host) that matches the
destination IP address in a packet.If you choose to omit either SOURCE or DEST, place "-" in
that column. Note that you may not omit both SOURCE and
DEST.PROVIDERThe provider to route the traffic through. May be
expressed either as the provider name or the provider
number.PRIORITYThe rule's priority which determines the order in which
the rules are processed.1000-1999 Before Shorewall-generated 'MARK' rules11000- 11999 After 'MARK' rules but before
Shorewall-generated rules for ISP interfaces.26000-26999 After ISP interface rules but before 'default'
rule.Rules with equal priority are applied in the order in
which they appear in the file.Example 1: You want all traffic entering the firewall on eth1 to
be routed through Comcast.#SOURCE DEST PROVIDER PRIORITY
eth1 - Comcast 1000With this entry, the output of ip rule ls
would be as follows.gateway:~ # ip rule ls
0: from all lookup local
1000: from all iif eth1 lookup Comcast
10001: from all fwmark 0x1 lookup Blarg
10002: from all fwmark 0x2 lookup Comcast
20000: from 206.124.146.176 lookup Blarg
20256: from 24.12.22.33 lookup Comcast
32766: from all lookup main
32767: from all lookup default
gateway:~ #Note that because we used a priority of 1000, the
test for eth1 is inserted
before the fwmark tests.Example 2: You use OpenVPN (routed setup /tunX) in combination
with multiple providers. In this case you have to set up a rule to
ensure that the OpenVPN traffic is routed back through the tunX
interface(s) rather than through any of the providers. 10.8.0.0/24 is
the subnet choosen in your OpenVPN configuration (server 10.8.0.0
255.255.255.0).#SOURCE DEST PROVIDER PRIORITY
- 10.8.0.0/24 main 1000