# Shorewall and Multiple Internet Connections

Tom
Eastep
2005
2006
Thomas M. Eastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled GNU Free Documentation License.

Make sure you are running a current, vendor-supported distribution, before attempting to perform this setup, older distributions do not meet the minimum requirements,and you will need to recompile iptables, kernel and other software on your system. If you don't follow this advice,we will not be able to help you.

Reading just Shorewall documentation is probably not going to give you enough background to use this material. Shorewall may make iptables easy but the Shorewall team doesn't have the resources to be able to spoon-feed Linux policy routing to you (please remember that the user's manual for a tractor doesn't teach you to grow corn either). You will likely need to refer to the following additional information:

- The LARTC HOWTO: http://www.lartc.org

- Output of `man ip`

- Output of `ip route help` and `ip rule help`

## Multiple Internet Connection Support

Beginning with Shorewall 2.3.2, limited support is included for multiple internet connections. Limitations of this support are as follows:

- It utilizes static routing configuration. As such, there is no provision for reacting to the failure of either of the uplinks.

- The routing changes are made and the route cache is purged when Shorewall is started and when Shorewall is restarted (unless you specify the "-n" option to `shorewall restart`). Ideally, restarting the packet filter should have no effect on routing.

- Prior to Shorewall 3.4.0, the routes and route rules added by this support were not completely removed during `shorewall stop`, `shorewall clear` or `shorewall restart`.

### Overview

Let's assume that a firewall is connected via two separate ethernet interfaces to two different ISPs as in the following diagram.

- eth0 connects to ISP1. The IP address of eth0 is 206.124.146.176 and the ISP's gateway router has IP address 206.124.146.254.

- eth1 connects to ISP 2. The IP address of eth1 is 130.252.99.27 and the ISP's gateway router has IP address 130.252.99.254.

- eth2 connects to the local LAN. Its IP configuration is not relevant to this discussion.

Each of these providers is described in an entry in the file `/etc/shorewall/providers`.

Entries in `/etc/shorewall/providers` can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in `/etc/shorewall/tcrules` can be used to direct particular outgoing connections to one ISP or the other. Use of `/etc/shorewall/tcrules` is not required for `/etc/shorewall/providers` to work, but you must select a unique MARK value for each provider so Shorewall can set up the correct marking rules for you.

When you use the track option in `/etc/shorewall/providers`, connections from the internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall.

Shorewall will set up the routing and will update the `/etc/iproute2/rt_tables` to include the table names and numbers of the tables that it adds.

This feature uses packet marking to control the routing. As a consequence, there are some restrictions concerning entries in `/etc/shorewall/tcrules`:

- Packet marking for traffic control purposes may not be done in the PREROUTING table for connections involving providers with 'track' specified (see below).

- You may not use the SAVE or RESTORE options.

- You may not use connection marking.

The `/etc/shorewall/providers` file can also be used in other routing scenarios. See the Squid documentation for an example.

### /etc/shorewall/providers File

Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value.

NAME

  The provider name. Must begin with a letter and consist of letters and digits. The provider name becomes the name of the generated routing table for this provider.

NUMBER

  A number between 1 and 252. This becomes the routing table number for the generated table for this provider.

MARK

  A mark value used in your /etc/shorewall/tcrules file to direct packets to this provider. Shorewall will also mark connections that have seen input from this provider with this value and will restore the packet mark in the PREROUTING CHAIN. Mark values must be in the range 1-255.

  Beginning with Shorewall version 3.2.0 Beta 6, you may use may set HIGH_ROUTE_MARKS=Yes in `/etc/shorewall/shorewall.conf`. This allows you to:

  - Use connection marks for traffic shaping, provided that you assign those marks in the FORWARD chain.

  - Use mark values > 255 for provider marks in this column. These mark values must be a multiple of 256 in the range 256-65280 (hex equivalent 0x100 - 0xFF00 with the low-order 8 bits being zero).

DUPLICATE

  Gives the name or number of a routing table to duplicate. May be 'main' or the name or number of a previously declared provider. For most applications, you want to specify 'main' here.

INTERFACE

  The name of the interface to the provider.

GATEWAY

  The IP address of the provider's Gateway router.

  You can enter detect here and Shorewall will attempt to automatically determine the gateway IP address.

  Hint: "detect" is appropriate for use in cases where the interface named in the INTERFACE column is dynamically configured via DHCP etc.

OPTIONS

  A comma-separated list from the following:

  track

    If specified, connections FROM this interface are to be tracked so that responses may be routed back out this same interface.

    You want to specify 'track' if internet hosts will be connecting to local servers through this provider. Any time that you specify 'track', you will also want to specify 'balance' (see below).

    Use of this feature requires that your kernel and iptables support CONNMARK target and connmark match support. It does not require the ROUTE target extension.

    iptables 1.3.1 is broken with respect to CONNMARK and iptables-save/iptables-restore. This means that if you configure multiple ISPs, `shorewall restore` may fail. If it does, you may patch your iptables using the patch at http://shorewall.net/pub/shorewall/contrib/iptables/CONNMARK.diff.

    If you are using `/etc/shorewall/providers` because you have multiple internet connections, we recommend that you specify 'track' even if you don't need it. It helps maintain long-term connections in which there are significant periods with no traffic.

  balance

    The providers that have 'balance' specified will get outbound traffic load-balanced among them. Balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider.

    By default, each provider is given the same weight (1) . You can change the weight of a given provider by following balance with "=" and the desired weight (e.g., balance=2). The weights reflect the relative bandwidth of the providers connections and should be small numbers since the kernel actually creates additional default routes for each weight increment.

    If you are using `/etc/shorewall/providers` because you have multiple internet connections, we recommend that you specify 'balance' even if you don't need it. You can still use entries in `/etc/shorewall/tcrules` to force traffic to one provider or another.

    If you don't heed this advice then be prepared to read FAQ 57 and FAQ 58.

    If you specify 'balance' and still find that all traffic is going out through only one provider, you may need to install a kernel built with CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have reported that this change has corrected similar problems.

    The SuSE 10.0 kernel is subject to this problem, and a kernel oops may result in this circumstance. SUSE 10.1 and SLES 10 have CONFIG_IP_ROUTE_MULTIPATH_CACHED=n set by default. The source of the problem seems to be an incompatibility between the LARTC patches and CONFIG_IP_ROUTE_MULTIPATH_CACHED.

  loose

    Do not include routing rules that force traffic whose source IP is an address of the INTERFACE to be routed to this provider. Useful for defining providers that are to be used only when the appropriate packet mark is applied.

  optional (added in Shorewall 3.2.2)

    Shorewall will determine of this interface is up and has a configured IPv4 address. If it is not, a warning is issued and this provider is not configured.

    'optional' is designed to detect interface states that will cause `shorewall start` or `shorewall restart` to fail; just because an interface is in a state that Shorewall can [re]start without error doesn't mean that traffic can actually be sent through the interface.

  For those of you who are termnally confused between track and balance:

  - track governs incoming connections.

  - balance governs outgoing connections.

COPY

  When you specify an existing table in the DUPLICATE column, Shorewall copies all routes through the interface specified in the INTERFACE column plus the interfaces listed in this column. At a minumum, you should list all interfaces on your firewall in this column except those internet interfaces specified in the INTERFACE column of entries in this file.

### What an entry in the Providers File Does

Adding another entry in the providers file simply creates an alternate routing table for you. In addition:

1. Unless loose is specified, an ip rule is generated for each IP address on the INTERFACE that routes traffic from that address through the associated routing table.

2. If you specify track, then connections which have had at least one packet arrive on the interface listed in the INTERFACE column have their connection mark set to the value in the MARK column. In the PREROUTING chain, packets with a connection mark have their packet mark set to the value of the associated connection mark; packets marked in this way bypass any prerouting rules that you create in `/etc/shorewall/tcrules`. This ensures that packets associated with connections from outside are always routed out of the correct interface.

3. If you specify balance, then Shorewall will replace the 'default' route with weight 100 in the 'main' routing table with a load-balancing route among those gateways where balance was specified. So if you configure default routes, be sure that their weight is less than 100 or the route added by Shorewall will not be used.

That's all that these entries do. You still have to follow the principle stated in the Shorewall Routing documentation:

1. Routing determines where packets are to be sent.

2. Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there.

The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic with the provider's MARK value in `/etc/shorewall/tcrules` and you must do that marking in the PREROUTING chain; or, you must provide the appropriate rules in `/etc/shorewall/route_rules`.

If you are running a Shorewall version prior to 3.4.0, entries in `/etc/shorewall/providers` permanently alter your firewall/gateway's routing; that is, the effect of these changes is not reversed by `shorewall stop` or `shorewall clear`. To restore routing to its original state, you may have to restart your network. This can usually be done by `/etc/init.d/network restart` or `/etc/init.d/networking restart`. Check your distribution's networking documentation.

Here are some additional things to consider:

- You can mitigate the effect of the Shorewall-generated changes to your routing table by specifying a metric for each default route that you configure. Shorewall will generate a load-balancing default route (assuming that balance has been specified for some of the providers) that does not include a metric and that will therefore not replace any existing route that has a non-zero metric.

- The `-n` option to `shorewall restart` and `shorewall restore` can be used to prevent the command from changing your routing.

- The `/etc/shorewall/stopped` file can also be used to restore routing when you stop Shorewall. With your firewall in its normal (single-table) routing configuration, you can capture the contents as follows:

      ip route ls > routes

  Here's what the `routes` file looked like after I did that on my firewall:

      192.168.1.1 dev eth3  scope link
      206.124.146.177 dev eth1  scope link
      192.168.2.2 dev tun0  proto kernel  scope link  src 192.168.2.1
      192.168.2.0/24 via 192.168.2.2 dev tun0
      192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.254
      206.124.146.0/24 dev eth3  proto kernel  scope link  src 206.124.146.176
      169.254.0.0/16 dev eth0  scope link
      127.0.0.0/8 dev lo  scope link
      default via 206.124.146.254 dev eth3

  Now edit the file as shown below:

      ip route flush table main
      ip route add 192.168.1.1 dev eth3  scope link
      ip route add 206.124.146.177 dev eth1  scope link
      ip route add 192.168.2.2 dev tun0  proto kernel  scope link  src 192.168.2.1
      ip route add 192.168.2.0/24 via 192.168.2.2 dev tun0
      ip route add 192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.254
      ip route add 206.124.146.0/24 dev eth3  proto kernel  scope link  src 206.124.146.176
      ip route add 169.254.0.0/16 dev eth0  scope link
      ip route add 127.0.0.0/8 dev lo  scope link
      ip route add default via 206.124.146.254 dev eth3
      ip route flush cache

  Now paste the contents of that file into `/etc/shorewall/stopped`.

  You might also want to consider adding the following to the file:

      ip rule ls | while read priority rule; do
          case ${priority%:} in
              0|3276[67])
                  ;;
              *)
                  ip rule del $rule
                  ;;
          esac
      done

  That code will delete all but the default routing rules.

### What an entry in the Providers File Does NOT Do

Given that Shorewall is simply a tool to configure Netfilter and does not run continuously in your system, entries in the providers file do not provide any automatic failover in the event of failure of one of your Internet connections.

### Example

The configuration in the figure at the top of this section would be specified in `/etc/shorewall/providers` as follows.

    #NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS          COPY
    ISP1    1       1       main            eth0            206.124.146.254 track,balance    eth2
    ISP2    2       2       main            eth1            130.252.99.254  track,balance    eth2

Other configuration files go something like this:

`/etc/shorewall/interfaces`:

    #ZONE    INTERFACE    BROADCAST       OPTIONS
    net      eth0         detect          …
    net      eth1         detect          …

`/etc/shorewall/policy`:

    #SOURCE    DESTINATION    POLICY     LIMIT:BURST
    net        net            DROP

Regardless of whether you have masqueraded hosts or not, YOU MUST ADD THESE TWO ENTRIES TO `/etc/shorewall/masq`:

    #INTERFACE       SUBNET            ADDRESS
    eth0             130.252.99.27     206.124.146.176
    eth1             206.124.146.176   130.252.99.27

Those entries ensure that traffic originating on the firewall always has the source IP address corresponding to the interface that it is routed out of.

If you have a Dynamic IP address on either of the interfaces, you can use shell variables to construct the above rules. For example, if `eth0` had a dynamic IP address, then:

`/etc/shorewall/params`:

    ETH0_IP=$(find_first_interface_address eth0)

/etc/shorewall/masq:

    #INTERFACE       SUBNET            ADDRESS
    eth0             130.252.99.27     $ETH0_IP
    eth1             $ETH0_IP          130.252.99.27

If you have masqueraded hosts, be sure to update `/etc/shorewall/masq` to masquerade to both ISPs. For example, if you masquerade all hosts connected to `eth2` then:

    #INTERFACE       SUBNET            ADDRESS
    eth0             eth2              206.124.146.176
    eth1             eth2              130.252.99.27

Entries in `/etc/shorewall/masq` have no effect on which ISP a particular connection will be sent through. That is rather the purpose of entries in `/etc/shorewall/tcrules` or `/etc/shorewall/route_rules`.

Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You would make this entry in /etc/shorewall/tcrules (and if you are running a version of Shorewall earlier than 3.0.0, you would set TC_ENABLED=Yes in /etc/shorewall/shorewall.conf).

    #MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
    #                                                               PORT(S)
    2:P             <local network> 0.0.0.0/0       tcp     25

### Applications running on the Firewall

Experience has shown that in some cases, problems occur with applications running on the firewall itself. When this happens, it is suggested that you have the application use specific local IP addresses rather than 0.

Examples:

- Squid: In `squid.conf`, set tcp_outgoing_address to the IP address of the interface that you want Squid to use.

- In OpenVPN, set local (--local on the command line) to the IP address that you want the server to receive connections on.

### /etc/shorewall/route_rules

The `/etc/shorewall/route_rules` file was added in Shorewall version 3.2.0. The `route_rules` file allows assigning certain traffic to a particular provider just as entries in the `tcrules` file. The difference between the two files is that entries in `route_rules` are independent of Netfilter.

Entries in `/etc/shorewall/route_rules` permanently alter your firewall/gateway's routing; that is, the effect of these changes is not reversed by `shorewall stop` or `shorewall clear`. To restore routing to its original state, you may have to restart your network. This can usually be done by `/etc/init.d/network restart` or `/etc/init.d/networking restart`. Check your distribution's networking documentation.

Please also see the warning in the section above entitled What an Entry in the Providers File Does.

#### Routing Rules

Routing rules are maintained by the Linux kernel and can be displayed using the `ip rule ls` command. When routing a packet, the rules are processed in turn until the packet is successfully routed.

    gateway:~ # ip rule ls
    0:      from all lookup local                <=== Local (to the firewall) IP addresses
    10001:  from all fwmark 0x1 lookup Blarg     <=== This and the next rule are generated by the
    10002:  from all fwmark 0x2 lookup Comcast        'MARK' values in /etc/shorewall/providers.
    20000:  from 206.124.146.176 lookup Blarg    <=== This and the next rule are generated unless
    20256:  from 24.12.22.33 lookup Comcast           'loose' is specified; based in the output of 'ip addr ls'
    32766:  from all lookup main                 <=== This is the routing table shown by 'iproute -n'
    32767:  from all lookup default              <=== This table is usually empty
    gateway:~ #

In the above example, there are two providers: Blarg and Comcast with MARK 1 going to Blarg and mark 2 going to Comcast.

#### Columns in the route_rules file

Columns in the file are:

SOURCE (Optional)

  An ip address (network or host) that matches the source IP address in a packet. May also be specified as an interface name optionally followed by ":" and an address. If the device 'lo' is specified, the packet must originate from the firewall itself.

DEST (Optional)

  An ip address (network or host) that matches the destination IP address in a packet.

  If you choose to omit either SOURCE or DEST, place "-" in that column. Note that you may not omit both SOURCE and DEST.

PROVIDER

  The provider to route the traffic through. May be expressed either as the provider name or the provider number.

PRIORITY

  The rule's priority which determines the order in which the rules are processed.

  1000-1999 Before Shorewall-generated 'MARK' rules

  11000- 11999 After 'MARK' rules but before Shorewall-generated rules for ISP interfaces.

  26000-26999 After ISP interface rules but before 'default' rule.

  Rules with equal priority are applied in the order in which they appear in the file.

Example 1: You want all traffic entering the firewall on eth1 to be routed through Comcast.

    #SOURCE            DEST      PROVIDER        PRIORITY
    eth1               -         Comcast         1000

With this entry, the output of `ip rule ls` would be as follows.

    gateway:~ # ip rule ls
    0:      from all lookup local
    1000:   from all iif eth1 lookup Comcast
    10001:  from all fwmark 0x1 lookup Blarg
    10002:  from all fwmark 0x2 lookup Comcast
    20000:  from 206.124.146.176 lookup Blarg
    20256:  from 24.12.22.33 lookup Comcast
    32766:  from all lookup main
    32767:  from all lookup default
    gateway:~ #

Note that because we used a priority of 1000, the test for `eth1` is inserted before the fwmark tests.

Example 2: You use OpenVPN (routed setup /tunX) in combination with multiple providers. In this case you have to set up a rule to ensure that the OpenVPN traffic is routed back through the tunX interface(s) rather than through any of the providers. 10.8.0.0/24 is the subnet choosen in your OpenVPN configuration (server 10.8.0.0 255.255.255.0).

    #SOURCE                 DEST            PROVIDER        PRIORITY
    -                       10.8.0.0/24     main            1000