Tom Eastep 2005-11-16 2005 Thomas M. Eastep Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled GNU Free Documentation License.

One of the most misunderstood aspects of Shorewall is its relationship with routing. This article attempts to clear some of the fog that surrounds this issue. As a general principle: Routing determines where packets are to be sent. Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there. There are ways that Shorewall can affect routing which are described in the following sections.

The following diagram shows the relationship between routing decisions and Netfilter. The light blue boxes indicate where routing decisions are made. Upon exit from one of these boxes, if the packet is being sent to another system then the interface and the next hop have been uniquely determined. The green boxes show where Netfilter processing takes place (as directed by Shorewall). You will notice that there are two different paths through this maze, depending on where the packet originates. We will look at each of these separately.

When a packet arrives from outside, it first undergoes Netfilter PREROUTING processing. In Shorewall terms: Packets may be marked using entries in the /etc/shorewall/tcrules file. Entries in that file containing ":P" in the mark column are applied here as are rules that default to the MARK_IN_FORWARD_CHAIN=No setting in /etc/shorewall/shorewall.conf. These marks may be used to specify that the packet should be routed using an alternate routing table; see the Shorewall Squid documentation for examples. Marking packets then using the fwmark selector in your "ip rule add" commands should NOT be your first choice. In most cases, you can use the from or dev selector instead. The destination IP address may be rewritten as a consequence of: DNAT[-] rules. REDIRECT[-] rules. Entries in /etc/shorewall/nat. So the only influence that Shorewall has over where these packets go is via NAT or by marking them so that they may be routed using an alternate routing table.

Processing of packets that originate on the firewall itself are initially routed using the default routing table then passed through the OUTPUT chains. Shorewall can influence what happens here: Packets may be marked using entries in the /etc/shorewall/tcrules file (rules with "$FW" in the SOURCE column). These marks may be used to specify that the packet should be re-routed using an alternate routing table. The destination IP address may be rewritten as a consequence of: DNAT[-] rules that specify $FW as the SOURCE. Entries in /etc/shorewall/nat that have "Yes" in LOCAL column. So again in this case, the only influence that Shorewall has over the packet destination is NAT or marking.

The Shorewall Squid documentation shows how alternate routing tables can be created and used. That documentation shows how you can use logic in /etc/shorewall/init to create and populate an alternate table and to add a routing rule for its use. It is fine to use that technique so long as you understand that you are basically just using the Shorewall init script (/etc/init.d/shorewall) to configure your alternate routing table at boot time and that other than as described in the previous section, there is no connection between Shorewall and routing when using Shorewall versions prior to 2.3.2.

There is one instance where Shorewall creates main routing table entries. When an entry in /etc/shorewall/proxyarp contains "No" in the HAVEROUTE column then Shorewall will create a host route to the IP address listed in the ADDRESS column through the interface named in the INTERFACE column. This is the only case where Shorewall directly manipulates the main routing table. Example: /etc/shorewall/proxyarp: #ADDRESS INTERFACE EXTERNAL HAVEROUTE PERSISTENT 206.124.146.177 eth1 eth0 No #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE The above entry will cause Shorewall to execute the following command: ip route add 206.124.146.177 dev eth1

Beginning with Shorewall 2.3.2, support is included for multiple internet connections. If you wish to use this feature, we recommend strongly that you upgrade to version 2.4.2 or later. This section assumes that you have so upgraded.

Let's assume that a firewall is connected via two separate ethernet interfaces to two different ISPs as in the following diagram. eth0 connects to ISP1. The IP address of eth0 is 206.124.146.176 and the ISP's gateway router has IP address 206.124.146.254. eth1 connects to ISP 2. The IP address of eth1 is 130.252.99.27 and the ISP's gateway router has IP address 130.252.99.254. Each of these providers is described in an entry in the file /etc/shorewall/providers. Entries in /etc/shorewall/providers can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in /etc/shorewall/tcrules can be used to direct particular outgoing connections to one ISP or the other. Use of /etc/shorewall/tcrules is not required for /etc/shorewall/providers to work, but you must select a unique MARK value for each provider so Shorewall can set up the correct marking rules for you. When using /etc/shorewall/providers, connections from the internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall. Shorewall will set up the routing and will update the /etc/iproute2/rt_tables to include the table names and number of the tables that it adds. This feature uses packet marking to control the routing. As a consequence, there are some restrictions concerning entries in /etc/shorewall/tcrules: Packet marking for traffic control purposes may not be done in the PREROUTING table for connections involving providers with 'track' specified (see below). You may not use the SAVE or RESTORE options. You may not use connection marking. Use of this feature requires that your kernel and iptables support CONNMARK target and conntrack match support. It does NOT require the ROUTE target extension. The current version of iptables (1.3.1) is broken with respect to CONNMARK and iptables-save/iptables-restore. This means that if you configure multiple ISPs, shorewall restore may fail. If it does, you may patch your iptables using the patch at http://shorewall.net/pub/shorewall/contrib/iptables/CONNMARK.diff. The /etc/shorewall/providers file can also be used in other routing scenarios. See the Squid documentation for an example.

Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value. NAME The provider name. Must begin with a letter and consist of letters and digits. The provider name becomes the name of the generated routing table for this provider. NUMBER A number between 1 and 252. This becomes the routing table number for the generated table for this provider. MARK A mark value used in your /etc/shorewall/tcrules file to direct packets to this provider. Shorewall will also mark connections that have seen input from this provider with this value and will restore the packet mark in the PREROUTING CHAIN. DUPLICATE Gives the name or number of a routing table to duplicate. May be 'main' or the name or number of a previously declared provider. For most applications, you want to specify 'main' here. INTERFACE The name of the interface to the provider. GATEWAY The IP address of the provider's Gateway router. You can enter detect here and Shorewall will attempt to automatically determine the gateway IP address. Hint: "detect" is appropriate for use in cases where the interface named in the INTERFACE column is dynamically configured via DHCP etc. OPTIONS A comma-separated list from the following: track If specified, connections FROM this interface are to be tracked so that responses may be routed back out this same interface. You want specify 'track' if internet hosts will be connecting to local servers through this provider. Any time that you specify 'track', you will also want to specify 'balance' (see below). balance The providers that have 'balance' specified will get outbound traffic load-balanced among them. Balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider. By default, each provider is given the same weight (1) . Beginning with 2.4.0-RC3, you can change the weight of a given provider by following balance with "=" and the desired weight (e.g., balance=2). The weights reflect the relative bandwidth of the providers connections and should be small numbers since the kernel actually creates additional default routes for each weight increment. loose Do not include routing rules that force traffic whose source IP is an address of the INTERFACE to be routed to this provider. Useful for defining providers that are to be used only when the appropriate packet mark is applied. COPY When you specify an existing table in the DUPLICATE column, Shorewall copies all routes through the interface specified in the INTERFACE column plus the interfaces listed in this column. At a minumum, you should list all interfaces on your firewall in this column except those internet interfaces specified in the INTERFACE column of entries in this file.

Adding another entry in the providers file simply creates an alternate routing table for you. In addition: Unless loose is specified, an ip rule is generated for each IP address on the INTERFACE that routes traffic from that address through the associated routing table. If you specify track, then connections which have had at least one packet arrive on the interface listed in the INTERFACE column have their connection mark set to the value in the MARK column. In the PREROUTING chain, packets with that connmark have their packet mark set to that value; packets so marked then bypass any prerouting rules that you create in /etc/shorewall/tcrules. This ensures that packets associated with connections from outside are always routed out of the correct interface. If you specify balance, then Shorewall will replace the 'default' route with weight 100 in the 'main' routing table with a load-balancing route among those gateways where balance was specified. So if you configure default routes, be sure that their weight is less than 100 or the route added by Shorewall will not be used. That's all that these entries do. You still have to follow the principle stated at the top of this article: Routing determines where packets are to be sent. Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there. The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic with the provider's MARK value in /etc/shorewall/tcrules and you must do that marking in the PREROUTING chain. Entries in /etc/shorewall/providers permanently alter your firewall/gateway's routing; that is, the effect of these changes is not reversed by shorewall stop or shorewall clear. To restore routing to its original state, you will have to restart your network. This can usually be done by /etc/init.d/network restart or /etc/init.d/networking restart. Check your distribution's networking documentation. You can mitigate the effect of the Shorewall-generated changes to your routing table by specifying a metric for each default route that you configure. Shorewall will generate a load-balancing default route (assuming that balance has been specified for some of the providers) that does not include a metric and that will therefore not replace any existing route that has a non-zero metric.

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers as follows. Assume tht there is a single internal interface, eth2. #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY ISP1 1 1 main eth0 206.124.146.254 track,balance eth2 ISP2 2 2 main eth1 130.252.99.254 track,balance eth2 Other configuration files go something like this: /etc/shorewall/interfaces: #ZONE INTERFACE BROADCAST OPTIONS net eth0 detect … net eth1 detect … /etc/shorewall/policy: #SOURCE DESTINATION POLICY LIMIT:BURST net net DROP If you have masqueraded hosts, be sure to update /etc/shorewall/masq to masquerade to both ISPs. For example, if you masquerade all hosts connected to eth2 then: #INTERFACE SUBNET ADDRESS eth0 eth2 206.124.146.176 eth1 eth2 130.252.99.27 Entries in /etc/shorewall/masq have no effect on which ISP a particular connection will be sent through. That is rather the purpuse of entries in /etc/shorewall/tcrules. Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You would make this entry in /etc/shorewall/tcrules (and you would set TC_ENABLED=Yes in /etc/shorewall/shorewall.conf). #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST # PORT(S) 2:P <local network> 0.0.0.0/0 tcp 25