Shorewall and Routing Tom Eastep 2005-08-11 2005 Thomas M. Eastep Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled GNU Free Documentation License.
Routing vs. Firewalling. One of the most misunderstood aspects of Shorewall is its relationship with routing. This article attempts to clear some of the fog that surrounds this issue. As a general principle: Routing determines where packets are to be sent. Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there. There are ways that Shorewall can affect routing which are described in the following sections.
Routing and Netfilter The following diagram shows the relationship between routing decisions and Netfilter. The light blue boxes indicate where routing decisions are made. Upon exit from one of these boxes, if the packet is being sent to another system then the interface and the next hop have been uniquely determined. The green boxes show where Netfilter processing takes place (as directed by Shorewall). You will notice that there are two different paths through this maze, depending on where the packet originates. We will look at each of these separately.
Packets Entering the Firewall from Outside When a packet arrives from outside, it first undergoes Netfilter PREROUTING processing. In Shorewall terms: Packets may be marked using entries in the /etc/shorewall/tcrules file. Entries in that file containing ":P" in the mark column are applied here as are rules that default to the MARK_IN_FORWARD_CHAIN=No setting in /etc/shorewall/shorewall.conf. These marks may be used to specify that the packet should be routed using an alternate routing table; see the Shorewall Squid documentation for examples. Marking packets then using the fwmark selector in your "ip rule add" commands should NOT be your first choice. In most cases, you can use the from or dev selector instead. The destination IP address may be rewritten as a consequence of: DNAT[-] rules. REDIRECT[-] rules. Entries in /etc/shorewall/nat. So the only influence that Shorewall has over where these packets go is via NAT or by marking them so that they may be routed using an alternate routing table.
Packets Originating on the Firewall Processing of packets that originate on the firewall itself are initially routed using the default routing table then passed through the OUTPUT chains. Shorewall can influence what happens here: Packets may be marked using entries in the /etc/shorewall/tcrules file (rules with "$FW" in the SOURCE column). These marks may be used to specify that the packet should be re-routed using an alternate routing table. The destination IP address may be rewritten as a consequence of: DNAT[-] rules that specify $FW as the SOURCE. Entries in /etc/shorewall/nat that have "Yes" in LOCAL column. So again in this case, the only influence that Shorewall has over the packet destination is NAT or marking.
Alternate Routing Table Configuration The Shorewall Squid documentation shows how alternate routing tables can be created and used. That documentation shows how you can use logic in /etc/shorewall/init to create and populate an alternate table and to add a routing rule for its use. It is fine to use that technique so long as you understand that you are basically just using the Shorewall init script (/etc/init.d/shorewall) to configure your alternate routing table at boot time and that other than as described in the previous section, there is no connection between Shorewall and routing when using Shorewall versions prior to 2.3.2.
Routing and Proxy ARP There is one instance where Shorewall creates main routing table entries. When an entry in /etc/shorewall/proxyarp contains "No" in the HAVEROUTE column then Shorewall will create a host route to the IP address listed in the ADDRESS column through the interface named in the INTERFACE column. This is the only case where Shorewall directly manipulates the main routing table. Example: /etc/shorewall/proxyarp: #ADDRESS INTERFACE EXTERNAL HAVEROUTE PERSISTENT 206.124.146.177 eth1 eth0 No #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE The above entry will cause Shorewall to execute the following command: ip route add 206.124.146.177 dev eth1
Multiple Internet Connection Support in Shorewall 2.4.2 and Later Beginning with Shorewall 2.3.2, support is included for multiple internet connections. If you wish to use this feature, we recommend strongly that you upgrade to version 2.4.2 or later. This section assumes that you have so upgraded.
Overview Let's assume that a firewall is connected via two separate ethernet interfaces to two different ISPs as in the following diagram. eth0 connects to ISP1. The IP address of eth0 is 206.124.146.176 and the ISP's gateway router has IP address 206.124.146.254. eth1 connects to ISP 2. The IP address of eth1 is 130.252.99.27 and the ISP's gateway router has IP address 130.252.99.254. Each of these providers is described in an entry in the file /etc/shorewall/providers. Entries in /etc/shorewall/providers can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in /etc/shorewall/tcrules can be used to direct particular outgoing connections to one ISP or the other. Connections from the internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall. Shorewall will set up the routing and will update the /etc/iproute2/rt_tables to include the table names and number of the tables that it adds. This feature uses packet marking to control the routing. As a consequence, there are some restrictions concerning entries in /etc/shorewall/tcrules: Packet marking for traffic control purposes may not be done in the PREROUTING table for connections involving providers with 'track' specified (see below). You may not use the SAVE or RESTORE options. You may not use connection marking. Use of this feature requires that your kernel and iptables support CONNMARK target and conntrack match support. It does NOT require the ROUTE target extension. The current version of iptables (1.3.1) is broken with respect to CONNMARK and iptables-save/iptables-restore. This means that if you configure multiple ISPs, shorewall restore may fail. If it does, you may patch your iptables using the patch at http://shorewall.net/pub/shorewall/contrib/iptables/CONNMARK.diff. The /etc/shorewall/providers file can also be used in other routing scenarios. See the Squid documentation for an example.
/etc/shorewall/providers File Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value. /etc/shorewall/providers: NAME The provider name. Must begin with a letter and consist of letters and digits. The provider name becomes the name of the generated routing table for this provider. NUMBER A number between 1 and 252. This becomes the routing table number for the generated table for this provider. MARK A mark value used in your /etc/shorewall/tcrules file to direct packets to this provider. Shorewall will also mark connections that have seen input from this provider with this value and will restore the packet mark in the PREROUTING CHAIN. DUPLICATE Gives the name or number of a routing table to duplicate. May be 'main' or the name or number of a previously declared provider. For most applications, you want to specify 'main' here. INTERFACE The name of the interface to the provider. GATEWAY The IP address of the provider's Gateway router. You can enter detect here and Shorewall will attempt to automatically determine the gateway IP address. Hint: "detect" is appropriate for use in cases where the interface named in the INTERFACE column is dynamically configured via DHCP etc. OPTIONS A comma-separated list from the following: track If specified, connections FROM this interface are to be tracked so that responses may be routed back out this same interface. You want specify 'track' if internet hosts will be connecting to local servers through this provider. Any time that you specify 'track', you will also want to specify 'balance' (see below). balance The providers that have 'balance' specified will get outbound traffic load-balanced among them. Balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider. By default, each provider is given the same weight (1) . Beginning with 2.4.0-RC3, you can change the weight of a given provider by following balance with "=" and the desired weight (e.g., balance=2). The weights reflect the relative bandwidth of the providers connections and should be small numbers since the kernel actually creates additional default routes for each weight increment. COPY When you specify an existing table in the DUPLICATE column, Shorewall copies all routes through the interface specified in the INTERFACE column plus the interfaces listed in this column. At a minumum, you should list all interfaces on your firewall in this column except those internet interfaces specified in the INTERFACE column of entries in this file.
What an entry in the Providers File Does Adding another entry in the providers file simply creates an alternate routing table for you. In addition: An ip rule is generated for each IP address on the INTERFACE that routes traffic from that address through the associated routing table. If you specify track, then connections which have had at least one packet arrive on the interface listed in the INTERFACE column have their connection mark set to the value in the MARK column. In the PREROUTING chain, packets with that connmark have their packet mark set to that value; packets so marked then bypass any prerouting rules that you create in /etc/shorewall/tcrules. This ensures that packets associated with connections from outside are always routed out of the correct interface. If you specify balance, then Shorewall will replace the 'default' route with weight 100 in the 'main' routing table with a load-balancing route among those gateways where balance was specified. So if you configure default routes, be sure that their weight is less than 100 or the route added by Shorewall will not be used. That's all that these entries do. You still have to follow the principle stated at the top of this article: Routing determines where packets are to be sent. Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there. The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic with the provider's MARK value in /etc/shorewall/tcrules and you must do that marking in the PREROUTING chain. Entries in /etc/shorewall/providers permanently alter your firewall/gateway's routing; that is, the effect of these changes is not reversed by shorewall stop or shorewall clear. To restore routing to its original state, you will have to restart your network. This can usually be done by /etc/init.d/network restart or /etc/init.d/networking restart. Check your distribution's networking documentation. You can mitigate the effect of the Shorewall-generated changes to your routing table by specifying a metric for each default route that you configure. Shorewall will generate a load-balancing default route (assuming that balance has been specified for some of the providers) that does not include a metric and that will therefore not replace any existing route that has a non-zero metric.
Example The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers as follows. Assume tht there is a single internal interface, eth2. #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY ISP1 1 1 main eth0 206.124.146.254 track,balance eth2 ISP2 2 2 main eth1 130.252.99.254 track,balance eth2 Other configuration files go something like this: /etc/shorewall/interfaces: #ZONE INTERFACE BROADCAST OPTIONS net eth0 detect … net eth1 detect … /etc/shorewall/policy: #SOURCE DESTINATION POLICY LIMIT:BURST net net DROP If you have masqueraded hosts, be sure to update /etc/shorewall/masq to masquerade to both ISPs. For example, if you masquerade all hosts connected to eth2 then: #INTERFACE SUBNET ADDRESS eth0 eth2 206.124.146.176 eth1 eth2 130.252.99.27 Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You would make this entry in /etc/shorewall/tcrules (and you would set TC_ENABLED=Yes in /etc/shorewall/shorewall.conf). #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST # PORT(S) 2:P <local network> 0.0.0.0/0 tcp 25
Experimental Routing with Shorewall 2.3.2 and Later Beginning with Shorewall 2.3.2, Shorewall is integrated with the ROUTE target extension available from Netfilter Patch-O-Matic-NG (http://www.netfilter.org). As of this writing, I know of no distribution that is shipping a kernel or iptables with the ROUTE target patch included. This means that you must patch and build your own kernel and iptables in order to be able to use the feature described in this section. This code remains experimental since there is no intent by the Netfilter team to ever submit the ROUTE target patch for inclusion in the official kernels from kernel.org. This support may also be removed from Shorewall in a future release. See Shorewall FAQ 42 for information about determining if your kernel and iptables have this support enabled. You must be running Shorewall 2.3.2 or later to make this determination. Routing with Shorewall is specified through entries in /etc/shorewall/routes. Note that entries in the /etc/shorewall/routes file override the routing specified in your routing tables. These rules generate Netfilter rules in the mangle tables FORWARD chain or OUTPUT chain depending whether the packets are being routed through the firewall or originate on the firewall itself (see the flow diagram at the top of this article). Columns in this file are as follows: SOURCE Source of the packet. May be any of the following: A host or network address A network interface name. The name of an ipset prefaced with "+" $FW (for packets originating on the firewall) A MAC address in Shorewall format A range of IP addresses (assuming that your kernel and iptables support range match) A network interface name followed by ":" and an address or address range. DEST Destination of the packet. May be any of the following: A host or network address A network interface name (determined from routing table(s)) The name of an ipset prefaced with "+" A network interface name followed by ":" and an address or address range. PROTO Protocol - Must be a protocol listed in /etc/protocols, a number or "ipp2p", a number, or "all". "ipp2p" require ipp2p match support in your kernel and iptables. PORT(S) Destination Ports. A comma-separated list of Port names (from /etc/services), port numbers or port ranges; if the protocol is "icmp", this column is interpreted as the destination icmp-type(s). If the protocol is ipp2p, this column is interpreted as an ipp2p option without the leading "--" (example "bit" for bit-torrent). If no PORT is given, "ipp2p" is assumed. This column is ignored if PROTOCOL = all but must be entered if any of the following field is supplied. In that case, it is suggested that this field contain "-" SOURCE PORT(S) Optional) Source port(s). If omitted, any source port is acceptable. Specified as a comma-separated list of port names, port numbers or port ranges. TEST Defines a test on the existing packet or connection mark. The rule will match only if the test returns true. Tests have the format
[!]<value>[/<mask>][:C]
where: ! Inverts the test (not equal) <value> Value of the packet or connection mark. <mask> A mask to be applied to the mark before testing :C Designates a connection mark. If omitted, the packet mark's value is tested INTERFACE The interface that the packet is to be routed out of. If you do not specify this field then you must place "-" in this column and enter an IP address in the GATEWAY column. GATEWAY The gateway that the packet is to be forwarded through. The idea here is that traffic that matches the SOURCE, DEST, PROTO, PORT(S), SOURCE PORT(S) and TEST columns is routed out of the INTERFACE through the optional GATEWAY.
Example: Your local interface is eth1 and your DMZ interface is eth2. You want to run Squid as a transparent proxy for HTTP on 192.168.3.22 in your DMZ. You would use the following entry in /etc/shorewall/routes: #SOURCE DEST PROTO PORT(S) SOURCE TEST INTERFACE GATEWAY # PORT(S) eth1 0.0.0.0/0 tcp 80 - - eth1 192.168.3.22 This entry specifies that "traffic coming in through eth1 to TCP port 80 is to be routed out of eth1 to gateway 192.168.3.22".