Tom Eastep 2009 Thomas M. Eastep Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled GNU Free Documentation License. The ruleset shown in this article uses Shorewall features that are not available in Shorewall versions prior to 4.4.0.

The configuration described in this article represents the network at shorewall.net during the summer of 2009. It uses the following Shorewall features: Two Internet Interfaces A DMZ with two "systems" using Proxy ARP and running in OpenVZ Virtual Environments IPv6 Access through a 6to4 Tunnel OpenVPN and IPSEC for access when we are on the road. Ipsets Dynamic Zones Transparent proxy using Squid Manual Chains Traffic Shaping Linux runs the firewall and the servers (although they run in OpenVZ containers on the firewall system). Linux is not used natively on any of our other systems except for an HP mini which runs HP Mobile Internet Experience (MIE) -- essentially Ubuntu Hardy. I rather run Windows natively (either Vista Home Premium or XP Professional) and run Linux in VMs under VirtualBox. This approach has a number of advantages: Efficient disk utilization. The virtual disks used by Linux are just files in the NTFS file system. There is no need to pre-allocate one or more partitions for use by Linux. Some large applications, like Google Earth, are installed only on Windows. Avoids proprietary hardware issues. The Linux VMs emulate standard hardware that is well-supported by Linux. Avoids DRM hassles All DRM-protected media can be handled under Windows. Websites that don't work with Firefox (or at least with Linux Firefox) VirtualBox is fast (when your processor supports virtualization extensions) and very easy to use. I highly recommend it!

Our network is diagrammed in the following graphic. We have accounts with two different ISPs: Comcast This is a high-speed (20mb/4mb) link with a single dynamic IPv4 address. We are not allowed to run servers accessible through this account. Avvanta This is a low-speec (1.5mb/384kbit) link with five static IP address. Our servers are accessed through this account. The wired local network is restricted to my home office. The wireless network is managed by a Linksys WRT300N pre-N router which we use only as an access point -- its WAN interface is unused and it is configured to not do NAT. The wireless network uses WPA2 personal security and MAC filtering is enabled in the router. These two factors make it a hassle when guests visit with a laptop but provide good security for the network.

This section contains exerpts from the Shorewall configuration. It is important to keep in mind that parts of my configuration are there just to provide a test bed for Shorewall features. So while they show correct usage, they don't necessarily provide any useful benefit. I have tried to point those out in the sub-sections that follow.

MIRRORS=62.216.169.37,\ 63.229.2.114,\ ... NTPSERVERS=... POPSERVERS=... LOG=ULOG INT_IF=eth1 EXT_IF=eth2 COM_IF=eth0 VPS_IF=venet0As shown, this file defines variables to hold the various lists of IP addresses that I need to maintain. To simplify network reconfiguration, I also use variables to define the log level and the network interfaces.

############################################################################### # S T A R T U P E N A B L E D ############################################################################### STARTUP_ENABLED=Yes ############################################################################### # V E R B O S I T Y ############################################################################### VERBOSITY=0 ############################################################################### # C O M P I L E R # (setting this to 'perl' requires installation of Shorewall-perl) ############################################################################### SHOREWALL_COMPILER=perl ############################################################################### # L O G G I N G ############################################################################### LOGFILE=/var/log/ulog/syslogemu.log STARTUP_LOG=/var/log/shorewall-init.log LOG_VERBOSITY=2 LOGFORMAT="%s:%s:" LOGTAGONLY=No LOGRATE= LOGBURST= LOGALLNEW= BLACKLIST_LOGLEVEL= MACLIST_LOG_LEVEL= TCP_FLAGS_LOG_LEVEL=$LOG SMURF_LOG_LEVEL=$LOG LOG_MARTIANS=No ############################################################################### # L O C A T I O N O F F I L E S A N D D I R E C T O R I E S ############################################################################### IPTABLES= IPSET= PATH=/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin SHOREWALL_SHELL=/bin/sh SUBSYSLOCK= MODULESDIR= CONFIG_PATH=/etc/shorewall:/usr/share/shorewall RESTOREFILE= IPSECFILE=zones LOCKFILE= ############################################################################### # D E F A U L T A C T I O N S / M A C R O S ############################################################################### DROP_DEFAULT="Drop" REJECT_DEFAULT="Reject" ACCEPT_DEFAULT="none" QUEUE_DEFAULT="none" ############################################################################### # R S H / R C P C O M M A N D S ############################################################################### RSH_COMMAND='ssh ${root}@${system} ${command}' RCP_COMMAND='scp ${files} ${root}@${system}:${destination}' ############################################################################### # F I R E W A L L O P T I O N S ############################################################################### IP_FORWARDING=Yes ADD_IP_ALIASES=No ADD_SNAT_ALIASES=No RETAIN_ALIASES=No TC_ENABLED=Internal TC_EXPERT=No CLEAR_TC=Yes MARK_IN_FORWARD_CHAIN=Yes CLAMPMSS=Yes ROUTE_FILTER=No DETECT_DNAT_IPADDRS=Yes MUTEX_TIMEOUT=60 ADMINISABSENTMINDED=Yes BLACKLISTNEWONLY=Yes DELAYBLACKLISTLOAD=No MODULE_SUFFIX=ko DONT_LOAD= DISABLE_IPV6=No BRIDGING=No DYNAMIC_ZONES=No PKTTYPE=No MACLIST_TABLE=mangle MACLIST_TTL=60 SAVE_IPSETS=No MAPOLDACTIONS=No FASTACCEPT=No IMPLICIT_CONTINUE=Yes HIGH_ROUTE_MARKS=Yes USE_ACTIONS=Yes OPTIMIZE=1 EXPORTPARAMS=Yes EXPAND_POLICIES=Yes KEEP_RT_TABLES=No DELETE_THEN_ADD=No MULTICAST=Yes AUTO_COMMENT=Yes MANGLE_ENABLED=Yes NULL_ROUTE_RFC1918=Yes USE_DEFAULT_RT=No RESTORE_DEFAULT_ROUTE=No FAST_STOP=Yes AUTOMAKE=No LOG_MARTIANS=Yes WIDE_TC_MARKS=Yes ############################################################################### # P A C K E T D I S P O S I T I O N ############################################################################### BLACKLIST_DISPOSITION=DROP MACLIST_DISPOSITION=ACCEPT TCP_FLAGS_DISPOSITION=DROP I don't believe that there is anything remarkable there

#ACTION Mirrors # Accept traffic from Shorewall Mirrors I make this into an action so the rather long list of rules go into their own chain.

#TARGET SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE # PORT PORT(S) DEST LIMIT COMMENT Accept traffic from Mirrors ACCEPT $MIRRORS See the rules file -- this action is used for rsync traffic.

fw firewall loc ipv4 #Local Zone dmz ipv4 #DMZ net ipv4 #Internet vpn:loc,net ipsec #IPSEC drct:loc ipv4 #Direct internet accessThe vpn zone is mostly for testing Shorewall IPSEC support. It is nested in loc and net to test a feature added in Shorewall 4.4.0. The drct zone is a dynamic zone whose members bypass the transparent proxy. Some applications (such as VirtualBox registration) don't work through the proxy.

#ZONE INTERFACE BROADCAST OPTIONS loc $INT_IF detect dhcp,logmartians=1,routefilter=1,tcpflags dmz $VPS_IF detect logmartians=1,routefilter=0,routeback net $EXT_IF detect dhcp,blacklist,tcpflags,optional,routefilter=0,nosmurfs,logmartians=0,proxyarp=1 net $COM_IF detect dhcp,blacklist,tcpflags,optional,upnp,routefilter=0,nosmurfs,logmartians=0 loc tun+ detectNotice that VPN clients are treated the same as local hosts. I set the proxyarp option on $EXT_IF so that The firewall will respond to ARP who-has requests for the servers in the DMZ. To keep OpenVZ happy (it issues dire warnings if the option is not set on the associated external interface).

#ZONE HOST(S) OPTIONS vpn $EXT_IF:0.0.0.0/0 vpn $COM_IF:0.0.0.0/0 vpn $INT_IF:0.0.0.0/0 drct $INT_IF:dynamicThe vpn zone includes ipsec hosts interfacing from either external interface as well as the local interface. drct is defined as dynamic through the local interface (recall that it is a sub-zone of loc).

#SOURCE DEST POLICY LOG LIMIT:BURST # LEVEL $FW dmz REJECT $LOG $FW all ACCEPT loc net ACCEPT - loc fw ACCEPT loc vpn ACCEPT vpn fw ACCEPT vpn loc ACCEPT net net NONE net all DROP $LOG 8/sec:30 dmz fw REJECT $LOG all fw DROP $LOG all all REJECT $LOGI'm a bit sloppy with my fw<->loc policies -- I should fix that someday...

#ACTION CHAIN SOURCE DESTINATION PROTO DEST SOURCE USER/ # PORT(S) PORT(S) GROUP hp:COUNT accounting $COM_IF $INT_IF:172.20.1.107 UDP hp:COUNT accounting $INT_IF:172.20.1.107 $COM_IF UDP DONE hp mail:COUNT - $EXT_IF $VPS_IF:206.124.146.0/24 tcp 25 mail:COUNT - $VPS_IF:206.124.146.0/24 $EXT_IF tcp 25 DONE mail web - $EXT_IF $VPS_IF:206.124.146.0/24 tcp 80 web - $EXT_IF $VPS_IF:206.124.146.0/24 tcp 443 web - $VPS_IF:206.124.146.0/24 $EXT_IF tcp - 80 web - $VPS_IF:206.124.146.0/24 $EXT_IF tcp - 443 COUNT web $EXT_IF $VPS_IF:206.124.146.0/24 COUNT web $VPS_IF:206.124.146.0/24 $EXT_IF The accounting chains are as follows: hp Counts traffic to/from my work laptop to HP. The VPN users NAT-Traversal (UDP 4500) so I just count all UDP traffic to/from my work system. mail Incoming and outgoing email web Website traffic (both HTTP and HTTPS)

#ADDRESS/SUBNET PROTOCOL PORT - udp 1024:1033,1434 - tcp 57,1433,1434,2401,2745,3127,3306,3410,4899,5554,6101,8081,9898This configuration silently drops a few ports that get lots of traffic.

use strict; use Shorewall::Chains; my $chainref = ensure_manual_chain qw/DNS_DDoS/; add_rule $chainref, q(-m string --algo bm --from 30 --to 31 --hex-string "|010000010000000000000000020001|" -j DROP); add_rule $chainref, q(-m string --algo bm --from 30 --to 31 --hex-string "|000000010000000000000000020001|" -j DROP); add_rule $chainref, q(-j ACCEPT); 1;The above was created during a recent DDOS incident that targeted DNS servers. It illustrates how manual chains can be created.

if [ -f /var/lib/dhcp3/dhclient.${1}.leases ]; then grep 'option routers' /var/lib/dhcp3/dhclient.${1}.leases | tail -n 1 | while read j1 j2 gateway; do echo $gateway | sed 's/;//'; return 0; done fiThe Comcast line has a dynamic IP address assigned with the help of dhclient.

local status status=0 [ -f /etc/shorewall/${1}.status ] && status=$(cat /etc/shorewall/${1}.status) return $statusFor use with lsm.

start_lsm() { killall lsm 2> /dev/null cat <<EOF > /etc/lsm/shorewall.conf connection { name=Avvanta checkip=206.124.146.254 device=$EXT_IF ttl=2 } connection { name=Comcast checkip=${ETH0_GATEWAY:-71.231.152.1} device=$COM_IF ttl=1 } EOF rm -f /etc/shorewall/*.status /usr/sbin/lsm /etc/lsm/lsm.conf >> /var/log/lsm } This function configures and starts lsm.

#INTERFACE SOURCE ADDRESS COMMENT Masquerade Local Network $COM_IF 0.0.0.0/0 $EXT_IF !206.124.146.0/24 206.124.146.179 All connections out through Comcast must have the dynamically assigned address as their source address. Traffic from hosts without an Avvanta public IP address get 206.124.146.179 as their source address.

#SOURCE DESTINATION PROTO DEST SOURCE USER/ # PORT(S) PORT(S) GROUP net:!192.88.99.1 - 41 dmz 206.124.146.255 udp dmz 255.255.255.255 udp loc 172.20.1.255 udp loc 255.255.255.255 udp $FW 255.255.255.255 udp $FW 172.20.1.255 udp $FW 206.124.146.255 udpThis file omits the 6to4 traffic originating from 6to4 relays as well as broadcast traffic (which Netfilter doesn't handle).

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY Avvanta 1 0x10000 main $EXT_IF 206.124.146.254 track,loose,fallback $INT_IF,$VPS_IF,tun* Comcast 2 0x20000 main $COM_IF detect track,balance $INT_IF,$VPS_IF,tun*See the Multi-ISP article for an explaination of the multi-ISP aspects of this configuration.

<empty>As mentioned above, I set the proxyarp on the associated external interface instead of defining proxy ARP in this file.

if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then start_lsm fi chmod 744 ${VARDIR}/stateIf lsm isn't running then start it. Make the state file world-readable.

#SOURCE DEST PROVIDER PRIORITY - 72.20.0.0/24 main 1000 #OpenVPN clients - 206.124.146.177 main 1001 #Servers -- Routes configured by OpenVZ - 206.124.146.178 main 1001 # - 216.168.3.44 Avvanta 1001 #NNTP -- Does source IP verification 206.124.146.176/30 - Avvanta 26000 #Avvanta public IP addresses 206.124.146.180 - Avvanta 26000 #These entries simply ensure that outgoing traffic uses the correct interface.

#INTERFACE HOST(S) OPTIONS PROTO $INT_IF 172.20.1.0/24 source,dest $VPS_IF 206.124.146.177,206.124.146.178 $EXT_IF - notrack 41Keep the lights on while Shorewall is stopped.

############################################################################################################################################################################### #ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/ # PORT PORT(S) DEST LIMIT GROUP ############################################################################################################################################################################### SECTION ESTABLISHED SECTION RELATED SECTION NEW REJECT:$LOG loc net tcp 25 #Stop direct loc->net SMTP (Comcast uses submission). REJECT:$LOG loc net udp 1025:1031 #MS Messaging COMMENT Stop NETBIOS crap REJECT loc net tcp 137,445 REJECT loc net udp 137:139 COMMENT Stop my idiotic work laptop from sending to the net with an HP source IP address DROP loc:!172.20.0.0/23 net COMMENT ############################################################################################################################################################################### # Local Network to Firewall # NONAT drct - REDIRECT- loc 3128 tcp 80 - !66.199.187.46,172.20.1.108,206.124.146.177,155.98.64.80,81.19.16.0/21 ############################################################################################################################################################################### # Local network to DMZ # ACCEPT loc dmz udp domain,177 ACCEPT loc dmz tcp ssh,smtp,465,587,www,ftp,imaps,domain,https,5901:5903 - ACCEPT loc dmz udp 33434:33524 ############################################################################################################################################################################### # Internet to ALL -- drop NewNotSyn packets # dropNotSyn net fw tcp dropNotSyn net loc tcp dropNotSyn net dmz tcp ############################################################################################################################################################################### # Internet to DMZ # DNS_DDoS net dmz udp domain ACCEPT net dmz tcp smtp,www,ftp,465,587,imaps,domain,https ACCEPT net dmz udp 33434:33454 Mirrors:none net dmz tcp 873 ACCEPT net dmz tcp 22 - - s:ssh:3/min:3 ############################################################################################################################################################# ################# # # Net to Local # Limit:$LOG:SSHA,3,60\ net loc tcp 22 # # BitTorrent from Wireless Network # #DNAT net:$COM_IF loc:172.20.1.102 tcp 6881:6889 #DNAT net:$COM_IF loc:172.20.1.102 udp 6881 # # UPnP # forwardUPnP net loc # # Silently Handle common probes # REJECT net loc tcp www,ftp,https DROP net loc icmp 8 ############################################################################################################################################################################### # DMZ to Internet # ACCEPT dmz net udp domain,ntp REJECT dmz net:$COM_IF tcp smtp ACCEPT dmz net tcp echo,ftp,ssh,smtp,whois,domain,www,81,nntp,https,2401,2702,2703,8080 ACCEPT dmz net:$POPSERVERS tcp pop3 # # Some FTP clients seem prone to sending the PORT command split over two packets. This prevents the FTP connection tracking # code from processing the command and setting up the proper expectation. The following rule allows active FTP to work in these cases # but logs the connection so I can keep an eye on this potential security hole. # ACCEPT:$LOG dmz net tcp 1024: 20 ############################################################################################################################################################################### # DMZ to Local # ACCEPT dmz loc tcp 22 - - s:ssh:3/min:3 ############################################################################################################################################################################### # DMZ to Firewall -- ntp & snmp Silently reject Auth # ACCEPT dmz fw tcp 161,ssh ACCEPT dmz fw udp 161,ntp REJECT dmz fw tcp auth ############################################################################################################################################################################### # Internet to Firewall # REJECT net fw tcp www,ftp,https DROP net fw icmp 8 ACCEPT net fw udp 33434:33454 ACCEPT net fw tcp 22 - - s:ssh:3/min:3 ACCEPT net fw udp 33434:33524 ############################################################################################################################################################################### # Firewall to DMZ # ACCEPT fw dmz tcp domain,www,ftp,ssh,smtp,https,993,465,587,5901 ACCEPT fw dmz udp domain REJECT fw dmz udp 137:139 ############################################################################################################################################################################## # COMMENT Freenode Probes DROP net:82.96.96.3,85.190.0.3 any COMMENT ############################################################################################################################################################################## # Allow Ping except where disallowed earlier # ACCEPT any any icmp 8

if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then start_lsm fi chmod 744 ${VARDIR}/stateIf lsm isn't running then start it. Make the state file world-readable.

if [ "$COMMAND" = stop -o "$COMMAND" = clear ]; then killall lsm 2> /dev/null fi chmod 744 ${VARDIR}/stateKill lsm if the command is stop or clear. Make the state file world-readable.

#INTERFACE IN-BANDWITH OUT-BANDWIDTH OPTIONS $EXT_IF - 300kbit classify $INT_IF - 80mbit classify $COM_IF - 4mbit classify,hfsc The use of HFSC on the Comcast link is largely to provide a test bed for that qdisc; I really don't have any real-time requirement such as VOIP.

INTERFACE MARK RATE CEIL PRIORITY OPTIONS 1:110 - full/4 full 1 tcp-ack,tos-minimize-delay 1:120 - full/4 full 2 flow=nfct-src 1:130 - full/4 230kbit 3 default,flow=nfct-src 1:140 - full/4 230kbit 4 flow=nfct-src 2:10 - 95*full/100 full 1 flow=dst 2:100 - 14mbit 20mbit 2 2:100:101 - 7mbit 20mbit 3 default,flow=dst 2:100:102 - 7mbit 20mbit 3 flow=dst 3:10 - 2mbit:4ms full 1 flow=nfct-src 3:100 - 2mbit full 2 3:100:101 - 1mbit full 3 default,flow=nfct-src 3:100:102 - 1mbit full 3 flow=nfct-src Note that most of the outgoing bandwidth on the local interface is allocated to one class. That class is used for local traffic.

#INTERFACE: SOURCE DEST PROTO DEST SOURCE TOS LENGTH #CLASS PORT(S) PORT(S) # =============================== AVVANTA ==================================== # # Give Highest priority to LSM's pings to the gateway and to DNS queries # 1:110 206.124.146.176 206.124.146.254 icmp 1:110 206.124.146.177 - udp 53 # # Second Highest priority to IPv6 Tunnel # 1:120 206.124.146.180 # # Lowest priority to bulk traffic # 1:140 206.124.146.177 - tcp - 873 - 2048 1:140 206.124.146.177 - - - - tos-minimize-cost The tcfilters file is only used for the Avvanta provider because it has static public IP addresses.

#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS # PORT(S) COMMENT Shape incoming traffic # # Most of the bandwidth is reserved for local traffic since the downlinks aren't that fast # 2:10 206.124.146.176/30 $INT_IF 2:10 206.124.146.177 $INT_IF 2:10 172.20.1.254 $INT_IF # # Guarantee 1/2 of the incoming bandwidth for my work system # 2:102 0.0.0.0/0 $INT_IF:172.20.1.107 COMMENT Shape outgoing traffic to Comcast # # Give 1/2 to my work system and add a latency guarantee # 3:10 172.20.1.107 $COM_IF # # Restrict Torrent uploads # 3:102 172.20.1.0/24 $COM_IF tcp - 6881:6889 The tcrules file is used to classify traffic that deals with the local network and/or with Comcast.

#TYPE ZONE GATEWAY GATEWAY # ZONE openvpnserver:udp net 6to4 net ipsec net ipsec loc ipip vpn 0.0.0.0/0The ipip tunnel from the vpn zone handles IP compression on IPSEC connections.