shorewall_code/Shorewall/releasenotes.txt

Shorewall 4.3.11

Shorewall 4.3 is the development thread for Shorewall 4.4 which will be
released late in 2009.

----------------------------------------------------------------------------
               R E L E A S E  4 . 3  H I G H L I G H T S
----------------------------------------------------------------------------

1)  Support for Shorewall-shell has been discontinued. Shorewall-perl
    has been combined with Shorewall-common to produce a single
    Shorewall package.

2)  The Shorewall documentation and man pages have been purged of
    information about earlier Shorewall releases. The documentation
    describes only the behavior of Shorewall 4.3 and later versions.

3)  The interfaces file OPTIONs have been extended to largely remove the
    need for the hosts file.

4)  It is now possible to define PREROUTING and OUTPUT marking rules
    that cause new connections to use the same provider as an existing
    connection of the same kind.

5)  Dynamic Zone support is once again available for IPv4; ipset support is
    required in your kernel and in iptables.

6)  A new AUTOMAKE option has been added to shorewall.conf and
    shorewall6.conf. Setting this option will allow Shorewall to skip
    the compilation phase during start/restart if no configuration
    changes have occurred since the last start/restart.

7)  The LIMIT:BURST column in /etc/shorewall/policy
    (/etc/shorewall6/policy) and the RATE LIMIT column in
    /etc/shorewall/rules (/etc/shorewall6/rules) may now be used to
    limit on a per source IP or per destination IP basis.

8)  Support for per-IP traffic shaping classes has been added.

----------------------------------------------------------------------------
                    M I G R A T I O N   I S S U E S
----------------------------------------------------------------------------

1)  The 'shorewall stop', 'shorewall clear', 'shorewall6 stop' and
    'shorewall6 clear' commands no longer read the 'routestopped'
    file. The 'routestopped' file used is the one that was present at
    the last 'start', 'restart' or 'restore' command.

2)  The old macro parameter syntax (e.g., SSH/ACCEPT) is now deprecated
    in favor of the new syntax (e.g., SSH(ACCEPT)). The 4.3 documentation
    uses the new syntax exclusively, although the old syntax
    continues to be supported.

3)  Support for the SAME target in /etc/shorewall/masq and
    /etc/shorewall/rules has been removed, following the removal of the
    underlying support in the Linux kernel.

4)  Supplying an interface name in the SOURCE column of
    /etc/shorewall/masq is now deprecated. Entering the name of an
    interface there will result in a compile-time warning:

    WARNING: Using an interface as the masq SOURCE requires the
             interface to be up and configured when Shorewall
             starts/restarts

    To avoid this warning, replace interface names by the corresponding
    network addresses (e.g., 192.168.144.0/24).

5)  Previously, Shorewall has treated traffic shaping class IDs as
    decimal numbers (or pairs of decimal numbers). That worked fine
    until IPMARK was implemented. IPMARK requires Shorewall to generate
    class Ids in numeric sequence. In 4.3.9, that didn't work correctly
    because Shorewall was generating the sequence "..8,9,10,11..." when
    the correct sequence was "...8,9,a,b,...". Shorewall now treats
    class IDs as hex, like 'tc' and 'iptables' do.

    This should only be an issue if you have more than 9 interfaces
    defined in /etc/shorewall/tcdevices and if you use class IDs in
    /etc/shorewall/tcrules. You will need to renumber the class IDs for
    devices 10 and greater.

----------------------------------------------------------------------------
          P R O B L E M S   C O R R E C T E D   I N   4 . 3 . 11
----------------------------------------------------------------------------

1.  Previously, specifying a source port for SCTP in
    /etc/shorewall/tcfilters would cause an invalid rule to be
    generated.

2.  The list of networks returned by the 'shorewall iprange' command
    was not minimal.

----------------------------------------------------------------------------
             K N O W N   P R O B L E M S   R E M A I N I N G
----------------------------------------------------------------------------

None.

----------------------------------------------------------------------------
                N E W   F E A T U R E S   I N   4 . 3 . 11
----------------------------------------------------------------------------

None.
   
----------------------------------------------------------------------------
                 N E W   F E A T U R E S   IN   4 . 3
----------------------------------------------------------------------------

1)  The Shorewall packaging has been completely revamped in Shorewall
    4.3.

    The new packages are:

    - Shorewall.   Includes the former Shorewall-common and
                   Shorewall-perl packages. Includes everything needed
                   to create an IPv4 firewall.

    - Shorewall6.  Requires Shorewall. Adds the components necessary to
      		   create an IPv6 firewall.

    - Shorewall-lite

		   May be installed on a firewall system to run
		   IPv4 firewall scripts generated by Shorewall.

    - Shorewall6-lite

		   May be installed on a firewall system to run
		   IPv6 firewall scripts generated by Shorewall6.

2)  The interfaces file supports a new 'nets=' option. This option
    allows users to restrict a zone's definition to particular networks
    through an interface without having to use the hosts file.

    Example interfaces file:

    #ZONE	INTERFACE	BROADCAST		OPTIONS
    loc		eth3		detect			dhcp,logmartians=1,routefilter=1,nets=172.20.1.0/24
    dmz     	eth4		detect			logmartians=1,routefilter=1,nets=206.124.146.177
    net		eth0		detect			dhcp,blacklist,tcpflags,optional,routefilter=0,nets=(!172.20.0.0/24,206.124.146.177)
    net		eth2		detect			dhcp,blacklist,tcpflags,optional,upnp,routefilter=0,nets=(!172.20.0.0/24,206.124.146.177)
    loc		tun+		detect			nets=172.20.0.0/24
    #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

    Note that when more than one network address is listed, the list
    must be enclosed in parentheses. Notice also that exclusion may be
    used.

    The first entry in the above interfaces file is equivalent to the
    following:

    interfaces:

    #ZONE	INTERFACE	BROADCAST		OPTIONS
    -		eth0		detect			dhcp,logmartians=1,routefilter=1

    hosts:

    #ZONE	HOST(S)					OPTIONS
    loc		$INT_IF:192.20.1.0/24			broadcast

    Note that the 'broadcast' option is automatically assumed and need
    not be explicitly specified.

3)  Some websites run applications that require multiple connections
    from a client browser. Where multiple 'balanced' providers are
    configured, this can lead to problems when some of the connections
    are routed through one provider and some through another.

    To work around this issue, the SAME target has been added to
    /etc/shorewall/tcrules. SAME may be used in the PREROUTING and
    OUTPUT chains. When used in PREROUTING, it causes matching
    connections from an individual local system to all use the same
    provider.

    For example:

    	SAME:P	192.168.1.0/24	-	tcp	80,443

    If a host in 192.168.1.0/24 attempts a connection on TCP port 80 or
    443 and it has sent a packet on either of those ports in the last
    five minutes then the new connection will use the same provider as
    the connection over which that last packet was sent.

    When used in the OUTPUT chain, it causes all matching connections
    to an individual remote system to all use the same provider.

    For example:

    	SAME	$FW	-	tcp	80,443

    If the firewall attempts a connection on TCP port 80 or
    443 and it has sent a packet on either of those ports in the last
    five minutes to the same remote system then the new connection will
    use the same provider as the connection over which that last packet
    was sent.

    Important note: SAME only works with providers that have the
    'track' option specified in /etc/shorewall/providers.

4)  The file /var/lib/shorewall/.restore has been renamed to
    /var/lib/shorewall/firewall. A similar change has been made in
    Shorewall6.

    When a successful start or restart is completed, the script that
    executed the command copies itself to to
    /var/lib/shorewall[6]/firewall.

5)  Dynamic zone support is once again available for IPv4. This support 
    is built on top of ipsets so you must have the xtables-addons
    installed on the firewall system.

    Dynamic zones are available when Shorewall-lite is used as well.

    Note that the dynamic zone support built into Shorewall provides no
    additional functionality over what is provided by simply defining a
    zone in terms of an ipset (see
    http://www1.shorewall.net/ipsets.html#Dynamic). 

    You define a zone as having dynamic content in one of two ways:

    - By specifying nets=dynamic in the OPTIONS column of an entry for
      the zone in /etc/shorewall/interfaces; or

    - By specifying <interface>:dynamic in the HOST(S) column of an
      entry for the zone in /etc/shorewall/hosts.

    When there are any dynamic zones present in your configuration, 
    Shorewall (Shorewall-lite) will:

    a) Execute the following commands during 'shorewall start' or
    'shorewall-lite start'.

           ipset -U :all: :all:
	   ipset -U :all: :default:
	   ipset -F
	   ipset -X   
	   ipset -R < ${VARDIR}/ipsets.save

       where $VARDIR normally contains /var/lib/shorewall
       (/var/lib/shorewall-lite) but may be modified by
       /etc/shorewall/vardir (/etc/shorewall-lite/vardir).


    b) During 'start', 'restart' and 'restore' processing, Shorewall
       will then attempt to create an ipset named <zone>_<interface>
       for each zone/interface pair that has been specified as
       dynamic. The type of ipset created is 'iphash' so that only
       individual IPv4 addresses may be added to the set.

    c) Execute the following commands during 'shorewall stop' or
       'shorewall-lite stop':

           if ipset -S > ${VARDIR}/ipsets.tmp; then
              mv -f ${VARDIR}/ipsets.tmp ${VARDIR}/ipsets.save
	   fi

    The 'shorewall add' and 'shorewall delete' commands are supported
    with their original syntax:

           add <interface>[:<host-list>] ... <zone>

           delete <interface>[:<host-list>] ... <zone>

    In addition, the 'show dynamic' command is added that lists the dynamic
    content of a zone.

    	    show dynamic <zone>

    These commands are supported by shorewall-lite as well.

6)  The generated program now attempts to detect all dynamic
    information when it first starts. If any of those steps fail, an
    error message is generated and the state of the firewall is not
    changed.

7)  Shorewall will now attempt to detect a dynamic gateway by reading
    the dhclient lease file for the interface
    (/var/run/dhcp/dhclient-<if>.lease).

8)  To improve readability of the configuration files, Shorewall now
    allows leading white space in continuation lines when the continued
    line ends in ":" or ",".

    Example (/etc/shorewall/rules):

    #ACTION	SOURCE		DEST		PROTO		DEST
    #                                                           PORT(S)
    ACCEPT	net:\
    		206.124.146.177,\
		206.124.146.178,\
		206.124.146.180\
				dmz		tcp		873

    The leading white space on the lines that contain just an IP
    address is ignored so the SOURCE column effectively contains
    "net:206.124.146.177,206.124.147.178,206.124.146.180".

9)  The generated script now uses iptables[6]-restore to instantiate
    the Netfilter ruleset during processing of the 'stop' command. As a
    consequence, the 'critical' option in /etc/shorewall/route_stopped
    is no longer needed and will result in a warning.

10) A new AUTOMAKE option has been added to shorewall.conf and
    shorewall6.conf. When set to 'Yes', this option causes new behavior
    during processing of the 'start' and 'restart' commands; if no
    files in /etc/shorewall/ (/etc/shorewall6) have changed since the last
    'start' or 'restart', then the compilation step is skipped and the
    script used during the last 'start' or 'restart' is used to
    start/restart the firewall.

    Note that if a <directory> is specified in the start/restart
    command (e.g., "shorewall restart /etc/shorewall.new") then the
    setting of AUTOMAKE is ignored.

    Note that the 'make' utility must be installed on the firewall
    system in order for AUTOMAKE=Yes to work correctly.

11) The 'compile' command now allows you to omit the <pathname>. When
    you do that, the <pathname> defaults to /var/lib/shorewall/firewall
    (/var/lib/shorewall6/firewall) unless you have overridden VARDIR
    using /etc/shorewall/vardir (/etc/shorewall6/vardir).

    When combined with AUTOMAKE=Yes, it allows the following:

    	 gateway:~ # shorewall compile
    	 Compiling...
    	 Shorewall configuration compiled to /root/shorewall/firewall
    	 gateway:~ #
	 ...
    	 gateway:~ # shorewall restart
    	 Restarting Shorewall....
    	 done.
    	 gateway:~ # 

    In other words, you can compile the current configuration then
    install it at a later time.

12) Thanks to I. Buijs, it is now possible to rate-limit connections by
    source IP or destination IP. The LIMIT:BURST column in
    /etc/shorewall/policy (/etc/shorewall6/policy) and the RATE LIMIT
    column /etc/shorewall/rules (/etc/shorewall6/rules) have been
    extended as follows:

    	[{s|d}:[[<name>]:]]<rate>/{sec|min}[:<burst>]

    When s: is specified, the rate is per source IP address.
    When d: is specified, the rate is per destination IP address.
    The <name> specifies the name of a hash table -- you get to choose
    the name. If you don't specify a name, the name 'shorewall' is
    assumed. Rules with the same name have their connection counts
    aggregated and the individual rates are applied to the aggregate.

    Example:

	ACCEPT	net   fw    tcp    22  - - s:ssh:3/min

    This will limit SSH connections from net->fw to 3 per minute.

    	ACCEPT	net   fw    tcp    25   - - s:mail:3/min
    	ACCEPT	net   fw    tcp    587  - - s:mail:3/min

    Since the same hash table name is used in both rules, the above is
    equivalent to this single rule:

    	ACCEPT	net   fw    tcp    25,587  - - s:mail:3/min

13) Rules that specify a log level with a target other than LOG or NFLOG
    are now implemented through a separate chain. While this may increase
    the processing cost slightly for packets that match these rules, it
    is expected to reduce the overall cost of such rules because each
    packet that doesn't match the rules only has to be processed once
    per rule rather than twice.

    Example:

    /etc/shorewall/rules:

	REJECT:info   loc       net      tcp    25

    This previously generated these two rules (long rules folded):

	 -A loc2net -p 6 --dport 25 -j LOG --log-level 6
	    --log-prefix "Shorewall:loc2net:reject:" 
	 -A loc2net -p 6 --dport 25 -j reject 

    It now generates these rules:

       	 :log0 - [0:0]
	 ...
	 -A loc2net -p 6 --dport 25 -g log0
	 ...
	 -A log0 -j LOG --log-level 6
	    --log-prefix "Shorewall:loc2net:REJECT:" 
	 -A log0 -p 6 --dport 25 -j reject 

    Notice that now there is only a single rule generated in the
    'loc2net' chain where before there were two. Packets for other than
    TCP port 25 had to be processed by both rules.

    Notice also that the new LOG rule reflects the original action
    ("REJECT") rather than what Shorewall maps that to ("reject").

14) Shorewall6 has now been tested on kernel 2.6.24 (Ubuntu Hardy) and
    hence will now start successfully when running on that kernel.

15) Three new options (IP, TC and IPSET) have been added to
    shorewall.conf and shorwall6.conf. These options specify the name
    of the executable for the 'ip', 'tc' and 'ipset' utilities
    respectively.

    If not specified, the default values are:

       IP=ip
       TC=tc
       IPSET=ipset

    In other words, the utilities will be located via the current PATH
    setting.

16) There has been a desire in the user community to limit traffic by
    IP address using Shorewall traffic shaping. Heretofore, that has
    required a very inefficient process:

    a) Define a tcclass for each internal host (two, if shaping both in
       and out).
    b) Define a tcrule for each host to mark to classify the packets
       accordingly.

    Beginning with Shorewall 4.3.9, this process is made easier IF YOU
    ARE WILLING TO INSTALL xtables-addons. The feature requires IPMARK
    support in iptables[6] and your kernel. That support is available
    in xtables-addons.

    The new facility has two components:

    a) A new IPMARK MARKing command in /etc/shorewall/tcrules.
    b) A new 'occurs' OPTION in /etc/shorewall/tcclasses.

    The facility is currently only available with IPv4.

    In a sense, the IPMARK target is more like an IPCLASSIFY target in
    that the mark value is later interpreted as a class ID. A packet
    mark is 32 bits wide; so is a class ID. The <major> class occupies
    the high-order 16 bits and the <minor> class occupies the low-order
    16 bits. So the class ID 1:4ff (remember that class IDs are always
    in hex) is equivalent to a mark value of 0x104ff. Remember that
    Shorewall uses the interface number as the <major> number where the
    first interface in tcdevices has <major> number 1, the second has
    <major> number 2, and so on.

    The IPMARK target assigns a mark to each matching packet based on
    the either the source or destination IP address. By default, it
    assigns a mark value equal to the low-order 8 bits of the source
    address.

    The syntax is as follows:

	IPMARK[([{src|dst}][,[<mask1>][,[<mask2>][,[<shift>]]]])]

    Default values are:

	    src
	    <mask1> = 0xFF
	    <mask2> = 0x00
	    <shift> = 0

     'src' and 'dst' specify whether the mark is to be based on the
     source or destination address respectively.

     The selected address is first shifted right by <shift>, then
     LANDed with <mask1> and then LORed with <mask2>. The <shift>
     argument is intended to be used primarily with IPv6 addresses.

     Example:

	IPMARK(src,0xff,0x10100)

        Destination IP address is 192.168.4.3 = 0xc0a80403

	0xc0a80403 >> 0         = 0xc0a80403
	0xc0a80403 LAND 0xFF    = 0x03
	0x03       LOR  0x10100 = 0x10103

	So the mark value is 0x10103 which corresponds to class id
	1:103.

    It is important to realize that, while class IDs are composed of a
    <major> and a <minor> value, the set of <minor> values must be
    unique. You must keep this in mind when deciding how to map IP
    addresses to class IDs. 

    For example, suppose that your internal network is 192.168.1.0/29
    (host IP addresses 192.168.1.1 - 192.168.1.6). Your first notion
    might be to use IPMARK(src,0xFF,0x10000) so as to produce class IDs
    1:1 through 1:6. But 1:1 is the class ID if the base HTB class on
    interface 1. So you might chose instent to use
    IPMARK(src,0xFF,0x10100) as shown in the example above so as to
    avoid minor class 1.

    The 'occurs' option in /etc/shorewall/tcclasses causes the class
    definition to be replicated many times. The synax is:

	 occurs=<number>

    When 'occurs' is used:

	 a) The associated device may not have the 'classify' option.
	 b) The class may not be the default class.
	 c) The class may not have any 'tos=' options (including
	    'tcp-ack').
	 d) The class should not specify a MARK value. Any MARK value
	    given is ignored with a warning.

    The 'RATE' and 'CEIL' parameters apply to each instance of the
    class. So the total RATE represented by an entry with 'occurs' will
    be the listed RATE multiplied by the 'occurs' number.

    Example:

	/etc/shorewall/tcdevices:

	#INTERFACE     IN-BANDWIDTH     OUT-BANDWIDTH
	eth0	       100mbit		100mbit

	/etc/shorewall/tcclasses:

	#DEVICE   MARK  RATE    CEIL    PRIORITY	OPTIONS
	eth0:101  -	1kbit	230kbit	       4	occurs=6

        The above defines 6 classes with class IDs 0x101-0x106. Each
        class has a guaranteed rate of 1kbit/second and a ceiling of
        230kbit.

	/etc/shoreall/tcrules:

	#MARK                      SOURCE              DEST
	IPMARK(src,0xff,0x10100):F 192.168.1.0/29      eth0

    This change also altered the way in which Shorewall generates a
    class number when none is given.

    - Prior to this change, the class number was constructed by concatinating
      the mark value with the either '1' or '10'. '10' is used when
      there are more than 10 devices defined in /etc/shorewall/tcdevices.

    - Beginning with this change, a new method is added; class numbers
      are assigned sequentially beginning with 2.

    The WIDE_TC_MARKS option in shorewall.conf selects which
    construction to use. WIDE_TC_MARKS=No (the default) produces
    pre-4.3.9 behavior. WIDE_TC_MARKS=Yes produces the new behavior.

    In addition to determining the method of constructing class Ids,
    WIDE_TC_MARKS=Yes provides for larger mark values for traffic
    shaping. Traffic shaping marks may have values up to 16383 (0x3fff)
    with WIDE_TC_MARKS=Yes. This means that when both WIDE_TC_MARKS=Yes and
    HIGH_ROUTE_MARKS=Yes, routing marks (/etc/shorewall/providers MARK
    column) must be >= 65536 (0x10000) and must be a multiple of 65536
    (0x1000, 0x20000, 0x30000, ...).

17) In the 'shorewall compile' command, the filename '-' is now causes
    the compiled script to be written to Standard Out. As a side
    effect, the effective VERBOSITY is set to -1 (silent).

    Examples:

	shorewall compile -v-1 -- -  # Compile the configuration in
                        	     # /etc/shorewall and send the
	                             # output to STDOUT
	shorewall compile -v-1 . -   # Compile the configuration in the
		  	  	     # current working directory
		  	    	     # and send the output to STDOUT

    Note that the '-v-1' suppresses the 'Compiling...' message normally
    issued by /sbin/shorewall (/sbin/shorewall6) when a compilation
    begins.

18) Supplying an interface name in the SOURCE column of
    /etc/shorewall/masq is now deprecated. Entering the name of an
    interface there will result in a compile-time warning.

19) Shorewall now supports nested HTB traffic shaping classes.  The
    nested classes within a class can borrow from their parent class in
    the same way as the first level classes can borrow from the root
    class.

    To use nested classes, you must explicitly number your
    classes. That does not imply that you must use the 'classify'
    option.

    Example:

    /etc/shorewall/tcdevices

    #INTERFACE	IN-BANDWITH	OUT-BANDWIDTH	OPTIONS
    eth2	-		100mbps		classify

    /etc/shorewall/tcclasses

    #INTERFACE	MARK	RATE		CEIL	 PRIORITY	OPTIONS
    1:10	-	full/2		full		1
    1:100	-	16mbit		20mbit		2
    1:100:101	-	 8mbit		20mbit		3	default
    1:100:102	-	 8mbit		20mbit		3
     
    /etc/shorewall/tcrules

    #MARK	SOURCE		DEST
    1:102	0.0.0.0/0	eth2:172.20.1.107
    1:10	206.124.146.177	eth2
    1:10	172.20.1.254	eth2

    The above controls download for internal interface eth2. The
    external interface has a download rate of 20mbit so we guarantee
    that to class 1:100. 1:100 has two subclasses, each of which is
    guaranteed half of their parent's bandwidth.

    Local traffic (that coming from the firewall and from the DMZ
    server) is placed in the effectively unrestricted class 1:10. The
    default class is guaranteed half of the download capacity and my
    work system (172.20.1.107) is guarandeed the other half.