Shorewall 3.2.0 RC 5

Note to users upgrading from Shorewall 2.x or 3.0

   Most problems associated with upgrades come from two causes:

   - The user didn't read and follow the migration considerations in these
     release notes.

   - The user mis-handled the /etc/shorewall/shorewall.conf file during
     upgrade. Shorewall is designed to allow the default behavior of
     the product to evolve over time. To make this possible, the design
     assumes that you will not replace your current shorewall.conf file
     during upgrades. If you feel absolutely compelled to have the latest
     comments and options in your shorewall.conf then you must proceed
     carefully.

   While you are at it, if you have a file named /etc/shorewall/rfc1918 then
   please check that file. If it has addresses listed that are NOT in one of
   these three ranges, then please rename the file to
   /etc/shorewall/rfc1918.old.

	10.0.0.0 - 10.255.255.255
	172.16.0.0 - 172.31.255.255
	192.168.0.0 - 192.168.255.255

   If you have a file named /etc/shorewall/modules, please remove
   it. The default modules file is now located in /usr/share/shorewall/
   (see the "Migration Considerations" below).

   Please see the "Migration Considerations" below for additional upgrade
   information.

Problems Corrected in 3.2.0 RC 5

1)  DOA /sbin/shorewall:

    wifitipper:/etc/test # shorewall restart
    Compiling...
	ERROR: LITEDIR not defined in /usr/share/shorewall/configpath
    wifitipper:

2)  The compiler was running iptables needlessly when DELAYBLACKLISTLOAD=Yes.
    This could also result in the blacklist not being loaded in some cases.

3)  Some versions of 'ash' generate an error message when a log level is
    included in a rule.

4)  Shorewall would use the incorrect shell for compilation in the
    following case:

    a) /etc/shorewall/shorewall.conf specified
    SHOREWALL_SHELL=/bin/shell1.

    b) /foo/bar/shorewall.conf specified SHOREWALL_SHELL=/bin/shell2.

    c) The command "shorewall try /foo/bar" was executed. Compilation
    of the configuration in /foo/bar would be done using /bin/shell2
    rather than /bin/shell1.

Other changes in 3.2.0 RC 5

None.

Migration Considerations:

1)  If you are upgrading from Shorewall 2.x, it is essential that you read
    the Shorewall 3.0.7 (or later) release notes:

    http://www.shorewall.net/pub/shorewall/3.0/shorewall-3.0.7/releasenotes.txt

2)  A number of macros have been split into two.  The macros affected are:

	IMAP LDAP NNTP POP3 SMTP

    Each of these macros now handles only traffic on the native (plaintext)
    port.  There is a corresponding macro with S added to the end of the
    name for the SSL version of the same protocol.  Thus each macro results
    in the insertion of only one port per invocation.

    The Web macro has not been split, but two new macros, HTTP and HTTPS have
    been created.  The Web macro is deprecated in favour of these new macros,
    and may be removed from future Shorewall releases.

    These changes have been made to ensure no unexpected ports are opened due
    to the use of macros.

3)  In previous Shorewall releases, DNAT and REDIRECT rules supported a
    special syntax for exclusion of a sub-zone from the effect of the rule.

    Example:

	Z2 is a subzone of Z1:

	DNAT     Z1!Z2        loc:192.168.1.4        ...

    That feature has never worked correctly when Z2 is a dynamic zone.
    Furthermore, now that Shorewall supports exclusion lists, the capability
    is redundant since the above rule can now be written in the form:

	DNAT     Z1:!<list of exclusions>   loc:192.168.1.4   ...

    Beginning with Shorewall 3.2.0, the special exclusion syntax will no
    longer be supported.

4)  Important if you use the QUEUE target.

    In the /etc/shorewall/rules file and in actions, you may now specify
    'tcp:syn' in the PROTO column. 'tcp:syn' is equivalent to 'tcp' but also
    requires that the SYN flag is set and the RST, FIN and ACK flags be
    off ("--syn" is added to the iptables rule).

    As part of this change, Shorewall no longer adds the "--syn" option
    to TCP rules that specify QUEUE as their target.

5)  Extension Scripts may require change

    In previous releases, extension scripts were executed during [re]start
    by using the Bourne Shell "." operator. In addition to executing commands
    during [re]start, these scripts had to "save" the commands to be executed
    during "shorewall restore".

    This clumsiness has been eliminated in Shorewall 3.2. In Shorewall 3.2,
    extension scripts are copied in-line into the compiled program and are
    executed in-line during "start", "restart" and "restore".

    This new approach has two implications for existing scripts.

    a)  It is no longer necessary to save the commands; so functions like
        'save_command', 'run_and_save_command' and 'ensure_and_save_command'
        need no longer be called. For convenience, the generated program will
        supply functions with these names:

	    save_command() - does nothing
	    run_and_save_command() - runs the passed command
	    ensure_and_save_command() - runs the passed command and
	                                stops/restores the firewall if the
	                                command fails.

	These functions should provide for transparent migration of
	scripts that use them until you can get around to eliminating
	their use completely.

    b)  When the extension script is copied into the compiled program, it
        is indented to line up with the surrounding code. If you have 'awk'
        installed on your system, the Shorewall compiler will correctly handle
        line continuation (last character on the line = "\"). If you do not
        have awk, it will not be possible to use line-continuation in your
        extension scripts.

        In no case is it possible to continue a quoted string over multiple lines
        without having additional whitespace inserted into the string.

6)  Beginning with this release, the way in which packet marking in the
    PREROUTING chain interracts with the 'track' option in /etc/shorewall/providers
    has changed in two ways:

    a)  Packets arriving on a tracked interface are now passed to the PREROUTING
	marking chain so that they may be marked with a mark other than the
	'track' mark (the connection still retains the 'track' mark).

    b)  When HIGH_ROUTE_MARKS=Yes, you can still clear the mark on packets
	in the PREROUTING chain (i.e., you can specify a mark value of zero).

7)  Kernel version 2.6.16 introduces 'xtables', a new common packet
    filtering and connection tracking facility that supports both IPv4
    and IPv6. Because a different set of kernel modules must be loaded
    for xtables, Shorewall now includes two 'modules' files:

    a) /usr/share/shorewall/modules -- the former
    /etc/shorewall/modules

    b) /usr/share/shorewall/xmodules -- a new file that support
    xtables.

    If you wish to use the new file, then simply execute this command:

    cp -f /usr/share/shorewall/xmodules /etc/shorewall/modules

New Features:

1)  Shorewall has always been very noisy (lots of messages). No longer.

    You set the default level of verbosity using the VERBOSITY option in
    shorewall.conf. If you don't set it (as would be the case of you use your
    old shorewall.conf file) then VERBOSITY defaults to a value of 2 which
    results in behavior compatible with previous Shorewall versions.
    A value of 1 suppresses some of the output (like the old -q option did)
    while a value of 0 makes Shorewall almost silent. A value of -1
    suppresses all output except warning and error messages.

    The value specified in the 3.2 shorewall.conf is 1. So you can make
    Shorewall as verbose as previously using a single -v and you can make it
    silent by using a single -q.

    If the default is set at 2, you can still make a command silent by using
    two "q"s (e.g., shorewall -qq restart).

    In summary, each "q" subtracts one from VERBOSITY while each "v" adds one
    to VERBOSITY.

    The "shorewall show log", "shorewall logwatch" and "shorewall dump"
    commands require VERBOSITY to be greater than or equal to 3 to display MAC
    addresses.This is consistent with the previous implementation which
    required a single -v to enable MAC display but means that if you set
    VERBOSITY=0 in shorewall.conf, then you will need to include -vvv in
    commands that display log records in order to have MACs displayed.

    To make the display of MAC addresses let cumbersome, a '-m' option has
    been added to the "show" and logwatch commands:

	shorewall show -m log
	shorewall logwatch -m

2)  A new 'shorewall compile' command has been added.

     shorewall compile [ -e ] [ <config directory> ] <script file>

    where:

	-e                    Allows the generated script to run
	                      on a system with Shorewall Lite installed.
			      Generates an error if the configuration uses
	                      an option that would prevent the generated
	                      script from running on a system other than
	                      where the 'compile' command is running (see
	                      additional consideration a) below).

	<config directory>    Is an optional directory to be searched for
	                      configuration files prior to those listed
	                      in CONFIG_DIR in
			      /etc/shorewall/shorewall.conf.
	<script file>         Is the name of the output file.

    The 'compile' command processes the configuration and generates a
    script file which may then be executed (either directly or using the
    'shorewall restore' command) to configure the firewall.

    The generated script contains error checking and will terminate if an
    important command fails. Before terminating:

    a) The script will check for the existence of the restore script
       specified by the RESTOREFILE variable in shorewall.conf. If that
       restore script exists, it is executed.

    b) If the restore script doesn't exist but Shorewall appears to be
       installed on the system, the equivalent of an
       "/sbin/shorewall stop" command is executed.

    Some additional considerations:

    a) When you run 'compile' on one system and then run the generated script
       on another system under Shorewall Lite, there are certain limitations.

       1) A compatible version of Shorewall Lite must be running on the remote
          system.
       2) The 'detectnets' interface option is not allowed.
       3) DYNAMIC_ZONES=Yes is not allowed.
       4) You must supply the file /etc/shorewall/capabilities to provide
          the compiler with knowledge of the capabilities of the system
          where the script is to be run. See below.
       5) If your /etc/shorewall/params file contains code other than simple
          assignment statements with contant values, then you should move
          that code to /etc/shorewall/init. That way, the code will be
          executed on the target system when the compiled script is run and
          not on the local system at compile time.

    b) If you run the "shorewall compile" or "shorewall check" commands under
       a user other than 'root', then you must supply
       /etc/shorewall/capabilities.

    c) To aid in building /etc/shorewall/capabilities, a 'shorecap' program
       is provided in the Shorewall Lite package and is installed in
       /usr/share/shorewall-lite/shorecap when you install Shorewall Lite.

       For instructions about running shorecap, see the comments at the
       top of the program file (it's a simple shell script).

    The "shorewall start" and "shorewall restart" commands have been
    rewritten to use compilation. They both compile a temporary program
    then run it. This results in a slightly longer elapsed time than the
    similar commands required under earlier versions of Shorewall but new
    connections are blocked for a much smaller percentage of that time.

    If an error is found during the compilation phase, /sbin/shorewall
    terminates and the Shorewall state is unchanged.

    Under Shorewall 3.1.5, "shorewall restart" takes roughly 16.5 seconds
    on my firewall:

	real    0m16.599s
	user    0m6.292s
	sys     0m9.885s

    Of the elapsed 16.5 seconds, new connections are disabled less than
    3.5 seconds. Here are some numbers for comparison:

    A) shorewall restart (Shorewall 3.0.4)

	real    0m17.540s
	user    0m5.956s
	sys     0m10.737s

    B) ./foo restart # foo created using "shorewall compile"

	real    0m3.297s
	user    0m1.444s
	sys     0m1.728s

    C) shorewall restore (Shorewall 3.0.4) # Restores from file generated by
                                           # "shorewall save"

	real    0m1.164s
	user    0m0.556s
	sys     0m0.608s

    D) shorewall restore (shorewall 3.1.5)

	real    0m1.637s
	user    0m0.728s
	sys     0m0.584s

    The time difference between B and C reflects the difference between
    "iptables-restore" and multiple executions of "iptables". The time
    difference between C and D results from the fact that the "restore"
    command in Shorewall 3.1 runs the compiled program in a way that
    turns all iptables commands into no-ops then invokes
    iptables-restore. The system is a 1.4Ghz Celeron with 512MB RAM.

    As a final part of this change, the "check" command now compiles the
    current configuration and writes the compiled output to /dev/null. So
    "check" performs all of the same validation that compile does. Note that
    there is still no guarantee that the generated script won't encounter
    run-time errors.

2)  The /etc/shorewall/maclist file has a new column layout. The first column
    is now DISPOSITION. This column determines what to do with matching
    packets and can have the value ACCEPT or DROP (if MACLIST_TABLE=filter, it
    can also contain REJECT). This change is upward compatible so your existing
    maclist file can still be used.

    ACCEPT, DROP and REJECT may be optionally followed by a log level to
    cause the packet to be logged.

4)  In macro files, you can now use the reserved words SOURCE and DEST
    in the columns of the same names. When Shorewall expands the
    macro, it will substitute the SOURCE from the macro invocation for
    SOURCE and the DEST from the invocation for DEST. This allows you2)  "load" and "reload" commands have been added. These commands allow
    a non-root user with ssh access to a remote system to compile a
    firewall script on the local system and to install that script on
    the remote system.

    Syntax is:

	   shorewall [re]load [ <directory> ] <system>

    If <directory> is omitted, the current working directory is
    assumed.

    The command is equivalent to:

    /sbin/shorewall compile -e . firewall &&\
    scp firewall root@<system>:/var/lib/shorewall-lite/ &&\
    ssh root@<system> '/sbin/shorewall-lite [re]start' # Note 1

    In other words, the configuration in the specified (or defaulted)
    directory is compiled to a file called firewall in that
    directory. If compilation succeeds, then 'firewall' is copied to the
    (usually remote) <system> using scp. If the copy succeeds,
    Shorewall Lite on <system> is started or restarted via ssh (
    load causes Shorewall Lite to be started and 'reload' causes
    Shorewall Lite to be re-started)

    Note 1: In Shorewall Lite 3.2.0 RC4, the 'firewall' script has moved
    from /usr/share/shorewall-lite/ to /var/lib/shorewall-lite in
    packages from shorewall.net. The package maintainers for the
    various distributions are free to choose the directory where the
    script will be stored under their distribution by altering the
    value of LITEDIR in /usr/share/shorewall/configpath. You can run the
    "shorewall show config" command to see how your distribution
    defines LITEDIR.


    to write macros that act in both directions (from source to destination
    and from destination to source).

    Example:

	macro.FOO:

	    PARAM      SOURCE        DEST      udp     500
	    PARAM      DEST          SOURCE    udp     500

	/etc/shorewall/rules:

	    FOO/ACCEPT    fw         net

	Resulting rules:

	    ACCEPT        fw         net        udp     500
	    ACCEPT        net        fw         udp     500

    This new feature has been used to implement the SMBBI macro.
    SMBBI is the same as the SMB macro with the exception that
    it passes SMB traffic in both directions whereas SMB only
    passes that traffic in one direction.

5)  In the /etc/shorewall/rules file and in actions, you may now specify
    'tcp:syn' in the PROTO column. 'tcp:syn' is equivalent to 'tcp' but also
    requires that the SYN flag is set and the RST, FIN and ACK flags be
    off ("--syn" is added to the iptables rule).

    As part of this change, Shorewall no longer adds the "--syn" option
    to TCP rules that specify QUEUE as their target.

6)  /sbin/shorewall now supports a "-t" option that causes all progress
    messages to be timestamped.

    Example (VERBOSITY=0 in shorewall.conf):

    gateway:/etc/shorewall # shorewall -t restart
    07:08:51 Compiling...
    07:09:05 Shorewall configuration compiled to /var/lib/shorewall/.restart
    07:09:05 Restarting Shorewall....
    07:09:08 done.
    gateway:/etc/shorewall #

7)  A 'refreshed' extension script has been added -- it is executed after
    "shorewall refresh" has finished.

8)  Two new dynamic blacklisting commands have been added:

	logdrop -- like 'drop' but causes the dropped packets to be logged.

	logreject -- like 'reject' but causes the rejected packets to be
	             logged.

    Packets are logged at the BLACKLIST_LOGLEVEL if one was specified at the
    last "shorewall [re]start"; otherwise, they are logged at the 'info'
    log level.

9)  A new IMPLICIT_CONTINUE option has been added to shorewall.conf. When
    this option is set to "Yes", it causes subzones to be treated differently
    with respect to policies.

    Subzones are defined by following their name with ":" and a list of parent
    zones (in /etc/shorewall/zones). Normally, you want to have a set of
    special rules for the subzone and if a connection doesn't match any of
    those subzone-specific rules then you want the parent zone rules and
    policies to be applied. With IMPLICIT_CONTINUE=Yes, that happens
    automatically.

    If IMPLICIT_CONTINUE=No or if IMPLICIT_CONTINUE is not set, then
    subzones are not subject to this special treatment.

    With IMPLICIT_CONTINUE=Yes, an implicit CONTINUE policy may be overridden
    by including an explicit policy (one that does not specify "all" in either
    the SOURCE or the DEST columns).

    Example:

    /etc/shorewall/zones:

	prnt       ipv4
	chld:prnt  ipv4

    Traffic to/from the 'chld' zone will first pass through the applicable
    'chld' rules and if none of those rules match then it will be passed through
    the appropriate 'prnt' rules. If the connection request does not match
    any of the 'prnt' rules then the relevant 'prnt' policy is applied.

    If you want the fw->chld policy to be ACCEPT, simply add this entry to
    /etc/shorewall/policy:

	$FW      chld        ACCEPT

    Traffic from all other zones to 'chld' will be subject to the implicit
    CONTINUE policy.

10) Shorewall now includes support for explicit routing rules when the
    /etc/shorewall/providers file is used. A new file,
    /etc/shorewall/route_rules can be used to add routing rules based on
    packet source and/or destination.

    The file has the following columns:

		SOURCE(optonal) An ip address (network or host) that
				matches the source IP address in a packet.
				May also be specified as an interface
				name optionally followed by ":" and an
				address. If the define 'lo' is specified,
				the packet must originate from the firewall
				itself.

		DEST(optional)  An ip address (network or host) that
				matches the destination IP address in a packet.

				If you choose to omit either SOURCE or DEST,
				place "-" in the column. Note that you
				may not omit both SOURCE and DEST.

		PROVIDER        The provider to route the traffic through.
				May be expressed either as the provider name
				or the provider number. You may also specify
				the 'main' routing table here, either by
				name or by number (254).

		PRIORITY
				The rule's priority which determines the order
				in which the rules are processed.

				1000-1999   Before Shorewall-generated
					    'MARK' rules

				11000- 11999 After 'MARK' rules but before
					    Shorewall-generated rules for
					    provider interfaces.

				26000-26999 After provider interface rules but
					    before 'default' rule.

				Rules with equal priority are applied in
				the order in which they appear in the file.

    Example 1: You want all traffic coming in on eth1 to be routed to the ISP1
	     provider:

	    #PROVIDER	PRIORITY	SOURCE			DEST
	    ISP1	1000		eth1

    Example 2: You use OpenVPN (routed setup /tunX) in combination with multiple
	       providers. In this case you have to set up a rule to ensure that
	       the OpenVPN traffic is routed back through the tunX interface(s)
	       rather than through any of the providers. 10.8.0.0/24 is the
	       subnet choosen in your OpenVPN configuration (server 10.8.0.0
	       255.255.255.0)

	#SOURCE			DEST		PROVIDER	PRIORITY
	-	    		10.8.0.0/24	main		1000

11) Prior to now, it has not been possible to use connection marking in
    /etc/shorewall/tcrules if you have a multi-ISP configuration that uses the
    'track' option.

    Beginning with this release, you may now set HIGH_ROUTE_MARKS=Yes in
    shorewall.conf to effectively divide the packet mark and connection mark
    into two 8-byte mark fields.

    When you do this:

	a) The MARK field in the providers file must have a value that is
	   less than 65536 and that is a multiple of 256 (using hex
	   representation, the values are 0x0100-0xFF00 with the low-order
	   8 bits being zero).

	b) You may only set those mark values in the PREROUTING chain.

	c) Marks used for traffic shaping must still be in the range of 1-255
	   and may still not be set in the PREROUTING chain.

	d) When you SAVE or RESTORE in tcrules, only the TC mark value is
	   saved or restored. Shorewall handles saving and restoring the
	   routing (provider) marks.

12) A TOS column has been added to /etc/shorewall/tcrules. This allows marking
    based on the contents of the TOS field in the packet header.

13) Beginning with this release, the way in which packet marking in the
    PREROUTING chain interracts with the 'track' option in /etc/shorewall/providers
    has changed in two ways:

    a)  Packets *arriving* on a tracked interface are now passed to the PREROUTING
	marking chain so that they may be marked with a mark other than the
	'track' mark (the connection still retains the 'track' mark).

    b)  When HIGH_ROUTE_MARKS=Yes, you can still clear the mark on packets
	in the PREROUTING chain (i.e., you can specify a mark value of zero).

14) Shorewall will now attempt to detect the MTU of devices listed in
    /etc/shorewall/tcdevices and will use the detected MTU in setting
    up traffic shaping.

15) In /etc/shorewall/rules, the values "all-" and "all+-" may now be
    used for zone names. "all-" means "All zones except the firewall";
    "all+-" means "All zones except the firewall" and intra-zone
    traffic is included.

16) Kernel version 2.6.16 introduces 'xtables', a new common packet
    filtering and connection tracking facility that supports both IPv4
    and IPv6. Because a different set of kernel modules must be loaded
    for xtables, Shorewall now includes two 'modules' files:

    a) /usr/share/shorewall/modules -- the former
    /etc/shorewall/modules

    b) /usr/share/shorewall/xmodules -- a new file that support
    xtables.

    If you wish to use the new file, then simply execute this command:

    cp -f /usr/share/shorewall/xmodules /etc/shorewall/modules

17) Shorewall now checks to see if devices in /etc/shorewall/tcdevices
    exist. If a device does not exist, a warning message is issued and
    that device's entries in /etc/shorewall/tcclasses are ignored. This
    applies to "shorewall start", "shorewall restart" and "shorewall
    refresh".

18) "load" and "reload" commands have been added. These commands allow
    a non-root user with ssh access to a remote system running
    Shorewall Lite to compile a firewall script on the local system and
    to install that script on the remote system.

    Syntax is:

	   shorewall [re]load [ <directory> ] <system>

    If <directory> is omitted, the current working directory is
    assumed.

    The command is equivalent to:

    /sbin/shorewall compile -e . firewall &&\
    scp firewall root@<system>:/var/lib/shorewall-lite/ &&\
    ssh root@<system> '/sbin/shorewall-lite [re]start' # Note 1

    In other words, the configuration in the specified (or defaulted)
    directory is compiled to a file called firewall in that
    directory. If compilation succeeds, then 'firewall' is copied to the
    (usually remote) <system> using scp. If the copy succeeds,
    Shorewall Lite on <system> is started or restarted via ssh (
    load causes Shorewall Lite to be started and 'reload' causes
    Shorewall Lite to be re-started)

    Note 1: In Shorewall Lite 3.2.0 RC4, the 'firewall' script has moved
    from /usr/share/shorewall-lite/ to /var/lib/shorewall-lite in
    packages from shorewall.net. The package maintainers for the
    various distributions are free to choose the directory where the
    script will be stored under their distribution by altering the
    value of LITEDIR in /usr/share/shorewall/configpath. You can run the
    "shorewall show config" command to see how your distribution
    defines LITEDIR.