shorewall_code/Shorewall-common/releasenotes.txt

Shorewall 4.0.0 Beta 7
----------------------------------------------------------------------------
                   R E L E A S E  H I G H L I G H T S
----------------------------------------------------------------------------
1) This is the first Shorewall release that fully integrates the new
   Shorewall-perl compiler. See the "New Features" section below.

2) You are now offered a choice as to which compiler(s) you install. In
   4.0.0, there are the following packages:

   - Shorewall-common ( common files )
   - Shorewall-shell ( the shell-based compiler )
   - Shorewall-perl (the Perl-based compiler )

You must install Shorewall-common and at least one of the compiler
packages (you may install them both).

3) The facilities for supporting bridge/firewalls under earlier
   releases are deprecated and their documentation is omitted from the
   4.0 distribution. New bridge support is implemented in the
   Shorewall-perl compiler. This support utilizes the reduced-function
   physdev match support available in Linux kernel 2.6.20 and later.

Problems corrected in 4.0.0 Beta 7.

1)  Installation of Shorewall-perl when Shorewall is not yet installs
    no longer suffers a failure in the generation of
    /usr/share/shorewall-perl/Shorewall/Ports.pm.

Other changes in Shorewall 4.0.0 Beta 7

1)  When an /sbin/shorewall command that begins with 'debug' or 'trace'
    invokes the Shorewall-perl compiler, the compiler will include
    additional debugging information in its warning and error
    messages. This additional information is intended to help the
    people supporting Shorewall to diagnose the cause of the message.

2)  The script generated by Shorewall-perl now assumes that
    iptables-restore and iptables-save are in the same directory as the
    program specified in the IPTABLES setting in Shorewall-conf.

    If IPTABLES is not specified, then the iptables utility is located
    using the PATH setting and the iptables-restore and iptables-save
    programs from the same directory are used.

3)  COMMENTs are now allowed in the accounting file under
    Shorewall-perl

Migration Considerations:

1)  You cannot simply upgrade your existing Shorewall package. You must
    upgrade Shorewall *and* install one or both of the compilers.

    If you attempt to upgrade using the RPM, you get this result:

    gateway:~ # rpm -Uvh shorewall-common-4.0.0.noarch.rpm 
    error: Failed dependencies:
        shorewall_compiler is needed by shorewall-common-4.0.0-1.noarch
    gateway:~ #

    You must either:

	rpm -U shorewall-common-4.0.0.noarch.rpm \
	    shorewall-shell-4.0.0.noarch.rpm

    or

	rpm -U shorewall-common-4.0.0.noarch.rpm \
	    shorewall-perl-4.0.0.noarch.rpm
	
    or

	rpm -U shorewall-common-4.0.0.noarch.rpm \
	    shorewall-perl-4.0.0.noarch.rpm \
	    shorewall-shell-4.0.0.noarch.rpm
 
    If you are upgrading using the tarball, you must install 
    shorewall-shell and/or shorewall-perl before you upgrade
    Shorewall. Otherwise, the install.sh script fails with:

         ERROR: No Shorewall compiler is installed

    The shorewall-shell and shorewall-perl packages are installed from
    the tarball in the expected way; untar the package, and run the
    install.sh script.

2)  The ROUTE_FILTER and LOG_MARTIANS options in shorewall.conf work
    slightly differently in Shorewall 4.0.0. In prior releases, leaving
    these options empty was equivalent to setting them to 'No' which
    caused the corresponding flag in /proc to be reset for all
    interfaces. Beginning in Shorewall 4.0.0, leaving these options
    empty causes Shorewall to leave the flags in /proc as they are. You
    must set the option to 'No' in order to obtain the old behavior. 

----------------------------------------------------------------------------
                         N E W  F E A T U R E S
----------------------------------------------------------------------------
1)  Shorewall-perl

    This companion product to Shorewall 3.4.2 and later includes a complete
    rewrite of the compiler in Perl.

    I decided to make Shorewall-perl a separate product for several reasons:

    a) Embedded applications are unlikely to adopt Shorewall-perl; even
       Mini-Perl has a substantial disk and RAM footprint.

    b) Because of the gross incompatibilities between the new compiler and the
       old (see below), migration to the new compiler must be voluntary.
    ------------------------------------------------------------------------
                      T H E   G O O D   N E W S:
    ------------------------------------------------------------------------
    a) The compiler has a small disk footprint.
    b) The compiler is very fast.
    c) The compiler generates a firewall script that uses iptables-restore;
       so the script is very fast.
    d) Use of the perl compiler is optional! The old slow clunky
       Bourne-shell compiler is still available.
    ------------------------------------------------------------------------
                      T H E   B A D   N E W S:
    ------------------------------------------------------------------------
    There are a number of incompatibilities between the Perl-based compiler
    and the Bourne-shell one. Some of these will probably go away by first
    official release but most will not.

    a) The Perl-based compiler requires the following capabilities in your
       kernel and iptables.

       - addrtype match (may be relaxed later)
       - multiport match (will not be relaxed)

       These capabilities are in current distributions.

    b) Now that Netfilter has features to deal reasonably with port lists,
       I see no reason to duplicate those features in Shorewall. The
       Bourne-shell compiler goes to great pain (in some cases) to
       break very long port lists ( > 15 where port ranges in lists
       count as two ports) into individual rules. In the new compiler, I'm
       avoiding the ugliness required to do that. The new compiler just
       generates an error if your list is too long. It will also produce
       an error if you insert a port range into a port list and you don't
       have extended multiport support.

    c) The old BRIDGING=Yes support has been replaced by new bridge
       support that uses the reduced 'physdev match' capabilities found
       in kernel 2.6.20 and later. This new implementation may be used
       where it is desired to control traffic through a bridge.

       The new implementation includes the following features:

       a)  A new "Bridge Port" zone type is defined. Specify 'bport' or
	   'bport4' in the TYPE column of /etc/shorewall/zones.

	   Bridge Port zones should be a sub-zone of a regular ipv4 zone
	   that represents all hosts attached to the bridge.

       b)  A new 'bridge' option is defined for entries in	
	   /etc/shorewall/interfaces. Bridges should have this option
	   specified.

       c)  Bridge ports must now be defined in
	   /etc/shorewall/interfaces. The INTERFACE column contains
	   both the bridge name and the port name separated by a colon
	   (e.g., "br0:eth1"). No OPTIONS are allowed for bridge
	   ports. The bridge must be defined before its ports and must
	   have the 'bridge' option.

       Bridge Port (BP) zones have a number of limitations:

       a)  Each BP zone may only be associated with ports on a single
	   bridge.

       b)  BP zones may not be associated with interfaces that are not
	   bridge ports.

       c)  You may not have policies or rules where the DEST is a BP
	   zone but the source is not a BP zone. If you need such
	   rules, you must use the BP zone's parent zone as the DEST.

       Example (Bridge br0 with ports eth1 and tap0):

       /etc/shorewall/zones:

	fw	firewall
	net	ipv4
	loc	ipv4
	lan:loc	bport
	vpn:loc bport

       /etc/shorewall/interfaces:

	net	eth0	-	...
	loc	br0	-	...
	lan	eth1
	vpn	tap0

       When using the /etc/shorewall/hosts file to define a bport4
       zone, you specify only the port name:

       Example:

       /etc/shorewall/zones:

	fw	firewall
	net	ipv4
	loc	ipv4
	lan:loc	bport
	vpn:loc bport

       /etc/shorewall/hosts

	lan	eth1:192.168.2.0/24	...

       The structure of the accounting rules changes slightly when
       there are bridges defined in the Shorewall
       configuration. Because of the restrictions imposed by Netfilter
       in kernel 2.6.21 and later, output accounting rules must be
       segregated from forwarding and input rules.

       To accomplish this separation, Shorewall-perl creates two
       accounting chains:

       - accounting - for input and forwarded traffic.
       - accountout - for output traffic.

       If the CHAIN column contains '-', then:

       - If the SOURCE column in a rule includes the name of the
         firewall zone (e.g., $FW), then the default chain to insert
         the rule into is accountout only.

       - Otherwise, if the DEST in the rule is any or all or 0.0.0.0/0,
	 then the rule is added to both accounting and accountout.

       - Otherwise, the rule is added to accounting only.	

    d) The BROADCAST column in the interfaces file is essentially unused;
       if you enter anything in this column but '-' or 'detect', you will
       receive a warning. This will be relaxed if and when the addrtype
       match requirement is relaxed.

    e) Because the compiler is now written in Perl, your compile-time
       extension scripts from earlier versions will no longer work.
       Compile-time extension scripts are executed using the Perl
       'eval `cat <file>`' mechanism. Be sure that each script returns a
       'true' value; otherwise, the compiler will assume that the script
       failed and will abort the compilation.

       All scripts will need to begin with the following line:

	   use Shorewall::Chains;

       For more complex scripts, you may need to 'use' other Shorewall
       Perl modules -- browse /usr/share/shorewall-perl/Shorewall/ to
       see what's available.

       When a script is invoked, the $chainref scalar variable will hold a
       reference to a chain table entry. 

           $chainref->{name}  contains the name of the chain
           $chainref->{table} holds the table name
       
           To add a rule to the chain:

	      add_rule( $chainref, <the rule> );

           Where

              <the rule> is a scalar argument holding the rule text. Do
              not include "-A <chain name>"

           Example:

              add_rule( $chainref, '-j ACCEPT' );

           To insert a rule into the chain:

              insert_rule( $chainref, <rulenum>, <the rule> );

           The log_rule_limit function works like it does in the shell
           compiler with two exceptions:

              - You pass the chain reference rather than the name of
                the chain.
	      - The commands are 'add' and 'insert' rather than '-A'
                and '-I'.
	      - There is only a single "pass as-is to iptables"
                argument (so you must quote that part).

           Example:

	      log_rule_limit(
	      	             'info' , 
			      $chainref ,	     
			      $chainref->{name},
			      'DROP' , 
			      '',      #Limit
			      '' ,     #Log tag
			      'add',   #Command
			      '-p tcp' #Pass as-is
                             );

    f) The 'refresh' command now works like 'restart' with the
       following exceptions:

       - The refresh command is rejected if Shorewall is not running.
       - A directory name may not be specified in the refresh command.

    g) Some run-time scripts have been converted to compile time
       scripts:

            initdone
	    maclog

       Note that in the 'initdone' script, there is no default chain
       ($chainref). You can objtain a reference to a standard chain by:

           my $chainref = $chain_table{<table>}{<chain name>};

       Example:

           my $chainref = $chain_table{'filter'}{'INPUT'};

       Some run-time scripts are simply eliminated because they no
       longer make any sense under Shorewall-perl:

	continue  - This script was designed to allow you to add
	            special rules during [re]start.
	            Shorewall-perl doesn't need such rules.
	refresh   - The 'refresh' command is the same as 'restart'
	refreshed

    h) The /etc/shorewall/tos file now has zone-independent SOURCE and
       DEST columns as do all other files except the rules and policy
       files.

       The SOURCE column may be one of the following:

           [all:]<address>[,...]
           [all:]<interface>[:<address>[,...]]
           $FW[:<address>[,...]]

       The DEST column may be one of the following:
       
           [all:]<address>[,...]
           [all:]<interface>[:<address>[,...]]

       This is a permanent change. The old zone-based rules have never
       worked right and this is a good time to replace them. I've tried
       to make the new syntax cover the most common cases without
       requiring change to existing files. In particular, it will
       handle the tos file released with Shorewall 1.4 and earlier.

    i) Shorewall is now out of the ipset load/reload business. With
       scripts generated by the Perl-based Compiler, the Netfilter
       ruleset is never cleared. That means that there is no
       opportunity for Shorewall to load/reload your ipsets since that
       cannot be done while there are any current rules using ipsets.

       So:

	i)   Your ipsets must be loaded before Shorewall starts. You
	     are free to try to do that with the following code in
	     /etc/shorewall/start:

	     if [ "$COMMAND" = start ]; then
		ipset -U :all: :all:
		ipset -F
		ipset -X
		ipset -R < /my/ipset/contents
	     fi

	     The file '/my/ipset/contents' (not its real name of
	     course) will normally be produced using the ipset -S
	     command.

	     The above will work most of the time but will fail in a
	     'shorewall stop' - 'shorewall start' sequence if you
	     use ipsets in your routestopped file (see below).

	ii)  Your ipsets may not be reloaded until Shorewall is stopped
	     or cleared.

	iii) If you specify ipsets in your routestopped file then
	     Shorewall must be cleared in order to reload your ipsets.

       As a consequence, scripts generated by the Perl-based compiler
       will ignore /etc/shorewall/ipsets and will issue a warning if
       you set SAVE_IPSETS=Yes in shorewall.conf.

    j) Because the configuration files (with the exception of
       /etc/shorewall/params) are now processed by the Perl-based
       compiler rather than by the shell, only the basic forms of Shell
       expansion ($variable and ${variable}) are supported. The more
       exotic forms such as ${variable:=default} are not
       supported. Both variables defined in /etc/shorewall/params and
       environmental variables (exported by the shell) can be used in
       configuration files.

    h) USE_ACTIONS=No is not supported. That option is intended to
       minimize Shorewall's footprint in embedded applications. As a
       consequence, Default Macros are not supported.

    i) DELAYBLACKLISTLOAD=Yes is not supported. The entire ruleset is
       atomically loaded with one execution of iptables-restore.

    j) MAPOLDACTIONS=Yes is not supported. People should have converted
    to using macros by now.

    k) The pre Shorewall-3.0 format of the zones file is not supported;
       neither is the /etc/shorewall/ipsec file.

    l) BLACKLISTNEWONLY=No is not permitted with FASTACCEPT=Yes. This
       combination doesn't work in previous versions of Shorewall so
       the Perl-based compiler simply rejects it.

    m) Shorewall-perl has a single rule generator that is used for all
       rule-oriented files. So it is important that the syntax is
       consistent between files.
       
       With shorewall-shell, there is a special syntax in the SOURCE
       column of /etc/shorewall/masq to designate "all traffic entering
       the firewall on this interface except...".

       Example:

       #INTERFACE	SOURCE			ADDRESSES
       eth0		eth1!192.168.4.9	...

       Shorewall-perl uses syntax that is consistent with the rest of
       Shorewall:

       #INTERFACE	SOURCE			ADDRESSES
       eth0		eth1:!192.168.4.9	...

    n) The 'allowoutUPnP' built-in action is no longer supported. The
       Netfilter team have removed support for '-m owner --owner-cmd'
       which that action depended on. 

    o) The treatment of the following interface options has changed under
       Shorewall-perl.

       - arp_filter
       - routefilter
       - logmartians
       - proxy_arp
       - sourceroute

       With the Shorewall-shell compiler, Shorewall resets these options
       on all interfaces then sets the option on those interfaces
       for which the option is defined in /etc/shorewall/interfaces.

       Under Shorewall-perl, these options can be specified with the value
       0 or 1 (e.g., proxy_arp=0). If no value is specified, the value 1
       is assumed. Shorewall will modify only the setting of those
       interfaces for which the option is specified and will set the
       option to the given value.

       A fatal compilation error is also generated if you specify one of
       these options with a wildcard interface (one ending with '+').

    p) The LOG_MARTIANS and ROUTE_FILTER options are now tri-valued in
       Shorewall-perl.

	Yes -  Same as before
	No  -  Same as before except that it applies regardless of
	       whether any interfaces have the logmartians/routefilter
	       option
	Keep - Shorewall ignores the option entirely.

2)  An 'optional' option has been added to
    /etc/shorewall/interfaces. This option is recognized by
    Shorewall-perl but not by Shorewall-shell. When 'optional' is
    specified for an interface, Shorewall will be silent when:

    - a /proc/sys/net/ipv4/conf/ entry for the interface cannot be
      modified (including for proxy ARP).

    - The first address of the interface cannot be obtained.

    I specify 'optional' on interfaces to Xen virtual machines that may
    or may not be running when Shorewall is [re]started.

    CAUTION: Use 'optional' at your own risk. If you [re]start
    Shorewall when an 'optional' interface is not available and then do
    a 'shorewall save', subsequent 'shorewall restore' and 'shorewall -f
    start' operations will instantiate a ruleset that does not support
    that interface, even if it is available at the time of the
    restore/start.

3)  Thanks to Paul Gear, an IPPServer macro has been added. Be sure to
    read the comments in the macro file before trying to use this
    macro.

4)  Eariler generations of Shorewall Lite required that remote root
    login via ssh be enabled in order to use the 'load' and 'reload'
    commands.

    Beginning with this release, you may define an alternative means
    for accessing the remote firewall system.

    Two new options have been added to shorewall.conf:

	RSH_COMMAND
	RCP_COMMAND

    The default values for these are as follows:

	RSH_COMMAND: ssh ${root}@${system} ${command}
	RCP_COMMAND: scp ${files} ${root}@${system}:${destination}

    Shell variables that will be set when the commands are envoked are
    as follows:

       root  - root user. Normally 'root' but may be overridden using
               the '-r' option.

       system - The name/IP address of the remote firewall system.

       command - For RSH_COMMAND, the command to be executed on the 
                 firewall system.

       files   - For RCP_COMMAND, a space-separated list of files to
                 be copied to the remote firewall system.

       destination - The directory on the remote system that the files 
                     are to be copied into. 

5)  The accounting, masq, rules and tos files now have a 'MARK' column
    similar to the column of the same name in the tcrules file. This
    column allows filtering by MARK and CONNMARK value (CONNMARK is
    only accepted under Shorewall Perl).

6)  SOURCE and DEST are now reserved zone names to avoid problems with
    bi-directional macro definitions which use these as names as key
    words.

7)  Shorewall-perl validates all IP addresses and addresses ranges
    in rules. DNS names are resolved and an error is issued for any
    name that cannot be resolved.

8)  Shorewall-perl checks configuration files for the presense of
    characters that can cause problems if they are allowed into the
    generated firewall script:

    -	Double Quotes. These are prohibited except in the
        shorewall.conf and params files.

    -	Single Quotes. These are prohibited except in the
        shorewall.conf and params files and in COMMENT lines.

    -   Single back quotes. These are prohibited except in the 
	shorewall.conf and params files.

    -   Backslash. Probibited except as the last character on a line to
        denote line continuation.

9)  Under Shorewall-perl, macros may invoke other macros with the
    restriction that such macros may not be invoked within an action
    body.

    When marcros are invoked recursively, the parameter passed to an
    invocation are automatically propagated to lower level macros.

    Macro invocations may be nested to a maximum level of 5.

12) The "shorewall show zones" command now flags zone members that have
    been added using "shorewall add" by preceding them with a plus sign
    ("+").

    Example:

    Shorewall 3.9.4 Zones at gateway - Mon May 14 07:48:16 PDT 2007

    fw (firewall)
    net (ipv4)
        eth0:0.0.0.0/0
    loc (ipv4)
	br0:0.0.0.0/0
	eth4:0.0.0.0/0
	eth5:0.0.0.0/0
	+eth1:0.0.0.0/0
    dmz (ipv4)
	eth3:0.0.0.0/0
    vpn (ipv4)
	tun+:0.0.0.0/0

    In the above output, "eth1:0.0.0.0/0" was dynamically added to the
    'loc' zone. As part of this change, "shorewall delete" will only
    delete entries that have been added dynamically. In earlier
    versions, any entry could be deleted although the ruleset was only
    changed by deleting entries that had been added dynamically.

13) The 'shorewall version' command now lists the version of the
    installed compiler(s):

    gateway:/bulk/backup # shorewall version
    4.0.0-Beta1
    Shorewall-shell 4.0.0-Beta1
    Shorewall-perl 4.0.0-Beta1
    gateway:/bulk/backup #

14) The Perl compiler is externalized. Both the compiler.pl program
    and the Perl Module interface are documented.

    The compiler program is /usr/share/shorewall-perl/compiler.pl:

	compiler.pl [ <option> ... ] [ <filename> ]

    If a <filename> is given, then the configuration will be compiled
    output placed in the named file. If <filename> is not given, then
    the configuration will simply be syntax checked.

    Options are:

    -v <verbosity>
    --verbosity=<verbosity>
    
	The <verbosity> is a number between 0 and 2 and corresponds to
	the VERBOSITY setting in shorewall.conf. This setting controls
	the verbosity of the compiler itself.

    -e
    --export

	If given, the configuration will be compiled for export to
	another system.

    -d <directory>
    --directory=<directory>

	If this option is omitted, the configuration in /etc/shorewall
	is compiled/checked. Otherwise, the configuration in the named
	directory will be compiled/checked.

    -t
    --timestamp

	If given, each progress message issued by the compiler and by
	the compiled program will be timestamped.

    --debug

	If given, when a warning or error message is issued, it is 
	supplimented with a stack trace. Requires the Carp Perl
	module.

    Example (compiles the configuration in the current directory 
             generating a script named 'firewall' and using VERBOSITY
             2).

       /usr/share/shorewall-perl/compiler.pl -v 2 -d . firewall

    Note: For compatibility with the Shorewall 3.4.2 and 3.4.3
    releases, options not passed on the run-line get their values from
    environmental variables:

	   Option                 Variable

	   --verbosity		  VERBOSE
	   --export		  EXPORT
	   --directory		  SHOREWALL_DIR
	   --timestamp		  TIMESTAMP

    The Perl Module is externalized as follows:

	use lib '/usr/share/shorewall-perl';
	use Shorewall::Compiler;
	
	    compiler $filename, $directory, $verbose, $options

    The arguments to the compiler function are as follows:

	$filename -   Name of the compiled script to be created.
                      If the arguments evaluates to false, the
                      configuration is syntax checked
		      
        $directory -  The directory containing the configuration.
		      If passed as '', then /etc/shorewall/ is assumed.

	$verbose -    The verbosity level (0-2).

	$options -    A bitmap of options. Shorewall::Compiler
                      exports two constants to help building this
		      argument:

			EXPORT    = 0x01
			TIMESTAMP = 0x02

    The compiler raises an exception with 'die' if it encounters an
    error; $@ contains the 'ERROR' messages describing the problem.
    
    The compiler function can be called repeatedly with different
    inputs.

15) When TC_ENABLED=Internal, Shorewall-perl now validates classids in
    the MARK/CLASSIFY column of /etc/shorewall/tcrules against the
    classes generated by /etc/shorewall/tcclasses.

16) During installation, Shorewall generates the Perl module
    /usr/share/shorewall-perl/Shorewall/Ports.pm, using your
    /etc/protocols and /etc/services as input.

    To re-generate the module from those two files:

	1. Backup your current /usr/share/shorewall-perl/Shorewall/Ports.pm
	   file.
        2. /usr/share/shorewall-perl/buildports.pl > \
	   /usr/share/shorewall-perl/Shorewall/Ports.pm

    Note: If the buildports.pl program fails to run to a successful
    completion during installation, a fallback version of
    module will be installed. That fallback module was generated from
    the /etc/protocols and /etc/services shipped with Ubuntu Feisty
    Fawn.

    Even if the buildports.pl program runs successfully, the fallback
    module is also installed as
    /usr/share/shorewall-perl/Shorewall/FallbackPorts.pm. So if you
    encounter problems with the generated module, simply copy the
    fallback module to /usr/share/shorewall-perl/Shorewall/Ports.pm.

----------------------------------------------------------------------------
                      P R E R E Q U I S I T E S
----------------------------------------------------------------------------
- Perl (I use Perl 5.8.8 but other versions should work fine)
- Perl Cwd Module
- Perl File::Basename Module
- Perl File::Temp Module
- Perl Getopt::Long Module
----------------------------------------------------------------------------
               U S I N G   T H E   N E W   C O M P I L E R
----------------------------------------------------------------------------
If you only install one compiler, then that compiler will be used.

If you install both compilers, then the compiler actually used depends
on the SHOREWALL_COMPILER setting in shorewall.conf.

The value of this new option can be either 'perl' or 'shell'.

If you add 'SHOREWALL_COMPILER=perl' to /etc/shorewall/shorewall.conf
then by default, the new compiler will be used on the system. If you
add it to shorewall.conf in a separate directory (such as a
Shorewall-lite export directory) then the new compiler will only be
used when you compile from that directory.

If you only install one compiler, it is suggested that you do not set
SHOREWALL_COMPILER.

You can also select the compiler to use on the command line using the
'C option:

       '-C shell' means use the shell compiler
       '-C perl' means use the perl compiler

The -C option overrides the setting in shorewall.conf. 

Example:

	shorewall restart -C perl

Regardless of the setting of SHOREWALL_COMPILER, there is one change in
Shorewall operation that is triggered simply by installing
shorewall-perl. Your params file will be processed during compilation 
with the shell's '-a' option  which causes any variables that you set
or create in that file to be automatically exported. Since the params
file is processed before shorewall.conf, using -a insures that the
settings of your params variables are available to the new compiler
should its use be specified in shorewall.conf.