diff --git a/docs/Internals.xml b/docs/Internals.xml index 690071397..a28779a05 100644 --- a/docs/Internals.xml +++ b/docs/Internals.xml @@ -65,9 +65,9 @@ released it as a separate Shorewall-perl packets in Shorewall 4.0.0 (July 2007). The shell-based compiler was packaged in a Shorewall-shell package. An option (SHOREWALL_COMPILER) in shorewall.conf specified - which compiler to use. The Perl-based compiler was siginificantly faster - and the compiled script also ran much faster, thanks to its use of - iptables-restore. + which compiler to use. The Perl-based compiler was siginificantly + faster, and the compiled script also ran much faster thanks to its use + of iptables-restore. Shorewall6 was introduced in Shorewall 4.2.4 (December 2008). @@ -82,7 +82,387 @@
Architecture - + The components of the Shorewall product suite fall into five broad + categories: + + + + Build/Install subsystem + + + + + + Command Line Interface (CLI) + + + + Run-time Libraries + + + + Compiler + + + + Configuration files (including actions and macros) + + + +
+ Build/Install Subsystem + + The Shorewall Build/Install subsystem packages the products for + release and installs them on an end-user's or a packager's system. It + is diagrammed in the following graphic. + + + + The build environment components are not released and are + discussed in the Shorewall Build + Article. + + The end-user/packager environment consists of the + configure and configure.pl + programs in Shorewall-core and an install.sh + program in each product. +
+ +
+ CLI + + The CLI is written entirely in Bourne Shell so as to allow it to + run on small embedded systems within the -lite products. The CLI + programs themselves are very small; then set global variables then + call into the CLI libraries. Here's an example + (/sbin/shorewall): + + PRODUCT=shorewall + +# +# This is modified by the installer when ${SHAREDIR} != /usr/share +# +. /usr/share/shorewall/shorewallrc + +g_program=$PRODUCT +g_libexec="$LIBEXECDIR" +g_sharedir="$SHAREDIR"/shorewall +g_sbindir="$SBINDIR" +g_perllib="$PERLLIBDIR" +g_confdir="$CONFDIR"/shorewall +g_readrc=1 + +. $g_sharedir/lib.cli + +shorewall_cli $@ + + As you can see, it sets the PRODUCT variable, loads the + shorewallrc file, sets the global variables (all of which have names + beginning with "g_", loads lib.cli, and calls + shorewall_cli passing its own arguments. + + There are two CLI libraries: lib.cli in + Shorewall Core and lib.cli-std in Shorewall. The + lib.cli library is always loaded by the CLI + programs; lib-cli-std is also loaded when the + product is 'shorewall' or 'shorewall6'. + lib.cli-std overloads some functions in + lib.cli and also provides logic for the + additional commands supported by the full products. + + The CLI libraries load two additional Shell libraries from + Shorewall.core: lib.base and + lib.common (actually, + lib.base loads lib.common). + These libraries are separete from lib.cli for + both historical and practicle reasons. lib.base + (aka functions) can be loaded by application programs, although this + was more common in the early years of Shorewall. In addition to being + loaded by the CLIs, lib.common is also copied + into the generated script by the compilers. +
+ +
+ Run-time Libraries + + Thare are two libraries that are copied into the generated + script by the compiler: lib.common from + Shorewall-core and lib.core from Shorewall. The + "outer block" of the generated script comes from the Shorewall file + prog.footer. +
+ +
+ Compiler + + With the exception of the getparams Shell + program, the compiler is written in Perl. The compiler main program is + compiler.pl from Shorewall.conf; it's run-line arguments are described + in the Shorewall Perl + Article. It is invoked by the compiler + function in lib.cli-std. + + The compiler is modularized as follows: + + + + Accounting.pm (Shorewall::Accounting). + Processes the accounting file. + + + + Chains.pm (Shorewall::Chains). This is + the module that provides an interface to iptables/Netfilter for + the other modules. The optimizer is included in this + module. + + + + Config.pm (Shorewall::Config). This is + a multi-purpose module that supplies several related + services: + + + + Error and Progress message production. + + + + Pre-processor. Supplies all configuration file handling + including variable expansion, ?IF...?ELSE...?ENDIF processing, + INCLUDE directives and embedded Shell and Perl. + + + + Output script file creation with functions to write into + the script. The latter functions are no-ops when the + check command is being executed. + + + + + + Compiler.pm (Shorewall::Compiler). The + compiler() function in this module contains the top-leve of the + compiler. + + + + IPAddrs.pm (Shorewall::IPAddrs) - IP + Address validation and manipulation (both IPv4 and IPv6). Also + interfaces to NSS for protocol/service name resolution. + + + + Misc.pm (Shorewall::Misc) - Provides + services that don't fit well into the other modules. + + + + Nat.pm (Shorewall::Nat) - Handles all + nat table rules. Processes the masq, + nat and netmap + files. + + + + Proc.pm (Shorewall::Proc) - Handles + manipulation of /proc/sys/. + + + + Providers.pm (Shorewall::Providers) - + Handles policy routing; processes the + providers file. + + + + Proxyarp.pm (Shorewall::Proxyarp) - + Processes the proxyarp file. + + + + Raw.pm (Shorewall::Raw) - Handles the + raw table; processes the conntrack (formerly + notrack) file. + + + + Rules.pm (Shorewall::Rules) - Contains + the logic for process the policy and + rules files, including + macros and + actions. + + + + Tc.pm (Shorewall::Tc) - Handles traffic + shaping. + + + + Tunnels.pm (Shorewall::Tunnels) - + Processes the tunnels file. + + + + Zones.pm (Shorewall::Zones) - Processes + the zones, interfaces + and hosts files. Provides the interface to + zones and interfaces to the other modules. + + + + Because the params file can contain arbitrary shell code, it + must be processed by a shell. The body of + getparams is as follows: + + # Parameters: +# +# $1 = Path name of params file +# $2 = $CONFIG_PATH +# $3 = Address family (4 or 6) +# +if [ "$3" = 6 ]; then + PRODUCT=shorewall6 +else + PRODUCT=shorewall +fi + +# +# This is modified by the installer when ${SHAREDIR} != /usr/share +# +. /usr/share/shorewall/shorewallrc + +g_program="$PRODUCT" +g_libexec="$LIBEXECDIR" +g_sharedir="$SHAREDIR"/shorewall +g_sbindir="$SBINDIR" +g_perllib="$PERLLIBDIR" +g_confdir="$CONFDIR/$PRODUCT" +g_readrc=1 + +. $g_sharedir/lib.cli + +CONFIG_PATH="$2" + +set -a + +. $1 >&2 # Avoid spurious output on STDOUT + +set +a + +export -p + + The program establishes the environment of the Shorewall or + Shoreall6 CLI program since that is the environment in which the + params file has been traditionally processed. It + then sets the - option so that all newly-created + variables will be exported and invokes the + params file. Because the + STDOUT file is a pipe back to the compiler, no spurious output must be + sent to that file; so getparams redirect + params output to STDOUT. After the script has + executed, an export -p command is executed to send + the contents of the environ array back to the compiler. + + Regrettably, the various shells (and even different versions of + the same shell) produce quite different output from export + -p. The Perl function Shorewall::Config::getparams() detects + which species of shell was being used and stores the variable settings + into the %params hash. Variables that are also in %ENV are only stored + in %params if there value in the output from the + getparams script is different from that in + %ENV. +
+ +
+ Configuration Files + + The configuration files are all well-documented. About the only + thing worth noting is that some macros and actions are duplicated in + the Shorewall and Shorewall6 packages. Because the Shorewall6 default + CONFIG_PATH looks in ${SHAREDIR}/shorewall6 before looking in + ${SHARDIR_/shorewall, this allows Shorewall6 to implement + IPv6-specific handling where required. +
+
+ +
+ The Generated Script + + The generated script is completely self-contained so as to avoid + version dependencies between the Shorewall version used to create the + script and the version of Shorewall-common installed on the remote + firewall. + + The operation of the generated script is illustrated in this + diagram. + + + + The Netfilter ruleset is sometimes dependent on the environment + when the script runs. Dynamic IP addresses and gateways, for example, + must be detected when the script runs. As a consequence, it is the + generated script and not the compiler that creates the input for + iptables-restore. While that input could be passed to iptables-restore + in a pipe, it is written to + ${VARDIR}/.iptables_restore-input so that it is + available for post-mortem analysis in the event that iptables-restore + fails. For the other utilities (ip, tc, ipset, etc), the script runs + them passing their input on the run-line. +
+ + +
+ Compiler Internals + + Because the compiler is the most complex part of the Shorewall + product suite, I've chosen to document it first. Before diving into the + details of the individual modules, lets take a look at a few general + things. + +
+ Modularization + + While the compiler is modularized and uses encapsulation, it is + not object-oriented. This is due to the fact that much of the compiler + was written by manually translating the earlier Shell code. + + Module data is not completely encapsulated. Heavily used tables, + most notably the Chain Table (%chain_table) in Shorewall::Chains is + exported for read access. Updates to module data is always + encapsulated. +
+ +
+ Module Initialization + + While currently unused and untested, the Compiler modules are + designed to be able to be loaded into a parent Perl program and the + compiler executed repeatedly without unloading the modules. To + accomodate that usage scenario, variable data is not initialized at + declaration time or in an INIT block, but is rather initialized in an + initialize function. Because off of these + functions have the same name ("initialize"), they are not exported but + are rather called using a fully-qualified name (e.g., + "Shorewall::Config::initialize"). + + Most of the the initialization functions accept arguements. Those + most common argument is the address family (4 or 6), depending on + whether an IPv4 or IPv6 firewall is being compiled. Each of the modules + that are address-family dependent have their own $family private (my) + variable. +
+ +
+ Module Dependence + + Here is the module dependency tree. To simplify the diagram, + direct dependencies are not shown where there is also a transitive + dependency. + +
diff --git a/docs/images/BuildInstall.png b/docs/images/BuildInstall.png new file mode 100644 index 000000000..4187ac028 Binary files /dev/null and b/docs/images/BuildInstall.png differ diff --git a/docs/images/ModuleDepencency.dia b/docs/images/ModuleDepencency.dia new file mode 100644 index 000000000..d4c850be4 Binary files /dev/null and b/docs/images/ModuleDepencency.dia differ diff --git a/docs/images/ModuleDepencency.png b/docs/images/ModuleDepencency.png new file mode 100644 index 000000000..c302f3b6e Binary files /dev/null and b/docs/images/ModuleDepencency.png differ diff --git a/docs/images/RunningScript.png b/docs/images/RunningScript.png new file mode 100644 index 000000000..08b10a549 Binary files /dev/null and b/docs/images/RunningScript.png differ