shorewall_code/docs/Accounting.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
  <!--$Id$-->

  <articleinfo>
    <title>Shorewall Traffic Accounting</title>

    <authorgroup>
      <author>
        <firstname>Tom</firstname>

        <surname>Eastep</surname>
      </author>
    </authorgroup>

    <pubdate><?dbtimestamp format="Y/m/d"?></pubdate>

    <copyright>
      <year>2003-2009</year>

      <holder>Thomas M. Eastep</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <caution>
    <para><emphasis role="bold">This article applies to Shorewall 4.0 and
    later. If you are running a version of Shorewall earlier than Shorewall
    4.0.0 then please see the documentation for that
    release</emphasis>.</para>
  </caution>

  <section id="Basics">
    <title>Accounting Basics</title>

    <para>Shorewall accounting rules are described in the file
    <filename><filename>/etc/shorewall/accounting</filename></filename>. By
    default, the accounting rules are placed in a chain called
    <quote>accounting</quote> and can thus be displayed using
    <quote>shorewall[-lite] show -x accounting</quote>. All traffic passing
    into, out of, or through the firewall traverses the accounting chain
    including traffic that will later be rejected by interface options such as
    <quote>tcpflags</quote> and <quote>maclist</quote>.</para>

    <para>The columns in the accounting file are described in <ulink
    url="manpages/shorewall-accounting.html">shorewall-accounting</ulink> (5)
    and <ulink
    url="manpages6/shorewall6-accounting.html">shorewall6-accounting</ulink>
    (5).</para>

    <para>In all columns except ACTION and CHAIN, the values <quote>-</quote>,
    <quote>any</quote> and <quote>all</quote> are treated as
    wild-cards.</para>

    <para>The accounting rules are evaluated in the Netfilter
    <quote>filter</quote> table. This is the same environment where the
    <quote>rules</quote> file rules are evaluated and in this environment,
    DNAT has already occurred in inbound packets and SNAT has not yet occurred
    on outbound packets.</para>

    <para>Accounting rules are not stateful -- each rule only handles traffic
    in one direction. For example, if eth0 is your Internet interface, and you
    have a web server in your DMZ connected to eth1, then to count HTTP
    traffic in both directions requires two rules:</para>

    <programlisting>        #ACTION CHAIN   SOURCE  DESTINATION     PROTOCOL        DEST            SOURCE
        #                                                       PORT            PORT
        DONE    -       eth0    eth1            tcp             80
        DONE    -       eth1    eth0            tcp             -               80</programlisting>

    <para>Associating a counter with a chain allows for nice reporting. For
    example:</para>

    <programlisting>        #ACTION         CHAIN   SOURCE  DESTINATION     PROTOCOL        DEST            SOURCE
        #                                                               PORT            PORT
        web:COUNT       -       eth0    eth1            tcp             80
        web:COUNT       -       eth1    eth0            tcp             -               80
        web:COUNT       -       eth0    eth1            tcp             443
        web:COUNT       -       eth1    eth0            tcp             -               443
        DONE            web</programlisting>

    <para>Now <command>shorewall show web</command> (or
    <command>shorewall-lite show web</command> for Shorewall Lite users) will
    give you a breakdown of your web traffic:</para>

    <programlisting>     [root@gateway shorewall]# shorewall show web
     Shorewall-1.4.6-20030821 Chain web at gateway.shorewall.net - Wed Aug 20 09:48:56 PDT 2003
     
     Counters reset Wed Aug 20 09:48:00 PDT 2003

     Chain web (4 references)
     pkts bytes target     prot opt in     out     source               destination
       11  1335            tcp  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0          tcp dpt:80
       18  1962            tcp  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0          tcp spt:80
        0     0            tcp  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0          tcp dpt:443
        0     0            tcp  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0          tcp spt:443
       29  3297 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
       [root@gateway shorewall]#</programlisting>

    <para>Here is a slightly different example:</para>

    <programlisting>        #ACTION         CHAIN   SOURCE  DESTINATION     PROTOCOL        DEST            SOURCE
        #                                                               PORT            PORT
        web             -       eth0    eth1            tcp             80
        web             -       eth1    eth0            tcp             -               80
        web             -       eth0    eth1            tcp             443
        web             -       eth1    eth0            tcp             -               443
        COUNT           web     eth0    eth1
        COUNT           web     eth1    eth0</programlisting>

    <para>Now <command>shorewall show web</command> (or
    <command>shorewall-lite show web</command> for Shorewall Lite users)
    simply gives you a breakdown by input and output:</para>

    <programlisting>     [root@gateway shorewall]# shorewall show accounting web
     Shorewall-1.4.6-20030821 Chains accounting web at gateway.shorewall.net - Wed Aug 20 10:27:21 PDT 2003

     Counters reset Wed Aug 20 10:24:33 PDT 2003

     Chain accounting (3 references)
         pkts bytes target     prot opt in     out     source               destination
         8767  727K web        tcp  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0          tcp dpt:80
            0     0 web        tcp  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0          tcp dpt:443</programlisting>

    <programlisting>
        11506   13M web        tcp  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0          tcp spt:80
            0     0 web        tcp  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0          tcp spt:443

     Chain web (4 references)
         pkts bytes target     prot opt in     out     source               destination
         8767  727K            all  --  eth0   eth1    0.0.0.0/0            0.0.0.0/0
        11506   13M            all  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0
     [root@gateway shorewall]#</programlisting>

    <para>Here's how the same example would be constructed on an HTTP server
    with only one interface (eth0).</para>

    <caution>
      <para>READ THE ABOVE CAREFULLY -- IT SAYS <emphasis
      role="bold">SERVER</emphasis>. If you want to account for web browsing,
      you have to reverse the rules below.</para>
    </caution>

    <programlisting>        #ACTION         CHAIN   SOURCE  DESTINATION     PROTOCOL        DEST            SOURCE
        #                                                               PORT            PORT
        web             -       eth0    -               tcp             80
        web             -       -       eth0            tcp             -               80
        web             -       eth0    -               tcp             443
        web             -       -       eth0            tcp             -               443
        COUNT           web     eth0
        COUNT           web     -       eth0</programlisting>

    <para>Note that with only one interface, only the SOURCE (for input rules)
    or the DESTINATION (for output rules) is specified in each rule.</para>

    <para>Here's the output:</para>

    <programlisting>     [root@mail shorewall]# shorewall show accounting web Shorewall-1.4.7
     Chains accounting web at mail.shorewall.net - Sun Oct 12 10:27:21 PDT 2003

     Counters reset Sat Oct 11 08:12:57 PDT 2003

     Chain accounting (3 references)
      pkts bytes target     prot opt in     out     source               destination
      8767  727K web        tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0          tcp dpt:80
     11506   13M web        tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0          tcp spt:80
         0     0 web        tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0          tcp dpt:443
         0     0 web        tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0          tcp spt:443

     Chain web (4 references)
      pkts bytes target     prot opt in     out     source               destination
      8767  727K            all  --  eth0   *       0.0.0.0/0            0.0.0.0/0
     11506   13M            all  --  *      eth0    0.0.0.0/0            0.0.0.0/0
     [root@mail shorewall]#</programlisting>

    <para>For an example of integrating Shorewall Accounting with MRTG, see
    <ulink
    url="http://www.nightbrawler.com/code/shorewall-stats/">http://www.nightbrawler.com/code/shorewall-stats/</ulink>.</para>
  </section>

  <section id="Bridge">
    <title>Accounting with Bridges</title>

    <para>The structure of the accounting rules changes slightly when there
    are <ulink url="bridge-Shorewall-perl.html">bridges</ulink> defined in the
    Shorewall configuration. Because of the restrictions imposed by Netfilter
    in kernel 2.6.21 and later, output accounting rules must be segregated
    from forwarding and input rules. To accomplish this separation,
    Shorewall-perl creates two accounting chains:</para>

    <itemizedlist>
      <listitem>
        <para><emphasis role="bold">accounting</emphasis> - for input and
        forwarded traffic.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">accountout</emphasis> - for output
        traffic.</para>
      </listitem>
    </itemizedlist>

    <para>If the CHAIN column contains <quote>-</quote>, then:</para>

    <itemizedlist>
      <listitem>
        <para>If the SOURCE column in a rule includes the name of the firewall
        zone (e.g., $FW), then the default chain to insert the rule into is
        <emphasis role="bold">accountout</emphasis> only.</para>
      </listitem>

      <listitem>
        <para>Otherwise, if the DEST in the rule is <emphasis
        role="bold">any</emphasis> or <emphasis role="bold">all</emphasis> or
        0.0.0.0/0, then the rule is added to both <emphasis
        role="bold">accounting</emphasis> and <emphasis
        role="bold">accountout</emphasis>.</para>
      </listitem>

      <listitem>
        <para>Otherwise, the rule is added to <emphasis
        role="bold">accounting</emphasis> only.</para>
      </listitem>
    </itemizedlist>
  </section>

  <section>
    <title>Sectioned Accounting Rules</title>

    <para>Traditionally, the root of the Shorewall accounting rules has been
    the <emphasis role="bold">accounting</emphasis> chain. Having a single
    root chain has drawbacks:</para>

    <itemizedlist>
      <listitem>
        <para>Many rules are traversed needlessly (they could not possibly
        match traffic).</para>
      </listitem>

      <listitem>
        <para>At any time, the Netfilter team could begin generating errors
        when loading those same rules (that has happened).</para>
      </listitem>

      <listitem>
        <para>MAC addresses may not be used in the accounting rules.</para>
      </listitem>

      <listitem>
        <para>The <emphasis role="bold">accounting</emphasis> chain cannot be
        optimized when OPTIMIZE_ACCOUNTING=Yes.</para>
      </listitem>

      <listitem>
        <para>The rules may be defined in any order so the rules compiler must
        post-process the ruleset to ensure that there are no loops and to
        alert the user to unreferenced chains.</para>
      </listitem>
    </itemizedlist>

    <para>Beginning with Shorewall 4.4.18, the accounting structure can be
    created with three root chains:</para>

    <itemizedlist>
      <listitem>
        <para><emphasis role="bold">accountin</emphasis>: Rules that are valid
        in the <emphasis role="bold">INPUT</emphasis> chain (may not specify
        an output interface).</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">accountout</emphasis>: Rules that are
        valid in the OUTPUT chain (may not specify an input interface or a MAC
        address).</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">accounting</emphasis>: Other rules.</para>
      </listitem>
    </itemizedlist>

    <para>The new structure is enabled by sectioning the accounting file in a
    manner similar to the <ulink url="manpages/shorewall-rules.html">rules
    file</ulink>. The sections are <emphasis role="bold">INPUT</emphasis>,
    <emphasis role="bold">OUTPUT</emphasis> and <emphasis
    role="bold">FORWARD</emphasis> and must appear in that order (although any
    of them may be omitted). The first non-commentary record in the accounting
    file must be a section header when sectioning is used.</para>

    <para>Beginning with Shorewall 4.4.20, the ACCOUNTING_TABLE setting was
    added to shorewall.conf and shorewall6.conf. That setting determines the
    Netfilter table (filter or mangle) where the accounting rules are added.
    When ACCOUNTING_TABLE=mangle is specified, the available sections are
    <emphasis role="bold">PREROUTING</emphasis>, <emphasis
    role="bold">INPUT</emphasis>, <emphasis role="bold">OUTPUT</emphasis>,
    <emphasis role="bold">FORWARD</emphasis> and <emphasis
    role="bold">POSTROUTING</emphasis>.</para>

    <para>Section headers have the form:</para>

    <para><option>SECTION</option>
    <replaceable>section-name</replaceable></para>

    <para>When sections are enabled:</para>

    <itemizedlist>
      <listitem>
        <para>You must jump to a user-defined accounting chain before you can
        add rules to that chain.</para>
      </listitem>

      <listitem>
        <para>This eliminates loops and unreferenced chains.</para>
      </listitem>

      <listitem>
        <para>You may not specify an output interface in the <emphasis
        role="bold">PREROUTING</emphasis> and <emphasis
        role="bold">INPUT</emphasis> sections.</para>
      </listitem>

      <listitem>
        <para>In the <emphasis role="bold">OUTPUT</emphasis> and <emphasis
        role="bold">POSTROUTING</emphasis> sections:</para>

        <itemizedlist>
          <listitem>
            <para>You may not specify an input interface</para>
          </listitem>

          <listitem>
            <para>You may not jump to a chain defined in the <emphasis
            role="bold">INPUT</emphasis> or <emphasis
            role="bold">PREROUTING</emphasis> sections that specifies an input
            interface</para>
          </listitem>

          <listitem>
            <para>You may not specify a MAC address</para>
          </listitem>

          <listitem>
            <para>You may not jump to a chain defined in the <emphasis
            role="bold">INPUT</emphasis> or <emphasis
            role="bold">PREROUTING</emphasis> section that specifies a MAC
            address.</para>
          </listitem>
        </itemizedlist>
      </listitem>

      <listitem>
        <para>The default value of the CHAIN column is:</para>

        <itemizedlist>
          <listitem>
            <para><emphasis role="bold">accountin</emphasis> in the <emphasis
            role="bold">INPUT</emphasis> section</para>
          </listitem>

          <listitem>
            <para><emphasis role="bold">accounout</emphasis> in the <emphasis
            role="bold">OUTPUT</emphasis> section</para>
          </listitem>

          <listitem>
            <para><emphasis role="bold">accountfwd</emphasis> in the <emphasis
            role="bold">FORWARD</emphasis> section</para>
          </listitem>

          <listitem>
            <para><emphasis role="bold">accountpre</emphasis> in the <emphasis
            role="bold">PREROUTING</emphasis> section</para>
          </listitem>

          <listitem>
            <para><emphasis role="bold">accountpost</emphasis> in the
            <emphasis role="bold">POSTROUTING</emphasis> section</para>
          </listitem>
        </itemizedlist>
      </listitem>

      <listitem>
        <para>Traffic addressed to the firewall goes through the rules defined
        in the INPUT section.</para>
      </listitem>

      <listitem>
        <para>Traffic originating on the firewall goes through the rules
        defined in the OUTPUT section.</para>
      </listitem>

      <listitem>
        <para>Traffic being forwarded through the firewall goes through the
        rules from the FORWARD sections.</para>
      </listitem>
    </itemizedlist>

    <para>Here is a sample sectioned file that used <link
    linkend="perIP">Per-IP Accounting</link>.</para>

    <caution>
      <para>In this example, the dmz net corresponds to a vserver zone so
      lives on the firewall itself.</para>
    </caution>

    <programlisting>#ACTION				CHAIN           SOURCE                          DESTINATION                   PROTO   DEST            SOURCE  USER/    MARK     IPSEC
#                                      		                                                              	      PORT(S)         PORT(S) GROUP
SECTION INPUT
ACCOUNT(fw-net,$FW_NET)		-		COM_IF
ACCOUNT(dmz-net,$DMZ_NET)  	-		COM_IF

SECTION OUTPUT
ACCOUNT(fw-net,$FW_NET)		-		-				COM_IF
ACCOUNT(dmz-net,$DMZ_NET)   	-		-				COM_IF

SECTION FORWARD
ACCOUNT(loc-net,$INT_NET)   	-		COM_IF				INT_IF
ACCOUNT(loc-net,$INT_NET)   	-		INT_IF				COM_IF
</programlisting>
  </section>

  <section id="Collectd">
    <title>Integrating Shorewall Accounting with Collectd</title>

    <para>Sergiusz Pawlowicz has written a nice article that shows how to
    integrate Shorewall Accounting with collectd to produce nice graphs of
    traffic activity. The article may be found at <ulink
    url="http://collectd.org/wiki/index.php/Plugin:IPTables">http://collectd.org/wiki/index.php/Plugin:IPTables</ulink>.</para>
  </section>

  <section id="perIP">
    <title>Per-IP Accounting</title>

    <para>Shorewall 4.4.17 added support for per-IP accounting using the
    ACCOUNT target. That target is only available when xtables-addons is
    installed. This support has been successfully tested with xtables-addons
    1.32 on:</para>

    <itemizedlist>
      <listitem>
        <para>Fedora 14</para>
      </listitem>

      <listitem>
        <para>Debian Squeeze</para>
      </listitem>

      <listitem>
        <para>OpenSuSE 11.3</para>
      </listitem>
    </itemizedlist>

    <para>and xtables-addons Version 1.21 on:</para>

    <itemizedlist>
      <listitem>
        <para>Debian Lenny</para>
      </listitem>
    </itemizedlist>

    <para>Information about xtables-addons installation may be found at <ulink
    url="Dynamic.html#xtables-addons">here</ulink>.</para>

    <para>Per-IP accounting is configured in <ulink
    url="manpages/shorewall-accounting.html">shorewall-accounting</ulink> (5)
    (it is currently not supported in IPv6). In the ACTION column,
    enter:</para>

    <simplelist>
      <member><emphasis
      role="bold">ACCOUNT(</emphasis><replaceable>table</replaceable>,<replaceable>network</replaceable><emphasis
      role="bold">)</emphasis></member>
    </simplelist>

    <para>where</para>

    <simplelist>
      <member><replaceable>table</replaceable> is the name of an accounting
      table (you choose the name). All rules specifying the same table will
      have their per-IP counters accumulated in that table.</member>

      <member><replaceable>network</replaceable> is an IPv4 network in CIDR
      notation. The network can be as large as a /8 (class A).</member>
    </simplelist>

    <para>One nice feature of per-IP accounting is that the counters survive
    <command>shorewall restart</command>. This has a downside, however. If you
    change the network associated with an accounting table, then you must
    <command>shorewall stop; shorewall start</command> to have a successful
    restart (counters will be cleared).</para>

    <para>Example: Suppose your WAN interface is eth0 and your LAN interface
    is eth1 with network 172.20.1.0/24. To account for all traffic between the
    WAN and LAN interfaces:</para>

    <programlisting>#ACTION                         CHAIN        SOURCE              DEST          ...
ACCOUNT(net-loc,172.20.1.0/24)  -            eth0                eth1
ACCOUNT(net-loc,172.20.1.0/24)  -            eth1                eth0</programlisting>

    <para>This will create a <emphasis role="bold">net-loc</emphasis> table
    for counting packets and bytes for traffic between the two
    interfaces.</para>

    <para>The table is dumped using the <command>iptaccount</command> utility
    (part of xtables-addons):</para>

    <programlisting><command>iptaccount [-f] -l net-loc</command></programlisting>

    <para>Example:</para>

    <programlisting>gateway:~# <emphasis role="bold">iptaccount -l net-loc</emphasis>

libxt_ACCOUNT_cl userspace accounting tool v1.3

Showing table: net-loc
Run #0 - 3 items found
IP: 172.20.1.105 SRC packets: 115 bytes: 131107 DST packets: 68 bytes: 20045
IP: 172.20.1.131 SRC packets: 47 bytes: 12729 DST packets: 38 bytes: 25304
IP: 172.20.1.145 SRC packets: 20747 bytes: 2779676 DST packets: 27050 bytes: 32286071
Finished.
gateway:~#</programlisting>

    <para>For each local IP address with non-zero counters, the packet and
    byte count for both incoming traffic (IP is DST) and outgoing traffic (IP
    is SRC) are listed. The -f option causes the table to be flushed (reset
    all counters to zero) after printing.</para>

    <para>For a command synopsis:</para>

    <programlisting><command>iptaccount --help</command></programlisting>

    <para><filename>/sbin/shorewall</filename> also supports a <command>show
    ipa</command> command (from my own gateway just after I flushed the
    counters using <command>iptaccount -f -l</command>.:</para>

    <programlisting>gateway:~# <command>shorewall show ipa</command>
Shorewall 4.4.18-Beta1 per-IP Accounting at gateway - Thu Feb 10 13:28:37 PST 2011

Showing table: loc-net
IP: 172.20.1.146 SRC packets: 9 bytes: 574 DST packets: 9 bytes: 770

Showing table: dmz-net
IP: 70.90.191.124 SRC packets: 243 bytes: 23726 DST packets: 248 bytes: 39036
IP: 70.90.191.125 SRC packets: 73 bytes: 10640 DST packets: 73 bytes: 4846

Showing table: fw-net
IP: 70.90.191.121 SRC packets: 0 bytes: 0 DST packets: 4 bytes: 243
IP: 70.90.191.122 SRC packets: 11 bytes: 1338 DST packets: 8 bytes: 5465
IP: 70.90.191.123 SRC packets: 42 bytes: 4604 DST packets: 44 bytes: 10662

gateway:~# 
</programlisting>
  </section>

  <section id="nfacct">
    <title>Accounting using nfacct</title>

    <para>Beginning with the 3.3 kernels, Netfilter supports a form of
    accounting (nfacct) that is triggered by iptables rules but that survives
    purging and/or reloading the Netfilter ruleset. Shorewall support for this
    form of accounting was added in Shorewall 4.5.7.</para>

    <para>As of this writing (late July 2012), Fedora 17 has partial support
    for this feature but not all. It is necessary to download and build the
    following:</para>

    <itemizedlist>
      <listitem>
        <para>libnetfilter_acct</para>
      </listitem>

      <listitem>
        <para>nfacct</para>
      </listitem>
    </itemizedlist>

    <para>The following Fedora packages are also required:</para>

    <itemizedlist>
      <listitem>
        <para>libnetlink and libnetlink-dev</para>
      </listitem>

      <listitem>
        <para>libmnl and libmnl-dev</para>
      </listitem>
    </itemizedlist>

    <para>The tarballs are available from the Netfilter download sites.</para>

    <para>The nfacct utility can create, delete and display <firstterm>nfacct
    objects</firstterm>. These named objects consist of a packet and byte
    counter. Packets matching those netfilter rules that use the nfacct match
    cause the packet and byte count in the object named in the match to be
    incremented.</para>

    <para>To use nfaccnt with Shorewall, use the NFACCT target. See <ulink
    url="manpages/shorewall-accounting.html">shorewall-accounting</ulink>(5)
    for details.</para>

    <para>The <command>shorewall show nfacct</command> command is a thin
    wrapper around the <command>nfacct list</command> command.</para>
  </section>
</article>