shorewall_code/docs/traffic_shaping.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<article>
  <!--$Id$-->

  <articleinfo>
    <title>Complex Traffic Shaping/Control</title>

    <authorgroup>
      <author>
        <firstname>Tom</firstname>

        <surname>Eastep</surname>
      </author>

      <author>
        <firstname>Arne</firstname>

        <surname>Bernin</surname>
      </author>
    </authorgroup>

    <pubdate><?dbtimestamp format="Y/m/d"?></pubdate>

    <copyright>
      <year>2001-2009</year>

      <holder>Thomas M. Eastep</holder>
    </copyright>

    <copyright>
      <year>2005</year>

      <holder>Arne Bernin &amp; Thomas M. Eastep</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <important>
    <para>Traffic shaping is complex and the Shorewall community is not well
    equipped to answer traffic shaping questions. So if you are the type of
    person who needs "insert tab A into slot B" instructions for everything
    that you do, then please don't try to implement traffic shaping using
    Shorewall. You will just frustrate yourself and we won't be able to help
    you.</para>
  </important>

  <warning>
    <para>Said another way, reading just Shorewall documentation is not going
    to give you enough background to use this material.</para>

    <para>At a minimum, you will need to refer to at least the following
    additional information:</para>

    <itemizedlist>
      <listitem>
        <para><emphasis>The LARTC HOWTO</emphasis>: <ulink
        url="http://www.lartc.org">http://www.lartc.org</ulink></para>
      </listitem>

      <listitem>
        <para>T<emphasis>he HTB User's Guide</emphasis>: <ulink
        url="http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm">http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm</ulink></para>
      </listitem>

      <listitem>
        <para><emphasis>HFSC Scheduling with Linux</emphasis>: <ulink
        url="http://linux-ip.net/articles/hfsc.en/">http://linux-ip.net/articles/hfsc.en/</ulink></para>
      </listitem>

      <listitem>
        <para>Some of the documents listed at <ulink
        url="http://www.netfilter.org/documentation/index.html#documentation-howto">http://www.netfilter.org/documentation/index.html#documentation-howto</ulink>.
        The tutorial by Oskar Andreasson is particularly good.</para>
      </listitem>

      <listitem>
        <para>The output of <command>man iptables</command></para>
      </listitem>
    </itemizedlist>
  </warning>

  <section id="Intro">
    <title>Introduction</title>

    <para>Beginning with Shorewall 4.4.6, Shorewall includes two separate
    implementations of traffic shaping. This document describes the original
    implementation which is complex and difficult to configure. A much simpler
    version is described in <ulink role="bold"
    url="simple_traffic_shaping.html">Simple Traffic Shaping/Control</ulink>
    and is highly recommended unless you really need to delay certain traffic
    passing through your firewall.</para>

    <para>Shorewall has builtin support for traffic shaping and control. This
    support does not cover all options available (and especially all
    algorithms that can be used to queue traffic) in the Linux kernel but it
    should fit most needs. If you are using your own script for traffic
    control and you still want to use it in the future, you will find
    information on how to do this, <link linkend="owntcstart">later in this
    document</link>. But for this to work, you will also need to enable
    traffic shaping in the kernel and Shorewall as covered by the next
    sections.</para>
  </section>

  <section id="LinuxTC">
    <title>Linux traffic shaping and control</title>

    <para>This section gives a brief introduction of how controlling traffic
    with the Linux kernel works. Although this might be enough for configuring
    it in the Shorewall configuration files, we strongly recommend that you
    take a deeper look into the <ulink url="http://lartc.org/howto/">Linux
    Advanced Routing and Shaping HOWTO</ulink>. At the time of writing this,
    the current version is 1.0.0.</para>

    <para>Since kernel 2.2, Linux has extensive support for controlling
    traffic. You can define different algorithms that are used to queue the
    traffic before it leaves an interface. The standard one is called pfifo
    and is (as the name suggests) of the type First In First out. This means,
    that it does not shape anything, if you have a connection that eats up all
    your bandwidth, this queuing algorithm will not stop it from doing
    so.</para>

    <para>For Shorewall traffic shaping we use three algorithms: HTB
    (Hierarchical Token Bucket), HFSC (Hierarchical Fair Service Curves) and
    SFQ (Stochastic Fairness Queuing). SFQ is easy to explain: it just tries
    to track your connections (tcp or udp streams) and balances the traffic
    between them. This normally works well. HTB and HFSC allow you to define a
    set of classes, and you can put the traffic you want into these classes.
    You can define minimum and maximum bandwidth settings for those classes
    and order them hierarchically (the less prioritized classes only get
    bandwidth if the more important have what they need). Additionally, HFSC
    allows you to specify the maximum queuing delay that a packet may
    experience. Shorewall builtin traffic shaping allows you to define these
    classes (and their bandwidth limits), and it uses SFQ inside these classes
    to make sure, that different data streams are handled equally. If SFQ's
    default notion of a 'stream' doesn't work well for you, you can change it
    using the <emphasis role="bold">flow</emphasis> option described <link
    linkend="tcclasses">below</link>.</para>

    <para>You can shape incoming traffic through use of an
    <firstterm>Intermediate Functional Block</firstterm> (IFB) device. <link
    linkend="IFB">See below</link>. <emphasis role="bold">But beware: using an
    IFB can result in queues building up both at your ISPs router and at your
    own.</emphasis></para>

    <para>You shape and control outgoing traffic by assigning the traffic to
    <firstterm>classes</firstterm>. Each class is associated with exactly one
    network interface and has a number of attributes:</para>

    <orderedlist>
      <listitem>
        <para>PRIORITY - Used to give preference to one class over another
        when selecting a packet to send. The priority is a numeric value with
        1 being the highest priority, 2 being the next highest, and so
        on.</para>
      </listitem>

      <listitem>
        <para>RATE - The minimum bandwidth this class should get, when the
        traffic load rises. Classes with a higher priority (lower PRIORITY
        value) are served even if there are others that have a guaranteed
        bandwidth but have a lower priority (higher PRIORITY value).</para>
      </listitem>

      <listitem>
        <para>CEIL - The maximum bandwidth the class is allowed to use when
        the link is idle.</para>
      </listitem>

      <listitem>
        <para>MARK - Netfilter has a facility for
        <firstterm>marking</firstterm> packets. Packet marks have a numeric
        value which is limited in Shorewall to the values 1-255 (1-16383 if
        you set WIDE_TC_MARKS=Yes in <ulink
        url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5) ). You
        assign packet marks to different types of traffic using entries in the
        <filename>/etc/shorewall/tcrules</filename> file.</para>

        <note>
          <para>In Shorewall 4.5.0, WIDE_TC_MARKS was superseded by TC_BITS
          which specifies the width in bits of the traffic shaping mark field.
          The default is based on the setting of WIDE_TC_MARKS so as to
          provide upward compatibility.</para>
        </note>
      </listitem>
    </orderedlist>

    <para>One class for each interface must be designated as the
    <firstterm>default class</firstterm>. This is the class to which unmarked
    traffic (packets to which you have not assigned a mark value in
    <filename>/etc/shorewall/tcrules</filename>) is assigned.</para>

    <para>Netfilter also supports a mark value on each connection. You can
    assign connection mark values in
    <filename>/etc/shorewall/tcrules</filename>, you can copy the current
    packet's mark to the connection mark (SAVE), or you can copy the
    connection mark value to the current packet's mark (RESTORE). For more
    information, see<ulink url="PacketMarking.html"> this
    article</ulink>.</para>
  </section>

  <section id="Kernel">
    <title>Linux Kernel Configuration</title>

    <para>You will need at least kernel 2.4.18 for this to work, please take a
    look at the following screenshot for what settings you need to enable. For
    builtin support, you need the HTB scheduler, the Ingress scheduler, the
    PRIO pseudoscheduler and SFQ queue. The other scheduler or queue
    algorithms are not needed.</para>

    <para>This screen shot shows how I configured QoS in a 2.6.16
    Kernel:</para>

    <graphic align="center" fileref="images/traffic_shaping2.6.png" />

    <para>And here's my recommendation for a 2.6.21 kernel:<graphic
    align="center" fileref="images/traffic_shaping2.6.21.png" /></para>
  </section>

  <section id="Shorewall">
    <title>Enable TC support in Shorewall</title>

    <para>You need this support whether you use the builtin support or whether
    you provide your own tcstart script.</para>

    <para>To enable the builtin traffic shaping and control in Shorewall, you
    have to do the following:</para>

    <itemizedlist>
      <listitem>
        <para>Set <emphasis role="bold">TC_ENABLED</emphasis> to "<emphasis
        role="bold">Internal</emphasis>" in /etc/shorewall/shorewall.conf.
        Setting <emphasis role="bold">TC_ENABLED=Yes</emphasis> causes
        Shorewall to look for an external tcstart file (See <link
        linkend="tcstart">a later section</link> for details).</para>
      </listitem>

      <listitem>
        <para>Setting <emphasis role="bold">CLEAR_TC</emphasis> parameter in
        /etc/shorewall/shorewall.conf to <emphasis role="bold">Yes</emphasis>
        will clear the traffic shaping configuration during Shorewall
        [re]start and Shorewall stop. This is normally what you want when
        using the builtin support (and also if you use your own tcstart
        script)</para>
      </listitem>

      <listitem>
        <para>The other steps that follow depend on whether you use your own
        script or the builtin solution. They will be explained in the
        following sections.</para>
      </listitem>
    </itemizedlist>
  </section>

  <section id="Builtin">
    <title>Using builtin traffic shaping/control</title>

    <para>Shorewall's builtin traffic shaping feature provides a thin layer on
    top of the ingress qdesc, HTB and SFQ. That translation layer allows you
    to:</para>

    <itemizedlist>
      <listitem>
        <para>Define HTB and/or HFSC classes using Shorewall-style
        column-oriented configuration files.</para>
      </listitem>

      <listitem>
        <para>Integrate the reloading of your traffic shaping configuration
        with the reloading of your packet-filtering and marking
        configuration.</para>
      </listitem>

      <listitem>
        <para>Assign traffic to HTB or HFSC classes by TOS value.</para>
      </listitem>

      <listitem>
        <para>Assign outgoing TCP ACK packets to an HTB or HFSC class.</para>
      </listitem>

      <listitem>
        <para>Assign traffic to HTB and/or HFSC classes based on packet mark
        value or based on packet contents.</para>
      </listitem>
    </itemizedlist>

    <para>Those few features are really all that builtin traffic
    shaping/control provides; consequently, you need to understand HTB and/or
    HFSC and Linux traffic shaping as well as Netfilter packet marking in
    order to use the facility. Again, please see the links at top of this
    article.</para>

    <para>For defining bandwidths (for either devices or classes) please use
    kbit or kbps (for Kilobytes per second) and make sure there is <emphasis
    role="bold">NO</emphasis> space between the number and the unit (it is
    100kbit <emphasis role="bold">not</emphasis> 100 kbit). Using mbit, mbps
    or a raw number (which means bytes) could be used, but note that only
    integer numbers are supported (0.5 is <emphasis role="bold">not
    valid</emphasis>).</para>

    <para><emphasis role="bold">To properly configure the settings for your
    devices you need to find out the real up- and downstream rates you
    have</emphasis>. This is especially the case, if you are using a DSL
    connection or one of another type that do not have a guaranteed bandwidth.
    Don't trust the values your provider tells you for this; especially
    measuring the real download speed is important! There are several online
    tools that help you find out; search for "dsl speed test" on google (For
    Germany you can use <ulink
    url="http://www.speedcheck.arcor.de/cgi-bin/speedcheck.cgi">arcor speed
    check</ulink>). Be sure to choose a test site located near you.</para>

    <section id="tcdevices">
      <title>/etc/shorewall/tcdevices</title>

      <para>This file allows you to define the incoming and outgoing bandwidth
      for the devices you want traffic shaping to be enabled. That means, if
      you want to use traffic shaping for a device, you have to define it
      here. For additional information, see <ulink
      url="manpages/shorewall-tcdevices.html">shorewall-tcdevices</ulink>
      (5).</para>

      <para>Columns in the file are as follows:</para>

      <itemizedlist>
        <listitem>
          <para>INTERFACE - Name of interface. Each interface may be listed
          only once in this file. You may NOT specify the name of an alias
          (e.g., eth0:0) here; see <ulink url="FAQ.htm#faq18">FAQ #18</ulink>.
          You man NOT specify wildcards here, e.g. if you have multiple ppp
          interfaces, you need to put them all in here! Shorewall will
          determine if the device exists and will only configure the device if
          it does exist. If it doesn't exist or it is DOWN, the following
          warning is issued:</para>

          <para><emphasis role="bold">WARNING: Device &lt;device name&gt; is
          not in the UP state -- traffic-shaping configuration
          skipped</emphasis></para>

          <para>Shorewall assigns a sequential <firstterm>interface
          number</firstterm> to each interface (the first entry in
          <filename>/etc/shorewall/tcdevices</filename> is interface 1, the
          second is interface 2 and so on) You can also explicitly specify the
          interface number by prefixing the interface name with the number and
          a colon (":"). Example: 1:eth0.</para>

          <warning>
            <para>Device numbers are expressed in hexidecimal. So the device
            following 9 is A, not 10.</para>
          </warning>
        </listitem>

        <listitem>
          <para>IN-BANDWIDTH - The incoming Bandwidth of that interface.
          Please note that when you use this column, you are not traffic
          shaping incoming traffic, as the traffic is already received before
          you could do so. This Column allows you to define the maximum
          traffic allowed for this interface in total, if the rate is
          exceeded, the excess packets are dropped. You want this mainly if
          you have a DSL or Cable Connection to avoid queuing at your
          providers side. If you don't want any traffic to be dropped set this
          to a value faster than your interface maximum rate (or to 0
          (zero).</para>

          <para>To determine the optimum value for this setting, we recommend
          that you start by setting it significantly below your measured
          download bandwidth (20% or so). While downloading, measure the
          <emphasis>ping</emphasis> response time from the firewall to the
          upstream router as you gradually increase the setting.The optimal
          setting is at the point beyond which the <emphasis>ping</emphasis>
          time increases sharply as you increase the setting.</para>
        </listitem>

        <listitem>
          <para>OUT-BANDWIDTH - Specify the outgoing bandwidth of that
          interface. This is the maximum speed your connection can handle. It
          is also the speed you can refer as "full" if you define the tc
          classes. Outgoing traffic above this rate will be dropped.</para>
        </listitem>

        <listitem>
          <para>OPTIONS — A comma-separated list of options from the following
          list:</para>

          <variablelist>
            <varlistentry>
              <term>classify</term>

              <listitem>
                <para>If specified, classification of traffic into the various
                classes is done by CLASSIFY entries in
                <filename>/etc/shorewall/tcrules</filename> or by entries in
                <filename>/etc/shorewall/tcfilters</filename>. No MARK value
                will be associated with classes on this interface.</para>
              </listitem>
            </varlistentry>

            <varlistentry>
              <term>hfsc</term>

              <listitem>
                <para>Shorewall normally uses the <firstterm>Hierarchical
                Token Bucket</firstterm> (HTB) queuing discipline. When
                <option>hfsc</option> is specified, the
                <firstterm>Hierarchical Fair Service Curves</firstterm> (HFSC)
                discipline is used instead.</para>
              </listitem>
            </varlistentry>
          </variablelist>
        </listitem>

        <listitem>
          <para>REDIRECTED INTERFACES — Entries are appropriate in this column
          only if the device in the INTERFACE column names a <link
          linkend="IFB">Intermediate Functional Block (IFB)</link>. It lists
          the physical interfaces that will have their input shaped using
          classes defined on the IFB. Neither the IFB nor any of the
          interfaces listed in this column may have an IN-BANDWIDTH specified.
          You may specify zero (0) or a dash ("-:) in the IN-BANDWIDTH
          column.</para>

          <para>IFB devices automatically get the <emphasis
          role="bold">classify</emphasis> option.</para>
        </listitem>
      </itemizedlist>

      <example id="Example0">
        <title></title>

        <para>Suppose you are using PPP over Ethernet (DSL) and ppp0 is the
        interface for this. The device has an outgoing bandwidth of 500kbit
        and an incoming bandwidth of 6000kbit</para>

        <programlisting>#INTERFACE    IN-BANDWITH      OUT-BANDWIDTH
ppp0           6000kbit         500kbit</programlisting>
      </example>
    </section>

    <section id="tcclasses">
      <title>/etc/shorewall/tcclasses</title>

      <para>This file allows you to define the actual classes that are used to
      split the outgoing traffic. For additional information, see <ulink
      url="manpages/shorewall-tcclasses.html">shorewall-tcclasses</ulink>
      (5).</para>

      <itemizedlist>
        <listitem>
          <para>INTERFACE - Name of interface. Users may also specify the
          interface number. Must match the name (or number) of an interface
          with an entry in <filename>/etc/shorewall/tcdevices</filename>. If
          the interface has the <emphasis role="bold">classify</emphasis>
          option in <filename>/etc/shorewall/tcdevices</filename>, then the
          interface name or number must be followed by a colon and a
          <firstterm>class number</firstterm>. Examples: eth0:1, 4:9. Class
          numbers must be unique for a given interface. Normally, all classes
          defined here are sub-classes of a root class that is implicitly
          defined from the entry in <ulink
          url="shorewall-tcdevices.html">shorewall-tcdevices</ulink>(5). You
          can establish a class hierarchy by specifying a
          <emphasis>parent</emphasis> class (e.g.,
          <emphasis>interface</emphasis>:<emphasis>parent-class</emphasis>:<emphasis>class</emphasis>)
          -- the number of a class that you have previously defined. The
          sub-class may borrow unused bandwidth from its parent.</para>

          <warning>
            <para>Class numbers are expressed in hexidecimal. So the class
            following class 9 is A, not 10.</para>
          </warning>
        </listitem>

        <listitem>
          <para>MARK - The mark value which is an integer in the range 1-255
          (1-16383 if you set WIDE_TC_MARKS=Yes in <ulink
          url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5) ). You
          define these marks in the tcrules file, marking the traffic you want
          to go into the queuing classes defined in here. You can use the same
          marks for different Interfaces. You must specify "-' in this column
          if the device specified in the INTERFACE column has the <emphasis
          role="bold">classify</emphasis> option in
          <filename>/etc/shorewall/tcdevices</filename>.</para>

          <note>
            <para>In Shorewall 4.5.0, WIDE_TC_MARKS was superseded by TC_BITS
            which specifies the width in bits of the traffic shaping mark
            field. The default is based on the setting of WIDE_TC_MARKS so as
            to provide upward compatibility.</para>
          </note>
        </listitem>

        <listitem>
          <para>RATE - The minimum bandwidth this class should get, when the
          traffic load rises. Please note that first the classes which equal
          or a lesser priority value are served even if there are others that
          have a guaranteed bandwidth but a lower priority. <emphasis
          role="bold">If the sum of the RATEs for all classes assigned to an
          INTERFACE exceed that interfaces's OUT-BANDWIDTH, then the
          OUT-BANDWIDTH limit will not be honored.</emphasis></para>

          <para>When using HFSC, this column may contain 1, 2 or 3 pieces of
          information separated by colons (":"). In addition to the minimum
          bandwidth, leaf classes may specify realtime criteria: DMAX (maximum
          delay in milliseconds) and optionally UMAX (the largest packet
          expected in the class). See <link linkend="HFSC">below</link> for
          details.</para>
        </listitem>

        <listitem>
          <para>CEIL - The maximum bandwidth this class is allowed to use when
          the link is idle. Useful if you have traffic which can get full
          speed when more important services (e.g. interactive like ssh) are
          not used. You can use the value "full" in here for setting the
          maximum bandwidth to the defined output bandwidth of that
          interface.</para>
        </listitem>

        <listitem>
          <para>PRIORITY - you have to define a priority for the class.
          packets in a class with a higher priority (=lesser value) are
          handled before less prioritized ones. You can just define the mark
          value here also, if you are increasing the mark values with lesser
          priority.</para>
        </listitem>

        <listitem>
          <para>OPTIONS - A comma-separated list of options including the
          following:</para>

          <itemizedlist>
            <listitem>
              <para>default - this is the default class for that interface
              where all traffic should go, that is not classified
              otherwise.</para>

              <note>
                <para>defining default for exactly <emphasis
                role="bold">one</emphasis> class per interface is
                mandatory!</para>
              </note>
            </listitem>

            <listitem>
              <para>tos-&lt;tosname&gt; - this lets you define a filter for
              the given &lt;tosname&gt; which lets you define a value of the
              Type Of Service bits in the ip package which causes the package
              to go in this class. Please note, that this filter overrides all
              mark settings, so if you define a tos filter for a class all
              traffic having that mark will go in it regardless of the mark on
              the package. You can use the following for this option:
              tos-minimize-delay (16) tos-maximize-throughput (8)
              tos-maximize-reliability (4) tos-minimize-cost (2)
              tos-normal-service (0)</para>

              <note>
                <para>Each of this options is only valid for <emphasis
                role="bold">one</emphasis> class per interface.</para>
              </note>
            </listitem>

            <listitem>
              <para>tcp-ack - if defined causes an tc filter to be created
              that puts all tcp ack packets on that interface that have an
              size of &lt;=64 Bytes to go in this class. This is useful for
              speeding up downloads. Please note that the size of the ack
              packets is limited to 64 bytes as some applications (p2p for
              example) use to make every package an ack package which would
              cause them all into here. We want only packets WITHOUT payload
              to match, so the size limit. Bigger packets just take their
              normal way into the classes.</para>

              <note>
                <para>This option is only valid for <emphasis
                role="bold">class</emphasis> per interface.</para>
              </note>
            </listitem>

            <listitem>
              <para>occurs=<emphasis>number</emphasis> - Typically used with
              an IPMARK entry in tcrules. Causes the rule to be replicated for
              a total of <emphasis>number</emphasis> rules. Each rule has a
              successively class number and mark value.</para>

              <para>When 'occurs' is used:</para>

              <itemizedlist>
                <listitem>
                  <para>The associated device may not have the 'classify'
                  option.</para>
                </listitem>

                <listitem>
                  <para>The class may not be the default class.</para>
                </listitem>

                <listitem>
                  <para>The class may not have any 'tos=' options (including
                  'tcp-ack').</para>
                </listitem>

                <listitem>
                  <para>The class should not specify a MARK value. If one is
                  specified, it will be ignored with a warning message.</para>
                </listitem>
              </itemizedlist>

              <para>The 'RATE' and 'CEIL' parameters apply to each instance of
              the class. So the total RATE represented by an entry with
              'occurs' will be the listed RATE multiplied by
              <emphasis>number</emphasis>. For additional information, see
              <ulink url="manpages/shorewall-tcrules.html">tcrules</ulink>
              (5).</para>
            </listitem>

            <listitem>
              <para>flow=<emphasis>keys</emphasis> - Shorewall attaches an SFQ
              queuing discipline to each leaf HTB and HFSC class. SFQ ensures
              that each <firstterm>flow</firstterm> gets equal access to the
              interface. The default definition of a flow corresponds roughly
              to a Netfilter connection. So if one internal system is running
              BitTorrent, for example, it can have lots of 'flows' and can
              thus take up a larger share of the bandwidth than a system
              having only a single active connection. The
              <option>flow</option> classifier (module cls_flow) works around
              this by letting you define what a 'flow' is. The clasifier must
              be used carefully or it can block off all traffic on an
              interface! The flow option can be specified for an HTB or HFSC
              leaf class (one that has no sub-classes). We recommend that you
              use the following:</para>

              <simplelist>
                <member>Shaping internet-bound traffic: <emphasis
                role="bold">flow=nfct-src</emphasis></member>

                <member>Shaping traffic bound for your local net: <emphasis
                role="bold">flow=dst</emphasis></member>
              </simplelist>

              <para>These will cause a 'flow' to consists of the traffic
              to/from each internal system.</para>

              <para>When more than one key is give, they must be enclosed in
              parenthesis and separated by commas.</para>

              <para>To see a list of the possible flow keys, run this
              command:</para>

              <blockquote>
                <para><command>tc filter add flow help</command></para>
              </blockquote>

              <para>Those that begin with "nfct-" are Netfilter connection
              tracking fields. As shown above, we recommend flow=nfct-src;
              that means that we want to use the source IP address
              <emphasis>before SNAT</emphasis> as the key.</para>

              <note>
                <para>Shorewall cannot determine ahead of time if the flow
                classifier is available in your kernel (especially if it was
                built into the kernel as opposed to being loaded as a module).
                Consequently, you should check ahead of time to ensure that
                both your kernel and 'tc' utility support the feature.</para>

                <para>You can test the 'tc' utility by typing (as
                root):</para>

                <blockquote>
                  <para><command>tc filter add flow help</command></para>
                </blockquote>

                <para>If flow is supported, you will see:</para>

                <programlisting>   Usage: ... flow ...

      [mapping mode]: map key KEY [ OPS ] ...
      [hashing mode]: hash keys KEY-LIST ...

   ...</programlisting>

                <para>If 'flow' is not supported, you will see:</para>

                <programlisting>   Unknown filter "flow", hence option "help" is unparsable</programlisting>

                <para>If your kernel supports module autoloading, just type
                (as root):</para>

                <blockquote>
                  <para><command>modprobe cls_flow</command></para>
                </blockquote>

                <para>If 'flow' is supported, no output is produced;
                otherwise, you will see:</para>

                <programlisting>   FATAL: Module cls_flow not found.</programlisting>

                <para>If your kernel is not modularized or does not support
                module autoloading, look at your kernel configuration (either
                <filename>/proc/config.gz</filename> or the
                <filename>.config</filename> file in <filename
                class="directory">/lib/modules/&lt;kernel-version&gt;/build/</filename></para>

                <para>If 'flow' is supported, you will see: NET_CLS_FLOW=m or
                NET_CLS_FLOW=y.</para>

                <para>For modularized kernels, Shorewall will attempt to load
                <filename>/lib/modules/&lt;kernel-version&gt;/net/sched/cls_flow.ko</filename>
                by default.</para>
              </note>
            </listitem>

            <listitem>
              <para>pfifo - When specified for a leaf class, the pfifo queing
              discipline is applied to the class rather than the sfq queuing
              discipline.</para>
            </listitem>

            <listitem>
              <para>limit=<emphasis>number</emphasis> - Added in Shorewall
              4.4.3. When specified for a leaf class, specifies the maximum
              number of packets that may be queued within the class. The
              <emphasis>number</emphasis> must be &gt; 2 and less than 128. If
              not specified, the value 127 is assumed</para>
            </listitem>
          </itemizedlist>
        </listitem>
      </itemizedlist>
    </section>

    <section id="tcrules">
      <title>/etc/shorewall/tcrules</title>

      <important>
        <para>Unlike rules in the <ulink
        url="manpages/shorewall-rules.html">shorewall-rules</ulink>(5) file,
        evaluation of rules in this file will continue after a match. So the
        final mark for each packet will be the one assigned by the LAST tcrule
        that matches.</para>

        <para>Also unlike rules in the <ulink
        url="manpages/shorewall-rules.html">shorewall-rules</ulink>(5) file,
        the tcrules file is not stateful. So every packet that goes into, out
        of or through your firewall is subject to entries in the tcrules
        file.</para>

        <para>Because tcrules are not stateful, it is necessary to understand
        basic IP socket operation. Here is an edited excerpt from a post on
        the Shorewall Users list:<blockquote>
            <para>For the purposes of this discussion, the world is separated
            into clients and servers. Servers provide services to
            clients.</para>

            <para>When a server starts, it creates a socket and
            <emphasis>binds</emphasis> the socket to an
            <emphasis>address</emphasis>. For AF_INET (IPv4) and AF_INET6
            (IPv6) sockets, that address is an ordered triple consisting of an
            IPv4 or IPv6 address, a protocol, and possibly a port number. Port
            numbers are only used when the protocol is TCP, UDP, SCTP or DCCP.
            The protocol and port number used by a server are typically
            well-known so that clients will be able to connect to it or send
            datagrams to it. So SSH servers bind to TCP port 22, SMTP servers
            bind to TCP port 25, etc. We will call this port the SERVER
            PORT.</para>

            <para>When a client want to use the service provided by a server,
            it also creates a socket and, like the server's socket, the
            client's socket must be bound to an address. But in the case of
            the client, the socket is usually given an automatic address
            binding. For AF_INET and AF_INET6 sockets. the IP address is the
            IP address of the client system (loose generalization) and the
            port number is selected from a <firstterm>local port
            range</firstterm>. On Linux systems, the local port range can be
            seen by <command>cat
            /proc/sys/net/ipv4/ip_local_port_range</command>. So it is not
            possible in advance to determine what port the client will be
            using. Whatever it is, we'll call it the CLIENT PORT.</para>

            <para>Now: <blockquote>
                <para>Packets sent from the client to the server will
                have:<blockquote>
                    <para>SOURCE PORT = CLIENT PORT</para>

                    <para>DEST PORT = SERVER PORT</para>
                  </blockquote></para>

                <para>Packets sent from the server to the client will have:
                <blockquote>
                    <para>SOURCE PORT = SERVER PORT</para>

                    <para>DEST PORT = CLIENT PORT</para>
                  </blockquote></para>
              </blockquote></para>

            <para>Since the SERVER PORT is generally the only port known ahead
            of time, we must categorize traffic from the server to the client
            using the SOURCE PORT.</para>
          </blockquote></para>
      </important>

      <para>The fwmark classifier provides a convenient way to classify
      packets for traffic shaping. The <quote>/etc/shorewall/tcrules</quote>
      file is used for specifying these marks in a tabular fashion. For an
      in-depth look at the packet marking facility in Netfilter/Shorewall,
      please see <ulink url="PacketMarking.html">this article</ulink>.</para>

      <para><emphasis role="bold">For marking forwarded traffic, you must
      either set MARK_IN_FORWARD_CHAIN=Yes shorewall.conf or by using the :F
      qualifier (see below).</emphasis></para>

      <para>Columns in the file are as follows:</para>

      <itemizedlist>
        <listitem>
          <para>MARK or CLASSIFY - MARK specifies the mark value is to be
          assigned in case of a match. This is an integer in the range 1-255
          (1-16383 if you set WIDE_TC_MARKS=Yes in <ulink
          url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5)
          ).</para>

          <note>
            <para>In Shorewall 4.5.0, WIDE_TC_MARKS was superseded by TC_BITS
            which specifies the width in bits of the traffic shaping mark
            field. The default is based on the setting of WIDE_TC_MARKS so as
            to provide upward compatibility.</para>
          </note>

          <para>This value may be optionally followed by <quote>:</quote> and
          either <quote>F</quote>, <quote>P</quote> or "T" to designate that
          the marking will occur in the FORWARD, PREROUTING or POSTROUTING
          chains respectively. If this additional specification is omitted,
          the chain used to mark packets will be determined as follows:</para>

          <itemizedlist>
            <listitem>
              <para>If the SOURCE is
              $FW[:&lt;<emphasis>address</emphasis>&gt;], then the rule is
              inserted in the OUTPUT chain.</para>
            </listitem>

            <listitem>
              <para>Otherwise, the chain is determined by the setting of the
              MARK_IN_FORWARD_CHAIN option in shorewall.conf.</para>
            </listitem>
          </itemizedlist>

          <note>
            <para><emphasis role="bold">Use the 'T' qualifier if you want the
            rule to apply equally to traffic being routed through the firewall
            and to traffic originating on the firewall
            itself.</emphasis></para>
          </note>

          <para>Normally, the mark is applied to the packet. If you follow the
          mark value with ":" and "C", then the mark is applied to the
          connection. "C" can be combined with "F", "P" or "T" to designate
          that the connection should be marked in a particular chain (e.g.,
          "CF", "CP", "CT").</para>

          <para>There are additional special values available:</para>

          <orderedlist numeration="loweralpha">
            <listitem>
              <para><emphasis
              role="bold">RESTORE</emphasis>[/<emphasis>mask</emphasis>] --
              restore the packet's mark from the connection's mark using the
              supplied mask if any. Your kernel and iptables must include
              CONNMARK support.</para>

              <para>As above, may be followed by <emphasis
              role="bold">:P</emphasis>, <emphasis role="bold">:F</emphasis>
              or <emphasis role="bold">:T</emphasis>.</para>
            </listitem>

            <listitem>
              <para><emphasis
              role="bold">SAVE</emphasis>[/<emphasis>mask</emphasis>] -- save
              the packet's mark to the connection's mark using the supplied
              mask if any. Your kernel and iptables must include CONNMARK
              support.</para>

              <para>As above, may be followed by <emphasis
              role="bold">:P</emphasis>, <emphasis role="bold">:F</emphasis>
              or <emphasis role="bold">:T</emphasis>.</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">CONTINUE</emphasis> Don't process
              any more marking rules in the table.</para>

              <para>As above, may be followed by <emphasis
              role="bold">:P</emphasis>, <emphasis role="bold">:F</emphasis>
              or <emphasis role="bold">:T</emphasis>.</para>
            </listitem>

            <listitem>
              <para><emphasis role="bold">COMMENT</emphasis> -- the rest of
              the line will be attached as a comment to the Netfilter rule(s)
              generated by the following entries. The comment will appear
              delimited by "/* ... */" in the output of <command>shorewall
              show mangle</command></para>

              <para>To stop the comment from being attached to further rules,
              simply include COMMENT on a line by itself.</para>
            </listitem>
          </orderedlist>

          <para>To use CLASSIFY, your kernel and iptables must include
          CLASSIFY target support. In that case, this column contains a
          classification (classid) of the form &lt;major&gt;:&lt;minor&gt;
          where &lt;major&gt; and &lt;minor&gt; are integers. Corresponds to
          the 'class' specification in these traffic shaping modules:</para>

          <simplelist>
            <member>atm</member>

            <member>cbq</member>

            <member>dsmark</member>

            <member>pfifo_fast</member>

            <member>htb</member>

            <member>prio</member>
          </simplelist>

          <para>Classification occurs in the POSTROUTING chain <emphasis
          role="bold">except</emphasis> when the SOURCE contains
          $FW[:&lt;<emphasis>address</emphasis>&gt;] in which case, the
          classify action takes place in the OUTPUT chain. When used with the
          builtin traffic shaper, the &lt;major&gt; class is the interface
          number and the &lt;minor&gt; class is either:</para>

          <orderedlist>
            <listitem>
              <para>Constructed by Shorewall. The method of construction
              depends on the setting of WIDE_TC_MARKS (<ulink
              url="manpages/shorewall.conf.html">shorewall.conf</ulink>
              (5)).</para>

              <para>When WIDE_TC_MARKS=No (the default), the &lt;minor&gt;
              class is:</para>

              <itemizedlist>
                <listitem>
                  <para>the MARK value of the class preceded by the number "1"
                  or "10" (MARK value 1 is &lt;minor&gt; class 11, MARK value
                  22 is &lt;minor&gt; class 122, and so on). "10" is used
                  where there are more than 10 devices defined in <link
                  linkend="tcdevices">/etc/shorewall/tcdevices</link>.</para>
                </listitem>
              </itemizedlist>

              <para>When WIDE_TC_MARKS=Yes, the &lt;minor&gt; class is
              assigned sequentially beginning with 2.</para>
            </listitem>

            <listitem>
              <para>The class number, if specified.</para>
            </listitem>
          </orderedlist>
        </listitem>

        <listitem>
          <para>SOURCE - Source of the packet.</para>

          <para>May be:</para>

          <orderedlist>
            <listitem>
              <para>An interface name - matches traffic entering the firewall
              on the specified interface. May not be used in classify rules or
              in rules using the :T chain qualifier.</para>
            </listitem>

            <listitem>
              <para>A comma-separated list of host or network IP addresses or
              MAC addresses. <emphasis role="bold">This form will not match
              traffic that originates on the firewall itself unless either
              &lt;major&gt;&lt;minor&gt; or the :T chain qualifier is used in
              the MARK column.</emphasis></para>

              <para>Examples:<simplelist>
                  <member>0.0.0.0/0</member>
                </simplelist></para>

              <para><simplelist>
                  <member>192.168.1.0/24, 172.20.4.0/24</member>
                </simplelist></para>
            </listitem>

            <listitem>
              <para>An interface name followed by a colon (":") followed by a
              comma-separated list of host or network IP addresses or MAC
              addresses. May not be used in classify rules or in rules using
              the :T chain qualifier.</para>
            </listitem>

            <listitem>
              <para>$FW optionally followed by a colon (":") and a
              comma-separated list of host or network IP addresses. matches
              packets originating on the firewall. May not be used with a
              chain qualifier (:P, :F, etc.) in the MARK column.</para>
            </listitem>
          </orderedlist>

          <para>MAC addresses must be prefixed with "~" and use "-" as a
          separator.</para>

          <para>Example: ~00-A0-C9-15-39-78</para>

          <para>If your kernel includes iprange match support, then address
          ranges may be included in the address lists.</para>
        </listitem>

        <listitem>
          <para>DEST - Destination of the packet.</para>

          <para>May be:</para>

          <orderedlist>
            <listitem>
              <para>An interface name. May not be used in the PREROUTING chain
              (:P in the mark column or no chain qualifier and
              MARK_IN_FORWARD_CHAIN=No in <ulink
              url="manpages/shorewall.conf">shorewall.conf</ulink> (5)). The
              interface name may be optionally followed by a colon (":") and
              an IP address list.</para>
            </listitem>

            <listitem>
              <para>A comma-separated list of host or network IP addresses.
              The list may include ip address ranges if your kernel and
              iptables include iprange support.</para>
            </listitem>
          </orderedlist>
        </listitem>

        <listitem>
          <para>PROTO - Protocol - Must be "tcp", "udp", "icmp", "ipp2p",
          "ipp2p:udp", "ipp2p:all" a number, or "all". "ipp2p" requires ipp2p
          match support in your kernel and iptables.</para>
        </listitem>

        <listitem>
          <para>PORT(S) - Destination Ports. A comma-separated list of Port
          names (from /etc/services), port numbers or port ranges; if the
          protocol is "icmp", this column is interpreted as the destination
          icmp-type(s).</para>

          <para>If the protocol is ipp2p, this column is interpreted as an
          ipp2p option without the leading "--" (example "bit" for
          bit-torrent). If no PORT is given, "ipp2p" is assumed. Note that the
          xtables-addons version of IPP2P does not support the "ipp2p" option;
          if the column is empty or contains "ipp2p" when using that version
          of IPP2P, Shorewall will substitute "edk,kazaa,gnu,dc".</para>

          <para>This column is ignored if PROTOCOL = all but must be entered
          if any of the following field is supplied. In that case, it is
          suggested that this field contain "-"</para>
        </listitem>

        <listitem>
          <para>CLIENT PORT(S) - (Optional) Port(s) used by the client. If
          omitted, any source port is acceptable. Specified as a
          comma-separate list of port names, port numbers or port
          ranges.</para>
        </listitem>

        <listitem>
          <para>USER/GROUP (Optional) This column may only be non-empty if the
          SOURCE is the firewall itself. When this column is non-empty, the
          rule applies only if the program generating the output is running
          under the effective user and/or group. It may contain :</para>

          <para>[!][&lt;user name or number&gt;]:[&lt;group name or
          number&gt;][+&lt;program name&gt;]</para>

          <para>The colon is optional when specifying only a user.</para>

          <para>Examples:</para>

          <programlisting>joe     #program must be run by joe
:kids   #program must be run by a member of the 'kids' group
!:kids  #program must not be run by a member of the 'kids' group
+upnpd  #program named upnpd (This feature was removed from Netfilter in kernel version 2.6.14).</programlisting>
        </listitem>

        <listitem>
          <para>TEST (Optional) Defines a test on the existing packet or
          connection mark. The rule will match only if the test returns true.
          Tests have the format [!]&lt;value&gt;[/&lt;mask&gt;][:C]</para>

          <para>Where:</para>

          <simplelist>
            <member>! Inverts the test (not equal)</member>

            <member>&lt;value&gt; Value of the packet or connection
            mark.</member>

            <member>&lt;mask&gt; A mask to be applied to the mark before
            testing</member>

            <member>:C Designates a connection mark. If omitted, the packet
            mark's value is tested.</member>
          </simplelist>
        </listitem>

        <listitem>
          <para>LENGTH (Optional) This field, if present, allows you to match
          the length of a packet against a specific value or range of values.
          A range is specified in the form &lt;min&gt;:&lt;max&gt; where
          either &lt;min&gt; or &lt;max&gt; (but not both) may be omitted. If
          &lt;min&gt; is omitted, then 0 is assumed; if &lt;max&gt; is
          omitted, than any packet that is &lt;min&gt; or longer will
          match.</para>

          <para>You must have iptables length support for this to work. If you
          let it empty or place an "-" here, no length match will be
          done.</para>

          <para>Examples: 1024, 64:1500, :100</para>
        </listitem>

        <listitem>
          <para>TOS (Optional) Type of Service. Either a standard name, or a
          numeric value to match.</para>

          <blockquote>
            <simplelist>
              <member>Minimize-Delay (16)</member>

              <member>Maximize-Throughput (8)</member>

              <member>Maximize-Reliability (4)</member>

              <member>Minimize-Cost (2)</member>

              <member>Normal-Service (0)</member>
            </simplelist>
          </blockquote>
        </listitem>

        <listitem>
          <para>HELPER (Optional). Names one of the Netfilter protocol helper
          modules such as <emphasis>ftp</emphasis>, <emphasis>sip</emphasis>,
          <emphasis>amanda</emphasis>, etc.</para>
        </listitem>

        <listitem>
          <para>HEADERS (Optioinal, Shorewall6 only, added in Shorewall
          4.4.15). List of IPv6 headers that may appear in packets. See <ulink
          url="manpages6/shorewall6-tcrules.html">shorewall6-tcrules</ulink>
          (5) for details.</para>
        </listitem>
      </itemizedlist>

      <example id="Example1">
        <title></title>

        <para>All packets arriving on eth1 should be marked with 1. All
        packets arriving on eth2 and eth3 should be marked with 2. All packets
        originating on the firewall itself should be marked with 3.</para>

        <programlisting>#MARK   SOURCE    DESTINATION    PROTOCOL   PORT(S)
1       eth1      0.0.0.0/0      all
2       eth2      0.0.0.0/0      all
2       eth3      0.0.0.0/0      all
3       $FW       0.0.0.0/0      all</programlisting>
      </example>

      <example id="Example2">
        <title></title>

        <para>All GRE (protocol 47) packets destined for 155.186.235.151
        should be marked with 12.</para>

        <programlisting>#MARK   SOURCE    DESTINATION     PROTOCOL   PORT(S)
12:T    0.0.0.0/0 155.182.235.151 47</programlisting>
      </example>

      <example id="Example3">
        <title></title>

        <para>All SSH request packets originating in 192.168.1.0/24 and
        destined for 155.186.235.151 should be marked with 22.</para>

        <programlisting>#MARK   SOURCE         DESTINATION     PROTOCOL   PORT(S)
22:T    192.168.1.0/24 155.182.235.151 tcp        22</programlisting>
      </example>

      <example id="Example4">
        <title></title>

        <para>All SSH packets packets going out of the first device in in
        /etc/shorewall/tcdevices should be assigned to the class with mark
        value 10.</para>

        <programlisting>#MARK   SOURCE         DESTINATION     PROTOCOL   PORT(S)         CLIENT
#                                                                 PORT(S)
1:110   0.0.0.0/0      0.0.0.0/0       tcp        22
1:110   0.0.0.0/0      0.0.0.0/0       tcp        -               22</programlisting>
      </example>

      <example id="Example5">
        <title></title>

        <para>Mark all ICMP echo traffic with packet mark 1. Mark all peer to
        peer traffic with packet mark 4.</para>

        <para>This is a little more complex than otherwise expected. Since the
        ipp2p module is unable to determine all packets in a connection are
        P2P packets, we mark the entire connection as P2P if any of the
        packets are determined to match. We assume packet/connection mark 0 to
        means unclassified. Traffic originating on the firewall is not covered
        by this example.</para>

        <programlisting>#MARK    SOURCE         DESTINATION     PROTOCOL   PORT(S)       CLIENT   USER/     TEST
#                                                                PORT(S)  GROUP
1        0.0.0.0/0      0.0.0.0/0       icmp       echo-request
1        0.0.0.0/0      0.0.0.0/0       icmp       echo-reply

RESTORE  0.0.0.0/0      0.0.0.0/0       all        -             -        -         0
CONTINUE 0.0.0.0/0      0.0.0.0/0       all        -             -        -         !0
4        0.0.0.0/0      0.0.0.0/0       ipp2p:all
SAVE     0.0.0.0/0      0.0.0.0/0       all        -             -        -         !0</programlisting>

        <para>The last four rules can be translated as:</para>

        <blockquote>
          <para>"If a packet hasn't been classified (packet mark is 0), copy
          the connection mark to the packet mark. If the packet mark is set,
          we're done. If the packet is P2P, set the packet mark to 4. If the
          packet mark has been set, save it to the connection mark."</para>
        </blockquote>
      </example>

      <example>
        <title></title>

        <para>Mark all forwarded VOIP connections with connection mark 1 and
        ensure that all VOIP packets also receive that mark (assumes that
        nf_conntrack_sip is loaded).</para>

        <programlisting>#MARK    SOURCE         DESTINATION     PROTOCOL   PORT(S)       CLIENT   USER/     TEST      CONNBYTES      TOS      HELPER
#                                                                PORT(S)  GROUP
RESTORE  0.0.0.0/0      0.0.0.0/0       all        -             -        -         0
CONTINUE 0.0.0.0/0      0.0.0.0/0       all        -             -        -         !0
1        0.0.0.0/0      0.0.0.0/0       all        -             -        -         -         -              -        sip
SAVE     0.0.0.0/0      0.0.0.0/0       all        -             -        -         !0</programlisting>
      </example>
    </section>

    <section id="ppp">
      <title>ppp devices</title>

      <para>If you use ppp/pppoe/pppoa) to connect to your Internet provider
      and you use traffic shaping you need to restart shorewall traffic
      shaping. The reason for this is, that if the ppp connection gets
      restarted (and it usually does this at least daily), all
      <quote>tc</quote> filters/qdiscs related to that interface are
      deleted.</para>

      <para>The easiest way to achieve this, is just to restart shorewall once
      the link is up. To achieve this add a small executable script
      to<quote>/etc/ppp/ip-up.d</quote>.</para>

      <programlisting>#! /bin/sh

/sbin/shorewall refresh</programlisting>
    </section>

    <section>
      <title>Sharing a TC configuration between Shorewall and
      Shorewall6</title>

      <para>Beginning with Shorewall 4.4.15, the traffic-shaping configuration
      in the tcdevices, tcclasses and tcfilters files can be shared between
      Shorewall and Shorewall6. Only one of the products can control the
      configuration but the other can configure CLASSIFY rules in its own
      tcrules file that refer to the shared classes.</para>

      <para>To defined the configuration in Shorewall and shared it with
      Shorewall6:</para>

      <itemizedlist>
        <listitem>
          <para>Set TC_ENABLED=Internal in <ulink
          url="manpages/shorewall.conf.html">shorewall.conf</ulink>
          (5).</para>
        </listitem>

        <listitem>
          <para>Set TC_ENABLED=SHARED in <ulink
          url="manpages6/shorewall6.conf.html">shorewall6.conf</ulink>
          (5).</para>
        </listitem>

        <listitem>
          <para>Create symbolic links from /etc/shorewall6 to
          /etc/shorewall/tcdevices and /etc/shorewall/tcclasses:</para>

          <programlisting>ln -s ../shorewall/tcdevices /etc/shorewall6/tcdevices
ln -s ../shorewall/tcclasses /etc/shorewall6/tcclasses</programlisting>
        </listitem>

        <listitem>
          <para>If you need to define IPv6 tcfilter entries, do so in
          /etc/shorewall/tcfilters. That file now allows entries that apply to
          IPv6.</para>
        </listitem>
      </itemizedlist>

      <para>Shorewall6 compilations to have access to the tcdevices and
      tcclasses files although it will create no output. That access allows
      CLASSIFY rules in /etc/shorewall6/tcrules to be validated against the TC
      configuration.</para>

      <para>In this configuration, it is Shorewall that controls TC
      configuration (except for IPv6 tcrules). You can reverse the settings in
      the files if you want to control the configuration using
      Shorewall6.</para>
    </section>

    <section id="perIP">
      <title>Per-IP Traffic Shaping</title>

      <para>Some network administrators feel that they have to divy up their
      available bandwidth by IP address rather than by prioritizing the
      traffic based on the type of traffic. This gets really awkward when
      there are a large number of local IP addresses.</para>

      <para>This section describes the Shorewall facility for making this
      configuration less tedious (and a lot more efficient). Note that it
      requires that you <ulink url="Dynamic.html#xtables-addons">install
      xtables-addons</ulink>. So before you try this facility, we suggest that
      first you add the following OPTION to each external interface described
      in /etc/shorewall/tcdevices:</para>

      <programlisting>flow=nfct-src</programlisting>

      <para>If you shape traffic on your internal interface(s), then add this
      to their entries:</para>

      <programlisting>flow=dst</programlisting>

      <para>You may find that this simple change is all that is needed to
      control bandwidth hogs like Bit Torrent. If it doesn't, then proceed as
      described in this section.</para>

      <para>The facility has two components:</para>

      <orderedlist>
        <listitem>
          <para>An IPMARK MARKing command in
          <filename>/etc/shorewall/tcrules</filename>.</para>
        </listitem>

        <listitem>
          <para>An <emphasis role="bold">occurs</emphasis> OPTION in
          /etc/shorewall/tcclasses.</para>
        </listitem>
      </orderedlist>

      <para>The facility is currently only available with IPv4.</para>

      <para>In a sense, the IPMARK target is more like an IPCLASSIFY target in
      that the mark value is later interpreted as a class ID. A packet mark is
      32 bits wide; so is a class ID. The <emphasis>major</emphasis> class
      occupies the high-order 16 bits and the <emphasis>minor</emphasis> class
      occupies the low-order 16 bits. So the class ID 1:4ff (remember that
      class IDs are always in hex) is equivalent to a mark value of 0x104ff.
      Remember that Shorewall uses the interface number as the
      <emphasis>major</emphasis> number where the first interface in tcdevices
      has <emphasis>major</emphasis> number 1, the second has
      <emphasis>major</emphasis> number 2, and so on.</para>

      <para>The IPMARK target assigns a mark to each matching packet based on
      the either the source or destination IP address. By default, it assigns
      a mark value equal to the low-order 8 bits of the source address.</para>

      <para>The syntax is as follows:</para>

      <blockquote>
        <para><emphasis role="bold">IPMARK</emphasis>[<emphasis
        role="bold">(</emphasis>[{<emphasis
        role="bold">src</emphasis>|<emphasis
        role="bold">dst</emphasis>}][<emphasis
        role="bold">,</emphasis>[<emphasis>mask1</emphasis>][,[<emphasis>mask2</emphasis>][<emphasis
        role="bold">,</emphasis>[<emphasis>shift</emphasis>]]]]<emphasis
        role="bold">)</emphasis>]</para>
      </blockquote>

      <para>Default values are:</para>

      <simplelist>
        <member>src</member>

        <member>mask1 = 0xFF</member>

        <member>mask2 = 0x00</member>

        <member>shift = 0</member>
      </simplelist>

      <para><emphasis role="bold">src</emphasis> and <emphasis
      role="bold">dst</emphasis> specify whether the mark is to be based on
      the source or destination address respectively. The selected address is
      first shifted right by <emphasis>shift</emphasis>, then LANDed with
      <emphasis>mask1</emphasis> and then LORed with
      <emphasis>mask2</emphasis>. The <emphasis>shift</emphasis> argument is
      intended to be used primarily with IPv6 addresses.</para>

      <para>Example:</para>

      <programlisting>IPMARK(src,0xff,0x10100)

Source IP address is 192.168.4.3 = 0xc0a80403

        0xc0a80403 &gt;&gt; 0         = 0xc0a80403
        0xc0a80403 LAND 0xFF    = 0x03
        0x03       LOR  0x10100 = 0x10103

        So the mark value is 0x10103 which corresponds to class id 1:103.</programlisting>

      <para>It is important to realize that, while class IDs are composed of a
      <emphasis>major</emphasis> and a <emphasis>minor</emphasis> value, the
      set of <emphasis>minor</emphasis> values must be unique. You must keep
      this in mind when deciding how to map IP addresses to class IDs. For
      example, suppose that your internal network is 192.168.1.0/29 (host IP
      addresses 192.168.1.1 - 192.168.1.6). Your first notion might be to use
      IPMARK(src,0xFF,0x10000) so as to produce class IDs 1:1 through 1:6. But
      1:1 is the class ID of the base HTB class on interface 1. So you might
      chose instead to use IPMARK(src,0xFF,0x10100) as shown in the example
      above so as to avoid minor class 1.</para>

      <para>The <emphasis role="bold">occurs</emphasis> option in
      <filename>/etc/shorewall/tcclasses</filename> causes the class
      definition to be replicated many times.</para>

      <para>The synax is:</para>

      <blockquote>
        <para><emphasis
        role="bold">occurs=</emphasis><emphasis>number</emphasis></para>
      </blockquote>

      <para>When <emphasis role="bold">occurs</emphasis> is used:</para>

      <orderedlist numeration="loweralpha">
        <listitem>
          <para>The associated device may not have the <emphasis
          role="bold">classify</emphasis> option.</para>
        </listitem>

        <listitem>
          <para>The class may not be the default class.</para>
        </listitem>

        <listitem>
          <para>The class may not have any <emphasis
          role="bold">tos=</emphasis> options (including <emphasis
          role="bold">tcp-ack</emphasis>).</para>
        </listitem>
      </orderedlist>

      <para>The class should not specify a MARK value. Any MARK value given is
      ignored with a warning. The RATE and CEIL parameters apply to each
      instance of the class. So the total RATE represented by an entry with
      <emphasis role="bold">occurs</emphasis> will be the listed RATE
      multiplied by <emphasis>number</emphasis>.</para>

      <para>Example:</para>

      <para><filename>/etc/shorewall/tcdevices</filename>:</para>

      <programlisting>#INTERFACE IN-BANDWIDTH OUT-BANDWIDTH
eth0       100mbit      100mbit</programlisting>

      <para><filename>/etc/shorewall/tcclasses</filename>:</para>

      <programlisting>#DEVICE   MARK RATE     CEIL PRIORITY    OPTIONS
eth0:101     - 1kbit 230kbit        4    occurs=6</programlisting>

      <para>The above defines 6 classes with class IDs 0x101-0x106. Each class
      has a guaranteed rate of 1kbit/second and a ceiling of 230kbit.</para>

      <para><filename>/etc/shoreall/tcrules</filename>:</para>

      <programlisting>#MARK                      SOURCE         DEST
IPMARK(src,0xff,0x10100):F 192.168.1.0/29 eth0</programlisting>

      <para>This facility also alters the way in which Shorewall generates a
      class number when none is given. Prior to the implementation of this
      facility, the class number was constructed by concatinating the MARK
      value with the either '1' or '10'. '10' was used when there were more
      than 10 devices defined in
      <filename>/etc/shorewall/tcdevices</filename>.</para>

      <para>With this facility, a new method is added; class numbers are
      assigned sequentially beginning with 2. The WIDE_TC_MARKS option in
      <filename>shorewall.conf</filename> selects which construction to use.
      WIDE_TC_MARKS=No (the default) produces pre-Shorewall 4.4 behavior.
      WIDE_TC_MARKS=Yes produces the new behavior.</para>
    </section>

    <section id="Real">
      <title>Real life examples</title>

      <section>
        <title>A Shorewall User's Experience</title>

        <para>Chuck Kollars has provided <ulink
        url="http://www.ckollars.org/shaping.html">an excellent
        writeup</ulink> about his traffic shaping experiences.</para>
      </section>

      <section id="Wondershaper">
        <title>Configuration to replace Wondershaper</title>

        <para>You are able to fully replace the wondershaper script by using
        the buitin traffic control.. In this example it is assumed that your
        interface for your Internet connection is ppp0 (for DSL), if you use
        another connection type, you have to change it. You also need to
        change the settings in the tcdevices.wondershaper file to reflect your
        line speed. The relevant lines of the config files follow here. Please
        note that this is just a 1:1 replacement doing exactly what
        wondershaper should do. You are free to change it...</para>

        <section id="realtcd">
          <title>tcdevices file</title>

          <programlisting>#INTERFACE    IN-BANDWITH      OUT-BANDWIDTH
ppp0            5000kbit        500kbit</programlisting>
        </section>

        <section id="realtcc">
          <title>tcclasses file</title>

          <programlisting>#INTERFACE      MARK    RATE            CEIL        PRIORITY    OPTIONS
ppp0            1       5*full/10       full            1       tcp-ack,tos-minimize-delay
ppp0            2       3*full/10       9*full/10       2       default
ppp0            3       2*full/10       8*full/10       2</programlisting>
        </section>

        <section id="realtcr">
          <title>tcrules file</title>

          <programlisting>#MARK           SOURCE          DEST            PROTO   PORT(S) CLIENT   USER
#                                                              PORT(S)
1:F             0.0.0.0/0       0.0.0.0/0       icmp    echo-request
1:F             0.0.0.0/0       0.0.0.0/0       icmp    echo-reply
# mark traffic which should have a lower priority with a 3:
# mldonkey
3               0.0.0.0/0       0.0.0.0/0       udp     -        4666</programlisting>

          <para>Wondershaper allows you to define a set of hosts and/or ports
          you want to classify as low priority. To achieve this , you have to
          add these hosts to tcrules and set the mark to 3 (true if you use
          the example configuration files).</para>
        </section>

        <section id="lowpro">
          <title>Setting hosts to low priority</title>

          <para>lets assume the following settings from your old wondershaper
          script (don't assume these example values are really useful, they
          are only used for demonstrating ;-):</para>

          <programlisting>
# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks
NOPRIOHOSTSRC="192.168.1.128/25 192.168.3.28"

# low priority destination netmasks
NOPRIOHOSTDST=60.0.0.0/24

# low priority source ports
NOPRIOPORTSRC="6662 6663"

# low priority destination ports
NOPRIOPORTDST="6662 6663"  </programlisting>

          <para>This would result in the following additional settings to the
          tcrules file:</para>

          <programlisting>3               192.168.1.128/25 0.0.0.0/0      all
3               192.168.3.28     0.0.0.0/0      all
3               0.0.0.0/0        60.0.0.0/24    all
3               0.0.0.0/0        0.0.0.0/0      udp     6662,6663
3               0.0.0.0/0        0.0.0.0/0      udp     -         6662,6663
3               0.0.0.0/0        0.0.0.0/0      tcp     6662,6663
3               0.0.0.0/0        0.0.0.0/0      tcp     -         6662,6663</programlisting>
        </section>
      </section>

      <section id="simiple">
        <title>A simple setup</title>

        <para>This is a simple setup for people sharing an Internet connection
        and using different computers for this. It just basically shapes
        between 2 hosts which have the ip addresses 192.168.2.23 and
        192.168.2.42</para>

        <section id="simpletcd">
          <title>tcdevices file</title>

          <programlisting>#INTERFACE    IN-BANDWITH      OUT-BANDWIDTH
ppp0            6000kbit        700kbit</programlisting>

          <para>We have 6mbit down and 700kbit upstream.</para>
        </section>

        <section id="simpletcc">
          <title>tcclasses file</title>

          <programlisting>#INTERFACE      MARK    RATE            CEIL            PRIORITY    OPTIONS
ppp0            1       10kbit          50kbit          1           tcp-ack,tos-minimize-delay
ppp0            2       300kbit         full            2
ppp0            3       300kbit         full            2
ppp0            4       90kbit          200kbit         3           default</programlisting>

          <para>We add a class for tcp ack packets with highest priority, so
          that downloads are fast. The following 2 classes share most of the
          bandwidth between the 2 hosts, if the connection is idle, they may
          use full speed. As the hosts should be treated equally they have the
          same priority. The last class is for the remaining traffic.</para>
        </section>

        <section id="simpletcr">
          <title>tcrules file</title>

          <programlisting>#MARK           SOURCE          DEST            PROTO   PORT(S)      CLIENT   USER
#                                                                    PORT(S)
1:F             0.0.0.0/0       0.0.0.0/0       icmp    echo-request
1:F             0.0.0.0/0       0.0.0.0/0       icmp    echo-reply
2:F             192.168.2.23    0.0.0.0/0       all
3:F             192.168.2.42    0.0.0.0/0       all</programlisting>

          <para>We mark icmp ping and replies so they will go into the fast
          interactive class and set a mark for each host.</para>
        </section>
      </section>
    </section>
  </section>

  <section id="Xen">
    <title>A Warning to Xen Users</title>

    <para>If you are running traffic shaping in your dom0 and traffic shaping
    doesn't seem to be limiting outgoing traffic properly, it may be due to
    "checksum offloading" in your domU(s). Check the output of "shorewall show
    tc". Here's an excerpt from the output of that command:</para>

    <programlisting>class htb 1:130 parent 1:1 leaf 130: prio 3 quantum 1500 rate 76000bit <emphasis
        role="bold">ceil 230000bit</emphasis> burst 1537b/8 mpu 0b overhead 0b cburst 1614b/8 mpu 0b overhead 0b level 0 
 Sent 559018700 bytes 75324 pkt (dropped 0, overlimits 0 requeues 0) 
 <emphasis role="bold">rate 299288bit</emphasis> 3pps backlog 0b 0p requeues 0 
 lended: 53963 borrowed: 21361 <emphasis role="bold">giants: 90174</emphasis>
 tokens: -26688 ctokens: -14783</programlisting>

    <para>There are two obvious problems in the above output:</para>

    <orderedlist>
      <listitem>
        <para>The rate (299288) is considerably larger than the ceiling
        (230000).</para>
      </listitem>

      <listitem>
        <para>There are a large number (90174) of giants reported.</para>
      </listitem>
    </orderedlist>

    <para>This problem will be corrected by disabling "checksum offloading" in
    your domU(s) using the <command>ethtool</command> utility. See the <ulink
    url="XenMyWay-Routed.html">one of the Xen articles</ulink> for
    instructions.</para>
  </section>

  <section id="HFSC">
    <title>An HFSC Example</title>

    <para>As mentioned at the top of this article, there is an excellent
    introduction to HFSC at <ulink
    url="http://linux-ip.net/articles/hfsc.en/">http://linux-ip.net/articles/hfsc.en/</ulink>.
    At the end of that article are 'tc' commands that implement the
    configuration in the article. Those tc commands correspond to the
    following Shorewall traffic shaping configuration.</para>

    <para><filename>/etc/shorewall/tcdevices</filename>:</para>

    <programlisting>#INTERFACE    IN-BANDWITH      OUT-BANDWIDTH          OPTIONS
eth0          -                1000kbit               hfsc</programlisting>

    <para><filename>/etc/shorewall/tcclasses</filename>:</para>

    <programlisting>#INTERFACE:CLASS     MARK      RATE:              CEIL     PRIORITY      OPTIONS
#                              DMAX:UMAX
1:10                 1         500kbit            full     1
1:20                 2         500kbit            full     1
1:10:11              3         400kbit:53ms:1500b full     2
1:10:12              4         100kbit:30ms:1500b full     2</programlisting>

    <para>The following sub-section offers some notes about the
    article.</para>

    <section id="MajicNumbers">
      <title>Where Did all of those Magic Numbers come from?</title>

      <para>As you read the article, numbers seem to be introduced out of thin
      air. I'll try to shed some light on those.</para>

      <para>There is very clear development of these numbers:</para>

      <itemizedlist>
        <listitem>
          <para>12ms to transfer a 1500b packet at 1000kbits/second.</para>
        </listitem>

        <listitem>
          <para>100kbits per second with 1500b packets, requires 8 packets per
          second.</para>
        </listitem>

        <listitem>
          <para>A packet from class 1:12 must be sent every 120ms.</para>
        </listitem>

        <listitem>
          <para>Total transmit delay can be no more than 132ms (120 +
          12).</para>
        </listitem>
      </itemizedlist>

      <para>We then learn that the queuing latency can be reduced to 30ms if
      we use a two-part service curve whose first part is 400kbits/second.
      Where did those come from?</para>

      <itemizedlist>
        <listitem>
          <para>The latency is calculated from the rate. If it takes 12ms to
          transmit a 1500 byte packet at 1000kbits/second, it takes 30ms to
          transmit a 1500b at 400kbits/second.</para>
        </listitem>

        <listitem>
          <para>For the slope of the first part of the service curve, in
          theory we can pick any number between 100 (the rate of class 1:12)
          and 500 (the rate of the parent class) with higher numbers providing
          lower latency.</para>
        </listitem>
      </itemizedlist>

      <para>The final curious number is the latency for class 1:11 - 52.5ms.
      It is a consequence of everything that has gone before.</para>

      <para>To acheive 400kbits/second with 1500-byte packets, 33.33 packets
      per second are required. So a packet from class 1:11 must be sent every
      30 ms. As the article says, "...the maximum transmission delay of this
      class increases from 30ms to a total of 52.5 ms.". So we are looking for
      an additional 22.5 ms.</para>

      <para>Assume that both class 1:11 and 1:12 transmit for 30 ms at
      400kbits/second. That is a total of 800kbits/second for 30ms. So Class
      1:11 is punished for the excess. How long is the punishment? The two
      classes sent 24,000 bits in 30ms; they are only allowed 0.030 * 500,000
      = 15,000. So they are 9,000 bits over their quota. The amount of time
      required to transmit 9,000 bits at 400,000 bits/second is
      22.5ms!.</para>
    </section>
  </section>

  <section id="IFB">
    <title>Intermediate Functional Block (IFB) Devices</title>

    <para>The principles behind an IFB is fairly simple:</para>

    <itemizedlist>
      <listitem>
        <para>It looks like a network interface although it is never given an
        IPv4 configuration.</para>
      </listitem>

      <listitem>
        <para>Because it is a network interface, queuing disciplines can be
        associated with an IFB.</para>
      </listitem>
    </itemizedlist>

    <para>The magic of an IFB comes in the fact that a filter may be defined
    on a real network interface such that each packet that arrives on that
    interface is queued for the IFB! In that way, the IFB provides a means for
    shaping input traffic.</para>

    <para>To use an IFB, you must have IFB support in your kernel
    (configuration option CONFIG_IFB). Assuming that you have a modular
    kernel, the name of the IFB module is 'ifb' and may be loaded using the
    command <command>modprobe ifb</command> (if you have modprobe installed)
    or <command>insmod /path/to/module/ifb</command>.</para>

    <para>By default, two IFB devices (ifb0 and ifb1) are created. You can
    control that using the numifbs option (e.g., <command>modprobe ifb
    numifbs=1</command>).</para>

    <para>To create a single IFB when Shorewall starts, place the following
    two commands in <filename>/etc/shorewall/init</filename>:</para>

    <programlisting><command>modprobe ifb numifbs=1
ip link set ifb0 up</command></programlisting>

    <para>Entries in <filename>/etc/shorewall/tcrules</filename> have no
    effect on shaping traffic through an IFB. To allow classification of such
    traffic, the /etc/shorewall/tcfilters file has been added. Entries in that
    file create <ulink url="http://b42.cz/notes/u32_classifier/">u32
    classification rules</ulink>.</para>

    <section id="tcfilters">
      <title>/etc/shorewall/tcfilters</title>

      <para>While this file was created to allow shaping of traffic through an
      IFB, the file may be used for general traffic classification as well.
      The file is similar to <ulink
      url="shorewall-tcrules.html">shorewall-tcrules</ulink>(5) with the
      following key exceptions:</para>

      <itemizedlist>
        <listitem>
          <para>The first match determines the classification, whereas in the
          tcrules file, the last match determines the classification.</para>
        </listitem>

        <listitem>
          <para>ipsets are not supported</para>
        </listitem>

        <listitem>
          <para>DNS Names are not supported</para>
        </listitem>

        <listitem>
          <para>Address ranges and lists are not supported</para>
        </listitem>

        <listitem>
          <para>Exclusion is not supported.</para>
        </listitem>

        <listitem>
          <para>filters are applied to packets as they <emphasis>appear on the
          wire</emphasis>. So incoming packets will not have DNAT applied yet
          (the destination IP address will be the external address) and
          outgoing packets will have had SNAT applied.</para>
        </listitem>
      </itemizedlist>

      <para>The last point warrants elaboration. When looking at traffic being
      shaped by an IFB, there are two cases to consider:</para>

      <orderedlist>
        <listitem>
          <para>Requests — packets being sent from remote clients to local
          servers. These packets may undergo subsequent DNAT, either as a
          result of entries in <filename>/etc/shorewall/nat</filename> or as a
          result of DNAT or REDIRECT rules.</para>

          <para>Example: <filename>/etc/shorewall/rules</filename>:</para>

          <programlisting>#ACTION       SOURCE           DEST            PROTO    DEST            SOURCE         ORIGINAL
#                                                       PORT(S)         PORT(S)        DEST
DNAT          net              dmz:192.168.4.5 tcp      80              -              206.124.146.177</programlisting>

          <para>Requests redirected by this rule will have destination IP
          address 206.124.146.177 and destination port 80.</para>
        </listitem>

        <listitem>
          <para>Responses — packets being sent from remote servers to local
          clients. These packets may undergo subsequent DNAT as a result of
          entries in <filename>/etc/shorewall/nat</filename> or in
          <filename>/etc/shorewall/masq</filename>. The packet's destination
          IP address will be the external address specified in the
          entry.</para>

          <para>Example:
          <filename>/etc/shorewall/masq</filename>:<programlisting>#INTERFACE        SOURCE           ADDRESS
eth0              192.168.1.0/24   206.124.146.179</programlisting></para>

          <para>HTTP response packets corresponding to requests that fall
          under that rule will have destination IP address 206.124.146.179 and
          <emphasis role="bold">source</emphasis> port 80.</para>
        </listitem>
      </orderedlist>

      <para>Beginning with Shorewall 4.4.15, both IPv4 and IPv6 rules can be
      defined in this file. See <ulink
      url="manpages/shorewall-tcfilters.html">shorewall-tcfilters</ulink> (5)
      for details.</para>

      <para>Columns in the file are as follow. As in all Shorewall
      configuration files, a hyphen ("-") may be used to indicate that no
      value is supplied in the column.</para>

      <variablelist>
        <varlistentry>
          <term>CLASS</term>

          <listitem>
            <para>The interface name or number followed by a colon (":") and
            the class number.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>SOURCE</term>

          <listitem>
            <para>SOURCE IP address (host or network). DNS names are not
            allowed.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>DEST</term>

          <listitem>
            <para>DESTINATION IP address (host or network). DNS names are not
            allowed.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>PROTO</term>

          <listitem>
            <para>Protocol name or number.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>DEST PORT(S)</term>

          <listitem>
            <para>Comma-separated list of destination port names or numbers.
            May only be specified if the protocol is TCP, UDP, SCTP or ICMP.
            Port ranges are supported except for ICMP.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>SOURCE PORT</term>

          <listitem>
            <para>Comma-separated list of source port names or numbers. May
            only be specified if the protocol is TCP, UDP or SCTP. Port ranges
            are supported.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>TOS</term>

          <listitem>
            <para>Specifies the value of the TOS field. The value can be any
            of the following:</para>

            <itemizedlist>
              <listitem>
                <para><option>tos-minimize-delay</option></para>
              </listitem>

              <listitem>
                <para><option>tos-maximuze-throughput</option></para>
              </listitem>

              <listitem>
                <para><option>tos-maximize-reliability</option></para>
              </listitem>

              <listitem>
                <para><option>tos-minimize-cost</option></para>
              </listitem>

              <listitem>
                <para><option>tos-normal-service</option></para>
              </listitem>

              <listitem>
                <para><replaceable>hex-number</replaceable></para>
              </listitem>

              <listitem>
                <para><replaceable>hex-number</replaceable>/<replaceable>hex-number</replaceable></para>
              </listitem>
            </itemizedlist>

            <para>The <replaceable>hex-number</replaceable>s must be exactly
            two digits (e.g., 0x04).</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>LENGTH</term>

          <listitem>
            <para>Must be a power of 2 between 32 and 8192 inclusive. Packets
            with a total length that is strictly less than the specified value
            will match the rule.</para>
          </listitem>
        </varlistentry>
      </variablelist>

      <para>Example:</para>

      <para>I've used this configuration on my own firewall. The IFB portion
      is more for test purposes rather than to serve any well-reasoned QOS
      strategy.</para>

      <para><filename>/etc/shorewall/init</filename>:<programlisting>qt modprobe ifb numifbs=1
qt ip link set dev ifb0 up</programlisting></para>

      <para><filename>/etc/shorewall/interfaces</filename>:</para>

      <programlisting>#ZONE          INTERFACE         BROADCAST
-              ifb0</programlisting>

      <para><filename>/etc/shorewall/tcdevices</filename>:</para>

      <para><programlisting>
#INTERFACE      IN-BANDWITH     OUT-BANDWIDTH   OPTIONS         REDIRECTED
#                                                               INTERFACES
1:eth0          -               384kbit         classify
2:ifb0          -               1300kbit        -               eth0</programlisting>
      <filename>/etc/shorewall/tcclasses</filename>:<programlisting>#INTERFACE      MARK    RATE            CEIL            PRIORITY        OPTIONS
1:110           -       5*full/10       full            1               tcp-ack,tos-minimize-delay
1:120           -       2*full/10       6*full/10       2               default
1:130           -       2*full/10       6*full/10       3
2:110           -       5*full/10       full            1               tcp-ack,tos-minimize-delay
2:120           -       2*full/10       6*full/10       2               default
2:130           -       2*full/10       6*full/10       3</programlisting><filename>/etc/shorewall/tcfilters</filename>:<programlisting>#INTERFACE:     SOURCE          DEST            PROTO   DEST    SOURCE
#CLASS                                                  PORT(S) PORT(S)
#
#                                  OUTGOING TRAFFIC
#
1:130           206.124.146.178 -               tcp     -       49441,49442    #BITTORRENT on wookie
1:110           206.124.146.178                                                #wookie
1:110           206.124.146.179                                                #SNAT of internal systems
1:110           206.124.146.180                                                #Work Laptop
1:110           -               -               icmp    echo-request,echo-reply
1:110           -               -               icmp    echo-reply
1:130           206.124.146.177 -               tcp     -       873,25         #Bulk Traffic
#
#                                   INCOMING TRAFFIC
#
2:110           -               206.124.146.178                          #Wookie
2:110           -               206.124.146.179                          #SNAT Responses
2:110           -               206.124.146.180                          #Work Laptop
2:130           -               206.124.146.177 tcp     25               #Incoming Email.</programlisting></para>

      <para>You can examine the installed filters with the <command>shorewall
      show filters</command> command. What follows shows the output for
      <filename class="devicefile">eth0</filename> with the filters shown
      above. <emphasis role="bold">Bold font</emphasis> are comments
      explaining the rules.<programlisting>gateway:~ # shorewall-lite show filters
Shorewall Lite 4.1.6 Classifiers at gateway - Fri Mar 21 08:06:47 PDT 2008

Device eth1:

Device eth2:

Device eth0:
filter parent 1: protocol ip pref 10 u32 
filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 3:</emphasis> ht divisor 1 <emphasis
            role="bold">  &lt;========= Start of table 3. parses TCP header</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 3</emphasis>::800 order 2048 key ht 3 bkt 0 <emphasis
            role="bold">flowid 1:130</emphasis>  (rule hit 102 success 0)
  match 03690000/ffff0000 at nexthdr+0 (success 0 )        <emphasis
            role="bold">   &lt;========= SOURCE PORT 873 goes to class 1:130</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 2:</emphasis> ht divisor 1 <emphasis
            role="bold">  &lt;========= Start of table 2. parses ICMP header</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 2</emphasis>::800 order 2048 key ht 2 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 0 success 0)
  match 08000000/ff000000 at nexthdr+0 (success 0 )        <emphasis
            role="bold">   &lt;========= ICMP Type 8 goes to class 1:110</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 2</emphasis>::801 order 2049 key ht 2 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 0 success 0)
  match 00000000/ff000000 at nexthdr+0 (success 0 )         <emphasis
            role="bold">  &lt;========= ICMP Type 0 goes to class 1:110</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 1:</emphasis> ht divisor 1 <emphasis
            role="bold">  &lt;========= Start of table 1. parses TCP header</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 1:</emphasis>:800 order 2048 key ht 1 bkt 0 <emphasis
            role="bold">flowid 1:130</emphasis>  (rule hit 0 success 0)
  match c1210000/ffff0000 at nexthdr+0 (success 0 )         <emphasis
            role="bold">  &lt;========= SOURCE PORT 49441 goes to class 1:130</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 1</emphasis>::801 order 2049 key ht 1 bkt 0 <emphasis
            role="bold">flowid 1:130</emphasis>  (rule hit 0 success 0)
  match c1220000/ffff0000 at nexthdr+0 (success 0 )         <emphasis
            role="bold">  &lt;========= SOURCE PORT 49442 goes to class 1:130</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis> ht divisor 1 <emphasis
            role="bold">&lt;========= Start of Table 800. Packets start here!</emphasis>

   <emphasis role="bold">=============== The following 2 rules are generated by the class definition in /etc/shorewall/classes ==================</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:800 order 2048 key ht 800 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 2204 success 138)
  match 00060000/00ff0000 at 8 (success 396 )                 <emphasis
            role="bold">&lt;========= TCP   </emphasis> 
  match 05000000/0f00ffc0 at 0 (success 250 )                 <emphasis
            role="bold">&lt;========= Header length 20 and Packet Length &lt; 64</emphasis> 
  match 00100000/00ff0000 at 32 (success 138 )                <emphasis
            role="bold">&lt;========= ACK</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:801 order 2049 key ht 800 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 2066 success 0)
  match 00100000/00100000 at 0 (success 0 )                  <emphasis
            role="bold">&lt;========= Minimize-delay</emphasis><emphasis
            role="bold"> goes to class 1:110</emphasis>

   <emphasis role="bold">                     =============== Jump to Table 1 if the matches are met ==================</emphasis>
 
filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:802 order 2050 key ht 800 bkt 0 <emphasis
            role="bold">link 1:</emphasis>  (rule hit 2066 success 0)
  match ce7c92b2/ffffffff at 12 (success 1039 )              <emphasis
            role="bold">&lt;========= SOURCE 206.124.146.178 </emphasis>         
  match 00060000/00ff0000 at 8 (success 0 )                  <emphasis
            role="bold">&lt;========= PROTO TCP</emphasis>
    offset 0f00&gt;&gt;6 at 0  eat 

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:803 order 2051 key ht 800 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 2066 success 1039)
  match ce7c92b2/ffffffff at 12 (success 1039 )               <emphasis
            role="bold">&lt;========= SOURCE 206.124.146.178 goes to class 1:110</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:804 order 2052 key ht 800 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 1027 success 132)
  match ce7c92b3/ffffffff at 12 (success 132 )               <emphasis
            role="bold"> &lt;========= SOURCE 206.124.146.179 goes to class 1:110</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:805 order 2053 key ht 800 bkt 0 <emphasis
            role="bold">flowid 1:110</emphasis>  (rule hit 895 success 603)
  match ce7c92b4/ffffffff at 12 (success 603 )                <emphasis
            role="bold">&lt;========= SOURCE 206.124.146.180 goes to class 1:110</emphasis>

   <emphasis role="bold">                     =============== Jump to Table 2 if the matches are met ==================</emphasis>

filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:806 order 2054 key ht 800 bkt 0 <emphasis
            role="bold">link 2:</emphasis>  (rule hit 292 success 0)
  match 00010000/00ff0000 at 8 (success 0 )                   <emphasis
            role="bold">&lt;========= PROTO ICMP</emphasis> 
    offset 0f00&gt;&gt;6 at 0  eat

   <emphasis role="bold">                     =============== Jump to Table 3 if the matches are met ==================</emphasis>
 
filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:807 order 2055 key ht 800 bkt 0 <emphasis
            role="bold">link 3:</emphasis>  (rule hit 292 success 0)
  match ce7c92b1/ffffffff at 12 (success 265 )                <emphasis
            role="bold">&lt;========= SOURCE 206.124.146.177</emphasis>
  match 00060000/00ff0000 at 8 (success 102 )                 <emphasis
            role="bold">&lt;========= PROTO TCP</emphasis>
    offset 0f00&gt;&gt;6 at 0  eat </programlisting></para>
    </section>
  </section>

  <section id="show">
    <title>Understanding the output of 'shorewall show tc'</title>

    <para>The <command>shorewall show tc</command> (<command>shorewall-lite
    show tc</command>) command displays information about the current state of
    traffic shaping. For each device, it executes the following
    commands:</para>

    <programlisting> echo Device $device:
 tc -s -d qdisc show dev $device
 echo
 tc -s -d class show dev $device
 echo </programlisting>

    <para>So, the traffic-shaping output is generated entirely by the
    <command>tc</command> utility.</para>

    <para>Here's the output of for eth0. The configuration is the one shown in
    the preceding section (the output was obtained almost 24 hours later than
    the <command>shorewall show filters</command> output shown above).</para>

    <programlisting>Device eth0:

<emphasis role="bold">       ============== The primary queuing discipline is HTB (Hierarchical Token Bucket) ==================== </emphasis>

qdisc htb 1: r2q 10 default 120 direct_packets_stat 9 ver 3.17
 Sent 2133336743 bytes 4484781 pkt (dropped 198, overlimits 4911403 requeues 21) <emphasis
        role="bold">&lt;=========== Note the overlimits and dropped counts</emphasis>
 rate 0bit 0pps backlog 0b 8p requeues 21

<emphasis role="bold">============== The ingress filter. If you specify IN-BANDWIDTH, you can see the 'dropped' count here. =========</emphasis>

                       <emphasis role="bold">In this case, the packets are being sent to the IFB for shaping</emphasis>
 
qdisc ingress ffff: ---------------- 
 Sent 4069015112 bytes 4997252 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0

 <emphasis role="bold">============ Each of the leaf HTB classes has an SFQ qdisc to ensure that each flow gets its turn ============</emphasis>
 
qdisc sfq 110: parent 1:110 limit 128p quantum 1514b flows 128/1024 perturb 10sec 
 Sent 613372519 bytes 2870225 pkt (dropped 0, overlimits 0 requeues 6) 
 rate 0bit 0pps backlog 0b 0p requeues 6 
qdisc sfq 120: parent 1:120 limit 128p quantum 1514b flows 128/1024 perturb 10sec 
 Sent 18434920 bytes 60961 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 130: parent 1:130 limit 128p quantum 1514b flows 128/1024 perturb 10sec 
 Sent 1501528722 bytes 1553586 pkt (dropped 198, overlimits 0 requeues 15) 
 rate 0bit 0pps backlog 11706b 8p requeues 15 

                           <emphasis role="bold">============= Class 1:110 -- the high-priority class ===========


                                   Note the rate and ceiling calculated from 'full'</emphasis>

class htb <emphasis role="bold">1:110</emphasis> parent 1:1 leaf 110: prio 1 quantum 4800 <emphasis
        role="bold">rate 192000bit ceil 384000bit</emphasis> burst 1695b/8 mpu 0b overhead 0b cburst 1791b/8 mpu 0b overhead 0b level 0 
 Sent 613372519 bytes 2870225 pkt (dropped 0, overlimits 0 requeues 0) 
 <emphasis role="bold">rate 195672bit 28pps backlog 0b 0p</emphasis> requeues 0 <emphasis
        role="bold">&lt;=========== Note the current rate of traffic. There is no queuing (no packet backlog)</emphasis> 
 lended: 2758458 borrowed: 111773 giants:
 tokens: 46263 ctokens: 35157

                                      <emphasis role="bold">============== The root class ============</emphasis>

class htb <emphasis role="bold">1:1 root</emphasis> rate 384000bit ceil 384000bit burst 1791b/8 mpu 0b overhead 0b cburst 1791b/8 mpu 0b overhead 0b level 7 
 <emphasis role="bold">Sent 2133276316 bytes 4484785 pkt</emphasis> (dropped 0, overlimits 0 requeues 0) <emphasis
        role="bold">&lt;==== Total output traffic since last 'restart'</emphasis>
 rate 363240bit 45pps backlog 0b 0p requeues 0 
 lended: 1081936 borrowed: 0 giants: 0
 tokens: -52226 ctokens: -52226

                      <emphasis role="bold">============= Bulk Class (outgoing rsync, email and bittorrent) ============</emphasis>

class htb 1:130 parent 1:1 leaf 130: prio 3 quantum 1900 rate 76000bit ceil 230000bit burst 1637b/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
 Sent 1501528722 bytes 1553586 pkt (dropped 198, overlimits 0 requeues 0) 
 <emphasis role="bold">rate 162528bit 14pps backlog 0b 8p</emphasis> requeues 0 <emphasis
        role="bold">&lt;======== Queuing is occurring (8 packet backlog). The rate is still below the ceiling.</emphasis>
 lended: 587134 borrowed: 966459 giants: 0               <emphasis role="bold">During peak activity, the rate tops out at around 231000 (just above ceiling).</emphasis>
 tokens: -30919 ctokens: -97657

                       <emphasis role="bold">================= Default class (mostly serving web pages) ===============</emphasis>

class htb 1:120 parent 1:1 leaf 120: prio 2 quantum 1900 rate 76000bit ceil 230000bit burst 1637b/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 
 Sent 18434920 bytes 60961 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 2240bit 2pps backlog 0b 0p requeues 0 
 lended: 57257 borrowed: 3704 giants: 0
 tokens: 156045 ctokens: 54178

</programlisting>
  </section>

  <section id="External">
    <title id="tcstart">Using your own tc script</title>

    <section id="owntcstart">
      <title>Replacing builtin tcstart file</title>

      <para>If you prefer your own tcstart file, just install it in
      /etc/shorewall/tcstart.</para>

      <para>In your tcstart script, when you want to run the <quote>tc</quote>
      utility, use the run_tc function supplied by Shorewall if you want tc
      errors to stop the firewall.</para>

      <orderedlist>
        <listitem>
          <para>Set TC_ENABLED=Yes and CLEAR_TC=Yes</para>
        </listitem>

        <listitem>
          <para>Supply an /etc/shorewall/tcstart script to configure your
          traffic shaping rules.</para>
        </listitem>

        <listitem>
          <para>Optionally supply an /etc/shorewall/tcclear script to stop
          traffic shaping. That is usually unnecessary.</para>
        </listitem>

        <listitem>
          <para>If your tcstart script uses the <quote>fwmark</quote>
          classifier, you can mark packets using entries in
          /etc/shorewall/tcrules.</para>
        </listitem>
      </orderedlist>
    </section>

    <section id="Start">
      <title>Traffic control outside Shorewall</title>

      <para>To start traffic shaping when you bring up your network
      interfaces, you will have to arrange for your traffic shaping
      configuration script to be run at that time. How you do that is
      distribution dependent and will not be covered here. You then
      should:</para>

      <orderedlist>
        <listitem>
          <para>Set TC_ENABLED=No and CLEAR_TC=No</para>
        </listitem>

        <listitem>
          <para>If your script uses the <quote>fwmark</quote> classifier, you
          can mark packets using entries in /etc/shorewall/tcrules.</para>
        </listitem>
      </orderedlist>
    </section>
  </section>

  <section id="Testing">
    <title>Testing Tools</title>

    <para>At least one Shorewall user has found this tool helpful: <ulink
    url="http://e2epi.internet2.edu/network-performance-toolkit.html">http://e2epi.internet2.edu/network-performance-toolkit.html</ulink></para>
  </section>
</article>