<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> <article> <!--$Id$--> <articleinfo> <title>Traffic Shaping/Control</title> <authorgroup> <author> <firstname>Tom</firstname> <surname>Eastep</surname> </author> <author> <firstname>Arne</firstname> <surname>Bernin</surname> </author> </authorgroup> <pubdate><?dbtimestamp format="Y/m/d"?></pubdate> <copyright> <year>2001-2008</year> <holder>Thomas M. Eastep</holder> </copyright> <copyright> <year>2005</year> <holder>Arne Bernin & Thomas M. Eastep</holder> </copyright> <legalnotice> <para>Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled <quote><ulink url="GnuCopyright.htm">GNU Free Documentation License</ulink></quote>.</para> </legalnotice> </articleinfo> <important> <para>Traffic shaping is complex and the Shorewall community is not well equipped to answer traffic shaping questions. So if you are the type of person who needs "insert tab A into slot B" instructions for everything that you do, then please don't try to implement traffic shaping using Shorewall. You will just frustrate yourself and we won't be able to help you.</para> </important> <warning> <para>Said another way, reading just Shorewall documentation is not going to give you enough background to use this material.</para> <para>At a minimum, you will need to refer to at least the following additional information:</para> <itemizedlist> <listitem> <para>The LARTC HOWTO: <ulink url="http://www.lartc.org">http://www.lartc.org</ulink></para> </listitem> <listitem> <para>The HTB User's Guide: <ulink url="http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm">http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm</ulink></para> </listitem> <listitem> <para>Some of the documents listed at <ulink url="http://www.netfilter.org/documentation/index.html#documentation-howto">http://www.netfilter.org/documentation/index.html#documentation-howto</ulink>. The tutorial by Oskar Andreasson is particularly good.</para> </listitem> <listitem> <para>The output of <command>man iptables</command></para> </listitem> </itemizedlist> </warning> <section id="Intro"> <title>Introduction</title> <para>Shorewall has builtin support for traffic shaping and control. Before this version, the support was quite limited. You were able to use your own tcstart script (and you still are), but besides the tcrules file it was not possible to define classes or queuing disciplines inside the Shorewall config files.</para> <para>The support for traffic shaping and control still does not cover all options available (and especially all algorithms that can be used to queue traffic) in the Linux kernel but it should fit most needs. If you are using your own script for traffic control and you still want to use it in the future, you will find information on how to do this, <link linkend="owntcstart"> later in this document</link>. But for this to work, you will also need to enable traffic shaping in the kernel and Shorewall as covered by the next sections.</para> </section> <section id="LinuxTC"> <title>Linux traffic shaping and control</title> <para>This section gives a brief introduction of how controlling traffic with the Linux kernel works. Although this might be enough for configuring it in the Shorewall configuration files, we strongly recommend that you take a deeper look into the <ulink url="http://lartc.org/howto/">Linux Advanced Routing and Shaping HOWTO</ulink>. At the time of writing this, the current version is 1.0.0.</para> <para>Since kernel 2.2 Linux has extensive support for controlling traffic. You can define different algorithms that are used to queue the traffic before it leaves an interface. The standard one is called pfifo and is (as the name suggests) of the type First In First out. This means, that it does not shape anything, if you have a connection that eats up all your bandwidth, this queuing algorithm will not stop it from doing so.</para> <para>For Shorewall traffic shaping we use two algorithms, one is called HTB (Hierarchical Token Bucket) and SFQ (Stochastic Fairness Queuing). SFQ is easy to explain: it just tries to track your connections (tcp or udp streams) and balances the traffic between them. This normally works well. HTB allows you to define a set of classes, and you can put the traffic you want into these classes. You can define minimum and maximum bandwidth settings for those classes and order them hierarchically (the less prioritized classes only get bandwidth if the more important have what they need). Shorewall builtin traffic shaping allows you to define these classes (and their bandwidth limits), and it uses SFQ inside these classes to make sure, that different data streams are handled equally.</para> <para>You can shape incoming traffic through use of an <firstterm>Intermediate Frame Block</firstterm> (IFB) device. <link linkend="IFB">See below</link>. <emphasis role="bold">But beware: using an IFB can result in queues building up both at your ISPs router and at your own.</emphasis></para> <para>If you wish to shape downloads, you can also configure traffic shaping on your firewall's local interface. An example appears <link linkend="Downloads">below</link>. Again, however, <emphasis role="bold">this can result in queues building up both at your ISPs router and at your own</emphasis>.</para> <para>You shape and control outgoing traffic by assigning the traffic to <firstterm>classes</firstterm>. Each class is associated with exactly one network interface and has a number of attributes:</para> <orderedlist> <listitem> <para>PRIORITY - Used to give preference to one class over another when selecting a packet to send. The priority is a numeric value with 1 being the highest priority, 2 being the next highest, and so on.</para> </listitem> <listitem> <para>RATE - The minimum bandwidth this class should get, when the traffic load rises. Classes with a higher priority (lower PRIORITY value) are served even if there are others that have a guaranteed bandwidth but have a lower priority (higher PRIORITY value).</para> </listitem> <listitem> <para>CEIL - The maximum bandwidth the class is allowed to use when the link is idle.</para> </listitem> <listitem> <para>MARK - Netfilter has a facility for <firstterm>marking</firstterm> packets. Packet marks have a numeric value which is limited in Shorewall to the values 1-255 (1-16383 if you set WIDE_TC_MARKS=Yes in <ulink url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5) ). You assign packet marks to different types of traffic using entries in the <filename>/etc/shorewall/tcrules</filename> file.</para> </listitem> </orderedlist> <para>One class for each interface must be designated as the <firstterm>default class</firstterm>. This is the class to which unmarked traffic (packets to which you have not assigned a mark value in <filename>/etc/shorewall/tcrules</filename>) is assigned.</para> <para>Netfilter also supports a mark value on each connection. You can assign connection mark values in <filename>/etc/shorewall/tcrules</filename>, you can copy the current packet's mark to the connection mark (SAVE), or you can copy the connection mark value to the current packet's mark (RESTORE). For more information, see<ulink url="PacketMarking.html"> this article</ulink>.</para> </section> <section id="Kernel"> <title>Linux Kernel Configuration</title> <para>You will need at least kernel 2.4.18 for this to work, please take a look at the following screenshot for what settings you need to enable. For builtin support, you need the HTB scheduler, the Ingress scheduler, the PRIO pseudoscheduler and SFQ queue. The other scheduler or queue algorithms are not needed.</para> <para>This screen shot shows how I configured QoS in a 2.6.16 Kernel:</para> <graphic align="center" fileref="images/traffic_shaping2.6.png" /> <para>And here's my recommendation for a 2.6.21 kernel:<graphic align="center" fileref="images/traffic_shaping2.6.21.png" /></para> </section> <section id="Shorewall"> <title>Enable TC support in Shorewall</title> <para>You need this support whether you use the builtin support or whether you provide your own tcstart script.</para> <para>To enable the builtin traffic shaping and control in Shorewall, you have to do the following:</para> <itemizedlist> <listitem> <para>Set <emphasis role="bold">TC_ENABLED</emphasis> to "<emphasis role="bold">Internal</emphasis>" in /etc/shorewall/shorewall.conf. Setting <emphasis role="bold">TC_ENABLED=Yes</emphasis> causes Shorewall to look for an external tcstart file (See <link linkend="tcstart">a later section</link> for details).</para> </listitem> <listitem> <para>Setting <emphasis role="bold">CLEAR_TC</emphasis> parameter in /etc/shorewall/shorewall.conf to <emphasis role="bold">Yes</emphasis> will clear the traffic shaping configuration during Shorewall [re]start and Shorewall stop. This is normally what you want when using the builtin support (and also if you use your own tcstart script)</para> </listitem> <listitem> <para>The other steps that follow depend on whether you use your own script or the builtin solution. They will be explained in the following sections.</para> </listitem> </itemizedlist> </section> <section id="Builtin"> <title>Using builtin traffic shaping/control</title> <para>Shorewall's builtin traffic shaping feature provides a thin layer on top of the ingress qdesc, HTB and SFQ. That translation layer allows you to:</para> <itemizedlist> <listitem> <para>Define HTB classes using Shorewall-style column-oriented configuration files.</para> </listitem> <listitem> <para>Integrate the reloading of your traffic shaping configuration with the reloading of your packet-filtering and marking configuration.</para> </listitem> <listitem> <para>Assign traffic to HTB classes by TOS value.</para> </listitem> <listitem> <para>Assign outgoing TCP ACK packets to an HTB class.</para> </listitem> <listitem> <para>Assign traffic to HTB classes based on packet mark value.</para> </listitem> </itemizedlist> <warning> <para>Shorewall's builtin traffic shaping feature is limited to ten (10) devices.</para> </warning> <para>Those few features are really all that builtin traffic shaping/control provides; consequently, you need to understand HTB and Linux traffic shaping as well as Netfilter packet marking in order to use the facility. Again, please see the links at top of this article.</para> <para>For defining bandwidths (for either devices or classes) please use kbit or kbps (for Kilobytes per second) and make sure there is <emphasis role="bold">NO</emphasis> space between the number and the unit (it is 100kbit <emphasis role="bold">not</emphasis> 100 kbit). Using mbit, mbps or a raw number (which means bytes) could be used, but note that only integer numbers are supported (0.5 is <emphasis role="bold">not valid</emphasis>).</para> <para><emphasis role="bold">To properly configure the settings for your devices you need to find out the real up- and downstream rates you have</emphasis>. This is especially the case, if you are using a DSL connection or one of another type that do not have a guaranteed bandwidth. Don't trust the values your provider tells you for this; especially measuring the real download speed is important! There are several online tools that help you find out; search for "dsl speed test" on google (For Germany you can use <ulink url="http://www.speedcheck.arcor.de/cgi-bin/speedcheck.cgi">arcor speed check</ulink>). Be sure to choose a test located near you.</para> <section id="tcdevices"> <title>/etc/shorewall/tcdevices</title> <para>This file allows you to define the incoming and outgoing bandwidth for the devices you want traffic shaping to be enabled. That means, if you want to use traffic shaping for a device, you have to define it here.</para> <para>Columns in the file are as follows:</para> <itemizedlist> <listitem> <para>INTERFACE - Name of interface. Each interface may be listed only once in this file. You may NOT specify the name of an alias (e.g., eth0:0) here; see <ulink url="FAQ.htm#faq18">FAQ #18</ulink>. You man NOT specify wildcards here, e.g. if you have multiple ppp interfaces, you need to put them all in here! Shorewall will determine if the device exists and will only configure the device if it does exist. If it doesn't exist, the following warning is issued:</para> <para><emphasis role="bold">WARNING: Device <device name> not found -- traffic-shaping configuration skipped</emphasis></para> <para>Shorewall assigns a sequential <firstterm>interface number</firstterm> to each interface (the first entry in <filename>/etc/shorewall/tcdevices</filename> is interface 1, the second is interface 2 and so on) You can also explicitly specify the interface number by prefixing the interface name with the number and a colon (":"). Example: 1:eth0.</para> </listitem> <listitem> <para>IN-BANDWIDTH - The incoming Bandwidth of that interface. Please note that when you use this column, you are not traffic shaping incoming traffic, as the traffic is already received before you could do so. This Column allows you to define the maximum traffic allowed for this interface in total, if the rate is exceeded, the excess packets are dropped. You want this mainly if you have a DSL or Cable Connection to avoid queuing at your providers side. If you don't want any traffic to be dropped set this to a value faster than your interface maximum rate (or to 0 (zero).</para> <para>To determine the optimum value for this setting, we recommend that you start by setting it significantly below your measured download bandwidth (20% or so). While downloading, measure the <emphasis>ping</emphasis> response time from the firewall to the upstream router as you gradually increase the setting.The optimal setting is at the point beyond which the <emphasis>ping</emphasis> time increases sharply as you increase the setting.</para> </listitem> <listitem> <para>OUT-BANDWIDTH - Specify the outgoing bandwidth of that interface. This is the maximum speed your connection can handle. It is also the speed you can refer as "full" if you define the tc classes. Outgoing traffic above this rate will be dropped.</para> </listitem> <listitem> <para>OPTIONS — A comma-separated list of options from the following list:</para> <variablelist> <varlistentry> <term>classify</term> <listitem> <para>If specified, classification of traffic into the various classes is done by CLASSIFY entries in <filename>/etc/shorewall/tcrules</filename> or by entries in <filename>/etc/shorewall/tcfilters</filename>. No MARK value will be associated with classes on this interface.</para> </listitem> </varlistentry> </variablelist> </listitem> <listitem> <para>REDIRECTED INTERFACES — Entries are appropriate in this column only if the device in the INTERFACE column names a <link linkend="IFB">Intermediate Frame Block (IFB)</link>. It lists the physical interfaces that will have their input shaped using classes defined on the IFB. Neither the IFB nor any of the interfaces listed in this column may have an IN-BANDWIDTH specified. You may specify zero (0) or a dash ("-:) in the IN-BANDWIDTH column.</para> <para>IFB devices automatically get the <emphasis role="bold">classify</emphasis> option.</para> </listitem> </itemizedlist> <example id="Example0"> <title></title> <para>Suppose you are using PPP over Ethernet (DSL) and ppp0 is the interface for this. The device has an outgoing bandwidth of 500kbit and an incoming bandwidth of 6000kbit</para> <programlisting>#INTERFACE IN-BANDWITH OUT-BANDWIDTH ppp0 6000kbit 500kbit</programlisting> </example> </section> <section id="tcclasses"> <title>/etc/shorewall/tcclasses</title> <para>This file allows you to define the actual classes that are used to split the outgoing traffic.</para> <itemizedlist> <listitem> <para>INTERFACE - Name of interface. Users may also specify the interface number. Must match the name (or number) of an interface with an entry in <filename>/etc/shorewall/tcdevices</filename>. If the interface has the <emphasis role="bold">classify</emphasis> option in <filename>/etc/shorewall/tcdevices</filename>, then the interface name or number must be followed by a colon and a <firstterm>class number</firstterm>. Examples: eth0:1, 4:9. Class numbers must be unique for a given interface.</para> </listitem> <listitem> <para>MARK - The mark value which is an integer in the range 1-255 (1-16383 if you set WIDE_TC_MARKS=Yes in <ulink url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5) ). You define these marks in the tcrules file, marking the traffic you want to go into the queuing classes defined in here. You can use the same marks for different Interfaces. You must specify "-' in this column if the device specified in the INTERFACE column has the <emphasis role="bold">classify</emphasis> option in <filename>/etc/shorewall/tcdevices</filename>.</para> </listitem> <listitem> <para>RATE - The minimum bandwidth this class should get, when the traffic load rises. Please note that first the classes which equal or a lesser priority value are served even if there are others that have a guaranteed bandwidth but a lower priority. <emphasis role="bold">If the sum of the RATEs for all classes assigned to an INTERFACE exceed that interfaces's OUT-BANDWIDTH, then the OUT-BANDWIDTH limit will not be honored.</emphasis></para> </listitem> <listitem> <para>CEIL - The maximum bandwidth this class is allowed to use when the link is idle. Useful if you have traffic which can get full speed when more important services (e.g. interactive like ssh) are not used. You can use the value "full" in here for setting the maximum bandwidth to the defined output bandwidth of that interface.</para> </listitem> <listitem> <para>PRIORITY - you have to define a priority for the class. packets in a class with a higher priority (=lesser value) are handled before less prioritized ones. You can just define the mark value here also, if you are increasing the mark values with lesser priority.</para> </listitem> <listitem> <para>OPTIONS - A comma-separated list of options including the following:</para> <itemizedlist> <listitem> <para>default - this is the default class for that interface where all traffic should go, that is not classified otherwise.</para> <note> <para>defining default for exactly <emphasis role="bold">one</emphasis> class per interface is mandatory!</para> </note> </listitem> <listitem> <para>tos-<tosname> - this lets you define a filter for the given <tosname> which lets you define a value of the Type Of Service bits in the ip package which causes the package to go in this class. Please note, that this filter overrides all mark settings, so if you define a tos filter for a class all traffic having that mark will go in it regardless of the mark on the package. You can use the following for this option: tos-minimize-delay (16) tos-maximize-throughput (8) tos-maximize-reliability (4) tos-minimize-cost (2) tos-normal-service (0)</para> <note> <para>Each of this options is only valid for <emphasis role="bold">one</emphasis> class per interface.</para> </note> </listitem> <listitem> <para>tcp-ack - if defined causes an tc filter to be created that puts all tcp ack packets on that interface that have an size of <=64 Bytes to go in this class. This is useful for speeding up downloads. Please note that the size of the ack packets is limited to 64 bytes as some applications (p2p for example) use to make every package an ack package which would cause them all into here. We want only packets WITHOUT payload to match, so the size limit. Bigger packets just take their normal way into the classes.</para> <note> <para>This option is only valid for <emphasis role="bold">class</emphasis> per interface.</para> </note> </listitem> </itemizedlist> </listitem> </itemizedlist> </section> <section id="tcrules"> <title>/etc/shorewall/tcrules</title> <important> <para>Unlike rules in the <ulink url="manpages/shorewall-rules.html">shorewall-rules</ulink>(5) file, evaluation of rules in this file will continue after a match. So the final mark for each packet will be the one assigned by the LAST tcrule that matches.</para> <para>Also unlike rules in the <ulink url="manpages/shorewall-rules.html">shorewall-rules</ulink>(5) file, the tcrules file is not stateful. So every packet that goes into, out of or through your firewall is subject to entries in the tcrules file.</para> <para>Because tcrules are not stateful, it is necessary to understand basic IP socket operation. Here is an edited excerpt from a post on the Shorewall Users list:<blockquote> <para>For the purposes of this discussion, the world is separated into clients and servers. Servers provide services to clients.</para> <para>When a server starts, it creates a socket and <emphasis>binds</emphasis> the socket to an <emphasis>address</emphasis>. For AF_INET (IPv4) and AF_INET6 (IPv6) sockets, that address is an ordered triple consisting of an IPv4 or IPv6 address, a protocol, and possibly a port number. Port numbers are only used when the protocol is TCP, UDP, SCTP or DCCP. The protocol and port number used by a server are typically well-known so that clients will be able to connect to it or send datagrams to it. So SSH servers bind to TCP port 22, SMTP servers bind to TCP port 25, etc. We will call this port the SERVER PORT.</para> <para>When a client want to use the service provided by a server, it also creates a socket and, like the server's socket, the client's socket must be bound to an address. But in the case of the client, the socket is usually given an automatic address binding. For AF_INET and AF_INET6 sockets. the IP address is the IP address of the client system (loose generalization) and the port number is selected from a <firstterm>local port range</firstterm>. On Linux systems, the local port range can be seen by <command>cat /proc/sys/net/ipv4/ip_local_port_range</command>. So it is not possible in advance to determine what port the client will be using. Whatever it is, we'll call it the CLIENT PORT.</para> <para>Now: <blockquote> <para>Packets sent from the client to the server will have:<blockquote> <para>SOURCE PORT = CLIENT PORT</para> <para>DEST PORT = SERVER PORT</para> </blockquote></para> <para>Packets sent from the server to the client will have: <blockquote> <para>SOURCE PORT = SERVER PORT</para> <para>DEST PORT = CLIENT PORT</para> </blockquote></para> </blockquote></para> <para>Since the SERVER PORT is generally the only port known ahead of time, we must categorize traffic from the server to the client using the SOURCE PORT.</para> </blockquote></para> </important> <para>The fwmark classifier provides a convenient way to classify packets for traffic shaping. The <quote>/etc/shorewall/tcrules</quote> file is used for specifying these marks in a tabular fashion. For an in-depth look at the packet marking facility in Netfilter/Shorewall, please see <ulink url="PacketMarking.html">this article</ulink>.</para> <para>Normally, packet marking occurs in the PREROUTING chain before any address rewriting takes place. This makes it impossible to mark inbound packets based on their destination address when SNAT or Masquerading are being used. You can cause packet marking to occur in the FORWARD chain by using the MARK_IN_FORWARD_CHAIN option in shorewall.conf.</para> <para>Columns in the file are as follows:</para> <itemizedlist> <listitem> <para>MARK or CLASSIFY - MARK specifies the mark value is to be assigned in case of a match. This is an integer in the range 1-255 (1-16383 if you set WIDE_TC_MARKS=Yes in <ulink url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5) ). This value may be optionally followed by <quote>:</quote> and either <quote>F</quote>, <quote>P</quote> or "T" to designate that the marking will occur in the FORWARD, PREROUTING or POSTROUTING chains respectively. If this additional specification is omitted, the chain used to mark packets will be determined as follows:</para> <itemizedlist> <listitem> <para>If the SOURCE is $FW[:<<emphasis>address</emphasis>>], then the rule is inserted in the OUTPUT chain.</para> </listitem> <listitem> <para>Otherwise, the chain is determined by the setting of the MARK_IN_FORWARD_CHAIN option in shorewall.conf.</para> </listitem> </itemizedlist> <note> <para><emphasis role="bold">Use the 'T' qualifier if you want the rule to apply equally to traffic being routed through the firewall and to traffic originating on the firewall itself.</emphasis></para> </note> <para>Normally, the mark is applied to the packet. If you follow the mark value with ":" and "C", then the mark is applied to the connection. "C" can be combined with "F", "P" or "T" to designate that the connection should be marked in a particular chain (e.g., "CF", "CP", "CT").</para> <para>There are additional special values available:</para> <orderedlist numeration="loweralpha"> <listitem> <para><emphasis role="bold">RESTORE</emphasis>[/<emphasis>mask</emphasis>] -- restore the packet's mark from the connection's mark using the supplied mask if any. Your kernel and iptables must include CONNMARK support.</para> <para>As above, may be followed by <emphasis role="bold">:P</emphasis>, <emphasis role="bold">:F</emphasis> or <emphasis role="bold">:T</emphasis>.</para> </listitem> <listitem> <para><emphasis role="bold">SAVE</emphasis>[/<emphasis>mask</emphasis>] -- save the packet's mark to the connection's mark using the supplied mask if any. Your kernel and iptables must include CONNMARK support.</para> <para>As above, may be followed by <emphasis role="bold">:P</emphasis>, <emphasis role="bold">:F</emphasis> or <emphasis role="bold">:T</emphasis>.</para> </listitem> <listitem> <para><emphasis role="bold">CONTINUE</emphasis> Don't process any more marking rules in the table.</para> <para>As above, may be followed by <emphasis role="bold">:P</emphasis>, <emphasis role="bold">:F</emphasis> or <emphasis role="bold">:T</emphasis>.</para> </listitem> <listitem> <para><emphasis role="bold">COMMENT</emphasis> -- the rest of the line will be attached as a comment to the Netfilter rule(s) generated by the following entries. The comment will appear delimited by "/* ... */" in the output of <command>shorewall show mangle</command></para> <para>To stop the comment from being attached to further rules, simply include COMMENT on a line by itself.</para> </listitem> </orderedlist> <para>To use CLASSIFY, your kernel and iptables must include CLASSIFY target support. In that case, this column contains a classification (classid) of the form <major>:<minor> where <major> and <minor> are integers. Corresponds to the 'class' specification in these traffic shaping modules:</para> <simplelist> <member>atm</member> <member>cbq</member> <member>dsmark</member> <member>pfifo_fast</member> <member>htb</member> <member>prio</member> </simplelist> <para>Classification occurs in the POSTROUTING chain <emphasis role="bold">except</emphasis> when the SOURCE contains $FW[:<<emphasis>address</emphasis>>] in which case, the classify action takes place in the OUTPUT chain. When used with the builtin traffic shaper, the <major> class is the interface number and the <minor> class is either:</para> <orderedlist> <listitem> <para>Constructed from the mark. The method of construction depends on the setting of WIDE_TC_MARKS (<ulink url="manpages/shorewall.conf.html">shorewall.conf</ulink> (5)).</para> <para>When WIDE_TC_MARKS=No (the default), the <minor> class is:</para> <itemizedlist> <listitem> <para>the MARK value of the class preceded by the number "1" or "10" (MARK value 1 is <minor> class 11, MARK value 22 is <minor> class 122, and so on). "10" is used where there are more than 10 devices defined in <link linkend="tcdevices">/etc/shorewall/tcdevices</link>.</para> </listitem> </itemizedlist> <para>When SIDE_TC_MARKS=Yes, the <minor> class is:</para> <itemizedlist> <listitem> <para>The number 0x4000 logically ORed with the MARK value. The <command>shorewall encode</command> and <command>shorewall decode</command> commands (<ulink url="manpages/shorewall.html">shorewall</ulink> (8)) may be used to translate a mark to/from a <minor> class.</para> <para>Examples:</para> <programlisting>$ <command>shorewall encode 100</command> Class Number = 16484 $ <command>shorewall decode 16484</command> Mark = 100</programlisting> </listitem> </itemizedlist> </listitem> <listitem> <para>The class number (if the <emphasis role="bold">classify</emphasis> option was specified for the interface in <filename>/etc/shorewall/interfaces</filename>)</para> </listitem> </orderedlist> </listitem> <listitem> <para>SOURCE - Source of the packet.</para> <para>May be:</para> <orderedlist> <listitem> <para>An interface name - matches traffic entering the firewall on the specified interface. May not be used in classify rules or in rules using the :T chain qualifier.</para> </listitem> <listitem> <para>A comma-separated list of host or network IP addresses or MAC addresses. <emphasis role="bold">This form will not match traffic that originates on the firewall itself unless either <major><minor> or the :T chain qualifier is used in the MARK column.</emphasis></para> <para>Examples:<simplelist> <member>0.0.0.0/0</member> </simplelist></para> <para><simplelist> <member>192.168.1.0/24, 172.20.4.0/24</member> </simplelist></para> </listitem> <listitem> <para>An interface name followed by a colon (":") followed by a comma-separated list of host or network IP addresses or MAC addresses. May not be used in classify rules or in rules using the :T chain qualifier.</para> </listitem> <listitem> <para>$FW optionally followed by a colon (":") and a comma-separated list of host or network IP addresses. matches packets originating on the firewall. May not be used with a chain qualifier (:P, :F, etc.) in the MARK column.</para> </listitem> </orderedlist> <para>MAC addresses must be prefixed with "~" and use "-" as a separator.</para> <para>Example: ~00-A0-C9-15-39-78</para> <para>If your kernel includes iprange match support, then address ranges may be included in the address lists.</para> </listitem> <listitem> <para>DEST - Destination of the packet.</para> <para>May be:</para> <orderedlist> <listitem> <para>An interface name. May not be used in the PREROUTING chain (:P in the mark column or no chain qualifier and MARK_IN_FORWARD_CHAIN=No in <ulink url="manpages/shorewall.conf">shorewall.conf</ulink> (5)). The interface name may be optionally followed by a colon (":") and an IP address list.</para> </listitem> <listitem> <para>A comma-separated list of host or network IP addresses. The list may include ip address ranges if your kernel and iptables include iprange support.</para> </listitem> </orderedlist> </listitem> <listitem> <para>PROTO - Protocol - Must be "tcp", "udp", "icmp", "ipp2p", "ipp2p:udp", "ipp2p:all" a number, or "all". "ipp2p" requires ipp2p match support in your kernel and iptables.</para> </listitem> <listitem> <para>PORT(S) - Destination Ports. A comma-separated list of Port names (from /etc/services), port numbers or port ranges; if the protocol is "icmp", this column is interpreted as the destination icmp-type(s).</para> <para>If the protocol is ipp2p, this column is interpreted as an ipp2p option without the leading "--" (example "bit" for bit-torrent). If no PORT is given, "ipp2p" is assumed. Note that the xtables-addons version of IPP2P does not support the "ipp2p" option; if the column is empty or contains "ipp2p" when using that version of IPP2P, Shorewall will substitute "edk,kazaa,gnu,dc".</para> <para>This column is ignored if PROTOCOL = all but must be entered if any of the following field is supplied. In that case, it is suggested that this field contain "-"</para> </listitem> <listitem> <para>CLIENT PORT(S) - (Optional) Port(s) used by the client. If omitted, any source port is acceptable. Specified as a comma-separate list of port names, port numbers or port ranges.</para> </listitem> <listitem> <para>USER/GROUP (Optional) This column may only be non-empty if the SOURCE is the firewall itself. When this column is non-empty, the rule applies only if the program generating the output is running under the effective user and/or group. It may contain :</para> <para>[!][<user name or number>]:[<group name or number>][+<program name>]</para> <para>The colon is optional when specifying only a user.</para> <para>Examples:</para> <programlisting>joe #program must be run by joe :kids #program must be run by a member of the 'kids' group !:kids #program must not be run by a member of the 'kids' group +upnpd #program named upnpd (This feature was removed from Netfilter in kernel version 2.6.14).</programlisting> </listitem> <listitem> <para>TEST (Optional) Defines a test on the existing packet or connection mark. The rule will match only if the test returns true. Tests have the format [!]<value>[/<mask>][:C]</para> <para>Where:</para> <simplelist> <member>! Inverts the test (not equal)</member> <member><value> Value of the packet or connection mark.</member> <member><mask> A mask to be applied to the mark before testing</member> <member>:C Designates a connection mark. If omitted, the packet mark's value is tested.</member> </simplelist> </listitem> <listitem> <para>LENGTH (Optional) This field, if present, allows you to match the length of a packet against a specific value or range of values. A range is specified in the form <min>:<max> where either <min> or <max> (but not both) may be omitted. If <min> is omitted, then 0 is assumed; if <max> is omitted, than any packet that is <min> or longer will match.</para> <para>You must have iptables length support for this to work. If you let it empty or place an "-" here, no length match will be done.</para> <para>Examples: 1024, 64:1500, :100</para> </listitem> <listitem> <para>TOS (Optional) Type of Service. Either a standard name, or a numeric value to match.</para> <blockquote> <simplelist> <member>Minimize-Delay (16)</member> <member>Maximize-Throughput (8)</member> <member>Maximize-Reliability (4)</member> <member>Minimize-Cost (2)</member> <member>Normal-Service (0)</member> </simplelist> </blockquote> </listitem> <listitem> <para>HELPER (Optional). Names one of the Netfilter protocol helper modules such as <emphasis>ftp</emphasis>, <emphasis>sip</emphasis>, <emphasis>amanda</emphasis>, etc.</para> </listitem> </itemizedlist> <example id="Example1"> <title></title> <para>All packets arriving on eth1 should be marked with 1. All packets arriving on eth2 and eth3 should be marked with 2. All packets originating on the firewall itself should be marked with 3.</para> <programlisting>#MARK SOURCE DESTINATION PROTOCOL PORT(S) 1 eth1 0.0.0.0/0 all 2 eth2 0.0.0.0/0 all 2 eth3 0.0.0.0/0 all 3 $FW 0.0.0.0/0 all</programlisting> </example> <example id="Example2"> <title></title> <para>All GRE (protocol 47) packets destined for 155.186.235.151 should be marked with 12.</para> <programlisting>#MARK SOURCE DESTINATION PROTOCOL PORT(S) 12:T 0.0.0.0/0 155.182.235.151 47</programlisting> </example> <example id="Example3"> <title></title> <para>All SSH request packets originating in 192.168.1.0/24 and destined for 155.186.235.151 should be marked with 22.</para> <programlisting>#MARK SOURCE DESTINATION PROTOCOL PORT(S) 22:T 192.168.1.0/24 155.182.235.151 tcp 22</programlisting> </example> <example id="Example4"> <title></title> <para>All SSH packets packets going out of the first device in in /etc/shorewall/tcdevices should be assigned to the class with mark value 10.</para> <programlisting>#MARK SOURCE DESTINATION PROTOCOL PORT(S) CLIENT # PORT(S) 1:110 0.0.0.0/0 0.0.0.0/0 tcp 22 1:110 0.0.0.0/0 0.0.0.0/0 tcp - 22</programlisting> </example> <example id="Example5"> <title></title> <para>Mark all ICMP echo traffic with packet mark 1. Mark all peer to peer traffic with packet mark 4.</para> <para>This is a little more complex than otherwise expected. Since the ipp2p module is unable to determine all packets in a connection are P2P packets, we mark the entire connection as P2P if any of the packets are determined to match. We assume packet/connection mark 0 to means unclassified. Traffic originating on the firewall is not covered by this example.</para> <programlisting>#MARK SOURCE DESTINATION PROTOCOL PORT(S) CLIENT USER/ TEST # PORT(S) GROUP 1 0.0.0.0/0 0.0.0.0/0 icmp echo-request 1 0.0.0.0/0 0.0.0.0/0 icmp echo-reply RESTORE 0.0.0.0/0 0.0.0.0/0 all - - - 0 CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - !0 4 0.0.0.0/0 0.0.0.0/0 ipp2p:all SAVE 0.0.0.0/0 0.0.0.0/0 all - - - !0</programlisting> <para>The last four rules can be translated as:</para> <blockquote> <para>"If a packet hasn't been classified (packet mark is 0), copy the connection mark to the packet mark. If the packet mark is set, we're done. If the packet is P2P, set the packet mark to 4. If the packet mark has been set, save it to the connection mark."</para> </blockquote> </example> <example> <title></title> <para>Mark all forwarded VOIP connections with connection mark 1 and ensure that all VOIP packets also receive that mark (assumes that nf_conntrack_sip is loaded).</para> <programlisting>#MARK SOURCE DESTINATION PROTOCOL PORT(S) CLIENT USER/ TEST CONNBYTES TOS HELPER # PORT(S) GROUP RESTORE 0.0.0.0/0 0.0.0.0/0 all - - - 0 CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - !0 1 0.0.0.0/0 0.0.0.0/0 all - - - - - - sip SAVE 0.0.0.0/0 0.0.0.0/0 all - - - !0</programlisting> </example> </section> <section id="ppp"> <title>ppp devices</title> <para>If you use ppp/pppoe/pppoa) to connect to your Internet provider and you use traffic shaping you need to restart shorewall traffic shaping. The reason for this is, that if the ppp connection gets restarted (and it usually does this at least daily), all <quote>tc</quote> filters/qdiscs related to that interface are deleted.</para> <para>The easiest way to achieve this, is just to restart shorewall once the link is up. To achieve this add a small executable script to<quote>/etc/ppp/ip-up.d</quote>.</para> <programlisting>#! /bin/sh /sbin/shorewall refresh</programlisting> </section> <section id="Real"> <title>Real life examples</title> <section> <title>A Shorewall User's Experience</title> <para>Chuck Kollars has provided <ulink url="http://www.ckollars.org/shaping.html">an excellent writeup</ulink> about his traffic shaping experiences.</para> </section> <section id="Wondershaper"> <title>Configuration to replace Wondershaper</title> <para>You are able to fully replace the wondershaper script by using the buitin traffic control.You can find example configuration files at <ulink url="http://www1.shorewall.net/pub/shorewall/Samples/tc4shorewall/">"http://www1.shorewall.net/pub/shorewall/Samples/tc4shorewall/</ulink>. Please note that they are just examples and need to be adjusted to work for you. In this example it is assumed that your interface for your Internet connection is ppp0 (for DSL), if you use another connection type, you have to change it. You also need to change the settings in the tcdevices.wondershaper file to reflect your line speed. The relevant lines of the config files follow here. Please note that this is just a 1:1 replacement doing exactly what wondershaper should do. You are free to change it...</para> <section id="realtcd"> <title>tcdevices file</title> <programlisting>#INTERFACE IN-BANDWITH OUT-BANDWIDTH ppp0 5000kbit 500kbit</programlisting> </section> <section id="realtcc"> <title>tcclasses file</title> <programlisting>#INTERFACE MARK RATE CEIL PRIORITY OPTIONS ppp0 1 5*full/10 full 1 tcp-ack,tos-minimize-delay ppp0 2 3*full/10 9*full/10 2 default ppp0 3 2*full/10 8*full/10 2</programlisting> </section> <section id="realtcr"> <title>tcrules file</title> <programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER # PORT(S) 1:F 0.0.0.0/0 0.0.0.0/0 icmp echo-request 1:F 0.0.0.0/0 0.0.0.0/0 icmp echo-reply # mark traffic which should have a lower priority with a 3: # mldonkey 3 0.0.0.0/0 0.0.0.0/0 udp - 4666</programlisting> <para>Wondershaper allows you to define a set of hosts and/or ports you want to classify as low priority. To achieve this , you have to add these hosts to tcrules and set the mark to 3 (true if you use the example configuration files).</para> </section> <section id="lowpro"> <title>Setting hosts to low priority</title> <para>lets assume the following settings from your old wondershaper script (don't assume these example values are really useful, they are only used for demonstrating ;-):</para> <programlisting> # low priority OUTGOING traffic - you can leave this blank if you want # low priority source netmasks NOPRIOHOSTSRC="192.168.1.128/25 192.168.3.28" # low priority destination netmasks NOPRIOHOSTDST=60.0.0.0/24 # low priority source ports NOPRIOPORTSRC="6662 6663" # low priority destination ports NOPRIOPORTDST="6662 6663" </programlisting> <para>This would result in the following additional settings to the tcrules file:</para> <programlisting>3 192.168.1.128/25 0.0.0.0/0 all 3 192.168.3.28 0.0.0.0/0 all 3 0.0.0.0/0 60.0.0.0/24 all 3 0.0.0.0/0 0.0.0.0/0 udp 6662,6663 3 0.0.0.0/0 0.0.0.0/0 udp - 6662,6663 3 0.0.0.0/0 0.0.0.0/0 tcp 6662,6663 3 0.0.0.0/0 0.0.0.0/0 tcp - 6662,6663</programlisting> </section> </section> <section id="simiple"> <title>A simple setup</title> <para>This is a simple setup for people sharing an Internet connection and using different computers for this. It just basically shapes between 2 hosts which have the ip addresses 192.168.2.23 and 192.168.2.42</para> <section id="simpletcd"> <title>tcdevices file</title> <programlisting>#INTERFACE IN-BANDWITH OUT-BANDWIDTH ppp0 6000kbit 700kbit</programlisting> <para>We have 6mbit down and 700kbit upstream.</para> </section> <section id="simpletcc"> <title>tcclasses file</title> <programlisting>#INTERFACE MARK RATE CEIL PRIORITY OPTIONS ppp0 1 10kbit 50kbit 1 tcp-ack,tos-minimize-delay ppp0 2 300kbit full 2 ppp0 3 300kbit full 2 ppp0 4 90kbit 200kbit 3 default</programlisting> <para>We add a class for tcp ack packets with highest priority, so that downloads are fast. The following 2 classes share most of the bandwidth between the 2 hosts, if the connection is idle, they may use full speed. As the hosts should be treated equally they have the same priority. The last class is for the remaining traffic.</para> </section> <section id="simpletcr"> <title>tcrules file</title> <programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER # PORT(S) 1:F 0.0.0.0/0 0.0.0.0/0 icmp echo-request 1:F 0.0.0.0/0 0.0.0.0/0 icmp echo-reply 2:F 192.168.2.23 0.0.0.0/0 all 3:F 192.168.2.42 0.0.0.0/0 all</programlisting> <para>We mark icmp ping and replies so they will go into the fast interactive class and set a mark for each host.</para> </section> </section> </section> </section> <section id="Xen"> <title>A Warning to Xen Users</title> <para>If you are running traffic shaping in your dom0 and traffic shaping doesn't seem to be limiting outgoing traffic properly, it may be due to "checksum offloading" in your domU(s). Check the output of "shorewall show tc". Here's an excerpt from the output of that command:</para> <programlisting>class htb 1:130 parent 1:1 leaf 130: prio 3 quantum 1500 rate 76000bit <emphasis role="bold">ceil 230000bit</emphasis> burst 1537b/8 mpu 0b overhead 0b cburst 1614b/8 mpu 0b overhead 0b level 0 Sent 559018700 bytes 75324 pkt (dropped 0, overlimits 0 requeues 0) <emphasis role="bold">rate 299288bit</emphasis> 3pps backlog 0b 0p requeues 0 lended: 53963 borrowed: 21361 <emphasis role="bold">giants: 90174</emphasis> tokens: -26688 ctokens: -14783</programlisting> <para>There are two obvious problems in the above output:</para> <orderedlist> <listitem> <para>The rate (299288) is considerably larger than the ceiling (230000).</para> </listitem> <listitem> <para>There are a large number (90174) of giants reported.</para> </listitem> </orderedlist> <para>This problem will be corrected by disabling "checksum offloading" in your domU(s) using the <command>ethtool</command> utility. See the <ulink url="XenMyWay-Routed.html">one of the Xen articles</ulink> for instructions.</para> </section> <section id="Downloads"> <title>Shaping Download Traffic</title> <para>As stated at the outset, traffic shaping works on traffic being sent by the firewall. Download traffic from the Internet to local hosts is sent by the firewall over a local interface. So it follows that if you want to shape such traffic, you must configure shaping on the local interface.</para> <para>Shaping of download traffic is most straightforward when there are only two interface. That way, traffic leaving the local interface falls into only two broad categories:</para> <itemizedlist> <listitem> <para>Traffic being forwarded from the Internet</para> </listitem> <listitem> <para>Traffic that originated on the firewall itself</para> </listitem> </itemizedlist> <para>In general, you will want to shape the forwarded traffic and leave the local traffic unrestricted.</para> <para>Extending the <link linkend="simiple">simple example</link> above:</para> <para><filename>/etc/shorewall/tcdevices</filename>:<programlisting>#INTERFACE IN-BANDWITH OUT-BANDWIDTH ppp0 6000kbit 700kbit eth1 - 100mbit</programlisting></para> <para>/etc/shorewall/tcclasses:<programlisting>#INTERFACE MARK RATE CEIL PRIORITY OPTIONS ppp0 1 10kbit 50kbit 1 tcp-ack,tos-minimize-delay ppp0 2 300kbit full 2 ppp0 3 300kbit full 2 ppp0 4 90kbit 200kbit 3 default eth0 1 100kbit 500kbit 1 tcp-ack eth0 2 3mbit 6mbit 2 eth0 3 3mbit 6mbit 3 eth0 4 94mbit full default #for local traffic</programlisting></para> <para>/etc/shorewall/tcrules:<programlisting>#MARK SOURCE DEST PROTO PORT(S) CLIENT USER # PORT(S) 1:F 0.0.0.0/0 0.0.0.0/0 icmp echo-request 1:F 0.0.0.0/0 0.0.0.0/0 icmp echo-reply 2:F 192.168.2.23 0.0.0.0/0 all 3:F 192.168.2.42 0.0.0.0/0 all 2:F ppp0 192.168.2.23 all 3:F ppp0 192.168.2.42 all</programlisting></para> </section> <section id="IFB"> <title>Intermediate Frame Block (IFB) Devices</title> <para>The principles behind an IFB is fairly simple:</para> <itemizedlist> <listitem> <para>It looks like a network interface although it is never given an IPv4 configuration.</para> </listitem> <listitem> <para>Because it is a network interface, queuing disciplines can be associated with an IFB.</para> </listitem> </itemizedlist> <para>The magic of an IFB comes in the fact that a filter may be defined on a real network interface such that each packet that arrives on that interface is queued for the IFB! In that way, the IFB provides a means for shaping input traffic.</para> <para>To use an IFB, you must have IFB support in your kernel (configuration option CONFIG_IFB). Assuming that you have a modular kernel, the name of the IFB module is 'ifb' and may be loaded using the command <command>modprobe ifb</command> (if you have modprobe installed) or <command>insmod /path/to/module/ifb</command>.</para> <para>By default, two IFB devices (ifb0 and ifb1) are created. You can control that using the numifbs option (e.g., <command>modprobe ifb numifbs=1</command>).</para> <para>To create a single IFB when Shorewall starts, place the following two commands in <filename>/etc/shorewall/init</filename>:</para> <programlisting><command>modprobe ifb numifbs=1 ip link set ifb0 up</command></programlisting> <para>Entries in <filename>/etc/shorewall/tcrules</filename> have no effect on shaping traffic through an IFB. To allow classification of such traffic, the /etc/shorewall/tcfilters file has been added. Entries in that file create <ulink url="http://b42.cz/notes/u32_classifier/">u32 classification rules</ulink>.</para> <section id="tcfilters"> <title>/etc/shorewall/tcfilters</title> <para>While this file was created to allow shaping of traffic through an IFB, the file may be used for general traffic classification as well. The file is similar to <ulink url="shorewall-tcrules.html">shorewall-tcrules</ulink>(5) with the following key exceptions:</para> <itemizedlist> <listitem> <para>The first match determines the classification, whereas in the tcrules file, the last match determines the classification.</para> </listitem> <listitem> <para>ipsets are not supported</para> </listitem> <listitem> <para>DNS Names are not supported</para> </listitem> <listitem> <para>filters are applied to packets as they <emphasis>appear on the wire</emphasis>. So incoming packets will not have DNAT applied yet (the destination IP address will be the external address) and outgoing packets will have had SNAT applied.</para> </listitem> </itemizedlist> <para>The last point warrants elaboration. When looking at traffic being shaped by an IFB, there are two cases to consider:</para> <orderedlist> <listitem> <para>Requests — packets being sent from remote clients to local servers. These packets may undergo subsequent DNAT, either as a result of entries in <filename>/etc/shorewall/nat</filename> or as a result of DNAT or REDIRECT rules.</para> <para>Example: <filename>/etc/shorewall/rules</filename>:</para> <programlisting>#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL # PORT(S) PORT(S) DEST DNAT net dmz:192.168.4.5 tcp 80 - 206.124.146.177</programlisting> <para>Requests redirected by this rule will have destination IP address 206.124.146.177 and destination port 80.</para> </listitem> <listitem> <para>Responses — packets being sent from remote servers to local clients. These packets may undergo subsequent DNAT as a result of entries in <filename>/etc/shorewall/nat</filename> or in <filename>/etc/shorewall/masq</filename>. The packet's destination IP address will be the external address specified in the entry.</para> <para>Example: <filename>/etc/shorewall/masq</filename>:<programlisting>#INTERFACE SOURCE ADDRESS eth0 192.168.1.0/24 206.124.146.179</programlisting></para> <para>HTTP response packets corresponding to requests that fall under that rule will have destination IP address 206.124.146.179 and <emphasis role="bold">source</emphasis> port 80.</para> </listitem> </orderedlist> <para>Columns in the file are as follow. As in all Shorewall configuration files, a hyphen ("-") may be used to indicate that no value is supplied in the column.</para> <variablelist> <varlistentry> <term>CLASS</term> <listitem> <para>The interface name or number followed by a colon (":") and the class number.</para> </listitem> </varlistentry> <varlistentry> <term>SOURCE</term> <listitem> <para>SOURCE IP address (host or network). DNS names are not allowed.</para> </listitem> </varlistentry> <varlistentry> <term>DEST</term> <listitem> <para>DESTINATION IP address (host or network). DNS names are not allowed.</para> </listitem> </varlistentry> <varlistentry> <term>PROTO</term> <listitem> <para>Protocol name or number.</para> </listitem> </varlistentry> <varlistentry> <term>DEST PORT(S)</term> <listitem> <para>Comma-separated list of destination port names or numbers. May only be specified if the protocol is TCP, UDP, SCTP or ICMP. Port ranges are supported except for ICMP.</para> </listitem> </varlistentry> <varlistentry> <term>SOURCE PORT</term> <listitem> <para>Comma-separated list of source port names or numbers. May only be specified if the protocol is TCP, UDP or SCTP. Port ranges are supported.</para> </listitem> </varlistentry> </variablelist> <para>Example:</para> <para>I've used this configuration on my own firewall. The IFB portion is more for test purposes rather than to serve any well-reasoned QOS strategy.</para> <para><filename>/etc/shorewall/init</filename>:<programlisting>qt modprobe ifb numifbs=1 qt ip link set dev ifb0 up</programlisting></para> <para><filename>/etc/shorewall/tcdevices</filename>:<programlisting>#INTERFACE IN-BANDWITH OUT-BANDWIDTH OPTIONS REDIRECTED # INTERFACES 1:eth0 - 384kbit classify 2:ifb0 - 1300kbit - eth0</programlisting> <filename>/etc/shorewall/tcclasses</filename>:<programlisting>#INTERFACE MARK RATE CEIL PRIORITY OPTIONS 1:110 - 5*full/10 full 1 tcp-ack,tos-minimize-delay 1:120 - 2*full/10 6*full/10 2 default 1:130 - 2*full/10 6*full/10 3 2:110 - 5*full/10 full 1 tcp-ack,tos-minimize-delay 2:120 - 2*full/10 6*full/10 2 default 2:130 - 2*full/10 6*full/10 3</programlisting><filename>/etc/shorewall/tcfilters</filename>:<programlisting>#INTERFACE: SOURCE DEST PROTO DEST SOURCE #CLASS PORT(S) PORT(S) # # OUTGOING TRAFFIC # 1:130 206.124.146.178 - tcp - 49441,49442 #BITTORRENT on wookie 1:110 206.124.146.178 #wookie 1:110 206.124.146.179 #SNAT of internal systems 1:110 206.124.146.180 #Work Laptop 1:110 - - icmp echo-request,echo-reply 1:110 - - icmp echo-reply 1:130 206.124.146.177 - tcp - 873,25 #Bulk Traffic # # INCOMING TRAFFIC # 2:110 - 206.124.146.178 #Wookie 2:110 - 206.124.146.179 #SNAT Responses 2:110 - 206.124.146.180 #Work Laptop 2:130 - 206.124.146.177 tcp 25 #Incoming Email.</programlisting></para> <para>You can examine the installed filters with the <command>shorewall show filters</command> command. What follows shows the output for <filename class="devicefile">eth0</filename> with the filters shown above. <emphasis role="bold">Bold font</emphasis> are comments explaining the rules.<programlisting>gateway:~ # shorewall-lite show filters Shorewall Lite 4.1.6 Classifiers at gateway - Fri Mar 21 08:06:47 PDT 2008 Device eth1: Device eth2: Device eth0: filter parent 1: protocol ip pref 10 u32 filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 3:</emphasis> ht divisor 1 <emphasis role="bold"> <========= Start of table 3. parses TCP header</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 3</emphasis>::800 order 2048 key ht 3 bkt 0 <emphasis role="bold">flowid 1:130</emphasis> (rule hit 102 success 0) match 03690000/ffff0000 at nexthdr+0 (success 0 ) <emphasis role="bold"> <========= SOURCE PORT 873 goes to class 1:130</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 2:</emphasis> ht divisor 1 <emphasis role="bold"> <========= Start of table 2. parses ICMP header</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 2</emphasis>::800 order 2048 key ht 2 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 0 success 0) match 08000000/ff000000 at nexthdr+0 (success 0 ) <emphasis role="bold"> <========= ICMP Type 8 goes to class 1:110</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 2</emphasis>::801 order 2049 key ht 2 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 0 success 0) match 00000000/ff000000 at nexthdr+0 (success 0 ) <emphasis role="bold"> <========= ICMP Type 0 goes to class 1:110</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 1:</emphasis> ht divisor 1 <emphasis role="bold"> <========= Start of table 1. parses TCP header</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 1:</emphasis>:800 order 2048 key ht 1 bkt 0 <emphasis role="bold">flowid 1:130</emphasis> (rule hit 0 success 0) match c1210000/ffff0000 at nexthdr+0 (success 0 ) <emphasis role="bold"> <========= SOURCE PORT 49441 goes to class 1:130</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 1</emphasis>::801 order 2049 key ht 1 bkt 0 <emphasis role="bold">flowid 1:130</emphasis> (rule hit 0 success 0) match c1220000/ffff0000 at nexthdr+0 (success 0 ) <emphasis role="bold"> <========= SOURCE PORT 49442 goes to class 1:130</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis> ht divisor 1 <emphasis role="bold"><========= Start of Table 800. Packets start here!</emphasis> <emphasis role="bold">=============== The following 2 rules are generated by the class definition in /etc/shorewall/classes ==================</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:800 order 2048 key ht 800 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 2204 success 138) match 00060000/00ff0000 at 8 (success 396 ) <emphasis role="bold"><========= TCP </emphasis> match 05000000/0f00ffc0 at 0 (success 250 ) <emphasis role="bold"><========= Header length 20 and Packet Length < 64</emphasis> match 00100000/00ff0000 at 32 (success 138 ) <emphasis role="bold"><========= ACK</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:801 order 2049 key ht 800 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 2066 success 0) match 00100000/00100000 at 0 (success 0 ) <emphasis role="bold"><========= Minimize-delay</emphasis><emphasis role="bold"> goes to class 1:110</emphasis> <emphasis role="bold"> =============== Jump to Table 1 if the matches are met ==================</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:802 order 2050 key ht 800 bkt 0 <emphasis role="bold">link 1:</emphasis> (rule hit 2066 success 0) match ce7c92b2/ffffffff at 12 (success 1039 ) <emphasis role="bold"><========= SOURCE 206.124.146.178 </emphasis> match 00060000/00ff0000 at 8 (success 0 ) <emphasis role="bold"><========= PROTO TCP</emphasis> offset 0f00>>6 at 0 eat filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:803 order 2051 key ht 800 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 2066 success 1039) match ce7c92b2/ffffffff at 12 (success 1039 ) <emphasis role="bold"><========= SOURCE 206.124.146.178 goes to class 1:110</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:804 order 2052 key ht 800 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 1027 success 132) match ce7c92b3/ffffffff at 12 (success 132 ) <emphasis role="bold"> <========= SOURCE 206.124.146.179 goes to class 1:110</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:805 order 2053 key ht 800 bkt 0 <emphasis role="bold">flowid 1:110</emphasis> (rule hit 895 success 603) match ce7c92b4/ffffffff at 12 (success 603 ) <emphasis role="bold"><========= SOURCE 206.124.146.180 goes to class 1:110</emphasis> <emphasis role="bold"> =============== Jump to Table 2 if the matches are met ==================</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:806 order 2054 key ht 800 bkt 0 <emphasis role="bold">link 2:</emphasis> (rule hit 292 success 0) match 00010000/00ff0000 at 8 (success 0 ) <emphasis role="bold"><========= PROTO ICMP</emphasis> offset 0f00>>6 at 0 eat <emphasis role="bold"> =============== Jump to Table 3 if the matches are met ==================</emphasis> filter parent 1: protocol ip pref 10 u32 <emphasis role="bold">fh 800:</emphasis>:807 order 2055 key ht 800 bkt 0 <emphasis role="bold">link 3:</emphasis> (rule hit 292 success 0) match ce7c92b1/ffffffff at 12 (success 265 ) <emphasis role="bold"><========= SOURCE 206.124.146.177</emphasis> match 00060000/00ff0000 at 8 (success 102 ) <emphasis role="bold"><========= PROTO TCP</emphasis> offset 0f00>>6 at 0 eat </programlisting></para> </section> </section> <section id="show"> <title>Understanding the output of 'shorewall show tc'</title> <para>The <command>shorewall show tc</command> (<command>shorewall-lite show tc</command>) command displays information about the current state of traffic shaping. For each device, it executes the following commands:</para> <programlisting> echo Device $device: tc -s -d qdisc show dev $device echo tc -s -d class show dev $device echo </programlisting> <para>So, the traffic-shaping output is generated entirely by the <command>tc</command> utility.</para> <para>Here's the output of for eth0. The configuration is the one shown in the preceding section (the output was obtained almost 24 hours later than the <command>shorewall show filters</command> output shown above).</para> <programlisting>Device eth0: <emphasis role="bold"> ============== The primary queuing discipline is HTB (Hierarchical Token Bucket) ==================== </emphasis> qdisc htb 1: r2q 10 default 120 direct_packets_stat 9 ver 3.17 Sent 2133336743 bytes 4484781 pkt (dropped 198, overlimits 4911403 requeues 21) <emphasis role="bold"><=========== Note the overlimits and dropped counts</emphasis> rate 0bit 0pps backlog 0b 8p requeues 21 <emphasis role="bold">============== The ingress filter. If you specify IN-BANDWIDTH, you can see the 'dropped' count here. =========</emphasis> <emphasis role="bold">In this case, the packets are being sent to the IFB for shaping</emphasis> qdisc ingress ffff: ---------------- Sent 4069015112 bytes 4997252 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 <emphasis role="bold">============ Each of the leaf HTB classes has an SFQ qdisc to ensure that each flow gets its turn ============</emphasis> qdisc sfq 110: parent 1:110 limit 128p quantum 1514b flows 128/1024 perturb 10sec Sent 613372519 bytes 2870225 pkt (dropped 0, overlimits 0 requeues 6) rate 0bit 0pps backlog 0b 0p requeues 6 qdisc sfq 120: parent 1:120 limit 128p quantum 1514b flows 128/1024 perturb 10sec Sent 18434920 bytes 60961 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc sfq 130: parent 1:130 limit 128p quantum 1514b flows 128/1024 perturb 10sec Sent 1501528722 bytes 1553586 pkt (dropped 198, overlimits 0 requeues 15) rate 0bit 0pps backlog 11706b 8p requeues 15 <emphasis role="bold">============= Class 1:110 -- the high-priority class =========== Note the rate and ceiling calculated from 'full'</emphasis> class htb <emphasis role="bold">1:110</emphasis> parent 1:1 leaf 110: prio 1 quantum 4800 <emphasis role="bold">rate 192000bit ceil 384000bit</emphasis> burst 1695b/8 mpu 0b overhead 0b cburst 1791b/8 mpu 0b overhead 0b level 0 Sent 613372519 bytes 2870225 pkt (dropped 0, overlimits 0 requeues 0) <emphasis role="bold">rate 195672bit 28pps backlog 0b 0p</emphasis> requeues 0 <emphasis role="bold"><=========== Note the current rate of traffic. There is no queuing (no packet backlog)</emphasis> lended: 2758458 borrowed: 111773 giants: tokens: 46263 ctokens: 35157 <emphasis role="bold">============== The root class ============</emphasis> class htb <emphasis role="bold">1:1 root</emphasis> rate 384000bit ceil 384000bit burst 1791b/8 mpu 0b overhead 0b cburst 1791b/8 mpu 0b overhead 0b level 7 <emphasis role="bold">Sent 2133276316 bytes 4484785 pkt</emphasis> (dropped 0, overlimits 0 requeues 0) <emphasis role="bold"><==== Total output traffic since last 'restart'</emphasis> rate 363240bit 45pps backlog 0b 0p requeues 0 lended: 1081936 borrowed: 0 giants: 0 tokens: -52226 ctokens: -52226 <emphasis role="bold">============= Bulk Class (outgoing rsync, email and bittorrent) ============</emphasis> class htb 1:130 parent 1:1 leaf 130: prio 3 quantum 1900 rate 76000bit ceil 230000bit burst 1637b/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 Sent 1501528722 bytes 1553586 pkt (dropped 198, overlimits 0 requeues 0) <emphasis role="bold">rate 162528bit 14pps backlog 0b 8p</emphasis> requeues 0 <emphasis role="bold"><======== Queuing is occurring (8 packet backlog). The rate is still below the ceiling.</emphasis> lended: 587134 borrowed: 966459 giants: 0 <emphasis role="bold">During peak activity, the rate tops out at around 231000 (just above ceiling).</emphasis> tokens: -30919 ctokens: -97657 <emphasis role="bold">================= Default class (mostly serving web pages) ===============</emphasis> class htb 1:120 parent 1:1 leaf 120: prio 2 quantum 1900 rate 76000bit ceil 230000bit burst 1637b/8 mpu 0b overhead 0b cburst 1714b/8 mpu 0b overhead 0b level 0 Sent 18434920 bytes 60961 pkt (dropped 0, overlimits 0 requeues 0) rate 2240bit 2pps backlog 0b 0p requeues 0 lended: 57257 borrowed: 3704 giants: 0 tokens: 156045 ctokens: 54178 </programlisting> </section> <section id="External"> <title id="tcstart">Using your own tc script</title> <section id="owntcstart"> <title>Replacing builtin tcstart file</title> <para>If you prefer your own tcstart file, just install it in /etc/shorewall/tcstart.</para> <para>In your tcstart script, when you want to run the <quote>tc</quote> utility, use the run_tc function supplied by Shorewall if you want tc errors to stop the firewall.</para> <orderedlist> <listitem> <para>Set TC_ENABLED=Yes and CLEAR_TC=Yes</para> </listitem> <listitem> <para>Supply an /etc/shorewall/tcstart script to configure your traffic shaping rules.</para> </listitem> <listitem> <para>Optionally supply an /etc/shorewall/tcclear script to stop traffic shaping. That is usually unnecessary.</para> </listitem> <listitem> <para>If your tcstart script uses the <quote>fwmark</quote> classifier, you can mark packets using entries in /etc/shorewall/tcrules.</para> </listitem> </orderedlist> </section> <section id="Start"> <title>Traffic control outside Shorewall</title> <para>To start traffic shaping when you bring up your network interfaces, you will have to arrange for your traffic shaping configuration script to be run at that time. How you do that is distribution dependent and will not be covered here. You then should:</para> <orderedlist> <listitem> <para>Set TC_ENABLED=No and CLEAR_TC=No</para> </listitem> <listitem> <para>If your script uses the <quote>fwmark</quote> classifier, you can mark packets using entries in /etc/shorewall/tcrules.</para> </listitem> </orderedlist> </section> </section> <section id="Testing"> <title>Testing Tools</title> <para>At least one Shorewall user has found this tool helpful: <ulink url="http://e2epi.internet2.edu/network-performance-toolkit.html">http://e2epi.internet2.edu/network-performance-toolkit.html</ulink></para> </section> </article>