2006-10-03 17:12:43 +02:00
<?xml version="1.0" encoding="UTF-8"?>
< !DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<article >
<!-- $Id$ -->
<articleinfo >
<title > Packet Marking using /etc/shorewall/tcrules</title>
<authorgroup >
<author >
<firstname > Tom</firstname>
<surname > Eastep</surname>
</author>
</authorgroup>
<pubdate > <?dbtimestamp format="Y/m/d"?> </pubdate>
<copyright >
<year > 2006</year>
<holder > Thomas M. Eastep</holder>
</copyright>
<legalnotice >
<para > Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
<quote > <ulink url= "GnuCopyright.htm" > GNU Free Documentation
License</ulink> </quote> .</para>
</legalnotice>
</articleinfo>
<caution >
<para > This article includes information that applies to Shorewall version
3.2.5 and later. Not all features described here will be available in
earlier releases.</para>
</caution>
<section >
<title > Packet and Connection Marks</title>
<para > Perhaps no aspect of Shorewall causes more confusion than packet
marking. This article will attempt to clear up some of that
confusion.</para>
<para > Each packet has a mark whose value is initially 0. Mark values are
stored in the <emphasis > skb</emphasis> (socket buffer) structure used by
the Linux kernel to track packets; the mark value is not part of the
packet itself and cannot be seen with <command > tcpdump</command> ,
<command > ethereal</command> or any other packet sniffing program.</para>
<para > Each active connection (even those that are not yet in ESTABLISHED
state) has a mark value that is distinct from the packet marks. Connection
mark values can be seen using the <command > shorewall show
connections</command> command. The default connection mark value is
0.</para>
<para > Example (output has been folded for display ):</para>
<programlisting > <command > shorewall show connections</command>
Shorewall 3.3.2 Connections at gateway - Mon Oct 2 09:08:18 PDT 2006
tcp 6 19 TIME_WAIT src=206.124.146.176 dst=192.136.34.98 sport=58597 dport=80
packets=23 bytes=4623 src=192.136.34.98 dst=206.124.146.176 sport=80 dport=58597
packets=23 bytes=22532 [ASSURED] <emphasis role= "bold" > mark=256</emphasis> use=1
…</programlisting>
<para > Packet marks are valid only while the packet is being processed by
the firewall. Once the packet has been given to a local process or sent on
to another system, the packet's mark value is no longer available.
Connection mark values, on the other hand, persist for the life of the
connection.</para>
2007-01-10 21:01:53 +01:00
<important >
<para > Other parts of the system such as <ulink
url="traffic_shaping.htm">Traffic Shaping</ulink> and <ulink
url="MultiISP.html">Policy Routing</ulink> cannot use connection marks —
they can only use packet marks.</para>
</important>
2006-10-03 17:12:43 +02:00
</section>
<section >
<title > Packet Marking "Programs"</title>
<para > Packet marking occurs in Netfilter's <emphasis > mangle</emphasis>
table. See the <ulink url= "NetfilterOverview.html" > Netfilter
Overview</ulink> article.</para>
<para > You can think of entries in the tcrules file like instructions in a
program coded in a crude assembly language. The program gets executed for
each packet.</para>
<para > That is another way of saying that <emphasis role= "bold" > if you
don't program, you may have difficulty making full use of
Netfilter/Shorewall's Packet Marking</emphasis> .</para>
<para > Actually, the tcrules define several programs. Each program
corresponds to one of the built-in chains in the mangle table.</para>
<itemizedlist >
<listitem >
<para > PREROUTING program — If MARK_IN_FORWARD_CHAIN=No in
<filename > shorewall.conf</filename> , then by default entries in
<filename > /etc/shorewall/tcrules</filename> are part of the PREROUTING
program. Entries specifying the ":P" suffix in the MARK column are
also part of the PREROUTING program. The PREROUTING program gets
executed for each packet entering the firewall.</para>
</listitem>
<listitem >
<para > FORWARD program — If MARK_IN_FORWARD_CHAIN=Yes in
<filename > shorewall.conf</filename> , then by default entries in
<filename > /etc/shorewall/tcrules</filename> are part of the FORWARD
program. Entries specifying the ":F" suffix in the MARK column are
also part of the FORWARD program. The FORWARD program gets executed
for each packet forwarded by the firewall.</para>
</listitem>
<listitem >
<para > OUTPUT program — Entries with $FW in the SOURCE column are part
of the OUTPUT program. The OUTPUT program is executed for each packet
originating on the firewall itself.</para>
</listitem>
<listitem >
<para > POSTROUTING program — Entries with a class-id in the MARK column
(and that don't specify $FW in the SOURCE column) are part of the
POSTROUTING program. These rules are executed for each packet leaving
2007-01-10 21:01:53 +01:00
the firewall. Entries specifying the ":T" suffix in the MARK column
are also part of the POSTROUTING program (Shorewall version 3.4.0 and
later). </para>
2006-10-03 17:12:43 +02:00
</listitem>
<listitem >
<para > INPUT program — No entries in tcrules will add entries to this
program. It is executed for each packet that is targeted to the
firewall itself.</para>
</listitem>
</itemizedlist>
<para > Note that a packet being forwarded by your firewall actually gets
processed by three different programs: PREROUTING, FORWARD and
2006-10-04 19:38:16 +02:00
POSTROUTING. Similarly, packets addressed to the firewall itself are
2006-10-03 17:12:43 +02:00
processed by two programs (PREROUTING and INPUT) while packets originating
on the firewall are likewise processed by two programs (OUTPUT and
POSTROUTING).</para>
<para > Rules in each program are <emphasis > executed</emphasis> as
follows:</para>
<itemizedlist >
<listitem >
<para > Rules are conditionally executed based on whether the current
packet matches the contents of the SOURCE, DEST, PROTO, PORT(S),
CLIENT PORT(S_, USER, TEST, LENGTH and TOS columns.</para>
</listitem>
<listitem >
<para > When a rule is executed, either:</para>
<orderedlist >
<listitem >
<para > the current packet receives a new mark value; or</para>
</listitem>
<listitem >
<para > the connection to which the current packet belongs receives
a new mark value (":C", ":CF" or ":CP" suffix in the MARK column);
or</para>
</listitem>
<listitem >
<para > the packet is classified for traffic shaping (class-id in
the MARK column); or</para>
</listitem>
<listitem >
<para > the packet mark in the current packet is moved to the
connection mark for the connection that the current packet is part
of ("SAVE" in the MARK column); or</para>
</listitem>
<listitem >
<para > the connection mark value for the connection that the
current packet is part of is moved to the current packet's mark
("RESTORE" in the MARK column); or</para>
</listitem>
<listitem >
<para > jump to a subroutine (another chain in the mangle table).
These jumps are generated by Shorewall; or</para>
</listitem>
<listitem >
<para > exit the current subroutine ("CONTINUE" in the MARK
column).</para>
</listitem>
</orderedlist>
</listitem>
<listitem >
<para > Unless the subroutine is exited using CONTINUE, <emphasis
role="bold">the current packet is always passed to the next tcrule in
the subroutine</emphasis> .</para>
</listitem>
</itemizedlist>
</section>
<section >
<title > Mark and Mask Values</title>
<para > The mark value is held in a 32-bit field. Because packet marking is
the Netfilter <emphasis > kludge of last resort</emphasis> for solving many
hard technical problems, Shorewall reserves half of this field (16 bits)
for future use. The remainder is split into two 8-bit values:</para>
<itemizedlist >
<listitem >
<para > The low-order eight bits are used for traffic shaping marks.
These eight bits are also used for selecting among multiple providers
when HIGH_ROUTE_MARKS=No in <filename > shorewall.conf</filename> . Some
rules that deal with only these bits use a mask value of 0xff.</para>
</listitem>
<listitem >
<para > The next 8 bits are used for selecting among multiple providers
when HIGH_ROUTE_MARKS=Yes in <filename > shorewall.conf</filename> .
These bits are manipulated using a mask value of 0xff00.</para>
</listitem>
</itemizedlist>
<para > As hinted above, marking rules can specify both a mark value and a
2006-10-04 20:04:24 +02:00
mask. The mask determines the subset of the 32 bits in the mark to be used
in the operation — only those bits that are on in the mask are manipulated
when the rule is executed. For entries in tcrules, Shorewall-generated
rules use a mask value that depends on which program the rule is part of,
what the rule does, and the setting of HIGH_ROUTE_MARKS.</para>
2006-10-03 17:12:43 +02:00
<para > For entries in tcrules, the default mask value is 0xffff except in
these cases:</para>
<itemizedlist >
<listitem >
<para > RESTORE rules use a default mask value of 0xff.</para>
</listitem>
<listitem >
<para > SAVE rules use a default mask value of 0xff.</para>
</listitem>
2006-10-10 20:59:18 +02:00
<listitem >
<para > Connection marking rules use a mask value of 0xff.</para>
</listitem>
2006-10-03 17:12:43 +02:00
</itemizedlist>
</section>
<section >
<title > Shorewall-defined Chains in the Mangle Table</title>
<para > Shorewall creates a set of chains in the mangle table to hold rules
defined in your <firstterm > /etc/shorewall/tcrules</firstterm> file. As
mentioned above, chains are like subroutines in the packet marking
programming language. By placing all of your rules in subroutines,
CONTINUE (which generates a Netfilter RETURN rule) can be used to stop
processing your rules while still allowing following Shorewall-generated
rules to be executed.</para>
<variablelist >
<varlistentry >
<term > tcpre</term>
<listitem >
<para > PREROUTING rules.</para>
</listitem>
</varlistentry>
<varlistentry >
<term > tcfor</term>
<listitem >
<para > FORWARD rules.</para>
</listitem>
</varlistentry>
<varlistentry >
<term > tcout</term>
<listitem >
<para > OUTPUT rules.</para>
</listitem>
</varlistentry>
<varlistentry >
<term > tcpost</term>
<listitem >
<para > POSTROUTING rules.</para>
</listitem>
</varlistentry>
</variablelist>
<para > Shorewall generates jumps to these chains from the built-in chains
(PREROUTING, FORWARD, etc.).</para>
</section>
2006-10-03 18:08:43 +02:00
<section >
<title > An Example</title>
2006-10-04 03:26:08 +02:00
<para > Here's the example (slightly expanded) from the comments at the top
of the <filename > /etc/shorewall/tcrules</filename> file.</para>
2006-10-03 18:08:43 +02:00
<programlisting > #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS
# PORT(S)
1 0.0.0.0/0 0.0.0.0/0 icmp echo-request #Rule 1
1 0.0.0.0/0 0.0.0.0/0 icmp echo-reply #Rule 2
2006-10-04 03:26:08 +02:00
1 $FW 0.0.0.0/0 icmp echo-request #Rule 3
1 $FW 0.0.0.0/0 icmp echo-reply #Rule 4
2006-10-03 18:08:43 +02:00
2006-10-04 03:26:08 +02:00
RESTORE 0.0.0.0/0 0.0.0.0/0 all - - - 0 #Rule 5
CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - !0 #Rule 6
4 0.0.0.0/0 0.0.0.0/0 ipp2p:all #Rule 7
SAVE 0.0.0.0/0 0.0.0.0/0 all - - - !0 #Rule 8
2006-10-03 18:08:43 +02:00
##LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE</programlisting>
<para > Let's take a look at each rule:</para>
<orderedlist >
<listitem >
<para > This straight-forward rule simply marks all 'ping' requests
passing through the firewall with mark value 1. Note that it does not
2006-10-04 03:26:08 +02:00
mark pings that originate on the firewall itself.</para>
2006-10-03 18:08:43 +02:00
</listitem>
<listitem >
<para > Similarly, this rule marks 'ping' replies.</para>
</listitem>
2006-10-04 03:26:08 +02:00
<listitem >
<para > This rule marks 'ping' requests that originate on the firewall.
This rule and the next ones are part of the OUTPUT program.</para>
</listitem>
<listitem >
<para > Similarly, this rule marks 'ping' replies from the firewall
itself.</para>
</listitem>
2006-10-03 18:08:43 +02:00
<listitem >
<para > Remember that even though 'ping' packets were marked in one of
2006-10-04 16:44:12 +02:00
the first two rules, they are still passed on to rule 5 (note that
2006-10-04 03:26:08 +02:00
packets marked by rules 3 and 4 are not processed by this rule since
it is in a different program). That rule moves the connection mark to
the packet mark, <emphasis > if the packet mark is still zero</emphasis>
(note the '0' in the TEST column). Without the '0' in the TEST column,
this rule would overwrite the marks assigned in the first two
rules.</para>
2006-10-03 18:08:43 +02:00
</listitem>
<listitem >
<para > If the packet mark is non-zero (note the '!0' in the TEST
column), then exit — The remaining rules will not be executed in this
case. The packet mark will be non-zero if this is a 'ping' packet, or
2006-10-04 03:26:08 +02:00
if the connection mark restored in rule 5 was non-zero.</para>
2006-10-03 18:08:43 +02:00
</listitem>
<listitem >
<para > The packet mark is still zero. This rule checks to see if this
is a P2P packet and if it is, the packet mark is set to 4.</para>
</listitem>
<listitem >
<para > If the packet mark is non-zero (meaning that it was set to 4 in
2006-10-04 03:26:08 +02:00
rule 7), then save the value (4) in the connection. The next time that
a packet from this same connection comes through this program, rule 6
2006-10-03 18:08:43 +02:00
will be executed and the P2P check will be avoided.</para>
</listitem>
</orderedlist>
</section>
2006-10-03 17:12:43 +02:00
<section >
<title > Examining the Marking Programs on a Running System</title>
<para > You can see the tcrules in action using the <command > shorewall show
2006-10-03 18:08:43 +02:00
mangle</command> command.</para>
2006-10-03 17:12:43 +02:00
<para > The sample output from that command shown below has the following in
<filename > /etc/shorewall/providers</filename> :</para>
<programlisting > #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
Blarg 1 0x100 main eth3 206.124.146.254 track,balance br0,eth1
#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE
</programlisting>
<para > Here is <filename > /etc/shorewall/tcrules</filename> :</para>
<programlisting > #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST
# PORT(S)
1:110 192.168.0.0/22 eth3 #Our internel nets get priority
#over the server
1:130 206.124.146.177 eth3 tcp - 873
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
</programlisting>
<para > And here is <filename > /etc/shorewall/tcdevices</filename> and
<filename > /etc/shorewall/tcclasses</filename> :</para>
<programlisting > #INTERFACE IN-BANDWITH OUT-BANDWIDTH
eth3 1.3mbit 384kbit
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
#INTERFACE MARK RATE CEIL PRIORITY OPTIONS
eth3 10 full full 1 tcp-ack,tos-minimize-delay
eth3 20 9*full/10 9*full/10 2 default
eth3 30 6*full/10 6*full/10 3
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
</programlisting>
<para > I've annotated the following output with comments beginning with
"< < < < " and ending with "> > > > ". This example uses
HIGH_ROUTE_MARKS=Yes and TC_EXPERT=No in
<filename > shorewall.conf</filename> .</para>
<programlisting > gateway:~ # <command > shorewall show mangle</command>
Shorewall 3.3.2 Mangle Table at gateway - Mon Oct 2 15:07:32 PDT 2006
Counters reset Mon Oct 2 07:49:52 PDT 2006
< < < < The PREROUTING Program > > > >
Chain PREROUTING (policy ACCEPT 409K packets, 122M bytes)
pkts bytes target prot opt in out source destination
< < < < Restore the provider mark from the connection, if any > > > >
185K 77M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00
< < < < If there is no mark in the connection and the packet came in on eth3, then jump to the <emphasis
role="bold">routemark</emphasis> chain
This rule is generated as a result of 'track' being specified in the providers file entry for eth3 > > > >
8804 1396K routemark all -- eth3 * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00
< < < < If the packet came in on eth3, jump the the <emphasis
role="bold">tcpre</emphasis> chain -- packets entering on a 'track'ed interface can have their mark set to zero there > > > >
102K 52M tcpre all -- eth3 * 0.0.0.0/0 0.0.0.0/0
< < < < Otherwise, jump to the tcpre chain if there is no current provider mark --
if we would have had TC_EXPERT=Yes, this jump would have been unconditional> > > >
215K 44M tcpre all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00
< < < < End of PREROUTING program > > > >
< < < < INPUT Program -- Shorewall generates the single rule here which turns off the provider mark in the packet after routing
The rule does that by logically ANDing the mark value with 0xff which will turn off all but the low-order 8 bits > > > >
Chain INPUT (policy ACCEPT 98238 packets, 16M bytes)
pkts bytes target prot opt in out source destination
98234 16M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK and 0xff
< < < < End of INPUT program > > > >
< < < < FORWARD Program -- Shorewall generates the first rule here which turns off the provider mark in the packet after routing > > > >
Chain FORWARD (policy ACCEPT 312K packets, 106M bytes)
pkts bytes target prot opt in out source destination
312K 106M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK and 0xff
< < < < Jump unconditionally to the <emphasis role= "bold" > tcfor</emphasis> chain > > > >
312K 106M tcfor all -- * * 0.0.0.0/0 0.0.0.0/0
< < < < End of FORWARD program > > > >
< < < < OUTPUT Program > > > >
Chain OUTPUT (policy ACCEPT 1462K packets, 396M bytes)
pkts bytes target prot opt in out source destination
< < < < Restore the provider mark from the connection -- this rule was generated by Shorewall because of the 'track' option > > > >
3339 615K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00
< < < < If there is no provider mark, then jump to the <emphasis
role="bold">tcout</emphasis> chain --
if we would have had TC_EXPERT=Yes, this jump would have been unconditional > > > >
92747 28M tcout all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00
< < < < End of FORWARD program > > > >
< < < < POSTROUTING Program -- Unconditionally jump to the <emphasis
role="bold">tcpost</emphasis> chain > > > >
Chain POSTROUTING (policy ACCEPT 407K packets, 135M bytes)
pkts bytes target prot opt in out source destination
407K 135M tcpost all -- * * 0.0.0.0/0 0.0.0.0/0
< < < < End of FORWARD program > > > >
Chain <emphasis role= "bold" > routemark</emphasis> (1 references)
pkts bytes target prot opt in out source destination
< < < < Set connection 'track' mark for packets coming in on eth3 > > > >
8804 1396K MARK all -- eth3 * 0.0.0.0/0 0.0.0.0/0 MARK or 0x100
< < < < Save any mark added above in the connection mark > > > >
8804 1396K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match !0x0/0xff00 CONNMARK save mask 0xff00
Chain <emphasis role= "bold" > tcfor</emphasis> (1 references)
pkts bytes target prot opt in out source destination
Chain <emphasis role= "bold" > tcout</emphasis> (1 references)
pkts bytes target prot opt in out source destination
Chain <emphasis role= "bold" > tcpost</emphasis> (1 references)
pkts bytes target prot opt in out source destination
< < < < The next two rules are the entries in the /etc/shorewall/tcrules file > > > >
65061 11M CLASSIFY all -- * eth3 192.168.0.0/22 0.0.0.0/0 CLASSIFY set 1:110
2224 2272K CLASSIFY tcp -- * eth3 206.124.146.177 0.0.0.0/0 tcp spt:873 CLASSIFY set 1:130
2006-10-03 20:58:54 +02:00
< < < < The following rules are generated by Shorewall and classify the traffic according to the marks in /etc/shorewall/classes > > > >
2006-10-03 17:12:43 +02:00
0 0 CLASSIFY all -- * eth3 0.0.0.0/0 0.0.0.0/0 MARK match 0xa/0xff CLASSIFY set 1:110
0 0 CLASSIFY all -- * eth3 0.0.0.0/0 0.0.0.0/0 MARK match 0x14/0xff CLASSIFY set 1:120
0 0 CLASSIFY all -- * eth3 0.0.0.0/0 0.0.0.0/0 MARK match 0x1e/0xff CLASSIFY set 1:130
Chain <emphasis role= "bold" > tcpre</emphasis> (2 references)
pkts bytes target prot opt in out source destination
gateway:~ #</programlisting>
</section>
</article>
