Tom Eastep 2006 Thomas M. Eastep Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled GNU Free Documentation License. This article includes information that applies to Shorewall version 3.2.5 and later. Not all features described here will be available in earlier releases.

Perhaps no aspect of Shorewall causes more confusion than packet marking. This article will attempt to clear up some of that confusion. Each packet has a mark whose value is initially 0. Mark values are stored in the skb (socket buffer) structure used by the Linux kernel to track packets; the mark value is not part of the packet itself and cannot be seen with tcpdump, ethereal or any other packet sniffing program. They can be seen in an iptables/ip6tables trace -- see the iptrace command in shorewal(8) and shorewall6(8). Example (output has been folded for display ): [11692.096077] TRACE: mangle:tcout:return:3 IN= OUT=eth0 SRC=172.20.1.130 DST=206.124.146.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=7212 SEQ=3 UID=0 GID=1000 MARK=0x10082 Each active connection (even those that are not yet in ESTABLISHED state) has a mark value that is distinct from the packet marks. Connection mark values can be seen using the shorewall show connections command. The default connection mark value is 0. Example (output has been folded for display ): shorewall show connections Shorewall 3.3.2 Connections at gateway - Mon Oct 2 09:08:18 PDT 2006 tcp 6 19 TIME_WAIT src=206.124.146.176 dst=192.136.34.98 sport=58597 dport=80 packets=23 bytes=4623 src=192.136.34.98 dst=206.124.146.176 sport=80 dport=58597 packets=23 bytes=22532 [ASSURED] mark=256 use=1 … Packet marks are valid only while the packet is being processed by the firewall. Once the packet has been given to a local process or sent on to another system, the packet's mark value is no longer available. Connection mark values, on the other hand, persist for the life of the connection. Other parts of the system such as Traffic Shaping and Policy Routing cannot use connection marks — they can only use packet marks.

Packet marking occurs in Netfilter's mangle table. See the Netfilter Overview article. You can think of entries in the tcrules file like instructions in a program coded in a crude assembly language. The program gets executed for each packet. That is another way of saying that if you don't program, you may have difficulty making full use of Netfilter/Shorewall's Packet Marking. Actually, the tcrules define several programs. Each program corresponds to one of the built-in chains in the mangle table. PREROUTING program — If MARK_IN_FORWARD_CHAIN=No in shorewall.conf, then by default entries in /etc/shorewall/tcrules are part of the PREROUTING program. Entries specifying the ":P" suffix in the MARK column are also part of the PREROUTING program. The PREROUTING program gets executed for each packet entering the firewall. FORWARD program — If MARK_IN_FORWARD_CHAIN=Yes in shorewall.conf, then by default entries in /etc/shorewall/tcrules are part of the FORWARD program. Entries specifying the ":F" suffix in the MARK column are also part of the FORWARD program. The FORWARD program gets executed for each packet forwarded by the firewall. OUTPUT program — Entries with $FW in the SOURCE column are part of the OUTPUT program. The OUTPUT program is executed for each packet originating on the firewall itself. POSTROUTING program — Entries with a class-id in the MARK column (and that don't specify $FW in the SOURCE column) are part of the POSTROUTING program. These rules are executed for each packet leaving the firewall. Entries specifying the ":T" suffix in the MARK column are also part of the POSTROUTING program (Shorewall version 3.4.0 and later). INPUT program — No entries in tcrules will add entries to this program. It is executed for each packet that is targeted to the firewall itself. Note that a packet being forwarded by your firewall actually gets processed by three different programs: PREROUTING, FORWARD and POSTROUTING. Similarly, packets addressed to the firewall itself are processed by two programs (PREROUTING and INPUT) while packets originating on the firewall are likewise processed by two programs (OUTPUT and POSTROUTING). Rules in each program are executed as follows: Rules are conditionally executed based on whether the current packet matches the contents of the SOURCE, DEST, PROTO, PORT(S), CLIENT PORT(S_, USER, TEST, LENGTH and TOS columns. When a rule is executed, either: the current packet receives a new mark value; or the connection to which the current packet belongs receives a new mark value (":C", ":CF" or ":CP" suffix in the MARK column); or the packet is classified for traffic shaping (class-id in the MARK column); or the packet mark in the current packet is moved to the connection mark for the connection that the current packet is part of ("SAVE" in the MARK column); or the connection mark value for the connection that the current packet is part of is moved to the current packet's mark ("RESTORE" in the MARK column); or jump to a subroutine (another chain in the mangle table). These jumps are generated by Shorewall; or exit the current subroutine ("CONTINUE" in the MARK column). Unless the subroutine is exited using CONTINUE, the current packet is always passed to the next tcrule in the subroutine.

The mark value is held in a 32-bit field. Because packet marking is the Netfilter kludge of last resort for solving many hard technical problems, Shorewall reserves half of this field (16 bits) for future use. The remainder is split into two 8-bit values: The low-order eight bits are used for traffic shaping marks. These eight bits are also used for selecting among multiple providers when HIGH_ROUTE_MARKS=No in shorewall.conf. Some rules that deal with only these bits use a mask value of 0xff. The next 8 bits are used for selecting among multiple providers when HIGH_ROUTE_MARKS=Yes in shorewall.conf. These bits are manipulated using a mask value of 0xff00. As hinted above, marking rules can specify both a mark value and a mask. The mask determines the subset of the 32 bits in the mark to be used in the operation — only those bits that are on in the mask are manipulated when the rule is executed. For entries in tcrules, Shorewall-generated rules use a mask value that depends on which program the rule is part of, what the rule does, and the setting of HIGH_ROUTE_MARKS. For entries in tcrules, the default mask value is 0xffff except in these cases: RESTORE rules use a default mask value of 0xff. SAVE rules use a default mask value of 0xff. Connection marking rules use a mask value of 0xff. Shorewall actually allows you to have complete control over the layout of the 32-bit mark using the following options in shorewall.conf (5): TC_BITS The number of bits at the low end of the mark to be used for traffic shaping marking. May be zero. PROVIDER_BITS The number of bits in the mark to be used for provider numbers. May be zero. PROVIDER_OFFSET The offset from the right (low-order end) of the provider number field. If non-zero, must be >= TC_BITS (Shorewall automatically adjusts PROVIDER_OFFSET's value). PROVIDER_OFFSET + PROVIDER_BITS must be <= 32. MASK_BITS Number of bits on the right of the mark to be masked when clearing the traffic shaping mark. Must be >= TC_BITS and <= PROVIDER_OFFSET (if PROVIDER_OFFSET > 0) The relationship between these options is shown in this diagram. The default values of these options are determined by the settings of other options as follows: WIDE_TC_MARKS=No, HIGH_ROUTE_MARKS=No TC_BITS=8, PROVIDER_BITS=0, PROVIDER_OFFSET=0, MASK_BITS=8 WIDE_TC_MARKS=No, HIGH_ROUTE_MARKS=Yes TC_BITS=8, PROVIDER_BITS=8, PROVIDER_OFFSET=8, MASK_BITS=8 WIDE_TC_MARKS=Yes, HIGH_ROUTE_MARKS=No TC_BITS=14, PROVIDER_BITS=8, PROVIDER_OFFSET=0, MASK_BITS=16 WIDE_TC_MARKS=Yes, HIGH_ROUTE_MARKS=Yes TC_BITS=14, PROVIDER_BITS=8, PROVIDER_OFFSET=16, MASK_BITS=16 The existence of both TC_BITS and MASK_BITS is owed to the way that WIDE_TC_MARKS was originally implemented. Note that TC_BITS is 14 rather than 16 when WIDE_TC_MARKS=Yes. Beginning with Shorewall 4.4.12, the field between MASK_BITS and PROVIDER_OFFSET can be used for any purpose you want.

Shorewall creates a set of chains in the mangle table to hold rules defined in your /etc/shorewall/tcrules file. As mentioned above, chains are like subroutines in the packet marking programming language. By placing all of your rules in subroutines, CONTINUE (which generates a Netfilter RETURN rule) can be used to stop processing your rules while still allowing following Shorewall-generated rules to be executed. tcpre PREROUTING rules. tcfor FORWARD rules. tcout OUTPUT rules. tcpost POSTROUTING rules. Shorewall generates jumps to these chains from the built-in chains (PREROUTING, FORWARD, etc.).

Here's the example (slightly expanded) from the comments at the top of the /etc/shorewall/tcrules file. #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS # PORT(S) 1 0.0.0.0/0 0.0.0.0/0 icmp echo-request #Rule 1 1 0.0.0.0/0 0.0.0.0/0 icmp echo-reply #Rule 2 1 $FW 0.0.0.0/0 icmp echo-request #Rule 3 1 $FW 0.0.0.0/0 icmp echo-reply #Rule 4 RESTORE 0.0.0.0/0 0.0.0.0/0 all - - - 0 #Rule 5 CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - !0 #Rule 6 4 0.0.0.0/0 0.0.0.0/0 ipp2p:all #Rule 7 SAVE 0.0.0.0/0 0.0.0.0/0 all - - - !0 #Rule 8 ##LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE Let's take a look at each rule: This straight-forward rule simply marks all 'ping' requests passing through the firewall with mark value 1. Note that it does not mark pings that originate on the firewall itself. Similarly, this rule marks 'ping' replies. This rule marks 'ping' requests that originate on the firewall. This rule and the next ones are part of the OUTPUT program. Similarly, this rule marks 'ping' replies from the firewall itself. Remember that even though 'ping' packets were marked in one of the first two rules, they are still passed on to rule 5 (note that packets marked by rules 3 and 4 are not processed by this rule since it is in a different program). That rule moves the connection mark to the packet mark, if the packet mark is still zero (note the '0' in the TEST column). Without the '0' in the TEST column, this rule would overwrite the marks assigned in the first two rules. If the packet mark is non-zero (note the '!0' in the TEST column), then exit — The remaining rules will not be executed in this case. The packet mark will be non-zero if this is a 'ping' packet, or if the connection mark restored in rule 5 was non-zero. The packet mark is still zero. This rule checks to see if this is a P2P packet and if it is, the packet mark is set to 4. If the packet mark is non-zero (meaning that it was set to 4 in rule 7), then save the value (4) in the connection. The next time that a packet from this same connection comes through this program, rule 6 will be executed and the P2P check will be avoided.

You can see the tcrules in action using the shorewall show mangle command. The sample output from that command shown below has the following in /etc/shorewall/providers: #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY Blarg 1 0x100 main eth3 206.124.146.254 track,balance br0,eth1 #LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE Here is /etc/shorewall/tcrules: #MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST # PORT(S) 1:110 192.168.0.0/22 eth3 #Our internal nets get priority #over the server 1:130 206.124.146.177 eth3 tcp - 873 #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE And here is /etc/shorewall/tcdevices and /etc/shorewall/tcclasses: #INTERFACE IN-BANDWITH OUT-BANDWIDTH eth3 1.3mbit 384kbit #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE #INTERFACE MARK RATE CEIL PRIORITY OPTIONS eth3 10 full full 1 tcp-ack,tos-minimize-delay eth3 20 9*full/10 9*full/10 2 default eth3 30 6*full/10 6*full/10 3 #LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE I've annotated the following output with comments beginning with "<<<<" and ending with ">>>>". This example uses HIGH_ROUTE_MARKS=Yes and TC_EXPERT=No in shorewall.conf. gateway:~ # shorewall show mangle Shorewall 3.3.2 Mangle Table at gateway - Mon Oct 2 15:07:32 PDT 2006 Counters reset Mon Oct 2 07:49:52 PDT 2006 <<<< The PREROUTING Program >>>> Chain PREROUTING (policy ACCEPT 409K packets, 122M bytes) pkts bytes target prot opt in out source destination <<<< Restore the provider mark from the connection, if any >>>> 185K 77M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00 <<<< If there is no mark in the connection and the packet came in on eth3, then jump to the routemark chain This rule is generated as a result of 'track' being specified in the providers file entry for eth3 >>>> 8804 1396K routemark all -- eth3 * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00 <<<< If the packet came in on eth3, jump the the tcpre chain -- packets entering on a 'track'ed interface can have their mark set to zero there >>>> 102K 52M tcpre all -- eth3 * 0.0.0.0/0 0.0.0.0/0 <<<< Otherwise, jump to the tcpre chain if there is no current provider mark -- if we would have had TC_EXPERT=Yes, this jump would have been unconditional>>>> 215K 44M tcpre all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00 <<<< End of PREROUTING program >>>> <<<< INPUT Program -- Shorewall generates the single rule here which turns off the provider mark in the packet after routing The rule does that by logically ANDing the mark value with 0xff which will turn off all but the low-order 8 bits >>>> Chain INPUT (policy ACCEPT 98238 packets, 16M bytes) pkts bytes target prot opt in out source destination 98234 16M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK and 0xff <<<< End of INPUT program >>>> <<<< FORWARD Program -- Shorewall generates the first rule here which turns off the provider mark in the packet after routing >>>> Chain FORWARD (policy ACCEPT 312K packets, 106M bytes) pkts bytes target prot opt in out source destination 312K 106M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK and 0xff <<<< Jump unconditionally to the tcfor chain >>>> 312K 106M tcfor all -- * * 0.0.0.0/0 0.0.0.0/0 <<<< End of FORWARD program >>>> <<<< OUTPUT Program >>>> Chain OUTPUT (policy ACCEPT 1462K packets, 396M bytes) pkts bytes target prot opt in out source destination <<<< Restore the provider mark from the connection -- this rule was generated by Shorewall because of the 'track' option >>>> 3339 615K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00 <<<< If there is no provider mark, then jump to the tcout chain -- if we would have had TC_EXPERT=Yes, this jump would have been unconditional >>>> 92747 28M tcout all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00 <<<< End of FORWARD program >>>> <<<< POSTROUTING Program -- Unconditionally jump to the tcpost chain >>>> Chain POSTROUTING (policy ACCEPT 407K packets, 135M bytes) pkts bytes target prot opt in out source destination 407K 135M tcpost all -- * * 0.0.0.0/0 0.0.0.0/0 <<<< End of FORWARD program >>>> Chain routemark (1 references) pkts bytes target prot opt in out source destination <<<< Set connection 'track' mark for packets coming in on eth3 >>>> 8804 1396K MARK all -- eth3 * 0.0.0.0/0 0.0.0.0/0 MARK or 0x100 <<<< Save any mark added above in the connection mark >>>> 8804 1396K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK match !0x0/0xff00 CONNMARK save mask 0xff00 Chain tcfor (1 references) pkts bytes target prot opt in out source destination Chain tcout (1 references) pkts bytes target prot opt in out source destination Chain tcpost (1 references) pkts bytes target prot opt in out source destination <<<< The next two rules are the entries in the /etc/shorewall/tcrules file >>>> 65061 11M CLASSIFY all -- * eth3 192.168.0.0/22 0.0.0.0/0 CLASSIFY set 1:110 2224 2272K CLASSIFY tcp -- * eth3 206.124.146.177 0.0.0.0/0 tcp spt:873 CLASSIFY set 1:130 <<<< The following rules are generated by Shorewall and classify the traffic according to the marks in /etc/shorewall/classes >>>> 0 0 CLASSIFY all -- * eth3 0.0.0.0/0 0.0.0.0/0 MARK match 0xa/0xff CLASSIFY set 1:110 0 0 CLASSIFY all -- * eth3 0.0.0.0/0 0.0.0.0/0 MARK match 0x14/0xff CLASSIFY set 1:120 0 0 CLASSIFY all -- * eth3 0.0.0.0/0 0.0.0.0/0 MARK match 0x1e/0xff CLASSIFY set 1:130 Chain tcpre (2 references) pkts bytes target prot opt in out source destination gateway:~ #