<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> <article id="usefull_links"> <!--$Id$--> <articleinfo> <title>Shorewall Troubleshooting Guide</title> <author> <firstname>Tom</firstname> <surname>Eastep</surname> </author> <pubdate><?dbtimestamp format="Y/m/d"?></pubdate> <copyright> <year>2001-2007</year> <holder>Thomas M. Eastep</holder> </copyright> <legalnotice> <para>Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled <quote><ulink type="" url="GnuCopyright.htm">GNU Free Documentation License</ulink></quote>.</para> </legalnotice> </articleinfo> <section id="Start"> <title><quote>shorewall start</quote> and <quote>shorewall restart</quote> Errors</title> <para>If the error is detected by the Shorewall compiler, it should be fairly obvious where the problem was found. Each error message includes the configuration file name and line number where the error was detected and often gives the particular item in error. The item is either enclosed in parentheses or is at the end following a colon (":").</para> <para>Example:<programlisting>gateway:~/test # shorewall restart . Compiling... ERROR: Invalid ICMP Type (0/400) : /root/test/rules (line 19) gateway:~/test # </programlisting>In this case, line 19 in the rules file specified an invalid ICMP Type (0/400).</para> <para>Additional information about the error can be obtained using the 'debug' keyword (Shorewall 4.4.19 and earlier) or using the (-T) option.</para> <para>Example (4.4.19 and earlier):</para> <para><programlisting>gateway:~/test # shorewall debug restart Compiling... ERROR: Invalid ICMP Type (0/400) : /root/test/rules (line 19) at /usr/share/shorewall/Shorewall/Config.pm line 338 Shorewall::Config::fatal_error('Invalid ICMP Type (0/400)') called at /usr/share/shorewall/Shorewall/Chains.pm line 885 Shorewall::Chains::validate_icmp('0/400') called at /usr/share/shorewall/Shorewall/Chains.pm line 949 Shorewall::Chains::do_proto('icmp', '0/400', '-') called at /usr/share/shorewall/Shorewall/Rules.pm line 1055 Shorewall::Rules::process_rule1('ACCEPT', 'loc', 'net', 'icmp', '0/400', '-', '-', '-', '-', ...) called at /usr/share/shorewall/Shorewall/Rules.pm line 1290 Shorewall::Rules::process_rule('ACCEPT', 'loc', 'net', 'icmp', '0/400', '-', '-', '-', '-', ...) called at /usr/share/shorewall/Shorewall/Rules.pm line 1336 Shorewall::Rules::process_rules() called at /usr/share/shorewall/Shorewall/Compiler.pm line 799 Shorewall::Compiler::compiler('/var/lib/shorewall/.restart', '/root/test', 0, 4) called at /usr/share/shorewall/compiler.pl line 86 gateway:~/test # </programlisting></para> <para>Example (4.4.20 and later):</para> <programlisting>gateway:~/test # shorewall restart -T Compiling... ERROR: Invalid ICMP Type (0/400) : /root/test/rules (line 19) at /usr/share/shorewall/Shorewall/Config.pm line 338 Shorewall::Config::fatal_error('Invalid ICMP Type (0/400)') called at /usr/share/shorewall/Shorewall/Chains.pm line 885 Shorewall::Chains::validate_icmp('0/400') called at /usr/share/shorewall/Shorewall/Chains.pm line 949 Shorewall::Chains::do_proto('icmp', '0/400', '-') called at /usr/share/shorewall/Shorewall/Rules.pm line 1055 Shorewall::Rules::process_rule1('ACCEPT', 'loc', 'net', 'icmp', '0/400', '-', '-', '-', '-', ...) called at /usr/share/shorewall/Shorewall/Rules.pm line 1290 Shorewall::Rules::process_rule('ACCEPT', 'loc', 'net', 'icmp', '0/400', '-', '-', '-', '-', ...) called at /usr/share/shorewall/Shorewall/Rules.pm line 1336 Shorewall::Rules::process_rules() called at /usr/share/shorewall/Shorewall/Compiler.pm line 799 Shorewall::Compiler::compiler('/var/lib/shorewall/.restart', '/root/test', 0, 4) called at /usr/share/shorewall/compiler.pl line 86 gateway:~/test # </programlisting> <para>This information is useful to Shorewall support if you need to <ulink url="support.html">file a problem report</ulink>.</para> <para>The end of the compile phase is signaled by a message such as the following:<programlisting>Shorewall configuration compiled to /var/lib/shorewall/.restart</programlisting>Errors occurring past that point are said to occur at <firstterm>run-time</firstterm> because they occur during the running of the compiled firewall script (/var/lib/shorewall/.restart in the case of the above message).</para> <para>One common run-time failure is that the iptables-restore program encounters an error. This will produce an error such as the following:<programlisting>... Restarting Shorewall.... iptables-restore v1.3.6: No chain/target/match by that name Error occurred at line: 83 Try `iptables-restore -h' or 'iptables-restore --help' for more information. ERROR: iptables-restore Failed. Input is in /var/lib/shorewall/.iptables-restore-input Restoring Shorewall... Shorewall restored from /var/lib/shorewall/restore Terminated gateway:~/test # </programlisting>A look at /var/lib/shorewall/restore at line 83 might show something like the following:<programlisting>-A reject -p tcp -j REJECT --reject-with tcp-reset</programlisting>In this case, the user had compiled his own kernel and had forgotten to include REJECT target support (see <ulink url="kernel.htm">kernel.htm</ulink>).</para> <para>You may also include the word <emphasis role="bold">debug</emphasis> as the first argument to the <filename>/sbin/shorewall</filename> and <filename>/sbin/shorewall-lite</filename> commands.<programlisting><command>shorewall debug restart</command></programlisting>In most cases, <emphasis role="bold">debug</emphasis> is a synonym for <emphasis role="bold">trace</emphasis>. The exceptions are:</para> <itemizedlist> <listitem> <para><emphasis role="bold">debug</emphasis> is ignored by the Shorewall compiler.</para> </listitem> <listitem> <para><emphasis role="bold">debug</emphasis> causes altered behavior of generated scripts. These scripts normally use<command> iptables-restore</command> to install the Netfilter ruleset but with <emphasis role="bold">debug</emphasis>, the commands normally passed to <command>iptables-restore</command> in its input file are passed individually to <command>iptables</command>. This is a diagnostic aid which allows identifying the individual command that is causing <command>iptables-restore</command> to fail; it should be used when iptables-restore fails when executing a <command>COMMIT</command> command.</para> </listitem> </itemizedlist> <warning> <para>The <emphasis role="bold">debug</emphasis> feature is strictly for problem analysis. When <emphasis role="bold">debug</emphasis> is used:</para> <orderedlist> <listitem> <para>The firewall is made 'wide open' before the rules are applied.</para> </listitem> <listitem> <para>The <filename>routestopped</filename> file is not consulted.</para> </listitem> <listitem> <para>The rules are applied in the canonical <command>iptables-restore</command> order. So if you need critical hosts to be always available during start/restart, you may not be able to use <emphasis role="bold">debug</emphasis>.</para> </listitem> </orderedlist> </warning> <para>In other run-time failure cases:<itemizedlist> <listitem> <para>Make a note of the error message that you see.</para> </listitem> <listitem> <para><command>shorewall debug start 2> /tmp/trace</command></para> </listitem> <listitem> <para>Look at the <filename>/tmp/trace</filename> file and see if that helps you determine what the problem is. Be sure you find the place in the log where the error message you saw is generated -- you should find the message near the end of the log.</para> </listitem> <listitem> <para>If you still can't determine what's wrong then see the <ulink url="support.htm">support page</ulink>.</para> </listitem> </itemizedlist></para> </section> <section id="Network"> <title>Your Network Environment</title> <para>Many times when people have problems with Shorewall, the problem is actually an ill-conceived network setup. Here are several popular snafus:</para> <itemizedlist> <listitem> <para>Port Forwarding where client and server are in the same subnet. See <ulink url="FAQ.htm#faq2">FAQ 2</ulink>.</para> </listitem> <listitem> <para>Trying to test net->loc DNAT rules from inside your firewall. You must test these rules from <emphasis role="bold">outside</emphasis> your firewall.</para> </listitem> <listitem> <para>Multiple interfaces connected to the same HUB or Switch. Given the way that the Linux kernel respond to ARP <quote>who-has</quote> requests, this type of setup <emphasis role="bold">does NOT work the way that you expect it to</emphasis>. You can test using this kind of configuration if you specify the <emphasis role="bold">arp_filter</emphasis> option or the <emphasis role="bold">arp_ignore</emphasis> option in <filename><ulink url="manpages/shorewall-interfaces.html">/etc/shorewall/interfaces</ulink></filename> for all interfaces connected to the common hub/switch. <emphasis role="bold">Using such a setup with a production firewall is strongly recommended against</emphasis>.</para> </listitem> </itemizedlist> </section> <section id="NewDevice"> <title>New Device Doesn't Work?</title> <para>If you have just added a new device such as VOIP and it doesn't work, be sure that you have assigned it an IP address in your local network and that its default gateway has been set to the IP address of your internal interface. For many of these devices, the simplest solution is to run a DHCP server; running it on your firewall is fine — be sure to set the <emphasis role="bold">dhcp</emphasis> option on your internal interface in <ulink url="manpages/shorewall-interfaces.html">/etc/shorewall/interfaces</ulink>.</para> </section> <section id="Connections"> <title>Connection Problems</title> <para>One very important thing to remember is that not all connection problems are Shorewall configuration problems. If the connection that is giving you problems is to or from the firewall system or if it doesn't rely on NAT or Proxy ARP then you can often eliminate Shorewall using a simple test:</para> <itemizedlist> <listitem> <para><command>/sbin/shorewall clear</command></para> </listitem> <listitem> <para>Try the connection. If it works then the problem is in your Shorewall configuration; if the connection still doesn't work then the problem is not with Shorewall or the way that it is configured.</para> </listitem> <listitem> <para>Be sure to <command>/sbin/shorewall start</command> after the test.</para> </listitem> </itemizedlist> <para>If you still suspect Shorewall and the appropriate policy for the connection that you are trying to make is ACCEPT, please DO NOT ADD ADDITIONAL ACCEPT RULES TRYING TO MAKE IT WORK. Such additional rules will NEVER make it work, they add clutter to your rule set and they represent a big security hole in the event that you forget to remove them later.</para> <para>I also recommend against setting all of your policies to ACCEPT in an effort to make something work. That robs you of one of your best diagnostic tools - the <quote>Shorewall</quote> messages that Netfilter will generate when you try to connect in a way that isn't permitted by your rule set.</para> <para>Check your log (<quote><command>/sbin/shorewall show log</command></quote>). If you don't see Shorewall messages, then your problem is probably NOT a Shorewall problem. If you DO see packet messages, it may be an indication that you are missing one or more rules -- see <ulink url="FAQ.htm#faq17">FAQ 17</ulink>.</para> <para>While you are troubleshooting, it is a good idea to clear two variables in <filename><filename>/etc/shorewall/shorewall.conf</filename></filename>:</para> <para><programlisting>LOGRATE= LOGBURST=""</programlisting>This way, you will see all of the log messages being generated (be sure to restart shorewall after clearing these variables).</para> <example id="Example2"> <title>Log Message</title> <programlisting>Jun 27 15:37:56 gateway kernel: Shorewall:all2all:REJECT:IN=eth2 OUT=eth1 SRC=192.168.2.2 DST=192.168.1.3 LEN=67 TOS=0x00 PREC=0x00 TTL=63 ID=5805 DF PROTO=UDP SPT=1803 DPT=53 LEN=47</programlisting> <para>Let's look at the important parts of this message:</para> <itemizedlist> <listitem> <para>all2all:REJECT - This packet was REJECTed out of the all2all chain -- the packet was rejected under the <quote>all</quote>-><quote>all</quote> REJECT policy (see <ulink url="FAQ.htm#faq17">FAQ 17</ulink>).</para> </listitem> <listitem> <para>IN=eth2 - the packet entered the firewall via eth2</para> </listitem> <listitem> <para>OUT=eth1 - if accepted, the packet would be sent on eth1</para> </listitem> <listitem> <para>SRC=192.168.2.2 - the packet was sent by 192.168.2.2</para> </listitem> <listitem> <para>DST=192.168.1.3 - the packet is destined for 192.168.1.3</para> </listitem> <listitem> <para>PROTO=UDP - UDP Protocol</para> </listitem> <listitem> <para>DPT=53 - DNS</para> </listitem> </itemizedlist> <para>In this case, 192.168.2.2 was in the <quote>dmz</quote> zone and 192.168.1.3 is in the <quote>loc</quote> zone. I was missing the rule:</para> <programlisting>#ACTION SOURCE DEST PROTO DEST # PORT(S) ACCEPT dmz loc udp 53</programlisting> </example> </section> <section id="Ping"> <title>Ping Problems</title> <para>Either can't ping when you think you should be able to or are able to ping when you think that you shouldn't be allowed? Shorewall's <quote>Ping</quote> Management is <ulink url="ping.html">described here</ulink>. Here are a couple of tips:</para> <itemizedlist> <listitem> <para>Remember that Shorewall doesn't automatically allow ICMP type 8 (<quote>ping</quote>) requests to be sent between zones. If you want pings to be allowed between zones, you need a rule of the form:</para> <programlisting>#ACTION SOURCE DEST PROTO DEST # PORT(S) Ping(ACCEPT)<emphasis><source zone></emphasis> <emphasis><destination zone></emphasis></programlisting> <para>The ramifications of this can be subtle. For example, if you have the following in <filename><ulink url="NAT.htm">/etc/shorewall/nat</ulink></filename>:</para> <programlisting>#EXTERNAL INTERFACE INTERNAL 10.1.1.2 eth0 130.252.100.18</programlisting> <para>and you ping 130.252.100.18, unless you have allowed icmp type 8 between the zone containing the system you are pinging from and the zone containing 10.1.1.2, the ping requests will be dropped.</para> </listitem> <listitem> <para>Ping requests are subject to logging under your policies. So ping floods can cause an equally big flood of log messages. To eliminate these, as the last rule in your /etc/shorewall/rules file add:</para> <programlisting>#ACTION SOURCE DEST PROTO DEST # PORT(S) Ping(DROP)net all</programlisting> </listitem> </itemizedlist> </section> <section id="Other"> <title>Some Things to Keep in Mind</title> <itemizedlist> <listitem> <para><emphasis role="bold">You cannot test your firewall from the inside</emphasis>. Just because you send requests to your firewall external IP address does not mean that the request will be associated with the external interface or the <quote>net</quote> zone. Any traffic that you generate from the local network will be associated with your local interface and will be treated as loc->fw traffic.</para> </listitem> <listitem> <para><emphasis role="bold">IP addresses are properties of systems, not of interfaces</emphasis>. It is a mistake to believe that your firewall is able to forward packets just because you can ping the IP address of all of the firewall's interfaces from the local network. The only conclusion you can draw from such pinging success is that the link between the local system and the firewall works and that you probably have the local system's default gateway set correctly.</para> </listitem> <listitem> <para><emphasis role="bold">All IP addresses configured on firewall interfaces are in the $FW (fw) zone</emphasis>. If 192.168.1.254 is the IP address of your internal interface then you can write <quote><emphasis role="bold">$FW:192.168.1.254</emphasis></quote> in a rule but you may not write <quote><emphasis role="bold">loc:192.168.1.254</emphasis></quote>. Similarly, it is nonsensical to add 192.168.1.254 to the <emphasis role="bold">loc</emphasis> zone using an entry in <filename>/etc/shorewall/hosts</filename>.</para> </listitem> <listitem> <para><emphasis role="bold">Reply packets do NOT automatically follow the reverse path of the one taken by the original request</emphasis>. All packets are routed according to the routing table of the host at each step of the way. This issue commonly comes up when people install a Shorewall firewall parallel to an existing gateway and try to use DNAT through Shorewall without changing the default gateway of the system receiving the forwarded requests. Requests come in through the Shorewall firewall where the destination IP address gets rewritten but replies go out unmodified through the old gateway.</para> </listitem> <listitem> <para><emphasis role="bold">Shorewall itself has no notion of inside or outside</emphasis>. These concepts are embodied in how Shorewall is configured.</para> </listitem> </itemizedlist> </section> <section id="More"> <title>Other Gotchas</title> <itemizedlist> <listitem> <para>Seeing rejected/dropped packets logged out of the INPUT or FORWARD chains? This means that:</para> <orderedlist> <listitem> <para>your zone definitions are screwed up and the host that is sending the packets or the destination host isn't in any zone (using an <ulink url="manpages/shorewall-hosts.html"><filename>/etc/shorewall/hosts</filename></ulink> file are you?); or</para> </listitem> <listitem> <para>the source and destination hosts are both connected to the same interface and you don't have a policy or rule for the source zone to or from the destination zone or you haven't set the <emphasis role="bold">routeback</emphasis> option for the interface in <ulink url="manpages/shorewall-interfaces.html"><filename>/etc/shorewall/interfaces</filename></ulink>.</para> </listitem> <listitem> <para>You have connected two firewall interfaces (from different zones) to the same hub or switch.</para> </listitem> </orderedlist> </listitem> <listitem> <para>If you specify <quote>routefilter</quote> for an interface, that interface must be up prior to starting the firewall.</para> </listitem> <listitem> <para>Is your routing correct? For example, internal systems usually need to be configured with their default gateway set to the IP address of their nearest firewall interface. One often overlooked aspect of routing is that in order for two hosts to communicate, the routing between them must be set up <emphasis role="bold">in both directions</emphasis>. So when setting up routing between <emphasis role="bold">A</emphasis> and <emphasis role="bold">B</emphasis>, be sure to verify that the route from <emphasis role="bold">B</emphasis> back to <emphasis role="bold">A</emphasis> is defined and correct.</para> </listitem> <listitem> <para>Do you have your kernel properly configured? <ulink url="kernel.htm">Click here to see kernel configuration information</ulink>.</para> </listitem> </itemizedlist> </section> <section id="Support"> <title>Still Having Problems?</title> <para>See the <ulink url="support.htm">Shorewall Support Page</ulink>.</para> </section> </article>