shorewall_code/docs/troubleshoot.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
<article id="usefull_links">
  <!--$Id$-->

  <articleinfo>
    <title>Shorewall Troubleshooting Guide</title>

    <author>
      <firstname>Tom</firstname>

      <surname>Eastep</surname>
    </author>

    <pubdate><?dbtimestamp format="Y/m/d"?></pubdate>

    <copyright>
      <year>2001-2007</year>

      <holder>Thomas M. Eastep</holder>
    </copyright>

    <legalnotice>
      <para>Permission is granted to copy, distribute and/or modify this
      document under the terms of the GNU Free Documentation License, Version
      1.2 or any later version published by the Free Software Foundation; with
      no Invariant Sections, with no Front-Cover, and with no Back-Cover
      Texts. A copy of the license is included in the section entitled
      <quote><ulink type="" url="GnuCopyright.htm">GNU Free Documentation
      License</ulink></quote>.</para>
    </legalnotice>
  </articleinfo>

  <section id="Start">
    <title><quote>shorewall start</quote> and <quote>shorewall restart</quote>
    Errors</title>

    <section id="Start-shell">
      <title>Shorewall-shell</title>

      <para>If you use the Shorewall-shell compiler and you receive an error
      message when starting or restarting the firewall and you can't determine
      the cause. First, if your VERBOSITY setting in shorewall.conf is less
      than 2, then try running with a higher verbosity level by using the "-v"
      option:</para>

      <blockquote>
        <programlisting><command>shorewall -vv [re]start</command></programlisting>
      </blockquote>

      <para>That will give you additional progress messages that may make it
      clear which entry in which file is generating the error.</para>

      <para>If that didn't help, then do the following:</para>

      <itemizedlist>
        <listitem>
          <para>Make a note of the error message that you see.</para>
        </listitem>

        <listitem>
          <para><command>shorewall debug start 2&gt;
          /tmp/trace</command></para>
        </listitem>

        <listitem>
          <para>Look at the <filename>/tmp/trace</filename> file and see if
          that helps you determine what the problem is. Be sure you find the
          place in the log where the error message you saw is generated -- If
          you are using Shorewall 1.4.0 or later, you should find the message
          near the end of the log.</para>
        </listitem>

        <listitem>
          <para>If you still can't determine what's wrong then see the <ulink
          url="support.htm">support page</ulink>.</para>
        </listitem>
      </itemizedlist>

      <example id="Example1">
        <title>Startup Error</title>

        <para>During startup, a user sees the following:</para>

        <programlisting>Adding Common Rules
iptables: No chain/target/match by that name
Terminated</programlisting>

        <para>A search through the trace for <quote>No chain/target/match by
        that name</quote> turned up the following:</para>

        <programlisting>+ echo 'Adding Common Rules'
+ add_common_rules
+ run_iptables -A reject -p tcp -j REJECT --reject-with tcp-reset
++ echo -A reject -p tcp -j REJECT --reject-with tcp-reset
++ sed 's/!/! /g'
+ iptables -A reject -p tcp -j REJECT --reject-with tcp-reset
iptables: No chain/target/match by that name
</programlisting>

        <para>The command that failed was: <quote><command>iptables -A reject
        -p tcp -j REJECT --reject-with tcp-reset</command></quote>. In this
        case, the user had compiled his own kernel and had forgotten to
        include REJECT target support (see <ulink
        url="kernel.htm">kernel.htm</ulink>)</para>
      </example>
    </section>

    <section id="Start-perl">
      <title>Shorewall-perl</title>

      <para>If the error is detected by the Shorewall-perl compiler, it should
      be fairly obvious where the problem was found. Each error message
      includes the configuration file name and line number where the error was
      detected and often gives the particular item in error. The item is
      either enclosed in parentheses or is at the end following a colon
      (":").</para>

      <para>Example:<programlisting>gateway:~/test # shorewall restart .
Compiling...
   ERROR: Invalid ICMP Type (0/400) : /root/test/rules (line 19)
gateway:~/test # </programlisting>In this case, line 19 in the rules file
      specified an invalid ICMP Type (0/400).</para>

      <para>Additional information about the error can be obtained using the
      'debug' keyword:<programlisting>gateway:~/test # shorewall debug restart .
Compiling...
   ERROR: Invalid ICMP Type (0/400) : /root/test/rules (line 19) at /usr/share/shorewall-perl/Shorewall/Config.pm line 338
        Shorewall::Config::fatal_error('Invalid ICMP Type (0/400)') called at /usr/share/shorewall-perl/Shorewall/Chains.pm line 885
        Shorewall::Chains::validate_icmp('0/400') called at /usr/share/shorewall-perl/Shorewall/Chains.pm line 949
        Shorewall::Chains::do_proto('icmp', '0/400', '-') called at /usr/share/shorewall-perl/Shorewall/Rules.pm line 1055
        Shorewall::Rules::process_rule1('ACCEPT', 'loc', 'net', 'icmp', '0/400', '-', '-', '-', '-', ...) called at /usr/share/shorewall-perl/Shorewall/Rules.pm line 1290
        Shorewall::Rules::process_rule('ACCEPT', 'loc', 'net', 'icmp', '0/400', '-', '-', '-', '-', ...) called at /usr/share/shorewall-perl/Shorewall/Rules.pm line 1336
        Shorewall::Rules::process_rules() called at /usr/share/shorewall-perl/Shorewall/Compiler.pm line 799
        Shorewall::Compiler::compiler('/var/lib/shorewall/.restart', '/root/test', 0, 4) called at /usr/share/shorewall-perl/compiler.pl line 86
gateway:~/test # </programlisting>This information is useful to Shorewall
      support if you need to <ulink url="support.html">file a problem
      report</ulink>.</para>

      <para>The end of the compile phase is signaled by a message such as the
      following:<programlisting>Shorewall configuration compiled to /var/lib/shorewall/.restart</programlisting>Errors
      occurring past that point are said to occur at
      <firstterm>run-time</firstterm> because they occur during the running of
      the compiled firewall script (/var/lib/shorewall/.restart in the case of
      the above message).</para>

      <para>One common run-time failure is that the iptables-restore program
      encounters an error. This will produce an error such as the
      following:<programlisting>...
Restarting Shorewall....
iptables-restore v1.3.6: No chain/target/match by that name
Error occurred at line: 83
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
   ERROR: iptables-restore Failed. Input is in /var/lib/shorewall/.iptables-restore-input
Restoring Shorewall...
Shorewall restored from /var/lib/shorewall/restore
Terminated
gateway:~/test # </programlisting>A look at /var/lib/shorewall/restore at line
      83 might show something like the following:<programlisting>-A reject -p tcp -j REJECT --reject-with tcp-reset</programlisting>In
      this case, the user had compiled his own kernel and had forgotten to
      include REJECT target support (see <ulink
      url="kernel.htm">kernel.htm</ulink>).</para>

      <para>f you are running Shorewall-perl 4.0.5 or later, you may also
      include the word <emphasis role="bold">debug</emphasis> as the first
      argument to the <filename>/sbin/shorewall</filename> and
      <filename>/sbin/shorewall-lite</filename> commands.<programlisting><command>shorewall debug restart</command></programlisting>In
      most cases, <emphasis role="bold">debug</emphasis> is a synonym for
      <emphasis role="bold">trace</emphasis>. The exceptions are:</para>

      <itemizedlist>
        <listitem>
          <para><emphasis role="bold">debug</emphasis> is ignored by the
          Shorewall-perl compiler.</para>
        </listitem>

        <listitem>
          <para><emphasis role="bold">debug</emphasis> causes altered behavior
          of scripts generated by the Shorewall-perl compiler. These scripts
          normally use<command> iptables-restore</command> to install the
          Netfilter ruleset but with <emphasis role="bold">debug</emphasis>,
          the commands normally passed to <command>iptables-restore</command>
          in its input file are passed individually to
          <command>iptables</command>. This is a diagnostic aid which allows
          identifying the individual command that is causing
          <command>iptables-restore</command> to fail; it should be used when
          iptables-restore fails when executing a <command>COMMIT</command>
          command.</para>
        </listitem>
      </itemizedlist>

      <warning>
        <para> The <emphasis role="bold">debug</emphasis> feature is strictly
        for problem analysis. When <emphasis role="bold">debug</emphasis> is
        used:</para>

        <orderedlist>
          <listitem>
            <para>The firewall is made 'wide open' before the rules are
            applied.</para>
          </listitem>

          <listitem>
            <para>The <filename>routestopped</filename> file is not
            consulted.</para>
          </listitem>

          <listitem>
            <para>The rules are applied in the canonical
            <command>iptables-restore</command> order. So if you need critical
            hosts to be always available during start/restart, you may not be
            able to use <emphasis role="bold">debug</emphasis>.</para>
          </listitem>
        </orderedlist>
      </warning>

      <para>In other run-time failure cases:<itemizedlist>
          <listitem>
            <para>Make a note of the error message that you see.</para>
          </listitem>

          <listitem>
            <para><command>shorewall debug start 2&gt;
            /tmp/trace</command></para>
          </listitem>

          <listitem>
            <para>Look at the <filename>/tmp/trace</filename> file and see if
            that helps you determine what the problem is. Be sure you find the
            place in the log where the error message you saw is generated --
            you should find the message near the end of the log.</para>
          </listitem>

          <listitem>
            <para>If you still can't determine what's wrong then see the
            <ulink url="support.htm">support page</ulink>.</para>
          </listitem>
        </itemizedlist></para>
    </section>
  </section>

  <section id="Network">
    <title>Your Network Environment</title>

    <para>Many times when people have problems with Shorewall, the problem is
    actually an ill-conceived network setup. Here are several popular
    snafus:</para>

    <itemizedlist>
      <listitem>
        <para>Port Forwarding where client and server are in the same subnet.
        See <ulink url="FAQ.htm#faq2">FAQ 2</ulink>.</para>
      </listitem>

      <listitem>
        <para>Trying to test net-&gt;loc DNAT rules from inside your firewall.
        You must test these rules from <emphasis
        role="bold">outside</emphasis> your firewall.</para>
      </listitem>

      <listitem>
        <para>Multiple interfaces connected to the same HUB or Switch. Given
        the way that the Linux kernel respond to ARP <quote>who-has</quote>
        requests, this type of setup <emphasis role="bold">does NOT work the
        way that you expect it to</emphasis>. You can test using this kind of
        configuration if you specify the <emphasis
        role="bold">arp_filter</emphasis> option or the <emphasis
        role="bold">arp_ignore</emphasis> option in <filename><ulink
        url="manpages/shorewall-interfaces.html">/etc/shorewall/interfaces</ulink></filename>
        for all interfaces connected to the common hub/switch. <emphasis
        role="bold">Using such a setup with a production firewall is strongly
        recommended against</emphasis>.</para>
      </listitem>
    </itemizedlist>
  </section>

  <section id="NewDevice">
    <title>New Device Doesn't Work?</title>

    <para>If you have just added a new device such as VOIP and it doesn't
    work, be sure that you have assigned it an IP address in your local
    network and that its default gateway has been set to the IP address of
    your internal interface. For many of these devices, the simplest solution
    is to run a DHCP server; running it on your firewall is fine — be sure to
    set the <emphasis role="bold">dhcp</emphasis> option on your internal
    interface in <ulink
    url="manpages/shorewall-interfaces.html">/etc/shorewall/interfaces</ulink>.</para>
  </section>

  <section id="Connections">
    <title>Connection Problems</title>

    <para>One very important thing to remember is that not all connection
    problems are Shorewall configuration problems. If the connection that is
    giving you problems is to or from the firewall system or if it doesn't
    rely on NAT or Proxy ARP then you can often eliminate Shorewall using a
    simple test:</para>

    <itemizedlist>
      <listitem>
        <para><command>/sbin/shorewall clear</command></para>
      </listitem>

      <listitem>
        <para>Try the connection. If it works then the problem is in your
        Shorewall configuration; if the connection still doesn't work then the
        problem is not with Shorewall or the way that it is configured.</para>
      </listitem>

      <listitem>
        <para>Be sure to <command>/sbin/shorewall start</command> after the
        test.</para>
      </listitem>
    </itemizedlist>

    <para>If you still suspect Shorewall and the appropriate policy for the
    connection that you are trying to make is ACCEPT, please DO NOT ADD
    ADDITIONAL ACCEPT RULES TRYING TO MAKE IT WORK. Such additional rules will
    NEVER make it work, they add clutter to your rule set and they represent a
    big security hole in the event that you forget to remove them
    later.</para>

    <para>I also recommend against setting all of your policies to ACCEPT in
    an effort to make something work. That robs you of one of your best
    diagnostic tools - the <quote>Shorewall</quote> messages that Netfilter
    will generate when you try to connect in a way that isn't permitted by
    your rule set.</para>

    <para>Check your log (<quote><command>/sbin/shorewall show
    log</command></quote>). If you don't see Shorewall messages, then your
    problem is probably NOT a Shorewall problem. If you DO see packet
    messages, it may be an indication that you are missing one or more rules
    -- see <ulink url="FAQ.htm#faq17">FAQ 17</ulink>.</para>

    <para>While you are troubleshooting, it is a good idea to clear two
    variables in
    <filename><filename>/etc/shorewall/shorewall.conf</filename></filename>:</para>

    <para><programlisting>LOGRATE=
LOGBURST=""</programlisting>This way, you will see all of the log messages
    being generated (be sure to restart shorewall after clearing these
    variables).</para>

    <example id="Example2">
      <title>Log Message</title>

      <programlisting>Jun 27 15:37:56 gateway kernel: Shorewall:all2all:REJECT:IN=eth2
                                OUT=eth1 SRC=192.168.2.2
                                DST=192.168.1.3 LEN=67 TOS=0x00
                                PREC=0x00 TTL=63 ID=5805 DF
                                PROTO=UDP SPT=1803 DPT=53 LEN=47</programlisting>

      <para>Let's look at the important parts of this message:</para>

      <itemizedlist>
        <listitem>
          <para>all2all:REJECT - This packet was REJECTed out of the all2all
          chain -- the packet was rejected under the
          <quote>all</quote>-&gt;<quote>all</quote> REJECT policy (see <ulink
          url="FAQ.htm#faq17">FAQ 17</ulink>).</para>
        </listitem>

        <listitem>
          <para>IN=eth2 - the packet entered the firewall via eth2</para>
        </listitem>

        <listitem>
          <para>OUT=eth1 - if accepted, the packet would be sent on
          eth1</para>
        </listitem>

        <listitem>
          <para>SRC=192.168.2.2 - the packet was sent by 192.168.2.2</para>
        </listitem>

        <listitem>
          <para>DST=192.168.1.3 - the packet is destined for
          192.168.1.3</para>
        </listitem>

        <listitem>
          <para>PROTO=UDP - UDP Protocol</para>
        </listitem>

        <listitem>
          <para>DPT=53 - DNS</para>
        </listitem>
      </itemizedlist>

      <para>In this case, 192.168.2.2 was in the <quote>dmz</quote> zone and
      192.168.1.3 is in the <quote>loc</quote> zone. I was missing the
      rule:</para>

      <programlisting>#ACTION   SOURCE           DEST                  PROTO   DEST
#                                                        PORT(S)
ACCEPT    dmz              loc                   udp     53</programlisting>
    </example>
  </section>

  <section id="Ping">
    <title>Ping Problems</title>

    <para>Either can't ping when you think you should be able to or are able
    to ping when you think that you shouldn't be allowed? Shorewall's
    <quote>Ping</quote> Management is <ulink url="ping.html">described
    here</ulink>. Here are a couple of tips:</para>

    <itemizedlist>
      <listitem>
        <para>Remember that Shorewall doesn't automatically allow ICMP type 8
        (<quote>ping</quote>) requests to be sent between zones. If you want
        pings to be allowed between zones, you need a rule of the form:</para>

        <programlisting>#ACTION  SOURCE          DEST                  PROTO   DEST
#                                                      PORT(S)
Ping/ACCEPT <emphasis>&lt;source zone&gt;</emphasis> <emphasis>&lt;destination zone&gt;</emphasis></programlisting>

        <para>The ramifications of this can be subtle. For example, if you
        have the following in <filename><ulink
        url="NAT.htm">/etc/shorewall/nat</ulink></filename>:</para>

        <programlisting>#EXTERNAL   INTERFACE  INTERNAL
10.1.1.2 eth0       130.252.100.18</programlisting>

        <para>and you ping 130.252.100.18, unless you have allowed icmp type 8
        between the zone containing the system you are pinging from and the
        zone containing 10.1.1.2, the ping requests will be dropped.</para>
      </listitem>

      <listitem>
        <para>Ping requests are subject to logging under your policies. So
        ping floods can cause an equally big flood of log messages. To
        eliminate these, as the last rule in your /etc/shorewall/rules file
        add:</para>

        <programlisting>#ACTION  SOURCE          DEST                  PROTO   DEST
#                                                      PORT(S)
Ping/DROP net             all</programlisting>
      </listitem>
    </itemizedlist>
  </section>

  <section id="Other">
    <title>Some Things to Keep in Mind</title>

    <itemizedlist>
      <listitem>
        <para><emphasis role="bold">You cannot test your firewall from the
        inside</emphasis>. Just because you send requests to your firewall
        external IP address does not mean that the request will be associated
        with the external interface or the <quote>net</quote> zone. Any
        traffic that you generate from the local network will be associated
        with your local interface and will be treated as loc-&gt;fw
        traffic.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">IP addresses are properties of systems,
        not of interfaces</emphasis>. It is a mistake to believe that your
        firewall is able to forward packets just because you can ping the IP
        address of all of the firewall's interfaces from the local network.
        The only conclusion you can draw from such pinging success is that the
        link between the local system and the firewall works and that you
        probably have the local system's default gateway set correctly.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">All IP addresses configured on firewall
        interfaces are in the $FW (fw) zone</emphasis>. If 192.168.1.254 is
        the IP address of your internal interface then you can write
        <quote><emphasis role="bold">$FW:192.168.1.254</emphasis></quote> in a
        rule but you may not write <quote><emphasis
        role="bold">loc:192.168.1.254</emphasis></quote>. Similarly, it is
        nonsensical to add 192.168.1.254 to the <emphasis
        role="bold">loc</emphasis> zone using an entry in
        <filename>/etc/shorewall/hosts</filename>.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">Reply packets do NOT automatically follow
        the reverse path of the one taken by the original request</emphasis>.
        All packets are routed according to the routing table of the host at
        each step of the way. This issue commonly comes up when people install
        a Shorewall firewall parallel to an existing gateway and try to use
        DNAT through Shorewall without changing the default gateway of the
        system receiving the forwarded requests. Requests come in through the
        Shorewall firewall where the destination IP address gets rewritten but
        replies go out unmodified through the old gateway.</para>
      </listitem>

      <listitem>
        <para><emphasis role="bold">Shorewall itself has no notion of inside
        or outside</emphasis>. These concepts are embodied in how Shorewall is
        configured.</para>
      </listitem>
    </itemizedlist>
  </section>

  <section id="More">
    <title>Other Gotchas</title>

    <itemizedlist>
      <listitem>
        <para>Seeing rejected/dropped packets logged out of the INPUT or
        FORWARD chains? This means that:</para>

        <orderedlist>
          <listitem>
            <para>your zone definitions are screwed up and the host that is
            sending the packets or the destination host isn't in any zone
            (using an <ulink
            url="manpages/shorewall-hosts.html"><filename>/etc/shorewall/hosts</filename></ulink>
            file are you?); or</para>
          </listitem>

          <listitem>
            <para>the source and destination hosts are both connected to the
            same interface and you don't have a policy or rule for the source
            zone to or from the destination zone or you haven't set the
            <emphasis role="bold">routeback</emphasis> option for the
            interface in <ulink
            url="manpages/shorewall-interfaces.html"><filename>/etc/shorewall/interfaces</filename></ulink>.</para>
          </listitem>

          <listitem>
            <para>You have connected two firewall interfaces (from different
            zones) to the same hub or switch.</para>
          </listitem>
        </orderedlist>
      </listitem>

      <listitem>
        <para>If you specify <quote>routefilter</quote> for an interface, that
        interface must be up prior to starting the firewall.</para>
      </listitem>

      <listitem>
        <para>Is your routing correct? For example, internal systems usually
        need to be configured with their default gateway set to the IP address
        of their nearest firewall interface. One often overlooked aspect of
        routing is that in order for two hosts to communicate, the routing
        between them must be set up <emphasis role="bold">in both
        directions</emphasis>. So when setting up routing between <emphasis
        role="bold">A</emphasis> and <emphasis role="bold">B</emphasis>, be
        sure to verify that the route from <emphasis role="bold">B</emphasis>
        back to <emphasis role="bold">A</emphasis> is defined and
        correct.</para>
      </listitem>

      <listitem>
        <para>Do you have your kernel properly configured? <ulink
        url="kernel.htm">Click here to see kernel configuration
        information</ulink>.</para>
      </listitem>
    </itemizedlist>
  </section>

  <section id="Support">
    <title>Still Having Problems?</title>

    <para>See the <ulink url="support.htm">Shorewall Support
    Page</ulink>.</para>
  </section>
</article>