mirror of
https://gitlab.com/shorewall/code.git
synced 2024-11-24 00:23:28 +01:00
afee989ee5
git-svn-id: https://shorewall.svn.sourceforge.net/svnroot/shorewall/trunk@1099 fbd18981-670d-0410-9b5c-8dc0c1a9a2bb
1568 lines
68 KiB
XML
1568 lines
68 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
|
<article>
|
|
<!--$Id$-->
|
|
|
|
<articleinfo>
|
|
<title>Shorewall FAQs</title>
|
|
|
|
<authorgroup>
|
|
<corpauthor>Shorewall Community</corpauthor>
|
|
|
|
<author>
|
|
<firstname>Tom</firstname>
|
|
|
|
<surname>Eastep</surname>
|
|
</author>
|
|
</authorgroup>
|
|
|
|
<pubdate>2004-01-25</pubdate>
|
|
|
|
<copyright>
|
|
<year>2001-2004</year>
|
|
|
|
<holder>Thomas M. Eastep</holder>
|
|
</copyright>
|
|
|
|
<legalnotice>
|
|
<para>Permission is granted to copy, distribute and/or modify this
|
|
document under the terms of the GNU Free Documentation License, Version
|
|
1.2 or any later version published by the Free Software Foundation; with
|
|
no Invariant Sections, with no Front-Cover, and with no Back-Cover
|
|
Texts. A copy of the license is included in the section entitled
|
|
<quote><ulink url="GnuCopyright.htm">GNU Free Documentation License</ulink></quote>.</para>
|
|
</legalnotice>
|
|
</articleinfo>
|
|
|
|
<section>
|
|
<title>Installing Shorewall</title>
|
|
|
|
<section>
|
|
<title>Where do I find Step by Step Installation and Configuration
|
|
Instructions?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Check out the <ulink
|
|
url="shorewall_quickstart_guide.htm">QuickStart Guides</ulink>.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Port Forwarding</title>
|
|
|
|
<section id="faq1">
|
|
<title>(FAQ 1) I want to forward UDP port 7777 to my my personal PC with
|
|
IP address 192.168.1.5. I've looked everywhere and can't find
|
|
how to do it.</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> The first example in the
|
|
<ulink url="Documentation.htm#Rules">rules file documentation</ulink>
|
|
shows how to do port forwarding under Shorewall. The format of a
|
|
port-forwarding rule to a local system is as follows:</para>
|
|
|
|
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT
|
|
DNAT net loc:<l<emphasis>ocal IP address</emphasis>>[:<<emphasis>local port</emphasis>>] <<emphasis>protocol</emphasis>> <<emphasis>port #</emphasis>></programlisting>
|
|
|
|
<para>So to forward UDP port 7777 to internal system 192.168.1.5, the
|
|
rule is:</para>
|
|
|
|
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT
|
|
DNAT net loc:192.168.1.5 udp 7777</programlisting>
|
|
|
|
<para>If you want to forward requests directed to a particular address (
|
|
<emphasis><external IP></emphasis> ) on your firewall to an
|
|
internal system:</para>
|
|
|
|
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT SOURCE ORIGINAL
|
|
# PORT DEST.
|
|
DNAT net loc:<l<emphasis>ocal IP address</emphasis>>[:<<emphasis>local port</emphasis>>] <<emphasis>protocol</emphasis>> <<emphasis>port #</emphasis>> - <<emphasis>external IP</emphasis>></programlisting>
|
|
|
|
<para>Finally, if you need to forward a range of ports, in the PORT
|
|
column specify the range as <emphasis><low-port>:<high-port></emphasis>.</para>
|
|
|
|
<section id="faq1a">
|
|
<title>(FAQ 1a) Ok -- I followed those instructions but it doesn't
|
|
work</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> That is usually the
|
|
result of one of four things:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>You are trying to test from inside your firewall (no, that
|
|
won't work -- see <xref linkend="faq2" />).</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>You have a more basic problem with your local system (the
|
|
one that you are trying to forward to) such as an incorrect
|
|
default gateway (it should be set to the IP address of your
|
|
firewall's internal interface).</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Your ISP is blocking that particular port inbound.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>You are running Mandrake Linux and have configured Internet
|
|
Connection Sharing. In that case, the name of your local zone is
|
|
'masq' rather than 'loc' (change all instances of
|
|
'loc' to 'masq' in your rules). You may want to
|
|
consider re-installing Shorewall in a configuration which matches
|
|
the Shorewall documentation. See the <ulink
|
|
url="two-interface.htm">two-interface QuickStart Guide</ulink> for
|
|
details.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
|
|
<section id="faq1b">
|
|
<title>(FAQ 1b) I'm still having problems with port forwarding</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> To further diagnose
|
|
this problem:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>As root, type <quote><command>iptables -t nat -Z</command></quote>.
|
|
This clears the NetFilter counters in the nat table.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Try to connect to the redirected port from an external host.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>As root type <quote><command>shorewall show nat</command></quote></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Locate the appropriate DNAT rule. It will be in a chain
|
|
called <emphasis><source zone></emphasis>_dnat (<quote>net_dnat</quote>
|
|
in the above examples).</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Is the packet count in the first column non-zero? If so, the
|
|
connection request is reaching the firewall and is being
|
|
redirected to the server. In this case, the problem is usually a
|
|
missing or incorrect default gateway setting on the local system
|
|
(the system you are trying to forward to -- its default gateway
|
|
should be the IP address of the firewall's interface to that
|
|
system).</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>If the packet count is zero:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>the connection request is not reaching your server
|
|
(possibly it is being blocked by your ISP); or</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>you are trying to connect to a secondary IP address on
|
|
your firewall and your rule is only redirecting the primary IP
|
|
address (You need to specify the secondary IP address in the
|
|
<quote>ORIG. DEST.</quote> column in your DNAT rule); or</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>your DNAT rule doesn't match the connection request
|
|
in some other way. In that case, you may have to use a packet
|
|
sniffer such as tcpdump or ethereal to further diagnose the
|
|
problem.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
|
|
<section id="faq1c">
|
|
<title>(FAQ 1c) From the internet, I want to connect to port 1022 on
|
|
my firewall and have the firewall forward the connection to port 22 on
|
|
local system 192.168.1.3. How do I do that?</title>
|
|
|
|
<para>In /<filename>etc/shorewall/rules</filename>:</para>
|
|
|
|
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT
|
|
DNAT net loc:192.168.3:22 tcp 1022</programlisting>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="faq30">
|
|
<title>(FAQ 30) I'm confused about when to use DNAT rules and when
|
|
to use ACCEPT rules.</title>
|
|
|
|
<para>It would be a good idea to review the <ulink
|
|
url="shorewall_quickstart_guide.htm">QuickStart Guide</ulink>
|
|
appropriate for your setup; the guides cover this topic in a tutorial
|
|
fashion. DNAT rules should be used for connections that need to go the
|
|
opposite direction from SNAT/MASQUERADE. So if you masquerade or use
|
|
SNAT from your local network to the internet then you will need to use
|
|
DNAT rules to allow connections from the internet to your local network.
|
|
In all other cases, you use ACCEPT unless you need to hijack connections
|
|
as they go through your firewall and handle them on the firewall box
|
|
itself; in that case, you use a REDIRECT rule.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>DNS and Port Forwarding/NAT</title>
|
|
|
|
<section id="faq2">
|
|
<title>(FAQ 2) I port forward www requests to www.mydomain.com (IP
|
|
130.151.100.69) to system 192.168.1.5 in my local network. External
|
|
clients can browse http://www.mydomain.com but internal clients
|
|
can't.</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> I have two objections to
|
|
this setup.</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>Having an internet-accessible server in your local network is
|
|
like raising foxes in the corner of your hen house. If the server is
|
|
compromised, there's nothing between that server and your other
|
|
internal systems. For the cost of another NIC and a cross-over
|
|
cable, you can put your server in a DMZ such that it is isolated
|
|
from your local systems - assuming that the Server can be located
|
|
near the Firewall, of course :-)</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>The accessibility problem is best solved using <ulink
|
|
url="shorewall_setup_guide.htm#DNS">Bind Version 9 <quote>views</quote></ulink>
|
|
(or using a separate DNS server for local clients) such that
|
|
www.mydomain.com resolves to 130.141.100.69 externally and
|
|
192.168.1.5 internally. That's what I do here at shorewall.net
|
|
for my local systems that use one-to-one NAT.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>If you insist on an IP solution to the accessibility problem
|
|
rather than a DNS solution, then assuming that your external interface
|
|
is eth0 and your internal interface is eth1 and that eth1 has IP address
|
|
192.168.1.254 with subnet 192.168.1.0/24.</para>
|
|
|
|
<para>If you are running Shorewall 1.4.0 or earlier see the <ulink
|
|
url="1.3/FAQ.htm#faq2">1.3 FAQ</ulink> for instructions suitable for
|
|
those releases.</para>
|
|
|
|
<para>If you are running Shorewall 1.4.1 or Shorewall 1.4.1a, please
|
|
upgrade to Shorewall 1.4.2 or later.</para>
|
|
|
|
<para>Otherwise:<warning><para>In this configuration, all loc->loc
|
|
traffic will look to the server as if it came from the firewall rather
|
|
than from the original client!</para></warning></para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>In <filename>/etc/shorewall/interfaces</filename>:</para>
|
|
|
|
<programlisting>#ZONE INTERFACE BROADCAST OPTIONS
|
|
loc eth1 detect <emphasis role="bold">routeback</emphasis></programlisting>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>In <filename>/etc/shorewall/rules</filename>:</para>
|
|
|
|
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT SOURCE ORIGINAL
|
|
# PORT DEST.
|
|
DNAT loc loc:192.168.1.5 tcp www - 130.151.100.69:192.168.1.254</programlisting>
|
|
|
|
<para>That rule only works of course if you have a static external
|
|
IP address. If you have a dynamic IP address and are running
|
|
Shorewall 1.3.4 or later then include this in <filename>/etc/shorewall/init</filename>:</para>
|
|
|
|
<programlisting><command>ETH0_IP=`find_interface_address eth0`</command></programlisting>
|
|
|
|
<para>and make your DNAT rule:</para>
|
|
|
|
<programlisting>#ACTION SOURCE DEST PROTO DEST PORT SOURCE ORIGINAL
|
|
# PORT DEST.
|
|
DNAT loc loc:192.168.1.5 tcp www - $ETH0_IP:192.168.1.254</programlisting>
|
|
|
|
<para>Using this technique, you will want to configure your
|
|
DHCP/PPPoE client to automatically restart Shorewall each time that
|
|
you get a new IP address.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<section id="faq2a">
|
|
<title>(FAQ 2a) I have a zone <quote>Z</quote> with an RFC1918 subnet
|
|
and I use one-to-one NAT to assign non-RFC1918 addresses to hosts in
|
|
Z. Hosts in Z cannot communicate with each other using their external
|
|
(non-RFC1918 addresses) so they can't access each other using
|
|
their DNS names.</title>
|
|
|
|
<note>
|
|
<para>If the ALL INTERFACES column in /etc/shorewall/nat is empty or
|
|
contains <quote>Yes</quote>, you will also see log messages like the
|
|
following when trying to access a host in Z from another host in Z
|
|
using the destination hosts's public address:</para>
|
|
|
|
<programlisting>Oct 4 10:26:40 netgw kernel:
|
|
Shorewall:FORWARD:REJECT:IN=eth1 OUT=eth1 SRC=192.168.118.200
|
|
DST=192.168.118.210 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=1342 DF
|
|
PROTO=TCP SPT=1494 DPT=1491 WINDOW=17472 RES=0x00 ACK SYN URGP=0</programlisting>
|
|
</note>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> This is another problem
|
|
that is best solved using Bind Version 9 <quote>views</quote>. It
|
|
allows both external and internal clients to access a NATed host using
|
|
the host's DNS name.</para>
|
|
|
|
<para>Another good way to approach this problem is to switch from
|
|
one-to-one NAT to Proxy ARP. That way, the hosts in Z have non-RFC1918
|
|
addresses and can be accessed externally and internally using the same
|
|
address.</para>
|
|
|
|
<para>If you don't like those solutions and prefer routing all
|
|
Z->Z traffic through your firewall then:</para>
|
|
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>Set the Z->Z policy to ACCEPT.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Masquerade Z to itself.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Set the routeback option on the interface to Z.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Set the ALL INTERFACES column in the nat file to
|
|
<quote>Yes</quote>.</para>
|
|
|
|
<warning>
|
|
<para>In this configuration, all Z->Z traffic will look to
|
|
the server as if it came from the firewall rather than from the
|
|
original client! I DO NOT RECOMMEND THIS SETUP.</para>
|
|
</warning>
|
|
</listitem>
|
|
</orderedlist>
|
|
|
|
<example>
|
|
<title>Example:</title>
|
|
|
|
<literallayout>Zone: dmz Interface: eth2 Subnet: 192.168.2.0/24</literallayout>
|
|
|
|
<para>In <filename>/etc/shorewall/interfaces</filename>:</para>
|
|
|
|
<programlisting>#ZONE INTERFACE BROADCAST OPTIONS
|
|
loc eth2 192.168.2.255 <emphasis role="bold">routeback</emphasis></programlisting>
|
|
|
|
<para>In <filename>/etc/shorewall/policy</filename>:</para>
|
|
|
|
<programlisting>#SOURCE DESTINATION POLICY LIMIT:BURST
|
|
dmz dmz ACCEPT</programlisting>
|
|
|
|
<para>In <filename>/etc/shorewall/masq</filename>:</para>
|
|
|
|
<programlisting>#INTERFACE SUBNET ADDRESS
|
|
eth2 192.168.2.0/24</programlisting>
|
|
|
|
<para>In <filename>/etc/shorewall/na</filename>t, be sure that you
|
|
have <quote>Yes</quote> in the ALL INTERFACES column.</para>
|
|
</example>
|
|
</section>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Netmeeting/MSN</title>
|
|
|
|
<section id="faq3">
|
|
<title>(FAQ 3) I want to use Netmeeting or MSN Instant Messenger with
|
|
Shorewall. What do I do?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> There is an <ulink
|
|
url="http://www.kfki.hu/%7Ekadlec/sw/netfilter/newnat-suite/">H.323
|
|
connection tracking/NAT module</ulink> that helps with Netmeeting. Look
|
|
<ulink url="http://linux-igd.sourceforge.net">here</ulink> for a
|
|
solution for MSN IM but be aware that there are significant security
|
|
risks involved with this solution. Also check the Netfilter mailing list
|
|
archives at <ulink url="http://www.netfilter.org">http://www.netfilter.org</ulink>.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Open Ports</title>
|
|
|
|
<section id="faq4">
|
|
<title>(FAQ 4) I just used an online port scanner to check my firewall
|
|
and it shows some ports as <quote>closed</quote> rather than
|
|
<quote>blocked</quote>. Why?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> The common.def included
|
|
with version 1.3.x always rejects connection requests on TCP port 113
|
|
rather than dropping them. This is necessary to prevent outgoing
|
|
connection problems to services that use the <quote>Auth</quote>
|
|
mechanism for identifying requesting users. Shorewall also rejects TCP
|
|
ports 135, 137, 139 and 445 as well as UDP ports 137-139. These are
|
|
ports that are used by Windows (Windows <emphasis>can</emphasis> be
|
|
configured to use the DCE cell locator on port 135). Rejecting these
|
|
connection requests rather than dropping them cuts down slightly on the
|
|
amount of Windows chatter on LAN segments connected to the Firewall.</para>
|
|
|
|
<para>If you are seeing port 80 being <quote>closed</quote>, that's
|
|
probably your ISP preventing you from running a web server in violation
|
|
of your Service Agreement.</para>
|
|
|
|
<tip>
|
|
<para>You can change the default behavior of Shorewall through use of
|
|
an /etc/shorewall/common file. See the <ulink
|
|
url="shorewall_extension_scripts.htm">Extension Script Section</ulink>.</para>
|
|
</tip>
|
|
|
|
<tip>
|
|
<para>Beginning with Shorewall 1.4.9, Shorewall no longer rejects the
|
|
Windows SMB ports (135-139 and 445) by default and silently drops them
|
|
instead.</para>
|
|
</tip>
|
|
|
|
<section id="faq4a">
|
|
<title>(FAQ 4a) I just ran an nmap UDP scan of my firewall and it
|
|
showed 100s of ports as open!!!!</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Take a deep breath and
|
|
read the nmap man page section about UDP scans. If nmap gets <emphasis
|
|
role="bold">nothing</emphasis> back from your firewall then it reports
|
|
the port as open. If you want to see which UDP ports are really open,
|
|
temporarily change your net->all policy to REJECT, restart
|
|
Shorewall and do the nmap UDP scan again.</para>
|
|
</section>
|
|
|
|
<section id="faq4b">
|
|
<title>(FAQ 4b) I have a port that I can't close no matter how I
|
|
change my rules.</title>
|
|
|
|
<para>I had a rule that allowed telnet from my local network to my
|
|
firewall; I removed that rule and restarted Shorewall but my telnet
|
|
session still works!!!</para>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Rules only govern the
|
|
establishment of new connections. Once a connection is established
|
|
through the firewall it will be usable until disconnected (tcp) or
|
|
until it times out (other protocols). If you stop telnet and try to
|
|
establish a new session your firerwall will block that attempt.</para>
|
|
</section>
|
|
|
|
<section id="faq4c">
|
|
<title>(FAQ 4c) How to I use Shorewall with PortSentry?</title>
|
|
|
|
<para><ulink
|
|
url="http://www.shorewall.net/pub/shorewall/contrib/PortsentryHOWTO.txt">Here's
|
|
a writeup</ulink> on a nice integration of Shorewall and PortSentry.</para>
|
|
</section>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Connection Problems</title>
|
|
|
|
<section id="faq5">
|
|
<title>(FAQ 5) I've installed Shorewall and now I can't ping
|
|
through the firewall</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> If you want your firewall
|
|
to be totally open for <quote>ping</quote>,</para>
|
|
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>Create <filename>/etc/shorewall/common</filename> if it
|
|
doesn't already exist.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Be sure that the first command in the file is <quote>.
|
|
<filename>/etc/shorewall/common.de</filename>f</quote></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Add the following to <filename>/etc/shorewall/common</filename></para>
|
|
|
|
<programlisting><command>run_iptables -A icmpdef -p ICMP --icmp-type echo-request -j ACCEPT</command></programlisting>
|
|
</listitem>
|
|
</orderedlist>
|
|
|
|
<para>For a complete description of Shorewall <quote>ping</quote>
|
|
management, see <ulink url="ping.html">this page</ulink>.</para>
|
|
</section>
|
|
|
|
<section id="faq15">
|
|
<title>(FAQ 15) My local systems can't see out to the net</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Every time I read
|
|
<quote>systems can't see out to the net</quote>, I wonder where the
|
|
poster bought computers with eyes and what those computers will
|
|
<quote>see</quote> when things are working properly. That aside, the
|
|
most common causes of this problem are:</para>
|
|
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>The default gateway on each local system isn't set to the
|
|
IP address of the local firewall interface.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>The entry for the local network in the /etc/shorewall/masq
|
|
file is wrong or missing.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>The DNS settings on the local systems are wrong or the user is
|
|
running a DNS server on the firewall and hasn't enabled UDP and
|
|
TCP port 53 from the firewall to the internet.</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</section>
|
|
|
|
<section id="faq29">
|
|
<title>(FAQ 29) FTP Doesn't Work</title>
|
|
|
|
<para>See the <ulink url="FTP.html">Shorewall and FTP page</ulink>.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Logging</title>
|
|
|
|
<section id="faq6">
|
|
<title>(FAQ 6) Where are the log messages written and how do I change
|
|
the destination?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> NetFilter uses the
|
|
kernel's equivalent of syslog (see <quote>man syslog</quote>) to log
|
|
messages. It always uses the LOG_KERN (kern) facility (see
|
|
<quote>man openlog</quote>) and you get to choose the log level (again,
|
|
see <quote>man syslog</quote>) in your <ulink
|
|
url="Documentation.htm#Policy">policies</ulink> and <ulink
|
|
url="Documentation.htm#Rules">rules</ulink>. The destination for
|
|
messaged logged by syslog is controlled by <filename>/etc/syslog.conf</filename>
|
|
(see <quote>man syslog.conf</quote>). When you have changed
|
|
/etc/syslog.conf, be sure to restart syslogd (on a RedHat system,
|
|
<quote>service syslog restart</quote>).</para>
|
|
|
|
<para>By default, older versions of Shorewall ratelimited log messages
|
|
through <ulink url="Documentation.htm#Conf">settings</ulink> in
|
|
<filename>/etc/shorewall/shorewall.conf</filename> -- If you want to log
|
|
all messages, set:</para>
|
|
|
|
<programlisting>LOGLIMIT=""
|
|
LOGBURST=""</programlisting>
|
|
|
|
<para>Beginning with Shorewall version 1.3.12, you can <ulink
|
|
url="shorewall_logging.html">set up Shorewall to log all of its messages
|
|
to a separate file</ulink>.</para>
|
|
|
|
<section id="faq6a">
|
|
<title>(FAQ 6a) Are there any log parsers that work with Shorewall?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Here are several links
|
|
that may be helpful:</para>
|
|
|
|
<literallayout><ulink
|
|
url="http://www.shorewall.net/pub/shorewall/parsefw/">http://www.shorewall.net/pub/shorewall/parsefw/</ulink>
|
|
<ulink url="http://www.fireparse.com">http://www.fireparse.com</ulink>
|
|
<ulink url="http://cert.uni-stuttgart.de/projects/fwlogwatch">http://cert.uni-stuttgart.de/projects/fwlogwatch</ulink>
|
|
<ulink url="http://www.logwatch.org">http://www.logwatch.org</ulink>
|
|
<ulink url="http://gege.org/iptables">http://gege.org/iptables</ulink>
|
|
<ulink url="http://home.regit.org/ulogd-php.html">http://home.regit.org/ulogd-php.html</ulink></literallayout>
|
|
|
|
<para>I personnaly use Logwatch. It emails me a report each day from
|
|
my various systems with each report summarizing the logged activity on
|
|
the corresponding system.</para>
|
|
</section>
|
|
|
|
<section id="faq6b">
|
|
<title>(FAQ 2b) DROP messages on port 10619 are flooding the logs with
|
|
their connect requests. Can i exclude these error messages for this
|
|
port temporarily from logging in Shorewall?</title>
|
|
|
|
<para>Temporarily add the following rule:</para>
|
|
|
|
<programlisting>DROP net fw udp 10619</programlisting>
|
|
</section>
|
|
|
|
<section id="faq6c">
|
|
<title>(FAQ 6c) All day long I get a steady flow of these DROP
|
|
messages from port 53 to some high numbered port. They get dropped,
|
|
but what the heck are they?</title>
|
|
|
|
<programlisting>Jan 8 15:50:48 norcomix kernel:
|
|
Shorewall:net2all:DROP:IN=eth0 OUT=
|
|
MAC=00:40:c7:2e:09:c0:00:01:64:4a:70:00:08:00 SRC=208.138.130.16
|
|
DST=24.237.22.45 LEN=53 TOS=0x00 PREC=0x00 TTL=251 ID=8288 DF
|
|
PROTO=UDP SPT=53 DPT=40275 LEN=33</programlisting>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> There are two
|
|
possibilities:</para>
|
|
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>They are late-arriving replies to DNS queries.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>They are corrupted reply packets.</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
|
|
<para>You can distinguish the difference by setting the <emphasis
|
|
role="bold">logunclean</emphasis> option (<filename><ulink
|
|
url="Documentation.htm#Interfaces">/etc/shorewall/interfaces</ulink></filename>)
|
|
on your external interface (eth0 in the above example). If they get
|
|
logged twice, they are corrupted. I solve this problem by using an
|
|
/etc/shorewall/common file like this:</para>
|
|
|
|
<programlisting>#
|
|
# Include the standard common.def file
|
|
#
|
|
<command>. /etc/shorewall/common.def</command>
|
|
#
|
|
# The following rule is non-standard and compensates for tardy
|
|
# DNS replies
|
|
#
|
|
<command>run_iptables -A common -p udp --sport 53 -mstate --state NEW -j DROP</command></programlisting>
|
|
|
|
<para>The above file is also include in all of my sample
|
|
configurations available in the <ulink
|
|
url="shorewall_quickstart_guide.htm">Quick Start Guides</ulink> and in
|
|
the common.def file in Shorewall 1.4.0 and later.</para>
|
|
</section>
|
|
|
|
<section id="faq6d">
|
|
<title>(FAQ 6d) Why is the MAC address in Shorewall log messages so
|
|
long? I thought MAC addresses were only 6 bytes in length.</title>
|
|
|
|
<para>What is labeled as the MAC address in a Shorewall log message is
|
|
actually the Ethernet frame header. It contains:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>the destination MAC address (6 bytes)</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>the source MAC address (6 bytes)</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>the ethernet frame type (2 bytes)</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para><example><title>Example</title><para><programlisting>MAC=00:04:4c:dc:e2:28:00:b0:8e:cf:3c:4c:08:00</programlisting><itemizedlist><listitem><para>Destination
|
|
MAC address = 00:04:4c:dc:e2:28</para></listitem><listitem><para>Source
|
|
MAC address = 00:b0:8e:cf:3c:4c</para></listitem><listitem><para>Ethernet
|
|
Frame Type = 08:00 (IP Version 4)</para></listitem></itemizedlist></para></example></para>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="faq16">
|
|
<title>(FAQ 16) Shorewall is writing log messages all over my console
|
|
making it unusable!</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> If you are running
|
|
Shorewall version 1.4.4 or 1.4.4a then check the <ulink url="errata.htm">errata</ulink>.
|
|
Otherwise:</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>Find where klogd is being started (it will be from one of the
|
|
files in /etc/init.d -- sysklogd, klogd, ...). Modify that file or
|
|
the appropriate configuration file so that klogd is started with
|
|
<quote>-c <emphasis><n></emphasis></quote> where
|
|
<emphasis><n></emphasis> is a log level of 5 or less; or</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>See the <quote>dmesg</quote> man page (<quote>man dmesg</quote>).
|
|
You must add a suitable <quote>dmesg</quote> command to your startup
|
|
scripts or place it in /etc/shorewall/start.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<tip>
|
|
<para>Under RedHat and Mandrake, the max <ulink
|
|
url="shorewall_logging.html">log level</ulink> that is sent to the
|
|
console is specified in /etc/sysconfig/init in the LOGLEVEL variable.
|
|
Set <quote>LOGLEVEL=5</quote> to suppress info (log level 6) messages
|
|
on the console.</para>
|
|
</tip>
|
|
|
|
<tip>
|
|
<para>Under Debian, you can set KLOGD=<quote>-c 5</quote> in
|
|
<filename>/etc/init.d/klogd</filename> to suppress info (log level 6)
|
|
messages on the console.</para>
|
|
</tip>
|
|
|
|
<tip>
|
|
<para>Under SuSE, add <quote>-c 5</quote> to KLOGD_PARAMS in
|
|
/etc/sysconfig/syslog to suppress info (log level 6) messages on the
|
|
console.</para>
|
|
</tip>
|
|
</section>
|
|
|
|
<section id="faq17">
|
|
<title>(FAQ 17) What does this log message mean?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Logging occurs out of a
|
|
number of chains (as indicated in the log message) in Shorewall:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>man1918 or logdrop</term>
|
|
|
|
<listitem>
|
|
<para>The destination address is listed in <filename>/etc/shorewall/rfc1918</filename>
|
|
with a <emphasis role="bold">logdrop</emphasis> target -- see
|
|
<filename><ulink url="Documentation.htm#rfc1918">/etc/shorewall/rfc1918</ulink></filename>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>rfc1918 or logdrop</term>
|
|
|
|
<listitem>
|
|
<para>The source address is listed in <filename>/etc/shorewall/rfc1918</filename>
|
|
with a <emphasis role="bold">logdrop</emphasis> target -- see
|
|
<filename><ulink url="Documentation.htm#rfc1918">/etc/shorewall/rfc1918</ulink></filename>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry id="all2all">
|
|
<term>all2<zone>, <zone>2all or all2all</term>
|
|
|
|
<listitem>
|
|
<para>You have a <ulink url="Documentation.htm#Policy">policy</ulink>
|
|
that specifies a log level and this packet is being logged under
|
|
that policy. If you intend to ACCEPT this traffic then you need a
|
|
<ulink url="Documentation.htm#Rules">rule</ulink> to that effect.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><zone1>2<zone2></term>
|
|
|
|
<listitem>
|
|
<para>Either you have a <ulink url="Documentation.htm#Policy">policy</ulink>
|
|
for <emphasis role="bold"><zone1></emphasis> to <emphasis
|
|
role="bold"><zone2></emphasis> that specifies a log level
|
|
and this packet is being logged under that policy or this packet
|
|
matches a <ulink url="Documentation.htm#Rules">rule</ulink> that
|
|
includes a log level.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><interface>_mac</term>
|
|
|
|
<listitem>
|
|
<para>The packet is being logged under the <emphasis role="bold">maclist</emphasis>
|
|
<ulink url="Documentation.htm#Interfaces">interface option</ulink>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>logpkt</term>
|
|
|
|
<listitem>
|
|
<para>The packet is being logged under the <emphasis role="bold">logunclean</emphasis>
|
|
<ulink url="Documentation.htm#Interfaces">interface option</ulink>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>badpkt</term>
|
|
|
|
<listitem>
|
|
<para>The packet is being logged under the <emphasis role="bold">dropunclean</emphasis>
|
|
<ulink url="Documentation.htm#Interfaces">interface option</ulink>
|
|
as specified in the <emphasis role="bold">LOGUNCLEAN</emphasis>
|
|
setting in <ulink url="Documentation.htm#Conf"><filename>/etc/shorewall/shorewall.conf</filename></ulink>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>blacklst</term>
|
|
|
|
<listitem>
|
|
<para>The packet is being logged because the source IP is
|
|
blacklisted in the <filename><ulink
|
|
url="Documentation.htm#Blacklist">/etc/shorewall/blacklist</ulink></filename>
|
|
file.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>newnotsyn</term>
|
|
|
|
<listitem>
|
|
<para>The packet is being logged because it is a TCP packet that
|
|
is not part of any current connection yet it is not a syn packet.
|
|
Options affecting the logging of such packets include <emphasis
|
|
role="bold">NEWNOTSYN</emphasis> and <emphasis role="bold">LOGNEWNOTSYN</emphasis>
|
|
in <ulink url="Documentation.htm#Conf"><filename>/etc/shorewall/shorewall.conf</filename></ulink>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>INPUT or FORWARD</term>
|
|
|
|
<listitem>
|
|
<para>The packet has a source IP address that isn't in any of
|
|
your defined zones (<quote>shorewall check</quote> and look at the
|
|
printed zone definitions) or the chain is FORWARD and the
|
|
destination IP isn't in any of your defined zones. Also see
|
|
<xref linkend="faq2a" /> for another cause of packets being logged
|
|
in the FORWARD chain.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>logflags</term>
|
|
|
|
<listitem>
|
|
<para>The packet is being logged because it failed the checks
|
|
implemented by the <emphasis role="bold">tcpflags</emphasis>
|
|
<ulink url="Documentation.htm#Interfaces">interface option</ulink>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<example>
|
|
<title>Here is an example:</title>
|
|
|
|
<programlisting>Jun 27 15:37:56 gateway kernel:
|
|
Shorewall:all2all:REJECT:IN=eth2 OUT=eth1 SRC=192.168.2.2
|
|
DST=192.168.1.3 LEN=67 TOS=0x00 PREC=0x00 TTL=63 ID=5805 DF PROTO=UDP
|
|
SPT=1803 DPT=53 LEN=47</programlisting>
|
|
|
|
<para>Let's look at the important parts of this message:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>all2all:REJECT</term>
|
|
|
|
<listitem>
|
|
<para>This packet was REJECTed out of the <emphasis role="bold">all2all</emphasis>
|
|
chain -- the packet was rejected under the <quote>all</quote>-><quote>all</quote>
|
|
REJECT policy (<xref linkend="all2all" /> above).</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>IN=eth2</term>
|
|
|
|
<listitem>
|
|
<para>the packet entered the firewall via eth2. If you see
|
|
<quote>IN=</quote> with no interface name, the packet originated
|
|
on the firewall itself.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>OUT=eth1</term>
|
|
|
|
<listitem>
|
|
<para>if accepted, the packet would be sent on eth1. If you see
|
|
<quote>OUT=</quote> with no interface name, the packet would be
|
|
processed by the firewall itself.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>SRC=192.168.2.2</term>
|
|
|
|
<listitem>
|
|
<para>the packet was sent by 192.168.2.2</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>DST=192.168.1.3</term>
|
|
|
|
<listitem>
|
|
<para>the packet is destined for 192.168.1.3</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>PROTO=UDP</term>
|
|
|
|
<listitem>
|
|
<para>UDP Protocol</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>DPT=53</term>
|
|
|
|
<listitem>
|
|
<para>The destination port is 53 (DNS)</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<para>For additional information about the log message, see <ulink
|
|
url="http://logi.cc/linux/netfilter-log-format.php3">http://logi.cc/linux/netfilter-log-format.php3</ulink>.</para>
|
|
|
|
<para>In this case, 192.168.2.2 was in the <quote>dmz</quote> zone and
|
|
192.168.1.3 is in the <quote>loc</quote> zone. I was missing the rule:</para>
|
|
|
|
<programlisting>ACCEPT dmz loc udp 53</programlisting>
|
|
</example>
|
|
</section>
|
|
|
|
<section id="faq21">
|
|
<title>(FAQ 21) I see these strange log entries occasionally; what are
|
|
they?</title>
|
|
|
|
<programlisting>Nov 25 18:58:52 linux kernel:
|
|
Shorewall:net2all:DROP:IN=eth1 OUT=
|
|
MAC=00:60:1d:f0:a6:f9:00:60:1d:f6:35:50:08:00 SRC=206.124.146.179
|
|
DST=192.0.2.3 LEN=56 TOS=0x00 PREC=0x00 TTL=110 ID=18558 PROTO=ICMP
|
|
TYPE=3 CODE=3 [SRC=192.0.2.3 DST=172.16.1.10 LEN=128 TOS=0x00 PREC=0x00
|
|
TTL=47 ID=0 DF PROTO=UDP SPT=53 DPT=2857 LEN=108 ]</programlisting>
|
|
|
|
<para>192.0.2.3 is external on my firewall... 172.16.0.0/24 is my
|
|
internal LAN</para>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> While most people
|
|
associate the Internet Control Message Protocol (ICMP) with
|
|
<quote>ping</quote>, ICMP is a key piece of the internet. ICMP is used
|
|
to report problems back to the sender of a packet; this is what is
|
|
happening here. Unfortunately, where NAT is involved (including SNAT,
|
|
DNAT and Masquerade), there are a lot of broken implementations. That is
|
|
what you are seeing with these messages.</para>
|
|
|
|
<para>Here is my interpretation of what is happening -- to confirm this
|
|
analysis, one would have to have packet sniffers placed a both ends of
|
|
the connection.</para>
|
|
|
|
<para>Host 172.16.1.10 behind NAT gateway 206.124.146.179 sent a UDP DNS
|
|
query to 192.0.2.3 and your DNS server tried to send a response (the
|
|
response information is in the brackets -- note source port 53 which
|
|
marks this as a DNS reply). When the response was returned to to
|
|
206.124.146.179, it rewrote the destination IP TO 172.16.1.10 and
|
|
forwarded the packet to 172.16.1.10 who no longer had a connection on
|
|
UDP port 2857. This causes a port unreachable (type 3, code 3) to be
|
|
generated back to 192.0.2.3. As this packet is sent back through
|
|
206.124.146.179, that box correctly changes the source address in the
|
|
packet to 206.124.146.179 but doesn't reset the DST IP in the
|
|
original DNS response similarly. When the ICMP reaches your firewall
|
|
(192.0.2.3), your firewall has no record of having sent a DNS reply to
|
|
172.16.1.10 so this ICMP doesn't appear to be related to anything
|
|
that was sent. The final result is that the packet gets logged and
|
|
dropped in the all2all chain. I have also seen cases where the source IP
|
|
in the ICMP itself isn't set back to the external IP of the remote
|
|
NAT gateway; that causes your firewall to log and drop the packet out of
|
|
the rfc1918 chain because the source IP is reserved by RFC 1918.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Routing</title>
|
|
|
|
<section id="faq32">
|
|
<title>(FAQ 32) My firewall has two connections to the internet from two
|
|
different ISPs. How do I set this up in Shorewall?</title>
|
|
|
|
<para>Setting this up in Shorewall is easy; setting up the routing is a
|
|
bit harder.</para>
|
|
|
|
<para>Assuming that <filename class="devicefile">eth0</filename> and
|
|
<filename class="devicefile">eth1</filename> are the interfaces to the
|
|
two ISPs then:</para>
|
|
|
|
<para><filename>/etc/shorewall/interfaces</filename>:</para>
|
|
|
|
<programlisting>#ZONE INTERFACE BROADCAST OPTIONS
|
|
net eth0 detect
|
|
net eth1 detect</programlisting>
|
|
|
|
<para><filename>/etc/shorewall/policy</filename>:</para>
|
|
|
|
<programlisting>#SOURCE DESTINATION POLICY LIMIT:BURST
|
|
net net DROP</programlisting>
|
|
|
|
<para>If you have masqueraded hosts, be sure to update
|
|
<filename>/etc/shorewall/masq</filename> to masquerade to both ISPs. For
|
|
example, if you masquerade all hosts connected to <filename
|
|
class="devicefile">eth2</filename> then:</para>
|
|
|
|
<programlisting>#INTERFACE SUBNET ADDRESS
|
|
eth0 eth2
|
|
eth1 eth2</programlisting>
|
|
|
|
<para><citetitle>There was an article in SysAdmin covering this topic.
|
|
It may be found at <ulink
|
|
url="http://www.samag.com/documents/s=1824/sam0201h/">http://www.samag.com/documents/s=1824/sam0201h/</ulink></citetitle></para>
|
|
|
|
<para><citetitle>The following information regarding setting up routing
|
|
for this configuration is reproduced from the <ulink
|
|
url="http://www.lartc.org">LARTC HOWTO</ulink> and has not been verified
|
|
by the author. If you have questions or problems with the instructions
|
|
given below, please post to the <ulink
|
|
url="http://www.lartc.org/#mailinglist">LARTC mailing list</ulink>.</citetitle></para>
|
|
|
|
<sidebar>
|
|
<para>A common configuration is the following, in which there are two
|
|
providers that connect a local network (or even a single machine) to
|
|
the big Internet.</para>
|
|
|
|
<programlisting> ________
|
|
+------------+ /
|
|
| | |
|
|
+-------------+ Provider 1 +-------
|
|
__ | | | /
|
|
___/ \_ +------+-------+ +------------+ |
|
|
_/ \__ | if1 | /
|
|
/ \ | | |
|
|
| Local network -----+ Linux router | | Internet
|
|
\_ __/ | | |
|
|
\__ __/ | if2 | \
|
|
\___/ +------+-------+ +------------+ |
|
|
| | | \
|
|
+-------------+ Provider 2 +-------
|
|
| | |
|
|
+------------+ \________
|
|
</programlisting>
|
|
|
|
<para>There are usually two questions given this setup.</para>
|
|
|
|
<para><emphasis role="bold">Split access</emphasis></para>
|
|
|
|
<para>The first is how to route answers to packets coming in over a
|
|
particular provider, say Provider 1, back out again over that same
|
|
provider.</para>
|
|
|
|
<para>Let us first set some symbolical names. Let <emphasis
|
|
role="bold">$IF1</emphasis> be the name of the first interface (if1 in
|
|
the picture above) and <emphasis role="bold">$IF2</emphasis> the name
|
|
of the second interface. Then let <emphasis role="bold">$IP1</emphasis>
|
|
be the IP address associated with <emphasis role="bold">$IF1</emphasis>
|
|
and <emphasis role="bold">$IP2</emphasis> the IP address associated
|
|
with <emphasis role="bold">$IF2</emphasis>. Next, let <emphasis
|
|
role="bold">$P1</emphasis> be the IP address of the gateway at
|
|
Provider 1, and <emphasis role="bold">$P2</emphasis> the IP address of
|
|
the gateway at provider 2. Finally, let <emphasis role="bold">$P1_NET</emphasis>
|
|
be the IP network <emphasis role="bold">$P1</emphasis> is in, and
|
|
<emphasis role="bold">$P2_NET</emphasis> the IP network <emphasis
|
|
role="bold">$P2</emphasis> is in.</para>
|
|
|
|
<para>One creates two additional routing tables, say <emphasis
|
|
role="bold">T1</emphasis> and <emphasis role="bold">T2</emphasis>.
|
|
These are added in /etc/iproute2/rt_tables. Then you set up routing in
|
|
these tables as follows:</para>
|
|
|
|
<programlisting>ip route add $P1_NET dev $IF1 src $IP1 table T1
|
|
ip route add default via $P1 table T1
|
|
ip route add $P2_NET dev $IF2 src $IP2 table T2
|
|
ip route add default via $P2 table T2</programlisting>
|
|
|
|
<para>Nothing spectacular, just build a route to the gateway and build
|
|
a default route via that gateway, as you would do in the case of a
|
|
single upstream provider, but put the routes in a separate table per
|
|
provider. Note that the network route suffices, as it tells you how to
|
|
find any host in that network, which includes the gateway, as
|
|
specified above.</para>
|
|
|
|
<para>Next you set up the main routing table. It is a good idea to
|
|
route things to the direct neighbour through the interface connected
|
|
to that neighbour. Note the `src' arguments, they make sure the
|
|
right outgoing IP address is chosen.</para>
|
|
|
|
<programlisting>ip route add $P1_NET dev $IF1 src $IP1
|
|
ip route add $P2_NET dev $IF2 src $IP2</programlisting>
|
|
|
|
<para>Then, your preference for default route:</para>
|
|
|
|
<programlisting>ip route add default via $P1</programlisting>
|
|
|
|
<para>Next, you set up the routing rules. These actually choose what
|
|
routing table to route with. You want to make sure that you route out
|
|
a given interface if you already have the corresponding source
|
|
address:</para>
|
|
|
|
<programlisting>ip rule add from $IP1 table T1
|
|
ip rule add from $IP2 table T2</programlisting>
|
|
|
|
<para>This set of commands makes sure all answers to traffic coming in
|
|
on a particular interface get answered from that interface.</para>
|
|
|
|
<note>
|
|
<para>'If $P0_NET is the local network and $IF0 is its
|
|
interface, the following additional entries are desirable:</para>
|
|
|
|
<programlisting format="linespecific">ip route add $P0_NET dev $IF0 table T1
|
|
ip route add $P2_NET dev $IF2 table T1
|
|
ip route add 127.0.0.0/8 dev lo table T1
|
|
ip route add $P0_NET dev $IF0 table T2
|
|
ip route add $P1_NET dev $IF1 table T2
|
|
ip route add 127.0.0.0/8 dev lo table T2</programlisting>
|
|
</note>
|
|
|
|
<para>Now, this is just the very basic setup. It will work for all
|
|
processes running on the router itself, and for the local network, if
|
|
it is masqueraded. If it is not, then you either have IP space from
|
|
both providers or you are going to want to masquerade to one of the
|
|
two providers. In both cases you will want to add rules selecting
|
|
which provider to route out from based on the IP address of the
|
|
machine in the local network.</para>
|
|
|
|
<para><emphasis role="bold">Load balancing</emphasis></para>
|
|
|
|
<para>The second question is how to balance traffic going out over the
|
|
two providers. This is actually not hard if you already have set up
|
|
split access as above.</para>
|
|
|
|
<para>Instead of choosing one of the two providers as your default
|
|
route, you now set up the default route to be a multipath route. In
|
|
the default kernel this will balance routes over the two providers. It
|
|
is done as follows (once more building on the example in the section
|
|
on split-access):</para>
|
|
|
|
<programlisting>ip route add default scope global nexthop via $P1 dev $IF1 weight 1 \
|
|
nexthop via $P2 dev $IF2 weight 1</programlisting>
|
|
|
|
<para>This will balance the routes over both providers. The <emphasis
|
|
role="bold">weight</emphasis> parameters can be tweaked to favor one
|
|
provider over the other.</para>
|
|
|
|
<note>
|
|
<para>balancing will not be perfect, as it is route based, and
|
|
routes are cached. This means that routes to often-used sites will
|
|
always be over the same provider.</para>
|
|
</note>
|
|
|
|
<para>Furthermore, if you really want to do this, you probably also
|
|
want to look at Julian Anastasov's patches at <ulink
|
|
url="http://www.ssi.bg/%7Eja/#routes">http://www.ssi.bg/~ja/#routes</ulink>
|
|
, Julian's route patch page. They will make things nicer to work
|
|
with.</para>
|
|
</sidebar>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Starting and Stopping</title>
|
|
|
|
<section id="faq7">
|
|
<title>(FAQ 7) When I stop Shorewall using <quote>shorewall stop</quote>,
|
|
I can't connect to anything. Why doesn't that command work?</title>
|
|
|
|
<para>The <quote><command>stop</command></quote> command is intended to
|
|
place your firewall into a safe state whereby only those hosts listed in
|
|
<filename>/etc/shorewall/routestopped</filename>' are activated. If
|
|
you want to totally open up your firewall, you must use the
|
|
<quote><command>shorewall clear</command></quote> command.</para>
|
|
</section>
|
|
|
|
<section id="faq8">
|
|
<title>(FAQ 8) When I try to start Shorewall on RedHat, I get messages
|
|
about insmod failing -- what's wrong?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> The output you will see
|
|
looks something like this:</para>
|
|
|
|
<programlisting>/lib/modules/2.4.17/kernel/net/ipv4/netfilter/ip_tables.o: init_module: Device or resource busy
|
|
Hint: insmod errors can be caused by incorrect module parameters, including invalid IO or IRQ parameters
|
|
/lib/modules/2.4.17/kernel/net/ipv4/netfilter/ip_tables.o: insmod
|
|
/lib/modules/2.4.17/kernel/net/ipv4/netfilter/ip_tables.o failed
|
|
/lib/modules/2.4.17/kernel/net/ipv4/netfilter/ip_tables.o: insmod ip_tables failed
|
|
iptables v1.2.3: can't initialize iptables table `nat': iptables who? (do you need to insmod?)
|
|
Perhaps iptables or your kernel needs to be upgraded.</programlisting>
|
|
|
|
<para>This problem is usually corrected through the following sequence
|
|
of commands</para>
|
|
|
|
<programlisting>service ipchains stop
|
|
chkconfig --delete ipchains
|
|
rmmod ipchains</programlisting>
|
|
|
|
<para>Also, be sure to check the <ulink url="errata.htm">errata</ulink>
|
|
for problems concerning the version of iptables (v1.2.3) shipped with
|
|
RH7.2.</para>
|
|
|
|
<section id="faq8a">
|
|
<title>(FAQ 8a) When I try to start Shorewall on RedHat I get a
|
|
message referring me to FAQ #8</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> This is usually cured
|
|
by the sequence of commands shown above in <xref linkend="faq8" />.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="faq9">
|
|
<title>(FAQ 9) Why can't Shorewall detect my interfaces properly at
|
|
startup?</title>
|
|
|
|
<para>I just installed Shorewall and when I issue the start command, I
|
|
see the following:</para>
|
|
|
|
<programlisting>Processing /etc/shorewall/params ...
|
|
Processing /etc/shorewall/shorewall.conf ...
|
|
Starting Shorewall...
|
|
Loading Modules...
|
|
Initializing...
|
|
Determining Zones...
|
|
Zones: net loc
|
|
Validating interfaces file...
|
|
Validating hosts file...
|
|
Determining Hosts in Zones...
|
|
<emphasis role="bold">Net Zone: eth0:0.0.0.0/0
|
|
</emphasis><emphasis role="bold">Local Zone: eth1:0.0.0.0/0</emphasis>
|
|
Deleting user chains...
|
|
Creating input Chains...
|
|
...</programlisting>
|
|
|
|
<para>Why can't Shorewall detect my interfaces properly?</para>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> The above output is
|
|
perfectly normal. The Net zone is defined as all hosts that are
|
|
connected through eth0 and the local zone is defined as all hosts
|
|
connected through <filename class="devicefile">eth1</filename>. If you
|
|
are running Shorewall 1.4.10 or later, you can consider setting the
|
|
<ulink url="Documentation.htm#Interfaces"><emphasis role="bold">detectnets</emphasis>
|
|
interface option</ulink> on your local interface (<filename
|
|
class="devicefile">eth1</filename> in the above example). That will
|
|
cause Shorewall to restrict the local zone to only those networks routed
|
|
through that interface.</para>
|
|
</section>
|
|
|
|
<section id="faq22">
|
|
<title>(FAQ 22) I have some iptables commands that I want to run when
|
|
Shorewall starts. Which file do I put them in?</title>
|
|
|
|
<para>You can place these commands in one of the <ulink
|
|
url="shorewall_extension_scripts.htm">Shorewall Extension Scripts</ulink>.
|
|
Be sure that you look at the contents of the chain(s) that you will be
|
|
modifying with your commands to be sure that the commands will do what
|
|
they are intended. Many iptables commands published in HOWTOs and other
|
|
instructional material use the -A command which adds the rules to the
|
|
end of the chain. Most chains that Shorewall constructs end with an
|
|
unconditional DROP, ACCEPT or REJECT rule and any rules that you add
|
|
after that will be ignored. Check <quote>man iptables</quote> and look
|
|
at the -I (--insert) command.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>About Shorewall</title>
|
|
|
|
<section id="faq10">
|
|
<title>(FAQ 10) What Distributions does it work with?</title>
|
|
|
|
<para>Shorewall works with any GNU/Linux distribution that includes the
|
|
<ulink url="shorewall_prerequisites.htm">proper prerequisites</ulink>.</para>
|
|
</section>
|
|
|
|
<section id="faq11">
|
|
<title>(FAQ 11) What Features does it have?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> See the <ulink
|
|
url="shorewall_features.htm">Shorewall Feature List</ulink>.</para>
|
|
</section>
|
|
|
|
<section id="faq12">
|
|
<title>(FAQ 12) Is there a GUI?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Yes. Shorewall support is
|
|
included in Webmin 1.060 and later versions. See <ulink
|
|
url="http://www.webmin.com">http://www.webmin.com</ulink></para>
|
|
</section>
|
|
|
|
<section id="faq13">
|
|
<title>(FAQ 13) Why do you call it <quote>Shorewall</quote>?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Shorewall is a
|
|
concatenation of <quote><emphasis>Shore</emphasis>line</quote> (<ulink
|
|
url="http://www.cityofshoreline.com">the city where I live</ulink>) and
|
|
<quote>Fire<emphasis>wall</emphasis></quote>. The full name of the
|
|
product is actually <quote>Shoreline Firewall</quote> but
|
|
<quote>Shorewall</quote> is must more commonly used.</para>
|
|
</section>
|
|
|
|
<section id="faq23">
|
|
<title>(FAQ 23) Why do you use such ugly fonts on your web site?</title>
|
|
|
|
<para>The Shorewall web site is almost font neutral (it doesn't
|
|
explicitly specify fonts except on a few pages) so the fonts you see are
|
|
largely the default fonts configured in your browser. If you don't
|
|
like them then reconfigure your browser.</para>
|
|
</section>
|
|
|
|
<section id="faq25">
|
|
<title>(FAQ 25) How to I tell which version of Shorewall I am running?</title>
|
|
|
|
<para>At the shell prompt, type:</para>
|
|
|
|
<programlisting><command>/sbin/shorewall version</command></programlisting>
|
|
</section>
|
|
|
|
<section id="faq31">
|
|
<title>(FAQ 31) Does Shorewall provide protection against....</title>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>IP Spoofing: Sending packets over the WAN interface using an
|
|
internal LAP IP address as the source address?</term>
|
|
|
|
<listitem>
|
|
<para>Answer: Yes.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Tear Drop: Sending packets that contain overlapping fragments?</term>
|
|
|
|
<listitem>
|
|
<para>Answer: This is the responsibility of the IP stack, not the
|
|
Netfilter-based firewall since fragment reassembly occurs before
|
|
the stateful packet filter ever touches each packet.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Smurf and Fraggle: Sending packets that use the WAN or LAN
|
|
broadcast address as the source address?</term>
|
|
|
|
<listitem>
|
|
<para>Answer: Shorewall can be configured to do that using the
|
|
<ulink url="blacklisting_support.htm">blacklisting</ulink>
|
|
facility.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>Land Attack: Sending packets that use the same address as the
|
|
source and destination address?</term>
|
|
|
|
<listitem>
|
|
<para>Answer: Yes, if the <ulink
|
|
url="Documentation.htm#Interfaces">routefilter interface option</ulink>
|
|
is selected.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term>DOS: - SYN Dos - ICMP Dos - Per-host Dos protection</term>
|
|
|
|
<listitem>
|
|
<para>Answer: Shorewall has facilities for limiting SYN and ICMP
|
|
packets. Netfilter as included in standard Linux kernels
|
|
doesn't support per-remote-host limiting except by explicit
|
|
rule that specifies the host IP address; that form of limiting is
|
|
supported by Shorewall.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>RFC 1918</title>
|
|
|
|
<section id="faq14">
|
|
<title>(FAQ 14) I'm connected via a cable modem and it has an
|
|
internal web server that allows me to configure/monitor it but as
|
|
expected if I enable rfc1918 blocking for my eth0 interface (the
|
|
internet one), it also blocks the cable modems web server.</title>
|
|
|
|
<para>Is there any way it can add a rule before the rfc1918 blocking
|
|
that will let all traffic to and from the 192.168.100.1 address of the
|
|
modem in/out but still block all other rfc1918 addresses?</para>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> If you are running a
|
|
version of Shorewall earlier than 1.3.1, create /etc/shorewall/start and
|
|
in it, place the following:</para>
|
|
|
|
<programlisting><command>run_iptables -I rfc1918 -s 192.168.100.1 -j ACCEPT</command></programlisting>
|
|
|
|
<para>If you are running version 1.3.1 or later, simply add the
|
|
following to <ulink url="Documentation.htm#rfc1918">/etc/shorewall/rfc1918</ulink>:</para>
|
|
|
|
<para>Be sure that you add the entry ABOVE the entry for 192.168.0.0/16.</para>
|
|
|
|
<programlisting>#SUBNET TARGET
|
|
192.168.100.1 RETURN</programlisting>
|
|
|
|
<note>
|
|
<para>If you add a second IP address to your external firewall
|
|
interface to correspond to the modem address, you must also make an
|
|
entry in /etc/shorewall/rfc1918 for that address. For example, if you
|
|
configure the address 192.168.100.2 on your firewall, then you would
|
|
add two entries to /etc/shorewall/rfc1918:</para>
|
|
|
|
<programlisting>#SUBNET TARGET
|
|
192.168.100.1 RETURN
|
|
192.168.100.2 RETURN</programlisting>
|
|
</note>
|
|
|
|
<section id="faq14a">
|
|
<title>(FAQ 14a) Even though it assigns public IP addresses, my
|
|
ISP's DHCP server has an RFC 1918 address. If I enable RFC 1918
|
|
filtering on my external interface, my DHCP client cannot renew its
|
|
lease.</title>
|
|
|
|
<para>The solution is the same as <xref linkend="faq14" /> above.
|
|
Simply substitute the IP address of your ISPs DHCP server.</para>
|
|
</section>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Alias IP Addresses/Virtual Interfaces</title>
|
|
|
|
<section id="faq18">
|
|
<title>(FAQ 18) Is there any way to use aliased ip addresses with
|
|
Shorewall, and maintain separate rulesets for different IPs?</title>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Yes. See <ulink
|
|
url="Shorewall_and_Aliased_Interfaces.html">Shorewall and Aliased
|
|
Interfaces</ulink>.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section>
|
|
<title>Miscellaneous</title>
|
|
|
|
<section id="faq19">
|
|
<title>(FAQ 19) I have added entries to /etc/shorewall/tcrules but they
|
|
don't seem to do anything. Why?</title>
|
|
|
|
<para>You probably haven't set TC_ENABLED=Yes in
|
|
/etc/shorewall/shorewall.conf so the contents of the tcrules file are
|
|
simply being ignored.</para>
|
|
</section>
|
|
|
|
<section id="faq20">
|
|
<title>(FAQ 20) I have just set up a server. Do I have to change
|
|
Shorewall to allow access to my server from the internet?</title>
|
|
|
|
<para>Yes. Consult the <ulink url="shorewall_quickstart_guide.htm">QuickStart
|
|
guide</ulink> that you used during your initial setup for information
|
|
about how to set up rules for your server.</para>
|
|
</section>
|
|
|
|
<section id="faq24">
|
|
<title>(FAQ 24) How can I allow conections to let's say the ssh port
|
|
only from specific IP Addresses on the internet?</title>
|
|
|
|
<para>In the SOURCE column of the rule, follow <quote>net</quote> by a
|
|
colon and a list of the host/subnet addresses as a comma-separated list.</para>
|
|
|
|
<programlisting>net:<ip1>,<ip2>,...</programlisting>
|
|
|
|
<example>
|
|
<title>Example:</title>
|
|
|
|
<programlisting>ACCEPT net:192.0.2.16/28,192.0.2.44 fw tcp 22</programlisting>
|
|
</example>
|
|
</section>
|
|
|
|
<section id="faq26">
|
|
<title>(FAQ 26) When I try to use any of the SYN options in nmap on or
|
|
behind the firewall, I get <quote>operation not permitted</quote>. How
|
|
can I use nmap with Shorewall?"</title>
|
|
|
|
<para>Edit /etc/shorewall/shorewall.conf and change <quote>NEWNOTSYN=No</quote>
|
|
to <quote>NEWNOTSYN=Yes</quote> then restart Shorewall.</para>
|
|
|
|
<section id="faq26a">
|
|
<title>(FAQ 26a) When I try to use the <quote>-O</quote> option of
|
|
nmap from the firewall system, I get <quote>operation not permitted</quote>.
|
|
How do I allow this option?</title>
|
|
|
|
<para>Add this command to your /etc/shorewall/start file:</para>
|
|
|
|
<programlisting><command>run_iptables -D OUTPUT -p ! icmp -m state --state INVALID -j DROP</command></programlisting>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="faq27">
|
|
<title>(FAQ 27) I'm compiling a new kernel for my firewall. What
|
|
should I look out for?</title>
|
|
|
|
<para>First take a look at the <ulink url="kernel.htm">Shorewall kernel
|
|
configuration page</ulink>. You probably also want to be sure that you
|
|
have selected the <quote><emphasis role="bold">NAT of local connections
|
|
(READ HELP)</emphasis></quote> on the Netfilter Configuration menu.
|
|
Otherwise, DNAT rules with your firewall as the source zone won't
|
|
work with your new kernel.</para>
|
|
|
|
<section id="faq27a">
|
|
<title>(FAQ 27a) I just built and installed a new kernel and now
|
|
Shorewall won't start. I know that my kernel options are correct.</title>
|
|
|
|
<para>The last few lines of <ulink url="troubleshoot.htm">a startup
|
|
trace</ulink> are these:</para>
|
|
|
|
<programlisting>+ run_iptables2 -t nat -A eth0_masq -s 192.168.2.0/24 -d 0.0.0.0/0 -j
|
|
MASQUERADE
|
|
+ '[' 'x-t nat -A eth0_masq -s 192.168.2.0/24 -d 0.0.0.0/0 -j
|
|
MASQUERADE' = 'x-t nat -A eth0_masq -s 192.168.2.0/24 -d 0.0.0.
|
|
0/0 -j MASQUERADE' ']'
|
|
+ run_iptables -t nat -A eth0_masq -s 192.168.2.0/24 -d 0.0.0.0/0 -j
|
|
MASQUERADE
|
|
+ iptables -t nat -A eth0_masq -s 192.168.2.0/24 -d 0.0.0.0/0 -j
|
|
MASQUERADE
|
|
iptables: Invalid argument
|
|
+ '[' -z '' ']'
|
|
+ stop_firewall
|
|
+ set +x</programlisting>
|
|
|
|
<para><emphasis role="bold">Answer:</emphasis> Your new kernel
|
|
contains headers that are incompatible with the ones used to compile
|
|
your <command>iptables</command> utility. You need to rebuild
|
|
<command>iptables</command> using your new kernel source.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="faq28">
|
|
<title>(FAQ 28) How do I use Shorewall as a Bridging Firewall?</title>
|
|
|
|
<para>Basically, you don't. While there are kernel patches that
|
|
allow you to route bridge traffic through Netfilter, the environment is
|
|
so different from the Layer 3 firewalling environment that very little
|
|
of Shorewall works. In fact, so much of Shorewall doesn't work that
|
|
my official position is that <quote>Shorewall doesn't work with
|
|
Layer 2 Bridging</quote>.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<appendix>
|
|
<title>Revision History</title>
|
|
|
|
<para><revhistory><revision><revnumber>1.15</revnumber><date>2004-01-25</date><authorinitials>TE</authorinitials><revremark>Updated
|
|
FAQ 32 to mention masquerading. Remove tables.</revremark></revision><revision><revnumber>1.14</revnumber><date>2004-01-24</date><authorinitials>TE</authorinitials><revremark>Added
|
|
FAQ 27a regarding kernel/iptables incompatibility.</revremark></revision><revision><revnumber>1.13</revnumber><date>2004-01-24</date><authorinitials>TE</authorinitials><revremark>Add
|
|
a note about the <emphasis role="bold">detectnets</emphasis> interface
|
|
option in FAQ 9.</revremark></revision><revision><revnumber>1.12</revnumber><date>2004-01-20</date><authorinitials>TE</authorinitials><revremark>Improve
|
|
FAQ 16 answer.</revremark></revision><revision><revnumber>1.11</revnumber><date>2004-01-14</date><authorinitials>TE</authorinitials><revremark>Corrected
|
|
broken link</revremark></revision><revision><revnumber>1.10</revnumber><date>2004-01-09</date><authorinitials>TE</authorinitials><revremark>Added
|
|
a couple of more legacy FAQ numbers.</revremark></revision><revision><revnumber>1.9</revnumber><date>2004-01-08</date><authorinitials>TE</authorinitials><revremark>Corrected
|
|
typo in FAQ 26a. Added warning to FAQ 2 regarding source address of
|
|
redirected requests.</revremark></revision><revision><revnumber>1.8</revnumber><date>2003-12-31</date><authorinitials>TE</authorinitials><revremark>Additions
|
|
to FAQ 4.</revremark></revision><revision><revnumber>1.7</revnumber><date>2003-12-30</date><authorinitials>TE</authorinitials><revremark>Remove
|
|
dead link from FAQ 1.</revremark></revision><revision><revnumber>1.6</revnumber><date>2003.12-18</date><authorinitials>TE</authorinitials><revremark>Add
|
|
external link reference to FAQ 17.</revremark></revision><revision><revnumber>1.5</revnumber><date>2003-12-16</date><authorinitials>TE</authorinitials><revremark>Added
|
|
a link to a Sys Admin article about multiple internet interfaces. Added
|
|
Legal Notice. Moved "abstract" to the body of the document. Moved
|
|
Revision History to this Appendix.</revremark></revision><revision><revnumber>1.4</revnumber><date>2003-12-13</date><authorinitials>TE</authorinitials><revremark>Corrected
|
|
formatting problems</revremark></revision><revision><revnumber>1.3</revnumber><date>2003-12-10</date><authorinitials>TE</authorinitials><revremark>Changed
|
|
the title of FAQ 17</revremark></revision><revision><revnumber>1.2</revnumber><date>2003-12-09</date><authorinitials>TE</authorinitials><revremark>Added
|
|
Copyright and legacy FAQ numbers</revremark></revision><revision><revnumber>1.1</revnumber><date>2003-12-04</date><authorinitials>MN</authorinitials><revremark>Converted
|
|
to Simplified DocBook XML</revremark></revision><revision><revnumber>1.0</revnumber><date>2002-08-13</date><authorinitials>TE</authorinitials><revremark>Initial
|
|
revision</revremark></revision></revhistory></para>
|
|
</appendix>
|
|
</article> |