* re-organized imports according to pep8
* fixed all remaining pep8 issues
* moved common config into setup.cfg, additionally test `tests`
* removed --select=X -- the errors selected where by default not in
flake8's --ignore list so effectively had no effect
* update .travis.yml to reflect changes in tox.ini
* make travis just use tox in order to avoid code duplaction
* replace py.test with pytest
* fixed .travis.yml
* try different pypy toxenv
* hopefully fixed testenv for pypy
* added pypy basepython, removed unused python2.6
* install dev package before testing (fixes missing coverage)
* fixed empty exception pass blocks with noqa
* Added dummy log message on empty try-except-pass blocks to make dodacy happy :(
* Replaced Exception with BaseException
There's a known issue that makes sshuttle crash if there are too
many routes on the remote host (that don't fit in 64KB). This patch
requests the routes only if auto-nets is specified on the command
line.
This was susceptible to the same deadlock issue that ipt_chain_exists
had and was fixed in d43db80 where if the command returned a significant
amount of output, it wouldn't all be read in, resulting in the
subprocess hanging waiting for the output to be read.
When running sshuttle with a large list of routes it's failing to clean
them up at exit. It returns the following:
$ sshuttle -r user@host.example.com -s /tmp/aws-cidrs.txt
user@host.example.com's password:
client: Connected.
^CAnother app is currently holding the xtables lock; still -9s 0us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still -19s 0us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still -29s 0us time ahead to have a chance to grab the lock...
This continues indefinitely. Looking in ps reveals that there are 2
iptables processes running. Killing -9 the first one, allows sshuttle to
continue and clean up successfully.
The problem lies with the use of Popen here. The function currently
returns as soon as it finds a match without consuming everything from
stdout. This means that if there's more output from iptables than will
fit in the buffer it doesn't exit, and therefore doesn't release the
kernel xtables lock.
Before this change, in pf, exclusions used a pass out quick which gave
them higher precedence than any other rule independent of subnet width.
As reported in #265 this causes exclusion from one instance of sshuttle
to also take effect on other instances because quick aborts the
evaluation of rules across all anchors.
This commit changes the precedence of rules so quick can now be
dropped. The new order is defined by the following rule, from
subnet_weight:
"We need to go from smaller, more specific, port ranges, to larger,
less-specific, port ranges. At each level, we order by subnet
width, from most-specific subnets (largest swidth) to
least-specific. On ties, excludes come first."
* Fixes support for OpenBSD (6.1+)
As reported in #219, new versions of OpenBSD ship with a different
pfioc_rule struct. This commit adjusts the offset to match the new struct.
* Fixes tests for OpenBSD 6.1+
Having the tests in a `tests` directory in root is the most common
approach. Also moved pytest's conftest.py into `tests` making the
fixture available for client and server tests.
This provides a way to avoid setting PYTHONPATH when invoking the
privileged part of sshuttle with sudo. This is useful if running
sshuttle as a PEX archive, as Telepresence does, as it enables
sshuttle's sudo access to be securely locked down.
PEX archives will extract themselves into the invoking user's home
directory, which means that the invoking user has full control over
the code in them. This makes restricting sudo access with
PYTHONPATH set completely pointless in this scenario -- an attacker
could put any code into ~/.pex and gain full root access anyway.
On the other hand, if sshuttle is a PEX archive, the privileged
invocation will simply extract itself into /root/.pex anyway, so
there is no need to set PYTHONPATH in this case.
Starting sshuttle without having to type in one's password requires to
put the sudo-ed command in the `/etc/sudoers` file. However, sshuttle
sets an environment variable, which cannot be done as-is in the sudoers
file. This fix prepend the /usr/bin/env command, which allows one to
pass fixed environment variables to a sudo-ed command.
In practice, the sub-command:
```
sudo PYTHONPATH=/usr/lib/python3/dist-packages -- \
/usr/bin/python3 /usr/bin/sshuttle --method auto --firewall
```
becomes
```
sudo /usr/bin/env PYTHONPATH=/usr/lib/python3/dist-packages \
/usr/bin/python3 /usr/bin/sshuttle --method auto --firewall
```
This small change will allow a file path to be passed as argument from which
the command line options will be loaded.
Extra command line options can be passed (in addition to those already in the
file) and existing ones can be overriden.
Example sshuttle.conf file:
192.168.0.0/16
--remote
user@example.com
Example sshuttle call:
sshuttle @/path/to/sshuttle.conf
Example sshuttle call with verbose flags added:
sshuttle @/path/to/sshuttle.conf -vvv
Example sshuttle call overriding the remote server:
sshuttle @/path/to/sshuttle.conf -r otheruser@test.example.com
When I starting sshuttle with option `--seed-hosts example.com`, got the following error:
```
hostwatch: Starting hostwatch with Python version 3.5.2
hostwatch: Traceback (most recent call last):
---> File "sshuttle.server", line 144, in start_hostwatch
---> File "sshuttle.hostwatch", line 272, in hw_main
---> File "sshuttle.hostwatch", line 234, in check_host
---> File "sshuttle.hostwatch", line 32, in _is_ip
---> File "/usr/lib/python3.5/re.py", line 163, in match
---> return _compile(pattern, flags).match(string)
---> TypeError: cannot use a string pattern on a bytes-like object
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "assembler.py", line 37, in <module>
File "sshuttle.server", line 393, in main
File "sshuttle.ssnet", line 596, in runonce
File "sshuttle.server", line 324, in hostwatch_ready
sshuttle.helpers.Fatal: hostwatch process died
```
It seems like the list of hosts is not properly decoded on the server side. This is an attempt to fix that.
As suggested by @colinmkeith the UDP and DNS proxies should listen on different
ports otherwise the DNS proxy can get traffic intended to the UDP proxy (or
vice-versa) and handle it incorrectly as reported in #178.
At first sight it seems that we had the code in place to try another port if
the one we are binding is already bound, however, with UDP and REUSEADDR the
OS will not refuse to bind two sockets to the same socket address, so both
the UDP proxy and DNS proxy were being bound to the same pair.
Some Linux distros, like Alpine, Arch, etc and some BSDs, like FreeBSD, are
now shipping with python3.6 as the default python3. Both the client and the
server are failing to run in this distros, because we are specifically looking
for python3.5.
These changes make the run shell script use python3 if the version is greater
than 3.5, otherwise falling back as usual.
On the server any version of python3 will do, use it before falling back to
python, as the server code can run with any version of python3.
When the pf module is not loaded our calls to pfctl will fail with
unhelpful messages.
This change spares the user the pain of decrypting those messages and manually
enabling pf. It also keeps track if pf was loaded by sshuttle and unloads on
exit if that was the case.
Also fixed the case where both ipv4 and ipv6 anchors were added by sshuttle
but the first call of disable would disable pf before the second call had the
chance of cleaning it's anchor.
If we receive no routes from server or if, for some reason, we receive
some empty lines, we should skip them instead of crashing.
Fixes on of the problems in #147.
Currently hostwatch only adds hostnames even when FQDNs are available.
This commit changes found_host so that when the name is a FQDN, both the FQDN
and an hostname are added, e.g., given api.foo.com both api and api.foo.com
will be added.
Fixes#151 if merged.
N.B.: I rarely use hostwatch, it would probably be a good idea to get feedback
from people who actually use it before merging. Not too sure about this...
While with AF_INET sockaddr is a 2-tuple composed by (address, port),
with AF_INET6 it is a 4-tuple with (address, port, flow info, scope id).
We were always passing a 2-tuple to socket.connect which would fail whenever
the address was, for instance, a link-local IPv6 address that needs a scope id.
With this change we now use getaddrinfo to correctly compute the full tuple.
Fixes#156.
When doing port forwarding on lo0 avoid the special case where the
traffic on lo0 did not came from sshuttle pass out rule but from the lo0
address itself. Fixes#159.
This change makes the subnets with the most specific port ranges come
before subnets with larger, least specific, port ranges. Before this
change subnets with smaller swidth would always come first and only for
subnets with the same width would the size of the port range be
considered.
Example:
188.0.0.0/8 -x 0.0.0.0/0:443
Before: 188.0.0.0/8 would come first meaning that all ports would be
routed through the VPN for the subnet 188.0.0.0/8
After: 0.0.0.0/0:443 comes first, meaning that port 443 will be
excluded for all subnets, including 188.0.0.0/8. All other ports of
188.0.0.0/8 will be routed.
* Adds support for tunneling specific port ranges
This set of changes implements the ability of specifying a port or port
range for an IP or subnet to only tunnel those ports for that subnet.
Also supports excluding a port or port range for a given IP or subnet.
When, for a given subnet, there are intercepting ranges being added and
excluded, the most specific, i.e., smaller range, takes precedence. In
case of a tie the exclusion wins.
For different subnets, the most specific, i.e., largest swidth, takes
precedence independent of any eventual port ranges.
Examples:
Tunnels all traffic to the 188.0.0.0/8 subnet except those to port 443.
```
sshuttle -r <server> 188.0.0.0/8 -x 188.0.0.0/8:443
```
Only tunnels traffic to port 80 of the 188.0.0.0/8 subnet.
```
sshuttle -r <server> 188.0.0.0/8:80
```
Tunnels traffic to the 188.0.0.0/8 subnet and the port range that goes
from 80 to 89.
```
sshuttle -r <server> 188.0.0.0/8:80-89 -x 188.0.0.0/8:80-90
```
* Allow subnets to be specified with domain names
Simplifies the implementation of address parsing by using
socket.getaddrinfo(), which can handle domain resolution, IPv4 and IPv6
addresses. This was proposed and mostly implemented by @DavidBuchanan314
in #146.
Signed-off-by: David Buchanan <DavidBuchanan314@users.noreply.github.com>
Signed-off-by: João Vieira <vieira@yubo.be>
* Also use getaddrinfo for parsing listen addr:port
* Fixes tests for tunneling a port range
* Updates documentation to include port/port range
Adds some examples with subnet:port and subnet:port-port.
Also clarifies the versions of Python supported on the server while
maintaining the recommendation for Python 2.7, 3.5 or later.
Mentions support for pfSense.
* In Py2 only named arguments may follow *expression
Fixes issue in Python 2.7 where *expression may only be followed by
named arguments.
* Use right regex to extract ip4/6, mask and ports
* Tests for parse_subnetport
netstat outputs some headers in BSD (that the Linux version does not)
that are not tabular and were breaking our 'split line into columns
and get nth column' logic. We now skip such headers.
Should fix#141.
There was runtime failure on UDP or DNS processing, because "socket" was redefined to PyXAPI's socket_ext in tproxy.py, but still was plain Python's socket in client.py
Fixed https://github.com/sshuttle/sshuttle/issues/134 for me
`netstat` has been deprecated for some time and some distros might
start shipping without it in the near future. This commit adds support
for `ip route` and uses it when available.
PfSense is based on FreeBSD and its pf is pretty close to the one
FreeBSD ships, however some structures have different fields and two
offsets had to be fixed.
We set it to true when we enable pf, but do not set it back to False
after disabling. When using IPv4 and IPv6 we end up trying to disable
twice which procudes an error while undoing changes in FreeBSD 11.
These changes introduce support for sdnotify allowing sshuttle to notify
systemd when it finishes connecting to the server and installing
firewall rules, and is ready to tunnel requests.
This should fix an issue introduced in #117 where when no subnets are
given via file (-s file) the variable is None instead of an empty list
and the concatenation with the subnets given as positional parameters
fails.
By just splitting at spaces, multi-word arguments are torn apart even if
quoted. In case of custom ssh-cmd, this makes it practically impossible
to set certian options through `ssh -o`.
shlex splits arguments like a shell and e.g. respects quotes.
This should fix#116. Handling this while still having the positional
arguments and -s both write to the same list turned out to be more
complicated than it's worth so each writes to their own variable and we
merge them at the end.
AF_INET is the same constant on Linux and BSD but AF_INET6
is different. As the client and server can be running on
different platforms we can not just set the socket family
to what comes in the wire.
A possible implementation for the change requested in #94, so that seed
hosts can be used without auto hosts. In this scenario only the
specified hosts (or ips) will be looked up (or rev looked up).
We shouldn't come up with a fatal error because of a ENETUNREACH when
trying to contact the DNS server. Although this error shouldn't happen
either.
Fixes#89.
Previously the sshuttle shell script would pass the python to use as the
first argument of the command. The new run script no longer does this.
Instead we can obtain the python being used via sys.executable.
Fixes#88.
It is often the case that the user has no administrative control over
the server that is being used. As such it is important to support as
many versions as possible, at least on the remote server end. These
fixes will allow sshuttle to be used with servers that have only
python 2.4 or python 2.6 installed while hopefully not breaking the
compatibility with 2.7 and 3.5.
When passing multiple subnet files, e.g., by using -s/--subnets
multiple times or by using it together with subnets passed as positional
arguments append the content from all sources instead of only using the
subnets from the last source. This makes the behaviour of -s/--subnets
consistent with -x/--exclude.
This allows disabling all client tests using a conftest.py file, if for
example #56 gets merged and the server supports more python versions
then the server.
The server side tests are very incomplete.
Just because we may have found IPv6 DNS servers from /etc/resolv.conf
doesn't mean we should force IPv6 support.
Instead we should disable the IPv6 DNS servers if IPv6 is disabled.
Note: this will also result in any IPv6 servers specified on the command
line being silently ignored too.
Specifying an IPv6 subnet will still require IPv6 support.
Closes#74
In some cases (see #43) it seems that some network configurations may
end up setting a skip on lo. As sshuttle adds rules that rely on
filtering/translating packets on lo, this causes problem. This fix
overrides the skip and makes the rules be applied again.
Should fix at least some of the problems reported on #43.
This is the error message that this commit fixes:
TypeError: sequence item 142: expected a bytes-like object, str found
Complete what 1c46f25e started, more or less.
Wew were trying to allocate an IPv6 socket even though we weren't using
IPv6, causing failures on systems without IPv6 support available.
This change means a number of methods on MultiListener, e.g. setsockopt,
should not be called until after the bind call.
Closes#68
Fix the following error. Looks like we have to have a function to call
for the entrypoint.
$ pip install dist/sshuttle-0.76.dev8_ngf59508f-py2-none-any.whl
Processing ./dist/sshuttle-0.76.dev8_ngf59508f-py2-none-any.whl
Installing collected packages: sshuttle
Exception:
Traceback (most recent call last):
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/commands/install.py", line 311, in run
root=options.root_path,
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/req/req_set.py", line 646, in install
**kwargs
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/req/req_install.py", line 803, in install
self.move_wheel_files(self.source_dir, root=root)
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/req/req_install.py", line 998, in move_wheel_files
isolated=self.isolated,
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/wheel.py", line 479, in move_wheel_files
maker.make_multiple(['%s = %s' % kv for kv in console.items()])
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/_vendor/distlib/scripts.py", line 364, in make_multiple
filenames.extend(self.make(specification, options))
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/_vendor/distlib/scripts.py", line 353, in make
self._make_script(entry, filenames, options=options)
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/_vendor/distlib/scripts.py", line 244, in _make_script
script = self._get_script_text(entry).encode('utf-8')
File "/tmp/ddd/local/lib/python2.7/site-packages/pip/wheel.py", line 396, in _get_script_text
"import_name": entry.suffix.split(".")[0],
AttributeError: 'NoneType' object has no attribute 'split'
The PF firewall that is included in the FreeBSD base system does not
have exactly the same data structures as the OSX version. This commit
fixes the offsets and some field types that are also different. Tested
with FreeBSD 10.2 and OSX 10.11.2.
When executing with the option --syslog start redirecting to
syslog immediately after the command line options are validated.
This way when using with some init daemon, e.g., upstart all the
relevant information (connection failures, etc) can be retrieved from
the log instead of being lost to stdout or stderr.
This is no longer used by modern MacOSX and not getting tested.
It also required a do_wait() function which was a complication for
sshuttle as a whole.
Can get resurrected if required.
We will log the errors, however no point in failing; not only can this
hide errors that occured setting up the firewall, but is pointless as we
can't actually handle these errors in a good way anyway.
Pull out firewall methods code into seperate files.
Fix problems starting with method=='auto'; we were making decisions
based on the method, before the method had been finalized by the
firewall.
Only very basic testing so far. What could go wrong?
