v0.5.0

Added example for .netrc usage (closes #139)

README.rst

@@ -16,6 +16,10 @@ for **testing, debugging**, and generally **interacting** with HTTP servers.
     :height: 835
     :align: center
 
+
+------
+
+
 .. image:: https://raw.github.com/claudiatd/httpie-artwork/master/images/httpie_logo_simple.png
     :alt: HTTPie logo
     :align: center
@@ -45,6 +49,7 @@ Main Features
 * Arbitrary request data
 * Custom headers
 * Persistent sessions
+* Wget-like downloads
 * Python 2.6, 2.7 and 3.x support
 * Linux, Mac OS X and Windows support
 * Documentation
@@ -138,7 +143,9 @@ See the request that is being sent using one of the `output options`_:
     $ http -v example.org
 
 
-Use `Github API`_ to post a comment on an issue with `authentication`_:
+Use `Github API`_ to post a comment on an
+`issue <https://github.com/jkbr/httpie/issues/83>`_
+with `authentication`_:
 
 .. code-block:: bash
 
@@ -158,6 +165,13 @@ Download a file and save it via `redirected output`_:
 
     $ http example.org/file > file
 
+
+Download a file ``wget`` style:
+
+.. code-block:: bash
+
+    $ http --download example.org/file
+
 Use named `sessions`_ to make certain aspects or the communication persistent
 between requests to the same host:
 
@@ -245,8 +259,8 @@ their type is distinguished only by the separator used:
 |                       | The ``==`` separator is used                        |
 +-----------------------+-----------------------------------------------------+
 | Data Fields           | Request data fields to be serialized as a JSON      |
-| ``field=value``       | object (default), or to be form encoded (``--form`` |
+| ``field=value``       | object (default), or to be form encoded             |
-|                       | / ``-f``).                                          |
+|                       | (``--form, -f``).                                   |
 +-----------------------+-----------------------------------------------------+
 | Raw JSON fields       | Useful when sending JSON and one or                 |
 | ``field:=json``       | more fields need to be a ``Boolean``, ``Number``,   |
@@ -254,7 +268,7 @@ their type is distinguished only by the separator used:
 |                       | ``meals:='["ham","spam"]'`` or ``pies:=[1,2,3]``    |
 |                       | (note the quotes).                                  |
 +-----------------------+-----------------------------------------------------+
-| Files                 | Only available with ``-f`` / ``--form``.            |
+| Files                 | Only available with ``--form, -f``.                 |
 | ``field@/dir/file``   | For example ``screenshot@~/Pictures/img.png``.      |
 |                       | The presence of a file field results                |
 |                       | in a ``multipart/form-data`` request.               |
@@ -285,7 +299,7 @@ both of which can be overwritten:
 ``Accept``          ``application/json``
 ================    =======================================
 
-You can use ``--json`` / ``-j`` to explicitly set ``Accept``
+You can use ``--json, -j`` to explicitly set ``Accept``
 to ``application/json`` regardless of whether you are sending data
 (it's a shortcut for setting the header via the usual header notation –
 ``http url Accept:application/json``).
@@ -303,7 +317,6 @@ Simple example:
     Accept-Encoding: identity, deflate, compress, gzip
     Content-Type: application/json; charset=utf-8
     Host: example.org
-    User-Agent: HTTPie/0.2.7dev
 
     {
         "name": "John",
@@ -325,7 +338,6 @@ into the resulting object:
     Accept: application/json
     Content-Type: application/json; charset=utf-8
     Host: api.example.com
-    User-Agent: HTTPie/0.2.7dev
 
     {
         "age": 29,
@@ -350,7 +362,7 @@ Forms
 =====
 
 Submitting forms is very similar to sending `JSON`_ requests. Often the only
-difference is in adding the ``--form`` / ``-f`` option, which ensures that
+difference is in adding the ``--form, -f`` option, which ensures that
 data fields are serialized as, and ``Content-Type`` is set to,
 ``application/x-www-form-urlencoded; charset=utf-8``.
 
@@ -370,7 +382,6 @@ Regular Forms
 .. code-block:: http
 
     POST /person/1 HTTP/1.1
-    User-Agent: HTTPie/0.2.7dev
     Content-Type: application/x-www-form-urlencoded; charset=utf-8
 
     name=John+Smith&email=john%40example.org
@@ -458,7 +469,7 @@ come). There are two flags that control authentication:
                         ``basic`` so it can often be omitted.
 ===================     ======================================================
 
-Authorization information from ``.netrc`` is honored as well.
+
 
 Basic auth:
 
@@ -483,15 +494,29 @@ With password prompt:
     $ http -a username example.org
 
 
+Authorization information from your ``.netrc`` file is honored as well:
+
+.. code-block:: bash
+
+    $ cat .netrc
+    machine httpbin.org
+    login httpie
+    password test
+    $ http httpbin.org/basic-auth/httpie/test
+    HTTP/1.1 200 OK
+    [...]
+
+
 =======
 Proxies
 =======
 
-You can specify proxies to be used through the ``--proxy`` argument:
+You can specify proxies to be used through the ``--proxy`` argument for each
+protocol (which is included in the value in case of redirects across protocols):
 
 .. code-block:: bash
 
-    $ http --proxy=http:10.10.1.10:3128 --https:10.10.1.10:1080 example.org
+    $ http --proxy=http:10.10.1.10:3128 --proxy=https:10.10.1.10:1080 example.org
 
 
 With Basic authentication:
@@ -570,7 +595,7 @@ documentation examples:
     }
 
 
-All the other options are just a shortcut for ``--print`` / ``-p``.
+All the other options are just a shortcut for ``--print, -p``.
 It accepts a string of characters each of which represents a specific part of
 the HTTP exchange:
 
@@ -811,6 +836,66 @@ by adding the following to your ``~/.bash_profile``:
     }
 
 
+=============
+Download Mode
+=============
+
+HTTPie features a download mode in which it acts similarly to ``wget``.
+
+When enabled using the ``--download, -d`` flag, response headers are printed to
+the terminal (``stderr``), and a progress bar is shown while the response body
+is being saved to a file.
+
+.. code-block:: bash
+
+    $ http --download https://github.com/jkbr/httpie/tarball/master
+
+.. code-block:: http
+
+    HTTP/1.1 200 OK
+    Connection: keep-alive
+    Content-Disposition: attachment; filename=jkbr-httpie-0.4.1-33-gfc4f70a.tar.gz
+    Content-Length: 505530
+    Content-Type: application/x-gzip
+    Server: GitHub.com
+    Vary: Accept-Encoding
+
+    Downloading 494.89 kB to "jkbr-httpie-0.4.1-33-gfc4f70a.tar.gz"
+    /  21.01% 104.00 kB   47.55 kB/s  0:00:08 ETA
+
+
+If not provided via ``--output, -o``, the output filename will be determined
+from ``Content-Disposition`` (if available), or from the URL and
+``Content-Type``. If the guessed filename already exists, HTTPie adds a unique
+suffix to it.
+
+You can also redirect the response body to another program while the response
+headers and progress are still shown in the terminal:
+
+.. code-block:: bash
+
+    $ http -d https://github.com/jkbr/httpie/tarball/master |  tar zxf -
+
+
+If ``--output, -o`` is specified, you can resume a partial download using the
+``--continue, -c`` option. This only works with servers that support
+``Range`` requests and ``206 Partial Content`` responses. If the server doesn't
+support that, the whole file will simply be downloaded:
+
+.. code-block:: bash
+
+    $ http -dco file.zip example.org/file
+
+Other notes:
+
+* The ``--download`` option only changes how the response body is treated.
+* You can still set custom headers, use sessions, ``--verbose, -v``, etc.
+* ``--download`` always implies ``--follow`` (redirects are followed).
+* HTTPie exits with status code ``1`` (error) if the body hasn't been fully
+  downloaded.
+* ``Accept-Encoding`` cannot be set with ``--download``.
+
+
 ==================
 Streamed Responses
 ==================
@@ -886,11 +971,7 @@ Session data are stored in JSON files in the directory
 **Warning:** All session data, including credentials, cookie data,
 and custom headers are stored in plain text.
 
-Session files can also be created or edited with a text editor.
+Session files can also be created and edited manually in a text editor.
-
-.. code-block:: bash
-
-    $ httpie session edit example.org user1
 
 
 See also `Config`_.
@@ -1081,6 +1162,10 @@ Changelog
 
 *You can click a version name to see a diff with the previous one.*
 
+* `0.6.0-dev`_
+* `0.5.0`_ (2013-04-27)
+    * Added a `download mode`_ via ``--download``.
+    * Bugfixes.
 * `0.4.1`_ (2013-02-26)
     * Fixed ``setup.py``.
 * `0.4.0`_ (2013-02-22)
@@ -1105,7 +1190,7 @@ Changelog
       ``--ugly`` has bee removed in favor of ``--pretty=none``.
 * `0.2.7`_ (2012-08-07)
     * Compatibility with Requests 0.13.6.
-    * Streamed terminal output. ``--stream`` / ``-S`` can be used to enable
+    * Streamed terminal output. ``--stream, -S`` can be used to enable
       streaming also with ``--pretty`` and to ensure a more frequent output
       flushing.
     * Support for efficient large file downloads.
@@ -1186,7 +1271,7 @@ Changelog
 .. _0.3.0: https://github.com/jkbr/httpie/compare/0.2.7...0.3.0
 .. _0.4.0: https://github.com/jkbr/httpie/compare/0.3.0...0.4.0
 .. _0.4.1: https://github.com/jkbr/httpie/compare/0.4.0...0.4.1
-.. _0.5.0-alpha: https://github.com/jkbr/httpie/compare/0.4.0...master
+.. _0.5.0: https://github.com/jkbr/httpie/compare/0.4.1...0.5.0
-.. _stable version: https://github.com/jkbr/httpie/tree/0.3.0#readme
+.. _0.6.0-dev: https://github.com/jkbr/httpie/compare/0.5.0...master
 .. _AUTHORS.rst: https://github.com/jkbr/httpie/blob/master/AUTHORS.rst
 .. _LICENSE: https://github.com/jkbr/httpie/blob/master/LICENSE

httpie/__init__.py

@@ -3,7 +3,7 @@ HTTPie - a CLI, cURL-like tool for humans.
 
 """
 __author__ = 'Jakub Roztocil'
-__version__ = '0.4.1'
+__version__ = '0.5.0'
 __licence__ = 'BSD'
 
 

httpie/cli.py

@@ -1,15 +1,12 @@
 """CLI arguments definition.
 
 NOTE: the CLI interface may change before reaching v1.0.
-TODO: make the options config friendly, i.e., no mutually exclusive groups to
-      allow options overwriting.
 
 """
 from argparse import FileType, OPTIONAL, ZERO_OR_MORE, SUPPRESS
 
 from . import __doc__
 from . import __version__
-from .compat import is_windows
 from .sessions import DEFAULT_SESSIONS_DIR, Session
 from .output import AVAILABLE_STYLES, DEFAULT_STYLE
 from .input import (Parser, AuthCredentialsArgType, KeyValueArgType,
@@ -45,7 +42,8 @@ positional = parser.add_argument_group(
     ''')
 )
 positional.add_argument(
-    'method', metavar='METHOD',
+    'method',
+    metavar='METHOD',
     nargs=OPTIONAL,
     default=None,
     help=_('''
@@ -57,14 +55,16 @@ positional.add_argument(
     ''')
 )
 positional.add_argument(
-    'url', metavar='URL',
+    'url',
+    metavar='URL',
     help=_('''
         The protocol defaults to http:// if the
         URL does not include one.
     ''')
 )
 positional.add_argument(
-    'items', metavar='REQUEST ITEM',
+    'items',
+    metavar='REQUEST ITEM',
     nargs=ZERO_OR_MORE,
     type=KeyValueArgType(*SEP_GROUP_ITEMS),
     help=_('''
@@ -90,7 +90,8 @@ content_type = parser.add_argument_group(
 )
 
 content_type.add_argument(
-    '--json', '-j', action='store_true',
+    '--json', '-j',
+    action='store_true',
     help=_('''
         (default) Data items from the command
         line are serialized as a JSON object.
@@ -99,7 +100,8 @@ content_type.add_argument(
     ''')
 )
 content_type.add_argument(
-    '--form', '-f', action='store_true',
+    '--form', '-f',
+    action='store_true',
     help=_('''
         Data items from the command line are serialized as form fields.
         The Content-Type is set to application/x-www-form-urlencoded
@@ -117,20 +119,9 @@ content_type.add_argument(
 output_processing = parser.add_argument_group(title='Output processing')
 
 output_processing.add_argument(
-    '--output', '-o', type=FileType('w+b'),
+    '--pretty',
-    metavar='FILE',
+    dest='prettify',
-    help=SUPPRESS if not is_windows else _(
+    default=PRETTY_STDOUT_TTY_ONLY,
-        '''
-        Save output to FILE.
-        This option is a replacement for piping output to FILE,
-        which would on Windows result in corrupted data
-        being saved.
-
-        '''
-    )
-)
-output_processing.add_argument(
-    '--pretty', dest='prettify', default=PRETTY_STDOUT_TTY_ONLY,
     choices=sorted(PRETTY_MAP.keys()),
     help=_('''
         Controls output processing. The value can be "none" to not prettify
@@ -140,7 +131,10 @@ output_processing.add_argument(
     ''')
 )
 output_processing.add_argument(
-    '--style', '-s', dest='style', default=DEFAULT_STYLE, metavar='STYLE',
+    '--style', '-s',
+    dest='style',
+    metavar='STYLE',
+    default=DEFAULT_STYLE,
     choices=AVAILABLE_STYLES,
     help=_('''
         Output coloring style. One of %s. Defaults to "%s".
@@ -157,7 +151,9 @@ output_processing.add_argument(
 output_options = parser.add_argument_group(title='Output options')
 
 output_options.add_argument(
-    '--print', '-p', dest='output_options', metavar='WHAT',
+    '--print', '-p',
+    dest='output_options',
+    metavar='WHAT',
     help=_('''
         String specifying what the output should contain:
         "{request_headers}" stands for the request headers,  and
@@ -174,24 +170,30 @@ output_options.add_argument(
                    response_body=OUT_RESP_BODY,))
 )
 output_options.add_argument(
-    '--verbose', '-v', dest='output_options',
+    '--verbose', '-v',
-    action='store_const', const=''.join(OUTPUT_OPTIONS),
+    dest='output_options',
+    action='store_const',
+    const=''.join(OUTPUT_OPTIONS),
     help=_('''
         Print the whole request as well as the response.
         Shortcut for --print={0}.
     '''.format(''.join(OUTPUT_OPTIONS)))
 )
 output_options.add_argument(
-    '--headers', '-h', dest='output_options',
+    '--headers', '-h',
-    action='store_const', const=OUT_RESP_HEAD,
+    dest='output_options',
+    action='store_const',
+    const=OUT_RESP_HEAD,
     help=_('''
         Print only the response headers.
         Shortcut for --print={0}.
     '''.format(OUT_RESP_HEAD))
 )
 output_options.add_argument(
-    '--body', '-b', dest='output_options',
+    '--body', '-b',
-    action='store_const', const=OUT_RESP_BODY,
+    dest='output_options',
+    action='store_const',
+    const=OUT_RESP_BODY,
     help=_('''
         Print only the response body.
         Shortcut for --print={0}.
@@ -199,7 +201,9 @@ output_options.add_argument(
 )
 
 output_options.add_argument(
-    '--stream', '-S', action='store_true', default=False,
+    '--stream', '-S',
+    action='store_true',
+    default=False,
     help=_('''
     Always stream the output by line, i.e., behave like `tail -f'.
 
@@ -214,7 +218,43 @@ output_options.add_argument(
 
     ''')
 )
+output_processing.add_argument(
+    '--output', '-o',
+    type=FileType('a+b'),
+    dest='output_file',
+    metavar='FILE',
+    help=_(
+        '''
+        Save output to FILE. If --download is set, then only the response
+        body is saved to the file. Other parts of the HTTP exchange are
+        printed to stderr.
 
+        '''
+    )
+)
+
+output_options.add_argument(
+    '--download', '-d',
+    action='store_true',
+    default=False,
+    help=_('''
+    Do not print the response body to stdout. Rather, download it and store it
+    in a file. The filename is guessed unless specified with --output
+    [filename]. This action is similar to the default behaviour of wget.
+
+    ''')
+)
+
+output_options.add_argument(
+    '--continue', '-c',
+    dest='download_resume',
+    action='store_true',
+    default=False,
+    help=_('''
+    Resume an interrupted download.
+    The --output option needs to be specified as well.
+    ''')
+)
 
 ###############################################################################
 # Sessions
@@ -223,7 +263,9 @@ sessions = parser.add_argument_group(title='Sessions')\
                  .add_mutually_exclusive_group(required=False)
 
 sessions.add_argument(
-    '--session', metavar='SESSION_NAME', type=RegexValidator(
+    '--session',
+    metavar='SESSION_NAME',
+    type=RegexValidator(
         Session.VALID_NAME_PATTERN,
         'Session name contains invalid characters.'
     ),
@@ -235,7 +277,8 @@ sessions.add_argument(
     ''' % DEFAULT_SESSIONS_DIR)
 )
 sessions.add_argument(
-    '--session-read-only', metavar='SESSION_NAME',
+    '--session-read-only',
+    metavar='SESSION_NAME',
     help=_('''
         Create or read a session without updating it form the
         request/response exchange.
@@ -249,7 +292,8 @@ sessions.add_argument(
 # ``requests.request`` keyword arguments.
 auth = parser.add_argument_group(title='Authentication')
 auth.add_argument(
-    '--auth', '-a', metavar='USER[:PASS]',
+    '--auth', '-a',
+    metavar='USER[:PASS]',
     type=AuthCredentialsArgType(SEP_CREDENTIALS),
     help=_('''
         If only the username is provided (-a username),
@@ -258,7 +302,9 @@ auth.add_argument(
 )
 
 auth.add_argument(
-    '--auth-type', choices=['basic', 'digest'], default='basic',
+    '--auth-type',
+    choices=['basic', 'digest'],
+    default='basic',
     help=_('''
         The authentication mechanism to be used.
         Defaults to "basic".
@@ -272,7 +318,10 @@ auth.add_argument(
 network = parser.add_argument_group(title='Network')
 
 network.add_argument(
-    '--proxy', default=[], action='append', metavar='PROTOCOL:HOST',
+    '--proxy',
+    default=[],
+    action='append',
+    metavar='PROTOCOL:HOST',
     type=KeyValueArgType(SEP_PROXY),
     help=_('''
         String mapping protocol to the URL of the proxy
@@ -281,14 +330,17 @@ network.add_argument(
     ''')
 )
 network.add_argument(
-    '--follow', default=False, action='store_true',
+    '--follow',
+    default=False,
+    action='store_true',
     help=_('''
         Set this flag if full redirects are allowed
         (e.g. re-POST-ing of data at new ``Location``)
     ''')
 )
 network.add_argument(
-    '--verify', default='yes',
+    '--verify',
+    default='yes',
     help=_('''
         Set to "no" to skip checking the host\'s SSL certificate.
         You can also pass the  path to a CA_BUNDLE
@@ -299,14 +351,19 @@ network.add_argument(
 )
 
 network.add_argument(
-    '--timeout', type=float, default=30, metavar='SECONDS',
+    '--timeout',
+    type=float,
+    default=30,
+    metavar='SECONDS',
     help=_('''
         The connection timeout of the request in seconds.
         The default value is 30 seconds.
     ''')
 )
 network.add_argument(
-    '--check-status', default=False, action='store_true',
+    '--check-status',
+    default=False,
+    action='store_true',
     help=_('''
         By default, HTTPie exits with 0 when no network or other fatal
         errors occur.
@@ -333,17 +390,25 @@ troubleshooting = parser.add_argument_group(title='Troubleshooting')
 
 troubleshooting.add_argument(
     '--help',
-    action='help', default=SUPPRESS,
+    action='help',
+    default=SUPPRESS,
     help='Show this help message and exit'
 )
 troubleshooting.add_argument(
-    '--version', action='version', version=__version__)
+    '--version',
+    action='version',
+    version=__version__
+)
 troubleshooting.add_argument(
-    '--traceback', action='store_true', default=False,
+    '--traceback',
+    action='store_true',
+    default=False,
     help='Prints exception traceback should one occur.'
 )
 troubleshooting.add_argument(
-    '--debug', action='store_true', default=False,
+    '--debug',
+    action='store_true',
+    default=False,
     help=_('''
         Prints exception traceback should one occur, and also other
         information that is useful for debugging HTTPie itself and

httpie/client.py

@@ -44,9 +44,10 @@ def get_requests_kwargs(args):
     }
 
     auto_json = args.data and not args.form
+    # FIXME: Accept is set to JSON with `http url @./file.txt`.
     if args.json or auto_json:
         implicit_headers['Accept'] = 'application/json'
-        if args.data:
+        if args.json or (auto_json and args.data):
             implicit_headers['Content-Type'] = JSON
 
         if isinstance(args.data, dict):

httpie/core.py

@@ -21,8 +21,9 @@ from pygments import __version__ as pygments_version
 from .cli import parser
 from .compat import str, is_py3
 from .client import get_response
+from .downloads import Download
 from .models import Environment
-from .output import build_output_stream, write, write_with_colors_win_p3k
+from .output import build_output_stream, write, write_with_colors_win_py3
 from . import ExitStatus
 
 
@@ -63,7 +64,7 @@ def main(args=sys.argv[1:], env=Environment()):
     def error(msg, *args, **kwargs):
         msg = msg % args
         level = kwargs.get('level', 'error')
-        env.stderr.write('http: %s: %s\n' % (level, msg))
+        env.stderr.write('\nhttp: %s: %s\n' % (level, msg))
 
     debug = '--debug' in args
     traceback = debug or '--traceback' in args
@@ -74,13 +75,28 @@ def main(args=sys.argv[1:], env=Environment()):
         if args == ['--debug']:
             return exit_status
 
+    download = None
+
     try:
         args = parser.parse_args(args=args, env=env)
 
+        if args.download:
+            args.follow = True  # --download implies --follow.
+            download = Download(
+                output_file=args.output_file,
+                progress_file=env.stderr,
+                resume=args.download_resume
+            )
+            download.pre_request(args.headers)
+
         response = get_response(args, config_dir=env.config.directory)
 
-        if args.check_status:
+        if args.check_status or download:
-            exit_status = get_exit_status(response.status_code, args.follow)
+
+            exit_status = get_exit_status(
+                http_status=response.status_code,
+                follow=args.follow
+            )
 
             if not env.stdout_isatty and exit_status != ExitStatus.OK:
                 error('HTTP %s %s',
@@ -89,25 +105,40 @@ def main(args=sys.argv[1:], env=Environment()):
                       level='warning')
 
         write_kwargs = {
-            'stream': build_output_stream(args, env,
+            'stream': build_output_stream(
-                                          response.request,
+                args, env, response.request, response),
-                                          response),
+
+            # This will in fact be `stderr` with `--download`
             'outfile': env.stdout,
+
             'flush': env.stdout_isatty or args.stream
         }
+
         try:
+
             if env.is_windows and is_py3 and 'colors' in args.prettify:
-                write_with_colors_win_p3k(**write_kwargs)
+                write_with_colors_win_py3(**write_kwargs)
             else:
                 write(**write_kwargs)
 
+            if download and exit_status == ExitStatus.OK:
+                # Response body download.
+                download_stream, download_to = download.start(response)
+                write(
+                    stream=download_stream,
+                    outfile=download_to,
+                    flush=False,
+                )
+                download.finish()
+                if download.interrupted:
+                    exit_status = ExitStatus.ERROR
+
         except IOError as e:
             if not traceback and e.errno == errno.EPIPE:
                 # Ignore broken pipes unless --traceback.
                 env.stderr.write('\n')
             else:
                 raise
-
     except (KeyboardInterrupt, SystemExit):
         if traceback:
             raise
@@ -126,4 +157,8 @@ def main(args=sys.argv[1:], env=Environment()):
         error('%s: %s', type(e).__name__, str(e))
         exit_status = ExitStatus.ERROR
 
+    finally:
+        if download and not download.finished:
+            download.failed()
+
     return exit_status

httpie/downloads.py

@@ -0,0 +1,423 @@
+# coding=utf-8
+"""
+Download mode implementation.
+
+"""
+from __future__ import division
+import os
+import re
+import sys
+import mimetypes
+import threading
+from time import time
+
+from .output import RawStream
+from .models import HTTPResponse
+from .utils import humanize_bytes
+from .compat import urlsplit
+
+
+PARTIAL_CONTENT = 206
+
+
+CLEAR_LINE = '\r\033[K'
+PROGRESS = (
+    '{percentage: 6.2f} %'
+    ' {downloaded: >10}'
+    ' {speed: >10}/s'
+    ' {eta: >8} ETA'
+)
+PROGRESS_NO_CONTENT_LENGTH = '{downloaded: >10} {speed: >10}/s'
+SUMMARY = 'Done. {downloaded} in {time:0.5f}s ({speed}/s)\n'
+SPINNER = '|/-\\'
+
+
+class ContentRangeError(ValueError):
+    pass
+
+
+def parse_content_range(content_range, resumed_from):
+    """
+    Parse and validate Content-Range header.
+
+    <http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>
+
+    :param content_range: the value of a Content-Range response header
+                          eg. "bytes 21010-47021/47022"
+    :param resumed_from: first byte pos. from the Range request header
+    :return: total size of the response body when fully downloaded.
+
+    """
+    if content_range is None:
+        raise ContentRangeError('Missing Content-Range')
+
+    pattern = (
+        '^bytes (?P<first_byte_pos>\d+)-(?P<last_byte_pos>\d+)'
+        '/(\*|(?P<instance_length>\d+))$'
+    )
+    match = re.match(pattern, content_range)
+
+    if not match:
+        raise ContentRangeError(
+            'Invalid Content-Range format %r' % content_range)
+
+    content_range_dict = match.groupdict()
+    first_byte_pos = int(content_range_dict['first_byte_pos'])
+    last_byte_pos = int(content_range_dict['last_byte_pos'])
+    instance_length = (
+        int(content_range_dict['instance_length'])
+        if content_range_dict['instance_length']
+        else None
+    )
+
+    # "A byte-content-range-spec with a byte-range-resp-spec whose
+    # last- byte-pos value is less than its first-byte-pos value,
+    # or whose instance-length value is less than or equal to its
+    # last-byte-pos value, is invalid. The recipient of an invalid
+    # byte-content-range- spec MUST ignore it and any content
+    # transferred along with it."
+    if (first_byte_pos >= last_byte_pos
+            or (instance_length is not None
+                and instance_length <= last_byte_pos)):
+        raise ContentRangeError(
+            'Invalid Content-Range returned: %r' % content_range)
+
+    if (first_byte_pos != resumed_from
+        or (instance_length is not None
+            and last_byte_pos + 1 != instance_length)):
+        # Not what we asked for.
+        raise ContentRangeError(
+            'Unexpected Content-Range returned (%r)'
+            ' for the requested Range ("bytes=%d-")'
+            % (content_range, resumed_from)
+        )
+
+    return last_byte_pos + 1
+
+
+def filename_from_content_disposition(content_disposition):
+    """
+    Extract and validate filename from a Content-Disposition header.
+
+    :param content_disposition: Content-Disposition value
+    :return: the filename if present and valid, otherwise `None`
+
+    """
+    # attachment; filename=jkbr-httpie-0.4.1-20-g40bd8f6.tar.gz
+    match = re.search('filename=(\S+)', content_disposition)
+    if match and match.group(1):
+        fn = match.group(1).strip('."')
+        if re.match('^[a-zA-Z0-9._-]+$', fn):
+            return fn
+
+
+def filename_from_url(url, content_type):
+    fn = urlsplit(url).path.rstrip('/')
+    fn = os.path.basename(fn) if fn else 'index'
+    if '.' not in fn and content_type:
+        content_type = content_type.split(';')[0]
+        if content_type == 'text/plain':
+            # mimetypes returns '.ksh'
+            ext = '.txt'
+        else:
+            ext = mimetypes.guess_extension(content_type)
+
+        if ext == '.htm':  # Python 3
+            ext = '.html'
+
+        if ext:
+            fn += ext
+
+    return fn
+
+
+def get_unique_filename(fn, exists=os.path.exists):
+    attempt = 0
+    while True:
+        suffix = '-' + str(attempt) if attempt > 0 else ''
+        if not exists(fn + suffix):
+            return fn + suffix
+        attempt += 1
+
+
+class Download(object):
+
+    def __init__(self, output_file=None,
+                 resume=False, progress_file=sys.stderr):
+        """
+        :param resume: Should the download resume if partial download
+                       already exists.
+        :type resume: bool
+
+        :param output_file: The file to store response body in. If not
+                            provided, it will be guessed from the response.
+        :type output_file: file
+
+        :param progress_file: Where to report download progress.
+        :type progress_file: file
+
+        """
+        self._output_file = output_file
+        self._resume = resume
+        self._resumed_from = 0
+        self.finished = False
+
+        self._status = Status()
+        self._progress_reporter = ProgressReporter(
+            status=self._status,
+            output=progress_file
+        )
+
+    def pre_request(self, request_headers):
+        """Called just before the HTTP request is sent.
+
+        Might alter `request_headers`.
+
+        :type request_headers: dict
+
+        """
+        # Disable content encoding so that we can resume, etc.
+        request_headers['Accept-Encoding'] = None
+        if self._resume:
+            bytes_have = os.path.getsize(self._output_file.name)
+            if bytes_have:
+                # Set ``Range`` header to resume the download
+                # TODO: Use "If-Range: mtime" to make sure it's fresh?
+                request_headers['Range'] = 'bytes=%d-' % bytes_have
+                self._resumed_from = bytes_have
+
+    def start(self, response):
+        """
+        Initiate and return a stream for `response` body  with progress
+        callback attached. Can be called only once.
+
+        :param response: Initiated response object with headers already fetched
+        :type response: requests.models.Response
+
+        :return: RawStream, output_file
+
+        """
+        assert not self._status.time_started
+
+        try:
+            total_size = int(response.headers['Content-Length'])
+        except (KeyError, ValueError, TypeError):
+            total_size = None
+
+        if self._output_file:
+            if self._resume and response.status_code == PARTIAL_CONTENT:
+                total_size = parse_content_range(
+                    response.headers.get('Content-Range'),
+                    self._resumed_from
+                )
+
+            else:
+                self._resumed_from = 0
+                try:
+                    self._output_file.seek(0)
+                    self._output_file.truncate()
+                except IOError:
+                    pass  # stdout
+        else:
+            # TODO: Should the filename be taken from response.history[0].url?
+            # Output file not specified. Pick a name that doesn't exist yet.
+            fn = None
+            if 'Content-Disposition' in response.headers:
+                fn = filename_from_content_disposition(
+                    response.headers['Content-Disposition'])
+            if not fn:
+                fn = filename_from_url(
+                    url=response.url,
+                    content_type=response.headers.get('Content-Type'),
+                )
+            self._output_file = open(get_unique_filename(fn), mode='a+b')
+
+        self._status.started(
+            resumed_from=self._resumed_from,
+            total_size=total_size
+        )
+
+        stream = RawStream(
+            msg=HTTPResponse(response),
+            with_headers=False,
+            with_body=True,
+            on_body_chunk_downloaded=self._chunk_downloaded,
+            chunk_size=1024 * 8
+        )
+
+        self._progress_reporter.output.write(
+            'Downloading %sto "%s"\n' % (
+                (humanize_bytes(total_size) + ' '
+                 if total_size is not None
+                 else ''),
+                self._output_file.name
+            )
+        )
+        self._progress_reporter.report()
+
+        return stream, self._output_file
+
+    def finish(self):
+        assert not self.finished
+        self.finished = True
+        self._status.finished()
+
+    def failed(self):
+        self.finish()
+        self._progress_reporter.stop()
+
+    @property
+    def interrupted(self):
+        return (
+            self.finished
+            and self._status.total_size
+            and self._status.total_size != self._status.downloaded
+        )
+
+    def _chunk_downloaded(self, chunk):
+        """
+        A download progress callback.
+
+        :param chunk: A chunk of response body data that has just
+                      been downloaded and written to the output.
+        :type chunk: bytes
+
+        """
+        self._status.chunk_downloaded(len(chunk))
+
+
+class Status(object):
+    """Holds details about the downland status."""
+
+    def __init__(self):
+        self.downloaded = 0
+        self.total_size = None
+        self.resumed_from = 0
+        self.time_started = None
+        self.time_finished = None
+
+    def started(self, resumed_from=0, total_size=None):
+        assert self.time_started is None
+        if total_size is not None:
+            self.total_size = total_size
+        self.downloaded = self.resumed_from = resumed_from
+        self.time_started = time()
+
+    def chunk_downloaded(self, size):
+        assert self.time_finished is None
+        self.downloaded += size
+
+    @property
+    def has_finished(self):
+        return self.time_finished is not None
+
+    def finished(self):
+        assert self.time_started is not None
+        assert self.time_finished is None
+        self.time_finished = time()
+
+
+class ProgressReporter(object):
+    """
+    Reports download progress based on its status.
+
+    Uses threading to periodically update the status (speed, ETA, etc.).
+
+    """
+    def __init__(self, status, output, tick=.1, update_interval=1):
+        """
+
+        :type status: Status
+        :type output: file
+        """
+        self.status = status
+        self.output = output
+        self._tick = tick
+        self._update_interval = update_interval
+        self._spinner_pos = 0
+        self._status_line = ''
+        self._prev_bytes = 0
+        self._prev_time = time()
+        self._stop = False
+
+    def stop(self):
+        """Stop reporting on next tick."""
+        self._stop = True
+
+    def report(self):
+        if self._stop:
+            return
+        if self.status.has_finished:
+            self.sum_up()
+        else:
+            self.report_speed()
+            threading.Timer(self._tick, self.report).start()
+
+    def report_speed(self):
+
+        now = time()
+
+        if now - self._prev_time >= self._update_interval:
+            downloaded = self.status.downloaded
+            try:
+                speed = ((downloaded - self._prev_bytes)
+                         / (now - self._prev_time))
+            except ZeroDivisionError:
+                speed = 0
+
+            if not self.status.total_size:
+                self._status_line = PROGRESS_NO_CONTENT_LENGTH.format(
+                    downloaded=humanize_bytes(downloaded),
+                    speed=humanize_bytes(speed),
+                )
+            else:
+                try:
+                    percentage = downloaded / self.status.total_size * 100
+                except ZeroDivisionError:
+                    percentage = 0
+
+                if not speed:
+                    eta = '-:--:--'
+                else:
+                    s = int((self.status.total_size - downloaded) / speed)
+                    h, s = divmod(s, 60 * 60)
+                    m, s = divmod(s, 60)
+                    eta = '{}:{:0>2}:{:0>2}'.format(h, m, s)
+
+                self._status_line = PROGRESS.format(
+                    percentage=percentage,
+                    downloaded=humanize_bytes(downloaded),
+                    speed=humanize_bytes(speed),
+                    eta=eta,
+                )
+
+            self._prev_time = now
+            self._prev_bytes = downloaded
+
+        self.output.write(
+            CLEAR_LINE
+            + ' '
+            + SPINNER[self._spinner_pos]
+            + ' '
+            + self._status_line
+        )
+        self.output.flush()
+
+        self._spinner_pos = (self._spinner_pos + 1
+                             if self._spinner_pos + 1 != len(SPINNER)
+                             else 0)
+
+    def sum_up(self):
+        actually_downloaded = (self.status.downloaded
+                               - self.status.resumed_from)
+        time_taken = self.status.time_finished - self.status.time_started
+
+        self.output.write(CLEAR_LINE)
+        self.output.write(SUMMARY.format(
+            downloaded=humanize_bytes(actually_downloaded),
+            total=(self.status.total_size
+                   and humanize_bytes(self.status.total_size)),
+            speed=humanize_bytes(actually_downloaded / time_taken),
+            time=time_taken,
+        ))
+        self.output.flush()

httpie/input.py

@@ -98,57 +98,103 @@ class Parser(ArgumentParser):
     def parse_args(self, env, args=None, namespace=None):
 
         self.env = env
+        self.args, no_options = super(Parser, self)\
+            .parse_known_args(args, namespace)
 
-        args, no_options = super(Parser, self).parse_known_args(args,
+        if self.args.debug:
-                                                                namespace)
+            self.args.traceback = True
-
-        self._apply_no_options(args, no_options)
-
-        if not args.json and env.config.implicit_content_type == 'form':
-            args.form = True
-
-        if args.debug:
-            args.traceback = True
-
-        if args.output:
-            env.stdout = args.output
-            env.stdout_isatty = False
-
-        self._process_output_options(args, env)
-        self._process_pretty_options(args, env)
-        self._guess_method(args, env)
-        self._parse_items(args)
 
+        # Arguments processing and environment setup.
+        self._apply_no_options(no_options)
+        self._apply_config()
+        self._validate_download_options()
+        self._setup_standard_streams()
+        self._process_output_options()
+        self._process_pretty_options()
+        self._guess_method()
+        self._parse_items()
         if not env.stdin_isatty:
-            self._body_from_file(args, env.stdin)
+            self._body_from_file(self.env.stdin)
+        if not (self.args.url.startswith((HTTP, HTTPS))):
+            # Default to 'https://' if invoked as `https args`.
+            scheme = HTTPS if self.env.progname == 'https' else HTTP
+            self.args.url = scheme + self.args.url
+        self._process_auth()
 
-        if not (args.url.startswith(HTTP) or args.url.startswith(HTTPS)):
+        return self.args
-            scheme = HTTPS if env.progname == 'https' else HTTP
-            args.url = scheme + args.url
 
-        self._process_auth(args)
+    # noinspection PyShadowingBuiltins
+    def _print_message(self, message, file=None):
+        # Sneak in our stderr/stdout.
+        file = {
+            sys.stdout: self.env.stdout,
+            sys.stderr: self.env.stderr,
+            None: self.env.stderr
+        }.get(file, file)
 
-        return args
+        super(Parser, self)._print_message(message, file)
 
-    def _process_auth(self, args):
+    def _setup_standard_streams(self):
-        url = urlsplit(args.url)
+        """
+        Modify `env.stdout` and `env.stdout_isatty` based on args, if needed.
 
-        if args.auth:
+        """
-            if not args.auth.has_password():
+        if not self.env.stdout_isatty and self.args.output_file:
+            self.error('Cannot use --output, -o with redirected output.')
+
+        # FIXME: Come up with a cleaner solution.
+        if self.args.download:
+
+            if not self.env.stdout_isatty:
+                # Use stdout as tge download output file.
+                self.args.output_file = self.env.stdout
+
+            # With `--download`, we write everything that would normally go to
+            # `stdout` to `stderr` instead. Let's replace the stream so that
+            # we don't have to use many `if`s throughout the codebase.
+            # The response body will be treated separately.
+            self.env.stdout = self.env.stderr
+            self.env.stdout_isatty = self.env.stderr_isatty
+
+        elif self.args.output_file:
+            # When not `--download`ing, then `--output` simply replaces
+            # `stdout`. The file is opened for appending, which isn't what
+            # we want in this case.
+            self.args.output_file.seek(0)
+            self.args.output_file.truncate()
+
+            self.env.stdout = self.args.output_file
+            self.env.stdout_isatty = False
+
+    def _apply_config(self):
+        if (not self.args.json
+                and self.env.config.implicit_content_type == 'form'):
+            self.args.form = True
+
+    def _process_auth(self):
+        """
+        If only a username provided via --auth, then ask for a password.
+        Or, take credentials from the URL, if provided.
+
+        """
+        url = urlsplit(self.args.url)
+
+        if self.args.auth:
+            if not self.args.auth.has_password():
                 # Stdin already read (if not a tty) so it's save to prompt.
-                args.auth.prompt_password(url.netloc)
+                self.args.auth.prompt_password(url.netloc)
 
         elif url.username is not None:
             # Handle http://username:password@hostname/
             username, password = url.username, url.password
-            args.auth = AuthCredentials(
+            self.args.auth = AuthCredentials(
                 key=username,
                 value=password,
                 sep=SEP_CREDENTIALS,
                 orig=SEP_CREDENTIALS.join([username, password])
             )
 
-    def _apply_no_options(self, args, no_options):
+    def _apply_no_options(self, no_options):
         """For every `--no-OPTION` in `no_options`, set `args.OPTION` to
         its default value. This allows for un-setting of options, e.g.,
         specified in config.
@@ -165,7 +211,7 @@ class Parser(ArgumentParser):
             inverted = '--' + option[5:]
             for action in self._actions:
                 if inverted in action.option_strings:
-                    setattr(args, action.dest, action.default)
+                    setattr(self.args, action.dest, action.default)
                     break
             else:
                 invalid.append(option)
@@ -174,123 +220,140 @@ class Parser(ArgumentParser):
             msg = 'unrecognized arguments: %s'
             self.error(msg % ' '.join(invalid))
 
-    def _print_message(self, message, file=None):
+    def _body_from_file(self, fd):
-        # Sneak in our stderr/stdout.
-        file = {
-            sys.stdout: self.env.stdout,
-            sys.stderr: self.env.stderr,
-            None: self.env.stderr
-        }.get(file, file)
-
-        super(Parser, self)._print_message(message, file)
-
-    def _body_from_file(self, args, fd):
         """There can only be one source of request data.
 
         Bytes are always read.
 
         """
-        if args.data:
+        if self.args.data:
             self.error('Request body (from stdin or a file) and request '
                        'data (key=value) cannot be mixed.')
-        args.data = getattr(fd, 'buffer', fd).read()
+        self.args.data = getattr(fd, 'buffer', fd).read()
 
-    def _guess_method(self, args, env):
+    def _guess_method(self):
         """Set `args.method` if not specified to either POST or GET
         based on whether the request has data or not.
 
         """
-        if args.method is None:
+        if self.args.method is None:
             # Invoked as `http URL'.
-            assert not args.items
+            assert not self.args.items
-            if not env.stdin_isatty:
+            if not self.env.stdin_isatty:
-                args.method = HTTP_POST
+                self.args.method = HTTP_POST
             else:
-                args.method = HTTP_GET
+                self.args.method = HTTP_GET
 
         # FIXME: False positive, e.g., "localhost" matches but is a valid URL.
-        elif not re.match('^[a-zA-Z]+$', args.method):
+        elif not re.match('^[a-zA-Z]+$', self.args.method):
             # Invoked as `http URL item+'. The URL is now in `args.method`
             # and the first ITEM is now incorrectly in `args.url`.
             try:
                 # Parse the URL as an ITEM and store it as the first ITEM arg.
-                args.items.insert(
+                self.args.items.insert(
-                    0, KeyValueArgType(*SEP_GROUP_ITEMS).__call__(args.url))
+                    0,
+                    KeyValueArgType(*SEP_GROUP_ITEMS).__call__(self.args.url)
+                )
 
             except ArgumentTypeError as e:
-                if args.traceback:
+                if self.args.traceback:
                     raise
                 self.error(e.message)
 
             else:
                 # Set the URL correctly
-                args.url = args.method
+                self.args.url = self.args.method
                 # Infer the method
-                has_data = not env.stdin_isatty or any(
+                has_data = not self.env.stdin_isatty or any(
-                    item.sep in SEP_GROUP_DATA_ITEMS for item in args.items)
+                    item.sep in SEP_GROUP_DATA_ITEMS
-                args.method = HTTP_POST if has_data else HTTP_GET
+                    for item in self.args.items
+                )
+                self.args.method = HTTP_POST if has_data else HTTP_GET
 
-    def _parse_items(self, args):
+    def _parse_items(self):
-        """Parse `args.items` into `args.headers`, `args.data`,
+        """Parse `args.items` into `args.headers`, `args.data`, `args.params`,
-        `args.`, and `args.files`.
+         and `args.files`.
 
         """
-        args.headers = CaseInsensitiveDict()
+        self.args.headers = CaseInsensitiveDict()
-        args.data = ParamDict() if args.form else OrderedDict()
+        self.args.data = ParamDict() if self.args.form else OrderedDict()
-        args.files = OrderedDict()
+        self.args.files = OrderedDict()
-        args.params = ParamDict()
+        self.args.params = ParamDict()
 
         try:
-            parse_items(items=args.items,
+            parse_items(items=self.args.items,
-                        headers=args.headers,
+                        headers=self.args.headers,
-                        data=args.data,
+                        data=self.args.data,
-                        files=args.files,
+                        files=self.args.files,
-                        params=args.params)
+                        params=self.args.params)
         except ParseError as e:
-            if args.traceback:
+            if self.args.traceback:
                 raise
             self.error(e.message)
 
-        if args.files and not args.form:
+        if self.args.files and not self.args.form:
             # `http url @/path/to/file`
-            file_fields = list(args.files.keys())
+            file_fields = list(self.args.files.keys())
             if file_fields != ['']:
                 self.error(
                     'Invalid file fields (perhaps you meant --form?): %s'
                     % ','.join(file_fields))
 
-            fn, fd = args.files['']
+            fn, fd = self.args.files['']
-            args.files = {}
+            self.args.files = {}
-            self._body_from_file(args, fd)
+
-            if 'Content-Type' not in args.headers:
+            self._body_from_file(fd)
+
+            if 'Content-Type' not in self.args.headers:
                 mime, encoding = mimetypes.guess_type(fn, strict=False)
                 if mime:
                     content_type = mime
                     if encoding:
                         content_type = '%s; charset=%s' % (mime, encoding)
-                    args.headers['Content-Type'] = content_type
+                    self.args.headers['Content-Type'] = content_type
 
-    def _process_output_options(self, args, env):
+    def _process_output_options(self):
-        """Apply defaults to output options or validate the provided ones.
+        """Apply defaults to output options, or validate the provided ones.
 
         The default output options are stdout-type-sensitive.
 
         """
-        if not args.output_options:
+        if not self.args.output_options:
-            args.output_options = (OUTPUT_OPTIONS_DEFAULT if env.stdout_isatty
+            self.args.output_options = (
-                                else OUTPUT_OPTIONS_DEFAULT_STDOUT_REDIRECTED)
+                OUTPUT_OPTIONS_DEFAULT
+                if self.env.stdout_isatty
+                else OUTPUT_OPTIONS_DEFAULT_STDOUT_REDIRECTED
+            )
 
-        unknown = set(args.output_options) - OUTPUT_OPTIONS
+        unknown_output_options = set(self.args.output_options) - OUTPUT_OPTIONS
-        if unknown:
+        if unknown_output_options:
-            self.error('Unknown output options: %s' % ','.join(unknown))
+            self.error(
+                'Unknown output options: %s' % ','.join(unknown_output_options)
+            )
 
-    def _process_pretty_options(self, args, env):
+        if self.args.download and OUT_RESP_BODY in self.args.output_options:
-        if args.prettify == PRETTY_STDOUT_TTY_ONLY:
+            # Response body is always downloaded with --download and it goes
-            args.prettify = PRETTY_MAP['all' if env.stdout_isatty else 'none']
+            # through a different routine, so we remove it.
-        elif args.prettify and env.is_windows:
+            self.args.output_options = str(
+                set(self.args.output_options) - set(OUT_RESP_BODY))
+
+    def _process_pretty_options(self):
+        if self.args.prettify == PRETTY_STDOUT_TTY_ONLY:
+            self.args.prettify = PRETTY_MAP[
+                'all' if self.env.stdout_isatty else 'none']
+        elif self.args.prettify and self.env.is_windows:
             self.error('Only terminal output can be colorized on Windows.')
         else:
-            args.prettify = PRETTY_MAP[args.prettify]
+            # noinspection PyTypeChecker
+            self.args.prettify = PRETTY_MAP[self.args.prettify]
+
+    def _validate_download_options(self):
+        if not self.args.download:
+            if self.args.download_resume:
+                self.error('--continue only works with --download')
+        if self.args.download_resume and not (
+                self.args.download and self.args.output_file):
+            self.error('--continue requires --output to be specified')
 
 
 class ParseError(Exception):
@@ -320,16 +383,6 @@ def session_name_arg_type(name):
     return name
 
 
-def host_name_arg_type(name):
-    from .sessions import Host
-    if not Host.is_valid_name(name):
-        raise ArgumentTypeError(
-            'special characters and spaces are not'
-            ' allowed in host names: "%s"'
-            % name)
-    return name
-
-
 class RegexValidator(object):
 
     def __init__(self, pattern, error_message):

httpie/models.py

@@ -19,23 +19,27 @@ class Environment(object):
     if progname not in ['http', 'https']:
         progname = 'http'
 
-    stdin_isatty = sys.stdin.isatty()
-    stdin = sys.stdin
-    stdout_isatty = sys.stdout.isatty()
-
     config_dir = DEFAULT_CONFIG_DIR
 
-    if stdout_isatty and is_windows:
-        from colorama.initialise import wrap_stream
-        stdout = wrap_stream(sys.stdout, convert=None,
-                             strip=None, autoreset=True, wrap=True)
-    else:
-        stdout = sys.stdout
-    stderr = sys.stderr
-
     # Can be set to 0 to disable colors completely.
     colors = 256 if '256color' in os.environ.get('TERM', '') else 88
 
+    stdin = sys.stdin
+    stdin_isatty = sys.stdin.isatty()
+
+    stdout_isatty = sys.stdout.isatty()
+    stderr_isatty = sys.stderr.isatty()
+    if is_windows:
+        # noinspection PyUnresolvedReferences
+        from colorama.initialise import wrap_stream
+        stdout = wrap_stream(sys.stdout, convert=None,
+                             strip=None, autoreset=True, wrap=True)
+        stderr = wrap_stream(sys.stderr, convert=None,
+                             strip=None, autoreset=True, wrap=True)
+    else:
+        stdout = sys.stdout
+        stderr = sys.stderr
+
     def __init__(self, **kwargs):
         assert all(hasattr(type(self), attr)
                    for attr in kwargs.keys())
@@ -84,7 +88,13 @@ class HTTPMessage(object):
     @property
     def content_type(self):
         """Return the message content type."""
-        ct = self._orig.headers.get('Content-Type', '')
+        ct = self._orig.headers.get(
+            b'Content-Type',
+            self._orig.headers.get(
+                'Content-Type',
+                ''
+            )
+        )
         if isinstance(ct, bytes):
             ct = ct.decode()
         return ct

httpie/output.py

@@ -61,7 +61,7 @@ def write(stream, outfile, flush):
             outfile.flush()
 
 
-def write_with_colors_win_p3k(stream, outfile, flush):
+def write_with_colors_win_py3(stream, outfile, flush):
     """Like `write`, but colorized chunks are written as text
     directly to `outfile` to ensure it gets processed by colorama.
     Applies only to Windows with Python 3 and colorized terminal output.
@@ -147,7 +147,8 @@ def get_stream_type(env, args):
 class BaseStream(object):
     """Base HTTP message output stream class."""
 
-    def __init__(self, msg, with_headers=True, with_body=True):
+    def __init__(self, msg, with_headers=True, with_body=True,
+                 on_body_chunk_downloaded=None):
         """
         :param msg: a :class:`models.HTTPMessage` subclass
         :param with_headers: if `True`, headers will be included
@@ -158,6 +159,7 @@ class BaseStream(object):
         self.msg = msg
         self.with_headers = with_headers
         self.with_body = with_body
+        self.on_body_chunk_downloaded = on_body_chunk_downloaded
 
     def _get_headers(self):
         """Return the headers' bytes."""
@@ -177,6 +179,8 @@ class BaseStream(object):
             try:
                 for chunk in self._iter_body():
                     yield chunk
+                    if self.on_body_chunk_downloaded:
+                        self.on_body_chunk_downloaded(chunk)
             except BinarySuppressedError as e:
                 if self.with_headers:
                     yield b'\n'
@@ -187,7 +191,7 @@ class RawStream(BaseStream):
     """The message is streamed in chunks with no processing."""
 
     CHUNK_SIZE = 1024 * 100
-    CHUNK_SIZE_BY_LINE = 1024 * 5
+    CHUNK_SIZE_BY_LINE = 1
 
     def __init__(self, chunk_size=CHUNK_SIZE, **kwargs):
         super(RawStream, self).__init__(**kwargs)
@@ -205,7 +209,7 @@ class EncodedStream(BaseStream):
     is suppressed. The body is always streamed by line.
 
     """
-    CHUNK_SIZE = 1024 * 5
+    CHUNK_SIZE = 1
 
     def __init__(self, env=Environment(), **kwargs):
 
@@ -241,7 +245,7 @@ class PrettyStream(EncodedStream):
 
     """
 
-    CHUNK_SIZE = 1024 * 5
+    CHUNK_SIZE = 1
 
     def __init__(self, processor, **kwargs):
         super(PrettyStream, self).__init__(**kwargs)

httpie/sessions.py

@@ -152,13 +152,11 @@ class Session(BaseConfigDict):
         # http://docs.python.org/2/library/cookielib.html#cookie-objects
         stored_attrs = ['value', 'path', 'secure', 'expires']
         self['cookies'] = {}
-        for host in jar._cookies.values():
+        for cookie in jar:
-            for path in host.values():
+            self['cookies'][cookie.name] = dict(
-                for name, cookie in path.items():
+                (attname, getattr(cookie, attname))
-                    self['cookies'][name] = dict(
+                for attname in stored_attrs
-                        (attname, getattr(cookie, attname))
+            )
-                        for attname in stored_attrs
-                    )
 
     @property
     def auth(self):

httpie/utils.py

@@ -0,0 +1,46 @@
+from __future__ import division
+
+
+def humanize_bytes(n, precision=2):
+    # Author: Doug Latornell
+    # Licence: MIT
+    # URL: http://code.activestate.com/recipes/577081/
+    """Return a humanized string representation of a number of bytes.
+
+    Assumes `from __future__ import division`.
+
+    >>> humanize_bytes(1)
+    '1 byte'
+    >>> humanize_bytes(1024)
+    '1.0 kB'
+    >>> humanize_bytes(1024 * 123)
+    '123.0 kB'
+    >>> humanize_bytes(1024 * 12342)
+    '12.1 MB'
+    >>> humanize_bytes(1024 * 12342, 2)
+    '12.05 MB'
+    >>> humanize_bytes(1024 * 1234, 2)
+    '1.21 MB'
+    >>> humanize_bytes(1024 * 1234 * 1111, 2)
+    '1.31 GB'
+    >>> humanize_bytes(1024 * 1234 * 1111, 1)
+    '1.3 GB'
+
+    """
+    abbrevs = [
+        (1 << 50, 'PB'),
+        (1 << 40, 'TB'),
+        (1 << 30, 'GB'),
+        (1 << 20, 'MB'),
+        (1 << 10, 'kB'),
+        (1, 'B')
+    ]
+
+    if n == 1:
+        return '1 B'
+
+    for factor, suffix in abbrevs:
+        if n >= factor:
+            break
+
+    return '%.*f %s' % (precision, n / factor, suffix)

requirements-dev.txt

@@ -0,0 +1,3 @@
+tox
+httpbin
+docutils

requirements.txt

@@ -1 +0,0 @@
-#

tests/tests.py

@@ -27,16 +27,20 @@ import argparse
 import tempfile
 import unittest
 import shutil
+import time
+from requests.structures import CaseInsensitiveDict
 
 try:
     from urllib.request import urlopen
 except ImportError:
+    # noinspection PyUnresolvedReferences
     from urllib2 import urlopen
 try:
     from unittest import skipIf, skip
 except ImportError:
     skip = lambda msg: lambda self: None
 
+    # noinspection PyUnusedLocal
     def skipIf(cond, reason):
         def decorator(test_method):
             if cond:
@@ -62,6 +66,14 @@ from httpie.core import main
 from httpie.output import BINARY_SUPPRESSED_NOTICE
 from httpie.input import ParseError
 from httpie.compat import is_windows, is_py26, bytes, str
+from httpie.downloads import (
+    parse_content_range,
+    filename_from_content_disposition,
+    filename_from_url,
+    get_unique_filename,
+    ContentRangeError,
+    Download,
+)
 
 
 CRLF = '\r\n'
@@ -180,20 +192,25 @@ def http(*args, **kwargs):
     if not env:
         env = kwargs['env'] = TestEnvironment()
 
+    stdout = env.stdout
+    stderr = env.stderr
     try:
 
         try:
             exit_status = main(args=['--traceback'] + list(args), **kwargs)
+            if '--download' in args:
+                # Let the progress reporter thread finish.
+                time.sleep(.5)
         except Exception:
-            sys.stderr.write(env.stderr.read())
+            sys.stderr.write(stderr.read())
             raise
         except SystemExit:
             exit_status = ExitStatus.ERROR
 
-        env.stdout.seek(0)
+        stdout.seek(0)
-        env.stderr.seek(0)
+        stderr.seek(0)
 
-        output = env.stdout.read()
+        output = stdout.read()
         try:
             r = StrResponse(output.decode('utf8'))
         except UnicodeDecodeError:
@@ -215,14 +232,14 @@ def http(*args, **kwargs):
                         except ValueError:
                             pass
 
-        r.stderr = env.stderr.read()
+        r.stderr = stderr.read()
         r.exit_status = exit_status
 
         return r
 
     finally:
-        env.stdout.close()
+        stdout.close()
-        env.stderr.close()
+        stderr.close()
 
 
 class BaseTestCase(unittest.TestCase):
@@ -240,11 +257,15 @@ class BaseTestCase(unittest.TestCase):
             self.assertEqual(set(d1.keys()), set(d2.keys()), msg)
             self.assertEqual(sorted(d1.values()), sorted(d2.values()), msg)
 
+    def assertIsNone(self, obj, msg=None):
+        self.assertEqual(obj, None, msg=msg)
+
 
 #################################################################
 # High-level tests using httpbin.
 #################################################################
 
+
 class HTTPieTest(BaseTestCase):
 
     def test_GET(self):
@@ -268,7 +289,7 @@ class HTTPieTest(BaseTestCase):
             'foo=bar'
         )
         self.assertIn(OK, r)
-        self.assertIn('"foo": "bar"', r)
+        self.assertIn(r'\"foo\": \"bar\"', r)
 
     def test_POST_JSON_data(self):
         r = http(
@@ -277,7 +298,7 @@ class HTTPieTest(BaseTestCase):
             'foo=bar'
         )
         self.assertIn(OK, r)
-        self.assertIn('"foo": "bar"', r)
+        self.assertIn(r'\"foo\": \"bar\"', r)
 
     def test_POST_form(self):
         r = http(
@@ -436,7 +457,9 @@ class AutoContentTypeAndAcceptHeadersTest(BaseTestCase):
         )
         self.assertIn(OK, r)
         self.assertEqual(r.json['headers']['Accept'], 'application/json')
-        self.assertFalse(r.json['headers'].get('Content-Type'))
+        # Make sure Content-Type gets set even with no data.
+        # https://github.com/jkbr/httpie/issues/137
+        self.assertIn('application/json', r.json['headers']['Content-Type'])
 
     def test_GET_explicit_JSON_explicit_headers(self):
         r = http(
@@ -519,7 +542,7 @@ class ImplicitHTTPMethodTest(BaseTestCase):
             'hello=world'
         )
         self.assertIn(OK, r)
-        self.assertIn('"hello": "world"', r)
+        self.assertIn(r'\"hello\": \"world\"', r)
 
     def test_implicit_POST_form(self):
         r = http(
@@ -658,8 +681,8 @@ class VerboseFlagTest(BaseTestCase):
             'baz=bar'
         )
         self.assertIn(OK, r)
-        #noinspection PyUnresolvedReferences
+        self.assertIn('"baz": "bar"', r)  # request
-        self.assertEqual(r.count('"baz": "bar"'), 2)
+        self.assertIn(r'\"baz\": \"bar\"', r)  # response
 
 
 class MultipartFormDataFileUploadTest(BaseTestCase):
@@ -782,6 +805,8 @@ class RequestBodyFromFilePathTest(BaseTestCase):
 
     """
     def test_request_body_from_file_by_path(self):
+        # FIXME: *sometimes* fails on py33, the content-type is form.
+        # https://github.com/jkbr/httpie/issues/140
         r = http(
             'POST',
             httpbin('/post'),
@@ -792,6 +817,8 @@ class RequestBodyFromFilePathTest(BaseTestCase):
         self.assertIn('"Content-Type": "text/plain"', r)
 
     def test_request_body_from_file_by_path_with_explicit_content_type(self):
+        # FIXME: *sometimes* fails on py33, the content-type is form.
+        # https://github.com/jkbr/httpie/issues/140
         r = http(
             'POST',
             httpbin('/post'),
@@ -845,8 +872,8 @@ class AuthTest(BaseTestCase):
             httpbin('/digest-auth/auth/user/password')
         )
         self.assertIn(OK, r)
-        self.assertIn('"authenticated": true', r)
+        self.assertIn(r'"authenticated": true', r)
-        self.assertIn('"user": "user"', r)
+        self.assertIn(r'"user": "user"', r)
 
     def test_password_prompt(self):
 
@@ -1195,71 +1222,79 @@ class ArgumentParserTestCase(unittest.TestCase):
         self.parser = input.Parser()
 
     def test_guess_when_method_set_and_valid(self):
-        args = argparse.Namespace()
+        self.parser.args = argparse.Namespace()
-        args.method = 'GET'
+        self.parser.args.method = 'GET'
-        args.url = 'http://example.com/'
+        self.parser.args.url = 'http://example.com/'
-        args.items = []
+        self.parser.args.items = []
 
-        self.parser._guess_method(args, TestEnvironment())
+        self.parser.env = TestEnvironment()
 
-        self.assertEqual(args.method, 'GET')
+        self.parser._guess_method()
-        self.assertEqual(args.url, 'http://example.com/')
+
-        self.assertEqual(args.items, [])
+        self.assertEqual(self.parser.args.method, 'GET')
+        self.assertEqual(self.parser.args.url, 'http://example.com/')
+        self.assertEqual(self.parser.args.items, [])
 
     def test_guess_when_method_not_set(self):
-        args = argparse.Namespace()
-        args.method = None
-        args.url = 'http://example.com/'
-        args.items = []
 
-        self.parser._guess_method(args, TestEnvironment())
+        self.parser.args = argparse.Namespace()
+        self.parser.args.method = None
+        self.parser.args.url = 'http://example.com/'
+        self.parser.args.items = []
+        self.parser.env = TestEnvironment()
 
-        self.assertEqual(args.method, 'GET')
+        self.parser._guess_method()
-        self.assertEqual(args.url, 'http://example.com/')
+
-        self.assertEqual(args.items, [])
+        self.assertEqual(self.parser.args.method, 'GET')
+        self.assertEqual(self.parser.args.url, 'http://example.com/')
+        self.assertEqual(self.parser.args.items, [])
 
     def test_guess_when_method_set_but_invalid_and_data_field(self):
-        args = argparse.Namespace()
+        self.parser.args = argparse.Namespace()
-        args.method = 'http://example.com/'
+        self.parser.args.method = 'http://example.com/'
-        args.url = 'data=field'
+        self.parser.args.url = 'data=field'
-        args.items = []
+        self.parser.args.items = []
+        self.parser.env = TestEnvironment()
+        self.parser._guess_method()
 
-        self.parser._guess_method(args, TestEnvironment())
+        self.assertEqual(self.parser.args.method, 'POST')
-
+        self.assertEqual(self.parser.args.url, 'http://example.com/')
-        self.assertEqual(args.method, 'POST')
-        self.assertEqual(args.url, 'http://example.com/')
         self.assertEqual(
-            args.items,
+            self.parser.args.items,
             [input.KeyValue(
                 key='data', value='field', sep='=', orig='data=field')])
 
     def test_guess_when_method_set_but_invalid_and_header_field(self):
-        args = argparse.Namespace()
+        self.parser.args = argparse.Namespace()
-        args.method = 'http://example.com/'
+        self.parser.args.method = 'http://example.com/'
-        args.url = 'test:header'
+        self.parser.args.url = 'test:header'
-        args.items = []
+        self.parser.args.items = []
 
-        self.parser._guess_method(args, TestEnvironment())
+        self.parser.env = TestEnvironment()
 
-        self.assertEqual(args.method, 'GET')
+        self.parser._guess_method()
-        self.assertEqual(args.url, 'http://example.com/')
+
+        self.assertEqual(self.parser.args.method, 'GET')
+        self.assertEqual(self.parser.args.url, 'http://example.com/')
         self.assertEqual(
-            args.items,
+            self.parser.args.items,
             [input.KeyValue(
                 key='test', value='header', sep=':', orig='test:header')])
 
     def test_guess_when_method_set_but_invalid_and_item_exists(self):
-        args = argparse.Namespace()
+        self.parser.args = argparse.Namespace()
-        args.method = 'http://example.com/'
+        self.parser.args.method = 'http://example.com/'
-        args.url = 'new_item=a'
+        self.parser.args.url = 'new_item=a'
-        args.items = [
+        self.parser.args.items = [
             input.KeyValue(
                 key='old_item', value='b', sep='=', orig='old_item=b')
         ]
 
-        self.parser._guess_method(args, TestEnvironment())
+        self.parser.env = TestEnvironment()
 
-        self.assertEqual(args.items, [
+        self.parser._guess_method()
+
+        self.assertEqual(self.parser.args.items, [
             input.KeyValue(
                 key='new_item', value='a', sep='=', orig='new_item=a'),
             input.KeyValue(
@@ -1410,5 +1445,139 @@ class SessionTest(BaseTestCase):
         self.assertDictEqual(r1.json, r3.json)
 
 
+class DownloadUtilsTest(BaseTestCase):
+
+    def test_Content_Range_parsing(self):
+
+        parse = parse_content_range
+
+        self.assertEqual(parse('bytes 100-199/200', 100), 200)
+        self.assertEqual(parse('bytes 100-199/*', 100), 200)
+
+        # missing
+        self.assertRaises(ContentRangeError, parse, None, 100)
+
+        # syntax error
+        self.assertRaises(ContentRangeError, parse, 'beers 100-199/*', 100)
+
+        # unexpected range
+        self.assertRaises(ContentRangeError, parse, 'bytes 100-199/*', 99)
+
+        # invalid instance-length
+        self.assertRaises(ContentRangeError, parse, 'bytes 100-199/199', 100)
+
+        # invalid byte-range-resp-spec
+        self.assertRaises(ContentRangeError, parse, 'bytes 100-99/199', 100)
+
+        # invalid byte-range-resp-spec
+        self.assertRaises(ContentRangeError, parse, 'bytes 100-100/*', 100)
+
+    def test_Content_Disposition_parsing(self):
+        parse = filename_from_content_disposition
+        self.assertEqual(
+            parse('attachment; filename=hello-WORLD_123.txt'),
+            'hello-WORLD_123.txt'
+        )
+        self.assertEqual(
+            parse('attachment; filename=".hello-WORLD_123.txt"'),
+            'hello-WORLD_123.txt'
+        )
+        self.assertIsNone(parse('attachment; filename='))
+        self.assertIsNone(parse('attachment; filename=/etc/hosts'))
+        self.assertIsNone(parse('attachment; filename=hello@world'))
+
+    def test_filename_from_url(self):
+        self.assertEqual(filename_from_url(
+            url='http://example.org/foo',
+            content_type='text/plain'
+        ), 'foo.txt')
+        self.assertEqual(filename_from_url(
+            url='http://example.org/foo',
+            content_type='text/html; charset=utf8'
+        ), 'foo.html')
+        self.assertEqual(filename_from_url(
+            url='http://example.org/foo',
+            content_type=None
+        ), 'foo')
+        self.assertEqual(filename_from_url(
+            url='http://example.org/foo',
+            content_type='x-foo/bar'
+        ), 'foo')
+
+    def test_unique_filename(self):
+
+        def make_exists(unique_on_attempt=0):
+            # noinspection PyUnresolvedReferences,PyUnusedLocal
+            def exists(filename):
+                if exists.attempt == unique_on_attempt:
+                    return False
+                exists.attempt += 1
+                return True
+            exists.attempt = 0
+            return exists
+
+        self.assertEqual(
+            get_unique_filename('foo.bar', exists=make_exists()),
+            'foo.bar'
+        )
+        self.assertEqual(
+            get_unique_filename('foo.bar', exists=make_exists(1)),
+            'foo.bar-1'
+        )
+        self.assertEqual(
+            get_unique_filename('foo.bar', exists=make_exists(10)),
+            'foo.bar-10'
+        )
+
+
+class Response(object):
+
+    # noinspection PyDefaultArgument
+    def __init__(self, url, headers={}, status_code=200):
+        self.url = url
+        self.headers = CaseInsensitiveDict(headers)
+        self.status_code = status_code
+
+
+# noinspection PyTypeChecker
+class DownloadTest(BaseTestCase):
+    # TODO: more tests
+
+    def test_actual_download(self):
+        url = httpbin('/robots.txt')
+        body = urlopen(url).read().decode()
+        r = http(
+            '--download',
+            url,
+            env=TestEnvironment(
+                stdin_isatty=True,
+                stdout_isatty=False
+            )
+        )
+        self.assertIn('Downloading', r.stderr)
+        self.assertIn('[K', r.stderr)
+        self.assertIn('Done', r.stderr)
+        self.assertEqual(body, r)
+
+    def test_download_no_Content_Length(self):
+        download = Download(output_file=open(os.devnull, 'w'))
+        download.start(Response(url=httpbin('/')))
+        download._chunk_downloaded(b'12345')
+        download.finish()
+        self.assertFalse(download.interrupted)
+
+    def test_download_interrupted(self):
+        download = Download(
+            output_file=open(os.devnull, 'w')
+        )
+        download.start(Response(
+            url=httpbin('/'),
+            headers={'Content-Length': 5}
+        ))
+        download._chunk_downloaded(b'1234')
+        download.finish()
+        self.assertTrue(download.interrupted)
+
+
 if __name__ == '__main__':
     unittest.main()

tox.ini

@@ -11,9 +11,3 @@ commands = {envpython} setup.py test
 
 [testenv:py26]
 deps = argparse
-
-[testenv:py30]
-deps = argparse
-
-[testenv:py31]
-deps = argparse