1.0.3

When ``--download`` without ``--output`` results in a redirect,
now only the initial URL is considered, not the final one.

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: https://paypal.me/roztocil

CHANGELOG.rst

@@ -6,10 +6,32 @@ This document records all notable changes to `HTTPie <http://httpie.org>`_.
 This project adheres to `Semantic Versioning <http://semver.org/>`_.
 
 
-`1.0.3-dev`_ (unreleased)
+`1.0.3`_ (2019-08-26)
 -------------------------
 
-* No changes yet.
+* Fixed CVE-2019-10751 — the way the output filename is generated for
+  ``--download`` requests without ``--output`` resulting in a redirect has
+  been changed to only consider the initial URL as the base for the generated
+  filename, and not the final one. This fixes a potential security issue under
+  the following scenario:
+
+  1. A ``--download`` request with no explicit ``--output`` is made (e.g.,
+     ``$ http -d example.org/file.txt``), instructing httpie to
+     `generate the output filename <https://httpie.org/doc#downloaded-file-name>`_
+     from the ``Content-Disposition`` response, or from the URL if the header
+     is not provided.
+  2. The server handling the request has been modified by an attacker and
+     instead of the expected response the URL returns a redirect to another
+     URL, e.g., ``attacker.example.org/.bash_profile``, whose response does
+     not provide  a ``Content-Disposition`` header (i.e., the base for the
+     generated filename becomes ``.bash_profile`` instead of ``file.txt``).
+  3. Your current directory doesn’t already contain ``.bash_profile``
+     (i.e., no unique suffix is added to the generated filename).
+  4. You don’t notice the potentially unexpected output filename
+     as reported by httpie in the console output
+     (e.g., ``Downloading 100.00 B to ".bash_profile"``).
+
+  Reported by Raul Onitza and Giulio Comi.
 
 
 `1.0.2`_ (2018-11-14)
@@ -361,4 +383,4 @@ This project adheres to `Semantic Versioning <http://semver.org/>`_.
 .. _1.0.0: https://github.com/jakubroztocil/httpie/compare/0.9.9...1.0.0
 .. _1.0.1: https://github.com/jakubroztocil/httpie/compare/1.0.0...1.0.1
 .. _1.0.2: https://github.com/jakubroztocil/httpie/compare/1.0.1...1.0.2
-.. _1.0.3-dev: https://github.com/jakubroztocil/httpie/compare/1.0.2...master
+.. _1.0.3: https://github.com/jakubroztocil/httpie/compare/1.0.2...1.0.3

LICENSE

@@ -1,4 +1,4 @@
-Copyright © 2012-2017 Jakub Roztocil <jakub@roztocil.co>
+Copyright © 2012-2019 Jakub Roztocil <jakub@roztocil.co>
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:

Makefile

@@ -93,9 +93,8 @@ publish-no-test:
 	@echo $(TAG)Testing wheel build an installation$(END)
 	@echo "$(VERSION)"
 	@echo "$(VERSION)" | grep -q "dev" && echo '!!!Not publishing dev version!!!' && exit 1 || echo ok
-	python setup.py register
-	python setup.py sdist upload
-	python setup.py bdist_wheel upload
+	python setup.py sdist bdist_wheel
+	twine upload dist/*
 	@echo
 
 
@@ -140,9 +139,14 @@ pdf:
 
 
 ###############################################################################
-# Utils
+# Homebrew
 ###############################################################################
 
+brew-deps:
+	extras/brew-deps.py
 
-homebrew-formula-vars:
-	extras/get-homebrew-formula-vars.py
+brew-test:
+	- brew uninstall httpie
+	brew install --build-from-source ./extras/httpie.rb
+	brew test httpie
+	brew audit --strict httpie

README.rst

@@ -11,6 +11,12 @@ generally interacting with HTTP servers.
 
 .. class:: no-web
 
+    .. image:: https://raw.githubusercontent.com/jakubroztocil/httpie/master/httpie.gif
+        :alt: HTTPie in action
+        :width: 100%
+        :align: center
+
+
     .. image:: https://raw.githubusercontent.com/jakubroztocil/httpie/master/httpie.png
         :alt: HTTPie compared to cURL
         :width: 100%
@@ -1106,6 +1112,13 @@ You can use ``echo`` for simple data:
     $ echo '{"name": "John"}' | http PATCH example.com/person/1 X-API-Token:123
 
 
+You can also use a Bash *here string*:
+
+.. code-block:: bash
+
+    $ http example.com/ <<<'{"name": "John"}'
+
+
 You can even pipe web services together using HTTPie:
 
 .. code-block:: bash
@@ -1300,13 +1313,25 @@ is being saved to a file.
     Done. 251.30 kB in 2.73862s (91.76 kB/s)
 
 
-Downloaded file name
+Downloaded filename
 --------------------
 
-If not provided via ``--output, -o``, the output filename will be determined
-from ``Content-Disposition`` (if available), or from the URL and
-``Content-Type``. If the guessed filename already exists, HTTPie adds a unique
-suffix to it.
+There are three mutually exclusive ways through which HTTPie determines
+the output filename (with decreasing priority):
+
+1. You can explicitly provide it via ``--output, -o``.
+   The file gets overwritten if it already exists
+   (or appended to with ``--continue, -c``).
+2. The server may specify the filename in the optional ``Content-Disposition``
+   response header. Any leading dots are stripped from a server-provided filename.
+3. The last resort HTTPie uses is to generate the filename from a combination
+   of the request URL and the response ``Content-Type``.
+   The initial URL is always used as the basis for
+   the generated filename — even if there has been one or more redirects.
+
+
+To prevent data loss by overwriting, HTTPie adds a unique numerical suffix to the
+filename when necessary (unless specified with ``--output, -o``).
 
 
 Piping while downloading
@@ -1553,7 +1578,7 @@ Best practices
 --------------
 
 The default behaviour of automatically reading ``stdin`` is typically not
-desirable during non-interactive invocations. You most likely want
+desirable during non-interactive invocations. You most likely want to
 use the ``--ignore-stdin`` option to disable it.
 
 It is a common gotcha that without this option HTTPie seemingly hangs.
@@ -1685,7 +1710,9 @@ See `CHANGELOG <https://github.com/jakubroztocil/httpie/blob/master/CHANGELOG.rs
 Artwork
 -------
 
-See `claudiatd/httpie-artwork`_
+* `Logo <https://github.com/claudiatd/httpie-artwork>`_ by `Cláudia Delgado <https://github.com/claudiatd>`_.
+* `Animation <https://raw.githubusercontent.com/jakubroztocil/httpie/master/httpie.gif>`_ by `Allen Smith <https://github.com/loranallensmith>`_ of GitHub.
+
 
 
 Licence
@@ -1707,7 +1734,6 @@ have contributed.
 .. _these fine people: https://github.com/jakubroztocil/httpie/contributors
 .. _Jakub Roztocil: https://roztocil.co
 .. _@jakubroztocil: https://twitter.com/jakubroztocil
-.. _claudiatd/httpie-artwork: https://github.com/claudiatd/httpie-artwork
 
 
 .. |pypi| image:: https://img.shields.io/pypi/v/httpie.svg?style=flat-square&label=latest%20stable%20version

extras/brew-deps.py

@@ -1,9 +1,11 @@
 #!/usr/bin/env python3
 """
-Generate URLs and file hashes to be included in the Homebrew formula
-after a new release of HTTPie has been published on PyPi.
+Generate Ruby code with URLs and file hashes for packages from PyPi
+(i.e., httpie itself as well as its dependencies) to be included
+in the Homebrew formula after a new release of HTTPie has been published
+on PyPi.
 
-https://github.com/Homebrew/homebrew-core/blob/master/Formula/httpie.rb
+<https://github.com/Homebrew/homebrew-core/blob/master/Formula/httpie.rb>
 
 """
 import hashlib
@@ -12,7 +14,7 @@ import requests
 
 PACKAGES = [
     'httpie',
-    'pygments',
+    'Pygments',
     'requests',
     'certifi',
     'urllib3',

extras/httpie.rb

@@ -9,43 +9,42 @@ class Httpie < Formula
 
   desc "User-friendly cURL replacement (command-line HTTP client)"
   homepage "https://httpie.org/"
-  url "https://files.pythonhosted.org/packages/44/ee/7177b743400d7f82a69bf30cb3c24ea4bb1f4aea68878bc540f732bf4940/httpie-1.0.0.tar.gz"
-  sha256 "1650342d2eca2622092196bf106ab8f68ea2dbb2ed265d37191185618e159a25"
+  url "https://files.pythonhosted.org/packages/09/8d/581ef7bd9a09dc30b621638a4fa805a2073bbfb45fa06ed37f998f172419/httpie-1.0.2.tar.gz"
+  sha256 "fc676c85febdf3d80abc1ef6fa71ec3764d8b838806a7ae4e55e5e5aa014a2ab"
   head "https://github.com/jakubroztocil/httpie.git"
 
   bottle do
     cellar :any_skip_relocation
-    sha256 "7e9db255e324dd63b66106ca62ed7e4e81f6634c624dec3ff49c293aba1072a6" => :mojave
-    sha256 "437504a11416284b17d3a801c267d0fd5e15416f38cff3abf7ed99b096b4828a" => :high_sierra
-    sha256 "10b25fc787076719b1f1f9c242c5e9d872ebd1c7a6d83e6f1af983a17cd8ca55" => :sierra
-    sha256 "1bd35480d1ef401bdad9c322e7c1624aefc9b5056530ab990e327d0bc397e4fb" => :el_capitan
+    sha256 "158258be68ac93de13860be2bef02da6fd8b68aa24b2e6609bcff1ec3f93e7a0" => :mojave
+    sha256 "54352116b6fa2c3bd65f26136fdcb57986dbff8a52de5febf7aea59c126d29e1" => :high_sierra
+    sha256 "9cce71768fe388808e11b26d651b44a6b54219f5406845b4273b5099f5c1f76f" => :sierra
   end
 
-  depends_on "python" ["3.6.5_1"]
+  depends_on "python"
 
-  resource "pygments" do
-    url "https://files.pythonhosted.org/packages/71/2a/2e4e77803a8bd6408a2903340ac498cb0a2181811af7c9ec92cb70b0308a/Pygments-2.2.0.tar.gz"
-    sha256 "dbae1046def0efb574852fab9e90209b23f556367b5a320c0bcb871c77c3e8cc"
+ resource "Pygments" do
+    url "https://files.pythonhosted.org/packages/64/69/413708eaf3a64a6abb8972644e0f20891a55e621c6759e2c3f3891e05d63/Pygments-2.3.1.tar.gz"
+    sha256 "5ffada19f6203563680669ee7f53b64dabbeb100eb51b61996085e99c03b284a"
   end
 
   resource "requests" do
-    url "https://files.pythonhosted.org/packages/97/10/92d25b93e9c266c94b76a5548f020f3f1dd0eb40649cb1993532c0af8f4c/requests-2.20.0.tar.gz"
-    sha256 "99dcfdaaeb17caf6e526f32b6a7b780461512ab3f1d992187801694cba42770c"
+    url "https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz"
+    sha256 "502a824f31acdacb3a35b6690b5fbf0bc41d63a24a45c4004352b0242707598e"
   end
 
   resource "certifi" do
-    url "https://files.pythonhosted.org/packages/41/b6/4f0cefba47656583217acd6cd797bc2db1fede0d53090fdc28ad2c8e0716/certifi-2018.10.15.tar.gz"
-    sha256 "6d58c986d22b038c8c0df30d639f23a3e6d172a05c3583e766f4c0b785c0986a"
+    url "https://files.pythonhosted.org/packages/55/54/3ce77783acba5979ce16674fc98b1920d00b01d337cfaaf5db22543505ed/certifi-2018.11.29.tar.gz"
+    sha256 "47f9c83ef4c0c621eaef743f133f09fa8a74a9b75f037e8624f83bd1b6626cb7"
   end
 
   resource "urllib3" do
-    url "https://files.pythonhosted.org/packages/a5/74/05ffd00b4b5c08306939c485869f5dc40cbc27357195b0a98b18e4c48893/urllib3-1.24.tar.gz"
-    sha256 "41c3db2fc01e5b907288010dec72f9d0a74e37d6994e6eb56849f59fea2265ae"
+    url "https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz"
+    sha256 "de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22"
   end
 
   resource "idna" do
-    url "https://files.pythonhosted.org/packages/65/c4/80f97e9c9628f3cac9b98bfca0402ede54e0563b56482e3e6e45c43c4935/idna-2.7.tar.gz"
-    sha256 "684a38a6f903c1d71d6d5fac066b58d7768af4de2b832e426ec79c30daa94a16"
+    url "https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz"
+    sha256 "c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407"
   end
 
   resource "chardet" do

httpie/__init__.py

@@ -2,7 +2,7 @@
 HTTPie - a CLI, cURL-like tool for humans.
 
 """
-__version__ = '1.0.2'
+__version__ = '1.0.3'
 __author__ = 'Jakub Roztocil'
 __licence__ = 'BSD'
 

httpie/downloads.py

@@ -224,13 +224,13 @@ class Downloader(object):
                 request_headers['Range'] = 'bytes=%d-' % bytes_have
                 self._resumed_from = bytes_have
 
-    def start(self, response):
+    def start(self, final_response):
         """
         Initiate and return a stream for `response` body  with progress
         callback attached. Can be called only once.
 
-        :param response: Initiated response object with headers already fetched
-        :type response: requests.models.Response
+        :param final_response: Initiated response object with headers already fetched
+        :type final_response: requests.models.Response
 
         :return: RawStream, output_file
 
@@ -240,14 +240,18 @@ class Downloader(object):
         # FIXME: some servers still might sent Content-Encoding: gzip
         # <https://github.com/jakubroztocil/httpie/issues/423>
         try:
-            total_size = int(response.headers['Content-Length'])
+            total_size = int(final_response.headers['Content-Length'])
         except (KeyError, ValueError, TypeError):
             total_size = None
 
-        if self._output_file:
-            if self._resume and response.status_code == PARTIAL_CONTENT:
+        if not self._output_file:
+            self._output_file = self._get_output_file_from_response(
+                final_response)
+        else:
+            # `--output, -o` provided
+            if self._resume and final_response.status_code == PARTIAL_CONTENT:
                 total_size = parse_content_range(
-                    response.headers.get('Content-Range'),
+                    final_response.headers.get('Content-Range'),
                     self._resumed_from
                 )
 
@@ -258,19 +262,6 @@ class Downloader(object):
                     self._output_file.truncate()
                 except IOError:
                     pass  # stdout
-        else:
-            # TODO: Should the filename be taken from response.history[0].url?
-            # Output file not specified. Pick a name that doesn't exist yet.
-            filename = None
-            if 'Content-Disposition' in response.headers:
-                filename = filename_from_content_disposition(
-                    response.headers['Content-Disposition'])
-            if not filename:
-                filename = filename_from_url(
-                    url=response.url,
-                    content_type=response.headers.get('Content-Type'),
-                )
-            self._output_file = open(get_unique_filename(filename), mode='a+b')
 
         self.status.started(
             resumed_from=self._resumed_from,
@@ -278,7 +269,7 @@ class Downloader(object):
         )
 
         stream = RawStream(
-            msg=HTTPResponse(response),
+            msg=HTTPResponse(final_response),
             with_headers=False,
             with_body=True,
             on_body_chunk_downloaded=self.chunk_downloaded,
@@ -324,6 +315,25 @@ class Downloader(object):
         """
         self.status.chunk_downloaded(len(chunk))
 
+    @staticmethod
+    def _get_output_file_from_response(final_response):
+        # Output file not specified. Pick a name that doesn't exist yet.
+        filename = None
+        if 'Content-Disposition' in final_response.headers:
+            filename = filename_from_content_disposition(
+                final_response.headers['Content-Disposition'])
+        if not filename:
+            initial_response = (
+                final_response.history[0] if final_response.history
+                else final_response
+            )
+            filename = filename_from_url(
+                url=initial_response.url,
+                content_type=final_response.headers.get('Content-Type'),
+            )
+        unique_filename = get_unique_filename(filename)
+        return open(unique_filename, mode='a+b')
+
 
 class Status(object):
     """Holds details about the downland status."""

httpie/input.py

@@ -752,7 +752,7 @@ def parse_items(items,
 
 def readable_file_arg(filename):
     try:
-        open(filename, 'rb')
+        with open(filename, 'rb'):
+            return filename
     except IOError as ex:
         raise ArgumentTypeError('%s: %s' % (filename, ex.args[1]))
-    return filename

requirements-dev.txt

@@ -6,3 +6,4 @@ pytest-httpbin>=0.0.6
 docutils
 wheel
 pycodestyle
+twine

setup.py

@@ -35,8 +35,8 @@ tests_require = [
 
 
 install_requires = [
-    'requests>=2.18.4',
-    'Pygments>=2.1.3'
+    'requests>=2.21.0',
+    'Pygments>=2.3.1'
 ]
 
 

tests/test_downloads.py

@@ -1,4 +1,5 @@
 import os
+import tempfile
 import time
 
 import pytest
@@ -22,6 +23,7 @@ class Response(object):
 
 
 class TestDownloadUtils:
+
     def test_Content_Range_parsing(self):
         parse = parse_content_range
 
@@ -131,35 +133,50 @@ class TestDownloads:
         assert body == r
 
     def test_download_with_Content_Length(self, httpbin_both):
-        devnull = open(os.devnull, 'w')
-        downloader = Downloader(output_file=devnull, progress_file=devnull)
-        downloader.start(Response(
-            url=httpbin_both.url + '/',
-            headers={'Content-Length': 10}
-        ))
-        time.sleep(1.1)
-        downloader.chunk_downloaded(b'12345')
-        time.sleep(1.1)
-        downloader.chunk_downloaded(b'12345')
-        downloader.finish()
-        assert not downloader.interrupted
+        with open(os.devnull, 'w') as devnull:
+            downloader = Downloader(output_file=devnull, progress_file=devnull)
+            downloader.start(Response(
+                url=httpbin_both.url + '/',
+                headers={'Content-Length': 10}
+            ))
+            time.sleep(1.1)
+            downloader.chunk_downloaded(b'12345')
+            time.sleep(1.1)
+            downloader.chunk_downloaded(b'12345')
+            downloader.finish()
+            assert not downloader.interrupted
+            downloader._progress_reporter.join()
 
     def test_download_no_Content_Length(self, httpbin_both):
-        devnull = open(os.devnull, 'w')
-        downloader = Downloader(output_file=devnull, progress_file=devnull)
-        downloader.start(Response(url=httpbin_both.url + '/'))
-        time.sleep(1.1)
-        downloader.chunk_downloaded(b'12345')
-        downloader.finish()
-        assert not downloader.interrupted
+        with open(os.devnull, 'w') as devnull:
+            downloader = Downloader(output_file=devnull, progress_file=devnull)
+            downloader.start(Response(url=httpbin_both.url + '/'))
+            time.sleep(1.1)
+            downloader.chunk_downloaded(b'12345')
+            downloader.finish()
+            assert not downloader.interrupted
+            downloader._progress_reporter.join()
 
     def test_download_interrupted(self, httpbin_both):
-        devnull = open(os.devnull, 'w')
-        downloader = Downloader(output_file=devnull, progress_file=devnull)
-        downloader.start(Response(
-            url=httpbin_both.url + '/',
-            headers={'Content-Length': 5}
-        ))
-        downloader.chunk_downloaded(b'1234')
-        downloader.finish()
-        assert downloader.interrupted
+        with open(os.devnull, 'w') as devnull:
+            downloader = Downloader(output_file=devnull, progress_file=devnull)
+            downloader.start(Response(
+                url=httpbin_both.url + '/',
+                headers={'Content-Length': 5}
+            ))
+            downloader.chunk_downloaded(b'1234')
+            downloader.finish()
+            assert downloader.interrupted
+            downloader._progress_reporter.join()
+
+    def test_download_with_redirect_original_url_used_for_filename(self, httpbin):
+        # Redirect from `/redirect/1` to `/get`.
+        expected_filename = '1.json'
+        orig_cwd = os.getcwd()
+        os.chdir(tempfile.mkdtemp(prefix='httpie_download_test_'))
+        try:
+            assert os.listdir('.') == []
+            http('--download', httpbin.url + '/redirect/1')
+            assert os.listdir('.') == [expected_filename]
+        finally:
+            os.chdir(orig_cwd)

tests/utils.py

@@ -62,6 +62,8 @@ class MockEnvironment(Environment):
         return super(MockEnvironment, self).config
 
     def cleanup(self):
+        self.stdout.close()
+        self.stderr.close()
         if self._delete_config_dir:
             assert self.config_dir.startswith(tempfile.gettempdir())
             from shutil import rmtree