Add nested JSON syntax to the HTTPie DSL (#1224)

* Add support for nested JSON syntax (#1169)

Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Jakub Roztocil <jakub@roztocil.co>

* minor improvements

* unpack top level lists

* Write more docs

* doc style changes

* fix double quotes

Co-authored-by: Mickaël Schoentgen <contact@tiger-222.fr>
Co-authored-by: Jakub Roztocil <jakub@roztocil.co>

CHANGELOG.md

@@ -6,6 +6,7 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 ## [3.0.0.dev0](https://github.com/httpie/httpie/compare/2.6.0...master) (unreleased)
 
 - Improved startup time by 40%. ([#1211](https://github.com/httpie/httpie/pull/1211))
+- Added support for nested JSON syntax. ([#1169](https://github.com/httpie/httpie/issues/1169))
 - Added `httpie plugins` interface for plugin management. ([#566](https://github.com/httpie/httpie/issues/566))
 - Added support for Bearer authentication via `--auth-type=bearer` ([#1215](https://github.com/httpie/httpie/issues/1215)).
 - Added support for quick conversions of pasted URLs into HTTPie calls by adding a space after the protocol name (`$ https ://pie.dev` → `https://pie.dev`). ([#1195](https://github.com/httpie/httpie/issues/1195))

docs/README.md

@@ -714,6 +714,246 @@ The `:=`/`:=@` syntax is JSON-specific. You can switch your request to `--form`
 ```
 
 As well as a nested one,
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  result[type]=success
+```
+
+```json
+{
+    "result": {"type": "success"}
+}
+```
+
+Or even multiple levels of nesting.
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  result[status][type]=ok
+```
+
+```json
+{
+    "result": {
+        "status": {
+            "type": "ok"
+        }
+    }
+}
+```
+
+The declaration also supports creating arrays; which can be either done by simply
+assigning the same path multiple times
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  ids:=1 ids:=2
+```
+
+```json
+{
+    "ids": [
+        1,
+        2
+    ]
+}
+```
+
+Or using the append suffix `[]`, which would create an array and append the items to the
+end of it.
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  ids[]:=1
+```
+
+```json
+{
+    "ids": [
+        1,
+        2
+    ]
+}
+```
+
+You can also use indexes to set items on an array,
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  items[0]=terminal items[1]=desktop
+```
+
+```json
+{
+    "items": [
+        "terminal",
+        "desktop"
+    ]
+}
+```
+
+If you don't set value for the indexes between, then those will be nullified.
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  items[1]=terminal items[3]=desktop
+```
+
+```json
+{
+    "items": [
+        null,
+        "terminal",
+        null,
+        "desktop"
+    ]
+}
+```
+
+It is permitted to mix index-access with append actions (`[]`), but be aware that appends will not fill
+the voids but instead they will append after the last item.
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  items[1]=terminal items[3]=desktop items[]=web
+```
+
+```json
+{
+    "items": [
+        null,
+        "terminal",
+        null,
+        "desktop",
+        "web"
+    ]
+}
+```
+
+If you need to send a top-level list (without any object that is encapsulating it), use the append operator (`[]`) without
+any keys.
+
+```bash
+$ http --offline --print=B pie.dev/post \
+  []:=1 []:=2 []:=3
+```
+
+```json
+[
+    1,
+    2,
+    3
+]
+```
+
+Here is a slightly unified example
+
+```bash
+$ http --offline --print=B pie.dev/post name=python version:=3 \
+  date[year]:=2021 date[month]=December \
+  systems=Linux systems=Mac systems=Windows \
+  people[known_ids][1]=1000 people[known_ids][5]=5000
+```
+
+```json
+{
+    "date": {
+        "month": "December",
+        "year": 2021
+    },
+    "name": "python",
+    "people": {
+        "known_ids": [
+            null,
+            "1000",
+            null,
+            null,
+            null,
+            "5000"
+        ]
+    },
+    "systems": [
+        "Linux",
+        "Mac",
+        "Windows"
+    ],
+    "version": 3
+}
+```
+
+And here is an even more comprehensive example to show all the features.
+
+```bash
+$ http PUT pie.dev/put \
+    'object=scalar' \                           # Object — blank key
+    'object[0]=array 1' \                       # Object — "0" key
+    'object[key]=key key' \                     # Object — "key" key
+    'array:=1' \                                # Array — first item
+    'array:=2' \                                # Array — second item
+    'array[]:=3' \                              # Array — append (third item)
+    'wow[such][deep][3][much][power][!]=Amaze'  # Nested object
+```
+
+```http
+PUT /person/1 HTTP/1.1
+Accept: application/json, */*;q=0.5
+Content-Type: application/json
+Host: pie.dev
+
+```
+
+### Raw and complex JSON
+
+Please note that with the [request items](#request-items) data field syntax, commands can quickly become unwieldy when sending complex structures.
+In such cases, it’s better to pass the full raw JSON data via [raw request body](#raw-request-body), for example:
+
+```bash
+$ echo -n '{"hello": "world"}' | http POST pie.dev/post
+```
+
+```bash
+$ http --raw '{"hello": "world"}' POST pie.dev/post
+```
+
+```bash
+$ http POST pie.dev/post < files/data.json
+```
+
+## Forms
+
+Submitting forms is very similar to sending [JSON](#json) requests.
+Often the only difference is in adding the `--form, -f` option, which ensures that data fields are serialized as, and `Content-Type` is set to `application/x-www-form-urlencoded; charset=utf-8`.
+It is possible to make form data the implicit content type instead of JSON via the [config](#config) file.
+
+### Regular forms
+
+```bash
+$ http --form POST pie.dev/post name='John Smith'
+```
+
+```http
+POST /post HTTP/1.1
+Content-Type: application/x-www-form-urlencoded; charset=utf-8
+
+name=John+Smith
+```
+
+### File upload forms
+
+If one or more file fields is present, the serialization and content type is `multipart/form-data`:
+
+```bash
+$ http -f POST pie.dev/post name='John Smith' cv@~/files/data.xml
+```
+
+The request above is the same as if the following HTML form were submitted:
+
+```html
+<form enctype="multipart/form-data" method="post" action="http://example.com/jobs">
+    <input type="text" name="name" />
+    <input type="file" name="cv" />
+</form>
 ```
 
 Please note that `@` is used to simulate a file upload form field, whereas `=@` just embeds the file content as a regular text field value.
@@ -731,9 +971,6 @@ $ http --raw '{"hello": "world"}' POST pie.dev/post
 ```
 
 ```http
-
-If one or more file fields is present, the serialization and content type is `multipart/form-data`:
-
 POST / HTTP/1.1
 Content-Length: 129
 Content-Type: multipart/form-data; boundary=c31279ab254f40aeb06df32b433cbccb

httpie/cli/constants.py

@@ -46,10 +46,10 @@ SEPARATOR_GROUP_DATA_EMBED_ITEMS = frozenset({
     SEPARATOR_DATA_EMBED_RAW_JSON_FILE,
 })
 
-# Separators for raw JSON items
+# Separators for nested JSON items
-SEPARATOR_GROUP_RAW_JSON_ITEMS = frozenset([
+SEPARATOR_GROUP_NESTED_JSON_ITEMS = frozenset([
+    SEPARATOR_DATA_STRING,
     SEPARATOR_DATA_RAW_JSON,
-    SEPARATOR_DATA_EMBED_RAW_JSON_FILE,
 ])
 
 # Separators allowed in ITEM arguments

httpie/cli/definition.py

@@ -120,7 +120,7 @@ positional.add_argument(
 
     '=@' A data field like '=', but takes a file path and embeds its content:
 
-         essay=@Documents/essay.txt
+        essay=@Documents/essay.txt
 
     ':=@' A raw JSON field like ':=', but takes a file path and embeds its content:
 

httpie/cli/json_form.py

@@ -0,0 +1,150 @@
+"""
+Routines for JSON form syntax, used to support nested JSON request items.
+
+Highly inspired from the great jarg project <https://github.com/jdp/jarg/blob/master/jarg/jsonform.py>.
+"""
+import re
+import operator
+from typing import Optional
+
+
+def step(value: str, is_escaped: bool) -> str:
+    if is_escaped:
+        value = value.replace(r'\[', '[').replace(r'\]', ']')
+    return value
+
+
+def find_opening_bracket(
+    value: str,
+    search=re.compile(r'(?<!\\)\[').search
+) -> Optional[int]:
+    match = search(value)
+    if not match:
+        return None
+    return match.start()
+
+
+def find_closing_bracket(
+    value: str,
+    search=re.compile(r'(?<!\\)\]').search
+) -> Optional[int]:
+    match = search(value)
+    if not match:
+        return None
+    return match.start()
+
+
+def parse_path(path):
+    """
+    Parse a string as a JSON path.
+
+    An implementation of 'steps to parse a JSON encoding path'.
+    <https://www.w3.org/TR/html-json-forms/#dfn-steps-to-parse-a-json-encoding-path>
+
+    """
+    original = path
+    is_escaped = r'\[' in original
+
+    opening_bracket = find_opening_bracket(original)
+    last_step = [(step(path, is_escaped), {'last': True, 'type': 'object'})]
+    if opening_bracket is None:
+        return last_step
+
+    steps = [(step(original[:opening_bracket], is_escaped), {'type': 'object'})]
+    path = original[opening_bracket:]
+    while path:
+        if path.startswith('[]'):
+            steps[-1][1]['append'] = True
+            path = path[2:]
+            if path:
+                return last_step
+        elif path[0] == '[':
+            path = path[1:]
+            closing_bracket = find_closing_bracket(path)
+            if closing_bracket is None:
+                return last_step
+            key = path[:closing_bracket]
+            path = path[closing_bracket + 1:]
+            try:
+                steps.append((int(key), {'type': 'array'}))
+            except ValueError:
+                steps.append((key, {'type': 'object'}))
+        elif path[:2] == r'\[':
+            key = step(path[1:path.index(r'\]') + 2], is_escaped)
+            path = path[path.index(r'\]') + 2:]
+            steps.append((key, {'type': 'object'}))
+        else:
+            return last_step
+
+    for i in range(len(steps) - 1):
+        steps[i][1]['type'] = steps[i + 1][1]['type']
+    steps[-1][1]['last'] = True
+    return steps
+
+
+def set_value(context, step, current_value, entry_value):
+    """Apply a JSON value to a context object.
+
+    An implementation of 'steps to set a JSON encoding value'.
+    <https://www.w3.org/TR/html-json-forms/#dfn-steps-to-set-a-json-encoding-value>
+
+    """
+    key, flags = step
+    if flags.get('last', False):
+        if current_value is None:
+            if flags.get('append', False):
+                context[key] = [entry_value]
+            else:
+                if isinstance(context, list) and len(context) <= key:
+                    context.extend([None] * (key - len(context) + 1))
+                context[key] = entry_value
+        elif isinstance(current_value, list):
+            context[key].append(entry_value)
+        else:
+            context[key] = [current_value, entry_value]
+        return context
+
+    if current_value is None:
+        if flags.get('type') == 'array':
+            context[key] = []
+        else:
+            if isinstance(context, list) and len(context) <= key:
+                context.extend([None] * (key - len(context) + 1))
+            context[key] = {}
+        return context[key]
+    elif isinstance(current_value, dict):
+        return context[key]
+    elif isinstance(current_value, list):
+        if flags.get('type') == 'array':
+            return current_value
+
+        obj = {}
+        for i, item in enumerate(current_value):
+            if item is not None:
+                obj[i] = item
+        else:
+            context[key] = obj
+        return obj
+    else:
+        obj = {'': current_value}
+        context[key] = obj
+        return obj
+
+
+def interpret_json_form(pairs):
+    """The application/json form encoding algorithm.
+
+    <https://www.w3.org/TR/html-json-forms/#dfn-application-json-encoding-algorithm>
+
+    """
+    result = {}
+    for key, value in pairs:
+        steps = parse_path(key)
+        context = result
+        for step in steps:
+            try:
+                current_value = operator.getitem(context, step[0])
+            except LookupError:
+                current_value = None
+            context = set_value(context, step, current_value, value)
+    return result

httpie/cli/requestitems.py

@@ -5,7 +5,7 @@ from typing import Callable, Dict, IO, List, Optional, Tuple, Union
 from .argtypes import KeyValueArg
 from .constants import (
     SEPARATORS_GROUP_MULTIPART, SEPARATOR_DATA_EMBED_FILE_CONTENTS,
-    SEPARATOR_DATA_EMBED_RAW_JSON_FILE,
+    SEPARATOR_DATA_EMBED_RAW_JSON_FILE, SEPARATOR_GROUP_NESTED_JSON_ITEMS,
     SEPARATOR_DATA_RAW_JSON, SEPARATOR_DATA_STRING, SEPARATOR_FILE_UPLOAD,
     SEPARATOR_FILE_UPLOAD_TYPE, SEPARATOR_HEADER, SEPARATOR_HEADER_EMPTY,
     SEPARATOR_QUERY_PARAM, SEPARATOR_QUERY_EMBED_FILE, RequestType
@@ -16,7 +16,8 @@ from .dicts import (
     RequestQueryParamsDict,
 )
 from .exceptions import ParseError
-from ..utils import get_content_type, load_json_preserve_order_and_dupe_keys
+from .json_form import interpret_json_form
+from ..utils import get_content_type, load_json_preserve_order_and_dupe_keys, split
 
 
 class RequestItems:
@@ -67,6 +68,10 @@ class RequestItems:
                 process_data_embed_file_contents_arg,
                 instance.data,
             ),
+            SEPARATOR_GROUP_NESTED_JSON_ITEMS: (
+                process_data_nested_json_embed_args,
+                instance.data,
+            ),
             SEPARATOR_DATA_RAW_JSON: (
                 json_only(instance, process_data_raw_json_embed_arg),
                 instance.data,
@@ -77,6 +82,21 @@ class RequestItems:
             ),
         }
 
+        if instance.is_json:
+            json_item_args, request_item_args = split(
+                request_item_args,
+                lambda arg: arg.sep in SEPARATOR_GROUP_NESTED_JSON_ITEMS
+            )
+            if json_item_args:
+                pairs = [
+                    (arg.key, rules[arg.sep][0](arg))
+                    for arg in json_item_args
+                ]
+                processor_func, target_dict = rules[SEPARATOR_GROUP_NESTED_JSON_ITEMS]
+                value = processor_func(pairs)
+                target_dict.update(value)
+
+        # Then handle all other items.
         for arg in request_item_args:
             processor_func, target_dict = rules[arg.sep]
             value = processor_func(arg)
@@ -172,6 +192,10 @@ def process_data_raw_json_embed_arg(arg: KeyValueArg) -> JSONType:
     return value
 
 
+def process_data_nested_json_embed_args(pairs) -> Dict[str, JSONType]:
+    return interpret_json_form(pairs)
+
+
 def load_text_file(item: KeyValueArg) -> str:
     path = item.value
     try:

httpie/client.py

@@ -287,6 +287,13 @@ def make_request_kwargs(
     data = args.data
     auto_json = data and not args.form
     if (args.json or auto_json) and isinstance(data, dict):
+        # Propagate the top-level list if there is only one
+        # item in the object, with an en empty key.
+        if len(data) == 1:
+            [(key, value)] = data.items()
+            if key == '' and isinstance(value, list):
+                data = value
+
         if data:
             data = json.dumps(data)
         else:

httpie/utils.py

@@ -9,13 +9,14 @@ from collections import OrderedDict
 from http.cookiejar import parse_ns_headers
 from pathlib import Path
 from pprint import pformat
-from typing import Any, List, Optional, Tuple
+from typing import Any, List, Optional, Tuple, Callable, Iterable, TypeVar
 
 import requests.auth
 
 RE_COOKIE_SPLIT = re.compile(r', (?=[^ ;]+=)')
 Item = Tuple[str, Any]
 Items = List[Item]
+T = TypeVar("T")
 
 
 class JsonDictPreservingDuplicateKeys(OrderedDict):
@@ -218,3 +219,13 @@ def as_site(path: Path) -> Path:
         vars={'base': str(path)}
     )
     return Path(site_packages_path)
+
+
+def split(iterable: Iterable[T], key: Callable[[T], bool]) -> Tuple[List[T], List[T]]:
+    left, right = [], []
+    for item in iterable:
+        if key(item):
+            left.append(item)
+        else:
+            right.append(item)
+    return left, right

tests/test_cli.py

@@ -79,6 +79,8 @@ class TestItemParsing:
             self.key_value_arg('Empty-Header;'),
             self.key_value_arg('list:=["a", 1, {}, false]'),
             self.key_value_arg('obj:={"a": "b"}'),
+            self.key_value_arg(r'nested\[2\][a][]=1'),
+            self.key_value_arg('nested[2][a][]:=1'),
             self.key_value_arg('ed='),
             self.key_value_arg('bool:=true'),
             self.key_value_arg('file@' + FILE_PATH_ARG),
@@ -105,7 +107,9 @@ class TestItemParsing:
             'ed': '',
             'string': 'value',
             'bool': True,
-            'list': ['a', 1, {}, False],
+            'list': ['a', 1, load_json_preserve_order_and_dupe_keys('{}'), False],
+            'nested[2]': {'a': ['1']},
+            'nested': [None, None, {'a': [1]}],
             'obj': load_json_preserve_order_and_dupe_keys('{"a": "b"}'),
             'string-embed': FILE_CONTENT
         }

tests/test_json.py

@@ -152,3 +152,60 @@ def test_complex_json_arguments_with_non_json(httpbin, request_type, value):
         )
 
     cm.match('Can\'t use complex JSON value types')
+
+
+@pytest.mark.parametrize('input_json, expected_json', [
+    # Examples taken from https://www.w3.org/TR/html-json-forms/
+    (
+        ['bottle-on-wall:=1', 'bottle-on-wall:=2', 'bottle-on-wall:=3'],
+        {'bottle-on-wall': [1, 2, 3]},
+    ),
+    (
+        ['pet[species]=Dahut', 'pet[name]:="Hypatia"', 'kids[1]=Thelma', 'kids[0]:="Ashley"'],
+        {'pet': {'species': 'Dahut', 'name': 'Hypatia'}, 'kids': ['Ashley', 'Thelma']},
+    ),
+    (
+        ['pet[0][species]=Dahut', 'pet[0][name]=Hypatia', 'pet[1][species]=Felis Stultus', 'pet[1][name]:="Billie"'],
+        {'pet': [{'species': 'Dahut', 'name': 'Hypatia'}, {'species': 'Felis Stultus', 'name': 'Billie'}]},
+    ),
+    (
+        ['wow[such][deep][3][much][power][!]=Amaze'],
+        {'wow': {'such': {'deep': [None, None, None, {'much': {'power': {'!': 'Amaze'}}}]}}},
+    ),
+    (
+        ['mix=scalar', 'mix[0]=array 1', 'mix[2]:="array 2"', 'mix[key]:="key key"', 'mix[car]=car key'],
+        {'mix': {'': 'scalar', '0': 'array 1', '2': 'array 2', 'key': 'key key', 'car': 'car key'}},
+    ),
+    (
+        ['highlander[]=one'],
+        {'highlander': ['one']},
+    ),
+    (
+        ['error[good]=BOOM!', 'error[bad:="BOOM BOOM!"'],
+        {'error': {'good': 'BOOM!'}, 'error[bad': 'BOOM BOOM!'},
+    ),
+    (
+        ['special[]:=true', 'special[]:=false', 'special[]:="true"', 'special[]:=null'],
+        {'special': [True, False, 'true', None]},
+    ),
+    (
+        [r'\[\]:=1', r'escape\[d\]:=1', r'escaped\[\]:=1', r'e\[s\][c][a][p]\[ed\][]:=1'],
+        {'[]': 1, 'escape[d]': 1, 'escaped[]': 1, 'e[s]': {'c': {'a': {'p': {'[ed]': [1]}}}}},
+    ),
+    (
+        ['[]:=1', '[]=foo'],
+        [1, 'foo'],
+    ),
+    (
+        [']:=1', '[]1:=1', '[1]]:=1'],
+        {']': 1, '[]1': 1, '[1]]': 1},
+    ),
+])
+def test_nested_json_syntax(input_json, expected_json, httpbin_both):
+    r = http(httpbin_both + '/post', *input_json)
+    assert r.json['json'] == expected_json
+
+
+def test_nested_json_sparse_array(httpbin_both):
+    r = http(httpbin_both + '/post', 'test[0]:=1', 'test[100]:=1')
+    assert len(r.json['json']['test']) == 101

tests/utils/__init__.py

@@ -228,7 +228,7 @@ def http(
     # noinspection PyUnresolvedReferences
     """
     Run HTTPie and capture stderr/out and exit status.
-    Content writtent to devnull will be captured only if
+    Content written to devnull will be captured only if
     env.devnull is set manually.
 
     Invoke `httpie.core.main()` with `args` and `kwargs`,