Add nested JSON syntax to the HTTPie DSL (#1224)

* Add support for nested JSON syntax (#1169)

Co-authored-by: Batuhan Taskaya <isidentical@gmail.com>
Co-authored-by: Jakub Roztocil <jakub@roztocil.co>

* minor improvements

* unpack top level lists

* Write more docs

* doc style changes

* fix double quotes

Co-authored-by: Mickaël Schoentgen <contact@tiger-222.fr>
Co-authored-by: Jakub Roztocil <jakub@roztocil.co>
This commit is contained in:
Batuhan Taskaya 2021-12-03 13:17:45 +03:00 committed by GitHub
parent 8fe1f08a37
commit df58ec683e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 503 additions and 12 deletions

View File

@ -6,6 +6,7 @@ This project adheres to [Semantic Versioning](https://semver.org/).
## [3.0.0.dev0](https://github.com/httpie/httpie/compare/2.6.0...master) (unreleased)
- Improved startup time by 40%. ([#1211](https://github.com/httpie/httpie/pull/1211))
- Added support for nested JSON syntax. ([#1169](https://github.com/httpie/httpie/issues/1169))
- Added `httpie plugins` interface for plugin management. ([#566](https://github.com/httpie/httpie/issues/566))
- Added support for Bearer authentication via `--auth-type=bearer` ([#1215](https://github.com/httpie/httpie/issues/1215)).
- Added support for quick conversions of pasted URLs into HTTPie calls by adding a space after the protocol name (`$ https ://pie.dev` → `https://pie.dev`). ([#1195](https://github.com/httpie/httpie/issues/1195))

View File

@ -714,6 +714,246 @@ The `:=`/`:=@` syntax is JSON-specific. You can switch your request to `--form`
and string, float, and number values will continue to be serialized (as string form values).
Other JSON types, however, are not allowed with `--form` or `--multipart`.
### Nested JSON fields
For creating nested JSON structures, you can simply declare the path for the object's new destination
and HTTPie will interpret it according to the [JSON form](https://www.w3.org/TR/html-json-forms/)
notation and create your object. It works directly with the existing data field (`=`) and raw JSON
field (`:=`) operators.
#### Path Declaration
A simple path can be a shallow key;
```bash
$ http --offline --print=B pie.dev/post \
type=success
```
```json
{
"type": "success"
}
```
As well as a nested one,
```bash
$ http --offline --print=B pie.dev/post \
result[type]=success
```
```json
{
"result": {"type": "success"}
}
```
Or even multiple levels of nesting.
```bash
$ http --offline --print=B pie.dev/post \
result[status][type]=ok
```
```json
{
"result": {
"status": {
"type": "ok"
}
}
}
```
The declaration also supports creating arrays; which can be either done by simply
assigning the same path multiple times
```bash
$ http --offline --print=B pie.dev/post \
ids:=1 ids:=2
```
```json
{
"ids": [
1,
2
]
}
```
Or using the append suffix `[]`, which would create an array and append the items to the
end of it.
```bash
$ http --offline --print=B pie.dev/post \
ids[]:=1
```
```json
{
"ids": [
1,
2
]
}
```
You can also use indexes to set items on an array,
```bash
$ http --offline --print=B pie.dev/post \
items[0]=terminal items[1]=desktop
```
```json
{
"items": [
"terminal",
"desktop"
]
}
```
If you don't set value for the indexes between, then those will be nullified.
```bash
$ http --offline --print=B pie.dev/post \
items[1]=terminal items[3]=desktop
```
```json
{
"items": [
null,
"terminal",
null,
"desktop"
]
}
```
It is permitted to mix index-access with append actions (`[]`), but be aware that appends will not fill
the voids but instead they will append after the last item.
```bash
$ http --offline --print=B pie.dev/post \
items[1]=terminal items[3]=desktop items[]=web
```
```json
{
"items": [
null,
"terminal",
null,
"desktop",
"web"
]
}
```
If you need to send a top-level list (without any object that is encapsulating it), use the append operator (`[]`) without
any keys.
```bash
$ http --offline --print=B pie.dev/post \
[]:=1 []:=2 []:=3
```
```json
[
1,
2,
3
]
```
Here is a slightly unified example
```bash
$ http --offline --print=B pie.dev/post name=python version:=3 \
date[year]:=2021 date[month]=December \
systems=Linux systems=Mac systems=Windows \
people[known_ids][1]=1000 people[known_ids][5]=5000
```
```json
{
"date": {
"month": "December",
"year": 2021
},
"name": "python",
"people": {
"known_ids": [
null,
"1000",
null,
null,
null,
"5000"
]
},
"systems": [
"Linux",
"Mac",
"Windows"
],
"version": 3
}
```
And here is an even more comprehensive example to show all the features.
```bash
$ http PUT pie.dev/put \
'object=scalar' \ # Object — blank key
'object[0]=array 1' \ # Object — "0" key
'object[key]=key key' \ # Object — "key" key
'array:=1' \ # Array — first item
'array:=2' \ # Array — second item
'array[]:=3' \ # Array — append (third item)
'wow[such][deep][3][much][power][!]=Amaze' # Nested object
```
```http
PUT /person/1 HTTP/1.1
Accept: application/json, */*;q=0.5
Content-Type: application/json
Host: pie.dev
{
"array": [
1,
2,
3
],
"object": {
"": "scalar",
"0": "array 1",
"key": "key key"
},
"wow": {
"such": {
"deep": [
null,
null,
null,
{
"much": {
"power": {
"!": "Amaze"
}
}
}
]
}
}
}
```
### Raw and complex JSON
Please note that with the [request items](#request-items) data field syntax, commands can quickly become unwieldy when sending complex structures.
@ -731,9 +971,6 @@ $ http --raw '{"hello": "world"}' POST pie.dev/post
$ http POST pie.dev/post < files/data.json
```
Furthermore, the structure syntax only allows you to send an object as the JSON document, but not an array, etc.
Here, again, the solution is to use [redirected input](#redirected-input).
## Forms
Submitting forms is very similar to sending [JSON](#json) requests.

View File

@ -46,10 +46,10 @@ SEPARATOR_GROUP_DATA_EMBED_ITEMS = frozenset({
SEPARATOR_DATA_EMBED_RAW_JSON_FILE,
})
# Separators for raw JSON items
SEPARATOR_GROUP_RAW_JSON_ITEMS = frozenset([
# Separators for nested JSON items
SEPARATOR_GROUP_NESTED_JSON_ITEMS = frozenset([
SEPARATOR_DATA_STRING,
SEPARATOR_DATA_RAW_JSON,
SEPARATOR_DATA_EMBED_RAW_JSON_FILE,
])
# Separators allowed in ITEM arguments

View File

@ -120,7 +120,7 @@ positional.add_argument(
'=@' A data field like '=', but takes a file path and embeds its content:
essay=@Documents/essay.txt
essay=@Documents/essay.txt
':=@' A raw JSON field like ':=', but takes a file path and embeds its content:

150
httpie/cli/json_form.py Normal file
View File

@ -0,0 +1,150 @@
"""
Routines for JSON form syntax, used to support nested JSON request items.
Highly inspired from the great jarg project <https://github.com/jdp/jarg/blob/master/jarg/jsonform.py>.
"""
import re
import operator
from typing import Optional
def step(value: str, is_escaped: bool) -> str:
if is_escaped:
value = value.replace(r'\[', '[').replace(r'\]', ']')
return value
def find_opening_bracket(
value: str,
search=re.compile(r'(?<!\\)\[').search
) -> Optional[int]:
match = search(value)
if not match:
return None
return match.start()
def find_closing_bracket(
value: str,
search=re.compile(r'(?<!\\)\]').search
) -> Optional[int]:
match = search(value)
if not match:
return None
return match.start()
def parse_path(path):
"""
Parse a string as a JSON path.
An implementation of 'steps to parse a JSON encoding path'.
<https://www.w3.org/TR/html-json-forms/#dfn-steps-to-parse-a-json-encoding-path>
"""
original = path
is_escaped = r'\[' in original
opening_bracket = find_opening_bracket(original)
last_step = [(step(path, is_escaped), {'last': True, 'type': 'object'})]
if opening_bracket is None:
return last_step
steps = [(step(original[:opening_bracket], is_escaped), {'type': 'object'})]
path = original[opening_bracket:]
while path:
if path.startswith('[]'):
steps[-1][1]['append'] = True
path = path[2:]
if path:
return last_step
elif path[0] == '[':
path = path[1:]
closing_bracket = find_closing_bracket(path)
if closing_bracket is None:
return last_step
key = path[:closing_bracket]
path = path[closing_bracket + 1:]
try:
steps.append((int(key), {'type': 'array'}))
except ValueError:
steps.append((key, {'type': 'object'}))
elif path[:2] == r'\[':
key = step(path[1:path.index(r'\]') + 2], is_escaped)
path = path[path.index(r'\]') + 2:]
steps.append((key, {'type': 'object'}))
else:
return last_step
for i in range(len(steps) - 1):
steps[i][1]['type'] = steps[i + 1][1]['type']
steps[-1][1]['last'] = True
return steps
def set_value(context, step, current_value, entry_value):
"""Apply a JSON value to a context object.
An implementation of 'steps to set a JSON encoding value'.
<https://www.w3.org/TR/html-json-forms/#dfn-steps-to-set-a-json-encoding-value>
"""
key, flags = step
if flags.get('last', False):
if current_value is None:
if flags.get('append', False):
context[key] = [entry_value]
else:
if isinstance(context, list) and len(context) <= key:
context.extend([None] * (key - len(context) + 1))
context[key] = entry_value
elif isinstance(current_value, list):
context[key].append(entry_value)
else:
context[key] = [current_value, entry_value]
return context
if current_value is None:
if flags.get('type') == 'array':
context[key] = []
else:
if isinstance(context, list) and len(context) <= key:
context.extend([None] * (key - len(context) + 1))
context[key] = {}
return context[key]
elif isinstance(current_value, dict):
return context[key]
elif isinstance(current_value, list):
if flags.get('type') == 'array':
return current_value
obj = {}
for i, item in enumerate(current_value):
if item is not None:
obj[i] = item
else:
context[key] = obj
return obj
else:
obj = {'': current_value}
context[key] = obj
return obj
def interpret_json_form(pairs):
"""The application/json form encoding algorithm.
<https://www.w3.org/TR/html-json-forms/#dfn-application-json-encoding-algorithm>
"""
result = {}
for key, value in pairs:
steps = parse_path(key)
context = result
for step in steps:
try:
current_value = operator.getitem(context, step[0])
except LookupError:
current_value = None
context = set_value(context, step, current_value, value)
return result

View File

@ -5,7 +5,7 @@ from typing import Callable, Dict, IO, List, Optional, Tuple, Union
from .argtypes import KeyValueArg
from .constants import (
SEPARATORS_GROUP_MULTIPART, SEPARATOR_DATA_EMBED_FILE_CONTENTS,
SEPARATOR_DATA_EMBED_RAW_JSON_FILE,
SEPARATOR_DATA_EMBED_RAW_JSON_FILE, SEPARATOR_GROUP_NESTED_JSON_ITEMS,
SEPARATOR_DATA_RAW_JSON, SEPARATOR_DATA_STRING, SEPARATOR_FILE_UPLOAD,
SEPARATOR_FILE_UPLOAD_TYPE, SEPARATOR_HEADER, SEPARATOR_HEADER_EMPTY,
SEPARATOR_QUERY_PARAM, SEPARATOR_QUERY_EMBED_FILE, RequestType
@ -16,7 +16,8 @@ from .dicts import (
RequestQueryParamsDict,
)
from .exceptions import ParseError
from ..utils import get_content_type, load_json_preserve_order_and_dupe_keys
from .json_form import interpret_json_form
from ..utils import get_content_type, load_json_preserve_order_and_dupe_keys, split
class RequestItems:
@ -67,6 +68,10 @@ class RequestItems:
process_data_embed_file_contents_arg,
instance.data,
),
SEPARATOR_GROUP_NESTED_JSON_ITEMS: (
process_data_nested_json_embed_args,
instance.data,
),
SEPARATOR_DATA_RAW_JSON: (
json_only(instance, process_data_raw_json_embed_arg),
instance.data,
@ -77,6 +82,21 @@ class RequestItems:
),
}
if instance.is_json:
json_item_args, request_item_args = split(
request_item_args,
lambda arg: arg.sep in SEPARATOR_GROUP_NESTED_JSON_ITEMS
)
if json_item_args:
pairs = [
(arg.key, rules[arg.sep][0](arg))
for arg in json_item_args
]
processor_func, target_dict = rules[SEPARATOR_GROUP_NESTED_JSON_ITEMS]
value = processor_func(pairs)
target_dict.update(value)
# Then handle all other items.
for arg in request_item_args:
processor_func, target_dict = rules[arg.sep]
value = processor_func(arg)
@ -172,6 +192,10 @@ def process_data_raw_json_embed_arg(arg: KeyValueArg) -> JSONType:
return value
def process_data_nested_json_embed_args(pairs) -> Dict[str, JSONType]:
return interpret_json_form(pairs)
def load_text_file(item: KeyValueArg) -> str:
path = item.value
try:

View File

@ -287,6 +287,13 @@ def make_request_kwargs(
data = args.data
auto_json = data and not args.form
if (args.json or auto_json) and isinstance(data, dict):
# Propagate the top-level list if there is only one
# item in the object, with an en empty key.
if len(data) == 1:
[(key, value)] = data.items()
if key == '' and isinstance(value, list):
data = value
if data:
data = json.dumps(data)
else:

View File

@ -9,13 +9,14 @@ from collections import OrderedDict
from http.cookiejar import parse_ns_headers
from pathlib import Path
from pprint import pformat
from typing import Any, List, Optional, Tuple
from typing import Any, List, Optional, Tuple, Callable, Iterable, TypeVar
import requests.auth
RE_COOKIE_SPLIT = re.compile(r', (?=[^ ;]+=)')
Item = Tuple[str, Any]
Items = List[Item]
T = TypeVar("T")
class JsonDictPreservingDuplicateKeys(OrderedDict):
@ -218,3 +219,13 @@ def as_site(path: Path) -> Path:
vars={'base': str(path)}
)
return Path(site_packages_path)
def split(iterable: Iterable[T], key: Callable[[T], bool]) -> Tuple[List[T], List[T]]:
left, right = [], []
for item in iterable:
if key(item):
left.append(item)
else:
right.append(item)
return left, right

View File

@ -79,6 +79,8 @@ class TestItemParsing:
self.key_value_arg('Empty-Header;'),
self.key_value_arg('list:=["a", 1, {}, false]'),
self.key_value_arg('obj:={"a": "b"}'),
self.key_value_arg(r'nested\[2\][a][]=1'),
self.key_value_arg('nested[2][a][]:=1'),
self.key_value_arg('ed='),
self.key_value_arg('bool:=true'),
self.key_value_arg('file@' + FILE_PATH_ARG),
@ -105,7 +107,9 @@ class TestItemParsing:
'ed': '',
'string': 'value',
'bool': True,
'list': ['a', 1, {}, False],
'list': ['a', 1, load_json_preserve_order_and_dupe_keys('{}'), False],
'nested[2]': {'a': ['1']},
'nested': [None, None, {'a': [1]}],
'obj': load_json_preserve_order_and_dupe_keys('{"a": "b"}'),
'string-embed': FILE_CONTENT
}

View File

@ -152,3 +152,60 @@ def test_complex_json_arguments_with_non_json(httpbin, request_type, value):
)
cm.match('Can\'t use complex JSON value types')
@pytest.mark.parametrize('input_json, expected_json', [
# Examples taken from https://www.w3.org/TR/html-json-forms/
(
['bottle-on-wall:=1', 'bottle-on-wall:=2', 'bottle-on-wall:=3'],
{'bottle-on-wall': [1, 2, 3]},
),
(
['pet[species]=Dahut', 'pet[name]:="Hypatia"', 'kids[1]=Thelma', 'kids[0]:="Ashley"'],
{'pet': {'species': 'Dahut', 'name': 'Hypatia'}, 'kids': ['Ashley', 'Thelma']},
),
(
['pet[0][species]=Dahut', 'pet[0][name]=Hypatia', 'pet[1][species]=Felis Stultus', 'pet[1][name]:="Billie"'],
{'pet': [{'species': 'Dahut', 'name': 'Hypatia'}, {'species': 'Felis Stultus', 'name': 'Billie'}]},
),
(
['wow[such][deep][3][much][power][!]=Amaze'],
{'wow': {'such': {'deep': [None, None, None, {'much': {'power': {'!': 'Amaze'}}}]}}},
),
(
['mix=scalar', 'mix[0]=array 1', 'mix[2]:="array 2"', 'mix[key]:="key key"', 'mix[car]=car key'],
{'mix': {'': 'scalar', '0': 'array 1', '2': 'array 2', 'key': 'key key', 'car': 'car key'}},
),
(
['highlander[]=one'],
{'highlander': ['one']},
),
(
['error[good]=BOOM!', 'error[bad:="BOOM BOOM!"'],
{'error': {'good': 'BOOM!'}, 'error[bad': 'BOOM BOOM!'},
),
(
['special[]:=true', 'special[]:=false', 'special[]:="true"', 'special[]:=null'],
{'special': [True, False, 'true', None]},
),
(
[r'\[\]:=1', r'escape\[d\]:=1', r'escaped\[\]:=1', r'e\[s\][c][a][p]\[ed\][]:=1'],
{'[]': 1, 'escape[d]': 1, 'escaped[]': 1, 'e[s]': {'c': {'a': {'p': {'[ed]': [1]}}}}},
),
(
['[]:=1', '[]=foo'],
[1, 'foo'],
),
(
[']:=1', '[]1:=1', '[1]]:=1'],
{']': 1, '[]1': 1, '[1]]': 1},
),
])
def test_nested_json_syntax(input_json, expected_json, httpbin_both):
r = http(httpbin_both + '/post', *input_json)
assert r.json['json'] == expected_json
def test_nested_json_sparse_array(httpbin_both):
r = http(httpbin_both + '/post', 'test[0]:=1', 'test[100]:=1')
assert len(r.json['json']['test']) == 101

View File

@ -228,7 +228,7 @@ def http(
# noinspection PyUnresolvedReferences
"""
Run HTTPie and capture stderr/out and exit status.
Content writtent to devnull will be captured only if
Content written to devnull will be captured only if
env.devnull is set manually.
Invoke `httpie.core.main()` with `args` and `kwargs`,