extern/nushell
1
0
Fork 1
mirror of https://github.com/nushell/nushell.git synced 2025-08-11 11:14:51 +02:00
Code Issues Packages Projects Releases Wiki Activity

Overhaul the coloring system

Browse Source 
This commit replaces the previous naive coloring system with a coloring
system that is more aligned with the parser.

The main benefit of this change is that it allows us to use parsing
rules to decide how to color tokens.

For example, consider the following syntax:

```
$ ps | where cpu > 10
```

Ideally, we could color `cpu` like a column name and not a string,
because `cpu > 10` is a shorthand block syntax that expands to
`{ $it.cpu > 10 }`.

The way that we know that it's a shorthand block is that the `where`
command declares that its first parameter is a `SyntaxShape::Block`,
which allows the shorthand block form.

In order to accomplish this, we need to color the tokens in a way that
corresponds to their expanded semantics, which means that high-fidelity
coloring requires expansion.

This commit adds a `ColorSyntax` trait that corresponds to the
`ExpandExpression` trait. The semantics are fairly similar, with a few
differences.

First `ExpandExpression` consumes N tokens and returns a single
`hir::Expression`. `ColorSyntax` consumes N tokens and writes M
`FlatShape` tokens to the output.

Concretely, for syntax like `[1 2 3]`

- `ExpandExpression` takes a single token node and produces a single
  `hir::Expression`
- `ColorSyntax` takes the same token node and emits 7 `FlatShape`s
  (open delimiter, int, whitespace, int, whitespace, int, close
  delimiter)

Second, `ColorSyntax` is more willing to plow through failures than
`ExpandExpression`.

In particular, consider syntax like

```
$ ps | where cpu >
```

In this case

- `ExpandExpression` will see that the `where` command is expecting a
  block, see that it's not a literal block and try to parse it as a
  shorthand block. It will successfully find a member followed by an
  infix operator, but not a following expression. That means that the
  entire pipeline part fails to parse and is a syntax error.
- `ColorSyntax` will also try to parse it as a shorthand block and
  ultimately fail, but it will fall back to "backoff coloring mode",
  which parsing any unidentified tokens in an unfallible, simple way. In
  this case, `cpu` will color as a string and `>` will color as an
  operator.

Finally, it's very important that coloring a pipeline infallibly colors
the entire string, doesn't fail, and doesn't get stuck in an infinite
loop.

In order to accomplish this, this PR separates `ColorSyntax`, which is
infallible from `FallibleColorSyntax`, which might fail. This allows the
type system to let us know if our coloring rules bottom out at at an
infallible rule.

It's not perfect: it's still possible for the coloring process to get
stuck or consume tokens non-atomically. I intend to reduce the
opportunity for those problems in a future commit. In the meantime, the
current system catches a number of mistakes (like trying to use a
fallible coloring rule in a loop without thinking about the possibility
that it will never terminate).
This commit is contained in:
Yehuda Katz
2019-10-06 13:22:50 -07:00
parent 1ad9d6f199
commit c2c10e2bc0
50 changed files with 3527 additions and 845 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
src/plugins/add.rs
View File

@ -1,7 +1,7 @@
use itertools::Itertools;
use nu::{
serve_plugin, CallInfo, Plugin, ReturnSuccess, ReturnValue, ShellError, Signature, SyntaxShape,
Tagged, Value,
Tagged, TaggedItem, Value,
};
pub type ColumnPath = Vec<Tagged<String>>;
@ -25,21 +25,27 @@ impl Add {
Some(f) => match obj.insert_data_at_column_path(value_tag, &f, v) {
Some(v) => return Ok(v),
None => {
return Err(ShellError::string(format!(
"add could not find place to insert field {:?} {}",
obj,
f.iter().map(|i| &i.item).join(".")
)))
return Err(ShellError::labeled_error(
format!(
"add could not find place to insert field {:?} {}",
obj,
f.iter().map(|i| &i.item).join(".")
),
"column name",
value_tag,
))
}
},
None => Err(ShellError::string(
None => Err(ShellError::labeled_error(
"add needs a column name when adding a value to a table",
"column name",
value_tag,
)),
},
x => Err(ShellError::string(format!(
"Unrecognized type in stream: {:?}",
x
))),
(value, _) => Err(ShellError::type_error(
"row",
value.type_name().tagged(value_tag),
)),
}
}
}
@ -64,12 +70,7 @@ impl Plugin for Add {
self.field = Some(table.as_column_path()?.item);
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
args[0]
)))
}
value => return Err(ShellError::type_error("table", value.tagged_type_name())),
}
match &args[1] {
Tagged { item: v, .. } => {

26
src/plugins/edit.rs
View File

@ -3,7 +3,7 @@ use nu::{
Tagged, Value,
};
pub type ColumnPath = Vec<Tagged<String>>;
pub type ColumnPath = Tagged<Vec<Tagged<String>>>;
struct Edit {
field: Option<ColumnPath>,
@ -24,19 +24,22 @@ impl Edit {
Some(f) => match obj.replace_data_at_column_path(value_tag, &f, v) {
Some(v) => return Ok(v),
None => {
return Err(ShellError::string(
return Err(ShellError::labeled_error(
"edit could not find place to insert column",
"column name",
f.tag,
))
}
},
None => Err(ShellError::string(
None => Err(ShellError::untagged_runtime_error(
"edit needs a column when changing a value in a table",
)),
},
x => Err(ShellError::string(format!(
"Unrecognized type in stream: {:?}",
x
))),
_ => Err(ShellError::labeled_error(
"Unrecognized type in stream",
"original value",
value_tag,
)),
}
}
}
@ -57,14 +60,9 @@ impl Plugin for Edit {
item: Value::Table(_),
..
} => {
self.field = Some(table.as_column_path()?.item);
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
args[0]
)))
self.field = Some(table.as_column_path()?);
}
value => return Err(ShellError::type_error("table", value.tagged_type_name())),
}
match &args[1] {
Tagged { item: v, .. } => {

11
src/plugins/embed.rs
View File

@ -25,8 +25,10 @@ impl Embed {
});
Ok(())
}
None => Err(ShellError::string(
None => Err(ShellError::labeled_error(
"embed needs a field when embedding a value",
"original value",
value.tag,
)),
},
}
@ -52,12 +54,7 @@ impl Plugin for Embed {
self.field = Some(s.clone());
self.values = Vec::new();
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
args[0]
)))
}
value => return Err(ShellError::type_error("string", value.tagged_type_name())),
}
}

41
src/plugins/inc.rs
View File

@ -14,7 +14,7 @@ pub enum SemVerAction {
Patch,
}
pub type ColumnPath = Vec<Tagged<String>>;
pub type ColumnPath = Tagged<Vec<Tagged<String>>>;
struct Inc {
field: Option<ColumnPath>,
@ -90,7 +90,11 @@ impl Inc {
let replacement = match value.item.get_data_by_column_path(value.tag(), f) {
Some(result) => self.inc(result.map(|x| x.clone()))?,
None => {
return Err(ShellError::string("inc could not find field to replace"))
return Err(ShellError::labeled_error(
"inc could not find field to replace",
"column name",
f.tag,
))
}
};
match value.item.replace_data_at_column_path(
@ -100,18 +104,22 @@ impl Inc {
) {
Some(v) => return Ok(v),
None => {
return Err(ShellError::string("inc could not find field to replace"))
return Err(ShellError::labeled_error(
"inc could not find field to replace",
"column name",
f.tag,
))
}
}
}
None => Err(ShellError::string(
None => Err(ShellError::untagged_runtime_error(
"inc needs a field when incrementing a column in a table",
)),
},
x => Err(ShellError::string(format!(
"Unrecognized type in stream: {:?}",
x
))),
_ => Err(ShellError::type_error(
"incrementable value",
value.tagged_type_name(),
)),
}
}
}
@ -145,14 +153,9 @@ impl Plugin for Inc {
item: Value::Table(_),
..
} => {
self.field = Some(table.as_column_path()?.item);
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
arg
)))
self.field = Some(table.as_column_path()?);
}
value => return Err(ShellError::type_error("table", value.tagged_type_name())),
}
}
}
@ -163,7 +166,11 @@ impl Plugin for Inc {
match &self.error {
Some(reason) => {
return Err(ShellError::string(format!("{}: {}", reason, Inc::usage())))
return Err(ShellError::untagged_runtime_error(format!(
"{}: {}",
reason,
Inc::usage()
)))
}
None => Ok(vec![]),
}
@ -308,7 +315,7 @@ mod tests {
assert_eq!(
plugin
.field
.map(|f| f.into_iter().map(|f| f.item).collect()),
.map(|f| f.iter().map(|f| f.item.clone()).collect()),
Some(vec!["package".to_string(), "version".to_string()])
);
}

44
src/plugins/match.rs
View File

@ -35,11 +35,12 @@ impl Plugin for Match {
} => {
self.column = s.clone();
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
args[0]
)));
Tagged { tag, .. } => {
return Err(ShellError::labeled_error(
"Unrecognized type in params",
"value",
tag,
));
}
}
match &args[1] {
@ -49,11 +50,12 @@ impl Plugin for Match {
} => {
self.regex = Regex::new(s).unwrap();
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
args[1]
)));
Tagged { tag, .. } => {
return Err(ShellError::labeled_error(
"Unrecognized type in params",
"value",
tag,
));
}
}
}
@ -65,7 +67,7 @@ impl Plugin for Match {
match &input {
Tagged {
item: Value::Row(dict),
..
tag,
} => {
if let Some(val) = dict.entries.get(&self.column) {
match val {
@ -75,22 +77,20 @@ impl Plugin for Match {
} => {
flag = self.regex.is_match(s);
}
_ => {
return Err(ShellError::string(format!(
"value is not a string! {:?}",
&val
)));
Tagged { tag, .. } => {
return Err(ShellError::labeled_error("expected string", "value", tag));
}
}
} else {
return Err(ShellError::string(format!(
"column not in row! {:?} {:?}",
&self.column, dict
)));
return Err(ShellError::labeled_error(
format!("column not in row! {:?} {:?}", &self.column, dict),
"row",
tag,
));
}
}
_ => {
return Err(ShellError::string(format!("Not a row! {:?}", &input)));
Tagged { tag, .. } => {
return Err(ShellError::labeled_error("Expected row", "value", tag));
}
}
if flag {

31
src/plugins/str.rs
View File

@ -105,20 +105,24 @@ impl Str {
) {
Some(v) => return Ok(v),
None => {
return Err(ShellError::string("str could not find field to replace"))
return Err(ShellError::type_error(
"column name",
value.tagged_type_name(),
))
}
}
}
None => Err(ShellError::string(format!(
None => Err(ShellError::untagged_runtime_error(format!(
"{}: {}",
"str needs a column when applied to a value in a row",
Str::usage()
))),
},
x => Err(ShellError::string(format!(
"Unrecognized type in stream: {:?}",
x
))),
_ => Err(ShellError::labeled_error(
"Unrecognized type in stream",
value.type_name(),
value.tag,
)),
}
}
}
@ -167,10 +171,11 @@ impl Plugin for Str {
self.field = Some(table.as_column_path()?.item);
}
_ => {
return Err(ShellError::string(format!(
"Unrecognized type in params: {:?}",
possible_field
)))
return Err(ShellError::labeled_error(
"Unrecognized type in params",
possible_field.type_name(),
possible_field.tag,
))
}
}
}
@ -187,7 +192,11 @@ impl Plugin for Str {
match &self.error {
Some(reason) => {
return Err(ShellError::string(format!("{}: {}", reason, Str::usage())))
return Err(ShellError::untagged_runtime_error(format!(
"{}: {}",
reason,
Str::usage()
)))
}
None => Ok(vec![]),
}

25
src/plugins/sum.rs
View File

@ -28,9 +28,11 @@ impl Sum {
self.total = Some(value.clone());
Ok(())
}
_ => Err(ShellError::string(format!(
"Could not sum non-integer or unrelated types"
))),
_ => Err(ShellError::labeled_error(
"Could not sum non-integer or unrelated types",
"source",
value.tag,
)),
}
}
Value::Primitive(Primitive::Bytes(b)) => {
@ -47,15 +49,18 @@ impl Sum {
self.total = Some(value);
Ok(())
}
_ => Err(ShellError::string(format!(
"Could not sum non-integer or unrelated types"
))),
_ => Err(ShellError::labeled_error(
"Could not sum non-integer or unrelated types",
"source",
value.tag,
)),
}
}
x => Err(ShellError::string(format!(
"Unrecognized type in stream: {:?}",
x
))),
x => Err(ShellError::labeled_error(
format!("Unrecognized type in stream: {:?}", x),
"source",
value.tag,
)),
}
}
}