Finish the job of moving shapes into the stream

This commit should finish the `coloring_in_tokens` feature, which moves
the shape accumulator into the token stream. This allows rollbacks of
the token stream to also roll back any shapes that were added.

This commit also adds a much nicer syntax highlighter trace, which shows
all of the paths the highlighter took to arrive at a particular coloring
output. This change is fairly substantial, but really improves the
understandability of the flow. I intend to update the normal parser with
a similar tracing view.

In general, this change also fleshes out the concept of "atomic" token
stream operations.

A good next step would be to try to make the parser more
error-correcting, using the coloring infrastructure. A follow-up step
would involve merging the parser and highlighter shapes themselves.
This commit is contained in:
Yehuda Katz
2019-10-21 08:18:43 -07:00
parent 82b24d9beb
commit 6a7c00eaef
22 changed files with 888 additions and 361 deletions

View File

@@ -246,13 +246,18 @@ fn it_arg_works_with_many_inputs_to_external_command() {
let (stdout, stderr) = nu_combined!(
cwd: dirs.test(), h::pipeline(
r#"
echo file1 file2
echo hello world
| split-row " "
| cat $it
| ^echo $it
"#
));
assert_eq!("text and more text", stdout);
#[cfg(windows)]
assert_eq!("hello world", stdout);
#[cfg(not(windows))]
assert_eq!("helloworld", stdout);
assert!(!stderr.contains("No such file or directory"));
})
}