forked from extern/nushell
IO and redirection overhaul (#11934)
# Description The PR overhauls how IO redirection is handled, allowing more explicit and fine-grain control over `stdout` and `stderr` output as well as more efficient IO and piping. To summarize the changes in this PR: - Added a new `IoStream` type to indicate the intended destination for a pipeline element's `stdout` and `stderr`. - The `stdout` and `stderr` `IoStream`s are stored in the `Stack` and to avoid adding 6 additional arguments to every eval function and `Command::run`. The `stdout` and `stderr` streams can be temporarily overwritten through functions on `Stack` and these functions will return a guard that restores the original `stdout` and `stderr` when dropped. - In the AST, redirections are now directly part of a `PipelineElement` as a `Option<Redirection>` field instead of having multiple different `PipelineElement` enum variants for each kind of redirection. This required changes to the parser, mainly in `lite_parser.rs`. - `Command`s can also set a `IoStream` override/redirection which will apply to the previous command in the pipeline. This is used, for example, in `ignore` to allow the previous external command to have its stdout redirected to `Stdio::null()` at spawn time. In contrast, the current implementation has to create an os pipe and manually consume the output on nushell's side. File and pipe redirections (`o>`, `e>`, `e>|`, etc.) have precedence over overrides from commands. This PR improves piping and IO speed, partially addressing #10763. Using the `throughput` command from that issue, this PR gives the following speedup on my setup for the commands below: | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | --------------------------- | -------------:| ------------:| -----------:| | `throughput o> /dev/null` | 1169 | 52938 | 54305 | | `throughput \| ignore` | 840 | 55438 | N/A | | `throughput \| null` | Error | 53617 | N/A | | `throughput \| rg 'x'` | 1165 | 3049 | 3736 | | `(throughput) \| rg 'x'` | 810 | 3085 | 3815 | (Numbers above are the median samples for throughput) This PR also paves the way to refactor our `ExternalStream` handling in the various commands. For example, this PR already fixes the following code: ```nushell ^sh -c 'echo -n "hello "; sleep 0; echo "world"' | find "hello world" ``` This returns an empty list on 0.90.1 and returns a highlighted "hello world" on this PR. Since the `stdout` and `stderr` `IoStream`s are available to commands when they are run, then this unlocks the potential for more convenient behavior. E.g., the `find` command can disable its ansi highlighting if it detects that the output `IoStream` is not the terminal. Knowing the output streams will also allow background job output to be redirected more easily and efficiently. # User-Facing Changes - External commands returned from closures will be collected (in most cases): ```nushell 1..2 | each {|_| nu -c "print a" } ``` This gives `["a", "a"]` on this PR, whereas this used to print "a\na\n" and then return an empty list. ```nushell 1..2 | each {|_| nu -c "print -e a" } ``` This gives `["", ""]` and prints "a\na\n" to stderr, whereas this used to return an empty list and print "a\na\n" to stderr. - Trailing new lines are always trimmed for external commands when piping into internal commands or collecting it as a value. (Failure to decode the output as utf-8 will keep the trailing newline for the last binary value.) In the current nushell version, the following three code snippets differ only in parenthesis placement, but they all also have different outputs: 1. `1..2 | each { ^echo a }` ``` a a ╭────────────╮ │ empty list │ ╰────────────╯ ``` 2. `1..2 | each { (^echo a) }` ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` 3. `1..2 | (each { ^echo a })` ``` ╭───┬───╮ │ 0 │ a │ │ │ │ │ 1 │ a │ │ │ │ ╰───┴───╯ ``` But in this PR, the above snippets will all have the same output: ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` - All existing flags on `run-external` are now deprecated. - File redirections now apply to all commands inside a code block: ```nushell (nu -c "print -e a"; nu -c "print -e b") e> test.out ``` This gives "a\nb\n" in `test.out` and prints nothing. The same result would happen when printing to stdout and using a `o>` file redirection. - External command output will (almost) never be ignored, and ignoring output must be explicit now: ```nushell (^echo a; ^echo b) ``` This prints "a\nb\n", whereas this used to print only "b\n". This only applies to external commands; values and internal commands not in return position will not print anything (e.g., `(echo a; echo b)` still only prints "b"). - `complete` now always captures stderr (`do` is not necessary). # After Submitting The language guide and other documentation will need to be updated.
This commit is contained in:
Ian Manske
GitHub
@ -1,212 +1,135 @@
|
||||
/// Lite parsing converts a flat stream of tokens from the lexer to a syntax element structure that
|
||||
/// can be parsed.
|
||||
//! Lite parsing converts a flat stream of tokens from the lexer to a syntax element structure that
|
||||
//! can be parsed.
|
||||
|
||||
use std::mem;
|
||||
|
||||
use crate::{Token, TokenContents};
|
||||
|
||||
use nu_protocol::{ast::Redirection, ParseError, Span};
|
||||
use nu_protocol::{ast::RedirectionSource, ParseError, Span};
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct LiteCommand {
|
||||
pub comments: Vec<Span>,
|
||||
pub parts: Vec<Span>,
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub enum LiteRedirectionTarget {
|
||||
File {
|
||||
connector: Span,
|
||||
file: Span,
|
||||
append: bool,
|
||||
},
|
||||
Pipe {
|
||||
connector: Span,
|
||||
},
|
||||
}
|
||||
|
||||
impl Default for LiteCommand {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
impl LiteRedirectionTarget {
|
||||
pub fn connector(&self) -> Span {
|
||||
match self {
|
||||
LiteRedirectionTarget::File { connector, .. }
|
||||
| LiteRedirectionTarget::Pipe { connector } => *connector,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum LiteRedirection {
|
||||
Single {
|
||||
source: RedirectionSource,
|
||||
target: LiteRedirectionTarget,
|
||||
},
|
||||
Separate {
|
||||
out: LiteRedirectionTarget,
|
||||
err: LiteRedirectionTarget,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct LiteCommand {
|
||||
pub pipe: Option<Span>,
|
||||
pub comments: Vec<Span>,
|
||||
pub parts: Vec<Span>,
|
||||
pub redirection: Option<LiteRedirection>,
|
||||
}
|
||||
|
||||
impl LiteCommand {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
comments: vec![],
|
||||
parts: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
pub fn push(&mut self, span: Span) {
|
||||
fn push(&mut self, span: Span) {
|
||||
self.parts.push(span);
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.parts.is_empty()
|
||||
fn try_add_redirection(
|
||||
&mut self,
|
||||
source: RedirectionSource,
|
||||
target: LiteRedirectionTarget,
|
||||
) -> Result<(), ParseError> {
|
||||
let redirection = match (self.redirection.take(), source) {
|
||||
(None, source) => Ok(LiteRedirection::Single { source, target }),
|
||||
(
|
||||
Some(LiteRedirection::Single {
|
||||
source: RedirectionSource::Stdout,
|
||||
target: out,
|
||||
}),
|
||||
RedirectionSource::Stderr,
|
||||
) => Ok(LiteRedirection::Separate { out, err: target }),
|
||||
(
|
||||
Some(LiteRedirection::Single {
|
||||
source: RedirectionSource::Stderr,
|
||||
target: err,
|
||||
}),
|
||||
RedirectionSource::Stdout,
|
||||
) => Ok(LiteRedirection::Separate { out: target, err }),
|
||||
(
|
||||
Some(LiteRedirection::Single {
|
||||
source,
|
||||
target: first,
|
||||
}),
|
||||
_,
|
||||
) => Err(ParseError::MultipleRedirections(
|
||||
source,
|
||||
first.connector(),
|
||||
target.connector(),
|
||||
)),
|
||||
(
|
||||
Some(LiteRedirection::Separate { out, .. }),
|
||||
RedirectionSource::Stdout | RedirectionSource::StdoutAndStderr,
|
||||
) => Err(ParseError::MultipleRedirections(
|
||||
RedirectionSource::Stdout,
|
||||
out.connector(),
|
||||
target.connector(),
|
||||
)),
|
||||
(Some(LiteRedirection::Separate { err, .. }), RedirectionSource::Stderr) => {
|
||||
Err(ParseError::MultipleRedirections(
|
||||
RedirectionSource::Stderr,
|
||||
err.connector(),
|
||||
target.connector(),
|
||||
))
|
||||
}
|
||||
}?;
|
||||
|
||||
self.redirection = Some(redirection);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// Note: the Span is the span of the connector not the whole element
|
||||
#[derive(Debug)]
|
||||
pub enum LiteElement {
|
||||
Command(Option<Span>, LiteCommand),
|
||||
// Similar to LiteElement::Command, except the previous command's output is stderr piped.
|
||||
// e.g: `e>| cmd`
|
||||
ErrPipedCommand(Option<Span>, LiteCommand),
|
||||
// Similar to LiteElement::Command, except the previous command's output is stderr + stdout piped.
|
||||
// e.g: `o+e>| cmd`
|
||||
OutErrPipedCommand(Option<Span>, LiteCommand),
|
||||
// final field indicates if it's in append mode
|
||||
Redirection(Span, Redirection, LiteCommand, bool),
|
||||
// SeparateRedirection variant can only be generated by two different Redirection variant
|
||||
// final bool field indicates if it's in append mode
|
||||
SeparateRedirection {
|
||||
out: (Span, LiteCommand, bool),
|
||||
err: (Span, LiteCommand, bool),
|
||||
},
|
||||
// SameTargetRedirection variant can only be generated by Command with Redirection::OutAndErr
|
||||
// redirection's final bool field indicates if it's in append mode
|
||||
SameTargetRedirection {
|
||||
cmd: (Option<Span>, LiteCommand),
|
||||
redirection: (Span, LiteCommand, bool),
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct LitePipeline {
|
||||
pub commands: Vec<LiteElement>,
|
||||
pub commands: Vec<LiteCommand>,
|
||||
}
|
||||
|
||||
impl LitePipeline {
|
||||
pub fn new() -> Self {
|
||||
Self { commands: vec![] }
|
||||
}
|
||||
|
||||
pub fn push(&mut self, element: LiteElement) {
|
||||
self.commands.push(element);
|
||||
}
|
||||
|
||||
pub fn insert(&mut self, index: usize, element: LiteElement) {
|
||||
self.commands.insert(index, element);
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.commands.is_empty()
|
||||
}
|
||||
|
||||
pub fn exists(&self, new_target: &Redirection) -> bool {
|
||||
for cmd in &self.commands {
|
||||
if let LiteElement::Redirection(_, exists_target, _, _) = cmd {
|
||||
if exists_target == new_target {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
fn push(&mut self, element: &mut LiteCommand) {
|
||||
if !element.parts.is_empty() || element.redirection.is_some() {
|
||||
self.commands.push(mem::take(element));
|
||||
}
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct LiteBlock {
|
||||
pub block: Vec<LitePipeline>,
|
||||
}
|
||||
|
||||
impl Default for LiteBlock {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl LiteBlock {
|
||||
pub fn new() -> Self {
|
||||
Self { block: vec![] }
|
||||
}
|
||||
|
||||
pub fn push(&mut self, mut pipeline: LitePipeline) {
|
||||
// once we push `pipeline` to our block
|
||||
// the block takes ownership of `pipeline`, which means that
|
||||
// our `pipeline` is complete on collecting commands.
|
||||
self.merge_redirections(&mut pipeline);
|
||||
self.merge_cmd_with_outerr_redirection(&mut pipeline);
|
||||
|
||||
self.block.push(pipeline);
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.block.is_empty()
|
||||
}
|
||||
|
||||
fn merge_cmd_with_outerr_redirection(&self, pipeline: &mut LitePipeline) {
|
||||
let mut cmd_index = None;
|
||||
let mut outerr_index = None;
|
||||
for (index, cmd) in pipeline.commands.iter().enumerate() {
|
||||
if let LiteElement::Command(..) = cmd {
|
||||
cmd_index = Some(index);
|
||||
}
|
||||
if let LiteElement::Redirection(
|
||||
_span,
|
||||
Redirection::StdoutAndStderr,
|
||||
_target_cmd,
|
||||
_is_append_mode,
|
||||
) = cmd
|
||||
{
|
||||
outerr_index = Some(index);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if let (Some(cmd_index), Some(outerr_index)) = (cmd_index, outerr_index) {
|
||||
// we can make sure that cmd_index is less than outerr_index.
|
||||
let outerr_redirect = pipeline.commands.remove(outerr_index);
|
||||
let cmd = pipeline.commands.remove(cmd_index);
|
||||
// `outerr_redirect` and `cmd` should always be `LiteElement::Command` and `LiteElement::Redirection`
|
||||
if let (
|
||||
LiteElement::Command(cmd_span, lite_cmd),
|
||||
LiteElement::Redirection(span, _, outerr_cmd, is_append_mode),
|
||||
) = (cmd, outerr_redirect)
|
||||
{
|
||||
pipeline.insert(
|
||||
cmd_index,
|
||||
LiteElement::SameTargetRedirection {
|
||||
cmd: (cmd_span, lite_cmd),
|
||||
redirection: (span, outerr_cmd, is_append_mode),
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn merge_redirections(&self, pipeline: &mut LitePipeline) {
|
||||
// In case our command may contains both stdout and stderr redirection.
|
||||
// We pick them out and Combine them into one LiteElement::SeparateRedirection variant.
|
||||
let mut stdout_index = None;
|
||||
let mut stderr_index = None;
|
||||
for (index, cmd) in pipeline.commands.iter().enumerate() {
|
||||
if let LiteElement::Redirection(_span, redirection, _target_cmd, _is_append_mode) = cmd
|
||||
{
|
||||
match *redirection {
|
||||
Redirection::Stderr => stderr_index = Some(index),
|
||||
Redirection::Stdout => stdout_index = Some(index),
|
||||
Redirection::StdoutAndStderr => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let (Some(out_indx), Some(err_indx)) = (stdout_index, stderr_index) {
|
||||
let (out_redirect, err_redirect, new_indx) = {
|
||||
// to avoid panic, we need to remove commands which have larger index first.
|
||||
if out_indx > err_indx {
|
||||
let out_redirect = pipeline.commands.remove(out_indx);
|
||||
let err_redirect = pipeline.commands.remove(err_indx);
|
||||
(out_redirect, err_redirect, err_indx)
|
||||
} else {
|
||||
let err_redirect = pipeline.commands.remove(err_indx);
|
||||
let out_redirect = pipeline.commands.remove(out_indx);
|
||||
(out_redirect, err_redirect, out_indx)
|
||||
}
|
||||
};
|
||||
// `out_redirect` and `err_redirect` should always be `LiteElement::Redirection`
|
||||
if let (
|
||||
LiteElement::Redirection(out_span, _, out_command, out_append_mode),
|
||||
LiteElement::Redirection(err_span, _, err_command, err_append_mode),
|
||||
) = (out_redirect, err_redirect)
|
||||
{
|
||||
// using insert with specific index to keep original
|
||||
// pipeline commands order.
|
||||
pipeline.insert(
|
||||
new_indx,
|
||||
LiteElement::SeparateRedirection {
|
||||
out: (out_span, out_command, out_append_mode),
|
||||
err: (err_span, err_command, err_append_mode),
|
||||
},
|
||||
)
|
||||
}
|
||||
fn push(&mut self, pipeline: &mut LitePipeline) {
|
||||
if !pipeline.commands.is_empty() {
|
||||
self.block.push(mem::take(pipeline));
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -226,162 +149,230 @@ fn last_non_comment_token(tokens: &[Token], cur_idx: usize) -> Option<TokenConte
|
||||
}
|
||||
|
||||
pub fn lite_parse(tokens: &[Token]) -> (LiteBlock, Option<ParseError>) {
|
||||
let mut block = LiteBlock::new();
|
||||
let mut curr_pipeline = LitePipeline::new();
|
||||
let mut curr_command = LiteCommand::new();
|
||||
|
||||
let mut last_token = TokenContents::Eol;
|
||||
|
||||
let mut last_connector = TokenContents::Pipe;
|
||||
let mut last_connector_span: Option<Span> = None;
|
||||
|
||||
if tokens.is_empty() {
|
||||
return (LiteBlock::new(), None);
|
||||
return (LiteBlock::default(), None);
|
||||
}
|
||||
|
||||
let mut curr_comment: Option<Vec<Span>> = None;
|
||||
let mut block = LiteBlock::default();
|
||||
let mut pipeline = LitePipeline::default();
|
||||
let mut command = LiteCommand::default();
|
||||
|
||||
let mut last_token = TokenContents::Eol;
|
||||
let mut file_redirection = None;
|
||||
let mut curr_comment: Option<Vec<Span>> = None;
|
||||
let mut error = None;
|
||||
|
||||
for (idx, token) in tokens.iter().enumerate() {
|
||||
match &token.contents {
|
||||
TokenContents::PipePipe => {
|
||||
error = error.or(Some(ParseError::ShellOrOr(token.span)));
|
||||
curr_command.push(token.span);
|
||||
last_token = TokenContents::Item;
|
||||
}
|
||||
TokenContents::Item => {
|
||||
// If we have a comment, go ahead and attach it
|
||||
if let Some(curr_comment) = curr_comment.take() {
|
||||
curr_command.comments = curr_comment;
|
||||
}
|
||||
curr_command.push(token.span);
|
||||
last_token = TokenContents::Item;
|
||||
}
|
||||
TokenContents::OutGreaterThan
|
||||
| TokenContents::OutGreaterGreaterThan
|
||||
| TokenContents::ErrGreaterThan
|
||||
| TokenContents::ErrGreaterGreaterThan
|
||||
| TokenContents::OutErrGreaterThan
|
||||
| TokenContents::OutErrGreaterGreaterThan => {
|
||||
if let Some(err) = push_command_to(
|
||||
&mut curr_pipeline,
|
||||
curr_command,
|
||||
last_connector,
|
||||
last_connector_span,
|
||||
) {
|
||||
error = Some(err);
|
||||
}
|
||||
if let Some((source, append, span)) = file_redirection.take() {
|
||||
if command.parts.is_empty() {
|
||||
error = error.or(Some(ParseError::LabeledError(
|
||||
"Redirection without command or expression".into(),
|
||||
"there is nothing to redirect".into(),
|
||||
span,
|
||||
)));
|
||||
|
||||
curr_command = LiteCommand::new();
|
||||
last_token = token.contents;
|
||||
last_connector = token.contents;
|
||||
last_connector_span = Some(token.span);
|
||||
}
|
||||
pipe_token @ (TokenContents::Pipe
|
||||
| TokenContents::ErrGreaterPipe
|
||||
| TokenContents::OutErrGreaterPipe) => {
|
||||
if let Some(err) = push_command_to(
|
||||
&mut curr_pipeline,
|
||||
curr_command,
|
||||
last_connector,
|
||||
last_connector_span,
|
||||
) {
|
||||
error = Some(err);
|
||||
}
|
||||
command.push(span);
|
||||
|
||||
curr_command = LiteCommand::new();
|
||||
last_token = *pipe_token;
|
||||
last_connector = *pipe_token;
|
||||
last_connector_span = Some(token.span);
|
||||
}
|
||||
TokenContents::Eol => {
|
||||
// Handle `[Command] [Pipe] ([Comment] | [Eol])+ [Command]`
|
||||
//
|
||||
// `[Eol]` branch checks if previous token is `[Pipe]` to construct pipeline
|
||||
// and so `[Comment] | [Eol]` should be ignore to make it work
|
||||
let actual_token = last_non_comment_token(tokens, idx);
|
||||
if actual_token != Some(TokenContents::Pipe)
|
||||
&& actual_token != Some(TokenContents::OutGreaterThan)
|
||||
{
|
||||
if let Some(err) = push_command_to(
|
||||
&mut curr_pipeline,
|
||||
curr_command,
|
||||
last_connector,
|
||||
last_connector_span,
|
||||
) {
|
||||
error = Some(err);
|
||||
match token.contents {
|
||||
TokenContents::Comment => {
|
||||
command.comments.push(token.span);
|
||||
curr_comment = None;
|
||||
}
|
||||
|
||||
curr_command = LiteCommand::new();
|
||||
if !curr_pipeline.is_empty() {
|
||||
block.push(curr_pipeline);
|
||||
|
||||
curr_pipeline = LitePipeline::new();
|
||||
last_connector = TokenContents::Pipe;
|
||||
last_connector_span = None;
|
||||
TokenContents::Pipe
|
||||
| TokenContents::ErrGreaterPipe
|
||||
| TokenContents::OutErrGreaterPipe => {
|
||||
pipeline.push(&mut command);
|
||||
command.pipe = Some(token.span);
|
||||
}
|
||||
TokenContents::Semicolon => {
|
||||
pipeline.push(&mut command);
|
||||
block.push(&mut pipeline);
|
||||
}
|
||||
TokenContents::Eol => {
|
||||
pipeline.push(&mut command);
|
||||
}
|
||||
_ => command.push(token.span),
|
||||
}
|
||||
} else {
|
||||
match &token.contents {
|
||||
TokenContents::PipePipe => {
|
||||
error = error.or(Some(ParseError::ShellOrOr(token.span)));
|
||||
command.push(span);
|
||||
command.push(token.span);
|
||||
}
|
||||
TokenContents::Item => {
|
||||
let target = LiteRedirectionTarget::File {
|
||||
connector: span,
|
||||
file: token.span,
|
||||
append,
|
||||
};
|
||||
if let Err(err) = command.try_add_redirection(source, target) {
|
||||
error = error.or(Some(err));
|
||||
command.push(span);
|
||||
command.push(token.span)
|
||||
}
|
||||
}
|
||||
TokenContents::OutGreaterThan
|
||||
| TokenContents::OutGreaterGreaterThan
|
||||
| TokenContents::ErrGreaterThan
|
||||
| TokenContents::ErrGreaterGreaterThan
|
||||
| TokenContents::OutErrGreaterThan
|
||||
| TokenContents::OutErrGreaterGreaterThan => {
|
||||
error =
|
||||
error.or(Some(ParseError::Expected("redirection target", token.span)));
|
||||
command.push(span);
|
||||
command.push(token.span);
|
||||
}
|
||||
TokenContents::Pipe
|
||||
| TokenContents::ErrGreaterPipe
|
||||
| TokenContents::OutErrGreaterPipe => {
|
||||
error =
|
||||
error.or(Some(ParseError::Expected("redirection target", token.span)));
|
||||
command.push(span);
|
||||
pipeline.push(&mut command);
|
||||
command.pipe = Some(token.span);
|
||||
}
|
||||
TokenContents::Eol => {
|
||||
error =
|
||||
error.or(Some(ParseError::Expected("redirection target", token.span)));
|
||||
command.push(span);
|
||||
pipeline.push(&mut command);
|
||||
}
|
||||
TokenContents::Semicolon => {
|
||||
error =
|
||||
error.or(Some(ParseError::Expected("redirection target", token.span)));
|
||||
command.push(span);
|
||||
pipeline.push(&mut command);
|
||||
block.push(&mut pipeline);
|
||||
}
|
||||
TokenContents::Comment => {
|
||||
error = error.or(Some(ParseError::Expected("redirection target", span)));
|
||||
command.push(span);
|
||||
command.comments.push(token.span);
|
||||
curr_comment = None;
|
||||
}
|
||||
}
|
||||
|
||||
if last_token == TokenContents::Eol {
|
||||
// Clear out the comment as we're entering a new comment
|
||||
curr_comment = None;
|
||||
}
|
||||
|
||||
last_token = TokenContents::Eol;
|
||||
}
|
||||
TokenContents::Semicolon => {
|
||||
if let Some(err) = push_command_to(
|
||||
&mut curr_pipeline,
|
||||
curr_command,
|
||||
last_connector,
|
||||
last_connector_span,
|
||||
) {
|
||||
error = Some(err);
|
||||
} else {
|
||||
match &token.contents {
|
||||
TokenContents::PipePipe => {
|
||||
error = error.or(Some(ParseError::ShellOrOr(token.span)));
|
||||
command.push(token.span);
|
||||
}
|
||||
TokenContents::Item => {
|
||||
// This is commented out to preserve old parser behavior,
|
||||
// but we should probably error here.
|
||||
//
|
||||
// if element.redirection.is_some() {
|
||||
// error = error.or(Some(ParseError::LabeledError(
|
||||
// "Unexpected positional".into(),
|
||||
// "cannot add positional arguments after output redirection".into(),
|
||||
// token.span,
|
||||
// )));
|
||||
// }
|
||||
//
|
||||
// For example, this is currently allowed: ^echo thing o> out.txt extra_arg
|
||||
|
||||
curr_command = LiteCommand::new();
|
||||
if !curr_pipeline.is_empty() {
|
||||
block.push(curr_pipeline);
|
||||
|
||||
curr_pipeline = LitePipeline::new();
|
||||
last_connector = TokenContents::Pipe;
|
||||
last_connector_span = None;
|
||||
// If we have a comment, go ahead and attach it
|
||||
if let Some(curr_comment) = curr_comment.take() {
|
||||
command.comments = curr_comment;
|
||||
}
|
||||
command.push(token.span);
|
||||
}
|
||||
TokenContents::OutGreaterThan => {
|
||||
file_redirection = Some((RedirectionSource::Stdout, false, token.span));
|
||||
}
|
||||
TokenContents::OutGreaterGreaterThan => {
|
||||
file_redirection = Some((RedirectionSource::Stdout, true, token.span));
|
||||
}
|
||||
TokenContents::ErrGreaterThan => {
|
||||
file_redirection = Some((RedirectionSource::Stderr, false, token.span));
|
||||
}
|
||||
TokenContents::ErrGreaterGreaterThan => {
|
||||
file_redirection = Some((RedirectionSource::Stderr, true, token.span));
|
||||
}
|
||||
TokenContents::OutErrGreaterThan => {
|
||||
file_redirection =
|
||||
Some((RedirectionSource::StdoutAndStderr, false, token.span));
|
||||
}
|
||||
TokenContents::OutErrGreaterGreaterThan => {
|
||||
file_redirection = Some((RedirectionSource::StdoutAndStderr, true, token.span));
|
||||
}
|
||||
TokenContents::ErrGreaterPipe => {
|
||||
let target = LiteRedirectionTarget::Pipe {
|
||||
connector: token.span,
|
||||
};
|
||||
if let Err(err) = command.try_add_redirection(RedirectionSource::Stderr, target)
|
||||
{
|
||||
error = error.or(Some(err));
|
||||
}
|
||||
pipeline.push(&mut command);
|
||||
command.pipe = Some(token.span);
|
||||
}
|
||||
TokenContents::OutErrGreaterPipe => {
|
||||
let target = LiteRedirectionTarget::Pipe {
|
||||
connector: token.span,
|
||||
};
|
||||
if let Err(err) =
|
||||
command.try_add_redirection(RedirectionSource::StdoutAndStderr, target)
|
||||
{
|
||||
error = error.or(Some(err));
|
||||
}
|
||||
pipeline.push(&mut command);
|
||||
command.pipe = Some(token.span);
|
||||
}
|
||||
TokenContents::Pipe => {
|
||||
pipeline.push(&mut command);
|
||||
command.pipe = Some(token.span);
|
||||
}
|
||||
TokenContents::Eol => {
|
||||
// Handle `[Command] [Pipe] ([Comment] | [Eol])+ [Command]`
|
||||
//
|
||||
// `[Eol]` branch checks if previous token is `[Pipe]` to construct pipeline
|
||||
// and so `[Comment] | [Eol]` should be ignore to make it work
|
||||
let actual_token = last_non_comment_token(tokens, idx);
|
||||
if actual_token != Some(TokenContents::Pipe) {
|
||||
pipeline.push(&mut command);
|
||||
block.push(&mut pipeline);
|
||||
}
|
||||
|
||||
last_token = TokenContents::Semicolon;
|
||||
}
|
||||
TokenContents::Comment => {
|
||||
// Comment is beside something
|
||||
if last_token != TokenContents::Eol {
|
||||
curr_command.comments.push(token.span);
|
||||
curr_comment = None;
|
||||
} else {
|
||||
// Comment precedes something
|
||||
if let Some(curr_comment) = &mut curr_comment {
|
||||
curr_comment.push(token.span);
|
||||
if last_token == TokenContents::Eol {
|
||||
// Clear out the comment as we're entering a new comment
|
||||
curr_comment = None;
|
||||
}
|
||||
}
|
||||
TokenContents::Semicolon => {
|
||||
pipeline.push(&mut command);
|
||||
block.push(&mut pipeline);
|
||||
}
|
||||
TokenContents::Comment => {
|
||||
// Comment is beside something
|
||||
if last_token != TokenContents::Eol {
|
||||
command.comments.push(token.span);
|
||||
curr_comment = None;
|
||||
} else {
|
||||
curr_comment = Some(vec![token.span]);
|
||||
// Comment precedes something
|
||||
if let Some(curr_comment) = &mut curr_comment {
|
||||
curr_comment.push(token.span);
|
||||
} else {
|
||||
curr_comment = Some(vec![token.span]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
last_token = TokenContents::Comment;
|
||||
}
|
||||
}
|
||||
|
||||
last_token = token.contents;
|
||||
}
|
||||
|
||||
if let Some(err) = push_command_to(
|
||||
&mut curr_pipeline,
|
||||
curr_command,
|
||||
last_connector,
|
||||
last_connector_span,
|
||||
) {
|
||||
error = Some(err);
|
||||
}
|
||||
if !curr_pipeline.is_empty() {
|
||||
block.push(curr_pipeline);
|
||||
if let Some((_, _, span)) = file_redirection {
|
||||
command.push(span);
|
||||
error = error.or(Some(ParseError::Expected("redirection target", span)));
|
||||
}
|
||||
|
||||
pipeline.push(&mut command);
|
||||
block.push(&mut pipeline);
|
||||
|
||||
if last_non_comment_token(tokens, tokens.len()) == Some(TokenContents::Pipe) {
|
||||
(
|
||||
block,
|
||||
@ -394,86 +385,3 @@ pub fn lite_parse(tokens: &[Token]) -> (LiteBlock, Option<ParseError>) {
|
||||
(block, error)
|
||||
}
|
||||
}
|
||||
|
||||
fn get_redirection(connector: TokenContents) -> Option<(Redirection, bool)> {
|
||||
match connector {
|
||||
TokenContents::OutGreaterThan => Some((Redirection::Stdout, false)),
|
||||
TokenContents::OutGreaterGreaterThan => Some((Redirection::Stdout, true)),
|
||||
TokenContents::ErrGreaterThan => Some((Redirection::Stderr, false)),
|
||||
TokenContents::ErrGreaterGreaterThan => Some((Redirection::Stderr, true)),
|
||||
TokenContents::OutErrGreaterThan => Some((Redirection::StdoutAndStderr, false)),
|
||||
TokenContents::OutErrGreaterGreaterThan => Some((Redirection::StdoutAndStderr, true)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// push a `command` to `pipeline`
|
||||
///
|
||||
/// It will return Some(err) if `command` is empty and we want to push a
|
||||
/// redirection command, or we have meet the same redirection in `pipeline`.
|
||||
fn push_command_to(
|
||||
pipeline: &mut LitePipeline,
|
||||
command: LiteCommand,
|
||||
last_connector: TokenContents,
|
||||
last_connector_span: Option<Span>,
|
||||
) -> Option<ParseError> {
|
||||
if !command.is_empty() {
|
||||
match get_redirection(last_connector) {
|
||||
Some((redirect, is_append_mode)) => {
|
||||
let span = last_connector_span
|
||||
.expect("internal error: redirection missing span information");
|
||||
if pipeline.exists(&redirect) {
|
||||
return Some(ParseError::LabeledError(
|
||||
"Redirection can be set only once".into(),
|
||||
"try to remove one".into(),
|
||||
span,
|
||||
));
|
||||
}
|
||||
pipeline.push(LiteElement::Redirection(
|
||||
last_connector_span
|
||||
.expect("internal error: redirection missing span information"),
|
||||
redirect,
|
||||
command,
|
||||
is_append_mode,
|
||||
))
|
||||
}
|
||||
None => {
|
||||
if last_connector == TokenContents::ErrGreaterPipe {
|
||||
pipeline.push(LiteElement::ErrPipedCommand(last_connector_span, command))
|
||||
} else if last_connector == TokenContents::OutErrGreaterPipe {
|
||||
// Don't allow o+e>| along with redirection.
|
||||
for cmd in &pipeline.commands {
|
||||
if matches!(
|
||||
cmd,
|
||||
LiteElement::Redirection { .. }
|
||||
| LiteElement::SameTargetRedirection { .. }
|
||||
| LiteElement::SeparateRedirection { .. }
|
||||
) {
|
||||
return Some(ParseError::LabeledError(
|
||||
"`o+e>|` pipe is not allowed to use with redirection".into(),
|
||||
"try to use different type of pipe, or remove redirection".into(),
|
||||
last_connector_span
|
||||
.expect("internal error: outerr pipe missing span information"),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
pipeline.push(LiteElement::OutErrPipedCommand(
|
||||
last_connector_span,
|
||||
command,
|
||||
))
|
||||
} else {
|
||||
pipeline.push(LiteElement::Command(last_connector_span, command))
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
} else if get_redirection(last_connector).is_some() {
|
||||
Some(ParseError::Expected(
|
||||
"redirection target",
|
||||
last_connector_span.expect("internal error: redirection missing span information"),
|
||||
))
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user