holm/nushell
1
1
Fork 0
forked from extern/nushell
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f70c6d5d48
nushell/src/parser/hir.rs

417 lines
13 KiB
Rust
Raw Normal View History

Replace crate visibility identifier with pub(crate) Result of running: find src -name *.rs -exec sed -i 's/crate /pub(crate) /g' {} \;
2019-08-29 13:08:28 +02:00
pub(crate) mod baseline_parse;
pub(crate) mod binary;
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
pub(crate) mod expand_external_tokens;
Replace crate visibility identifier with pub(crate) Result of running: find src -name *.rs -exec sed -i 's/crate /pub(crate) /g' {} \;
2019-08-29 13:08:28 +02:00
pub(crate) mod external_command;
pub(crate) mod named;
pub(crate) mod path;
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
pub(crate) mod syntax_shape;
pub(crate) mod tokens_iterator;
Things work
2019-06-22 03:36:57 +02:00
--no-edit
2019-11-04 16:47:03 +01:00
use crate::parser::hir::path::PathMember;
use crate::parser::hir::syntax_shape::Member;
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
use crate::parser::{registry, Operator, Unit};
Tests pass!
2019-07-24 00:22:11 +02:00
use crate::prelude::*;
Things work
2019-06-22 03:36:57 +02:00
use derive_new::new;
use getset::Getters;
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
use nu_source::Spanned;
Restructuring
2019-08-02 21:15:07 +02:00
use serde::{Deserialize, Serialize};
Add support for ~ expansion This ended up being a bit of a yak shave. The basic idea in this commit is to expand `~` in paths, but only in paths. The way this is accomplished is by doing the expansion inside of the code that parses literal syntax for `SyntaxType::Path`. As a quick refresher: every command is entitled to expand its arguments in a custom way. While this could in theory be used for general-purpose macros, today the expansion facility is limited to syntactic hints. For example, the syntax `where cpu > 0` expands under the hood to `where { $it.cpu > 0 }`. This happens because the first argument to `where` is defined as a `SyntaxType::Block`, and the parser coerces binary expressions whose left-hand-side looks like a member into a block when the command is expecting one. This is mildly more magical than what most programming languages would do, but we believe that it makes sense to allow commands to fine-tune the syntax because of the domain nushell is in (command-line shells). The syntactic expansions supported by this facility are relatively limited. For example, we don't allow `$it` to become a bare word, simply because the command asks for a string in the relevant position. That would quickly become more confusing than it's worth. This PR adds a new `SyntaxType` rule: `SyntaxType::Path`. When a command declares a parameter as a `SyntaxType::Path`, string literals and bare words passed as an argument to that parameter are processed using the path expansion rules. Right now, that only means that `~` is expanded into the home directory, but additional rules are possible in the future. By restricting this expansion to a syntactic expansion when passed as an argument to a command expecting a path, we avoid making `~` a generally reserved character. This will also allow us to give good tab completion for paths with `~` characters in them when a command is expecting a path. In order to accomplish the above, this commit changes the parsing functions to take a `Context` instead of just a `CommandRegistry`. From the perspective of macro expansion, you can think of the `CommandRegistry` as a dictionary of in-scope macros, and the `Context` as the compile-time state used in expansion. This could gain additional functionality over time as we find more uses for the expansion system.
2019-08-26 21:21:03 +02:00
use std::path::PathBuf;
Things work
2019-06-22 03:36:57 +02:00
Add support for external escape valve (`^dir`) This commit makes it possible to force nu to treat a command as an external command by prefixing it with `^`. For example `^dir` will force `dir` to run an external command, even if `dir` is also a registered nu command. This ensures that users don't need to leave nu just because we happened to use a command they need. This commit adds a new token type for external commands, which, among other things, makes it pretty straight forward to syntax highlight external commands uniquely, and generally to treat them as special.
2019-08-16 00:18:18 +02:00
use crate::evaluate::Scope;
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
use crate::parser::parse::tokens::RawNumber;
Add support for external escape valve (`^dir`) This commit makes it possible to force nu to treat a command as an external command by prefixing it with `^`. For example `^dir` will force `dir` to run an external command, even if `dir` is also a registered nu command. This ensures that users don't need to leave nu just because we happened to use a command they need. This commit adds a new token type for external commands, which, among other things, makes it pretty straight forward to syntax highlight external commands uniquely, and generally to treat them as special.
2019-08-16 00:18:18 +02:00
Replace crate visibility identifier with pub(crate) Result of running: find src -name *.rs -exec sed -i 's/crate /pub(crate) /g' {} \;
2019-08-29 13:08:28 +02:00
pub(crate) use self::binary::Binary;
pub(crate) use self::external_command::ExternalCommand;
pub(crate) use self::named::NamedArguments;
pub(crate) use self::path::Path;
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
pub(crate) use self::syntax_shape::ExpandContext;
pub(crate) use self::tokens_iterator::TokensIterator;
Things work
2019-06-22 03:36:57 +02:00
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
pub use self::syntax_shape::SyntaxShape;
Support evaluating most expressions Blocks, paths, and others Plus a bunch of other infra improvements
2019-06-29 10:55:42 +02:00
Tests pass!
2019-07-24 00:22:11 +02:00
#[derive(Debug, Clone, Eq, PartialEq, Getters, Serialize, Deserialize, new)]
Things work
2019-06-22 03:36:57 +02:00
pub struct Call {
Get rid of feature(crate_visibility_modifier) (see #362)
2019-09-01 21:56:17 +02:00
#[get = "pub(crate)"]
WIP supporting from/to automatically
2019-08-29 05:53:45 +02:00
pub head: Box<Expression>,
Get rid of feature(crate_visibility_modifier) (see #362)
2019-09-01 21:56:17 +02:00
#[get = "pub(crate)"]
WIP supporting from/to automatically
2019-08-29 05:53:45 +02:00
pub positional: Option<Vec<Expression>>,
Get rid of feature(crate_visibility_modifier) (see #362)
2019-09-01 21:56:17 +02:00
#[get = "pub(crate)"]
WIP supporting from/to automatically
2019-08-29 05:53:45 +02:00
pub named: Option<NamedArguments>,
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
pub span: Span,
}
impl PrettyDebugWithSource for Call {
fn pretty_debug(&self, source: &str) -> DebugDocBuilder {
b::delimit(
"(",
self.head.pretty_debug(source)
+ b::preceded_option(
Some(b::space()),
self.positional.as_ref().map(|pos| {
b::intersperse(pos.iter().map(|expr| expr.pretty_debug(source)), b::space())
}),
)
+ b::preceded_option(
Some(b::space()),
self.named.as_ref().map(|named| named.pretty_debug(source)),
),
")",
)
}
Things work
2019-06-22 03:36:57 +02:00
}
Tests pass!
2019-07-24 00:22:11 +02:00
impl Call {
pub fn evaluate(
&self,
registry: &registry::CommandRegistry,
scope: &Scope,
source: &Text,
) -> Result<registry::EvaluatedArgs, ShellError> {
registry::evaluate_args(self, registry, scope, source)
}
}
#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
Things work
2019-06-22 03:36:57 +02:00
pub enum RawExpression {
Literal(Literal),
Close a bunch of holes in external command args Previously, there was a single parsing rule for "bare words" that applied to both internal and external commands. This meant that, because `cargo +nightly` needed to work, we needed to add `+` as a valid character in bare words. The number of characters continued to grow, and the situation was becoming untenable. The current strategy would eventually eat up all syntax and make it impossible to add syntax like `@foo` to internal commands. This patch significantly restricts bare words and introduces a new token type (`ExternalWord`). An `ExternalWord` expands to an error in the internal syntax, but expands to a bare word in the external syntax. `ExternalWords` are highlighted in grey in the shell.
2019-09-09 19:43:10 +02:00
ExternalWord,
Tests pass!
2019-07-24 00:22:11 +02:00
Synthetic(Synthetic),
Things work
2019-06-22 03:36:57 +02:00
Variable(Variable),
Binary(Box<Binary>),
Support evaluating most expressions Blocks, paths, and others Plus a bunch of other infra improvements
2019-06-29 10:55:42 +02:00
Block(Vec<Expression>),
Restructuring
2019-08-02 21:15:07 +02:00
List(Vec<Expression>),
Things work
2019-06-22 03:36:57 +02:00
Path(Box<Path>),
Added glob patterns to the syntax shapes Bare words now represent literal file names, and globs are a different syntax shape called "Pattern". This allows commands like `cp` to ask for a pattern as a source and a literal file as a target. This also means that attempting to pass a glob to a command that expects a literal path will produce an error.
2019-09-10 17:31:21 +02:00
Add support for ~ expansion This ended up being a bit of a yak shave. The basic idea in this commit is to expand `~` in paths, but only in paths. The way this is accomplished is by doing the expansion inside of the code that parses literal syntax for `SyntaxType::Path`. As a quick refresher: every command is entitled to expand its arguments in a custom way. While this could in theory be used for general-purpose macros, today the expansion facility is limited to syntactic hints. For example, the syntax `where cpu > 0` expands under the hood to `where { $it.cpu > 0 }`. This happens because the first argument to `where` is defined as a `SyntaxType::Block`, and the parser coerces binary expressions whose left-hand-side looks like a member into a block when the command is expecting one. This is mildly more magical than what most programming languages would do, but we believe that it makes sense to allow commands to fine-tune the syntax because of the domain nushell is in (command-line shells). The syntactic expansions supported by this facility are relatively limited. For example, we don't allow `$it` to become a bare word, simply because the command asks for a string in the relevant position. That would quickly become more confusing than it's worth. This PR adds a new `SyntaxType` rule: `SyntaxType::Path`. When a command declares a parameter as a `SyntaxType::Path`, string literals and bare words passed as an argument to that parameter are processed using the path expansion rules. Right now, that only means that `~` is expanded into the home directory, but additional rules are possible in the future. By restricting this expansion to a syntactic expansion when passed as an argument to a command expecting a path, we avoid making `~` a generally reserved character. This will also allow us to give good tab completion for paths with `~` characters in them when a command is expecting a path. In order to accomplish the above, this commit changes the parsing functions to take a `Context` instead of just a `CommandRegistry`. From the perspective of macro expansion, you can think of the `CommandRegistry` as a dictionary of in-scope macros, and the `Context` as the compile-time state used in expansion. This could gain additional functionality over time as we find more uses for the expansion system.
2019-08-26 21:21:03 +02:00
FilePath(PathBuf),
Add support for external escape valve (`^dir`) This commit makes it possible to force nu to treat a command as an external command by prefixing it with `^`. For example `^dir` will force `dir` to run an external command, even if `dir` is also a registered nu command. This ensures that users don't need to leave nu just because we happened to use a command they need. This commit adds a new token type for external commands, which, among other things, makes it pretty straight forward to syntax highlight external commands uniquely, and generally to treat them as special.
2019-08-16 00:18:18 +02:00
ExternalCommand(ExternalCommand),
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
Command(Span),
Things work
2019-06-22 03:36:57 +02:00
Boolean(bool),
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
impl ShellTypeName for RawExpression {
fn type_name(&self) -> &'static str {
Produce ArgumentError for signature mismatch ArgumentError also automatically produces diagnostics
2019-06-30 08:14:40 +02:00
match self {
RawExpression::Literal(literal) => literal.type_name(),
Tests pass!
2019-07-24 00:22:11 +02:00
RawExpression::Synthetic(synthetic) => synthetic.type_name(),
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
RawExpression::Command(..) => "command",
RawExpression::ExternalWord => "external word",
RawExpression::FilePath(..) => "file path",
Produce ArgumentError for signature mismatch ArgumentError also automatically produces diagnostics
2019-06-30 08:14:40 +02:00
RawExpression::Variable(..) => "variable",
Restructuring
2019-08-02 21:15:07 +02:00
RawExpression::List(..) => "list",
Produce ArgumentError for signature mismatch ArgumentError also automatically produces diagnostics
2019-06-30 08:14:40 +02:00
RawExpression::Binary(..) => "binary",
RawExpression::Block(..) => "block",
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
RawExpression::Path(..) => "variable path",
Produce ArgumentError for signature mismatch ArgumentError also automatically produces diagnostics
2019-06-30 08:14:40 +02:00
RawExpression::Boolean(..) => "boolean",
Add support for external escape valve (`^dir`) This commit makes it possible to force nu to treat a command as an external command by prefixing it with `^`. For example `^dir` will force `dir` to run an external command, even if `dir` is also a registered nu command. This ensures that users don't need to leave nu just because we happened to use a command they need. This commit adds a new token type for external commands, which, among other things, makes it pretty straight forward to syntax highlight external commands uniquely, and generally to treat them as special.
2019-08-16 00:18:18 +02:00
RawExpression::ExternalCommand(..) => "external",
Produce ArgumentError for signature mismatch ArgumentError also automatically produces diagnostics
2019-06-30 08:14:40 +02:00
}
}
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
pub enum Synthetic {
String(String),
}
impl ShellTypeName for Synthetic {
fn type_name(&self) -> &'static str {
match self {
Synthetic::String(_) => "string",
}
}
}
impl RawExpression {
pub fn into_expr(self, span: impl Into<Span>) -> Expression {
Expression {
expr: self,
span: span.into(),
}
}
pub fn into_unspanned_expr(self) -> Expression {
Expression {
expr: self,
span: Span::unknown(),
}
}
}
#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
pub struct Expression {
pub expr: RawExpression,
pub span: Span,
}
impl std::ops::Deref for Expression {
type Target = RawExpression;
fn deref(&self) -> &RawExpression {
&self.expr
}
}
impl HasSpan for Expression {
fn span(&self) -> Span {
self.span
}
}
impl PrettyDebugWithSource for Expression {
fn pretty_debug(&self, source: &str) -> DebugDocBuilder {
match &self.expr {
RawExpression::Literal(literal) => literal.spanned(self.span).pretty_debug(source),
RawExpression::ExternalWord => {
b::typed("external word", b::primitive(self.span.slice(source)))
}
RawExpression::Synthetic(s) => match s {
Synthetic::String(s) => b::typed("synthetic", b::primitive(format!("{:?}", s))),
},
RawExpression::Variable(_) => b::keyword(self.span.slice(source)),
RawExpression::Binary(binary) => binary.pretty_debug(source),
RawExpression::Block(_) => b::opaque("block"),
RawExpression::List(list) => b::delimit(
"[",
b::intersperse(
list.iter().map(|item| item.pretty_debug(source)),
b::space(),
),
"]",
),
RawExpression::Path(path) => path.pretty_debug(source),
RawExpression::FilePath(path) => b::typed("path", b::primitive(path.display())),
RawExpression::ExternalCommand(external) => b::typed(
"external command",
b::primitive(external.name.slice(source)),
),
RawExpression::Command(command) => {
b::typed("command", b::primitive(command.slice(source)))
}
RawExpression::Boolean(boolean) => match boolean {
true => b::primitive("$yes"),
false => b::primitive("$no"),
},
}
}
}
Things work
2019-06-22 03:36:57 +02:00
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
impl std::fmt::Display for Expression {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
let span = self.span;
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
match &self.expr {
RawExpression::Literal(literal) => write!(f, "{:?}", literal),
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
RawExpression::Synthetic(Synthetic::String(s)) => write!(f, "{}", s),
RawExpression::Command(_) => write!(f, "Command{{ {}..{} }}", span.start(), span.end()),
RawExpression::ExternalWord => {
write!(f, "ExternalWord{{ {}..{} }}", span.start(), span.end())
}
RawExpression::FilePath(file) => write!(f, "Path{{ {} }}", file.display()),
RawExpression::Variable(variable) => write!(f, "{}", variable),
RawExpression::List(list) => f
.debug_list()
.entries(list.iter().map(|e| format!("{}", e)))
.finish(),
RawExpression::Binary(binary) => write!(f, "{}", binary),
RawExpression::Block(items) => {
write!(f, "Block")?;
f.debug_set()
.entries(items.iter().map(|i| format!("{}", i)))
.finish()
}
RawExpression::Path(path) => write!(f, "{}", path),
RawExpression::Boolean(b) => write!(f, "${}", b),
RawExpression::ExternalCommand(..) => {
write!(f, "ExternalComment{{ {}..{} }}", span.start(), span.end())
}
}
}
}
Things work
2019-06-22 03:36:57 +02:00
impl Expression {
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn number(i: impl Into<Number>, span: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let span = span.into();
RawExpression::Literal(RawLiteral::Number(i.into()).into_literal(span)).into_expr(span)
Things work
2019-06-22 03:36:57 +02:00
}
Basic support for decimal numbers This commit is more substantial than it looks: there was basically no real support for decimals before, and that impacted values all the way through. I also made Size contain a decimal instead of an integer (`1.6kb` is a reasonable thing to type), which impacted a bunch of code. The biggest impact of this commit is that it creates many more possible ways for valid nu types to fail to serialize as toml, json, etc. which typically can't support the full range of Decimal (or Bigint, which I also think we should support). This commit makes to-toml fallible, and a similar effort is necessary for the rest of the serializations. We also need to figure out how to clearly communicate to users what has happened, but failing to serialize to toml seems clearly superior to me than weird errors in basic math operations.
2019-08-30 19:29:04 +02:00
pub(crate) fn size(
i: impl Into<Number>,
unit: impl Into<Unit>,
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
span: impl Into<Span>,
Basic support for decimal numbers This commit is more substantial than it looks: there was basically no real support for decimals before, and that impacted values all the way through. I also made Size contain a decimal instead of an integer (`1.6kb` is a reasonable thing to type), which impacted a bunch of code. The biggest impact of this commit is that it creates many more possible ways for valid nu types to fail to serialize as toml, json, etc. which typically can't support the full range of Decimal (or Bigint, which I also think we should support). This commit makes to-toml fallible, and a similar effort is necessary for the rest of the serializations. We also need to figure out how to clearly communicate to users what has happened, but failing to serialize to toml seems clearly superior to me than weird errors in basic math operations.
2019-08-30 19:29:04 +02:00
) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let span = span.into();
RawExpression::Literal(RawLiteral::Size(i.into(), unit.into()).into_literal(span))
.into_expr(span)
Things work
2019-06-22 03:36:57 +02:00
}
Replace crate visibility identifier with pub(crate) Result of running: find src -name *.rs -exec sed -i 's/crate /pub(crate) /g' {} \;
2019-08-29 13:08:28 +02:00
pub(crate) fn synthetic_string(s: impl Into<String>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::Synthetic(Synthetic::String(s.into())).into_unspanned_expr()
Tests pass!
2019-07-24 00:22:11 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn string(inner: impl Into<Span>, outer: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let outer = outer.into();
RawExpression::Literal(RawLiteral::String(inner.into()).into_literal(outer))
.into_expr(outer)
Things work
2019-06-22 03:36:57 +02:00
}
--no-edit
2019-11-04 16:47:03 +01:00
pub(crate) fn column_path(members: Vec<Member>, span: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let span = span.into();
RawExpression::Literal(RawLiteral::ColumnPath(members).into_literal(span)).into_expr(span)
--no-edit
2019-11-04 16:47:03 +01:00
}
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
pub(crate) fn path(
head: Expression,
--no-edit
2019-11-04 16:47:03 +01:00
tail: Vec<impl Into<PathMember>>,
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
span: impl Into<Span>,
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
) -> Expression {
--no-edit
2019-11-04 16:47:03 +01:00
let tail = tail.into_iter().map(|t| t.into()).collect();
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::Path(Box::new(Path::new(head, tail))).into_expr(span.into())
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
--no-edit
2019-11-04 16:47:03 +01:00
pub(crate) fn dot_member(head: Expression, next: impl Into<PathMember>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let Expression { expr: item, span } = head;
--no-edit
2019-11-04 16:47:03 +01:00
let next = next.into();
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
let new_span = head.span.until(next.span);
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
match item {
RawExpression::Path(path) => {
let (head, mut tail) = path.parts();
--no-edit
2019-11-04 16:47:03 +01:00
tail.push(next);
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
Expression::path(head, tail, new_span)
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
other => Expression::path(other.into_expr(span), vec![next], new_span),
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
}
pub(crate) fn infix(
left: Expression,
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
op: Spanned<impl Into<Operator>>,
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
right: Expression,
) -> Expression {
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
let new_span = left.span.until(right.span);
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
RawExpression::Binary(Box::new(Binary::new(left, op.map(|o| o.into()), right)))
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
.into_expr(new_span)
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn file_path(path: impl Into<PathBuf>, outer: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::FilePath(path.into()).into_expr(outer)
Add support for ~ expansion This ended up being a bit of a yak shave. The basic idea in this commit is to expand `~` in paths, but only in paths. The way this is accomplished is by doing the expansion inside of the code that parses literal syntax for `SyntaxType::Path`. As a quick refresher: every command is entitled to expand its arguments in a custom way. While this could in theory be used for general-purpose macros, today the expansion facility is limited to syntactic hints. For example, the syntax `where cpu > 0` expands under the hood to `where { $it.cpu > 0 }`. This happens because the first argument to `where` is defined as a `SyntaxType::Block`, and the parser coerces binary expressions whose left-hand-side looks like a member into a block when the command is expecting one. This is mildly more magical than what most programming languages would do, but we believe that it makes sense to allow commands to fine-tune the syntax because of the domain nushell is in (command-line shells). The syntactic expansions supported by this facility are relatively limited. For example, we don't allow `$it` to become a bare word, simply because the command asks for a string in the relevant position. That would quickly become more confusing than it's worth. This PR adds a new `SyntaxType` rule: `SyntaxType::Path`. When a command declares a parameter as a `SyntaxType::Path`, string literals and bare words passed as an argument to that parameter are processed using the path expansion rules. Right now, that only means that `~` is expanded into the home directory, but additional rules are possible in the future. By restricting this expansion to a syntactic expansion when passed as an argument to a command expecting a path, we avoid making `~` a generally reserved character. This will also allow us to give good tab completion for paths with `~` characters in them when a command is expecting a path. In order to accomplish the above, this commit changes the parsing functions to take a `Context` instead of just a `CommandRegistry`. From the perspective of macro expansion, you can think of the `CommandRegistry` as a dictionary of in-scope macros, and the `Context` as the compile-time state used in expansion. This could gain additional functionality over time as we find more uses for the expansion system.
2019-08-26 21:21:03 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn list(list: Vec<Expression>, span: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::List(list).into_expr(span)
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn bare(span: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let span = span.into();
RawExpression::Literal(RawLiteral::Bare.into_literal(span)).into_expr(span)
Things work
2019-06-22 03:36:57 +02:00
}
Expand tilde in patterns.
2019-10-27 02:01:58 +01:00
pub(crate) fn pattern(inner: impl Into<String>, outer: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
let outer = outer.into();
RawExpression::Literal(RawLiteral::GlobPattern(inner.into()).into_literal(outer))
.into_expr(outer)
Added glob patterns to the syntax shapes Bare words now represent literal file names, and globs are a different syntax shape called "Pattern". This allows commands like `cp` to ask for a pattern as a source and a literal file as a target. This also means that attempting to pass a glob to a command that expects a literal path will produce an error.
2019-09-10 17:31:21 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn variable(inner: impl Into<Span>, outer: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::Variable(Variable::Other(inner.into())).into_expr(outer)
Things work
2019-06-22 03:36:57 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn external_command(inner: impl Into<Span>, outer: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::ExternalCommand(ExternalCommand::new(inner.into())).into_expr(outer)
Add support for external escape valve (`^dir`) This commit makes it possible to force nu to treat a command as an external command by prefixing it with `^`. For example `^dir` will force `dir` to run an external command, even if `dir` is also a registered nu command. This ensures that users don't need to leave nu just because we happened to use a command they need. This commit adds a new token type for external commands, which, among other things, makes it pretty straight forward to syntax highlight external commands uniquely, and generally to treat them as special.
2019-08-16 00:18:18 +02:00
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
pub(crate) fn it_variable(inner: impl Into<Span>, outer: impl Into<Span>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::Variable(Variable::It(inner.into())).into_expr(outer)
Tests pass!
2019-07-24 00:22:11 +02:00
}
}
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
impl From<Spanned<Path>> for Expression {
fn from(path: Spanned<Path>) -> Expression {
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
RawExpression::Path(Box::new(path.item)).into_expr(path.span)
Support evaluating most expressions Blocks, paths, and others Plus a bunch of other infra improvements
2019-06-29 10:55:42 +02:00
}
}
Close a bunch of holes in external command args Previously, there was a single parsing rule for "bare words" that applied to both internal and external commands. This meant that, because `cargo +nightly` needed to work, we needed to add `+` as a valid character in bare words. The number of characters continued to grow, and the situation was becoming untenable. The current strategy would eventually eat up all syntax and make it impossible to add syntax like `@foo` to internal commands. This patch significantly restricts bare words and introduces a new token type (`ExternalWord`). An `ExternalWord` expands to an error in the internal syntax, but expands to a bare word in the external syntax. `ExternalWords` are highlighted in grey in the shell.
2019-09-09 19:43:10 +02:00
/// Literals are expressions that are:
///
/// 1. Copy
/// 2. Can be evaluated without additional context
/// 3. Evaluation cannot produce an error
Migrated numerics to BigInt/BigDecimal This commit migrates Value's numeric types to BigInt and BigDecimal. The basic idea is that overflow errors aren't great in a shell environment, and not really necessary. The main immediate consequence is that new errors can occur when serializing Nu values to other formats. You can see this in changes to the various serialization formats (JSON, TOML, etc.). There's a new `CoerceInto` trait that uses the `ToPrimitive` trait from `num_traits` to attempt to coerce a `BigNum` or `BigDecimal` into a target type, and produces a `RangeError` (kind of `ShellError`) if the coercion fails. Another possible future consequence is that certain performance-critical numeric operations might be too slow. If that happens, we can introduce specialized numeric types to help improve the performance of those situations, based on the real-world experience.
2019-09-01 18:20:31 +02:00
#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
pub enum RawLiteral {
Basic support for decimal numbers This commit is more substantial than it looks: there was basically no real support for decimals before, and that impacted values all the way through. I also made Size contain a decimal instead of an integer (`1.6kb` is a reasonable thing to type), which impacted a bunch of code. The biggest impact of this commit is that it creates many more possible ways for valid nu types to fail to serialize as toml, json, etc. which typically can't support the full range of Decimal (or Bigint, which I also think we should support). This commit makes to-toml fallible, and a similar effort is necessary for the rest of the serializations. We also need to figure out how to clearly communicate to users what has happened, but failing to serialize to toml seems clearly superior to me than weird errors in basic math operations.
2019-08-30 19:29:04 +02:00
Number(Number),
Size(Number, Unit),
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
String(Span),
Expand tilde in patterns.
2019-10-27 02:01:58 +01:00
GlobPattern(String),
--no-edit
2019-11-04 16:47:03 +01:00
ColumnPath(Vec<Member>),
Things work
2019-06-22 03:36:57 +02:00
Bare,
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
impl RawLiteral {
pub fn into_literal(self, span: impl Into<Span>) -> Literal {
Literal {
literal: self,
span: span.into(),
}
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
pub struct Literal {
pub literal: RawLiteral,
pub span: Span,
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
impl ShellTypeName for Literal {
fn type_name(&self) -> &'static str {
match &self.literal {
RawLiteral::Number(..) => "number",
RawLiteral::Size(..) => "size",
RawLiteral::String(..) => "string",
RawLiteral::ColumnPath(..) => "column path",
RawLiteral::Bare => "string",
RawLiteral::GlobPattern(_) => "pattern",
Tests pass!
2019-07-24 00:22:11 +02:00
}
}
}
Extract nu_source into a crate This commit extracts Tag, Span, Text, as well as source-related debug facilities into a new crate called nu_source. This change is much bigger than one might have expected because the previous code relied heavily on implementing inherent methods on `Tagged<T>` and `Spanned<T>`, which is no longer possible. As a result, this change creates more concrete types instead of using `Tagged<T>`. One notable example: Tagged<Value> became Value, and Value became UntaggedValue. This change clarifies the intent of the code in many places, but it does make it a big change.
2019-11-21 15:33:14 +01:00
impl PrettyDebugWithSource for Literal {
fn pretty_debug(&self, source: &str) -> DebugDocBuilder {
match &self.literal {
RawLiteral::Number(number) => number.pretty(),
RawLiteral::Size(number, unit) => (number.pretty() + unit.pretty()).group(),
RawLiteral::String(string) => b::primitive(format!("{:?}", string.slice(source))),
RawLiteral::GlobPattern(pattern) => b::typed("pattern", b::primitive(pattern)),
RawLiteral::ColumnPath(path) => b::typed(
"column path",
b::intersperse_with_source(path.iter(), b::space(), source),
),
RawLiteral::Bare => b::primitive(self.span.slice(source)),
Produce ArgumentError for signature mismatch ArgumentError also automatically produces diagnostics
2019-06-30 08:14:40 +02:00
}
}
}
Tests pass!
2019-07-24 00:22:11 +02:00
#[derive(Debug, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)]
Things work
2019-06-22 03:36:57 +02:00
pub enum Variable {
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
It(Span),
Other(Span),
Things work
2019-06-22 03:36:57 +02:00
}
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
impl std::fmt::Display for Variable {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Variable::It(_) => write!(f, "$it"),
Stream support (#812) * Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy. * Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt. * Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag. * To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 06:12:43 +02:00
Variable::Other(span) => write!(f, "${{ {}..{} }}", span.start(), span.end()),
Overhaul the expansion system The main thrust of this (very large) commit is an overhaul of the expansion system. The parsing pipeline is: - Lightly parse the source file for atoms, basic delimiters and pipeline structure into a token tree - Expand the token tree into a HIR (high-level intermediate representation) based upon the baseline syntax rules for expressions and the syntactic shape of commands. Somewhat non-traditionally, nu doesn't have an AST at all. It goes directly from the token tree, which doesn't represent many important distinctions (like the difference between `hello` and `5KB`) directly into a high-level representation that doesn't have a direct correspondence to the source code. At a high level, nu commands work like macros, in the sense that the syntactic shape of the invocation of a command depends on the definition of a command. However, commands do not have the ability to perform unrestricted expansions of the token tree. Instead, they describe their arguments in terms of syntactic shapes, and the expander expands the token tree into HIR based upon that definition. For example, the `where` command says that it takes a block as its first required argument, and the description of the block syntactic shape expands the syntax `cpu > 10` into HIR that represents `{ $it.cpu > 10 }`. This commit overhauls that system so that the syntactic shapes are described in terms of a few new traits (`ExpandSyntax` and `ExpandExpression` are the primary ones) that are more composable than the previous system. The first big win of this new system is the addition of the `ColumnPath` shape, which looks like `cpu."max ghz"` or `package.version`. Previously, while a variable path could look like `$it.cpu."max ghz"`, the tail of a variable path could not be easily reused in other contexts. Now, that tail is its own syntactic shape, and it can be used as part of a command's signature. This cleans up commands like `inc`, `add` and `edit` as well as shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-09-18 00:26:27 +02:00
}
}
}
Reference in New Issue Copy Permalink