merge w/ upstream

crates/nu-parser/Cargo.toml

@@ -4,5 +4,6 @@ version = "0.1.0"
 edition = "2018"
 
 [dependencies]
-codespan-reporting = "0.11.1"
-nu-protocol = { path = "../nu-protocol"}
+miette = "3.0.0"
+thiserror = "1.0.29"
+nu-protocol = { path = "../nu-protocol"}

crates/nu-parser/README.md

@@ -0,0 +1,99 @@
+# nu-parser, the Nushell parser
+
+Nushell's parser is a type-directed parser, meaning that the parser will use type information available during parse time to configure the parser. This allows it to handle a broader range of techniques to handle the arguments of a command.
+
+Nushell's base language is whitespace-separated tokens with the command (Nushell's term for a function) name in the head position:
+
+```
+head1 arg1 arg2 | head2
+```
+
+## Lexing
+
+The first job of the parser is to a lexical analysis to find where the tokens start and end in the input. This turns the above into:
+
+```
+<item: "head1">, <item: "arg1">, <item: "arg2">, <pipe>, <item: "head2">
+```
+
+At this point, the parser has little to no understanding of the shape of the command or how to parse its arguments.
+
+## Lite parsing
+
+As nushell is a language of pipelines, pipes form a key role in both separating commands from each other as well as denoting the flow of information between commands. The lite parse phase, as the name suggests, helps to group the lexed tokens into units.
+
+The above tokens are converted the following during the lite parse phase:
+
+```
+Pipeline:
+  Command #1:
+    <item: "head1">, <item: "arg1">, <item: "arg2">
+  Command #2:
+    <item: "head2">
+```
+
+## Parsing
+
+The real magic begins to happen when the parse moves on to the parsing stage. At this point, it traverses the lite parse tree and for each command makes a decision:
+
+* If the command looks like an internal/external command literal: eg) `foo` or `/usr/bin/ls`, it parses it as an internal or external command
+* Otherwise, it parses the command as part of a mathematical expression
+
+### Types/shapes
+
+Each command has a shape assigned to each of the arguments in reads in. These shapes help define how the parser will handle the parse.
+
+For example, if the command is written as:
+
+```sql
+where $x > 10
+```
+
+When the parsing happens, the parser will look up the `where` command and find its Signature. The Signature states what flags are allowed and what positional arguments are allowed (both required and optional). Each argument comes with it a Shape that defines how to parse values to get that position.
+
+In the above example, if the Signature of `where` said that it took three String values, the result would be:
+
+```
+CallInfo:
+  Name: `where`
+  Args:
+    Expression($x), a String
+    Expression(>), a String
+    Expression(10), a String
+```
+
+Or, the Signature could state that it takes in three positional arguments: a Variable, an Operator, and a Number, which would give:
+
+```
+CallInfo:
+  Name: `where`
+  Args:
+    Expression($x), a Variable
+    Expression(>), an Operator
+    Expression(10), a Number
+```
+
+Note that in this case, each would be checked at compile time to confirm that the expression has the shape requested. For example, `"foo"` would fail to parse as a Number.
+
+Finally, some Shapes can consume more than one token. In the above, if the `where` command stated it took in a single required argument, and that the Shape of this argument was a MathExpression, then the parser would treat the remaining tokens as part of the math expression.
+
+```
+CallInfo:
+  Name: `where`
+  Args:
+    MathExpression:
+      Op: >
+      LHS: Expression($x)
+      RHS: Expression(10)
+```
+
+When the command runs, it will now be able to evaluate the whole math expression as a single step rather than doing any additional parsing to understand the relationship between the parameters.
+
+## Making space
+
+As some Shapes can consume multiple tokens, it's important that the parser allow for multiple Shapes to coexist as peacefully as possible.
+
+The simplest way it does this is to ensure there is at least one token for each required parameter. If the Signature of the command says that it takes a MathExpression and a Number as two required arguments, then the parser will stop the math parser one token short. This allows the second Shape to consume the final token.
+
+Another way that the parser makes space is to look for Keyword shapes in the Signature. A Keyword is a word that's special to this command. For example in the `if` command, `else` is a keyword. When it is found in the arguments, the parser will use it as a signpost for where to make space for each Shape. The tokens leading up to the `else` will then feed into the parts of the Signature before the `else`, and the tokens following are consumed by the `else` and the Shapes that follow.
+

crates/nu-parser/src/errors.rs

@@ -1,31 +1,162 @@
+use miette::Diagnostic;
 use nu_protocol::{Span, Type};
+use thiserror::Error;
 
-#[derive(Debug)]
+#[derive(Clone, Debug, Error, Diagnostic)]
 pub enum ParseError {
-    ExtraTokens(Span),
-    ExtraPositional(Span),
-    UnexpectedEof(String, Span),
-    Unclosed(String, Span),
-    UnknownStatement(Span),
-    Expected(String, Span),
-    Mismatch(String, String, Span), // expected, found, span
-    UnsupportedOperation(Span, Span, Type, Span, Type),
-    ExpectedKeyword(String, Span),
-    MultipleRestParams(Span),
-    VariableNotFound(Span),
-    UnknownCommand(Span),
-    NonUtf8(Span),
-    UnknownFlag(Span),
-    UnknownType(Span),
-    MissingFlagParam(Span),
-    ShortFlagBatchCantTakeArg(Span),
-    MissingPositional(String, Span),
-    KeywordMissingArgument(String, Span),
-    MissingType(Span),
-    TypeMismatch(Type, Type, Span), // expected, found, span
-    MissingRequiredFlag(String, Span),
-    IncompleteMathExpression(Span),
-    UnknownState(String, Span),
-    IncompleteParser(Span),
-    RestNeedsName(Span),
+    /// The parser encountered unexpected tokens, when the code should have
+    /// finished. You should remove these or finish adding what you intended
+    /// to add.
+    #[error("Extra tokens in code.")]
+    #[diagnostic(
+        code(nu::parser::extra_tokens),
+        url(docsrs),
+        help("Try removing them.")
+    )]
+    ExtraTokens(#[label = "extra tokens"] Span),
+
+    #[error("Extra positional argument.")]
+    #[diagnostic(code(nu::parser::extra_positional), url(docsrs))]
+    ExtraPositional(#[label = "extra positional argument"] Span),
+
+    #[error("Unexpected end of code.")]
+    #[diagnostic(code(nu::parser::unexpected_eof), url(docsrs))]
+    UnexpectedEof(String, #[label("expected closing {0}")] Span),
+
+    #[error("Unclosed delimiter.")]
+    #[diagnostic(code(nu::parser::unclosed_delimiter), url(docsrs))]
+    Unclosed(String, #[label("unclosed {0}")] Span),
+
+    #[error("Unknown statement.")]
+    #[diagnostic(code(nu::parser::unknown_statement), url(docsrs))]
+    UnknownStatement(#[label("unknown statement")] Span),
+
+    #[error("Parse mismatch during operation.")]
+    #[diagnostic(code(nu::parser::parse_mismatch), url(docsrs))]
+    Expected(String, #[label("expected {0}")] Span),
+
+    #[error("Type mismatch during operation.")]
+    #[diagnostic(code(nu::parser::type_mismatch), url(docsrs))]
+    Mismatch(String, String, #[label("expected {0}, found {1}")] Span), // expected, found, span
+
+    #[error("Types mismatched for operation.")]
+    #[diagnostic(
+        code(nu::parser::unsupported_operation),
+        url(docsrs),
+        help("Change {2} or {4} to be the right types and try again.")
+    )]
+    UnsupportedOperation(
+        #[label = "doesn't support these values."] Span,
+        #[label("{2}")] Span,
+        Type,
+        #[label("{4}")] Span,
+        Type,
+    ),
+
+    #[error("Expected keyword.")]
+    #[diagnostic(code(nu::parser::expected_keyword), url(docsrs))]
+    ExpectedKeyword(String, #[label("expected {0}")] Span),
+
+    #[error("Multiple rest params.")]
+    #[diagnostic(code(nu::parser::multiple_rest_params), url(docsrs))]
+    MultipleRestParams(#[label = "multiple rest params"] Span),
+
+    #[error("Variable not found.")]
+    #[diagnostic(code(nu::parser::variable_not_found), url(docsrs))]
+    VariableNotFound(#[label = "variable not found"] Span),
+
+    #[error("Module not found.")]
+    #[diagnostic(code(nu::parser::module_not_found), url(docsrs))]
+    ModuleNotFound(#[label = "module not found"] Span),
+
+    #[error("Unknown command.")]
+    #[diagnostic(
+        code(nu::parser::unknown_command),
+        url(docsrs),
+        // TODO: actual suggestions
+        // help("Did you mean `foo`?")
+    )]
+    UnknownCommand(#[label = "unknown command"] Span),
+
+    #[error("Non-UTF8 code.")]
+    #[diagnostic(code(nu::parser::non_utf8), url(docsrs))]
+    NonUtf8(#[label = "non-UTF8 code"] Span),
+
+    #[error("The `{0}` command doesn't have flag `{1}`.")]
+    #[diagnostic(code(nu::parser::unknown_flag), url(docsrs))]
+    UnknownFlag(String, String, #[label = "unknown flag"] Span),
+
+    #[error("Unknown type.")]
+    #[diagnostic(code(nu::parser::unknown_type), url(docsrs))]
+    UnknownType(#[label = "unknown type"] Span),
+
+    #[error("Missing flag param.")]
+    #[diagnostic(code(nu::parser::missing_flag_param), url(docsrs))]
+    MissingFlagParam(#[label = "flag missing param"] Span),
+
+    #[error("Batches of short flags can't take arguments.")]
+    #[diagnostic(code(nu::parser::short_flag_arg_cant_take_arg), url(docsrs))]
+    ShortFlagBatchCantTakeArg(#[label = "short flag batches can't take args"] Span),
+
+    #[error("Missing required positional argument.")]
+    #[diagnostic(code(nu::parser::missing_positional), url(docsrs))]
+    MissingPositional(String, #[label("missing {0}")] Span),
+
+    #[error("Missing argument to `{0}`.")]
+    #[diagnostic(code(nu::parser::keyword_missing_arg), url(docsrs))]
+    KeywordMissingArgument(String, #[label("missing value that follows {0}")] Span),
+
+    #[error("Missing type.")]
+    #[diagnostic(code(nu::parser::missing_type), url(docsrs))]
+    MissingType(#[label = "expected type"] Span),
+
+    #[error("Type mismatch.")]
+    #[diagnostic(code(nu::parser::type_mismatch), url(docsrs))]
+    TypeMismatch(Type, Type, #[label("expected {0:?}, found {1:?}")] Span), // expected, found, span
+
+    #[error("Missing required flag.")]
+    #[diagnostic(code(nu::parser::missing_required_flag), url(docsrs))]
+    MissingRequiredFlag(String, #[label("missing required flag {0}")] Span),
+
+    #[error("Incomplete math expression.")]
+    #[diagnostic(code(nu::parser::incomplete_math_expression), url(docsrs))]
+    IncompleteMathExpression(#[label = "incomplete math expression"] Span),
+
+    #[error("Unknown state.")]
+    #[diagnostic(code(nu::parser::unknown_state), url(docsrs))]
+    UnknownState(String, #[label("{0}")] Span),
+
+    #[error("Parser incomplete.")]
+    #[diagnostic(code(nu::parser::parser_incomplete), url(docsrs))]
+    IncompleteParser(#[label = "parser support missing for this expression"] Span),
+
+    #[error("Rest parameter needs a name.")]
+    #[diagnostic(code(nu::parser::rest_needs_name), url(docsrs))]
+    RestNeedsName(#[label = "needs a parameter name"] Span),
+
+    #[error("Extra columns.")]
+    #[diagnostic(code(nu::parser::extra_columns), url(docsrs))]
+    ExtraColumns(
+        usize,
+        #[label("expected {0} column{}", if *.0 == 1 { "" } else { "s" })] Span,
+    ),
+
+    #[error("Missing columns.")]
+    #[diagnostic(code(nu::parser::missing_columns), url(docsrs))]
+    MissingColumns(
+        usize,
+        #[label("expected {0} column{}", if *.0 == 1 { "" } else { "s" })] Span,
+    ),
+
+    #[error("{0}")]
+    #[diagnostic(code(nu::parser::assignment_mismatch), url(docsrs))]
+    AssignmentMismatch(String, String, #[label("{1}")] Span),
+
+    #[error("Missing import pattern.")]
+    #[diagnostic(code(nu::parser::missing_import_pattern), url(docsrs))]
+    MissingImportPattern(#[label = "needs an import pattern"] Span),
+
+    #[error("Module export not found.")]
+    #[diagnostic(code(nu::parser::export_not_found), url(docsrs))]
+    ExportNotFound(#[label = "could not find imports"] Span),
 }

crates/nu-parser/src/flatten.rs

@@ -10,11 +10,13 @@ pub enum FlatShape {
     Range,
     InternalCall,
     External,
+    ExternalArg,
     Literal,
     Operator,
     Signature,
     String,
     Variable,
+    Custom(String),
 }
 
 pub fn flatten_block(working_set: &StateWorkingSet, block: &Block) -> Vec<(Span, FlatShape)> {
@@ -39,6 +41,10 @@ pub fn flatten_expression(
     working_set: &StateWorkingSet,
     expr: &Expression,
 ) -> Vec<(Span, FlatShape)> {
+    if let Some(custom_completion) = &expr.custom_completion {
+        return vec![(expr.span, FlatShape::Custom(custom_completion.clone()))];
+    }
+
     match &expr.expr {
         Expr::BinaryOp(lhs, op, rhs) => {
             let mut output = vec![];
@@ -55,8 +61,14 @@ pub fn flatten_expression(
             }
             output
         }
-        Expr::ExternalCall(..) => {
-            vec![(expr.span, FlatShape::External)]
+        Expr::ExternalCall(name, args) => {
+            let mut output = vec![(*name, FlatShape::External)];
+
+            for arg in args {
+                output.push((*arg, FlatShape::ExternalArg));
+            }
+
+            output
         }
         Expr::Garbage => {
             vec![(expr.span, FlatShape::Garbage)]
@@ -67,10 +79,10 @@ pub fn flatten_expression(
         Expr::Float(_) => {
             vec![(expr.span, FlatShape::Float)]
         }
-        Expr::FullCellPath(column_path) => {
+        Expr::FullCellPath(cell_path) => {
             let mut output = vec![];
-            output.extend(flatten_expression(working_set, &column_path.head));
-            for path_element in &column_path.tail {
+            output.extend(flatten_expression(working_set, &cell_path.head));
+            for path_element in &cell_path.tail {
                 match path_element {
                     PathMember::String { span, .. } => output.push((*span, FlatShape::String)),
                     PathMember::Int { span, .. } => output.push((*span, FlatShape::Int)),
@@ -78,15 +90,19 @@ pub fn flatten_expression(
             }
             output
         }
-        Expr::Range(from, to, op) => {
+        Expr::Range(from, next, to, op) => {
             let mut output = vec![];
             if let Some(f) = from {
                 output.extend(flatten_expression(working_set, f));
             }
+            if let Some(s) = next {
+                output.extend(vec![(op.next_op_span, FlatShape::Operator)]);
+                output.extend(flatten_expression(working_set, s));
+            }
+            output.extend(vec![(op.span, FlatShape::Operator)]);
             if let Some(t) = to {
                 output.extend(flatten_expression(working_set, t));
             }
-            output.extend(vec![(op.span, FlatShape::Operator)]);
             output
         }
         Expr::Bool(_) => {
@@ -114,6 +130,7 @@ pub fn flatten_expression(
         Expr::String(_) => {
             vec![(expr.span, FlatShape::String)]
         }
+        Expr::RowCondition(_, expr) => flatten_expression(working_set, expr),
         Expr::Subexpression(block_id) => {
             flatten_block(working_set, working_set.get_block(*block_id))
         }

crates/nu-parser/src/lex.rs

@@ -168,7 +168,7 @@ pub fn lex_item(
             (delim as char).to_string(),
             Span {
                 start: span.end,
-                end: span.end + 1,
+                end: span.end,
             },
         );
 
@@ -181,7 +181,13 @@ pub fn lex_item(
         // correct information from the non-lite parse.
         return (
             span,
-            Some(ParseError::UnexpectedEof((delim as char).to_string(), span)),
+            Some(ParseError::UnexpectedEof(
+                (delim as char).to_string(),
+                Span {
+                    start: span.end,
+                    end: span.end,
+                },
+            )),
         );
     }
 

crates/nu-parser/src/lib.rs

@@ -2,6 +2,7 @@ mod errors;
 mod flatten;
 mod lex;
 mod lite_parse;
+mod parse_keywords;
 mod parser;
 mod type_check;
 
@@ -9,4 +10,7 @@ pub use errors::ParseError;
 pub use flatten::{flatten_block, FlatShape};
 pub use lex::{lex, Token, TokenContents};
 pub use lite_parse::{lite_parse, LiteBlock};
+pub use parse_keywords::{
+    parse_alias, parse_def, parse_def_predecl, parse_let, parse_module, parse_use,
+};
 pub use parser::{parse, Import, VarDecl};

crates/nu-parser/src/parse_keywords.rs

@@ -0,0 +1,541 @@
+use nu_protocol::{
+    ast::{Block, Call, Expr, Expression, ImportPatternMember, Pipeline, Statement},
+    engine::StateWorkingSet,
+    span, DeclId, Span, SyntaxShape, Type,
+};
+
+use crate::{
+    lex, lite_parse,
+    parser::{
+        check_name, garbage, garbage_statement, parse_block_expression, parse_import_pattern,
+        parse_internal_call, parse_signature, parse_string,
+    },
+    ParseError,
+};
+
+pub fn parse_def_predecl(working_set: &mut StateWorkingSet, spans: &[Span]) {
+    let name = working_set.get_span_contents(spans[0]);
+
+    if name == b"def" && spans.len() >= 4 {
+        let (name_expr, ..) = parse_string(working_set, spans[1]);
+        let name = name_expr.as_string();
+
+        working_set.enter_scope();
+        // FIXME: because parse_signature will update the scope with the variables it sees
+        // we end up parsing the signature twice per def. The first time is during the predecl
+        // so that we can see the types that are part of the signature, which we need for parsing.
+        // The second time is when we actually parse the body itworking_set.
+        // We can't reuse the first time because the variables that are created during parse_signature
+        // are lost when we exit the scope below.
+        let (sig, ..) = parse_signature(working_set, spans[2]);
+        let signature = sig.as_signature();
+        working_set.exit_scope();
+
+        if let (Some(name), Some(mut signature)) = (name, signature) {
+            signature.name = name;
+            let decl = signature.predeclare();
+
+            working_set.add_decl(decl);
+        }
+    }
+}
+
+pub fn parse_def(
+    working_set: &mut StateWorkingSet,
+    spans: &[Span],
+) -> (Statement, Option<ParseError>) {
+    let mut error = None;
+    let name = working_set.get_span_contents(spans[0]);
+
+    if name == b"def" {
+        let def_decl_id = working_set
+            .find_decl(b"def")
+            .expect("internal error: missing def command");
+
+        let mut call = Box::new(Call {
+            head: spans[0],
+            decl_id: def_decl_id,
+            positional: vec![],
+            named: vec![],
+        });
+
+        let call = if let Some(name_span) = spans.get(1) {
+            let (name_expr, err) = parse_string(working_set, *name_span);
+            error = error.or(err);
+
+            let name = name_expr.as_string();
+            call.positional.push(name_expr);
+
+            if let Some(sig_span) = spans.get(2) {
+                working_set.enter_scope();
+                let (sig, err) = parse_signature(working_set, *sig_span);
+                error = error.or(err);
+
+                let signature = sig.as_signature();
+
+                call.positional.push(sig);
+
+                if let Some(block_span) = spans.get(3) {
+                    let (block, err) = parse_block_expression(
+                        working_set,
+                        &SyntaxShape::Block(Some(vec![])),
+                        *block_span,
+                    );
+                    error = error.or(err);
+
+                    let block_id = block.as_block();
+
+                    call.positional.push(block);
+
+                    if let (Some(name), Some(mut signature), Some(block_id)) =
+                        (name, signature, block_id)
+                    {
+                        let decl_id = working_set
+                            .find_decl(name.as_bytes())
+                            .expect("internal error: predeclaration failed to add definition");
+
+                        let declaration = working_set.get_decl_mut(decl_id);
+
+                        signature.name = name;
+
+                        *declaration = signature.into_block_command(block_id);
+                    }
+                } else {
+                    let err_span = Span {
+                        start: sig_span.end,
+                        end: sig_span.end,
+                    };
+
+                    error = error
+                        .or_else(|| Some(ParseError::MissingPositional("block".into(), err_span)));
+                }
+                working_set.exit_scope();
+
+                call
+            } else {
+                let err_span = Span {
+                    start: name_span.end,
+                    end: name_span.end,
+                };
+
+                error = error
+                    .or_else(|| Some(ParseError::MissingPositional("parameters".into(), err_span)));
+
+                call
+            }
+        } else {
+            let err_span = Span {
+                start: spans[0].end,
+                end: spans[0].end,
+            };
+
+            error = error.or_else(|| {
+                Some(ParseError::MissingPositional(
+                    "definition name".into(),
+                    err_span,
+                ))
+            });
+
+            call
+        };
+
+        (
+            Statement::Pipeline(Pipeline::from_vec(vec![Expression {
+                expr: Expr::Call(call),
+                span: span(spans),
+                ty: Type::Unknown,
+                custom_completion: None,
+            }])),
+            error,
+        )
+    } else {
+        (
+            garbage_statement(spans),
+            Some(ParseError::UnknownState(
+                "Expected structure: def <name> [] {}".into(),
+                span(spans),
+            )),
+        )
+    }
+}
+
+pub fn parse_alias(
+    working_set: &mut StateWorkingSet,
+    spans: &[Span],
+) -> (Statement, Option<ParseError>) {
+    let name = working_set.get_span_contents(spans[0]);
+
+    if name == b"alias" {
+        if let Some((span, err)) = check_name(working_set, spans) {
+            return (
+                Statement::Pipeline(Pipeline::from_vec(vec![garbage(*span)])),
+                Some(err),
+            );
+        }
+
+        if let Some(decl_id) = working_set.find_decl(b"alias") {
+            let (call, call_span, _) =
+                parse_internal_call(working_set, spans[0], &spans[1..], decl_id);
+
+            if spans.len() >= 4 {
+                let alias_name = working_set.get_span_contents(spans[1]);
+
+                let alias_name = if alias_name.starts_with(b"\"")
+                    && alias_name.ends_with(b"\"")
+                    && alias_name.len() > 1
+                {
+                    alias_name[1..(alias_name.len() - 1)].to_vec()
+                } else {
+                    alias_name.to_vec()
+                };
+                let _equals = working_set.get_span_contents(spans[2]);
+
+                let replacement = spans[3..].to_vec();
+
+                //println!("{:?} {:?}", alias_name, replacement);
+
+                working_set.add_alias(alias_name, replacement);
+            }
+
+            return (
+                Statement::Pipeline(Pipeline::from_vec(vec![Expression {
+                    expr: Expr::Call(call),
+                    span: call_span,
+                    ty: Type::Unknown,
+                    custom_completion: None,
+                }])),
+                None,
+            );
+        }
+    }
+
+    (
+        garbage_statement(spans),
+        Some(ParseError::UnknownState(
+            "internal error: alias statement unparseable".into(),
+            span(spans),
+        )),
+    )
+}
+
+pub fn parse_module(
+    working_set: &mut StateWorkingSet,
+    spans: &[Span],
+) -> (Statement, Option<ParseError>) {
+    // TODO: Currently, module is closing over its parent scope (i.e., defs in the parent scope are
+    // visible and usable in this module's scope). We might want to disable that. How?
+
+    let mut error = None;
+    let bytes = working_set.get_span_contents(spans[0]);
+
+    // parse_def() equivalent
+    if bytes == b"module" && spans.len() >= 3 {
+        let (module_name_expr, err) = parse_string(working_set, spans[1]);
+        error = error.or(err);
+
+        let module_name = module_name_expr
+            .as_string()
+            .expect("internal error: module name is not a string");
+
+        // parse_block_expression() equivalent
+        let block_span = spans[2];
+        let block_bytes = working_set.get_span_contents(block_span);
+        let mut start = block_span.start;
+        let mut end = block_span.end;
+
+        if block_bytes.starts_with(b"{") {
+            start += 1;
+        } else {
+            return (
+                garbage_statement(spans),
+                Some(ParseError::Expected("block".into(), block_span)),
+            );
+        }
+
+        if block_bytes.ends_with(b"}") {
+            end -= 1;
+        } else {
+            error = error.or_else(|| {
+                Some(ParseError::Unclosed(
+                    "}".into(),
+                    Span {
+                        start: end,
+                        end: end + 1,
+                    },
+                ))
+            });
+        }
+
+        let block_span = Span { start, end };
+
+        let source = working_set.get_span_contents(block_span);
+
+        let (output, err) = lex(source, start, &[], &[]);
+        error = error.or(err);
+
+        working_set.enter_scope();
+
+        // Do we need block parameters?
+
+        let (output, err) = lite_parse(&output);
+        error = error.or(err);
+
+        // We probably don't need $it
+
+        // we're doing parse_block() equivalent
+        // let (mut output, err) = parse_block(working_set, &output, false);
+
+        for pipeline in &output.block {
+            if pipeline.commands.len() == 1 {
+                parse_def_predecl(working_set, &pipeline.commands[0].parts);
+            }
+        }
+
+        let mut exports: Vec<(Vec<u8>, DeclId)> = vec![];
+
+        let block: Block = output
+            .block
+            .iter()
+            .map(|pipeline| {
+                if pipeline.commands.len() == 1 {
+                    // this one here is doing parse_statement() equivalent
+                    // let (stmt, err) = parse_statement(working_set, &pipeline.commands[0].parts);
+                    let name = working_set.get_span_contents(pipeline.commands[0].parts[0]);
+
+                    let (stmt, err) = match name {
+                        // TODO: Here we can add other stuff that's alowed for modules
+                        b"def" => {
+                            let (stmt, err) = parse_def(working_set, &pipeline.commands[0].parts);
+
+                            if err.is_none() {
+                                let decl_name =
+                                    working_set.get_span_contents(pipeline.commands[0].parts[1]);
+
+                                let decl_id = working_set
+                                    .find_decl(decl_name)
+                                    .expect("internal error: failed to find added declaration");
+
+                                // TODO: Later, we want to put this behind 'export'
+                                exports.push((decl_name.into(), decl_id));
+                            }
+
+                            (stmt, err)
+                        }
+                        _ => (
+                            garbage_statement(&pipeline.commands[0].parts),
+                            Some(ParseError::Expected(
+                                "def".into(),
+                                pipeline.commands[0].parts[0],
+                            )),
+                        ),
+                    };
+
+                    if error.is_none() {
+                        error = err;
+                    }
+
+                    stmt
+                } else {
+                    error = Some(ParseError::Expected("not a pipeline".into(), block_span));
+                    garbage_statement(spans)
+                }
+            })
+            .into();
+
+        let block = block.with_exports(exports);
+
+        working_set.exit_scope();
+
+        let block_id = working_set.add_module(&module_name, block);
+
+        let block_expr = Expression {
+            expr: Expr::Block(block_id),
+            span: block_span,
+            ty: Type::Block,
+            custom_completion: None,
+        };
+
+        let module_decl_id = working_set
+            .find_decl(b"module")
+            .expect("internal error: missing module command");
+
+        let call = Box::new(Call {
+            head: spans[0],
+            decl_id: module_decl_id,
+            positional: vec![module_name_expr, block_expr],
+            named: vec![],
+        });
+
+        (
+            Statement::Pipeline(Pipeline::from_vec(vec![Expression {
+                expr: Expr::Call(call),
+                span: span(spans),
+                ty: Type::Unknown,
+                custom_completion: None,
+            }])),
+            error,
+        )
+    } else {
+        (
+            garbage_statement(spans),
+            Some(ParseError::UnknownState(
+                "Expected structure: module <name> {}".into(),
+                span(spans),
+            )),
+        )
+    }
+}
+
+pub fn parse_use(
+    working_set: &mut StateWorkingSet,
+    spans: &[Span],
+) -> (Statement, Option<ParseError>) {
+    let mut error = None;
+    let bytes = working_set.get_span_contents(spans[0]);
+
+    // TODO: Currently, this directly imports the module's definitions into the current scope.
+    // Later, we want to put them behind the module's name and add selective importing
+    if bytes == b"use" && spans.len() >= 2 {
+        let (module_name_expr, err) = parse_string(working_set, spans[1]);
+        error = error.or(err);
+
+        let (import_pattern, err) = parse_import_pattern(working_set, spans[1]);
+        error = error.or(err);
+
+        let exports = if let Some(block_id) = working_set.find_module(&import_pattern.head) {
+            // TODO: Since we don't use the Block at all, we might just as well create a separate
+            // Module that holds only the exports, without having Blocks in the way.
+            working_set.get_block(block_id).exports.clone()
+        } else {
+            return (
+                garbage_statement(spans),
+                Some(ParseError::ModuleNotFound(spans[1])),
+            );
+        };
+
+        let exports = if import_pattern.members.is_empty() {
+            exports
+                .into_iter()
+                .map(|(name, id)| {
+                    let mut new_name = import_pattern.head.to_vec();
+                    new_name.push(b'.');
+                    new_name.extend(&name);
+                    (new_name, id)
+                })
+                .collect()
+        } else {
+            match &import_pattern.members[0] {
+                ImportPatternMember::Glob { .. } => exports,
+                ImportPatternMember::Name { name, span } => {
+                    let new_exports: Vec<(Vec<u8>, usize)> =
+                        exports.into_iter().filter(|x| &x.0 == name).collect();
+
+                    if new_exports.is_empty() {
+                        error = error.or(Some(ParseError::ExportNotFound(*span)))
+                    }
+
+                    new_exports
+                }
+                ImportPatternMember::List { names } => {
+                    let mut output = vec![];
+
+                    for (name, span) in names {
+                        let mut new_exports: Vec<(Vec<u8>, usize)> = exports
+                            .iter()
+                            .filter_map(|x| if &x.0 == name { Some(x.clone()) } else { None })
+                            .collect();
+
+                        if new_exports.is_empty() {
+                            error = error.or(Some(ParseError::ExportNotFound(*span)))
+                        } else {
+                            output.append(&mut new_exports)
+                        }
+                    }
+
+                    output
+                }
+            }
+        };
+
+        // Extend the current scope with the module's exports
+        working_set.activate_overlay(exports);
+
+        // Create the Use command call
+        let use_decl_id = working_set
+            .find_decl(b"use")
+            .expect("internal error: missing use command");
+
+        let call = Box::new(Call {
+            head: spans[0],
+            decl_id: use_decl_id,
+            positional: vec![module_name_expr],
+            named: vec![],
+        });
+
+        (
+            Statement::Pipeline(Pipeline::from_vec(vec![Expression {
+                expr: Expr::Call(call),
+                span: span(spans),
+                ty: Type::Unknown,
+                custom_completion: None,
+            }])),
+            error,
+        )
+    } else {
+        (
+            garbage_statement(spans),
+            Some(ParseError::UnknownState(
+                "Expected structure: use <name>".into(),
+                span(spans),
+            )),
+        )
+    }
+}
+
+pub fn parse_let(
+    working_set: &mut StateWorkingSet,
+    spans: &[Span],
+) -> (Statement, Option<ParseError>) {
+    let name = working_set.get_span_contents(spans[0]);
+
+    if name == b"let" {
+        if let Some((span, err)) = check_name(working_set, spans) {
+            return (
+                Statement::Pipeline(Pipeline::from_vec(vec![garbage(*span)])),
+                Some(err),
+            );
+        }
+
+        if let Some(decl_id) = working_set.find_decl(b"let") {
+            let (call, call_span, err) =
+                parse_internal_call(working_set, spans[0], &spans[1..], decl_id);
+
+            // Update the variable to the known type if we can.
+            if err.is_none() {
+                let var_id = call.positional[0]
+                    .as_var()
+                    .expect("internal error: expected variable");
+                let rhs_type = call.positional[1].ty.clone();
+
+                working_set.set_variable_type(var_id, rhs_type);
+            }
+
+            return (
+                Statement::Pipeline(Pipeline::from_vec(vec![Expression {
+                    expr: Expr::Call(call),
+                    span: call_span,
+                    ty: Type::Unknown,
+                    custom_completion: None,
+                }])),
+                err,
+            );
+        }
+    }
+    (
+        garbage_statement(spans),
+        Some(ParseError::UnknownState(
+            "internal error: let statement unparseable".into(),
+            span(spans),
+        )),
+    )
+}

crates/nu-parser/src/type_check.rs

@@ -20,6 +20,7 @@ pub fn math_result_type(
     op: &mut Expression,
     rhs: &mut Expression,
 ) -> (Type, Option<ParseError>) {
+    //println!("checking: {:?} {:?} {:?}", lhs, op, rhs);
     match &op.expr {
         Expr::Operator(operator) => match operator {
             Operator::Plus => match (&lhs.ty, &rhs.ty) {
@@ -31,6 +32,7 @@ pub fn math_result_type(
                 (Type::Unknown, _) => (Type::Unknown, None),
                 (_, Type::Unknown) => (Type::Unknown, None),
                 (Type::Int, _) => {
+                    let ty = rhs.ty.clone();
                     *rhs = Expression::garbage(rhs.span);
                     (
                         Type::Unknown,
@@ -39,7 +41,7 @@ pub fn math_result_type(
                             lhs.span,
                             lhs.ty.clone(),
                             rhs.span,
-                            rhs.ty.clone(),
+                            ty,
                         )),
                     )
                 }

crates/nu-parser/tests/test_parser.rs

@@ -2,10 +2,43 @@ use nu_parser::ParseError;
 use nu_parser::*;
 use nu_protocol::{
     ast::{Expr, Expression, Pipeline, Statement},
-    engine::{EngineState, StateWorkingSet},
+    engine::{Command, EngineState, StateWorkingSet},
     Signature, SyntaxShape,
 };
 
+#[cfg(test)]
+pub struct Let;
+
+#[cfg(test)]
+impl Command for Let {
+    fn name(&self) -> &str {
+        "let"
+    }
+
+    fn usage(&self) -> &str {
+        "Create a variable and give it a value."
+    }
+
+    fn signature(&self) -> nu_protocol::Signature {
+        Signature::build("let")
+            .required("var_name", SyntaxShape::VarWithOptType, "variable name")
+            .required(
+                "initial_value",
+                SyntaxShape::Keyword(b"=".to_vec(), Box::new(SyntaxShape::Expression)),
+                "equals sign followed by value",
+            )
+    }
+
+    fn run(
+        &self,
+        _context: &nu_protocol::engine::EvaluationContext,
+        _call: &nu_protocol::ast::Call,
+        _input: nu_protocol::Value,
+    ) -> Result<nu_protocol::Value, nu_protocol::ShellError> {
+        todo!()
+    }
+}
+
 #[test]
 pub fn parse_int() {
     let engine_state = EngineState::new();
@@ -164,6 +197,7 @@ mod range {
                     Expression {
                         expr: Expr::Range(
                             Some(_),
+                            None,
                             Some(_),
                             RangeOperator {
                                 inclusion: RangeInclusion::Inclusive,
@@ -195,6 +229,7 @@ mod range {
                     Expression {
                         expr: Expr::Range(
                             Some(_),
+                            None,
                             Some(_),
                             RangeOperator {
                                 inclusion: RangeInclusion::RightExclusive,
@@ -209,6 +244,38 @@ mod range {
         }
     }
 
+    #[test]
+    fn parse_reverse_range() {
+        let engine_state = EngineState::new();
+        let mut working_set = StateWorkingSet::new(&engine_state);
+
+        let (block, err) = parse(&mut working_set, None, b"10..0", true);
+
+        assert!(err.is_none());
+        assert!(block.len() == 1);
+        match &block[0] {
+            Statement::Pipeline(Pipeline { expressions }) => {
+                assert!(expressions.len() == 1);
+                assert!(matches!(
+                    expressions[0],
+                    Expression {
+                        expr: Expr::Range(
+                            Some(_),
+                            None,
+                            Some(_),
+                            RangeOperator {
+                                inclusion: RangeInclusion::Inclusive,
+                                ..
+                            }
+                        ),
+                        ..
+                    }
+                ))
+            }
+            _ => panic!("No match"),
+        }
+    }
+
     #[test]
     fn parse_subexpression_range() {
         let engine_state = EngineState::new();
@@ -226,6 +293,7 @@ mod range {
                     Expression {
                         expr: Expr::Range(
                             Some(_),
+                            None,
                             Some(_),
                             RangeOperator {
                                 inclusion: RangeInclusion::RightExclusive,
@@ -245,6 +313,8 @@ mod range {
         let engine_state = EngineState::new();
         let mut working_set = StateWorkingSet::new(&engine_state);
 
+        working_set.add_decl(Box::new(Let));
+
         let (block, err) = parse(&mut working_set, None, b"let a = 2; $a..10", true);
 
         assert!(err.is_none());
@@ -257,6 +327,7 @@ mod range {
                     Expression {
                         expr: Expr::Range(
                             Some(_),
+                            None,
                             Some(_),
                             RangeOperator {
                                 inclusion: RangeInclusion::Inclusive,
@@ -276,6 +347,8 @@ mod range {
         let engine_state = EngineState::new();
         let mut working_set = StateWorkingSet::new(&engine_state);
 
+        working_set.add_decl(Box::new(Let));
+
         let (block, err) = parse(&mut working_set, None, b"let a = 2; $a..<($a + 10)", true);
 
         assert!(err.is_none());
@@ -288,6 +361,7 @@ mod range {
                     Expression {
                         expr: Expr::Range(
                             Some(_),
+                            None,
                             Some(_),
                             RangeOperator {
                                 inclusion: RangeInclusion::RightExclusive,
@@ -320,6 +394,39 @@ mod range {
                         expr: Expr::Range(
                             Some(_),
                             None,
+                            None,
+                            RangeOperator {
+                                inclusion: RangeInclusion::Inclusive,
+                                ..
+                            }
+                        ),
+                        ..
+                    }
+                ))
+            }
+            _ => panic!("No match"),
+        }
+    }
+
+    #[test]
+    fn parse_left_unbounded_range() {
+        let engine_state = EngineState::new();
+        let mut working_set = StateWorkingSet::new(&engine_state);
+
+        let (block, err) = parse(&mut working_set, None, b"..10", true);
+
+        assert!(err.is_none());
+        assert!(block.len() == 1);
+        match &block[0] {
+            Statement::Pipeline(Pipeline { expressions }) => {
+                assert!(expressions.len() == 1);
+                assert!(matches!(
+                    expressions[0],
+                    Expression {
+                        expr: Expr::Range(
+                            None,
+                            None,
+                            Some(_),
                             RangeOperator {
                                 inclusion: RangeInclusion::Inclusive,
                                 ..
@@ -349,6 +456,39 @@ mod range {
                     expressions[0],
                     Expression {
                         expr: Expr::Range(
+                            Some(_),
+                            None,
+                            Some(_),
+                            RangeOperator {
+                                inclusion: RangeInclusion::Inclusive,
+                                ..
+                            }
+                        ),
+                        ..
+                    }
+                ))
+            }
+            _ => panic!("No match"),
+        }
+    }
+
+    #[test]
+    fn parse_float_range() {
+        let engine_state = EngineState::new();
+        let mut working_set = StateWorkingSet::new(&engine_state);
+
+        let (block, err) = parse(&mut working_set, None, b"2.0..4.0..10.0", true);
+
+        assert!(err.is_none());
+        assert!(block.len() == 1);
+        match &block[0] {
+            Statement::Pipeline(Pipeline { expressions }) => {
+                assert!(expressions.len() == 1);
+                assert!(matches!(
+                    expressions[0],
+                    Expression {
+                        expr: Expr::Range(
+                            Some(_),
                             Some(_),
                             Some(_),
                             RangeOperator {