Internal representation (IR) compiler and evaluator (#13330)

# Description

This PR adds an internal representation language to Nushell, offering an
alternative evaluator based on simple instructions, stream-containing
registers, and indexed control flow. The number of registers required is
determined statically at compile-time, and the fixed size required is
allocated upon entering the block.

Each instruction is associated with a span, which makes going backwards
from IR instructions to source code very easy.

Motivations for IR:

1. **Performance.** By simplifying the evaluation path and making it
more cache-friendly and branch predictor-friendly, code that does a lot
of computation in Nushell itself can be sped up a decent bit. Because
the IR is fairly easy to reason about, we can also implement
optimization passes in the future to eliminate and simplify code.
2. **Correctness.** The instructions mostly have very simple and
easily-specified behavior, so hopefully engine changes are a little bit
easier to reason about, and they can be specified in a more formal way
at some point. I have made an effort to document each of the
instructions in the docs for the enum itself in a reasonably specific
way. Some of the errors that would have happened during evaluation
before are now moved to the compilation step instead, because they don't
make sense to check during evaluation.
3. **As an intermediate target.** This is a good step for us to bring
the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at
some point, as code generated from new AST can be directly compared to
code generated from old AST. If the IR code is functionally equivalent,
it will behave the exact same way.
4. **Debugging.** With a little bit more work, we can probably give
control over advancing the virtual machine that `IrBlock`s run on to
some sort of external driver, making things like breakpoints and single
stepping possible. Tools like `view ir` and [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than
before to see what exactly is going on with your Nushell code.

The goal is to eventually replace the AST evaluator entirely, once we're
sure it's working just as well. You can help dogfood this by running
Nushell with `$env.NU_USE_IR` set to some value. The environment
variable is checked when Nushell starts, so config runs with IR, or it
can also be set on a line at the REPL to change it dynamically. It is
also checked when running `do` in case within a script you want to just
run a specific piece of code with or without IR.

# Example

```nushell
view ir { |data|
  mut sum = 0
  for n in $data {
    $sum += $n
  }
  $sum
}
```
  
```gas
# 3 registers, 19 instructions, 0 bytes of data
   0: load-literal           %0, int(0)
   1: store-variable         var 904, %0 # let
   2: drain                  %0
   3: drop                   %0
   4: load-variable          %1, var 903
   5: iterate                %0, %1, end 15 # for, label(1), from(14:)
   6: store-variable         var 905, %0
   7: load-variable          %0, var 904
   8: load-variable          %2, var 905
   9: binary-op              %0, Math(Plus), %2
  10: span                   %0
  11: store-variable         var 904, %0
  12: load-literal           %0, nothing
  13: drain                  %0
  14: jump                   5
  15: drop                   %0          # label(0), from(5:)
  16: drain                  %0
  17: load-variable          %0, var 904
  18: return                 %0
```

# Benchmarks

All benchmarks run on a base model Mac Mini M1.

## Iterative Fibonacci sequence

This is about as best case as possible, making use of the much faster
control flow. Most code will not experience a speed improvement nearly
this large.

```nushell
def fib [n: int] {
  mut a = 0
  mut b = 1
  for _ in 2..=$n {
    let c = $a + $b
    $a = $b
    $b = $c
  }
  $b
}
use std bench
bench { 0..50 | each { |n| fib $n } }
```

IR disabled:

```
╭───────┬─────────────────╮
│ mean  │ 1ms 924µs 665ns │
│ min   │ 1ms 700µs 83ns  │
│ max   │ 3ms 450µs 125ns │
│ std   │ 395µs 759ns     │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```

IR enabled:

```
╭───────┬─────────────────╮
│ mean  │ 452µs 820ns     │
│ min   │ 427µs 417ns     │
│ max   │ 540µs 167ns     │
│ std   │ 17µs 158ns      │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```

![explore ir
view](https://github.com/nushell/nushell/assets/10729/d7bccc03-5222-461c-9200-0dce71b83b83)

##
[gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu)

IR disabled:

```
╭───┬──────────────────╮
│ 0 │ 27ms 929µs 958ns │
│ 1 │ 21ms 153µs 459ns │
│ 2 │ 18ms 639µs 666ns │
│ 3 │ 19ms 554µs 583ns │
│ 4 │ 13ms 383µs 375ns │
│ 5 │ 11ms 328µs 208ns │
│ 6 │  5ms 659µs 542ns │
╰───┴──────────────────╯
```

IR enabled:

```
╭───┬──────────────────╮
│ 0 │       22ms 662µs │
│ 1 │ 17ms 221µs 792ns │
│ 2 │ 14ms 786µs 708ns │
│ 3 │ 13ms 876µs 834ns │
│ 4 │  13ms 52µs 875ns │
│ 5 │ 11ms 269µs 666ns │
│ 6 │  6ms 942µs 500ns │
╰───┴──────────────────╯
```

##
[random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu)

I got pretty random results out of this benchmark so I decided not to
include it. Not clear why.

# User-Facing Changes
- IR compilation errors may appear even if the user isn't evaluating
with IR.
- IR evaluation can be enabled by setting the `NU_USE_IR` environment
variable to any value.
- New command `view ir` pretty-prints the IR for a block, and `view ir
--json` can be piped into an external tool like [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir).

# Tests + Formatting
All tests are passing with `NU_USE_IR=1`, and I've added some more eval
tests to compare the results for some very core operations. I will
probably want to add some more so we don't have to always check
`NU_USE_IR=1 toolkit test --workspace` on a regular basis.

# After Submitting
- [ ] release notes
- [ ] further documentation of instructions?
- [ ] post-release: publish `nu_plugin_explore_ir`

crates/nu-protocol/src/alias.rs

@@ -1,6 +1,6 @@
 use crate::{
-    ast::{Call, Expression},
-    engine::{Command, CommandType, EngineState, Stack},
+    ast::Expression,
+    engine::{Call, Command, CommandType, EngineState, Stack},
     PipelineData, ShellError, Signature,
 };
 

crates/nu-protocol/src/ast/block.rs

@@ -1,5 +1,5 @@
 use super::Pipeline;
-use crate::{engine::EngineState, OutDest, Signature, Span, Type, VarId};
+use crate::{engine::StateWorkingSet, ir::IrBlock, OutDest, Signature, Span, Type, VarId};
 use serde::{Deserialize, Serialize};
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -8,6 +8,8 @@ pub struct Block {
     pub pipelines: Vec<Pipeline>,
     pub captures: Vec<VarId>,
     pub redirect_env: bool,
+    /// The block compiled to IR instructions. Not available for subexpressions.
+    pub ir_block: Option<IrBlock>,
     pub span: Option<Span>, // None option encodes no span to avoid using test_span()
 }
 
@@ -22,10 +24,10 @@ impl Block {
 
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
         if let Some(first) = self.pipelines.first() {
-            first.pipe_redirection(engine_state)
+            first.pipe_redirection(working_set)
         } else {
             (None, None)
         }
@@ -45,6 +47,7 @@ impl Block {
             pipelines: vec![],
             captures: vec![],
             redirect_env: false,
+            ir_block: None,
             span: None,
         }
     }
@@ -55,6 +58,7 @@ impl Block {
             pipelines: Vec::with_capacity(capacity),
             captures: vec![],
             redirect_env: false,
+            ir_block: None,
             span: None,
         }
     }
@@ -86,6 +90,7 @@ where
             pipelines: pipelines.collect(),
             captures: vec![],
             redirect_env: false,
+            ir_block: None,
             span: None,
         }
     }

crates/nu-protocol/src/ast/expr.rs

@@ -5,7 +5,9 @@ use super::{
     Call, CellPath, Expression, ExternalArgument, FullCellPath, Keyword, MatchPattern, Operator,
     Range, Table, ValueWithUnit,
 };
-use crate::{ast::ImportPattern, engine::EngineState, BlockId, OutDest, Signature, Span, VarId};
+use crate::{
+    ast::ImportPattern, engine::StateWorkingSet, BlockId, OutDest, Signature, Span, VarId,
+};
 
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub enum Expr {
@@ -60,17 +62,17 @@ const _: () = assert!(std::mem::size_of::<Expr>() <= 40);
 impl Expr {
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
         // Usages of `$in` will be wrapped by a `collect` call by the parser,
         // so we do not have to worry about that when considering
         // which of the expressions below may consume pipeline output.
         match self {
-            Expr::Call(call) => engine_state.get_decl(call.decl_id).pipe_redirection(),
-            Expr::Subexpression(block_id) | Expr::Block(block_id) => engine_state
+            Expr::Call(call) => working_set.get_decl(call.decl_id).pipe_redirection(),
+            Expr::Subexpression(block_id) | Expr::Block(block_id) => working_set
                 .get_block(*block_id)
-                .pipe_redirection(engine_state),
-            Expr::FullCellPath(cell_path) => cell_path.head.expr.pipe_redirection(engine_state),
+                .pipe_redirection(working_set),
+            Expr::FullCellPath(cell_path) => cell_path.head.expr.pipe_redirection(working_set),
             Expr::Bool(_)
             | Expr::Int(_)
             | Expr::Float(_)

crates/nu-protocol/src/ast/match_pattern.rs

@@ -5,7 +5,7 @@ use serde::{Deserialize, Serialize};
 #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
 pub struct MatchPattern {
     pub pattern: Pattern,
-    pub guard: Option<Expression>,
+    pub guard: Option<Box<Expression>>,
     pub span: Span,
 }
 
@@ -19,7 +19,9 @@ impl MatchPattern {
 pub enum Pattern {
     Record(Vec<(String, MatchPattern)>),
     List(Vec<MatchPattern>),
-    Value(Expression),
+    // TODO: it would be nice if this didn't depend on AST
+    // maybe const evaluation can get us to a Value instead?
+    Value(Box<Expression>),
     Variable(VarId),
     Or(Vec<MatchPattern>),
     Rest(VarId), // the ..$foo pattern

crates/nu-protocol/src/ast/pipeline.rs

@@ -1,8 +1,4 @@
-use crate::{
-    ast::Expression,
-    engine::{EngineState, StateWorkingSet},
-    OutDest, Span,
-};
+use crate::{ast::Expression, engine::StateWorkingSet, OutDest, Span};
 use serde::{Deserialize, Serialize};
 use std::fmt::Display;
 
@@ -120,9 +116,9 @@ impl PipelineElement {
 
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
-        self.expr.expr.pipe_redirection(engine_state)
+        self.expr.expr.pipe_redirection(working_set)
     }
 }
 
@@ -166,10 +162,10 @@ impl Pipeline {
 
     pub fn pipe_redirection(
         &self,
-        engine_state: &EngineState,
+        working_set: &StateWorkingSet,
     ) -> (Option<OutDest>, Option<OutDest>) {
         if let Some(first) = self.elements.first() {
-            first.pipe_redirection(engine_state)
+            first.pipe_redirection(working_set)
         } else {
             (None, None)
         }

crates/nu-protocol/src/engine/argument.rs

@@ -0,0 +1,124 @@
+use std::sync::Arc;
+
+use crate::{ast::Expression, ir::DataSlice, Span, Value};
+
+/// Represents a fully evaluated argument to a call.
+#[derive(Debug, Clone)]
+pub enum Argument {
+    /// A positional argument
+    Positional {
+        span: Span,
+        val: Value,
+        ast: Option<Arc<Expression>>,
+    },
+    /// A spread argument, e.g. `...$args`
+    Spread {
+        span: Span,
+        vals: Value,
+        ast: Option<Arc<Expression>>,
+    },
+    /// A named argument with no value, e.g. `--flag`
+    Flag {
+        data: Arc<[u8]>,
+        name: DataSlice,
+        short: DataSlice,
+        span: Span,
+    },
+    /// A named argument with a value, e.g. `--flag value` or `--flag=`
+    Named {
+        data: Arc<[u8]>,
+        name: DataSlice,
+        short: DataSlice,
+        span: Span,
+        val: Value,
+        ast: Option<Arc<Expression>>,
+    },
+    /// Information generated by the parser for use by certain keyword commands
+    ParserInfo {
+        data: Arc<[u8]>,
+        name: DataSlice,
+        // TODO: rather than `Expression`, this would probably be best served by a specific enum
+        // type for this purpose.
+        info: Box<Expression>,
+    },
+}
+
+impl Argument {
+    /// The span encompassing the argument's usage within the call, distinct from the span of the
+    /// actual value of the argument.
+    pub fn span(&self) -> Option<Span> {
+        match self {
+            Argument::Positional { span, .. } => Some(*span),
+            Argument::Spread { span, .. } => Some(*span),
+            Argument::Flag { span, .. } => Some(*span),
+            Argument::Named { span, .. } => Some(*span),
+            // Because `ParserInfo` is generated, its span shouldn't be used
+            Argument::ParserInfo { .. } => None,
+        }
+    }
+
+    /// The original AST [`Expression`] for the argument's value. This is not usually available;
+    /// declarations have to opt-in if they require this.
+    pub fn ast_expression(&self) -> Option<&Arc<Expression>> {
+        match self {
+            Argument::Positional { ast, .. } => ast.as_ref(),
+            Argument::Spread { ast, .. } => ast.as_ref(),
+            Argument::Flag { .. } => None,
+            Argument::Named { ast, .. } => ast.as_ref(),
+            Argument::ParserInfo { .. } => None,
+        }
+    }
+}
+
+/// Stores the argument context for calls in IR evaluation.
+#[derive(Debug, Clone)]
+pub struct ArgumentStack {
+    arguments: Vec<Argument>,
+}
+
+impl ArgumentStack {
+    /// Create a new, empty argument stack.
+    pub const fn new() -> Self {
+        ArgumentStack { arguments: vec![] }
+    }
+
+    /// Returns the index of the end of the argument stack. Call and save this before adding
+    /// arguments.
+    pub fn get_base(&self) -> usize {
+        self.arguments.len()
+    }
+
+    /// Calculates the number of arguments past the given [previously retrieved](.get_base) base
+    /// pointer.
+    pub fn get_len(&self, base: usize) -> usize {
+        self.arguments.len().checked_sub(base).unwrap_or_else(|| {
+            panic!(
+                "base ({}) is beyond the end of the arguments stack ({})",
+                base,
+                self.arguments.len()
+            );
+        })
+    }
+
+    /// Push an argument onto the end of the argument stack.
+    pub fn push(&mut self, argument: Argument) {
+        self.arguments.push(argument);
+    }
+
+    /// Clear all of the arguments after the given base index, to prepare for the next frame.
+    pub fn leave_frame(&mut self, base: usize) {
+        self.arguments.truncate(base);
+    }
+
+    /// Get arguments for the frame based on the given [`base`](`.get_base()`) and
+    /// [`len`](`.get_len()`) parameters.
+    pub fn get_args(&self, base: usize, len: usize) -> &[Argument] {
+        &self.arguments[base..(base + len)]
+    }
+
+    /// Move arguments for the frame based on the given [`base`](`.get_base()`) and
+    /// [`len`](`.get_len()`) parameters.
+    pub fn drain_args(&mut self, base: usize, len: usize) -> impl Iterator<Item = Argument> + '_ {
+        self.arguments.drain(base..(base + len))
+    }
+}

crates/nu-protocol/src/engine/call.rs

@@ -0,0 +1,223 @@
+use crate::{
+    ast::{self, Expression},
+    ir, DeclId, FromValue, ShellError, Span, Value,
+};
+
+use super::{EngineState, Stack, StateWorkingSet};
+
+/// This is a HACK to help [`Command`](super::Command) support both the old AST evaluator and the
+/// new IR evaluator at the same time. It should be removed once we are satisfied with the new
+/// evaluator.
+#[derive(Debug, Clone)]
+pub struct Call<'a> {
+    pub head: Span,
+    pub decl_id: DeclId,
+    pub inner: CallImpl<'a>,
+}
+
+#[derive(Debug, Clone)]
+pub enum CallImpl<'a> {
+    AstRef(&'a ast::Call),
+    AstBox(Box<ast::Call>),
+    IrRef(&'a ir::Call),
+    IrBox(Box<ir::Call>),
+}
+
+impl Call<'_> {
+    /// Returns a new AST call with the given span. This is often used by commands that need an
+    /// empty call to pass to a command. It's not easily possible to add anything to this.
+    pub fn new(span: Span) -> Self {
+        // this is using the boxed variant, which isn't so efficient... but this is only temporary
+        // anyway.
+        Call {
+            head: span,
+            decl_id: 0,
+            inner: CallImpl::AstBox(Box::new(ast::Call::new(span))),
+        }
+    }
+
+    /// Convert the `Call` from any lifetime into `'static`, by cloning the data within onto the
+    /// heap.
+    pub fn to_owned(&self) -> Call<'static> {
+        Call {
+            head: self.head,
+            decl_id: self.decl_id,
+            inner: self.inner.to_owned(),
+        }
+    }
+
+    /// Assert that the call is `ast::Call`, and fail with an error if it isn't.
+    ///
+    /// Provided as a stop-gap for commands that can't work with `ir::Call`, or just haven't been
+    /// implemented yet. Eventually these issues should be resolved and then this can be removed.
+    pub fn assert_ast_call(&self) -> Result<&ast::Call, ShellError> {
+        match &self.inner {
+            CallImpl::AstRef(call) => Ok(call),
+            CallImpl::AstBox(call) => Ok(call),
+            _ => Err(ShellError::NushellFailedSpanned {
+                msg: "Can't be used in IR context".into(),
+                label: "this command is not yet supported by IR evaluation".into(),
+                span: self.head,
+            }),
+        }
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn has_flag_const(
+        &self,
+        working_set: &StateWorkingSet,
+        flag_name: &str,
+    ) -> Result<bool, ShellError> {
+        self.assert_ast_call()?
+            .has_flag_const(working_set, flag_name)
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn get_flag_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        name: &str,
+    ) -> Result<Option<T>, ShellError> {
+        self.assert_ast_call()?.get_flag_const(working_set, name)
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn req_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        pos: usize,
+    ) -> Result<T, ShellError> {
+        self.assert_ast_call()?.req_const(working_set, pos)
+    }
+
+    /// FIXME: implementation asserts `ast::Call` and proxies to that
+    pub fn rest_const<T: FromValue>(
+        &self,
+        working_set: &StateWorkingSet,
+        starting_pos: usize,
+    ) -> Result<Vec<T>, ShellError> {
+        self.assert_ast_call()?
+            .rest_const(working_set, starting_pos)
+    }
+
+    /// Returns a span covering the call's arguments.
+    pub fn arguments_span(&self) -> Span {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.arguments_span(),
+            CallImpl::AstBox(call) => call.arguments_span(),
+            CallImpl::IrRef(call) => call.arguments_span(),
+            CallImpl::IrBox(call) => call.arguments_span(),
+        }
+    }
+
+    /// Returns a span covering the whole call.
+    pub fn span(&self) -> Span {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.span(),
+            CallImpl::AstBox(call) => call.span(),
+            CallImpl::IrRef(call) => call.span(),
+            CallImpl::IrBox(call) => call.span(),
+        }
+    }
+
+    /// Get a parser info argument by name.
+    pub fn get_parser_info<'a>(&'a self, stack: &'a Stack, name: &str) -> Option<&'a Expression> {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.get_parser_info(name),
+            CallImpl::AstBox(call) => call.get_parser_info(name),
+            CallImpl::IrRef(call) => call.get_parser_info(stack, name),
+            CallImpl::IrBox(call) => call.get_parser_info(stack, name),
+        }
+    }
+
+    /// Evaluator-agnostic implementation of `rest_iter_flattened()`. Evaluates or gets all of the
+    /// positional and spread arguments, flattens spreads, and then returns one list of values.
+    pub fn rest_iter_flattened(
+        &self,
+        engine_state: &EngineState,
+        stack: &mut Stack,
+        eval_expression: fn(
+            &EngineState,
+            &mut Stack,
+            &ast::Expression,
+        ) -> Result<Value, ShellError>,
+        starting_pos: usize,
+    ) -> Result<Vec<Value>, ShellError> {
+        fn by_ast(
+            call: &ast::Call,
+            engine_state: &EngineState,
+            stack: &mut Stack,
+            eval_expression: fn(
+                &EngineState,
+                &mut Stack,
+                &ast::Expression,
+            ) -> Result<Value, ShellError>,
+            starting_pos: usize,
+        ) -> Result<Vec<Value>, ShellError> {
+            call.rest_iter_flattened(starting_pos, |expr| {
+                eval_expression(engine_state, stack, expr)
+            })
+        }
+
+        fn by_ir(
+            call: &ir::Call,
+            stack: &Stack,
+            starting_pos: usize,
+        ) -> Result<Vec<Value>, ShellError> {
+            call.rest_iter_flattened(stack, starting_pos)
+        }
+
+        match &self.inner {
+            CallImpl::AstRef(call) => {
+                by_ast(call, engine_state, stack, eval_expression, starting_pos)
+            }
+            CallImpl::AstBox(call) => {
+                by_ast(call, engine_state, stack, eval_expression, starting_pos)
+            }
+            CallImpl::IrRef(call) => by_ir(call, stack, starting_pos),
+            CallImpl::IrBox(call) => by_ir(call, stack, starting_pos),
+        }
+    }
+
+    /// Get the original AST expression for a positional argument. Does not usually work for IR
+    /// unless the decl specified `requires_ast_for_arguments()`
+    pub fn positional_nth<'a>(&'a self, stack: &'a Stack, index: usize) -> Option<&'a Expression> {
+        match &self.inner {
+            CallImpl::AstRef(call) => call.positional_nth(index),
+            CallImpl::AstBox(call) => call.positional_nth(index),
+            CallImpl::IrRef(call) => call.positional_ast(stack, index).map(|arc| arc.as_ref()),
+            CallImpl::IrBox(call) => call.positional_ast(stack, index).map(|arc| arc.as_ref()),
+        }
+    }
+}
+
+impl CallImpl<'_> {
+    pub fn to_owned(&self) -> CallImpl<'static> {
+        match self {
+            CallImpl::AstRef(call) => CallImpl::AstBox(Box::new((*call).clone())),
+            CallImpl::AstBox(call) => CallImpl::AstBox(call.clone()),
+            CallImpl::IrRef(call) => CallImpl::IrBox(Box::new((*call).clone())),
+            CallImpl::IrBox(call) => CallImpl::IrBox(call.clone()),
+        }
+    }
+}
+
+impl<'a> From<&'a ast::Call> for Call<'a> {
+    fn from(call: &'a ast::Call) -> Self {
+        Call {
+            head: call.head,
+            decl_id: call.decl_id,
+            inner: CallImpl::AstRef(call),
+        }
+    }
+}
+
+impl<'a> From<&'a ir::Call> for Call<'a> {
+    fn from(call: &'a ir::Call) -> Self {
+        Call {
+            head: call.head,
+            decl_id: call.decl_id,
+            inner: CallImpl::IrRef(call),
+        }
+    }
+}

crates/nu-protocol/src/engine/command.rs

@@ -1,5 +1,5 @@
 use super::{EngineState, Stack, StateWorkingSet};
-use crate::{ast::Call, Alias, BlockId, Example, OutDest, PipelineData, ShellError, Signature};
+use crate::{engine::Call, Alias, BlockId, Example, OutDest, PipelineData, ShellError, Signature};
 use std::fmt::Display;
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -124,6 +124,12 @@ pub trait Command: Send + Sync + CommandClone {
     fn pipe_redirection(&self) -> (Option<OutDest>, Option<OutDest>) {
         (None, None)
     }
+
+    /// Return true if the AST nodes for the arguments are required for IR evaluation. This is
+    /// currently inefficient so is not generally done.
+    fn requires_ast_for_arguments(&self) -> bool {
+        false
+    }
 }
 
 pub trait CommandClone {

crates/nu-protocol/src/engine/error_handler.rs

@@ -0,0 +1,55 @@
+use crate::RegId;
+
+/// Describes an error handler stored during IR evaluation.
+#[derive(Debug, Clone, Copy)]
+pub struct ErrorHandler {
+    /// Instruction index within the block that will handle the error
+    pub handler_index: usize,
+    /// Register to put the error information into, when an error occurs
+    pub error_register: Option<RegId>,
+}
+
+/// Keeps track of error handlers pushed during evaluation of an IR block.
+#[derive(Debug, Clone)]
+pub struct ErrorHandlerStack {
+    handlers: Vec<ErrorHandler>,
+}
+
+impl ErrorHandlerStack {
+    pub const fn new() -> ErrorHandlerStack {
+        ErrorHandlerStack { handlers: vec![] }
+    }
+
+    /// Get the current base of the stack, which establishes a frame.
+    pub fn get_base(&self) -> usize {
+        self.handlers.len()
+    }
+
+    /// Push a new error handler onto the stack.
+    pub fn push(&mut self, handler: ErrorHandler) {
+        self.handlers.push(handler);
+    }
+
+    /// Try to pop an error handler from the stack. Won't go below `base`, to avoid retrieving a
+    /// handler belonging to a parent frame.
+    pub fn pop(&mut self, base: usize) -> Option<ErrorHandler> {
+        if self.handlers.len() > base {
+            self.handlers.pop()
+        } else {
+            None
+        }
+    }
+
+    /// Reset the stack to the state it was in at the beginning of the frame, in preparation to
+    /// return control to the parent frame.
+    pub fn leave_frame(&mut self, base: usize) {
+        if self.handlers.len() >= base {
+            self.handlers.truncate(base);
+        } else {
+            panic!(
+                "ErrorHandlerStack bug: tried to leave frame at {base}, but current base is {}",
+                self.get_base()
+            )
+        }
+    }
+}

crates/nu-protocol/src/engine/mod.rs

@@ -1,8 +1,11 @@
+mod argument;
 mod cached_file;
+mod call;
 mod call_info;
 mod capture_block;
 mod command;
 mod engine_state;
+mod error_handler;
 mod overlay;
 mod pattern_match;
 mod stack;
@@ -14,10 +17,13 @@ mod variable;
 
 pub use cached_file::CachedFile;
 
+pub use argument::*;
+pub use call::*;
 pub use call_info::*;
 pub use capture_block::*;
 pub use command::*;
 pub use engine_state::*;
+pub use error_handler::*;
 pub use overlay::*;
 pub use pattern_match::*;
 pub use stack::*;

crates/nu-protocol/src/engine/stack.rs

@@ -1,7 +1,7 @@
 use crate::{
     engine::{
-        EngineState, Redirection, StackCallArgGuard, StackCaptureGuard, StackIoGuard, StackOutDest,
-        DEFAULT_OVERLAY_NAME,
+        ArgumentStack, EngineState, ErrorHandlerStack, Redirection, StackCallArgGuard,
+        StackCaptureGuard, StackIoGuard, StackOutDest, DEFAULT_OVERLAY_NAME,
     },
     OutDest, ShellError, Span, Value, VarId, ENV_VARIABLE_ID, NU_VARIABLE_ID,
 };
@@ -41,6 +41,12 @@ pub struct Stack {
     pub env_hidden: HashMap<String, HashSet<String>>,
     /// List of active overlays
     pub active_overlays: Vec<String>,
+    /// Argument stack for IR evaluation
+    pub arguments: ArgumentStack,
+    /// Error handler stack for IR evaluation
+    pub error_handlers: ErrorHandlerStack,
+    /// Set true to always use IR mode
+    pub use_ir: bool,
     pub recursion_count: u64,
     pub parent_stack: Option<Arc<Stack>>,
     /// Variables that have been deleted (this is used to hide values from parent stack lookups)
@@ -68,6 +74,9 @@ impl Stack {
             env_vars: Vec::new(),
             env_hidden: HashMap::new(),
             active_overlays: vec![DEFAULT_OVERLAY_NAME.to_string()],
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: false,
             recursion_count: 0,
             parent_stack: None,
             parent_deletions: vec![],
@@ -85,6 +94,9 @@ impl Stack {
             env_vars: parent.env_vars.clone(),
             env_hidden: parent.env_hidden.clone(),
             active_overlays: parent.active_overlays.clone(),
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: parent.use_ir,
             recursion_count: parent.recursion_count,
             vars: vec![],
             parent_deletions: vec![],
@@ -254,6 +266,9 @@ impl Stack {
             env_vars,
             env_hidden: self.env_hidden.clone(),
             active_overlays: self.active_overlays.clone(),
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: self.use_ir,
             recursion_count: self.recursion_count,
             parent_stack: None,
             parent_deletions: vec![],
@@ -284,6 +299,9 @@ impl Stack {
             env_vars,
             env_hidden: self.env_hidden.clone(),
             active_overlays: self.active_overlays.clone(),
+            arguments: ArgumentStack::new(),
+            error_handlers: ErrorHandlerStack::new(),
+            use_ir: self.use_ir,
             recursion_count: self.recursion_count,
             parent_stack: None,
             parent_deletions: vec![],

crates/nu-protocol/src/engine/state_working_set.rs

@@ -4,8 +4,8 @@ use crate::{
         usage::build_usage, CachedFile, Command, CommandType, EngineState, OverlayFrame,
         StateDelta, Variable, VirtualPath, Visibility,
     },
-    BlockId, Category, Config, DeclId, FileId, GetSpan, Module, ModuleId, ParseError, ParseWarning,
-    Span, SpanId, Type, Value, VarId, VirtualPathId,
+    BlockId, Category, CompileError, Config, DeclId, FileId, GetSpan, Module, ModuleId, ParseError,
+    ParseWarning, Span, SpanId, Type, Value, VarId, VirtualPathId,
 };
 use core::panic;
 use std::{
@@ -31,6 +31,7 @@ pub struct StateWorkingSet<'a> {
     pub search_predecls: bool,
     pub parse_errors: Vec<ParseError>,
     pub parse_warnings: Vec<ParseWarning>,
+    pub compile_errors: Vec<CompileError>,
 }
 
 impl<'a> StateWorkingSet<'a> {
@@ -50,6 +51,7 @@ impl<'a> StateWorkingSet<'a> {
             search_predecls: true,
             parse_errors: vec![],
             parse_warnings: vec![],
+            compile_errors: vec![],
         }
     }
 
@@ -260,6 +262,12 @@ impl<'a> StateWorkingSet<'a> {
     }
 
     pub fn add_block(&mut self, block: Arc<Block>) -> BlockId {
+        log::trace!(
+            "block id={} added, has IR = {:?}",
+            self.num_blocks(),
+            block.ir_block.is_some()
+        );
+
         self.delta.blocks.push(block);
 
         self.num_blocks() - 1

crates/nu-protocol/src/errors/cli_error.rs

@@ -107,4 +107,8 @@ impl<'src> miette::Diagnostic for CliError<'src> {
     fn related<'a>(&'a self) -> Option<Box<dyn Iterator<Item = &'a dyn miette::Diagnostic> + 'a>> {
         self.0.related()
     }
+
+    fn diagnostic_source(&self) -> Option<&dyn miette::Diagnostic> {
+        self.0.diagnostic_source()
+    }
 }

crates/nu-protocol/src/errors/compile_error.rs

@@ -0,0 +1,238 @@
+use crate::{RegId, Span};
+use miette::Diagnostic;
+use serde::{Deserialize, Serialize};
+use thiserror::Error;
+
+/// An internal compiler error, generally means a Nushell bug rather than an issue with user error
+/// since parsing and typechecking has already passed.
+#[derive(Debug, Clone, Error, Diagnostic, PartialEq, Serialize, Deserialize)]
+pub enum CompileError {
+    #[error("Register overflow.")]
+    #[diagnostic(code(nu::compile::register_overflow))]
+    RegisterOverflow {
+        #[label("the code being compiled is probably too large")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Register {reg_id} was uninitialized when used, possibly reused.")]
+    #[diagnostic(
+        code(nu::compile::register_uninitialized),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new\nfrom: {caller}"),
+    )]
+    RegisterUninitialized { reg_id: RegId, caller: String },
+
+    #[error("Register {reg_id} was uninitialized when used, possibly reused.")]
+    #[diagnostic(
+        code(nu::compile::register_uninitialized),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new\nfrom: {caller}"),
+    )]
+    RegisterUninitializedWhilePushingInstruction {
+        reg_id: RegId,
+        caller: String,
+        instruction: String,
+        #[label("while adding this instruction: {instruction}")]
+        span: Span,
+    },
+
+    #[error("Block contains too much string data: maximum 4 GiB exceeded.")]
+    #[diagnostic(
+        code(nu::compile::data_overflow),
+        help("try loading the string data from a file instead")
+    )]
+    DataOverflow {
+        #[label("while compiling this block")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Block contains too many files.")]
+    #[diagnostic(
+        code(nu::compile::register_overflow),
+        help("try using fewer file redirections")
+    )]
+    FileOverflow {
+        #[label("while compiling this block")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Invalid redirect mode: File should not be specified by commands.")]
+    #[diagnostic(
+        code(nu::compile::invalid_redirect_mode),
+        help("this is a command bug. Please report it at https://github.com/nushell/nushell/issues/new")
+    )]
+    InvalidRedirectMode {
+        #[label("while compiling this expression")]
+        span: Span,
+    },
+
+    #[error("Encountered garbage, likely due to parse error.")]
+    #[diagnostic(code(nu::compile::garbage))]
+    Garbage {
+        #[label("garbage found here")]
+        span: Span,
+    },
+
+    #[error("Unsupported operator expression.")]
+    #[diagnostic(code(nu::compile::unsupported_operator_expression))]
+    UnsupportedOperatorExpression {
+        #[label("this expression is in operator position but is not an operator")]
+        span: Span,
+    },
+
+    #[error("Attempted access of $env by integer path.")]
+    #[diagnostic(code(nu::compile::access_env_by_int))]
+    AccessEnvByInt {
+        #[label("$env keys should be strings")]
+        span: Span,
+    },
+
+    #[error("Encountered invalid `{keyword}` keyword call.")]
+    #[diagnostic(code(nu::compile::invalid_keyword_call))]
+    InvalidKeywordCall {
+        keyword: String,
+        #[label("this call is not properly formed")]
+        span: Span,
+    },
+
+    #[error("Attempted to set branch target of non-branch instruction.")]
+    #[diagnostic(
+        code(nu::compile::set_branch_target_of_non_branch_instruction),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new"),
+    )]
+    SetBranchTargetOfNonBranchInstruction {
+        instruction: String,
+        #[label("tried to modify: {instruction}")]
+        span: Span,
+    },
+
+    /// You're trying to run an unsupported external command.
+    ///
+    /// ## Resolution
+    ///
+    /// Make sure there's an appropriate `run-external` declaration for this external command.
+    #[error("External calls are not supported.")]
+    #[diagnostic(
+        code(nu::compile::run_external_not_found),
+        help("`run-external` was not found in scope")
+    )]
+    RunExternalNotFound {
+        #[label("can't be run in this context")]
+        span: Span,
+    },
+
+    /// Invalid assignment left-hand side
+    ///
+    /// ## Resolution
+    ///
+    /// Assignment requires that you assign to a variable or variable cell path.
+    #[error("Assignment operations require a variable.")]
+    #[diagnostic(
+        code(nu::compile::assignment_requires_variable),
+        help("try assigning to a variable or a cell path of a variable")
+    )]
+    AssignmentRequiresVar {
+        #[label("needs to be a variable")]
+        span: Span,
+    },
+
+    /// Invalid assignment left-hand side
+    ///
+    /// ## Resolution
+    ///
+    /// Assignment requires that you assign to a mutable variable or cell path.
+    #[error("Assignment to an immutable variable.")]
+    #[diagnostic(
+        code(nu::compile::assignment_requires_mutable_variable),
+        help("declare the variable with `mut`, or shadow it again with `let`")
+    )]
+    AssignmentRequiresMutableVar {
+        #[label("needs to be a mutable variable")]
+        span: Span,
+    },
+
+    /// This environment variable cannot be set manually.
+    ///
+    /// ## Resolution
+    ///
+    /// This environment variable is set automatically by Nushell and cannot not be set manually.
+    #[error("{envvar_name} cannot be set manually.")]
+    #[diagnostic(
+        code(nu::compile::automatic_env_var_set_manually),
+        help(
+            r#"The environment variable '{envvar_name}' is set automatically by Nushell and cannot be set manually."#
+        )
+    )]
+    AutomaticEnvVarSetManually {
+        envvar_name: String,
+        #[label("cannot set '{envvar_name}' manually")]
+        span: Span,
+    },
+
+    /// It is not possible to replace the entire environment at once
+    ///
+    /// ## Resolution
+    ///
+    /// Setting the entire environment is not allowed. Change environment variables individually
+    /// instead.
+    #[error("Cannot replace environment.")]
+    #[diagnostic(
+        code(nu::compile::cannot_replace_env),
+        help("Assigning a value to '$env' is not allowed.")
+    )]
+    CannotReplaceEnv {
+        #[label("setting '$env' not allowed")]
+        span: Span,
+    },
+
+    #[error("Unexpected expression.")]
+    #[diagnostic(code(nu::compile::unexpected_expression))]
+    UnexpectedExpression {
+        expr_name: String,
+        #[label("{expr_name} is not allowed in this context")]
+        span: Span,
+    },
+
+    #[error("Missing required declaration: `{decl_name}`")]
+    #[diagnostic(code(nu::compile::missing_required_declaration))]
+    MissingRequiredDeclaration {
+        decl_name: String,
+        #[label("`{decl_name}` must be in scope to compile this expression")]
+        span: Span,
+    },
+
+    #[error("Invalid literal")]
+    #[diagnostic(code(nu::compile::invalid_literal))]
+    InvalidLiteral {
+        msg: String,
+        #[label("{msg}")]
+        span: Span,
+    },
+
+    #[error("{msg}")]
+    #[diagnostic(code(nu::compile::not_in_a_loop))]
+    NotInALoop {
+        msg: String,
+        #[label("can't be used outside of a loop")]
+        span: Option<Span>,
+    },
+
+    #[error("Incoherent loop state: the loop that ended was not the one we were expecting.")]
+    #[diagnostic(
+        code(nu::compile::incoherent_loop_state),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new"),
+    )]
+    IncoherentLoopState {
+        #[label("while compiling this block")]
+        block_span: Option<Span>,
+    },
+
+    #[error("Undefined label `{label_id}`.")]
+    #[diagnostic(
+        code(nu::compile::undefined_label),
+        help("this is a compiler bug. Please report it at https://github.com/nushell/nushell/issues/new"),
+    )]
+    UndefinedLabel {
+        label_id: usize,
+        #[label("label was used while compiling this code")]
+        span: Option<Span>,
+    },
+}

crates/nu-protocol/src/errors/mod.rs

@@ -1,10 +1,12 @@
 pub mod cli_error;
+mod compile_error;
 mod labeled_error;
 mod parse_error;
 mod parse_warning;
 mod shell_error;
 
 pub use cli_error::{format_error, report_error, report_error_new};
+pub use compile_error::CompileError;
 pub use labeled_error::{ErrorLabel, LabeledError};
 pub use parse_error::{DidYouMean, ParseError};
 pub use parse_warning::ParseWarning;

crates/nu-protocol/src/errors/shell_error.rs

@@ -1376,6 +1376,23 @@ On Windows, this would be %USERPROFILE%\AppData\Roaming"#
         help("Set XDG_CONFIG_HOME to an absolute path, or set it to an empty string to ignore it")
     )]
     InvalidXdgConfig { xdg: String, default: String },
+
+    /// An unexpected error occurred during IR evaluation.
+    ///
+    /// ## Resolution
+    ///
+    /// This is most likely a correctness issue with the IR compiler or evaluator. Please file a
+    /// bug with the minimum code needed to reproduce the issue, if possible.
+    #[error("IR evaluation error: {msg}")]
+    #[diagnostic(
+        code(nu::shell::ir_eval_error),
+        help("this is a bug, please report it at https://github.com/nushell/nushell/issues/new along with the code you were running if able")
+    )]
+    IrEvalError {
+        msg: String,
+        #[label = "while running this code"]
+        span: Option<Span>,
+    },
 }
 
 // TODO: Implement as From trait

crates/nu-protocol/src/eval_const.rs

@@ -307,7 +307,7 @@ fn eval_const_call(
         return Err(ShellError::NotAConstHelp { span: call.head });
     }
 
-    decl.run_const(working_set, call, input)
+    decl.run_const(working_set, &call.into(), input)
 }
 
 pub fn eval_const_subexpression(

crates/nu-protocol/src/id.rs

@@ -7,5 +7,19 @@ pub type ModuleId = usize;
 pub type OverlayId = usize;
 pub type FileId = usize;
 pub type VirtualPathId = usize;
+
 #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
 pub struct SpanId(pub usize); // more robust ID style used in the new parser
+
+/// An ID for an [IR](crate::ir) register. `%n` is a common shorthand for `RegId(n)`.
+///
+/// Note: `%0` is allocated with the block input at the beginning of a compiled block.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+#[repr(transparent)]
+pub struct RegId(pub u32);
+
+impl std::fmt::Display for RegId {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "%{}", self.0)
+    }
+}

crates/nu-protocol/src/ir/call.rs

@@ -0,0 +1,351 @@
+use std::sync::Arc;
+
+use crate::{
+    ast::Expression,
+    engine::{self, Argument, Stack},
+    DeclId, ShellError, Span, Spanned, Value,
+};
+
+use super::DataSlice;
+
+/// Contains the information for a call being made to a declared command.
+#[derive(Debug, Clone)]
+pub struct Call {
+    /// The declaration ID of the command to be invoked.
+    pub decl_id: DeclId,
+    /// The span encompassing the command name, before the arguments.
+    pub head: Span,
+    /// The span encompassing the command name and all arguments.
+    pub span: Span,
+    /// The base index of the arguments for this call within the
+    /// [argument stack](crate::engine::ArgumentStack).
+    pub args_base: usize,
+    /// The number of [`Argument`]s for the call. Note that this just counts the number of
+    /// `Argument` entries on the stack, and has nothing to do with the actual number of positional
+    /// or spread arguments.
+    pub args_len: usize,
+}
+
+impl Call {
+    /// Build a new call with arguments.
+    pub fn build(decl_id: DeclId, head: Span) -> CallBuilder {
+        CallBuilder {
+            inner: Call {
+                decl_id,
+                head,
+                span: head,
+                args_base: 0,
+                args_len: 0,
+            },
+        }
+    }
+
+    /// Get the arguments for this call from the arguments stack.
+    pub fn arguments<'a>(&self, stack: &'a Stack) -> &'a [Argument] {
+        stack.arguments.get_args(self.args_base, self.args_len)
+    }
+
+    /// The span encompassing the arguments
+    ///
+    /// If there are no arguments the span covers where the first argument would exist
+    ///
+    /// If there are one or more arguments the span encompasses the start of the first argument to
+    /// end of the last argument
+    pub fn arguments_span(&self) -> Span {
+        let past = self.head.past();
+        Span::new(past.start, self.span.end)
+    }
+
+    /// The number of named arguments, with or without values.
+    pub fn named_len(&self, stack: &Stack) -> usize {
+        self.arguments(stack)
+            .iter()
+            .filter(|arg| matches!(arg, Argument::Named { .. } | Argument::Flag { .. }))
+            .count()
+    }
+
+    /// Iterate through named arguments, with or without values.
+    pub fn named_iter<'a>(
+        &'a self,
+        stack: &'a Stack,
+    ) -> impl Iterator<Item = (Spanned<&'a str>, Option<&'a Value>)> + 'a {
+        self.arguments(stack).iter().filter_map(
+            |arg: &Argument| -> Option<(Spanned<&str>, Option<&Value>)> {
+                match arg {
+                    Argument::Flag {
+                        data, name, span, ..
+                    } => Some((
+                        Spanned {
+                            item: std::str::from_utf8(&data[*name]).expect("invalid arg name"),
+                            span: *span,
+                        },
+                        None,
+                    )),
+                    Argument::Named {
+                        data,
+                        name,
+                        span,
+                        val,
+                        ..
+                    } => Some((
+                        Spanned {
+                            item: std::str::from_utf8(&data[*name]).expect("invalid arg name"),
+                            span: *span,
+                        },
+                        Some(val),
+                    )),
+                    _ => None,
+                }
+            },
+        )
+    }
+
+    /// Get a named argument's value by name. Returns [`None`] for named arguments with no value as
+    /// well.
+    pub fn get_named_arg<'a>(&self, stack: &'a Stack, flag_name: &str) -> Option<&'a Value> {
+        // Optimized to avoid str::from_utf8()
+        self.arguments(stack)
+            .iter()
+            .find_map(|arg: &Argument| -> Option<Option<&Value>> {
+                match arg {
+                    Argument::Flag { data, name, .. } if &data[*name] == flag_name.as_bytes() => {
+                        Some(None)
+                    }
+                    Argument::Named {
+                        data, name, val, ..
+                    } if &data[*name] == flag_name.as_bytes() => Some(Some(val)),
+                    _ => None,
+                }
+            })
+            .flatten()
+    }
+
+    /// The number of positional arguments, excluding spread arguments.
+    pub fn positional_len(&self, stack: &Stack) -> usize {
+        self.arguments(stack)
+            .iter()
+            .filter(|arg| matches!(arg, Argument::Positional { .. }))
+            .count()
+    }
+
+    /// Iterate through positional arguments. Does not include spread arguments.
+    pub fn positional_iter<'a>(&self, stack: &'a Stack) -> impl Iterator<Item = &'a Value> {
+        self.arguments(stack).iter().filter_map(|arg| match arg {
+            Argument::Positional { val, .. } => Some(val),
+            _ => None,
+        })
+    }
+
+    /// Get a positional argument by index. Does not include spread arguments.
+    pub fn positional_nth<'a>(&self, stack: &'a Stack, index: usize) -> Option<&'a Value> {
+        self.positional_iter(stack).nth(index)
+    }
+
+    /// Get the AST node for a positional argument by index. Not usually available unless the decl
+    /// required it.
+    pub fn positional_ast<'a>(
+        &self,
+        stack: &'a Stack,
+        index: usize,
+    ) -> Option<&'a Arc<Expression>> {
+        self.arguments(stack)
+            .iter()
+            .filter_map(|arg| match arg {
+                Argument::Positional { ast, .. } => Some(ast),
+                _ => None,
+            })
+            .nth(index)
+            .and_then(|option| option.as_ref())
+    }
+
+    /// Returns every argument to the rest parameter, as well as whether each argument
+    /// is spread or a normal positional argument (true for spread, false for normal)
+    pub fn rest_iter<'a>(
+        &self,
+        stack: &'a Stack,
+        start: usize,
+    ) -> impl Iterator<Item = (&'a Value, bool)> + 'a {
+        self.arguments(stack)
+            .iter()
+            .filter_map(|arg| match arg {
+                Argument::Positional { val, .. } => Some((val, false)),
+                Argument::Spread { vals, .. } => Some((vals, true)),
+                _ => None,
+            })
+            .skip(start)
+    }
+
+    /// Returns all of the positional arguments including and after `start`, with spread arguments
+    /// flattened into a single `Vec`.
+    pub fn rest_iter_flattened(
+        &self,
+        stack: &Stack,
+        start: usize,
+    ) -> Result<Vec<Value>, ShellError> {
+        let mut acc = vec![];
+        for (rest_val, spread) in self.rest_iter(stack, start) {
+            if spread {
+                match rest_val {
+                    Value::List { vals, .. } => acc.extend(vals.iter().cloned()),
+                    Value::Error { error, .. } => return Err(ShellError::clone(error)),
+                    _ => {
+                        return Err(ShellError::CannotSpreadAsList {
+                            span: rest_val.span(),
+                        })
+                    }
+                }
+            } else {
+                acc.push(rest_val.clone());
+            }
+        }
+        Ok(acc)
+    }
+
+    /// Get a parser info argument by name.
+    pub fn get_parser_info<'a>(&self, stack: &'a Stack, name: &str) -> Option<&'a Expression> {
+        self.arguments(stack)
+            .iter()
+            .find_map(|argument| match argument {
+                Argument::ParserInfo {
+                    data,
+                    name: name_slice,
+                    info: expr,
+                } if &data[*name_slice] == name.as_bytes() => Some(expr.as_ref()),
+                _ => None,
+            })
+    }
+
+    /// Returns a span encompassing the entire call.
+    pub fn span(&self) -> Span {
+        self.span
+    }
+
+    /// Resets the [`Stack`] to its state before the call was made.
+    pub fn leave(&self, stack: &mut Stack) {
+        stack.arguments.leave_frame(self.args_base);
+    }
+}
+
+/// Utility struct for building a [`Call`] with arguments on the [`Stack`].
+pub struct CallBuilder {
+    inner: Call,
+}
+
+impl CallBuilder {
+    /// Add an argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_argument(&mut self, stack: &mut Stack, argument: Argument) -> &mut Self {
+        if self.inner.args_len == 0 {
+            self.inner.args_base = stack.arguments.get_base();
+        }
+        self.inner.args_len += 1;
+        if let Some(span) = argument.span() {
+            self.inner.span = self.inner.span.append(span);
+        }
+        stack.arguments.push(argument);
+        self
+    }
+
+    /// Add a positional argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_positional(&mut self, stack: &mut Stack, span: Span, val: Value) -> &mut Self {
+        self.add_argument(
+            stack,
+            Argument::Positional {
+                span,
+                val,
+                ast: None,
+            },
+        )
+    }
+
+    /// Add a spread argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_spread(&mut self, stack: &mut Stack, span: Span, vals: Value) -> &mut Self {
+        self.add_argument(
+            stack,
+            Argument::Spread {
+                span,
+                vals,
+                ast: None,
+            },
+        )
+    }
+
+    /// Add a flag (no-value named) argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_flag(
+        &mut self,
+        stack: &mut Stack,
+        name: impl AsRef<str>,
+        short: impl AsRef<str>,
+        span: Span,
+    ) -> &mut Self {
+        let (data, name, short) = data_from_name_and_short(name.as_ref(), short.as_ref());
+        self.add_argument(
+            stack,
+            Argument::Flag {
+                data,
+                name,
+                short,
+                span,
+            },
+        )
+    }
+
+    /// Add a named argument to the [`Stack`] and reference it from the [`Call`].
+    pub fn add_named(
+        &mut self,
+        stack: &mut Stack,
+        name: impl AsRef<str>,
+        short: impl AsRef<str>,
+        span: Span,
+        val: Value,
+    ) -> &mut Self {
+        let (data, name, short) = data_from_name_and_short(name.as_ref(), short.as_ref());
+        self.add_argument(
+            stack,
+            Argument::Named {
+                data,
+                name,
+                short,
+                span,
+                val,
+                ast: None,
+            },
+        )
+    }
+
+    /// Produce the finished [`Call`] from the builder.
+    ///
+    /// The call should be entered / run before any other calls are constructed, because the
+    /// argument stack will be reset when they exit.
+    pub fn finish(&self) -> Call {
+        self.inner.clone()
+    }
+
+    /// Run a closure with the [`Call`] as an [`engine::Call`] reference, and then clean up the
+    /// arguments that were added to the [`Stack`] after.
+    ///
+    /// For convenience. Calls [`Call::leave`] after the closure ends.
+    pub fn with<T>(
+        self,
+        stack: &mut Stack,
+        f: impl FnOnce(&mut Stack, &engine::Call<'_>) -> T,
+    ) -> T {
+        let call = engine::Call::from(&self.inner);
+        let result = f(stack, &call);
+        self.inner.leave(stack);
+        result
+    }
+}
+
+fn data_from_name_and_short(name: &str, short: &str) -> (Arc<[u8]>, DataSlice, DataSlice) {
+    let data: Vec<u8> = name.bytes().chain(short.bytes()).collect();
+    let data: Arc<[u8]> = data.into();
+    let name = DataSlice {
+        start: 0,
+        len: name.len().try_into().expect("flag name too big"),
+    };
+    let short = DataSlice {
+        start: name.start.checked_add(name.len).expect("flag name too big"),
+        len: short.len().try_into().expect("flag short name too big"),
+    };
+    (data, name, short)
+}

crates/nu-protocol/src/ir/display.rs

@@ -0,0 +1,452 @@
+use std::fmt;
+
+use crate::{ast::Pattern, engine::EngineState, DeclId, VarId};
+
+use super::{DataSlice, Instruction, IrBlock, Literal, RedirectMode};
+
+pub struct FmtIrBlock<'a> {
+    pub(super) engine_state: &'a EngineState,
+    pub(super) ir_block: &'a IrBlock,
+}
+
+impl<'a> fmt::Display for FmtIrBlock<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        let plural = |count| if count == 1 { "" } else { "s" };
+        writeln!(
+            f,
+            "# {} register{}, {} instruction{}, {} byte{} of data",
+            self.ir_block.register_count,
+            plural(self.ir_block.register_count as usize),
+            self.ir_block.instructions.len(),
+            plural(self.ir_block.instructions.len()),
+            self.ir_block.data.len(),
+            plural(self.ir_block.data.len()),
+        )?;
+        if self.ir_block.file_count > 0 {
+            writeln!(
+                f,
+                "# {} file{} used for redirection",
+                self.ir_block.file_count,
+                plural(self.ir_block.file_count as usize)
+            )?;
+        }
+        for (index, instruction) in self.ir_block.instructions.iter().enumerate() {
+            let formatted = format!(
+                "{:-4}: {}",
+                index,
+                FmtInstruction {
+                    engine_state: self.engine_state,
+                    instruction,
+                    data: &self.ir_block.data,
+                }
+            );
+            let comment = &self.ir_block.comments[index];
+            if comment.is_empty() {
+                writeln!(f, "{formatted}")?;
+            } else {
+                writeln!(f, "{formatted:40} # {comment}")?;
+            }
+        }
+        Ok(())
+    }
+}
+
+pub struct FmtInstruction<'a> {
+    pub(super) engine_state: &'a EngineState,
+    pub(super) instruction: &'a Instruction,
+    pub(super) data: &'a [u8],
+}
+
+impl<'a> fmt::Display for FmtInstruction<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        const WIDTH: usize = 22;
+
+        match self.instruction {
+            Instruction::Unreachable => {
+                write!(f, "{:WIDTH$}", "unreachable")
+            }
+            Instruction::LoadLiteral { dst, lit } => {
+                let lit = FmtLiteral {
+                    literal: lit,
+                    data: self.data,
+                };
+                write!(f, "{:WIDTH$} {dst}, {lit}", "load-literal")
+            }
+            Instruction::LoadValue { dst, val } => {
+                let val = val.to_debug_string();
+                write!(f, "{:WIDTH$} {dst}, {val}", "load-value")
+            }
+            Instruction::Move { dst, src } => {
+                write!(f, "{:WIDTH$} {dst}, {src}", "move")
+            }
+            Instruction::Clone { dst, src } => {
+                write!(f, "{:WIDTH$} {dst}, {src}", "clone")
+            }
+            Instruction::Collect { src_dst } => {
+                write!(f, "{:WIDTH$} {src_dst}", "collect")
+            }
+            Instruction::Span { src_dst } => {
+                write!(f, "{:WIDTH$} {src_dst}", "span")
+            }
+            Instruction::Drop { src } => {
+                write!(f, "{:WIDTH$} {src}", "drop")
+            }
+            Instruction::Drain { src } => {
+                write!(f, "{:WIDTH$} {src}", "drain")
+            }
+            Instruction::LoadVariable { dst, var_id } => {
+                let var = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "{:WIDTH$} {dst}, {var}", "load-variable")
+            }
+            Instruction::StoreVariable { var_id, src } => {
+                let var = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "{:WIDTH$} {var}, {src}", "store-variable")
+            }
+            Instruction::LoadEnv { dst, key } => {
+                let key = FmtData(self.data, *key);
+                write!(f, "{:WIDTH$} {dst}, {key}", "load-env")
+            }
+            Instruction::LoadEnvOpt { dst, key } => {
+                let key = FmtData(self.data, *key);
+                write!(f, "{:WIDTH$} {dst}, {key}", "load-env-opt")
+            }
+            Instruction::StoreEnv { key, src } => {
+                let key = FmtData(self.data, *key);
+                write!(f, "{:WIDTH$} {key}, {src}", "store-env")
+            }
+            Instruction::PushPositional { src } => {
+                write!(f, "{:WIDTH$} {src}", "push-positional")
+            }
+            Instruction::AppendRest { src } => {
+                write!(f, "{:WIDTH$} {src}", "append-rest")
+            }
+            Instruction::PushFlag { name } => {
+                let name = FmtData(self.data, *name);
+                write!(f, "{:WIDTH$} {name}", "push-flag")
+            }
+            Instruction::PushShortFlag { short } => {
+                let short = FmtData(self.data, *short);
+                write!(f, "{:WIDTH$} {short}", "push-short-flag")
+            }
+            Instruction::PushNamed { name, src } => {
+                let name = FmtData(self.data, *name);
+                write!(f, "{:WIDTH$} {name}, {src}", "push-named")
+            }
+            Instruction::PushShortNamed { short, src } => {
+                let short = FmtData(self.data, *short);
+                write!(f, "{:WIDTH$} {short}, {src}", "push-short-named")
+            }
+            Instruction::PushParserInfo { name, info } => {
+                let name = FmtData(self.data, *name);
+                write!(f, "{:WIDTH$} {name}, {info:?}", "push-parser-info")
+            }
+            Instruction::RedirectOut { mode } => {
+                write!(f, "{:WIDTH$} {mode}", "redirect-out")
+            }
+            Instruction::RedirectErr { mode } => {
+                write!(f, "{:WIDTH$} {mode}", "redirect-err")
+            }
+            Instruction::CheckErrRedirected { src } => {
+                write!(f, "{:WIDTH$} {src}", "check-err-redirected")
+            }
+            Instruction::OpenFile {
+                file_num,
+                path,
+                append,
+            } => {
+                write!(
+                    f,
+                    "{:WIDTH$} file({file_num}), {path}, append = {append:?}",
+                    "open-file"
+                )
+            }
+            Instruction::WriteFile { file_num, src } => {
+                write!(f, "{:WIDTH$} file({file_num}), {src}", "write-file")
+            }
+            Instruction::CloseFile { file_num } => {
+                write!(f, "{:WIDTH$} file({file_num})", "close-file")
+            }
+            Instruction::Call { decl_id, src_dst } => {
+                let decl = FmtDecl::new(self.engine_state, *decl_id);
+                write!(f, "{:WIDTH$} {decl}, {src_dst}", "call")
+            }
+            Instruction::StringAppend { src_dst, val } => {
+                write!(f, "{:WIDTH$} {src_dst}, {val}", "string-append")
+            }
+            Instruction::GlobFrom { src_dst, no_expand } => {
+                let no_expand = if *no_expand { "no-expand" } else { "expand" };
+                write!(f, "{:WIDTH$} {src_dst}, {no_expand}", "glob-from",)
+            }
+            Instruction::ListPush { src_dst, item } => {
+                write!(f, "{:WIDTH$} {src_dst}, {item}", "list-push")
+            }
+            Instruction::ListSpread { src_dst, items } => {
+                write!(f, "{:WIDTH$} {src_dst}, {items}", "list-spread")
+            }
+            Instruction::RecordInsert { src_dst, key, val } => {
+                write!(f, "{:WIDTH$} {src_dst}, {key}, {val}", "record-insert")
+            }
+            Instruction::RecordSpread { src_dst, items } => {
+                write!(f, "{:WIDTH$} {src_dst}, {items}", "record-spread")
+            }
+            Instruction::Not { src_dst } => {
+                write!(f, "{:WIDTH$} {src_dst}", "not")
+            }
+            Instruction::BinaryOp { lhs_dst, op, rhs } => {
+                write!(f, "{:WIDTH$} {lhs_dst}, {op:?}, {rhs}", "binary-op")
+            }
+            Instruction::FollowCellPath { src_dst, path } => {
+                write!(f, "{:WIDTH$} {src_dst}, {path}", "follow-cell-path")
+            }
+            Instruction::CloneCellPath { dst, src, path } => {
+                write!(f, "{:WIDTH$} {dst}, {src}, {path}", "clone-cell-path")
+            }
+            Instruction::UpsertCellPath {
+                src_dst,
+                path,
+                new_value,
+            } => {
+                write!(
+                    f,
+                    "{:WIDTH$} {src_dst}, {path}, {new_value}",
+                    "upsert-cell-path"
+                )
+            }
+            Instruction::Jump { index } => {
+                write!(f, "{:WIDTH$} {index}", "jump")
+            }
+            Instruction::BranchIf { cond, index } => {
+                write!(f, "{:WIDTH$} {cond}, {index}", "branch-if")
+            }
+            Instruction::BranchIfEmpty { src, index } => {
+                write!(f, "{:WIDTH$} {src}, {index}", "branch-if-empty")
+            }
+            Instruction::Match {
+                pattern,
+                src,
+                index,
+            } => {
+                let pattern = FmtPattern {
+                    engine_state: self.engine_state,
+                    pattern,
+                };
+                write!(f, "{:WIDTH$} ({pattern}), {src}, {index}", "match")
+            }
+            Instruction::CheckMatchGuard { src } => {
+                write!(f, "{:WIDTH$} {src}", "check-match-guard")
+            }
+            Instruction::Iterate {
+                dst,
+                stream,
+                end_index,
+            } => {
+                write!(f, "{:WIDTH$} {dst}, {stream}, end {end_index}", "iterate")
+            }
+            Instruction::OnError { index } => {
+                write!(f, "{:WIDTH$} {index}", "on-error")
+            }
+            Instruction::OnErrorInto { index, dst } => {
+                write!(f, "{:WIDTH$} {index}, {dst}", "on-error-into")
+            }
+            Instruction::PopErrorHandler => {
+                write!(f, "{:WIDTH$}", "pop-error-handler")
+            }
+            Instruction::CheckExternalFailed { dst, src } => {
+                write!(f, "{:WIDTH$} {dst}, {src}", "check-external-failed")
+            }
+            Instruction::ReturnEarly { src } => {
+                write!(f, "{:WIDTH$} {src}", "return-early")
+            }
+            Instruction::Return { src } => {
+                write!(f, "{:WIDTH$} {src}", "return")
+            }
+        }
+    }
+}
+
+struct FmtDecl<'a>(DeclId, &'a str);
+
+impl<'a> FmtDecl<'a> {
+    fn new(engine_state: &'a EngineState, decl_id: DeclId) -> Self {
+        FmtDecl(decl_id, engine_state.get_decl(decl_id).name())
+    }
+}
+
+impl fmt::Display for FmtDecl<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "decl {} {:?}", self.0, self.1)
+    }
+}
+
+struct FmtVar<'a>(DeclId, Option<&'a str>);
+
+impl<'a> FmtVar<'a> {
+    fn new(engine_state: &'a EngineState, var_id: VarId) -> Self {
+        // Search for the name of the variable
+        let name: Option<&str> = engine_state
+            .active_overlays(&[])
+            .flat_map(|overlay| overlay.vars.iter())
+            .find(|(_, v)| **v == var_id)
+            .map(|(k, _)| std::str::from_utf8(k).unwrap_or("<utf-8 error>"));
+        FmtVar(var_id, name)
+    }
+}
+
+impl fmt::Display for FmtVar<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        if let Some(name) = self.1 {
+            write!(f, "var {} {:?}", self.0, name)
+        } else {
+            write!(f, "var {}", self.0)
+        }
+    }
+}
+
+impl fmt::Display for RedirectMode {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            RedirectMode::Pipe => write!(f, "pipe"),
+            RedirectMode::Capture => write!(f, "capture"),
+            RedirectMode::Null => write!(f, "null"),
+            RedirectMode::Inherit => write!(f, "inherit"),
+            RedirectMode::File { file_num } => write!(f, "file({file_num})"),
+            RedirectMode::Caller => write!(f, "caller"),
+        }
+    }
+}
+
+struct FmtData<'a>(&'a [u8], DataSlice);
+
+impl<'a> fmt::Display for FmtData<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        if let Ok(s) = std::str::from_utf8(&self.0[self.1]) {
+            // Write as string
+            write!(f, "{s:?}")
+        } else {
+            // Write as byte array
+            write!(f, "0x{:x?}", self.0)
+        }
+    }
+}
+
+struct FmtLiteral<'a> {
+    literal: &'a Literal,
+    data: &'a [u8],
+}
+
+impl<'a> fmt::Display for FmtLiteral<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self.literal {
+            Literal::Bool(b) => write!(f, "bool({b:?})"),
+            Literal::Int(i) => write!(f, "int({i:?})"),
+            Literal::Float(fl) => write!(f, "float({fl:?})"),
+            Literal::Filesize(q) => write!(f, "filesize({q}b)"),
+            Literal::Duration(q) => write!(f, "duration({q}ns)"),
+            Literal::Binary(b) => write!(f, "binary({})", FmtData(self.data, *b)),
+            Literal::Block(id) => write!(f, "block({id})"),
+            Literal::Closure(id) => write!(f, "closure({id})"),
+            Literal::RowCondition(id) => write!(f, "row_condition({id})"),
+            Literal::Range {
+                start,
+                step,
+                end,
+                inclusion,
+            } => write!(f, "range({start}, {step}, {end}, {inclusion:?})"),
+            Literal::List { capacity } => write!(f, "list(capacity = {capacity})"),
+            Literal::Record { capacity } => write!(f, "record(capacity = {capacity})"),
+            Literal::Filepath { val, no_expand } => write!(
+                f,
+                "filepath({}, no_expand = {no_expand:?})",
+                FmtData(self.data, *val)
+            ),
+            Literal::Directory { val, no_expand } => write!(
+                f,
+                "directory({}, no_expand = {no_expand:?})",
+                FmtData(self.data, *val)
+            ),
+            Literal::GlobPattern { val, no_expand } => write!(
+                f,
+                "glob-pattern({}, no_expand = {no_expand:?})",
+                FmtData(self.data, *val)
+            ),
+            Literal::String(s) => write!(f, "string({})", FmtData(self.data, *s)),
+            Literal::RawString(rs) => write!(f, "raw-string({})", FmtData(self.data, *rs)),
+            Literal::CellPath(p) => write!(f, "cell-path({p})"),
+            Literal::Date(dt) => write!(f, "date({dt})"),
+            Literal::Nothing => write!(f, "nothing"),
+        }
+    }
+}
+
+struct FmtPattern<'a> {
+    engine_state: &'a EngineState,
+    pattern: &'a Pattern,
+}
+
+impl<'a> fmt::Display for FmtPattern<'a> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self.pattern {
+            Pattern::Record(bindings) => {
+                f.write_str("{")?;
+                for (name, pattern) in bindings {
+                    write!(
+                        f,
+                        "{}: {}",
+                        name,
+                        FmtPattern {
+                            engine_state: self.engine_state,
+                            pattern: &pattern.pattern,
+                        }
+                    )?;
+                }
+                f.write_str("}")
+            }
+            Pattern::List(bindings) => {
+                f.write_str("[")?;
+                for pattern in bindings {
+                    write!(
+                        f,
+                        "{}",
+                        FmtPattern {
+                            engine_state: self.engine_state,
+                            pattern: &pattern.pattern
+                        }
+                    )?;
+                }
+                f.write_str("]")
+            }
+            Pattern::Value(expr) => {
+                let string =
+                    String::from_utf8_lossy(self.engine_state.get_span_contents(expr.span));
+                f.write_str(&string)
+            }
+            Pattern::Variable(var_id) => {
+                let variable = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "{}", variable)
+            }
+            Pattern::Or(patterns) => {
+                for (index, pattern) in patterns.iter().enumerate() {
+                    if index > 0 {
+                        f.write_str(" | ")?;
+                    }
+                    write!(
+                        f,
+                        "{}",
+                        FmtPattern {
+                            engine_state: self.engine_state,
+                            pattern: &pattern.pattern
+                        }
+                    )?;
+                }
+                Ok(())
+            }
+            Pattern::Rest(var_id) => {
+                let variable = FmtVar::new(self.engine_state, *var_id);
+                write!(f, "..{}", variable)
+            }
+            Pattern::IgnoreRest => f.write_str(".."),
+            Pattern::IgnoreValue => f.write_str("_"),
+            Pattern::Garbage => f.write_str("<garbage>"),
+        }
+    }
+}

crates/nu-protocol/src/ir/mod.rs

@@ -0,0 +1,419 @@
+use std::{fmt, sync::Arc};
+
+use crate::{
+    ast::{CellPath, Expression, Operator, Pattern, RangeInclusion},
+    engine::EngineState,
+    BlockId, DeclId, RegId, Span, Value, VarId,
+};
+
+use chrono::{DateTime, FixedOffset};
+use serde::{Deserialize, Serialize};
+
+mod call;
+mod display;
+
+pub use call::*;
+pub use display::{FmtInstruction, FmtIrBlock};
+
+#[derive(Clone, Serialize, Deserialize)]
+pub struct IrBlock {
+    pub instructions: Vec<Instruction>,
+    pub spans: Vec<Span>,
+    #[serde(with = "serde_arc_u8_array")]
+    pub data: Arc<[u8]>,
+    pub ast: Vec<Option<IrAstRef>>,
+    /// Additional information that can be added to help with debugging
+    pub comments: Vec<Box<str>>,
+    pub register_count: u32,
+    pub file_count: u32,
+}
+
+impl fmt::Debug for IrBlock {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        // the ast field is too verbose and doesn't add much
+        f.debug_struct("IrBlock")
+            .field("instructions", &self.instructions)
+            .field("spans", &self.spans)
+            .field("data", &self.data)
+            .field("comments", &self.comments)
+            .field("register_count", &self.register_count)
+            .field("file_count", &self.register_count)
+            .finish_non_exhaustive()
+    }
+}
+
+impl IrBlock {
+    /// Returns a value that can be formatted with [`Display`](std::fmt::Display) to show a detailed
+    /// listing of the instructions contained within this [`IrBlock`].
+    pub fn display<'a>(&'a self, engine_state: &'a EngineState) -> FmtIrBlock<'a> {
+        FmtIrBlock {
+            engine_state,
+            ir_block: self,
+        }
+    }
+}
+
+/// A slice into the `data` array of a block. This is a compact and cache-friendly way to store
+/// string data that a block uses.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+pub struct DataSlice {
+    pub start: u32,
+    pub len: u32,
+}
+
+impl DataSlice {
+    /// A data slice that contains no data. This slice is always valid.
+    pub const fn empty() -> DataSlice {
+        DataSlice { start: 0, len: 0 }
+    }
+}
+
+impl std::ops::Index<DataSlice> for [u8] {
+    type Output = [u8];
+
+    fn index(&self, index: DataSlice) -> &Self::Output {
+        &self[index.start as usize..(index.start as usize + index.len as usize)]
+    }
+}
+
+/// A possible reference into the abstract syntax tree for an instruction. This is not present for
+/// most instructions and is just added when needed.
+#[derive(Debug, Clone)]
+pub struct IrAstRef(pub Arc<Expression>);
+
+impl Serialize for IrAstRef {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        self.0.as_ref().serialize(serializer)
+    }
+}
+
+impl<'de> Deserialize<'de> for IrAstRef {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        Expression::deserialize(deserializer).map(|expr| IrAstRef(Arc::new(expr)))
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum Instruction {
+    /// Unreachable code path (error)
+    Unreachable,
+    /// Load a literal value into the `dst` register
+    LoadLiteral { dst: RegId, lit: Literal },
+    /// Load a clone of a boxed value into the `dst` register (e.g. from const evaluation)
+    LoadValue { dst: RegId, val: Box<Value> },
+    /// Move a register. Value is taken from `src` (used by this instruction).
+    Move { dst: RegId, src: RegId },
+    /// Copy a register (must be a collected value). Value is still in `src` after this instruction.
+    Clone { dst: RegId, src: RegId },
+    /// Collect a stream in a register to a value
+    Collect { src_dst: RegId },
+    /// Change the span of the contents of a register to the span of this instruction.
+    Span { src_dst: RegId },
+    /// Drop the value/stream in a register, without draining
+    Drop { src: RegId },
+    /// Drain the value/stream in a register and discard (e.g. semicolon).
+    ///
+    /// If passed a stream from an external command, sets $env.LAST_EXIT_CODE to the resulting exit
+    /// code, and invokes any available error handler with Empty, or if not available, returns an
+    /// exit-code-only stream, leaving the block.
+    Drain { src: RegId },
+    /// Load the value of a variable into the `dst` register
+    LoadVariable { dst: RegId, var_id: VarId },
+    /// Store the value of a variable from the `src` register
+    StoreVariable { var_id: VarId, src: RegId },
+    /// Load the value of an environment variable into the `dst` register
+    LoadEnv { dst: RegId, key: DataSlice },
+    /// Load the value of an environment variable into the `dst` register, or `Nothing` if it
+    /// doesn't exist
+    LoadEnvOpt { dst: RegId, key: DataSlice },
+    /// Store the value of an environment variable from the `src` register
+    StoreEnv { key: DataSlice, src: RegId },
+    /// Add a positional arg to the next (internal) call.
+    PushPositional { src: RegId },
+    /// Add a list of args to the next (internal) call (spread/rest).
+    AppendRest { src: RegId },
+    /// Add a named arg with no value to the next (internal) call.
+    PushFlag { name: DataSlice },
+    /// Add a short named arg with no value to the next (internal) call.
+    PushShortFlag { short: DataSlice },
+    /// Add a named arg with a value to the next (internal) call.
+    PushNamed { name: DataSlice, src: RegId },
+    /// Add a short named arg with a value to the next (internal) call.
+    PushShortNamed { short: DataSlice, src: RegId },
+    /// Add parser info to the next (internal) call.
+    PushParserInfo {
+        name: DataSlice,
+        info: Box<Expression>,
+    },
+    /// Set the redirection for stdout for the next call (only).
+    ///
+    /// The register for a file redirection is not consumed.
+    RedirectOut { mode: RedirectMode },
+    /// Set the redirection for stderr for the next call (only).
+    ///
+    /// The register for a file redirection is not consumed.
+    RedirectErr { mode: RedirectMode },
+    /// Throw an error if stderr wasn't redirected in the given stream. `src` is preserved.
+    CheckErrRedirected { src: RegId },
+    /// Open a file for redirection, pushing it onto the file stack.
+    OpenFile {
+        file_num: u32,
+        path: RegId,
+        append: bool,
+    },
+    /// Write data from the register to a file. This is done to finish a file redirection, in case
+    /// an internal command or expression was evaluated rather than an external one.
+    WriteFile { file_num: u32, src: RegId },
+    /// Pop a file used for redirection from the file stack.
+    CloseFile { file_num: u32 },
+    /// Make a call. The input is taken from `src_dst`, and the output is placed in `src_dst`,
+    /// overwriting it. The argument stack is used implicitly and cleared when the call ends.
+    Call { decl_id: DeclId, src_dst: RegId },
+    /// Append a value onto the end of a string. Uses `to_expanded_string(", ", ...)` on the value.
+    /// Used for string interpolation literals. Not the same thing as the `++` operator.
+    StringAppend { src_dst: RegId, val: RegId },
+    /// Convert a string into a glob. Used for glob interpolation and setting glob variables. If the
+    /// value is already a glob, it won't be modified (`no_expand` will have no effect).
+    GlobFrom { src_dst: RegId, no_expand: bool },
+    /// Push a value onto the end of a list. Used to construct list literals.
+    ListPush { src_dst: RegId, item: RegId },
+    /// Spread a value onto the end of a list. Used to construct list literals.
+    ListSpread { src_dst: RegId, items: RegId },
+    /// Insert a key-value pair into a record. Used to construct record literals. Raises an error if
+    /// the key already existed in the record.
+    RecordInsert {
+        src_dst: RegId,
+        key: RegId,
+        val: RegId,
+    },
+    /// Spread a record onto a record. Used to construct record literals. Any existing value for the
+    /// key is overwritten.
+    RecordSpread { src_dst: RegId, items: RegId },
+    /// Negate a boolean.
+    Not { src_dst: RegId },
+    /// Do a binary operation on `lhs_dst` (left) and `rhs` (right) and write the result to
+    /// `lhs_dst`.
+    BinaryOp {
+        lhs_dst: RegId,
+        op: Operator,
+        rhs: RegId,
+    },
+    /// Follow a cell path on the value in `src_dst`, storing the result back to `src_dst`
+    FollowCellPath { src_dst: RegId, path: RegId },
+    /// Clone the value at a cell path in `src`, storing the result to `dst`. The original value
+    /// remains in `src`. Must be a collected value.
+    CloneCellPath { dst: RegId, src: RegId, path: RegId },
+    /// Update/insert a cell path to `new_value` on the value in `src_dst`, storing the modified
+    /// value back to `src_dst`
+    UpsertCellPath {
+        src_dst: RegId,
+        path: RegId,
+        new_value: RegId,
+    },
+    /// Jump to an offset in this block
+    Jump { index: usize },
+    /// Branch to an offset in this block if the value of the `cond` register is a true boolean,
+    /// otherwise continue execution
+    BranchIf { cond: RegId, index: usize },
+    /// Branch to an offset in this block if the value of the `src` register is Empty or Nothing,
+    /// otherwise continue execution. The original value in `src` is preserved.
+    BranchIfEmpty { src: RegId, index: usize },
+    /// Match a pattern on `src`. If the pattern matches, branch to `index` after having set any
+    /// variables captured by the pattern. If the pattern doesn't match, continue execution. The
+    /// original value is preserved in `src` through this instruction.
+    Match {
+        pattern: Box<Pattern>,
+        src: RegId,
+        index: usize,
+    },
+    /// Check that a match guard is a boolean, throwing
+    /// [`MatchGuardNotBool`](crate::ShellError::MatchGuardNotBool) if it isn't. Preserves `src`.
+    CheckMatchGuard { src: RegId },
+    /// Iterate on register `stream`, putting the next value in `dst` if present, or jumping to
+    /// `end_index` if the iterator is finished
+    Iterate {
+        dst: RegId,
+        stream: RegId,
+        end_index: usize,
+    },
+    /// Push an error handler, without capturing the error value
+    OnError { index: usize },
+    /// Push an error handler, capturing the error value into `dst`. If the error handler is not
+    /// called, the register should be freed manually.
+    OnErrorInto { index: usize, dst: RegId },
+    /// Pop an error handler. This is not necessary when control flow is directed to the error
+    /// handler due to an error.
+    PopErrorHandler,
+    /// Check if an external command failed. Boolean value into `dst`. `src` is preserved, but it
+    /// does require waiting for the command to exit.
+    CheckExternalFailed { dst: RegId, src: RegId },
+    /// Return early from the block, raising a `ShellError::Return` instead.
+    ///
+    /// Collecting the value is unavoidable.
+    ReturnEarly { src: RegId },
+    /// Return from the block with the value in the register
+    Return { src: RegId },
+}
+
+impl Instruction {
+    /// Returns a value that can be formatted with [`Display`](std::fmt::Display) to show a detailed
+    /// listing of the instruction.
+    pub fn display<'a>(
+        &'a self,
+        engine_state: &'a EngineState,
+        data: &'a [u8],
+    ) -> FmtInstruction<'a> {
+        FmtInstruction {
+            engine_state,
+            instruction: self,
+            data,
+        }
+    }
+
+    /// Returns the branch target index of the instruction if this is a branching instruction.
+    pub fn branch_target(&self) -> Option<usize> {
+        match self {
+            Instruction::Jump { index } => Some(*index),
+            Instruction::BranchIf { cond: _, index } => Some(*index),
+            Instruction::BranchIfEmpty { src: _, index } => Some(*index),
+            Instruction::Match {
+                pattern: _,
+                src: _,
+                index,
+            } => Some(*index),
+
+            Instruction::Iterate {
+                dst: _,
+                stream: _,
+                end_index,
+            } => Some(*end_index),
+            Instruction::OnError { index } => Some(*index),
+            Instruction::OnErrorInto { index, dst: _ } => Some(*index),
+            _ => None,
+        }
+    }
+
+    /// Sets the branch target of the instruction if this is a branching instruction.
+    ///
+    /// Returns `Err(target_index)` if it isn't a branching instruction.
+    pub fn set_branch_target(&mut self, target_index: usize) -> Result<(), usize> {
+        match self {
+            Instruction::Jump { index } => *index = target_index,
+            Instruction::BranchIf { cond: _, index } => *index = target_index,
+            Instruction::BranchIfEmpty { src: _, index } => *index = target_index,
+            Instruction::Match {
+                pattern: _,
+                src: _,
+                index,
+            } => *index = target_index,
+
+            Instruction::Iterate {
+                dst: _,
+                stream: _,
+                end_index,
+            } => *end_index = target_index,
+            Instruction::OnError { index } => *index = target_index,
+            Instruction::OnErrorInto { index, dst: _ } => *index = target_index,
+            _ => return Err(target_index),
+        }
+        Ok(())
+    }
+}
+
+// This is to document/enforce the size of `Instruction` in bytes.
+// We should try to avoid increasing the size of `Instruction`,
+// and PRs that do so will have to change the number below so that it's noted in review.
+const _: () = assert!(std::mem::size_of::<Instruction>() <= 24);
+
+/// A literal value that can be embedded in an instruction.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub enum Literal {
+    Bool(bool),
+    Int(i64),
+    Float(f64),
+    Filesize(i64),
+    Duration(i64),
+    Binary(DataSlice),
+    Block(BlockId),
+    Closure(BlockId),
+    RowCondition(BlockId),
+    Range {
+        start: RegId,
+        step: RegId,
+        end: RegId,
+        inclusion: RangeInclusion,
+    },
+    List {
+        capacity: usize,
+    },
+    Record {
+        capacity: usize,
+    },
+    Filepath {
+        val: DataSlice,
+        no_expand: bool,
+    },
+    Directory {
+        val: DataSlice,
+        no_expand: bool,
+    },
+    GlobPattern {
+        val: DataSlice,
+        no_expand: bool,
+    },
+    String(DataSlice),
+    RawString(DataSlice),
+    CellPath(Box<CellPath>),
+    Date(Box<DateTime<FixedOffset>>),
+    Nothing,
+}
+
+/// A redirection mode for the next call. See [`OutDest`](crate::OutDest).
+///
+/// This is generated by:
+///
+/// 1. Explicit redirection in a [`PipelineElement`](crate::ast::PipelineElement), or
+/// 2. The [`pipe_redirection()`](crate::engine::Command::pipe_redirection) of the command being
+///    piped into.
+///
+/// Not setting it uses the default, determined by [`Stack`](crate::engine::Stack).
+#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
+pub enum RedirectMode {
+    Pipe,
+    Capture,
+    Null,
+    Inherit,
+    /// Use the given numbered file.
+    File {
+        file_num: u32,
+    },
+    /// Use the redirection mode requested by the caller, for a pre-return call.
+    Caller,
+}
+
+/// Just a hack to allow `Arc<[u8]>` to be serialized and deserialized
+mod serde_arc_u8_array {
+    use serde::{Deserialize, Serialize};
+    use std::sync::Arc;
+
+    pub fn serialize<S>(data: &Arc<[u8]>, ser: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        data.as_ref().serialize(ser)
+    }
+
+    pub fn deserialize<'de, D>(de: D) -> Result<Arc<[u8]>, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        let data: Vec<u8> = Deserialize::deserialize(de)?;
+        Ok(data.into())
+    }
+}

crates/nu-protocol/src/lib.rs

@@ -9,6 +9,7 @@ pub mod eval_base;
 pub mod eval_const;
 mod example;
 mod id;
+pub mod ir;
 mod lev_distance;
 mod module;
 pub mod parser_path;

crates/nu-protocol/src/pipeline/byte_stream.rs

@@ -352,6 +352,12 @@ impl ByteStream {
         self.span
     }
 
+    /// Changes the [`Span`] associated with the [`ByteStream`].
+    pub fn with_span(mut self, span: Span) -> Self {
+        self.span = span;
+        self
+    }
+
     /// Returns the [`ByteStreamType`] associated with the [`ByteStream`].
     pub fn type_(&self) -> ByteStreamType {
         self.type_

crates/nu-protocol/src/pipeline/list_stream.rs

@@ -31,11 +31,22 @@ impl ListStream {
         self.span
     }
 
+    /// Changes the [`Span`] associated with this [`ListStream`].
+    pub fn with_span(mut self, span: Span) -> Self {
+        self.span = span;
+        self
+    }
+
     /// Convert a [`ListStream`] into its inner [`Value`] `Iterator`.
     pub fn into_inner(self) -> ValueIterator {
         self.stream
     }
 
+    /// Take a single value from the inner `Iterator`, modifying the stream.
+    pub fn next_value(&mut self) -> Option<Value> {
+        self.stream.next()
+    }
+
     /// Converts each value in a [`ListStream`] into a string and then joins the strings together
     /// using the given separator.
     pub fn into_string(self, separator: &str, config: &Config) -> String {

crates/nu-protocol/src/pipeline/pipeline_data.rs

@@ -96,6 +96,24 @@ impl PipelineData {
         }
     }
 
+    /// Change the span of the [`PipelineData`].
+    ///
+    /// Returns `Value(Nothing)` with the given span if it was [`PipelineData::Empty`].
+    pub fn with_span(self, span: Span) -> Self {
+        match self {
+            PipelineData::Empty => PipelineData::Value(Value::nothing(span), None),
+            PipelineData::Value(value, metadata) => {
+                PipelineData::Value(value.with_span(span), metadata)
+            }
+            PipelineData::ListStream(stream, metadata) => {
+                PipelineData::ListStream(stream.with_span(span), metadata)
+            }
+            PipelineData::ByteStream(stream, metadata) => {
+                PipelineData::ByteStream(stream.with_span(span), metadata)
+            }
+        }
+    }
+
     /// Get a type that is representative of the `PipelineData`.
     ///
     /// The type returned here makes no effort to collect a stream, so it may be a different type
@@ -129,7 +147,8 @@ impl PipelineData {
     /// without consuming input and without writing anything.
     ///
     /// For the other [`OutDest`]s, the given `PipelineData` will be completely consumed
-    /// and `PipelineData::Empty` will be returned.
+    /// and `PipelineData::Empty` will be returned, unless the data is from an external stream,
+    /// in which case an external stream containing only that exit code will be returned.
     pub fn write_to_out_dests(
         self,
         engine_state: &EngineState,
@@ -137,7 +156,11 @@ impl PipelineData {
     ) -> Result<PipelineData, ShellError> {
         match (self, stack.stdout()) {
             (PipelineData::ByteStream(stream, ..), stdout) => {
-                stream.write_to_out_dests(stdout, stack.stderr())?;
+                if let Some(exit_status) = stream.write_to_out_dests(stdout, stack.stderr())? {
+                    return Ok(PipelineData::new_external_stream_with_only_exit_code(
+                        exit_status.code(),
+                    ));
+                }
             }
             (data, OutDest::Pipe | OutDest::Capture) => return Ok(data),
             (PipelineData::Empty, ..) => {}
@@ -570,7 +593,7 @@ impl PipelineData {
                         self.write_all_and_flush(engine_state, no_newline, to_stderr)
                     } else {
                         let call = Call::new(Span::new(0, 0));
-                        let table = command.run(engine_state, stack, &call, self)?;
+                        let table = command.run(engine_state, stack, &(&call).into(), self)?;
                         table.write_all_and_flush(engine_state, no_newline, to_stderr)
                     }
                 } else {

crates/nu-protocol/src/signature.rs

@@ -1,6 +1,5 @@
 use crate::{
-    ast::Call,
-    engine::{Command, CommandType, EngineState, Stack},
+    engine::{Call, Command, CommandType, EngineState, Stack},
     BlockId, PipelineData, ShellError, SyntaxShape, Type, Value, VarId,
 };
 use serde::{Deserialize, Serialize};

crates/nu-protocol/src/span.rs

@@ -53,6 +53,22 @@ impl<T> Spanned<T> {
     }
 }
 
+impl<T, E> Spanned<Result<T, E>> {
+    /// Move the `Result` to the outside, resulting in a spanned `Ok` or unspanned `Err`.
+    pub fn transpose(self) -> Result<Spanned<T>, E> {
+        match self {
+            Spanned {
+                item: Ok(item),
+                span,
+            } => Ok(Spanned { item, span }),
+            Spanned {
+                item: Err(err),
+                span: _,
+            } => Err(err),
+        }
+    }
+}
+
 /// Helper trait to create [`Spanned`] more ergonomically.
 pub trait IntoSpanned: Sized {
     /// Wrap items together with a span into [`Spanned`].