mirror of
https://github.com/nushell/nushell.git
synced 2025-05-19 01:10:48 +02:00
833471241a
21 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Piepmatz
|
833471241a
|
Refactor: Construct IoError from std::io::Error instead of std::io::ErrorKind (#15777)
|
||
Jack Wright
|
c2ac8f730e
|
Rust 1.85, edition=2024 (#15741) | ||
Wind
|
2f18b9c856
|
Enable nushell error with backtrace (#14945)
# Description After this pr, nushell is able to raise errors with a backtrace, which should make users easier to debug. To enable the feature, users need to set env variable via `$env.NU_BACKTRACE = 1`. But yeah it might not work perfectly, there are some corner cases which might not be handled. I think it should close #13379 in another way. ### About the change The implementation mostly contained with 2 parts: 1. introduce a new `ChainedError` struct as well as a new `ShellError::ChainedError` variant. If `eval_instruction` returned an error, it converts the error to `ShellError::ChainedError`. `ChainedError` struct is responsable to display errors properly. It needs to handle the following 2 cases: - if we run a function which runs `error make` internally, it needs to display the error itself along with caller span. - if we run a `error make` directly, or some commands directly returns an error, we just want nushell raise an error about `error make`. 2. Attach caller spans to `ListStream` and `ByteStream`, because they are lazy streams, and *only* contains the span that runs it directly(like `^false`, for example), so nushell needs to add all caller spans to the stream. For example: in `def a [] { ^false }; def b [] { a; 33 }; b`, when we run `b`, which runs `a`, which runs `^false`, the `ByteStream` only contains the span of `^false`, we need to make it contains the span of `a`, so nushell is able to get all spans if something bad happened. This behavior is happened after running `Instruction::Call`, if it returns a `ByteStream` and `ListStream`, it will call `push_caller_span` method to attach call spans. # User-Facing Changes It's better to demostrate how it works by examples, given the following definition: ```nushell > $env.NU_BACKTRACE = 1 > def a [x] { if $x == 3 { error make {msg: 'a custom error'}}} > def a_2 [x] { if $x == 3 { ^false } else { $x } } > def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } } > def b [--list-stream --external] { if $external == true { # error with non-zero exit code, which is generated from external command. a_2 1; a_2 3; a_2 2 } else if $list_stream == true { # error generated by list-stream a_3 1; a_3 3; a_3 2 } else { # error generated by command directly a 1; a 2; a 3 } } ``` Run `b` directly shows the following error: <details> ```nushell Error: chained_error × oops ╭─[entry #27:1:1] 1 │ b · ┬ · ╰── error happened when running this ╰──── Error: chained_error × oops ╭─[entry #26:10:19] 9 │ # error generated by command directly 10 │ a 1; a 2; a 3 · ┬ · ╰── error happened when running this 11 │ } ╰──── Error: × a custom error ╭─[entry #6:1:26] 1 │ def a [x] { if $x == 3 { error make {msg: 'a custom error'}}} · ─────┬──── · ╰── originates from here ╰──── ``` </details> Run `b --list-stream` shows the following error <details> ```nushell Error: chained_error × oops ╭─[entry #28:1:1] 1 │ b --list-stream · ┬ · ╰── error happened when running this ╰──── Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #26:7:16] 6 │ # error generated by list-stream 7 │ a_3 1; a_3 3; a_3 2 · ─┬─ · ╰── source value 8 │ } else { ╰──── Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #23:1:29] 1 │ def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } } · ┬ · ╰── source value ╰──── Error: × a custom error inside list stream ╭─[entry #23:1:44] 1 │ def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } } · ─────┬──── · ╰── originates from here ╰──── ``` </details> Run `b --external` shows the following error: <details> ```nushell Error: chained_error × oops ╭─[entry #29:1:1] 1 │ b --external · ┬ · ╰── error happened when running this ╰──── Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #26:4:16] 3 │ # error with non-zero exit code, which is generated from external command. 4 │ a_2 1; a_2 3; a_2 2 · ─┬─ · ╰── source value 5 │ } else if $list_stream == true { ╰──── Error: nu:🐚:non_zero_exit_code × External command had a non-zero exit code ╭─[entry #7:1:29] 1 │ def a_2 [x] { if $x == 3 { ^false } else { $x } } · ──┬── · ╰── exited with code 1 ╰──── ``` </details> It also added a message to guide the usage of NU_BACKTRACE, see the last line in the following example: ```shell ls asdfasd Error: nu:🐚:io::not_found × I/O error ╰─▶ × Entity not found ╭─[entry #17:1:4] 1 │ ls asdfasd · ───┬─── · ╰── Entity not found ╰──── help: The error occurred at '/home/windsoilder/projects/nushell/asdfasd' set the `NU_BACKTRACE=1` environment variable to display a backtrace. ``` # Tests + Formatting Added some tests for the behavior. # After Submitting |
||
Bahex
|
b55ed69c92
|
fix range bugs in str substring, str index-of, slice, bytes at (#14863)
- fixes #14769 # Description ## Bugs - `str substring 0..<0` When passed a range containing no elements, for non-zero cases `str substring` behaves correctly: ```nushell ("hello world" | str substring 1..<1) == "" # => true ``` but if the range is `0..<0`, it returns the whole string instead ```nushell "hello world" | str substring 0..<0 # => hello world ``` - `[0 1 2] | range 0..<0` Similar behavior to `str substring` - `str index-of` - off-by-one on end bounds - underflow on negative start bounds - `bytes at` has inconsistent behavior, works correctly when the size is known, returns one byte less when it's not known (streaming) This can be demonstrated by comparing the outputs of following snippets ```nushell "hello world" | into binary | bytes at ..<5 | decode # => hello "hello world" | into binary | chunks 1 | bytes collect | bytes at ..<5 | decode # => hell ``` - `bytes at` panics on decreasing (`5..3`) ranges if the input size is known. Does not panic with streaming input. ## Changes - implement `FromValue` for `IntRange`, as it is very common to use integer ranges as arguments - `IntRange::absolute_start` can now point one-past-end - `IntRange::absolute_end` converts relative `Included` bounds to absolute `Excluded` bounds - `IntRange::absolute_bounds` is a convenience method that calls the other `absolute_*` methods and transforms reverse ranges to empty at `start` (`5..3` => `5..<5`) - refactored `str substring` tests to allow empty exclusive range tests - fix the `0..<0` case for `str substring` and `str index-of` - `IntRange::distance` never returns `Included(0)` As a general rule `Included(n) == Excluded(n + 1)`. This makes returning `Included(0)` bug prone as users of the function will likely rely on this general rule and cause bugs. - `ByteStream::slice` no longer has an off-by-one on inputs without a known size. This affected `bytes at`. - `bytes at` no longer panics on reverse ranges - `bytes at` is now consistent between streaming and non streaming inputs. # User-Facing Changes There should be no noticeable changes other than the bugfix. # Tests + Formatting - 🟢 toolkit fmt - 🟢 toolkit clippy - 🟢 toolkit test - 🟢 toolkit test stdlib # After Submitting N/A |
||
|
Piepmatz
|
66bc0542e0
|
Refactor I/O Errors (#14927)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
As mentioned in #10698, we have too many `ShellError` variants, with
some even overlapping in meaning. This PR simplifies and improves I/O
error handling by restructuring `ShellError` related to I/O issues.
Previously, `ShellError::IOError` only contained a message string,
making it convenient but overly generic. It was widely used without
providing spans (#4323).
This PR introduces a new `ShellError::Io` variant that consolidates
multiple I/O-related errors (except for `ShellError::NetworkFailure`,
which remains distinct for now). The new `ShellError::Io` variant
replaces the following:
- `FileNotFound`
- `FileNotFoundCustom`
- `IOInterrupted`
- `IOError`
- `IOErrorSpanned`
- `NotADirectory`
- `DirectoryNotFound`
- `MoveNotPossible`
- `CreateNotPossible`
- `ChangeAccessTimeNotPossible`
- `ChangeModifiedTimeNotPossible`
- `RemoveNotPossible`
- `ReadingFile`
## The `IoError`
`IoError` includes the following fields:
1. **`kind`**: Extends `std::io::ErrorKind` to specify the type of I/O
error without needing new `ShellError` variants. This aligns with the
approach used in `std::io::Error`. This adds a second dimension to error
reporting by combining the `kind` field with `ShellError` variants,
making it easier to describe errors in more detail. As proposed by
@kubouch in [#design-discussion on
Discord](https://discord.com/channels/601130461678272522/615329862395101194/1323699197165178930),
this helps reduce the number of `ShellError` variants. In the error
report, the `kind` field is displayed as the "source" of the error,
e.g., "I/O error," followed by the specific kind of I/O error.
2. **`span`**: A non-optional field to encourage providing spans for
better error reporting (#4323).
3. **`path`**: Optional `PathBuf` to give context about the file or
directory involved in the error (#7695). If provided, it’s shown as a
help entry in error reports.
4. **`additional_context`**: Allows adding custom messages when the
span, kind, and path are insufficient. This is rendered in the error
report at the labeled span.
5. **`location`**: Sometimes, I/O errors occur in the engine itself and
are not caused directly by user input. In such cases, if we don’t have a
span and must set it to `Span::unknown()`, we need another way to
reference the error. For this, the `location` field uses the new
`Location` struct, which records the Rust file and line number where the
error occurred. This ensures that we at least know the Rust code
location that failed, helping with debugging. To make this work, a new
`location!` macro was added, which retrieves `file!`, `line!`, and
`column!` values accurately. If `Location::new` is used directly, it
issues a warning to remind developers to use the macro instead, ensuring
consistent and correct usage.
### Constructor Behavior
`IoError` provides five constructor methods:
- `new` and `new_with_additional_context`: Used for errors caused by
user input and require a valid (non-unknown) span to ensure precise
error reporting.
- `new_internal` and `new_internal_with_path`: Used for internal errors
where a span is not available. These methods require additional context
and the `Location` struct to pinpoint the source of the error in the
engine code.
- `factory`: Returns a closure that maps an `std::io::Error` to an
`IoError`. This is useful for handling multiple I/O errors that share
the same span and path, streamlining error handling in such cases.
## New Report Look
This is simulation how the I/O errors look like (the `open crates` is
simulated to show how internal errors are referenced now):
## `Span::test_data()`
To enable better testing, `Span::test_data()` now returns a value
distinct from `Span::unknown()`. Both `Span::test_data()` and
`Span::unknown()` refer to invalid source code, but having a separate
value for test data helps identify issues during testing while keeping
spans unique.
## Cursed Sneaky Error Transfers
I removed the conversions between `std::io::Error` and `ShellError` as
they often removed important information and were used too broadly to
handle I/O errors. This also removed the problematic implementation
found here:
|
||
Simon Curtis
|
f05162811c
|
Implementing ByteStream interuption on infinite stream (#13552)
# Description This PR should address #13530 by explicitly handling ByteStreams. The issue can be replicated easily on linux by running: ```nushell open /dev/urandom | into binary | bytes at ..10 ``` Would leave the output hanging and with no way to cancel it, this was likely because it was trying to collect the input stream and would not complete. I have also put in an error to say that using negative offsets for a bytestream without a length cannot be used. ```nushell ~/git/nushell> open /dev/urandom | into binary | bytes at (-1).. Error: nu:🐚:incorrect_value × Incorrect value. ╭─[entry #3:1:35] 1 │ open /dev/urandom | into binary | bytes at (-1).. · ────┬─── ───┬── · │ ╰── encountered here · ╰── Negative range values cannot be used with streams that don't specify a length ╰──── ``` # User-Facing Changes No operation changes, only the warning you get back for negative offsets # Tests + Formatting Ran `toolkit check pr ` with no errors or warnings Manual testing of the example commands above --------- Co-authored-by: Ian Manske <ian.manske@pm.me> Co-authored-by: Simon Curtis <simon.curtis@candc-uk.com> |
||
|
Bahex
|
469e23cae4
|
Add bytes split command (#14652)
Related #10708 # Description Add `bytes split` command. `bytes split` splits its input on the provided separator on binary values _and_ binary streams without collecting. The separator can be a multiple character string or multiple byte binary. It can be used when neither `split row` (not streaming over raw input) nor `lines` (streaming, but can only split on newlines) is right. The backing iterator implemented in this PR, `SplitRead`, can be used to implement a streaming `split row` in the future. # User-Facing Changes `bytes split` command added, which can be used to split binary values and raw streams using a separator. # Tests + Formatting - 🟢 toolkit fmt - 🟢 toolkit clippy - 🟢 toolkit test - 🟢 toolkit test stdlib # After Submitting Mention in release notes. |
||
|
Piepmatz
|
3d5f853b03
|
Start to Add WASM Support Again (#14418)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> The [nushell/demo](https://github.com/nushell/demo) project successfully demonstrated running Nushell in the browser using WASM. However, the current version of Nushell cannot be easily built for the `wasm32-unknown-unknown` target, the default for `wasm-bindgen`. This PR introduces initial support for the `wasm32-unknown-unknown` target by disabling OS-dependent features such as filesystem access, IO, and platform/system-specific functionality. This separation is achieved using a new `os` feature in the following crates: - `nu-cmd-lang` - `nu-command` - `nu-engine` - `nu-protocol` The `os` feature includes all functionality that interacts with an operating system. It is enabled by default, but can be disabled using `--no-default-features`. All crates that depend on these core crates now use `--no-default-features` to allow compilation for WASM. To demonstrate compatibility, the following script builds all crates expected to work with WASM. Direct user interaction, running external commands, working with plugins, and features requiring `openssl` are out of scope for now due to their complexity or reliance on C libraries, which are difficult to compile and link in a WASM environment. ```nushell [ # compatible crates "nu-cmd-base", "nu-cmd-extra", "nu-cmd-lang", "nu-color-config", "nu-command", "nu-derive-value", "nu-engine", "nu-glob", "nu-json", "nu-parser", "nu-path", "nu-pretty-hex", "nu-protocol", "nu-std", "nu-system", "nu-table", "nu-term-grid", "nu-utils", "nuon" ] | each {cargo build -p $in --target wasm32-unknown-unknown --no-default-features} ``` ## Caveats This PR has a few caveats: 1. **`miette` and `terminal-size` Dependency Issue** `miette` depends on `terminal-size`, which uses `rustix` when the target is not Windows. However, `rustix` requires `std::os::unix`, which is unavailable in WASM. To address this, I opened a [PR](https://github.com/eminence/terminal-size/pull/68) for `terminal-size` to conditionally compile `rustix` only when the target is Unix. For now, the `Cargo.toml` includes patches to: - Use my forked version of `terminal-size`. - ~~Use an unreleased version of `miette` that depends on `terminal-size@0.4`.~~ These patches are temporary and can be removed once the upstream changes are merged and released. 2. **Test Output Adjustments** Due to the slight bump in the `miette` version, one test required adjustments to accommodate minor formatting changes in the error output, such as shifted newlines. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> This shouldn't break anything but allows using some crates for targeting `wasm32-unknown-unknown` to revive the demo page eventually. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` I did not add any extra tests, I just checked that compiling works, also when using the host target but unselecting the `os` feature. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> ~~Breaking the wasm support can be easily done by adding some `use`s or by adding a new dependency, we should definitely add some CI that also at least builds against wasm to make sure that building for it keep working.~~ I added a job to build wasm. --------- Co-authored-by: Ian Manske <ian.manske@pm.me> |
||
Darren Schroeder
|
af9c31152a
|
Add metadata on open --raw with bytestreams (#14141)
# Description This PR closes #14137 and allows the display hook to be set on byte streams. So, with a hook like this below. ```nushell display_output: { metadata access {|meta| match $meta.content_type? { "application/x-nuscript" | "application/x-nuon" | "text/x-nushell" => { nu-highlight }, "application/json" => { ^bat --language=json --color=always --style=plain --paging=never }, _ => {}, } } | table } ``` You could type `open toolkit.nu` and the text of toolkit.nu would be highlighted by nu-highlight. This PR also changes the way content-type is assigned with `open`. Previously it would only assign it if `--raw` was specified. Lastly, it changes the `is_external()` function to only say `ByteStreamSource::Child`'s are external instead of both Child and `ByteStreamSource::File`. Again, this was to allow the hook to function properly. I'm not sure what negative ramifications changing `is_external()` could have, but there may be some? # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Ian Manske
|
de08b68ba8
|
Fix try printing when it is not the last pipeline element (#13992)
# Description Fixes #13991. This was done by more clearly separating the case when a pipeline is drained vs when it is being written (to a file). I also added an `OutDest::Print` case which might not be strictly necessary, but is a helpful addition. # User-Facing Changes Bug fix. # Tests + Formatting Added a test. # After Submitting There are still a few redirection bugs that I found, but they require larger code changes, so I'll leave them until after the release. |
||
|
Ian Manske
|
03ee54a4df
|
Fix try not working with let, etc. (#13885)
# Description Partialy addresses #13868. `try` does not catch non-zero exit code errors from the last command in a pipeline if the result is assigned to a variable using `let` (or `mut`). This was fixed by adding a new `OutDest::Value` case. This is used when the pipeline is in a "value" position. I.e., it will be collected into a value. This ended up replacing most of the usages of `OutDest::Capture`. So, this PR also renames `OutDest::Capture` to `OutDest::PipeSeparate` to better fit the few remaining use cases for it. # User-Facing Changes Bug fix. # Tests + Formatting Added two tests. |
||
|
Ian Manske
|
3d008e2c4e
|
Error on non-zero exit statuses (#13515)
# Description This PR makes it so that non-zero exit codes and termination by signal are treated as a normal `ShellError`. Currently, these are silent errors. That is, if an external command fails, then it's code block is aborted, but the parent block can sometimes continue execution. E.g., see #8569 and this example: ```nushell [1 2] | each { ^false } ``` Before this would give: ``` ╭───┬──╮ │ 0 │ │ │ 1 │ │ ╰───┴──╯ ``` Now, this shows an error: ``` Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #1:1:2] 1 │ [1 2] | each { ^false } · ┬ · ╰── source value ╰──── Error: nu:🐚:non_zero_exit_code × External command had a non-zero exit code ╭─[entry #1:1:17] 1 │ [1 2] | each { ^false } · ──┬── · ╰── exited with code 1 ╰──── ``` This PR fixes #12874, fixes #5960, fixes #10856, and fixes #5347. This PR also partially addresses #10633 and #10624 (only the last command of a pipeline is currently checked). It looks like #8569 is already fixed, but this PR will make sure it is definitely fixed (fixes #8569). # User-Facing Changes - Non-zero exit codes and termination by signal now cause an error to be thrown. - The error record value passed to a `catch` block may now have an `exit_code` column containing the integer exit code if the error was due to an external command. - Adds new config values, `display_errors.exit_code` and `display_errors.termination_signal`, which determine whether an error message should be printed in the respective error cases. For non-interactive sessions, these are set to `true`, and for interactive sessions `display_errors.exit_code` is false (via the default config). # Tests Added a few tests. # After Submitting - Update docs and book. - Future work: - Error if other external commands besides the last in a pipeline exit with a non-zero exit code. Then, deprecate `do -c` since this will be the default behavior everywhere. - Add a better mechanism for exit codes and deprecate `$env.LAST_EXIT_CODE` (it's buggy). |
||
Stefan Holderbach
|
c5aa15c7f6
|
Add top-level crate documentation/READMEs (#12907)
# Description Add `README.md` files to each crate in our workspace (-plugins) and also include it in the `lib.rs` documentation for <docs.rs> (if there is no existing `lib.rs` crate documentation) In all new README I added the defensive comment that the crates are not considered stable for public consumption. If necessary we can adjust this if we deem a crate useful for plugin authors. |
||
|
Stefan Holderbach
|
076a29ae19
|
Document public types in nu-protocol (#12906)
- **Doc-comment public `nu-protocol` modules** - **Doccomment argument/signature/call stuff** - **Doccomment cell path types** - **Doccomment expression stuff** - **Doccomment import patterns** - **Doccomment pattern matching AST nodes** |
||
Devyn Cairns
|
d7392f1f3b
|
Internal representation (IR) compiler and evaluator (#13330)
# Description This PR adds an internal representation language to Nushell, offering an alternative evaluator based on simple instructions, stream-containing registers, and indexed control flow. The number of registers required is determined statically at compile-time, and the fixed size required is allocated upon entering the block. Each instruction is associated with a span, which makes going backwards from IR instructions to source code very easy. Motivations for IR: 1. **Performance.** By simplifying the evaluation path and making it more cache-friendly and branch predictor-friendly, code that does a lot of computation in Nushell itself can be sped up a decent bit. Because the IR is fairly easy to reason about, we can also implement optimization passes in the future to eliminate and simplify code. 2. **Correctness.** The instructions mostly have very simple and easily-specified behavior, so hopefully engine changes are a little bit easier to reason about, and they can be specified in a more formal way at some point. I have made an effort to document each of the instructions in the docs for the enum itself in a reasonably specific way. Some of the errors that would have happened during evaluation before are now moved to the compilation step instead, because they don't make sense to check during evaluation. 3. **As an intermediate target.** This is a good step for us to bring the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at some point, as code generated from new AST can be directly compared to code generated from old AST. If the IR code is functionally equivalent, it will behave the exact same way. 4. **Debugging.** With a little bit more work, we can probably give control over advancing the virtual machine that `IrBlock`s run on to some sort of external driver, making things like breakpoints and single stepping possible. Tools like `view ir` and [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than before to see what exactly is going on with your Nushell code. The goal is to eventually replace the AST evaluator entirely, once we're sure it's working just as well. You can help dogfood this by running Nushell with `$env.NU_USE_IR` set to some value. The environment variable is checked when Nushell starts, so config runs with IR, or it can also be set on a line at the REPL to change it dynamically. It is also checked when running `do` in case within a script you want to just run a specific piece of code with or without IR. # Example ```nushell view ir { |data| mut sum = 0 for n in $data { $sum += $n } $sum } ``` ```gas # 3 registers, 19 instructions, 0 bytes of data 0: load-literal %0, int(0) 1: store-variable var 904, %0 # let 2: drain %0 3: drop %0 4: load-variable %1, var 903 5: iterate %0, %1, end 15 # for, label(1), from(14:) 6: store-variable var 905, %0 7: load-variable %0, var 904 8: load-variable %2, var 905 9: binary-op %0, Math(Plus), %2 10: span %0 11: store-variable var 904, %0 12: load-literal %0, nothing 13: drain %0 14: jump 5 15: drop %0 # label(0), from(5:) 16: drain %0 17: load-variable %0, var 904 18: return %0 ``` # Benchmarks All benchmarks run on a base model Mac Mini M1. ## Iterative Fibonacci sequence This is about as best case as possible, making use of the much faster control flow. Most code will not experience a speed improvement nearly this large. ```nushell def fib [n: int] { mut a = 0 mut b = 1 for _ in 2..=$n { let c = $a + $b $a = $b $b = $c } $b } use std bench bench { 0..50 | each { |n| fib $n } } ``` IR disabled: ``` ╭───────┬─────────────────╮ │ mean │ 1ms 924µs 665ns │ │ min │ 1ms 700µs 83ns │ │ max │ 3ms 450µs 125ns │ │ std │ 395µs 759ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ``` IR enabled: ``` ╭───────┬─────────────────╮ │ mean │ 452µs 820ns │ │ min │ 427µs 417ns │ │ max │ 540µs 167ns │ │ std │ 17µs 158ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ```  ## [gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu) IR disabled: ``` ╭───┬──────────────────╮ │ 0 │ 27ms 929µs 958ns │ │ 1 │ 21ms 153µs 459ns │ │ 2 │ 18ms 639µs 666ns │ │ 3 │ 19ms 554µs 583ns │ │ 4 │ 13ms 383µs 375ns │ │ 5 │ 11ms 328µs 208ns │ │ 6 │ 5ms 659µs 542ns │ ╰───┴──────────────────╯ ``` IR enabled: ``` ╭───┬──────────────────╮ │ 0 │ 22ms 662µs │ │ 1 │ 17ms 221µs 792ns │ │ 2 │ 14ms 786µs 708ns │ │ 3 │ 13ms 876µs 834ns │ │ 4 │ 13ms 52µs 875ns │ │ 5 │ 11ms 269µs 666ns │ │ 6 │ 6ms 942µs 500ns │ ╰───┴──────────────────╯ ``` ## [random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu) I got pretty random results out of this benchmark so I decided not to include it. Not clear why. # User-Facing Changes - IR compilation errors may appear even if the user isn't evaluating with IR. - IR evaluation can be enabled by setting the `NU_USE_IR` environment variable to any value. - New command `view ir` pretty-prints the IR for a block, and `view ir --json` can be piped into an external tool like [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir). # Tests + Formatting All tests are passing with `NU_USE_IR=1`, and I've added some more eval tests to compare the results for some very core operations. I will probably want to add some more so we don't have to always check `NU_USE_IR=1 toolkit test --workspace` on a regular basis. # After Submitting - [ ] release notes - [ ] further documentation of instructions? - [ ] post-release: publish `nu_plugin_explore_ir` |
||
|
Ian Manske
|
399a7c8836
|
Add and use new Signals struct (#13314)
# Description This PR introduces a new `Signals` struct to replace our adhoc passing around of `ctrlc: Option<Arc<AtomicBool>>`. Doing so has a few benefits: - We can better enforce when/where resetting or triggering an interrupt is allowed. - Consolidates `nu_utils::ctrl_c::was_pressed` and other ad-hoc re-implementations into a single place: `Signals::check`. - This allows us to add other types of signals later if we want. E.g., exiting or suspension. - Similarly, we can more easily change the underlying implementation if we need to in the future. - Places that used to have a `ctrlc` of `None` now use `Signals::empty()`, so we can double check these usages for correctness in the future. |
||
|
Ian Manske
|
c5d716951f
|
Allow byte streams with unknown type to be compatiable with binary (#12959)
# Description Currently, this pipeline doesn't work `open --raw file | take 100`, since the type of the byte stream is `Unknown`, but `take` expects `Binary` streams. This PR changes commands that expect `ByteStreamType::Binary` to also work with `ByteStreamType::Unknown`. This was done by adding two new methods to `ByteStreamType`: `is_binary_coercible` and `is_string_coercible`. These return true if the type is `Unknown` or matches the type in the method name. |
||
|
Ian Manske
|
c98960d053
|
Take owned Read and Write (#12909)
# Description As @YizhePKU pointed out, the [Rust API guidelines](https://rust-lang.github.io/api-guidelines/interoperability.html#generic-readerwriter-functions-take-r-read-and-w-write-by-value-c-rw-value) recommend that generic functions take readers and writers by value and not by reference. This PR changes `copy_with_interupt` and few other places to take owned `Read` and `Write` instead of mutable references. |
||
|
Devyn Cairns
|
c61075e20e
|
Add string/binary type color to ByteStream (#12897)
# Description
This PR allows byte streams to optionally be colored as being
specifically binary or string data, which guarantees that they'll be
converted to `Binary` or `String` appropriately on `into_value()`,
making them compatible with `Type` guarantees. This makes them
significantly more broadly usable for command input and output.
There is still an `Unknown` type for byte streams coming from external
commands, which uses the same behavior as we previously did where it's a
string if it's UTF-8.
A small number of commands were updated to take advantage of this, just
to prove the point. I will be adding more after this merges.
# User-Facing Changes
- New types in `describe`: `string (stream)`, `binary (stream)`
- These commands now return a stream if their input was a stream:
- `into binary`
- `into string`
- `bytes collect`
- `str join`
- `first` (binary)
- `last` (binary)
- `take` (binary)
- `skip` (binary)
- Streams that are explicitly binary colored will print as a streaming
hexdump
- example:
```nushell
1.. | each { into binary } | bytes collect
```
# Tests + Formatting
I've added some tests to cover it at a basic level, and it doesn't break
anything existing, but I do think more would be nice. Some of those will
come when I modify more commands to stream.
# After Submitting
There are a few things I'm not quite satisfied with:
- **String trimming behavior.** We automatically trim newlines from
streams from external commands, but I don't think we should do this with
internal commands. If I call a command that happens to turn my string
into a stream, I don't want the newline to suddenly disappear. I changed
this to specifically do it only on `Child` and `File`, but I don't know
if this is quite right, and maybe we should bring back the old flag for
`trim_end_newline`
- **Known binary always resulting in a hexdump.** It would be nice to
have a `print --raw`, so that we can put binary data on stdout
explicitly if we want to. This PR doesn't change how external commands
work though - they still dump straight to stdout.
Otherwise, here's the normal checklist:
- [ ] release notes
- [ ] docs update for plugin protocol changes (added `type` field)
---------
Co-authored-by: Ian Manske <ian.manske@pm.me>
|
||
|
Ian Manske
|
2a09dccc11
|
Bytestream touchup (#12886)
# Description Adds some docs and a small fix to `Chunks`. |
||
|
Ian Manske
|
6fd854ed9f
|
Replace ExternalStream with new ByteStream type (#12774)
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes #7017, fixes #10763, and fixes #12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
|