mirror of
https://github.com/nushell/nushell.git
synced 2025-05-28 22:07:40 +02:00
41 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Solomon
|
942030199d
|
check signals while printing values (#14980)
Fixes #14960 # User-Facing Changes - The output of non-streaming values can now be interrupted with ctrl-c: ```nushell ~> use std repeat; random chars --length 100kb | repeat 2000 | str join ' ' | collect <data omitted>^C Error: × Operation interrupted ╭─[entry #1:1:61] 1 │ use std repeat; random chars --length 100kb | repeat 2000 | str join ' ' | collect · ────┬─── · ╰── This operation was interrupted ╰──── ``` - When IO errors occur while printing data, nushell no longer panics: ```diff $ nu -c "true | print" | - -Error: - x Main thread panicked. - |-> at crates/nu-protocol/src/errors/shell_error/io.rs:198:13 - `-> for unknown spans with paths, use `new_internal_with_path` +Error: nu:🐚:io::broken_pipe + + x I/O error + `-> x Broken pipe + + ,-[source:1:1] + 1 | true | print + : ^^|^ + : `-| Writing to stdout failed + : | Broken pipe + `---- ``` |
||
Wind
|
2f18b9c856
|
Enable nushell error with backtrace (#14945)
# Description After this pr, nushell is able to raise errors with a backtrace, which should make users easier to debug. To enable the feature, users need to set env variable via `$env.NU_BACKTRACE = 1`. But yeah it might not work perfectly, there are some corner cases which might not be handled. I think it should close #13379 in another way. ### About the change The implementation mostly contained with 2 parts: 1. introduce a new `ChainedError` struct as well as a new `ShellError::ChainedError` variant. If `eval_instruction` returned an error, it converts the error to `ShellError::ChainedError`. `ChainedError` struct is responsable to display errors properly. It needs to handle the following 2 cases: - if we run a function which runs `error make` internally, it needs to display the error itself along with caller span. - if we run a `error make` directly, or some commands directly returns an error, we just want nushell raise an error about `error make`. 2. Attach caller spans to `ListStream` and `ByteStream`, because they are lazy streams, and *only* contains the span that runs it directly(like `^false`, for example), so nushell needs to add all caller spans to the stream. For example: in `def a [] { ^false }; def b [] { a; 33 }; b`, when we run `b`, which runs `a`, which runs `^false`, the `ByteStream` only contains the span of `^false`, we need to make it contains the span of `a`, so nushell is able to get all spans if something bad happened. This behavior is happened after running `Instruction::Call`, if it returns a `ByteStream` and `ListStream`, it will call `push_caller_span` method to attach call spans. # User-Facing Changes It's better to demostrate how it works by examples, given the following definition: ```nushell > $env.NU_BACKTRACE = 1 > def a [x] { if $x == 3 { error make {msg: 'a custom error'}}} > def a_2 [x] { if $x == 3 { ^false } else { $x } } > def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } } > def b [--list-stream --external] { if $external == true { # error with non-zero exit code, which is generated from external command. a_2 1; a_2 3; a_2 2 } else if $list_stream == true { # error generated by list-stream a_3 1; a_3 3; a_3 2 } else { # error generated by command directly a 1; a 2; a 3 } } ``` Run `b` directly shows the following error: <details> ```nushell Error: chained_error × oops ╭─[entry #27:1:1] 1 │ b · ┬ · ╰── error happened when running this ╰──── Error: chained_error × oops ╭─[entry #26:10:19] 9 │ # error generated by command directly 10 │ a 1; a 2; a 3 · ┬ · ╰── error happened when running this 11 │ } ╰──── Error: × a custom error ╭─[entry #6:1:26] 1 │ def a [x] { if $x == 3 { error make {msg: 'a custom error'}}} · ─────┬──── · ╰── originates from here ╰──── ``` </details> Run `b --list-stream` shows the following error <details> ```nushell Error: chained_error × oops ╭─[entry #28:1:1] 1 │ b --list-stream · ┬ · ╰── error happened when running this ╰──── Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #26:7:16] 6 │ # error generated by list-stream 7 │ a_3 1; a_3 3; a_3 2 · ─┬─ · ╰── source value 8 │ } else { ╰──── Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #23:1:29] 1 │ def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } } · ┬ · ╰── source value ╰──── Error: × a custom error inside list stream ╭─[entry #23:1:44] 1 │ def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } } · ─────┬──── · ╰── originates from here ╰──── ``` </details> Run `b --external` shows the following error: <details> ```nushell Error: chained_error × oops ╭─[entry #29:1:1] 1 │ b --external · ┬ · ╰── error happened when running this ╰──── Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #26:4:16] 3 │ # error with non-zero exit code, which is generated from external command. 4 │ a_2 1; a_2 3; a_2 2 · ─┬─ · ╰── source value 5 │ } else if $list_stream == true { ╰──── Error: nu:🐚:non_zero_exit_code × External command had a non-zero exit code ╭─[entry #7:1:29] 1 │ def a_2 [x] { if $x == 3 { ^false } else { $x } } · ──┬── · ╰── exited with code 1 ╰──── ``` </details> It also added a message to guide the usage of NU_BACKTRACE, see the last line in the following example: ```shell ls asdfasd Error: nu:🐚:io::not_found × I/O error ╰─▶ × Entity not found ╭─[entry #17:1:4] 1 │ ls asdfasd · ───┬─── · ╰── Entity not found ╰──── help: The error occurred at '/home/windsoilder/projects/nushell/asdfasd' set the `NU_BACKTRACE=1` environment variable to display a backtrace. ``` # Tests + Formatting Added some tests for the behavior. # After Submitting |
||
Bahex
|
b55ed69c92
|
fix range bugs in str substring, str index-of, slice, bytes at (#14863)
- fixes #14769 # Description ## Bugs - `str substring 0..<0` When passed a range containing no elements, for non-zero cases `str substring` behaves correctly: ```nushell ("hello world" | str substring 1..<1) == "" # => true ``` but if the range is `0..<0`, it returns the whole string instead ```nushell "hello world" | str substring 0..<0 # => hello world ``` - `[0 1 2] | range 0..<0` Similar behavior to `str substring` - `str index-of` - off-by-one on end bounds - underflow on negative start bounds - `bytes at` has inconsistent behavior, works correctly when the size is known, returns one byte less when it's not known (streaming) This can be demonstrated by comparing the outputs of following snippets ```nushell "hello world" | into binary | bytes at ..<5 | decode # => hello "hello world" | into binary | chunks 1 | bytes collect | bytes at ..<5 | decode # => hell ``` - `bytes at` panics on decreasing (`5..3`) ranges if the input size is known. Does not panic with streaming input. ## Changes - implement `FromValue` for `IntRange`, as it is very common to use integer ranges as arguments - `IntRange::absolute_start` can now point one-past-end - `IntRange::absolute_end` converts relative `Included` bounds to absolute `Excluded` bounds - `IntRange::absolute_bounds` is a convenience method that calls the other `absolute_*` methods and transforms reverse ranges to empty at `start` (`5..3` => `5..<5`) - refactored `str substring` tests to allow empty exclusive range tests - fix the `0..<0` case for `str substring` and `str index-of` - `IntRange::distance` never returns `Included(0)` As a general rule `Included(n) == Excluded(n + 1)`. This makes returning `Included(0)` bug prone as users of the function will likely rely on this general rule and cause bugs. - `ByteStream::slice` no longer has an off-by-one on inputs without a known size. This affected `bytes at`. - `bytes at` no longer panics on reverse ranges - `bytes at` is now consistent between streaming and non streaming inputs. # User-Facing Changes There should be no noticeable changes other than the bugfix. # Tests + Formatting - 🟢 toolkit fmt - 🟢 toolkit clippy - 🟢 toolkit test - 🟢 toolkit test stdlib # After Submitting N/A |
||
Piepmatz
|
080b501ba8
|
Fix cargo doc Warnings (#14948)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> As an avid `cargo doc` enjoyer I realized we had some doc warnings, so I fixed them. After this PR `cargo doc --workspace` should stop throwing warnings. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No code changes. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> We could add a `cargo doc` CI pipeline but usually running a full `cargo doc` takes like forever, so maybe we don't want that. |
||
|
Piepmatz
|
66bc0542e0
|
Refactor I/O Errors (#14927)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
As mentioned in #10698, we have too many `ShellError` variants, with
some even overlapping in meaning. This PR simplifies and improves I/O
error handling by restructuring `ShellError` related to I/O issues.
Previously, `ShellError::IOError` only contained a message string,
making it convenient but overly generic. It was widely used without
providing spans (#4323).
This PR introduces a new `ShellError::Io` variant that consolidates
multiple I/O-related errors (except for `ShellError::NetworkFailure`,
which remains distinct for now). The new `ShellError::Io` variant
replaces the following:
- `FileNotFound`
- `FileNotFoundCustom`
- `IOInterrupted`
- `IOError`
- `IOErrorSpanned`
- `NotADirectory`
- `DirectoryNotFound`
- `MoveNotPossible`
- `CreateNotPossible`
- `ChangeAccessTimeNotPossible`
- `ChangeModifiedTimeNotPossible`
- `RemoveNotPossible`
- `ReadingFile`
## The `IoError`
`IoError` includes the following fields:
1. **`kind`**: Extends `std::io::ErrorKind` to specify the type of I/O
error without needing new `ShellError` variants. This aligns with the
approach used in `std::io::Error`. This adds a second dimension to error
reporting by combining the `kind` field with `ShellError` variants,
making it easier to describe errors in more detail. As proposed by
@kubouch in [#design-discussion on
Discord](https://discord.com/channels/601130461678272522/615329862395101194/1323699197165178930),
this helps reduce the number of `ShellError` variants. In the error
report, the `kind` field is displayed as the "source" of the error,
e.g., "I/O error," followed by the specific kind of I/O error.
2. **`span`**: A non-optional field to encourage providing spans for
better error reporting (#4323).
3. **`path`**: Optional `PathBuf` to give context about the file or
directory involved in the error (#7695). If provided, it’s shown as a
help entry in error reports.
4. **`additional_context`**: Allows adding custom messages when the
span, kind, and path are insufficient. This is rendered in the error
report at the labeled span.
5. **`location`**: Sometimes, I/O errors occur in the engine itself and
are not caused directly by user input. In such cases, if we don’t have a
span and must set it to `Span::unknown()`, we need another way to
reference the error. For this, the `location` field uses the new
`Location` struct, which records the Rust file and line number where the
error occurred. This ensures that we at least know the Rust code
location that failed, helping with debugging. To make this work, a new
`location!` macro was added, which retrieves `file!`, `line!`, and
`column!` values accurately. If `Location::new` is used directly, it
issues a warning to remind developers to use the macro instead, ensuring
consistent and correct usage.
### Constructor Behavior
`IoError` provides five constructor methods:
- `new` and `new_with_additional_context`: Used for errors caused by
user input and require a valid (non-unknown) span to ensure precise
error reporting.
- `new_internal` and `new_internal_with_path`: Used for internal errors
where a span is not available. These methods require additional context
and the `Location` struct to pinpoint the source of the error in the
engine code.
- `factory`: Returns a closure that maps an `std::io::Error` to an
`IoError`. This is useful for handling multiple I/O errors that share
the same span and path, streamlining error handling in such cases.
## New Report Look
This is simulation how the I/O errors look like (the `open crates` is
simulated to show how internal errors are referenced now):
## `Span::test_data()`
To enable better testing, `Span::test_data()` now returns a value
distinct from `Span::unknown()`. Both `Span::test_data()` and
`Span::unknown()` refer to invalid source code, but having a separate
value for test data helps identify issues during testing while keeping
spans unique.
## Cursed Sneaky Error Transfers
I removed the conversions between `std::io::Error` and `ShellError` as
they often removed important information and were used too broadly to
handle I/O errors. This also removed the problematic implementation
found here:
|
||
Simon Curtis
|
f05162811c
|
Implementing ByteStream interuption on infinite stream (#13552)
# Description This PR should address #13530 by explicitly handling ByteStreams. The issue can be replicated easily on linux by running: ```nushell open /dev/urandom | into binary | bytes at ..10 ``` Would leave the output hanging and with no way to cancel it, this was likely because it was trying to collect the input stream and would not complete. I have also put in an error to say that using negative offsets for a bytestream without a length cannot be used. ```nushell ~/git/nushell> open /dev/urandom | into binary | bytes at (-1).. Error: nu:🐚:incorrect_value × Incorrect value. ╭─[entry #3:1:35] 1 │ open /dev/urandom | into binary | bytes at (-1).. · ────┬─── ───┬── · │ ╰── encountered here · ╰── Negative range values cannot be used with streams that don't specify a length ╰──── ``` # User-Facing Changes No operation changes, only the warning you get back for negative offsets # Tests + Formatting Ran `toolkit check pr ` with no errors or warnings Manual testing of the example commands above --------- Co-authored-by: Ian Manske <ian.manske@pm.me> Co-authored-by: Simon Curtis <simon.curtis@candc-uk.com> |
||
132ikl
|
214714e0ab
|
Add run-time type checking for command pipeline input (#14741)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds type checking of all command input types at run-time. Generally, these errors should be caught by the parser, but sometimes we can't know the type of a value at parse-time. The simplest example is using the `echo` command, which has an output type of `any`, so prefixing a literal with `echo` will bypass parse-time type checking. Before this PR, each command has to individually check its input types. This can result in scenarios where the input/output types don't match the actual command behavior. This can cause valid usage with an non-`any` type to become a parse-time error if a command is missing that type in its pipeline input/output (`drop nth` and `history import` do this before this PR). Alternatively, a command may not list a type in its input/output types, but doesn't actually reject that type in its code, which can have unintended side effects (`get` does this on an empty pipeline input, and `sort` used to before #13154). After this PR, the type of the pipeline input is checked to ensure it matches one of the input types listed in the proceeding command's input/output types. While each of the issues in the "before this PR" section could be addressed with each command individually, this PR solves this issue for _all_ commands. **This will likely cause some breakage**, as some commands have incorrect input/output types, and should be adjusted. Also, some scripts may have erroneous usage of commands. In writing this PR, I discovered that `toolkit.nu` was passing `null` values to `str join`, which doesn't accept nothing types (if folks think it should, we can adjust it in this PR or in a different PR). I found some issues in the standard library and its tests. I also found that carapace's vendor script had an incorrect chaining of `get -i`: ```nushell let expanded_alias = (scope aliases | where name == $spans.0 | get -i 0 | get -i expansion) ``` Before this PR, if the `get -i 0` ever actually did evaluate to `null`, the second `get` invocation would error since `get` doesn't operate on `null` values. After this PR, this is immediately a run-time error, alerting the user to the problematic code. As a side note, we'll need to PR this fix (`get -i 0 | get -i expansion` -> `get -i 0.expansion`) to carapace. A notable exception to the type checking is commands with input type of `nothing -> <type>`. In this case, any input type is allowed. This allows piping values into the command without an error being thrown. For example, `123 | echo $in` would be an error without this exception. Additionally, custom types bypass type checking (I believe this also happens during parsing, but not certain) I added a `is_subtype` method to `Value` and `PipelineData`. It functions slightly differently than `get_type().is_subtype()`, as noted in the doccomments. Notably, it respects structural typing of lists and tables. For example, the type of a value `[{a: 123} {a: 456, b: 789}]` is a subtype of `table<a: int>`, whereas the type returned by `Value::get_type` is a `list<any>`. Similarly, `PipelineData` has some special handling for `ListStream`s and `ByteStream`s. The latter was needed for this PR to work properly with external commands. Here's some examples. Before: ```nu 1..2 | drop nth 1 Error: nu::parser::input_type_mismatch × Command does not support range input. ╭─[entry #9:1:8] 1 │ 1..2 | drop nth 1 · ────┬─── · ╰── command doesn't support range input ╰──── echo 1..2 | drop nth 1 # => ╭───┬───╮ # => │ 0 │ 1 │ # => ╰───┴───╯ ``` After this PR, I've adjusted `drop nth`'s input/output types to accept range input. Before this PR, zip accepted any value despite not being listed in its input/output types. This caused different behavior depending on if you triggered a parse error or not: ```nushell 1 | zip [2] # => Error: nu::parser::input_type_mismatch # => # => × Command does not support int input. # => ╭─[entry #3:1:5] # => 1 │ 1 | zip [2] # => · ─┬─ # => · ╰── command doesn't support int input # => ╰──── echo 1 | zip [2] # => ╭───┬───────────╮ # => │ 0 │ ╭───┬───╮ │ # => │ │ │ 0 │ 1 │ │ # => │ │ │ 1 │ 2 │ │ # => │ │ ╰───┴───╯ │ # => ╰───┴───────────╯ ``` After this PR, it works the same in both cases. For cases like this, if we do decide we want `zip` or other commands to accept any input value, then we should explicitly add that to the input types. ```nushell 1 | zip [2] # => Error: nu::parser::input_type_mismatch # => # => × Command does not support int input. # => ╭─[entry #3:1:5] # => 1 │ 1 | zip [2] # => · ─┬─ # => · ╰── command doesn't support int input # => ╰──── echo 1 | zip [2] # => Error: nu:🐚:only_supports_this_input_type # => # => × Input type not supported. # => ╭─[entry #14:2:6] # => 2 │ echo 1 | zip [2] # => · ┬ ─┬─ # => · │ ╰── only list<any> and range input data is supported # => · ╰── input type: int # => ╰──── ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> **Breaking change**: The type of a command's input is now checked against the input/output types of that command at run-time. While these errors should mostly be caught at parse-time, in cases where they can't be detected at parse-time they will be caught at run-time instead. This applies to both internal commands and custom commands. Example function and corresponding parse-time error (same before and after PR): ```nushell def foo []: int -> nothing { print $"my cool int is ($in)" } 1 | foo # => my cool int is 1 "evil string" | foo # => Error: nu::parser::input_type_mismatch # => # => × Command does not support string input. # => ╭─[entry #16:1:17] # => 1 │ "evil string" | foo # => · ─┬─ # => · ╰── command doesn't support string input # => ╰──── # => ``` Before: ```nu echo "evil string" | foo # => my cool int is evil string ``` After: ```nu echo "evil string" | foo # => Error: nu:🐚:only_supports_this_input_type # => # => × Input type not supported. # => ╭─[entry #17:1:6] # => 1 │ echo "evil string" | foo # => · ──────┬────── ─┬─ # => · │ ╰── only int input data is supported # => · ╰── input type: string # => ╰──── ``` Known affected internal commands which erroneously accepted any type: * `str join` * `zip` * `reduce` # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> * Play whack-a-mole with the commands and scripts this will inevitably break |
||
|
132ikl
|
ed1381adc4
|
Change PipelineData::into_value to use internal Value's span before passed in span (#14757)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Changes the `Value` variant match arm of `PipelineData::into_value` to use the internal `Value`'s span instead of the span passed in by the user. This aligns more closely with the `ListStream` and `ByteStream` match arms, which already use their internal span, and allows errors to provide better diagnostics since the span information doesn't get lost when `into_value` is called. At the suggestion of @cptpiepmatz, if the `Value` has `Span::unknown` for some reason, then we replace the `Value`'s span with the passed in span. Before: ```nushell {} | get foo bar # => Error: nu:🐚:column_not_found # => # => × Cannot find column 'foo' # => ╭─[entry #43:2:6] # => 2 │ {} | get foo bar # => · ─┬─ ─┬─ # => · │ ╰── cannot find column 'foo' # => · ╰── value originates here # => ╰──── ``` After: ```nushell {} | get foo bar # => Error: nu:🐚:column_not_found # => # => × Cannot find column 'foo' # => ╭─[entry #2:2:1] # => 2 │ {} | get foo bar # => · ─┬ ─┬─ # => · │ ╰── cannot find column 'foo' # => · ╰── value originates here # => ╰──── ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> * Some errors may have more accurate info about where the value originates from # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> N/A |
||
|
Bahex
|
469e23cae4
|
Add bytes split command (#14652)
Related #10708 # Description Add `bytes split` command. `bytes split` splits its input on the provided separator on binary values _and_ binary streams without collecting. The separator can be a multiple character string or multiple byte binary. It can be used when neither `split row` (not streaming over raw input) nor `lines` (streaming, but can only split on newlines) is right. The backing iterator implemented in this PR, `SplitRead`, can be used to implement a streaming `split row` in the future. # User-Facing Changes `bytes split` command added, which can be used to split binary values and raw streams using a separator. # Tests + Formatting - 🟢 toolkit fmt - 🟢 toolkit clippy - 🟢 toolkit test - 🟢 toolkit test stdlib # After Submitting Mention in release notes. |
||
|
Piepmatz
|
3d5f853b03
|
Start to Add WASM Support Again (#14418)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> The [nushell/demo](https://github.com/nushell/demo) project successfully demonstrated running Nushell in the browser using WASM. However, the current version of Nushell cannot be easily built for the `wasm32-unknown-unknown` target, the default for `wasm-bindgen`. This PR introduces initial support for the `wasm32-unknown-unknown` target by disabling OS-dependent features such as filesystem access, IO, and platform/system-specific functionality. This separation is achieved using a new `os` feature in the following crates: - `nu-cmd-lang` - `nu-command` - `nu-engine` - `nu-protocol` The `os` feature includes all functionality that interacts with an operating system. It is enabled by default, but can be disabled using `--no-default-features`. All crates that depend on these core crates now use `--no-default-features` to allow compilation for WASM. To demonstrate compatibility, the following script builds all crates expected to work with WASM. Direct user interaction, running external commands, working with plugins, and features requiring `openssl` are out of scope for now due to their complexity or reliance on C libraries, which are difficult to compile and link in a WASM environment. ```nushell [ # compatible crates "nu-cmd-base", "nu-cmd-extra", "nu-cmd-lang", "nu-color-config", "nu-command", "nu-derive-value", "nu-engine", "nu-glob", "nu-json", "nu-parser", "nu-path", "nu-pretty-hex", "nu-protocol", "nu-std", "nu-system", "nu-table", "nu-term-grid", "nu-utils", "nuon" ] | each {cargo build -p $in --target wasm32-unknown-unknown --no-default-features} ``` ## Caveats This PR has a few caveats: 1. **`miette` and `terminal-size` Dependency Issue** `miette` depends on `terminal-size`, which uses `rustix` when the target is not Windows. However, `rustix` requires `std::os::unix`, which is unavailable in WASM. To address this, I opened a [PR](https://github.com/eminence/terminal-size/pull/68) for `terminal-size` to conditionally compile `rustix` only when the target is Unix. For now, the `Cargo.toml` includes patches to: - Use my forked version of `terminal-size`. - ~~Use an unreleased version of `miette` that depends on `terminal-size@0.4`.~~ These patches are temporary and can be removed once the upstream changes are merged and released. 2. **Test Output Adjustments** Due to the slight bump in the `miette` version, one test required adjustments to accommodate minor formatting changes in the error output, such as shifted newlines. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> This shouldn't break anything but allows using some crates for targeting `wasm32-unknown-unknown` to revive the demo page eventually. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` I did not add any extra tests, I just checked that compiling works, also when using the host target but unselecting the `os` feature. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> ~~Breaking the wasm support can be easily done by adding some `use`s or by adding a new dependency, we should definitely add some CI that also at least builds against wasm to make sure that building for it keep working.~~ I added a job to build wasm. --------- Co-authored-by: Ian Manske <ian.manske@pm.me> |
||
Ian Manske
|
4edce44689
|
Remove ListStream type (#14425)
# Description List values and list streams have the same type (`list<>`). Rather, streaming is a separate property of the pipeline/command output. This PR removes the unnecessary `ListStream` type. # User-Facing Changes Should be none, except `random dice` now has a more specific output type. |
||
|
132ikl
|
d69e131450
|
Rely on display_output hook for formatting values from evaluations (#14361)
# Description I was reading through the documentation yesterday, when I stumbled upon [this section](https://www.nushell.sh/book/pipelines.html#behind-the-scenes) explaining how command output is formatted using the `table` command. I was surprised that this section didn't mention the `display_output` hook, so I took a look in the code and was shocked to discovered that the documentation was correct, and the `table` command _is_ automatically applied to printed pipelines. This auto-tabling has two ramifications for the `display_output` hook: 1. The `table` command is called on the output of a pipeline after the `display_output` has run, even if `display_output` contains the table command. This means each pipeline output is roughly equivalent to the following (using `ls` as an example): ```nushell ls | do $config.hooks.display_output | table ``` 2. If `display_output` returns structured data, it will _still_ be formatted through the table command. This PR removes the auto-table when the `display_output` hook is set. The auto-table made sense before `display_output` was introduced, but to me, it now seems like unnecessary "automagic" which can be accomplished using existing Nushell features. This means that you can now pull back the curtain a bit, and replace your `display_output` hook with an empty closure (`$env.config.hooks.display_output = {||}`, setting it to null retains the previous behavior) to see the values printed normally without the table formatting. I think this is a good thing, and makes it easier to understand Nushell fundamentals. It is important to note that this PR does not change how `print` and other commands (well, specifically only `watch`) print out values. They continue to use `table` with no arguments, so changing your config/`display_output` hook won't affect what `print`ing a value does. Rel: [Discord discussion](https://discord.com/channels/601130461678272522/615329862395101194/1307102690848931904) (cc @dcarosone) # User-Facing Changes Pipelines are no longer automatically formatted using the `table` command. Instead, the `display_output` hook is used to format pipeline output. Most users should see no impact, as the default `display_output` hook already uses the `table` command. # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting Will update mentioned docs page to call out `display_output` hook. |
||
Darren Schroeder
|
af9c31152a
|
Add metadata on open --raw with bytestreams (#14141)
# Description This PR closes #14137 and allows the display hook to be set on byte streams. So, with a hook like this below. ```nushell display_output: { metadata access {|meta| match $meta.content_type? { "application/x-nuscript" | "application/x-nuon" | "text/x-nushell" => { nu-highlight }, "application/json" => { ^bat --language=json --color=always --style=plain --paging=never }, _ => {}, } } | table } ``` You could type `open toolkit.nu` and the text of toolkit.nu would be highlighted by nu-highlight. This PR also changes the way content-type is assigned with `open`. Previously it would only assign it if `--raw` was specified. Lastly, it changes the `is_external()` function to only say `ByteStreamSource::Child`'s are external instead of both Child and `ByteStreamSource::File`. Again, this was to allow the hook to function properly. I'm not sure what negative ramifications changing `is_external()` could have, but there may be some? # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Ian Manske
|
de08b68ba8
|
Fix try printing when it is not the last pipeline element (#13992)
# Description Fixes #13991. This was done by more clearly separating the case when a pipeline is drained vs when it is being written (to a file). I also added an `OutDest::Print` case which might not be strictly necessary, but is a helpful addition. # User-Facing Changes Bug fix. # Tests + Formatting Added a test. # After Submitting There are still a few redirection bugs that I found, but they require larger code changes, so I'll leave them until after the release. |
||
|
Ian Manske
|
03ee54a4df
|
Fix try not working with let, etc. (#13885)
# Description Partialy addresses #13868. `try` does not catch non-zero exit code errors from the last command in a pipeline if the result is assigned to a variable using `let` (or `mut`). This was fixed by adding a new `OutDest::Value` case. This is used when the pipeline is in a "value" position. I.e., it will be collected into a value. This ended up replacing most of the usages of `OutDest::Capture`. So, this PR also renames `OutDest::Capture` to `OutDest::PipeSeparate` to better fit the few remaining use cases for it. # User-Facing Changes Bug fix. # Tests + Formatting Added two tests. |
||
|
Ian Manske
|
3d008e2c4e
|
Error on non-zero exit statuses (#13515)
# Description This PR makes it so that non-zero exit codes and termination by signal are treated as a normal `ShellError`. Currently, these are silent errors. That is, if an external command fails, then it's code block is aborted, but the parent block can sometimes continue execution. E.g., see #8569 and this example: ```nushell [1 2] | each { ^false } ``` Before this would give: ``` ╭───┬──╮ │ 0 │ │ │ 1 │ │ ╰───┴──╯ ``` Now, this shows an error: ``` Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #1:1:2] 1 │ [1 2] | each { ^false } · ┬ · ╰── source value ╰──── Error: nu:🐚:non_zero_exit_code × External command had a non-zero exit code ╭─[entry #1:1:17] 1 │ [1 2] | each { ^false } · ──┬── · ╰── exited with code 1 ╰──── ``` This PR fixes #12874, fixes #5960, fixes #10856, and fixes #5347. This PR also partially addresses #10633 and #10624 (only the last command of a pipeline is currently checked). It looks like #8569 is already fixed, but this PR will make sure it is definitely fixed (fixes #8569). # User-Facing Changes - Non-zero exit codes and termination by signal now cause an error to be thrown. - The error record value passed to a `catch` block may now have an `exit_code` column containing the integer exit code if the error was due to an external command. - Adds new config values, `display_errors.exit_code` and `display_errors.termination_signal`, which determine whether an error message should be printed in the respective error cases. For non-interactive sessions, these are set to `true`, and for interactive sessions `display_errors.exit_code` is false (via the default config). # Tests Added a few tests. # After Submitting - Update docs and book. - Future work: - Error if other external commands besides the last in a pipeline exit with a non-zero exit code. Then, deprecate `do -c` since this will be the default behavior everywhere. - Add a better mechanism for exit codes and deprecate `$env.LAST_EXIT_CODE` (it's buggy). |
||
Devyn Cairns
|
39bda8986e
|
Make tee work more nicely with non-collections (#13652)
# Description This changes the behavior of `tee` to be more transparent when given a value that isn't a list or range. Previously, anything that wasn't a byte stream would converted to a list stream using the iterator implementation, which led to some surprising results. Instead, now, if the value is a string or binary, it will be treated the same way a byte stream is, and the output of `tee` is a byte stream instead of the original value. This is done so that we can synchronize with the other thread on collect, and potentially capture any error produced by the closure. For values that can't be converted to streams, the closure is just run with a clone of the value instead on another thread. Because we can't wait for the other thread, there is no way to send an error back to the original thread, so instead it's just written to stderr using `report_error_new()`. There are a couple of follow up edge cases I see where byte streams aren't necessarily treated exactly the same way strings are, but this should mostly be a good experience. Fixes #13489. # User-Facing Changes Breaking change. - `tee` now outputs and sends string/binary stream for string/binary input. - `tee` now outputs and sends the original value for any other input other than lists/ranges. # Tests + Formatting Added for new behavior. # After Submitting - [ ] release notes: breaking change, command change |
||
|
Devyn Cairns
|
4e205cd9a7
|
Add --raw switch to print for binary data (#13597)
# Description
Something I meant to add a long time ago. We currently don't have a
convenient way to print raw binary data intentionally. You can pipe it
through `cat` to turn it into an unknown stream, or write it to a file
and read it again, but we can't really just e.g. generate msgpack and
write it to stdout without this. For example:
```nushell
[abc def] | to msgpack | print --raw
```
This is useful for nushell scripts that will be piped into something
else. It also means that `nu_plugin_nu_example` probably doesn't need to
do this anymore, but I haven't adjusted it yet:
```nushell
def tell_nushell_encoding [] {
print -n "\u{0004}json"
}
```
This happens to work because 0x04 is a valid UTF-8 character, but it
wouldn't be possible if it were something above 0x80.
`--raw` also formats other things without `table`, I figured the two
things kind of go together. The output is kind of like `to text`.
Debatable whether that should share the same flag, but it was easier
that way and seemed reasonable.
# User-Facing Changes
- `print` new flag: `--raw`
# Tests + Formatting
Added tests.
# After Submitting
- [ ] release notes (command modified)
|
||
Andy Gayton
|
1cd0544a3f
|
fix: relay Signals reset to plugins (#13510)
This PR will close #13501 # Description This PR expands on [the relay of signals to running plugin processes](https://github.com/nushell/nushell/pull/13181). The Ctrlc relay has been generalized to SignalAction::Interrupt and when reset_signal is called on the main EngineState, a SignalAction::Reset is now relayed to running plugins. # User-Facing Changes The signal handler closure now takes a `signals::SignalAction`, while previously it took no arguments. The handler will now be called on both interrupt and reset. The method to register a handler on the plugin side is now called `register_signal_handler` instead of `register_ctrlc_handler` [example](https://github.com/nushell/nushell/pull/13510/files#diff-3e04dff88fd0780a49778a3d1eede092ec729a1264b4ef07ca0d2baa859dad05L38). This will only affect plugin authors who have started making use of https://github.com/nushell/nushell/pull/13181, which isn't currently part of an official release. The change will also require all of user's plugins to be recompiled in order that they don't error when a signal is received on the PluginInterface. # Testing ``` : example ctrlc interrupt status: false waiting for interrupt signal... ^Cinterrupt status: true peace. Error: × Operation interrupted ╭─[display_output hook:1:1] 1 │ if (term size).columns >= 100 { table -e } else { table } · ─┬ · ╰── This operation was interrupted ╰──── : example ctrlc interrupt status: false <-- NOTE status is false waiting for interrupt signal... ^Cinterrupt status: true peace. Error: × Operation interrupted ╭─[display_output hook:1:1] 1 │ if (term size).columns >= 100 { table -e } else { table } · ─┬ · ╰── This operation was interrupted ╰──── ``` |
||
Jack Wright
|
d081e3386f
|
Make pipeline metadata available to plugins (#13495)
# Description Fixes an issue with pipeline metadata not being passed to plugins. |
||
Stefan Holderbach
|
42531e017c
|
Clippy fixes from stable and nightly (#13455)
- **Doccomment style fixes** - **Forgotten stuff in `nu-pretty-hex`** - **Don't `for` around an `Option`** - and more I think the suggestions here are a net positive, some of the suggestions moved into #13498 feel somewhat arbitrary, I also raised https://github.com/rust-lang/rust-clippy/issues/13188 as the nightly `byte_char_slices` would require either a global allow or otherwise a ton of granular allows or possibly confusing bytestring literals. |
||
|
Andy Gayton
|
7b82c6b482
|
feat: make ctrlc available to plugins (#13181)
# Description This PR adds a new method to `EngineInterface`: `register_ctrlc_handler` which takes a closure to run when the plugin's driving engine receives a ctrlc-signal. It also adds a mirror of the `signals` attribute from the main shell `EngineState`. This is an example of how a plugin which makes a long poll http request can end the request on ctrlc: https://github.com/cablehead/nu_plugin_http/blob/main/src/commands/request.rs#L68-L77 To facilitate the feature, a new attribute has been added to `EngineState`: `ctrlc_handlers`. This is a Vec of closures that will be run when the engine's process receives a ctrlc signal. When plugins are added to an `engine_state` during a `merge_delta`, the engine passes the ctrlc_handlers to the plugin's `.configure_ctrlc_handler` method, which gives the plugin a chance to register a handler that sends a ctrlc packet through the `PluginInterface`, if an instance of the plugin is currently running. On the plugin side: `EngineInterface` also has a ctrlc_handlers Vec of closures. Plugin calls can use `register_ctrlc_handler` to register a closure that will be called in the plugin process when the PluginInput::Ctrlc command is received. For future reference these are some alternate places that were investigated for tying the ctrlc trigger to transmitting a Ctrlc packet through the `PluginInterface`: - Directly from `src/signals.rs`: the handler there would need a reference to the Vec<Arc<RegisteredPlugins>>, which would require us to wrap the plugins in a Mutex, which we don't want to do. - have `PersistentPlugin.get_plugin` pass down the engine's CtrlcHandlers to .get and then to .spawn (if the plugin isn't already running). Once we have CtrlcHandlers in spawn, we can register a handler to write directly to PluginInterface. We don't want to double down on passing engine_state to spawn this way though, as it's unpredictable because it would depend on whether the plugin has already been spawned or not. - pass `ctrlc_handlers` to PersistentPlugin::new so it can store it on itself so it's available to spawn. - in `PersistentPlugin.spawn`, create a handler that sends to a clone of the GC event loop's tx. this has the same issues with regards to how to get CtrlcHandlers to the spawn method, and is more complicated than a handler that writes directly to PluginInterface # User-Facing Changes No breaking changes --------- Co-authored-by: Ian Manske <ian.manske@pm.me> |
||
|
Devyn Cairns
|
d618fd0527
|
Fix bad method links in docstrings (#13471)
# Description Seems like I developed a bit of a bad habit of trying to link ```rust /// [`.foo()`] ``` in docstrings, and this just doesn't work automatically; you have to do ```rust /// [`.foo()`](Self::foo) ``` if you want it to actually link. I think I found and replaced all of these. # User-Facing Changes Just docs. |
||
|
Ian Manske
|
0918050ac8
|
Deprecate group in favor of chunks (#13377)
# Description The name of the `group` command is a little unclear/ambiguous. Everything I look at it, I think of `group-by`. I think `chunks` more clearly conveys what the `group` command does. Namely, it divides the input list into chunks of a certain size. For example, [`slice::chunks`](https://doc.rust-lang.org/std/primitive.slice.html#method.chunks) has the same name. So, this PR adds a new `chunks` command to replace the now deprecated `group` command. The `chunks` command is a refactored version of `group`. As such, there is a small performance improvement: ```nushell # $data is a very large list > bench { $data | chunks 2 } --rounds 30 | get mean 474ms 921µs 190ns # deprecation warning was disabled here for fairness > bench { $data | group 2 } --rounds 30 | get mean 592ms 702µs 440ns > bench { $data | chunks 200 } --rounds 30 | get mean 374ms 188µs 318ns > bench { $data | group 200 } --rounds 30 | get mean 481ms 264µs 869ns > bench { $data | chunks 1 } --rounds 30 | get mean 642ms 574µs 42ns > bench { $data | group 1 } --rounds 30 | get mean 981ms 602µs 513ns ``` # User-Facing Changes - `group` command has been deprecated in favor of new `chunks` command. - `chunks` errors when given a chunk size of `0` whereas `group` returns chunks with one element. # Tests + Formatting Added tests for `chunks`, since `group` did not have any tests. # After Submitting Update book if necessary. |
||
|
Stefan Holderbach
|
c5aa15c7f6
|
Add top-level crate documentation/READMEs (#12907)
# Description Add `README.md` files to each crate in our workspace (-plugins) and also include it in the `lib.rs` documentation for <docs.rs> (if there is no existing `lib.rs` crate documentation) In all new README I added the defensive comment that the crates are not considered stable for public consumption. If necessary we can adjust this if we deem a crate useful for plugin authors. |
||
|
Stefan Holderbach
|
076a29ae19
|
Document public types in nu-protocol (#12906)
- **Doc-comment public `nu-protocol` modules** - **Doccomment argument/signature/call stuff** - **Doccomment cell path types** - **Doccomment expression stuff** - **Doccomment import patterns** - **Doccomment pattern matching AST nodes** |
||
|
Devyn Cairns
|
d7392f1f3b
|
Internal representation (IR) compiler and evaluator (#13330)
# Description This PR adds an internal representation language to Nushell, offering an alternative evaluator based on simple instructions, stream-containing registers, and indexed control flow. The number of registers required is determined statically at compile-time, and the fixed size required is allocated upon entering the block. Each instruction is associated with a span, which makes going backwards from IR instructions to source code very easy. Motivations for IR: 1. **Performance.** By simplifying the evaluation path and making it more cache-friendly and branch predictor-friendly, code that does a lot of computation in Nushell itself can be sped up a decent bit. Because the IR is fairly easy to reason about, we can also implement optimization passes in the future to eliminate and simplify code. 2. **Correctness.** The instructions mostly have very simple and easily-specified behavior, so hopefully engine changes are a little bit easier to reason about, and they can be specified in a more formal way at some point. I have made an effort to document each of the instructions in the docs for the enum itself in a reasonably specific way. Some of the errors that would have happened during evaluation before are now moved to the compilation step instead, because they don't make sense to check during evaluation. 3. **As an intermediate target.** This is a good step for us to bring the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at some point, as code generated from new AST can be directly compared to code generated from old AST. If the IR code is functionally equivalent, it will behave the exact same way. 4. **Debugging.** With a little bit more work, we can probably give control over advancing the virtual machine that `IrBlock`s run on to some sort of external driver, making things like breakpoints and single stepping possible. Tools like `view ir` and [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than before to see what exactly is going on with your Nushell code. The goal is to eventually replace the AST evaluator entirely, once we're sure it's working just as well. You can help dogfood this by running Nushell with `$env.NU_USE_IR` set to some value. The environment variable is checked when Nushell starts, so config runs with IR, or it can also be set on a line at the REPL to change it dynamically. It is also checked when running `do` in case within a script you want to just run a specific piece of code with or without IR. # Example ```nushell view ir { |data| mut sum = 0 for n in $data { $sum += $n } $sum } ``` ```gas # 3 registers, 19 instructions, 0 bytes of data 0: load-literal %0, int(0) 1: store-variable var 904, %0 # let 2: drain %0 3: drop %0 4: load-variable %1, var 903 5: iterate %0, %1, end 15 # for, label(1), from(14:) 6: store-variable var 905, %0 7: load-variable %0, var 904 8: load-variable %2, var 905 9: binary-op %0, Math(Plus), %2 10: span %0 11: store-variable var 904, %0 12: load-literal %0, nothing 13: drain %0 14: jump 5 15: drop %0 # label(0), from(5:) 16: drain %0 17: load-variable %0, var 904 18: return %0 ``` # Benchmarks All benchmarks run on a base model Mac Mini M1. ## Iterative Fibonacci sequence This is about as best case as possible, making use of the much faster control flow. Most code will not experience a speed improvement nearly this large. ```nushell def fib [n: int] { mut a = 0 mut b = 1 for _ in 2..=$n { let c = $a + $b $a = $b $b = $c } $b } use std bench bench { 0..50 | each { |n| fib $n } } ``` IR disabled: ``` ╭───────┬─────────────────╮ │ mean │ 1ms 924µs 665ns │ │ min │ 1ms 700µs 83ns │ │ max │ 3ms 450µs 125ns │ │ std │ 395µs 759ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ``` IR enabled: ``` ╭───────┬─────────────────╮ │ mean │ 452µs 820ns │ │ min │ 427µs 417ns │ │ max │ 540µs 167ns │ │ std │ 17µs 158ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ```  ## [gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu) IR disabled: ``` ╭───┬──────────────────╮ │ 0 │ 27ms 929µs 958ns │ │ 1 │ 21ms 153µs 459ns │ │ 2 │ 18ms 639µs 666ns │ │ 3 │ 19ms 554µs 583ns │ │ 4 │ 13ms 383µs 375ns │ │ 5 │ 11ms 328µs 208ns │ │ 6 │ 5ms 659µs 542ns │ ╰───┴──────────────────╯ ``` IR enabled: ``` ╭───┬──────────────────╮ │ 0 │ 22ms 662µs │ │ 1 │ 17ms 221µs 792ns │ │ 2 │ 14ms 786µs 708ns │ │ 3 │ 13ms 876µs 834ns │ │ 4 │ 13ms 52µs 875ns │ │ 5 │ 11ms 269µs 666ns │ │ 6 │ 6ms 942µs 500ns │ ╰───┴──────────────────╯ ``` ## [random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu) I got pretty random results out of this benchmark so I decided not to include it. Not clear why. # User-Facing Changes - IR compilation errors may appear even if the user isn't evaluating with IR. - IR evaluation can be enabled by setting the `NU_USE_IR` environment variable to any value. - New command `view ir` pretty-prints the IR for a block, and `view ir --json` can be piped into an external tool like [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir). # Tests + Formatting All tests are passing with `NU_USE_IR=1`, and I've added some more eval tests to compare the results for some very core operations. I will probably want to add some more so we don't have to always check `NU_USE_IR=1 toolkit test --workspace` on a regular basis. # After Submitting - [ ] release notes - [ ] further documentation of instructions? - [ ] post-release: publish `nu_plugin_explore_ir` |
||
|
Ian Manske
|
399a7c8836
|
Add and use new Signals struct (#13314)
# Description This PR introduces a new `Signals` struct to replace our adhoc passing around of `ctrlc: Option<Arc<AtomicBool>>`. Doing so has a few benefits: - We can better enforce when/where resetting or triggering an interrupt is allowed. - Consolidates `nu_utils::ctrl_c::was_pressed` and other ad-hoc re-implementations into a single place: `Signals::check`. - This allows us to add other types of signals later if we want. E.g., exiting or suspension. - Similarly, we can more easily change the underlying implementation if we need to in the future. - Places that used to have a `ctrlc` of `None` now use `Signals::empty()`, so we can double check these usages for correctness in the future. |
||
|
Devyn Cairns
|
948b90299d
|
Preserve attributes on external ByteStreams (#13305)
# Description Bug fix: `PipelineData::check_external_failed()` was not preserving the original `type_` and `known_size` attributes of the stream passed in for streams that come from children, so `external-command | into binary` did not work properly and always ended up still being unknown type. # User-Facing Changes The following test case now works as expected: ```nushell > head -c 2 /dev/urandom | into binary # Expected: pretty hex dump of binary # Previous behavior: just raw binary in the terminal ``` # Tests + Formatting Added a test to cover this to `into binary` |
||
Jakub Žádník
|
3fae77209a
|
Revert "Span ID Refactor (Step 2): Make Call SpanId-friendly (#13268)" (#13292)
This reverts commit 0cfd5fbece6f25b54ab9dc417a9e06af9d83f282. The original PR messed up syntax higlighting of aliases and causes panics of completion in the presence of alias. <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jakub Žádník
|
0cfd5fbece
|
Span ID Refactor (Step 2): Make Call SpanId-friendly (#13268)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Part of https://github.com/nushell/nushell/issues/12963, step 2. This PR refactors Call and related argument structures to remove their dependency on `Expression::span` which will be removed in the future. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Should be none. If you see some error messages that look broken, please report. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jack Wright
|
0d060aeae8
|
Use pipeline data for http post|put|patch|delete commands. (#13254)
# Description Provides the ability to use http commands as part of a pipeline. Additionally, this pull requests extends the pipeline metadata to add a content_type field. The content_type metadata field allows commands such as `to json` to set the metadata in the pipeline allowing the http commands to use it when making requests. This pull request also introduces the ability to directly stream http requests from streaming pipelines. One other small change is that Content-Type will always be set if it is passed in to the http commands, either indirectly or throw the content type flag. Previously it was not preserved with requests that were not of type json or form data. # User-Facing Changes * `http post`, `http put`, `http patch`, `http delete` can be used as part of a pipeline * `to text`, `to json`, `from json` all set the content_type metadata field and the http commands will utilize them when making requests. |
||
|
Ian Manske
|
c5d716951f
|
Allow byte streams with unknown type to be compatiable with binary (#12959)
# Description Currently, this pipeline doesn't work `open --raw file | take 100`, since the type of the byte stream is `Unknown`, but `take` expects `Binary` streams. This PR changes commands that expect `ByteStreamType::Binary` to also work with `ByteStreamType::Unknown`. This was done by adding two new methods to `ByteStreamType`: `is_binary_coercible` and `is_string_coercible`. These return true if the type is `Unknown` or matches the type in the method name. |
||
|
Devyn Cairns
|
b06f31d3c6
|
Make from json --objects streaming (#12949)
# Description Makes the `from json --objects` command produce a stream, and read lazily from an input stream to produce its output. Also added a helper, `PipelineData::get_type()`, to make it easier to construct a wrong type error message when matching on `PipelineData`. I expect checking `PipelineData` for either a string value or an `Unknown` or `String` typed `ByteStream` will be very, very common. I would have liked to have a helper that just returns a readable stream from either, but that would either be a bespoke enum or a `Box<dyn BufRead>`, which feels like it wouldn't be so great for performance. So instead, taking the approach I did here is probably better - having a function that accepts the `impl BufRead` and matching to use it. # User-Facing Changes - `from json --objects` no longer collects its input, and can be used for large datasets or streams that produce values over time. # Tests + Formatting All passing. # After Submitting - [ ] release notes --------- Co-authored-by: Ian Manske <ian.manske@pm.me> |
||
|
Ian Manske
|
c98960d053
|
Take owned Read and Write (#12909)
# Description As @YizhePKU pointed out, the [Rust API guidelines](https://rust-lang.github.io/api-guidelines/interoperability.html#generic-readerwriter-functions-take-r-read-and-w-write-by-value-c-rw-value) recommend that generic functions take readers and writers by value and not by reference. This PR changes `copy_with_interupt` and few other places to take owned `Read` and `Write` instead of mutable references. |
||
|
Devyn Cairns
|
c61075e20e
|
Add string/binary type color to ByteStream (#12897)
# Description
This PR allows byte streams to optionally be colored as being
specifically binary or string data, which guarantees that they'll be
converted to `Binary` or `String` appropriately on `into_value()`,
making them compatible with `Type` guarantees. This makes them
significantly more broadly usable for command input and output.
There is still an `Unknown` type for byte streams coming from external
commands, which uses the same behavior as we previously did where it's a
string if it's UTF-8.
A small number of commands were updated to take advantage of this, just
to prove the point. I will be adding more after this merges.
# User-Facing Changes
- New types in `describe`: `string (stream)`, `binary (stream)`
- These commands now return a stream if their input was a stream:
- `into binary`
- `into string`
- `bytes collect`
- `str join`
- `first` (binary)
- `last` (binary)
- `take` (binary)
- `skip` (binary)
- Streams that are explicitly binary colored will print as a streaming
hexdump
- example:
```nushell
1.. | each { into binary } | bytes collect
```
# Tests + Formatting
I've added some tests to cover it at a basic level, and it doesn't break
anything existing, but I do think more would be nice. Some of those will
come when I modify more commands to stream.
# After Submitting
There are a few things I'm not quite satisfied with:
- **String trimming behavior.** We automatically trim newlines from
streams from external commands, but I don't think we should do this with
internal commands. If I call a command that happens to turn my string
into a stream, I don't want the newline to suddenly disappear. I changed
this to specifically do it only on `Child` and `File`, but I don't know
if this is quite right, and maybe we should bring back the old flag for
`trim_end_newline`
- **Known binary always resulting in a hexdump.** It would be nice to
have a `print --raw`, so that we can put binary data on stdout
explicitly if we want to. This PR doesn't change how external commands
work though - they still dump straight to stdout.
Otherwise, here's the normal checklist:
- [ ] release notes
- [ ] docs update for plugin protocol changes (added `type` field)
---------
Co-authored-by: Ian Manske <ian.manske@pm.me>
|
||
|
Ian Manske
|
cc9f41e553
|
Use CommandType in more places (#12832)
# Description
Kind of a vague title, but this PR does two main things:
1. Rather than overriding functions like `Command::is_parser_keyword`,
this PR instead changes commands to override `Command::command_type`.
The `CommandType` returned by `Command::command_type` is then used to
automatically determine whether `Command::is_parser_keyword` and the
other `is_{type}` functions should return true. These changes allow us
to remove the `CommandType::Other` case and should also guarantee than
only one of the `is_{type}` functions on `Command` will return true.
2. Uses the new, reworked `Command::command_type` function in the `scope
commands` and `which` commands.
# User-Facing Changes
- Breaking change for `scope commands`: multiple columns (`is_builtin`,
`is_keyword`, `is_plugin`, etc.) have been merged into the `type`
column.
- Breaking change: the `which` command can now report `plugin` or
`keyword` instead of `built-in` in the `type` column. It may also now
report `external` instead of `custom` in the `type` column for known
`extern`s.
|
||
|
Ian Manske
|
580c60bb82
|
Preserve metadata in more places (#12848)
# Description This PR makes some commands and areas of code preserve pipeline metadata. This is in an attempt to make the issue described in #12599 and #9456 less likely to occur. That is, reading and writing to the same file in a pipeline will result in an empty file. Since we preserve metadata in more places now, there will be a higher chance that we successfully detect this error case and abort the pipeline. |
||
|
Devyn Cairns
|
59f7c523fa
|
Fix the way the output of table is printed in print() (#12895)
# Description Forgot that I fixed this already on my branch, but when printing without a display output hook, the implicit call to `table` gets its output mangled with newlines (since #12774). This happens when running `nu -c` or a script file. Here's that fix in one PR so it can be merged easily. # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` |
||
|
Ian Manske
|
2a09dccc11
|
Bytestream touchup (#12886)
# Description Adds some docs and a small fix to `Chunks`. |
||
|
Ian Manske
|
6fd854ed9f
|
Replace ExternalStream with new ByteStream type (#12774)
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes #7017, fixes #10763, and fixes #12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
|