mirror of
https://github.com/nushell/nushell.git
synced 2025-05-08 12:04:25 +02:00
080b501ba8
25 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Piepmatz
|
080b501ba8
|
Fix cargo doc Warnings (#14948)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> As an avid `cargo doc` enjoyer I realized we had some doc warnings, so I fixed them. After this PR `cargo doc --workspace` should stop throwing warnings. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No code changes. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> We could add a `cargo doc` CI pipeline but usually running a full `cargo doc` takes like forever, so maybe we don't want that. |
||
|
Piepmatz
|
66bc0542e0
|
Refactor I/O Errors (#14927)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
As mentioned in #10698, we have too many `ShellError` variants, with
some even overlapping in meaning. This PR simplifies and improves I/O
error handling by restructuring `ShellError` related to I/O issues.
Previously, `ShellError::IOError` only contained a message string,
making it convenient but overly generic. It was widely used without
providing spans (#4323).
This PR introduces a new `ShellError::Io` variant that consolidates
multiple I/O-related errors (except for `ShellError::NetworkFailure`,
which remains distinct for now). The new `ShellError::Io` variant
replaces the following:
- `FileNotFound`
- `FileNotFoundCustom`
- `IOInterrupted`
- `IOError`
- `IOErrorSpanned`
- `NotADirectory`
- `DirectoryNotFound`
- `MoveNotPossible`
- `CreateNotPossible`
- `ChangeAccessTimeNotPossible`
- `ChangeModifiedTimeNotPossible`
- `RemoveNotPossible`
- `ReadingFile`
## The `IoError`
`IoError` includes the following fields:
1. **`kind`**: Extends `std::io::ErrorKind` to specify the type of I/O
error without needing new `ShellError` variants. This aligns with the
approach used in `std::io::Error`. This adds a second dimension to error
reporting by combining the `kind` field with `ShellError` variants,
making it easier to describe errors in more detail. As proposed by
@kubouch in [#design-discussion on
Discord](https://discord.com/channels/601130461678272522/615329862395101194/1323699197165178930),
this helps reduce the number of `ShellError` variants. In the error
report, the `kind` field is displayed as the "source" of the error,
e.g., "I/O error," followed by the specific kind of I/O error.
2. **`span`**: A non-optional field to encourage providing spans for
better error reporting (#4323).
3. **`path`**: Optional `PathBuf` to give context about the file or
directory involved in the error (#7695). If provided, it’s shown as a
help entry in error reports.
4. **`additional_context`**: Allows adding custom messages when the
span, kind, and path are insufficient. This is rendered in the error
report at the labeled span.
5. **`location`**: Sometimes, I/O errors occur in the engine itself and
are not caused directly by user input. In such cases, if we don’t have a
span and must set it to `Span::unknown()`, we need another way to
reference the error. For this, the `location` field uses the new
`Location` struct, which records the Rust file and line number where the
error occurred. This ensures that we at least know the Rust code
location that failed, helping with debugging. To make this work, a new
`location!` macro was added, which retrieves `file!`, `line!`, and
`column!` values accurately. If `Location::new` is used directly, it
issues a warning to remind developers to use the macro instead, ensuring
consistent and correct usage.
### Constructor Behavior
`IoError` provides five constructor methods:
- `new` and `new_with_additional_context`: Used for errors caused by
user input and require a valid (non-unknown) span to ensure precise
error reporting.
- `new_internal` and `new_internal_with_path`: Used for internal errors
where a span is not available. These methods require additional context
and the `Location` struct to pinpoint the source of the error in the
engine code.
- `factory`: Returns a closure that maps an `std::io::Error` to an
`IoError`. This is useful for handling multiple I/O errors that share
the same span and path, streamlining error handling in such cases.
## New Report Look
This is simulation how the I/O errors look like (the `open crates` is
simulated to show how internal errors are referenced now):
## `Span::test_data()`
To enable better testing, `Span::test_data()` now returns a value
distinct from `Span::unknown()`. Both `Span::test_data()` and
`Span::unknown()` refer to invalid source code, but having a separate
value for test data helps identify issues during testing while keeping
spans unique.
## Cursed Sneaky Error Transfers
I removed the conversions between `std::io::Error` and `ShellError` as
they often removed important information and were used too broadly to
handle I/O errors. This also removed the problematic implementation
found here:
|
||
132ikl
|
214714e0ab
|
Add run-time type checking for command pipeline input (#14741)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds type checking of all command input types at run-time. Generally, these errors should be caught by the parser, but sometimes we can't know the type of a value at parse-time. The simplest example is using the `echo` command, which has an output type of `any`, so prefixing a literal with `echo` will bypass parse-time type checking. Before this PR, each command has to individually check its input types. This can result in scenarios where the input/output types don't match the actual command behavior. This can cause valid usage with an non-`any` type to become a parse-time error if a command is missing that type in its pipeline input/output (`drop nth` and `history import` do this before this PR). Alternatively, a command may not list a type in its input/output types, but doesn't actually reject that type in its code, which can have unintended side effects (`get` does this on an empty pipeline input, and `sort` used to before #13154). After this PR, the type of the pipeline input is checked to ensure it matches one of the input types listed in the proceeding command's input/output types. While each of the issues in the "before this PR" section could be addressed with each command individually, this PR solves this issue for _all_ commands. **This will likely cause some breakage**, as some commands have incorrect input/output types, and should be adjusted. Also, some scripts may have erroneous usage of commands. In writing this PR, I discovered that `toolkit.nu` was passing `null` values to `str join`, which doesn't accept nothing types (if folks think it should, we can adjust it in this PR or in a different PR). I found some issues in the standard library and its tests. I also found that carapace's vendor script had an incorrect chaining of `get -i`: ```nushell let expanded_alias = (scope aliases | where name == $spans.0 | get -i 0 | get -i expansion) ``` Before this PR, if the `get -i 0` ever actually did evaluate to `null`, the second `get` invocation would error since `get` doesn't operate on `null` values. After this PR, this is immediately a run-time error, alerting the user to the problematic code. As a side note, we'll need to PR this fix (`get -i 0 | get -i expansion` -> `get -i 0.expansion`) to carapace. A notable exception to the type checking is commands with input type of `nothing -> <type>`. In this case, any input type is allowed. This allows piping values into the command without an error being thrown. For example, `123 | echo $in` would be an error without this exception. Additionally, custom types bypass type checking (I believe this also happens during parsing, but not certain) I added a `is_subtype` method to `Value` and `PipelineData`. It functions slightly differently than `get_type().is_subtype()`, as noted in the doccomments. Notably, it respects structural typing of lists and tables. For example, the type of a value `[{a: 123} {a: 456, b: 789}]` is a subtype of `table<a: int>`, whereas the type returned by `Value::get_type` is a `list<any>`. Similarly, `PipelineData` has some special handling for `ListStream`s and `ByteStream`s. The latter was needed for this PR to work properly with external commands. Here's some examples. Before: ```nu 1..2 | drop nth 1 Error: nu::parser::input_type_mismatch × Command does not support range input. ╭─[entry #9:1:8] 1 │ 1..2 | drop nth 1 · ────┬─── · ╰── command doesn't support range input ╰──── echo 1..2 | drop nth 1 # => ╭───┬───╮ # => │ 0 │ 1 │ # => ╰───┴───╯ ``` After this PR, I've adjusted `drop nth`'s input/output types to accept range input. Before this PR, zip accepted any value despite not being listed in its input/output types. This caused different behavior depending on if you triggered a parse error or not: ```nushell 1 | zip [2] # => Error: nu::parser::input_type_mismatch # => # => × Command does not support int input. # => ╭─[entry #3:1:5] # => 1 │ 1 | zip [2] # => · ─┬─ # => · ╰── command doesn't support int input # => ╰──── echo 1 | zip [2] # => ╭───┬───────────╮ # => │ 0 │ ╭───┬───╮ │ # => │ │ │ 0 │ 1 │ │ # => │ │ │ 1 │ 2 │ │ # => │ │ ╰───┴───╯ │ # => ╰───┴───────────╯ ``` After this PR, it works the same in both cases. For cases like this, if we do decide we want `zip` or other commands to accept any input value, then we should explicitly add that to the input types. ```nushell 1 | zip [2] # => Error: nu::parser::input_type_mismatch # => # => × Command does not support int input. # => ╭─[entry #3:1:5] # => 1 │ 1 | zip [2] # => · ─┬─ # => · ╰── command doesn't support int input # => ╰──── echo 1 | zip [2] # => Error: nu:🐚:only_supports_this_input_type # => # => × Input type not supported. # => ╭─[entry #14:2:6] # => 2 │ echo 1 | zip [2] # => · ┬ ─┬─ # => · │ ╰── only list<any> and range input data is supported # => · ╰── input type: int # => ╰──── ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> **Breaking change**: The type of a command's input is now checked against the input/output types of that command at run-time. While these errors should mostly be caught at parse-time, in cases where they can't be detected at parse-time they will be caught at run-time instead. This applies to both internal commands and custom commands. Example function and corresponding parse-time error (same before and after PR): ```nushell def foo []: int -> nothing { print $"my cool int is ($in)" } 1 | foo # => my cool int is 1 "evil string" | foo # => Error: nu::parser::input_type_mismatch # => # => × Command does not support string input. # => ╭─[entry #16:1:17] # => 1 │ "evil string" | foo # => · ─┬─ # => · ╰── command doesn't support string input # => ╰──── # => ``` Before: ```nu echo "evil string" | foo # => my cool int is evil string ``` After: ```nu echo "evil string" | foo # => Error: nu:🐚:only_supports_this_input_type # => # => × Input type not supported. # => ╭─[entry #17:1:6] # => 1 │ echo "evil string" | foo # => · ──────┬────── ─┬─ # => · │ ╰── only int input data is supported # => · ╰── input type: string # => ╰──── ``` Known affected internal commands which erroneously accepted any type: * `str join` * `zip` * `reduce` # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> * Play whack-a-mole with the commands and scripts this will inevitably break |
||
|
132ikl
|
ed1381adc4
|
Change PipelineData::into_value to use internal Value's span before passed in span (#14757)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Changes the `Value` variant match arm of `PipelineData::into_value` to use the internal `Value`'s span instead of the span passed in by the user. This aligns more closely with the `ListStream` and `ByteStream` match arms, which already use their internal span, and allows errors to provide better diagnostics since the span information doesn't get lost when `into_value` is called. At the suggestion of @cptpiepmatz, if the `Value` has `Span::unknown` for some reason, then we replace the `Value`'s span with the passed in span. Before: ```nushell {} | get foo bar # => Error: nu:🐚:column_not_found # => # => × Cannot find column 'foo' # => ╭─[entry #43:2:6] # => 2 │ {} | get foo bar # => · ─┬─ ─┬─ # => · │ ╰── cannot find column 'foo' # => · ╰── value originates here # => ╰──── ``` After: ```nushell {} | get foo bar # => Error: nu:🐚:column_not_found # => # => × Cannot find column 'foo' # => ╭─[entry #2:2:1] # => 2 │ {} | get foo bar # => · ─┬ ─┬─ # => · │ ╰── cannot find column 'foo' # => · ╰── value originates here # => ╰──── ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> * Some errors may have more accurate info about where the value originates from # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> N/A |
||
Ian Manske
|
4edce44689
|
Remove ListStream type (#14425)
# Description List values and list streams have the same type (`list<>`). Rather, streaming is a separate property of the pipeline/command output. This PR removes the unnecessary `ListStream` type. # User-Facing Changes Should be none, except `random dice` now has a more specific output type. |
||
|
132ikl
|
d69e131450
|
Rely on display_output hook for formatting values from evaluations (#14361)
# Description I was reading through the documentation yesterday, when I stumbled upon [this section](https://www.nushell.sh/book/pipelines.html#behind-the-scenes) explaining how command output is formatted using the `table` command. I was surprised that this section didn't mention the `display_output` hook, so I took a look in the code and was shocked to discovered that the documentation was correct, and the `table` command _is_ automatically applied to printed pipelines. This auto-tabling has two ramifications for the `display_output` hook: 1. The `table` command is called on the output of a pipeline after the `display_output` has run, even if `display_output` contains the table command. This means each pipeline output is roughly equivalent to the following (using `ls` as an example): ```nushell ls | do $config.hooks.display_output | table ``` 2. If `display_output` returns structured data, it will _still_ be formatted through the table command. This PR removes the auto-table when the `display_output` hook is set. The auto-table made sense before `display_output` was introduced, but to me, it now seems like unnecessary "automagic" which can be accomplished using existing Nushell features. This means that you can now pull back the curtain a bit, and replace your `display_output` hook with an empty closure (`$env.config.hooks.display_output = {||}`, setting it to null retains the previous behavior) to see the values printed normally without the table formatting. I think this is a good thing, and makes it easier to understand Nushell fundamentals. It is important to note that this PR does not change how `print` and other commands (well, specifically only `watch`) print out values. They continue to use `table` with no arguments, so changing your config/`display_output` hook won't affect what `print`ing a value does. Rel: [Discord discussion](https://discord.com/channels/601130461678272522/615329862395101194/1307102690848931904) (cc @dcarosone) # User-Facing Changes Pipelines are no longer automatically formatted using the `table` command. Instead, the `display_output` hook is used to format pipeline output. Most users should see no impact, as the default `display_output` hook already uses the `table` command. # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting Will update mentioned docs page to call out `display_output` hook. |
||
|
Ian Manske
|
de08b68ba8
|
Fix try printing when it is not the last pipeline element (#13992)
# Description Fixes #13991. This was done by more clearly separating the case when a pipeline is drained vs when it is being written (to a file). I also added an `OutDest::Print` case which might not be strictly necessary, but is a helpful addition. # User-Facing Changes Bug fix. # Tests + Formatting Added a test. # After Submitting There are still a few redirection bugs that I found, but they require larger code changes, so I'll leave them until after the release. |
||
|
Ian Manske
|
03ee54a4df
|
Fix try not working with let, etc. (#13885)
# Description Partialy addresses #13868. `try` does not catch non-zero exit code errors from the last command in a pipeline if the result is assigned to a variable using `let` (or `mut`). This was fixed by adding a new `OutDest::Value` case. This is used when the pipeline is in a "value" position. I.e., it will be collected into a value. This ended up replacing most of the usages of `OutDest::Capture`. So, this PR also renames `OutDest::Capture` to `OutDest::PipeSeparate` to better fit the few remaining use cases for it. # User-Facing Changes Bug fix. # Tests + Formatting Added two tests. |
||
|
Ian Manske
|
3d008e2c4e
|
Error on non-zero exit statuses (#13515)
# Description This PR makes it so that non-zero exit codes and termination by signal are treated as a normal `ShellError`. Currently, these are silent errors. That is, if an external command fails, then it's code block is aborted, but the parent block can sometimes continue execution. E.g., see #8569 and this example: ```nushell [1 2] | each { ^false } ``` Before this would give: ``` ╭───┬──╮ │ 0 │ │ │ 1 │ │ ╰───┴──╯ ``` Now, this shows an error: ``` Error: nu:🐚:eval_block_with_input × Eval block failed with pipeline input ╭─[entry #1:1:2] 1 │ [1 2] | each { ^false } · ┬ · ╰── source value ╰──── Error: nu:🐚:non_zero_exit_code × External command had a non-zero exit code ╭─[entry #1:1:17] 1 │ [1 2] | each { ^false } · ──┬── · ╰── exited with code 1 ╰──── ``` This PR fixes #12874, fixes #5960, fixes #10856, and fixes #5347. This PR also partially addresses #10633 and #10624 (only the last command of a pipeline is currently checked). It looks like #8569 is already fixed, but this PR will make sure it is definitely fixed (fixes #8569). # User-Facing Changes - Non-zero exit codes and termination by signal now cause an error to be thrown. - The error record value passed to a `catch` block may now have an `exit_code` column containing the integer exit code if the error was due to an external command. - Adds new config values, `display_errors.exit_code` and `display_errors.termination_signal`, which determine whether an error message should be printed in the respective error cases. For non-interactive sessions, these are set to `true`, and for interactive sessions `display_errors.exit_code` is false (via the default config). # Tests Added a few tests. # After Submitting - Update docs and book. - Future work: - Error if other external commands besides the last in a pipeline exit with a non-zero exit code. Then, deprecate `do -c` since this will be the default behavior everywhere. - Add a better mechanism for exit codes and deprecate `$env.LAST_EXIT_CODE` (it's buggy). |
||
Devyn Cairns
|
39bda8986e
|
Make tee work more nicely with non-collections (#13652)
# Description This changes the behavior of `tee` to be more transparent when given a value that isn't a list or range. Previously, anything that wasn't a byte stream would converted to a list stream using the iterator implementation, which led to some surprising results. Instead, now, if the value is a string or binary, it will be treated the same way a byte stream is, and the output of `tee` is a byte stream instead of the original value. This is done so that we can synchronize with the other thread on collect, and potentially capture any error produced by the closure. For values that can't be converted to streams, the closure is just run with a clone of the value instead on another thread. Because we can't wait for the other thread, there is no way to send an error back to the original thread, so instead it's just written to stderr using `report_error_new()`. There are a couple of follow up edge cases I see where byte streams aren't necessarily treated exactly the same way strings are, but this should mostly be a good experience. Fixes #13489. # User-Facing Changes Breaking change. - `tee` now outputs and sends string/binary stream for string/binary input. - `tee` now outputs and sends the original value for any other input other than lists/ranges. # Tests + Formatting Added for new behavior. # After Submitting - [ ] release notes: breaking change, command change |
||
|
Devyn Cairns
|
4e205cd9a7
|
Add --raw switch to print for binary data (#13597)
# Description
Something I meant to add a long time ago. We currently don't have a
convenient way to print raw binary data intentionally. You can pipe it
through `cat` to turn it into an unknown stream, or write it to a file
and read it again, but we can't really just e.g. generate msgpack and
write it to stdout without this. For example:
```nushell
[abc def] | to msgpack | print --raw
```
This is useful for nushell scripts that will be piped into something
else. It also means that `nu_plugin_nu_example` probably doesn't need to
do this anymore, but I haven't adjusted it yet:
```nushell
def tell_nushell_encoding [] {
print -n "\u{0004}json"
}
```
This happens to work because 0x04 is a valid UTF-8 character, but it
wouldn't be possible if it were something above 0x80.
`--raw` also formats other things without `table`, I figured the two
things kind of go together. The output is kind of like `to text`.
Debatable whether that should share the same flag, but it was easier
that way and seemed reasonable.
# User-Facing Changes
- `print` new flag: `--raw`
# Tests + Formatting
Added tests.
# After Submitting
- [ ] release notes (command modified)
|
||
Stefan Holderbach
|
42531e017c
|
Clippy fixes from stable and nightly (#13455)
- **Doccomment style fixes** - **Forgotten stuff in `nu-pretty-hex`** - **Don't `for` around an `Option`** - and more I think the suggestions here are a net positive, some of the suggestions moved into #13498 feel somewhat arbitrary, I also raised https://github.com/rust-lang/rust-clippy/issues/13188 as the nightly `byte_char_slices` would require either a global allow or otherwise a ton of granular allows or possibly confusing bytestring literals. |
||
|
Devyn Cairns
|
d618fd0527
|
Fix bad method links in docstrings (#13471)
# Description Seems like I developed a bit of a bad habit of trying to link ```rust /// [`.foo()`] ``` in docstrings, and this just doesn't work automatically; you have to do ```rust /// [`.foo()`](Self::foo) ``` if you want it to actually link. I think I found and replaced all of these. # User-Facing Changes Just docs. |
||
|
Ian Manske
|
0918050ac8
|
Deprecate group in favor of chunks (#13377)
# Description The name of the `group` command is a little unclear/ambiguous. Everything I look at it, I think of `group-by`. I think `chunks` more clearly conveys what the `group` command does. Namely, it divides the input list into chunks of a certain size. For example, [`slice::chunks`](https://doc.rust-lang.org/std/primitive.slice.html#method.chunks) has the same name. So, this PR adds a new `chunks` command to replace the now deprecated `group` command. The `chunks` command is a refactored version of `group`. As such, there is a small performance improvement: ```nushell # $data is a very large list > bench { $data | chunks 2 } --rounds 30 | get mean 474ms 921µs 190ns # deprecation warning was disabled here for fairness > bench { $data | group 2 } --rounds 30 | get mean 592ms 702µs 440ns > bench { $data | chunks 200 } --rounds 30 | get mean 374ms 188µs 318ns > bench { $data | group 200 } --rounds 30 | get mean 481ms 264µs 869ns > bench { $data | chunks 1 } --rounds 30 | get mean 642ms 574µs 42ns > bench { $data | group 1 } --rounds 30 | get mean 981ms 602µs 513ns ``` # User-Facing Changes - `group` command has been deprecated in favor of new `chunks` command. - `chunks` errors when given a chunk size of `0` whereas `group` returns chunks with one element. # Tests + Formatting Added tests for `chunks`, since `group` did not have any tests. # After Submitting Update book if necessary. |
||
|
Devyn Cairns
|
d7392f1f3b
|
Internal representation (IR) compiler and evaluator (#13330)
# Description This PR adds an internal representation language to Nushell, offering an alternative evaluator based on simple instructions, stream-containing registers, and indexed control flow. The number of registers required is determined statically at compile-time, and the fixed size required is allocated upon entering the block. Each instruction is associated with a span, which makes going backwards from IR instructions to source code very easy. Motivations for IR: 1. **Performance.** By simplifying the evaluation path and making it more cache-friendly and branch predictor-friendly, code that does a lot of computation in Nushell itself can be sped up a decent bit. Because the IR is fairly easy to reason about, we can also implement optimization passes in the future to eliminate and simplify code. 2. **Correctness.** The instructions mostly have very simple and easily-specified behavior, so hopefully engine changes are a little bit easier to reason about, and they can be specified in a more formal way at some point. I have made an effort to document each of the instructions in the docs for the enum itself in a reasonably specific way. Some of the errors that would have happened during evaluation before are now moved to the compilation step instead, because they don't make sense to check during evaluation. 3. **As an intermediate target.** This is a good step for us to bring the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at some point, as code generated from new AST can be directly compared to code generated from old AST. If the IR code is functionally equivalent, it will behave the exact same way. 4. **Debugging.** With a little bit more work, we can probably give control over advancing the virtual machine that `IrBlock`s run on to some sort of external driver, making things like breakpoints and single stepping possible. Tools like `view ir` and [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than before to see what exactly is going on with your Nushell code. The goal is to eventually replace the AST evaluator entirely, once we're sure it's working just as well. You can help dogfood this by running Nushell with `$env.NU_USE_IR` set to some value. The environment variable is checked when Nushell starts, so config runs with IR, or it can also be set on a line at the REPL to change it dynamically. It is also checked when running `do` in case within a script you want to just run a specific piece of code with or without IR. # Example ```nushell view ir { |data| mut sum = 0 for n in $data { $sum += $n } $sum } ``` ```gas # 3 registers, 19 instructions, 0 bytes of data 0: load-literal %0, int(0) 1: store-variable var 904, %0 # let 2: drain %0 3: drop %0 4: load-variable %1, var 903 5: iterate %0, %1, end 15 # for, label(1), from(14:) 6: store-variable var 905, %0 7: load-variable %0, var 904 8: load-variable %2, var 905 9: binary-op %0, Math(Plus), %2 10: span %0 11: store-variable var 904, %0 12: load-literal %0, nothing 13: drain %0 14: jump 5 15: drop %0 # label(0), from(5:) 16: drain %0 17: load-variable %0, var 904 18: return %0 ``` # Benchmarks All benchmarks run on a base model Mac Mini M1. ## Iterative Fibonacci sequence This is about as best case as possible, making use of the much faster control flow. Most code will not experience a speed improvement nearly this large. ```nushell def fib [n: int] { mut a = 0 mut b = 1 for _ in 2..=$n { let c = $a + $b $a = $b $b = $c } $b } use std bench bench { 0..50 | each { |n| fib $n } } ``` IR disabled: ``` ╭───────┬─────────────────╮ │ mean │ 1ms 924µs 665ns │ │ min │ 1ms 700µs 83ns │ │ max │ 3ms 450µs 125ns │ │ std │ 395µs 759ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ``` IR enabled: ``` ╭───────┬─────────────────╮ │ mean │ 452µs 820ns │ │ min │ 427µs 417ns │ │ max │ 540µs 167ns │ │ std │ 17µs 158ns │ │ times │ [list 50 items] │ ╰───────┴─────────────────╯ ```  ## [gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu) IR disabled: ``` ╭───┬──────────────────╮ │ 0 │ 27ms 929µs 958ns │ │ 1 │ 21ms 153µs 459ns │ │ 2 │ 18ms 639µs 666ns │ │ 3 │ 19ms 554µs 583ns │ │ 4 │ 13ms 383µs 375ns │ │ 5 │ 11ms 328µs 208ns │ │ 6 │ 5ms 659µs 542ns │ ╰───┴──────────────────╯ ``` IR enabled: ``` ╭───┬──────────────────╮ │ 0 │ 22ms 662µs │ │ 1 │ 17ms 221µs 792ns │ │ 2 │ 14ms 786µs 708ns │ │ 3 │ 13ms 876µs 834ns │ │ 4 │ 13ms 52µs 875ns │ │ 5 │ 11ms 269µs 666ns │ │ 6 │ 6ms 942µs 500ns │ ╰───┴──────────────────╯ ``` ## [random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu) I got pretty random results out of this benchmark so I decided not to include it. Not clear why. # User-Facing Changes - IR compilation errors may appear even if the user isn't evaluating with IR. - IR evaluation can be enabled by setting the `NU_USE_IR` environment variable to any value. - New command `view ir` pretty-prints the IR for a block, and `view ir --json` can be piped into an external tool like [`explore ir`](https://github.com/devyn/nu_plugin_explore_ir). # Tests + Formatting All tests are passing with `NU_USE_IR=1`, and I've added some more eval tests to compare the results for some very core operations. I will probably want to add some more so we don't have to always check `NU_USE_IR=1 toolkit test --workspace` on a regular basis. # After Submitting - [ ] release notes - [ ] further documentation of instructions? - [ ] post-release: publish `nu_plugin_explore_ir` |
||
|
Ian Manske
|
399a7c8836
|
Add and use new Signals struct (#13314)
# Description This PR introduces a new `Signals` struct to replace our adhoc passing around of `ctrlc: Option<Arc<AtomicBool>>`. Doing so has a few benefits: - We can better enforce when/where resetting or triggering an interrupt is allowed. - Consolidates `nu_utils::ctrl_c::was_pressed` and other ad-hoc re-implementations into a single place: `Signals::check`. - This allows us to add other types of signals later if we want. E.g., exiting or suspension. - Similarly, we can more easily change the underlying implementation if we need to in the future. - Places that used to have a `ctrlc` of `None` now use `Signals::empty()`, so we can double check these usages for correctness in the future. |
||
|
Devyn Cairns
|
948b90299d
|
Preserve attributes on external ByteStreams (#13305)
# Description Bug fix: `PipelineData::check_external_failed()` was not preserving the original `type_` and `known_size` attributes of the stream passed in for streams that come from children, so `external-command | into binary` did not work properly and always ended up still being unknown type. # User-Facing Changes The following test case now works as expected: ```nushell > head -c 2 /dev/urandom | into binary # Expected: pretty hex dump of binary # Previous behavior: just raw binary in the terminal ``` # Tests + Formatting Added a test to cover this to `into binary` |
||
Jakub Žádník
|
3fae77209a
|
Revert "Span ID Refactor (Step 2): Make Call SpanId-friendly (#13268)" (#13292)
This reverts commit
|
||
|
Jakub Žádník
|
0cfd5fbece
|
Span ID Refactor (Step 2): Make Call SpanId-friendly (#13268)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Part of https://github.com/nushell/nushell/issues/12963, step 2. This PR refactors Call and related argument structures to remove their dependency on `Expression::span` which will be removed in the future. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Should be none. If you see some error messages that look broken, please report. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Devyn Cairns
|
b06f31d3c6
|
Make from json --objects streaming (#12949)
# Description Makes the `from json --objects` command produce a stream, and read lazily from an input stream to produce its output. Also added a helper, `PipelineData::get_type()`, to make it easier to construct a wrong type error message when matching on `PipelineData`. I expect checking `PipelineData` for either a string value or an `Unknown` or `String` typed `ByteStream` will be very, very common. I would have liked to have a helper that just returns a readable stream from either, but that would either be a bespoke enum or a `Box<dyn BufRead>`, which feels like it wouldn't be so great for performance. So instead, taking the approach I did here is probably better - having a function that accepts the `impl BufRead` and matching to use it. # User-Facing Changes - `from json --objects` no longer collects its input, and can be used for large datasets or streams that produce values over time. # Tests + Formatting All passing. # After Submitting - [ ] release notes --------- Co-authored-by: Ian Manske <ian.manske@pm.me> |
||
|
Devyn Cairns
|
c61075e20e
|
Add string/binary type color to ByteStream (#12897)
# Description
This PR allows byte streams to optionally be colored as being
specifically binary or string data, which guarantees that they'll be
converted to `Binary` or `String` appropriately on `into_value()`,
making them compatible with `Type` guarantees. This makes them
significantly more broadly usable for command input and output.
There is still an `Unknown` type for byte streams coming from external
commands, which uses the same behavior as we previously did where it's a
string if it's UTF-8.
A small number of commands were updated to take advantage of this, just
to prove the point. I will be adding more after this merges.
# User-Facing Changes
- New types in `describe`: `string (stream)`, `binary (stream)`
- These commands now return a stream if their input was a stream:
- `into binary`
- `into string`
- `bytes collect`
- `str join`
- `first` (binary)
- `last` (binary)
- `take` (binary)
- `skip` (binary)
- Streams that are explicitly binary colored will print as a streaming
hexdump
- example:
```nushell
1.. | each { into binary } | bytes collect
```
# Tests + Formatting
I've added some tests to cover it at a basic level, and it doesn't break
anything existing, but I do think more would be nice. Some of those will
come when I modify more commands to stream.
# After Submitting
There are a few things I'm not quite satisfied with:
- **String trimming behavior.** We automatically trim newlines from
streams from external commands, but I don't think we should do this with
internal commands. If I call a command that happens to turn my string
into a stream, I don't want the newline to suddenly disappear. I changed
this to specifically do it only on `Child` and `File`, but I don't know
if this is quite right, and maybe we should bring back the old flag for
`trim_end_newline`
- **Known binary always resulting in a hexdump.** It would be nice to
have a `print --raw`, so that we can put binary data on stdout
explicitly if we want to. This PR doesn't change how external commands
work though - they still dump straight to stdout.
Otherwise, here's the normal checklist:
- [ ] release notes
- [ ] docs update for plugin protocol changes (added `type` field)
---------
Co-authored-by: Ian Manske <ian.manske@pm.me>
|
||
|
Ian Manske
|
cc9f41e553
|
Use CommandType in more places (#12832)
# Description
Kind of a vague title, but this PR does two main things:
1. Rather than overriding functions like `Command::is_parser_keyword`,
this PR instead changes commands to override `Command::command_type`.
The `CommandType` returned by `Command::command_type` is then used to
automatically determine whether `Command::is_parser_keyword` and the
other `is_{type}` functions should return true. These changes allow us
to remove the `CommandType::Other` case and should also guarantee than
only one of the `is_{type}` functions on `Command` will return true.
2. Uses the new, reworked `Command::command_type` function in the `scope
commands` and `which` commands.
# User-Facing Changes
- Breaking change for `scope commands`: multiple columns (`is_builtin`,
`is_keyword`, `is_plugin`, etc.) have been merged into the `type`
column.
- Breaking change: the `which` command can now report `plugin` or
`keyword` instead of `built-in` in the `type` column. It may also now
report `external` instead of `custom` in the `type` column for known
`extern`s.
|
||
|
Ian Manske
|
580c60bb82
|
Preserve metadata in more places (#12848)
# Description This PR makes some commands and areas of code preserve pipeline metadata. This is in an attempt to make the issue described in #12599 and #9456 less likely to occur. That is, reading and writing to the same file in a pipeline will result in an empty file. Since we preserve metadata in more places now, there will be a higher chance that we successfully detect this error case and abort the pipeline. |
||
|
Devyn Cairns
|
59f7c523fa
|
Fix the way the output of table is printed in print() (#12895)
# Description Forgot that I fixed this already on my branch, but when printing without a display output hook, the implicit call to `table` gets its output mangled with newlines (since #12774). This happens when running `nu -c` or a script file. Here's that fix in one PR so it can be merged easily. # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` |
||
|
Ian Manske
|
6fd854ed9f
|
Replace ExternalStream with new ByteStream type (#12774)
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes #7017, fixes #10763, and fixes #12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
|