# Description
Fixes#15485
This PR changes pipeline checking to keep track of all possible output
types instead of only first type matching input type which appears in
the input/output types. For example, in this command:
```nushell
def foo []: [int -> string, int -> record] {
# ...
}
```
An `int` input to the command may result in a string or a record to be
output. Before this PR, Nushell would always assume that an `int` input
would cause a `string` output because it's the first matching
input/output type pair. This would cause issues during type checking
where the parser would incorrectly determine the output type. After this
PR, Nushell considers the command to output either a string or a record.
# User-Facing Changes
* Parse-time pipeline type checking now properly supports commands with
multiple pipeline output types for the same pipeline input type
# Tests + Formatting
Added a couple tests
# After Submitting
N/A
---------
Co-authored-by: Bahex <Bahex@users.noreply.github.com>
# Description
As title: this pr is try to introduce 2 functions to `PipelineData`:
1. PipelineData::list_stream --> create a PipelineData::ListStream
2. PipelineData::byte_stream -> create a PipelineData::ByteStream
And use these functions everywhere.
### Reason behind this change
I tried to implement `pipefail` feature, but this would required to
change `PipelineData` from enum to struct. So use these functions can
reduce diff if I finally change to struct. [Discord message
here](https://discord.com/channels/601130461678272522/615962413203718156/1396999539000479784)
is my plan.
# User-Facing Changes
NaN
# Tests + Formatting
NaN
# After Submitting
NaN
Restricts custom completion from universal to internal arguments only.
Pros:
1. Less memory
2. More flexible for later customizations, e.g. #14923
Cons:
1. limited customization capabilities, but at least covers all currently
existing features in nushell.
# Description
Mostly vibe coded by [Zed AI](https://zed.dev/ai) with a single prompt.
LGTM, but I'm not so sure @ysthakur
# User-Facing Changes
Hopefully none.
# Tests + Formatting
+3
# After Submitting
---------
Co-authored-by: Yash Thakur <45539777+ysthakur@users.noreply.github.com>
# Description
Fixes#15308. Surprisingly, I can't find any more issues that bring this
up.
This PR adds a check for interrupt before evaluating each IR
instruction. This makes it possible to ctrl-c out of things that were
difficult/impossible previously, like:
```nushell
loop {}
```
@devyn also [mentioned
previously](https://discord.com/channels/601130461678272522/615329862395101194/1268674828492083327)
that this might be a feasible option, but mentioned some performance
concerns.
I did some benchmarking previously but it turns out tango isn't the
greatest for this. I don't have hard numbers anymore but based on some
experimenting I shared in the Discord it seems like no experimental
option and limiting this to specific instructions should keep the
performance impact minimal.
<details>
<summary>Old benchmarking info</summary>
I did some benchmarks against main. It seems like most benchmarks are
between 1-6% slower, with some oddball exceptions.
The worst case seems to a giant loop, which makes sense since it's
mostly just jumping back to the beginning, meaning we hit the check over
and over again.
With `bench --pretty -n 100 { for i in 0..1000000 { 1 } }`, it seems
like the `ir-interrupt` branch is ~10% slower for this stress case:
```
main: 94ms 323µs 29ns +/- 1ms 291µs 759ns
ir-interrupt: 103ms 54µs 708ns +/- 1ms 606µs 571ns
(103ms + 54µs + 708ns) / (94ms + 323µs + 29ns) = 1.0925720801438639
```
The performance numbers here aren't great, but they're not terrible
either, and I think the usability improvement is probably worth it here.
<details>
<summary>Tango benchmarks</summary>
```
load_standard_lib [ 2.8 ms ... 2.9 ms ] +3.53%*
record_create_1 [ 101.8 us ... 107.0 us ] +5.12%*
record_create_10 [ 132.3 us ... 135.6 us ] +2.48%*
record_create_100 [ 349.7 us ... 354.6 us ] +1.39%*
record_create_1000 [ 3.7 ms ... 3.6 ms ] -1.30%
record_flat_access_1 [ 94.8 us ... 99.9 us ] +5.35%*
record_flat_access_10 [ 96.2 us ... 101.3 us ] +5.33%*
record_flat_access_100 [ 106.0 us ... 111.4 us ] +5.07%*
record_flat_access_1000 [ 203.0 us ... 208.5 us ] +2.69%
record_nested_access_1 [ 95.3 us ... 100.1 us ] +5.03%*
record_nested_access_2 [ 98.4 us ... 104.6 us ] +6.26%*
record_nested_access_4 [ 100.8 us ... 105.4 us ] +4.56%*
record_nested_access_8 [ 104.7 us ... 108.1 us ] +3.23%
record_nested_access_16 [ 110.6 us ... 115.8 us ] +4.71%*
record_nested_access_32 [ 119.0 us ... 124.0 us ] +4.20%*
record_nested_access_64 [ 137.4 us ... 142.2 us ] +3.47%*
record_nested_access_128 [ 176.7 us ... 181.8 us ] +2.87%*
record_insert_1_1 [ 106.2 us ... 111.9 us ] +5.35%*
record_insert_10_1 [ 111.2 us ... 116.0 us ] +4.28%*
record_insert_100_1 [ 134.3 us ... 139.8 us ] +4.06%*
record_insert_1000_1 [ 387.8 us ... 417.1 us ] +7.55%
record_insert_1_10 [ 162.9 us ... 171.5 us ] +5.23%*
record_insert_10_10 [ 159.5 us ... 166.2 us ] +4.23%*
record_insert_100_10 [ 183.3 us ... 190.7 us ] +4.04%*
record_insert_1000_10 [ 411.7 us ... 418.7 us ] +1.71%*
table_create_1 [ 117.2 us ... 120.3 us ] +2.66%
table_create_10 [ 144.9 us ... 152.2 us ] +5.02%*
table_create_100 [ 476.0 us ... 484.8 us ] +1.86%*
table_create_1000 [ 3.7 ms ... 3.7 ms ] +1.55%
table_get_1 [ 121.6 us ... 127.0 us ] +4.40%*
table_get_10 [ 112.1 us ... 117.4 us ] +4.71%*
table_get_100 [ 124.0 us ... 129.5 us ] +4.41%*
table_get_1000 [ 229.9 us ... 245.5 us ] +6.75%*
table_select_1 [ 111.1 us ... 115.6 us ] +4.03%*
table_select_10 [ 112.1 us ... 118.4 us ] +5.65%*
table_select_100 [ 142.2 us ... 147.2 us ] +3.53%*
table_select_1000 [ 383.5 us ... 370.3 us ] -3.43%*
table_insert_row_1_1 [ 122.5 us ... 127.8 us ] +4.35%*
table_insert_row_10_1 [ 123.6 us ... 128.6 us ] +3.99%*
table_insert_row_100_1 [ 124.9 us ... 131.2 us ] +5.05%*
table_insert_row_1000_1 [ 177.3 us ... 182.7 us ] +3.07%*
table_insert_row_1_10 [ 225.7 us ... 234.5 us ] +3.90%*
table_insert_row_10_10 [ 229.2 us ... 235.4 us ] +2.69%*
table_insert_row_100_10 [ 253.3 us ... 257.6 us ] +1.69%
table_insert_row_1000_10 [ 311.1 us ... 318.3 us ] +2.32%*
table_insert_col_1_1 [ 107.6 us ... 113.3 us ] +5.35%*
table_insert_col_10_1 [ 109.6 us ... 115.3 us ] +5.27%*
table_insert_col_100_1 [ 155.5 us ... 159.8 us ] +2.71%*
table_insert_col_1000_1 [ 476.6 us ... 480.8 us ] +0.88%*
table_insert_col_1_10 [ 167.7 us ... 177.4 us ] +5.75%
table_insert_col_10_10 [ 178.5 us ... 194.8 us ] +9.16%*
table_insert_col_100_10 [ 314.4 us ... 322.3 us ] +2.53%*
table_insert_col_1000_10 [ 1.7 ms ... 1.7 ms ] +1.35%*
eval_interleave_100 [ 485.8 us ... 506.5 us ] +4.27%
eval_interleave_1000 [ 3.3 ms ... 3.2 ms ] -1.51%
eval_interleave_10000 [ 31.5 ms ... 31.0 ms ] -1.80%
eval_interleave_with_interrupt_100 [ 473.2 us ... 479.6 us ] +1.35%
eval_interleave_with_interrupt_1000 [ 3.2 ms ... 3.2 ms ] -1.26%
eval_interleave_with_interrupt_10000 [ 32.3 ms ... 31.1 ms ] -3.81%
eval_for_1 [ 124.4 us ... 130.1 us ] +4.60%*
eval_for_10 [ 124.4 us ... 130.3 us ] +4.80%*
eval_for_100 [ 134.5 us ... 141.5 us ] +5.18%*
eval_for_1000 [ 222.7 us ... 244.0 us ] +9.59%*
eval_for_10000 [ 1.0 ms ... 1.2 ms ] +13.86%*
eval_each_1 [ 146.9 us ... 153.0 us ] +4.15%*
eval_each_10 [ 152.3 us ... 158.8 us ] +4.26%*
eval_each_100 [ 169.3 us ... 175.6 us ] +3.76%*
eval_each_1000 [ 346.8 us ... 357.4 us ] +3.06%*
eval_each_10000 [ 2.1 ms ... 2.2 ms ] +2.37%*
eval_par_each_1 [ 194.3 us ... 203.5 us ] +4.73%*
eval_par_each_10 [ 186.4 us ... 193.1 us ] +3.59%*
eval_par_each_100 [ 213.5 us ... 222.2 us ] +4.08%*
eval_par_each_1000 [ 433.4 us ... 437.4 us ] +0.93%
eval_par_each_10000 [ 2.4 ms ... 2.4 ms ] -0.40%
eval_default_config [ 406.3 us ... 414.9 us ] +2.12%*
eval_default_env [ 475.7 us ... 495.1 us ] +4.08%*
encode_json_100_5 [ 132.7 us ... 128.9 us ] -2.88%*
encode_json_10000_15 [ 35.5 ms ... 34.0 ms ] -4.15%*
encode_msgpack_100_5 [ 85.0 us ... 83.9 us ] -1.20%*
encode_msgpack_10000_15 [ 22.2 ms ... 21.7 ms ] -2.17%*
decode_json_100_5 [ 431.3 us ... 421.0 us ] -2.40%*
decode_json_10000_15 [ 119.7 ms ... 117.6 ms ] -1.76%*
decode_msgpack_100_5 [ 160.6 us ... 151.1 us ] -5.90%*
decode_msgpack_10000_15 [ 44.0 ms ... 43.1 ms ] -2.12%*
```
</details>
</details>
# User-Facing Changes
* It should be possible to ctrl-c in situations where it was not
previously possible
# Tests + Formatting
N/A
# After Submitting
N/A
# Description
Adds a proper `ShellWarning` enum which has the same functionality as
`ParseWarning`.
Also moves the deprecation from #15806 into `ShellWarning::Deprecated`
with `ReportMode::FirstUse`, so that warning will only pop up once now.
# User-Facing Changes
Technically the change to the deprecation warning from #15806 is user
facing but it's really not worth listing in the changelog
# Description
Adds an `extern` definition (`KnownExternal`) for the `nu` command. This
way you can do `help nu` and get tab completions for flags on the `nu`
executable
# User-Facing Changes
* You can now view the flags for Nushell itself with `help nu`, and tab
completion for flags on `nu` works
1.88.0 was released today, clippy now lints (machine-applicable)
against:
- format strings with empty braces that could be inlined
- easy win
- `manual_abs_diff`
- returning of a stored result of the last expression.
- this can be somewhat contentious but touched only a few places
# Description
Adds formatting for code in backticks in `help` output. If it's possible
to highlight syntax (`nu-highlight` is available and there's no invalid
syntax) then it's highlighted. If the syntax is invalid or not an
internal command, then it's dimmed and italicized. like some of the
output from `std/help`. If `use_ansi_coloring` is `false`, then we leave
the backticks alone. Here's a couple examples:
(note on this one: usually we can highlight partial commands, like `get`
in the `select` help page which is invalid according to `nu-check` but
is still properly highlighted, however `where` is special cased and just
typing `where` with no row condition is highlighted with the garbage
style so `where` alone isn't highlighted here)
here's the `where` page with `$env.config.use_ansi_coloring = false`:
Technically, some syntax is valid but isn't really "Nushell code". For
example, the `select` help page has a line that says "Select just the
\`name\` column". If you just type `name` in the REPL, Nushell treats it
as an external command, but for the purposes of highlighted we actually
want this to fall back to the generic dimmed/italic style. This is
accomplished by temporarily setting the `shape_external` and
`shape_externalarg` color config to the generic/fallback style, and then
restoring the color config after highlighting. This is a bit hack-ish
but it seems to work pretty well.
# User-Facing Changes
- `help` command now supports code backtick formatting. Code will be
highlighted using `nu-highlight` if possible, otherwise it will fall
back to a generic format.
- Adds `--reject-garbage` flag to `nu-highlight` which will return an
error on invalid syntax (which would otherwise be highlighted with
`$env.config.color_config.shape_garbage`)
# Tests + Formatting
Added tests for the regex. I don't think tests for the actual
highlighting are very necessary since the failure mode is graceful and
it would be difficult to meaningfully test.
# After Submitting
N/A
---------
Co-authored-by: Piepmatz <git+github@cptpiepmatz.de>
# Description
Use hashset for existence checking.
Still needs a vector collection to keep the column order for tables.
# User-Facing Changes
Should be None
# Description
Closes#15848. Currently, we expand unquoted strings to absolute paths
if they are of type `path` or `directory`. This PR makes this no longer
happen. `~`, `.`, and `..+` are still expanded, but a path like
`.../foo/bar/..` will only be turned into `../../foo`, rather than a
full absolute path.
This is mostly so that paths don't get modified before being sent to
known external commands (as in the linked issue). But also, it seems
unnecessary to make all unquoted paths absolute.
After feedback from @132ikl, this PR also makes it so that unquoted
paths are expanded at parse time, so that it matches the runtime
behavior. Previously, `path` expressions were turned into strings
verbatim, while `directory` expressions were treated as not being const.
API change: `nu_path::expansions::expand_path` is now exposed as
`nu_path::expand_path`.
# User-Facing Changes
This has the potential to silently break a lot of scripts. For example,
if someone has a command that expects an already-expanded absolute path,
changes the current working directory, and then passes the path
somewhere, they will now need to use `path expand` to expand the path
themselves before changing the current working directory.
# Tests + Formatting
Just added one test to make sure unquoted `path` arguments aren't made
absolute.
# After Submitting
This is a breaking change, so will need to be mentioned in the release
notes.
# Description
Bump nushell to development version.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
- related #10698
# Description
In my endeavor to make the `ShellError` less crowded I moved the job
related errors into `ShellError::Job` with a `JobError` enum. Mostly I
just moved the codes, errors and labels over to the new enum.
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
Bump dev version to 0.104.2
Related:
- #15683
- #14551
- #849
- #12701
- #11527
# Description
Currently various commands have differing behavior regarding cell-paths
```nushell
{a: 1, A: 2} | get a A
# => ╭───┬───╮
# => │ 0 │ 2 │
# => │ 1 │ 2 │
# => ╰───┴───╯
{a: 1, A: 2} | select a A
# => ╭───┬───╮
# => │ a │ 1 │
# => │ A │ 2 │
# => ╰───┴───╯
{A: 1} | update a 2
# => Error: nu:🐚:column_not_found
# =>
# => × Cannot find column 'a'
# => ╭─[entry #62:1:1]
# => 1 │ {A: 1} | update a 2
# => · ───┬── ┬
# => · │ ╰── cannot find column 'a'
# => · ╰── value originates here
# => ╰────
```
Proposal: making cell-path access case-sensitive by default and adding
new syntax for case-insensitive parts, similar to optional (?) parts.
```nushell
{FOO: BAR}.foo
# => Error: nu:🐚:name_not_found
# =>
# => × Name not found
# => ╭─[entry #60:1:21]
# => 1 │ {FOO: BAR}.foo
# => · ─┬─
# => · ╰── did you mean 'FOO'?
# => ╰────
{FOO: BAR}.foo!
# => BAR
```
This would solve the problem of case sensitivity for all commands
without causing an explosion of flags _and_ make it more granular
Assigning to a field using a case-insensitive path is case-preserving.
```nushell
mut val = {FOO: "I'm FOO"}; $val
# => ╭─────┬─────────╮
# => │ FOO │ I'm FOO │
# => ╰─────┴─────────╯
$val.foo! = "I'm still FOO"; $val
# => ╭─────┬───────────────╮
# => │ FOO │ I'm still FOO │
# => ╰─────┴───────────────╯
```
For `update`, case-insensitive is case-preserving.
```nushell
{FOO: 1} | update foo! { $in + 1 }
# => ╭─────┬───╮
# => │ FOO │ 2 │
# => ╰─────┴───╯
```
`insert` can insert values into nested values so accessing into existing
columns is case-insensitive, but creating new columns uses the cell-path
as it is.
So `insert foo! ...` and `insert FOO! ...` would work exactly as they do
without `!`
```nushell
{FOO: {quox: 0}}
# => ╭─────┬──────────────╮
# => │ │ ╭──────┬───╮ │
# => │ FOO │ │ quox │ 0 │ │
# => │ │ ╰──────┴───╯ │
# => ╰─────┴──────────────╯
{FOO: {quox: 0}} | insert foo.bar 1
# => ╭─────┬──────────────╮
# => │ │ ╭──────┬───╮ │
# => │ FOO │ │ quox │ 0 │ │
# => │ │ ╰──────┴───╯ │
# => │ │ ╭─────┬───╮ │
# => │ foo │ │ bar │ 1 │ │
# => │ │ ╰─────┴───╯ │
# => ╰─────┴──────────────╯
{FOO: {quox: 0}} | insert foo!.bar 1
# => ╭─────┬──────────────╮
# => │ │ ╭──────┬───╮ │
# => │ FOO │ │ quox │ 0 │ │
# => │ │ │ bar │ 1 │ │
# => │ │ ╰──────┴───╯ │
# => ╰─────┴──────────────╯
```
`upsert` is tricky, depending on the input, the data might end up with
different column names in rows. We can either forbid case-insensitive
cell-paths for `upsert` or trust the user to keep their data in a
sensible shape.
This would be a breaking change as it would make existing cell-path
accesses case-sensitive, however the case-sensitivity is already
inconsistent and any attempt at making it consistent would be a breaking
change.
> What about `$env`?
1. Initially special case it so it keeps its current behavior.
2. Accessing environment variables with non-matching paths gives a
deprecation warning urging users to either use exact casing or use the
new explicit case-sensitivity syntax
3. Eventuall remove `$env`'s special case, making `$env` accesses
case-sensitive by default as well.
> `$env.ENV_CONVERSIONS`?
In addition to `from_string` and `to_string` add an optional field to
opt into case insensitive/preserving behavior.
# User-Facing Changes
- `get`, `where` and other previously case-insensitive commands are now
case-sensitive by default.
- `get`'s `--sensitive` flag removed, similar to `--ignore-errors` there
is now an `--ignore-case` flag that treats all parts of the cell-path as
case-insensitive.
- Users can explicitly choose the case case-sensitivity of cell-path
accesses or commands.
# Tests + Formatting
Existing tests required minimal modification. ***However, new tests are
not yet added***.
- 🟢 toolkit fmt
- 🟢 toolkit clippy
- 🟢 toolkit test
- 🟢 toolkit test stdlib
# After Submitting
- Update the website to include the new syntax
- Update [tree-sitter-nu](https://github.com/nushell/tree-sitter-nu)
---------
Co-authored-by: Bahex <17417311+Bahex@users.noreply.github.com>
# Description
While working on something else, I noticed that
`Value::follow_cell_path` receives `self`.
While it would be ideal for the signature to be `(&'a self, cell_path)
-> &'a Value`, that's not possible because:
1. Selecting a row from a list and field from a record can be done with
a reference but selecting a column from a table requires creating a new
list.
2. `Value::Custom` returns new `Value`s when indexed.
So the signature becomes `(&'a self, cell_path) -> Cow<'a, Value>`.
Another complication that arises is, once a new `Value` is created, and
it is further indexed, the `current` variable
1. can't be `&'a Value`, as the lifetime requirement means it can't
refer to local variables
2. _shouldn't_ be `Cow<'a, Value>`, as once it becomes an owned value,
it can't be borrowed ever again, as `current` is derived from its
previous value in further iterations. So once it's owned, it can't be
indexed by reference, leading to more clones
We need `current` to have _two_ possible lifetimes
1. `'out`: references derived from `&self`
2. `'local`: references derived from an owned value stored in a local
variable
```rust
enum MultiLife<'out, 'local, T>
where
'out: 'local,
T: ?Sized,
{
Out(&'out T),
Local(&'local T),
}
```
With `current: MultiLife<'out, '_, Value>`, we can traverse values with
minimal clones, and we can transform it to `Cow<'out, Value>` easily
(`MultiLife::Out -> Cow::Borrowed, MultiLife::Local -> Cow::Owned`) to
return it
# User-Facing Changes
# Tests + Formatting
# After Submitting
---------
Co-authored-by: Bahex <17417311+Bahex@users.noreply.github.com>
# Description
Fixes: #15510
I think it's introduced by #14653, which changes `and/or` to `match`
expression.
After looking into `compile_match`, it's important to collect the value
before matching this.
```rust
// Important to collect it first
builder.push(Instruction::Collect { src_dst: match_reg }.into_spanned(match_expr.span))?;
```
This pr is going to apply the logic while compiling `and/or` operation.
# User-Facing Changes
The following will raise a reasonable error:
```nushell
> (nu --testbin cococo false) and true
Error: nu:🐚:operator_unsupported_type
× The 'and' operator does not work on values of type 'string'.
╭─[entry #7:1:2]
1 │ (nu --testbin cococo false) and true
· ─┬ ─┬─
· │ ╰── does not support 'string'
· ╰── string
╰────
```
# Tests + Formatting
Added 1 test.
# After Submitting
Maybe need to update doc
https://github.com/nushell/nushell.github.io/pull/1876
---------
Co-authored-by: Stefan Holderbach <sholderbach@users.noreply.github.com>
# Description
Try to fixes https://github.com/nushell/nushell/issues/15326 in another
way.
The main point of this change is to avoid duplicate `write` and `close`
a redirected file. So during compile, if compiler know current element
is a sub-expression(defined by private `is_subexpression` function), it
will no longer invoke `finish_redirection`.
In this way, we can avoid duplicate `finish_redirection`.
# User-Facing Changes
`(^echo aa) o> /tmp/aaa` will no longer raise an error.
Here is the IR after the pr:
```
# 3 registers, 12 instructions, 11 bytes of data
# 1 file used for redirection
0: load-literal %1, string("aaa")
1: open-file file(0), %1, append = false
2: load-literal %1, glob-pattern("echo", no_expand = false)
3: load-literal %2, glob-pattern("true", no_expand = false)
4: push-positional %1
5: push-positional %2
6: redirect-out file(0)
7: redirect-err caller
8: call decl 135 "run-external", %0
9: write-file file(0), %0
10: close-file file(0)
11: return %0
```
# Tests + Formatting
Added 3 tests.
# After Submitting
Maybe need to update doc
https://github.com/nushell/nushell.github.io/pull/1876
---------
Co-authored-by: Stefan Holderbach <sholderbach@users.noreply.github.com>
# Description
There are some clippy(version 0.1.86) errors on nushell repo. This pr is
trying to fix it.
# User-Facing Changes
Hopefully none.
# Tests + Formatting
NaN
# After Submitting
NaN
Closes#15395
# User-Facing Changes
Certain errors no longer leave the argument stack in an unexpected
state:
```diff
let x: any = 1; try { $x | get path } catch { print caught }
-$.path # extra `print` argument from the failed `get` call
caught
```
# Description
If `eval_call` fails in `check_input_types` or `gather_arguments`, the
cleanup code is still executed.
# Description
This is an attempt to improve the nushell situation with regard to issue
#247.
This PR implements:
- [X] spawning jobs: `job spawn { do_background_thing }`
Jobs will be implemented as threads and not forks, to maintain a
consistent behavior between unix and windows.
- [X] listing running jobs: `job list`
This should allow users to list what background tasks they currently
have running.
- [X] killing jobs: `job kill <id>`
- [X] interupting nushell code in the job's background thread
- [X] interrupting the job's currently-running process, if any.
Things that should be taken into consideration for implementation:
- [X] (unix-only) Handling `TSTP` signals while executing code and
turning the current program into a background job, and unfreezing them
in foreground `job unfreeze`.
- [X] Ensuring processes spawned by background jobs get distinct process
groups from the nushell shell itself
This PR originally aimed to implement some of the following, but it is
probably ideal to be left for another PR (scope creep)
- Disowning external process jobs (`job dispatch`)
- Inter job communication (`job send/recv`)
Roadblocks encountered so far:
- Nushell does some weird terminal sequence magics which make so that
when a background process or thread prints something to stderr and the
prompt is idle, the stderr output ends up showing up weirdly
In this PR, the two new variants for `ErrorKind`, `FileNotFound`
and `DirectoryNotFound` with a nice `not_found_as` method for the
`ErrorKind` to easily specify the `NotFound` errors. I also updated some
places where I could of think of with these new variants and the message
for `NotFound` is no longer "Entity not found" but "Not found" to be
less strange.
closes#15142closes#15055
# Description
This PR adds two new `ParseError` and `ShellError` cases for type errors
relating to operators.
- `OperatorUnsupportedType` is used when a type is not supported by an
operator in any way, shape, or form. E.g., `+` does not support `bool`.
- `OperatorIncompatibleTypes` is used when a operator is used with types
it supports, but the combination of types provided cannot be used
together. E.g., `filesize + duration` is not a valid combination.
The other preexisting error cases related to operators have been removed
and replaced with the new ones above. Namely:
- `ShellError::OperatorMismatch`
- `ShellError::UnsupportedOperator`
- `ParseError::UnsupportedOperationLHS`
- `ParseError::UnsupportedOperationRHS`
- `ParseError::UnsupportedOperationTernary`
# User-Facing Changes
- `help operators` now lists the precedence of `not` as 55 instead of 0
(above the other boolean operators). Fixes#13675.
- `math median` and `math mode` now ignore NaN values so that `[NaN NaN]
| math median` and `[NaN NaN] | math mode` no longer trigger a type
error. Instead, it's now an empty input error. Fixing this in earnest
can be left for a future PR.
- Comparisons with `nan` now return false instead of causing an error.
E.g., `1 == nan` is now `false`.
- All the operator type errors have been standardized and reworked. In
particular, they can now have a help message, which is currently used
for types errors relating to `++`.
```nu
[1] ++ 2
```
```
Error: nu::parser::operator_unsupported_type
× The '++' operator does not work on values of type 'int'.
╭─[entry #1:1:5]
1 │ [1] ++ 2
· ─┬ ┬
· │ ╰── int
· ╰── does not support 'int'
╰────
help: if you meant to append a value to a list or a record to a table, use the `append` command or wrap the value in a list. For example: `$list ++ $value` should be
`$list ++ [$value]` or `$list | append $value`.
```
# Description
Add custom command attributes.
- Attributes are placed before a command definition and start with a `@`
character.
- Attribute invocations consist of const command call. The command's
name must start with "attr ", but this prefix is not used in the
invocation.
- A command named `attr example` is invoked as an attribute as
`@example`
- Several built-in attribute commands are provided as part of this PR
- `attr example`: Attaches an example to the commands help text
```nushell
# Double numbers
@example "double an int" { 5 | double } --result 10
@example "double a float" { 0.5 | double } --result 1.0
def double []: [number -> number] {
$in * 2
}
```
- `attr search-terms`: Adds search terms to a command
- ~`attr env`: Equivalent to using `def --env`~
- ~`attr wrapped`: Equivalent to using `def --wrapped`~ shelved for
later discussion
- several testing related attributes in `std/testing`
- If an attribute has no internal/special purpose, it's stored as
command metadata that can be obtained with `scope commands`.
- This allows having attributes like `@test` which can be used by test
runners.
- Used the `@example` attribute for `std` examples.
- Updated the std tests and test runner to use `@test` attributes
- Added completions for attributes
# User-Facing Changes
Users can add examples to their own command definitions, and add other
arbitrary attributes.
# Tests + Formatting
- 🟢 toolkit fmt
- 🟢 toolkit clippy
- 🟢 toolkit test
- 🟢 toolkit test stdlib
# After Submitting
- Add documentation about the attribute syntax and built-in attributes
- `help attributes`
---------
Co-authored-by: 132ikl <132@ikl.sh>
# Description
This PR makes `match` no longer run closures as if they were blocks.
This also allows returning closures from `match` without needing to wrap
in an outer subexpression or block.
Before PR:
```nushell
match 1 { _ => {|| print hi} }
# => hi
```
After PR:
```nushell
match 1 { _ => {|| print hi} }
# => closure_1090
```
# User-Facing Changes
* `match` no longer runs closures as if they were blocks
# Tests + Formatting
N/A
# After Submitting
N/A
# Description
After this pr, nushell is able to raise errors with a backtrace, which
should make users easier to debug. To enable the feature, users need to
set env variable via `$env.NU_BACKTRACE = 1`. But yeah it might not work
perfectly, there are some corner cases which might not be handled.
I think it should close#13379 in another way.
### About the change
The implementation mostly contained with 2 parts:
1. introduce a new `ChainedError` struct as well as a new
`ShellError::ChainedError` variant. If `eval_instruction` returned an
error, it converts the error to `ShellError::ChainedError`.
`ChainedError` struct is responsable to display errors properly. It
needs to handle the following 2 cases:
- if we run a function which runs `error make` internally, it needs to
display the error itself along with caller span.
- if we run a `error make` directly, or some commands directly returns
an error, we just want nushell raise an error about `error make`.
2. Attach caller spans to `ListStream` and `ByteStream`, because they
are lazy streams, and *only* contains the span that runs it
directly(like `^false`, for example), so nushell needs to add all caller
spans to the stream.
For example: in `def a [] { ^false }; def b [] { a; 33 }; b`, when we
run `b`, which runs `a`, which runs `^false`, the `ByteStream` only
contains the span of `^false`, we need to make it contains the span of
`a`, so nushell is able to get all spans if something bad happened.
This behavior is happened after running `Instruction::Call`, if it
returns a `ByteStream` and `ListStream`, it will call `push_caller_span`
method to attach call spans.
# User-Facing Changes
It's better to demostrate how it works by examples, given the following
definition:
```nushell
> $env.NU_BACKTRACE = 1
> def a [x] { if $x == 3 { error make {msg: 'a custom error'}}}
> def a_2 [x] { if $x == 3 { ^false } else { $x } }
> def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } }
> def b [--list-stream --external] {
if $external == true {
# error with non-zero exit code, which is generated from external command.
a_2 1; a_2 3; a_2 2
} else if $list_stream == true {
# error generated by list-stream
a_3 1; a_3 3; a_3 2
} else {
# error generated by command directly
a 1; a 2; a 3
}
}
```
Run `b` directly shows the following error:
<details>
```nushell
Error: chained_error
× oops
╭─[entry #27:1:1]
1 │ b
· ┬
· ╰── error happened when running this
╰────
Error: chained_error
× oops
╭─[entry #26:10:19]
9 │ # error generated by command directly
10 │ a 1; a 2; a 3
· ┬
· ╰── error happened when running this
11 │ }
╰────
Error:
× a custom error
╭─[entry #6:1:26]
1 │ def a [x] { if $x == 3 { error make {msg: 'a custom error'}}}
· ─────┬────
· ╰── originates from here
╰────
```
</details>
Run `b --list-stream` shows the following error
<details>
```nushell
Error: chained_error
× oops
╭─[entry #28:1:1]
1 │ b --list-stream
· ┬
· ╰── error happened when running this
╰────
Error: nu:🐚:eval_block_with_input
× Eval block failed with pipeline input
╭─[entry #26:7:16]
6 │ # error generated by list-stream
7 │ a_3 1; a_3 3; a_3 2
· ─┬─
· ╰── source value
8 │ } else {
╰────
Error: nu:🐚:eval_block_with_input
× Eval block failed with pipeline input
╭─[entry #23:1:29]
1 │ def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } }
· ┬
· ╰── source value
╰────
Error:
× a custom error inside list stream
╭─[entry #23:1:44]
1 │ def a_3 [x] { if $x == 3 { [1 2 3] | each {error make {msg: 'a custom error inside list stream'} } } }
· ─────┬────
· ╰── originates from here
╰────
```
</details>
Run `b --external` shows the following error:
<details>
```nushell
Error: chained_error
× oops
╭─[entry #29:1:1]
1 │ b --external
· ┬
· ╰── error happened when running this
╰────
Error: nu:🐚:eval_block_with_input
× Eval block failed with pipeline input
╭─[entry #26:4:16]
3 │ # error with non-zero exit code, which is generated from external command.
4 │ a_2 1; a_2 3; a_2 2
· ─┬─
· ╰── source value
5 │ } else if $list_stream == true {
╰────
Error: nu:🐚:non_zero_exit_code
× External command had a non-zero exit code
╭─[entry #7:1:29]
1 │ def a_2 [x] { if $x == 3 { ^false } else { $x } }
· ──┬──
· ╰── exited with code 1
╰────
```
</details>
It also added a message to guide the usage of NU_BACKTRACE, see the last
line in the following example:
```shell
ls asdfasd
Error: nu:🐚:io::not_found
× I/O error
╰─▶ × Entity not found
╭─[entry #17:1:4]
1 │ ls asdfasd
· ───┬───
· ╰── Entity not found
╰────
help: The error occurred at '/home/windsoilder/projects/nushell/asdfasd'
set the `NU_BACKTRACE=1` environment variable to display a backtrace.
```
# Tests + Formatting
Added some tests for the behavior.
# After Submitting
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR makes two changes related to [run-time pipeline input type
checking](https://github.com/nushell/nushell/pull/14741):
1. The check which bypasses type checking for commands with only
`Type::Nothing` input types has been expanded to work with commands with
multiple `Type::Nothing` inputs for different outputs. For example,
`ast` has three input/output type pairs, but all of the inputs are
`Type::Nothing`:
```
╭───┬─────────┬────────╮
│ # │ input │ output │
├───┼─────────┼────────┤
│ 0 │ nothing │ table │
│ 1 │ nothing │ record │
│ 2 │ nothing │ string │
╰───┴─────────┴────────╯
```
Before this PR, passing a value (which would otherwise be ignored) to
`ast` caused a run-time type error:
```
Error: nu:🐚:only_supports_this_input_type
× Input type not supported.
╭─[entry #1:1:6]
1 │ echo 123 | ast -j -f "hi"
· ─┬─ ─┬─
· │ ╰── only nothing, nothing, and nothing input data is supported
· ╰── input type: int
╰────
```
After this PR, no error is raised.
This doesn't really matter for `ast` (the only other built-in command
with a similar input/output type signature is `cal`), but it's more
logically consistent.
2. Bypasses input type-checking (parse-time ***and*** run-time) for some
(not all, see below) commands which have both a `Type::Nothing` input
and some other non-nothing `Type` input. This is accomplished by adding
a `Type::Any` input with the same output as the corresponding
`Type::Nothing` input/output pair.
This is necessary because some commands are intended to operate on an
argument with empty pipeline input, or operate on an empty pipeline
input with no argument. This causes issues when a value is implicitly
passed to one of these commands. I [discovered this
issue](https://discord.com/channels/601130461678272522/615962413203718156/1329945784346611712)
when working with an example where the `open` command is used in
`sort-by` closure:
```nushell
ls | sort-by { open -r $in.name | lines | length }
```
Before this PR (but after the run-time input type checking PR), this
error is raised:
```
Error: nu:🐚:only_supports_this_input_type
× Input type not supported.
╭─[entry #1:1:1]
1 │ ls | sort-by { open -r $in.name | lines | length }
· ─┬ ──┬─
· │ ╰── only nothing and string input data is supported
· ╰── input type: record<name: string, type: string, size: filesize, modified: date>
╰────
```
While this error is technically correct, we don't actually want to
return an error here since `open` ignores its pipeline input when an
argument is passed. This would be a parse-time error as well if the
parser was able to infer that the closure input type was a record, but
our type inference isn't that robust currently, so this technically
incorrect form snuck by type checking until #14741.
However, there are some commands with the same kind of type signature
where this behavior is actually desirable. This means we can't just
bypass type-checking for any command with a `Type::Nothing` input. These
commands operate on true `null` values, rather than ignoring their
input. For example, `length` returns `0` when passed a `null` value.
It's correct, and even desirable, to throw a run-time error when
`length` is passed an unexpected type. For example, a string, which
should instead be measured with `str length`:
```nushell
["hello" "world"] | sort-by { length }
# => Error: nu:🐚:only_supports_this_input_type
# =>
# => × Input type not supported.
# => ╭─[entry #32:1:10]
# => 1 │ ["hello" "world"] | sort-by { length }
# => · ───┬─── ───┬──
# => · │ ╰── only list<any>, binary, and nothing input data is supported
# => · ╰── input type: string
# => ╰────
```
We need a more robust way for commands to express how they handle the
`Type::Nothing` input case. I think a possible solution here is to allow
commands to express that they operate on `PipelineData::Empty`, rather
than `Value::Nothing`. Then, a command like `open` could have an empty
pipeline input type rather than a `Type::Nothing`, and the parse-time
and run-time pipeline input type checks know that `open` will safely
ignore an incorrectly typed input.
That being said, we have a release coming up and the above solution
might take a while to implement, so while unfortunate, bypassing input
type-checking for these problematic commands serves as a workaround to
avoid breaking changes in the release until a more robust solution is
implemented.
This PR bypasses input type-checking for the following commands:
* `load-env`: can take record of envvars as input or argument
* `nu-check`: checks input string or filename argument
* `open`: can take filename as input or argument
* `polars when`: can be used with input, or can be chained with another
`polars when`
* `stor insert`: data record can be passed as input or argument
* `stor update`: data record can be passed as input or argument
* `format date`: `--list` ignores input value
* `into datetime`: `--list` ignores input value (also added a
`Type::Nothing` input which was missing from this command)
These commands have a similar input/output signature to the above
commands, but are working as intended:
* `cd`: The input/output signature was actually incorrect, `cd` always
ignores its input. I fixed this in this PR.
* `generate`
* `get`
* `history import`
* `interleave`
* `into bool`
* `length`
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
As a temporary workaround, pipeline input type-checking for the
following commands has been bypassed to avoid undesirable run-time input
type checking errors which were previously not caught at parse-time:
* `open`
* `load-env`
* `format date`
* `into datetime`
* `nu-check`
* `stor insert`
* `stor update`
* `polars when`
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
CI became green in the time it took me to type the description 😄
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
N/A
# Description
I'm on rust toolchain 1.8.4, and I can see clippy warnings that can't be
caught by the ci workflow, primarily related to lifetime params.
I think it doesn't hurt to fix those in advance.
# User-Facing Changes
# Tests + Formatting
# After Submitting
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR returns error values while checking pipeline input types and
positional argument types. This should help return non-nested errors
earlier and prevent confusing errors.
The positional argument change is directly related to an example given
on Discord. Before this PR, this is the error shown:
```
Error: nu:🐚:cant_convert
× Can't convert to record.
╭─[/home/rose/tmp/script.nu:23:5]
22 │ let entry = $in
23 │ ╭─▶ {
24 │ │ name: $entry,
25 │ │ details: {
26 │ │ context: $context
27 │ │ }
28 │ ├─▶ }
· ╰──── can't convert error to record
29 │ }
╰────
```
After this PR, this is the error shown:
```
Error: nu:🐚:eval_block_with_input
× Eval block failed with pipeline input
╭─[/home/rose/tmp/script.nu:23:5]
22 │ let entry = $in
23 │ ╭─▶ {
24 │ │ name: $entry,
25 │ │ details: {
26 │ │ context: $context
27 │ │ }
28 │ ├─▶ }
· ╰──── source value
29 │ }
╰────
Error: nu:🐚:type_mismatch
× Type mismatch.
╭─[/home/rose/tmp/much.nu:3:38]
2 │ $in | each { |elem|
3 │ print $elem.details.context.yaml.0
· ┬
· ╰── Can't access record values with a row index. Try specifying a column name instead
4 │ } | each { |elem|
╰────
```
I'm not certain if the pipeline input error check actually can ever be
triggered, but it seems to be a good defensive error handling strategy
regardless. My addition of the `Value::Error` case in the first place
would suggest it can be, but after looking at it more closely the error
that caused me to add the case in the first place was actually unrelated
to input typechecking.
Additionally, this PR does not affect the handling of nested errors, so
something like:
```nushell
try { ... } catch {|e| $e | reject raw | to nuon }
```
works the same before and after this PR.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
Errors values detected as arguments to commands or as pipeline input to
commands are immediately thrown, rather than passed to the command.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
N/A
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
It seems in my PR #14927 I missed a few calls to `IoError::new` that had
`Span::unknown` inside them, which shouldn't be used but rather
`IoError::new_internal`. I replaced these calls.
Thanks to @132ikl to finding out that I forgot some. 😄
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
Pretty much none really.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
