<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR adds type checking of all command input types at run-time.
Generally, these errors should be caught by the parser, but sometimes we
can't know the type of a value at parse-time. The simplest example is
using the `echo` command, which has an output type of `any`, so
prefixing a literal with `echo` will bypass parse-time type checking.
Before this PR, each command has to individually check its input types.
This can result in scenarios where the input/output types don't match
the actual command behavior. This can cause valid usage with an
non-`any` type to become a parse-time error if a command is missing that
type in its pipeline input/output (`drop nth` and `history import` do
this before this PR). Alternatively, a command may not list a type in
its input/output types, but doesn't actually reject that type in its
code, which can have unintended side effects (`get` does this on an
empty pipeline input, and `sort` used to before #13154).
After this PR, the type of the pipeline input is checked to ensure it
matches one of the input types listed in the proceeding command's
input/output types. While each of the issues in the "before this PR"
section could be addressed with each command individually, this PR
solves this issue for _all_ commands.
**This will likely cause some breakage**, as some commands have
incorrect input/output types, and should be adjusted. Also, some scripts
may have erroneous usage of commands. In writing this PR, I discovered
that `toolkit.nu` was passing `null` values to `str join`, which doesn't
accept nothing types (if folks think it should, we can adjust it in this
PR or in a different PR). I found some issues in the standard library
and its tests. I also found that carapace's vendor script had an
incorrect chaining of `get -i`:
```nushell
let expanded_alias = (scope aliases | where name == $spans.0 | get -i 0 | get -i expansion)
```
Before this PR, if the `get -i 0` ever actually did evaluate to `null`,
the second `get` invocation would error since `get` doesn't operate on
`null` values. After this PR, this is immediately a run-time error,
alerting the user to the problematic code. As a side note, we'll need to
PR this fix (`get -i 0 | get -i expansion` -> `get -i 0.expansion`) to
carapace.
A notable exception to the type checking is commands with input type of
`nothing -> <type>`. In this case, any input type is allowed. This
allows piping values into the command without an error being thrown. For
example, `123 | echo $in` would be an error without this exception.
Additionally, custom types bypass type checking (I believe this also
happens during parsing, but not certain)
I added a `is_subtype` method to `Value` and `PipelineData`. It
functions slightly differently than `get_type().is_subtype()`, as noted
in the doccomments. Notably, it respects structural typing of lists and
tables. For example, the type of a value `[{a: 123} {a: 456, b: 789}]`
is a subtype of `table<a: int>`, whereas the type returned by
`Value::get_type` is a `list<any>`. Similarly, `PipelineData` has some
special handling for `ListStream`s and `ByteStream`s. The latter was
needed for this PR to work properly with external commands.
Here's some examples.
Before:
```nu
1..2 | drop nth 1
Error: nu::parser::input_type_mismatch
× Command does not support range input.
╭─[entry #9:1:8]
1 │ 1..2 | drop nth 1
· ────┬───
· ╰── command doesn't support range input
╰────
echo 1..2 | drop nth 1
# => ╭───┬───╮
# => │ 0 │ 1 │
# => ╰───┴───╯
```
After this PR, I've adjusted `drop nth`'s input/output types to accept
range input.
Before this PR, zip accepted any value despite not being listed in its
input/output types. This caused different behavior depending on if you
triggered a parse error or not:
```nushell
1 | zip [2]
# => Error: nu::parser::input_type_mismatch
# =>
# => × Command does not support int input.
# => ╭─[entry #3:1:5]
# => 1 │ 1 | zip [2]
# => · ─┬─
# => · ╰── command doesn't support int input
# => ╰────
echo 1 | zip [2]
# => ╭───┬───────────╮
# => │ 0 │ ╭───┬───╮ │
# => │ │ │ 0 │ 1 │ │
# => │ │ │ 1 │ 2 │ │
# => │ │ ╰───┴───╯ │
# => ╰───┴───────────╯
```
After this PR, it works the same in both cases. For cases like this, if
we do decide we want `zip` or other commands to accept any input value,
then we should explicitly add that to the input types.
```nushell
1 | zip [2]
# => Error: nu::parser::input_type_mismatch
# =>
# => × Command does not support int input.
# => ╭─[entry #3:1:5]
# => 1 │ 1 | zip [2]
# => · ─┬─
# => · ╰── command doesn't support int input
# => ╰────
echo 1 | zip [2]
# => Error: nu:🐚:only_supports_this_input_type
# =>
# => × Input type not supported.
# => ╭─[entry #14:2:6]
# => 2 │ echo 1 | zip [2]
# => · ┬ ─┬─
# => · │ ╰── only list<any> and range input data is supported
# => · ╰── input type: int
# => ╰────
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
**Breaking change**: The type of a command's input is now checked
against the input/output types of that command at run-time. While these
errors should mostly be caught at parse-time, in cases where they can't
be detected at parse-time they will be caught at run-time instead. This
applies to both internal commands and custom commands.
Example function and corresponding parse-time error (same before and
after PR):
```nushell
def foo []: int -> nothing {
print $"my cool int is ($in)"
}
1 | foo
# => my cool int is 1
"evil string" | foo
# => Error: nu::parser::input_type_mismatch
# =>
# => × Command does not support string input.
# => ╭─[entry #16:1:17]
# => 1 │ "evil string" | foo
# => · ─┬─
# => · ╰── command doesn't support string input
# => ╰────
# =>
```
Before:
```nu
echo "evil string" | foo
# => my cool int is evil string
```
After:
```nu
echo "evil string" | foo
# => Error: nu:🐚:only_supports_this_input_type
# =>
# => × Input type not supported.
# => ╭─[entry #17:1:6]
# => 1 │ echo "evil string" | foo
# => · ──────┬────── ─┬─
# => · │ ╰── only int input data is supported
# => · ╰── input type: string
# => ╰────
```
Known affected internal commands which erroneously accepted any type:
* `str join`
* `zip`
* `reduce`
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
* Play whack-a-mole with the commands and scripts this will inevitably
break
# Description
This PR updates `group-by` and `split-by` to allow other nushell Values
to be used, namely bools.
### Before
```nushell
❯ [false, false, true, false, true, false] | group-by | table -e
Error: nu:🐚:cant_convert
× Can't convert to string.
╭─[entry #1:1:2]
1 │ [false, false, true, false, true, false] | group-by | table -e
· ──┬──
· ╰── can't convert bool to string
╰────
```
### After
```nushell
❯ [false, false, true, false, true, false] | group-by | table -e
╭───────┬───────────────╮
│ │ ╭───┬───────╮ │
│ false │ │ 0 │ false │ │
│ │ │ 1 │ false │ │
│ │ │ 2 │ false │ │
│ │ │ 3 │ false │ │
│ │ ╰───┴───────╯ │
│ │ ╭───┬──────╮ │
│ true │ │ 0 │ true │ │
│ │ │ 1 │ true │ │
│ │ ╰───┴──────╯ │
╰───────┴───────────────╯
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
This PR updates `group-by` and `split-by` to allow other nushell Values
to be used, namely bools.
### Before
```nushell
❯ [false, false, true, false, true, false] | group-by | table -e
Error: nu:🐚:cant_convert
× Can't convert to string.
╭─[entry #1:1:2]
1 │ [false, false, true, false, true, false] | group-by | table -e
· ──┬──
· ╰── can't convert bool to string
╰────
```
### After
```nushell
❯ [false, false, true, false, true, false] | group-by | table -e
╭───────┬───────────────╮
│ │ ╭───┬───────╮ │
│ false │ │ 0 │ false │ │
│ │ │ 1 │ false │ │
│ │ │ 2 │ false │ │
│ │ │ 3 │ false │ │
│ │ ╰───┴───────╯ │
│ │ ╭───┬──────╮ │
│ true │ │ 0 │ true │ │
│ │ │ 1 │ true │ │
│ │ ╰───┴──────╯ │
╰───────┴───────────────╯
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
When a closure if provided to `group-by`, errors that occur in the
closure are currently ignored. That is, `group-by` will fall back and
use the `"error"` key if an error occurs. For example, the code snippet
below will group all `ls` entries under the `"error"` column.
```nushell
ls | group-by { get nope }
```
This PR changes `group-by` to instead bubble up any errors triggered
inside the closure. In addition, this PR also does some refactoring and
cleanup inside `group-by`.
# User-Facing Changes
Errors are now returned from the closure provided to `group-by` instead
of falling back to the `"error"` group/key.
# Description
This PR implements modifications to command tests that write unnecessary
json and csv to disk then load it with open, by using nuon literals
instead.
- Fixes#7189
# User-Facing Changes
None
# Tests + Formatting
This only affects existing tests, which still pass.
# Description
Changed `group-by` behavior to accept empty list as input and return an
empty record instead of throwing an error. I also replaced
`errors_if_input_empty()` test to reflect the new expected behavior.
See #10713
# User-Facing Changes
`[] | group-by` or `[] | group-by a` now returns empty record
# Tests + Formatting
1 test for emptied table i.e. list
---------
Signed-off-by: Oscar <71343264+0scvr@users.noreply.github.com>
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR is related to **Tests: clean up unnecessary use of cwd,
pipeline(), etc.
[#8670](https://github.com/nushell/nushell/issues/8670)**
- Removed the `r#"..."#` raw string literal syntax, which is unnecessary
when there are no special characters that need quoting from the tests
that use the `nu!` macro.
- `cwd:` and `pipeline()` has not changed
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
Closes#9003.
This PR changes `group-by` so that its optional argument is interpreted
as a cell path. In turn, this lets users use `?` to ignore rows that are
missing the column they wish to group on. For example:
```
> [{foo: 123}, {foo: 234}, {bar: 345}] | group-by foo
Error: nu:🐚:column_not_found
× Cannot find column
╭─[entry #3:1:1]
1 │ [{foo: 123}, {foo: 234}, {bar: 345}] | group-by foo
· ─────┬──── ─┬─
· │ ╰── cannot find column 'foo'
· ╰── value originates here
╰────
> [{foo: 123}, {foo: 234}, {bar: 345}] | group-by foo?
╭─────┬───────────────╮
│ 123 │ [table 1 row] │
│ 234 │ [table 1 row] │
╰─────┴───────────────╯
```
~~This removes the ability to pass `group-by` a closure or block (I
wasn't able to figure out how to make the 2 features coexist), and so it
is a breaking change. I think this is OK; I didn't even know `group-by`
could accept a closure or block because there was no example for that
functionality.~~
# Description
As title, closes: #7921closes: #8273
# User-Facing Changes
when define a closure without pipe, nushell will raise error for now:
```
❯ let x = {ss ss}
Error: nu::parser::closure_missing_pipe
× Missing || inside closure
╭─[entry #2:1:1]
1 │ let x = {ss ss}
· ───┬───
· ╰── Parsing as a closure, but || is missing
╰────
help: Try add || to the beginning of closure
```
`any`, `each`, `all`, `where` command accepts closure, it forces user
input closure like `{||`, or parse error will returned.
```
❯ {major:2, minor:1, patch:4} | values | each { into string }
Error: nu::parser::closure_missing_pipe
× Missing || inside closure
╭─[entry #4:1:1]
1 │ {major:2, minor:1, patch:4} | values | each { into string }
· ───────┬───────
· ╰── Parsing as a closure, but || is missing
╰────
help: Try add || to the beginning of closure
```
`with-env`, `do`, `def`, `try` are special, they still remain the same,
although it says that it accepts a closure, but they don't need to be
written like `{||`, it's more likely a block but can capture variable
outside of scope:
```
❯ def test [input] { echo [0 1 2] | do { do { echo $input } } }; test aaa
aaa
```
Just realize that It's a big breaking change, we need to update config
and scripts...
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
# Description
This fixes up some clippy warnings and removes some old names/info from
our unit tests
# User-Facing Changes
Internal changes only
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
* Fix "index out of bounds" when input to group-by is empty #4369
* Fix formatting #4369
* Adds test for empty input
Co-authored-by: Ray Henry <ray.henry@thermofisher.com>
* update docs to refer to length instead of count
* rename count to length
* change all occurrences of 'count' to 'length' in tests
* format length command
* move commands, futures.rs, script.rs, utils
* move over maybe_print_errors
* add nu_command crate references to nu_cli
* in commands.rs open up to pub mod from pub(crate)
* nu-cli, nu-command, and nu tests are now passing
* cargo fmt
* clean up nu-cli/src/prelude.rs
* code cleanup
* for some reason lex.rs was not formatted, may be causing my error
* remove mod completion from lib.rs which was not being used along with quickcheck macros
* add in allow unused imports
* comment out one failing external test; comment out one failing internal test
* revert commenting out failing tests; something else might be going on; someone with a windows machine should check and see what is going on with these failing windows tests
* Update Cargo.toml
Extend the optional features to nu-command
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
