# Description
This pull request does a lot of the heavy lifting needed to supported
more complex dtypes like categorical dtypes. It introduces a new
CustomValue, NuDataType and makes NuSchema a full CustomValue. Further
more it introduces a new command `polars into-dtype` that allows a dtype
to be created. This can then be passed into schemas when they are
created.
```nu
> ❯ : let dt = ("str" | polars to-dtype)
> ❯ : [[a b]; ["one" "two"]] | polars into-df -s {a: $dt, b: str} | polars schema
╭───┬─────╮
│ a │ str │
│ b │ str │
╰───┴─────╯
```
# User-Facing Changes
- Introduces new command `polars into-dtype`, allows dtype variables to
be passed in during schema creation.
Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.8.0 to
2.9.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/indexmap-rs/indexmap/blob/main/RELEASES.md">indexmap's
changelog</a>.</em></p>
<blockquote>
<h2>2.9.0 (2025-04-04)</h2>
<ul>
<li>Added a <code>get_disjoint_mut</code> method to
<code>IndexMap</code>, matching Rust 1.86's
<code>HashMap</code> method.</li>
<li>Added a <code>get_disjoint_indices_mut</code> method to
<code>IndexMap</code> and <code>map::Slice</code>,
matching Rust 1.86's <code>get_disjoint_mut</code> method on
slices.</li>
<li>Deprecated the <code>borsh</code> feature in favor of their own
<code>indexmap</code> feature,
solving a cyclic dependency that occured via
<code>borsh-derive</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="1818d4140d"><code>1818d41</code></a>
Merge pull request <a
href="https://redirect.github.com/indexmap-rs/indexmap/issues/387">#387</a>
from cuviper/release-2.9.0</li>
<li><a
href="9f4998341b"><code>9f49983</code></a>
Release 2.9.0</li>
<li><a
href="582a90fda3"><code>582a90f</code></a>
Merge pull request <a
href="https://redirect.github.com/indexmap-rs/indexmap/issues/386">#386</a>
from cuviper/de-borsh</li>
<li><a
href="90117397b6"><code>9011739</code></a>
Deprecate the "borsh" feature</li>
<li><a
href="0a836e8648"><code>0a836e8</code></a>
Merge pull request <a
href="https://redirect.github.com/indexmap-rs/indexmap/issues/238">#238</a>
from NiklasJonsson/get_many_mut</li>
<li><a
href="434d7ac6d1"><code>434d7ac</code></a>
Avoid let-else for MSRV's sake</li>
<li><a
href="5be552d557"><code>5be552d</code></a>
Implement additional suggestions from review</li>
<li><a
href="4e1d8cef47"><code>4e1d8ce</code></a>
Address review feedback</li>
<li><a
href="5aec9ec674"><code>5aec9ec</code></a>
Implement get_disjoint_mut for arrays of keys</li>
<li><a
href="d10de30e74"><code>d10de30</code></a>
Merge pull request <a
href="https://redirect.github.com/indexmap-rs/indexmap/issues/385">#385</a>
from iajoiner/docs/macros</li>
<li>Additional commits viewable in <a
href="https://github.com/indexmap-rs/indexmap/compare/2.8.0...2.9.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# Description
The current implementation of `polars into-df` and `polars into-lazy`
will throw an error if `--schema` is provided but not all columns are
defined. This PR seeks to remove this requirement so that when a partial
`--schema` is provided, the types on the defined columns are overridden
while the remaining columns take on their default types.
**Current Implementation**
```
$ [[a b]; [1 "foo"] [2 "bar"]] | polars into-df -s {a: str} | polars schema
Error: × Schema does not contain column: b
╭─[entry #88:1:12]
1 │ [[a b]; [1 "foo"] [2 "bar"]] | polars into-df -s {a: str} | polars schema
· ─────
╰────
```
**New Implementation (no error thrown on partial schema definition)**
Column b is not defined in `--schema`
```
$ [[a b]; [1 "foo"] [2 "bar"]] | polars into-df --schema {a: str} | polars schema
╭───┬─────╮
│ a │ str │
│ b │ str │
╰───┴─────╯
```
# User-Facing Changes
Soft breaking change: The user's previous (erroneous) code that would
have thrown an error would no longer throw an error. The user's previous
working code will still work.
# Tests + Formatting
# After Submitting
# Description
There are some clippy(version 0.1.86) errors on nushell repo. This pr is
trying to fix it.
# User-Facing Changes
Hopefully none.
# Tests + Formatting
NaN
# After Submitting
NaN
# Description
This PR seeks to fix an error in `polars as-datetime` where timezone
information is entirely ignored. This behavior raises a host of silent
errors when dealing with datetime conversions (see example below).
## Current Implementation
Timezones are entirely ignored and datetimes with different timezones
are converted to the same naive datetimes even when the user
specifically indicates that the timezone should be parsed. For example,
"2021-12-30 00:00:00 +0000" and "2021-12-30 00:00:00 -0400" will both be
parsed to "2021-12-30 00:00:00" even when the format string specifically
includes "%z".
```
$ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z"
╭───┬───────────────────────╮
│ # │ datetime │
├───┼───────────────────────┤
│ 0 │ 12/30/2021 12:00:00AM │
│ 1 │ 12/30/2021 12:00:00AM │ <-- Same datetime even though the first is +0000 and second is -0400
╰───┴───────────────────────╯
$ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" | polars schema
╭──────────┬──────────────╮
│ datetime │ datetime<ns> │
╰──────────┴──────────────╯
```
## New Implementation
Datetimes are converted to UTC and timezone information is retained.
```
$ "2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z"
╭───┬───────────────────────╮
│ # │ datetime │
├───┼───────────────────────┤
│ 0 │ 12/30/2021 12:00:00AM │
│ 1 │ 12/30/2021 04:00:00AM │ <-- Converted to UTC
╰───┴───────────────────────╯
$ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" | polars schema
╭──────────┬───────────────────╮
│ datetime │ datetime<ns, UTC> │
╰──────────┴───────────────────╯
```
The user may intentionally ignore timezone information by setting the
`--naive` flag.
```
$ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" --naive
╭───┬───────────────────────╮
│ # │ datetime │
├───┼───────────────────────┤
│ 0 │ 12/30/2021 12:00:00AM │
│ 1 │ 12/30/2021 12:00:00AM │ <-- the -0400 offset is ignored when --naive is set
╰───┴───────────────────────╯
$ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" --naive | polars schema
╭──────────┬──────────────╮
│ datetime │ datetime<ns> │
╰──────────┴──────────────╯
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
`polars as-datetime` will now account for timezone information and
return type `datetime<ns,UTC>` rather than `datetime<ns>` by default.
The user can replicate the previous behavior by setting `--naive`.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
Tests that incorporated `polars as-datetime` had to be tweaked to
include `--naive` flag to replicate previous behavior.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
This PR expands the `dtype` parameter of the `polars cast` command to
include `decimal<precision, scale>` type. Setting precision to "*" will
compel inferring the value. Note, however, setting scale to a
non-integer value will throw an explicit error (the underlying polars
crate assigns scale = 0 in such a case, but I opted for throwing an
error instead). .
```
$ [[a b]; [1 2] [3 4]] | polars into-df | polars cast decimal<4,2> a | polars schema
╭───┬──────────────╮
│ a │ decimal<4,2> │
│ b │ i64 │
╰───┴──────────────╯
$ [[a b]; [10.5 2] [3.1 4]] | polars into-df | polars cast decimal<*,2> a | polars schema
╭───┬──────────────╮
│ a │ decimal<*,2> │
│ b │ i64 │
╰───┴──────────────╯
$ [[a b]; [10.05 2] [3.1 4]] | polars into-df | polars cast decimal<5,*> a | polars schema
rror: × Invalid polars data type
╭─[entry #25:1:47]
1 │ [[a b]; [10.05 2] [3.1 4]] | polars into-df | polars cast decimal<5,*> a | polars schema
· ─────┬─────
· ╰── `*` is not a permitted value for scale
╰────
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
There are no breaking changes. The user has the additional option to
`polars cast` to a decimal type
# Tests + Formatting
Tests have been added to
`nu_plugin_polars/src/dataframe/values/nu_schema.rs`
# Description
This PR removes the mimalloc allocator due to run-away memory leaks
recently found.
closes#15311
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# fixes https://github.com/nushell/nushell/issues/15281
# Description
Provides the ability read dataframes with Categorical and Enum data
The ability to write Categorical and Enum data will provided in a future
PR
This PR (based on #15249 and #15248 because it mentions them) adds extra
documentation to the main polars command outlining the main datatypes
that are used by the plugin. The lack of a description of the types
involved in `polars xxx` commands was quite confusing to me when I
started using the plugin and this is a first try improving it.
I didn't find a better place but please let me know what you think.
# Description
This PR adds `polars str-strip-chars-end`
# User-Facing Changes
New function that can be used as follows:
```
~/Projects/nushell> [[text]; [hello!!!] [world!!!]] | polars into-df | polars select (polars col text | polars str-strip-chars-end "!") | polars collect
╭───┬───────╮
│ # │ text │
├───┼───────┤
│ 0 │ hello │
│ 1 │ world │
╰───┴───────╯
```
# Tests + Formatting
tests ran locally.
I ran the formatter.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
This PR adds two new `ParseError` and `ShellError` cases for type errors
relating to operators.
- `OperatorUnsupportedType` is used when a type is not supported by an
operator in any way, shape, or form. E.g., `+` does not support `bool`.
- `OperatorIncompatibleTypes` is used when a operator is used with types
it supports, but the combination of types provided cannot be used
together. E.g., `filesize + duration` is not a valid combination.
The other preexisting error cases related to operators have been removed
and replaced with the new ones above. Namely:
- `ShellError::OperatorMismatch`
- `ShellError::UnsupportedOperator`
- `ParseError::UnsupportedOperationLHS`
- `ParseError::UnsupportedOperationRHS`
- `ParseError::UnsupportedOperationTernary`
# User-Facing Changes
- `help operators` now lists the precedence of `not` as 55 instead of 0
(above the other boolean operators). Fixes#13675.
- `math median` and `math mode` now ignore NaN values so that `[NaN NaN]
| math median` and `[NaN NaN] | math mode` no longer trigger a type
error. Instead, it's now an empty input error. Fixing this in earnest
can be left for a future PR.
- Comparisons with `nan` now return false instead of causing an error.
E.g., `1 == nan` is now `false`.
- All the operator type errors have been standardized and reworked. In
particular, they can now have a help message, which is currently used
for types errors relating to `++`.
```nu
[1] ++ 2
```
```
Error: nu::parser::operator_unsupported_type
× The '++' operator does not work on values of type 'int'.
╭─[entry #1:1:5]
1 │ [1] ++ 2
· ─┬ ┬
· │ ╰── int
· ╰── does not support 'int'
╰────
help: if you meant to append a value to a list or a record to a table, use the `append` command or wrap the value in a list. For example: `$list ++ $value` should be
`$list ++ [$value]` or `$list | append $value`.
```
# Description
Parquet, CSV, NDJSON, and Arrow files can be written to AWS S3 via
`polars save`. This mirrors the s3 functionality provided by `polars
open`.
```nushell
ls | polars into-df | polars save s3://my-bucket/test.parquet
```
# User-Facing Changes
- S3 urls are now supported by `polars save`
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR makes two changes related to [run-time pipeline input type
checking](https://github.com/nushell/nushell/pull/14741):
1. The check which bypasses type checking for commands with only
`Type::Nothing` input types has been expanded to work with commands with
multiple `Type::Nothing` inputs for different outputs. For example,
`ast` has three input/output type pairs, but all of the inputs are
`Type::Nothing`:
```
╭───┬─────────┬────────╮
│ # │ input │ output │
├───┼─────────┼────────┤
│ 0 │ nothing │ table │
│ 1 │ nothing │ record │
│ 2 │ nothing │ string │
╰───┴─────────┴────────╯
```
Before this PR, passing a value (which would otherwise be ignored) to
`ast` caused a run-time type error:
```
Error: nu:🐚:only_supports_this_input_type
× Input type not supported.
╭─[entry #1:1:6]
1 │ echo 123 | ast -j -f "hi"
· ─┬─ ─┬─
· │ ╰── only nothing, nothing, and nothing input data is supported
· ╰── input type: int
╰────
```
After this PR, no error is raised.
This doesn't really matter for `ast` (the only other built-in command
with a similar input/output type signature is `cal`), but it's more
logically consistent.
2. Bypasses input type-checking (parse-time ***and*** run-time) for some
(not all, see below) commands which have both a `Type::Nothing` input
and some other non-nothing `Type` input. This is accomplished by adding
a `Type::Any` input with the same output as the corresponding
`Type::Nothing` input/output pair.
This is necessary because some commands are intended to operate on an
argument with empty pipeline input, or operate on an empty pipeline
input with no argument. This causes issues when a value is implicitly
passed to one of these commands. I [discovered this
issue](https://discord.com/channels/601130461678272522/615962413203718156/1329945784346611712)
when working with an example where the `open` command is used in
`sort-by` closure:
```nushell
ls | sort-by { open -r $in.name | lines | length }
```
Before this PR (but after the run-time input type checking PR), this
error is raised:
```
Error: nu:🐚:only_supports_this_input_type
× Input type not supported.
╭─[entry #1:1:1]
1 │ ls | sort-by { open -r $in.name | lines | length }
· ─┬ ──┬─
· │ ╰── only nothing and string input data is supported
· ╰── input type: record<name: string, type: string, size: filesize, modified: date>
╰────
```
While this error is technically correct, we don't actually want to
return an error here since `open` ignores its pipeline input when an
argument is passed. This would be a parse-time error as well if the
parser was able to infer that the closure input type was a record, but
our type inference isn't that robust currently, so this technically
incorrect form snuck by type checking until #14741.
However, there are some commands with the same kind of type signature
where this behavior is actually desirable. This means we can't just
bypass type-checking for any command with a `Type::Nothing` input. These
commands operate on true `null` values, rather than ignoring their
input. For example, `length` returns `0` when passed a `null` value.
It's correct, and even desirable, to throw a run-time error when
`length` is passed an unexpected type. For example, a string, which
should instead be measured with `str length`:
```nushell
["hello" "world"] | sort-by { length }
# => Error: nu:🐚:only_supports_this_input_type
# =>
# => × Input type not supported.
# => ╭─[entry #32:1:10]
# => 1 │ ["hello" "world"] | sort-by { length }
# => · ───┬─── ───┬──
# => · │ ╰── only list<any>, binary, and nothing input data is supported
# => · ╰── input type: string
# => ╰────
```
We need a more robust way for commands to express how they handle the
`Type::Nothing` input case. I think a possible solution here is to allow
commands to express that they operate on `PipelineData::Empty`, rather
than `Value::Nothing`. Then, a command like `open` could have an empty
pipeline input type rather than a `Type::Nothing`, and the parse-time
and run-time pipeline input type checks know that `open` will safely
ignore an incorrectly typed input.
That being said, we have a release coming up and the above solution
might take a while to implement, so while unfortunate, bypassing input
type-checking for these problematic commands serves as a workaround to
avoid breaking changes in the release until a more robust solution is
implemented.
This PR bypasses input type-checking for the following commands:
* `load-env`: can take record of envvars as input or argument
* `nu-check`: checks input string or filename argument
* `open`: can take filename as input or argument
* `polars when`: can be used with input, or can be chained with another
`polars when`
* `stor insert`: data record can be passed as input or argument
* `stor update`: data record can be passed as input or argument
* `format date`: `--list` ignores input value
* `into datetime`: `--list` ignores input value (also added a
`Type::Nothing` input which was missing from this command)
These commands have a similar input/output signature to the above
commands, but are working as intended:
* `cd`: The input/output signature was actually incorrect, `cd` always
ignores its input. I fixed this in this PR.
* `generate`
* `get`
* `history import`
* `interleave`
* `into bool`
* `length`
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
As a temporary workaround, pipeline input type-checking for the
following commands has been bypassed to avoid undesirable run-time input
type checking errors which were previously not caught at parse-time:
* `open`
* `load-env`
* `format date`
* `into datetime`
* `nu-check`
* `stor insert`
* `stor update`
* `polars when`
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
CI became green in the time it took me to type the description 😄
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
N/A
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
As mentioned in #10698, we have too many `ShellError` variants, with
some even overlapping in meaning. This PR simplifies and improves I/O
error handling by restructuring `ShellError` related to I/O issues.
Previously, `ShellError::IOError` only contained a message string,
making it convenient but overly generic. It was widely used without
providing spans (#4323).
This PR introduces a new `ShellError::Io` variant that consolidates
multiple I/O-related errors (except for `ShellError::NetworkFailure`,
which remains distinct for now). The new `ShellError::Io` variant
replaces the following:
- `FileNotFound`
- `FileNotFoundCustom`
- `IOInterrupted`
- `IOError`
- `IOErrorSpanned`
- `NotADirectory`
- `DirectoryNotFound`
- `MoveNotPossible`
- `CreateNotPossible`
- `ChangeAccessTimeNotPossible`
- `ChangeModifiedTimeNotPossible`
- `RemoveNotPossible`
- `ReadingFile`
## The `IoError`
`IoError` includes the following fields:
1. **`kind`**: Extends `std::io::ErrorKind` to specify the type of I/O
error without needing new `ShellError` variants. This aligns with the
approach used in `std::io::Error`. This adds a second dimension to error
reporting by combining the `kind` field with `ShellError` variants,
making it easier to describe errors in more detail. As proposed by
@kubouch in [#design-discussion on
Discord](https://discord.com/channels/601130461678272522/615329862395101194/1323699197165178930),
this helps reduce the number of `ShellError` variants. In the error
report, the `kind` field is displayed as the "source" of the error,
e.g., "I/O error," followed by the specific kind of I/O error.
2. **`span`**: A non-optional field to encourage providing spans for
better error reporting (#4323).
3. **`path`**: Optional `PathBuf` to give context about the file or
directory involved in the error (#7695). If provided, it’s shown as a
help entry in error reports.
4. **`additional_context`**: Allows adding custom messages when the
span, kind, and path are insufficient. This is rendered in the error
report at the labeled span.
5. **`location`**: Sometimes, I/O errors occur in the engine itself and
are not caused directly by user input. In such cases, if we don’t have a
span and must set it to `Span::unknown()`, we need another way to
reference the error. For this, the `location` field uses the new
`Location` struct, which records the Rust file and line number where the
error occurred. This ensures that we at least know the Rust code
location that failed, helping with debugging. To make this work, a new
`location!` macro was added, which retrieves `file!`, `line!`, and
`column!` values accurately. If `Location::new` is used directly, it
issues a warning to remind developers to use the macro instead, ensuring
consistent and correct usage.
### Constructor Behavior
`IoError` provides five constructor methods:
- `new` and `new_with_additional_context`: Used for errors caused by
user input and require a valid (non-unknown) span to ensure precise
error reporting.
- `new_internal` and `new_internal_with_path`: Used for internal errors
where a span is not available. These methods require additional context
and the `Location` struct to pinpoint the source of the error in the
engine code.
- `factory`: Returns a closure that maps an `std::io::Error` to an
`IoError`. This is useful for handling multiple I/O errors that share
the same span and path, streamlining error handling in such cases.
## New Report Look
This is simulation how the I/O errors look like (the `open crates` is
simulated to show how internal errors are referenced now):
## `Span::test_data()`
To enable better testing, `Span::test_data()` now returns a value
distinct from `Span::unknown()`. Both `Span::test_data()` and
`Span::unknown()` refer to invalid source code, but having a separate
value for test data helps identify issues during testing while keeping
spans unique.
## Cursed Sneaky Error Transfers
I removed the conversions between `std::io::Error` and `ShellError` as
they often removed important information and were used too broadly to
handle I/O errors. This also removed the problematic implementation
found here:
7ea4895513/crates/nu-protocol/src/errors/shell_error.rs (L1534-L1583)
which hid some downcasting from I/O errors and made it hard to trace
where `ShellError` was converted into `std::io::Error`. To address this,
I introduced a new struct called `ShellErrorBridge`, which explicitly
defines this transfer behavior. With `ShellErrorBridge`, we can now
easily grep the codebase to locate and manage such conversions.
## Miscellaneous
- Removed the OS error added in #14640, as it’s no longer needed.
- Improved error messages in `glob_from` (#14679).
- Trying to open a directory with `open` caused a permissions denied
error (it's just what the OS provides). I added a `is_dir` check to
provide a better error in that case.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
- Error outputs now include more detailed information and are formatted
differently, including updated error codes.
- The structure of `ShellError` has changed, requiring plugin authors
and embedders to update their implementations.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
I updated tests to account for the new I/O error structure and
formatting changes.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
This PR closes#7695 and closes#14892 and partially addresses #4323 and
#10698.
---------
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR adds a new command that outputs a NuDataFrame or NuLazyFrame in
its repr format, which can then be ingested in another polars instance.
Advantages of serializing a dataframe in this format are that it can be
viewed as a table, carries type information, and can easily be copied to
the clipboard.
```nushell
# In Nushell
> [[a b]; [2025-01-01 2] [2025-01-02 4]] | polars into-df | polars into-lazy | polars into-repr
shape: (2, 2)
┌─────────────────────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ datetime[ns] ┆ i64 │
╞═════════════════════╪═════╡
│ 2025-01-01 00:00:00 ┆ 2 │
│ 2025-01-02 00:00:00 ┆ 4 │
└─────────────────────┴─────┘
```
```python
# In python
>>> import polars as pl
>>> df = pl.from_repr("""
... shape: (2, 2)
... ┌─────────────────────┬─────┐
... │ a ┆ b │
... │ --- ┆ --- │
... │ datetime[ns] ┆ i64 │
... ╞═════════════════════╪═════╡
... │ 2025-01-01 00:00:00 ┆ 2 │
... │ 2025-01-02 00:00:00 ┆ 4 │
... └─────────────────────┴─────┘""")
shape: (2, 2)
┌─────────────────────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ datetime[ns] ┆ i64 │
╞═════════════════════╪═════╡
│ 2025-01-01 00:00:00 ┆ 2 │
│ 2025-01-02 00:00:00 ┆ 4 │
└─────────────────────┴─────┘
>>> df.select(pl.col("a").dt.offset_by("12m"))
shape: (2, 1)
┌─────────────────────┐
│ a │
│ --- │
│ datetime[ns] │
╞═════════════════════╡
│ 2025-01-01 00:12:00 │
│ 2025-01-02 00:12:00 │
└─────────────────────┘
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
A new command `polars into-repr` is added. No other commands are
impacted by the changes in this PR.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
Examples were added in the command definition.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR adds type checking of all command input types at run-time.
Generally, these errors should be caught by the parser, but sometimes we
can't know the type of a value at parse-time. The simplest example is
using the `echo` command, which has an output type of `any`, so
prefixing a literal with `echo` will bypass parse-time type checking.
Before this PR, each command has to individually check its input types.
This can result in scenarios where the input/output types don't match
the actual command behavior. This can cause valid usage with an
non-`any` type to become a parse-time error if a command is missing that
type in its pipeline input/output (`drop nth` and `history import` do
this before this PR). Alternatively, a command may not list a type in
its input/output types, but doesn't actually reject that type in its
code, which can have unintended side effects (`get` does this on an
empty pipeline input, and `sort` used to before #13154).
After this PR, the type of the pipeline input is checked to ensure it
matches one of the input types listed in the proceeding command's
input/output types. While each of the issues in the "before this PR"
section could be addressed with each command individually, this PR
solves this issue for _all_ commands.
**This will likely cause some breakage**, as some commands have
incorrect input/output types, and should be adjusted. Also, some scripts
may have erroneous usage of commands. In writing this PR, I discovered
that `toolkit.nu` was passing `null` values to `str join`, which doesn't
accept nothing types (if folks think it should, we can adjust it in this
PR or in a different PR). I found some issues in the standard library
and its tests. I also found that carapace's vendor script had an
incorrect chaining of `get -i`:
```nushell
let expanded_alias = (scope aliases | where name == $spans.0 | get -i 0 | get -i expansion)
```
Before this PR, if the `get -i 0` ever actually did evaluate to `null`,
the second `get` invocation would error since `get` doesn't operate on
`null` values. After this PR, this is immediately a run-time error,
alerting the user to the problematic code. As a side note, we'll need to
PR this fix (`get -i 0 | get -i expansion` -> `get -i 0.expansion`) to
carapace.
A notable exception to the type checking is commands with input type of
`nothing -> <type>`. In this case, any input type is allowed. This
allows piping values into the command without an error being thrown. For
example, `123 | echo $in` would be an error without this exception.
Additionally, custom types bypass type checking (I believe this also
happens during parsing, but not certain)
I added a `is_subtype` method to `Value` and `PipelineData`. It
functions slightly differently than `get_type().is_subtype()`, as noted
in the doccomments. Notably, it respects structural typing of lists and
tables. For example, the type of a value `[{a: 123} {a: 456, b: 789}]`
is a subtype of `table<a: int>`, whereas the type returned by
`Value::get_type` is a `list<any>`. Similarly, `PipelineData` has some
special handling for `ListStream`s and `ByteStream`s. The latter was
needed for this PR to work properly with external commands.
Here's some examples.
Before:
```nu
1..2 | drop nth 1
Error: nu::parser::input_type_mismatch
× Command does not support range input.
╭─[entry #9:1:8]
1 │ 1..2 | drop nth 1
· ────┬───
· ╰── command doesn't support range input
╰────
echo 1..2 | drop nth 1
# => ╭───┬───╮
# => │ 0 │ 1 │
# => ╰───┴───╯
```
After this PR, I've adjusted `drop nth`'s input/output types to accept
range input.
Before this PR, zip accepted any value despite not being listed in its
input/output types. This caused different behavior depending on if you
triggered a parse error or not:
```nushell
1 | zip [2]
# => Error: nu::parser::input_type_mismatch
# =>
# => × Command does not support int input.
# => ╭─[entry #3:1:5]
# => 1 │ 1 | zip [2]
# => · ─┬─
# => · ╰── command doesn't support int input
# => ╰────
echo 1 | zip [2]
# => ╭───┬───────────╮
# => │ 0 │ ╭───┬───╮ │
# => │ │ │ 0 │ 1 │ │
# => │ │ │ 1 │ 2 │ │
# => │ │ ╰───┴───╯ │
# => ╰───┴───────────╯
```
After this PR, it works the same in both cases. For cases like this, if
we do decide we want `zip` or other commands to accept any input value,
then we should explicitly add that to the input types.
```nushell
1 | zip [2]
# => Error: nu::parser::input_type_mismatch
# =>
# => × Command does not support int input.
# => ╭─[entry #3:1:5]
# => 1 │ 1 | zip [2]
# => · ─┬─
# => · ╰── command doesn't support int input
# => ╰────
echo 1 | zip [2]
# => Error: nu:🐚:only_supports_this_input_type
# =>
# => × Input type not supported.
# => ╭─[entry #14:2:6]
# => 2 │ echo 1 | zip [2]
# => · ┬ ─┬─
# => · │ ╰── only list<any> and range input data is supported
# => · ╰── input type: int
# => ╰────
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
**Breaking change**: The type of a command's input is now checked
against the input/output types of that command at run-time. While these
errors should mostly be caught at parse-time, in cases where they can't
be detected at parse-time they will be caught at run-time instead. This
applies to both internal commands and custom commands.
Example function and corresponding parse-time error (same before and
after PR):
```nushell
def foo []: int -> nothing {
print $"my cool int is ($in)"
}
1 | foo
# => my cool int is 1
"evil string" | foo
# => Error: nu::parser::input_type_mismatch
# =>
# => × Command does not support string input.
# => ╭─[entry #16:1:17]
# => 1 │ "evil string" | foo
# => · ─┬─
# => · ╰── command doesn't support string input
# => ╰────
# =>
```
Before:
```nu
echo "evil string" | foo
# => my cool int is evil string
```
After:
```nu
echo "evil string" | foo
# => Error: nu:🐚:only_supports_this_input_type
# =>
# => × Input type not supported.
# => ╭─[entry #17:1:6]
# => 1 │ echo "evil string" | foo
# => · ──────┬────── ─┬─
# => · │ ╰── only int input data is supported
# => · ╰── input type: string
# => ╰────
```
Known affected internal commands which erroneously accepted any type:
* `str join`
* `zip`
* `reduce`
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
* Play whack-a-mole with the commands and scripts this will inevitably
break
# Description
Fix the docs repo CI build error here:
https://github.com/nushell/nushell.github.io/actions/runs/12425087184/job/34691291790#step:5:18
The doc generated by `make_docs.nu` for `polars profile` command will
make the CI build fail due to the indention error of markdown front
matters. I used to fix it manually before, for the long run, it's better
to fix it from the source code.
- fixes#14572
# Description
This allowed columns to be coalesced on full joins with `polars join`,
providing functionality simlar to the old `--outer` join behavior.
# User-Facing Changes
- Provides a new flag `--coalesce-columns` on the `polars join` command
Addresses some null handling issues in #6882
# Description
This changes the implementation of guessing a column type when a schema
is not specified.
New behavior:
1. Use the first non-Value::Nothing value type for the columns data type
2. If the value type changes (ignoring Value::Nothing) in subsequent
values, the datatype will be changed to DataType::Object("Value", None)
3. If a column type does not have a value type,
DataType::Object("Value", None) will be assumed.
The `--name` flag of `polars with-column` only works when used with an
eager dataframe. I will not work with lazy dataframes and it will not
work when used with expressions (which forces a conversion to a
lazyframe). This pull request adds better documentation to the flags and
errors messages when used in cases where it will not work.
Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.6.0 to
2.7.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/indexmap-rs/indexmap/blob/master/RELEASES.md">indexmap's
changelog</a>.</em></p>
<blockquote>
<h2>2.7.0 (2024-11-30)</h2>
<ul>
<li>Added methods <code>Entry::insert_entry</code> and
<code>VacantEntry::insert_entry</code>, returning
an <code>OccupiedEntry</code> after insertion.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="539b401151"><code>539b401</code></a>
Merge pull request <a
href="https://redirect.github.com/indexmap-rs/indexmap/issues/361">#361</a>
from cuviper/insert_entry</li>
<li><a
href="998edb12fe"><code>998edb1</code></a>
Release 2.7.0</li>
<li><a
href="2a0ca97417"><code>2a0ca97</code></a>
Add <code>{Entry,VacantEntry}::insert_entry</code></li>
<li><a
href="dceb0f0598"><code>dceb0f0</code></a>
Merge pull request <a
href="https://redirect.github.com/indexmap-rs/indexmap/issues/360">#360</a>
from cuviper/collect_vec_list</li>
<li><a
href="c095322249"><code>c095322</code></a>
ci: downgrade hashbrown for 1.63</li>
<li><a
href="7d8cef8b4b"><code>7d8cef8</code></a>
Use rayon-1.9.0's <code>collect_vec_list</code></li>
<li>See full diff in <a
href="https://github.com/indexmap-rs/indexmap/compare/2.6.0...2.7.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
