mirror of
https://github.com/nushell/nushell.git
synced 2025-05-10 04:54:28 +02:00
78de505280
188 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
b0d68c31e8
|
build(deps): bump tokio from 1.44.2 to 1.45.0 (#15710)
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.44.2 to 1.45.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tokio-rs/tokio/releases">tokio's releases</a>.</em></p> <blockquote> <h2>Tokio v1.45.0</h2> <h3>Added</h3> <ul> <li>metrics: stabilize <code>worker_total_busy_duration</code>, <code>worker_park_count</code>, and <code>worker_unpark_count</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/6899">#6899</a>, <a href="https://redirect.github.com/tokio-rs/tokio/issues/7276">#7276</a>)</li> <li>process: add <code>Command::spawn_with</code> (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7249">#7249</a>)</li> </ul> <h3>Changed</h3> <ul> <li>io: do not require <code>Unpin</code> for some trait impls (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7204">#7204</a>)</li> <li>rt: mark <code>runtime::Handle</code> as unwind safe (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7230">#7230</a>)</li> <li>time: revert internal sharding implementation (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7226">#7226</a>)</li> </ul> <h3>Unstable</h3> <ul> <li>rt: remove alt multi-threaded runtime (<a href="https://redirect.github.com/tokio-rs/tokio/issues/7275">#7275</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/tokio/issues/6899">#6899</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/6899">tokio-rs/tokio#6899</a> <a href="https://redirect.github.com/tokio-rs/tokio/issues/7276">#7276</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/7276">tokio-rs/tokio#7276</a> <a href="https://redirect.github.com/tokio-rs/tokio/issues/7249">#7249</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/7249">tokio-rs/tokio#7249</a> <a href="https://redirect.github.com/tokio-rs/tokio/issues/7204">#7204</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/7204">tokio-rs/tokio#7204</a> <a href="https://redirect.github.com/tokio-rs/tokio/issues/7230">#7230</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/7230">tokio-rs/tokio#7230</a> <a href="https://redirect.github.com/tokio-rs/tokio/issues/7226">#7226</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/7226">tokio-rs/tokio#7226</a> <a href="https://redirect.github.com/tokio-rs/tokio/issues/7275">#7275</a>: <a href="https://redirect.github.com/tokio-rs/tokio/pull/7275">tokio-rs/tokio#7275</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
Jack Wright
|
ff8831318d
|
Added polars struct-encode-json, providing the ability to encode structs as json (#15678)
# Description
This PR introduces `polars struct-encode-json`. This exposes the ability
to encode struct columns as json strings. This is useful when converting
things to formats like CSV that do not support complex types.
```nushell
> ❯ : [[id person]; [1 {name: "Bob", age: 36}] [2 {name: "Betty", age: 63}]]
| polars into-df -s {id: i64, person: {name: str, age: u8}}
| polars select id (polars col person | polars struct-json-encode | polars as encoded)
| polars collect
╭───┬────┬───────────────────────────╮
│ # │ id │ encoded │
├───┼────┼───────────────────────────┤
│ 0 │ 1 │ {"age":36,"name":"Bob"} │
│ 1 │ 2 │ {"age":63,"name":"Betty"} │
╰───┴────┴───────────────────────────╯
```
# User-Facing Changes
* Added `polars struct-encode-json`, providing the ability to encode
structs as json
|
||
pyz4
|
ce582cdafb
|
feat(polars): add polars horizontal aggregation command (#15656)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR seeks to port over the `*_horizontal` commands in polars rust/python (e.g., https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.sum_horizontal.html), which aggregate across multiple columns (as opposed to rows). See below for several examples. ```nushell # Horizontal sum across two columns (ignore nulls by default) > [[a b]; [1 2] [2 3] [3 4] [4 5] [5 null]] | polars into-df | polars select (polars horizontal sum a b) | polars collect ╭───┬─────╮ │ # │ sum │ ├───┼─────┤ │ 0 │ 3 │ │ 1 │ 5 │ │ 2 │ 7 │ │ 3 │ 9 │ │ 4 │ 5 │ ╰───┴─────╯ # Horizontal sum across two columns while accounting for nulls > [[a b]; [1 2] [2 3] [3 4] [4 5] [5 null]] | polars into-df | polars select (polars horizontal sum a b --nulls) | polars collect ╭───┬─────╮ │ # │ sum │ ├───┼─────┤ │ 0 │ 3 │ │ 1 │ 5 │ │ 2 │ 7 │ │ 3 │ 9 │ │ 4 │ │ ╰───┴─────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users have access to a new command, `polars horizontal`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Example tests were added to `polars horizontal`. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Yash Thakur
|
eadb8da9f7
|
Bump to 0.104.1 dev version (#15669)
Marks development or hotfix |
||
|
Yash Thakur
|
cda15d91dd
|
Bump version for 0.104.0 release (#15664) | ||
Douglas
|
a1b7574306
|
Renamed join_where to join-where (#15660)
Renames the new `polars join_where` to `polars join-where` so that it conforms to the other Polars commands. |
||
|
pyz4
|
d1d6518ece
|
feat(polars): enable parsing strings as dates and datetime in polars schema (#15645)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR seeks to add a quality-of-life feature that enables date and datetime parsing of strings in `polars into-df`, `polars into-lazy`, and `polars open`, and avoid the more verbose method of casting each column into date/datetime. Currently, setting the schema to `date` on a `str` column would silently error as a null column. See a comparison of the current and proposed implementations. The proposed implementation assumes a date format "%Y-%m-%d" and a datetime format of "%Y-%m-%d %H:%M:%S" for naive datetimes and "%Y-%m-%d %H:%M:%S%:z" for timezone-aware datetimes. Other formats must be specified via parsing through `polars as-date` and `polars as-datetime`. ```nushell # Current Implementations > [[a]; ["2025-04-01"]] | polars into-df --schema {a: date} ╭───┬───╮ │ # │ a │ ├───┼───┤ │ 0 │ │ ╰───┴───╯ > [[a]; ["2025-04-01 01:00:00"]] | polars into-df --schema {a: "datetime<ns,*>"} ╭───┬───╮ │ # │ a │ ├───┼───┤ │ 0 │ │ ╰───┴───╯ # Proposed Implementation > [[a]; ["2025-04-01"]] | polars into-df --schema {a: date} ╭───┬─────────────────────╮ │ # │ a │ ├───┼─────────────────────┤ │ 0 │ 04/01/25 12:00:00AM │ ╰───┴─────────────────────╯ > [[a]; ["2025-04-01 01:00:00"]] | polars into-df --schema {a: "datetime<ns,*>"} ╭───┬─────────────────────╮ │ # │ a │ ├───┼─────────────────────┤ │ 0 │ 04/01/25 01:00:00AM │ ╰───┴─────────────────────╯ > [[a]; ["2025-04-01 01:00:00-04:00"]] | polars into-df --schema {a: "datetime<ns,UTC>"} ╭───┬─────────────────────╮ │ # │ a │ ├───┼─────────────────────┤ │ 0 │ 04/01/25 05:00:00AM │ ╰───┴─────────────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users have the added option to parse string columns into date/datetimes. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> No tests were added to any examples. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
715b0d90a9
|
fix(polars): conversion from nanoseconds to time_units in Datetime and Duration parsing (#15637)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> The current implementation improperly inverts the conversion from nanoseconds to the specified time units, resulting in nonsensical Datetime and Duration parsing and integer overflows when the specified time unit is not nanoseconds. This PR seeks to correct this conversion by changing the multiplication to an integer division. Below are examples highlighting the current and proposed implementations. ## Current Implementation Specifying a different time unit incorrectly changes the returned value. ```nushell > [[a]; [2024-04-01]] | polars into-df --schema {a: "datetime<ns,UTC>"} ╭───┬───────────────────────╮ │ # │ a │ ├───┼───────────────────────┤ │ 0 │ 04/01/2024 12:00:00AM │ > [[a]; [2024-04-01]] | polars into-df --schema {a: "datetime<ms,UTC>"} ╭───┬───────────────────────╮ │ # │ a │ ├───┼───────────────────────┤ │ 0 │ 06/27/2035 11:22:33PM │ <-- changing the time unit should not change the actual value > [[a]; [1day]] | polars into-df --schema {a: "duration<ns>"} ╭───┬────────────────╮ │ # │ a │ ├───┼────────────────┤ │ 0 │ 86400000000000 │ ╰───┴────────────────╯ > [[a]; [1day]] | polars into-df --schema {a: "duration<ms>"} ╭───┬──────────────────────╮ │ # │ a │ ├───┼──────────────────────┤ │ 0 │ -5833720368547758080 │ <-- i64 overflow ╰───┴──────────────────────╯ ``` ## Proposed Implementation ```nushell > [[a]; [2024-04-01]] | polars into-df --schema {a: "datetime<ns,UTC>"} ╭───┬───────────────────────╮ │ # │ a │ ├───┼───────────────────────┤ │ 0 │ 04/01/2024 12:00:00AM │ ╰───┴───────────────────────╯ > [[a]; [2024-04-01]] | polars into-df --schema {a: "datetime<ms,UTC>"} ╭───┬───────────────────────╮ │ # │ a │ ├───┼───────────────────────┤ │ 0 │ 04/01/2024 12:00:00AM │ ╰───┴───────────────────────╯ > [[a]; [1day]] | polars into-df --schema {a: "duration<ns>"} ╭───┬────────────────╮ │ # │ a │ ├───┼────────────────┤ │ 0 │ 86400000000000 │ ╰───┴────────────────╯ > [[a]; [1day]] | polars into-df --schema {a: "duration<ms>"} ╭───┬──────────╮ │ # │ a │ ├───┼──────────┤ │ 0 │ 86400000 │ ╰───┴──────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No user-facing breaking change. Developer breaking change: to mitigate the silent overflow in nanoseconds conversion functions `nanos_from_timeunit` and `nanos_to_timeunit` (new), the function signatures were changed from `i64` to `Result<i64, ShellError>`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> No additional examples were added, but I'd be happy to add a few if needed. The covering tests just didn't fit well into any examples. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Matthias Meschede
|
05c36d1bc7
|
add polars join_where command (#15635)
# Description This adds `polars join_where` which allows joining two dataframes based on a conditions. The command can be used as: ``` ➜ let df_a = [[name cash];[Alice 5] [Bob 10]] | polars into-lazy ➜ let df_b = [[item price];[A 3] [B 7] [C 12]] | polars into-lazy ➜ $df_a | polars join_where $df_b ((polars col cash) > (polars col price)) | polars collect ╭───┬───────┬──────┬──────┬───────╮ │ # │ name │ cash │ item │ price │ ├───┼───────┼──────┼──────┼───────┤ │ 0 │ Bob │ 10 │ B │ 7 │ │ 1 │ Bob │ 10 │ A │ 3 │ │ 2 │ Alice │ 5 │ A │ 3 │ ╰───┴───────┴──────┴──────┴───────╯ ``` # User-Facing Changes - new command `polars join_where` |
||
|
pyz4
|
208ebeefab
|
feat(polars): enable parsing decimals in polars schemas (#15632)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR enables the option to set a column type to `decimal` in the `--schema` parameter of `polars into-df` and `polars into-lazy` commands. This option was already available in `polars open`, which used the underlying polars io commands that already accounted for decimal types when specified in the schema. See below for a comparison of the current and proposed implementation. ```nushell # Current Implementation > [[a b]; [1 1.618]]| polars into-df -s {a: u8, b: 'decimal<4,3>'} Error: × Error creating dataframe: Unsupported type: Decimal(Some(4), Some(3)) # Proposed Implementation > [[a b]; [1 1.618]]| polars into-df -s {a: u8, b: 'decimal<4,3>'} | polars schema ╭───┬──────────────╮ │ a │ u8 │ │ b │ decimal<4,3> │ ╰───┴──────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking change. Users has the new option to specify decimal in `--schema` in `polars into-df` and `polars into-lazy`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> An example in `polars into-df` was modified to showcase the decimal type. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Matthias Meschede
|
717081bd2f
|
fix mistake in description of polars pivot command (#15621)
Very small change to fix a typo/mistake in the polars pivot command description. |
||
|
pyz4
|
1db4be12d1
|
fix(polars): remove requirement that pivot columns must be same type in polars pivot (#15608)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Contrary to the underlying implementation in polars rust/python, `polars pivot` throws an error if the user tries to pivot on multiple columns of different types. This PR seeks to remove this type-check. See comparison below. ```nushell # Current implementation: throws error when pivoting on multiple values of different types. > [[name subject date test_1 test_2 grade_1 grade_2]; [Cady maths 2025-04-01 98 100 A A] [Cady physics 2025-04-01 99 100 A A] [Karen maths 2025-04-02 61 60 D D] [Karen physics 2025-04-02 58 60 D D]] | polars into-df | polars pivot --on [subject] --index [name] --values [test_1 grade_1] Error: × Merge error ╭─[entry #291:1:271] 1 │ [[name subject date test_1 test_2 grade_1 grade_2]; [Cady maths 2025-04-01 98 100 A A] [Cady physics 2025-04-01 99 100 A A] [Karen maths 2025-04-02 61 60 D D] [Karen physics 2025-04-02 58 60 D D]] | polars into-df | polars pivot --on [subject] --index [name] --values [test_1 grade_1] · ───────┬────── · ╰── found different column types in list ╰──── help: datatypes i64 and str are incompatible # Proposed implementation > [[name subject date test_1 test_2 grade_1 grade_2]; [Cady maths 2025-04-01 98 100 A A] [Cady physics 2025-04-01 99 100 A A] [Karen maths 2025-04-02 61 60 D D] [Karen physics 2025-04-02 58 60 D D]] | polars into-df | polars pivot --on [subject] --index [name] --values [test_1 grade_1] ╭───┬───────┬──────────────┬────────────────┬───────────────┬─────────────────╮ │ # │ name │ test_1_maths │ test_1_physics │ grade_1_maths │ grade_1_physics │ ├───┼───────┼──────────────┼────────────────┼───────────────┼─────────────────┤ │ 0 │ Cady │ 98 │ 99 │ A │ A │ │ 1 │ Karen │ 61 │ 58 │ D │ D │ ╰───┴───────┴──────────────┴────────────────┴───────────────┴─────────────────╯ ``` Additionally, this PR ports over the `separator` parameter in `pivot`, which allows the user to specify how to delimit multiple `values` column names: ```nushell > [[name subject date test_1 test_2 grade_1 grade_2]; [Cady maths 2025-04-01 98 100 A A] [Cady physics 2025-04-01 99 100 A A] [Karen maths 2025-04-02 61 60 D D] [Karen physics 2025-04-02 58 60 D D]] | polars into-df | polars pivot --on [subject] --index [name] --values [test_1 grade_1] --separator / ╭───┬───────┬──────────────┬────────────────┬───────────────┬─────────────────╮ │ # │ name │ test_1/maths │ test_1/physics │ grade_1/maths │ grade_1/physics │ ├───┼───────┼──────────────┼────────────────┼───────────────┼─────────────────┤ │ 0 │ Cady │ 98 │ 99 │ A │ A │ │ 1 │ Karen │ 61 │ 58 │ D │ D │ ╰───┴───────┴──────────────┴────────────────┴───────────────┴─────────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Soft breaking change: where a user may have previously expected an error (pivoting on multiple columns with different types), no error is thrown. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Examples were added to `polars pivot`. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
a2dc3e3b33
|
feat(polars): enable as_date and as_datetime to handle expressions as inputs (#15590)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR is a follow-up to the previous PR #15557 and part of a wider campaign to enable certain polars commands that only operated on the entire dataframe to also operate on expressions. Here, we enable two commands `polars as-date` and `polars as-datetime` to receive expressions as inputs so that they may be used on specific columns in a dataframe with multiple columns of different types. See examples below. ```nushell > [[a b]; ["2025-04-01" 1] ["2025-04-02" 2] ["2025-04-03" 3]] | polars into-df | polars select (polars col a | polars as-date %Y-%m-%d) b | polars collect ╭───┬───────────────────────┬───╮ │ # │ a │ b │ ├───┼───────────────────────┼───┤ │ 0 │ 04/01/2025 12:00:00AM │ 1 │ │ 1 │ 04/02/2025 12:00:00AM │ 2 │ │ 2 │ 04/03/2025 12:00:00AM │ 3 │ ╰───┴───────────────────────┴───╯ > seq date -b 2025-04-01 --periods 4 --increment 25min -o "%Y-%m-%d %H:%M:%S" | polars into-df | polars select (polars col 0 | polars as-datetime "%Y-%m-%d %H:%M:%S") | polars collect ╭───┬───────────────────────╮ │ # │ 0 │ ├───┼───────────────────────┤ │ 0 │ 04/01/2025 12:00:00AM │ │ 1 │ 04/01/2025 12:25:00AM │ │ 2 │ 04/01/2025 12:50:00AM │ │ 3 │ 04/01/2025 01:15:00AM │ ╰───┴───────────────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users have the additional option to use `polars as-date` and `polars as-datetime` in expressions that operate on specific columns. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Examples have been added to `polars as-date` and `polars as-datetime`. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
95998bdd53
|
fix(custom_value) + fix(polars): map // operator to FloorDivide for custom values and in polars (#15599)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR fixes an issue where, for custom values, the `//` operator was incorrectly mapped to `Math::Divide` instead of `Math::FloorDivide`. This PR also fixes the same mis-mapping in the `polars` plugin. ```nushell > [[a b c]; [x 1 1.1] [y 2 2.2] [z 3 3.3]] | polars into-df | polars select {div: ((polars col c) / (polars col b)), floor_div: ((polars col c) // (polars col b))} | polars collect ╭───┬───────┬───────────╮ │ # │ div │ floor_div │ ├───┼───────┼───────────┤ │ 0 │ 1.100 │ 1.000 │ │ 1 │ 1.100 │ 1.000 │ │ 2 │ 1.100 │ 1.000 │ ╰───┴───────┴───────────╯ ``` **Note:** the number of line changes in this PR is inflated because of auto-formatting in `nu_plugin_polars/Cargo.toml`. Substantively, I've only added the `round_series` feature to the polars dependency list. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Breaking change: users who expected the operator `//` to function the same as `/` for custom values will not get the expected result. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> No tests were yet added, but let me know if we should put something into one of the polars examples. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
bd5de023a1
|
feat(polars): add pow (**) operator for polars expressions (#15598)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds the exponent operator ("**") to polars expressions. ```nushell > [[a b]; [6 2] [4 2] [2 2]] | polars into-df | polars select a b {c: ((polars col a) ** 2)} ╭───┬───┬───┬────╮ │ # │ a │ b │ c │ ├───┼───┼───┼────┤ │ 0 │ 6 │ 2 │ 36 │ │ 1 │ 4 │ 2 │ 16 │ │ 2 │ 2 │ 2 │ 4 │ ╰───┴───┴───┴────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users are enabled to use the `**` operator in polars expressions. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> An example in `polars select` was modified to showcase the `**` operator. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
0e9927ea4d
|
polars: expand polars col to handle multiple columns and by types (#15570)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR seeks to expand `polars col` functionality to allow selecting multiple columns and columns by type, which is particularly useful when piping to subsequent expressions that should be applied to each column selected (e.g., `polars col int --type | polars sum` as a shorthand for `[(polars col a | polars sum), (polars col b | polars sum)]`). See examples below. ```nushell # Select multiple columns (cannot be used with asterisk wildcard) > [[a b c]; [x 1 1.1] [y 2 2.2] [z 3 3.3]] | polars into-df | polars select (polars col b c | polars sum) | polars collect ╭───┬───┬──────╮ │ # │ b │ c │ ├───┼───┼──────┤ │ 0 │ 6 │ 6.60 │ ╰───┴───┴──────╯ # Select multiple columns by types (cannot be used with asterisk wildcard) > [[a b c]; [x o 1.1] [y p 2.2] [z q 3.3]] | polars into-df | polars select (polars col str f64 --type | polars max) | polars collect ╭───┬───┬───┬──────╮ │ # │ a │ b │ c │ ├───┼───┼───┼──────┤ │ 0 │ z │ q │ 3.30 │ ╰───┴───┴───┴──────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users have the additional capability to select multiple columns in `polars col`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Examples have been added to `polars col`. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
2dc5c19b71
|
feat(polars): loosen constraints on accepted expressions in polars group-by (#15583)
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR lifts the constraint that expressions in the `polars group-by`
command must be limited only to the type `Expr::Column` rather than most
`Expr` types, which is what the underlying polars crate allows. This
change enables more complex expressions to group by.
In the example below, we group by even or odd days of column `a`. While
we can reach the same result by creating and grouping by a new column in
two separate steps, integrating these steps in a single group-by allows
for better delegation to the polars optimizer.
```nushell
# Group by an expression and perform an aggregation
> [[a b]; [2025-04-01 1] [2025-04-02 2] [2025-04-03 3] [2025-04-04 4]]
| polars into-lazy
| polars group-by (polars col a | polars get-day | $in mod 2)
| polars agg [
(polars col b | polars min | polars as "b_min")
(polars col b | polars max | polars as "b_max")
(polars col b | polars sum | polars as "b_sum")
]
| polars collect
| polars sort-by a
╭───┬───┬───────┬───────┬───────╮
│ # │ a │ b_min │ b_max │ b_sum │
├───┼───┼───────┼───────┼───────┤
│ 0 │ 0 │ 2 │ 4 │ 6 │
│ 1 │ 1 │ 1 │ 3 │ 4 │
╰───┴───┴───────┴───────┴───────╯
```
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
No breaking changes. The user is empowered to use more complex
expressions in `polars group-by`
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
An example is added to `polars group-by`.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
|
||
|
pyz4
|
669b44ad7d
|
feat(polars): add polars truncate for rounding datetimes (#15582)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR directly ports the polars function `polars.Expr.dt.truncate` (https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.dt.truncate.html), which rounds a datetime to an arbitrarily specified period length. This function is particularly useful when rounding to variable period lengths such as months or quarters. See below for examples. ```nushell # Truncate a series of dates by period length > seq date -b 2025-01-01 --periods 4 --increment 6wk -o "%Y-%m-%d %H:%M:%S" | polars into-df | polars as-datetime "%F %H:%M:%S" --naive | polars select datetime (polars col datetime | polars truncate 5d37m | polars as truncated) | polars collect ╭───┬───────────────────────┬───────────────────────╮ │ # │ datetime │ truncated │ ├───┼───────────────────────┼───────────────────────┤ │ 0 │ 01/01/2025 12:00:00AM │ 12/30/2024 04:49:00PM │ │ 1 │ 02/12/2025 12:00:00AM │ 02/08/2025 09:45:00PM │ │ 2 │ 03/26/2025 12:00:00AM │ 03/21/2025 02:41:00AM │ │ 3 │ 05/07/2025 12:00:00AM │ 05/05/2025 08:14:00AM │ ╰───┴───────────────────────┴───────────────────────╯ # Truncate based on period length measured in quarters and months > seq date -b 2025-01-01 --periods 4 --increment 6wk -o "%Y-%m-%d %H:%M:%S" | polars into-df | polars as-datetime "%F %H:%M:%S" --naive | polars select datetime (polars col datetime | polars truncate 1q5mo | polars as truncated) | polars collect ╭───┬───────────────────────┬───────────────────────╮ │ # │ datetime │ truncated │ ├───┼───────────────────────┼───────────────────────┤ │ 0 │ 01/01/2025 12:00:00AM │ 09/01/2024 12:00:00AM │ │ 1 │ 02/12/2025 12:00:00AM │ 09/01/2024 12:00:00AM │ │ 2 │ 03/26/2025 12:00:00AM │ 09/01/2024 12:00:00AM │ │ 3 │ 05/07/2025 12:00:00AM │ 05/01/2025 12:00:00AM │ ╰───┴───────────────────────┴───────────────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. This PR introduces a new command `polars truncate` # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Example test was added. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jack Wright
|
8f81812ef9
|
fix cannot find issue when performing collect on an eager dataframe (#15577)
# Description Performing a `polars collect` on an eager dataframe should be a no-op operation. However, when used with a pipeline and not saving to a value a cache error occurs. This addresses that cache error. |
||
|
pyz4
|
a33650a69e
|
fix(polars): cast as date now returns Date type instead of Datetime<ns> (#15574)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR fixes the bug where various commands that cast a column as a `date` type would return `datetime<ns>` rather than the intended type `date`. Affected commands include `polars into-df --schema`, `polars into-lazy --schema`, `polars as-date`, and `polars cast date`. This bug derives from the fact that Nushell uses the `date` type to denote a datetime type whereas polars differentiates between `Date` and `Datetime` types. By default, this PR retains the behavior that a Nushell `date` type will be mapped to a polars `Datetime<ns>` unless otherwise specified. ```nushell # Current (erroneous) implementation > [[a]; [2025-03-20]] | polars into-df --schema {a: "date"} | polars schema ╭───┬──────────────╮ │ a │ datetime<ns> │ ╰───┴──────────────╯ # Fixed implementation > [[a]; [2025-03-20]] | polars into-df --schema {a: "date"} | polars schema ╭───┬──────╮ │ a │ date │ ╰───┴──────╯ # Fixed implementation: by default, Nushell dates map to datetime<ns> > [[a]; [2025-03-20]] | polars into-df | polars schema ╭───┬───────────────────╮ │ a │ datetime<ns, UTC> │ ╰───┴───────────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Soft breaking change: users previously who wanted to cast a date column to type `date` can now expect the output to be type `date` instead of `datetime<ns>`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Example test added to `polars as-date` command. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jack Wright
|
89322f59f2
|
Fix output type of polars schema (#15572)
# Description Output type of `polars schema` signature output type is of dataframe. It should be of type record. # User-Facing Changes - `polars schema` - how has an output type of record |
||
|
pyz4
|
4e307480e4
|
polars: extend NuExpression::extract_exprs to handle records (#15553)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR seeks to simplify the syntax for commands that handle a list of expressions (e.g., `select`, `with-column`, and `agg`) by enabling the user to replace a list of expressions each aliased with `polars as` to a single record where the key is the alias for the value. See below for examples in several contexts. ```nushell # Select a column from a dataframe using a record > [[a b]; [6 2] [4 2] [2 2]] | polars into-df | polars select {c: ((polars col a) * 2)} ╭───┬────╮ │ # │ c │ ├───┼────┤ │ 0 │ 12 │ │ 1 │ 8 │ │ 2 │ 4 │ ╰───┴────╯ # Select a column from a dataframe using a mix of expressions and record of expressions > [[a b]; [6 2] [4 2] [2 2]] | polars into-df | polars select a b {c: ((polars col a) * 2)} ╭───┬───┬───┬────╮ │ # │ a │ b │ c │ ├───┼───┼───┼────┤ │ 0 │ 6 │ 2 │ 12 │ │ 1 │ 4 │ 2 │ 8 │ │ 2 │ 2 │ 2 │ 4 │ ╰───┴───┴───┴────╯ # Add series to the dataframe using a record > [[a b]; [1 2] [3 4]] | polars into-lazy | polars with-column { c: ((polars col a) * 2) d: ((polars col a) * 3) } | polars collect ╭───┬───┬───┬───┬───╮ │ # │ a │ b │ c │ d │ ├───┼───┼───┼───┼───┤ │ 0 │ 1 │ 2 │ 2 │ 3 │ │ 1 │ 3 │ 4 │ 6 │ 9 │ ╰───┴───┴───┴───┴───╯ # Group by and perform an aggregation using a record > [[a b]; [1 2] [1 4] [2 6] [2 4]] | polars into-lazy | polars group-by a | polars agg { b_min: (polars col b | polars min) b_max: (polars col b | polars max) b_sum: (polars col b | polars sum) } | polars collect | polars sort-by a ╭───┬───┬───────┬───────┬───────╮ │ # │ a │ b_min │ b_max │ b_sum │ ├───┼───┼───────┼───────┼───────┤ │ 0 │ 1 │ 2 │ 4 │ 6 │ │ 1 │ 2 │ 4 │ 6 │ 10 │ ╰───┴───┴───────┴───────┴───────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users now can use a mix of lists of expressions and records of expressions where previously only lists of expressions were accepted (e.g., in `select`, `with-column`, and `agg`). # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Example tests were added to `select`, `with-column`, and `agg`. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
ceaa0f9375
|
polars: add new command polars over (#15551)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Introducing a basic implementation of the polars expression for window functions: `over` (https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.over.html). Note that this PR only implements the default values for the sorting and `mapping_strategy` parameters. Implementations for other values for these parameters may be added in a future PR, as the demand arises. ```nushell # Compute expression over an aggregation window > [[a b]; [x 2] [x 4] [y 6] [y 4]] | polars into-lazy | polars select a (polars col b | polars cumulative sum | polars over a | polars as cum_b) | polars collect ╭───┬───┬───────╮ │ # │ a │ cum_b │ ├───┼───┼───────┤ │ 0 │ x │ 2 │ │ 1 │ x │ 6 │ │ 2 │ y │ 6 │ │ 3 │ y │ 10 │ ╰───┴───┴───────╯ # Compute expression over an aggregation window where partitions are defined by expressions > [[a b]; [x 2] [X 4] [Y 6] [y 4]] | polars into-lazy | polars select a (polars col b | polars cumulative sum | polars over (polars col a | polars lowercase) | polars as cum_b) | polars collect ╭───┬───┬───────╮ │ # │ a │ cum_b │ ├───┼───┼───────┤ │ 0 │ x │ 2 │ │ 1 │ X │ 6 │ │ 2 │ Y │ 6 │ │ 3 │ y │ 10 │ ╰───┴───┴───────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. This PR seeks to add a new command only. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Example tests are included. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
d31b7024d8
|
polars: update get- datetime components commands to allow expressions as inputs (#15557)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR updates the following functions so they may also be used in a polars expression: - `polars get-day` - `polars get-hour` - `polars get-minute` - `polars get-month` - `polars get-nanosecond` - `polars get-ordinal` - `polars get-second` - `polars get-week` - `polars get-weekday` - `polars get-year` Below examples provide a comparison of the two contexts in which each of these commands may be used: ```nushell # Returns day from a date (current use case) > let dt = ('2020-08-04T16:39:18+00:00' | into datetime --timezone 'UTC'); let df = ([$dt $dt] | polars into-df); $df | polars get-day ╭───┬───╮ │ # │ 0 │ ├───┼───┤ │ 0 │ 4 │ │ 1 │ 4 │ ╰───┴───╯ # Returns day from a date in an expression (additional use case provided by this PR) > let dt = ('2020-08-04T16:39:18+00:00' | into datetime --timezone 'UTC'); let df = ([$dt $dt] | polars into-df); $df | polars select (polars col 0 | polars get-day) ╭───┬───╮ │ # │ 0 │ ├───┼───┤ │ 0 │ 4 │ │ 1 │ 4 │ ╰───┴───╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Each of these functions retains its current behavior and gains the benefit that they can now be used in an expression as well. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Tests have been added to each of the examples. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
9dd30d7756
|
polars: update polars lit to handle nushell Value::Duration and Value::Date types (#15564)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR seeks to expand `polars lit` to handle additional nushell types: Value::Date and Value::Duration. This change is especially relevant to the `polars filter` command, where expressions would then directly incorporate Value::Date and Value::Duration types as literals. See one such example below. ```nushell # Filter dataframe for rows where dt is within the last 2 days of the maximum dt value > [[dt val]; [2025-04-01 1] [2025-04-02 2] [2025-04-03 3] [2025-04-04 4]] | polars into-df | polars filter ((polars col dt) > ((polars col dt | polars max | $in - 2day))) ╭───┬─────────────────────┬─────╮ │ # │ dt │ val │ ├───┼─────────────────────┼─────┤ │ 0 │ 04/03/25 12:00:00AM │ 3 │ │ 1 │ 04/04/25 12:00:00AM │ 4 │ ╰───┴─────────────────────┴─────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users now can directly access Value::Date and Value::Duration types as literals in polars expressions. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Several additional examples added to `polars lit` and `polars filter` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
885b87a842
|
polars: add new command polars convert-time-zone (#15550)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This is a direct port of the python polars command `convert_time_zone` (https://docs.pola.rs/api/python/stable/reference/series/api/polars.Series.dt.convert_time_zone.html). Consistent with the rust/python implementation, naive datetimes are treated as if they are in UTC time. ```nushell # Convert timezone for timezone-aware datetime > ["2025-04-10 09:30:00 -0400" "2025-04-10 10:30:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" | polars select (polars col datetime | polars convert-time-zone "Europe/Lisbon") ╭───┬───────────────────────╮ │ # │ datetime │ ├───┼───────────────────────┤ │ 0 │ 04/10/2025 02:30:00PM │ │ 1 │ 04/10/2025 03:30:00PM │ ╰───┴───────────────────────╯ # Timezone conversions for timezone-naive datetime will assume the original timezone is UTC > ["2025-04-10 09:30:00" "2025-04-10 10:30:00"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S" --naive | polars select (polars col datetime | polars convert-time-zone "America/New_York") ╭───┬───────────────────────╮ │ # │ datetime │ ├───┼───────────────────────┤ │ 0 │ 04/10/2025 05:30:00AM │ │ 1 │ 04/10/2025 06:30:00AM │ ╰───┴───────────────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> No breaking changes. Users have access to a new command `polars convert-time-zone` # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Example tests have been added. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
pyz4
|
1a0778d77e
|
polars: add new command polars replace-time-zone (#15538)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR seeks to add a direct port of the python polars `replace_time_zone` command in the `dt` namespace (https://docs.pola.rs/api/python/stable/reference/series/api/polars.Series.dt.replace_time_zone.html). Please note: I opted for two keywords "dt" and "replace-time-zone" to map directly with the implementation in both the rust and python packages, but I'm open to simplifying it to just one keyword, or `polars replace-time-zone` ```nushell # Apply timezone to a naive datetime > ["2021-12-30 00:00:00" "2021-12-31 00:00:00"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S" --naive | polars select (polars col datetime | polars dt replace-time-zone "America/New_York") ╭───┬─────────────────────╮ │ # │ datetime │ ├───┼─────────────────────┤ │ 0 │ 12/30/21 12:00:00AM │ │ 1 │ 12/31/21 12:00:00AM │ ╰───┴─────────────────────╯ # Apply timezone with ambiguous datetime > ["2025-11-02 00:00:00", "2025-11-02 01:00:00", "2025-11-02 02:00:00", "2025-11-02 03:00:00"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S" --naive | polars select (polars col datetime | polars dt replace-time-zone "America/New_York" --ambiguous null) ╭───┬─────────────────────╮ │ # │ datetime │ ├───┼─────────────────────┤ │ 0 │ 11/02/25 12:00:00AM │ │ 1 │ │ │ 2 │ 11/02/25 02:00:00AM │ │ 3 │ 11/02/25 03:00:00AM │ ╰───┴─────────────────────╯ # Apply timezone with nonexistent datetime > ["2025-03-09 01:00:00", "2025-03-09 02:00:00", "2025-03-09 03:00:00", "2025-03-09 04:00:00"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S" --naive | polars select (polars col datetime | polars dt replace-time-zone "America/New_York" --nonexistent null) ╭───┬─────────────────────╮ │ # │ datetime │ ├───┼─────────────────────┤ │ 0 │ 03/09/25 01:00:00AM │ │ 1 │ │ │ 2 │ 03/09/25 03:00:00AM │ │ 3 │ 03/09/25 04:00:00AM │ ╰───┴─────────────────────╯ ``` # User-Facing Changes No breaking changes. The user will be able to access the new command. # Tests + Formatting See example tests. # After Submitting |
||
|
Jack Wright
|
f8ed4b45fd
|
Introducing polars into-schema (#15534)
# Description
Introduces `polars into-schema` which allows converting Values such as
records to a schema. This implicitly happens when when passing records
into commands like `polars into-df` today. This allows you to convert to
a schema object ahead of time and reuse the schema object. This can be
useful for guaranteeing your schema object is correct.
```nu
> ❯ : let schema = ({name: str, type: str} | polars into-schema)
> ❯ : ls | select name type | polars into-lazy -s $schema | polars schema
╭──────┬─────╮
│ name │ str │
│ type │ str │
╰──────┴─────╯
```
# User-Facing Changes
- Introduces `polars into-schema` allowing records to be converted to
schema objects.
|
||
|
Jack Wright
|
b0f9cda9b5
|
Introduction of NuDataType and polars dtype (#15529)
# Description
This pull request does a lot of the heavy lifting needed to supported
more complex dtypes like categorical dtypes. It introduces a new
CustomValue, NuDataType and makes NuSchema a full CustomValue. Further
more it introduces a new command `polars into-dtype` that allows a dtype
to be created. This can then be passed into schemas when they are
created.
```nu
> ❯ : let dt = ("str" | polars to-dtype)
> ❯ : [[a b]; ["one" "two"]] | polars into-df -s {a: $dt, b: str} | polars schema
╭───┬─────╮
│ a │ str │
│ b │ str │
╰───┴─────╯
```
# User-Facing Changes
- Introduces new command `polars into-dtype`, allows dtype variables to
be passed in during schema creation.
|
||
|
dependabot[bot]
|
c0b944edb6
|
build(deps): bump indexmap from 2.8.0 to 2.9.0 (#15531)
Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.8.0 to 2.9.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/indexmap-rs/indexmap/blob/main/RELEASES.md">indexmap's changelog</a>.</em></p> <blockquote> <h2>2.9.0 (2025-04-04)</h2> <ul> <li>Added a <code>get_disjoint_mut</code> method to <code>IndexMap</code>, matching Rust 1.86's <code>HashMap</code> method.</li> <li>Added a <code>get_disjoint_indices_mut</code> method to <code>IndexMap</code> and <code>map::Slice</code>, matching Rust 1.86's <code>get_disjoint_mut</code> method on slices.</li> <li>Deprecated the <code>borsh</code> feature in favor of their own <code>indexmap</code> feature, solving a cyclic dependency that occured via <code>borsh-derive</code>.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
pyz4
|
147009a161
|
polars into-df/polars into-lazy: --schema will not throw error if only some columns are defined (#15473)
# Description
The current implementation of `polars into-df` and `polars into-lazy`
will throw an error if `--schema` is provided but not all columns are
defined. This PR seeks to remove this requirement so that when a partial
`--schema` is provided, the types on the defined columns are overridden
while the remaining columns take on their default types.
**Current Implementation**
```
$ [[a b]; [1 "foo"] [2 "bar"]] | polars into-df -s {a: str} | polars schema
Error: × Schema does not contain column: b
╭─[entry #88:1:12]
1 │ [[a b]; [1 "foo"] [2 "bar"]] | polars into-df -s {a: str} | polars schema
· ─────
╰────
```
**New Implementation (no error thrown on partial schema definition)**
Column b is not defined in `--schema`
```
$ [[a b]; [1 "foo"] [2 "bar"]] | polars into-df --schema {a: str} | polars schema
╭───┬─────╮
│ a │ str │
│ b │ str │
╰───┴─────╯
```
# User-Facing Changes
Soft breaking change: The user's previous (erroneous) code that would
have thrown an error would no longer throw an error. The user's previous
working code will still work.
# Tests + Formatting
# After Submitting
|
||
Wind
|
1c6c85d35d
|
Fix clippy (#15489)
# Description There are some clippy(version 0.1.86) errors on nushell repo. This pr is trying to fix it. # User-Facing Changes Hopefully none. # Tests + Formatting NaN # After Submitting NaN |
||
|
pyz4
|
7ca2a6f8ac
|
FIX polars as-datetime: ignores timezone information on conversion (#15490)
# Description This PR seeks to fix an error in `polars as-datetime` where timezone information is entirely ignored. This behavior raises a host of silent errors when dealing with datetime conversions (see example below). ## Current Implementation Timezones are entirely ignored and datetimes with different timezones are converted to the same naive datetimes even when the user specifically indicates that the timezone should be parsed. For example, "2021-12-30 00:00:00 +0000" and "2021-12-30 00:00:00 -0400" will both be parsed to "2021-12-30 00:00:00" even when the format string specifically includes "%z". ``` $ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" ╭───┬───────────────────────╮ │ # │ datetime │ ├───┼───────────────────────┤ │ 0 │ 12/30/2021 12:00:00AM │ │ 1 │ 12/30/2021 12:00:00AM │ <-- Same datetime even though the first is +0000 and second is -0400 ╰───┴───────────────────────╯ $ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" | polars schema ╭──────────┬──────────────╮ │ datetime │ datetime<ns> │ ╰──────────┴──────────────╯ ``` ## New Implementation Datetimes are converted to UTC and timezone information is retained. ``` $ "2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" ╭───┬───────────────────────╮ │ # │ datetime │ ├───┼───────────────────────┤ │ 0 │ 12/30/2021 12:00:00AM │ │ 1 │ 12/30/2021 04:00:00AM │ <-- Converted to UTC ╰───┴───────────────────────╯ $ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" | polars schema ╭──────────┬───────────────────╮ │ datetime │ datetime<ns, UTC> │ ╰──────────┴───────────────────╯ ``` The user may intentionally ignore timezone information by setting the `--naive` flag. ``` $ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" --naive ╭───┬───────────────────────╮ │ # │ datetime │ ├───┼───────────────────────┤ │ 0 │ 12/30/2021 12:00:00AM │ │ 1 │ 12/30/2021 12:00:00AM │ <-- the -0400 offset is ignored when --naive is set ╰───┴───────────────────────╯ $ ["2021-12-30 00:00:00 +0000" "2021-12-30 00:00:00 -0400"] | polars into-df | polars as-datetime "%Y-%m-%d %H:%M:%S %z" --naive | polars schema ╭──────────┬──────────────╮ │ datetime │ datetime<ns> │ ╰──────────┴──────────────╯ ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> `polars as-datetime` will now account for timezone information and return type `datetime<ns,UTC>` rather than `datetime<ns>` by default. The user can replicate the previous behavior by setting `--naive`. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> Tests that incorporated `polars as-datetime` had to be tweaked to include `--naive` flag to replicate previous behavior. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Darren Schroeder
|
2bf0397d80
|
bump to the latest rust version (#15483)
# Description This PR bumps nushell to use the latest rust version 1.84.1. |
||
|
pyz4
|
470d130289
|
polars cast: add decimal option for dtype parameter (#15464)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description This PR expands the `dtype` parameter of the `polars cast` command to include `decimal<precision, scale>` type. Setting precision to "*" will compel inferring the value. Note, however, setting scale to a non-integer value will throw an explicit error (the underlying polars crate assigns scale = 0 in such a case, but I opted for throwing an error instead). . ``` $ [[a b]; [1 2] [3 4]] | polars into-df | polars cast decimal<4,2> a | polars schema ╭───┬──────────────╮ │ a │ decimal<4,2> │ │ b │ i64 │ ╰───┴──────────────╯ $ [[a b]; [10.5 2] [3.1 4]] | polars into-df | polars cast decimal<*,2> a | polars schema ╭───┬──────────────╮ │ a │ decimal<*,2> │ │ b │ i64 │ ╰───┴──────────────╯ $ [[a b]; [10.05 2] [3.1 4]] | polars into-df | polars cast decimal<5,*> a | polars schema rror: × Invalid polars data type ╭─[entry #25:1:47] 1 │ [[a b]; [10.05 2] [3.1 4]] | polars into-df | polars cast decimal<5,*> a | polars schema · ─────┬───── · ╰── `*` is not a permitted value for scale ╰──── ``` # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> There are no breaking changes. The user has the additional option to `polars cast` to a decimal type # Tests + Formatting Tests have been added to `nu_plugin_polars/src/dataframe/values/nu_schema.rs` |
||
|
Jack Wright
|
eaf522b41f
|
Polars cut (#15431)
- fixes #15366 # Description Introducing binning commands, `polars cut` and `polars qcut` # User-Facing Changes - New command `polars cut` - New command `polars qcut` |
||
|
dependabot[bot]
|
1979b61a92
|
build(deps): bump tokio from 1.43.0 to 1.44.1 (#15419) | ||
|
dependabot[bot]
|
946cef77f1
|
build(deps): bump uuid from 1.12.0 to 1.16.0 (#15346) | ||
|
dependabot[bot]
|
c99c8119fe
|
build(deps): bump indexmap from 2.7.0 to 2.8.0 (#15345) | ||
|
Yash Thakur
|
2c7ab6e898
|
Bump to 0.103.1 dev version (#15347)
# Description Marks development or hotfix |
||
|
Yash Thakur
|
c986426478
|
Bump version for 0.103.0 release (#15340) | ||
|
Darren Schroeder
|
42aa2ff5ba
|
remove mimalloc allocator (#15317)
# Description This PR removes the mimalloc allocator due to run-away memory leaks recently found. closes #15311 # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jack Wright
|
0f6996b70d
|
Support for reading Categorical and Enum types (#15292)
# fixes https://github.com/nushell/nushell/issues/15281 # Description Provides the ability read dataframes with Categorical and Enum data The ability to write Categorical and Enum data will provided in a future PR |
||
|
Matthias Meschede
|
966cebec34
|
Adds polars list-contains command (#15304)
# Description
This PR adds the `polars list-contains` command. It works like this:
```
~/Projects/nushell/nushell> let df = [[a]; [[a,b,c]] [[b,c,d]] [[c,d,f]]] | polars into-df -s {a: list<str>};
~/Projects/nushell/nushell> $df | polars with-column [(polars col a | polars list-contains (polars lit a) | polars as b)] | polars collect
╭───┬───────────┬───────╮
│ # │ a │ b │
├───┼───────────┼───────┤
│ 0 │ ╭───┬───╮ │ true │
│ │ │ 0 │ a │ │ │
│ │ │ 1 │ b │ │ │
│ │ │ 2 │ c │ │ │
│ │ ╰───┴───╯ │ │
│ 1 │ ╭───┬───╮ │ false │
│ │ │ 0 │ b │ │ │
│ │ │ 1 │ c │ │ │
│ │ │ 2 │ d │ │ │
│ │ ╰───┴───╯ │ │
│ 2 │ ╭───┬───╮ │ false │
│ │ │ 0 │ c │ │ │
│ │ │ 1 │ d │ │ │
│ │ │ 2 │ f │ │ │
│ │ ╰───┴───╯ │ │
╰───┴───────────┴───────╯
```
or
```
~/Projects/nushell/nushell> let df = [[a, b]; [[a,b,c], a] [[b,c,d], f] [[c,d,f], f]] | polars into-df -s {a: list<str>, b: str}
~/Projects/nushell/nushell> $df | polars with-column [(polars col a | polars list-contains b | polars as c)] | polars collect
╭───┬───────────┬───┬───────╮
│ # │ a │ b │ c │
├───┼───────────┼───┼───────┤
│ 0 │ ╭───┬───╮ │ a │ true │
│ │ │ 0 │ a │ │ │ │
│ │ │ 1 │ b │ │ │ │
│ │ │ 2 │ c │ │ │ │
│ │ ╰───┴───╯ │ │ │
│ 1 │ ╭───┬───╮ │ f │ false │
│ │ │ 0 │ b │ │ │ │
│ │ │ 1 │ c │ │ │ │
│ │ │ 2 │ d │ │ │ │
│ │ ╰───┴───╯ │ │ │
│ 2 │ ╭───┬───╮ │ f │ true │
│ │ │ 0 │ c │ │ │ │
│ │ │ 1 │ d │ │ │ │
│ │ │ 2 │ f │ │ │ │
│ │ ╰───┴───╯ │ │ │
╰───┴───────────┴───┴───────╯
```
or
```
~/Projects/nushell/nushell> let df = [[a, b]; [[1,2,3], 4] [[2,4,1], 2] [[2,1,6], 3]] | polars into-df -s {a: list<i64>, b: i64}
~/Projects/nushell/nushell> $df | polars with-column [(polars col a | polars list-contains ((polars col b) * 2) | polars as c)] | polars collect
╭───┬───────────┬───┬───────╮
│ # │ a │ b │ c │
├───┼───────────┼───┼───────┤
│ 0 │ ╭───┬───╮ │ 4 │ false │
│ │ │ 0 │ 1 │ │ │ │
│ │ │ 1 │ 2 │ │ │ │
│ │ │ 2 │ 3 │ │ │ │
│ │ ╰───┴───╯ │ │ │
│ 1 │ ╭───┬───╮ │ 2 │ true │
│ │ │ 0 │ 2 │ │ │ │
│ │ │ 1 │ 4 │ │ │ │
│ │ │ 2 │ 1 │ │ │ │
│ │ ╰───┴───╯ │ │ │
│ 2 │ ╭───┬───╮ │ 3 │ true │
│ │ │ 0 │ 2 │ │ │ │
│ │ │ 1 │ 1 │ │ │ │
│ │ │ 2 │ 6 │ │ │ │
│ │ ╰───┴───╯ │ │ │
╰───┴───────────┴───┴───────╯
```
Let me know what you think. I'm a bit surprised that a list by default
seems to get converted to "object" when doing `into-df` which is why I
added the extra `-s` flag every time to explicitly force it into a list.
|
||
|
Jack Wright
|
e926919582
|
polars open: exposing the ability to configure hive settings. (#15255)
# Description Exposes parameters for working with [hive](https://docs.pola.rs/user-guide/io/hive/#scanning-hive-partitioned-data) partitioning. # User-Facing Changes - Added flags `--hive-enabled`, `--hive-start-idx`, `--hive-schema`, `--hive-try-parse-dates` to `polars open` |
||
|
Jack Wright
|
2dab65f852
|
Polars: Map pq extension to parquet files (#15284)
# Description Files with the extension pq will automatically be treated as parquet files. closes #15282 |
||
|
Matthias Meschede
|
087fe484f6
|
Enhance polars plugin documentation (#15250)
This PR (based on #15249 and #15248 because it mentions them) adds extra documentation to the main polars command outlining the main datatypes that are used by the plugin. The lack of a description of the types involved in `polars xxx` commands was quite confusing to me when I started using the plugin and this is a first try improving it. I didn't find a better place but please let me know what you think. |
||
|
Matthias Meschede
|
88bbe4abaa
|
Add Xor to polars plugin nu_expressions (#15249)
solution for #15242 , based on PR #15248 . Allows doing this: ``` ~/Projects/nushell> [[a, b]; [1., 2.], [3.,3.], [4., 6.]] | polars into-df | polars filter (((polars col a) < 2) xor ((polars col b) > 5)) ╭───┬──────┬──────╮ │ # │ a │ b │ ├───┼──────┼──────┤ │ 0 │ 1.00 │ 2.00 │ │ 1 │ 4.00 │ 6.00 │ ╰───┴──────┴──────╯ ``` |
||
|
Jack Wright
|
7939fb05ea
|
polars strip-chars: Allow any polars expression for pattern argument (#15178)
# Description Allow any polars expression for pattern argument for `polars strip-chars` |
||
|
Matthias Meschede
|
53d30ee7ea
|
add polars str strip chars (with --end / --start options) (#15118)
# Description This PR adds `polars str-strip-chars-end` # User-Facing Changes New function that can be used as follows: ``` ~/Projects/nushell> [[text]; [hello!!!] [world!!!]] | polars into-df | polars select (polars col text | polars str-strip-chars-end "!") | polars collect ╭───┬───────╮ │ # │ text │ ├───┼───────┤ │ 0 │ hello │ │ 1 │ world │ ╰───┴───────╯ ``` # Tests + Formatting tests ran locally. I ran the formatter. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |