Provides support for reading Polars structs. This allows opening of
supported files (jsonl, parquet, etc) that contain rows with structured
data.
The following attached json lines
file([receipts.jsonl.gz](https://github.com/nushell/nushell/files/13311476/receipts.jsonl.gz))
contains a customer column with structured data. This json lines file
can now be loaded via `dfr open` and will render as follows:
<img width="525" alt="Screenshot 2023-11-09 at 10 09 18"
src="https://github.com/nushell/nushell/assets/56345/4b26ccdc-c230-43ae-a8d5-8af88a1b72de">
This also addresses some cleanup of date handling and utilizing
timezones where provided.
This pull request only addresses reading data from polars structs. I
will address converting nushell data to polars structs in a future
request as this change is large enough as it is.
---------
Co-authored-by: Jack Wright <jack.wright@disqo.com>
# Description
Open question:
Undocumented behavior for the new argument `ambiguous` to the
`as_datetime`
methods. I cheated by passing a default (assuming empty string).
This appears like an API primarily serving the python impl:
https://pola-rs.github.io/polars/py-polars/html/reference/expressions/api/polars.Expr.str.to_datetime.html#polars-expr-str-to-datetime
# User-Facing Changes
Only dependent on breaking changes to the behavior of polars.
# Tests + Formatting
No observed changes to tests
Manually checked `dfr as-datetime`, doesn't seem to panic.
Same logic as in #9971
Prevents building the heavy polars and arrow dependencies when just
running `cargo test --workspace` or `rust-analyzer`
`polars-io` dependency was introduced in #10019
Polars and SQLParser upgrade.
I have exposed features that have been added to polars as command args
where appropriate.
---------
Co-authored-by: Jack Wright <jack.wright@disqo.com>
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
Co-authored-by: sholderbach <sholderbach@users.noreply.github.com>
# Description
This PR bumps nushell from release version 0.84.0 to dev version 0.84.1.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
Context: https://github.com/serde-rs/serde/issues/2538
As other projects are investigating, this should pin serde to the last
stable release before binary requirements were introduced.
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
Context from Discord:
https://discord.com/channels/601130461678272522/615962413203718156/1138694933545504819
I was working on Nu for the first time in a while and I noticed that
sometimes rust-analyzer takes a really long time to run `cargo check` on
the entire workspace. I dug in and it was checking a bunch of
dataframe-related dependencies even though the `dataframe` feature is
not built by default.
It looks like this is a regression of sorts, introduced by
https://github.com/nushell/nushell/pull/9241. Thankfully the fix is
pretty easy, we can make it so everything important in
`nu-cmd-dataframe` is only used when the `dataframe` feature is enabled.
### Impact on `cargo check --workspace`
Before this PR: 635 crates, 33.59s
After this PR: 498 crates, ~20s
(with the `mold` linker and a `cargo clean` before each run, the
relative difference for incremental checks will likely be much larger)
# Description
This bumps nushell to the dev version of 0.83.1 and updates the default
config files with the proper version.
# User-Facing Changes
# Tests + Formatting
# After Submitting
# Description
Bump 0.83
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
Apart from `polars` (only used with `--features dataframe`) and the
dev-dependencies our deps use `indexmap 2.0`.
Thus the default or `extra` `cargo build` will reduce deps.
This also will help deduplicating `hashbrown` and `ahash`.
For #8060
- Bump `indexmap` to 2.0
- Remove unneeded `serde` feature from `indexmap`
# User-Facing Changes
None
# Description
This PR updates the sqlparser dep to 0.34.0.
closes#9525
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Checklist
- `nu-ansi-term` remains the same
- [x] `reedline` is released and updated
- [x] release scripts are updated for `nu-cmd-base`
- [x] info blog post is online
- [ ] release notes are ready
This PR reverts https://github.com/nushell/nushell/pull/9391
We try not to revert PRs like this, though after discussion with the
Nushell team, we decided to revert this one.
The main reason is that Nushell, as a codebase, isn't ready for these
kinds of optimisations. It's in the part of the development cycle where
our main focus should be on improving the algorithms inside of Nushell
itself. Once we have matured our algorithms, then we can look for
opportunities to switch out technologies we're using for alternate
forms.
Much of Nushell still has lots of opportunities for tuning the codebase,
paying down technical debt, and making the codebase generally cleaner
and more robust. This should be the focus. Performance improvements
should flow out of that work.
Said another, optimisation that isn't part of tuning the codebase is
premature at this stage. We need to focus on doing the hard work of
making the engine, parser, etc better.
# User-Facing Changes
Reverts the HashMap -> ahash change.
cc @FilipAndersson245
# Description
Make sure that our different crates that contain commands can be
compiled in parallel.
This can under certain circumstances accelerate the compilation with
sufficient multithreading available.
## Details
- Move `help` commands from `nu-cmd-lang` back to `nu-command`
- This also makes sense as the commands are implemented in an
ANSI-terminal specific way
- Make `nu-cmd-lang` only a dev dependency for `nu-command`
- Change context creation helpers for `nu-cmd-extra` and
`nu-cmd-dataframe` to have a consistent api used in
`src/main.rs`:`get_engine_state()`
- `nu-command` now indepedent from `nu-cmd-extra` and `nu-cmd-dataframe`
that are now dependencies of `nu` directly. (change to internal
features)
- Fix tests that previously used `nu-command::create_default_context()`
with replacement functions
## From scratch compilation times:
just debug (dev) build and default features
```
cargo clean --profile dev && cargo build --timings
```
### before
### after
# User-Facing Changes
None direct, only change to compilation on multithreaded jobs expected.
# Tests + Formatting
Tests that previously chose to use `nu-command` for their scope will
still use `nu-cmd-lang` + `nu-command` (command list in the granularity
at the time)
# Description
Update the polars dependency for dataframes to stay on-version.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
see https://github.com/nushell/nushell/issues/9390
using `ahash` instead of the default hasher. this will not affect
compile time as we where already building `ahash`.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
This PR updates most dependencies and tries to get in sync with
reedline.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
All of the dataframe commands ported over with no issues...
### 11 tests are commented out (for now)
So 100 of the original 111 tests are passing with only 11 tests being
ignored for now..
As per our conversation in the core team meeting on Wednesday
I took @jntrnr suggestion and just commented out the tests dealing
with
[IntoDatetime](https://github.com/nushell/nushell/blob/main/crates/nu-command/src/conversions/into/mod.rs)
Later on we can move this functionality out of nu-command if we decide
it makes sense...
### The following tests were ignored...
```rust
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_day.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_hour.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_minute.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_month.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_nanosecond.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_ordinal.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_second.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_week.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_weekday.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/date/get_year.rs
modified: crates/nu-cmd-dataframe/src/dataframe/series/string/strftime.rs
```
