74 Commits

Author SHA1 Message Date
Jack Wright
23ba613b00
Polars AWS S3 support (#14648)
# Description

Provides Amazon S3 support.

- Utilizes your existing AWS cli configuration. 
- Supports AWS SSO
- Supports
[gimme-aws-creds](https://github.com/Nike-Inc/gimme-aws-creds).
- respects the settings of AWS_PROFILE environment variable for
selecting profile config
- AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION environment
variables for configuring without an AWS config

Usage:
```nushell
polars open s3://bucket/and/path.parquet
```

Supports:
- CSV
- Parquet
- NDJSON / json lines
- Arrow

Doesn't support:
- eager dataframes
-  Avro
- JSON
2024-12-25 06:15:50 -06:00
Justin Ma
039d0a685a
Fix the document CI error for polars profile command (#14642)
# Description

Fix the docs repo CI build error here:
https://github.com/nushell/nushell.github.io/actions/runs/12425087184/job/34691291790#step:5:18

The doc generated by `make_docs.nu` for `polars profile` command will
make the CI build fail due to the indention error of markdown front
matters. I used to fix it manually before, for the long run, it's better
to fix it from the source code.
2024-12-20 13:47:02 +01:00
Jack Wright
981a000ee8
Added flag --coalesce-columns to allow columns to be coalsced on full joins (#14578)
- fixes #14572

# Description
This allowed columns to be coalesced on full joins with `polars join`,
providing functionality simlar to the old `--outer` join behavior.

# User-Facing Changes
- Provides a new flag `--coalesce-columns` on the `polars join` command
2024-12-17 09:55:42 -08:00
Darren Schroeder
baf86dfb0e
tweak polars join for better cross joins (#14586)
# Description

closes #14585

This PR tries to make `polars join --cross` work better. Example taken
from
https://docs.pola.rs/user-guide/transformations/joins/#cartesian-product

### Before
```nushell
❯ let tokens = [[monopoly_token]; [hat] [shoe] [boat]] | polars into-df
❯ let players = [[name, cash]; [Alice, 78] [Bob, 135]] | polars into-df
❯ $players | polars into-lazy | polars select (polars col name) | polars join --cross $tokens | polars collect
Error: nu::parser::missing_positional

  × Missing required positional argument.
   ╭─[entry #3:1:92]
 1 │ $players | polars into-lazy | polars select (polars col name) | polars join --cross $tokens
   ╰────
  help: Usage: polars join {flags} <other> <left_on> <right_on> . Use `--help` for more information.
```
### After
```nushell
❯ let players = [[name, cash]; [Alice, 78] [Bob, 135]] | polars into-df
❯ let tokens = [[monopoly_token]; [hat] [shoe] [boat]] | polars into-df
❯ $players | polars into-lazy | polars select (polars col name) | polars join --cross $tokens | polars collect
╭─#─┬─name──┬─monopoly_token─╮
│ 0 │ Alice │ hat            │
│ 1 │ Alice │ shoe           │
│ 2 │ Alice │ boat           │
│ 3 │ Bob   │ hat            │
│ 4 │ Bob   │ shoe           │
│ 5 │ Bob   │ boat           │
╰─#─┴─name──┴─monopoly_token─╯
```
Other examples
```nushell
❯ 1..3 | polars into-df | polars join --cross (4..6 | polars into-df)
╭─#─┬─0─┬─0_x─╮
│ 0 │ 1 │   4 │
│ 1 │ 1 │   5 │
│ 2 │ 1 │   6 │
│ 3 │ 2 │   4 │
│ 4 │ 2 │   5 │
│ 5 │ 2 │   6 │
│ 6 │ 3 │   4 │
│ 7 │ 3 │   5 │
│ 8 │ 3 │   6 │
╰─#─┴─0─┴─0_x─╯
❯ 1..3 | each {|x| {x: $x}} | polars into-df | polars join --cross (4..6 | each {|y| {y: $y}} | polars into-df) x y
╭─#─┬─x─┬─y─╮
│ 0 │ 1 │ 4 │
│ 1 │ 1 │ 5 │
│ 2 │ 1 │ 6 │
│ 3 │ 2 │ 4 │
│ 4 │ 2 │ 5 │
│ 5 │ 2 │ 6 │
│ 6 │ 3 │ 4 │
│ 7 │ 3 │ 5 │
│ 8 │ 3 │ 6 │
╰─#─┴─x─┴─y─╯
```
/cc @ayax79 
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-12-14 21:58:47 -06:00
Jack Wright
81d68cd478
Documentation and error handling around polars with-column --name (#14527)
The `--name` flag of `polars with-column` only works when used with an
eager dataframe. I will not work with lazy dataframes and it will not
work when used with expressions (which forces a conversion to a
lazyframe). This pull request adds better documentation to the flags and
errors messages when used in cases where it will not work.
2024-12-06 05:17:18 -06:00
Jack Wright
0172ad8461
Upgrading to polars 0.44 (#14478)
Upgrading to polars 0.44
2024-11-29 19:39:07 -06:00
Jan Klass
6c36bd822c
Fix doc and code comment typos (#14366)
# User-Facing Changes

* Fixes `polars value-counts --column` help text typo
* Fixes `polars agg-groups` help text typo
2024-11-17 19:17:35 +01:00
Jack Wright
ae54d05930
Upgrade to polars 0.43 (#14148)
Upgrades the polars plugin to polars version 0.43
2024-10-23 19:14:24 +02:00
Solomon
b0427ca9ff run ensure_flag_arg_type for short flag values (#14074)
Closes #13654

# User-Facing Changes

- Short flags are now fully type-checked,
  including null and record signatures for literal arguments:

```nushell
def test [-v: record<l: int>] {};
test -v null # error
test -v {l: ""} # error

def test2 [-v: int] {};
let v = ""
test2 -v $v # error
```

- `polars unpivot` `--index`/`--on` and `into value --columns`
now accept `list` values
2024-10-20 23:12:57 +02:00
Jack Wright
f787d272e6 Implemented polars unnest (#14104)
# Description
Provides the ability to decomes struct columns into seperate columns for
each field:
<img width="655" alt="Screenshot 2024-10-16 at 09 57 22"
src="https://github.com/user-attachments/assets/6706bd36-8d38-4365-b58d-ba82f2d5ba9a">

# User-Facing Changes
- provides a new command `polars unnest` for decomposing struct fields
into separate columns.
2024-10-20 23:12:57 +02:00
Ian Manske
28b6db115a
Revert PRs for 0.99.1 patch (#14119)
# Description

Temporarily reverts PRs merged after the 0.99.1 bump.
2024-10-18 02:51:14 +00:00
Solomon
2eef42c6b9
run ensure_flag_arg_type for short flag values (#14074)
Closes #13654

# User-Facing Changes

- Short flags are now fully type-checked,
  including null and record signatures for literal arguments:

```nushell
def test [-v: record<l: int>] {};
test -v null # error
test -v {l: ""} # error

def test2 [-v: int] {};
let v = ""
test2 -v $v # error
```

- `polars unpivot` `--index`/`--on` and `into value --columns`
now accept `list` values
2024-10-16 21:25:17 -05:00
Jack Wright
c9d54f821b
Implemented polars unnest (#14104)
# Description
Provides the ability to decomes struct columns into seperate columns for
each field:
<img width="655" alt="Screenshot 2024-10-16 at 09 57 22"
src="https://github.com/user-attachments/assets/6706bd36-8d38-4365-b58d-ba82f2d5ba9a">

# User-Facing Changes
- provides a new command `polars unnest` for decomposing struct fields
into separate columns.
2024-10-16 21:24:14 -05:00
Jack Wright
1d6ac16530
polars into-df struct fix (#13977)
# Description
This fixes an issue with converting to a dataframe when specifying a
struct in the schema. Things like the following now work correctly:
```nushell
 [[foo bar]; [{a: "a_0", b:"b_0"} 1] [{a: "a_1", b: "b_1" } 2]] | polars into-df -s {foo: {a: str, b: str}, bar: u8}
```
2024-10-02 05:59:14 -05:00
Jack Wright
bcaef8959c
Expose flag truncate-ragged-lines in polars open (#13939)
# Description
Introduces a new flag `--truncate-ragged-lines` for `polars open` that
will truncate lines that are longer than the schema.

# User-Facing Changes
- Introduction of the flag `--truncate-ragged-lines` for `polars open`
2024-09-27 06:54:46 -05:00
Jack Wright
5bef81a059
Added command polars len for performing count(*) like operations. (#13941)
# Description
This request exposes the prelude::polars::len expression. It is ended
for doing fast select count(*) like operations:

<img width="626" alt="Screenshot 2024-09-26 at 18 14 45"
src="https://github.com/user-attachments/assets/74285fc6-f99c-46e0-9226-9a7d41738d78">

# User-Facing Changes
- Introduction of the `polars len` command
2024-09-27 06:54:28 -05:00
Jack Wright
28a7461057
Added command polars profile for profiling lazy dataframes (#13904)
# Description
Introduce a new command `polars profile` for profiling lazy dataframes:

<img width="965" alt="Screenshot 2024-09-22 at 23 46 18"
src="https://github.com/user-attachments/assets/11402dd3-8256-43df-a986-64241c15354f">

# User-Facing Changes
- Introduces new command `polars profile`
2024-09-23 07:57:59 -05:00
Jack Wright
2541a712e4
Added polars concat to allow concatenation of multiple dataframes (#13879)
# Description
Provides the ability to concatenate multiple dataframes together

# User-Facing Changes
- Introduces new command `polars concat`
2024-09-23 06:43:43 -05:00
Jack Wright
af77bc60e2
Improved null handling when converting from nu -> dataframe. (#13855)
# Description
Fixes: #12726 and #13185

Previously converting columns that contained null caused polars to force
a dtype of object even when using a schema.

Now:
1. When using a schema, the type the schema defines for the column will
always be used.
2. When a schema is not used, the previous type is used when a value is
null.

# User-Facing Changes
- The type defined by the schema we be respected when passing in a null
value `[a]; [null] | polars into-df -s {a: str}` will create a df with
an str dtype column with one null value versus a column of type object.
- *BREAKING CHANGE* If you define a schema, all columns must be in the
schema.
2024-09-16 18:07:13 -05:00
Jack Wright
fb34a4fc6c
Make polars save return an empty pipeline (#13833)
# Description
In order to be more consistent with it's nu counterpart, `polars save`
now returns an empty pipeline instead of a message of the saved file.

# User-Facing Changes
- `polars save` no longer displays a save message, making it consistent
with `save` behavior.
2024-09-12 06:23:40 -05:00
Jack Wright
8d60c0d35d
Migrating polars commands away from macros, removed custom DataFrame comparison. (#13829)
# Description
This PR:
- Removes the lazy_command, expr_command macros and migrates the
commands that were utilizing them.
- Removes the custom logic in DataFrameValues::is_equals to use the
polars DataFrame version of PartialEq
- Adds examples to commands that previously did not have examples or had
inadequate ones.

NOTE: A lot of examples now have a `polars sort` at the end. This is
needed due to the comparison in the result. The new polars version of
equals cares about the ordering. I removed the custom equals logic as it
causes comparisons to lock up when comparing dataframes that contain a
row that contains a list. I discovered this issue when adding examples
to `polars implode`
2024-09-11 10:33:05 -07:00
Jack Wright
1e64f59220
Added documentation explanation explaining how to select all columns with polars col (#13806)
# Description
Previously there were no examples or explanations that you can use '*'
to select all columns. Updated description and added a new example.
2024-09-08 17:12:03 +00:00
Jack Wright
6c1c7f9509
Added expression support for polars cumulative (#13799)
# Description
Provides the ability to use `polars cumulative` as an expression:

<img width="1266" alt="Screenshot 2024-09-06 at 17 47 15"
src="https://github.com/user-attachments/assets/73c11f79-598c-4efa-bfcd-755e536ead66">

# User-Facing Changes
- `polars cumulative` can now be used as an expression.
2024-09-06 22:03:51 -05:00
Jack Wright
f531cc2058
Polars command reorg (#13798)
# Description
House keeping. Restructures polars modules as discussed in:
https://docs.google.com/spreadsheets/d/1gyA58i_yTXKCJ5DbO_RxBNAlK6S7C1M22ppKwVLZltc/edit?usp=sharing
2024-09-06 13:46:37 -07:00