Commit Graph

9 Commits

Author SHA1 Message Date
b9dd47ebb7 Polars 0.38 upgrade (#12506)
# Description
Polars 0.38 upgrade for both the dataframe crate and the polars plugin.

---------

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-13 13:00:04 -05:00
211d9c685c Fix clippy lint (#12504)
Just fixes a clippy lint.
2024-04-13 16:19:32 +00:00
1bded8572c Ensure that two columns named index don't exist when converting a Dataframe to a nu Value. (#12501)
# Description
@maxim-uvarov discovered an issue with the current implementation. When
executing [[index a]; [1 1]] | polars into-df, a plugin_failed_to_decode
error occurs. This happens because a Record is created with two columns
named "index" as an index column is added during conversion. This pull
request addresses the problem by not adding an index column if there is
already a column named "index" in the dataframe.

---------

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-13 06:33:29 -05:00
f975c9923a Handle relative paths correctly on polars to-(parquet|jsonl|arrow|etc) commands (#12486)
# Description

All polars commands that output a file were not handling relative paths
correctly.

A command like
``` [[a b]; [6 2] [1 4] [4 1]] | polars into-df | polars to-parquet foo.json``` 
was outputting the foo.json to the directory of the plugin executable. 

This pull request pulls in nu-path and using it for resolving the file paths.

Related discussion
https://discord.com/channels/601130461678272522/1227612017171501136/1227889870358183966

# User-Facing Changes
None

# Tests + Formatting
Done, added tests for each of the polars to-* commands.

---------

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-12 19:30:37 -05:00
50fb8243c8 Added a short flag -c to polars append --col (#12487)
# Description
`dfr append --col` had a short version -c. This polar requests adds the
short flag back.

Reference Conversation:
https://discord.com/channels/601130461678272522/1227612017171501136/1227902980628676688

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-12 10:55:36 -05:00
b9c2f9ee56 displaying span information, creation time, and size with polars ls (#12472)
# Description
`polars ls` is already different that `dfr ls`. Currently it just shows
the cache key, columns, rows, and type. I have added:
- creation time
- size
- span contents
-  span start and end

<img width="1471" alt="Screenshot 2024-04-10 at 17 27 06"
src="https://github.com/nushell/nushell/assets/56345/545918b7-7c96-4c25-bc01-b9e2b659a408">

# Tests + Formatting
Done

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-12 09:23:46 -05:00
872945ae8e Bump version to 0.92.3 (#12476) 2024-04-12 08:00:43 -05:00
81c61f3243 Showing full help when running the polars command (#12462)
Displays the full help message for all sub commands.

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-10 07:26:33 -05:00
efc1cfa939 Move dataframes support to a plugin (#12220)
WIP

This PR covers migration crates/nu-cmd-dataframes to a new plugin
./crates/nu_plugin_polars

## TODO List

Other:
- [X] Fix examples
- [x] Fix Plugin Test Harness
- [X] Move Cache to Mutex<BTreeMap>
- [X] Logic for disabling/enabling plugin GC based off whether items are
cached.
- [x] NuExpression custom values
- [X] Optimize caching (don't cache every object creation). 
- [x] Fix dataframe operations (in NuDataFrameCustomValue::operations)
- [x] Added plugin_debug! macro that for checking an env variable
POLARS_PLUGIN_DEBUG

Fix duplicated commands:
- [x] There are two polars median commands, one for lazy and one for
expr.. there should only be one that works for both. I temporarily
called on polars expr-median (inside expressions_macros.rs)
- [x] polars quantile (lazy, and expr). the expr one is temporarily
expr-median
- [x] polars is-in (renamed one series-is-in)

Commands:
- [x] AppendDF
- [x] CastDF
- [X] ColumnsDF
- [x] DataTypes
- [x] Summary
- [x] DropDF
- [x] DropDuplicates
- [x] DropNulls
- [x] Dummies
- [x] FilterWith
- [X] FirstDF
- [x] GetDF
- [x] LastDF
- [X] ListDF
- [x] MeltDF
- [X] OpenDataFrame
- [x] QueryDf
- [x] RenameDF
- [x] SampleDF
- [x] SchemaDF
- [x] ShapeDF
- [x] SliceDF
- [x] TakeDF
- [X] ToArrow
- [x] ToAvro
- [X] ToCSV
- [X] ToDataFrame
- [X] ToNu
- [x] ToParquet
- [x] ToJsonLines
- [x] WithColumn
- [x] ExprAlias
- [x] ExprArgWhere
- [x] ExprCol
- [x] ExprConcatStr
- [x] ExprCount
- [x] ExprLit
- [x] ExprWhen
- [x] ExprOtherwise
- [x] ExprQuantile
- [x] ExprList
- [x] ExprAggGroups
- [x] ExprCount
- [x] ExprIsIn
- [x] ExprNot
- [x] ExprMax
- [x] ExprMin
- [x] ExprSum
- [x] ExprMean
- [x] ExprMedian
- [x] ExprStd
- [x] ExprVar
- [x] ExprDatePart
- [X] LazyAggregate
- [x] LazyCache
- [X] LazyCollect
- [x] LazyFetch
- [x] LazyFillNA
- [x] LazyFillNull
- [x] LazyFilter
- [x] LazyJoin
- [x] LazyQuantile
- [x] LazyMedian
- [x] LazyReverse
- [x] LazySelect
- [x] LazySortBy
- [x] ToLazyFrame
- [x] ToLazyGroupBy
- [x] LazyExplode
- [x] LazyFlatten
- [x] AllFalse
- [x] AllTrue
- [x] ArgMax
- [x] ArgMin
- [x] ArgSort
- [x] ArgTrue
- [x] ArgUnique
- [x] AsDate
- [x] AsDateTime
- [x] Concatenate
- [x] Contains
- [x] Cumulative
- [x] GetDay
- [x] GetHour
- [x] GetMinute
- [x] GetMonth
- [x] GetNanosecond
- [x] GetOrdinal
- [x] GetSecond
- [x] GetWeek
- [x] GetWeekDay
- [x] GetYear
- [x] IsDuplicated
- [x] IsIn
- [x] IsNotNull
- [x] IsNull
- [x] IsUnique
- [x] NNull
- [x] NUnique
- [x] NotSeries
- [x] Replace
- [x] ReplaceAll
- [x] Rolling
- [x] SetSeries
- [x] SetWithIndex
- [x] Shift
- [x] StrLengths
- [x] StrSlice
- [x] StrFTime
- [x] ToLowerCase
- [x] ToUpperCase
- [x] Unique
- [x] ValueCount

---------

Co-authored-by: Jack Wright <jack.wright@disqo.com>
2024-04-09 19:31:43 -05:00