We very well support `nth 0 2 3 --skip 1 4` to select particular rows and skip some using a flag. However, in practice we deal with tables (whether they come from parsing or loading files and whatnot) where we don't know the size of the table up front (and everytime we have these, they may have different sizes). There are also other use cases when we use intermediate tables during processing and wish to always drop certain rows and **keep the rest**.
Usage:
```
... | drop nth 0
... | drop nth 3 8
```
The nu-serde crate allows us to become much more generic with respect to how we
convert output to `nu-protocol::Value`s. This allows us to remove a lot of the
special-case code that we wrote for deserializing JSON values.
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
* Refactor Hash code to simplify md5 and sha256 implementations
Md5 and Sha256 (and other future digests) require less boilerplate code
now. Error reporting includues the name of the hash again.
* Add missing hash sha256 test
Select used to ignore absent values resulting in "squashing" where
columns for multiple rows could be squashed into a single row.
This fixes the problem by inserting null/nothing into absent value, thus
preserving the structure of rows. This makes sure that all selected
columns are present in each row.
Hashers now uses on Rust Crypto Digest trait which makes it trivial to
implement additional hash functions.
The original `md5` crate does not implement the Digest trait and was
replaced by `md-5` crate which does. Sha256 uses already included `sha2`
crate.
* nuframe in its own type in UntaggedValue
* Removed eager dataframe from enum
* Dataframe created from list of values
* Corrected order in dataframe columns
* Returned tag from stream collection
* Removed series from dataframe commands
* Arithmetic operators
* forced push
* forced push
* Replace all command
* String commands
* appending operations with dfs
* Testing suite for dataframes
* Unit test for dataframe commands
* improved equality for dataframes
* Read from standard input in `rm`
With this change, rm falls back to reading from the standard input if no
arguments are supplied. This leads to more intuitive pipes as seen in
https://github.com/nushell/nushell/issues/2824
ls | get name | sort-by | first 20 | each {rm $it} becomes
ls | get name | sort-by | first 20 | rm
* [Fix] Run cargo fmt, make files a rest parameter, and fix cargo build warnings
* [Fix] Fix clippy suggestions
* Fix swapped PATH env var separators
* Support pathvar to manipulate other vars than PATH
* Add tests for pathvar and its subcommands
* Adjust pathvar tests to comply with env quirks
* Make pathvar tests work on non-Windows as well
* Compact the comments in pathvar tests
* Fix last failing test
Co-authored-by: Jakub Žádník <jakub.zadnik@tuni.fi>
This allows converting strings to filepaths without having to use
`path expand` roundtrip.
Filepaths are taken as-is without any validation/conversion.
* Fix swapped PATH env var separators
* Support pathvar to manipulate other vars than PATH
* Add tests for pathvar and its subcommands
* Fix PATH env name for Windows
Seems like Windows uses PATH as well.
Co-authored-by: Jakub Žádník <jakub.zadnik@tuni.fi>
* kind of works but not what we really want
* updated `into binary` and `first` to work better together
* attempt to fix wasm build problem
* attempt #2 to fix wasm stuff
Nothing used the `ptree` feature or optional dependency within
`nu-command` except to include it within the `version` output. This
may be related to when `nu-cli` also had a `ptree` feature, but
I'm not sure.
That leaves the code within `nu_plugin_tree` as the sole remaining
user of `ptree`, which is already covered by the feature `tree`
and included in the `version` output.
* Type in command description
* filter name change
* Clean column name
* Clippy error and updated polars version
* Lint correction in file
* CSV Infer schema optional
* Correct float operations
* changes in series castings to allow other types
* Clippy error correction
* Removed lists from command signatures
* Added not command for series
* take command with args
* set with idx command
* Type in command description
* filter name change
* Clean column name
* Clippy error and updated polars version
* Lint correction in file
* CSV Infer schema optional
* Correct float operations
* changes in series castings to allow other types
* Clippy error correction
* Removed lists from command signatures
* Added not command for series
Seems we do `ctrl` feature checks in `nu-cli` and `nu-command`. We should find a better way to report the enabled features un the `version` command without using the conditionals (or somewhere else)
* Add paste command
* fix build and format failures
* Add examples
* Make tests pass
* Format
* add cfg annotation for Clip
* format code
* remove additional import for clip
* Remove test
* Type in command description
* filter name change
* Clean column name
* Clippy error and updated polars version
* Lint correction in file
* CSV Infer schema optional
We've relied on `clap` for building our cli app bootstrapping that figures out the positionals, flags, and other convenient facilities. Nu has been capable of solving this problem for quite some time. Given this and much more reasons (including the build time caused by `clap`) we start here working with our own.
* Fix and clarify description of 'exec'
Most importantly, I added the information that it replaces the current
process.
* Convert exec to OutputStream; Remove unused trait
* Remove dead code & unused imports on non-unix
* Sample command
* Join command with checks
* More dataframes commands
* Groupby and aggregate commands
* Missing feature dataframe flag
* Renamed file
* New commands for dataframes
* error parser and df reference
* filter command for dataframes
* removed name from nu_dataframe
* commands to save to parquet and csv
* polars new version
* new dataframe commands
* series type and print
* Series basic arithmetics
* Add new column to dataframe
* Command names changed to nushell standard
* Throw an error if path failed to expand
Previously, it just repeated the non-expanded path.
* Allow expanding non-existent paths
This commit has a strange error in examples.
* Specify span manually in examples; Add an example
* Expand relative path without requiring cwd
* Remove redundant tilde expansion
This makes the tilde expansion in relative paths dependant on "dirs"
feature.
* Add missing example result
* Adjust path expand description
* Fix import error with missing feature
* Sample command
* Join command with checks
* More dataframes commands
* Groupby and aggregate commands
* Missing feature dataframe flag
* Renamed file
* New commands for dataframes
* error parser and df reference
* filter command for dataframes
* removed name from nu_dataframe
* commands to save to parquet and csv
* Add the load-env command
load-env can be used to add environment variables dynamically via an
InputStream. This allows developers to create tools that output environment
variables as key-value pairs, then have the user load those variables in using
load-env. This supplants most of the need for an `eval` command, which is
mostly used in POSIX envs for setting env vars.
Fixes#3481
* fixup! Add the load-env command
* Allow querying the current path separator
* Convert char command to engine-p
* Wrap char args into struct
* Add --list option to char command
This lists all the available character names, along with the character
and its unicode points.
* commands to engine
* Correction of error in parser
* Added detailed regex error to parse
* better regex error parsing
* clippy corrections
* parse example with test
* secondary error for regex
* removed clone in error parser
* Secondary error message
The `date utc` command was removed in this PR:
https://github.com/nushell/nushell/pull/2780
The file was left but is no longer referenced from the parent module
and was not used.
Co-authored-by: Henrik Sjööh <henrik.sjooh@configura.com>
It was too error prone when positional arguments were used with the rest
arguments. Now, you need to explicitly state from which position you
want to count the rest args (e.g., `rest(0)`).
* Dataframe MVP
* Removed test csv file
* Dataframe MVP
* Removed test csv file
* New revision polars
* New revision polars
* csv file reader
* argument parser for file reader
* Parser from Row primitive
* Column conversion
* Added as f32 and f64
* Parsing row to dataframe
* Removed repeated push to vector
* Accept table values to create dataframe
* Removed default serde
* Dataframe to rows to show data
* Save name of file with dataframe
* Usage example
* Upgrade polars version
* Clippy changes
* Added print function with head and tail
* Move dataframe struct to folder
* Lock file after running tests and merge
* Optional feature for dataframe
* Removed dataframe from plugins
* Update primitive.rs
Co-authored-by: JT <jonathandturner@users.noreply.github.com>
* Convert "random bool" to engine-p
Also implements FromValue for Tagged<BigDecimal> and Tagged<f64>.
* Convert "random dice" to engine-p
* Convert "random uuid" to engine-p
* Covert "random chars" to engine-p
* Convert "random" command to engine-p
* Implement minmax for Range; Simplify range command
* Port random integer to enginep; New FromValue impl
Now, FromValue is implemented for Tagged<Range> to allow extracting args
into this type.
* Make sure range value extraction fails properly
The range endpoint extraction methods now return error instead of
silently clipping the value. This now makes `random integer ..-4` fail
properly since -4 can't be cast as u64.
* Port random decimal to enginep & Refactor
This added a way to interpret Range limits as f64 and a Primitive helper
to get its value as f64.
A side effect of this commit is that it is now possible to specify the
command bounds as true decimals. E.g., `random decimal 0.0..3.14` does
not clip 3.14 to 3.
* Removes arg serialization
* action stream -> output stream
* uses nu_protocol::Range instead of NumericRange
* random missing newline I found in the code
* add nu-pretty-hex, add into binary, update binaryview
* updated parameter name, updated examples
* fixed nu-pretty-hex test
* fixed tests again! and added a no color option to pretty-hex
It compiles and works without errors, but the pager is not asynchronous
anymore (i.e., you need to wait for `seq 1 1000000` to finish before the
pager is displayed).
* Add new path parse subcommand
This includes a slight refactor to all the path subcommand `action()`
functions.
* Remove filestem and extension; Fix example
* Add additional description to path parse
* Put join arg behind flag; Fix missing import (Win)
* Fix error when column path is passed as arg
* Add structured path joining
Structured path is implicitly joined at every patch subcommand call.
* Fix existing path join tests; Fix rustfmt
* Remove redundant 'static lifetime (clippy)
* Add initial impl of path split subcommand
* Add ability to join path from parts
* Fix wrong results in path split examples
* Fix remaining asyncs after engine change
* Do not wrap split path parts into table
When the input is just a list of values, the `path split` command will
split each value directly into the output stream, similar to
`split-row`. Column path--specified values are still wrapped into a
table so they can still be used to replace table fields.
* Join list of values instead of going one-by-one
When `path join` encounters a list of values, it attempts to join them,
instead of going one-by-one like the rest of the path commands. You can
still `each { echo $it | path join }` to join them one-by-one, if the
values are, e.g., tables.
Now, the behavior of `path split` and `path join` should match the
`split-row` and `str collect` counterparts and should hopefully align
better with user's expectations.
* Make sure path join detects structured path
* Fix panic on empty input stream
Also, doesn't collect input into vector unnecessarily.
* Fix path join not appending value
* Remove argument serialization
* Make better errors; Misc refactor
* OsStr -> String encoding is now lossy, instead of throwing an error
* The consequence is action() now always returns Value instead of Result
* Removed redundant handle_value() call in `path join`
* Fix possible incorrect error detection in `path split`
* Applied rustfmt + clippy
* Add more usage, examples & test; Fix type error
The 'parent' column was required to be a path but didn't work with
string.
* Add more help & examples; Maybe fix Windows error
* Refactor operate function
Reducing code repetition
* Review usages and examples
* Add the option to manually specify the extension
* Add more tests; Fix failures on Windows
* Move path commands to engine-p
* Small refactor
* Add iter-extensions
* Move all to enginep style
* Remove iter extensions
* Fix clippy lints
* Add comment and make ? more visible
* Remove try_all
* Remove all because it cant return err
* Output error when ls into a file without permission
* math sqrt
* added test to check fails when ls into prohibited dir
* fix lint
* math sqrt with tests and doc
* trigger wasm build
* Update filesystem_shell.rs
* converted math commands to outputstream and new method for arg evaluation
* fmt
* clippy
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
* Use ctx.configs in all config commands
* Remove all setting/accessing of vars.("config-path")
* Add tests
* Add comment
* Reload cfg on remove
* Hypocratic ws change
* Use history_path in hist_or_default
* Make clippy happy
* Fix rebase stuff
* Fix clippy lint
* add support for timestamp-based time conversion by specifing timezone or 'UTC/Local'
* [fix] fix the wrong test sample
* code formating
* code formating and import missing mod to test
* code formating again
* [fix] it won't crash when given timestamp is too big.
* [fix] code formatting =_=b
* commands: any? all?
We can check if `any` (or `all`) rows of tables match predicates.
Small `all?` example: Given the following table with `services` running:
```
> echo [[status]; [UP] [UP]]
───┬────────
# │ status
───┼────────
0 │ UP
1 │ UP
───┴────────
```
We can ask if all services are UP, like so:
```
> echo [[status]; [UP] [UP]] | all? status == UP
true
```
* Fix any? signature.
* Output error when ls into a file without permission
* math sqrt
* added test to check fails when ls into prohibited dir
* fix lint
* math sqrt with tests and doc
* trigger wasm build
* Update filesystem_shell.rs
* always forgetting the linting
* fix clippy complaining
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
* Revert "History, more test coverage improvements, and refactorings. (#3217)"
This reverts commit 8fc8fc89aa.
* Add tests
* Refactor .nu-env
* Change logic of Config write to logic of read()
* Fix reload always appends to old vars
* Fix reload always takes last_modified of global config
* Add reload_config in evaluation context
* Reload config after writing to it in cfg set / cfg set_into
* Add --no-history to cli options
* Use --no-history in tests
* Add comment about maybe_print_errors
* Get ctrl_exit var from context.global_config
* Use context.global_config in command "config"
* Add Readme in engine how env vars are now handled
* Update docs from autoenv command
* Move history_path from engine to nu_data
* Move load history out of if
* No let before return
* Add import for indexmap
Improvements overall to Nu. Also among the changes here, we can also be more confident towards incorporating `3041`. End to end tests for checking envs properly exported to externals is not added here (since it's in the other PR)
A few things added in this PR (probably forgetting some too)
* no writes happen to history during test runs.
* environment syncing end to end coverage added.
* clean up / refactorings few areas.
* testing API for finer control (can write tests passing more than one pipeline)
* can pass environment variables in tests that nu will inherit when running.
* No longer needed.
* no longer under a module. No need to use super.
* add support for timestamp-based time conversion by specifing timezone or 'UTC/Local'
* [fix] fix the wrong test sample
* code formating
* code formating and import missing mod to test
* code formating again
While the "main" separator in Windows is the backslash, it supports the
forward slash as a separator too.
Add support for this so that the behavior is similar to the way Windows
PowerShell handles the forward slash: it is recognized as a separator,
and when using <tab> for path completion the slash is reversed.
* Playground infraestructure (tests, etc) additions.
A few things to note:
* Nu can be started with a custom configuration file (`nu --config-file /path/to/sample_config.toml`). Useful for mocking the configuration on test runs.
* When given a custom configuration file Nu will save any changes to the file supplied appropiately.
* The `$nu.config-path` variable either shows the default configuration file (or the custom one, if given)
* We can now run end to end tests with finer grained control (currently, since this is baseline work, standard out) This will allow to check things like exit status, assert the contents with a format, etc)
* Remove (for another PR)
* Make opening a directory enter it.
Not sure if this change is wanted, but I'm not sure what else opening a directory could mean.
And I find myself accidentally using `open <dir>` to mean `enter <dir>`
* Add example to open directory
* Open dir should list it's contents
* Update example description and fix style
* update docs to refer to length instead of count
* rename count to length
* change all occurrences of 'count' to 'length' in tests
* format length command
* Move run_script to engine
* Add which dep and feature to engine
* Change unwrap to expect
* Add wasm specification
* Remove which from default, add specification correctly
* Add nu-platform-specifics
* Move is_external_cmd to platform_specifics
* Add is_external_cmd to host and use it instead of nu_platform directly
* Clean up if else logic in is_external_cmd
* Bump nu-platform-specifics version
* Pass context to print_err
* Commit cargo.lock
* Move print functions to own module inside nu-engine
* Hypocratic change to run windows-nightly again
* Add import for Ordering
* Move printing of error to host
* Move platform specific which functionality to basic host
* Allow no use of cmd_name
* Fix windows compile issue
* Split help message into brief and full help
Demonstrate on ansi command
Brief help is printed when running `help commands` so it doesn't clutter
the table. Full help is printed when normal help message is requested
(e.g., `help ansi`, `ansi --help`, etc.).
* Split long command descriptions
Some are not split, just edited to be shorter.
* Capitalize the usage of all commands
* Make sure every usage ends with dot
* Fix random typo
* use the InputHandler functionality from minus
* respond to Q and ESC character to quit
* use arijit79/minus main branch until new release is pushed
* rename NushellMinusInputHandler to MinusInputHandler
* Refactor path subcommand argument handling
DefaultArguments are no longer passed to each subcommand. Instead, each
subcommand has its own Path<xxx>Arguments. This means that it is no
longer necessary to edit every single path subcommand source file when
changing the arguments struct.
* Add new path join subcommand
Makes it easier to create new paths. It's just a wrapper around Rust's
Path.join().
* This adds table paging, relying on minus to perform the paging functionality
This is gated behind the table-pager feature
* fix problem with long running InputStreams blocking table() returning
* some comments regarding Arc clones, and callback from minus
`drop` is used for removing the last row. Passing a number allows dropping N rows.
Here we introduce the same logic for dropping columns instead.
You can certainly remove columns by using `reject`, however, there could be cases
where we are interested in removing columns from tables that contain, say, a big
number of columns. Using `reject` becomes impractical, especially when you don't
care about the column names that could either be known or not known when exploring
tables.
```
> echo [[lib, extension]; [nu-core, rs] [rake, rb]]
─────────┬───────────
lib │ extension
─────────┼───────────
nu-core │ rs
rake │ rb
─────────┴───────────
```
```
> echo [[lib, extension]; [nu-core, rs] [rake, rb]] | drop column
─────────
lib
─────────
nu-core
rake
─────────
```
There are many use cases. Here we introduce the following:
- The rows can be rolled `... | roll` (up) or `... | roll down`
- Columns can be rolled too (the default is on the `left`, you can pass `... | roll column --opposite` to roll in the other direction)
- You can `roll` the cells of a table and keeping the header names in the same order (`... | roll column --cells-only`)
- Above examples can also be passed (Ex. `... | roll down 3`) a number to tell how many places to roll.
Basic working example with rolling columns:
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| rename bit1 bit2 bit3 bit4 bit5 bit6 bit7 bit8
───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────
# │ bit1 │ bit2 │ bit3 │ bit4 │ bit5 │ bit6 │ bit7 │ bit8
───┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────
0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 1 │ 0 │ 0
───┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────
```
We want to "shift" three bits to the left of the bitstring (four in decimal), let's try it:
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| rename bit1 bit2 bit3 bit4 bit5 bit6 bit7 bit8
| roll column 3
───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────
# │ bit4 │ bit5 │ bit6 │ bit7 │ bit8 │ bit1 │ bit2 │ bit3
───┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────
0 │ 0 │ 0 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0
───┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────
```
The tables was rolled correctly (32 in decimal, for above bitstring). However, the *last three header names* look confusing.
We can roll the cell contents only to fix it.
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| rename bit1 bit2 bit3 bit4 bit5 bit6 bit7 bit8
| roll column 3 --cells-only
───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────
# │ bit1 │ bit2 │ bit3 │ bit4 │ bit5 │ bit6 │ bit7 │ bit8
───┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────
0 │ 0 │ 0 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0
───┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────
```
There we go. Let's compute it's decimal value now (should be 32)
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| roll column 3 --cells-only
| pivot bit --ignore-titles
| get bit
| reverse
| each --numbered { = $it.item * (2 ** $it.index) }
| math sum
32
```
The autoenv logic mutates environment variables in the running session as
it operates and decides what to do for trusted directories containing `.nu-env`
files. Few of the ways to interact with it were all in a single test function.
We separate out all the ways that were done in the single test function to document
it better. This will greatly help once we start refactoring our way out from setting
environment variables this way to just setting them to `Scope`.
This is part of an on-going effort to keep variables (`PATH` and `ENV`)
in our `Scope` and rely on it for everything related to variables.
We expect to move away from setting (`std::*`) envrironment variables in the current
running process. This is non-trivial since we need to handle cases from vars
coming in from the outside world, prioritize, and also compare to the ones
we have both stored in memory and in configuration files.
Also to send out our in-memory (in `Scope`) variables properly to external
programs once we no longer rely on `std::env` vars from the running process.
* Use expand_path to handle the path including tilda
* Publish path::expand_path for using in nu-command
* cargo fmt
Co-authored-by: Wataru Yamaguchi <nagisamark2@gmail.com>
* Document the lexer and lightly improve its names
The bulk of this pull request adds a substantial amount of new inline
documentation for the lexer. Along the way, I made a few minor changes
to the names in the lexer, most of which were internal.
The main change that affects other files is renaming `group` to `block`,
since the function is actually parsing a block (a list of groups).
* Further clean up the lexer
- Consolidate the logic of the various token builders into a single type
- Improve and clean up the event-driven BlockParser
- Clean up comment parsing. Comments now contain their original leading
whitespace as well as trailing whitespace, and know how to move some
leading whitespace back into the body based on how the lexer decides
to dedent the comments. This preserves the original whitespace
information while still making it straight-forward to eliminate leading
whitespace in help comments.
* Update meta.rs
* WIP
* fix clippy
* remove unwraps
* remove unwraps
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
Co-authored-by: Jonathan Turner <jonathan.d.turner@gmail.com>
* standardize on how to get file size
* forgot to remove comment
* make specified size lowercase
* fix the test due to precision
* added another test
* Update README.md
add contributors graphic
* clippy - test adjustment
* tweaked matching
* move commands, futures.rs, script.rs, utils
* move over maybe_print_errors
* add nu_command crate references to nu_cli
* in commands.rs open up to pub mod from pub(crate)
* nu-cli, nu-command, and nu tests are now passing
* cargo fmt
* clean up nu-cli/src/prelude.rs
* code cleanup
* for some reason lex.rs was not formatted, may be causing my error
* remove mod completion from lib.rs which was not being used along with quickcheck macros
* add in allow unused imports
* comment out one failing external test; comment out one failing internal test
* revert commenting out failing tests; something else might be going on; someone with a windows machine should check and see what is going on with these failing windows tests
* Update Cargo.toml
Extend the optional features to nu-command
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
