* A first working version of flatten. Needs a lot of cleanup. Committing to have a working version
* Typo fix
* Flatten tests pass
* Final cleanup, ready for push
* Final cleanup, ready for push
* Final cleanup, ready for push
* Final cleanup, ready for push
* Update flatten.rs
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
* Proof of concept treating env vars as Values
* Refactor env var collection and method name
* Remove unnecessary pub
* Move env translations into a new file
* Fix LS_COLORS to support any Value
* Fix spans during env var translation
* Add span to env var in cd
* Improve error diagnostics
* Fix non-string env vars failing string conversion
* Make PROMPT_COMMAND a Block instead of String
* Record host env vars to a fake file
This will give spans to env vars that would otherwise be without one.
Makes errors less confusing.
* Add 'env' command to list env vars
It will list also their values translated to strings
* Sort env command by name; Add env var type
* Remove obsolete test
* Porting 'ansi' command from nushell to engine-q
* Added StrCollect to example_test.rs to allow example tests to run
* Run 'cargo fmt' to fix formatting
* Update command.rs
* Update command.rs
* Update command.rs
* Added a category
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
I removed the Inflector dependency in favor of heck for two reasons:
- to close#3674.
- heck seems simpler and actively maintained
We could probably alter the structure of the `str_` module to expose the
individual casing behaviors better.
I did not feel as confident on changing those signatures.
So I took a lazier approach of a macro in the `mod.rs` that creates the public
shimming function to heck's traits.
* First iteration of the version command
* Cleanup
* Fix the installed plugins bug
* Fix fmt check issue
* Fix clippy warning
* Fixing all clippy warnings
* Remove old code
* Sort default context items categorically
* Separate commands in multiple statements
* Use curly braces instead of square brackets
This prevents undesired reformatting.
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* First draft of these commands
* To MD
* To md and to html
* Fixed cargo and to_md
* `into_abbreviated_string` instead of `into_string`
* Changed how inner tables are displayed
* port over the nth command from nushell
* remove a line of redundant code
* must sort the rows or else if the rows are not from low to high this crashes engine-q
* Port str to-decimal to into decimal command. Add also a Value::test_float function for tests only
* Add support for handling integers into decimals and fix issues with error span
* Remember environment variables from previous scope
* Re-introduce env var hiding
Right now, hiding decls is broken
* Re-introduce hidden field of import patterns
All tests pass now.
* Remove/Address tests TODOs
* Fix test typo; Report hiding error
* Add a few more tests
* Fix wrong expected test result
* add icons to grid output. still needs cleanup
* working but adds a dependency on ansi_term - need to fix that
* update styling, added lots of green code to icons
* clippy
* add config point for grid icons
* fix#4140
We are passing commands into a shell underneath but we were not
escaping arguments correctly. This new version of the code also takes
into consideration the ";" and "&" characters, which have special
meaning in shells.
We would probably benefit from a more robust way to join arguments to
shell programs. Python's stdlib has shlex.join, and perhaps we can
take that implementation as a reference.
* clean up escaping of posix shell args
I believe the right place to do escaping of arguments was in the
spawn_sh_command function. Note that this change prevents things like:
^echo "$(ls)"
from executing the ls command. Instead, this will just print
$(ls)
The regex has been taken from the python stdlib implementation of shlex.quote
* fix non-literal parameters and single quotes
* address clippy's comments
* fixup! address clippy's comments
* test that subshell commands are sanitized properly
* option to replace command same name
* moved order of custom value declarations
* arranged dataframe folders and objects
* sort help commands by name
* added dtypes function for debugging
* corrected name for dataframe commands
* command names using function
* have save --append create file if not exists
Currently, doing:
echo a | save --raw --append file.txt
will fail if file.txt does not exist. This PR changes that
* test that `save --append` will create new file
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* `from toml` command
* From ods
* From XLSX
* From ics
* From ini
* From vcf
* Forgot a eprintln!
* custom value trait
* functions for custom value trait
* custom trait behind flag
* open dataframe command
* command to-df for basic types
* follow path for dataframe
* dataframe operations
* dataframe not default feature
* custom as default feature
* corrected examples in command
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* `from toml` command
* From ods
* From XLSX
It's no longer attached to a Block. Makes access to overlays more
streamlined by removing this one indirection. Also makes it easier to
create standalone overlays without a block which might come in handy.
* Add 'expor env' dummy command
* (WIP) Abstract away module exportables as Overlay
* Switch to Overlays for use/hide
Works for decls only right now.
* Fix passing import patterns of hide to eval
* Simplify use/hide of decls
* Add ImportPattern as Expr; Add use env eval
Still no parsing of "export env" so I can't test it yet.
* Refactor export parsing; Add InternalError
* Add env var export and activation; Misc changes
Now it is possible to `use` env var that was exported from a module.
This commit also adds some new errors and other small changes.
* Add env var hiding
* Fix eval not recognizing hidden decls
Without this change, calling `hide foo`, the evaluator does not know
whether a custom command named "foo" was hidden during parsing,
therefore, it is not possible to reliably throw an error about the "foo"
name not found.
* Add use/hide/export env var tests; Cleanup; Notes
* Ignore hide env related tests for now
* Fix main branch merge mess
* Fixed multi-word export def
* Fix hiding tests on Windows
* Remove env var hiding for now
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* FromEml and FromUrl
Added tests for from eml
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* `from yaml` and `from yml`
`from yaml` and `from yml`
from yaml and from yml
* Fix collect_string
* Fix tests and linting
* Port str contains command
* Add another test case / example for str contains
* Port str downcase to engine-q
Co-authored-by: Stefan Stanciulescu <contact@stefanstanciulescu.com>
* initial commit of reverse
* reverse is working, now move on to the examples
* add in working examples for reverse
* #[allow(clippy::needless_collect)]
* Port camel case and kebab case
* Port pascal case
* Port snake case and screaming snake case
* Cleanup before PR
* Add back cell path support for str case commands
* Add cell path tests for str case command
* Revert "Add cell path tests for str case command"
This reverts commit a0906318d95fd2b5e4f8ca42f547a7e4c5db381a.
* Add cell path test cases for str case command
* Move cell path tests from tests.rs to Examples in each of the command's file
Co-authored-by: Stefan Stanciulescu <contact@stefanstanciulescu.com>
Now, Echo converts multiple values in a ValueStream, but it simply
forwards a single Value; if no PipelineData is detected as an input, an
empty string is returned as a single Value.
Values to echo need to be extracted from the call, and then converted
into PipelineData.
I also updated the first example so that its result is a List,
as in the reference implementation.
See cc3653cfd9 for more on the `-c` flag.
Co-authored-by: Andrés N. Robalino <andres@androbtech.com>
Co-authored-by: Andrés N. Robalino <andres@androbtech.com>
* compiles on nightly now. (breaking change)
* less deps
* Switch over to new resolver
(it's been stable for a while.)
* let's leave num-format for another PR
Very often we need to work with tables (say extracted from unstructured data or some
kind of final report, timeseries, and the like).
It's inevitable we will be having columns that we can't know beforehand what their names
will be, or how many.
Also, we may end up with certain cells having values we may want to remove as we explore.
Here, `update cells` fundamentally goes over every cell in the table coming in and updates
the cell's contents with the output of the block passed. Basic example here:
```
> [
[ ty1, t2, ty];
[ 1, a, $nothing]
[(wrap), (0..<10), 1Mb]
[ 1s, ({}), 1000000]
[ $true, $false, ([[]])]
] | update cells { describe }
───┬───────────────────────┬───────────────────────────┬──────────
# │ ty1 │ t2 │ ty
───┼───────────────────────┼───────────────────────────┼──────────
0 │ integer │ string │ nothing
1 │ row Column(table of ) │ range[[integer, integer)] │ filesize
2 │ string │ nothing │ integer
3 │ boolean │ boolean │ table of
───┴───────────────────────┴───────────────────────────┴──────────
```
and another one (in the examples) for cases, say we have a timeseries table generated and
we want to remove the zeros and have empty strings and save it out to something like CSV.
```
> [
[2021-04-16, 2021-06-10, 2021-09-18, 2021-10-15, 2021-11-16, 2021-11-17, 2021-11-18];
[ 37, 0, 0, 0, 37, 0, 0]
] | update cells {|value| i
if ($value | into int) == 0 {
""
} {
$value
}
}
───┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────
# │ 2021-04-16 │ 2021-06-10 │ 2021-09-18 │ 2021-10-15 │ 2021-11-16 │ 2021-11-17 │ 2021-11-18
───┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────
0 │ 37 │ │ │ │ 37 │ │
───┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────
```
* Change path join signature
* Appending now works without flag
* Column path operation is behind a -c flag
* Move column path arg retrieval to a function
Also improves errors
* Fix path join tests
* Propagate column path changes to all path commands
* Update path command examples with columns paths
* Modernize path command examples by removing "echo"
* Improve structured path error message
* Fix typo
* Expand path when converting value -> PathBuf
Also includes Tagged<PathBuf>.
Fixes#3605
* Expand path for PATH env. variable
Fixes#1834
* Remove leftover Cows after nu-path refactor
There were some unnecessary Cow conversions leftover from the old
nu-path implementation.
* Use canonicalize in source command; Improve errors
Previously, `source` used `expand_path()` which does not follow
symlinks.
As a follow up, I improved the source error messages so they now tell
why the source file could not be canonicalized or read into string.
```
> [
[ msg, labels, span];
["The message", "Helpful message here", ([[start, end]; [0, 141]])]
] | error make
error: The message
┌─ shell:1:1
│
1 │ ╭ [
2 │ │ [ msg, labels, span];
3 │ │ ["The message", "Helpful message here", ([[start, end]; [0, 141]])]
│ ╰─────────────────────────────────────────────────────────────────────^ Helpful message here
```
Adding a more flexible approach for creating error values. One use case, for instance is the
idea of a test framework. A failed assertion instead of printing to the screen it could create
tables with more details of the failed assertion and pass it to this command for making a full
fledge error that Nu can show. This can (and should) be extended for capturing error values as well
in the pipeline. One could also use it for inspection.
For example: `.... | error inspect { # inspection here }`
or "error handling" as well, like so: `.... | error capture { fix here }`
However, we start here only with `error make` that creates an error value for you with limited support for the time being.
* Add subcommand `into filesize`
It's currently not possible to convert a number or a string containing a number
into a filesize. The only way to create an instance of filesize type today is
with a literal in nushell syntax. This commit adds the `into filesize`
subcommand so that file sizes can be created from the outputs of programs
producing numbers or strings, like standard unix tools.
There is a limitation with this - it doesn't currently parse values like `10 MB`
or `10 MiB`, it can only look at the number itself. If the desire is there, more
flexible parsing can be added.
* fixup! Add subcommand `into filesize`
* fixup! Add subcommand `into filesize`
* feat: spawn the executables directly if possible
This pull request changes nu-command so that it spawns the process directly if:
- They are a `.exe` on Windows
- They are not a `.sh` or `.bash` on not windows.
Benefits:
- As I explained in [this comment](https://github.com/nushell/nushell/issues/3898#issuecomment-894000812), this is another step towards making Nushell a standalone shell, that doesn't need to shell out unless it is running a script for a particular shell (cmd, sh, ps1, etc.).
- Fixes the bug with multiline strings
- Better performance due to direct spawning.
For example, this script shows ~20 ms less latency.
After:
```nu
C:\> benchmark { node -e 'console.log("sssss")' }
───┬──────────────────
# │ real time
───┼──────────────────
0 │ 63ms 921us 600ns
───┴──────────────────
```
Before
```nu
C:\> benchmark { node -e 'console.log("sssss")' }
───┬──────────────────
# │ real time
───┼──────────────────
0 │ 79ms 136us 800ns
───┴──────────────────
```
Fixes#3898
* fix: make which dependency optional
Also fixes clippy warnings
* refactor: refactor spawn_exe, spawn_cmd, spawn_sh, and spawn_any
* fix: use which feature instead of which-support
* fix: use which_in to use the cwd of nu
* fix: use case insensitive comparison of the extensions
Sometimes the case of the extension is uppercased by the "which_in" function
Also use unix instead of not windows. Some os might not have sh support
* Resolve rebase artifacts
* Remove leftover dependencies on removed feature
* Remove unnecessary 'pub'
* Start taking notes and fooling around
* Split canonicalize to two versions; Add TODOs
One that takes `relative_to` and one that doesn't.
More TODO notes.
* Merge absolutize to and rename resolve_dots
* Add custom absolutize fn and use it in path expand
* Convert a couple of dunce::canonicalize to ours
* Update nu-path description
* Replace all canonicalize with nu-path version
* Remove leftover dunce dependencies
* Fix broken autocd with trailing slash
Trailing slash is preserved *only* in paths that do not contain "." or
"..". This should be fixed in the future to cover all paths but for now
it at least covers basic cases.
* Use dunce::canonicalize for canonicalizing
* Alow cd recovery from non-existent cwd
* Disable removed canonicalize functionality tests
Remove unused import
* Break down nu-path into separate modules
* Remove unused public imports
* Remove abundant cow mapping
* Fix clippy warning
* Reformulate old canonicalize tests to expand_path
They wouldn't work with the new canonicalize.
* Canonicalize also ~ and ndots; Unify path joining
Also, add doc comments in nu_path::expansions.
* Add comment
* Avoid expanding ndots if path is not valid UTF-8
With this change, no lossy path->string conversion should happen in the
nu-path crate.
* Fmt
* Slight expand_tilde refactor; Add doc comments
* Start nu-path integration tests
* Add tests TODO
* Fix docstring typo
* Fix some doc strings
* Add README for nu-path crate
* Add a couple of canonicalize tests
* Add nu-path integration tests
* Add trim trailing slashes tests
* Update nu-path dependency
* Remove unused import
* Regenerate lockfile
* Allow different names for ...rest
* Resolves#3945
* This change requires an explicit name for the rest argument in `WholeStreamCommand`,
which is why there are so many changed files.
* Remove redundant clone
* Add tests
* Allow environment variables to be hidden
This change allows environment variables in Nushell to have a value of
`Nothing`, which can be set by the user by passing `$nothing` to
`let-env` and friends.
Environment variables with a value of Nothing behave as if they are not
set at all. This allows a user to shadow the value of an environment
variable in a parent scope, effectively removing it from their current
scope. This was not possible before, because a scope can not affect its
parent scopes.
This is a workaround for issues like #3920.
Additionally, this allows a user to simultaneously set, change and
remove multiple environment variables via `load-env`. Any environment
variables set to $nothing will be hidden and thus act as if they are
removed. This simplifies working with virtual environments, which rely
on setting multiple environment variables, including PATH, to specific
values, and remove/change them on deactivation.
One surprising behavior is that an environment variable set to $nothing
will act as if it is not set when querying it (via $nu.env.X), but it is
still possible to remove it entirely via `unlet-env`. If the same
environment variable is present in the parent scope, the value in the
parent scope will be visible to the user. This might be surprising
behavior to users who are not familiar with the implementation details.
An additional corner case is the the shorthand form of `with-env` does
not work with this feature. Using `X=$nothing` will set $nu.env.X to the
string "$nothing". The long-form works as expected: `with-env [X
$nothing] {...}`.
* Remove unused import
* Allow all primitives to be convert to strings
Given we can write nu scripts. As the codebase grows, splitting into many smaller nu scripts is necessary.
In general, when we work with paths and files we seem to face quite a few difficulties. Here we just tackle one of them and it involves sourcing
files that also source other nu files and so forth. The current working directory becomes important here and being on a different directory
when sourcing scripts will not work. Mostly because we expand the path on the current working directory and parse the files when a source command
call is done.
For the moment, we introduce a `lib_dirs` configuration value and, unfortunately, introduce a new dependency in `nu-parser` (`nu-data`) to get
a handle of the configuration file to retrieve it. This should give clues and ideas as the new parser engine continues (introduce a way to also know paths)
With this PR we can do the following:
Let's assume we want to write a nu library called `my_library`. We will have the code in a directory called `project`: The file structure will looks like this:
```
project/my_library.nu
project/my_library/hello.nu
project/my_library/name.nu
```
This "pattern" works well, that is, when creating a library have a directory named `my_library` and next to it a `my_library.nu` file. Filling them like this:
```
source my_library/hello.nu
source my_library/name.nu
```
```
def hello [] {
"hello world"
}
```
```
def name [] {
"Nu"
end
```
Assuming this `project` directory is stored at `/path/to/lib/project`, we can do:
```
config set lib_dirs ['path/to/lib/project']
```
Given we have this `lib_dirs` configuration value, we can be anywhere while using Nu and do the following:
```
source my_library.nu
echo (hello) (name)
```
* Allow sourcing paths with emojis
* Add source command tests for emoji paths
* Fmt
* Disable source tests on Windows with illegal paths
* Test sourcing also ASCII and single-quoted paths
We introduce it here and allow it to work with regular lists (tables with no columns) as well as symmetric tables. Say we have two lists and wish to zip them, like so:
```
[0 2 4 6 8] | zip {
[1 3 5 7 9]
} | flatten
───┬───
0 │ 0
1 │ 1
2 │ 2
3 │ 3
4 │ 4
5 │ 5
6 │ 6
7 │ 7
8 │ 8
9 │ 9
───┴───
```
In the case for two tables instead:
```
[[symbol]; ['('] ['['] ['{']] | zip {
[[symbol]; [')'] [']'] ['}']]
} | each {
get symbol | $'($in.0)nushell($in.1)'
}
───┬───────────
0 │ (nushell)
1 │ [nushell]
2 │ {nushell}
───┴───────────
```
We very well support `nth 0 2 3 --skip 1 4` to select particular rows and skip some using a flag. However, in practice we deal with tables (whether they come from parsing or loading files and whatnot) where we don't know the size of the table up front (and everytime we have these, they may have different sizes). There are also other use cases when we use intermediate tables during processing and wish to always drop certain rows and **keep the rest**.
Usage:
```
... | drop nth 0
... | drop nth 3 8
```
The nu-serde crate allows us to become much more generic with respect to how we
convert output to `nu-protocol::Value`s. This allows us to remove a lot of the
special-case code that we wrote for deserializing JSON values.
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
* Refactor Hash code to simplify md5 and sha256 implementations
Md5 and Sha256 (and other future digests) require less boilerplate code
now. Error reporting includues the name of the hash again.
* Add missing hash sha256 test
Select used to ignore absent values resulting in "squashing" where
columns for multiple rows could be squashed into a single row.
This fixes the problem by inserting null/nothing into absent value, thus
preserving the structure of rows. This makes sure that all selected
columns are present in each row.
* nuframe in its own type in UntaggedValue
* Removed eager dataframe from enum
* Dataframe created from list of values
* Corrected order in dataframe columns
* Returned tag from stream collection
* Removed series from dataframe commands
* Arithmetic operators
* forced push
* forced push
* Replace all command
* String commands
* appending operations with dfs
* Testing suite for dataframes
* Unit test for dataframe commands
* improved equality for dataframes
* moving all dataframe operations to protocol
Hashers now uses on Rust Crypto Digest trait which makes it trivial to
implement additional hash functions.
The original `md5` crate does not implement the Digest trait and was
replaced by `md-5` crate which does. Sha256 uses already included `sha2`
crate.
* nuframe in its own type in UntaggedValue
* Removed eager dataframe from enum
* Dataframe created from list of values
* Corrected order in dataframe columns
* Returned tag from stream collection
* Removed series from dataframe commands
* Arithmetic operators
* forced push
* forced push
* Replace all command
* String commands
* appending operations with dfs
* Testing suite for dataframes
* Unit test for dataframe commands
* improved equality for dataframes
* Read from standard input in `rm`
With this change, rm falls back to reading from the standard input if no
arguments are supplied. This leads to more intuitive pipes as seen in
https://github.com/nushell/nushell/issues/2824
ls | get name | sort-by | first 20 | each {rm $it} becomes
ls | get name | sort-by | first 20 | rm
* [Fix] Run cargo fmt, make files a rest parameter, and fix cargo build warnings
* [Fix] Fix clippy suggestions
* Fix swapped PATH env var separators
* Support pathvar to manipulate other vars than PATH
* Add tests for pathvar and its subcommands
* Adjust pathvar tests to comply with env quirks
* Make pathvar tests work on non-Windows as well
* Compact the comments in pathvar tests
* Fix last failing test
Co-authored-by: Jakub Žádník <jakub.zadnik@tuni.fi>
This allows converting strings to filepaths without having to use
`path expand` roundtrip.
Filepaths are taken as-is without any validation/conversion.
* Fix swapped PATH env var separators
* Support pathvar to manipulate other vars than PATH
* Add tests for pathvar and its subcommands
* Fix PATH env name for Windows
Seems like Windows uses PATH as well.
Co-authored-by: Jakub Žádník <jakub.zadnik@tuni.fi>
* kind of works but not what we really want
* updated `into binary` and `first` to work better together
* attempt to fix wasm build problem
* attempt #2 to fix wasm stuff
Nothing used the `ptree` feature or optional dependency within
`nu-command` except to include it within the `version` output. This
may be related to when `nu-cli` also had a `ptree` feature, but
I'm not sure.
That leaves the code within `nu_plugin_tree` as the sole remaining
user of `ptree`, which is already covered by the feature `tree`
and included in the `version` output.
* Type in command description
* filter name change
* Clean column name
* Clippy error and updated polars version
* Lint correction in file
* CSV Infer schema optional
* Correct float operations
* changes in series castings to allow other types
* Clippy error correction
* Removed lists from command signatures
* Added not command for series
* take command with args
* set with idx command
* Type in command description
* filter name change
* Clean column name
* Clippy error and updated polars version
* Lint correction in file
* CSV Infer schema optional
* Correct float operations
* changes in series castings to allow other types
* Clippy error correction
* Removed lists from command signatures
* Added not command for series
Seems we do `ctrl` feature checks in `nu-cli` and `nu-command`. We should find a better way to report the enabled features un the `version` command without using the conditionals (or somewhere else)
* Add paste command
* fix build and format failures
* Add examples
* Make tests pass
* Format
* add cfg annotation for Clip
* format code
* remove additional import for clip
* Remove test
* Type in command description
* filter name change
* Clean column name
* Clippy error and updated polars version
* Lint correction in file
* CSV Infer schema optional
* Output error when ls into a file without permission
* math sqrt
* added test to check fails when ls into prohibited dir
* fix lint
* math sqrt with tests and doc
* trigger wasm build
* Update filesystem_shell.rs
* Fix Running echo .. starts printing integers forever
* Fixed panic on operations with very large durations
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
We've relied on `clap` for building our cli app bootstrapping that figures out the positionals, flags, and other convenient facilities. Nu has been capable of solving this problem for quite some time. Given this and much more reasons (including the build time caused by `clap`) we start here working with our own.
* Fix and clarify description of 'exec'
Most importantly, I added the information that it replaces the current
process.
* Convert exec to OutputStream; Remove unused trait
* Remove dead code & unused imports on non-unix
* Sample command
* Join command with checks
* More dataframes commands
* Groupby and aggregate commands
* Missing feature dataframe flag
* Renamed file
* New commands for dataframes
* error parser and df reference
* filter command for dataframes
* removed name from nu_dataframe
* commands to save to parquet and csv
* polars new version
* new dataframe commands
* series type and print
* Series basic arithmetics
* Add new column to dataframe
* Command names changed to nushell standard
* Throw an error if path failed to expand
Previously, it just repeated the non-expanded path.
* Allow expanding non-existent paths
This commit has a strange error in examples.
* Specify span manually in examples; Add an example
* Expand relative path without requiring cwd
* Remove redundant tilde expansion
This makes the tilde expansion in relative paths dependant on "dirs"
feature.
* Add missing example result
* Adjust path expand description
* Fix import error with missing feature
* Sample command
* Join command with checks
* More dataframes commands
* Groupby and aggregate commands
* Missing feature dataframe flag
* Renamed file
* New commands for dataframes
* error parser and df reference
* filter command for dataframes
* removed name from nu_dataframe
* commands to save to parquet and csv
Using the `*` wildcard should not attempt to delete files with a leading dot
unless the more explicit `.*` is used. `rm *` should also not attempt to delete
the current directory or its parent directory (`.` and `..`). I have resolved
this bug as well in a less satisfactory way. I think it may be the case that we
can only disambiguate the `.` and `..` path segments by using `Path::display`.
Here is a short list of alternatives that I tried:
- `Path::ends_with()` can detect `/..` but not `/.`.
- `Path::iter()` and `Path::components()` leave out `/.`.
- `Path::file_name()` normalizes `/.` to the parent component's file name.
Fixes#3508
* Add the load-env command
load-env can be used to add environment variables dynamically via an
InputStream. This allows developers to create tools that output environment
variables as key-value pairs, then have the user load those variables in using
load-env. This supplants most of the need for an `eval` command, which is
mostly used in POSIX envs for setting env vars.
Fixes#3481
* fixup! Add the load-env command
* Allow querying the current path separator
* Convert char command to engine-p
* Wrap char args into struct
* Add --list option to char command
This lists all the available character names, along with the character
and its unicode points.
* commands to engine
* Correction of error in parser
* Added detailed regex error to parse
* better regex error parsing
* clippy corrections
* parse example with test
* secondary error for regex
* removed clone in error parser
* Secondary error message
The `date utc` command was removed in this PR:
https://github.com/nushell/nushell/pull/2780
The file was left but is no longer referenced from the parent module
and was not used.
Co-authored-by: Henrik Sjööh <henrik.sjooh@configura.com>
It was too error prone when positional arguments were used with the rest
arguments. Now, you need to explicitly state from which position you
want to count the rest args (e.g., `rest(0)`).
* Dataframe MVP
* Removed test csv file
* Dataframe MVP
* Removed test csv file
* New revision polars
* New revision polars
* csv file reader
* argument parser for file reader
* Parser from Row primitive
* Column conversion
* Added as f32 and f64
* Parsing row to dataframe
* Removed repeated push to vector
* Accept table values to create dataframe
* Removed default serde
* Dataframe to rows to show data
* Save name of file with dataframe
* Usage example
* Upgrade polars version
* Clippy changes
* Added print function with head and tail
* Move dataframe struct to folder
* Lock file after running tests and merge
* Optional feature for dataframe
* Removed dataframe from plugins
* Update primitive.rs
Co-authored-by: JT <jonathandturner@users.noreply.github.com>
* Convert "random bool" to engine-p
Also implements FromValue for Tagged<BigDecimal> and Tagged<f64>.
* Convert "random dice" to engine-p
* Convert "random uuid" to engine-p
* Covert "random chars" to engine-p
* Convert "random" command to engine-p
* Implement minmax for Range; Simplify range command
* Port random integer to enginep; New FromValue impl
Now, FromValue is implemented for Tagged<Range> to allow extracting args
into this type.
* Make sure range value extraction fails properly
The range endpoint extraction methods now return error instead of
silently clipping the value. This now makes `random integer ..-4` fail
properly since -4 can't be cast as u64.
* Port random decimal to enginep & Refactor
This added a way to interpret Range limits as f64 and a Primitive helper
to get its value as f64.
A side effect of this commit is that it is now possible to specify the
command bounds as true decimals. E.g., `random decimal 0.0..3.14` does
not clip 3.14 to 3.
* Removes arg serialization
* action stream -> output stream
* uses nu_protocol::Range instead of NumericRange
* random missing newline I found in the code
* add nu-pretty-hex, add into binary, update binaryview
* updated parameter name, updated examples
* fixed nu-pretty-hex test
* fixed tests again! and added a no color option to pretty-hex
It compiles and works without errors, but the pager is not asynchronous
anymore (i.e., you need to wait for `seq 1 1000000` to finish before the
pager is displayed).
* Add new path parse subcommand
This includes a slight refactor to all the path subcommand `action()`
functions.
* Remove filestem and extension; Fix example
* Add additional description to path parse
* Put join arg behind flag; Fix missing import (Win)
* Fix error when column path is passed as arg
* Add structured path joining
Structured path is implicitly joined at every patch subcommand call.
* Fix existing path join tests; Fix rustfmt
* Remove redundant 'static lifetime (clippy)
* Add initial impl of path split subcommand
* Add ability to join path from parts
* Fix wrong results in path split examples
* Fix remaining asyncs after engine change
* Do not wrap split path parts into table
When the input is just a list of values, the `path split` command will
split each value directly into the output stream, similar to
`split-row`. Column path--specified values are still wrapped into a
table so they can still be used to replace table fields.
* Join list of values instead of going one-by-one
When `path join` encounters a list of values, it attempts to join them,
instead of going one-by-one like the rest of the path commands. You can
still `each { echo $it | path join }` to join them one-by-one, if the
values are, e.g., tables.
Now, the behavior of `path split` and `path join` should match the
`split-row` and `str collect` counterparts and should hopefully align
better with user's expectations.
* Make sure path join detects structured path
* Fix panic on empty input stream
Also, doesn't collect input into vector unnecessarily.
* Fix path join not appending value
* Remove argument serialization
* Make better errors; Misc refactor
* OsStr -> String encoding is now lossy, instead of throwing an error
* The consequence is action() now always returns Value instead of Result
* Removed redundant handle_value() call in `path join`
* Fix possible incorrect error detection in `path split`
* Applied rustfmt + clippy
* Add more usage, examples & test; Fix type error
The 'parent' column was required to be a path but didn't work with
string.
* Add more help & examples; Maybe fix Windows error
* Refactor operate function
Reducing code repetition
* Review usages and examples
* Add the option to manually specify the extension
* Add more tests; Fix failures on Windows
* Move path commands to engine-p
* Small refactor
* Add iter-extensions
* Move all to enginep style
* Remove iter extensions
* Fix clippy lints
* Add comment and make ? more visible
* Remove try_all
* Remove all because it cant return err
* Output error when ls into a file without permission
* math sqrt
* added test to check fails when ls into prohibited dir
* fix lint
* math sqrt with tests and doc
* trigger wasm build
* Update filesystem_shell.rs
* converted math commands to outputstream and new method for arg evaluation
* fmt
* clippy
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
* Use ctx.configs in all config commands
* Remove all setting/accessing of vars.("config-path")
* Add tests
* Add comment
* Reload cfg on remove
* Hypocratic ws change
* Use history_path in hist_or_default
* Make clippy happy
* Fix rebase stuff
* Fix clippy lint
* add support for timestamp-based time conversion by specifing timezone or 'UTC/Local'
* [fix] fix the wrong test sample
* code formating
* code formating and import missing mod to test
* code formating again
* [fix] it won't crash when given timestamp is too big.
* [fix] code formatting =_=b
* commands: any? all?
We can check if `any` (or `all`) rows of tables match predicates.
Small `all?` example: Given the following table with `services` running:
```
> echo [[status]; [UP] [UP]]
───┬────────
# │ status
───┼────────
0 │ UP
1 │ UP
───┴────────
```
We can ask if all services are UP, like so:
```
> echo [[status]; [UP] [UP]] | all? status == UP
true
```
* Fix any? signature.
* Output error when ls into a file without permission
* math sqrt
* added test to check fails when ls into prohibited dir
* fix lint
* math sqrt with tests and doc
* trigger wasm build
* Update filesystem_shell.rs
* always forgetting the linting
* fix clippy complaining
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
* Output error when ls into a file without permission
* added test to check fails when ls into prohibited dir
* fix lint
* trigger wasm build
* Update filesystem_shell.rs
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
* Revert "History, more test coverage improvements, and refactorings. (#3217)"
This reverts commit 8fc8fc89aa.
* Add tests
* Refactor .nu-env
* Change logic of Config write to logic of read()
* Fix reload always appends to old vars
* Fix reload always takes last_modified of global config
* Add reload_config in evaluation context
* Reload config after writing to it in cfg set / cfg set_into
* Add --no-history to cli options
* Use --no-history in tests
* Add comment about maybe_print_errors
* Get ctrl_exit var from context.global_config
* Use context.global_config in command "config"
* Add Readme in engine how env vars are now handled
* Update docs from autoenv command
* Move history_path from engine to nu_data
* Move load history out of if
* No let before return
* Add import for indexmap
Improvements overall to Nu. Also among the changes here, we can also be more confident towards incorporating `3041`. End to end tests for checking envs properly exported to externals is not added here (since it's in the other PR)
A few things added in this PR (probably forgetting some too)
* no writes happen to history during test runs.
* environment syncing end to end coverage added.
* clean up / refactorings few areas.
* testing API for finer control (can write tests passing more than one pipeline)
* can pass environment variables in tests that nu will inherit when running.
* No longer needed.
* no longer under a module. No need to use super.
* add support for timestamp-based time conversion by specifing timezone or 'UTC/Local'
* [fix] fix the wrong test sample
* code formating
* code formating and import missing mod to test
* code formating again
While the "main" separator in Windows is the backslash, it supports the
forward slash as a separator too.
Add support for this so that the behavior is similar to the way Windows
PowerShell handles the forward slash: it is recognized as a separator,
and when using <tab> for path completion the slash is reversed.
* Playground infraestructure (tests, etc) additions.
A few things to note:
* Nu can be started with a custom configuration file (`nu --config-file /path/to/sample_config.toml`). Useful for mocking the configuration on test runs.
* When given a custom configuration file Nu will save any changes to the file supplied appropiately.
* The `$nu.config-path` variable either shows the default configuration file (or the custom one, if given)
* We can now run end to end tests with finer grained control (currently, since this is baseline work, standard out) This will allow to check things like exit status, assert the contents with a format, etc)
* Remove (for another PR)
* Make opening a directory enter it.
Not sure if this change is wanted, but I'm not sure what else opening a directory could mean.
And I find myself accidentally using `open <dir>` to mean `enter <dir>`
* Add example to open directory
* Open dir should list it's contents
* Update example description and fix style
* update docs to refer to length instead of count
* rename count to length
* change all occurrences of 'count' to 'length' in tests
* format length command
* Move run_script to engine
* Add which dep and feature to engine
* Change unwrap to expect
* Add wasm specification
* Remove which from default, add specification correctly
* Add nu-platform-specifics
* Move is_external_cmd to platform_specifics
* Add is_external_cmd to host and use it instead of nu_platform directly
* Clean up if else logic in is_external_cmd
* Bump nu-platform-specifics version
* Pass context to print_err
* Commit cargo.lock
* Move print functions to own module inside nu-engine
* Hypocratic change to run windows-nightly again
* Add import for Ordering
* Move printing of error to host
* Move platform specific which functionality to basic host
* Allow no use of cmd_name
* Fix windows compile issue
* Split help message into brief and full help
Demonstrate on ansi command
Brief help is printed when running `help commands` so it doesn't clutter
the table. Full help is printed when normal help message is requested
(e.g., `help ansi`, `ansi --help`, etc.).
* Split long command descriptions
Some are not split, just edited to be shorter.
* Capitalize the usage of all commands
* Make sure every usage ends with dot
* Fix random typo
* use the InputHandler functionality from minus
* respond to Q and ESC character to quit
* use arijit79/minus main branch until new release is pushed
* rename NushellMinusInputHandler to MinusInputHandler
* Refactor path subcommand argument handling
DefaultArguments are no longer passed to each subcommand. Instead, each
subcommand has its own Path<xxx>Arguments. This means that it is no
longer necessary to edit every single path subcommand source file when
changing the arguments struct.
* Add new path join subcommand
Makes it easier to create new paths. It's just a wrapper around Rust's
Path.join().
* This adds table paging, relying on minus to perform the paging functionality
This is gated behind the table-pager feature
* fix problem with long running InputStreams blocking table() returning
* some comments regarding Arc clones, and callback from minus
* fix case where parent_name was {nu, term} and possibly others in the future by doing an extra test first to see if if the *parent_name key actually exists in cmap
* update with help generate_docs testing
`drop` is used for removing the last row. Passing a number allows dropping N rows.
Here we introduce the same logic for dropping columns instead.
You can certainly remove columns by using `reject`, however, there could be cases
where we are interested in removing columns from tables that contain, say, a big
number of columns. Using `reject` becomes impractical, especially when you don't
care about the column names that could either be known or not known when exploring
tables.
```
> echo [[lib, extension]; [nu-core, rs] [rake, rb]]
─────────┬───────────
lib │ extension
─────────┼───────────
nu-core │ rs
rake │ rb
─────────┴───────────
```
```
> echo [[lib, extension]; [nu-core, rs] [rake, rb]] | drop column
─────────
lib
─────────
nu-core
rake
─────────
```
There are many use cases. Here we introduce the following:
- The rows can be rolled `... | roll` (up) or `... | roll down`
- Columns can be rolled too (the default is on the `left`, you can pass `... | roll column --opposite` to roll in the other direction)
- You can `roll` the cells of a table and keeping the header names in the same order (`... | roll column --cells-only`)
- Above examples can also be passed (Ex. `... | roll down 3`) a number to tell how many places to roll.
Basic working example with rolling columns:
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| rename bit1 bit2 bit3 bit4 bit5 bit6 bit7 bit8
───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────
# │ bit1 │ bit2 │ bit3 │ bit4 │ bit5 │ bit6 │ bit7 │ bit8
───┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────
0 │ 0 │ 0 │ 0 │ 0 │ 0 │ 1 │ 0 │ 0
───┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────
```
We want to "shift" three bits to the left of the bitstring (four in decimal), let's try it:
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| rename bit1 bit2 bit3 bit4 bit5 bit6 bit7 bit8
| roll column 3
───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────
# │ bit4 │ bit5 │ bit6 │ bit7 │ bit8 │ bit1 │ bit2 │ bit3
───┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────
0 │ 0 │ 0 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0
───┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────
```
The tables was rolled correctly (32 in decimal, for above bitstring). However, the *last three header names* look confusing.
We can roll the cell contents only to fix it.
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| rename bit1 bit2 bit3 bit4 bit5 bit6 bit7 bit8
| roll column 3 --cells-only
───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────
# │ bit1 │ bit2 │ bit3 │ bit4 │ bit5 │ bit6 │ bit7 │ bit8
───┼──────┼──────┼──────┼──────┼──────┼──────┼──────┼──────
0 │ 0 │ 0 │ 1 │ 0 │ 0 │ 0 │ 0 │ 0
───┴──────┴──────┴──────┴──────┴──────┴──────┴──────┴──────
```
There we go. Let's compute it's decimal value now (should be 32)
```
> echo '00000100'
| split chars
| each { str to-int }
| rotate counter-clockwise _
| reject _
| roll column 3 --cells-only
| pivot bit --ignore-titles
| get bit
| reverse
| each --numbered { = $it.item * (2 ** $it.index) }
| math sum
32
```
The autoenv logic mutates environment variables in the running session as
it operates and decides what to do for trusted directories containing `.nu-env`
files. Few of the ways to interact with it were all in a single test function.
We separate out all the ways that were done in the single test function to document
it better. This will greatly help once we start refactoring our way out from setting
environment variables this way to just setting them to `Scope`.
This is part of an on-going effort to keep variables (`PATH` and `ENV`)
in our `Scope` and rely on it for everything related to variables.
We expect to move away from setting (`std::*`) envrironment variables in the current
running process. This is non-trivial since we need to handle cases from vars
coming in from the outside world, prioritize, and also compare to the ones
we have both stored in memory and in configuration files.
Also to send out our in-memory (in `Scope`) variables properly to external
programs once we no longer rely on `std::env` vars from the running process.
* Use expand_path to handle the path including tilda
* Publish path::expand_path for using in nu-command
* cargo fmt
Co-authored-by: Wataru Yamaguchi <nagisamark2@gmail.com>
* Document the lexer and lightly improve its names
The bulk of this pull request adds a substantial amount of new inline
documentation for the lexer. Along the way, I made a few minor changes
to the names in the lexer, most of which were internal.
The main change that affects other files is renaming `group` to `block`,
since the function is actually parsing a block (a list of groups).
* Further clean up the lexer
- Consolidate the logic of the various token builders into a single type
- Improve and clean up the event-driven BlockParser
- Clean up comment parsing. Comments now contain their original leading
whitespace as well as trailing whitespace, and know how to move some
leading whitespace back into the body based on how the lexer decides
to dedent the comments. This preserves the original whitespace
information while still making it straight-forward to eliminate leading
whitespace in help comments.
* Update meta.rs
* WIP
* fix clippy
* remove unwraps
* remove unwraps
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
Co-authored-by: Jonathan Turner <jonathan.d.turner@gmail.com>
* standardize on how to get file size
* forgot to remove comment
* make specified size lowercase
* fix the test due to precision
* added another test
* Update README.md
add contributors graphic
* clippy - test adjustment
* tweaked matching
* move commands, futures.rs, script.rs, utils
* move over maybe_print_errors
* add nu_command crate references to nu_cli
* in commands.rs open up to pub mod from pub(crate)
* nu-cli, nu-command, and nu tests are now passing
* cargo fmt
* clean up nu-cli/src/prelude.rs
* code cleanup
* for some reason lex.rs was not formatted, may be causing my error
* remove mod completion from lib.rs which was not being used along with quickcheck macros
* add in allow unused imports
* comment out one failing external test; comment out one failing internal test
* revert commenting out failing tests; something else might be going on; someone with a windows machine should check and see what is going on with these failing windows tests
* Update Cargo.toml
Extend the optional features to nu-command
Co-authored-by: Jonathan Turner <jonathandturner@users.noreply.github.com>
