* Fix "index out of bounds" when input to group-by is empty #4369
* Fix formatting #4369
* Adds test for empty input
Co-authored-by: Ray Henry <ray.henry@thermofisher.com>
Same rationale as in #4378
Also accelerate `grid`
before:
```
Command being timed: "./eager_nu -c for i in 0..100000 { echo whatever } | grid"
User time (seconds): 0.21
System time (seconds): 0.05
Percent of CPU this job got: 36%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.71
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 48112
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 10580
Voluntary context switches: 266
Involuntary context switches: 2595
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
after:
```
Command being timed: "./lazy_nu -c for i in 0..100000 { echo whatever } | grid"
User time (seconds): 0.14
System time (seconds): 0.05
Percent of CPU this job got: 33%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.60
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 48272
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 10582
Voluntary context switches: 286
Involuntary context switches: 831
Swaps: 0
File system inputs: 56
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
For the width calculations for table layout the `strip_ansi` function
has to be called frequently. By checking for the ASCII control chars
(0x00 to 0x1f) except `\n` that are stripped by `strip_ansi_escapes` the number of
necessary allocations can be reduced significantly for the simple case
of text not containing ANSI escapes.
**Benchmark:**
```
nu -c "for i in 0..1000 { ls } | flatten | table"
```
**Allocation reduction**
Running on the nushell repo root as the directory, this change reduces the
allocation volume by approximately 400 MB
(Measured run via KDE heaptrack)
**Speed improvement to output**
Measured via `/usr/bin/time -v`
*before*
```
Command being timed: "./eager_nu -c for i in 0..1000 {ls} | flatten | table"
User time (seconds): 0.87
System time (seconds): 0.14
Percent of CPU this job got: 87%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.16
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 18888
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 4809
Voluntary context switches: 38
Involuntary context switches: 14
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
*after*
```
Command being timed: "./lazy_nu -c for i in 0..1000 {ls} | flatten | table"
User time (seconds): 0.63
System time (seconds): 0.14
Percent of CPU this job got: 80%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.97
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 18660
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 5149
Voluntary context switches: 24
Involuntary context switches: 5
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
* Allow range in 'drop nth'
* Unit tests for drop nth range
* Add range case to the description
* Fix description 2
* format fixes
* Fix example and some refactoring
* clippy fixes
* fix into filesize tests and filesize
* tweaks
* added span back for like the 10th time
* Update filesize.rs
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
* query command with json, web, xml
* query xml now working
* clippy
* comment out web tests
* Initial work on query web
For now we can query everything except tables
* Support for querying tables
Now we can query multiple tables just like before, now the only thing
missing is the test coverage
* Revert "Query plugin"
* augment `ps -l` on windows to display more info
Co-authored-by: Luccas Mateus de Medeiros Gomes <luccasmmg@gmail.com>
* query command with json, web, xml
* query xml now working
* clippy
* comment out web tests
* Initial work on query web
For now we can query everything except tables
* Support for querying tables
Now we can query multiple tables just like before, now the only thing
missing is the test coverage
* finish off
* comment out web test
Co-authored-by: Luccas Mateus de Medeiros Gomes <luccasmmg@gmail.com>
* Port update cells command
Clean up, nicer match statements in UpdateCellsIterator
Return columns flag into HashSet errors
Add FIXME: for update cell behavior on nested lists
* Fix: process cells for Record when no columns are specified
* Fix: address clippy lints for unwrap and into_iter
* Fix: don't step into lists and don't bind $it var
* Maybe solve the none bug?
* cargo fmt
* use nothing, not string
* check at last
* I check it at last
* Use error which has span
* use not found error
* fix error
* use a empty value length?
* * Add commit about what I change and fmt
Now all test passed, but I do not know if it is right
* update the test
* check if it is nothing
* update commit
* Rename test
Co-authored-by: Jakub Žádník <kubouch@gmail.com>
* Add rotate command
* Add rotate counter clockwise
* Fix comments in the code
* Fix clippy warnings
* Fix comment
* Fix wrong step for non even table sizes
* Fix comment for moving through array
* Refactor rotate and have only one command with a --ccw flag for counter-clockwise rotation. By default, rotate is clockwise
* Update usage description
* Add `Find` command
* Complete rustdoc for test `Value` constructors
* Use `Option::unwrap_or` instead of match
* Add `Value::test_filesize` constructor
* Handle searching for terms in `find`
* Fix `find` command signature
* Return multiple elements when `find`ing by predicate
* Do not accept rest parameter with predicate
* Handle `CellPath` in `r#in` and `not_in` for `Find`
* Use `PipelineData::filter`
* Add custom error for path not being a directory
* Fix cd issue with cd-ing into a file
* Keep formatting style as before
* Check if path is not a directory and return error if that's the case
* Add example test to zip
* Port merge command from Nushell
On top of the original merge, this one should not collect a stream
returned from the merged block and allows merging records.
Ansi-cut expects ranges of character numbers (of the non-ansi control
sequence characters) instead of byte indices.
This fixes the panics when wrapping of non-unicode lines (which exceed
the demanded number of characters as byte indices).
Also rectifies some wrong wrapping of unicdoe containing lines that
don't panic
* Switch to short-names when the path is a relative_path (a dir) and exit with an error if the path does not exist
* Remove debugging print line
* Show relative filenames... It does not work yet for ls ../
* Try something else to fix relative paths... it works, but the ../ code part is not very pretty
* Add canonicalize check and remove code clones
* Fix the canonicalize_with issue pointed out by kubouch. Not sure the prefix_str is what kubouch suggested
* Fix the canonicalize_with issue pointed out by kubouch. Not sure the prefix_str is what kubouch suggested
* Add single-dot expansion to nu-path
* Move value path expansion from parser to eval
Fixes#745
* Remove single dot expansion from parser
It is not necessary since it will get expanded anyway in the eval.
* Fix ls to display globs with relative paths
* Use pathdiff crate to get relative paths for ls
Co-authored-by: Stefan Stanciulescu <contact@stefanstanciulescu.com>
* Switch to short-names when the path is a relative_path (a dir) and exit with an error if the path does not exist
* Remove debugging print line
* Show relative filenames... It does not work yet for ls ../
* Try something else to fix relative paths... it works, but the ../ code part is not very pretty
* Add canonicalize check and remove code clones
* Fix the canonicalize_with issue pointed out by kubouch. Not sure the prefix_str is what kubouch suggested
* Fix the canonicalize_with issue pointed out by kubouch. Not sure the prefix_str is what kubouch suggested
* Make env var eval order during "use" deterministic
Fixes#726.
* Merge delta after getting config
To make sure env vars are all in the engine state and not in the stack.
* Move fetch to extra
* Move byte stream code to a function instead of copying it twice
* Fix formatting issues
* Make fetch a default command
* Fix formatting
* Use only $nu.env.PWD for getting current directory
Because setting and reading to/from std::env changes the global state
shich is problematic if we call `cd` from multiple threads (e.g., in a
`par-each` block).
With this change, when engine-q starts, it will either inherit existing
PWD env var, or create a new one from `std::env::current_dir()`.
Otherwise, everything that needs the current directory will get it from
`$nu.env.PWD`. Each spawned external command will get its current
directory per-process which should be thread-safe.
One thing left to do is to patch nu-path for this as well since it uses
`std::env::current_dir()` in its expansions.
* Rename nu-path functions
*_with is not *_relative which should be more descriptive and frees
"with" for use in a followup commit.
* Clone stack every each iter; Fix some commands
Cloning the stack each iteration of `each` makes sure we're not reusing
PWD between iterations.
Some fixes in commands to make them use the new PWD.
* Post-rebase cleanup, fmt, clippy
* Change back _relative to _with in nu-path funcs
Didn't use the idea I had for the new "_with".
* Remove leftover current_dir from rebase
* Add cwd sync at merge_delta()
This makes sure the parser and completer always have up-to-date cwd.
* Always pass absolute path to glob in ls
* Do not allow PWD a relative path; Allow recovery
Makes it possible to recover PWD by proceeding with the REPL cycle.
* Clone stack in each also for byte/string stream
* (WIP) Start moving env variables to engine state
* (WIP) Move env vars to engine state (ugly)
Quick and dirty code.
* (WIP) Remove unused mut and args; Fmt
* (WIP) Fix dataframe tests
* (WIP) Fix missing args after rebase
* (WIP) Clone only env vars, not the whole stack
* (WIP) Add env var clone to `for` loop as well
* Minor edits
* Refactor merge_delta() to include stack merging.
Less error-prone than doing it manually.
* Clone env for each `update` command iteration
* Mark env var hidden only when found in eng. state
* Fix clippt warnings
* Add TODO about env var reading
* Do not clone empty environment in loops
* Remove extra cwd collection
* Split current_dir() into str and path; Fix autocd
* Make completions respect PWD env var
* Port fetch to engine-q
* Fix check for path as a string
* Add a timeout flag and fix some span issues
* Add a temporary fetch command that returns byte streams. Got rid of async stuff as we're using the blocking feature of tokio
* More tweaks for the bytestream
* Rewrite fetch using ByteStreams
* buffer read on bytes directly
Co-authored-by: Stefan Stanciulescu <contact@stefanstanciulescu.com>
* get_columns is working in the columns command
* the new location of the get_columns method is nu-protocol/src/column.rs
* reference the new location of the get_columns method
* move get_columns to nu-engine
* fix#4161
println! and friends will panic on BrokenPipe. The solution is to use
writeln! instead, and ignore the error (or do we want to do something else?)
* test that nu doesn't panic in case of BrokenPipe error
* fixup! test that nu doesn't panic in case of BrokenPipe error
* make do_not_panic_if_broken_pipe only run on UNIX systems
* feat(into): add example of into-bool
* feat(into): add convert from int and float
* feat(into): add converting string to bool
* feat(into): add converting value in table
* fix(into): update error
* fix(into): update span for example
* chore(into): update signature description
* float comparison using epsilon
* Update bool.rs
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
* A first working version of flatten. Needs a lot of cleanup. Committing to have a working version
* Typo fix
* Flatten tests pass
* Final cleanup, ready for push
* Final cleanup, ready for push
* Final cleanup, ready for push
* Final cleanup, ready for push
* Update flatten.rs
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
* Proof of concept treating env vars as Values
* Refactor env var collection and method name
* Remove unnecessary pub
* Move env translations into a new file
* Fix LS_COLORS to support any Value
* Fix spans during env var translation
* Add span to env var in cd
* Improve error diagnostics
* Fix non-string env vars failing string conversion
* Make PROMPT_COMMAND a Block instead of String
* Record host env vars to a fake file
This will give spans to env vars that would otherwise be without one.
Makes errors less confusing.
* Add 'env' command to list env vars
It will list also their values translated to strings
* Sort env command by name; Add env var type
* Remove obsolete test
* Porting 'ansi' command from nushell to engine-q
* Added StrCollect to example_test.rs to allow example tests to run
* Run 'cargo fmt' to fix formatting
* Update command.rs
* Update command.rs
* Update command.rs
* Added a category
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
I removed the Inflector dependency in favor of heck for two reasons:
- to close#3674.
- heck seems simpler and actively maintained
We could probably alter the structure of the `str_` module to expose the
individual casing behaviors better.
I did not feel as confident on changing those signatures.
So I took a lazier approach of a macro in the `mod.rs` that creates the public
shimming function to heck's traits.
* First iteration of the version command
* Cleanup
* Fix the installed plugins bug
* Fix fmt check issue
* Fix clippy warning
* Fixing all clippy warnings
* Remove old code
* Sort default context items categorically
* Separate commands in multiple statements
* Use curly braces instead of square brackets
This prevents undesired reformatting.
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* First draft of these commands
* To MD
* To md and to html
* Fixed cargo and to_md
* `into_abbreviated_string` instead of `into_string`
* Changed how inner tables are displayed
* port over the nth command from nushell
* remove a line of redundant code
* must sort the rows or else if the rows are not from low to high this crashes engine-q
* Port str to-decimal to into decimal command. Add also a Value::test_float function for tests only
* Add support for handling integers into decimals and fix issues with error span
* Remember environment variables from previous scope
* Re-introduce env var hiding
Right now, hiding decls is broken
* Re-introduce hidden field of import patterns
All tests pass now.
* Remove/Address tests TODOs
* Fix test typo; Report hiding error
* Add a few more tests
* Fix wrong expected test result
* add icons to grid output. still needs cleanup
* working but adds a dependency on ansi_term - need to fix that
* update styling, added lots of green code to icons
* clippy
* add config point for grid icons
* fix#4140
We are passing commands into a shell underneath but we were not
escaping arguments correctly. This new version of the code also takes
into consideration the ";" and "&" characters, which have special
meaning in shells.
We would probably benefit from a more robust way to join arguments to
shell programs. Python's stdlib has shlex.join, and perhaps we can
take that implementation as a reference.
* clean up escaping of posix shell args
I believe the right place to do escaping of arguments was in the
spawn_sh_command function. Note that this change prevents things like:
^echo "$(ls)"
from executing the ls command. Instead, this will just print
$(ls)
The regex has been taken from the python stdlib implementation of shlex.quote
* fix non-literal parameters and single quotes
* address clippy's comments
* fixup! address clippy's comments
* test that subshell commands are sanitized properly
* option to replace command same name
* moved order of custom value declarations
* arranged dataframe folders and objects
* sort help commands by name
* added dtypes function for debugging
* corrected name for dataframe commands
* command names using function
* have save --append create file if not exists
Currently, doing:
echo a | save --raw --append file.txt
will fail if file.txt does not exist. This PR changes that
* test that `save --append` will create new file
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* `from toml` command
* From ods
* From XLSX
* From ics
* From ini
* From vcf
* Forgot a eprintln!
* custom value trait
* functions for custom value trait
* custom trait behind flag
* open dataframe command
* command to-df for basic types
* follow path for dataframe
* dataframe operations
* dataframe not default feature
* custom as default feature
* corrected examples in command
* MathEval Variance and Stddev
* Fix tests and linting
* Typo
* Deal with streams when they are not tables
* `from toml` command
* From ods
* From XLSX
It's no longer attached to a Block. Makes access to overlays more
streamlined by removing this one indirection. Also makes it easier to
create standalone overlays without a block which might come in handy.
* Add 'expor env' dummy command
* (WIP) Abstract away module exportables as Overlay
* Switch to Overlays for use/hide
Works for decls only right now.
* Fix passing import patterns of hide to eval
* Simplify use/hide of decls
* Add ImportPattern as Expr; Add use env eval
Still no parsing of "export env" so I can't test it yet.
* Refactor export parsing; Add InternalError
* Add env var export and activation; Misc changes
Now it is possible to `use` env var that was exported from a module.
This commit also adds some new errors and other small changes.
* Add env var hiding
* Fix eval not recognizing hidden decls
Without this change, calling `hide foo`, the evaluator does not know
whether a custom command named "foo" was hidden during parsing,
therefore, it is not possible to reliably throw an error about the "foo"
name not found.
* Add use/hide/export env var tests; Cleanup; Notes
* Ignore hide env related tests for now
* Fix main branch merge mess
* Fixed multi-word export def
* Fix hiding tests on Windows
* Remove env var hiding for now
