# Description
Add `README.md` files to each crate in our workspace (-plugins) and also
include it in the `lib.rs` documentation for <docs.rs> (if there is no
existing `lib.rs` crate documentation)
In all new README I added the defensive comment that the crates are not
considered stable for public consumption. If necessary we can adjust
this if we deem a crate useful for plugin authors.
# Description
This PR adds an internal representation language to Nushell, offering an
alternative evaluator based on simple instructions, stream-containing
registers, and indexed control flow. The number of registers required is
determined statically at compile-time, and the fixed size required is
allocated upon entering the block.
Each instruction is associated with a span, which makes going backwards
from IR instructions to source code very easy.
Motivations for IR:
1. **Performance.** By simplifying the evaluation path and making it
more cache-friendly and branch predictor-friendly, code that does a lot
of computation in Nushell itself can be sped up a decent bit. Because
the IR is fairly easy to reason about, we can also implement
optimization passes in the future to eliminate and simplify code.
2. **Correctness.** The instructions mostly have very simple and
easily-specified behavior, so hopefully engine changes are a little bit
easier to reason about, and they can be specified in a more formal way
at some point. I have made an effort to document each of the
instructions in the docs for the enum itself in a reasonably specific
way. Some of the errors that would have happened during evaluation
before are now moved to the compilation step instead, because they don't
make sense to check during evaluation.
3. **As an intermediate target.** This is a good step for us to bring
the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at
some point, as code generated from new AST can be directly compared to
code generated from old AST. If the IR code is functionally equivalent,
it will behave the exact same way.
4. **Debugging.** With a little bit more work, we can probably give
control over advancing the virtual machine that `IrBlock`s run on to
some sort of external driver, making things like breakpoints and single
stepping possible. Tools like `view ir` and [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than
before to see what exactly is going on with your Nushell code.
The goal is to eventually replace the AST evaluator entirely, once we're
sure it's working just as well. You can help dogfood this by running
Nushell with `$env.NU_USE_IR` set to some value. The environment
variable is checked when Nushell starts, so config runs with IR, or it
can also be set on a line at the REPL to change it dynamically. It is
also checked when running `do` in case within a script you want to just
run a specific piece of code with or without IR.
# Example
```nushell
view ir { |data|
mut sum = 0
for n in $data {
$sum += $n
}
$sum
}
```
```gas
# 3 registers, 19 instructions, 0 bytes of data
0: load-literal %0, int(0)
1: store-variable var 904, %0 # let
2: drain %0
3: drop %0
4: load-variable %1, var 903
5: iterate %0, %1, end 15 # for, label(1), from(14:)
6: store-variable var 905, %0
7: load-variable %0, var 904
8: load-variable %2, var 905
9: binary-op %0, Math(Plus), %2
10: span %0
11: store-variable var 904, %0
12: load-literal %0, nothing
13: drain %0
14: jump 5
15: drop %0 # label(0), from(5:)
16: drain %0
17: load-variable %0, var 904
18: return %0
```
# Benchmarks
All benchmarks run on a base model Mac Mini M1.
## Iterative Fibonacci sequence
This is about as best case as possible, making use of the much faster
control flow. Most code will not experience a speed improvement nearly
this large.
```nushell
def fib [n: int] {
mut a = 0
mut b = 1
for _ in 2..=$n {
let c = $a + $b
$a = $b
$b = $c
}
$b
}
use std bench
bench { 0..50 | each { |n| fib $n } }
```
IR disabled:
```
╭───────┬─────────────────╮
│ mean │ 1ms 924µs 665ns │
│ min │ 1ms 700µs 83ns │
│ max │ 3ms 450µs 125ns │
│ std │ 395µs 759ns │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```
IR enabled:
```
╭───────┬─────────────────╮
│ mean │ 452µs 820ns │
│ min │ 427µs 417ns │
│ max │ 540µs 167ns │
│ std │ 17µs 158ns │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```
##
[gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu)
IR disabled:
```
╭───┬──────────────────╮
│ 0 │ 27ms 929µs 958ns │
│ 1 │ 21ms 153µs 459ns │
│ 2 │ 18ms 639µs 666ns │
│ 3 │ 19ms 554µs 583ns │
│ 4 │ 13ms 383µs 375ns │
│ 5 │ 11ms 328µs 208ns │
│ 6 │ 5ms 659µs 542ns │
╰───┴──────────────────╯
```
IR enabled:
```
╭───┬──────────────────╮
│ 0 │ 22ms 662µs │
│ 1 │ 17ms 221µs 792ns │
│ 2 │ 14ms 786µs 708ns │
│ 3 │ 13ms 876µs 834ns │
│ 4 │ 13ms 52µs 875ns │
│ 5 │ 11ms 269µs 666ns │
│ 6 │ 6ms 942µs 500ns │
╰───┴──────────────────╯
```
##
[random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu)
I got pretty random results out of this benchmark so I decided not to
include it. Not clear why.
# User-Facing Changes
- IR compilation errors may appear even if the user isn't evaluating
with IR.
- IR evaluation can be enabled by setting the `NU_USE_IR` environment
variable to any value.
- New command `view ir` pretty-prints the IR for a block, and `view ir
--json` can be piped into an external tool like [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir).
# Tests + Formatting
All tests are passing with `NU_USE_IR=1`, and I've added some more eval
tests to compare the results for some very core operations. I will
probably want to add some more so we don't have to always check
`NU_USE_IR=1 toolkit test --workspace` on a regular basis.
# After Submitting
- [ ] release notes
- [ ] further documentation of instructions?
- [ ] post-release: publish `nu_plugin_explore_ir`
# Description
Fixes: https://github.com/nushell/nushell/issues/12099
Currently if user run `use voice.nu`, and file is unchanged, then run
`use voice.nu` again. nushell will use the module directly, even if
submodule inside `voice.nu` is changed.
After discussed with @kubouch, I think it's ok to re-parse the module
file when:
1. It exports sub modules which are defined by a file
2. It uses other modules which are defined by a file
## About the change:
To achieve the behavior, we need to add 2 attributes to `Module`:
1. `imported_modules`: it tracks the other modules is imported by the
givem `module`, e.g: `use foo.nu`
2. `file`: the path of a module, if a module is defined by a file, it
will be `Some(path)`, or else it will be `None`.
After the change:
use voice.nu always read the file and parse it.
use voice will still use the module which is saved in EngineState.
# User-Facing Changes
use `xxx.nu` will read the file and parse it if it exports submodules or
uses submodules
# Tests + Formatting
Done
---------
Co-authored-by: Jakub Žádník <kubouch@gmail.com>
# Description
After discussing with @sholderbach the cumbersome usage of
`nu_protocol::Value` in Rust, I created a derive macro to simplify it.
I’ve added a new crate called `nu-derive-value`, which includes two
macros, `IntoValue` and `FromValue`. These are re-exported in
`nu-protocol` and should be encouraged to be used via that re-export.
The macros ensure that all types can easily convert from and into
`Value`. For example, as a plugin author, you can define your plugin
configuration using a Rust struct and easily convert it using
`FromValue`. This makes plugin configuration less of a hassle.
I introduced the `IntoValue` trait for a standardized approach to
converting values into `Value` (and a fallible variant `TryIntoValue`).
This trait could potentially replace existing `into_value` methods.
Along with this, I've implemented `FromValue` for several standard types
and refined other implementations to use blanket implementations where
applicable.
I made these design choices with input from @devyn.
There are more improvements possible, but this is a solid start and the
PR is already quite substantial.
# User-Facing Changes
For `nu-protocol` users, these changes simplify the handling of
`Value`s. There are no changes for end-users of nushell itself.
# Tests + Formatting
Documenting the macros itself is not really possible, as they cannot
really reference any other types since they are the root of the
dependency graph. The standard library has the same problem
([std::Debug](https://doc.rust-lang.org/stable/std/fmt/derive.Debug.html)).
However I documented the `FromValue` and `IntoValue` traits completely.
For testing, I made of use `proc-macro2` in the derive macro code. This
would allow testing the generated source code. Instead I just tested
that the derived functionality is correct. This is done in
`nu_protocol::value::test_derive`, as a consumer of `nu-derive-value`
needs to do the testing of the macro usage. I think that these tests
should provide a stable baseline so that users can be sure that the impl
works.
# After Submitting
With these macros available, we can probably use them in some examples
for plugins to showcase the use of them.
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes#7017, fixes#10763, and fixes#12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
This is partially "feng-shui programming" of moving things to new
separate places.
The later commits include "`git blame` tollbooths" by moving out chunks
of code into new files, which requires an extra step to track things
with `git blame`. We can negiotiate if you want to keep particular
things in their original place.
If egregious I tried to add a bit of documentation. If I see something
that is unused/unnecessarily `pub` I will try to remove that.
- Move `nu_protocol::Exportable` to `nu-parser`
- Guess doccomment for `Exportable`
- Move `Unit` enum from `value` to `AST`
- Move engine state `Variable` def into its folder
- Move error-related files in `nu-protocol` subdir
- Move `pipeline_data` module into its own folder
- Move `stream.rs` over into the `pipeline_data` mod
- Move `PipelineMetadata` into its own file
- Doccomment `PipelineMetadata`
- Remove unused `is_leap_year` in `value/mod`
- Note about criminal `type_compatible` helper
- Move duration fmting into new `value/duration.rs`
- Move filesize fmting logic to new `value/filesize`
- Split reexports from standard imports in `value/mod`
- Doccomment trait `CustomValue`
- Polish doccomments and intradoc links
# Description
This PR uses the new plugin protocol to intelligently keep plugin
processes running in the background for further plugin calls.
Running plugins can be seen by running the new `plugin list` command,
and stopped by running the new `plugin stop` command.
This is an enhancement for the performance of plugins, as starting new
plugin processes has overhead, especially for plugins in languages that
take a significant amount of time on startup. It also enables plugins
that have persistent state between commands, making the migration of
features like dataframes and `stor` to plugins possible.
Plugins are automatically stopped by the new plugin garbage collector,
configurable with `$env.config.plugin_gc`:
```nushell
$env.config.plugin_gc = {
# Configuration for plugin garbage collection
default: {
enabled: true # true to enable stopping of inactive plugins
stop_after: 10sec # how long to wait after a plugin is inactive to stop it
}
plugins: {
# alternate configuration for specific plugins, by name, for example:
#
# gstat: {
# enabled: false
# }
}
}
```
If garbage collection is enabled, plugins will be stopped after
`stop_after` passes after they were last active. Plugins are counted as
inactive if they have no running plugin calls. Reading the stream from
the response of a plugin call is still considered to be activity, but if
a plugin holds on to a stream but the call ends without an active
streaming response, it is not counted as active even if it is reading
it. Plugins can explicitly disable the GC as appropriate with
`engine.set_gc_disabled(true)`.
The `version` command now lists plugin names rather than plugin
commands. The list of plugin commands is accessible via `plugin list`.
Recommend doing this together with #12029, because it will likely force
plugin developers to do the right thing with mutability and lead to less
unexpected behavior when running plugins nested / in parallel.
# User-Facing Changes
- new command: `plugin list`
- new command: `plugin stop`
- changed command: `version` (now lists plugin names, rather than
commands)
- new config: `$env.config.plugin_gc`
- Plugins will keep running and be reused, at least for the configured
GC period
- Plugins that used mutable state in weird ways like `inc` did might
misbehave until fixed
- Plugins can disable GC if they need to
- Had to change plugin signature to accept `&EngineInterface` so that
the GC disable feature works. #12029 does this anyway, and I'm expecting
(resolvable) conflicts with that
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
Because there is some specific OS behavior required for plugins to not
respond to Ctrl-C directly, I've developed against and tested on both
Linux and Windows to ensure that works properly.
# After Submitting
I think this probably needs to be in the book somewhere
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This PR adds a new evaluator path with callbacks to a mutable trait
object implementing a Debugger trait. The trait object can do anything,
e.g., profiling, code coverage, step debugging. Currently,
entering/leaving a block and a pipeline element is marked with
callbacks, but more callbacks can be added as necessary. Not all
callbacks need to be used by all debuggers; unused ones are simply empty
calls. A simple profiler is implemented as a proof of concept.
The debugging support is implementing by making `eval_xxx()` functions
generic depending on whether we're debugging or not. This has zero
computational overhead, but makes the binary slightly larger (see
benchmarks below). `eval_xxx()` variants called from commands (like
`eval_block_with_early_return()` in `each`) are chosen with a dynamic
dispatch for two reasons: to not grow the binary size due to duplicating
the code of many commands, and for the fact that it isn't possible
because it would make Command trait objects object-unsafe.
In the future, I hope it will be possible to allow plugin callbacks such
that users would be able to implement their profiler plugins instead of
having to recompile Nushell.
[DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be
interesting to explore.
Try `help debug profile`.
## Screenshots
Basic output:
To profile with more granularity, increase the profiler depth (you'll
see that repeated `is-windows` calls take a large chunk of total time,
making it a good candidate for optimizing):
## Benchmarks
### Binary size
Binary size increase vs. main: **+40360 bytes**. _(Both built with
`--release --features=extra,dataframe`.)_
### Time
```nushell
# bench_debug.nu
use std bench
let test = {
1..100
| each {
ls | each {|row| $row.name | str length }
}
| flatten
| math avg
}
print 'debug:'
let res2 = bench { debug profile $test } --pretty
print $res2
```
```nushell
# bench_nodebug.nu
use std bench
let test = {
1..100
| each {
ls | each {|row| $row.name | str length }
}
| flatten
| math avg
}
print 'no debug:'
let res1 = bench { do $test } --pretty
print $res1
```
`cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower
than `cargo run --release -- bench_nodebug.nu` due to the collection
overhead + gathering the report. This is expected. When gathering more
stuff, the overhead is obviously higher.
`cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I
didn't measure any difference. Both benchmarks report times between 97
and 103 ms randomly, without one being consistently higher than the
other. This suggests that at least in this particular case, when not
running any debugger, there is no runtime overhead.
## API changes
This PR adds a generic parameter to all `eval_xxx` functions that forces
you to specify whether you use the debugger. You can resolve it in two
ways:
* Use a provided helper that will figure it out for you. If you wanted
to use `eval_block(&engine_state, ...)`, call `let eval_block =
get_eval_block(&engine_state); eval_block(&engine_state, ...)`
* If you know you're in an evaluation path that doesn't need debugger
support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is
the case of hooks, for example).
I tried to add more explanation in the docstring of `debugger_trait.rs`.
## TODO
- [x] Better profiler output to reduce spam of iterative commands like
`each`
- [x] Resolve `TODO: DEBUG` comments
- [x] Resolve unwraps
- [x] Add doc comments
- [x] Add usage and extra usage for `debug profile`, explaining all
columns
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
Hopefully none.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
While #11057 is merged, it's hard to tell the difference between
`--flag: bool` and `--flag`, and it makes user hard to read custom
commands' signature, and hard to use them correctly.
After discussion, I think we can deprecate `--flag: bool` usage, and
encourage using `--flag` instead.
# User-Facing Changes
The following code will raise warning message, but don't stop from
running.
```nushell
❯ def florb [--dry-run: bool, --another-flag] { "aaa" }; florb
Error: × Deprecated: --flag: bool
╭─[entry #7:1:1]
1 │ def florb [--dry-run: bool, --another-flag] { "aaa" }; florb
· ──┬─
· ╰── `--flag: bool` is deprecated. Please use `--flag` instead, more info: https://www.nushell.sh/book/custom_commands.html
╰────
aaa
```
cc @kubouch
# Tests + Formatting
Done
# After Submitting
- [ ] Add more information under
https://www.nushell.sh/book/custom_commands.html to indicate `--dry-run:
bool` is not allowed,
- [ ] remove `: bool` from custom commands between 0.89 and 0.90
---------
Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com>
related to
- https://github.com/nushell/nushell/pull/10478
# Description
this PR is the followup removal to
https://github.com/nushell/nushell/pull/10478.
# User-Facing Changes
`$nothing` is now an undefined variable, unless define by the user.
```nushell
> $nothing
Error: nu::parser::variable_not_found
× Variable not found.
╭─[entry #1:1:1]
1 │ $nothing
· ────┬───
· ╰── variable not found.
╰────
```
# Tests + Formatting
# After Submitting
mention that in release notes
related to
- https://github.com/nushell/nushell/pull/9973
- https://github.com/nushell/nushell/pull/9918
thanks to @jntrnr and their super useful tips on this PR, i learned
about the parser + evaluation, so 🙏
# Description
because we already have `null` as the value of the type `nothing` and as
a followup to the two other attempts of mine, i propose to remove the
redundant `$nothing` built-in variable 😋
this PR is the first step, deprecating `$nothing`.
a followup PR will remove it altogether and wait for 0.87 👍⚙️ **details**: a new `NOTHING_VARIABLE_ID = 3` has been added,
parsing `$nothing` will create it, finally a `Value::Nothing` will be
produced and a warning will be reported.
this PR already fixes the `toolkit.nu` module so that it does not throw
a bunch of warnings each time 👌
# User-Facing Changes
`$nothing` is now deprecated and will be removed in 0.87
```nushell
> $nothing
Error: × Deprecated variable
╭─[entry #1:1:1]
1 │ $nothing
· ────┬───
· ╰── `$nothing` is deprecated and will be removed in 0.87.
╰────
help: Use `null` instead
```
# Tests + Formatting
tests have been updated, especially
- `nothing_fails_string`
- `nothing_fails_int`
which use a variable called `nil` now to make sure `nothing` does not
support cell paths 👍
# After Submitting
classic deprecation mention 👍
now nu_std only depends on nu_parser, nu_protocol and miette
and removes the nu_cli dependency
this enables developers moving forward to come along and implement their
own CLI's without having to pull in a redundant nu-cli which will not be
needed for them.
I did this by moving report_error into nu_protocol
which nu_std already has a dependency on anyway....
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
# Description
This is a pretty heavy refactor of the parser to support multiple parser
errors. It has a few issues we should address before landing:
- [x] In some cases, error quality has gotten worse `1 / "bob"` for
example
- [x] if/else isn't currently parsing correctly
- probably others
# User-Facing Changes
This may have error quality degradation as we adjust to the new error
reporting mechanism.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-utils/standard_library/tests.nu` to run the
tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
# Description
This PR adds an alternative alias implementation. Old aliases still work
but you need to use `old-alias` instead of `alias`.
Instead of replacing spans in the original code and re-parsing, which
proved to be extremely error-prone and a constant source of panics, the
new implementation creates a new command that references the old
command. Consider the new alias defined as `alias ll = ls -l`. The
parser creates a new command called `ll` and remembers that it is
actually a `ls` command called with the `-l` flag. Then, when the parser
sees the `ll` command, it will translate it to `ls -l` and passes to it
any parameters that were passed to the call to `ll`. It works quite
similar to how known externals defined with `extern` are implemented.
The new alias implementation should work the same way as the old
aliases, including exporting from modules, referencing both known and
unknown externals. It seems to preserve custom completions and pipeline
metadata. It is quite robust in most cases but there are some rough
edges (see later).
Fixes https://github.com/nushell/nushell/issues/7648,
https://github.com/nushell/nushell/issues/8026,
https://github.com/nushell/nushell/issues/7512,
https://github.com/nushell/nushell/issues/5780,
https://github.com/nushell/nushell/issues/7754
No effect: https://github.com/nushell/nushell/issues/8122 (we might
revisit the completions code after this PR)
Should use custom command instead:
https://github.com/nushell/nushell/issues/6048
# User-Facing Changes
Since aliases are now basically commands, it has some new implications:
1. `alias spam = "spam"` (requires command call)
* **workaround**: use `alias spam = echo "spam"`
2. `def foo [] { 'foo' }; alias foo = ls -l` (foo defined more than
once)
* **workaround**: use different name (commands also have this
limitation)
4. `alias ls = (ls | sort-by type name -i)`
* **workaround**: Use custom command. _The common issue with this is
that it is currently not easy to pass flags through custom commands and
command referencing itself will lead to stack overflow. Both of these
issues are meant to be addressed._
5. TODO: Help messages, `which` command, `$nu.scope.aliases`, etc.
* Should we treat the aliases as commands or should they be separated
from regular commands?
6. Needs better error message and syntax highlight for recursed alias
(`alias f = f`)
7. Can't create alias with the same name as existing command (`alias ls
= ls -a`)
* Might be possible to add support for it (not 100% sure)
8. Standalone `alias` doesn't list aliases anymore
9. Can't alias parser keywords (e.g., stuff like `alias ou = overlay
use` won't work)
* TODO: Needs a better error message when attempting to do so
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
# Description
As title, we can't provide examples for plugin commands, this pr would
make it possible
# User-Facing Changes
Take plugin `nu-example-1` as example:
```
❯ nu-example-1 -h
PluginSignature test 1 for plugin. Returns Value::Nothing
Usage:
> nu-example-1 {flags} <a> <b> (opt) ...(rest)
Flags:
-h, --help - Display the help message for this command
-f, --flag - a flag for the signature
-n, --named <String> - named string
Parameters:
a <int>: required integer value
b <string>: required string value
(optional) opt <int>: Optional number
...rest <string>: rest value string
Examples:
running example with an int value and string value
> nu-example-1 3 bb
```
The examples session is newly added.
## Basic idea behind these changes
when nushell query plugin signatures, plugin just returns it's signature
without any examples, so nushell have no idea about the examples of
plugin commands.
To adding the feature, we just making plugin returns it's signature with
examples.
Before:
```
1. get signature
---------------->
Nushell ------------------ Plugin
<-----------------
2. returns Vec<Signature>
```
After:
```
1. get signature
---------------->
Nushell ------------------ Plugin
<-----------------
2. returns Vec<PluginSignature>
```
When writing plugin signature to $nu.plugin-path:
Serialize `<PluginSignature>` rather than `<Signature>`, which would
enable us to serialize examples to `$nu.plugin-path`
## Shortcoming
It's a breaking changes because `Plugin::signature` is changed, and it
requires plugin authors to change their code for new signatures.
Fortunally it should be easy to change, for rust based plugin, we just
need to make a global replace from word `Signature` to word
`PluginSignature` in their plugin project.
Our content of plugin-path is really large, if one plugin have many
examples, it'd results to larger body of $nu.plugin-path, which is not
really scale. A solution would be save register information in other
binary formats rather than `json`. But I think it'd be another story.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
# Description
This PR allows the configuration of cursor shapes in nushell for each
edit mode. This is the change that is in the default_config.nu file.
```
cursor_shape: {
emacs: line # block, underscore, line (line is the default)
vi_insert: block # block, underscore, line (block is the default)
vi_normal: underscore # block, underscore, line (underscore is the default)
}
```
# User-Facing Changes
See above. If you'd prefer a different default, please speak up and let
us know.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
src/main.rs has a dependency on BufferedReader
which is currently located in nu_command.
I am moving BufferedReader to a more relevant
location (crate) which will allow / eliminate main's dependency
on nu_command in a benchmark / testing environment...
now that @rgwood has landed benches I want
to start experimenting with benchmarks related
to the parser.
For benchmark purposes when dealing with parsing
you need a very simple set of commands that show
how well the parser is doing, in other words
just the core commands... Not all of nu_command...
Having a smaller nu binary when running the benchmark CI
would enable building nushell quickly, yet still show us
how well the parser is performing...
Once this PR lands the only dependency main will have
on nu_command is create_default_context ---
meaning for benchmark purposes we can swap in a tiny
crate of commands instead of the gigantic nu_command
which has its "own" create_default_context...
It will also enable other crates going forward to
use BufferedReader. Right now it is not accessible
to other lower level crates because it is located in a
"top of the stack crate".
* Copy lev_distance.rs from the rust compiler
* Minor changes to code from rust compiler
* "Did you mean" suggestions: test instrumented to generate markdown report
* Did you mean suggestions: delete test instrumentation
* Fix tests
* Fix test
`foo` has a genuine match: `for`
* Improve tests
* WIP: Start laying overlays
* Rename Overlay->Module; Start adding overlay
* Revamp adding overlay
* Add overlay add tests; Disable debug print
* Fix overlay add; Add overlay remove
* Add overlay remove tests
* Add missing overlay remove file
* Add overlay list command
* (WIP?) Enable overlays for env vars
* Move OverlayFrames to ScopeFrames
* (WIP) Move everything to overlays only
ScopeFrame contains nothing but overlays now
* Fix predecls
* Fix wrong overlay id translation and aliases
* Fix broken env lookup logic
* Remove TODOs
* Add overlay add + remove for environment
* Add a few overlay tests; Fix overlay add name
* Some cleanup; Fix overlay add/remove names
* Clippy
* Fmt
* Remove walls of comments
* List overlays from stack; Add debugging flag
Currently, the engine state ordering is somehow broken.
* Fix (?) overlay list test
* Fix tests on Windows
* Fix activated overlay ordering
* Check for active overlays equality in overlay list
This removes the -p flag: Either both parser and engine will have the
same overlays, or the command will fail.
* Add merging on overlay remove
* Change help message and comment
* Add some remove-merge/discard tests
* (WIP) Track removed overlays properly
* Clippy; Fmt
* Fix getting last overlay; Fix predecls in overlays
* Remove merging; Fix re-add overwriting stuff
Also some error message tweaks.
* Fix overlay error in the engine
* Update variable_completions.rs
* Adds flags and optional arguments to view-source (#5446)
* added flags and optional arguments to view-source
* removed redundant code
* removed redundant code
* fmt
* fix bug in shell_integration (#5450)
* fix bug in shell_integration
* add some comments
* enable cd to work with directory abbreviations (#5452)
* enable cd to work with abbreviations
* add abbreviation example
* fix tests
* make it configurable
* make cd recornize symblic link (#5454)
* implement seq char command to generate single character sequence (#5453)
* add tmp code
* add seq char command
* Add split number flag in `split row` (#5434)
Signed-off-by: Yuheng Su <gipsyh.icu@gmail.com>
* Add two more overlay tests
* Add ModuleId to OverlayFrame
* Fix env conversion accidentally activating overlay
It activated overlay from permanent state prematurely which would
cause `overlay add` to misbehave.
* Remove unused parameter; Add overlay list test
* Remove added traces
* Add overlay commands examples
* Modify TODO
* Fix $nu.scope iteration
* Disallow removing default overlay
* Refactor some parser errors
* Remove last overlay if no argument
* Diversify overlay examples
* Make it possible to update overlay's module
In case the origin module updates, the overlay add loads the new module,
makes it overlay's origin and applies the changes. Before, it was
impossible to update the overlay if the module changed.
Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
Co-authored-by: pwygab <88221256+merelymyself@users.noreply.github.com>
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
Co-authored-by: WindSoilder <WindSoilder@outlook.com>
Co-authored-by: Yuheng Su <gipsyh.icu@gmail.com>
* option to replace command same name
* moved order of custom value declarations
* arranged dataframe folders and objects
* sort help commands by name
* added dtypes function for debugging
* corrected name for dataframe commands
* command names using function
* custom value trait
* functions for custom value trait
* custom trait behind flag
* open dataframe command
* command to-df for basic types
* follow path for dataframe
* dataframe operations
* dataframe not default feature
* custom as default feature
* corrected examples in command
* Add 'expor env' dummy command
* (WIP) Abstract away module exportables as Overlay
* Switch to Overlays for use/hide
Works for decls only right now.
* Fix passing import patterns of hide to eval
* Simplify use/hide of decls
* Add ImportPattern as Expr; Add use env eval
Still no parsing of "export env" so I can't test it yet.
* Refactor export parsing; Add InternalError
* Add env var export and activation; Misc changes
Now it is possible to `use` env var that was exported from a module.
This commit also adds some new errors and other small changes.
* Add env var hiding
* Fix eval not recognizing hidden decls
Without this change, calling `hide foo`, the evaluator does not know
whether a custom command named "foo" was hidden during parsing,
therefore, it is not possible to reliably throw an error about the "foo"
name not found.
* Add use/hide/export env var tests; Cleanup; Notes
* Ignore hide env related tests for now
* Fix main branch merge mess
* Fixed multi-word export def
* Fix hiding tests on Windows
* Remove env var hiding for now
In Nu we have variables (E.g. $var-name) and these contain `Value` types.
This means we can bind to variables any structured data and column path syntax
(E.g. `$variable.path.to`) allows flexibility for "querying" said structures.
Here we offer completions for these. For example, in a Nushell session the
variable `$nu` contains environment values among other things. If we wanted to
see in the screen some environment variable (say the var `SHELL`) we do:
```
> echo $nu.env.SHELL
```
with completions we can now do: `echo $nu.env.S[\TAB]` and we get suggestions
that start at the column path `$nu.env` with vars starting with the letter `S`
in this case `SHELL` appears in the suggestions.
* Dataframe MVP
* Removed test csv file
* Dataframe MVP
* Removed test csv file
* New revision polars
* New revision polars
* csv file reader
* argument parser for file reader
* Parser from Row primitive
* Column conversion
* Added as f32 and f64
* Parsing row to dataframe
* Removed repeated push to vector
* Accept table values to create dataframe
* Removed default serde
* Dataframe to rows to show data
* Save name of file with dataframe
* Usage example
* Upgrade polars version
* Clippy changes
* Added print function with head and tail
* Move dataframe struct to folder
* Lock file after running tests and merge
* Optional feature for dataframe
* Removed dataframe from plugins
* Update primitive.rs
Co-authored-by: JT <jonathandturner@users.noreply.github.com>
* Revert "History, more test coverage improvements, and refactorings. (#3217)"
This reverts commit 8fc8fc89aa.
* Add tests
* Refactor .nu-env
* Change logic of Config write to logic of read()
* Fix reload always appends to old vars
* Fix reload always takes last_modified of global config
* Add reload_config in evaluation context
* Reload config after writing to it in cfg set / cfg set_into
* Add --no-history to cli options
* Use --no-history in tests
* Add comment about maybe_print_errors
* Get ctrl_exit var from context.global_config
* Use context.global_config in command "config"
* Add Readme in engine how env vars are now handled
* Update docs from autoenv command
* Move history_path from engine to nu_data
* Move load history out of if
* No let before return
* Add import for indexmap
We split off the evaluation engine part of nu-cli into its own crate. This helps improve build times for nu-cli by 17% in my tests. It also helps us see a bit better what's the core engine portion vs the part specific to the interactive CLI piece.
There's more than can be done here, but I think it's a good start in the right direction.
* Begin allowing comments and multiline scripts.
* clippy
* Finish moving to groups. Test pass
* Keep going
* WIP
* WIP
* BROKEN WIP
* WIP
* WIP
* Fix more tests
* WIP: alias starts working
* Broken WIP
* Broken WIP
* Variables begin to work
* captures start working
* A little better but needs fixed scope
* Shorthand env setting
* Update main merge
* Broken WIP
* WIP
* custom command parsing
* Custom commands start working
* Fix coloring and parsing of block
* Almost there
* Add some tests
* Add more param types
* Bump version
* Fix benchmark
* Fix stuff
* seq command - WIP
* why, oh why
* works with parameters
* widths should've been optional
* dbg messages
* working. rest had to be first.
* updated so that it outputs a table instead of just strings
* made to work with floats, allowed separator be more than 1 char
* clippy
* fixed tests
* changed terminator help desc
* commit to get ci moving again
