mirror of
https://github.com/nushell/nushell.git
synced 2024-11-28 19:33:47 +01:00
ec528c0626
1351 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Stefan Holderbach
|
ec528c0626
|
Refactor source cache into CachedFile struct (#12240)
# Description Get rid of two parallel `Vec`s in `StateDelta` and `EngineState`, that also duplicated span information. Use a struct with documenting fields. Also use `Arc<str>` and `Arc<[u8]>` for the allocations as they are never modified and cloned often (see #12229 for the first improvement). This also makes the representation more compact as no capacity is necessary. # User-Facing Changes API breakage on `EngineState`/`StateWorkingSet`/`StateDelta` that should not really affect plugin authors. |
||
Devyn Cairns
|
cf321ab510
|
Make EngineState clone cheaper with Arc on all of the heavy objects (#12229)
# Description This makes many of the larger objects in `EngineState` into `Arc`, and uses `Arc::make_mut` to do clone-on-write if the reference is not unique. This is generally very cheap, giving us the best of both worlds - allowing us to mutate without cloning if we have an exclusive reference, and cloning if we don't. This started as more of a curiosity for me after remembering that `Arc::make_mut` exists and can make using `Arc` for mostly immutable data that sometimes needs to be changed very convenient, and also after hearing someone complain about memory usage on Discord - this is a somewhat significant win for that. The exact objects that were wrapped in `Arc`: - `files`, `file_contents` - the strings and byte buffers - `decls` - the whole `Vec`, but mostly to avoid lots of individual `malloc()` calls on Clone rather than for memory usage - `blocks` - the blocks themselves, rather than the outer Vec - `modules` - the modules themselves, rather than the outer Vec - `env_vars`, `previous_env_vars` - the entire maps - `config` The changes required were relatively minimal, but this is a breaking API change. In particular, blocks are added as Arcs, to allow the parser cache functionality to work. With my normal nu config, running on Linux, this saves me about 15 MiB of process memory usage when running interactively (65 MiB → 50 MiB). This also makes quick command executions cheaper, particularly since every REPL loop now involves a clone of the engine state so that we can recover from a panic. It also reduces memory usage where engine state needs to be cloned and sent to another thread or kept within an iterator. # User-Facing Changes Shouldn't be any, since it's all internal stuff, but it does change some public interfaces so it's a breaking change |
||
|
Devyn Cairns
|
6795ad7e33
|
Make custom value type handling more consistent (#12230)
[Context on Discord](https://discord.com/channels/601130461678272522/855947301380947968/1219425984990806207) # Description - Rename `CustomValue::value_string()` to `type_name()` to reflect its usage better. - Change print behavior to always call `to_base_value()` first, to give the custom value better control over the output. - Change `describe --detailed` to show the type name as the subtype, rather than trying to describe the base value. - Change custom `Type` to use `type_name()` rather than `typetag_name()` to make things like `PluginCustomValue` more transparent One question: should `describe --detailed` still include a description of the base value somewhere? I'm torn on it, it seems possibly useful for some things (maybe sqlite databases?), but having `describe -d` not include the custom type name anywhere felt weird. Another option would be to add another method to `CustomValue` for info to be displayed in `describe`, so that it can be more type-specific? # User-Facing Changes Everything above has implications for printing and `describe` on custom values # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` |
||
|
Devyn Cairns
|
1cb5221f01
|
Add Value::recurse_mut() to save duplicated code in PluginCustomValue (#12218)
# Description We do a lot of visiting contained values in the serialization / validity functions within `PluginCustomValue` utils. This adds `Value::recurse_mut()` which wraps up most of that logic into something that can be reused. |
||
Ian Manske
|
c950269575
|
Fix $in value for insert closure (#12209)
# Description Fixes #12193 where the `$in` value may be null for closures provided to `insert`. # User-Facing Changes The `$in` value will now always be the same as the closure parameter for `insert`. |
||
|
Devyn Cairns
|
9cf2e873b5
|
Reorganize plugin API around commands (#12170)
[Context on Discord](https://discord.com/channels/601130461678272522/855947301380947968/1216517833312309419) # Description This is a significant breaking change to the plugin API, but one I think is worthwhile. @ayax79 mentioned on Discord that while trying to start on a dataframes plugin, he was a little disappointed that more wasn't provided in terms of code organization for commands, particularly since there are *a lot* of `dfr` commands. This change treats plugins more like miniatures of the engine, with dispatch of the command name being handled inherently, each command being its own type, and each having their own signature within the trait impl for the command type rather than having to find a way to centralize it all into one `Vec`. For the example plugins that have multiple commands, I definitely like how this looks a lot better. This encourages doing code organization the right way and feels very good. For the plugins that have only one command, it's just a little bit more boilerplate - but still worth it, in my opinion. The `Box<dyn PluginCommand<Plugin = Self>>` type in `commands()` is a little bit hairy, particularly for Rust beginners, but ultimately not so bad, and it gives the desired flexibility for shared state for a whole plugin + the individual commands. # User-Facing Changes Pretty big breaking change to plugin API, but probably one that's worth making. ```rust use nu_plugin::*; use nu_protocol::{PluginSignature, PipelineData, Type, Value}; struct LowercasePlugin; struct Lowercase; // Plugins can now have multiple commands impl PluginCommand for Lowercase { type Plugin = LowercasePlugin; // The signature lives with the command fn signature(&self) -> PluginSignature { PluginSignature::build("lowercase") .usage("Convert each string in a stream to lowercase") .input_output_type(Type::List(Type::String.into()), Type::List(Type::String.into())) } // We also provide SimplePluginCommand which operates on Value like before fn run( &self, plugin: &LowercasePlugin, engine: &EngineInterface, call: &EvaluatedCall, input: PipelineData, ) -> Result<PipelineData, LabeledError> { let span = call.head; Ok(input.map(move |value| { value.as_str() .map(|string| Value::string(string.to_lowercase(), span)) // Errors in a stream should be returned as values. .unwrap_or_else(|err| Value::error(err, span)) }, None)?) } } // Plugin now just has a list of commands, and the custom value op stuff still goes here impl Plugin for LowercasePlugin { fn commands(&self) -> Vec<Box<dyn PluginCommand<Plugin=Self>>> { vec![Box::new(Lowercase)] } } fn main() { serve_plugin(&LowercasePlugin{}, MsgPackSerializer) } ``` Time this however you like - we're already breaking stuff for 0.92, so it might be good to do it now, but if it feels like a lot all at once, it could wait. # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting - [ ] Update examples in the book - [x] Fix #12088 to match - this change would actually simplify it a lot, because the methods are currently just duplicated between `Plugin` and `StreamingPlugin`, but they only need to be on `Plugin` with this change |
||
|
Ian Manske
|
b6c7656194
|
IO and redirection overhaul (#11934)
# Description The PR overhauls how IO redirection is handled, allowing more explicit and fine-grain control over `stdout` and `stderr` output as well as more efficient IO and piping. To summarize the changes in this PR: - Added a new `IoStream` type to indicate the intended destination for a pipeline element's `stdout` and `stderr`. - The `stdout` and `stderr` `IoStream`s are stored in the `Stack` and to avoid adding 6 additional arguments to every eval function and `Command::run`. The `stdout` and `stderr` streams can be temporarily overwritten through functions on `Stack` and these functions will return a guard that restores the original `stdout` and `stderr` when dropped. - In the AST, redirections are now directly part of a `PipelineElement` as a `Option<Redirection>` field instead of having multiple different `PipelineElement` enum variants for each kind of redirection. This required changes to the parser, mainly in `lite_parser.rs`. - `Command`s can also set a `IoStream` override/redirection which will apply to the previous command in the pipeline. This is used, for example, in `ignore` to allow the previous external command to have its stdout redirected to `Stdio::null()` at spawn time. In contrast, the current implementation has to create an os pipe and manually consume the output on nushell's side. File and pipe redirections (`o>`, `e>`, `e>|`, etc.) have precedence over overrides from commands. This PR improves piping and IO speed, partially addressing #10763. Using the `throughput` command from that issue, this PR gives the following speedup on my setup for the commands below: | Command | Before (MB/s) | After (MB/s) | Bash (MB/s) | | --------------------------- | -------------:| ------------:| -----------:| | `throughput o> /dev/null` | 1169 | 52938 | 54305 | | `throughput \| ignore` | 840 | 55438 | N/A | | `throughput \| null` | Error | 53617 | N/A | | `throughput \| rg 'x'` | 1165 | 3049 | 3736 | | `(throughput) \| rg 'x'` | 810 | 3085 | 3815 | (Numbers above are the median samples for throughput) This PR also paves the way to refactor our `ExternalStream` handling in the various commands. For example, this PR already fixes the following code: ```nushell ^sh -c 'echo -n "hello "; sleep 0; echo "world"' | find "hello world" ``` This returns an empty list on 0.90.1 and returns a highlighted "hello world" on this PR. Since the `stdout` and `stderr` `IoStream`s are available to commands when they are run, then this unlocks the potential for more convenient behavior. E.g., the `find` command can disable its ansi highlighting if it detects that the output `IoStream` is not the terminal. Knowing the output streams will also allow background job output to be redirected more easily and efficiently. # User-Facing Changes - External commands returned from closures will be collected (in most cases): ```nushell 1..2 | each {|_| nu -c "print a" } ``` This gives `["a", "a"]` on this PR, whereas this used to print "a\na\n" and then return an empty list. ```nushell 1..2 | each {|_| nu -c "print -e a" } ``` This gives `["", ""]` and prints "a\na\n" to stderr, whereas this used to return an empty list and print "a\na\n" to stderr. - Trailing new lines are always trimmed for external commands when piping into internal commands or collecting it as a value. (Failure to decode the output as utf-8 will keep the trailing newline for the last binary value.) In the current nushell version, the following three code snippets differ only in parenthesis placement, but they all also have different outputs: 1. `1..2 | each { ^echo a }` ``` a a ╭────────────╮ │ empty list │ ╰────────────╯ ``` 2. `1..2 | each { (^echo a) }` ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` 3. `1..2 | (each { ^echo a })` ``` ╭───┬───╮ │ 0 │ a │ │ │ │ │ 1 │ a │ │ │ │ ╰───┴───╯ ``` But in this PR, the above snippets will all have the same output: ``` ╭───┬───╮ │ 0 │ a │ │ 1 │ a │ ╰───┴───╯ ``` - All existing flags on `run-external` are now deprecated. - File redirections now apply to all commands inside a code block: ```nushell (nu -c "print -e a"; nu -c "print -e b") e> test.out ``` This gives "a\nb\n" in `test.out` and prints nothing. The same result would happen when printing to stdout and using a `o>` file redirection. - External command output will (almost) never be ignored, and ignoring output must be explicit now: ```nushell (^echo a; ^echo b) ``` This prints "a\nb\n", whereas this used to print only "b\n". This only applies to external commands; values and internal commands not in return position will not print anything (e.g., `(echo a; echo b)` still only prints "b"). - `complete` now always captures stderr (`do` is not necessary). # After Submitting The language guide and other documentation will need to be updated. |
||
Yash Thakur
|
0ff36dfe42
|
Canonicalize each component of config files (#12167)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Because `std::fs::canonicalize` requires the path to exist, this PR makes it so that when canonicalizing any config file, the `$nu.default-config-dir/nushell` part is canonicalized first, then `$nu.default-config-dir/nushell/foo.nu` is canonicalized. This should also fix the issue @devyn pointed out [here](https://github.com/nushell/nushell/pull/12118#issuecomment-1989546708) where a couple of the tests failed if one's `~/.config/nushell` folder was actually a symlink to a different folder. The tests previously didn't canonicalize the expected paths. I was going to make a PR that caches the config directory on startup (as suggested by fdncred and Ian in Discord), but I can make that part of this PR if we want to avoid creating unnecessary PRs. I think it probably makes more sense to separate them though. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Devyn Cairns
|
73f3c0b60b
|
Support for all custom value operations on plugin custom values (#12088)
# Description Adds support for the following operations on plugin custom values, in addition to `to_base_value` which was already present: - `follow_path_int()` - `follow_path_string()` - `partial_cmp()` - `operation()` - `Drop` (notification, if opted into with `CustomValue::notify_plugin_on_drop`) There are additionally customizable methods within the `Plugin` and `StreamingPlugin` traits for implementing these functions in a way that requires access to the plugin state, as a registered handle model such as might be used in a dataframes plugin would. `Value::append` was also changed to handle custom values correctly. # User-Facing Changes - Signature of `CustomValue::follow_path_string` and `CustomValue::follow_path_int` changed to give access to the span of the custom value itself, useful for some errors. - Plugins using custom values have to be recompiled because the engine will try to do custom value operations that aren't supported - Plugins can do more things 🎉 # Tests + Formatting Tests were added for all of the new custom values functionality. - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting - [ ] Document protocol reference `CustomValueOp` variants: - [ ] `FollowPathInt` - [ ] `FollowPathString` - [ ] `PartialCmp` - [ ] `Operation` - [ ] `Dropped` - [ ] Document `notify_on_drop` optional field in `PluginCustomValue` |
||
|
Ian Manske
|
26786a759e
|
Fix ignored clippy lints (#12160)
# Description Fixes some ignored clippy lints. # User-Facing Changes Changes some signatures and return types to `&dyn Command` instead of `&Box<dyn Command`, but I believe this is only an internal change. |
||
|
Stefan Holderbach
|
77379d7b3d
|
Remove outdated doccomment on EngineState (#12158)
Part of the doccomment was an implementation note on the `im` crate that hasn't been used for ages. (If I recall we maybe even received a comment on discord on this) |
||
|
Yash Thakur
|
f6853fd636
|
Use XDG_CONFIG_HOME before default config directory (#12118)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Closes #12103 # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> As described in #12103, this PR makes Nushell use `XDG_CONFIG_HOME` as the config directory if it exists. Otherwise, it uses the old behavior, which was to use `dirs_next::config_dir()`. Edit: We discussed choosing between `XDG_CONFIG_HOME` and the default config directory in Discord and decided against it, at least for now. <s>@kubouch also suggested letting users choose between `XDG_CONFIG_HOME` and the default config directory if config files aren't found on startup and `XDG_CONFIG_HOME` is set to a value different from the default config directory</s> On Windows and MacOS, if the `XDG_CONFIG_HOME` variable is set but `XDG_CONFIG_HOME` is either empty or doesn't exist *and* the old config directory is non-empty, Nushell will issue a warning on startup saying that it won't move files from the old config directory to the new one. To do this, I had to add a `nu_path::config_dir_old()` function. I assume that at some point, we will remove the warning message and the function can be removed too. Alternatively, instead of having that function there, `main.rs` could directly call `dirs_next::config_dir()`. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> When `$env.XDG_CONFIG_HOME` is set to an absolute path, Nushell will use `$"($env.XDG_CONFIG_HOME)/nushell"` as its config directory (previously, this only worked on Linux). To use `App Data\Roaming` (Windows) or `Library/Application Support` (MacOS) instead (the old behavior), one can either leave `XDG_CONFIG_HOME` unset or set it to an empty string. If `XDG_CONFIG_HOME` is set, but to a non-absolute/invalid path, Nushell will report an error on startup and use the default config directory instead:  On Windows and MacOS, if the `XDG_CONFIG_HOME` variable is set but `XDG_CONFIG_HOME` is either empty or doesn't exist *and* the old config directory is non-empty, Nushell will issue a warning on startup saying that it won't move files from the old config directory to the new one.  # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> The existing config path tests have been modified to use `XDG_CONFIG_HOME` to change the config directory on all OSes, not just Linux. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> The documentation will have to be updated to note that Nushell uses `XDG_CONFIG_HOME` now. As @fdncred pointed out, it's possible for people to set `XDG_CONFIG_HOME` to, say, `~/.config/nushell` rather than `~/.config`, so the documentation could warn about that mistake. |
||
|
Stefan Holderbach
|
27edef4874
|
Bump reedline to dev (and strum) (#12150)
Resolve version duplication around `strum(_macros)` - Pull recent reedline (`strum` update) - Update `strum` in `nu-protocol` |
||
|
Stefan Holderbach
|
f695ba408a
|
Restructure nu-protocol in more meaningful units (#11917)
This is partially "feng-shui programming" of moving things to new separate places. The later commits include "`git blame` tollbooths" by moving out chunks of code into new files, which requires an extra step to track things with `git blame`. We can negiotiate if you want to keep particular things in their original place. If egregious I tried to add a bit of documentation. If I see something that is unused/unnecessarily `pub` I will try to remove that. - Move `nu_protocol::Exportable` to `nu-parser` - Guess doccomment for `Exportable` - Move `Unit` enum from `value` to `AST` - Move engine state `Variable` def into its folder - Move error-related files in `nu-protocol` subdir - Move `pipeline_data` module into its own folder - Move `stream.rs` over into the `pipeline_data` mod - Move `PipelineMetadata` into its own file - Doccomment `PipelineMetadata` - Remove unused `is_leap_year` in `value/mod` - Note about criminal `type_compatible` helper - Move duration fmting into new `value/duration.rs` - Move filesize fmting logic to new `value/filesize` - Split reexports from standard imports in `value/mod` - Doccomment trait `CustomValue` - Polish doccomments and intradoc links |
||
|
Devyn Cairns
|
bc19be25b1
|
Keep plugins persistently running in the background (#12064)
# Description
This PR uses the new plugin protocol to intelligently keep plugin
processes running in the background for further plugin calls.
Running plugins can be seen by running the new `plugin list` command,
and stopped by running the new `plugin stop` command.
This is an enhancement for the performance of plugins, as starting new
plugin processes has overhead, especially for plugins in languages that
take a significant amount of time on startup. It also enables plugins
that have persistent state between commands, making the migration of
features like dataframes and `stor` to plugins possible.
Plugins are automatically stopped by the new plugin garbage collector,
configurable with `$env.config.plugin_gc`:
```nushell
$env.config.plugin_gc = {
# Configuration for plugin garbage collection
default: {
enabled: true # true to enable stopping of inactive plugins
stop_after: 10sec # how long to wait after a plugin is inactive to stop it
}
plugins: {
# alternate configuration for specific plugins, by name, for example:
#
# gstat: {
# enabled: false
# }
}
}
```
If garbage collection is enabled, plugins will be stopped after
`stop_after` passes after they were last active. Plugins are counted as
inactive if they have no running plugin calls. Reading the stream from
the response of a plugin call is still considered to be activity, but if
a plugin holds on to a stream but the call ends without an active
streaming response, it is not counted as active even if it is reading
it. Plugins can explicitly disable the GC as appropriate with
`engine.set_gc_disabled(true)`.
The `version` command now lists plugin names rather than plugin
commands. The list of plugin commands is accessible via `plugin list`.
Recommend doing this together with #12029, because it will likely force
plugin developers to do the right thing with mutability and lead to less
unexpected behavior when running plugins nested / in parallel.
# User-Facing Changes
- new command: `plugin list`
- new command: `plugin stop`
- changed command: `version` (now lists plugin names, rather than
commands)
- new config: `$env.config.plugin_gc`
- Plugins will keep running and be reused, at least for the configured
GC period
- Plugins that used mutable state in weird ways like `inc` did might
misbehave until fixed
- Plugins can disable GC if they need to
- Had to change plugin signature to accept `&EngineInterface` so that
the GC disable feature works. #12029 does this anyway, and I'm expecting
(resolvable) conflicts with that
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
Because there is some specific OS behavior required for plugins to not
respond to Ctrl-C directly, I've developed against and tested on both
Linux and Windows to ensure that works properly.
# After Submitting
I think this probably needs to be in the book somewhere
|
||
|
Devyn Cairns
|
430fb1fcb6
|
Add support for engine calls from plugins (#12029)
# Description This allows plugins to make calls back to the engine to get config, evaluate closures, and do other things that must be done within the engine process. Engine calls can both produce and consume streams as necessary. Closures passed to plugins can both accept stream input and produce stream output sent back to the plugin. Engine calls referring to a plugin call's context can be processed as long either the response hasn't been received, or the response created streams that haven't ended yet. This is a breaking API change for plugins. There are some pretty major changes to the interface that plugins must implement, including: 1. Plugins now run with `&self` and must be `Sync`. Executing multiple plugin calls in parallel is supported, and there's a chance that a closure passed to a plugin could invoke the same plugin. Supporting state across plugin invocations is left up to the plugin author to do in whichever way they feel best, but the plugin object itself is still shared. Even though the engine doesn't run multiple plugin calls through the same process yet, I still considered it important to break the API in this way at this stage. We might want to consider an optional threadpool feature for performance. 2. Plugins take a reference to `EngineInterface`, which can be cloned. This interface allows plugins to make calls back to the engine, including for getting config and running closures. 3. Plugins no longer take the `config` parameter. This can be accessed from the interface via the `.get_plugin_config()` engine call. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Not only does this have plugin protocol changes, it will require plugins to make some code changes before they will work again. But on the plus side, the engine call feature is extensible, and we can add more things to it as needed. Plugin maintainers will have to change the trait signature at the very least. If they were using `config`, they will have to call `engine.get_plugin_config()` instead. If they were using the mutable reference to the plugin, they will have to come up with some strategy to work around it (for example, for `Inc` I just cloned it). This shouldn't be such a big deal at the moment as it's not like plugins have ever run as daemons with persistent state in the past, and they don't in this PR either. But I thought it was important to make the change before we support plugins as daemons, as an exclusive mutable reference is not compatible with parallel plugin calls. I suggest this gets merged sometime *after* the current pending release, so that we have some time to adjust to the previous plugin protocol changes that don't require code changes before making ones that do. # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting I will document the additional protocol features (`EngineCall`, `EngineCallResponse`), and constraints on plugin call processing if engine calls are used - basically, to be aware that an engine call could result in a nested plugin call, so the plugin should be able to handle that. |
||
Raphael Gaschignard
|
d8f13b36b1
|
Allow for stacks to have parents (#11654)
This is another attempt on #11288 This allows for a `Stack` to have a parent stack (behind an `Arc`). This is being added to avoid constant stack copying in REPL code. Concretely the following changes are included here: - `Stack` can now have a `parent_stack`, pointing to another stack - variable lookups can fallback to this parent stack (env vars and everything else is still copied) - REPL code has been reworked so that we use parenting rather than cloning. A REPL-code-specific trait helps to ensure that we do not accidentally trigger a full clone of the main stack - A property test has been added to make sure that parenting "looks the same" as cloning for consumers of `Stack` objects --------- Co-authored-by: Raphael Gaschignard <rtpg@rokkenjima.local> Co-authored-by: Ian Manske <ian.manske@pm.me> |
||
Jakub Žádník
|
14d1c67863
|
Debugger experiments (#11441)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR adds a new evaluator path with callbacks to a mutable trait object implementing a Debugger trait. The trait object can do anything, e.g., profiling, code coverage, step debugging. Currently, entering/leaving a block and a pipeline element is marked with callbacks, but more callbacks can be added as necessary. Not all callbacks need to be used by all debuggers; unused ones are simply empty calls. A simple profiler is implemented as a proof of concept. The debugging support is implementing by making `eval_xxx()` functions generic depending on whether we're debugging or not. This has zero computational overhead, but makes the binary slightly larger (see benchmarks below). `eval_xxx()` variants called from commands (like `eval_block_with_early_return()` in `each`) are chosen with a dynamic dispatch for two reasons: to not grow the binary size due to duplicating the code of many commands, and for the fact that it isn't possible because it would make Command trait objects object-unsafe. In the future, I hope it will be possible to allow plugin callbacks such that users would be able to implement their profiler plugins instead of having to recompile Nushell. [DAP](https://microsoft.github.io/debug-adapter-protocol/) would also be interesting to explore. Try `help debug profile`. ## Screenshots Basic output:  To profile with more granularity, increase the profiler depth (you'll see that repeated `is-windows` calls take a large chunk of total time, making it a good candidate for optimizing):  ## Benchmarks ### Binary size Binary size increase vs. main: **+40360 bytes**. _(Both built with `--release --features=extra,dataframe`.)_ ### Time ```nushell # bench_debug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'debug:' let res2 = bench { debug profile $test } --pretty print $res2 ``` ```nushell # bench_nodebug.nu use std bench let test = { 1..100 | each { ls | each {|row| $row.name | str length } } | flatten | math avg } print 'no debug:' let res1 = bench { do $test } --pretty print $res1 ``` `cargo run --release -- bench_debug.nu` is consistently 1--2 ms slower than `cargo run --release -- bench_nodebug.nu` due to the collection overhead + gathering the report. This is expected. When gathering more stuff, the overhead is obviously higher. `cargo run --release -- bench_nodebug.nu` vs. `nu bench_nodebug.nu` I didn't measure any difference. Both benchmarks report times between 97 and 103 ms randomly, without one being consistently higher than the other. This suggests that at least in this particular case, when not running any debugger, there is no runtime overhead. ## API changes This PR adds a generic parameter to all `eval_xxx` functions that forces you to specify whether you use the debugger. You can resolve it in two ways: * Use a provided helper that will figure it out for you. If you wanted to use `eval_block(&engine_state, ...)`, call `let eval_block = get_eval_block(&engine_state); eval_block(&engine_state, ...)` * If you know you're in an evaluation path that doesn't need debugger support, call `eval_block::<WithoutDebug>(&engine_state, ...)` (this is the case of hooks, for example). I tried to add more explanation in the docstring of `debugger_trait.rs`. ## TODO - [x] Better profiler output to reduce spam of iterative commands like `each` - [x] Resolve `TODO: DEBUG` comments - [x] Resolve unwraps - [x] Add doc comments - [x] Add usage and extra usage for `debug profile`, explaining all columns # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Hopefully none. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Antoine Büsch
|
979a97c455
|
Introduce workspace dependencies (#12043)
# Description This PR introduces [workspaces dependencies](https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table). The advantages are: - a single place where dependency versions are declared - reduces the number of files to change when upgrading a dependency - reduces the risk of accidentally depending on 2 different versions of the same dependency I've only done a few so far. If this PR is accepted, I might continue and progressively do the rest. # User-Facing Changes N/A # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting N/A |
||
|
Ian Manske
|
dfe072fd30
|
Fix chrono deprecation warnings (#12091)
# Description Bumps `chrono` to 0.4.35 and fixes any deprecation warnings. |
||
|
Ian Manske
|
a18de999c2
|
Fix broken doc link (#12092)
Fixes a doc comment link in `Value::to_parsable_string`. |
||
|
Stefan Holderbach
|
e5f086cfb4
|
Bump version to 0.91.1 (#12085)
|
||
|
Stefan Holderbach
|
3016d7a64c
|
Bump version for 0.91.0 release (#12070) | ||
|
Yash Thakur
|
4cda183103
|
Canonicalize default-config-dir and plugin-path (#11999)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR makes sure `$nu.default-config-dir` and `$nu.plugin-path` are canonicalized. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> `$nu.default-config-dir` (and `$nu.plugin-path`) will now give canonical paths, with symlinks and whatnot resolved. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> I've added a couple of tests to check that even if the config folder and/or any of the config files within are symlinks, the `$nu.*` variables are properly canonicalized. These tests unfortunately only run on Linux and MacOS, because I couldn't figure out how to change the config directory on Windows. Also, given that they involve creating files, I'm not sure if they're excessive, so I could remove one or two of them. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Devyn Cairns
|
626d597527
|
Replace panics with errors in thread spawning (#12040)
# Description Replace panics with errors in thread spawning. Also adds `IntoSpanned` trait for easily constructing `Spanned`, and an implementation of `From<Spanned<std::io::Error>>` for `ShellError`, which is used to provide context for the error wherever there was a span conveniently available. In general this should make it more convenient to do the right thing with `std::io::Error` and always add a span to it when it's possible to do so. # User-Facing Changes Fewer panics! # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` |
||
Wind
|
387328fe73
|
Glob: don't allow implicit casting between glob and string (#11992)
# Description
As title, currently on latest main, nushell confused user if it allows
implicit casting between glob and string:
```nushell
let x = "*.txt"
def glob-test [g: glob] { open $g }
glob-test $x
```
It always expand the glob although `$x` is defined as a string.
This pr implements a solution from @kubouch :
> We could make it really strict and disallow all autocasting between
globs and strings because that's what's causing the "magic" confusion.
Then, modify all builtins that accept globs to accept oneof(glob,
string) and the rules would be that globs always expand and strings
never expand
# User-Facing Changes
After this pr, user needs to use `into glob` to invoke `glob-test`, if
user pass a string variable:
```nushell
let x = "*.txt"
def glob-test [g: glob] { open $g }
glob-test ($x | into glob)
```
Or else nushell will return an error.
```
3 │ glob-test $x
· ─┬
· ╰── can't convert string to glob
```
# Tests + Formatting
Done
# After Submitting
Nan
|
||
|
Yash Thakur
|
c0ff0f12f0
|
Add ConfigDirNotFound error (#11849)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Currently, there's multiple places that look for a config directory, and each of them has different error messages when it can't be found. This PR makes a `ConfigDirNotFound` error to standardize the error message for all of these cases. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Previously, the errors in `create_nu_constant()` would say which config file Nushell was trying to get when it couldn't find the config directory. Now it doesn't. However, I think that's fine, given that it doesn't matter whether it couldn't find the config directory while looking for `login.nu` or `env.nu`, it only matters that it couldn't find it. This is what the error looks like:  # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Antoine Stevan <44101798+amtoine@users.noreply.github.com> |
||
dependabot[bot]
|
123547444c
|
Bump strum_macros from 0.25.3 to 0.26.1 (#11979)
Bumps [strum_macros](https://github.com/Peternator7/strum) from 0.25.3 to 0.26.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/Peternator7/strum/releases">strum_macros's releases</a>.</em></p> <blockquote> <h2>v0.26.1</h2> <h2>0.26.1</h2> <ul> <li><a href="https://redirect.github.com/Peternator7/strum/pull/325">#325</a>: use <code>core</code> instead of <code>std</code> in VariantArray.</li> </ul> <h2>0.26.0</h2> <h3>Breaking Changes</h3> <ul> <li>The <code>EnumVariantNames</code> macro has been renamed <code>VariantNames</code>. The deprecation warning should steer you in the right direction for fixing the warning.</li> <li>The Iterator struct generated by EnumIter now has new bounds on it. This shouldn't break code unless you manually added the implementation in your code.</li> <li><code>Display</code> now supports format strings using named fields in the enum variant. This should be a no-op for most code. However, if you were outputting a string like <code>"Hello {field}"</code>, this will now be interpretted as a format string.</li> <li>EnumDiscriminant now inherits the repr and discriminant values from your main enum. This makes the discriminant type closer to a mirror of the original and that's always the goal.</li> </ul> <h3>New features</h3> <ul> <li> <p>The <code>VariantArray</code> macro has been added. This macro adds an associated constant <code>VARIANTS</code> to your enum. The constant is a <code>&'static [Self]</code> slice so that you can access all the variants of your enum. This only works on enums that only have unit variants.</p> <pre lang="rust"><code>use strum::VariantArray; <p>#[derive(Debug, VariantArray)] enum Color { Red, Blue, Green, }</p> <p>fn main() { println!("{:?}", Color::VARIANTS); // prints: ["Red", "Blue", "Green"] } </code></pre></p> </li> <li> <p>The <code>EnumTable</code> macro has been <em>experimentally</em> added. This macro adds a new type that stores an item for each variant of the enum. This is useful for storing a value for each variant of an enum. This is an experimental feature because I'm not convinced the current api surface area is correct.</p> <pre lang="rust"><code>use strum::EnumTable; <p>#[derive(Copy, Clone, Debug, EnumTable)] enum Color { Red, Blue, </code></pre></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Peternator7/strum/blob/master/CHANGELOG.md">strum_macros's changelog</a>.</em></p> <blockquote> <h2>0.26.1</h2> <ul> <li><a href="https://redirect.github.com/Peternator7/strum/pull/325">#325</a>: use <code>core</code> instead of <code>std</code> in VariantArray.</li> </ul> <h2>0.26.0</h2> <h3>Breaking Changes</h3> <ul> <li>The <code>EnumVariantNames</code> macro has been renamed <code>VariantNames</code>. The deprecation warning should steer you in the right direction for fixing the warning.</li> <li>The Iterator struct generated by EnumIter now has new bounds on it. This shouldn't break code unless you manually added the implementation in your code.</li> <li><code>Display</code> now supports format strings using named fields in the enum variant. This should be a no-op for most code. However, if you were outputting a string like <code>"Hello {field}"</code>, this will now be interpretted as a format string.</li> <li>EnumDiscriminant now inherits the repr and discriminant values from your main enum. This makes the discriminant type closer to a mirror of the original and that's always the goal.</li> </ul> <h3>New features</h3> <ul> <li> <p>The <code>VariantArray</code> macro has been added. This macro adds an associated constant <code>VARIANTS</code> to your enum. The constant is a <code>&'static [Self]</code> slice so that you can access all the variants of your enum. This only works on enums that only have unit variants.</p> <pre lang="rust"><code>use strum::VariantArray; <p>#[derive(Debug, VariantArray)] enum Color { Red, Blue, Green, }</p> <p>fn main() { println!("{:?}", Color::VARIANTS); // prints: ["Red", "Blue", "Green"] } </code></pre></p> </li> <li> <p>The <code>EnumTable</code> macro has been <em>experimentally</em> added. This macro adds a new type that stores an item for each variant of the enum. This is useful for storing a value for each variant of an enum. This is an experimental feature because I'm not convinced the current api surface area is correct.</p> <pre lang="rust"><code>use strum::EnumTable; <p>#[derive(Copy, Clone, Debug, EnumTable)] enum Color { Red, Blue, Green, </code></pre></p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/Peternator7/strum/commits/v0.26.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Devyn Cairns
|
88f1f386bb
|
Bidirectional communication and streams for plugins (#11911) | ||
|
Devyn Cairns
|
461f69ac5d
|
Rename spans in the serialized form of Value (#11972)
[Discord context](https://discord.com/channels/601130461678272522/615962413203718156/1211158641793695744) <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Span fields were previously renamed to `internal_span` to discourage their use in Rust code, but this change also affected the serde I/O for Value. I don't believe the Python plugin was ever updated to reflect this change. This effectively changes it back, but just for the serialized form. There are good reasons for doing this: 1. `internal_span` is a much longer name, and would be one of the most common strings found in serialized Value data, probably bulking up the plugin I/O 2. This change was never really meant to have implications for plugins, and was just meant to be a hint that `.span()` should be used instead in Rust code. When Span refactoring is complete, the serialized form of Value will probably change again in some significant way, so I think for now it's best that it's left like this. This has implications for #11911, particularly for documentation and for the Python plugin as that was already updated in that PR to reflect `internal_span`. If this is merged first, I will update that PR. This would probably be considered a breaking change as it would break plugin I/O compatibility (but not Rust code). I think it can probably go in any major release though - all things considered, it's pretty minor, and users are already expected to recompile plugins for new major versions. However, it may also be worth holding off to do it together with #11911 as that PR makes breaking changes in general a little bit friendlier. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Requires plugin recompile. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` Nothing outside of `Value` itself had to be changed to make tests pass. I did not check the Python plugin and whether it works now, but it was broken before. It may work again as I think the main incompatibility it had was expecting to use `span` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Stefan Holderbach
|
7884de1941
|
Remove some unnecessary static Vecs (#11947)
Avoid unnecessary allocations or larger iterator structs - Turn static `Vec`s into arrays when possible - Use `std::iter::once`/`empty` where applicable - Use `bool::then_some` in `detect column` `.chain` - Drop in the bucket: de-vec-ing tests |
||
|
Devyn Cairns
|
098527b263
|
Print stderr streams to stderr in pipeline_data::print_if_stream() (#11929)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Related to #11928 - `tee --stderr` doesn't really work as expected without it # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Print stderr streams to stderr in `pipeline_data::print_if_stream()` This corrects unexpected behavior if a stream from an external program is transformed while still preserving its stderr output. Before this change, that output is just drained and discarded. Worse, it's drained to a buffer, which could be really slow and memory hungry if there's a lot of output on stderr. This is needed to make `tee --stderr` function in a non-surprising way. See #11928 # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> A script that was erroneously not producing stderr output before might now, but I can't think of a lot of examples of an external stream being transformed without being converted. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Wind
|
f7d647ac3c
|
open, rm, umv, cp, rm and du: Don't globs if inputs are variables or string interpolation (#11886)
# Description
This is a follow up to
https://github.com/nushell/nushell/pull/11621#issuecomment-1937484322
Also Fixes: #11838
## About the code change
It applys the same logic when we pass variables to external commands:
|
||
|
Devyn Cairns
|
28f58057b6
|
Replace debug_assert! with assert! in Signature::check_names (#11937)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Debug assertions don't run at release, which means that `cargo test --release` fails because the tests for name checks don't run properly. These checks are not really expensive, and there shouldn't be any noticeable difference to startup time, so there isn't much reason not to just leave them in. It's valuable to be able to run `cargo test --release`, as that can expose race conditions and dependencies on undefined behavior that aren't exposed in debug builds. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> This shouldn't affect anything. Any violations of this rule were being caught with debug tests, which are run by the CI. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Jack Wright
|
f17f857b1f
|
wrapping run_repl with catch_unwind and restarting the repl on panic (#11860)
Provides the ability to cleanly recover from panics, falling back to the last known good state of EngineState and Stack. This pull request also utilizes miette's panic handler for better formatting of panics. <img width="642" alt="Screenshot 2024-02-21 at 08 34 35" src="https://github.com/nushell/nushell/assets/56345/f81efaba-aa45-4e47-991c-1a2cf99e06ff"> --------- Co-authored-by: Jack Wright <jack.wright@disqo.com> |
||
|
Yash Thakur
|
6ff3a4180b
|
Specify which file not found in error (#11868)
# Description Currently, `ShellError::FileNotFound` shows the span where the error occurred but doesn't say which file wasn't found. This PR makes it so the help includes that (like the `DirectoryNotFound` error). # User-Facing Changes No breaking changes, it's just that when a file can't be found, the help will say which file couldn't be found:  |
||
|
Stefan Holderbach
|
6e590fe0a2
|
Remove unused Index(Mut) impls on AST types (#11903)
# Description Both `Block` and `Pipeline` had `Index`/`IndexMut` implementations to access their elements, that are currently unused. Explicit helpers or iteration would generally be preferred anyways but in the current state the inner containers are `pub` and are liberally used. (Sometimes with potentially panicking indexing or also iteration) As it is potentially unclear what the meaning of the element from a block or pipeline queried by a usize is, let's remove it entirely until we come up with a better API. # User-Facing Changes None Plugin authors shouldn't dig into AST internals |
||
David Matos
|
123bf2d736
|
fix format date based on users locale (#11908)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description Hi, Fixes #10838, where before the `date` would be formatted incorrectly, and was not picking `LC_TIME` for time formatting, but it picked the first locale returned by the `sys-locale` crate instead. Now it will format time based on `LC_TIME`. For example, ``` // my locale `nl_NL.UTF-8` ❯ date now | format date '%x %X' 20-02-24 17:17:12 $env.LC_TIME = "en_US.UTF-8" ❯ date now | format date '%x %X' 02/20/2024 05:16:28 PM ``` Note that I also changed the `default_env.nu` as otherwise the Time will show AM/PM twice. Also reason for the `chrono` update is because this relies on a fix to upstream repo, which i initially submitted an [issue](https://github.com/chronotope/chrono/issues/1349#event-11765363286) <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - [X] `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - [X] `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - [X] `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - [X] `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
dependabot[bot]
|
a4ef7c1ac4
|
Bump fancy-regex from 0.12.0 to 0.13.0 (#11893)
[//]: # (dependabot-start) ⚠️ **Dependabot is rebasing this PR** ⚠️ Rebasing might not happen immediately, so don't worry if this takes some time. Note: if you make any changes to this PR yourself, they will take precedence over the rebase. --- [//]: # (dependabot-end) Bumps [fancy-regex](https://github.com/fancy-regex/fancy-regex) from 0.12.0 to 0.13.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/fancy-regex/fancy-regex/releases">fancy-regex's releases</a>.</em></p> <blockquote> <h2>0.13.0</h2> <h3>Added</h3> <ul> <li>Support for relative backreferences using <code>\k<-1></code> (-1 references the previous group) (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Add <code>try_replacen</code> to <code>Regex</code> which returns a <code>Result</code> instead of panicking when matching errors (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/130">#130</a>)</li> </ul> <h3>Changed</h3> <ul> <li>Switch from regex crate to regex-automata and regex-syntax (lower level APIs) to simplify internals (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Allow escaping some letters in character classes, e.g. <code>[\A]</code> used to error but now matches the same as <code>[A]</code> (for compatibility with Oniguruma)</li> <li>MSRV (minimum supported Rust version) is now 1.66.1 (from 1.61.0)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix index out of bounds panic when parsing unclosed <code>(?(</code> (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/125">#125</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/fancy-regex/fancy-regex/blob/main/CHANGELOG.md">fancy-regex's changelog</a>.</em></p> <blockquote> <h2>[0.13.0] - 2023-12-22</h2> <h3>Added</h3> <ul> <li>Support for relative backreferences using <code>\k<-1></code> (-1 references the previous group) (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Add <code>try_replacen</code> to <code>Regex</code> which returns a <code>Result</code> instead of panicking when matching errors (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/130">#130</a>)</li> </ul> <h3>Changed</h3> <ul> <li>Switch from regex crate to regex-automata and regex-syntax (lower level APIs) to simplify internals (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Allow escaping some letters in character classes, e.g. <code>[\A]</code> used to error but now matches the same as <code>[A]</code> (for compatibility with Oniguruma)</li> <li>MSRV (minimum supported Rust version) is now 1.66.1 (from 1.61.0)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix index out of bounds panic when parsing unclosed <code>(?(</code> (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/125">#125</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6a1691f378
|
Bump miette from 7.0.0 to 7.1.0 (#11892)
Bumps [miette](https://github.com/zkat/miette) from 7.0.0 to 7.1.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/zkat/miette/releases">miette's releases</a>.</em></p> <blockquote> <h2>v7.1.0</h2> <h3>Features</h3> <ul> <li><strong>derive:</strong> enable more boxed types to be #[diagnostic_source] (<a href="https://redirect.github.com/zkat/miette/issues/338">#338</a>) (<a href=" |
||
|
Ian Manske
|
68fcd71898
|
Add Value::coerce_str (#11885)
# Description Following #11851, this PR adds one final conversion function for `Value`. `Value::coerce_str` takes a `&Value` and converts it to a `Cow<str>`, creating an owned `String` for types that needed converting. Otherwise, it returns a borrowed `str` for `String` and `Binary` `Value`s which avoids a clone/allocation. Where possible, `coerce_str` and `coerce_into_string` should be used instead of `coerce_string`, since `coerce_string` always allocates a new `String`. |
||
|
Ian Manske
|
fb4251aba7
|
Remove Record::from_raw_cols_vals_unchecked (#11810)
# Description Follows from #11718 and replaces all usages of `Record::from_raw_cols_vals_unchecked` with iterator or `record!` equivalents. |
||
|
Stefan Holderbach
|
28f0f32ae7
|
Prune unused ShellError variants (#11883)
# Description Same procedure as #11881 Remove unused variants to avoid confusion and foster better practices around error variants. - Remove `SE::PermissionDeniedError` - Remove `SE::OutOfMemoryError` - Remove `SE::DirectoryNotFoundCustom` - Remove `SE::MoveNotPossibleSingle` - Remove `SE::NonUnicodeInput` # User-Facing Changes Plugin authors may have matched against or emitted those variants |
||
|
Stefan Holderbach
|
06c590d894
|
Prune unused ParseError variants (#11881)
# Description Error variants never raised should be removed to avoid confusion and make sure we choose the proper variants in the future - Remove unused `ParseError::InvalidModuleFileName` - Remove unused `ParseError::NotFound` - Remove unused `ParseError::MissingImportPattern` - Remove unused `ParseError::ReadingFile` # User-Facing Changes None for users. Insignificant for plugin authors as they interact only with `ShellError` |
||
|
Ian Manske
|
1c49ca503a
|
Name the Value conversion functions more clearly (#11851)
# Description This PR renames the conversion functions on `Value` to be more consistent. It follows the Rust [API guidelines](https://rust-lang.github.io/api-guidelines/naming.html#ad-hoc-conversions-follow-as_-to_-into_-conventions-c-conv) for ad-hoc conversions. The conversion functions on `Value` now come in a few forms: - `coerce_{type}` takes a `&Value` and attempts to convert the value to `type` (e.g., `i64` are converted to `f64`). This is the old behavior of some of the `as_{type}` functions -- these functions have simply been renamed to better reflect what they do. - The new `as_{type}` functions take a `&Value` and returns an `Ok` result only if the value is of `type` (no conversion is attempted). The returned value will be borrowed if `type` is non-`Copy`, otherwise an owned value is returned. - `into_{type}` exists for non-`Copy` types, but otherwise does not attempt conversion just like `as_type`. It takes an owned `Value` and always returns an owned result. - `coerce_into_{type}` has the same relationship with `coerce_{type}` as `into_{type}` does with `as_{type}`. - `to_{kind}_string`: conversion to different string formats (debug, abbreviated, etc.). Only two of the old string conversion functions were removed, the rest have been renamed only. - `to_{type}`: other conversion functions. Currently, only `to_path` exists. (And `to_string` through `Display`.) This table summaries the above: | Form | Cost | Input Ownership | Output Ownership | Converts `Value` case/`type` | | ---------------------------- | ----- | --------------- | ---------------- | -------- | | `as_{type}` | Cheap | Borrowed | Borrowed/Owned | No | | `into_{type}` | Cheap | Owned | Owned | No | | `coerce_{type}` | Cheap | Borrowed | Borrowed/Owned | Yes | | `coerce_into_{type}` | Cheap | Owned | Owned | Yes | | `to_{kind}_string` | Expensive | Borrowed | Owned | Yes | | `to_{type}` | Expensive | Borrowed | Owned | Yes | # User-Facing Changes Breaking API change for `Value` in `nu-protocol` which is exposed as part of the plugin API. |
||
|
Yash Thakur
|
cb67de675e
|
Disallow spreading lists automatically when calling externals (#11857)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> Spreading lists automatically when calling externals was deprecated in 0.89 (#11289), and this PR is to remove it in 0.91. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> The new error message looks like this: ``` > ^echo [1 2] Error: nu:🐚:cannot_pass_list_to_external × Lists are not automatically spread when calling external commands ╭─[entry #13:1:8] 1 │ ^echo [1 2] · ──┬── · ╰── Spread operator (...) is necessary to spread lists ╰──── help: Either convert the list to a string or use the spread operator, like so: ...[1 2] ``` The old error message didn't say exactly where to put the `...` and seemed to confuse a lot of people, so hopefully this helps. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> There was one test to check that implicit spread was deprecated before, updated that to check that it's disallowed now. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Darren Schroeder
|
a603b067e5
|
update default_config with new defaults (#11856)
# Description Update a few defaults. 1. use_ls_colors_completeions defaults to true. 2. make ide_menu only offer 10 completions at a time with `max_completion_height = 10` instead of taking the entire screen. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Steven
|
5042f19d1b
|
colored file-like completions (#11702)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> `ls` and other file completions uses `LS_COLORS`.  # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com> |
||
|
Wind
|
58c6fea60b
|
Support redirect stderr and stdout+stderr with a pipe (#11708)
# Description Close: #9673 Close: #8277 Close: #10944 This pr introduces the following syntax: 1. `e>|`, pipe stderr to next command. Example: `$env.FOO=bar nu --testbin echo_env_stderr FOO e>| str length` 2. `o+e>|` and `e+o>|`, pipe both stdout and stderr to next command, example: `$env.FOO=bar nu --testbin echo_env_mixed out-err FOO FOO e+o>| str length` Note: it only works for external commands. ~There is no different for internal commands, that is, the following three commands do the same things:~ Edit: it raises errors if we want to pipes for internal commands ``` ❯ ls e>| str length Error: × `e>|` only works with external streams ╭─[entry #1:1:1] 1 │ ls e>| str length · ─┬─ · ╰── `e>|` only works on external streams ╰──── ❯ ls e+o>| str length Error: × `o+e>|` only works with external streams ╭─[entry #2:1:1] 1 │ ls e+o>| str length · ──┬── · ╰── `o+e>|` only works on external streams ╰──── ``` This can help us to avoid some strange issues like the following: `$env.FOO=bar (nu --testbin echo_env_stderr FOO) e>| str length` Which is hard to understand and hard to explain to users. # User-Facing Changes Nan # Tests + Formatting To be done # After Submitting Maybe update documentation about these syntax. |
||
|
dependabot[bot]
|
e7f1bf8535
|
Bump indexmap from 2.1.0 to 2.2.2 (#11746) |