# Description
This PR does a few things to help improve type hovers and, in the
process, fixes a few outstanding issues in the type system. Here's a
list of the changes:
* `for` now will try to infer the type of the iteration variable based
on the expression it's given. This fixes things like `for x in [1, 2, 3]
{ }` where `x` now properly gets the int type.
* Removed old input/output type fields from the signature, focuses on
the vec of signatures. Updated a bunch of dataframe commands that hadn't
moved over. This helps tie things together a bit better
* Fixed inference of types from subexpressions to use the last
expression in the block
* Fixed handling of explicit types in `let` and `mut` calls, so we now
respect that as the authoritative type
I also tried to add `def` input/output type inference, but unfortunately
we only know the predecl types universally, which means we won't have
enough information to properly know what the types of the custom
commands are.
# User-Facing Changes
Script typechecking will get tighter in some cases
Hovers should be more accurate in some cases that previously resorted to
any.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
---------
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
This PR reverts https://github.com/nushell/nushell/pull/9391
We try not to revert PRs like this, though after discussion with the
Nushell team, we decided to revert this one.
The main reason is that Nushell, as a codebase, isn't ready for these
kinds of optimisations. It's in the part of the development cycle where
our main focus should be on improving the algorithms inside of Nushell
itself. Once we have matured our algorithms, then we can look for
opportunities to switch out technologies we're using for alternate
forms.
Much of Nushell still has lots of opportunities for tuning the codebase,
paying down technical debt, and making the codebase generally cleaner
and more robust. This should be the focus. Performance improvements
should flow out of that work.
Said another, optimisation that isn't part of tuning the codebase is
premature at this stage. We need to focus on doing the hard work of
making the engine, parser, etc better.
# User-Facing Changes
Reverts the HashMap -> ahash change.
cc @FilipAndersson245
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx
you can also mention related issues, PRs or discussions!
-->
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
Added comments to support API docs for the `nu-plugin` crate. Removed a
few items that I'd expect should only be used internally to Nushell from
the documentation and reduced the visibility of some items that did not
need to be public.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
There should be no user facing impact.
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
Standard tests run. Additionally numerous doctests were added to the
`nu-plugin` crate.
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
No changes to the website necessary.
# Description
see https://github.com/nushell/nushell/issues/9390
using `ahash` instead of the default hasher. this will not affect
compile time as we where already building `ahash`.
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
(*third* try at posting this PR, #9104, like #9084, got polluted with
unrelated commits. I'm never going to pull from the github feature
branch again!)
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.
Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
Show parameter defaults in scope command signature, where they're
available for display by help.
per https://github.com/nushell/nushell/issues/8928.
I found unexpected ramifications in one completer (NuHelpCompleter) and
plugins, which both use the flag-formatting routine from builtin help.
For the moment I made the minimum necessary changes to get the mainline
scenario to pass tests and run. But we should circle back on what to do
with plugins and help completer..
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
1. New `parameter_default` column to `signatures` table in
`$nu.scope.commands`
It is populated with whatever parameters can be defaulted: currently
positional args and named flags.
2. Built in help (both `help <command>` and `<command> --help` will
display the defaults
3. Help completer will display defaults for flags, but not for
positionals.
Example:
A custom command with some default parameters:
```
〉cat ~/work/dflts.nu
# sample function to show defaults in help
export def main [
arg1: string # mandatory positional
arg2:string=abc # optional positional
--switch # no default here
--named:int # named flag, no default
--other:string=def # flag
--hard:record<foo:int bar:string, bas:bool> # default can be compound type
= {foo:22, bar:"other worlds", bas:false}
] { {arg1: $arg1,
arg2: $arg2,
switch: $switch,
named: $named,
other: $other,
hard: $hard, }
}
〉use ~/work/dflts.nu
〉$nu.scope.commands | where name == 'dflts' | get signatures.0.any | reject short_flag description custom_completion
╭───┬────────────────┬────────────────┬──────────────────────────────────────────┬─────────────┬───────────────────────────╮
│ # │ parameter_name │ parameter_type │ syntax_shape │ is_optional │ parameter_default │
├───┼────────────────┼────────────────┼──────────────────────────────────────────┼─────────────┼───────────────────────────┤
│ 0 │ │ input │ any │ false │ │
│ 1 │ arg1 │ positional │ string │ false │ │
│ 2 │ arg2 │ positional │ string │ true │ abc │
│ 3 │ switch │ switch │ │ true │ │
│ 4 │ named │ named │ int │ true │ │
│ 5 │ other │ named │ string │ true │ def │
│ 6 │ hard │ named │ record<foo: int, bar: string, bas: bool> │ true │ ╭───────┬───────────────╮ │
│ │ │ │ │ │ │ foo │ 22 │ │
│ │ │ │ │ │ │ bar │ other worlds │ │
│ │ │ │ │ │ │ bas │ false │ │
│ │ │ │ │ │ ╰───────┴───────────────╯ │
│ 7 │ │ output │ any │ false │ │
╰───┴────────────────┴────────────────┴──────────────────────────────────────────┴─────────────┴───────────────────────────╯
〉help dflts
sample function to show defaults in help
Usage:
> dflts {flags} <arg1> (arg2)
Flags:
--switch - switch -- no default here
--named <Int> - named flag, typed, but no default
--other <String> - flag with default (default: 'def')
--hard <Record([("foo", Int), ("bar", String), ("bas", Boolean)])> - default can be compound type (default: {foo: 22, bar: 'other worlds', bas: false})
-h, --help - Display the help message for this command
Parameters:
arg1 <string>: mandatory positional
arg2 <string>: optional positional (optional, default: 'abc')
```
Compared to (relevant bits of) help output previously:
```
Flags:
-h, --help - Display the help message for this command
-, --switch - no default here
-, --named <int> - named flag, no default
-, --other <string> - flag
-, --hard <record<foo: int, bar: string, bas: bool>> - default can be compound type
Signatures:
<any> | dflts <string> <string> -> <any>
Parameters:
arg1 <string>: mandatory positional
(optional) arg2 <string>: optional positional
```
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> [x] toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
Continuation of #8229 and #8326
# Description
The `ShellError` enum at the moment is kind of messy.
Many variants are basic tuple structs where you always have to reference
the implementation with its macro invocation to know which field serves
which purpose.
Furthermore we have both variants that are kind of redundant or either
overly broad to be useful for the user to match on or overly specific
with few uses.
So I set out to start fixing the lacking documentation and naming to
make it feasible to critically review the individual usages and fix
those.
Furthermore we can decide to join or split up variants that don't seem
to be fit for purpose.
# Call to action
**Everyone:** Feel free to add review comments if you spot inconsistent
use of `ShellError` variants.
# User-Facing Changes
(None now, end goal more explicit and consistent error messages)
# Tests + Formatting
(No additional tests needed so far)
# Commits (so far)
- Remove `ShellError::FeatureNotEnabled`
- Name fields on `SE::ExternalNotSupported`
- Name field on `SE::InvalidProbability`
- Name fields on `SE::NushellFailed` variants
- Remove unused `SE::NushellFailedSpannedHelp`
- Name field on `SE::VariableNotFoundAtRuntime`
- Name fields on `SE::EnvVarNotFoundAtRuntime`
- Name fields on `SE::ModuleNotFoundAtRuntime`
- Remove usused `ModuleOrOverlayNotFoundAtRuntime`
- Name fields on `SE::OverlayNotFoundAtRuntime`
- Name field on `SE::NotFound`
# Description
As title, we can't provide examples for plugin commands, this pr would
make it possible
# User-Facing Changes
Take plugin `nu-example-1` as example:
```
❯ nu-example-1 -h
PluginSignature test 1 for plugin. Returns Value::Nothing
Usage:
> nu-example-1 {flags} <a> <b> (opt) ...(rest)
Flags:
-h, --help - Display the help message for this command
-f, --flag - a flag for the signature
-n, --named <String> - named string
Parameters:
a <int>: required integer value
b <string>: required string value
(optional) opt <int>: Optional number
...rest <string>: rest value string
Examples:
running example with an int value and string value
> nu-example-1 3 bb
```
The examples session is newly added.
## Basic idea behind these changes
when nushell query plugin signatures, plugin just returns it's signature
without any examples, so nushell have no idea about the examples of
plugin commands.
To adding the feature, we just making plugin returns it's signature with
examples.
Before:
```
1. get signature
---------------->
Nushell ------------------ Plugin
<-----------------
2. returns Vec<Signature>
```
After:
```
1. get signature
---------------->
Nushell ------------------ Plugin
<-----------------
2. returns Vec<PluginSignature>
```
When writing plugin signature to $nu.plugin-path:
Serialize `<PluginSignature>` rather than `<Signature>`, which would
enable us to serialize examples to `$nu.plugin-path`
## Shortcoming
It's a breaking changes because `Plugin::signature` is changed, and it
requires plugin authors to change their code for new signatures.
Fortunally it should be easy to change, for rust based plugin, we just
need to make a global replace from word `Signature` to word
`PluginSignature` in their plugin project.
Our content of plugin-path is really large, if one plugin have many
examples, it'd results to larger body of $nu.plugin-path, which is not
really scale. A solution would be save register information in other
binary formats rather than `json`. But I think it'd be another story.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
~~I happened to be reviewing our uses of `thread::spawn()` and came to
the conclusion that we're spawning a thread unnecessarily for plugin
calls. We were basically doing this:~~
~~1. Spawn a background thread to send data to the plugin over stdin~~
~~2. Immediately do a blocking wait for the plugin's response~~
~~As far as I can tell, there's no point in spawning a thread for 1 (and
it may harm error handling) given that we're blocking right away for the
response.~~
**Update:** the logic is correct, as confirmed by @WindSoilder
[here](https://discord.com/channels/601130461678272522/855947301380947968/1072743414795350037).
I've added a comment explaining the thread usage.
# Description
Lint: `clippy::uninlined_format_args`
More readable in most situations.
(May be slightly confusing for modifier format strings
https://doc.rust-lang.org/std/fmt/index.html#formatting-parameters)
Alternative to #7865
# User-Facing Changes
None intended
# Tests + Formatting
(Ran `cargo +stable clippy --fix --workspace -- -A clippy::all -D
clippy::uninlined_format_args` to achieve this. Depends on Rust `1.67`)
This is an attempt to implement a new `Value::LazyRecord` variant for
performance reasons.
`LazyRecord` is like a regular `Record`, but it's possible to access
individual columns without evaluating other columns. I've implemented
`LazyRecord` for the special `$nu` variable; accessing `$nu` is
relatively slow because of all the information in `scope`, and [`$nu`
accounts for about 2/3 of Nu's startup time on
Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122).
### Benchmarks
I ran some benchmarks on my desktop (Linux, 12900K) and the results are
very pleasing.
Nu's time to start up and run a command (`cargo build --release;
hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none
--warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**.
Tests are also much faster! Running `cargo nextest` (with our very slow
`proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**,
because most tests involve launching a new instance of Nu.
### Design (updated)
I've added a new `LazyRecord` trait and added a `Value` variant wrapping
those trait objects, much like `CustomValue`. `LazyRecord`
implementations must implement these 2 functions:
```rust
// All column names
fn column_names(&self) -> Vec<&'static str>;
// Get 1 specific column value
fn get_column_value(&self, column: &str) -> Result<Value, ShellError>;
```
### Serializability
`Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system:
1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable
2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]`
3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
I have changed `assert!(a == b)` calls to `assert_eq!(a, b)`, which give
better error messages. Similarly for `assert!(a != b)` and
`assert_ne!(a, b)`. Basically all instances were comparing primitives
(string slices or integers), so there is no loss of generality from
special-case macros,
I have also fixed a number of typos in comments, variable names, and a
few user-facing messages.
Also enforce this by #[non_exhaustive] span such that going forward we
cannot, in debug builds (1), construct invalid spans.
The motivation for this stems from #6431 where I've seen crashes due to
invalid slice indexing.
My hope is this will mitigate such senarios
1. https://github.com/nushell/nushell/pull/6431#issuecomment-1278147241
# Description
(description of your pull request here)
# Tests
Make sure you've done the following:
- [ ] Add tests that cover your changes, either in the command examples,
the crate/tests folder, or in the /tests folder.
- [ ] Try to think about corner cases and various ways how your changes
could break. Cover them with tests.
- [ ] If adding tests is not possible, please document in the PR body a
minimal example with steps on how to reproduce so one can verify your
change works.
Make sure you've run and fixed any issues with these commands:
- [x] `cargo fmt --all -- --check` to check standard code formatting
(`cargo fmt --all` applies these changes)
- [ ] `cargo clippy --workspace --features=extra -- -D warnings -D
clippy::unwrap_used -A clippy::needless_collect` to check that you're
using the standard code style
- [ ] `cargo test --workspace --features=extra` to check that all the
tests pass
# Documentation
- [ ] If your PR touches a user-facing nushell feature then make sure
that there is an entry in the documentation
(https://github.com/nushell/nushell.github.io) for the feature, and
update it if necessary.
# Description
This makes the help messages cleaner for keyword-style arguments.
Before:
```
(optional) else_expression <Keyword([101, 108, 115, 101], Expression)>: expression or block to run if check fails
```
Now:
```
(optional) "else" + <expression>: expression or block to run if check fails
```
# User-Facing Changes
Changes how help is printed, so we use slightly different shape names
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
* when spawned process during register plugin, pass env to child process
* tweak comment
* tweak comment
* remove trailing whitespace
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
* Skeleton implementation
Lots and lots of TODOs
* Bootstrap simple CustomValue plugin support test
* Create nu_plugin_custom_value
* Skeleton for nu_plugin_custom_values
* Return a custom value from plugin
* Encode CustomValues from plugin calls as PluginResponse::PluginData
* Add new PluginCall variant CollapseCustomValue
* Handle CollapseCustomValue plugin calls
* Add CallInput::Data variant to CallInfo inputs
* Handle CallInfo with CallInput::Data plugin calls
* Send CallInput::Data if Value is PluginCustomValue from plugin calls
* Remove unnecessary boxing of plugins CallInfo
* Add fields needed to collapse PluginCustomValue to it
* Document PluginCustomValue and its purpose
* Impl collapsing using plugin calls in PluginCustomValue::to_base_value
* Implement proper typetag based deserialization for CoolCustomValue
* Test demonstrating that passing back a custom value to plugin works
* Added a failing test for describing plugin CustomValues
* Support describe for PluginCustomValues
- Add name to PluginResponse::PluginData
- Also turn it into a struct for clarity
- Add name to PluginCustomValue
- Return name field from PluginCustomValue
* Demonstrate that plugins can create and handle multiple CustomValues
* Add bincode to nu-plugin dependencies
This is for demonstration purposes, any schemaless binary seralization
format will work. I picked bincode since it's the most popular for Rust
but there are defintely better options out there for this usecase
* serde_json::Value -> Vec<u8>
* Update capnp schema for new CallInfo.input field
* Move call_input capnp serialization and deserialization into new file
* Deserialize Value's span from Value itself instead of passing call.head
I am not sure if this was correct and I am breaking it or if it was a
bug, I don't fully understand how nu creates and uses Spans. What should
reuse spans and what should recreate new ones?
But yeah it felt weird that the Value's Span was being ignored since in
the json serializer just uses the Value's Span
* Add call_info value round trip test
* Add capnp CallInput::Data serialization and deserialization support
* Add CallInfo::CollapseCustomValue to capnp schema
* Add capnp PluginCall::CollapseCustomValue serialization and deserialization support
* Add PluginResponse::PluginData to capnp schema
* Add capnp PluginResponse::PluginData serialization and deserialization support
* Switch plugins::custom_values tests to capnp
Both json and capnp would work now! Sadly I can't choose both at the
same time :(
* Add missing JsonSerializer round trip tests
* Handle plugin returning PluginData as a response to CollapseCustomValue
* Refactor plugin calling into a reusable function
Many less levels of indentation now!
* Export PluginData from nu_plugin
So plugins can create their very own serve_plugin with whatever
CustomValue behavior they may desire
* Error if CustomValue cannot be handled by Plugin
* Initial implementation of ordered call args
* Run cargo fmt
* Fix some clippy lints
* Add positional len and nth
* Cargo fmt
* Remove more old nth calls
* Good ole rustfmt
* Add named len
Co-authored-by: Hristo Filaretov <h.filaretov@protonmail.com>
* Add search terms to command
* Rename Signature desc to usage
To be named uniformly with extra_usage
* Throw in foldl search term for reduce
* Add missing usage to post
* Add search terms to signature
* Try to add capnp Signature serialization
* query command with json, web, xml
* query xml now working
* clippy
* comment out web tests
* Initial work on query web
For now we can query everything except tables
* Support for querying tables
Now we can query multiple tables just like before, now the only thing
missing is the test coverage
* finish off
* comment out web test
Co-authored-by: Luccas Mateus de Medeiros Gomes <luccasmmg@gmail.com>
