Commit Graph

8920 Commits

Author SHA1 Message Date
YizhePKU
7ede90cba5
Remove std::env::set_current_dir() call from EngineState::merge_env() (#12922)
As discussed in https://github.com/nushell/nushell/pull/12749, we no
longer need to call `std::env::set_current_dir()` to sync `$env.PWD`
with the actual working directory. This PR removes the call from
`EngineState::merge_env()`.
2024-05-22 19:58:27 +03:00
Jakub Žádník
75689ec98a
Small improvements to debug profile (#12930)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx

you can also mention related issues, PRs or discussions!
-->

# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.

Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->

1. With the `-l` flag, `debug profile` now collects files and line
numbers of profiled pipeline elements

![profiler_lines](https://github.com/nushell/nushell/assets/25571562/b400a956-d958-4aff-aa4c-7e65da3f78fa)

2. Error from the profiled closure will be reported instead of silently
ignored.

![profiler_lines_error](https://github.com/nushell/nushell/assets/25571562/54f7ad7a-06a3-4d56-92c2-c3466917bee8)


# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

New `--lines(-l)` flag to `debug profile`. The command will also fail if
the profiled closure fails, so technically it is a breaking change.

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-22 19:56:51 +03:00
Devyn Cairns
7de513a4e0
Implement streaming I/O for CSV and TSV commands (#12918)
# Description

Implements streaming for:

- `from csv`
- `from tsv`
- `to csv`
- `to tsv`

via the new string-typed ByteStream support.

# User-Facing Changes
Commands above. Also:

- `to csv` and `to tsv` now have `--columns <List(String)>`, to provide
the exact columns desired in the output. This is required for them to
have streaming output, because otherwise collecting the entire list is
necessary to determine the output columns. If we introduce
`TableStream`, this may become less necessary.

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

# After Submitting
- [ ] release notes

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-22 16:55:24 +00:00
Devyn Cairns
758c5d447a
Add support for the ps command on FreeBSD, NetBSD, and OpenBSD (#12892)
# Description

I feel like it's a little sad that BSDs get to enjoy almost everything
other than the `ps` command, and there are some tests that rely on this
command, so I figured it would be fun to patch that and make it work.

The different BSDs have diverged from each other somewhat, but generally
have a similar enough API for reading process information via
`sysctl()`, with some slightly different args.

This supports FreeBSD with the `freebsd` module, and NetBSD and OpenBSD
with the `netbsd` module. OpenBSD is a fork of NetBSD and the interface
has some minor differences but many things are the same.

I had wanted to try to support DragonFlyBSD too, but their Rust version
in the latest release is only 1.72.0, which is too old for me to want to
try to compile rustc up to 1.77.2... but I will revisit this whenever
they do update it. Dragonfly is a fork of FreeBSD, so it's likely to be
more or less the same - I just don't want to enable it without testing
it.

Fixes #6862 (partially, we probably won't be adding `zfs list`)

# User-Facing Changes
`ps` added for FreeBSD, NetBSD, and OpenBSD.

# Tests + Formatting
The CI doesn't run tests for BSDs, so I'm not entirely sure if
everything was already passing before. (Frankly, it's unlikely.) But
nothing appears to be broken.

# After Submitting
- [ ] release notes?
- [ ] DragonflyBSD, whenever they do update Rust to something close
enough for me to try it
2024-05-22 08:13:45 -07:00
dependabot[bot]
d7e75c0b70
Bump shadow-rs from 0.27.1 to 0.28.0 (#12932)
Bumps [shadow-rs](https://github.com/baoyachi/shadow-rs) from 0.27.1 to
0.28.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/baoyachi/shadow-rs/releases">shadow-rs's
releases</a>.</em></p>
<blockquote>
<h2>fix cargo clippy</h2>
<p><a
href="https://redirect.github.com/baoyachi/shadow-rs/issues/160">#160</a></p>
<p>Thx <a href="https://github.com/qartik"><code>@​qartik</code></a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="ba9f8b0c2b"><code>ba9f8b0</code></a>
Update Cargo.toml</li>
<li><a
href="d1b724c1e7"><code>d1b724c</code></a>
Merge pull request <a
href="https://redirect.github.com/baoyachi/shadow-rs/issues/160">#160</a>
from qartik/patch-1</li>
<li><a
href="505108d5d6"><code>505108d</code></a>
Allow missing_docs for deprecated CLAP_VERSION constant</li>
<li>See full diff in <a
href="https://github.com/baoyachi/shadow-rs/compare/v0.27.1...v0.28.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=shadow-rs&package-manager=cargo&previous-version=0.27.1&new-version=0.28.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-22 15:59:33 +08:00
dependabot[bot]
3cf150727c
Bump actions/checkout from 4.1.5 to 4.1.6 (#12934)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.5
to 4.1.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/releases">actions/checkout's
releases</a>.</em></p>
<blockquote>
<h2>v4.1.6</h2>
<h2>What's Changed</h2>
<ul>
<li>Check platform to set archive extension appropriately by <a
href="https://github.com/cory-miller"><code>@​cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1732">actions/checkout#1732</a></li>
<li>Update for 4.1.6 release by <a
href="https://github.com/cory-miller"><code>@​cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1733">actions/checkout#1733</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/checkout/compare/v4.1.5...v4.1.6">https://github.com/actions/checkout/compare/v4.1.5...v4.1.6</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/checkout/blob/main/CHANGELOG.md">actions/checkout's
changelog</a>.</em></p>
<blockquote>
<h2>v4.1.6</h2>
<ul>
<li>Check platform to set archive extension appropriately by <a
href="https://github.com/cory-miller"><code>@​cory-miller</code></a> in
<a
href="https://redirect.github.com/actions/checkout/pull/1732">actions/checkout#1732</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="a5ac7e51b4"><code>a5ac7e5</code></a>
Update for 4.1.6 release (<a
href="https://redirect.github.com/actions/checkout/issues/1733">#1733</a>)</li>
<li><a
href="24ed1a3528"><code>24ed1a3</code></a>
Check platform for extension (<a
href="https://redirect.github.com/actions/checkout/issues/1732">#1732</a>)</li>
<li>See full diff in <a
href="https://github.com/actions/checkout/compare/v4.1.5...v4.1.6">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/checkout&package-manager=github_actions&previous-version=4.1.5&new-version=4.1.6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-22 11:22:02 +08:00
NotTheDr01ds
f83439fdda
Add completer for std help (#12929)
# Description

While each of the `help <subcommands>` in `std` had completers, there
wasn't one for the main `help` command.

This adds all internals and custom commands (as with `help commands`) as
possible completions.

# User-Facing Changes

`help ` + <kbd>Tab</kbd> will now suggest completions for both the `help
<subcommands>` as well as all internal and custom commands.

# Tests + Formatting

Note: Cannot add tests for completion functions since they are
module-internal and not visible to test cases, that I can see.

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-05-21 10:31:14 -05:00
Devyn Cairns
1cdc39bc2a
Update mimalloc to 0.1.42 (#12919)
# Description

This update fixes mimalloc for NetBSD

# Tests + Formatting

Tests are passing just fine
2024-05-21 07:47:24 +02:00
Stefan Holderbach
db37bead64
Remove unused dependencies (#12917)
- **Remove unused `pathdiff` dep in `nu-cli`**
- **Remove unused `serde_json` dep on `nu-protocol`**
- Unnecessary after moving the plugin file to msgpack (still a
dev-dependency)
2024-05-21 01:09:28 +00:00
Reilly Wood
6e050f5634
explore: consolidate padding config, handle ByteStream, tweak naming+comments (#12915)
Some minor changes to `explore`, continuing on my mission to simplify
the command in preparation for a larger UX overhaul:

1. Consolidate padding configuration. I don't think we need separate
config points for the (optional) index column and regular data columns
in the normal pager, they can share padding configuration. Likewise, in
the binary viewer all 3 columns (index, data, ASCII) had their
left+right padding configured independently.
2. Update `explore` so we use the binary viewer for the new `ByteStream`
type. `cat foo.txt | into binary | explore` was not using the binary
viewer after the `ByteStream` changes.
3. Tweak the naming of a few helper functions, add a comment

I've put the changes in separate commits to make them easier to review.

---------

Co-authored-by: Stefan Holderbach <sholderbach@users.noreply.github.com>
2024-05-20 22:03:21 +02:00
Ian Manske
905e3d0715
Remove dataframes crate and feature (#12889)
# Description
Removes the old `nu-cmd-dataframe` crate in favor of the polars plugin.
As such, this PR also removes the `dataframe` feature, related CI, and
full releases of nushell.
2024-05-20 17:22:08 +00:00
Darren Schroeder
4f69ba172e
add math min and math max to bench command (#12913)
# Description

This PR adds min and max to the bench command.
```nushell
❯ use std bench
❯ bench { dply -c 'parquet("./data.parquet") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null } --rounds 100 --verbose
100 / 100
╭───────┬───────────────────╮
│ mean  │ 71ms 358µs 850ns  │
│ min   │ 66ms 457µs 583ns  │
│ max   │ 120ms 338µs 167ns │
│ std   │ 6ms 553µs 949ns   │
│ times │ [list 100 items]  │
╰───────┴───────────────────╯
```

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-05-20 10:08:03 -05:00
Ian Manske
c98960d053
Take owned Read and Write (#12909)
# Description
As @YizhePKU pointed out, the [Rust API
guidelines](https://rust-lang.github.io/api-guidelines/interoperability.html#generic-readerwriter-functions-take-r-read-and-w-write-by-value-c-rw-value)
recommend that generic functions take readers and writers by value and
not by reference. This PR changes `copy_with_interupt` and few other
places to take owned `Read` and `Write` instead of mutable references.
2024-05-20 15:10:36 +02:00
Devyn Cairns
c61075e20e
Add string/binary type color to ByteStream (#12897)
# Description

This PR allows byte streams to optionally be colored as being
specifically binary or string data, which guarantees that they'll be
converted to `Binary` or `String` appropriately on `into_value()`,
making them compatible with `Type` guarantees. This makes them
significantly more broadly usable for command input and output.

There is still an `Unknown` type for byte streams coming from external
commands, which uses the same behavior as we previously did where it's a
string if it's UTF-8.

A small number of commands were updated to take advantage of this, just
to prove the point. I will be adding more after this merges.

# User-Facing Changes
- New types in `describe`: `string (stream)`, `binary (stream)`
- These commands now return a stream if their input was a stream:
  - `into binary`
  - `into string`
  - `bytes collect`
  - `str join`
  - `first` (binary)
  - `last` (binary)
  - `take` (binary)
  - `skip` (binary)
- Streams that are explicitly binary colored will print as a streaming
hexdump
  - example:
    ```nushell
    1.. | each { into binary } | bytes collect
    ```

# Tests + Formatting
I've added some tests to cover it at a basic level, and it doesn't break
anything existing, but I do think more would be nice. Some of those will
come when I modify more commands to stream.

# After Submitting
There are a few things I'm not quite satisfied with:

- **String trimming behavior.** We automatically trim newlines from
streams from external commands, but I don't think we should do this with
internal commands. If I call a command that happens to turn my string
into a stream, I don't want the newline to suddenly disappear. I changed
this to specifically do it only on `Child` and `File`, but I don't know
if this is quite right, and maybe we should bring back the old flag for
`trim_end_newline`
- **Known binary always resulting in a hexdump.** It would be nice to
have a `print --raw`, so that we can put binary data on stdout
explicitly if we want to. This PR doesn't change how external commands
work though - they still dump straight to stdout.

Otherwise, here's the normal checklist:

- [ ] release notes
- [ ] docs update for plugin protocol changes (added `type` field)

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-20 00:35:32 +00:00
Ian Manske
baeba19b22
Make get_full_help take &dyn Command (#12903)
# Description
Changes `get_full_help` to take a `&dyn Command` instead of multiple
arguments (`&Signature`, `&Examples` `is_parser_keyword`). All of these
arguments can be gathered from a `Command`, so there is no need to pass
the pieces to `get_full_help`.

This PR also fixes an issue where the search terms are not shown if
`--help` is used on a command.
2024-05-19 19:56:33 +02:00
Ian Manske
474293bf1c
Clear environment for child Commands (#12901)
# Description
There is a bug when `hide-env` is used on environment variables that
were present at shell startup. Namely, child processes still inherit the
hidden environment variable. This PR fixes #12900, fixes #11495, and
fixes #7937.

# Tests + Formatting
Added a test.
2024-05-19 15:35:07 +00:00
Ian Manske
cc9f41e553
Use CommandType in more places (#12832)
# Description
Kind of a vague title, but this PR does two main things:
1. Rather than overriding functions like `Command::is_parser_keyword`,
this PR instead changes commands to override `Command::command_type`.
The `CommandType` returned by `Command::command_type` is then used to
automatically determine whether `Command::is_parser_keyword` and the
other `is_{type}` functions should return true. These changes allow us
to remove the `CommandType::Other` case and should also guarantee than
only one of the `is_{type}` functions on `Command` will return true.
2. Uses the new, reworked `Command::command_type` function in the `scope
commands` and `which` commands.


# User-Facing Changes
- Breaking change for `scope commands`: multiple columns (`is_builtin`,
`is_keyword`, `is_plugin`, etc.) have been merged into the `type`
column.
- Breaking change: the `which` command can now report `plugin` or
`keyword` instead of `built-in` in the `type` column. It may also now
report `external` instead of `custom` in the `type` column for known
`extern`s.
2024-05-18 23:37:31 +00:00
Ian Manske
580c60bb82
Preserve metadata in more places (#12848)
# Description
This PR makes some commands and areas of code preserve pipeline
metadata. This is in an attempt to make the issue described in #12599
and #9456 less likely to occur. That is, reading and writing to the same
file in a pipeline will result in an empty file. Since we preserve
metadata in more places now, there will be a higher chance that we
successfully detect this error case and abort the pipeline.
2024-05-17 17:59:32 +00:00
Devyn Cairns
c10aa2cf09
collect: don't require a closure (#12788)
# Description

This changes the `collect` command so that it doesn't require a closure.
Still allowed, optionally.

Before:

```nushell
open foo.json | insert foo bar | collect { save -f foo.json }
```

After:

```nushell
open foo.json | insert foo bar | collect | save -f foo.json
```

The closure argument isn't really necessary, as collect values are also
supported as `PipelineData`.

# User-Facing Changes
- `collect` command changed

# Tests + Formatting
Example changed to reflect.

# After Submitting
- [ ] release notes
- [ ] we may want to deprecate the closure arg?
2024-05-17 18:46:03 +02:00
Devyn Cairns
e3db6ea04a
Exclude polars from ensure_plugins_built(), for performance reasons (#12896)
# Description

We have been building `nu_plugin_polars` unnecessarily during `cargo
test`, which is very slow. All of its tests are run within its own
crate, which happens during the plugins CI phase.

This should speed up the CI a bit.
2024-05-17 15:04:59 +00:00
Devyn Cairns
59f7c523fa
Fix the way the output of table is printed in print() (#12895)
# Description

Forgot that I fixed this already on my branch, but when printing without
a display output hook, the implicit call to `table` gets its output
mangled with newlines (since #12774). This happens when running `nu -c`
or a script file.

Here's that fix in one PR so it can be merged easily.

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-05-17 07:18:18 -07:00
Wind
8adf3406e5
allow define it as a variable inside closure (#12888)
# Description
Fixes: #12690 

The issue is happened after
https://github.com/nushell/nushell/pull/12056 is merged. It will raise
error if user doesn't supply required parameter when run closure with
do.
And parser adds a `$it` parameter when parsing closure or block
expression.

I believe the previous behavior is because we allow such syntax on
previous version(0.44):
```nushell
let x = { print $it }
```
But it's no longer allowed after 0.60.  So I think they can be removed.

# User-Facing Changes
```nushell
let tmp = {
  let it = 42
  print $it
}

do -c $tmp
```
should be possible again.

# Tests + Formatting
Added 1 test
2024-05-17 00:03:13 +00:00
Ian Manske
6891267b53
Support ByteStreams in bytes starts-with and bytes ends-with (#12887)
# Description
Restores `bytes starts-with` so that it is able to work with byte
streams once again. For parity/consistency, this PR also adds byte
stream support to `bytes ends-with`.

# User-Facing Changes
- `bytes ends-with` now supports byte streams.

# Tests + Formatting
Re-enabled tests for `bytes starts-with` and added tests for `bytes
ends-with`.
2024-05-17 07:59:08 +08:00
Ian Manske
aec41f3df0
Add Span merging functions (#12511)
# Description
This PR adds a few functions to `Span` for merging spans together:
- `Span::append`: merges two spans that are known to be in order.
- `Span::concat`: returns a span that encompasses all the spans in a
slice. The spans must be in order.
- `Span::merge`: merges two spans (no order necessary).
- `Span::merge_many`: merges an iterator of spans into a single span (no
order necessary).

These are meant to replace the free-standing `nu_protocol::span`
function.

The spans in a `LiteCommand` (the `parts`) should always be in order
based on the lite parser and lexer. So, the parser code sees the most
usage of `Span::append` and `Span::concat` where the order is known. In
other code areas, `Span::merge` and `Span::merge_many` are used since
the order between spans is often not known.
2024-05-16 22:34:49 +00:00
Ian Manske
2a09dccc11
Bytestream touchup (#12886)
# Description
Adds some docs and a small fix to `Chunks`.
2024-05-16 21:15:20 +00:00
Darren Schroeder
1c00a6ca5e
sync up with reedline changes (#12881)
# Description

sync-up nushell to reedline's latest minor changes. Not quite sure why
itertools downgraded to 0.11.0 when nushell and reedline have it set to
0.12.0.
2024-05-16 22:26:03 +02:00
Ian Manske
6fd854ed9f
Replace ExternalStream with new ByteStream type (#12774)
# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
    Read(Box<dyn Read + Send + 'static>),
    File(File),
    Child(ChildProcess),
}
```

This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.

Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
    Empty,
    Value(Value, Option<PipelineMetadata>),
    ListStream(ListStream, Option<PipelineMetadata>),
    ByteStream(ByteStream, Option<PipelineMetadata>),
}
```

The PR is relatively large, but a decent amount of it is just repetitive
changes.

This PR fixes #7017, fixes #10763, and fixes #12369.

This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |

# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.

# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.

---------

Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
2024-05-16 07:11:18 -07:00
Wind
1b8eb23785
allow passing float value to custom command (#12879)
# Description
Fixes: #12691 

In `parse_short_flag`, it only checks special cases for
`SyntaxShape::Int`, `SyntaxShape::Number` to allow a flag to be a
number. This pr adds `SyntaxShape::Float` to allow a flag to be float
number.

# User-Facing Changes
This is possible after this pr:
```nushell
def spam [val: float] { $val }; 
spam -1.4
```

# Tests + Formatting
Added 1 test
2024-05-16 10:50:29 +02:00
Ian Manske
e20113a0eb
Remove stack debug assert (#12861)
# Description
In order for `Stack::unwrap_unique` to work as intended, we currently
manually track all references to the parent stack and ensure that they
are cleared before calling `Stack::unwrap_unique` in the REPL. We also
only call `Stack::unwrap_unique` after all code from the current REPL
entry has finished executing. Since `Value`s cannot store `Stack`
references, then this should have worked in theory. However, we forgot
to account for threads. `run-external` (and maybe the plugin writers)
can spawn threads that clone the `Stack`, holding on to references of
the parent stack. These threads are not waited/joined upon, and so may
finish after the eval has already returned. This PR removes the
`Stack::unwrap_unique` function and associated debug assert that was
[causing
panics](https://gist.github.com/cablehead/f3d2608a1629e607c2d75290829354f7)
like @cablehead found.

# After Submitting
Make values cheaper to clone as a more robust solution to the
performance issues with cloning the stack.

---------

Co-authored-by: Wind <WindSoilder@outlook.com>
2024-05-15 22:59:10 +00:00
Jack Wright
6f3dbc97bb
fixed syntax shape requirements for --quantiles option for polars summary (#12878)
Fix for #12730

All of the code expected a list of floats, but the syntax shape expected
a table. Resolved by changing the syntax shape to list of floats.

cc: @maxim-uvarov
2024-05-15 16:55:07 -05:00
Ian Manske
06fe7d1e16
Remove usages of Call::positional_nth (#12871)
# Description
Following from #12867, this PR replaces usages of `Call::positional_nth`
with existing spans. This removes several `expect`s from the code.

Also remove unused `positional_nth_mut` and `positional_iter_mut`
2024-05-15 19:59:42 +02:00
NotTheDr01ds
b08135d877
Fixed small error in the help-examples for the get command (#12877)
# Description

Another small error in Help, this time for the `get` command example.

# User-Facing Changes

Help only
2024-05-15 19:49:08 +02:00
NotTheDr01ds
72b880662b
Fixed a nitpick usage-help error - closure v. block (#12876)
# Description

So minor, but had to be fixed sometime. `help each while` used the term
"block" in the "usage", but the argument type is a closure.

# User-Facing Changes

help-only
2024-05-15 18:16:59 +02:00
Darren Schroeder
defed3001d
make it clearer what is being loaded with --log-level info (#12875)
# Description

A common question we get is what config files are loaded when and with
what parameters. It's for this reason that I wrote [this
gist](https://gist.github.com/fdncred/b87b784f04984dc31a150baed9ad2447).
Another way to figure this out is to use `nu --log-level info`. This
will show some performance timings but will also show what is being
loaded when. For the most part the `[INFO]` lines show the performance
timings and the `[WARN]` lines show the files.

This PR tries to make things a little bit clearer when using the
`--log-level info` parameter.

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-05-15 09:44:09 -05:00
Ian Manske
0cfbdc909e
Fix sys panic (#12846)
# Description
This should fix #10155 where the `sys` command can panic due to date
math in certain cases / on certain systems.

# User-Facing Changes
The `boot_time` column now has a date value instead of a formatted date
string. This is technically a breaking change.
2024-05-15 15:40:04 +08:00
Andy Gayton
a7807735b1
Add a passing test for interactivity on slow pipelines (#12865)
# Description

This PR adds a single test to assert interactivity on slow pipelines

Currently the timeout is set to 6 seconds, as the test can sometimes
take ~3secs to run on my local m1 mac air, which I don't think is an
indication of a slow pipeline, but rather slow test start up time...
2024-05-15 01:48:27 +00:00
Wind
155934f783
make better messages for incomplete string (#12868)
# Description
Fixes: #12795

The issue is caused by an empty position of `ParseError::UnexpectedEof`.
So no detailed message is displayed.
To fix the issue, I adjust the start of span to `span.end - 1`. In this
way, we can make sure that it never points to an empty position.

After lexing item, I also reorder the unclosed character checking . Now
it will be checking unclosed opening delimiters first.

# User-Facing Changes
After this pr, it outputs detailed error message for incomplete string
when running scripts.

## Before
```
❯ nu -c "'ab"
Error: nu::parser::unexpected_eof

  × Unexpected end of code.
   ╭─[source:1:4]
 1 │ 'ab
   ╰────
> ./target/debug/nu -c "r#'ab"
Error: nu::parser::unexpected_eof

  × Unexpected end of code.
   ╭─[source:1:6]
 1 │ r#'ab
   ╰────
```
## After
```
> nu -c "'ab"
Error: nu::parser::unexpected_eof

  × Unexpected end of code.
   ╭─[source:1:3]
 1 │ 'ab
   ·   ┬
   ·   ╰── expected closing '
   ╰────
> ./target/debug/nu -c "r#'ab"
Error: nu::parser::unexpected_eof

  × Unexpected end of code.
   ╭─[source:1:5]
 1 │ r#'ab
   ·     ┬
   ·     ╰── expected closing '#
   ╰────
```


# Tests + Formatting
Added some tests for incomplete string.

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-15 01:14:11 +00:00
dependabot[bot]
9bf4d3ece6
Bump rust-embed from 8.3.0 to 8.4.0 (#12870)
Bumps [rust-embed](https://github.com/pyros2097/rust-embed) from 8.3.0
to 8.4.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pyrossh/rust-embed/blob/master/changelog.md">rust-embed's
changelog</a>.</em></p>
<blockquote>
<h2>[8.4.0] - 2024-05-11</h2>
<ul>
<li>Re-export RustEmbed as Embed <a
href="https://redirect.github.com/pyrossh/rust-embed/pull/245/files">#245</a>.
Thanks to <a href="https://github.com/pyrossh">pyrossh</a></li>
<li>Do not build glob matchers repeatedly when include-exclude feature
is enabled <a
href="https://redirect.github.com/pyrossh/rust-embed/pull/244/files">#244</a>.
Thanks to <a href="https://github.com/osiewicz">osiewicz</a></li>
<li>Add <code>metadata_only</code> attribute <a
href="https://redirect.github.com/pyrossh/rust-embed/pull/241/files">#241</a>.
Thanks to <a href="https://github.com/ddfisher">ddfisher</a></li>
<li>Replace <code>expect</code> with a safer alternative that returns
<code>None</code> instead <a
href="https://redirect.github.com/pyrossh/rust-embed/pull/240/files">#240</a>.
Thanks to <a href="https://github.com/costinsin">costinsin</a></li>
<li>Eliminate unnecessary <code>to_path</code> call <a
href="https://redirect.github.com/pyrossh/rust-embed/pull/239/files">#239</a>.
Thanks to <a href="https://github.com/smoelius">smoelius</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/pyros2097/rust-embed/commits">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=rust-embed&package-manager=cargo&previous-version=8.3.0&new-version=8.4.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-15 09:06:09 +08:00
dependabot[bot]
cb64c78a3b
Bump interprocess from 2.0.1 to 2.1.0 (#12869)
Bumps [interprocess](https://github.com/kotauskas/interprocess) from
2.0.1 to 2.1.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/kotauskas/interprocess/releases">interprocess's
releases</a>.</em></p>
<blockquote>
<h2>2.1.0 – listeners are now iterators</h2>
<ul>
<li>Fixes <a
href="https://redirect.github.com/kotauskas/interprocess/issues/49">#49</a></li>
<li>Adds <code>Iterator</code> impl on local socket listeners (closes <a
href="https://redirect.github.com/kotauskas/interprocess/issues/64">#64</a>)</li>
<li>Miscellaneous documentation fixes</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b79d363615"><code>b79d363</code></a>
Thank you Windows, very cool</li>
<li><a
href="2ab1418db7"><code>2ab1418</code></a>
Move a bunch of goalposts</li>
<li><a
href="d26ed39bd2"><code>d26ed39</code></a>
Use macro in Windows <code>UnnamedPipe</code> builder</li>
<li><a
href="f9528885e4"><code>f952888</code></a>
I'm not adding a <code>build.rs</code> to silence a warning</li>
<li><a
href="5233ffb7c9"><code>5233ffb</code></a>
Fix <a
href="https://redirect.github.com/kotauskas/interprocess/issues/49">#49</a>
and add test</li>
<li><a
href="85d3e1861a"><code>85d3e18</code></a>
Complete half-done move to <code>os</code> module in tests</li>
<li><a
href="7715dbdd49"><code>7715dbd</code></a>
Listeners are now iterators</li>
<li><a
href="8a47261ddc"><code>8a47261</code></a>
Bump version</li>
<li>See full diff in <a
href="https://github.com/kotauskas/interprocess/compare/2.0.1...2.1.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=interprocess&package-manager=cargo&previous-version=2.0.1&new-version=2.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-15 09:05:55 +08:00
Ian Manske
c3da44cbb7
Fix char panic (#12867)
# Description
The `char` command can panic due to a failed `expect`: `char --integer
...[77 78 79]`

This PR fixes the panic for the `--integer` flag and also the
`--unicode` flag.

# After Submitting
Check other commands and places where similar bugs can occur due to
usages of `Call::positional_nth` and related methods.
2024-05-14 21:10:06 +00:00
NotTheDr01ds
aa46bc97b3
Search terms for compact command (#12864)
# Description

There was a question in Discord today about how to remove empty rows
from a table. The user found the `compact` command on their own, but I
realized that there were no search terms on the command. I've added
'empty' and 'remove', although I subsequently figured out that 'empty'
is found in the "usage" anyway. That said, I don't think it hurts to
have good search terms behind it regardless.

# User-Facing Changes

Just the help

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

# After Submitting
2024-05-14 09:21:50 -05:00
Maxime Jacob
2ed77aef1d
Fix panic when exploring empty dictionary (#12860)
- fixes #12841 

# Description
Add boundary checks to ensure that the row and column chosen in
RecordView are not over the length of the possible row and columns. If
we are out of bounds, we default to Value::nothing.

# Tests + Formatting
Tests ran and formatting done
2024-05-14 14:13:49 +00:00
Maxime Jacob
cd381b74e0
Fix improperly escaped strings in stor insert (#12820)
- fixes #12764 

Replaced the custom logic with values_to_sql method that is already used
in crate::database.
This will ensure that handling of parameters is the same between sqlite
and stor.
2024-05-13 20:22:39 -05:00
Jack Wright
98369985b1
Allow custom value operations to work on eager and lazy dataframes interchangeably. (#12819)
Fixes Bug #12809 

The example that @maxim-uvarov posted now works as expected:

<img width="1223" alt="Screenshot 2024-05-09 at 16 21 01"
src="https://github.com/nushell/nushell/assets/56345/a4df62e3-e432-4c09-8e25-9a6c198741a3">
2024-05-13 18:17:31 -05:00
Piepmatz
aaf973bbba
Add Stack::stdout_file and Stack::stderr_file to capture stdout/-err of external commands (#12857)
# Description
In this PR I added two new methods to `Stack`, `stdout_file` and
`stderr_file`. These two modify the inner `StackOutDest` and set a
`File` into the `stdout` and `stderr` respectively. Different to the
`push_redirection` methods, these do not require to hold a guard up all
the time but require ownership of the stack.

This is primarly useful for applications that use `nu` as a language but
not the `nushell`.

This PR replaces my first attempt #12851 to add a way to capture
stdout/-err of external commands. Capturing the stdout without having to
write into a file is possible with crates like
[`os_pipe`](https://docs.rs/os_pipe), an example for this is given in
the doc comment of the `stdout_file` command and can be executed as a
doctest (although it doesn't validate that you actually got any data).

This implementation takes `File` as input to make it easier to implement
on different operating systems without having to worry about
`OwnedHandle` or `OwnedFd`. Also this doesn't expose any use `os_pipe`
to not leak its types into this API, making it depend on it.

As in my previous attempt, @IanManske guided me here.

# User-Facing Changes
This change has no effect on `nushell` and therefore no user-facing
changes.

# Tests + Formatting
This only exposes a new way of using already existing code and has
therefore no further testing. The doctest succeeds on my machine at
least (x86 Windows, 64 Bit).

# After Submitting
All the required documentation is already part of this PR.
2024-05-13 18:48:38 +00:00
Ian Manske
905ec88091
Update PR template (#12838)
# Description
Updates the command listed in the PR template to test the standard
library, following from #11151.
2024-05-13 08:45:44 -05:00
francesco-gaglione
c4dca5fe03
Merged tests to produce a single binary (#12826)
This PR should close #7147 

# Description
Merged src/tests into /tests to produce a single binary.

![image](https://github.com/nushell/nushell/assets/94604837/84726469-d447-4619-b6d1-2d1415d0f42e)

# User-Facing Changes
No user facing changes

# Tests + Formatting
Moved tests. Tollkit check pr pass.

# After Submitting

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-13 13:37:53 +00:00
NotTheDr01ds
c70c43aae9
Add example and search term for 'repeat' to the fill command (#12844)
# Description

It's commonly forgotten or overlooked that a lot of `std repeat`
functionality can be handled with the built-in `fill`. Added 'repeat` as
a search term for `fill` to improve discoverability.

Also replaced one of the existing examples with one `fill`ing an empty
string, a la `repeat`. There were 6 examples already, and 3 of them
pretty much were variations on the same theme, so I repurposed one of
those rather than adding a 7th.

# User-Facing Changes

Changes to `help` only

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

# After Submitting

I assume the "Commands" doc is auto-generated from the `help`, but I'll
double-check that assumption.
2024-05-12 20:55:07 -05:00
Ian Manske
30fc832035
Fix custom converters with save (#12833)
# Description
Fixes #10429 where `save` fails if a custom command is used as the file
format converter.

# Tests + Formatting
Added a test.
2024-05-12 13:19:28 +02:00
Brage Ingebrigtsen
075535f869
remove --not flag for 'str contains' (#12837)
# Description
This PR resolves an inconsistency between different `str` subcommands,
notably `str contains`, `str starts-with` and `str ends-with`. Only the
`str contains` command has the `--not` flag and a desicion was made in
this #12781 PR to remove the `--not` flag and use the `not` operator
instead.

Before:
`"blob" | str contains --not o`
After:
`not ("blob" | str contains o)` OR `"blob" | str contains o | not $in`

> Note, you can currently do all three, but the first will be broken
after this PR is merged.

# User-Facing Changes
- remove `--not(-n)` flag from `str contains` command
  - This is a breaking change!

# Tests + Formatting
- [x] Added tests
- [x] Ran `cargo fmt --all`
- [x] Ran `cargo clippy --workspace -- -D warnings -D
clippy::unwrap_used`
- [x] Ran `cargo test --workspace`
- [ ] Ran `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"`
    - I was unable to get this working.
```
Error: nu::parser::export_not_found

  × Export not found.
   ╭─[source:1:9]
 1 │ use std testing; testing run-tests --path crates/nu-std
   ·         ───┬───
   ·            ╰── could not find imports
   ╰────
```
^ I still can't figure out how to make this work 😂 

# After Submitting
Requires update of documentation
2024-05-11 23:13:36 +00:00