# Description
This PR introduces a `ByteStream` type which is a `Read`-able stream of
bytes. Internally, it has an enum over three different byte stream
sources:
```rust
pub enum ByteStreamSource {
Read(Box<dyn Read + Send + 'static>),
File(File),
Child(ChildProcess),
}
```
This is in comparison to the current `RawStream` type, which is an
`Iterator<Item = Vec<u8>>` and has to allocate for each read chunk.
Currently, `PipelineData::ExternalStream` serves a weird dual role where
it is either external command output or a wrapper around `RawStream`.
`ByteStream` makes this distinction more clear (via `ByteStreamSource`)
and replaces `PipelineData::ExternalStream` in this PR:
```rust
pub enum PipelineData {
Empty,
Value(Value, Option<PipelineMetadata>),
ListStream(ListStream, Option<PipelineMetadata>),
ByteStream(ByteStream, Option<PipelineMetadata>),
}
```
The PR is relatively large, but a decent amount of it is just repetitive
changes.
This PR fixes#7017, fixes#10763, and fixes#12369.
This PR also improves performance when piping external commands. Nushell
should, in most cases, have competitive pipeline throughput compared to,
e.g., bash.
| Command | Before (MB/s) | After (MB/s) | Bash (MB/s) |
| -------------------------------------------------- | -------------:|
------------:| -----------:|
| `throughput \| rg 'x'` | 3059 | 3744 | 3739 |
| `throughput \| nu --testbin relay o> /dev/null` | 3508 | 8087 | 8136 |
# User-Facing Changes
- This is a breaking change for the plugin communication protocol,
because the `ExternalStreamInfo` was replaced with `ByteStreamInfo`.
Plugins now only have to deal with a single input stream, as opposed to
the previous three streams: stdout, stderr, and exit code.
- The output of `describe` has been changed for external/byte streams.
- Temporary breaking change: `bytes starts-with` no longer works with
byte streams. This is to keep the PR smaller, and `bytes ends-with`
already does not work on byte streams.
- If a process core dumped, then instead of having a `Value::Error` in
the `exit_code` column of the output returned from `complete`, it now is
a `Value::Int` with the negation of the signal number.
# After Submitting
- Update docs and book as necessary
- Release notes (e.g., plugin protocol changes)
- Adapt/convert commands to work with byte streams (high priority is
`str length`, `bytes starts-with`, and maybe `bytes ends-with`).
- Refactor the `tee` code, Devyn has already done some work on this.
---------
Co-authored-by: Devyn Cairns <devyn.cairns@gmail.com>
# Description
Adds support for running plugins using local socket communication
instead of stdio. This will be an optional thing that not all plugins
have to support.
This frees up stdio for use to make plugins that use stdio to create
terminal UIs, cc @amtoine, @fdncred.
This uses the [`interprocess`](https://crates.io/crates/interprocess)
crate (298 stars, MIT license, actively maintained), which seems to be
the best option for cross-platform local socket support in Rust. On
Windows, a local socket name is provided. On Unixes, it's a path. The
socket name is kept to a relatively small size because some operating
systems have pretty strict limits on the whole path (~100 chars), so on
macOS for example we prefer `/tmp/nu.{pid}.{hash64}.sock` where the hash
includes the plugin filename and timestamp to be unique enough.
This also adds an API for moving plugins in and out of the foreground
group, which is relevant for Unixes where direct terminal control
depends on that.
TODO:
- [x] Generate local socket path according to OS conventions
- [x] Add support for passing `--local-socket` to the plugin executable
instead of `--stdio`, and communicating over that instead
- [x] Test plugins that were broken, including
[amtoine/nu_plugin_explore](https://github.com/amtoine/nu_plugin_explore)
- [x] Automatically upgrade to using local sockets when supported,
falling back if it doesn't work, transparently to the user without any
visible error messages
- Added protocol feature: `LocalSocket`
- [x] Reset preferred mode to `None` on `register`
- [x] Allow plugins to detect whether they're running on a local socket
and can use stdio freely, so that TUI plugins can just produce an error
message otherwise
- Implemented via `EngineInterface::is_using_stdio()`
- [x] Clean up foreground state when plugin command exits on the engine
side too, not just whole plugin
- [x] Make sure tests for failure cases work as intended
- `nu_plugin_stress_internals` added
# User-Facing Changes
- TUI plugins work
- Non-Rust plugins could optionally choose to use this
- This might behave differently, so will need to test it carefully
across different operating systems
# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
# After Submitting
- [ ] Document local socket option in plugin contrib docs
- [ ] Document how to do a terminal UI plugin in plugin contrib docs
- [ ] Document: `EnterForeground` engine call
- [ ] Document: `LeaveForeground` engine call
- [ ] Document: `LocalSocket` protocol feature
# Description
Where possible, this PR replaces usages of raw `libc` bindings to
instead use safe interfaces from the `nix` crate. Where not possible,
the `libc` version reexported through `nix` was used instead of having a
separate `libc` dependency.
# Description
Now we only use `nix 0.28.0`
Achieved by
- updating `ctrlc` to `3.4.4`
- updating `wl-clipboard-rs` to `0.8.1`
- update our own dependency on `nix` from `0.27` to `0.28`
- required fixing uses of `nix::unistd::{tcgetpgrp,tcsetpgrp}`
- now requires an I/O safe file descriptor
- fake one pointing to `libc::STDIN_FILENO` (we were only accessing
`0` previously, dito for fish)
# User-Facing Changes
Better compile times and less to download as source dependencies
# Description / User-Facing Changes
Signals are no longer blocked for child processes launched from both
interactive and non-interactive mode. The only exception is that
`SIGTSTP`, `SIGTTIN`, and `SIGTTOU` remain blocked for child processes
launched only from **interactive** mode. This is to help prevent nushell
from getting into an unrecoverable state, since we don't support
background jobs. Anyways, this fully fixes#9026.
# Other Notes
- Needs Rust version `>= 1.66` for a fix in
`std::process::Command::spawn`, but it looks our current Rust version is
way above this.
- Uses `sigaction` instead of `signal`, since the behavior of `signal`
can apparently differ across systems. Also, the `sigaction` man page
says:
> The sigaction() function supersedes the signal() function, and should
be used in preference.
Additionally, using both `sigaction` and `signal` is not recommended.
Since we were already using `sigaction` in some places (and possibly
some of our dependencies as well), this PR replaces all usages of
`signal`.
# Tests
Might want to wait for #11178 for testing.
# Description
This PR tries to remove ~atty~ is-terminal from the entire code base,
since ~[atty is
unmaintained](https://rustsec.org/advisories/RUSTSEC-2021-0145) and~
[`is_terminal` has been
stabilized](https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html#isterminal)
in rust 1.70.0.
cc @fdncred
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- crates/nu-std/tests/run.nu` to run the tests for the
standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
Currently, foreground process management is disabled for macOS, since
the original code had issues (see #7068).
This PR re-enables process management on macOS in combination with the
changes from #9693.
# User-Facing Changes
Fixes hang on exit for nested nushells on macOS (issue #9859). Nushell
should now manage processes in the same way on macOS and other unix
systems.
# Description
This reverts #9693 as it lead to CPU hangs. (btw, did the revert by hand
as it couldn't be done automatically. Hopefully I didn't miss anything 😅
)
Fixes#9859
cc @IanManske
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library
> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->
# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
# Description
Fixes a regression from #9681 where nushell will attempt to place itself
into the background or take control of the terminal even in
non-interactive mode.
Using the same
[reference](https://www.gnu.org/software/libc/manual/html_node/Initializing-the-Shell.html)
from #6584:
>A subshell that runs *interactively* has to ensure that it has been
placed in the foreground...
>A subshell that runs *non-interactively* cannot and should not support
job control.
`fish`
[code](54fa1ad6ec/src/reader.cpp (L4862))
also seems to follow this.
This *partially* fixes
[9026](https://github.com/nushell/nushell/issues/9026). That is, nushell
will no longer set the foreground process group in non-interactive mode.
# Description
Until we bump our minimal Rust version to `1.70.0` we can't use
`std::io::IsTerminal`. The crate `is-terminal` (depending on `rustix` or
`windows-sys`) can provide the same.
Get's rid of the dependency on the outdated `atty` crate.
We already transitively depend on it (e.g. through `miette`)
As soon as we reach the new Rust version we can supersede this with
@nibon7's #9550
Co-authored-by: nibon7 <nibon7@163.com>
* Revert "Revert "Try again: in unix like system, set foreground process while running external command (#6273)" (#6542)"
This reverts commit 2bb367f570502cd26d6b7a81425d9f3b3b9eb432.
* Make foreground job control hopefully work correctly
These changes are mostly inspired by the glibc manual.
* Fix typo in external command description
* Only restore tty control to shell when no fg procs are left; reuse pgrp
* Rework terminal acquirement code to be like fish
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
* Revert "Fix intermittent test crash (#6268)"
This reverts commit 555d9ee7630e0d347e0d8d35e7dd028ff860f3f7.
* make a working version again
* try second impl
* add
* fmt
* check stdin is atty before acquire stdin
* add libc
* clean comment
* fix typo
* while executing external command, make it as foreground
* remove useless file
* add comment, make var more readable
* add comment
* fmt code
* fix windows
* fix func name
* fix clippy
* fix windows clippy
* add comments, introduce `ForegroundProcess and ForegroundChild
* fix windows clippy
* fix on windows
* no need fg_process_setup module
* Revert "no need fg_process_setup module"
This reverts commit 21ee4ffbf65df93bbc65a08a725755c081c6a0da.
* restrict visibility for helper functions
