extern/nushell
1
0
Fork 1
mirror of https://github.com/nushell/nushell.git synced 2024-11-24 17:34:00 +01:00
Code Issues Packages Projects Releases Wiki Activity

Rewrite run_external.rs (#12921)

Browse Source 
This PR is a complete rewrite of `run_external.rs`. The main goal of the
rewrite is improving readability, but it also fixes some bugs related to
argument handling and the PATH variable (fixes
https://github.com/nushell/nushell/issues/6011).

I'll discuss some technical details to make reviewing easier.

## Argument handling

Quoting arguments for external commands is hard. Like, *really* hard.
We've had more than a dozen issues and PRs dedicated to quoting
arguments (see Appendix) but the current implementation is still buggy.

Here's a demonstration of the buggy behavior:

```nu
let foo = "'bar'"
^touch $foo            # This creates a file named `bar`, but it should be `'bar'`
^touch ...[ "'bar'" ]  # Same
```

I'll describe how this PR deals with argument handling.

First, we'll introduce the concept of **bare strings**. Bare strings are
**string literals** that are either **unquoted** or **quoted by
backticks** [^1]. Strings within a list literal are NOT considered bare
strings, even if they are unquoted or quoted by backticks.

When a bare string is used as an argument to external process, we need
to perform tilde-expansion, glob-expansion, and inner-quotes-removal, in
that order. "Inner-quotes-removal" means transforming from
`--option="value"` into `--option=value`.

## `.bat` files and CMD built-ins

On Windows, `.bat` files and `.cmd` files are considered executable, but
they need `CMD.exe` as the interpreter. The Rust standard library
supports running `.bat` files directly and will spawn `CMD.exe` under
the hood (see
[documentation](https://doc.rust-lang.org/std/process/index.html#windows-argument-splitting)).
However, other extensions are not supported [^2].

Nushell also supports a selected number of CMD built-ins. The problem
with CMD is that it uses a different set of quoting rules. Correctly
quoting for CMD requires using
[Command::raw_arg()](https://doc.rust-lang.org/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg)
and manually quoting CMD special characters, on top of quoting from the
Nushell side. ~~I decided that this is too complex and chose to reject
special characters in CMD built-ins instead [^3]. Hopefully this will
not affact real-world use cases.~~ I've implemented escaping that works
reasonably well.

## `which-support` feature

The `which` crate is now a hard dependency of `nu-command`, making the
`which-support` feature essentially useless. The `which` crate is
already a hard dependency of `nu-cli`, and we should consider removing
the `which-support` feature entirely.

## Appendix

Here's a list of quoting-related issues and PRs in rough chronological
order.

* https://github.com/nushell/nushell/issues/4609
* https://github.com/nushell/nushell/issues/4631
* https://github.com/nushell/nushell/issues/4601
  * https://github.com/nushell/nushell/pull/5846
* https://github.com/nushell/nushell/issues/5978
  * https://github.com/nushell/nushell/pull/6014
* https://github.com/nushell/nushell/issues/6154
  * https://github.com/nushell/nushell/pull/6161
* https://github.com/nushell/nushell/issues/6399
  * https://github.com/nushell/nushell/pull/6420
  * https://github.com/nushell/nushell/pull/6426
* https://github.com/nushell/nushell/issues/6465
* https://github.com/nushell/nushell/issues/6559
  * https://github.com/nushell/nushell/pull/6560

[^1]: The idea that backtick-quoted strings act like bare strings was
introduced by Kubouch and briefly mentioned in [the language
reference](https://www.nushell.sh/lang-guide/chapters/strings_and_text.html#backtick-quotes).

[^2]: The documentation also said "running .bat scripts in this way may
be removed in the future and so should not be relied upon", which is
another reason to move away from this. But again, quoting for CMD is
hard.

[^3]: If anyone wants to try, the best resource I found on the topic is
[this](https://daviddeley.com/autohotkey/parameters/parameters.htm).
This commit is contained in:
YizhePKU 2024-05-23 10:05:27 +08:00 committed by GitHub
parent 64afb52ffa
commit 6c649809d3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
20 changed files with 803 additions and 798 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Cargo.lock generated
View File

@ -3012,6 +3012,7 @@ dependencies = [
"sha2",
"sysinfo",
"tabled",
"tempfile",
"terminal_size",
"titlecase",
"toml 0.8.12",

5
crates/nu-command/Cargo.toml
View File

@ -99,7 +99,7 @@ uu_whoami = { workspace = true }
uuid = { workspace = true, features = ["v4"] }
v_htmlescape = { workspace = true }
wax = { workspace = true }
which = { workspace = true, optional = true }
which = { workspace = true }
unicode-width = { workspace = true }
[target.'cfg(windows)'.dependencies]
@ -134,7 +134,7 @@ workspace = true
plugin = ["nu-parser/plugin"]
sqlite = ["rusqlite"]
trash-support = ["trash"]
which-support = ["which"]
which-support = []
[dev-dependencies]
nu-cmd-lang = { path = "../nu-cmd-lang", version = "0.93.1" }
@ -146,3 +146,4 @@ quickcheck = { workspace = true }
quickcheck_macros = { workspace = true }
rstest = { workspace = true, default-features = false }
pretty_assertions = { workspace = true }
tempfile = { workspace = true }

75
crates/nu-command/src/env/config/config_env.rs vendored
View File

@ -1,6 +1,7 @@
use super::utils::gen_command;
use nu_cmd_base::util::get_editor;
use nu_engine::{command_prelude::*, env_to_strings};
use nu_protocol::{process::ChildProcess, ByteStream};
use nu_system::ForegroundChild;
#[derive(Clone)]
pub struct ConfigEnv;
@ -47,7 +48,7 @@ impl Command for ConfigEnv {
engine_state: &EngineState,
stack: &mut Stack,
call: &Call,
input: PipelineData,
_input: PipelineData,
) -> Result<PipelineData, ShellError> {
// `--default` flag handling
if call.has_flag(engine_state, stack, "default")? {
@ -55,27 +56,59 @@ impl Command for ConfigEnv {
return Ok(Value::string(nu_utils::get_default_env(), head).into_pipeline_data());
}
let env_vars_str = env_to_strings(engine_state, stack)?;
let nu_config = match engine_state.get_config_path("env-path") {
Some(path) => path,
None => {
return Err(ShellError::GenericError {
error: "Could not find $nu.env-path".into(),
msg: "Could not find $nu.env-path".into(),
span: None,
help: None,
inner: vec![],
});
}
// Find the editor executable.
let (editor_name, editor_args) = get_editor(engine_state, stack, call.head)?;
let paths = nu_engine::env::path_str(engine_state, stack, call.head)?;
let cwd = engine_state.cwd(Some(stack))?;
let editor_executable =
crate::which(&editor_name, &paths, &cwd).ok_or(ShellError::ExternalCommand {
label: format!("`{editor_name}` not found"),
help: "Failed to find the editor executable".into(),
span: call.head,
})?;
let Some(env_path) = engine_state.get_config_path("env-path") else {
return Err(ShellError::GenericError {
error: "Could not find $nu.env-path".into(),
msg: "Could not find $nu.env-path".into(),
span: None,
help: None,
inner: vec![],
});
};
let env_path = env_path.to_string_lossy().to_string();
let (item, config_args) = get_editor(engine_state, stack, call.head)?;
// Create the command.
let mut command = std::process::Command::new(editor_executable);
gen_command(call.head, nu_config, item, config_args, env_vars_str).run_with_input(
engine_state,
stack,
input,
true,
)
// Configure PWD.
command.current_dir(cwd);
// Configure environment variables.
let envs = env_to_strings(engine_state, stack)?;
command.env_clear();
command.envs(envs);
// Configure args.
command.arg(env_path);
command.args(editor_args);
// Spawn the child process. On Unix, also put the child process to
// foreground if we're in an interactive session.
#[cfg(windows)]
let child = ForegroundChild::spawn(command)?;
#[cfg(unix)]
let child = ForegroundChild::spawn(
command,
engine_state.is_interactive,
&engine_state.pipeline_externals_state,
)?;
// Wrap the output into a `PipelineData::ByteStream`.
let child = ChildProcess::new(child, None, false, call.head)?;
Ok(PipelineData::ByteStream(
ByteStream::child(child, call.head),
None,
))
}
}

75
crates/nu-command/src/env/config/config_nu.rs vendored
View File

@ -1,6 +1,7 @@
use super::utils::gen_command;
use nu_cmd_base::util::get_editor;
use nu_engine::{command_prelude::*, env_to_strings};
use nu_protocol::{process::ChildProcess, ByteStream};
use nu_system::ForegroundChild;
#[derive(Clone)]
pub struct ConfigNu;
@ -51,7 +52,7 @@ impl Command for ConfigNu {
engine_state: &EngineState,
stack: &mut Stack,
call: &Call,
input: PipelineData,
_input: PipelineData,
) -> Result<PipelineData, ShellError> {
// `--default` flag handling
if call.has_flag(engine_state, stack, "default")? {
@ -59,27 +60,59 @@ impl Command for ConfigNu {
return Ok(Value::string(nu_utils::get_default_config(), head).into_pipeline_data());
}
let env_vars_str = env_to_strings(engine_state, stack)?;
let nu_config = match engine_state.get_config_path("config-path") {
Some(path) => path,
None => {
return Err(ShellError::GenericError {
error: "Could not find $nu.config-path".into(),
msg: "Could not find $nu.config-path".into(),
span: None,
help: None,
inner: vec![],
});
}
// Find the editor executable.
let (editor_name, editor_args) = get_editor(engine_state, stack, call.head)?;
let paths = nu_engine::env::path_str(engine_state, stack, call.head)?;
let cwd = engine_state.cwd(Some(stack))?;
let editor_executable =
crate::which(&editor_name, &paths, &cwd).ok_or(ShellError::ExternalCommand {
label: format!("`{editor_name}` not found"),
help: "Failed to find the editor executable".into(),
span: call.head,
})?;
let Some(config_path) = engine_state.get_config_path("config-path") else {
return Err(ShellError::GenericError {
error: "Could not find $nu.config-path".into(),
msg: "Could not find $nu.config-path".into(),
span: None,
help: None,
inner: vec![],
});
};
let config_path = config_path.to_string_lossy().to_string();
let (item, config_args) = get_editor(engine_state, stack, call.head)?;
// Create the command.
let mut command = std::process::Command::new(editor_executable);
gen_command(call.head, nu_config, item, config_args, env_vars_str).run_with_input(
engine_state,
stack,
input,
true,
)
// Configure PWD.
command.current_dir(cwd);
// Configure environment variables.
let envs = env_to_strings(engine_state, stack)?;
command.env_clear();
command.envs(envs);
// Configure args.
command.arg(config_path);
command.args(editor_args);
// Spawn the child process. On Unix, also put the child process to
// foreground if we're in an interactive session.
#[cfg(windows)]
let child = ForegroundChild::spawn(command)?;
#[cfg(unix)]
let child = ForegroundChild::spawn(
command,
engine_state.is_interactive,
&engine_state.pipeline_externals_state,
)?;
// Wrap the output into a `PipelineData::ByteStream`.
let child = ChildProcess::new(child, None, false, call.head)?;
Ok(PipelineData::ByteStream(
ByteStream::child(child, call.head),
None,
))
}
}

1
crates/nu-command/src/env/config/mod.rs vendored
View File

@ -2,7 +2,6 @@ mod config_;
mod config_env;
mod config_nu;
mod config_reset;
mod utils;
pub use config_::ConfigMeta;
pub use config_env::ConfigEnv;
pub use config_nu::ConfigNu;

36
crates/nu-command/src/env/config/utils.rs vendored
View File

@ -1,36 +0,0 @@
use crate::ExternalCommand;
use nu_protocol::{OutDest, Span, Spanned};
use std::{collections::HashMap, path::Path};
pub(crate) fn gen_command(
span: Span,
config_path: &Path,
item: String,
config_args: Vec<String>,
env_vars_str: HashMap<String, String>,
) -> ExternalCommand {
let name = Spanned { item, span };
let mut args = vec![Spanned {
item: config_path.to_string_lossy().to_string(),
span: Span::unknown(),
}];
let number_of_args = config_args.len() + 1;
for arg in config_args {
args.push(Spanned {
item: arg,
span: Span::unknown(),
})
}
ExternalCommand {
name,
args,
arg_keep_raw: vec![false; number_of_args],
out: OutDest::Inherit,
err: OutDest::Inherit,
env_vars: env_vars_str,
}
}

120
crates/nu-command/src/system/exec.rs
View File

@ -1,7 +1,4 @@
use super::run_external::create_external_command;
#[allow(deprecated)]
use nu_engine::{command_prelude::*, current_dir};
use nu_protocol::OutDest;
use nu_engine::{command_prelude::*, env_to_strings};
#[derive(Clone)]
pub struct Exec;
@ -35,7 +32,66 @@ On Windows based systems, Nushell will wait for the command to finish and then e
call: &Call,
_input: PipelineData,
) -> Result<PipelineData, ShellError> {
exec(engine_state, stack, call)
let cwd = engine_state.cwd(Some(stack))?;
// Find the absolute path to the executable. If the command is not
// found, display a helpful error message.
let name: Spanned<String> = call.req(engine_state, stack, 0)?;
let executable = {
let paths = nu_engine::env::path_str(engine_state, stack, call.head)?;
let Some(executable) = crate::which(&name.item, &paths, &cwd) else {
return Err(crate::command_not_found(
&name.item,
call.head,
engine_state,
stack,
));
};
executable
};
// Create the command.
let mut command = std::process::Command::new(executable);
// Configure PWD.
command.current_dir(cwd);
// Configure environment variables.
let envs = env_to_strings(engine_state, stack)?;
command.env_clear();
command.envs(envs);
// Configure args.
let args = crate::eval_arguments_from_call(engine_state, stack, call)?;
command.args(args.into_iter().map(|s| s.item));
// Execute the child process, replacing/terminating the current process
// depending on platform.
#[cfg(unix)]
{
use std::os::unix::process::CommandExt;
let err = command.exec();
Err(ShellError::ExternalCommand {
label: "Failed to exec into new process".into(),
help: err.to_string(),
span: call.head,
})
}
#[cfg(windows)]
{
let mut child = command.spawn().map_err(|err| ShellError::ExternalCommand {
label: "Failed to exec into new process".into(),
help: err.to_string(),
span: call.head,
})?;
let status = child.wait().map_err(|err| ShellError::ExternalCommand {
label: "Failed to wait for child process".into(),
help: err.to_string(),
span: call.head,
})?;
std::process::exit(status.code().expect("status.code() succeeds on Windows"))
}
}
fn examples(&self) -> Vec<Example> {
@ -53,57 +109,3 @@ On Windows based systems, Nushell will wait for the command to finish and then e
]
}
}
fn exec(
engine_state: &EngineState,
stack: &mut Stack,
call: &Call,
) -> Result<PipelineData, ShellError> {
let mut external_command = create_external_command(engine_state, stack, call)?;
external_command.out = OutDest::Inherit;
external_command.err = OutDest::Inherit;
#[allow(deprecated)]
let cwd = current_dir(engine_state, stack)?;
let mut command = external_command.spawn_simple_command(&cwd.to_string_lossy())?;
command.current_dir(cwd);
command.envs(external_command.env_vars);
// this either replaces our process and should not return,
// or the exec fails and we get an error back
exec_impl(command, call.head)
}
#[cfg(unix)]
fn exec_impl(mut command: std::process::Command, span: Span) -> Result<PipelineData, ShellError> {
use std::os::unix::process::CommandExt;
let error = command.exec();
Err(ShellError::GenericError {
error: "Error on exec".into(),
msg: error.to_string(),
span: Some(span),
help: None,
inner: vec![],
})
}
#[cfg(windows)]
fn exec_impl(mut command: std::process::Command, span: Span) -> Result<PipelineData, ShellError> {
match command.spawn() {
Ok(mut child) => match child.wait() {
Ok(status) => std::process::exit(status.code().unwrap_or(0)),
Err(e) => Err(ShellError::ExternalCommand {
label: "Error in external command".into(),
help: e.to_string(),
span,
}),
},
Err(e) => Err(ShellError::ExternalCommand {
label: "Error spawning external command".into(),
help: e.to_string(),
span,
}),
}
}

2
crates/nu-command/src/system/mod.rs
View File

@ -33,7 +33,7 @@ pub use nu_check::NuCheck;
pub use ps::Ps;
#[cfg(windows)]
pub use registry_query::RegistryQuery;
pub use run_external::{External, ExternalCommand};
pub use run_external::{command_not_found, eval_arguments_from_call, which, External};
pub use sys::*;
pub use uname::UName;
pub use which_::Which;

1217
crates/nu-command/src/system/run_external.rs
View File

File diff suppressed because it is too large Load Diff

4
crates/nu-command/tests/commands/all.rs
View File

@ -55,7 +55,9 @@ fn checks_if_all_returns_error_with_invalid_command() {
"#
));
assert!(actual.err.contains("can't run executable") || actual.err.contains("did you mean"));
assert!(
actual.err.contains("Command `st` not found") && actual.err.contains("Did you mean `ast`?")
);
}
#[test]

4
crates/nu-command/tests/commands/any.rs
View File

@ -41,7 +41,9 @@ fn checks_if_any_returns_error_with_invalid_command() {
"#
));
assert!(actual.err.contains("can't run executable") || actual.err.contains("did you mean"));
assert!(
actual.err.contains("Command `st` not found") && actual.err.contains("Did you mean `ast`?")
);
}
#[test]

2
crates/nu-command/tests/commands/complete.rs
View File

@ -24,7 +24,7 @@ fn basic_exit_code() {
#[test]
fn error() {
let actual = nu!("not-found | complete");
assert!(actual.err.contains("executable was not found"));
assert!(actual.err.contains("Command `not-found` not found"));
}
#[test]

8
crates/nu-command/tests/commands/source_env.rs
View File

@ -283,13 +283,17 @@ fn source_env_is_scoped() {
let actual = nu!(cwd: dirs.test(), &inp.join("; "));
assert!(actual.err.contains("executable was not found"));
assert!(actual
.err
.contains("Command `no-name-similar-to-this` not found"));
let inp = &[r#"source-env spam.nu"#, r#"nor-similar-to-this"#];
let actual = nu!(cwd: dirs.test(), &inp.join("; "));
assert!(actual.err.contains("executable was not found"));
assert!(actual
.err
.contains("Command `nor-similar-to-this` not found"));
})
}

10
crates/nu-command/tests/commands/use_.rs
View File

@ -189,9 +189,7 @@ fn use_module_creates_accurate_did_you_mean_1() {
let actual = nu!(r#"
module spam { export def foo [] { "foo" } }; use spam; foo
"#);
assert!(actual.err.contains(
"command 'foo' was not found but it was imported from module 'spam'; try using `spam foo`"
));
assert!(actual.err.contains("Did you mean `spam foo`"));
}
#[test]
@ -199,9 +197,9 @@ fn use_module_creates_accurate_did_you_mean_2() {
let actual = nu!(r#"
module spam { export def foo [] { "foo" } }; foo
"#);
assert!(actual.err.contains(
"command 'foo' was not found but it exists in module 'spam'; try importing it with `use`"
));
assert!(actual
.err
.contains("A command with that name exists in module `spam`"));
}
#[test]

1
src/test_bins.rs
View File

@ -242,6 +242,7 @@ pub fn nu_repl() {
let mut top_stack = Arc::new(Stack::new());
engine_state.add_env_var("PWD".into(), Value::test_string(cwd.to_string_lossy()));
engine_state.add_env_var("PATH".into(), Value::test_string(""));
let mut last_output = String::new();

12
tests/repl/test_known_external.rs
View File

@ -108,14 +108,14 @@ fn known_external_misc_values() -> TestResult {
/// GitHub issue #7822
#[test]
fn known_external_subcommand_from_module() -> TestResult {
let output = Command::new("cargo").arg("check").arg("-h").output()?;
let output = Command::new("cargo").arg("add").arg("-h").output()?;
run_test(
r#"
module cargo {
export extern check []
export extern add []
};
use cargo;
cargo check -h
cargo add -h
"#,
String::from_utf8(output.stdout)?.trim(),
)
@ -124,14 +124,14 @@ fn known_external_subcommand_from_module() -> TestResult {
/// GitHub issue #7822
#[test]
fn known_external_aliased_subcommand_from_module() -> TestResult {
let output = Command::new("cargo").arg("check").arg("-h").output()?;
let output = Command::new("cargo").arg("add").arg("-h").output()?;
run_test(
r#"
module cargo {
export extern check []
export extern add []
};
use cargo;
alias cc = cargo check;
alias cc = cargo add;
cc -h
"#,
String::from_utf8(output.stdout)?.trim(),

4
tests/repl/test_parser.rs
View File

@ -647,7 +647,7 @@ fn duration_with_underscores_2() -> TestResult {
#[test]
fn duration_with_underscores_3() -> TestResult {
fail_test("1_000_d_ay", "executable was not found")
fail_test("1_000_d_ay", "Command `1_000_d_ay` not found")
}
#[test]
@ -667,7 +667,7 @@ fn filesize_with_underscores_2() -> TestResult {
#[test]
fn filesize_with_underscores_3() -> TestResult {
fail_test("42m_b", "executable was not found")
fail_test("42m_b", "Command `42m_b` not found")
}
#[test]

7
tests/repl/test_spread.rs
View File

@ -183,17 +183,14 @@ fn explain_spread_args() -> TestResult {
#[test]
fn disallow_implicit_spread_for_externals() -> TestResult {
fail_test(
r#"nu --testbin cococo [1 2]"#,
"Lists are not automatically spread",
)
fail_test(r#"^echo [1 2]"#, "Lists are not automatically spread")
}
#[test]
fn respect_shape() -> TestResult {
fail_test(
"def foo [...rest] { ...$rest }; foo bar baz",
"executable was not found",
"Command `...$rest` not found",
)
.unwrap();
fail_test("module foo { ...$bar }", "expected_keyword").unwrap();

6
tests/shell/pipeline/commands/external.rs
View File

@ -72,7 +72,9 @@ fn correctly_escape_external_arguments() {
fn escape_also_escapes_equals() {
let actual = nu!("^MYFOONAME=MYBARVALUE");
assert!(actual.err.contains("executable was not found"));
assert!(actual
.err
.contains("Command `MYFOONAME=MYBARVALUE` not found"));
}
#[test]
@ -127,7 +129,7 @@ fn command_not_found_error_shows_not_found_1() {
export extern "foo" [];
foo
"#);
assert!(actual.err.contains("'foo' was not found"));
assert!(actual.err.contains("Command `foo` not found"));
}
#[test]

11
tests/shell/pipeline/commands/internal.rs
View File

@ -989,7 +989,9 @@ fn hide_alias_hides_alias() {
"
));
assert!(actual.err.contains("did you mean 'all'?"));
assert!(
actual.err.contains("Command `ll` not found") && actual.err.contains("Did you mean `all`?")
);
}
mod parse {
@ -1035,7 +1037,7 @@ mod parse {
fn ensure_backticks_are_bareword_command() {
let actual = nu!("`8abc123`");
assert!(actual.err.contains("was not found"),);
assert!(actual.err.contains("Command `8abc123` not found"),);
}
}
@ -1146,5 +1148,8 @@ fn command_not_found_error_shows_not_found_2() {
export def --wrapped my-foo [...rest] { foo };
my-foo
"#);
assert!(actual.err.contains("did you mean"));
assert!(
actual.err.contains("Command `foo` not found")
&& actual.err.contains("Did you mean `for`?")
);
}