extern/nushell
1
0
Fork 1
mirror of https://github.com/nushell/nushell.git synced 2024-11-07 09:04:18 +01:00
Code Issues Packages Projects Releases Wiki Activity

Keep plugins persistently running in the background (#12064)

Browse Source 
# Description
This PR uses the new plugin protocol to intelligently keep plugin
processes running in the background for further plugin calls.

Running plugins can be seen by running the new `plugin list` command,
and stopped by running the new `plugin stop` command.

This is an enhancement for the performance of plugins, as starting new
plugin processes has overhead, especially for plugins in languages that
take a significant amount of time on startup. It also enables plugins
that have persistent state between commands, making the migration of
features like dataframes and `stor` to plugins possible.

Plugins are automatically stopped by the new plugin garbage collector,
configurable with `$env.config.plugin_gc`:

```nushell
  $env.config.plugin_gc = {
      # Configuration for plugin garbage collection
      default: {
          enabled: true # true to enable stopping of inactive plugins
          stop_after: 10sec # how long to wait after a plugin is inactive to stop it
      }
      plugins: {
          # alternate configuration for specific plugins, by name, for example:
          #
          # gstat: {
          #     enabled: false
          # }
      }
  }
```

If garbage collection is enabled, plugins will be stopped after
`stop_after` passes after they were last active. Plugins are counted as
inactive if they have no running plugin calls. Reading the stream from
the response of a plugin call is still considered to be activity, but if
a plugin holds on to a stream but the call ends without an active
streaming response, it is not counted as active even if it is reading
it. Plugins can explicitly disable the GC as appropriate with
`engine.set_gc_disabled(true)`.

The `version` command now lists plugin names rather than plugin
commands. The list of plugin commands is accessible via `plugin list`.

Recommend doing this together with #12029, because it will likely force
plugin developers to do the right thing with mutability and lead to less
unexpected behavior when running plugins nested / in parallel.

# User-Facing Changes
- new command: `plugin list`
- new command: `plugin stop`
- changed command: `version` (now lists plugin names, rather than
commands)
- new config: `$env.config.plugin_gc`
- Plugins will keep running and be reused, at least for the configured
GC period
- Plugins that used mutable state in weird ways like `inc` did might
misbehave until fixed
- Plugins can disable GC if they need to
- Had to change plugin signature to accept `&EngineInterface` so that
the GC disable feature works. #12029 does this anyway, and I'm expecting
(resolvable) conflicts with that

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

Because there is some specific OS behavior required for plugins to not
respond to Ctrl-C directly, I've developed against and tested on both
Linux and Windows to ensure that works properly.

# After Submitting
I think this probably needs to be in the book somewhere
This commit is contained in:
Devyn Cairns 2024-03-09 15:10:22 -08:00 committed by GitHub
parent 430fb1fcb6
commit bc19be25b1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
44 changed files with 2131 additions and 304 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Cargo.lock generated
View File

@ -3072,6 +3072,7 @@ dependencies = [
"serde",
"serde_json",
"typetag",
"windows 0.52.0",
]
[[package]]

9
crates/nu-cmd-lang/src/core_commands/mod.rs
View File

@ -71,8 +71,13 @@ pub use try_::Try;
pub use use_::Use;
pub use version::Version;
pub use while_::While;
//#[cfg(feature = "plugin")]
mod plugin;
mod plugin_list;
mod plugin_stop;
mod register;
//#[cfg(feature = "plugin")]
pub use plugin::PluginCommand;
pub use plugin_list::PluginList;
pub use plugin_stop::PluginStop;
pub use register::Register;

64
crates/nu-cmd-lang/src/core_commands/plugin.rs Normal file
View File

@ -0,0 +1,64 @@
use nu_engine::get_full_help;
use nu_protocol::{
ast::Call,
engine::{Command, EngineState, Stack},
Category, Example, IntoPipelineData, PipelineData, ShellError, Signature, Type, Value,
};
#[derive(Clone)]
pub struct PluginCommand;
impl Command for PluginCommand {
fn name(&self) -> &str {
"plugin"
}
fn signature(&self) -> Signature {
Signature::build("plugin")
.input_output_types(vec![(Type::Nothing, Type::Nothing)])
.category(Category::Core)
}
fn usage(&self) -> &str {
"Commands for managing plugins."
}
fn extra_usage(&self) -> &str {
"To load a plugin, see `register`."
}
fn run(
&self,
engine_state: &EngineState,
stack: &mut Stack,
call: &Call,
_input: PipelineData,
) -> Result<PipelineData, ShellError> {
Ok(Value::string(
get_full_help(
&PluginCommand.signature(),
&PluginCommand.examples(),
engine_state,
stack,
self.is_parser_keyword(),
),
call.head,
)
.into_pipeline_data())
}
fn examples(&self) -> Vec<Example> {
vec![
Example {
example: "plugin list",
description: "List installed plugins",
result: None,
},
Example {
example: "plugin stop inc",
description: "Stop the plugin named `inc`.",
result: None,
},
]
}
}

101
crates/nu-cmd-lang/src/core_commands/plugin_list.rs Normal file
View File

@ -0,0 +1,101 @@
use itertools::Itertools;
use nu_protocol::{
ast::Call,
engine::{Command, EngineState, Stack},
record, Category, Example, IntoInterruptiblePipelineData, PipelineData, ShellError, Signature,
Type, Value,
};
#[derive(Clone)]
pub struct PluginList;
impl Command for PluginList {
fn name(&self) -> &str {
"plugin list"
}
fn signature(&self) -> Signature {
Signature::build("plugin list")
.input_output_type(
Type::Nothing,
Type::Table(vec![
("name".into(), Type::String),
("is_running".into(), Type::Bool),
("pid".into(), Type::Int),
("filename".into(), Type::String),
("shell".into(), Type::String),
("commands".into(), Type::List(Type::String.into())),
]),
)
.category(Category::Core)
}
fn usage(&self) -> &str {
"List installed plugins."
}
fn examples(&self) -> Vec<nu_protocol::Example> {
vec![
Example {
example: "plugin list",
description: "List installed plugins.",
result: Some(Value::test_list(vec![Value::test_record(record! {
"name" => Value::test_string("inc"),
"is_running" => Value::test_bool(true),
"pid" => Value::test_int(106480),
"filename" => if cfg!(windows) {
Value::test_string(r"C:\nu\plugins\nu_plugin_inc.exe")
} else {
Value::test_string("/opt/nu/plugins/nu_plugin_inc")
},
"shell" => Value::test_nothing(),
"commands" => Value::test_list(vec![Value::test_string("inc")]),
})])),
},
Example {
example: "ps | where pid in (plugin list).pid",
description: "Get process information for running plugins.",
result: None,
},
]
}
fn run(
&self,
engine_state: &EngineState,
_stack: &mut Stack,
call: &Call,
_input: PipelineData,
) -> Result<PipelineData, ShellError> {
let span = call.span();
// Group plugin decls by plugin identity
let decls = engine_state.plugin_decls().into_group_map_by(|decl| {
decl.plugin_identity()
.expect("plugin decl should have identity")
});
// Build plugins list
let list = engine_state.plugins().iter().map(|plugin| {
// Find commands that belong to the plugin
let commands = decls.get(plugin.identity())
.into_iter()
.flat_map(|decls| {
decls.iter().map(|decl| Value::string(decl.name(), span))
})
.collect();
Value::record(record! {
"name" => Value::string(plugin.identity().name(), span),
"is_running" => Value::bool(plugin.is_running(), span),
"pid" => plugin.pid()
.map(|p| Value::int(p as i64, span))
.unwrap_or(Value::nothing(span)),
"filename" => Value::string(plugin.identity().filename().to_string_lossy(), span),
"shell" => plugin.identity().shell()
.map(|s| Value::string(s.to_string_lossy(), span))
.unwrap_or(Value::nothing(span)),
"commands" => Value::list(commands, span),
}, span)
}).collect::<Vec<Value>>();
Ok(list.into_pipeline_data(engine_state.ctrlc.clone()))
}
}

75
crates/nu-cmd-lang/src/core_commands/plugin_stop.rs Normal file
View File

@ -0,0 +1,75 @@
use nu_engine::CallExt;
use nu_protocol::{
ast::Call,
engine::{Command, EngineState, Stack},
Category, Example, PipelineData, ShellError, Signature, Spanned, SyntaxShape, Type,
};
#[derive(Clone)]
pub struct PluginStop;
impl Command for PluginStop {
fn name(&self) -> &str {
"plugin stop"
}
fn signature(&self) -> Signature {
Signature::build("plugin stop")
.input_output_type(Type::Nothing, Type::Nothing)
.required(
"name",
SyntaxShape::String,
"The name of the plugin to stop.",
)
.category(Category::Core)
}
fn usage(&self) -> &str {
"Stop an installed plugin if it was running."
}
fn examples(&self) -> Vec<nu_protocol::Example> {
vec![
Example {
example: "plugin stop inc",
description: "Stop the plugin named `inc`.",
result: None,
},
Example {
example: "plugin list | each { |p| plugin stop $p.name }",
description: "Stop all plugins.",
result: None,
},
]
}
fn run(
&self,
engine_state: &EngineState,
stack: &mut Stack,
call: &Call,
_input: PipelineData,
) -> Result<PipelineData, ShellError> {
let name: Spanned<String> = call.req(engine_state, stack, 0)?;
let mut found = false;
for plugin in engine_state.plugins() {
if plugin.identity().name() == name.item {
plugin.stop()?;
found = true;
}
}
if found {
Ok(PipelineData::Empty)
} else {
Err(ShellError::GenericError {
error: format!("Failed to stop the `{}` plugin", name.item),
msg: "couldn't find a plugin with this name".into(),
span: Some(name.span),
help: Some("you may need to `register` the plugin first".into()),
inner: vec![],
})
}
}
}

8
crates/nu-cmd-lang/src/core_commands/version.rs
View File

@ -130,11 +130,11 @@ pub fn version(engine_state: &EngineState, call: &Call) -> Result<PipelineData,
Value::string(features_enabled().join(", "), call.head),
);
// Get a list of command names and check for plugins
// Get a list of plugin names
let installed_plugins = engine_state
.plugin_decls()
.filter(|x| x.is_plugin().is_some())
.map(|x| x.name())
.plugins()
.iter()
.map(|x| x.identity().name())
.collect::<Vec<_>>();
record.push(

2
crates/nu-cmd-lang/src/default_context.rs
View File

@ -65,7 +65,7 @@ pub fn create_default_context() -> EngineState {
};
//#[cfg(feature = "plugin")]
bind_command!(Register);
bind_command!(PluginCommand, PluginList, PluginStop, Register,);
working_set.render()
};

2
crates/nu-engine/src/scope.rs
View File

@ -115,7 +115,7 @@ impl<'e, 's> ScopeData<'e, 's> {
// we can only be a is_builtin or is_custom, not both
"is_builtin" => Value::bool(!decl.is_custom_command(), span),
"is_sub" => Value::bool(decl.is_sub(), span),
"is_plugin" => Value::bool(decl.is_plugin().is_some(), span),
"is_plugin" => Value::bool(decl.is_plugin(), span),
"is_custom" => Value::bool(decl.is_custom_command(), span),
"is_keyword" => Value::bool(decl.is_parser_keyword(), span),
"is_extern" => Value::bool(decl.is_known_external(), span),

82
crates/nu-parser/src/parse_keywords.rs
View File

@ -13,8 +13,8 @@ use nu_protocol::{
},
engine::{StateWorkingSet, DEFAULT_OVERLAY_NAME},
eval_const::eval_constant,
span, Alias, BlockId, DeclId, Exportable, Module, ModuleId, ParseError, PositionalArg,
ResolvedImportPattern, Span, Spanned, SyntaxShape, Type, Value, VarId,
span, Alias, BlockId, DeclId, Exportable, IntoSpanned, Module, ModuleId, ParseError,
PositionalArg, ResolvedImportPattern, Span, Spanned, SyntaxShape, Type, Value, VarId,
};
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf};
@ -3543,8 +3543,10 @@ pub fn parse_where(working_set: &mut StateWorkingSet, spans: &[Span]) -> Pipelin
#[cfg(feature = "plugin")]
pub fn parse_register(working_set: &mut StateWorkingSet, spans: &[Span]) -> Pipeline {
use nu_plugin::{get_signature, PluginDeclaration};
use nu_protocol::{engine::Stack, PluginSignature};
use std::sync::Arc;
use nu_plugin::{get_signature, PersistentPlugin, PluginDeclaration};
use nu_protocol::{engine::Stack, PluginIdentity, PluginSignature, RegisteredPlugin};
let cwd = working_set.get_cwd();
@ -3671,35 +3673,61 @@ pub fn parse_register(working_set: &mut StateWorkingSet, spans: &[Span]) -> Pipe
// We need the current environment variables for `python` based plugins
// Or we'll likely have a problem when a plugin is implemented in a virtual Python environment.
let stack = Stack::new();
let current_envs =
nu_engine::env::env_to_strings(working_set.permanent_state, &stack).unwrap_or_default();
let get_envs = || {
let stack = Stack::new();
nu_engine::env::env_to_strings(working_set.permanent_state, &stack)
};
let error = arguments.and_then(|(path, path_span)| {
let path = path.path_buf();
// restrict plugin file name starts with `nu_plugin_`
let valid_plugin_name = path
.file_name()
.map(|s| s.to_string_lossy().starts_with("nu_plugin_"));
let Some(true) = valid_plugin_name else {
return Err(ParseError::LabeledError(
"Register plugin failed".into(),
"plugin name must start with nu_plugin_".into(),
path_span,
));
};
// Create the plugin identity. This validates that the plugin name starts with `nu_plugin_`
let identity =
PluginIdentity::new(path, shell).map_err(|err| err.into_spanned(path_span))?;
// Find garbage collection config
let gc_config = working_set
.get_config()
.plugin_gc
.get(identity.name())
.clone();
// Add it to the working set
let plugin = working_set.find_or_create_plugin(&identity, || {
Arc::new(PersistentPlugin::new(identity.clone(), gc_config))
});
// Downcast the plugin to `PersistentPlugin` - we generally expect this to succeed. The
// trait object only exists so that nu-protocol can contain plugins without knowing anything
// about their implementation, but we only use `PersistentPlugin` in practice.
let plugin: Arc<PersistentPlugin> = plugin.as_any().downcast().map_err(|_| {
ParseError::InternalError(
"encountered unexpected RegisteredPlugin type".into(),
spans[0],
)
})?;
let signatures = signature.map_or_else(
|| {
let signatures =
get_signature(&path, shell.as_deref(), &current_envs).map_err(|err| {
ParseError::LabeledError(
"Error getting signatures".into(),
err.to_string(),
spans[0],
)
});
// It's important that the plugin is restarted if we're going to get signatures
//
// The user would expect that `register` would always run the binary to get new
// signatures, in case it was replaced with an updated binary
plugin.stop().map_err(|err| {
ParseError::LabeledError(
"Failed to restart plugin to get new signatures".into(),
err.to_string(),
spans[0],
)
})?;
let signatures = get_signature(plugin.clone(), get_envs).map_err(|err| {
ParseError::LabeledError(
"Error getting signatures".into(),
err.to_string(),
spans[0],
)
});
if signatures.is_ok() {
// mark plugins file as dirty only when the user is registering plugins
@ -3715,7 +3743,7 @@ pub fn parse_register(working_set: &mut StateWorkingSet, spans: &[Span]) -> Pipe
for signature in signatures {
// create plugin command declaration (need struct impl Command)
// store declaration in working set
let plugin_decl = PluginDeclaration::new(path.clone(), signature, shell.clone());
let plugin_decl = PluginDeclaration::new(&plugin, signature);
working_set.add_decl(Box::new(plugin_decl));
}

6
crates/nu-plugin/Cargo.toml
View File

@ -22,3 +22,9 @@ log = "0.4"
miette = { workspace = true }
semver = "1.0"
typetag = "0.2"
[target.'cfg(target_os = "windows")'.dependencies]
windows = { version = "0.52", features = [
# For setting process creation flags
"Win32_System_Threading",
] }

4
crates/nu-plugin/src/lib.rs
View File

@ -16,7 +16,7 @@
//! invoked by Nushell.
//!
//! ```rust,no_run
//! use nu_plugin::*;
//! use nu_plugin::{EvaluatedCall, LabeledError, MsgPackSerializer, Plugin, EngineInterface, serve_plugin};
//! use nu_protocol::{PluginSignature, Value};
//!
//! struct MyPlugin;
@ -55,7 +55,7 @@ pub use serializers::{json::JsonSerializer, msgpack::MsgPackSerializer};
// Used by other nu crates.
#[doc(hidden)]
pub use plugin::{get_signature, PluginDeclaration};
pub use plugin::{get_signature, PersistentPlugin, PluginDeclaration};
#[doc(hidden)]
pub use serializers::EncodingType;

6
crates/nu-plugin/src/plugin/context.rs
View File

@ -4,11 +4,9 @@ use nu_engine::get_eval_block_with_early_return;
use nu_protocol::{
ast::Call,
engine::{Closure, EngineState, Stack},
Config, PipelineData, ShellError, Span, Spanned, Value,
Config, PipelineData, PluginIdentity, ShellError, Span, Spanned, Value,
};
use super::PluginIdentity;
/// Object safe trait for abstracting operations required of the plugin context.
pub(crate) trait PluginExecutionContext: Send + Sync {
/// The [Span] for the command execution (`call.head`)
@ -81,7 +79,7 @@ impl PluginExecutionContext for PluginExecutionCommandContext {
Ok(self
.get_config()?
.plugins
.get(&self.identity.plugin_name)
.get(self.identity.name())
.cloned()
.map(|value| {
let span = value.span();

61
crates/nu-plugin/src/plugin/declaration.rs
View File

@ -1,28 +1,27 @@
use super::{PluginExecutionCommandContext, PluginIdentity};
use super::{PersistentPlugin, PluginExecutionCommandContext, PluginSource};
use crate::protocol::{CallInfo, EvaluatedCall};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use nu_engine::get_eval_expression;
use nu_protocol::engine::{Command, EngineState, Stack};
use nu_protocol::{ast::Call, PluginSignature, Signature};
use nu_protocol::{Example, PipelineData, ShellError};
use nu_protocol::{Example, PipelineData, PluginIdentity, RegisteredPlugin, ShellError};
#[doc(hidden)] // Note: not for plugin authors / only used in nu-parser
#[derive(Clone)]
pub struct PluginDeclaration {
name: String,
signature: PluginSignature,
identity: Arc<PluginIdentity>,
source: PluginSource,
}
impl PluginDeclaration {
pub fn new(filename: PathBuf, signature: PluginSignature, shell: Option<PathBuf>) -> Self {
pub fn new(plugin: &Arc<PersistentPlugin>, signature: PluginSignature) -> Self {
Self {
name: signature.sig.name.clone(),
signature,
identity: Arc::new(PluginIdentity::new(filename, shell)),
source: PluginSource::new(plugin),
}
}
}
@ -79,25 +78,37 @@ impl Command for PluginDeclaration {
let evaluated_call =
EvaluatedCall::try_from_call(call, engine_state, stack, eval_expression)?;
// We need the current environment variables for `python` based plugins
// Or we'll likely have a problem when a plugin is implemented in a virtual Python environment.
let current_envs = nu_engine::env::env_to_strings(engine_state, stack).unwrap_or_default();
// Get the engine config
let engine_config = nu_engine::get_config(engine_state, stack);
// Start the plugin
let plugin = self.identity.clone().spawn(current_envs).map_err(|err| {
let decl = engine_state.get_decl(call.decl_id);
ShellError::GenericError {
error: format!("Unable to spawn plugin for `{}`", decl.name()),
msg: err.to_string(),
span: Some(call.head),
help: None,
inner: vec![],
}
})?;
// Get, or start, the plugin.
let plugin = self
.source
.persistent(None)
.and_then(|p| {
// Set the garbage collector config from the local config before running
p.set_gc_config(engine_config.plugin_gc.get(p.identity().name()));
p.get(|| {
// We need the current environment variables for `python` based plugins. Or
// we'll likely have a problem when a plugin is implemented in a virtual Python
// environment.
nu_engine::env::env_to_strings(engine_state, stack)
})
})
.map_err(|err| {
let decl = engine_state.get_decl(call.decl_id);
ShellError::GenericError {
error: format!("Unable to spawn plugin for `{}`", decl.name()),
msg: err.to_string(),
span: Some(call.head),
help: None,
inner: vec![],
}
})?;
// Create the context to execute in - this supports engine calls and custom values
let context = Arc::new(PluginExecutionCommandContext::new(
self.identity.clone(),
self.source.identity.clone(),
engine_state,
stack,
call,
@ -113,7 +124,11 @@ impl Command for PluginDeclaration {
)
}
fn is_plugin(&self) -> Option<(&Path, Option<&Path>)> {
Some((&self.identity.filename, self.identity.shell.as_deref()))
fn is_plugin(&self) -> bool {
true
}
fn plugin_identity(&self) -> Option<&PluginIdentity> {
Some(&self.source.identity)
}
}

290
crates/nu-plugin/src/plugin/gc.rs Normal file
View File

@ -0,0 +1,290 @@
use std::{
sync::{mpsc, Arc, Weak},
thread,
time::{Duration, Instant},
};
use nu_protocol::{PluginGcConfig, RegisteredPlugin};
use crate::PersistentPlugin;
/// Plugin garbage collector
///
/// Many users don't want all of their plugins to stay running indefinitely after using them, so
/// this runs a thread that monitors the plugin's usage and stops it automatically if it meets
/// certain conditions of inactivity.
#[derive(Debug, Clone)]
pub struct PluginGc {
sender: mpsc::Sender<PluginGcMsg>,
}
impl PluginGc {
/// Start a new plugin garbage collector. Returns an error if the thread failed to spawn.
pub fn new(
config: PluginGcConfig,
plugin: &Arc<PersistentPlugin>,
) -> std::io::Result<PluginGc> {
let (sender, receiver) = mpsc::channel();
let mut state = PluginGcState {
config,
last_update: None,
locks: 0,
disabled: false,
plugin: Arc::downgrade(plugin),
name: plugin.identity().name().to_owned(),
};
thread::Builder::new()
.name(format!("plugin gc ({})", plugin.identity().name()))
.spawn(move || state.run(receiver))?;
Ok(PluginGc { sender })
}
/// Update the garbage collector config
pub fn set_config(&self, config: PluginGcConfig) {
let _ = self.sender.send(PluginGcMsg::SetConfig(config));
}
/// Increment the number of locks held by the plugin
pub fn increment_locks(&self, amount: i64) {
let _ = self.sender.send(PluginGcMsg::AddLocks(amount));
}
/// Decrement the number of locks held by the plugin
pub fn decrement_locks(&self, amount: i64) {
let _ = self.sender.send(PluginGcMsg::AddLocks(-amount));
}
/// Set whether the GC is disabled by explicit request from the plugin. This is separate from
/// the `enabled` option in the config, and overrides that option.
pub fn set_disabled(&self, disabled: bool) {
let _ = self.sender.send(PluginGcMsg::SetDisabled(disabled));
}
/// Tell the GC to stop tracking the plugin. The plugin will not be stopped. The GC cannot be
/// reactivated after this request - a new one must be created instead.
pub fn stop_tracking(&self) {
let _ = self.sender.send(PluginGcMsg::StopTracking);
}
/// Tell the GC that the plugin exited so that it can remove it from the persistent plugin.
///
/// The reason the plugin tells the GC rather than just stopping itself via `source` is that
/// it can't guarantee that the plugin currently pointed to by `source` is itself, but if the
/// GC is still running, it hasn't received [`.stop_tracking()`] yet, which means it should be
/// the right plugin.
pub fn exited(&self) {
let _ = self.sender.send(PluginGcMsg::Exited);
}
}
#[derive(Debug)]
enum PluginGcMsg {
SetConfig(PluginGcConfig),
AddLocks(i64),
SetDisabled(bool),
StopTracking,
Exited,
}
#[derive(Debug)]
struct PluginGcState {
config: PluginGcConfig,
last_update: Option<Instant>,
locks: i64,
disabled: bool,
plugin: Weak<PersistentPlugin>,
name: String,
}
impl PluginGcState {
fn next_timeout(&self, now: Instant) -> Option<Duration> {
if self.locks <= 0 && !self.disabled {
self.last_update
.zip(self.config.enabled.then_some(self.config.stop_after))
.map(|(last_update, stop_after)| {
// If configured to stop, and used at some point, calculate the difference
let stop_after_duration = Duration::from_nanos(stop_after.max(0) as u64);
let duration_since_last_update = now.duration_since(last_update);
stop_after_duration.saturating_sub(duration_since_last_update)
})
} else {
// Don't timeout if there are locks set, or disabled
None
}
}
// returns `Some()` if the GC should not continue to operate, with `true` if it should stop the
// plugin, or `false` if it should not
fn handle_message(&mut self, msg: PluginGcMsg) -> Option<bool> {
match msg {
PluginGcMsg::SetConfig(config) => {
self.config = config;
}
PluginGcMsg::AddLocks(amount) => {
self.locks += amount;
if self.locks < 0 {
log::warn!(
"Plugin GC ({name}) problem: locks count below zero after adding \
{amount}: locks={locks}",
name = self.name,
locks = self.locks,
);
}
// Any time locks are modified, that counts as activity
self.last_update = Some(Instant::now());
}
PluginGcMsg::SetDisabled(disabled) => {
self.disabled = disabled;
}
PluginGcMsg::StopTracking => {
// Immediately exit without stopping the plugin
return Some(false);
}
PluginGcMsg::Exited => {
// Exit and stop the plugin
return Some(true);
}
}
None
}
fn run(&mut self, receiver: mpsc::Receiver<PluginGcMsg>) {
let mut always_stop = false;
loop {
let Some(msg) = (match self.next_timeout(Instant::now()) {
Some(duration) => receiver.recv_timeout(duration).ok(),
None => receiver.recv().ok(),
}) else {
// If the timeout was reached, or the channel is disconnected, break the loop
break;
};
log::trace!("Plugin GC ({name}) message: {msg:?}", name = self.name);
if let Some(should_stop) = self.handle_message(msg) {
// Exit the GC
if should_stop {
// If should_stop = true, attempt to stop the plugin
always_stop = true;
break;
} else {
// Don't stop the plugin
return;
}
}
}
// Upon exiting the loop, if the timeout reached zero, or we are exiting due to an Exited
// message, stop the plugin
if always_stop
|| self
.next_timeout(Instant::now())
.is_some_and(|t| t.is_zero())
{
// We only hold a weak reference, and it's not an error if we fail to upgrade it -
// that just means the plugin is definitely stopped anyway.
if let Some(plugin) = self.plugin.upgrade() {
let name = &self.name;
if let Err(err) = plugin.stop() {
log::warn!("Plugin `{name}` failed to be stopped by GC: {err}");
} else {
log::debug!("Plugin `{name}` successfully stopped by GC");
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn test_state() -> PluginGcState {
PluginGcState {
config: PluginGcConfig::default(),
last_update: None,
locks: 0,
disabled: false,
plugin: Weak::new(),
name: "test".into(),
}
}
#[test]
fn timeout_configured_as_zero() {
let now = Instant::now();
let mut state = test_state();
state.config.enabled = true;
state.config.stop_after = 0;
state.last_update = Some(now);
assert_eq!(Some(Duration::ZERO), state.next_timeout(now));
}
#[test]
fn timeout_past_deadline() {
let now = Instant::now();
let mut state = test_state();
state.config.enabled = true;
state.config.stop_after = Duration::from_secs(1).as_nanos() as i64;
state.last_update = Some(now - Duration::from_secs(2));
assert_eq!(Some(Duration::ZERO), state.next_timeout(now));
}
#[test]
fn timeout_with_deadline_in_future() {
let now = Instant::now();
let mut state = test_state();
state.config.enabled = true;
state.config.stop_after = Duration::from_secs(1).as_nanos() as i64;
state.last_update = Some(now);
assert_eq!(Some(Duration::from_secs(1)), state.next_timeout(now));
}
#[test]
fn no_timeout_if_disabled_by_config() {
let now = Instant::now();
let mut state = test_state();
state.config.enabled = false;
state.last_update = Some(now);
assert_eq!(None, state.next_timeout(now));
}
#[test]
fn no_timeout_if_disabled_by_plugin() {
let now = Instant::now();
let mut state = test_state();
state.config.enabled = true;
state.disabled = true;
state.last_update = Some(now);
assert_eq!(None, state.next_timeout(now));
}
#[test]
fn no_timeout_if_locks_count_over_zero() {
let now = Instant::now();
let mut state = test_state();
state.config.enabled = true;
state.locks = 1;
state.last_update = Some(now);
assert_eq!(None, state.next_timeout(now));
}
#[test]
fn adding_locks_changes_last_update() {
let mut state = test_state();
let original_last_update = Some(Instant::now() - Duration::from_secs(1));
state.last_update = original_last_update;
state.handle_message(PluginGcMsg::AddLocks(1));
assert_ne!(original_last_update, state.last_update, "not updated");
}
}

110
crates/nu-plugin/src/plugin/identity.rs
View File

@ -1,110 +0,0 @@
use std::{
ffi::OsStr,
path::{Path, PathBuf},
sync::Arc,
};
use nu_protocol::ShellError;
use super::{create_command, make_plugin_interface, PluginInterface};
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PluginIdentity {
/// The filename used to start the plugin
pub(crate) filename: PathBuf,
/// The shell used to start the plugin, if required
pub(crate) shell: Option<PathBuf>,
/// The friendly name of the plugin (e.g. `inc` for `C:\nu_plugin_inc.exe`)
pub(crate) plugin_name: String,
}
impl PluginIdentity {
pub(crate) fn new(filename: impl Into<PathBuf>, shell: Option<PathBuf>) -> PluginIdentity {
let filename = filename.into();
// `C:\nu_plugin_inc.exe` becomes `inc`
// `/home/nu/.cargo/bin/nu_plugin_inc` becomes `inc`
// any other path, including if it doesn't start with nu_plugin_, becomes
// `<invalid plugin name>`
let plugin_name = filename
.file_stem()
.map(|stem| stem.to_string_lossy().into_owned())
.and_then(|stem| stem.strip_prefix("nu_plugin_").map(|s| s.to_owned()))
.unwrap_or_else(|| {
log::warn!(
"filename `{}` is not a valid plugin name, must start with nu_plugin_",
filename.display()
);
"<invalid plugin name>".into()
});
PluginIdentity {
filename,
shell,
plugin_name,
}
}
#[cfg(all(test, windows))]
pub(crate) fn new_fake(name: &str) -> Arc<PluginIdentity> {
Arc::new(PluginIdentity::new(
format!(r"C:\fake\path\nu_plugin_{name}.exe"),
None,
))
}
#[cfg(all(test, not(windows)))]
pub(crate) fn new_fake(name: &str) -> Arc<PluginIdentity> {
Arc::new(PluginIdentity::new(
format!(r"/fake/path/nu_plugin_{name}"),
None,
))
}
/// Run the plugin command stored in this [`PluginIdentity`], then set up and return the
/// [`PluginInterface`] attached to it.
pub(crate) fn spawn(
self: Arc<Self>,
envs: impl IntoIterator<Item = (impl AsRef<OsStr>, impl AsRef<OsStr>)>,
) -> Result<PluginInterface, ShellError> {
let source_file = Path::new(&self.filename);
let mut plugin_cmd = create_command(source_file, self.shell.as_deref());
// We need the current environment variables for `python` based plugins
// Or we'll likely have a problem when a plugin is implemented in a virtual Python environment.
plugin_cmd.envs(envs);
let program_name = plugin_cmd.get_program().to_os_string().into_string();
// Run the plugin command
let child = plugin_cmd.spawn().map_err(|err| {
let error_msg = match err.kind() {
std::io::ErrorKind::NotFound => match program_name {
Ok(prog_name) => {
format!("Can't find {prog_name}, please make sure that {prog_name} is in PATH.")
}
_ => {
format!("Error spawning child process: {err}")
}
},
_ => {
format!("Error spawning child process: {err}")
}
};
ShellError::PluginFailedToLoad { msg: error_msg }
})?;
make_plugin_interface(child, self)
}
}
#[test]
fn parses_name_from_path() {
assert_eq!("test", PluginIdentity::new_fake("test").plugin_name);
assert_eq!(
"<invalid plugin name>",
PluginIdentity::new("other", None).plugin_name
);
assert_eq!(
"<invalid plugin name>",
PluginIdentity::new("", None).plugin_name
);
}

15
crates/nu-plugin/src/plugin/interface/engine.rs
View File

@ -13,7 +13,8 @@ use nu_protocol::{
use crate::{
protocol::{
CallInfo, CustomValueOp, EngineCall, EngineCallId, EngineCallResponse, PluginCall,
PluginCallId, PluginCallResponse, PluginCustomValue, PluginInput, ProtocolInfo,
PluginCallId, PluginCallResponse, PluginCustomValue, PluginInput, PluginOption,
ProtocolInfo,
},
LabeledError, PluginOutput,
};
@ -670,6 +671,18 @@ impl EngineInterface {
value => Ok(value),
}
}
/// Tell the engine whether to disable garbage collection for this plugin.
///
/// The garbage collector is enabled by default, but plugins can turn it off (ideally
/// temporarily) as necessary to implement functionality that requires the plugin to stay
/// running for longer than the engine can automatically determine.
///
/// The user can still stop the plugin if they want to with the `plugin stop` command.
pub fn set_gc_disabled(&self, disabled: bool) -> Result<(), ShellError> {
self.write(PluginOutput::Option(PluginOption::GcDisabled(disabled)))?;
self.flush()
}
}
impl Interface for EngineInterface {

148
crates/nu-plugin/src/plugin/interface/plugin.rs
View File

@ -2,7 +2,7 @@
use std::{
collections::{btree_map, BTreeMap},
sync::{mpsc, Arc},
sync::{mpsc, Arc, OnceLock},
};
use nu_protocol::{
@ -11,11 +11,11 @@ use nu_protocol::{
};
use crate::{
plugin::{context::PluginExecutionContext, PluginIdentity},
plugin::{context::PluginExecutionContext, gc::PluginGc, PluginSource},
protocol::{
CallInfo, CustomValueOp, EngineCall, EngineCallId, EngineCallResponse, PluginCall,
PluginCallId, PluginCallResponse, PluginCustomValue, PluginInput, PluginOutput,
ProtocolInfo, StreamId, StreamMessage,
PluginCallId, PluginCallResponse, PluginCustomValue, PluginInput, PluginOption,
PluginOutput, ProtocolInfo, StreamId, StreamMessage,
},
sequence::Sequence,
};
@ -63,14 +63,16 @@ impl std::ops::Deref for Context {
/// Internal shared state between the manager and each interface.
struct PluginInterfaceState {
/// The identity of the plugin being interfaced with
identity: Arc<PluginIdentity>,
/// The source to be used for custom values coming from / going to the plugin
source: Arc<PluginSource>,
/// Sequence for generating plugin call ids
plugin_call_id_sequence: Sequence,
/// Sequence for generating stream ids
stream_id_sequence: Sequence,
/// Sender to subscribe to a plugin call response
plugin_call_subscription_sender: mpsc::Sender<(PluginCallId, PluginCallSubscription)>,
/// An error that should be propagated to further plugin calls
error: OnceLock<ShellError>,
/// The synchronized output writer
writer: Box<dyn PluginWrite<PluginInput>>,
}
@ -78,7 +80,7 @@ struct PluginInterfaceState {
impl std::fmt::Debug for PluginInterfaceState {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("PluginInterfaceState")
.field("identity", &self.identity)
.field("source", &self.source)
.field("plugin_call_id_sequence", &self.plugin_call_id_sequence)
.field("stream_id_sequence", &self.stream_id_sequence)
.field(
@ -118,21 +120,24 @@ pub(crate) struct PluginInterfaceManager {
///
/// This is necessary so we know when we can remove context for plugin calls
plugin_call_input_streams: BTreeMap<StreamId, PluginCallId>,
/// Garbage collector handle, to notify about the state of the plugin
gc: Option<PluginGc>,
}
impl PluginInterfaceManager {
pub(crate) fn new(
identity: Arc<PluginIdentity>,
source: Arc<PluginSource>,
writer: impl PluginWrite<PluginInput> + 'static,
) -> PluginInterfaceManager {
let (subscription_tx, subscription_rx) = mpsc::channel();
PluginInterfaceManager {
state: Arc::new(PluginInterfaceState {
identity,
source,
plugin_call_id_sequence: Sequence::default(),
stream_id_sequence: Sequence::default(),
plugin_call_subscription_sender: subscription_tx,
error: OnceLock::new(),
writer: Box::new(writer),
}),
stream_manager: StreamManager::new(),
@ -140,9 +145,17 @@ impl PluginInterfaceManager {
plugin_call_subscriptions: BTreeMap::new(),
plugin_call_subscription_receiver: subscription_rx,
plugin_call_input_streams: BTreeMap::new(),
gc: None,
}
}
/// Add a garbage collector to this plugin. The manager will notify the garbage collector about
/// the state of the plugin so that it can be automatically cleaned up if the plugin is
/// inactive.
pub(crate) fn set_garbage_collector(&mut self, gc: Option<PluginGc>) {
self.gc = gc;
}
/// Consume pending messages in the `plugin_call_subscription_receiver`
fn receive_plugin_call_subscriptions(&mut self) {
while let Ok((id, subscription)) = self.plugin_call_subscription_receiver.try_recv() {
@ -154,16 +167,21 @@ impl PluginInterfaceManager {
}
}
/// Track the start of stream(s)
/// Track the start of incoming stream(s)
fn recv_stream_started(&mut self, call_id: PluginCallId, stream_id: StreamId) {
self.plugin_call_input_streams.insert(stream_id, call_id);
// Increment the number of streams on the subscription so context stays alive
self.receive_plugin_call_subscriptions();
if let Some(sub) = self.plugin_call_subscriptions.get_mut(&call_id) {
self.plugin_call_input_streams.insert(stream_id, call_id);
sub.remaining_streams_to_read += 1;
}
// Add a lock to the garbage collector for each stream
if let Some(ref gc) = self.gc {
gc.increment_locks(1);
}
}
/// Track the end of a stream
/// Track the end of an incoming stream
fn recv_stream_ended(&mut self, stream_id: StreamId) {
if let Some(call_id) = self.plugin_call_input_streams.remove(&stream_id) {
if let btree_map::Entry::Occupied(mut e) = self.plugin_call_subscriptions.entry(call_id)
@ -174,6 +192,11 @@ impl PluginInterfaceManager {
e.remove();
}
}
// Streams read from the plugin are tracked with locks on the GC so plugins don't get
// stopped if they have active streams
if let Some(ref gc) = self.gc {
gc.decrement_locks(1);
}
}
}
@ -336,12 +359,17 @@ impl PluginInterfaceManager {
&mut self,
mut reader: impl PluginRead<PluginOutput>,
) -> Result<(), ShellError> {
let mut result = Ok(());
while let Some(msg) = reader.read().transpose() {
if self.is_finished() {
break;
}
// We assume an error here is unrecoverable (at least, without restarting the plugin)
if let Err(err) = msg.and_then(|msg| self.consume(msg)) {
// Put the error in the state so that new calls see it
let _ = self.state.error.set(err.clone());
// Error to streams
let _ = self.stream_manager.broadcast_read_error(err.clone());
// Error to call waiters
@ -354,10 +382,16 @@ impl PluginInterfaceManager {
.as_ref()
.map(|s| s.send(ReceivedPluginCallMessage::Error(err.clone())));
}
return Err(err);
result = Err(err);
break;
}
}
Ok(())
// Tell the GC we are exiting so that the plugin doesn't get stuck open
if let Some(ref gc) = self.gc {
gc.exited();
}
result
}
}
@ -369,6 +403,7 @@ impl InterfaceManager for PluginInterfaceManager {
PluginInterface {
state: self.state.clone(),
stream_manager_handle: self.stream_manager.get_handle(),
gc: self.gc.clone(),
}
}
@ -387,7 +422,9 @@ impl InterfaceManager for PluginInterfaceManager {
msg: format!(
"Plugin `{}` is compiled for nushell version {}, \
which is not compatible with version {}",
self.state.identity.plugin_name, info.version, local_info.version,
self.state.source.name(),
info.version,
local_info.version,
),
})
}
@ -398,11 +435,20 @@ impl InterfaceManager for PluginInterfaceManager {
msg: format!(
"Failed to receive initial Hello message from `{}`. \
This plugin might be too old",
self.state.identity.plugin_name
self.state.source.name()
),
})
}
PluginOutput::Stream(message) => self.consume_stream_message(message),
PluginOutput::Option(option) => match option {
PluginOption::GcDisabled(disabled) => {
// Turn garbage collection off/on.
if let Some(ref gc) = self.gc {
gc.set_disabled(disabled);
}
Ok(())
}
},
PluginOutput::CallResponse(id, response) => {
// Handle reading the pipeline data, if any
let response = match response {
@ -413,17 +459,26 @@ impl InterfaceManager for PluginInterfaceManager {
// error response, but send it anyway
let exec_context = self.get_context(id)?;
let ctrlc = exec_context.as_ref().and_then(|c| c.0.ctrlc());
// Register the streams in the response
for stream_id in data.stream_ids() {
self.recv_stream_started(id, stream_id);
}
match self.read_pipeline_data(data, ctrlc) {
Ok(data) => PluginCallResponse::PipelineData(data),
Err(err) => PluginCallResponse::Error(err.into()),
}
}
};
self.send_plugin_call_response(id, response)
let result = self.send_plugin_call_response(id, response);
if result.is_ok() {
// When a call ends, it releases a lock on the GC
if let Some(ref gc) = self.gc {
gc.decrement_locks(1);
}
}
result
}
PluginOutput::EngineCall { context, id, call } => {
// Handle reading the pipeline data, if any
@ -441,7 +496,7 @@ impl InterfaceManager for PluginInterfaceManager {
} => {
// Add source to any plugin custom values in the arguments
for arg in positional.iter_mut() {
PluginCustomValue::add_source(arg, &self.state.identity);
PluginCustomValue::add_source(arg, &self.state.source);
}
self.read_pipeline_data(input, ctrlc)
.map(|input| EngineCall::EvalClosure {
@ -472,14 +527,14 @@ impl InterfaceManager for PluginInterfaceManager {
// Add source to any values
match data {
PipelineData::Value(ref mut value, _) => {
PluginCustomValue::add_source(value, &self.state.identity);
PluginCustomValue::add_source(value, &self.state.source);
Ok(data)
}
PipelineData::ListStream(ListStream { stream, ctrlc, .. }, meta) => {
let identity = self.state.identity.clone();
let source = self.state.source.clone();
Ok(stream
.map(move |mut value| {
PluginCustomValue::add_source(&mut value, &identity);
PluginCustomValue::add_source(&mut value, &source);
value
})
.into_pipeline_data_with_metadata(meta, ctrlc))
@ -489,7 +544,7 @@ impl InterfaceManager for PluginInterfaceManager {
}
fn consume_stream_message(&mut self, message: StreamMessage) -> Result<(), ShellError> {
// Keep track of streams that end so we know if we don't need the context anymore
// Keep track of streams that end
if let StreamMessage::End(id) = message {
self.recv_stream_ended(id);
}
@ -504,6 +559,8 @@ pub(crate) struct PluginInterface {
state: Arc<PluginInterfaceState>,
/// Handle to stream manager
stream_manager_handle: StreamManagerHandle,
/// Handle to plugin garbage collector
gc: Option<PluginGc>,
}
impl PluginInterface {
@ -580,7 +637,7 @@ impl PluginInterface {
mut call,
input,
}) => {
verify_call_args(&mut call, &self.state.identity)?;
verify_call_args(&mut call, &self.state.source)?;
let (header, writer) = self.init_write_pipeline_data(input)?;
(
PluginCall::Run(CallInfo {
@ -604,9 +661,28 @@ impl PluginInterface {
remaining_streams_to_read: 0,
},
))
.map_err(|_| ShellError::NushellFailed {
msg: "PluginInterfaceManager hung up and is no longer accepting plugin calls"
.into(),
.map_err(|_| ShellError::GenericError {
error: format!("Plugin `{}` closed unexpectedly", self.state.source.name()),
msg: "can't complete this operation because the plugin is closed".into(),
span: match &call {
PluginCall::CustomValueOp(value, _) => Some(value.span),
PluginCall::Run(info) => Some(info.call.head),
_ => None,
},
help: Some(format!(
"the plugin may have experienced an error. Try registering the plugin again \
with `{}`",
if let Some(shell) = self.state.source.shell() {
format!(
"register --shell '{}' '{}'",
shell.display(),
self.state.source.filename().display(),
)
} else {
format!("register '{}'", self.state.source.filename().display())
}
)),
inner: vec![],
})?;
// Write request
@ -681,6 +757,18 @@ impl PluginInterface {
call: PluginCall<PipelineData>,
context: &Option<Context>,
) -> Result<PluginCallResponse<PipelineData>, ShellError> {
// Check for an error in the state first, and return it if set.
if let Some(error) = self.state.error.get() {
return Err(error.clone());
}
// Starting a plugin call adds a lock on the GC. Locks are not added for streams being read
// by the plugin, so the plugin would have to explicitly tell us if it expects to stay alive
// while reading streams in the background after the response ends.
if let Some(ref gc) = self.gc {
gc.increment_locks(1);
}
let (writer, rx) = self.write_plugin_call(call, context.clone())?;
// Finish writing stream in the background
@ -737,7 +825,7 @@ impl PluginInterface {
/// Check that custom values in call arguments come from the right source
fn verify_call_args(
call: &mut crate::EvaluatedCall,
source: &Arc<PluginIdentity>,
source: &Arc<PluginSource>,
) -> Result<(), ShellError> {
for arg in call.positional.iter_mut() {
PluginCustomValue::verify_source(arg, source)?;
@ -772,14 +860,14 @@ impl Interface for PluginInterface {
// Validate the destination of values in the pipeline data
match data {
PipelineData::Value(mut value, meta) => {
PluginCustomValue::verify_source(&mut value, &self.state.identity)?;
PluginCustomValue::verify_source(&mut value, &self.state.source)?;
Ok(PipelineData::Value(value, meta))
}
PipelineData::ListStream(ListStream { stream, ctrlc, .. }, meta) => {
let identity = self.state.identity.clone();
let source = self.state.source.clone();
Ok(stream
.map(move |mut value| {
match PluginCustomValue::verify_source(&mut value, &identity) {
match PluginCustomValue::verify_source(&mut value, &source) {
Ok(()) => value,
// Put the error in the stream instead
Err(err) => Value::error(err, value.span()),

36
crates/nu-plugin/src/plugin/interface/plugin/tests.rs
View File

@ -12,7 +12,7 @@ use crate::{
plugin::{
context::PluginExecutionBogusContext,
interface::{test_util::TestCase, Interface, InterfaceManager},
PluginIdentity,
PluginSource,
},
protocol::{
test_util::{expected_test_custom_value, test_plugin_custom_value},
@ -214,17 +214,22 @@ fn manager_consume_all_propagates_io_error_to_plugin_calls() -> Result<(), Shell
.consume_all(&mut test)
.expect_err("consume_all did not error");
// We have to hold interface until now otherwise consume_all won't try to process the message
drop(interface);
let message = rx.try_recv().expect("failed to get plugin call message");
match message {
ReceivedPluginCallMessage::Error(error) => {
check_test_io_error(&error);
Ok(())
}
_ => panic!("received something other than an error: {message:?}"),
}
// Check that further calls also cause the error
match interface.get_signature() {
Ok(_) => panic!("plugin call after exit did not cause error somehow"),
Err(err) => {
check_test_io_error(&err);
Ok(())
}
}
}
#[test]
@ -242,17 +247,22 @@ fn manager_consume_all_propagates_message_error_to_plugin_calls() -> Result<(),
.consume_all(&mut test)
.expect_err("consume_all did not error");
// We have to hold interface until now otherwise consume_all won't try to process the message
drop(interface);
let message = rx.try_recv().expect("failed to get plugin call message");
match message {
ReceivedPluginCallMessage::Error(error) => {
check_invalid_output_error(&error);
Ok(())
}
_ => panic!("received something other than an error: {message:?}"),
}
// Check that further calls also cause the error
match interface.get_signature() {
Ok(_) => panic!("plugin call after exit did not cause error somehow"),
Err(err) => {
check_invalid_output_error(&err);
Ok(())
}
}
}
#[test]
@ -640,7 +650,7 @@ fn manager_prepare_pipeline_data_adds_source_to_values() -> Result<(), ShellErro
.expect("custom value is not a PluginCustomValue");
if let Some(source) = &custom_value.source {
assert_eq!("test", source.plugin_name);
assert_eq!("test", source.name());
} else {
panic!("source was not set");
}
@ -670,7 +680,7 @@ fn manager_prepare_pipeline_data_adds_source_to_list_streams() -> Result<(), She
.expect("custom value is not a PluginCustomValue");
if let Some(source) = &custom_value.source {
assert_eq!("test", source.plugin_name);
assert_eq!("test", source.name());
} else {
panic!("source was not set");
}
@ -1086,7 +1096,7 @@ fn normal_values(interface: &PluginInterface) -> Vec<Value> {
name: "SomeTest".into(),
data: vec![1, 2, 3],
// Has the same source, so it should be accepted
source: Some(interface.state.identity.clone()),
source: Some(interface.state.source.clone()),
})),
]
}
@ -1144,7 +1154,7 @@ fn bad_custom_values() -> Vec<Value> {
Value::test_custom_value(Box::new(PluginCustomValue {
name: "SomeTest".into(),
data: vec![1, 2, 3],
source: Some(PluginIdentity::new_fake("pluto")),
source: Some(PluginSource::new_fake("pluto").into()),
})),
]
}

4
crates/nu-plugin/src/plugin/interface/test_util.rs
View File

@ -5,7 +5,7 @@ use std::{
use nu_protocol::ShellError;
use crate::{plugin::PluginIdentity, protocol::PluginInput, PluginOutput};
use crate::{plugin::PluginSource, protocol::PluginInput, PluginOutput};
use super::{EngineInterfaceManager, PluginInterfaceManager, PluginRead, PluginWrite};
@ -131,7 +131,7 @@ impl<I, O> TestCase<I, O> {
impl TestCase<PluginOutput, PluginInput> {
/// Create a new [`PluginInterfaceManager`] that writes to this test case.
pub(crate) fn plugin(&self, name: &str) -> PluginInterfaceManager {
PluginInterfaceManager::new(PluginIdentity::new_fake(name), self.clone())
PluginInterfaceManager::new(PluginSource::new_fake(name).into(), self.clone())
}
}

75
crates/nu-plugin/src/plugin/mod.rs
View File

@ -1,7 +1,5 @@
mod declaration;
pub use declaration::PluginDeclaration;
use nu_engine::documentation::get_flags_section;
use std::collections::HashMap;
use std::ffi::OsStr;
use std::sync::mpsc::TrySendError;
use std::sync::{mpsc, Arc, Mutex};
@ -17,22 +15,35 @@ use std::path::Path;
use std::process::{Child, ChildStdout, Command as CommandSys, Stdio};
use std::{env, thread};
#[cfg(unix)]
use std::os::unix::process::CommandExt;
#[cfg(windows)]
use std::os::windows::process::CommandExt;
use nu_protocol::{PipelineData, PluginSignature, ShellError, Spanned, Value};
mod interface;
pub use interface::EngineInterface;
pub(crate) use interface::PluginInterface;
mod context;
pub(crate) use context::PluginExecutionCommandContext;
mod identity;
pub(crate) use identity::PluginIdentity;
use self::interface::{InterfaceManager, PluginInterfaceManager};
use self::gc::PluginGc;
use super::EvaluatedCall;
mod context;
mod declaration;
mod gc;
mod interface;
mod persistent;
mod source;
pub use declaration::PluginDeclaration;
pub use interface::EngineInterface;
pub use persistent::PersistentPlugin;
pub(crate) use context::PluginExecutionCommandContext;
pub(crate) use interface::PluginInterface;
pub(crate) use source::PluginSource;
use interface::{InterfaceManager, PluginInterfaceManager};
pub(crate) const OUTPUT_BUFFER_SIZE: usize = 8192;
/// Encoder for a specific message type. Usually implemented on [`PluginInput`]
@ -119,12 +130,19 @@ fn create_command(path: &Path, shell: Option<&Path>) -> CommandSys {
// Both stdout and stdin are piped so we can receive information from the plugin
process.stdout(Stdio::piped()).stdin(Stdio::piped());
// The plugin should be run in a new process group to prevent Ctrl-C from stopping it
#[cfg(unix)]
process.process_group(0);
#[cfg(windows)]
process.creation_flags(windows::Win32::System::Threading::CREATE_NEW_PROCESS_GROUP.0);
process
}
fn make_plugin_interface(
mut child: Child,
identity: Arc<PluginIdentity>,
source: Arc<PluginSource>,
gc: Option<PluginGc>,
) -> Result<PluginInterface, ShellError> {
let stdin = child
.stdin
@ -144,7 +162,9 @@ fn make_plugin_interface(
let reader = BufReader::with_capacity(OUTPUT_BUFFER_SIZE, stdout);
let mut manager = PluginInterfaceManager::new(identity, (Mutex::new(stdin), encoder));
let mut manager = PluginInterfaceManager::new(source.clone(), (Mutex::new(stdin), encoder));
manager.set_garbage_collector(gc);
let interface = manager.get_interface();
interface.hello()?;
@ -152,7 +172,10 @@ fn make_plugin_interface(
// we write, because we are expected to be able to handle multiple messages coming in from the
// plugin at any time, including stream messages like `Drop`.
std::thread::Builder::new()
.name("plugin interface reader".into())
.name(format!(
"plugin interface reader ({})",
source.identity.name()
))
.spawn(move || {
if let Err(err) = manager.consume_all((reader, encoder)) {
log::warn!("Error in PluginInterfaceManager: {err}");
@ -170,14 +193,16 @@ fn make_plugin_interface(
}
#[doc(hidden)] // Note: not for plugin authors / only used in nu-parser
pub fn get_signature(
path: &Path,
shell: Option<&Path>,
current_envs: &HashMap<String, String>,
) -> Result<Vec<PluginSignature>, ShellError> {
Arc::new(PluginIdentity::new(path, shell.map(|s| s.to_owned())))
.spawn(current_envs)?
.get_signature()
pub fn get_signature<E, K, V>(
plugin: Arc<PersistentPlugin>,
envs: impl FnOnce() -> Result<E, ShellError>,
) -> Result<Vec<PluginSignature>, ShellError>
where
E: IntoIterator<Item = (K, V)>,
K: AsRef<OsStr>,
V: AsRef<OsStr>,
{
plugin.get(envs)?.get_signature()
}
/// The basic API for a Nushell plugin

186
crates/nu-plugin/src/plugin/persistent.rs Normal file
View File

@ -0,0 +1,186 @@
use std::{
ffi::OsStr,
sync::{Arc, Mutex},
};
use nu_protocol::{PluginGcConfig, PluginIdentity, RegisteredPlugin, ShellError};
use super::{create_command, gc::PluginGc, make_plugin_interface, PluginInterface, PluginSource};
/// A box that can keep a plugin that was spawned persistent for further uses. The plugin may or
/// may not be currently running. [`.get()`] gets the currently running plugin, or spawns it if it's
/// not running.
///
/// Note: used in the parser, not for plugin authors
#[doc(hidden)]
#[derive(Debug)]
pub struct PersistentPlugin {
/// Identity (filename, shell, name) of the plugin
identity: PluginIdentity,
/// Reference to the plugin if running
running: Mutex<Option<RunningPlugin>>,
/// Garbage collector config
gc_config: Mutex<PluginGcConfig>,
}
#[derive(Debug)]
struct RunningPlugin {
/// Process ID of the running plugin
pid: u32,
/// Interface (which can be cloned) to the running plugin
interface: PluginInterface,
/// Garbage collector for the plugin
gc: PluginGc,
}
impl PersistentPlugin {
/// Create a new persistent plugin. The plugin will not be spawned immediately.
pub fn new(identity: PluginIdentity, gc_config: PluginGcConfig) -> PersistentPlugin {
PersistentPlugin {
identity,
running: Mutex::new(None),
gc_config: Mutex::new(gc_config),
}
}
/// Get the plugin interface of the running plugin, or spawn it if it's not currently running.
///
/// Will call `envs` to get environment variables to spawn the plugin if the plugin needs to be
/// spawned.
pub(crate) fn get<E, K, V>(
self: Arc<Self>,
envs: impl FnOnce() -> Result<E, ShellError>,
) -> Result<PluginInterface, ShellError>
where
E: IntoIterator<Item = (K, V)>,
K: AsRef<OsStr>,
V: AsRef<OsStr>,
{
let mut running = self.running.lock().map_err(|_| ShellError::NushellFailed {
msg: format!(
"plugin `{}` running mutex poisoned, probably panic during spawn",
self.identity.name()
),
})?;
if let Some(ref running) = *running {
// It exists, so just clone the interface
Ok(running.interface.clone())
} else {
// Try to spawn, and then store the spawned plugin if we were successful.
//
// We hold the lock the whole time to prevent others from trying to spawn and ending
// up with duplicate plugins
let new_running = self.clone().spawn(envs()?)?;
let interface = new_running.interface.clone();
*running = Some(new_running);
Ok(interface)
}
}
/// Run the plugin command, then set up and return [`RunningPlugin`].
fn spawn(
self: Arc<Self>,
envs: impl IntoIterator<Item = (impl AsRef<OsStr>, impl AsRef<OsStr>)>,
) -> Result<RunningPlugin, ShellError> {
let source_file = self.identity.filename();
let mut plugin_cmd = create_command(source_file, self.identity.shell());
// We need the current environment variables for `python` based plugins
// Or we'll likely have a problem when a plugin is implemented in a virtual Python environment.
plugin_cmd.envs(envs);
let program_name = plugin_cmd.get_program().to_os_string().into_string();
// Run the plugin command
let child = plugin_cmd.spawn().map_err(|err| {
let error_msg = match err.kind() {
std::io::ErrorKind::NotFound => match program_name {
Ok(prog_name) => {
format!("Can't find {prog_name}, please make sure that {prog_name} is in PATH.")
}
_ => {
format!("Error spawning child process: {err}")
}
},
_ => {
format!("Error spawning child process: {err}")
}
};
ShellError::PluginFailedToLoad { msg: error_msg }
})?;
// Start the plugin garbage collector
let gc_config =
self.gc_config
.lock()
.map(|c| c.clone())
.map_err(|_| ShellError::NushellFailed {
msg: "plugin gc mutex poisoned".into(),
})?;
let gc = PluginGc::new(gc_config, &self)?;
let pid = child.id();
let interface =
make_plugin_interface(child, Arc::new(PluginSource::new(&self)), Some(gc.clone()))?;
Ok(RunningPlugin { pid, interface, gc })
}
}
impl RegisteredPlugin for PersistentPlugin {
fn identity(&self) -> &PluginIdentity {
&self.identity
}
fn is_running(&self) -> bool {
// If the lock is poisoned, we return false here. That may not be correct, but this is a
// failure state anyway that would be noticed at some point
self.running.lock().map(|r| r.is_some()).unwrap_or(false)
}
fn pid(&self) -> Option<u32> {
// Again, we return None for a poisoned lock.
self.running
.lock()
.ok()
.and_then(|r| r.as_ref().map(|r| r.pid))
}
fn stop(&self) -> Result<(), ShellError> {
let mut running = self.running.lock().map_err(|_| ShellError::NushellFailed {
msg: format!(
"plugin `{}` running mutex poisoned, probably panic during spawn",
self.identity.name()
),
})?;
// If the plugin is running, stop its GC, so that the GC doesn't accidentally try to stop
// a future plugin
if let Some(running) = running.as_ref() {
running.gc.stop_tracking();
}
// We don't try to kill the process or anything, we just drop the RunningPlugin. It should
// exit soon after
*running = None;
Ok(())
}
fn set_gc_config(&self, gc_config: &PluginGcConfig) {
if let Ok(mut conf) = self.gc_config.lock() {
// Save the new config for future calls
*conf = gc_config.clone();
}
if let Ok(running) = self.running.lock() {
if let Some(running) = running.as_ref() {
// If the plugin is already running, propagate the config change to the running GC
running.gc.set_config(gc_config.clone());
}
}
}
fn as_any(self: Arc<Self>) -> Arc<dyn std::any::Any + Send + Sync> {
self
}
}

67
crates/nu-plugin/src/plugin/source.rs Normal file
View File

@ -0,0 +1,67 @@
use std::sync::{Arc, Weak};
use nu_protocol::{PluginIdentity, RegisteredPlugin, ShellError, Span};
use super::PersistentPlugin;
#[derive(Debug, Clone)]
pub(crate) struct PluginSource {
/// The identity of the plugin
pub(crate) identity: Arc<PluginIdentity>,
/// A weak reference to the persistent plugin that might hold an interface to the plugin.
///
/// This is weak to avoid cyclic references, but it does mean we might fail to upgrade if
/// the engine state lost the [`PersistentPlugin`] at some point.
pub(crate) persistent: Weak<PersistentPlugin>,
}
impl PluginSource {
/// Create from an `Arc<PersistentPlugin>`
pub(crate) fn new(plugin: &Arc<PersistentPlugin>) -> PluginSource {
PluginSource {
identity: plugin.identity().clone().into(),
persistent: Arc::downgrade(plugin),
}
}
/// Create a new fake source with a fake identity, for testing
///
/// Warning: [`.persistent()`] will always return an error.
#[cfg(test)]
pub(crate) fn new_fake(name: &str) -> PluginSource {
PluginSource {
identity: PluginIdentity::new_fake(name).into(),
persistent: Weak::new(),
}
}
/// Try to upgrade the persistent reference, and return an error referencing `span` as the
/// object that referenced it otherwise
pub(crate) fn persistent(
&self,
span: Option<Span>,
) -> Result<Arc<PersistentPlugin>, ShellError> {
self.persistent
.upgrade()
.ok_or_else(|| ShellError::GenericError {
error: format!("The `{}` plugin is no longer present", self.identity.name()),
msg: "removed since this object was created".into(),
span,
help: Some("try recreating the object that came from the plugin".into()),
inner: vec![],
})
}
/// Sources are compatible if their identities are equal
pub(crate) fn is_compatible(&self, other: &PluginSource) -> bool {
self.identity == other.identity
}
}
impl std::ops::Deref for PluginSource {
type Target = PluginIdentity;
fn deref(&self) -> &PluginIdentity {
&self.identity
}
}

12
crates/nu-plugin/src/protocol/mod.rs
View File

@ -320,6 +320,16 @@ impl PluginCallResponse<PipelineDataHeader> {
}
}
/// Options that can be changed to affect how the engine treats the plugin
#[derive(Serialize, Deserialize, Debug, Clone)]
pub enum PluginOption {
/// Send `GcDisabled(true)` to stop the plugin from being automatically garbage collected, or
/// `GcDisabled(false)` to enable it again.
///
/// See [`EngineInterface::set_gc_disabled`] for more information.
GcDisabled(bool),
}
/// Information received from the plugin
///
/// Note: exported for internal use, not public.
@ -328,6 +338,8 @@ impl PluginCallResponse<PipelineDataHeader> {
pub enum PluginOutput {
/// This must be the first message. Indicates supported protocol
Hello(ProtocolInfo),
/// Set option. No response expected
Option(PluginOption),
/// A response to a [`PluginCall`]. The ID should be the same sent with the plugin call this
/// is a response to
CallResponse(PluginCallId, PluginCallResponse<PipelineDataHeader>),

37
crates/nu-plugin/src/protocol/plugin_custom_value.rs
View File

@ -3,7 +3,7 @@ use std::sync::Arc;
use nu_protocol::{CustomValue, ShellError, Span, Spanned, Value};
use serde::{Deserialize, Serialize};
use crate::plugin::PluginIdentity;
use crate::plugin::PluginSource;
#[cfg(test)]
mod tests;
@ -17,7 +17,7 @@ mod tests;
/// that local plugin custom values are converted to and from [`PluginCustomData`] on the boundary.
///
/// [`PluginInterface`](crate::interface::PluginInterface) is responsible for adding the
/// appropriate [`PluginIdentity`](crate::plugin::PluginIdentity), ensuring that only
/// appropriate [`PluginSource`](crate::plugin::PluginSource), ensuring that only
/// [`PluginCustomData`] is contained within any values sent, and that the `source` of any
/// values sent matches the plugin it is being sent to.
#[derive(Clone, Debug, Serialize, Deserialize)]
@ -30,7 +30,7 @@ pub struct PluginCustomValue {
/// Which plugin the custom value came from. This is not defined on the plugin side. The engine
/// side is responsible for maintaining it, and it is not sent over the serialization boundary.
#[serde(skip, default)]
pub source: Option<Arc<PluginIdentity>>,
pub(crate) source: Option<Arc<PluginSource>>,
}
#[typetag::serde]
@ -52,7 +52,7 @@ impl CustomValue for PluginCustomValue {
"Unable to spawn plugin `{}` to get base value",
self.source
.as_ref()
.map(|s| s.plugin_name.as_str())
.map(|s| s.name())
.unwrap_or("<unknown>")
),
msg: err.to_string(),
@ -61,14 +61,18 @@ impl CustomValue for PluginCustomValue {
inner: vec![err],
};
let identity = self.source.clone().ok_or_else(|| {
let source = self.source.clone().ok_or_else(|| {
wrap_err(ShellError::NushellFailed {
msg: "The plugin source for the custom value was not set".into(),
})
})?;
let empty_env: Option<(String, String)> = None;
let plugin = identity.spawn(empty_env).map_err(wrap_err)?;
// Envs probably should be passed here, but it's likely that the plugin is already running
let empty_envs = std::iter::empty::<(&str, &str)>();
let plugin = source
.persistent(Some(span))
.and_then(|p| p.get(|| Ok(empty_envs)))
.map_err(wrap_err)?;
plugin
.custom_value_to_base_value(Spanned {
@ -117,8 +121,8 @@ impl PluginCustomValue {
})
}
/// Add a [`PluginIdentity`] to all [`PluginCustomValue`]s within a value, recursively.
pub(crate) fn add_source(value: &mut Value, source: &Arc<PluginIdentity>) {
/// Add a [`PluginSource`] to all [`PluginCustomValue`]s within a value, recursively.
pub(crate) fn add_source(value: &mut Value, source: &Arc<PluginSource>) {
let span = value.span();
match value {
// Set source on custom value
@ -179,21 +183,26 @@ impl PluginCustomValue {
/// since `LazyRecord` could return something different the next time it is called.
pub(crate) fn verify_source(
value: &mut Value,
source: &PluginIdentity,
source: &PluginSource,
) -> Result<(), ShellError> {
let span = value.span();
match value {
// Set source on custom value
Value::CustomValue { val, .. } => {
if let Some(custom_value) = val.as_any().downcast_ref::<PluginCustomValue>() {
if custom_value.source.as_deref() == Some(source) {
if custom_value
.source
.as_ref()
.map(|s| s.is_compatible(source))
.unwrap_or(false)
{
Ok(())
} else {
Err(ShellError::CustomValueIncorrectForPlugin {
name: custom_value.name.clone(),
span,
dest_plugin: source.plugin_name.clone(),
src_plugin: custom_value.source.as_ref().map(|s| s.plugin_name.clone()),
dest_plugin: source.name().to_owned(),
src_plugin: custom_value.source.as_ref().map(|s| s.name().to_owned()),
})
}
} else {
@ -201,7 +210,7 @@ impl PluginCustomValue {
Err(ShellError::CustomValueIncorrectForPlugin {
name: val.value_string(),
span,
dest_plugin: source.plugin_name.clone(),
dest_plugin: source.name().to_owned(),
src_plugin: None,
})
}

47
crates/nu-plugin/src/protocol/plugin_custom_value/tests.rs
View File

@ -1,9 +1,11 @@
use std::sync::Arc;
use nu_protocol::{
ast::RangeInclusion, engine::Closure, record, CustomValue, Range, ShellError, Span, Value,
};
use crate::{
plugin::PluginIdentity,
plugin::PluginSource,
protocol::test_util::{
expected_test_custom_value, test_plugin_custom_value, test_plugin_custom_value_with_source,
TestCustomValue,
@ -45,7 +47,7 @@ fn expected_serialize_output() -> Result<(), ShellError> {
#[test]
fn add_source_at_root() -> Result<(), ShellError> {
let mut val = Value::test_custom_value(Box::new(test_plugin_custom_value()));
let source = PluginIdentity::new_fake("foo");
let source = Arc::new(PluginSource::new_fake("foo"));
PluginCustomValue::add_source(&mut val, &source);
let custom_value = val.as_custom_value()?;
@ -53,7 +55,10 @@ fn add_source_at_root() -> Result<(), ShellError> {
.as_any()
.downcast_ref()
.expect("not PluginCustomValue");
assert_eq!(Some(source), plugin_custom_value.source);
assert_eq!(
Some(Arc::as_ptr(&source)),
plugin_custom_value.source.as_ref().map(Arc::as_ptr)
);
Ok(())
}
@ -84,7 +89,7 @@ fn add_source_nested_range() -> Result<(), ShellError> {
to: orig_custom_val.clone(),
inclusion: RangeInclusion::Inclusive,
});
let source = PluginIdentity::new_fake("foo");
let source = Arc::new(PluginSource::new_fake("foo"));
PluginCustomValue::add_source(&mut val, &source);
check_range_custom_values(&val, |name, custom_value| {
@ -93,8 +98,8 @@ fn add_source_nested_range() -> Result<(), ShellError> {
.downcast_ref()
.unwrap_or_else(|| panic!("{name} not PluginCustomValue"));
assert_eq!(
Some(&source),
plugin_custom_value.source.as_ref(),
Some(Arc::as_ptr(&source)),
plugin_custom_value.source.as_ref().map(Arc::as_ptr),
"{name} source not set correctly"
);
Ok(())
@ -126,7 +131,7 @@ fn add_source_nested_record() -> Result<(), ShellError> {
"foo" => orig_custom_val.clone(),
"bar" => orig_custom_val.clone(),
});
let source = PluginIdentity::new_fake("foo");
let source = Arc::new(PluginSource::new_fake("foo"));
PluginCustomValue::add_source(&mut val, &source);
check_record_custom_values(&val, &["foo", "bar"], |key, custom_value| {
@ -135,8 +140,8 @@ fn add_source_nested_record() -> Result<(), ShellError> {
.downcast_ref()
.unwrap_or_else(|| panic!("'{key}' not PluginCustomValue"));
assert_eq!(
Some(&source),
plugin_custom_value.source.as_ref(),
Some(Arc::as_ptr(&source)),
plugin_custom_value.source.as_ref().map(Arc::as_ptr),
"'{key}' source not set correctly"
);
Ok(())
@ -165,7 +170,7 @@ fn check_list_custom_values(
fn add_source_nested_list() -> Result<(), ShellError> {
let orig_custom_val = Value::test_custom_value(Box::new(test_plugin_custom_value()));
let mut val = Value::test_list(vec![orig_custom_val.clone(), orig_custom_val.clone()]);
let source = PluginIdentity::new_fake("foo");
let source = Arc::new(PluginSource::new_fake("foo"));
PluginCustomValue::add_source(&mut val, &source);
check_list_custom_values(&val, 0..=1, |index, custom_value| {
@ -174,8 +179,8 @@ fn add_source_nested_list() -> Result<(), ShellError> {
.downcast_ref()
.unwrap_or_else(|| panic!("[{index}] not PluginCustomValue"));
assert_eq!(
Some(&source),
plugin_custom_value.source.as_ref(),
Some(Arc::as_ptr(&source)),
plugin_custom_value.source.as_ref().map(Arc::as_ptr),
"[{index}] source not set correctly"
);
Ok(())
@ -209,7 +214,7 @@ fn add_source_nested_closure() -> Result<(), ShellError> {
block_id: 0,
captures: vec![(0, orig_custom_val.clone()), (1, orig_custom_val.clone())],
});
let source = PluginIdentity::new_fake("foo");
let source = Arc::new(PluginSource::new_fake("foo"));
PluginCustomValue::add_source(&mut val, &source);
check_closure_custom_values(&val, 0..=1, |index, custom_value| {
@ -218,8 +223,8 @@ fn add_source_nested_closure() -> Result<(), ShellError> {
.downcast_ref()
.unwrap_or_else(|| panic!("[{index}] not PluginCustomValue"));
assert_eq!(
Some(&source),
plugin_custom_value.source.as_ref(),
Some(Arc::as_ptr(&source)),
plugin_custom_value.source.as_ref().map(Arc::as_ptr),
"[{index}] source not set correctly"
);
Ok(())
@ -233,10 +238,10 @@ fn verify_source_error_message() -> Result<(), ShellError> {
let mut native_val = Value::custom_value(Box::new(TestCustomValue(32)), span);
let mut foreign_val = {
let mut val = test_plugin_custom_value();
val.source = Some(PluginIdentity::new_fake("other"));
val.source = Some(Arc::new(PluginSource::new_fake("other")));
Value::custom_value(Box::new(val), span)
};
let source = PluginIdentity::new_fake("test");
let source = PluginSource::new_fake("test");
PluginCustomValue::verify_source(&mut ok_val, &source).expect("ok_val should be verified ok");
@ -266,7 +271,7 @@ fn verify_source_error_message() -> Result<(), ShellError> {
#[test]
fn verify_source_nested_range() -> Result<(), ShellError> {
let native_val = Value::test_custom_value(Box::new(TestCustomValue(32)));
let source = PluginIdentity::new_fake("test");
let source = PluginSource::new_fake("test");
for (name, mut val) in [
(
"from",
@ -315,7 +320,7 @@ fn verify_source_nested_range() -> Result<(), ShellError> {
#[test]
fn verify_source_nested_record() -> Result<(), ShellError> {
let native_val = Value::test_custom_value(Box::new(TestCustomValue(32)));
let source = PluginIdentity::new_fake("test");
let source = PluginSource::new_fake("test");
for (name, mut val) in [
(
"first element foo",
@ -346,7 +351,7 @@ fn verify_source_nested_record() -> Result<(), ShellError> {
#[test]
fn verify_source_nested_list() -> Result<(), ShellError> {
let native_val = Value::test_custom_value(Box::new(TestCustomValue(32)));
let source = PluginIdentity::new_fake("test");
let source = PluginSource::new_fake("test");
for (name, mut val) in [
(
"first element",
@ -371,7 +376,7 @@ fn verify_source_nested_list() -> Result<(), ShellError> {
#[test]
fn verify_source_nested_closure() -> Result<(), ShellError> {
let native_val = Value::test_custom_value(Box::new(TestCustomValue(32)));
let source = PluginIdentity::new_fake("test");
let source = PluginSource::new_fake("test");
for (name, mut val) in [
(
"first capture",

4
crates/nu-plugin/src/protocol/test_util.rs
View File

@ -1,7 +1,7 @@
use nu_protocol::{CustomValue, ShellError, Span, Value};
use serde::{Deserialize, Serialize};
use crate::plugin::PluginIdentity;
use crate::plugin::PluginSource;
use super::PluginCustomValue;
@ -44,7 +44,7 @@ pub(crate) fn expected_test_custom_value() -> TestCustomValue {
pub(crate) fn test_plugin_custom_value_with_source() -> PluginCustomValue {
PluginCustomValue {
source: Some(PluginIdentity::new_fake("test")),
source: Some(PluginSource::new_fake("test").into()),
..test_plugin_custom_value()
}
}

26
crates/nu-plugin/src/serializers/tests.rs
View File

@ -2,8 +2,8 @@ macro_rules! generate_tests {
($encoder:expr) => {
use crate::protocol::{
CallInfo, CustomValueOp, EvaluatedCall, LabeledError, PipelineDataHeader, PluginCall,
PluginCallResponse, PluginCustomValue, PluginInput, PluginOutput, StreamData,
StreamMessage,
PluginCallResponse, PluginCustomValue, PluginInput, PluginOption, PluginOutput,
StreamData, StreamMessage,
};
use nu_protocol::{PluginSignature, Span, Spanned, SyntaxShape, Value};
@ -531,6 +531,28 @@ macro_rules! generate_tests {
_ => panic!("decoded into wrong value: {returned:?}"),
}
}
#[test]
fn output_round_trip_option() {
let plugin_output = PluginOutput::Option(PluginOption::GcDisabled(true));
let encoder = $encoder;
let mut buffer: Vec<u8> = Vec::new();
encoder
.encode(&plugin_output, &mut buffer)
.expect("unable to serialize message");
let returned = encoder
.decode(&mut buffer.as_slice())
.expect("unable to deserialize message")
.expect("eof");
match returned {
PluginOutput::Option(PluginOption::GcDisabled(disabled)) => {
assert!(disabled);
}
_ => panic!("decoded into wrong value: {returned:?}"),
}
}
};
}

8
crates/nu-protocol/src/config/mod.rs
View File

@ -13,6 +13,7 @@ pub use self::completer::CompletionAlgorithm;
pub use self::helper::extract_value;
pub use self::hooks::Hooks;
pub use self::output::ErrorStyle;
pub use self::plugin_gc::{PluginGcConfig, PluginGcConfigs};
pub use self::reedline::{
create_menus, EditBindings, HistoryFileFormat, NuCursorShape, ParsedKeybinding, ParsedMenu,
};
@ -22,6 +23,7 @@ mod completer;
mod helper;
mod hooks;
mod output;
mod plugin_gc;
mod reedline;
mod table;
@ -96,6 +98,8 @@ pub struct Config {
/// match the registered plugin name so `register nu_plugin_example` will be able to place
/// its configuration under a `nu_plugin_example` column.
pub plugins: HashMap<String, Value>,
/// Configuration for plugin garbage collection.
pub plugin_gc: PluginGcConfigs,
}
impl Default for Config {
@ -162,6 +166,7 @@ impl Default for Config {
highlight_resolved_externals: false,
plugins: HashMap::new(),
plugin_gc: PluginGcConfigs::default(),
}
}
}
@ -671,6 +676,9 @@ impl Value {
);
}
}
"plugin_gc" => {
config.plugin_gc.process(&[key], value, &mut errors);
}
// Menus
"menus" => match create_menus(value) {
Ok(map) => config.menus = map,

252
crates/nu-protocol/src/config/plugin_gc.rs Normal file
View File

@ -0,0 +1,252 @@
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use crate::{record, ShellError, Span, Value};
use super::helper::{
process_bool_config, report_invalid_key, report_invalid_value, ReconstructVal,
};
/// Configures when plugins should be stopped if inactive
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default)]
pub struct PluginGcConfigs {
/// The config to use for plugins not otherwise specified
pub default: PluginGcConfig,
/// Specific configs for plugins (by name)
pub plugins: HashMap<String, PluginGcConfig>,
}
impl PluginGcConfigs {
/// Get the plugin GC configuration for a specific plugin name. If not specified by name in the
/// config, this is `default`.
pub fn get(&self, plugin_name: &str) -> &PluginGcConfig {
self.plugins.get(plugin_name).unwrap_or(&self.default)
}
pub(super) fn process(
&mut self,
path: &[&str],
value: &mut Value,
errors: &mut Vec<ShellError>,
) {
if let Value::Record { val, .. } = value {
// Handle resets to default if keys are missing
if !val.contains("default") {
self.default = PluginGcConfig::default();
}
if !val.contains("plugins") {
self.plugins = HashMap::new();
}
val.retain_mut(|key, value| {
let span = value.span();
match key {
"default" => {
self.default
.process(&join_path(path, &["default"]), value, errors)
}
"plugins" => process_plugins(
&join_path(path, &["plugins"]),
value,
errors,
&mut self.plugins,
),
_ => {
report_invalid_key(&join_path(path, &[key]), span, errors);
return false;
}
}
true
});
} else {
report_invalid_value("should be a record", value.span(), errors);
*value = self.reconstruct_value(value.span());
}
}
}
impl ReconstructVal for PluginGcConfigs {
fn reconstruct_value(&self, span: Span) -> Value {
Value::record(
record! {
"default" => self.default.reconstruct_value(span),
"plugins" => reconstruct_plugins(&self.plugins, span),
},
span,
)
}
}
fn process_plugins(
path: &[&str],
value: &mut Value,
errors: &mut Vec<ShellError>,
plugins: &mut HashMap<String, PluginGcConfig>,
) {
if let Value::Record { val, .. } = value {
// Remove any plugin configs that aren't in the value
plugins.retain(|key, _| val.contains(key));
val.retain_mut(|key, value| {
if matches!(value, Value::Record { .. }) {
plugins.entry(key.to_owned()).or_default().process(
&join_path(path, &[key]),
value,
errors,
);
true
} else {
report_invalid_value("should be a record", value.span(), errors);
if let Some(conf) = plugins.get(key) {
// Reconstruct the value if it existed before
*value = conf.reconstruct_value(value.span());
true
} else {
// Remove it if it didn't
false
}
}
});
}
}
fn reconstruct_plugins(plugins: &HashMap<String, PluginGcConfig>, span: Span) -> Value {
Value::record(
plugins
.iter()
.map(|(key, val)| (key.to_owned(), val.reconstruct_value(span)))
.collect(),
span,
)
}
/// Configures when a plugin should be stopped if inactive
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PluginGcConfig {
/// True if the plugin should be stopped automatically
pub enabled: bool,
/// When to stop the plugin if not in use for this long (in nanoseconds)
pub stop_after: i64,
}
impl Default for PluginGcConfig {
fn default() -> Self {
PluginGcConfig {
enabled: true,
stop_after: 10_000_000_000, // 10sec
}
}
}
impl PluginGcConfig {
fn process(&mut self, path: &[&str], value: &mut Value, errors: &mut Vec<ShellError>) {
if let Value::Record { val, .. } = value {
// Handle resets to default if keys are missing
if !val.contains("enabled") {
self.enabled = PluginGcConfig::default().enabled;
}
if !val.contains("stop_after") {
self.stop_after = PluginGcConfig::default().stop_after;
}
val.retain_mut(|key, value| {
let span = value.span();
match key {
"enabled" => process_bool_config(value, errors, &mut self.enabled),
"stop_after" => match value {
Value::Duration { val, .. } => {
if *val >= 0 {
self.stop_after = *val;
} else {
report_invalid_value("must not be negative", span, errors);
*val = self.stop_after;
}
}
_ => {
report_invalid_value("should be a duration", span, errors);
*value = Value::duration(self.stop_after, span);
}
},
_ => {
report_invalid_key(&join_path(path, &[key]), span, errors);
return false;
}
}
true
})
} else {
report_invalid_value("should be a record", value.span(), errors);
*value = self.reconstruct_value(value.span());
}
}
}
impl ReconstructVal for PluginGcConfig {
fn reconstruct_value(&self, span: Span) -> Value {
Value::record(
record! {
"enabled" => Value::bool(self.enabled, span),
"stop_after" => Value::duration(self.stop_after, span),
},
span,
)
}
}
fn join_path<'a>(a: &[&'a str], b: &[&'a str]) -> Vec<&'a str> {
a.iter().copied().chain(b.iter().copied()).collect()
}
#[cfg(test)]
mod tests {
use super::*;
fn test_pair() -> (PluginGcConfigs, Value) {
(
PluginGcConfigs {
default: PluginGcConfig {
enabled: true,
stop_after: 30_000_000_000,
},
plugins: [(
"my_plugin".to_owned(),
PluginGcConfig {
enabled: false,
stop_after: 0,
},
)]
.into_iter()
.collect(),
},
Value::test_record(record! {
"default" => Value::test_record(record! {
"enabled" => Value::test_bool(true),
"stop_after" => Value::test_duration(30_000_000_000),
}),
"plugins" => Value::test_record(record! {
"my_plugin" => Value::test_record(record! {
"enabled" => Value::test_bool(false),
"stop_after" => Value::test_duration(0),
}),
}),
}),
)
}
#[test]
fn process() {
let (expected, mut input) = test_pair();
let mut errors = vec![];
let mut result = PluginGcConfigs::default();
result.process(&[], &mut input, &mut errors);
assert!(errors.is_empty(), "errors: {errors:#?}");
assert_eq!(expected, result);
}
#[test]
fn reconstruct() {
let (input, expected) = test_pair();
assert_eq!(expected, input.reconstruct_value(Span::test_data()));
}
}

14
crates/nu-protocol/src/engine/command.rs
View File

@ -1,5 +1,3 @@
use std::path::Path;
use crate::{ast::Call, Alias, BlockId, Example, PipelineData, ShellError, Signature};
use super::{EngineState, Stack, StateWorkingSet};
@ -91,8 +89,14 @@ pub trait Command: Send + Sync + CommandClone {
false
}
// Is a plugin command (returns plugin's path, type of shell if the declaration is a plugin)
fn is_plugin(&self) -> Option<(&Path, Option<&Path>)> {
/// Is a plugin command
fn is_plugin(&self) -> bool {
false
}
/// The identity of the plugin, if this is a plugin command
#[cfg(feature = "plugin")]
fn plugin_identity(&self) -> Option<&crate::PluginIdentity> {
None
}
@ -118,7 +122,7 @@ pub trait Command: Send + Sync + CommandClone {
self.is_parser_keyword(),
self.is_known_external(),
self.is_alias(),
self.is_plugin().is_some(),
self.is_plugin(),
) {
(true, false, false, false, false, false) => CommandType::Builtin,
(true, true, false, false, false, false) => CommandType::Custom,

73
crates/nu-protocol/src/engine/engine_state.rs
View File

@ -23,6 +23,9 @@ use std::sync::{
type PoisonDebuggerError<'a> = PoisonError<MutexGuard<'a, Box<dyn Debugger>>>;
#[cfg(feature = "plugin")]
use crate::RegisteredPlugin;
pub static PWD_ENV: &str = "PWD";
#[derive(Clone, Debug)]
@ -113,6 +116,8 @@ pub struct EngineState {
pub table_decl_id: Option<usize>,
#[cfg(feature = "plugin")]
pub plugin_signatures: Option<PathBuf>,
#[cfg(feature = "plugin")]
plugins: Vec<Arc<dyn RegisteredPlugin>>,
config_path: HashMap<String, PathBuf>,
pub history_enabled: bool,
pub history_session_id: i64,
@ -171,6 +176,8 @@ impl EngineState {
table_decl_id: None,
#[cfg(feature = "plugin")]
plugin_signatures: None,
#[cfg(feature = "plugin")]
plugins: vec![],
config_path: HashMap::new(),
history_enabled: true,
history_session_id: 0,
@ -255,14 +262,27 @@ impl EngineState {
self.scope.active_overlays.append(&mut activated_ids);
#[cfg(feature = "plugin")]
if delta.plugins_changed {
let result = self.update_plugin_file();
if result.is_ok() {
delta.plugins_changed = false;
if !delta.plugins.is_empty() {
// Replace plugins that overlap in identity.
for plugin in std::mem::take(&mut delta.plugins) {
if let Some(existing) = self
.plugins
.iter_mut()
.find(|p| p.identity() == plugin.identity())
{
// Stop the existing plugin, so that the new plugin definitely takes over
existing.stop()?;
*existing = plugin;
} else {
self.plugins.push(plugin);
}
}
}
return result;
#[cfg(feature = "plugin")]
if delta.plugins_changed {
// Update the plugin file with the new signatures.
self.update_plugin_file()?;
}
Ok(())
@ -274,6 +294,8 @@ impl EngineState {
stack: &mut Stack,
cwd: impl AsRef<Path>,
) -> Result<(), ShellError> {
let mut config_updated = false;
for mut scope in stack.env_vars.drain(..) {
for (overlay_name, mut env) in scope.drain() {
if let Some(env_vars) = self.env_vars.get_mut(&overlay_name) {
@ -285,6 +307,7 @@ impl EngineState {
let mut new_record = v.clone();
let (config, error) = new_record.into_config(&self.config);
self.config = config;
config_updated = true;
env_vars.insert(k, new_record);
if let Some(e) = error {
return Err(e);
@ -303,6 +326,12 @@ impl EngineState {
// TODO: better error
std::env::set_current_dir(cwd)?;
if config_updated {
// Make plugin GC config changes take effect immediately.
#[cfg(feature = "plugin")]
self.update_plugin_gc_configs(&self.config.plugin_gc);
}
Ok(())
}
@ -465,6 +494,11 @@ impl EngineState {
None
}
#[cfg(feature = "plugin")]
pub fn plugins(&self) -> &[Arc<dyn RegisteredPlugin>] {
&self.plugins
}
#[cfg(feature = "plugin")]
pub fn update_plugin_file(&self) -> Result<(), ShellError> {
use std::io::Write;
@ -490,8 +524,9 @@ impl EngineState {
self.plugin_decls().try_for_each(|decl| {
// A successful plugin registration already includes the plugin filename
// No need to check the None option
let (path, shell) = decl.is_plugin().expect("plugin should have file name");
let mut file_name = path
let identity = decl.plugin_identity().expect("plugin should have identity");
let mut file_name = identity
.filename()
.to_str()
.expect("path was checked during registration as a str")
.to_string();
@ -518,8 +553,8 @@ impl EngineState {
serde_json::to_string_pretty(&sig_with_examples)
.map(|signature| {
// Extracting the possible path to the shell used to load the plugin
let shell_str = shell
.as_ref()
let shell_str = identity
.shell()
.map(|path| {
format!(
"-s {}",
@ -558,6 +593,14 @@ impl EngineState {
})
}
/// Update plugins with new garbage collection config
#[cfg(feature = "plugin")]
fn update_plugin_gc_configs(&self, plugin_gc: &crate::PluginGcConfigs) {
for plugin in &self.plugins {
plugin.set_gc_config(plugin_gc.get(plugin.identity().name()));
}
}
pub fn num_files(&self) -> usize {
self.files.len()
}
@ -650,7 +693,7 @@ impl EngineState {
let mut unique_plugin_decls = HashMap::new();
// Make sure there are no duplicate decls: Newer one overwrites the older one
for decl in self.decls.iter().filter(|d| d.is_plugin().is_some()) {
for decl in self.decls.iter().filter(|d| d.is_plugin()) {
unique_plugin_decls.insert(decl.name(), decl);
}
@ -733,6 +776,12 @@ impl EngineState {
}
pub fn set_config(&mut self, conf: Config) {
#[cfg(feature = "plugin")]
if conf.plugin_gc != self.config.plugin_gc {
// Make plugin GC config changes take effect immediately.
self.update_plugin_gc_configs(&conf.plugin_gc);
}
self.config = conf;
}
@ -841,7 +890,7 @@ impl EngineState {
(
signature,
decl.examples(),
decl.is_plugin().is_some(),
decl.is_plugin(),
decl.get_block_id().is_some(),
decl.is_parser_keyword(),
)

10
crates/nu-protocol/src/engine/state_delta.rs
View File

@ -2,6 +2,12 @@ use super::{usage::Usage, Command, EngineState, OverlayFrame, ScopeFrame, Virtua
use crate::ast::Block;
use crate::{Module, Variable};
#[cfg(feature = "plugin")]
use std::sync::Arc;
#[cfg(feature = "plugin")]
use crate::RegisteredPlugin;
/// A delta (or change set) between the current global state and a possible future global state. Deltas
/// can be applied to the global state to update it to contain both previous state and the state held
/// within the delta.
@ -17,6 +23,8 @@ pub struct StateDelta {
pub scope: Vec<ScopeFrame>,
#[cfg(feature = "plugin")]
pub(super) plugins_changed: bool, // marks whether plugin file should be updated
#[cfg(feature = "plugin")]
pub(super) plugins: Vec<Arc<dyn RegisteredPlugin>>,
}
impl StateDelta {
@ -40,6 +48,8 @@ impl StateDelta {
usage: Usage::new(),
#[cfg(feature = "plugin")]
plugins_changed: false,
#[cfg(feature = "plugin")]
plugins: vec![],
}
}

28
crates/nu-protocol/src/engine/state_working_set.rs
View File

@ -11,6 +11,12 @@ use core::panic;
use std::collections::{HashMap, HashSet};
use std::path::PathBuf;
#[cfg(feature = "plugin")]
use std::sync::Arc;
#[cfg(feature = "plugin")]
use crate::{PluginIdentity, RegisteredPlugin};
/// A temporary extension to the global state. This handles bridging between the global state and the
/// additional declarations and scope changes that are not yet part of the global scope.
///
@ -155,6 +161,28 @@ impl<'a> StateWorkingSet<'a> {
self.delta.plugins_changed = true;
}
#[cfg(feature = "plugin")]
pub fn find_or_create_plugin(
&mut self,
identity: &PluginIdentity,
make: impl FnOnce() -> Arc<dyn RegisteredPlugin>,
) -> Arc<dyn RegisteredPlugin> {
// Check in delta first, then permanent_state
if let Some(plugin) = self
.delta
.plugins
.iter()
.chain(self.permanent_state.plugins())
.find(|p| p.identity() == identity)
{
plugin.clone()
} else {
let plugin = make();
self.delta.plugins.push(plugin.clone());
plugin
}
}
pub fn merge_predecl(&mut self, name: &[u8]) -> Option<DeclId> {
self.move_predecls_to_overlay();

4
crates/nu-protocol/src/lib.rs
View File

@ -16,7 +16,7 @@ mod parse_error;
mod parse_warning;
mod pipeline_data;
#[cfg(feature = "plugin")]
mod plugin_signature;
mod plugin;
mod shell_error;
mod signature;
pub mod span;
@ -40,7 +40,7 @@ pub use parse_error::{DidYouMean, ParseError};
pub use parse_warning::ParseWarning;
pub use pipeline_data::*;
#[cfg(feature = "plugin")]
pub use plugin_signature::*;
pub use plugin::*;
pub use shell_error::*;
pub use signature::*;
pub use span::*;

105
crates/nu-protocol/src/plugin/identity.rs Normal file
View File

@ -0,0 +1,105 @@
use std::path::{Path, PathBuf};
use crate::{ParseError, Spanned};
/// Error when an invalid plugin filename was encountered. This can be converted to [`ParseError`]
/// if a span is added.
#[derive(Debug, Clone)]
pub struct InvalidPluginFilename;
impl std::fmt::Display for InvalidPluginFilename {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("invalid plugin filename")
}
}
impl From<Spanned<InvalidPluginFilename>> for ParseError {
fn from(error: Spanned<InvalidPluginFilename>) -> ParseError {
ParseError::LabeledError(
"Invalid plugin filename".into(),
"must start with `nu_plugin_`".into(),
error.span,
)
}
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct PluginIdentity {
/// The filename used to start the plugin
filename: PathBuf,
/// The shell used to start the plugin, if required
shell: Option<PathBuf>,
/// The friendly name of the plugin (e.g. `inc` for `C:\nu_plugin_inc.exe`)
name: String,
}
impl PluginIdentity {
/// Create a new plugin identity from a path to plugin executable and shell option.
pub fn new(
filename: impl Into<PathBuf>,
shell: Option<PathBuf>,
) -> Result<PluginIdentity, InvalidPluginFilename> {
let filename = filename.into();
let name = filename
.file_stem()
.map(|stem| stem.to_string_lossy().into_owned())
.and_then(|stem| stem.strip_prefix("nu_plugin_").map(|s| s.to_owned()))
.ok_or(InvalidPluginFilename)?;
Ok(PluginIdentity {
filename,
shell,
name,
})
}
/// The filename of the plugin executable.
pub fn filename(&self) -> &Path {
&self.filename
}
/// The shell command used by the plugin.
pub fn shell(&self) -> Option<&Path> {
self.shell.as_deref()
}
/// The name of the plugin, determined by the part of the filename after `nu_plugin_` excluding
/// the extension.
///
/// - `C:\nu_plugin_inc.exe` becomes `inc`
/// - `/home/nu/.cargo/bin/nu_plugin_inc` becomes `inc`
pub fn name(&self) -> &str {
&self.name
}
/// Create a fake identity for testing.
#[cfg(windows)]
#[doc(hidden)]
pub fn new_fake(name: &str) -> PluginIdentity {
PluginIdentity::new(format!(r"C:\fake\path\nu_plugin_{name}.exe"), None)
.expect("fake plugin identity path is invalid")
}
/// Create a fake identity for testing.
#[cfg(not(windows))]
#[doc(hidden)]
pub fn new_fake(name: &str) -> PluginIdentity {
PluginIdentity::new(format!(r"/fake/path/nu_plugin_{name}"), None)
.expect("fake plugin identity path is invalid")
}
}
#[test]
fn parses_name_from_path() {
assert_eq!("test", PluginIdentity::new_fake("test").name());
assert_eq!("test_2", PluginIdentity::new_fake("test_2").name());
assert_eq!(
"foo",
PluginIdentity::new("nu_plugin_foo.sh", Some("sh".into()))
.expect("should be valid")
.name()
);
PluginIdentity::new("other", None).expect_err("should be invalid");
PluginIdentity::new("", None).expect_err("should be invalid");
}

7
crates/nu-protocol/src/plugin/mod.rs Normal file
View File

@ -0,0 +1,7 @@
mod identity;
mod registered;
mod signature;
pub use identity::*;
pub use registered::*;
pub use signature::*;

27
crates/nu-protocol/src/plugin/registered.rs Normal file
View File

@ -0,0 +1,27 @@
use std::{any::Any, sync::Arc};
use crate::{PluginGcConfig, PluginIdentity, ShellError};
/// Trait for plugins registered in the [`EngineState`](crate::EngineState).
pub trait RegisteredPlugin: Send + Sync {
/// The identity of the plugin - its filename, shell, and friendly name.
fn identity(&self) -> &PluginIdentity;
/// True if the plugin is currently running.
fn is_running(&self) -> bool;
/// Process ID of the plugin executable, if running.
fn pid(&self) -> Option<u32>;
/// Set garbage collection config for the plugin.
fn set_gc_config(&self, gc_config: &PluginGcConfig);
/// Stop the plugin.
fn stop(&self) -> Result<(), ShellError>;
/// Cast the pointer to an [`Any`] so that its concrete type can be retrieved.
///
/// This is necessary in order to allow `nu_plugin` to handle the implementation details of
/// plugins.
fn as_any(self: Arc<Self>) -> Arc<dyn Any + Send + Sync>;
}

0
crates/nu-protocol/src/plugin_signature.rs → crates/nu-protocol/src/plugin/signature.rs
View File

15
crates/nu-utils/src/sample_config/default_config.nu
View File

@ -241,6 +241,21 @@ $env.config = {
plugins: {} # Per-plugin configuration. See https://www.nushell.sh/contributor-book/plugins.html#configuration.
plugin_gc: {
# Configuration for plugin garbage collection
default: {
enabled: true # true to enable stopping of inactive plugins
stop_after: 10sec # how long to wait after a plugin is inactive to stop it
}
plugins: {
# alternate configuration for specific plugins, by name, for example:
#
# gstat: {
# enabled: false
# }
}
}
hooks: {
pre_prompt: [{ null }] # run before the prompt is shown
pre_execution: [{ null }] # run before the repl input is run

16
crates/nu_plugin_example/src/example.rs
View File

@ -99,4 +99,20 @@ impl Example {
span: Some(call.head),
})
}
pub fn disable_gc(
&self,
engine: &EngineInterface,
call: &EvaluatedCall,
) -> Result<Value, LabeledError> {
let disabled = !call.has_flag("reset")?;
engine.set_gc_disabled(disabled)?;
Ok(Value::string(
format!(
"The plugin garbage collector for `example` is now *{}*.",
if disabled { "disabled" } else { "enabled" }
),
call.head,
))
}
}

22
crates/nu_plugin_example/src/nu/mod.rs
View File

@ -48,6 +48,27 @@ impl Plugin for Example {
.category(Category::Experimental)
.search_terms(vec!["example".into(), "configuration".into()])
.input_output_type(Type::Nothing, Type::Table(vec![])),
PluginSignature::build("nu-example-disable-gc")
.usage("Disable the plugin garbage collector for `example`")
.extra_usage(
"\
Plugins are garbage collected by default after a period of inactivity. This
behavior is configurable with `$env.config.plugin_gc.default`, or to change it
specifically for the example plugin, use
`$env.config.plugin_gc.plugins.example`.
This command demonstrates how plugins can control this behavior and disable GC
temporarily if they need to. It is still possible to stop the plugin explicitly
using `plugin stop example`.",
)
.search_terms(vec![
"example".into(),
"gc".into(),
"plugin_gc".into(),
"garbage".into(),
])
.switch("reset", "Turn the garbage collector back on", None)
.category(Category::Experimental),
]
}
@ -64,6 +85,7 @@ impl Plugin for Example {
"nu-example-2" => self.test2(call, input),
"nu-example-3" => self.test3(call, input),
"nu-example-config" => self.config(engine, call),
"nu-example-disable-gc" => self.disable_gc(engine, call),
_ => Err(LabeledError {
label: "Plugin call with wrong name signature".into(),
msg: "the signature used to call the plugin does not match any name in the plugin signature vector".into(),

49
src/tests/test_config.rs
View File

@ -1,4 +1,4 @@
use super::{fail_test, run_test_std};
use super::{fail_test, run_test, run_test_std};
use crate::tests::TestResult;
#[test]
@ -122,3 +122,50 @@ fn mutate_nu_config_plugin() -> TestResult {
fn reject_nu_config_plugin_non_record() -> TestResult {
fail_test(r#"$env.config.plugins = 5"#, "should be a record")
}
#[test]
fn mutate_nu_config_plugin_gc_default_enabled() -> TestResult {
run_test(
r#"
$env.config.plugin_gc.default.enabled = false
$env.config.plugin_gc.default.enabled
"#,
"false",
)
}
#[test]
fn mutate_nu_config_plugin_gc_default_stop_after() -> TestResult {
run_test(
r#"
$env.config.plugin_gc.default.stop_after = 20sec
$env.config.plugin_gc.default.stop_after
"#,
"20sec",
)
}
#[test]
fn mutate_nu_config_plugin_gc_default_stop_after_negative() -> TestResult {
fail_test(
r#"
$env.config.plugin_gc.default.stop_after = -1sec
$env.config.plugin_gc.default.stop_after
"#,
"must not be negative",
)
}
#[test]
fn mutate_nu_config_plugin_gc_plugins() -> TestResult {
run_test(
r#"
$env.config.plugin_gc.plugins.inc = {
enabled: true
stop_after: 0sec
}
$env.config.plugin_gc.plugins.inc.stop_after
"#,
"0sec",
)
}

2
tests/main.rs
View File

@ -8,6 +8,8 @@ mod overlays;
mod parsing;
mod path;
#[cfg(feature = "plugin")]
mod plugin_persistence;
#[cfg(feature = "plugin")]
mod plugins;
mod scope;
mod shell;

325
tests/plugin_persistence/mod.rs Normal file
View File

@ -0,0 +1,325 @@
//! The tests in this file check the soundness of plugin persistence. When a plugin is needed by Nu,
//! it is spawned only if it was not already running. Plugins that are spawned are kept running and
//! are referenced in the engine state. Plugins can be stopped by the user if desired, but not
//! removed.
use nu_test_support::{nu, nu_with_plugins};
#[test]
fn plugin_list_shows_installed_plugins() {
let out = nu_with_plugins!(
cwd: ".",
plugins: [("nu_plugin_inc"), ("nu_plugin_custom_values")],
r#"(plugin list).name | str join ','"#
);
assert_eq!("inc,custom_values", out.out);
assert!(out.status.success());
}
#[test]
fn plugin_keeps_running_after_calling_it() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
plugin stop inc
(plugin list).0.is_running | print
print ";"
"2.0.0" | inc -m | ignore
(plugin list).0.is_running | print
"#
);
assert_eq!(
"false;true", out.out,
"plugin list didn't show is_running = true"
);
assert!(out.status.success());
}
#[test]
fn plugin_process_exits_after_stop() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
"2.0.0" | inc -m | ignore
let pid = (plugin list).0.pid
ps | where pid == $pid | length | print
print ";"
plugin stop inc
sleep 10ms
ps | where pid == $pid | length | print
"#
);
assert_eq!("1;0", out.out, "plugin process did not stop running");
assert!(out.status.success());
}
#[test]
fn plugin_process_exits_when_nushell_exits() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
"2.0.0" | inc -m | ignore
(plugin list).0.pid | print
"#
);
assert!(!out.out.is_empty());
assert!(out.status.success());
let pid = out.out.parse::<u32>().expect("failed to parse pid");
// use nu to check if process exists
assert_eq!(
"0",
nu!(format!("ps | where pid == {pid} | length")).out,
"plugin process {pid} is still running"
);
}
#[test]
fn plugin_commands_run_without_error() {
let out = nu_with_plugins!(
cwd: ".",
plugins: [
("nu_plugin_inc"),
("nu_plugin_stream_example"),
("nu_plugin_custom_values"),
],
r#"
"2.0.0" | inc -m | ignore
stream_example seq 1 10 | ignore
custom-value generate | ignore
"#
);
assert!(out.err.is_empty());
assert!(out.status.success());
}
#[test]
fn plugin_commands_run_multiple_times_without_error() {
let out = nu_with_plugins!(
cwd: ".",
plugins: [
("nu_plugin_inc"),
("nu_plugin_stream_example"),
("nu_plugin_custom_values"),
],
r#"
["2.0.0" "2.1.0" "2.2.0"] | each { inc -m } | print
stream_example seq 1 10 | ignore
custom-value generate | ignore
stream_example seq 1 20 | ignore
custom-value generate2 | ignore
"#
);
assert!(out.err.is_empty());
assert!(out.status.success());
}
#[test]
fn multiple_plugin_commands_run_with_the_same_plugin_pid() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_custom_values"),
r#"
custom-value generate | ignore
(plugin list).0.pid | print
print ";"
custom-value generate2 | ignore
(plugin list).0.pid | print
"#
);
assert!(out.status.success());
let pids: Vec<&str> = out.out.split(';').collect();
assert_eq!(2, pids.len());
assert_eq!(pids[0], pids[1]);
}
#[test]
fn plugin_pid_changes_after_stop_then_run_again() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_custom_values"),
r#"
custom-value generate | ignore
(plugin list).0.pid | print
print ";"
plugin stop custom_values
custom-value generate2 | ignore
(plugin list).0.pid | print
"#
);
assert!(out.status.success());
let pids: Vec<&str> = out.out.split(';').collect();
assert_eq!(2, pids.len());
assert_ne!(pids[0], pids[1]);
}
#[test]
fn custom_values_can_still_be_passed_to_plugin_after_stop() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_custom_values"),
r#"
let cv = custom-value generate
plugin stop custom_values
$cv | custom-value update
"#
);
assert!(!out.out.is_empty());
assert!(out.err.is_empty());
assert!(out.status.success());
}
#[test]
fn custom_values_can_still_be_collapsed_after_stop() {
// print causes a collapse (ToBaseValue) call.
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_custom_values"),
r#"
let cv = custom-value generate
plugin stop custom_values
$cv | print
"#
);
assert!(!out.out.is_empty());
assert!(out.err.is_empty());
assert!(out.status.success());
}
#[test]
fn plugin_gc_can_be_configured_to_stop_plugins_immediately() {
// I know the test is to stop "immediately", but if we actually check immediately it could
// lead to a race condition. So there's a 1ms sleep just to be fair.
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
$env.config.plugin_gc = { default: { stop_after: 0sec } }
"2.3.0" | inc -M
sleep 1ms
(plugin list | where name == inc).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("false", out.out, "with config as default");
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
$env.config.plugin_gc = {
plugins: {
inc: { stop_after: 0sec }
}
}
"2.3.0" | inc -M
sleep 1ms
(plugin list | where name == inc).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("false", out.out, "with inc-specific config");
}
#[test]
fn plugin_gc_can_be_configured_to_stop_plugins_after_delay() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
$env.config.plugin_gc = { default: { stop_after: 50ms } }
"2.3.0" | inc -M
sleep 100ms
(plugin list | where name == inc).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("false", out.out, "with config as default");
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
$env.config.plugin_gc = {
plugins: {
inc: { stop_after: 50ms }
}
}
"2.3.0" | inc -M
sleep 100ms
(plugin list | where name == inc).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("false", out.out, "with inc-specific config");
}
#[test]
fn plugin_gc_can_be_configured_as_disabled() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
$env.config.plugin_gc = { default: { enabled: false, stop_after: 0sec } }
"2.3.0" | inc -M
(plugin list | where name == inc).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("true", out.out, "with config as default");
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_inc"),
r#"
$env.config.plugin_gc = {
default: { enabled: true, stop_after: 0sec }
plugins: {
inc: { enabled: false, stop_after: 0sec }
}
}
"2.3.0" | inc -M
(plugin list | where name == inc).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("true", out.out, "with inc-specific config");
}
#[test]
fn plugin_gc_can_be_disabled_by_plugin() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_example"),
r#"
nu-example-disable-gc
$env.config.plugin_gc = { default: { stop_after: 0sec } }
nu-example-1 1 foo | ignore # ensure we've run the plugin with the new config
sleep 10ms
(plugin list | where name == example).0.is_running
"#
);
assert!(out.status.success());
assert_eq!("true", out.out);
}
#[test]
fn plugin_gc_does_not_stop_plugin_while_stream_output_is_active() {
let out = nu_with_plugins!(
cwd: ".",
plugin: ("nu_plugin_stream_example"),
r#"
$env.config.plugin_gc = { default: { stop_after: 10ms } }
# This would exceed the configured time
stream_example seq 1 500 | each { |n| sleep 1ms; $n } | length | print
"#
);
assert!(out.status.success());
assert_eq!("500", out.out);
}