Keep plugins persistently running in the background (#12064)

# Description
This PR uses the new plugin protocol to intelligently keep plugin
processes running in the background for further plugin calls.

Running plugins can be seen by running the new `plugin list` command,
and stopped by running the new `plugin stop` command.

This is an enhancement for the performance of plugins, as starting new
plugin processes has overhead, especially for plugins in languages that
take a significant amount of time on startup. It also enables plugins
that have persistent state between commands, making the migration of
features like dataframes and `stor` to plugins possible.

Plugins are automatically stopped by the new plugin garbage collector,
configurable with `$env.config.plugin_gc`:

```nushell
  $env.config.plugin_gc = {
      # Configuration for plugin garbage collection
      default: {
          enabled: true # true to enable stopping of inactive plugins
          stop_after: 10sec # how long to wait after a plugin is inactive to stop it
      }
      plugins: {
          # alternate configuration for specific plugins, by name, for example:
          #
          # gstat: {
          #     enabled: false
          # }
      }
  }
```

If garbage collection is enabled, plugins will be stopped after
`stop_after` passes after they were last active. Plugins are counted as
inactive if they have no running plugin calls. Reading the stream from
the response of a plugin call is still considered to be activity, but if
a plugin holds on to a stream but the call ends without an active
streaming response, it is not counted as active even if it is reading
it. Plugins can explicitly disable the GC as appropriate with
`engine.set_gc_disabled(true)`.

The `version` command now lists plugin names rather than plugin
commands. The list of plugin commands is accessible via `plugin list`.

Recommend doing this together with #12029, because it will likely force
plugin developers to do the right thing with mutability and lead to less
unexpected behavior when running plugins nested / in parallel.

# User-Facing Changes
- new command: `plugin list`
- new command: `plugin stop`
- changed command: `version` (now lists plugin names, rather than
commands)
- new config: `$env.config.plugin_gc`
- Plugins will keep running and be reused, at least for the configured
GC period
- Plugins that used mutable state in weird ways like `inc` did might
misbehave until fixed
- Plugins can disable GC if they need to
- Had to change plugin signature to accept `&EngineInterface` so that
the GC disable feature works. #12029 does this anyway, and I'm expecting
(resolvable) conflicts with that

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`

Because there is some specific OS behavior required for plugins to not
respond to Ctrl-C directly, I've developed against and tested on both
Linux and Windows to ensure that works properly.

# After Submitting
I think this probably needs to be in the book somewhere

crates/nu-protocol/src/plugin/identity.rs

@@ -0,0 +1,105 @@
+use std::path::{Path, PathBuf};
+
+use crate::{ParseError, Spanned};
+
+/// Error when an invalid plugin filename was encountered. This can be converted to [`ParseError`]
+/// if a span is added.
+#[derive(Debug, Clone)]
+pub struct InvalidPluginFilename;
+
+impl std::fmt::Display for InvalidPluginFilename {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str("invalid plugin filename")
+    }
+}
+
+impl From<Spanned<InvalidPluginFilename>> for ParseError {
+    fn from(error: Spanned<InvalidPluginFilename>) -> ParseError {
+        ParseError::LabeledError(
+            "Invalid plugin filename".into(),
+            "must start with `nu_plugin_`".into(),
+            error.span,
+        )
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
+pub struct PluginIdentity {
+    /// The filename used to start the plugin
+    filename: PathBuf,
+    /// The shell used to start the plugin, if required
+    shell: Option<PathBuf>,
+    /// The friendly name of the plugin (e.g. `inc` for `C:\nu_plugin_inc.exe`)
+    name: String,
+}
+
+impl PluginIdentity {
+    /// Create a new plugin identity from a path to plugin executable and shell option.
+    pub fn new(
+        filename: impl Into<PathBuf>,
+        shell: Option<PathBuf>,
+    ) -> Result<PluginIdentity, InvalidPluginFilename> {
+        let filename = filename.into();
+
+        let name = filename
+            .file_stem()
+            .map(|stem| stem.to_string_lossy().into_owned())
+            .and_then(|stem| stem.strip_prefix("nu_plugin_").map(|s| s.to_owned()))
+            .ok_or(InvalidPluginFilename)?;
+
+        Ok(PluginIdentity {
+            filename,
+            shell,
+            name,
+        })
+    }
+
+    /// The filename of the plugin executable.
+    pub fn filename(&self) -> &Path {
+        &self.filename
+    }
+
+    /// The shell command used by the plugin.
+    pub fn shell(&self) -> Option<&Path> {
+        self.shell.as_deref()
+    }
+
+    /// The name of the plugin, determined by the part of the filename after `nu_plugin_` excluding
+    /// the extension.
+    ///
+    /// - `C:\nu_plugin_inc.exe` becomes `inc`
+    /// - `/home/nu/.cargo/bin/nu_plugin_inc` becomes `inc`
+    pub fn name(&self) -> &str {
+        &self.name
+    }
+
+    /// Create a fake identity for testing.
+    #[cfg(windows)]
+    #[doc(hidden)]
+    pub fn new_fake(name: &str) -> PluginIdentity {
+        PluginIdentity::new(format!(r"C:\fake\path\nu_plugin_{name}.exe"), None)
+            .expect("fake plugin identity path is invalid")
+    }
+
+    /// Create a fake identity for testing.
+    #[cfg(not(windows))]
+    #[doc(hidden)]
+    pub fn new_fake(name: &str) -> PluginIdentity {
+        PluginIdentity::new(format!(r"/fake/path/nu_plugin_{name}"), None)
+            .expect("fake plugin identity path is invalid")
+    }
+}
+
+#[test]
+fn parses_name_from_path() {
+    assert_eq!("test", PluginIdentity::new_fake("test").name());
+    assert_eq!("test_2", PluginIdentity::new_fake("test_2").name());
+    assert_eq!(
+        "foo",
+        PluginIdentity::new("nu_plugin_foo.sh", Some("sh".into()))
+            .expect("should be valid")
+            .name()
+    );
+    PluginIdentity::new("other", None).expect_err("should be invalid");
+    PluginIdentity::new("", None).expect_err("should be invalid");
+}

crates/nu-protocol/src/plugin/mod.rs

@@ -0,0 +1,7 @@
+mod identity;
+mod registered;
+mod signature;
+
+pub use identity::*;
+pub use registered::*;
+pub use signature::*;

crates/nu-protocol/src/plugin/registered.rs

@@ -0,0 +1,27 @@
+use std::{any::Any, sync::Arc};
+
+use crate::{PluginGcConfig, PluginIdentity, ShellError};
+
+/// Trait for plugins registered in the [`EngineState`](crate::EngineState).
+pub trait RegisteredPlugin: Send + Sync {
+    /// The identity of the plugin - its filename, shell, and friendly name.
+    fn identity(&self) -> &PluginIdentity;
+
+    /// True if the plugin is currently running.
+    fn is_running(&self) -> bool;
+
+    /// Process ID of the plugin executable, if running.
+    fn pid(&self) -> Option<u32>;
+
+    /// Set garbage collection config for the plugin.
+    fn set_gc_config(&self, gc_config: &PluginGcConfig);
+
+    /// Stop the plugin.
+    fn stop(&self) -> Result<(), ShellError>;
+
+    /// Cast the pointer to an [`Any`] so that its concrete type can be retrieved.
+    ///
+    /// This is necessary in order to allow `nu_plugin` to handle the implementation details of
+    /// plugins.
+    fn as_any(self: Arc<Self>) -> Arc<dyn Any + Send + Sync>;
+}

crates/nu-protocol/src/plugin/signature.rs

@@ -0,0 +1,226 @@
+use crate::{PluginExample, Signature};
+use serde::Deserialize;
+use serde::Serialize;
+
+use crate::engine::Command;
+use crate::{BlockId, Category, Flag, PositionalArg, SyntaxShape, Type};
+
+/// A simple wrapper for Signature that includes examples.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct PluginSignature {
+    pub sig: Signature,
+    pub examples: Vec<PluginExample>,
+}
+
+impl PluginSignature {
+    pub fn new(sig: Signature, examples: Vec<PluginExample>) -> Self {
+        Self { sig, examples }
+    }
+
+    /// Add a default help option to a signature
+    pub fn add_help(mut self) -> PluginSignature {
+        self.sig = self.sig.add_help();
+        self
+    }
+
+    /// Build an internal signature with default help option
+    pub fn build(name: impl Into<String>) -> PluginSignature {
+        let sig = Signature::new(name.into()).add_help();
+        Self::new(sig, vec![])
+    }
+
+    /// Add a description to the signature
+    pub fn usage(mut self, msg: impl Into<String>) -> PluginSignature {
+        self.sig = self.sig.usage(msg);
+        self
+    }
+
+    /// Add an extra description to the signature
+    pub fn extra_usage(mut self, msg: impl Into<String>) -> PluginSignature {
+        self.sig = self.sig.extra_usage(msg);
+        self
+    }
+
+    /// Add search terms to the signature
+    pub fn search_terms(mut self, terms: Vec<String>) -> PluginSignature {
+        self.sig = self.sig.search_terms(terms);
+        self
+    }
+
+    /// Update signature's fields from a Command trait implementation
+    pub fn update_from_command(mut self, command: &dyn Command) -> PluginSignature {
+        self.sig = self.sig.update_from_command(command);
+        self
+    }
+
+    /// Allow unknown signature parameters
+    pub fn allows_unknown_args(mut self) -> PluginSignature {
+        self.sig = self.sig.allows_unknown_args();
+        self
+    }
+
+    /// Add a required positional argument to the signature
+    pub fn required(
+        mut self,
+        name: impl Into<String>,
+        shape: impl Into<SyntaxShape>,
+        desc: impl Into<String>,
+    ) -> PluginSignature {
+        self.sig = self.sig.required(name, shape, desc);
+        self
+    }
+
+    /// Add an optional positional argument to the signature
+    pub fn optional(
+        mut self,
+        name: impl Into<String>,
+        shape: impl Into<SyntaxShape>,
+        desc: impl Into<String>,
+    ) -> PluginSignature {
+        self.sig = self.sig.optional(name, shape, desc);
+        self
+    }
+
+    pub fn rest(
+        mut self,
+        name: &str,
+        shape: impl Into<SyntaxShape>,
+        desc: impl Into<String>,
+    ) -> PluginSignature {
+        self.sig = self.sig.rest(name, shape, desc);
+        self
+    }
+
+    /// Is this command capable of operating on its input via cell paths?
+    pub fn operates_on_cell_paths(&self) -> bool {
+        self.sig.operates_on_cell_paths()
+    }
+
+    /// Add an optional named flag argument to the signature
+    pub fn named(
+        mut self,
+        name: impl Into<String>,
+        shape: impl Into<SyntaxShape>,
+        desc: impl Into<String>,
+        short: Option<char>,
+    ) -> PluginSignature {
+        self.sig = self.sig.named(name, shape, desc, short);
+        self
+    }
+
+    /// Add a required named flag argument to the signature
+    pub fn required_named(
+        mut self,
+        name: impl Into<String>,
+        shape: impl Into<SyntaxShape>,
+        desc: impl Into<String>,
+        short: Option<char>,
+    ) -> PluginSignature {
+        self.sig = self.sig.required_named(name, shape, desc, short);
+        self
+    }
+
+    /// Add a switch to the signature
+    pub fn switch(
+        mut self,
+        name: impl Into<String>,
+        desc: impl Into<String>,
+        short: Option<char>,
+    ) -> PluginSignature {
+        self.sig = self.sig.switch(name, desc, short);
+        self
+    }
+
+    /// Changes the input type of the command signature
+    pub fn input_output_type(mut self, input_type: Type, output_type: Type) -> PluginSignature {
+        self.sig.input_output_types.push((input_type, output_type));
+        self
+    }
+
+    /// Set the input-output type signature variants of the command
+    pub fn input_output_types(mut self, input_output_types: Vec<(Type, Type)>) -> PluginSignature {
+        self.sig = self.sig.input_output_types(input_output_types);
+        self
+    }
+
+    /// Changes the signature category
+    pub fn category(mut self, category: Category) -> PluginSignature {
+        self.sig = self.sig.category(category);
+        self
+    }
+
+    /// Sets that signature will create a scope as it parses
+    pub fn creates_scope(mut self) -> PluginSignature {
+        self.sig = self.sig.creates_scope();
+        self
+    }
+
+    // Is it allowed for the type signature to feature a variant that has no corresponding example?
+    pub fn allow_variants_without_examples(mut self, allow: bool) -> PluginSignature {
+        self.sig = self.sig.allow_variants_without_examples(allow);
+        self
+    }
+
+    pub fn call_signature(&self) -> String {
+        self.sig.call_signature()
+    }
+
+    /// Get list of the short-hand flags
+    pub fn get_shorts(&self) -> Vec<char> {
+        self.sig.get_shorts()
+    }
+
+    /// Get list of the long-hand flags
+    pub fn get_names(&self) -> Vec<&str> {
+        self.sig.get_names()
+    }
+
+    pub fn get_positional(&self, position: usize) -> Option<PositionalArg> {
+        self.sig.get_positional(position)
+    }
+
+    pub fn num_positionals(&self) -> usize {
+        self.sig.num_positionals()
+    }
+
+    pub fn num_positionals_after(&self, idx: usize) -> usize {
+        self.sig.num_positionals_after(idx)
+    }
+
+    /// Find the matching long flag
+    pub fn get_long_flag(&self, name: &str) -> Option<Flag> {
+        self.sig.get_long_flag(name)
+    }
+
+    /// Find the matching long flag
+    pub fn get_short_flag(&self, short: char) -> Option<Flag> {
+        self.sig.get_short_flag(short)
+    }
+
+    /// Set the filter flag for the signature
+    pub fn filter(mut self) -> PluginSignature {
+        self.sig = self.sig.filter();
+        self
+    }
+
+    /// Create a placeholder implementation of Command as a way to predeclare a definition's
+    /// signature so other definitions can see it. This placeholder is later replaced with the
+    /// full definition in a second pass of the parser.
+    pub fn predeclare(self) -> Box<dyn Command> {
+        self.sig.predeclare()
+    }
+
+    /// Combines a signature and a block into a runnable block
+    pub fn into_block_command(self, block_id: BlockId) -> Box<dyn Command> {
+        self.sig.into_block_command(block_id)
+    }
+
+    pub fn formatted_flags(self) -> String {
+        self.sig.formatted_flags()
+    }
+
+    pub fn plugin_examples(mut self, examples: Vec<PluginExample>) -> PluginSignature {
+        self.examples = examples;
+        self
+    }
+}