Add support for engine calls from plugins (#12029)

# Description

This allows plugins to make calls back to the engine to get config,
evaluate closures, and do other things that must be done within the
engine process.

Engine calls can both produce and consume streams as necessary. Closures
passed to plugins can both accept stream input and produce stream output
sent back to the plugin.

Engine calls referring to a plugin call's context can be processed as
long either the response hasn't been received, or the response created
streams that haven't ended yet.

This is a breaking API change for plugins. There are some pretty major
changes to the interface that plugins must implement, including:

1. Plugins now run with `&self` and must be `Sync`. Executing multiple
plugin calls in parallel is supported, and there's a chance that a
closure passed to a plugin could invoke the same plugin. Supporting
state across plugin invocations is left up to the plugin author to do in
whichever way they feel best, but the plugin object itself is still
shared. Even though the engine doesn't run multiple plugin calls through
the same process yet, I still considered it important to break the API
in this way at this stage. We might want to consider an optional
threadpool feature for performance.

2. Plugins take a reference to `EngineInterface`, which can be cloned.
This interface allows plugins to make calls back to the engine,
including for getting config and running closures.

3. Plugins no longer take the `config` parameter. This can be accessed
from the interface via the `.get_plugin_config()` engine call.


# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->
Not only does this have plugin protocol changes, it will require plugins
to make some code changes before they will work again. But on the plus
side, the engine call feature is extensible, and we can add more things
to it as needed.

Plugin maintainers will have to change the trait signature at the very
least. If they were using `config`, they will have to call
`engine.get_plugin_config()` instead.

If they were using the mutable reference to the plugin, they will have
to come up with some strategy to work around it (for example, for `Inc`
I just cloned it). This shouldn't be such a big deal at the moment as
it's not like plugins have ever run as daemons with persistent state in
the past, and they don't in this PR either. But I thought it was
important to make the change before we support plugins as daemons, as an
exclusive mutable reference is not compatible with parallel plugin
calls.

I suggest this gets merged sometime *after* the current pending release,
so that we have some time to adjust to the previous plugin protocol
changes that don't require code changes before making ones that do.

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`


# After Submitting
I will document the additional protocol features (`EngineCall`,
`EngineCallResponse`), and constraints on plugin call processing if
engine calls are used - basically, to be aware that an engine call could
result in a nested plugin call, so the plugin should be able to handle
that.

crates/nu_plugin_stream_example/README.md

@@ -46,3 +46,18 @@ strings on input will be concatenated into an external stream (raw input) on std
 
     Hello
     worldhowareyou
+
+## `stream_example for-each`
+
+This command demonstrates executing closures on values in streams. Each value received on the input
+will be printed to the plugin's stderr. This works even with external commands.
+
+> ```nushell
+> ls | get name | stream_example for-each { |f| ^file $f }
+> ```
+
+    CODE_OF_CONDUCT.md: ASCII text
+
+    CONTRIBUTING.md: ASCII text, with very long lines (303)
+
+    ...

crates/nu_plugin_stream_example/src/example.rs

@@ -1,5 +1,5 @@
-use nu_plugin::{EvaluatedCall, LabeledError};
-use nu_protocol::{ListStream, PipelineData, RawStream, Value};
+use nu_plugin::{EngineInterface, EvaluatedCall, LabeledError};
+use nu_protocol::{IntoInterruptiblePipelineData, ListStream, PipelineData, RawStream, Value};
 
 pub struct Example;
 
@@ -64,4 +64,52 @@ impl Example {
             trim_end_newline: false,
         })
     }
+
+    pub fn for_each(
+        &self,
+        engine: &EngineInterface,
+        call: &EvaluatedCall,
+        input: PipelineData,
+    ) -> Result<PipelineData, LabeledError> {
+        let closure = call.req(0)?;
+        let config = engine.get_config()?;
+        for value in input {
+            let result = engine.eval_closure(&closure, vec![value.clone()], Some(value))?;
+            eprintln!("{}", result.to_expanded_string(", ", &config));
+        }
+        Ok(PipelineData::Empty)
+    }
+
+    pub fn generate(
+        &self,
+        engine: &EngineInterface,
+        call: &EvaluatedCall,
+    ) -> Result<PipelineData, LabeledError> {
+        let engine = engine.clone();
+        let call = call.clone();
+        let initial: Value = call.req(0)?;
+        let closure = call.req(1)?;
+
+        let mut next = (!initial.is_nothing()).then_some(initial);
+
+        Ok(std::iter::from_fn(move || {
+            next.take()
+                .and_then(|value| {
+                    engine
+                        .eval_closure(&closure, vec![value.clone()], Some(value))
+                        .and_then(|record| {
+                            if record.is_nothing() {
+                                Ok(None)
+                            } else {
+                                let record = record.as_record()?;
+                                next = record.get("next").cloned();
+                                Ok(record.get("out").cloned())
+                            }
+                        })
+                        .transpose()
+                })
+                .map(|result| result.unwrap_or_else(|err| Value::error(err, call.head)))
+        })
+        .into_pipeline_data(None))
+    }
 }

crates/nu_plugin_stream_example/src/main.rs

@@ -6,7 +6,7 @@ fn main() {
     // used to encode and decode the messages. The available options are
     // MsgPackSerializer and JsonSerializer. Both are defined in the serializer
     // folder in nu-plugin.
-    serve_plugin(&mut Example {}, MsgPackSerializer {})
+    serve_plugin(&Example {}, MsgPackSerializer {})
 
     // Note
     // When creating plugins in other languages one needs to consider how a plugin

crates/nu_plugin_stream_example/src/nu/mod.rs

@@ -1,5 +1,5 @@
 use crate::Example;
-use nu_plugin::{EvaluatedCall, LabeledError, StreamingPlugin};
+use nu_plugin::{EngineInterface, EvaluatedCall, LabeledError, StreamingPlugin};
 use nu_protocol::{
     Category, PipelineData, PluginExample, PluginSignature, Span, SyntaxShape, Type, Value,
 };
@@ -57,13 +57,50 @@ impl StreamingPlugin for Example {
                     result: Some(Value::string("ab", span)),
                 }])
                 .category(Category::Experimental),
+            PluginSignature::build("stream_example for-each")
+                .usage("Example execution of a closure with a stream")
+                .extra_usage("Prints each value the closure returns to stderr")
+                .input_output_type(Type::ListStream, Type::Nothing)
+                .required(
+                    "closure",
+                    SyntaxShape::Closure(Some(vec![SyntaxShape::Any])),
+                    "The closure to run for each input value",
+                )
+                .plugin_examples(vec![PluginExample {
+                    example: "ls | get name | stream_example for-each { |f| ^file $f }".into(),
+                    description: "example with an external command".into(),
+                    result: None,
+                }])
+                .category(Category::Experimental),
+            PluginSignature::build("stream_example generate")
+                .usage("Example execution of a closure to produce a stream")
+                .extra_usage("See the builtin `generate` command")
+                .input_output_type(Type::Nothing, Type::ListStream)
+                .required(
+                    "initial",
+                    SyntaxShape::Any,
+                    "The initial value to pass to the closure"
+                )
+                .required(
+                    "closure",
+                    SyntaxShape::Closure(Some(vec![SyntaxShape::Any])),
+                    "The closure to run to generate values",
+                )
+                .plugin_examples(vec![PluginExample {
+                    example: "stream_example generate 0 { |i| if $i <= 10 { {out: $i, next: ($i + 2)} } }".into(),
+                    description: "Generate a sequence of numbers".into(),
+                    result: Some(Value::test_list(
+                        [0, 2, 4, 6, 8, 10].into_iter().map(Value::test_int).collect(),
+                    )),
+                }])
+                .category(Category::Experimental),
         ]
     }
 
     fn run(
-        &mut self,
+        &self,
         name: &str,
-        _config: &Option<Value>,
+        engine: &EngineInterface,
         call: &EvaluatedCall,
         input: PipelineData,
     ) -> Result<PipelineData, LabeledError> {
@@ -76,6 +113,8 @@ impl StreamingPlugin for Example {
             "stream_example seq" => self.seq(call, input),
             "stream_example sum" => self.sum(call, input),
             "stream_example collect-external" => self.collect_external(call, input),
+            "stream_example for-each" => self.for_each(engine, call, input),
+            "stream_example generate" => self.generate(engine, call),
             _ => Err(LabeledError {
                 label: "Plugin call with wrong name signature".into(),
                 msg: "the signature used to call the plugin does not match any name in the plugin signature vector".into(),