nushell/crates/nu-engine/src/nu_variable.rs

220 lines
7.7 KiB
Rust
Raw Normal View History

LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
use crate::scope::create_scope;
use core::fmt;
use nu_protocol::{
engine::{EngineState, Stack},
LazyRecord, ShellError, Span, Value,
};
use serde::{Deserialize, Serialize};
use sysinfo::SystemExt;
// NuVariable: a LazyRecord for the special $nu variable
// $nu used to be a plain old Record, but LazyRecord lets us load different fields/columns lazily. This is important for performance;
// collecting all the information in $nu is expensive and unnecessary if you just want a subset of the data
// Note: NuVariable is not meaningfully serializable, this #[derive] is a lie to satisfy the type checker.
// Make sure to collect() the record before serializing it
#[derive(Serialize, Deserialize)]
pub struct NuVariable {
#[serde(skip)]
pub engine_state: EngineState,
#[serde(skip)]
pub stack: Stack,
pub span: Span,
}
impl LazyRecord for NuVariable {
fn column_names(&self) -> Vec<&'static str> {
let mut cols = vec!["config-path", "env-path", "history-path", "loginshell-path"];
#[cfg(feature = "plugin")]
if self.engine_state.plugin_signatures.is_some() {
cols.push("plugin-path");
}
cols.push("scope");
cols.push("home-path");
cols.push("temp-path");
cols.push("pid");
cols.push("os-info");
FEATURE: add the startup time to `$nu` (#8353) # Description in https://github.com/nushell/nushell/issues/8311 and the discord server, the idea of moving the default banner from the `rust` source to the `nushell` standar library has emerged :yum: however, in order to do this, one need to have access to all the variables used in the default banner => all of them are accessible because known constants, except for the startup time of the shell, which is not anywhere in the shell... #### this PR adds exactly this, i.e. the new `startup_time` to the `$nu` variable, which is computed to have the exact same value as the value shown in the banner. ## the changes in order to achieve this, i had to - add `startup_time` as an `i64` to the `EngineState` => this is, to the best of my knowledge, the easiest way to pass such an information around down to where the banner startup time is computed and where the `$nu` variable is evaluated - add `startup-time` to the `$nu` variable and use the `EngineState` getter for `startup_time` to show it as a `Value::Duration` - pass `engine_state` as a `&mut`able argument from `main.rs` down to `repl.rs` to allow the setter to change the value of `startup_time` => without this, the value would not change and would show `-1ns` as the default value... - the value of the startup time is computed in `evaluate_repl` in `repl.rs`, only once at the beginning, and the same value is used in the default banner :ok_hand: # User-Facing Changes one can now access to the same time as shown in the default banner with ```bash $nu.startup-time ``` # Tests + Formatting - :green_circle: `cargo fmt --all` - :green_circle: `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` - :green_circle: `cargo test --workspace` # After Submitting ``` $nothing ```
2023-03-09 21:18:58 +01:00
cols.push("startup-time");
LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
cols.push("is-interactive");
cols.push("is-login");
LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
cols
}
fn get_column_value(&self, column: &str) -> Result<Value, ShellError> {
let err = |message: &str| -> Result<Value, ShellError> {
Err(ShellError::LazyRecordAccessFailed {
message: message.into(),
column_name: column.to_string(),
span: self.span,
})
};
match column {
"config-path" => {
if let Some(path) = self.engine_state.get_config_path("config-path") {
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else if let Some(mut path) = nu_path::config_dir() {
path.push("nushell");
path.push("config.nu");
LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else {
err("Could not get config directory")
}
}
"env-path" => {
if let Some(path) = self.engine_state.get_config_path("env-path") {
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else if let Some(mut path) = nu_path::config_dir() {
path.push("nushell");
path.push("env.nu");
LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else {
err("Could not get config directory")
}
}
"history-path" => {
if let Some(mut path) = nu_path::config_dir() {
path.push("nushell");
match self.engine_state.config.history_file_format {
nu_protocol::HistoryFileFormat::Sqlite => {
path.push("history.sqlite3");
}
nu_protocol::HistoryFileFormat::PlainText => {
path.push("history.txt");
}
}
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else {
err("Could not get config directory")
}
}
"loginshell-path" => {
if let Some(mut path) = nu_path::config_dir() {
path.push("nushell");
path.push("login.nu");
LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else {
err("Could not get config directory")
}
}
"plugin-path" => {
#[cfg(feature = "plugin")]
{
if let Some(path) = &self.engine_state.plugin_signatures {
Ok(Value::String {
val: path.to_string_lossy().to_string(),
span: self.span,
})
} else {
err("Could not get plugin signature location")
}
}
#[cfg(not(feature = "plugin"))]
{
err("Plugin feature not enabled")
}
}
"scope" => Ok(create_scope(&self.engine_state, &self.stack, self.span())?),
"home-path" => {
if let Some(home_path) = nu_path::home_dir() {
Ok(Value::String {
val: home_path.to_string_lossy().into(),
span: self.span(),
})
} else {
err("Could not get home path")
}
}
"temp-path" => {
let temp_path = std::env::temp_dir();
Ok(Value::String {
val: temp_path.to_string_lossy().into(),
span: self.span(),
})
}
"pid" => Ok(Value::int(std::process::id().into(), self.span())),
"os-info" => {
let sys = sysinfo::System::new();
let ver = match sys.kernel_version() {
Some(v) => v,
None => "unknown".into(),
};
let os_record = Value::Record {
cols: vec![
"name".into(),
"arch".into(),
"family".into(),
"kernel_version".into(),
],
vals: vec![
Value::string(std::env::consts::OS, self.span()),
Value::string(std::env::consts::ARCH, self.span()),
Value::string(std::env::consts::FAMILY, self.span()),
Value::string(ver, self.span()),
],
span: self.span(),
};
Ok(os_record)
}
"is-interactive" => Ok(Value::Bool {
val: self.engine_state.is_interactive,
span: self.span,
}),
"is-login" => Ok(Value::Bool {
val: self.engine_state.is_login,
span: self.span,
}),
FEATURE: add the startup time to `$nu` (#8353) # Description in https://github.com/nushell/nushell/issues/8311 and the discord server, the idea of moving the default banner from the `rust` source to the `nushell` standar library has emerged :yum: however, in order to do this, one need to have access to all the variables used in the default banner => all of them are accessible because known constants, except for the startup time of the shell, which is not anywhere in the shell... #### this PR adds exactly this, i.e. the new `startup_time` to the `$nu` variable, which is computed to have the exact same value as the value shown in the banner. ## the changes in order to achieve this, i had to - add `startup_time` as an `i64` to the `EngineState` => this is, to the best of my knowledge, the easiest way to pass such an information around down to where the banner startup time is computed and where the `$nu` variable is evaluated - add `startup-time` to the `$nu` variable and use the `EngineState` getter for `startup_time` to show it as a `Value::Duration` - pass `engine_state` as a `&mut`able argument from `main.rs` down to `repl.rs` to allow the setter to change the value of `startup_time` => without this, the value would not change and would show `-1ns` as the default value... - the value of the startup time is computed in `evaluate_repl` in `repl.rs`, only once at the beginning, and the same value is used in the default banner :ok_hand: # User-Facing Changes one can now access to the same time as shown in the default banner with ```bash $nu.startup-time ``` # Tests + Formatting - :green_circle: `cargo fmt --all` - :green_circle: `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` - :green_circle: `cargo test --workspace` # After Submitting ``` $nothing ```
2023-03-09 21:18:58 +01:00
"startup-time" => Ok(Value::Duration {
val: self.engine_state.get_startup_time(),
span: self.span(),
}),
LazyRecord (#7619) This is an attempt to implement a new `Value::LazyRecord` variant for performance reasons. `LazyRecord` is like a regular `Record`, but it's possible to access individual columns without evaluating other columns. I've implemented `LazyRecord` for the special `$nu` variable; accessing `$nu` is relatively slow because of all the information in `scope`, and [`$nu` accounts for about 2/3 of Nu's startup time on Linux](https://github.com/nushell/nushell/issues/6677#issuecomment-1364618122). ### Benchmarks I ran some benchmarks on my desktop (Linux, 12900K) and the results are very pleasing. Nu's time to start up and run a command (`cargo build --release; hyperfine 'target/release/nu -c "echo \"Hello, world!\""' --shell=none --warmup 10`) goes from **8.8ms to 3.2ms, about 2.8x faster**. Tests are also much faster! Running `cargo nextest` (with our very slow `proptest` tests disabled) goes from **7.2s to 4.4s (1.6x faster)**, because most tests involve launching a new instance of Nu. ### Design (updated) I've added a new `LazyRecord` trait and added a `Value` variant wrapping those trait objects, much like `CustomValue`. `LazyRecord` implementations must implement these 2 functions: ```rust // All column names fn column_names(&self) -> Vec<&'static str>; // Get 1 specific column value fn get_column_value(&self, column: &str) -> Result<Value, ShellError>; ``` ### Serializability `Value` variants must implement `Serializable` and `Deserializable`, which poses some problems because I want to use unserializable things like `EngineState` in `LazyRecord`s. To work around this, I basically lie to the type system: 1. Add `#[typetag::serde(tag = "type")]` to `LazyRecord` to make it serializable 2. Any unserializable fields in `LazyRecord` implementations get marked with `#[serde(skip)]` 3. At the point where a `LazyRecord` normally would get serialized and sent to a plugin, I instead collect it into a regular `Value::Record` (which can be serialized)
2023-01-19 04:27:26 +01:00
_ => err(&format!("Could not find column '{column}'")),
}
}
fn span(&self) -> Span {
self.span
}
fn typetag_name(&self) -> &'static str {
"nu_variable"
}
fn typetag_deserialize(&self) {
unimplemented!("typetag_deserialize")
}
}
// manually implemented so we can skip engine_state which doesn't implement Debug
// FIXME: find a better way
impl fmt::Debug for NuVariable {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("NuVariable").finish()
}
}