holm/nushell
1
1
Fork 0
forked from extern/nushell
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Overlays (#5375)

Browse Source 
* WIP: Start laying overlays

* Rename Overlay->Module; Start adding overlay

* Revamp adding overlay

* Add overlay add tests; Disable debug print

* Fix overlay add; Add overlay remove

* Add overlay remove tests

* Add missing overlay remove file

* Add overlay list command

* (WIP?) Enable overlays for env vars

* Move OverlayFrames to ScopeFrames

* (WIP) Move everything to overlays only

ScopeFrame contains nothing but overlays now

* Fix predecls

* Fix wrong overlay id translation and aliases

* Fix broken env lookup logic

* Remove TODOs

* Add overlay add + remove for environment

* Add a few overlay tests; Fix overlay add name

* Some cleanup; Fix overlay add/remove names

* Clippy

* Fmt

* Remove walls of comments

* List overlays from stack; Add debugging flag

Currently, the engine state ordering is somehow broken.

* Fix (?) overlay list test

* Fix tests on Windows

* Fix activated overlay ordering

* Check for active overlays equality in overlay list

This removes the -p flag: Either both parser and engine will have the
same overlays, or the command will fail.

* Add merging on overlay remove

* Change help message and comment

* Add some remove-merge/discard tests

* (WIP) Track removed overlays properly

* Clippy; Fmt

* Fix getting last overlay; Fix predecls in overlays

* Remove merging; Fix re-add overwriting stuff

Also some error message tweaks.

* Fix overlay error in the engine

* Update variable_completions.rs

* Adds flags and optional arguments to view-source (#5446)

* added flags and optional arguments to view-source

* removed redundant code

* removed redundant code

* fmt

* fix bug in shell_integration (#5450)

* fix bug in shell_integration

* add some comments

* enable cd to work with directory abbreviations (#5452)

* enable cd to work with abbreviations

* add abbreviation example

* fix tests

* make it configurable

* make cd recornize symblic link (#5454)

* implement seq char command to generate single character sequence (#5453)

* add tmp code

* add seq char command

* Add split number flag in `split row` (#5434)

Signed-off-by: Yuheng Su <gipsyh.icu@gmail.com>

* Add two more overlay tests

* Add ModuleId to OverlayFrame

* Fix env conversion accidentally activating overlay

It activated overlay from permanent state prematurely which would
cause `overlay add` to misbehave.

* Remove unused parameter; Add overlay list test

* Remove added traces

* Add overlay commands examples

* Modify TODO

* Fix $nu.scope iteration

* Disallow removing default overlay

* Refactor some parser errors

* Remove last overlay if no argument

* Diversify overlay examples

* Make it possible to update overlay's module

In case the origin module updates, the overlay add loads the new module,
makes it overlay's origin and applies the changes. Before, it was
impossible to update the overlay if the module changed.

Co-authored-by: JT <547158+jntrnr@users.noreply.github.com>
Co-authored-by: pwygab <88221256+merelymyself@users.noreply.github.com>
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
Co-authored-by: WindSoilder <WindSoilder@outlook.com>
Co-authored-by: Yuheng Su <gipsyh.icu@gmail.com>
This commit is contained in:
Jakub Žádník
2022-05-07 22:39:22 +03:00
committed by GitHub
parent 1cb449b2d1
commit 9b99b2f6ac
46 changed files with 2638 additions and 607 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
crates/nu-protocol/src/ast/import_pattern.rs
View File

@ -1,6 +1,6 @@
use serde::{Deserialize, Serialize};
use crate::{span, OverlayId, Span};
use crate::{span, ModuleId, Span};
use std::collections::HashSet;
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
@ -13,7 +13,7 @@ pub enum ImportPatternMember {
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ImportPatternHead {
pub name: Vec<u8>,
pub id: Option<OverlayId>,
pub id: Option<ModuleId>,
pub span: Span,
}

1253
crates/nu-protocol/src/engine/engine_state.rs
View File

File diff suppressed because it is too large Load Diff

327
crates/nu-protocol/src/engine/stack.rs
View File

@ -1,8 +1,11 @@
use std::collections::{HashMap, HashSet};
use crate::engine::EngineState;
use crate::engine::{EngineState, DEFAULT_OVERLAY_NAME};
use crate::{ShellError, Span, Value, VarId};
/// Environment variables per overlay
pub type EnvVars = HashMap<String, HashMap<String, Value>>;
/// A runtime value stack used during evaluation
///
/// A note on implementation:
@ -25,16 +28,11 @@ pub struct Stack {
/// Variables
pub vars: HashMap<VarId, Value>,
/// Environment variables arranged as a stack to be able to recover values from parent scopes
pub env_vars: Vec<HashMap<String, Value>>,
/// Tells which environment variables from engine state are hidden. We don't need to track the
/// env vars in the stack since we can just delete them.
pub env_hidden: HashSet<String>,
}
impl Default for Stack {
fn default() -> Self {
Self::new()
}
pub env_vars: Vec<EnvVars>,
/// Tells which environment variables from engine state are hidden, per overlay.
pub env_hidden: HashMap<String, HashSet<String>>,
/// List of active overlays
pub active_overlays: Vec<String>,
}
impl Stack {
@ -42,18 +40,23 @@ impl Stack {
Stack {
vars: HashMap::new(),
env_vars: vec![],
env_hidden: HashSet::new(),
env_hidden: HashMap::new(),
active_overlays: vec![DEFAULT_OVERLAY_NAME.to_string()],
}
}
pub fn with_env(&mut self, env_vars: &[HashMap<String, Value>], env_hidden: &HashSet<String>) {
pub fn with_env(
&mut self,
env_vars: &[EnvVars],
env_hidden: &HashMap<String, HashSet<String>>,
) {
// Do not clone the environment if it hasn't changed
if self.env_vars.iter().any(|scope| !scope.is_empty()) {
self.env_vars = env_vars.to_owned();
}
if !self.env_hidden.is_empty() {
self.env_hidden = env_hidden.clone();
self.env_hidden = env_hidden.to_owned();
}
}
@ -78,30 +81,52 @@ impl Stack {
}
pub fn add_env_var(&mut self, var: String, value: Value) {
// if the env var was hidden, let's activate it again
self.env_hidden.remove(&var);
if let Some(last_overlay) = self.active_overlays.last() {
if let Some(env_hidden) = self.env_hidden.get_mut(last_overlay) {
// if the env var was hidden, let's activate it again
env_hidden.remove(&var);
}
if let Some(scope) = self.env_vars.last_mut() {
scope.insert(var, value);
if let Some(scope) = self.env_vars.last_mut() {
if let Some(env_vars) = scope.get_mut(last_overlay) {
env_vars.insert(var, value);
} else {
scope.insert(last_overlay.into(), HashMap::from([(var, value)]));
}
} else {
self.env_vars.push(HashMap::from([(
last_overlay.into(),
HashMap::from([(var, value)]),
)]));
}
} else {
self.env_vars.push(HashMap::from([(var, value)]));
// TODO: Remove panic
panic!("internal error: no active overlay");
}
}
pub fn last_overlay_name(&self) -> Result<String, ShellError> {
self.active_overlays
.last()
.cloned()
.ok_or_else(|| ShellError::NushellFailed("No active overlay".into()))
}
pub fn captures_to_stack(&self, captures: &HashMap<VarId, Value>) -> Stack {
let mut output = Stack::new();
output.vars = captures.clone();
// FIXME: this is probably slow
output.env_vars = self.env_vars.clone();
output.env_vars.push(HashMap::new());
let mut env_vars = self.env_vars.clone();
env_vars.push(HashMap::new());
output
Stack {
vars: captures.clone(),
env_vars,
env_hidden: HashMap::new(),
active_overlays: self.active_overlays.clone(),
}
}
pub fn gather_captures(&self, captures: &[VarId]) -> Stack {
let mut output = Stack::new();
let mut vars = HashMap::new();
let fake_span = Span::new(0, 0);
@ -109,30 +134,59 @@ impl Stack {
// Note: this assumes we have calculated captures correctly and that commands
// that take in a var decl will manually set this into scope when running the blocks
if let Ok(value) = self.get_var(*capture, fake_span) {
output.vars.insert(*capture, value);
vars.insert(*capture, value);
}
}
// FIXME: this is probably slow
output.env_vars = self.env_vars.clone();
output.env_vars.push(HashMap::new());
let mut env_vars = self.env_vars.clone();
env_vars.push(HashMap::new());
output
Stack {
vars,
env_vars,
env_hidden: HashMap::new(),
active_overlays: self.active_overlays.clone(),
}
}
/// Flatten the env var scope frames into one frame
pub fn get_env_vars(&self, engine_state: &EngineState) -> HashMap<String, Value> {
// TODO: We're collecting im::HashMap to HashMap here. It might make sense to make these
// the same data structure.
let mut result: HashMap<String, Value> = engine_state
.env_vars
.iter()
.filter(|(k, _)| !self.env_hidden.contains(*k))
.map(|(k, v)| (k.clone(), v.clone()))
.collect();
let mut result = HashMap::new();
for active_overlay in self.active_overlays.iter() {
if let Some(env_vars) = engine_state.env_vars.get(active_overlay) {
result.extend(
env_vars
.iter()
.filter(|(k, _)| {
if let Some(env_hidden) = self.env_hidden.get(active_overlay) {
!env_hidden.contains(*k)
} else {
// nothing has been hidden in this overlay
true
}
})
.map(|(k, v)| (k.clone(), v.clone()))
.collect::<HashMap<String, Value>>(),
);
}
}
result.extend(self.get_stack_env_vars());
result
}
/// Get flattened environment variables only from the stack
pub fn get_stack_env_vars(&self) -> HashMap<String, Value> {
let mut result = HashMap::new();
for scope in &self.env_vars {
result.extend(scope.clone());
for active_overlay in self.active_overlays.iter() {
if let Some(env_vars) = scope.get(active_overlay) {
result.extend(env_vars.clone());
}
}
}
result
@ -140,16 +194,33 @@ impl Stack {
/// Same as get_env_vars, but returns only the names as a HashSet
pub fn get_env_var_names(&self, engine_state: &EngineState) -> HashSet<String> {
let mut result: HashSet<String> = engine_state
.env_vars
.keys()
.filter(|k| !self.env_hidden.contains(*k))
.cloned()
.collect();
let mut result = HashSet::new();
for active_overlay in self.active_overlays.iter() {
if let Some(env_vars) = engine_state.env_vars.get(active_overlay) {
result.extend(
env_vars
.keys()
.filter(|k| {
if let Some(env_hidden) = self.env_hidden.get(active_overlay) {
!env_hidden.contains(*k)
} else {
// nothing has been hidden in this overlay
true
}
})
.cloned()
.collect::<HashSet<String>>(),
);
}
}
for scope in &self.env_vars {
let scope_keys: HashSet<String> = scope.keys().cloned().collect();
result.extend(scope_keys);
for active_overlay in self.active_overlays.iter() {
if let Some(env_vars) = scope.get(active_overlay) {
result.extend(env_vars.keys().cloned().collect::<HashSet<String>>());
}
}
}
result
@ -157,83 +228,123 @@ impl Stack {
pub fn get_env_var(&self, engine_state: &EngineState, name: &str) -> Option<Value> {
for scope in self.env_vars.iter().rev() {
if let Some(v) = scope.get(name) {
return Some(v.clone());
for active_overlay in self.active_overlays.iter().rev() {
if let Some(env_vars) = scope.get(active_overlay) {
if let Some(v) = env_vars.get(name) {
return Some(v.clone());
}
}
}
}
if self.env_hidden.contains(name) {
None
} else {
engine_state.env_vars.get(name).cloned()
for active_overlay in self.active_overlays.iter().rev() {
let is_hidden = if let Some(env_hidden) = self.env_hidden.get(active_overlay) {
env_hidden.contains(name)
} else {
false
};
if !is_hidden {
if let Some(env_vars) = engine_state.env_vars.get(active_overlay) {
if let Some(v) = env_vars.get(name) {
return Some(v.clone());
}
}
}
}
None
}
pub fn has_env_var(&self, engine_state: &EngineState, name: &str) -> bool {
for scope in self.env_vars.iter().rev() {
for active_overlay in self.active_overlays.iter().rev() {
if let Some(env_vars) = scope.get(active_overlay) {
if env_vars.contains_key(name) {
return true;
}
}
}
}
for active_overlay in self.active_overlays.iter().rev() {
let is_hidden = if let Some(env_hidden) = self.env_hidden.get(active_overlay) {
env_hidden.contains(name)
} else {
false
};
if !is_hidden {
if let Some(env_vars) = engine_state.env_vars.get(active_overlay) {
if env_vars.contains_key(name) {
return true;
}
}
}
}
false
}
pub fn remove_env_var(&mut self, engine_state: &EngineState, name: &str) -> Option<Value> {
for scope in self.env_vars.iter_mut().rev() {
for active_overlay in self.active_overlays.iter().rev() {
if let Some(env_vars) = scope.get_mut(active_overlay) {
if let Some(v) = env_vars.remove(name) {
return Some(v);
}
}
}
}
for active_overlay in self.active_overlays.iter().rev() {
if let Some(env_vars) = engine_state.env_vars.get(active_overlay) {
if let Some(val) = env_vars.get(name) {
if let Some(env_hidden) = self.env_hidden.get_mut(active_overlay) {
env_hidden.insert(name.into());
} else {
self.env_hidden
.insert(active_overlay.into(), HashSet::from([name.into()]));
}
return Some(val.clone());
}
}
}
None
}
pub fn has_env_overlay(&self, name: &str, engine_state: &EngineState) -> bool {
for scope in self.env_vars.iter().rev() {
if scope.contains_key(name) {
return true;
}
}
if self.env_hidden.contains(name) {
false
} else {
engine_state.env_vars.contains_key(name)
}
engine_state.env_vars.contains_key(name)
}
pub fn remove_env_var(&mut self, engine_state: &EngineState, name: &str) -> Option<Value> {
for scope in self.env_vars.iter_mut().rev() {
if let Some(v) = scope.remove(name) {
return Some(v);
}
}
pub fn add_overlay(&mut self, name: String) {
self.env_hidden.remove(&name);
if self.env_hidden.contains(name) {
// the environment variable is already hidden
None
} else if let Some(val) = engine_state.env_vars.get(name) {
// the environment variable was found in the engine state => mark it as hidden
self.env_hidden.insert(name.to_string());
Some(val.clone())
} else {
None
}
self.active_overlays.retain(|o| o != &name);
self.active_overlays.push(name);
}
// pub fn get_config(&self) -> Result<Config, ShellError> {
// let config = self.get_var(CONFIG_VARIABLE_ID, Span::new(0, 0));
// match config {
// Ok(config) => config.into_config(),
// Err(e) => Err(e),
// }
// }
// pub fn update_config(&mut self, name: &str, value: Value) {
// if let Some(Value::Record { cols, vals, .. }) = self.vars.get_mut(&CONFIG_VARIABLE_ID) {
// for col_val in cols.iter().zip(vals.iter_mut()) {
// if col_val.0 == name {
// *col_val.1 = value;
// return;
// }
// }
// cols.push(name.to_string());
// vals.push(value);
// }
// }
pub fn print_stack(&self) {
println!("vars:");
for (var, val) in &self.vars {
println!(" {}: {:?}", var, val);
}
for (i, scope) in self.env_vars.iter().rev().enumerate() {
println!("env vars, scope {} (from the last);", i);
for (var, val) in scope {
println!(" {}: {:?}", var, val.clone().debug_value());
}
pub fn remove_overlay(&mut self, name: &String, span: &Span) -> Result<(), ShellError> {
if !self.active_overlays.contains(name) {
return Err(ShellError::OverlayNotFoundAtRuntime(name.into(), *span));
}
self.active_overlays.retain(|o| o != name);
Ok(())
}
}
impl Default for Stack {
fn default() -> Self {
Self::new()
}
}

1
crates/nu-protocol/src/id.rs
View File

@ -2,4 +2,5 @@ pub type VarId = usize;
pub type DeclId = usize;
pub type AliasId = usize;
pub type BlockId = usize;
pub type ModuleId = usize;
pub type OverlayId = usize;

4
crates/nu-protocol/src/lib.rs
View File

@ -5,7 +5,7 @@ pub mod engine;
mod example;
mod exportable;
mod id;
mod overlay;
mod module;
mod pipeline_data;
mod shell_error;
mod signature;
@ -21,7 +21,7 @@ pub use engine::{ENV_VARIABLE_ID, IN_VARIABLE_ID, NU_VARIABLE_ID};
pub use example::*;
pub use exportable::*;
pub use id::*;
pub use overlay::*;
pub use module::*;
pub use pipeline_data::*;
pub use shell_error::*;
pub use signature::*;

12
crates/nu-protocol/src/overlay.rs → crates/nu-protocol/src/module.rs
View File

@ -7,16 +7,16 @@ use indexmap::IndexMap;
/// Collection of definitions that can be exported from a module
#[derive(Debug, Clone)]
pub struct Overlay {
pub struct Module {
pub decls: IndexMap<Vec<u8>, DeclId>,
pub aliases: IndexMap<Vec<u8>, AliasId>,
pub env_vars: IndexMap<Vec<u8>, BlockId>,
pub span: Option<Span>,
}
impl Overlay {
impl Module {
pub fn new() -> Self {
Overlay {
Module {
decls: IndexMap::new(),
aliases: IndexMap::new(),
env_vars: IndexMap::new(),
@ -25,7 +25,7 @@ impl Overlay {
}
pub fn from_span(span: Span) -> Self {
Overlay {
Module {
decls: IndexMap::new(),
aliases: IndexMap::new(),
env_vars: IndexMap::new(),
@ -45,7 +45,7 @@ impl Overlay {
self.env_vars.insert(name.to_vec(), block_id)
}
pub fn extend(&mut self, other: &Overlay) {
pub fn extend(&mut self, other: &Module) {
self.decls.extend(other.decls.clone());
self.env_vars.extend(other.env_vars.clone());
}
@ -201,7 +201,7 @@ impl Overlay {
}
}
impl Default for Overlay {
impl Default for Module {
fn default() -> Self {
Self::new()
}

2
crates/nu-protocol/src/pipeline_data.rs
View File

@ -448,7 +448,7 @@ impl PipelineData {
return Ok(());
}
match engine_state.find_decl("table".as_bytes()) {
match engine_state.find_decl("table".as_bytes(), &[]) {
Some(decl_id) => {
let table = engine_state.get_decl(decl_id).run(
engine_state,

53
crates/nu-protocol/src/shell_error.rs
View File

@ -168,7 +168,7 @@ pub enum ShellError {
/// It is very likely that this is a bug. Please file an issue at https://github.com/nushell/nushell/issues with relevant information.
#[error("Nushell failed: {0}.")]
#[diagnostic(code(nu::shell::nushell_failed), url(docsrs))]
// Only use this one if we Nushell completely falls over and hits a state that isn't possible or isn't recoverable
// Only use this one if Nushell completely falls over and hits a state that isn't possible or isn't recoverable
NushellFailed(String),
/// Catastrophic nushell failure. This reflects a completely unexpected or unrecoverable error.
@ -177,10 +177,30 @@ pub enum ShellError {
///
/// It is very likely that this is a bug. Please file an issue at https://github.com/nushell/nushell/issues with relevant information.
#[error("Nushell failed: {0}.")]
#[diagnostic(code(nu::shell::nushell_failed), url(docsrs))]
// Only use this one if we Nushell completely falls over and hits a state that isn't possible or isn't recoverable
#[diagnostic(code(nu::shell::nushell_failed_spanned), url(docsrs))]
// Only use this one if Nushell completely falls over and hits a state that isn't possible or isn't recoverable
NushellFailedSpanned(String, String, #[label = "{1}"] Span),
/// Catastrophic nushell failure. This reflects a completely unexpected or unrecoverable error.
///
/// ## Resolution
///
/// It is very likely that this is a bug. Please file an issue at https://github.com/nushell/nushell/issues with relevant information.
#[error("Nushell failed: {0}.")]
#[diagnostic(code(nu::shell::nushell_failed_help), url(docsrs))]
// Only use this one if Nushell completely falls over and hits a state that isn't possible or isn't recoverable
NushellFailedHelp(String, #[help] String),
/// Catastrophic nushell failure. This reflects a completely unexpected or unrecoverable error.
///
/// ## Resolution
///
/// It is very likely that this is a bug. Please file an issue at https://github.com/nushell/nushell/issues with relevant information.
#[error("Nushell failed: {0}.")]
#[diagnostic(code(nu::shell::nushell_failed_spanned_help), url(docsrs))]
// Only use this one if Nushell completely falls over and hits a state that isn't possible or isn't recoverable
NushellFailedSpannedHelp(String, String, #[label = "{1}"] Span, #[help] String),
/// A referenced variable was not found at runtime.
///
/// ## Resolution
@ -199,6 +219,33 @@ pub enum ShellError {
#[diagnostic(code(nu::shell::env_variable_not_found), url(docsrs))]
EnvVarNotFoundAtRuntime(String, #[label = "environment variable not found"] Span),
/// A referenced module was not found at runtime.
///
/// ## Resolution
///
/// Check the module name. Did you typo it? Did you forget to declare it? Is the casing right?
#[error("Module '{0}' not found")]
#[diagnostic(code(nu::shell::module_not_found), url(docsrs))]
ModuleNotFoundAtRuntime(String, #[label = "module not found"] Span),
/// A referenced module or overlay was not found at runtime.
///
/// ## Resolution
///
/// Check the module name. Did you typo it? Did you forget to declare it? Is the casing right?
#[error("Module or overlay'{0}' not found")]
#[diagnostic(code(nu::shell::module_not_found), url(docsrs))]
ModuleOrOverlayNotFoundAtRuntime(String, #[label = "not a module or overlay"] Span),
/// A referenced overlay was not found at runtime.
///
/// ## Resolution
///
/// Check the overlay name. Did you typo it? Did you forget to declare it? Is the casing right?
#[error("Overlay '{0}' not found")]
#[diagnostic(code(nu::shell::overlay_not_found), url(docsrs))]
OverlayNotFoundAtRuntime(String, #[label = "overlay not found"] Span),
/// The given item was not found. This is a fairly generic error that depends on context.
///
/// ## Resolution