feat!: Explicit cell-path case sensitivity syntax (#15692)

Related:
- #15683
- #14551
- #849
- #12701
- #11527

# Description
Currently various commands have differing behavior regarding cell-paths

```nushell
{a: 1, A: 2} | get a A
# => ╭───┬───╮
# => │ 0 │ 2 │
# => │ 1 │ 2 │
# => ╰───┴───╯
{a: 1, A: 2} | select a A
# => ╭───┬───╮
# => │ a │ 1 │
# => │ A │ 2 │
# => ╰───┴───╯
{A: 1} | update a 2
# => Error: nu:🐚:column_not_found
# => 
# =>   × Cannot find column 'a'
# =>    ╭─[entry #62:1:1]
# =>  1 │ {A: 1} | update a 2
# =>    · ───┬──          ┬
# =>    ·    │            ╰── cannot find column 'a'
# =>    ·    ╰── value originates here
# =>    ╰────
```

Proposal: making cell-path access case-sensitive by default and adding
new syntax for case-insensitive parts, similar to optional (?) parts.

```nushell
{FOO: BAR}.foo
# => Error: nu:🐚:name_not_found
# => 
# =>   × Name not found
# =>    ╭─[entry #60:1:21]
# =>  1 │ {FOO: BAR}.foo
# =>    ·            ─┬─
# =>    ·             ╰── did you mean 'FOO'?
# =>    ╰────
{FOO: BAR}.foo!
# => BAR
```

This would solve the problem of case sensitivity for all commands
without causing an explosion of flags _and_ make it more granular

Assigning to a field using a case-insensitive path is case-preserving.
```nushell
mut val = {FOO: "I'm FOO"}; $val
# => ╭─────┬─────────╮
# => │ FOO │ I'm FOO │
# => ╰─────┴─────────╯
$val.foo! = "I'm still FOO"; $val
# => ╭─────┬───────────────╮
# => │ FOO │ I'm still FOO │
# => ╰─────┴───────────────╯
```

For `update`, case-insensitive is case-preserving.
```nushell
{FOO: 1} | update foo! { $in + 1 }
# => ╭─────┬───╮
# => │ FOO │ 2 │
# => ╰─────┴───╯
```

`insert` can insert values into nested values so accessing into existing
columns is case-insensitive, but creating new columns uses the cell-path
as it is.
So `insert foo! ...` and `insert FOO! ...` would work exactly as they do
without `!`
```nushell
{FOO: {quox: 0}}
# => ╭─────┬──────────────╮
# => │     │ ╭──────┬───╮ │
# => │ FOO │ │ quox │ 0 │ │
# => │     │ ╰──────┴───╯ │
# => ╰─────┴──────────────╯
{FOO: {quox: 0}} | insert foo.bar 1
# => ╭─────┬──────────────╮
# => │     │ ╭──────┬───╮ │
# => │ FOO │ │ quox │ 0 │ │
# => │     │ ╰──────┴───╯ │
# => │     │ ╭─────┬───╮  │
# => │ foo │ │ bar │ 1 │  │
# => │     │ ╰─────┴───╯  │
# => ╰─────┴──────────────╯
{FOO: {quox: 0}} | insert foo!.bar 1
# => ╭─────┬──────────────╮
# => │     │ ╭──────┬───╮ │
# => │ FOO │ │ quox │ 0 │ │
# => │     │ │ bar  │ 1 │ │
# => │     │ ╰──────┴───╯ │
# => ╰─────┴──────────────╯
```

`upsert` is tricky, depending on the input, the data might end up with
different column names in rows. We can either forbid case-insensitive
cell-paths for `upsert` or trust the user to keep their data in a
sensible shape.

This would be a breaking change as it would make existing cell-path
accesses case-sensitive, however the case-sensitivity is already
inconsistent and any attempt at making it consistent would be a breaking
change.

> What about `$env`?

1. Initially special case it so it keeps its current behavior.
2. Accessing environment variables with non-matching paths gives a
deprecation warning urging users to either use exact casing or use the
new explicit case-sensitivity syntax
3. Eventuall remove `$env`'s special case, making `$env` accesses
case-sensitive by default as well.

> `$env.ENV_CONVERSIONS`?

In addition to `from_string` and `to_string` add an optional field to
opt into case insensitive/preserving behavior.

# User-Facing Changes

- `get`, `where` and other previously case-insensitive commands are now
case-sensitive by default.
- `get`'s `--sensitive` flag removed, similar to `--ignore-errors` there
is now an `--ignore-case` flag that treats all parts of the cell-path as
case-insensitive.
- Users can explicitly choose the case case-sensitivity of cell-path
accesses or commands.

# Tests + Formatting

Existing tests required minimal modification. ***However, new tests are
not yet added***.

- 🟢 toolkit fmt
- 🟢 toolkit clippy
- 🟢 toolkit test
- 🟢 toolkit test stdlib

# After Submitting

- Update the website to include the new syntax
- Update [tree-sitter-nu](https://github.com/nushell/tree-sitter-nu)

---------

Co-authored-by: Bahex <17417311+Bahex@users.noreply.github.com>
This commit is contained in:
Bahex
2025-05-18 12:19:09 +03:00
committed by GitHub
parent 1e8876b076
commit c4dcfdb77b
34 changed files with 466 additions and 207 deletions

View File

@ -1,5 +1,5 @@
use nu_engine::command_prelude::*;
use nu_protocol::ast::PathMember;
use nu_protocol::{ast::PathMember, casing::Casing};
#[derive(Clone)]
pub struct IntoCellPath;
@ -17,7 +17,12 @@ impl Command for IntoCellPath {
(Type::List(Box::new(Type::Any)), Type::CellPath),
(
Type::List(Box::new(Type::Record(
[("value".into(), Type::Any), ("optional".into(), Type::Bool)].into(),
[
("value".into(), Type::Any),
("optional".into(), Type::Bool),
("insensitive".into(), Type::Bool),
]
.into(),
))),
Type::CellPath,
),
@ -69,8 +74,8 @@ impl Command for IntoCellPath {
example: "'some.path' | split row '.' | into cell-path",
result: Some(Value::test_cell_path(CellPath {
members: vec![
PathMember::test_string("some".into(), false),
PathMember::test_string("path".into(), false),
PathMember::test_string("some".into(), false, Casing::Sensitive),
PathMember::test_string("path".into(), false, Casing::Sensitive),
],
})),
},
@ -80,19 +85,20 @@ impl Command for IntoCellPath {
result: Some(Value::test_cell_path(CellPath {
members: vec![
PathMember::test_int(5, false),
PathMember::test_string("c".into(), false),
PathMember::test_string("c".into(), false, Casing::Sensitive),
PathMember::test_int(7, false),
PathMember::test_string("h".into(), false),
PathMember::test_string("h".into(), false, Casing::Sensitive),
],
})),
},
Example {
description: "Convert table into cell path",
example: "[[value, optional]; [5 true] [c false]] | into cell-path",
example: "[[value, optional, insensitive]; [5 true false] [c false false] [d false true]] | into cell-path",
result: Some(Value::test_cell_path(CellPath {
members: vec![
PathMember::test_int(5, true),
PathMember::test_string("c".into(), false),
PathMember::test_string("c".into(), false, Casing::Sensitive),
PathMember::test_string("d".into(), false, Casing::Insensitive),
],
})),
},
@ -175,6 +181,12 @@ fn record_to_path_member(
}
};
if let Some(insensitive) = record.get("insensitive") {
if insensitive.as_bool()? {
member.make_insensitive();
}
};
Ok(member)
}
@ -196,7 +208,9 @@ fn value_to_path_member(val: &Value, span: Span) -> Result<PathMember, ShellErro
let val_span = val.span();
let member = match val {
Value::Int { val, .. } => int_to_path_member(*val, val_span)?,
Value::String { val, .. } => PathMember::string(val.into(), false, val_span),
Value::String { val, .. } => {
PathMember::string(val.into(), false, Casing::Sensitive, val_span)
}
Value::Record { val, .. } => record_to_path_member(val, val_span, span)?,
other => {
return Err(ShellError::CantConvert {

View File

@ -1,5 +1,5 @@
use nu_engine::command_prelude::*;
use nu_protocol::{IntoValue, ast::PathMember};
use nu_protocol::{IntoValue, ast::PathMember, casing::Casing};
#[derive(Clone)]
pub struct SplitCellPath;
@ -16,7 +16,12 @@ impl Command for SplitCellPath {
(
Type::CellPath,
Type::List(Box::new(Type::Record(
[("value".into(), Type::Any), ("optional".into(), Type::Bool)].into(),
[
("value".into(), Type::Any),
("optional".into(), Type::Bool),
("insensitive".into(), Type::Bool),
]
.into(),
))),
),
])
@ -72,36 +77,43 @@ impl Command for SplitCellPath {
Value::test_record(record! {
"value" => Value::test_int(5),
"optional" => Value::test_bool(true),
"insensitive" => Value::test_bool(false),
}),
Value::test_record(record! {
"value" => Value::test_string("c"),
"optional" => Value::test_bool(false),
"insensitive" => Value::test_bool(false),
}),
])),
},
Example {
description: "Split a complex cell-path",
example: r#"$.a.b?.1."2"."c.d" | split cell-path"#,
example: r#"$.a!.b?.1."2"."c.d" | split cell-path"#,
result: Some(Value::test_list(vec![
Value::test_record(record! {
"value" => Value::test_string("a"),
"optional" => Value::test_bool(false),
"insensitive" => Value::test_bool(true),
}),
Value::test_record(record! {
"value" => Value::test_string("b"),
"optional" => Value::test_bool(true),
"insensitive" => Value::test_bool(false),
}),
Value::test_record(record! {
"value" => Value::test_int(1),
"optional" => Value::test_bool(false),
"insensitive" => Value::test_bool(false),
}),
Value::test_record(record! {
"value" => Value::test_string("2"),
"optional" => Value::test_bool(false),
"insensitive" => Value::test_bool(false),
}),
Value::test_record(record! {
"value" => Value::test_string("c.d"),
"optional" => Value::test_bool(false),
"insensitive" => Value::test_bool(false),
}),
])),
},
@ -114,19 +126,29 @@ fn split_cell_path(val: CellPath, span: Span) -> Result<Value, ShellError> {
struct PathMemberRecord {
value: Value,
optional: bool,
insensitive: bool,
}
impl PathMemberRecord {
fn from_path_member(pm: PathMember) -> Self {
let (optional, internal_span) = match pm {
PathMember::String { optional, span, .. }
| PathMember::Int { optional, span, .. } => (optional, span),
let (optional, insensitive, internal_span) = match pm {
PathMember::String {
optional,
casing,
span,
..
} => (optional, casing == Casing::Insensitive, span),
PathMember::Int { optional, span, .. } => (optional, false, span),
};
let value = match pm {
PathMember::String { val, .. } => Value::string(val, internal_span),
PathMember::Int { val, .. } => Value::int(val as i64, internal_span),
};
Self { value, optional }
Self {
value,
optional,
insensitive,
}
}
}

View File

@ -15,7 +15,7 @@ pub fn empty(
if !columns.is_empty() {
for val in input {
for column in &columns {
if !val.follow_cell_path(&column.members, false)?.is_nothing() {
if !val.follow_cell_path(&column.members)?.is_nothing() {
return Ok(Value::bool(negate, head).into_pipeline_data());
}
}

View File

@ -1,7 +1,7 @@
use std::borrow::Cow;
use nu_engine::command_prelude::*;
use nu_protocol::{Signals, ast::PathMember};
use nu_protocol::{Signals, ast::PathMember, report_shell_warning};
#[derive(Clone)]
pub struct Get;
@ -47,7 +47,7 @@ If multiple cell paths are given, this will produce a list of values."#
)
.switch(
"sensitive",
"get path in a case sensitive manner",
"get path in a case sensitive manner (deprecated)",
Some('s'),
)
.allow_variants_without_examples(true)
@ -86,12 +86,12 @@ If multiple cell paths are given, this will produce a list of values."#
},
Example {
description: "Getting Path/PATH in a case insensitive way",
example: "$env | get paTH",
example: "$env | get paTH!",
result: None,
},
Example {
description: "Getting Path in a case sensitive way, won't work for 'PATH'",
example: "$env | get --sensitive Path",
example: "$env | get Path",
result: None,
},
]
@ -110,14 +110,12 @@ If multiple cell paths are given, this will produce a list of values."#
let cell_path: CellPath = call.req_const(working_set, 0)?;
let rest: Vec<CellPath> = call.rest_const(working_set, 1)?;
let ignore_errors = call.has_flag_const(working_set, "ignore-errors")?;
let sensitive = call.has_flag_const(working_set, "sensitive")?;
let metadata = input.metadata();
action(
input,
cell_path,
rest,
ignore_errors,
sensitive,
working_set.permanent().signals().clone(),
call.head,
)
@ -134,14 +132,24 @@ If multiple cell paths are given, this will produce a list of values."#
let cell_path: CellPath = call.req(engine_state, stack, 0)?;
let rest: Vec<CellPath> = call.rest(engine_state, stack, 1)?;
let ignore_errors = call.has_flag(engine_state, stack, "ignore-errors")?;
let sensitive = call.has_flag(engine_state, stack, "sensitive")?;
let sensitive_span = call.get_flag_span(stack, "sensitive");
let metadata = input.metadata();
if let Some(span) = sensitive_span {
report_shell_warning(
engine_state,
&ShellError::Deprecated {
deprecated: "sensitive flag",
suggestion: "",
span,
help: Some("cell-paths are case-sensitive by default"),
},
);
}
action(
input,
cell_path,
rest,
ignore_errors,
sensitive,
engine_state.signals().clone(),
call.head,
)
@ -154,7 +162,6 @@ fn action(
mut cell_path: CellPath,
mut rest: Vec<CellPath>,
ignore_errors: bool,
sensitive: bool,
signals: Signals,
span: Span,
) -> Result<PipelineData, ShellError> {
@ -180,7 +187,7 @@ fn action(
}
if rest.is_empty() {
follow_cell_path_into_stream(input, signals, cell_path.members, span, !sensitive)
follow_cell_path_into_stream(input, signals, cell_path.members, span)
} else {
let mut output = vec![];
@ -189,11 +196,7 @@ fn action(
let input = input.into_value(span)?;
for path in paths {
output.push(
input
.follow_cell_path(&path.members, !sensitive)?
.into_owned(),
);
output.push(input.follow_cell_path(&path.members)?.into_owned());
}
Ok(output.into_iter().into_pipeline_data(span, signals))
@ -212,7 +215,6 @@ pub fn follow_cell_path_into_stream(
signals: Signals,
cell_path: Vec<PathMember>,
head: Span,
insensitive: bool,
) -> Result<PipelineData, ShellError> {
// when given an integer/indexing, we fallback to
// the default nushell indexing behaviour
@ -227,7 +229,7 @@ pub fn follow_cell_path_into_stream(
let span = value.span();
value
.follow_cell_path(&cell_path, insensitive)
.follow_cell_path(&cell_path)
.map(Cow::into_owned)
.unwrap_or_else(|error| Value::error(error, span))
})
@ -237,7 +239,7 @@ pub fn follow_cell_path_into_stream(
}
_ => data
.follow_cell_path(&cell_path, head, insensitive)
.follow_cell_path(&cell_path, head)
.map(|x| x.into_pipeline_data()),
}
}

View File

@ -322,7 +322,7 @@ fn group_cell_path(
let mut groups = IndexMap::<_, Vec<_>>::new();
for value in values.into_iter() {
let key = value.follow_cell_path(&column_name.members, false)?;
let key = value.follow_cell_path(&column_name.members)?;
if key.is_nothing() {
continue; // likely the result of a failed optional access, ignore this value

View File

@ -301,7 +301,7 @@ fn insert_value_by_closure(
) -> Result<(), ShellError> {
let value_at_path = if first_path_member_int {
value
.follow_cell_path(cell_path, false)
.follow_cell_path(cell_path)
.map(Cow::into_owned)
.unwrap_or(Value::nothing(span))
} else {
@ -321,7 +321,7 @@ fn insert_single_value_by_closure(
) -> Result<(), ShellError> {
let value_at_path = if first_path_member_int {
value
.follow_cell_path(cell_path, false)
.follow_cell_path(cell_path)
.map(Cow::into_owned)
.unwrap_or(Value::nothing(span))
} else {

View File

@ -1,5 +1,5 @@
use nu_engine::command_prelude::*;
use nu_protocol::ast::PathMember;
use nu_protocol::{ast::PathMember, casing::Casing};
use std::{cmp::Reverse, collections::HashSet};
#[derive(Clone)]
@ -63,6 +63,7 @@ impl Command for Reject {
val: val.clone(),
span: *col_span,
optional: false,
casing: Casing::Sensitive,
}],
};
new_columns.push(cv.clone());

View File

@ -1,5 +1,5 @@
use nu_engine::command_prelude::*;
use nu_protocol::{PipelineIterator, ast::PathMember};
use nu_protocol::{PipelineIterator, ast::PathMember, casing::Casing};
use std::collections::BTreeSet;
#[derive(Clone)]
@ -67,6 +67,7 @@ produce a table, a list will produce a list, and a record will produce a record.
val,
span: col_span,
optional: false,
casing: Casing::Sensitive,
}],
};
new_columns.push(cv);
@ -233,7 +234,7 @@ fn select(
if !columns.is_empty() {
let mut record = Record::new();
for path in &columns {
match input_val.follow_cell_path(&path.members, false) {
match input_val.follow_cell_path(&path.members) {
Ok(fetcher) => {
record.push(path.to_column_name(), fetcher.into_owned());
}
@ -256,7 +257,7 @@ fn select(
let mut record = Record::new();
for cell_path in columns {
let result = v.follow_cell_path(&cell_path.members, false)?;
let result = v.follow_cell_path(&cell_path.members)?;
record.push(cell_path.to_column_name(), result.into_owned());
}
@ -273,7 +274,7 @@ fn select(
if !columns.is_empty() {
let mut record = Record::new();
for path in &columns {
match x.follow_cell_path(&path.members, false) {
match x.follow_cell_path(&path.members) {
Ok(value) => {
record.push(path.to_column_name(), value.into_owned());
}

View File

@ -1,5 +1,5 @@
use nu_engine::command_prelude::*;
use nu_protocol::ast::PathMember;
use nu_protocol::{ast::PathMember, casing::Casing};
use crate::Comparator;
@ -164,7 +164,14 @@ impl Command for Sort {
if let Type::Table(cols) = r#type {
let columns: Vec<Comparator> = cols
.iter()
.map(|col| vec![PathMember::string(col.0.clone(), false, Span::unknown())])
.map(|col| {
vec![PathMember::string(
col.0.clone(),
false,
Casing::Sensitive,
Span::unknown(),
)]
})
.map(|members| CellPath { members })
.map(Comparator::CellPath)
.collect();

View File

@ -243,7 +243,7 @@ fn update_value_by_closure(
cell_path: &[PathMember],
first_path_member_int: bool,
) -> Result<(), ShellError> {
let value_at_path = value.follow_cell_path(cell_path, false)?;
let value_at_path = value.follow_cell_path(cell_path)?;
let arg = if first_path_member_int {
value_at_path.as_ref()
@ -266,7 +266,7 @@ fn update_single_value_by_closure(
cell_path: &[PathMember],
first_path_member_int: bool,
) -> Result<(), ShellError> {
let value_at_path = value.follow_cell_path(cell_path, false)?;
let value_at_path = value.follow_cell_path(cell_path)?;
let arg = if first_path_member_int {
value_at_path.as_ref()

View File

@ -321,7 +321,7 @@ fn upsert_value_by_closure(
cell_path: &[PathMember],
first_path_member_int: bool,
) -> Result<(), ShellError> {
let value_at_path = value.follow_cell_path(cell_path, false);
let value_at_path = value.follow_cell_path(cell_path);
let arg = if first_path_member_int {
value_at_path
@ -352,7 +352,7 @@ fn upsert_single_value_by_closure(
cell_path: &[PathMember],
first_path_member_int: bool,
) -> Result<(), ShellError> {
let value_at_path = value.follow_cell_path(cell_path, false);
let value_at_path = value.follow_cell_path(cell_path);
let arg = if first_path_member_int {
value_at_path

View File

@ -89,7 +89,7 @@ impl Command for InputList {
.into_iter()
.map(move |val| {
let display_value = if let Some(ref cellpath) = display_path {
val.follow_cell_path(&cellpath.members, false)?
val.follow_cell_path(&cellpath.members)?
.to_expanded_string(", ", &config)
} else {
val.to_expanded_string(", ", &config)

View File

@ -239,8 +239,8 @@ pub fn compare_cell_path(
insensitive: bool,
natural: bool,
) -> Result<Ordering, ShellError> {
let left = left.follow_cell_path(&cell_path.members, false)?;
let right = right.follow_cell_path(&cell_path.members, false)?;
let left = left.follow_cell_path(&cell_path.members)?;
let right = right.follow_cell_path(&cell_path.members)?;
compare_values(&left, &right, insensitive, natural)
}

View File

@ -1,6 +1,6 @@
use chrono::{DateTime, FixedOffset};
use nu_path::AbsolutePathBuf;
use nu_protocol::{Span, Value, ast::PathMember, engine::EngineState, record};
use nu_protocol::{Span, Value, ast::PathMember, casing::Casing, engine::EngineState, record};
use nu_test_support::{
fs::{Stub, line_ending},
nu, pipeline,
@ -327,6 +327,7 @@ fn into_sqlite_big_insert() {
val: "somedate".into(),
span: Span::unknown(),
optional: false,
casing: Casing::Sensitive,
}],
Box::new(|dateval| {
Value::string(dateval.coerce_string().unwrap(), dateval.span())

View File

@ -2,6 +2,7 @@ use nu_command::{Comparator, sort, sort_by, sort_record};
use nu_protocol::{
Record, Span, Value,
ast::{CellPath, PathMember},
casing::Casing,
record,
};
@ -527,6 +528,7 @@ fn test_sort_equivalent() {
val: "value".to_string(),
span: Span::test_data(),
optional: false,
casing: Casing::Sensitive,
}],
});