Commit Graph

9281 Commits

Author SHA1 Message Date
Bruce Weirdan
fcb8e36caa
Remove default list-diving behaviour (#13386)
# Description

Before this change `default` would dive into lists and replace `null`
elements with the default value.
Therefore there was no easy way to process `null|list<null|string>`
input and receive `list<null|string>`
(IOW to apply the default on the top level only).

However it's trivially easy to apply default values to list elements
with the `each` command:
```nushell
[null, "a", null] | each { default "b" }
```

So there's no need to have this behavior in `default` command.

# User-Facing Changes

* `default` no longer dives into lists to replace `null` elements with
the default value.

# Tests + Formatting

Added a couple of tests for the new (and old) behavior.

# After Submitting

* Update docs.
2024-07-16 03:54:24 +00:00
Ian Manske
0918050ac8
Deprecate group in favor of chunks (#13377)
# Description
The name of the `group` command is a little unclear/ambiguous.
Everything I look at it, I think of `group-by`. I think `chunks` more
clearly conveys what the `group` command does. Namely, it divides the
input list into chunks of a certain size. For example,
[`slice::chunks`](https://doc.rust-lang.org/std/primitive.slice.html#method.chunks)
has the same name. So, this PR adds a new `chunks` command to replace
the now deprecated `group` command.

The `chunks` command is a refactored version of `group`. As such, there
is a small performance improvement:
```nushell
# $data is a very large list
> bench { $data | chunks 2 } --rounds 30 | get mean
474ms 921µs 190ns

# deprecation warning was disabled here for fairness
> bench { $data | group 2 } --rounds 30 | get mean
592ms 702µs 440ns



> bench { $data | chunks 200 } --rounds 30 | get mean
374ms 188µs 318ns

> bench { $data | group 200 } --rounds 30 | get mean
481ms 264µs 869ns 



> bench { $data | chunks 1 } --rounds 30 | get mean
642ms 574µs 42ns

> bench { $data | group 1 } --rounds 30 | get mean
981ms 602µs 513ns
```

# User-Facing Changes
- `group` command has been deprecated in favor of new `chunks` command.
- `chunks` errors when given a chunk size of `0` whereas `group` returns
chunks with one element.

# Tests + Formatting
Added tests for `chunks`, since `group` did not have any tests.

# After Submitting
Update book if necessary.
2024-07-16 03:49:00 +00:00
Stefan Holderbach
3d1145e759
Fix CI test failure on main (nu-json) (#13374)
Conflict resulting from #13329 and #13326
2024-07-14 10:37:57 +02:00
David
cf4864a9cd
JSON format output keeps braces on same line (issue #13326) (#13352)
# Description
This is a minor breaking change to JSON output syntax/style of the `to
json` command.

This fixes #13326 by setting `braces_same_line` to true when creating a
new `HjsonFormatter`.
This then simply tells `HjsonFormatter` to keep the braces on the same
line when outputting which is what I expected nu's `to json` command to
do.
There are almost no changes to nushell itself, all changes are contained
within `nu-json` crate (minus any documentation updates).

Oh, almost forgot to mention, to get the tests compiling, I added
fancy_regex as a _dev_ dependency to nu-json. I could look into
eliminating that if desirable.

# User-Facing Changes

**Breaking Change**
nushell now outputs the desired result using the reproduction command
from the issue:
```
echo '{"version": "v0.4.4","notes": "blablabla","pub_date": "2024-05-04T16:05:00Z","platforms":{"windows-x86_64":{"signature": "blablabla","url": "https://blablabla"}}}' | from json | to json
```

outputs:
```
{
  "version": "v0.4.4",
  "notes": "blablabla",
  "pub_date": "2024-05-04T16:05:00Z",
  "platforms": {
    "windows-x86_64": {
      "signature": "blablabla",
      "url": "https://blablabla"
    }
  }
}
```

whereas previously it would push the opening braces onto a new line:
```
{
  "version": "v0.4.4",
  "notes": "blablabla",
  "pub_date": "2024-05-04T16:05:00Z",
  "platforms":
  {
    "windows-x86_64":
    {
      "signature": "blablabla",
      "url": "https://blablabla"
    }
  }
}
```

# Tests + Formatting

toolkit check pr mostly passes - there are regrettably some tests not
passing on my windows machine _before making any changes_ (I may look
into this as a separate issue)

I have re-enabled the [hjson
tests](https://github.com/nushell/nushell/blob/main/crates/nu-json/tests/main.rs).
This is done in the second commit 🙂 
They have a crucial difference to what they were previously asserting:
  * nu-json outputs in json syntax, not hjson syntax
I think this is desirable, but I'm not aware of the history of these
tests.

# After Submitting

I suspect there `to json` command examples will need updating to match,
haven't checked yet!
2024-07-14 10:19:09 +02:00
Devyn Cairns
ae40d56fc5
Report parse warns and compile errs when running script files (#13369)
# Description

Report parse warnings and compile errors when running script files.

It's useful to report this information to script authors and users so
they can know if they need to update something in the near future.

I also found that moving some errors from eval to compile time meant
some error tests can fail (in `tests/repl`) so this is a good idea to
keep those passing.

# User-Facing Changes
- Report parse warnings when running script files
- Report compile errors when running script files
2024-07-14 10:12:55 +02:00
Stefan Holderbach
c5aa15c7f6
Add top-level crate documentation/READMEs (#12907)
# Description
Add `README.md` files to each crate in our workspace (-plugins) and also
include it in the `lib.rs` documentation for <docs.rs> (if there is no
existing `lib.rs` crate documentation)

In all new README I added the defensive comment that the crates are not
considered stable for public consumption. If necessary we can adjust
this if we deem a crate useful for plugin authors.
2024-07-14 10:10:41 +02:00
Stefan Holderbach
f5bff8c9c8
Fix select cell path renaming behavior (#13361)
# Description

Fixes #13359

In an attempt to generate names for flat columns resulting from a nested
accesses #3016 generated new column names on nested selection, out of
convenience, that composed the cell path as a string (including `.`) and
then simply replaced all `.` with `_`. As we permit `.` in column names
as long as you quote this surprisingly alters `select`ed columns.


# User-Facing Changes
New columns generated by selection with nested cell paths will for now
be named with a string containing the keys separated by `.` instead of
`_`. We may want to reconsider the semantics for nested access.

# Tests + Formatting
- Alter test to breaking change on nested `select`
2024-07-13 16:54:34 +02:00
Yash Thakur
b0bf54614f
Don't add touch command to default context twice (#13371)
# Description

Touch was added to the shell command context twice, once with the other
filesystem commands and once with the format commands. I removed that
second occurrence of touch, because I'm assuming it was only added there
because "Touch" starts with "To."

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

None
2024-07-13 16:52:39 +02:00
Devyn Cairns
a2758e6c40
Add IR support to the debugger (#13345)
# Description

This adds tracing for each individual instruction to the `Debugger`
trait. Register contents can be inspected both when entering and leaving
an instruction, and if an instruction produced an error, a reference to
the error is also available. It's not the full `EvalContext` but it's
most of the important parts for getting an idea of what's going on.

Added support for all of this to the `Profiler` / `debug profile` as
well, and the output is quite incredible - super verbose, but you can
see every instruction that's executed and also what the result was if
it's an instruction that has a clearly defined output (many do).

# User-Facing Changes

- Added `--instructions` to `debug profile`, which adds the `pc` and
`instruction` columns to the output.
- `--expr` only works in AST mode, and `--instructions` only works in IR
mode. In the wrong mode, the output for those columns is just blank.

# Tests + Formatting

All passing.

# After Submitting

- [ ] release notes
2024-07-13 01:58:21 -07:00
Devyn Cairns
d42cf55431
fix file_count in Debug implementation of IrBlock (#13367)
# Description

Oops.
2024-07-12 21:27:23 -05:00
Darren Schroeder
46b5e510ac
tweak parse usage and examples to be more clear (#13363)
# Description

This PR just tweaks the `parse` command's usage and examples to make it
clearer what's going on "under the hood".

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-07-12 09:48:27 -05:00
Devyn Cairns
02659b1c8a
Mention the actual output type on an OutputMismatch error (#13355)
# Description

This improves the error when the determined output of a custom command
doesn't match the specified output type by adding the actual determined
output type.

# User-Facing Changes

Previous: `command doesn't output {0}`

New: `expected {0}, but command outputs {1}`

# Tests + Formatting
Passing.

# After Submitting
- [ ] release notes? (minor change, but helpful)
2024-07-12 11:45:53 +02:00
Stefan Holderbach
8f981c1eb4
Use conventional generic bounds (#13360)
https://rust-lang.github.io/rust-clippy/master/index.html#/multiple_bound_locations
2024-07-12 17:13:07 +08:00
Devyn Cairns
0b8d0bcd7a
Fix order of I/O types in take until (#13356)
# Description
Just a quick one, but `List(Any)` has to come before `Table`, because
`List(Any)` is a valid match for `Table`, so it will choose `Table`
output even if the input was actually `List(Any)`. I ended up removing
`Table` because it's just not needed at all anyway.

Though, I'm not really totally sure this is correct - I think the parser
should probably actually just have some idea of what the more specific
type is, and choose the most specific type match, rather than just doing
it in order. I guess this will result in the output just always being
`List(Any)` for now. Still better than a bad typecheck error

# User-Facing Changes

Fixes the following contrived example:

```nushell
def foo []: nothing -> list<int> {
  seq 1 10 | # list<int>
    each { |n| $n * 20 } | # this causes the type to become list<any>
    take until { |x| $x < 10 } } # table is first, so now this is type table
    # ...but table is not compatible with list<int>
}
```

# After Submitting
- [ ] make typechecker type choice more robust
- [ ] release notes
2024-07-12 10:25:44 +02:00
Ian Manske
ee875bb8a3
Edit path form doc comments (#13358)
# Description
Fixes outdated/inaccurate doc comments for `PathForm`s in `nu_path`.
2024-07-12 10:23:51 +02:00
Ian Manske
d56457d63e
Path migration part 2: nu-test-support (#13329)
# Description
Part 2 of replacing `std::path` types with `nu_path` types added in
#13115. This PR targets `nu-test-support`.
2024-07-12 02:43:10 +00:00
Maxim Zhiburt
4bd87d0496
Fix unused space when truncation is used and header on border is configured (#13353)
Somehow this logic was missed on my end. ( I mean I was not even
thinking about it in original patch 😄 )

Please recheck

Added a regression test too.

close #13336

cc: @fdncred
2024-07-11 15:13:16 -05:00
Devyn Cairns
ccd0160c32
Make the store-env IR instruction also update config (#13351)
# Description

Follow up fix to #13332, so that changes to config when running under IR
actually happen as well. Since I merged them around the same time, I
forgot about this.
2024-07-11 10:49:46 -07:00
dependabot[bot]
acd4cb83e8
Bump ureq from 2.9.7 to 2.10.0 (#13348) 2024-07-11 14:08:42 +00:00
dependabot[bot]
a7a5315638
Bump crate-ci/typos from 1.23.1 to 1.23.2 (#13347) 2024-07-11 13:29:34 +00:00
Stefan Holderbach
9fec5883c0
Group dependabot bumps for uutils/coreutils (#13346)
Similar to #13084 for polars. This familiy of crates has shared
versioning and tends to at least depend on the same `uucore` crate.
2024-07-11 15:18:58 +02:00
Devyn Cairns
f65bc97a54
Update config directly at assignment (#13332)
# Description

Allows `Stack` to have a modified local `Config`, which is updated
immediately when `$env.config` is assigned to. This means that even
within a script, commands that come after `$env.config` changes will
always see those changes in `Stack::get_config()`.

Also fixed a lot of cases where `engine_state.get_config()` was used
even when `Stack` was available.

Closes #13324.

# User-Facing Changes
- Config changes apply immediately after the assignment is executed,
rather than whenever config is read by a command that needs it.
- Potentially slower performance when executing a lot of lines that
change `$env.config` one after another. Recommended to get `$env.config`
into a `mut` variable first and do modifications, then assign it back.
- Much faster performance when executing a script that made
modifications to `$env.config`, as the changes are only parsed once.

# Tests + Formatting
All passing.

# After Submitting
- [ ] release notes
2024-07-11 06:09:33 -07:00
Stefan Holderbach
deaa711ca6
Bump yanked libc version (#13344)
Fixes #13244
2024-07-11 14:20:28 +02:00
Stefan Holderbach
076a29ae19
Document public types in nu-protocol (#12906)
- **Doc-comment public `nu-protocol` modules**
- **Doccomment argument/signature/call stuff**
- **Doccomment cell path types**
- **Doccomment expression stuff**
- **Doccomment import patterns**
- **Doccomment pattern matching AST nodes**
2024-07-11 13:30:12 +02:00
Devyn Cairns
9de7f931c0
Add more argument types to view ir (#13343)
# Description

Add a few more options to `view ir` for finding blocks, which I found
myself wanting while trying to trace through the generated code.

If we end up adding support for plugins to call commands that are in
scope by name, this will also make it possible for
`nu_plugin_explore_ir` to just step through IR automatically (by passing
the block/decl ids) without exposing too many internals. With that I
could potentially add keys that allow you to step in to closures or
decls with the press of a button, just by calling `view ir --json`
appropriately.

# User-Facing Changes

- `view ir` can now take names of custom commands that are in scope.
- integer arguments are treated as block IDs, which sometimes show up in
IR (closure, block, row condition literals).
- `--decl-id` provided to treat the argument as a decl ID instead, which
is also sometimes necessary to access something that isn't in scope.
2024-07-11 06:05:06 -05:00
Devyn Cairns
d97512df8e
Fix the signature of view ir (#13342)
# Description

Fix `view ir` to use `Signature::build()` rather than `new()`, which is
required for `--help` to work. Also add `Category::Debug`, as that's
most appropriate.
2024-07-11 06:00:59 -05:00
Devyn Cairns
801cfae279
Avoid clone in Signature::get_positional() (#13338)
# Description
`Signature::get_positional()` was returning an owned `PositionalArg`,
which contains a bunch of strings. `ClosureEval` uses this in
`try_add_arg`, making all of that unnecessary cloning a little bit hot.

# User-Facing Changes
Slightly better performance
2024-07-11 02:14:05 +00:00
Devyn Cairns
f87cf895c2
Set the capacity of the Vec used in gather_captures() to the number of captures expected (#13339)
# Description

Just more efficient allocation during `Stack::gather_captures()` so that
we don't have to grow the `Vec` needlessly.

# User-Facing Changes
Slightly better performance.
2024-07-11 02:13:35 +00:00
Darren Schroeder
ac561b1b0e
quick fix up for ir pr as_refs (#13340)
# Description

Was having an issue compiling main after the IR pr. Talked to devyn and
he led me to change a couple things real quick and we're compiling once
again.
2024-07-11 09:19:06 +08:00
Devyn Cairns
1a5bf2447a
Use Arc for environment variables on the stack (#13333)
# Description

This is another easy performance lift that just changes `env_vars` and
`env_hidden` on `Stack` to use `Arc`. I noticed that these were being
cloned on essentially every closure invocation during captures
gathering, so we're paying the cost for all of that even when we don't
change anything. On top of that, for `env_vars`, there's actually an
entirely fresh `HashMap` created for each child scope, so it's highly
unlikely that we'll modify the parent ones.

Uses `Arc::make_mut` instead to take care of things when we need to
mutate something, and most of the time nothing has to be cloned at all.

# Benchmarks

The benefits are greater the more calls there are to env-cloning
functions like `captures_to_stack()`. Calling custom commands in a loop
is basically best case for a performance improvement. Plain `each` with
a literal block isn't so badly affected because the stack is set up
once.

## random_bytes.nu

```nushell
use std bench
do {
  const SCRIPT = ../nu_scripts/benchmarks/random-bytes.nu
  let before_change = bench { nu $SCRIPT }
  let after_change = bench { target/release/nu $SCRIPT }
  {
    before: ($before_change | reject times),
    after: ($after_change | reject times)
  }
}
```

```
╭────────┬──────────────────────────────╮
│        │ ╭──────┬───────────────────╮ │
│ before │ │ mean │ 603ms 759µs 727ns │ │
│        │ │ min  │ 593ms 298µs 167ns │ │
│        │ │ max  │ 648ms 612µs 291ns │ │
│        │ │ std  │ 9ms 335µs 251ns   │ │
│        │ ╰──────┴───────────────────╯ │
│        │ ╭──────┬───────────────────╮ │
│ after  │ │ mean │ 518ms 400µs 557ns │ │
│        │ │ min  │ 507ms 762µs 583ns │ │
│        │ │ max  │ 566ms 695µs 166ns │ │
│        │ │ std  │ 9ms 554µs 767ns   │ │
│        │ ╰──────┴───────────────────╯ │
╰────────┴──────────────────────────────╯
```

## gradient_benchmark_no_check.nu

```nushell
use std bench
do {
  const SCRIPT = ../nu_scripts/benchmarks/gradient_benchmark_no_check.nu
  let before_change = bench { nu $SCRIPT }
  let after_change = bench { target/release/nu $SCRIPT }
  {
    before: ($before_change | reject times),
    after: ($after_change | reject times)
  }
}
```

```
╭────────┬──────────────────────────────╮
│        │ ╭──────┬───────────────────╮ │
│ before │ │ mean │ 146ms 543µs 380ns │ │
│        │ │ min  │ 142ms 416µs 166ns │ │
│        │ │ max  │ 189ms 595µs       │ │
│        │ │ std  │ 7ms 140µs 342ns   │ │
│        │ ╰──────┴───────────────────╯ │
│        │ ╭──────┬───────────────────╮ │
│ after  │ │ mean │ 134ms 211µs 678ns │ │
│        │ │ min  │ 132ms 433µs 125ns │ │
│        │ │ max  │ 135ms 722µs 583ns │ │
│        │ │ std  │ 793µs 134ns       │ │
│        │ ╰──────┴───────────────────╯ │
╰────────┴──────────────────────────────╯
```

# User-Facing Changes
Better performance, particularly for custom commands, especially if
there are a lot of environment variables. Nothing else.

# Tests + Formatting
All passing.
2024-07-10 17:34:50 -07:00
Devyn Cairns
d7392f1f3b
Internal representation (IR) compiler and evaluator (#13330)
# Description

This PR adds an internal representation language to Nushell, offering an
alternative evaluator based on simple instructions, stream-containing
registers, and indexed control flow. The number of registers required is
determined statically at compile-time, and the fixed size required is
allocated upon entering the block.

Each instruction is associated with a span, which makes going backwards
from IR instructions to source code very easy.

Motivations for IR:

1. **Performance.** By simplifying the evaluation path and making it
more cache-friendly and branch predictor-friendly, code that does a lot
of computation in Nushell itself can be sped up a decent bit. Because
the IR is fairly easy to reason about, we can also implement
optimization passes in the future to eliminate and simplify code.
2. **Correctness.** The instructions mostly have very simple and
easily-specified behavior, so hopefully engine changes are a little bit
easier to reason about, and they can be specified in a more formal way
at some point. I have made an effort to document each of the
instructions in the docs for the enum itself in a reasonably specific
way. Some of the errors that would have happened during evaluation
before are now moved to the compilation step instead, because they don't
make sense to check during evaluation.
3. **As an intermediate target.** This is a good step for us to bring
the [`new-nu-parser`](https://github.com/nushell/new-nu-parser) in at
some point, as code generated from new AST can be directly compared to
code generated from old AST. If the IR code is functionally equivalent,
it will behave the exact same way.
4. **Debugging.** With a little bit more work, we can probably give
control over advancing the virtual machine that `IrBlock`s run on to
some sort of external driver, making things like breakpoints and single
stepping possible. Tools like `view ir` and [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir) make it easier than
before to see what exactly is going on with your Nushell code.

The goal is to eventually replace the AST evaluator entirely, once we're
sure it's working just as well. You can help dogfood this by running
Nushell with `$env.NU_USE_IR` set to some value. The environment
variable is checked when Nushell starts, so config runs with IR, or it
can also be set on a line at the REPL to change it dynamically. It is
also checked when running `do` in case within a script you want to just
run a specific piece of code with or without IR.

# Example

```nushell
view ir { |data|
  mut sum = 0
  for n in $data {
    $sum += $n
  }
  $sum
}
```
  
```gas
# 3 registers, 19 instructions, 0 bytes of data
   0: load-literal           %0, int(0)
   1: store-variable         var 904, %0 # let
   2: drain                  %0
   3: drop                   %0
   4: load-variable          %1, var 903
   5: iterate                %0, %1, end 15 # for, label(1), from(14:)
   6: store-variable         var 905, %0
   7: load-variable          %0, var 904
   8: load-variable          %2, var 905
   9: binary-op              %0, Math(Plus), %2
  10: span                   %0
  11: store-variable         var 904, %0
  12: load-literal           %0, nothing
  13: drain                  %0
  14: jump                   5
  15: drop                   %0          # label(0), from(5:)
  16: drain                  %0
  17: load-variable          %0, var 904
  18: return                 %0
```

# Benchmarks

All benchmarks run on a base model Mac Mini M1.

## Iterative Fibonacci sequence

This is about as best case as possible, making use of the much faster
control flow. Most code will not experience a speed improvement nearly
this large.

```nushell
def fib [n: int] {
  mut a = 0
  mut b = 1
  for _ in 2..=$n {
    let c = $a + $b
    $a = $b
    $b = $c
  }
  $b
}
use std bench
bench { 0..50 | each { |n| fib $n } }
```

IR disabled:

```
╭───────┬─────────────────╮
│ mean  │ 1ms 924µs 665ns │
│ min   │ 1ms 700µs 83ns  │
│ max   │ 3ms 450µs 125ns │
│ std   │ 395µs 759ns     │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```

IR enabled:

```
╭───────┬─────────────────╮
│ mean  │ 452µs 820ns     │
│ min   │ 427µs 417ns     │
│ max   │ 540µs 167ns     │
│ std   │ 17µs 158ns      │
│ times │ [list 50 items] │
╰───────┴─────────────────╯
```

![explore ir
view](https://github.com/nushell/nushell/assets/10729/d7bccc03-5222-461c-9200-0dce71b83b83)

##
[gradient_benchmark_no_check.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/gradient_benchmark_no_check.nu)

IR disabled:

```
╭───┬──────────────────╮
│ 0 │ 27ms 929µs 958ns │
│ 1 │ 21ms 153µs 459ns │
│ 2 │ 18ms 639µs 666ns │
│ 3 │ 19ms 554µs 583ns │
│ 4 │ 13ms 383µs 375ns │
│ 5 │ 11ms 328µs 208ns │
│ 6 │  5ms 659µs 542ns │
╰───┴──────────────────╯
```

IR enabled:

```
╭───┬──────────────────╮
│ 0 │       22ms 662µs │
│ 1 │ 17ms 221µs 792ns │
│ 2 │ 14ms 786µs 708ns │
│ 3 │ 13ms 876µs 834ns │
│ 4 │  13ms 52µs 875ns │
│ 5 │ 11ms 269µs 666ns │
│ 6 │  6ms 942µs 500ns │
╰───┴──────────────────╯
```

##
[random-bytes.nu](https://github.com/nushell/nu_scripts/blob/main/benchmarks/random-bytes.nu)

I got pretty random results out of this benchmark so I decided not to
include it. Not clear why.

# User-Facing Changes
- IR compilation errors may appear even if the user isn't evaluating
with IR.
- IR evaluation can be enabled by setting the `NU_USE_IR` environment
variable to any value.
- New command `view ir` pretty-prints the IR for a block, and `view ir
--json` can be piped into an external tool like [`explore
ir`](https://github.com/devyn/nu_plugin_explore_ir).

# Tests + Formatting
All tests are passing with `NU_USE_IR=1`, and I've added some more eval
tests to compare the results for some very core operations. I will
probably want to add some more so we don't have to always check
`NU_USE_IR=1 toolkit test --workspace` on a regular basis.

# After Submitting
- [ ] release notes
- [ ] further documentation of instructions?
- [ ] post-release: publish `nu_plugin_explore_ir`
2024-07-10 17:33:59 -07:00
Devyn Cairns
ea8c4e3af2
Make pipe redirections consistent, add err>| etc. forms (#13334)
# Description

Fixes the lexer to recognize `out>|`, `err>|`, `out+err>|`, etc.

Previously only the short-style forms were recognized, which was
inconsistent with normal file redirections.

I also integrated it all more into the normal lex path by checking `|`
in a special way, which should be more performant and consistent, and
cleans up the code a bunch.

Closes #13331.

# User-Facing Changes
- Adds `out>|` (error), `err>|`, `out+err>|`, `err+out>|` as recognized
forms of the pipe redirection.

# Tests + Formatting
All passing. Added tests for the new forms.

# After Submitting
- [ ] release notes
2024-07-11 07:16:22 +08:00
132ikl
616e9faaf1
Fix main binary being rebuilt when nothing has changed (#13337)
# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.

Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->

The build script is currently re-run on each `cargo build` even when it
has not changed. The `rerun-if-changed` line points to `/build.rs`, but
`build.rs` is actually located at `/scripts/build.rs`. This updates that
path.

# User-Facing Changes
N/A

# Tests + Formatting
N/A
2024-07-10 18:05:24 -05:00
Jack Wright
b68c7cf3fa
Make polars unpivot consistent with polars pivot (#13335)
# Description
Makes `polars unpivot` use the same arguments as `polars pivot` and
makes it consistent with the polars' rust api. Additionally, support for
the polar's streaming engine has been exposed on eager dataframes.
Previously, it would only work with lazy dataframes.


# User-Facing Changes
* `polars unpivot` argument `--columns`|`-c` has been renamed to
`--index`|`-i`
* `polars unpivot` argument `--values`|`-v` has been renamed to
`--on`|`-o`
* `polars unpivot` short argument for `--streamable` is now `-t` to make
it consistent with `polars pivot`. It was made `-t` for `polars pivot`
because `-s` is short for `--short`
2024-07-10 16:36:38 -05:00
dependabot[bot]
0178295363
Bump crate-ci/typos from 1.22.9 to 1.23.1 (#13328)
Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.22.9 to
1.23.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/crate-ci/typos/releases">crate-ci/typos's
releases</a>.</em></p>
<blockquote>
<h2>v1.23.1</h2>
<h2>[1.23.1] - 2024-07-05</h2>
<h3>Fixes</h3>
<ul>
<li>Add missing <a
href="https://redirect.github.com/crate-ci/typos/issues/1024">June
2024</a> changes</li>
</ul>
<h2>v1.23.0</h2>
<h2>[1.23.0] - 2024-07-05</h2>
<h3>Fixes</h3>
<ul>
<li>Updated the dictionary with the <a
href="https://redirect.github.com/crate-ci/typos/issues/1024">June
2024</a> changes</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/crate-ci/typos/blob/master/CHANGELOG.md">crate-ci/typos's
changelog</a>.</em></p>
<blockquote>
<h2>[1.23.1] - 2024-07-05</h2>
<h3>Fixes</h3>
<ul>
<li>Add missing <a
href="https://redirect.github.com/crate-ci/typos/issues/1024">June
2024</a> changes</li>
</ul>
<h2>[1.23.0] - 2024-07-05</h2>
<h3>Fixes</h3>
<ul>
<li>Updated the dictionary with the <a
href="https://redirect.github.com/crate-ci/typos/issues/1024">June
2024</a> changes</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="81a34f1ca2"><code>81a34f1</code></a>
chore: Release</li>
<li><a
href="1aa7c985e4"><code>1aa7c98</code></a>
docs: Update changelog</li>
<li><a
href="4d4121ea86"><code>4d4121e</code></a>
chore: Release</li>
<li><a
href="4edcc6aa95"><code>4edcc6a</code></a>
Merge pull request <a
href="https://redirect.github.com/crate-ci/typos/issues/1053">#1053</a>
from epage/june</li>
<li><a
href="fa7786ec69"><code>fa7786e</code></a>
fix(dict): Add more june typos</li>
<li><a
href="04eea79695"><code>04eea79</code></a>
chore: Release</li>
<li><a
href="d3b2a6eb90"><code>d3b2a6e</code></a>
docs: Update changelog</li>
<li><a
href="494a98f93e"><code>494a98f</code></a>
chore: Release</li>
<li><a
href="bdc571d921"><code>bdc571d</code></a>
Merge pull request <a
href="https://redirect.github.com/crate-ci/typos/issues/1052">#1052</a>
from epage/june</li>
<li><a
href="eac884cf3b"><code>eac884c</code></a>
fix(dict): June updates</li>
<li>Additional commits viewable in <a
href="https://github.com/crate-ci/typos/compare/v1.22.9...v1.23.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=crate-ci/typos&package-manager=github_actions&previous-version=1.22.9&new-version=1.23.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
Dependabot will merge this PR once CI passes on it, as requested by
@fdncred.

[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-09 19:53:46 -05:00
Darren Schroeder
ad8054ebed
update table comments 2024-07-09 19:52:57 -05:00
Jack Wright
ff27d6a18e
Implemented a command to expose polar's pivot functionality (#13282)
# Description
Implementing pivot support 

The example below is a port of the [python API
example](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.pivot.html)

<img width="1079" alt="Screenshot 2024-07-01 at 14 29 27"
src="https://github.com/nushell/nushell/assets/56345/277eb7a2-233b-4070-9d24-c2183805c1b8">

# User-Facing Changes
* Introduction of the `polars pivot` command
2024-07-09 10:17:20 -07:00
Maxim Zhiburt
4cdceca1f7
Fix kv table width issue with header_on_border configuration (#13325)
GOOD CATCH.............................................................
SORRY

I've added a test to catch regression just in case.

close #13319

cc: @fdncred
2024-07-09 09:49:04 -05:00
Wind
1964dacaef
Raise error when using o>| pipe (#13323)
# Description
From the feedbacks from @amtoine , it's good to make nushell shows error
for `o>|` syntax.

# User-Facing Changes
## Before
```nushell
'foo' o>| print                                                                                                                                                                                                                     07/09/2024 06:44:23 AM
Error: nu::parser::parse_mismatch

  × Parse mismatch during operation.
   ╭─[entry #6:1:9]
 1 │ 'foo' o>| print
   ·         ┬
   ·         ╰── expected redirection target
```

## After
```nushell
'foo' o>| print                                                                                                                                                                                                                     07/09/2024 06:47:26 AM
Error: nu::parser::parse_mismatch

  × Parse mismatch during operation.
   ╭─[entry #1:1:7]
 1 │ 'foo' o>| print
   ·       ─┬─
   ·        ╰── expected `|`.  Redirection stdout to pipe is the same as piping directly.
   ╰────
```

# Tests + Formatting
Added one test

---------

Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
2024-07-09 07:11:25 -05:00
Ian Manske
e98b2ceb8c
Path migration 1 (#13309)
# Description
Part 1 of replacing `std::path` types with `nu_path` types added in
#13115.
2024-07-09 17:25:23 +08:00
Ian Manske
399a7c8836
Add and use new Signals struct (#13314)
# Description
This PR introduces a new `Signals` struct to replace our adhoc passing
around of `ctrlc: Option<Arc<AtomicBool>>`. Doing so has a few benefits:
- We can better enforce when/where resetting or triggering an interrupt
is allowed.
- Consolidates `nu_utils::ctrl_c::was_pressed` and other ad-hoc
re-implementations into a single place: `Signals::check`.
- This allows us to add other types of signals later if we want. E.g.,
exiting or suspension.
- Similarly, we can more easily change the underlying implementation if
we need to in the future.
- Places that used to have a `ctrlc` of `None` now use
`Signals::empty()`, so we can double check these usages for correctness
in the future.
2024-07-07 22:29:01 +00:00
Justin Ma
c6b6b1b7a8
Upgrade Ubuntu runners to 22.04 to fix nightly build errors, fix #13255 (#13273)
# Description
Upgrade Ubuntu runners to 22.04 to fix nightly build errors related to
`GLIBC` versions like:

https://github.com/nushell/nushell/actions/runs/9727979044/job/26848189346
Should close #13255

BTW: The [workflow of
`nushell/nightly`](6faf3c3aed/.github/workflows/nightly-build.yml (L108))
has been upgraded a long time ago

# User-Facing Changes

Older linux system users can download
`*-x86_64-unknown-linux-musl.tar.gz` binaries if any `GLIBC` error found
while starting `nu` binary
2024-07-07 22:15:03 +00:00
YizhePKU
152fb5be39
Fix PWD-aware command hints (#13024)
This PR fixes PWD-aware command hints by sending PWD to the Reedline
state in every REPL loop. This PR should be merged along with
https://github.com/nushell/reedline/pull/796.

Fixes https://github.com/nushell/nushell/issues/12951.
2024-07-07 11:43:22 -05:00
Reilly Wood
83081f9852
explore: pass config to views at creation time (#13312)
cc: @zhiburt

This is an internal refactoring for `explore`.

Previously, views inside `explore` were created with default/incorrect
configuration and then the correct configuration was passed to them
using a function called `setup()`. I believe this was because
configuration was dynamic and could change while `explore` was running.

After https://github.com/nushell/nushell/pull/10259, configuration can
no longer be changed on the fly. So we can clean this up by removing
`setup()` and passing configuration to views when they are created.
2024-07-07 08:09:59 -05:00
Devyn Cairns
6ce5530fc2
Make into bits produce bitstring stream (#13310)
# Description

Fix `into bits` to have consistent behavior when passed a byte stream.

# User-Facing Changes

Previously, it was returning a binary on stream, even though its
input/output types don't describe this possibility. We don't need this
since we have `into binary` anyway.

# Tests + Formatting
Tests added
2024-07-07 08:00:57 -05:00
goldfish
5af8d62666
Fix from toml to handle toml datetime correctly (#13315)
# Description

fixed #12699

When bare dates or naive times are specified in toml files, `from toml`
returns invalid dates or times.
This PR fixes the problem to correctly handle toml datetime.

The current version command returns the default datetime
(`chrono::DateTime::default()`) if the datetime parse fails. However, I
felt that this behavior was a bit unfriendly, so I changed it to return
`Value::string`.

# User-Facing Changes

The command returns a date with default time and timezone if a bare date
is specified.

```
~/Development/nushell> "dob = 2023-05-27" | from toml
╭─────┬────────────╮
│ dob │ a year ago │
╰─────┴────────────╯
~/Development/nushell> "dob = 2023-05-27" | from toml |
Sat, 27 May 2023 00:00:00 +0000 (a year ago)
~/Development/nushell>                            
```

If a bare time is given, a time string is returned.

```
~/Development/nushell> "tm = 11:00:00" | from toml
╭────┬──────────╮
│ tm │ 11:00:00 │
╰────┴──────────╯
~/Development/nushell> "tm = 11:00:00" | from toml | get tm
11:00:00
~/Development/nushell>  
```

# Tests + Formatting

When I ran tests, `commands::touch::change_file_mtime_to_reference`
failed with the following error.
The error also occurs in the master branch, so it's probably unrelated
to these changes.
(maybe a problem with my dev environment)

```
$ ~/Development/nushell> toolkit check pr

~~~~~~~~

test usage_start_uppercase ... ok
test format_conversions::yaml::convert_dict_to_yaml_with_integer_floats_key ... ok
test format_conversions::yaml::convert_dict_to_yaml_with_boolean_key ... ok
test format_conversions::yaml::table_to_yaml_text_and_from_yaml_text_back_into_table ... ok
test quickcheck_parse ... ok
test format_conversions::yaml::convert_dict_to_yaml_with_integer_key ... ok

failures:

---- commands::touch::change_file_mtime_to_reference stdout ----
=== stderr

thread 'commands::touch::change_file_mtime_to_reference' panicked at crates/nu-command/tests/commands/touch.rs:298:9:
assertion `left == right` failed
  left: SystemTime { tv_sec: 1720344745, tv_nsec: 862392750 }
 right: SystemTime { tv_sec: 1720344745, tv_nsec: 887670417 }


failures:
    commands::touch::change_file_mtime_to_reference

test result: FAILED. 1542 passed; 1 failed; 32 ignored; 0 measured; 0 filtered out; finished in 12.04s

error: test failed, to rerun pass `-p nu-command --test main`
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🔴 `toolkit test`
-  `toolkit test stdlib`

~/Development/nushell> toolkit test stdlib
   Compiling nu v0.95.1 (/Users/hiroki/Development/nushell)
   Compiling nu-cmd-lang v0.95.1 (/Users/hiroki/Development/nushell/crates/nu-cmd-lang)
    Finished dev [unoptimized + debuginfo] target(s) in 6.64s
     Running `target/debug/nu --no-config-file -c '
        use crates/nu-std/testing.nu
        testing run-tests --path crates/nu-std
    '`
2024-07-07T19:00:20.423|INF|Running from_jsonl_invalid_object in module test_formats
2024-07-07T19:00:20.436|INF|Running env_log-prefix in module test_logger_env

~~~~~~~~~~~

2024-07-07T19:00:22.196|INF|Running debug_short in module test_basic_commands
~/Development/nushell> 
```

# After Submitting

nothing
2024-07-07 07:55:06 -05:00
Maxim Zhiburt
32db5d3aa3
Fix issue with head on separation lines (#13291)
Hi there,

Seems to work

Though I haven't done a lot of testing.


![image](https://github.com/nushell/nushell/assets/20165848/c95aa8d4-a8d2-462c-afc9-35c48f8825f4)

![image](https://github.com/nushell/nushell/assets/20165848/1859dfe5-4a76-4776-a4e0-d3f53fc86862)

![image](https://github.com/nushell/nushell/assets/20165848/b46bb62b-a951-412d-b8fa-65cebcfbfed6)

![image](https://github.com/nushell/nushell/assets/20165848/bff0762e-42d4-41bf-b2c2-641c0436ca2e)

![image](https://github.com/nushell/nushell/assets/20165848/2c3c5664-9b90-44e4-befc-c250174cb630)


close #13287
cc: @fdncred 

PS: Yessssss I do remember about emojie issue..... 😞
2024-07-06 14:47:39 -05:00
Ian Manske
fa183b6669
help operators refactor (#13307)
# Description
Refactors `help operators` so that its output is always up to date with
the parser.

# User-Facing Changes
- The order of output rows for `help operators` was changed.
- `not` is now listed as a boolean operator instead of a comparison
operator.
- Edited some of the descriptions for the operators.
2024-07-06 13:09:12 -05:00
Darren Schroeder
d2a1f96dbd
update to latest reedline commit (#13313)
# Description

Update to the latest reedline version.

(don't ask me why libloading changed. `cargo update -p reedline`
sometimes does weird things)

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
/cc @YizhePKU (this include your reedline pwd change)
2024-07-06 12:04:19 -05:00
Yash Thakur
de2b752771
Fix variable completion sort order (#13306)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx

you can also mention related issues, PRs or discussions!
-->

# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.

Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->

My last PR (https://github.com/nushell/nushell/pull/13242) made it so
that the last branch in the variable completer doesn't sort suggestions.
Sorry about that. This should fix it.

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

Variables will now be sorted properly.

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use toolkit.nu; toolkit test stdlib"` to run the
tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

Added one test case to verify this won't happen again.

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-07-05 17:58:35 -05:00