Commit Graph

5678 Commits

Author SHA1 Message Date
YizhePKU
7a86b98f61
Migrate to a new PWD API (part 2) (#12749)
Refer to #12603 for part 1.

We need to be careful when migrating to the new API, because the new API
has slightly different semantics (PWD can contain symlinks). This PR
handles the "obviously safe" part of the migrations. Namely, it handles
two specific use cases:

* Passing PWD into `canonicalize_with()`
* Passing PWD into `EngineState::merge_env()`

The first case is safe because symlinks are canonicalized away. The
second case is safe because `EngineState::merge_env()` only uses PWD to
call `std::env::set_current_dir()`, which shouldn't affact Nushell. The
commit message contains detailed stats on the updated files.

Because these migrations touch a lot of files, I want to keep these PRs
small to avoid merge conflicts.
2024-05-07 18:17:49 +03:00
Ian Manske
b9331d1b08
Add sys users command (#12787)
# Description
Add a new `sys users` command which returns a table of the users of the
system. This is the same table that is currently present as
`(sys).host.sessions`. The same table has been removed from the recently
added `sys host` command.

# User-Facing Changes
Adds a new command. (The old `sys` command is left as is.)
2024-05-07 07:52:02 -05:00
Ian Manske
c54d223ea0
Fix list spread syntax highlighting (#12793)
# Description
I broke syntax highlighting for list spreads in #12529. This should fix
#12792 😅. I just copied the code for highlighting record
spreads.
2024-05-07 13:41:47 +08:00
Ian Manske
eccc558a4e
describe refactor (#12770)
# Description

Refactors `describe` a bit. Namely, I added a `Description` enum to get
rid of `compact_primitive_description` and its awkward `Value` pattern
matching.
2024-05-06 23:20:46 +00:00
Ian Manske
1038c64f80
Add sys subcommands (#12747)
# Description
Adds subcommands to `sys` corresponding to each column of the record
returned by `sys`. This is to alleviate the fact that `sys` now returns
a regular record, meaning that it must compute every column which might
take a noticeable amount of time. The subcommands, on the other hand,
only need to compute and return a subset of the data which should be
much faster. In fact, it should be as fast as before, since this is how
the lazy record worked (it would compute only each column as necessary).

I choose to add subcommands instead of having an optional cell-path
parameter on `sys`, since the cell-path parameter would:
- increase the code complexity (can access any value at any row or
nested column)
- prevents discovery with tab-completion
- hinders type checking and allows users to pass potentially invalid
columns

# User-Facing Changes
Deprecates `sys` in favor of the new `sys` subcommands.
2024-05-06 23:20:27 +00:00
Jack Wright
68adc4657f
Polars lazy refactor (#12669)
This moves to predominantly supporting only lazy dataframes for most
operations. It removes a lot of the type conversion between lazy and
eager dataframes based on what was inputted into the command.

For the most part the changes will mean:
* You will need to run `polars collect` after performing operations
* The into-lazy command has been removed as it is redundant.
* When opening files a lazy frame will be outputted by default if the
reader supports lazy frames

A list of individual command changes can be found
[here](https://hackmd.io/@nucore/Bk-3V-hW0)

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-06 23:19:11 +00:00
Wind
97fc190cc5
allow raw string to be used inside subexpression, list, and closure (#12776)
# Description
Fixes: #12744

This pr is moving raw string lex logic into `lex_item` function, so we
can use raw string inside subexpression, list, closure.
```nushell
> [r#'abc'#]
╭───┬─────╮
│ 0 │ abc │
╰───┴─────╯
> (r#'abc'#)
abc
> do {r#'aa'#}
aa
```

# Tests + Formatting
Done

# After Submitting
NaN
2024-05-06 15:53:58 -05:00
Wind
460a1c8f87
Allow ls works inside dir with [] brackets (#12625)
# Description
Fixes: #12429

To fix the issue, we need to pass the `input pattern` itself to
`glob_from` function, but currently on latest main, nushell pass
`expanded path of input pattern` to `glob_from` function.
It causes globbing failed if expanded path includes `[]` brackets.

It's a pity that I have to duplicate `nu_engine::glob_from` function
into `ls`, because `ls` might convert from `NuGlob::NotExpand` to
`NuGlob::Expand`, in that case, `nu_engine::glob_from` won't work if
user want to ls for a directory which includes tilde:
```
mkdir "~abc"
ls "~abc"
```
So I need to duplicate `glob_from` function and pass original
`expand_tilde` information.

# User-Facing Changes
Nan

# Tests + Formatting
Done

# After Submitting
Nan
2024-05-06 14:01:32 +08:00
Ian Manske
e879d4ecaf
ListStream touchup (#12524)
# Description

Does some misc changes to `ListStream`:
- Moves it into its own module/file separate from `RawStream`.
- `ListStream`s now have an associated `Span`.
- This required changes to `ListStreamInfo` in `nu-plugin`. Note sure if
this is a breaking change for the plugin protocol.
- Hides the internals of `ListStream` but also adds a few more methods.
- This includes two functions to more easily alter a stream (these take
a `ListStream` and return a `ListStream` instead of having to go through
the whole `into_pipeline_data(..)` route).
  -  `map`: takes a `FnMut(Value) -> Value`
  - `modify`: takes a function to modify the inner stream.
2024-05-05 16:00:59 +00:00
Antoine Stevan
ce3bc470ba
improve NUON documentation (#12717)
# Description
this PR
- moves the documentation from `lib.rs` to `README.md` while still
including it in the lib file, so that both the [crates.io
page](https://crates.io/crates/nuon) and the
[documentation](https://docs.rs/nuon/latest/nuon/) show the top-level
doc
- mention that comments are allowed in NUON
- add a JSON-NUON example
- put back the formatting of NOTE blocks in the doc

# User-Facing Changes

# Tests + Formatting

# After Submitting
2024-05-05 15:34:22 +02:00
Ian Manske
2f8e397365
Refactor flattening to reduce intermediate allocations (#12756)
# Description
Our current flattening code creates a bunch of intermediate `Vec`s for
each function call. These intermediate `Vec`s are then usually appended
to the current `output` `Vec`. By instead passing a mutable reference of
the `output` `Vec` to each flattening function, this `Vec` can be
reused/appended to directly thereby eliminating the need for
intermediate `Vec`s in most cases.
2024-05-05 10:43:20 +02:00
Viktor Szépe
8eefb7313e
Minimize future false positive typos (#12751)
# Description

Make typos config more strict: ignore false positives where they occur.

1. Ignore only files with typos
2. Add regexp-s with context
3. Ignore variable names only in Rust code
4. Ignore only 1 "identifier"
5. Check dot files

🎁 Extra bonus: fix typos!!
2024-05-04 15:00:44 +00:00
Maxime Jacob
3ae6fe2114
Enable columns with spaces for into_sqlite by adding quotes to column names (#12759)
# Description
Spaces were causing an issue with into_sqlite when they appeared in
column names.

This is because the column names were not properly wrapped with
backticks that allow sqlite to properly interpret the column.

The issue has been addressed by adding backticks to the column names of
into sqlite. The output of the column names when using open is
unchanged, and the column names appear without backticks as expected.

fixes #12700 

# User-Facing Changes
N/A

# Tests + Formatting
Formatting has been respected.

Repro steps from the issue have been done, and ran multiple times. New
values get added to the correct columns as expected.
2024-05-04 08:12:44 -05:00
Ian Manske
1e71cd4777
Bump base64 to 0.22.1 (#12757)
# Description
Bumps `base64` to 0.22.1 which fixes the alphabet used for binhex
encoding and decoding. This required updating some test expected output.

Related to PR #12469 where `base64` was also bumped and ran into the
failing tests.

# User-Facing Changes
Bug fix, but still changes binhex encoding and decoding output.

# Tests + Formatting
Updated test expected output.
2024-05-04 15:56:16 +03:00
YizhePKU
0d6fbdde4a
Fix PWD cannot point to root paths (#12761)
PR https://github.com/nushell/nushell/pull/12603 made it so that PWD can
never contain a trailing slash. However, a root path (such as `/` or
`C:\`) technically counts as "having a trailing slash", so now `cd /`
doesn't work.

I feel dumb for missing such an obvious edge case. Let's just merge this
quickly before anyone else finds out...

EDIT: It appears I'm too late.
2024-05-04 13:05:54 +03:00
Devyn Cairns
709b2479d9
Fix trailing slash in PWD set by cd (#12760)
# Description

Fixes #12758.

#12662 introduced a bug where calling `cd` with a path with a trailing
slash would cause `PWD` to be set to a path including a trailing slash,
which is not allowed. This adds a helper to `nu_path` to remove this,
and uses it in the `cd` command to clean it up before setting `PWD`.

# Tests + Formatting
I added some tests to make sure we don't regress on this in the future.

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-05-04 12:38:37 +03:00
Himadri Bhattacharjee
35a0f7a369
fix: prevent relative directory traversal from crashing (#12438)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx

you can also mention related issues, PRs or discussions!
-->

- fixes #11922
- fixes #12203

# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.

Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->
This is a rewrite for some parts of the recursive completion system. The
Rust `std::path` structures often ignores things like a trailing `.`
because for a complete path, it implies the current directory. We are
replacing the use of some of these structs for Strings.

A side effect is the slashes being normalized in Windows. For example if
we were to type `foo/bar/b`, it would complete it to `foo\bar\baz`
because a backward slash is the main separator in windows.

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

Relative paths are preserved. `..`s in the paths won't eagerly show
completions from the parent path. For example, `asd/foo/../b` will now
complete to `asd/foo/../bar` instead of `asd/bar`.

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-05-03 20:17:50 -05:00
Maxim Uvarov
a1287f7b3f
add more tests to the polars plugin (#12719)
# Description

I added some more tests to our mighty `polars` ~~, yet I don't know how
to add expected results in some of them. I would like to ask for help.~~

~~My experiments are in the last commit: [polars:
experiments](f7e5e72019).
Without those experiments `cargo test` goes well.~~
 
UPD. I moved out my unsuccessful test experiments into a separate
[branch](https://github.com/maxim-uvarov/nushell/blob/polars-tests-broken2/).
So, this branch seems ready for a merge.

@ayax79, maybe you'll find time for me please? It's not urgent for sure.

P.S. I'm very new to git. Please feel free to give me any suggestions on
how I should use it better
2024-05-03 20:14:55 -05:00
Stefan Holderbach
406df7f208
Avoid taking unnecessary ownership of intermediates (#12740)
# Description

Judiciously try to avoid allocations/clone by changing the signature of
functions

- **Don't pass str by value unnecessarily if only read**
- **Don't require a vec in `Sandbox::with_files`**
- **Remove unnecessary string clone**
- **Fixup unnecessary borrow**
- **Use `&str` in shape color instead**
- **Vec -> Slice**
- **Elide string clone**
- **Elide `Path` clone**
- **Take &str to elide clone in tests**

# User-Facing Changes
None

# Tests + Formatting
This touches many tests purely in changing from owned to borrowed/static
data
2024-05-04 00:53:15 +00:00
Ian Manske
e6f473695c
Fix typo (#12752) 2024-05-03 16:14:13 -05:00
Raphael Gaschignard
eff7f33086
Report errors that occur on file operations in ls (#12033)
Currently errors just create empty entries inside of resulting
dataframes.

This changeset is meant to help debug #12004, though generally speaking
I do think it's worth having ways to make errors be visible in this kind
of pipeline be visible

An example of what this looks like

<img width="954" alt="image"
src="https://github.com/nushell/nushell/assets/1408472/2c3c9167-2aaf-4f87-bab5-e8302d7a1170">
2024-05-03 10:12:43 -05:00
YizhePKU
bdb6daa4b5
Migrate to a new PWD API (#12603)
This is the first PR towards migrating to a new `$env.PWD` API that
returns potentially un-canonicalized paths. Refer to PR #12515 for
motivations.

## New API: `EngineState::cwd()`

The goal of the new API is to cover both parse-time and runtime use
case, and avoid unintentional misuse. It takes an `Option<Stack>` as
argument, which if supplied, will search for `$env.PWD` on the stack in
additional to the engine state. I think with this design, there's less
confusion over parse-time and runtime environments. If you have access
to a stack, just supply it; otherwise supply `None`.

## Deprecation of other PWD-related APIs

Other APIs are re-implemented using `EngineState::cwd()` and properly
documented. They're marked deprecated, but their behavior is unchanged.
Unused APIs are deleted, and code that accesses `$env.PWD` directly
without using an API is rewritten.

Deprecated APIs:

* `EngineState::current_work_dir()`
* `StateWorkingSet::get_cwd()`
* `env::current_dir()`
* `env::current_dir_str()`
* `env::current_dir_const()`
* `env::current_dir_str_const()`

Other changes:

* `EngineState::get_cwd()` (deleted)
* `StateWorkingSet::list_env()` (deleted)
* `repl::do_run_cmd()` (rewritten with `env::current_dir_str()`)

## `cd` and `pwd` now use logical paths by default

This pulls the changes from PR #12515. It's currently somewhat broken
because using non-canonicalized paths exposed a bug in our path
normalization logic (Issue #12602). Once that is fixed, this should
work.

## Future plans

This PR needs some tests. Which test helpers should I use, and where
should I put those tests?

I noticed that unquoted paths are expanded within `eval_filepath()` and
`eval_directory()` before they even reach the `cd` command. This means
every paths is expanded twice. Is this intended?

Once this PR lands, the plan is to review all usages of the deprecated
APIs and migrate them to `EngineState::cwd()`. In the meantime, these
usages are annotated with `#[allow(deprecated)]` to avoid breaking CI.

---------

Co-authored-by: Jakub Žádník <kubouch@gmail.com>
2024-05-03 14:33:09 +03:00
Ian Manske
f32ecc641f
Remove some macros (#12742)
# Description
Replaces some macros with regular functions or other code.
2024-05-03 10:35:37 +02:00
Devyn Cairns
72f3942c37
Upgrade to interprocess 2.0.0 (#12729)
# Description

This fixes #12724. NetBSD confirmed to work with this change.

The update also behaves a bit better in some ways - it automatically
unlinks and reclaims sockets on Unix, and doesn't try to flush/sync the
socket on Windows, so I was able to remove that platform-specific logic.

They also have a way to split the socket so I could just use one socket
now, but I haven't tried to do that yet. That would be more of a
breaking change but I think it's more straightforward.

# User-Facing Changes

- Hopefully more platforms work

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-05-02 22:31:33 -07:00
Antoine Stevan
bc6d934fa1
add support for cell-paths to NUON (#12718)
# Description
_cell paths_ can be easily serialized back and forth to NUON with the
leading `$.` syntax.

# User-Facing Changes
```nushell
$.foo.bar.0 | to nuon
```
and
```nushell
"$.foo.bar.0" | from nuon
```
are now possible

# Tests + Formatting
a new `cell_path` test has been added to `nuon`

# After Submitting
2024-05-03 09:25:19 +08:00
Stefan Holderbach
944ebac1c2
Eliminate dead code in nu-explore (#12735)
# Description
Nightly clippy found some unused fields leading me down a rabbit hole of
dead code hidden behind `pub`

Generally removing any already dead code or premature configurability
that is not exposed to the user.

# User-Facing Changes

None in effect.

Removed some options from the `$env.config.explore.hex-dump` record that
were only read into a struct but never used and also not validated.
2024-05-03 08:36:58 +08:00
Ian Manske
847646e44e
Remove lazy records (#12682)
# Description
Removes lazy records from the language, following from the reasons
outlined in #12622. Namely, this should make semantics more clear and
will eliminate concerns regarding maintainability.

# User-Facing Changes
- Breaking change: `lazy make` is removed.
- Breaking change: `describe --collect-lazyrecords` flag is removed.
- `sys` and `debug info` now return regular records.

# After Submitting
- Update nushell book if necessary.
- Explore new `sys` and `debug info` APIs to prevent them from taking
too long (e.g., subcommands or taking an optional column/cell-path
argument).
2024-05-03 08:36:10 +08:00
Devyn Cairns
ad6deadf24
Flush on every plugin Data message (#12728)
# Description

This helps to ensure data produced on a stream is immediately available
to the consumer of the stream. The BufWriter introduced for performance
reasons in 0.93 exposed the behavior that data messages wouldn't make it
to the other side until they filled the buffer in @cablehead's
[`nu_plugin_from_sse`](https://github.com/cablehead/nu_plugin_from_sse).

I had originally not flushed on every `Data` message because I figured
that it isn't really critical that the other side sees those messages
immediately, since they're not used for control and they are flushed
when waiting for acknowledgement or when the buffer is too full anyway.

Increasing the amount of data that can be sent with a single underlying
write increases performance, but this interferes with some plugins that
want to use streams in a more real-time way. In the future I would like
to make this configurable, maybe even per-command, so that a command can
decide what the priority is. But for now I think this is reasonable.

In the worst case, this decreases performance by about 40%, when sending
very small values (just numbers). But for larger values, this PR
actually increases performance by about 20%, because I've increased the
buffer size about 2x to 16,384 bytes. The previous value of 8,192 bytes
was too small to fit a full buffer coming from an external command, so
doubling it makes sense, and now a write of a buffer from an external
command can be done in exactly one write call, which I think makes
sense. I'm doing this at the same time because flushing each data
message would make it very likely that each individual data message from
an external stream would require exactly two writes rather than
approximately one (amortized).

Again, hopefully the tradeoff isn't too bad, and if it is I'll just make
it configurable.

# User-Facing Changes

- Performance of plugin streams will be a bit different
- Plugins that expect to send streams in real-time will work again

# Tests + Formatting

- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-05-02 23:51:16 +00:00
Stefan Holderbach
be6137d136
Fix clippy::wrong_self_convention in polars plugin (#12737)
Expected `into_` for `fn(self) -> T`
2024-05-02 19:31:51 +02:00
Stefan Holderbach
b88d8726d0
Rework for new clippy lints (#12736)
- **Clippy lint `assigning_clones`**
- **Clippy lint `legacy_numeric_constants`**
- **`clippy::float_equality_without_abs`**
- **`nu-table`: clippy::zero_repeat_side_effects**

---------

Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-02 19:29:03 +02:00
Darren Schroeder
0805f1fd90
overhaul shell_integration to enable individual control over ansi escape sequences (#12629)
# Description

This PR overhauls the shell_integration system by allowing individual
control over which ansi escape sequences are used. As we continue to
broaden our support for more ansi escape sequences, we can't really have
an all-or-nothing strategy. Some ansi escapes cause problems in certain
operating systems or terminals. We should allow the user to choose which
escapes they want.

TODO:
* Gather feedback
* Should osc7, osc9_9 and osc633p be mutually exclusive?
* Is the naming convention for these settings too nerdy osc2, osc7, etc?

closes #11301

# User-Facing Changes
shell_integration is no longer a boolean value. This is what is
supported in the default_config.nu
```nushell
  shell_integration: {
    # osc2 abbreviates the path if in the home_dir, sets the tab/window title, shows the running command in the tab/window title
    osc2: true
    # osc7 is a way to communicate the path to the terminal, this is helpful for spawning new tabs in the same directory
    osc7: true
    # osc8 is also implemented as the deprecated setting ls.show_clickable_links, it shows clickable links in ls output if your terminal supports it
    osc8: true
    # osc9_9 is from ConEmu and is starting to get wider support. It's similar to osc7 in that it communicates the path to the terminal
    osc9_9: false
    # osc133 is several escapes invented by Final Term which include the supported ones below.
    # 133;A - Mark prompt start
    # 133;B - Mark prompt end
    # 133;C - Mark pre-execution
    # 133;D;exit - Mark execution finished with exit code
    # This is used to enable terminals to know where the prompt is, the command is, where the command finishes, and where the output of the command is
    osc133: true
    # osc633 is closely related to osc133 but only exists in visual studio code (vscode) and supports their shell integration features
    # 633;A - Mark prompt start
    # 633;B - Mark prompt end
    # 633;C - Mark pre-execution
    # 633;D;exit - Mark execution finished with exit code
    # 633;E - NOT IMPLEMENTED - Explicitly set the command line with an optional nonce
    # 633;P;Cwd=<path> - Mark the current working directory and communicate it to the terminal
    # and also helps with the run recent menu in vscode
    osc633: true
    # reset_application_mode is escape \x1b[?1l and was added to help ssh work better
    reset_application_mode: true
  }
```

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-05-02 09:56:50 -04:00
Darren Schroeder
8ed0d84d6a
add raw-string literal support (#9956)
# Description

This PR adds raw string support by using `r#` at the beginning of single
quoted strings and `#` at the end.

Notice that escapes do not process, even within single quotes,
parentheses don't mean anything, $variables don't mean anything. It's
just a string.
```nushell
❯ echo r#'one\ntwo (blah) ($var)'#
one\ntwo (blah) ($var)
```
Notice how they work without `echo` or `print` and how they work without
carriage returns.
```nushell
❯ r#'adsfa'#
adsfa
❯ r##"asdfa'@qpejq'##
asdfa'@qpejq
❯ r#'asdfasdfasf
∙ foqwejfqo@'23rfjqf'#
```
They also have a special configurable color in the repl. (use single
quotes though)

![image](https://github.com/nushell/nushell/assets/343840/8780e21d-de4c-45b3-9880-2425f5fe10ef)

They should work like rust raw literals and allow `r##`, `r###`,
`r####`, etc, to help with having one or many `#`'s in the middle of
your raw-string.

They should work with `let` as well.

```nushell
r#'some\nraw\nstring'# | str upcase
```

closes https://github.com/nushell/nushell/issues/5091
# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect -A clippy::result_large_err` to check that
you're using the standard code style
- `cargo test --workspace` to check that all tests pass
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->

---------

Co-authored-by: WindSoilder <WindSoilder@outlook.com>
Co-authored-by: Ian Manske <ian.manske@pm.me>
2024-05-02 09:36:37 -04:00
Reilly Wood
3d340657b5
explore: adopt anyhow, support CustomValue, remove help system (#12692)
This PR:
1. Adds basic support for `CustomValue` to `explore`. Previously `open
foo.db | explore` didn't really work, now we "materialize" the whole
database to a `Value` before loading it
2. Adopts `anyhow` for error handling in `explore`. Previously we were
kind of rolling our own version of `anyhow` by shoving all errors into a
`std::io::Error`; I think this is much nicer. This was necessary because
as part of 1), collecting input is now fallible...
3. Removes a lot of `explore`'s fancy command help system.
- Previously each command (`:help`, `:try`, etc.) had a sophisticated
help system with examples etc... but this was not very visible to users.
You had to know to run `:help :try` or view a list of commands with
`:help :`
- As discussed previously, we eventually want to move to a less modal
approach for `explore`, without the Vim-like commands. And so I don't
think it's worth keeping this command help system around (it's
intertwined with other stuff, and making these changes would have been
harder if keeping it).
4. Rename the `--reverse` flag to `--tail`. The flag scrolls to the end
of the data, which IMO is described better by "tail"
5. Does some renaming+commenting to clear up things I found difficult to
understand when navigating the `explore` code


I initially thought 1) would be just a few lines, and then this PR blew
up into much more extensive changes 😅


## Before
The whole database was being displayed as a single Nuon/JSON line 🤔 

![image](https://github.com/nushell/nushell/assets/26268125/6383f43b-fdff-48b4-9604-398438ad1499)


## After
The database gets displayed like a record

![image](https://github.com/nushell/nushell/assets/26268125/2f00ed7b-a3c4-47f4-a08c-98d07efc7bb4)


## Future work

It is sort of annoying that we have to load a whole SQLite database into
memory to make this work; it will be impractical for large databases.
I'd like to explore improvements to `CustomValue` that can make this
work more efficiently.
2024-05-01 17:34:37 -05:00
Devyn Cairns
2970d48d41
Make bytes build accept integer values as individual bytes (#12685)
# Description
This creates an option for building binary data from byte integers.
Previously I think you could only do this by formatting the integers to
hex and using `decode hex`.

One potentially confusing thing is that this is different from the `into
binary` behavior. But since this doesn't support any of the other `into
binary` behaviors, it might be okay.

# User-Facing Changes
- `bytes build` accepts single byte arguments as integers

# Tests + Formatting
Example added.

# After Submitting
- [ ] release notes
2024-05-01 17:29:33 -05:00
YizhePKU
f184a77fe1
Path expansion no longer removes trailing slashes (#12662)
This PR changes `nu_path::expand_path_with()` to no longer remove
trailing slashes. It also fixes bugs in the current implementation due
to ineffective tests (Fixes #12602).
2024-05-01 17:28:54 -05:00
pwygab
b22d131279
Prevent each from swallowing errors when eval_block returns a ListStream (#12412)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx

you can also mention related issues, PRs or discussions!
-->

# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.

Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->

Prior, it seemed that nested errors would not get detected and shown.
This PR fixes that.

Resolves #10176:
```
~/CodingProjects/nushell> [[1,2]] | each {|x| $x | each {|y| error make {msg: "oh noes"} } }                        05/04/2024 21:34:08
Error: nu:🐚:eval_block_with_input

  × Eval block failed with pipeline input
   ╭─[entry #1:1:3]
 1 │ [[1,2]] | each {|x| $x | each {|y| error make {msg: "oh noes"} } }
   ·   ┬
   ·   ╰── source value
   ╰────

Error:   × oh noes
   ╭─[entry #1:1:36]
 1 │ [[1,2]] | each {|x| $x | each {|y| error make {msg: "oh noes"} } }
   ·                                    ─────┬────
   ·                                         ╰── originates from here
   ╰────
```

Resolves #11224:
```
~/CodingProjects/nushell> [0] | each { |_|                                                                          05/04/2024 21:35:40
:::     [0] | each { |_|
:::         non-existent-command
:::     }
::: }
Error: nu:🐚:eval_block_with_input

  × Eval block failed with pipeline input
   ╭─[entry #1:2:6]
 1 │ [0] | each { |_|
 2 │     [0] | each { |_|
   ·      ┬
   ·      ╰── source value
 3 │         non-existent-command
   ╰────

Error: nu:🐚:external_command

  × External command failed
   ╭─[entry #1:3:9]
 2 │     [0] | each { |_|
 3 │         non-existent-command
   ·         ──────────┬─────────
   ·                   ╰── executable was not found
 4 │     }
   ╰────
  help: No such file or directory (os error 2)
```

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-05-01 17:24:54 -05:00
YizhePKU
52d99cc60c
Change environment variables to be case-preserving (#12701)
This PR changes `$env` to be **case-preserving** instead of
case-sensitive. That is, it preserves the case of the environment
variable when it is first assigned, but subsequent retrieval and update
ignores the case.

Notably, both `$env.PATH` and `$env.Path` can now be used to read or set
the environment variable, but child processes will always see the
correct case based on the platform.

Fixes #11268.

---

This feature was surprising simple to implement, because most of the
infrastructure to support case-insensitive cell path access already
exists. The `get` command extracts data using a cell path in a
case-insensitive way (!), but accepts a `--sensitive` flag. (I think
this should be flipped around?)
2024-05-01 17:22:34 -05:00
Devyn Cairns
21ebdfe8d7
Bump version to 0.93.1 (#12710)
# Description

Next patch/dev release, `0.93.1`
2024-05-01 17:19:20 -05:00
Devyn Cairns
3b220e07e3
Bump version to 0.93.0 (#12709)
# Description

Bump version to `0.93.0`
2024-04-30 15:51:13 -07:00
Ian Manske
e83123dcca
Add fs feature to nix dependency (#12702)
# Description
Caught a compilation error using `cargo hack` -- `nu-utils` will not
compile without the `fs` feature enabled for `nix`.
2024-04-29 22:50:39 +00:00
Devyn Cairns
648486400c
Fix missing local socket feature (#12698)
# Description

So sorry to do this during the pre-release freeze, but my plugin crate
split PR broke local socket mode, because `nu-plugin-protocol` didn't
have the compile feature to advertise the `LocalSocket` protocol
feature.

This is a very simple, configuration-only bugfix that I think really
needs to be merged before the release, or else local socket mode won't
work at all.

# Tests + Formatting

There's an oversight in my testing that caused this to not be caught:
the engine really did have the feature, but it just wasn't advertising
it, so for `stress_internals` it was still able to use it successfully.
Post-release I'll try to make sure this is properly handled somehow.
2024-04-29 15:02:56 +08:00
Darren Schroeder
59ee96c70d
fixes a rust-analyzer warning (#12694)
# Description

Minor change but fixes a few rust-analyzer warnings.

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-04-28 04:52:05 -05:00
Darren Schroeder
365f5954ee
restore query web --as-table to working order (#12693)
# Description

This PR fixes a problem introduced with PR
https://github.com/nushell/nushell/pull/12236. That PR accidentally
stopped `--as-table` from working.

Closes https://github.com/nushell/nushell/issues/12689

It works again.

![image](https://github.com/nushell/nushell/assets/343840/b517507f-6b92-4e39-a389-5c69907d77c0)

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

# After Submitting
<!-- If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
-->
2024-04-28 04:21:26 -05:00
Devyn Cairns
0c4d5330ee
Split the plugin crate (#12563)
# Description

This breaks `nu-plugin` up into four crates:

- `nu-plugin-protocol`: just the type definitions for the protocol, no
I/O. If someone wanted to wire up something more bare metal, maybe for
async I/O, they could use this.
- `nu-plugin-core`: the shared stuff between engine/plugin. Less stable
interface.
- `nu-plugin-engine`: everything required for the engine to talk to
plugins. Less stable interface.
- `nu-plugin`: everything required for the plugin to talk to the engine,
what plugin developers use. Should be the most stable interface.

No changes are made to the interface exposed by `nu-plugin` - it should
all still be there. Re-exports from `nu-plugin-protocol` or
`nu-plugin-core` are used as required. Plugins shouldn't ever have to
use those crates directly.

This should be somewhat faster to compile as `nu-plugin-engine` and
`nu-plugin` can compile in parallel, and the engine doesn't need
`nu-plugin` and plugins don't need `nu-plugin-engine` (except for test
support), so that should reduce what needs to be compiled too.

The only significant change here other than splitting stuff up was to
break the `source` out of `PluginCustomValue` and create a new
`PluginCustomValueWithSource` type that contains that instead. One bonus
of that is we get rid of the option and it's now more type-safe, but it
also means that the logic for that stuff (actually running the plugin
for custom value ops) can live entirely within the `nu-plugin-engine`
crate.

# User-Facing Changes
- New crates.
- Added `local-socket` feature for `nu` to try to make it possible to
compile without that support if needed.

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-04-27 12:08:12 -05:00
Maxim Uvarov
884d5312bb
add tests to polars unique (#12683)
# Description

I would like to help with `polars` plugin development and add tests to
all the `polars` command's existing params.

Since I have never written any lines of Rust, even though the task of
creating tests is relatively simple, I would like to ask for feedback to
ensure I did everything correctly here.
2024-04-27 12:04:54 -05:00
Devyn Cairns
02de69de92
Fix inconsistent print behavior (#12675)
# Description

I found a bunch of issues relating to the specialized reimplementation
of `print()` that's done in `nu-cli` and it just didn't seem necessary.
So I tried to unify the behavior reasonably. `PipelineData::print()`
already handles the call to `table` and it even has a `no_newline`
option.

One of the most major issues before was that we were using the value
iterator, and then converting to string, and then printing each with
newlines. This doesn't work well for an external stream, because its
iterator ends up creating `Value::binary()` with each buffer... so we
were doing lossy UTF-8 conversion on those and then printing them with
newlines, which was very weird:


![Screenshot_2024-04-26_02-02-29](https://github.com/nushell/nushell/assets/10729/131c2224-08ee-4582-8617-6ecbb3ce8da5)

You can see the random newline inserted in a break between buffers, but
this would be even worse if it were on a multibyte UTF-8 character. You
can produce this by writing a large amount of text to a text file, and
then doing `nu -c 'open file.txt'` - in my case I just wrote `^find .`;
it just has to be large enough to trigger a buffer break.

Using `print()` instead led to a new issue though, because it doesn't
abort on errors. This is so that certain commands can produce a stream
of errors and have those all printed. There are tests for e.g. `rm` that
depend on this behavior. I assume we want to keep that, so instead I
made my target `BufferedReader`, and had that fuse closed if an error
was encountered. I can't imagine we want to keep reading from a wrapped
I/O stream if an error occurs; more often than not the error isn't going
to magically resolve itself, it's not going to be a different error each
time, and it's just going to lead to an infinite stream of the same
error.

The test that broke without that was `open . | lines`, because `lines`
doesn't fuse closed on error. But I don't know if it's expected or not
for it to do that, so I didn't target that.

I think this PR makes things better but I'll keep looking for ways to
improve on how errors and streams interact, especially trying to
eliminate cases where infinite error loops can happen.

# User-Facing Changes
- **Breaking**: `BufferedReader` changes + no more public fields
- A raw I/O stream from e.g. `open` won't produce infinite errors
anymore, but I consider that to be a plus
- the implicit `print` on script output is the same as the normal one
now

# Tests + Formatting
Everything passes but I didn't add anything specific.
2024-04-27 00:25:11 +00:00
Ian Manske
533603b72c
Add deprecation warning to describe --collect-lazyrecords (#12667)
# Description
Missed a spot for lazy record deprecation. This adds a warning for the
`--collect-lazyrecords` flag on `describe`.
2024-04-26 16:36:30 +00:00
Ian Manske
1ecbb3e09f
Make exit code available in catch block (#12648)
# Description
Bandaid fix for #12643, where it is not possible to get the exit code of
a failed external command while also having the external command inherit
nushell's stdout and stderr. This changes `try` so that the exit code of
external command is available in the `catch` block via the usual
`$env.LAST_EXIT_CODE`.

# Tests + Formatting
Added one test.

# After Submitting
Rework I/O redirection and possibly exit codes.
2024-04-26 16:35:08 +00:00
pwygab
d23a3737c0
make grid throw an error when not enough columns (#12672)
<!--
if this PR closes one or more issues, you can automatically link the PR
with
them by using one of the [*linking
keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword),
e.g.
- this PR should close #xxxx
- fixes #xxxx

you can also mention related issues, PRs or discussions!
-->

# Description
<!--
Thank you for improving Nushell. Please, check our [contributing
guide](../CONTRIBUTING.md) and talk to the core team before making major
changes.

Description of your pull request goes here. **Provide examples and/or
screenshots** if your changes affect the user experience.
-->

Resolves #12654. 

# User-Facing Changes
<!-- List of all changes that impact the user experience here. This
helps us keep track of breaking changes. -->

`grid` can now throw an error.

# Tests + Formatting
<!--
Don't forget to add tests that cover your changes.

Make sure you've run and fixed any issues with these commands:

- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to
check that you're using the standard code style
- `cargo test --workspace` to check that all tests pass (on Windows make
sure to [enable developer
mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging))
- `cargo run -- -c "use std testing; testing run-tests --path
crates/nu-std"` to run the tests for the standard library

> **Note**
> from `nushell` you can also use the `toolkit` as follows
> ```bash
> use toolkit.nu # or use an `env_change` hook to activate it
automatically
> toolkit check pr
> ```
-->

Added relevant test.
2024-04-26 06:33:00 -05:00
Devyn Cairns
d126793290
Add plugin error propagation on write/flush (#12670)
# Description
Yet another attempt to fix the `stress_internals::test_wrong_version()`
test...

This time I think it's probably because we are getting a broken pipe
write error when we try to send `Hello` or perhaps something after it,
because the pipe has already been closed by the reader when it saw the
invalid version. In that case, an error should be available in state. It
probably makes more sense to send that back to the user rather than an
unhelpful I/O error.

# Tests + Formatting
- 🟢 `toolkit fmt`
- 🟢 `toolkit clippy`
- 🟢 `toolkit test`
- 🟢 `toolkit test stdlib`
2024-04-26 06:23:58 -05:00