mirror of
https://github.com/nushell/nushell.git
synced 2024-11-25 18:03:51 +01:00
b06f31d3c6
556 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
YizhePKU
|
6c649809d3
|
Rewrite run_external.rs (#12921)
This PR is a complete rewrite of `run_external.rs`. The main goal of the rewrite is improving readability, but it also fixes some bugs related to argument handling and the PATH variable (fixes https://github.com/nushell/nushell/issues/6011). I'll discuss some technical details to make reviewing easier. ## Argument handling Quoting arguments for external commands is hard. Like, *really* hard. We've had more than a dozen issues and PRs dedicated to quoting arguments (see Appendix) but the current implementation is still buggy. Here's a demonstration of the buggy behavior: ```nu let foo = "'bar'" ^touch $foo # This creates a file named `bar`, but it should be `'bar'` ^touch ...[ "'bar'" ] # Same ``` I'll describe how this PR deals with argument handling. First, we'll introduce the concept of **bare strings**. Bare strings are **string literals** that are either **unquoted** or **quoted by backticks** [^1]. Strings within a list literal are NOT considered bare strings, even if they are unquoted or quoted by backticks. When a bare string is used as an argument to external process, we need to perform tilde-expansion, glob-expansion, and inner-quotes-removal, in that order. "Inner-quotes-removal" means transforming from `--option="value"` into `--option=value`. ## `.bat` files and CMD built-ins On Windows, `.bat` files and `.cmd` files are considered executable, but they need `CMD.exe` as the interpreter. The Rust standard library supports running `.bat` files directly and will spawn `CMD.exe` under the hood (see [documentation](https://doc.rust-lang.org/std/process/index.html#windows-argument-splitting)). However, other extensions are not supported [^2]. Nushell also supports a selected number of CMD built-ins. The problem with CMD is that it uses a different set of quoting rules. Correctly quoting for CMD requires using [Command::raw_arg()](https://doc.rust-lang.org/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg) and manually quoting CMD special characters, on top of quoting from the Nushell side. ~~I decided that this is too complex and chose to reject special characters in CMD built-ins instead [^3]. Hopefully this will not affact real-world use cases.~~ I've implemented escaping that works reasonably well. ## `which-support` feature The `which` crate is now a hard dependency of `nu-command`, making the `which-support` feature essentially useless. The `which` crate is already a hard dependency of `nu-cli`, and we should consider removing the `which-support` feature entirely. ## Appendix Here's a list of quoting-related issues and PRs in rough chronological order. * https://github.com/nushell/nushell/issues/4609 * https://github.com/nushell/nushell/issues/4631 * https://github.com/nushell/nushell/issues/4601 * https://github.com/nushell/nushell/pull/5846 * https://github.com/nushell/nushell/issues/5978 * https://github.com/nushell/nushell/pull/6014 * https://github.com/nushell/nushell/issues/6154 * https://github.com/nushell/nushell/pull/6161 * https://github.com/nushell/nushell/issues/6399 * https://github.com/nushell/nushell/pull/6420 * https://github.com/nushell/nushell/pull/6426 * https://github.com/nushell/nushell/issues/6465 * https://github.com/nushell/nushell/issues/6559 * https://github.com/nushell/nushell/pull/6560 [^1]: The idea that backtick-quoted strings act like bare strings was introduced by Kubouch and briefly mentioned in [the language reference](https://www.nushell.sh/lang-guide/chapters/strings_and_text.html#backtick-quotes). [^2]: The documentation also said "running .bat scripts in this way may be removed in the future and so should not be relied upon", which is another reason to move away from this. But again, quoting for CMD is hard. [^3]: If anyone wants to try, the best resource I found on the topic is [this](https://daviddeley.com/autohotkey/parameters/parameters.htm). |
||
Devyn Cairns
|
21ebdfe8d7
|
Bump version to 0.93.1 (#12710)
# Description Next patch/dev release, `0.93.1` |
||
|
Devyn Cairns
|
3b220e07e3
|
Bump version to 0.93.0 (#12709)
# Description Bump version to `0.93.0` |
||
|
Devyn Cairns
|
adf38c7c76
|
Msgpack commands (#12664)
# Description I thought about bringing `nu_plugin_msgpack` in, but that is MPL with a clause that prevents other licenses, so rather than adapt that code I decided to take a crack at just doing it straight from `rmp` to `Value` without any `rmpv` in the middle. It seems like it's probably faster, though I can't say for sure how much with the plugin overhead. @IanManske I started on a `Read` implementation for `RawStream` but just specialized to `from msgpack` here, but I'm thinking after release maybe we can polish it up and make it a real one. It works! # User-Facing Changes New commands: - `from msgpack` - `from msgpackz` - `to msgpack` - `to msgpackz` # Tests + Formatting Pretty thorough tests added for the format deserialization, with a roundtrip for serialization. Some example tests too for both `from msgpack` and `to msgpack`. - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting - [ ] update release notes |
||
Antoine Stevan
|
55edef5dda
|
create nuon crate from from nuon and to nuon (#12553)
# Description
playing with the NUON format in Rust code in some plugins, we agreed
with the team it was a great time to create a standalone NUON format to
allow Rust devs to use this Nushell file format.
> **Note**
> this PR almost copy-pastes the code from
`nu_commands/src/formats/from/nuon.rs` and
`nu_commands/src/formats/to/nuon.rs` to `nuon/src/from.rs` and
`nuon/src/to.rs`, with minor tweaks to make then standalone functions,
e.g. remove the rest of the command implementations
### TODO
- [x] add tests
- [x] add documentation
# User-Facing Changes
devs will have access to a new crate, `nuon`, and two functions,
`from_nuon` and `to_nuon`
```rust
from_nuon(
input: &str,
span: Option<Span>,
) -> Result<Value, ShellError>
```
```rust
to_nuon(
input: &Value,
raw: bool,
tabs: Option<usize>,
indent: Option<usize>,
span: Option<Span>,
) -> Result<String, ShellError>
```
# Tests + Formatting
i've basically taken all the tests from
`crates/nu-command/tests/format_conversions/nuon.rs` and converted them
to use `from_nuon` and `to_nuon` instead of Nushell commands
- i've created a `nuon_end_to_end` to run both conversions with an
optional middle value to check that all is fine
> **Note**
> the `nuon::tests::read_code_should_fail_rather_than_panic` test does
give different results locally and in the CI...
> i've left it ignored with comments to help future us :)
# After Submitting
mention that in the release notes for sure!!
|
||
Stefan Holderbach
|
872945ae8e
|
Bump version to 0.92.3 (#12476)
|
||
|
Stefan Holderbach
|
394487b3a7
|
Bump version to 0.92.2 (#12402)
|
||
|
Stefan Holderbach
|
c00a05a762
|
Bump version to 0.92.1 (#12380)
|
||
|
Stefan Holderbach
|
c3428b891a
|
Bump version for 0.92.0 release (#12349)
- [x] `cargo hack` feature flag compatibility run - [x] reedline released and pinned - [x] `nu-plugin-test-support` added to release script - [x] dependency tree checked - [x] release notes |
||
Ian Manske
|
251599c507
|
Use safe nix API instead of libc (#12315)
# Description Where possible, this PR replaces usages of raw `libc` bindings to instead use safe interfaces from the `nix` crate. Where not possible, the `libc` version reexported through `nix` was used instead of having a separate `libc` dependency. |
||
Wind
|
a15462fd00
|
Change default algorithm in detect columns (#12277)
# Description @fdncred found another histogram based algorithm to detect columns, and rewrite it in rust: https://github.com/fdncred/guess-width I have tested it manually, and it works good with `df`, `docker ps`, `^ps`. This pr is going to use the algorithm in `detect columns` Fix: #4183 The pitfall of new algorithm: 1. it may not works well if there isn't too much rows of input 2. it may not works well if the length of value is less than the header to value, e.g: ``` c1 c2 c3 c4 c5 a b c d e g h i j k g a a q d a v c q q | detect columns ``` In this case, users might need to use ~~`--old`~~ `--legacy` to make it works well. # User-Facing Changes User might need to add ~~`--old`~~ `--legacy` to scripts if they find `detect columns` in their scripts broken. # Tests + Formatting Done # After Submitting NaN |
||
David Matos
|
838fc7e098
|
Initial implementation for uutils uname (#11684)
Hi, This PR aims at implementing the first iteration for `uname` using `uutils`. Couple of things: * Currently my [PR](https://github.com/uutils/coreutils/pull/5921) to make the required changes is pending in `uutils` repo. * I guess the number of flags has to be investigated. Still the tests cover all of them. <!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - [X] `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - [X] `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - [X] `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - [X] `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com> |
||
Antoine Büsch
|
4ddc35cdad
|
Move more dependencies to workspace level (#12270)
# Description This is a followup to #12043 that moves more dependency versions to workspace dependencies. # User-Facing Changes N/A # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` |
||
dependabot[bot]
|
6e2c41a5b5
|
Bump base64 from 0.21.7 to 0.22.0 (#12188)
Bumps [base64](https://github.com/marshallpierce/rust-base64) from 0.21.7 to 0.22.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md">base64's changelog</a>.</em></p> <blockquote> <h1>0.22.0</h1> <ul> <li><code>DecodeSliceError::OutputSliceTooSmall</code> is now conservative rather than precise. That is, the error will only occur if the decoded output <em>cannot</em> fit, meaning that <code>Engine::decode_slice</code> can now be used with exactly-sized output slices. As part of this, <code>Engine::internal_decode</code> now returns <code>DecodeSliceError</code> instead of <code>DecodeError</code>, but that is not expected to affect any external callers.</li> <li><code>DecodeError::InvalidLength</code> now refers specifically to the <em>number of valid symbols</em> being invalid (i.e. <code>len % 4 == 1</code>), rather than just the number of input bytes. This avoids confusing scenarios when based on interpretation you could make a case for either <code>InvalidLength</code> or <code>InvalidByte</code> being appropriate.</li> <li>Decoding is somewhat faster (5-10%)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
sarubo
|
687fbc49c8
|
Adjust permissions using umask in mkdir (#12207)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> With this change, `mkdir` mirrors coreutils works. Closes #12161 I referred to the implementation of `mkdir` in uutils/coreutils. I add `uucore` required for implementation to dependencies. Since `uucore` is already included in dependencies of `uu_mkdir`, I don't think there will be any additional dependencies. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> - Directories are created according to `umask` except for Windows. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> I add `mkdir` test considering permissions. The test assumes that the default `umask` is `022`. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
dependabot[bot]
|
8725bd3112
|
Bump rayon from 1.8.1 to 1.9.0 (#12186)
Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.8.1 to 1.9.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rayon-rs/rayon/blob/main/RELEASES.md">rayon's changelog</a>.</em></p> <blockquote> <h1>Release rayon 1.9.0 (2024-02-27)</h1> <ul> <li>The new methods <code>IndexedParallelIterator::by_exponential_blocks</code> and <code>by_uniform_blocks</code> allow processing items in smaller groups at a time.</li> <li>The new <code>iter::walk_tree</code>, <code>walk_tree_prefix</code>, and <code>walk_tree_postfix</code> functions enable custom parallel iteration over tree-like structures.</li> <li>The new method <code>ParallelIterator::collect_vec_list</code> returns items as a linked list of vectors, which is an efficient mode of parallel collection used by many of the internal implementations of <code>collect</code>.</li> <li>The new methods <code>ParallelSliceMut::par_split_inclusive_mut</code>, <code>ParallelSlice::par_split_inclusive</code>, and <code>ParallelString::par_split_inclusive</code> all work like a normal split but keeping the separator as part of the left slice.</li> <li>The new <code>ParallelString::par_split_ascii_whitespace</code> splits only on ASCII whitespace, which is faster than including Unicode multi-byte whitespace.</li> <li><code>OsString</code> now implements <code>FromParallelIterator<_></code> and <code>ParallelExtend<_></code> for a few item types similar to the standard <code>FromIterator</code> and <code>Extend</code>.</li> <li>The internal <code>Pattern</code> trait for string methods is now implemented for <code>[char; N]</code> and <code>&[char; N]</code>, matching any of the given characters.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
b2b4562376
|
Bump windows from 0.52.0 to 0.54.0 (#12037)
Bumps [windows](https://github.com/microsoft/windows-rs) from 0.52.0 to 0.54.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Antoine Büsch
|
979a97c455
|
Introduce workspace dependencies (#12043)
# Description This PR introduces [workspaces dependencies](https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table). The advantages are: - a single place where dependency versions are declared - reduces the number of files to change when upgrading a dependency - reduces the risk of accidentally depending on 2 different versions of the same dependency I've only done a few so far. If this PR is accepted, I might continue and progressively do the rest. # User-Facing Changes N/A # Tests + Formatting - 🟢 `toolkit fmt` - 🟢 `toolkit clippy` - 🟢 `toolkit test` - 🟢 `toolkit test stdlib` # After Submitting N/A |
||
|
dependabot[bot]
|
9c00757a5e
|
Bump open from 5.0.1 to 5.1.1 (#12061)
Bumps [open](https://github.com/Byron/open-rs) from 5.0.1 to 5.1.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/Byron/open-rs/releases">open's releases</a>.</em></p> <blockquote> <h2>v5.1.1</h2> <h3>Bug Fixes</h3> <ul> <li>add <code>shellexecute-on-windows</code> feature. That way, it's possible to toggle on a feature that might cause issues in some dependency trees that contain <code>flate2</code> with <code>zlib-ng</code> backend.</li> </ul> <h3>Commit Statistics</h3> <ul> <li>3 commits contributed to the release.</li> <li>2 days passed between releases.</li> <li>1 commit was understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>0 issues like '(#ID)' were seen in commit messages</li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong>Uncategorized</strong> <ul> <li>Merge branch 'validate-linkage' (59886df)</li> <li>Add <code>shellexecute-on-windows</code> feature. (74fd8ec)</li> <li>Try to validate linkage on all platforms (8f26da4)</li> </ul> </li> </ul> <!-- raw HTML omitted --> <h2>v5.1.0</h2> <h3>New Features</h3> <ul> <li>use <code>ShellExecuteW</code> for detached spawning on Windows</li> </ul> <h3>Commit Statistics</h3> <ul> <li>3 commits contributed to the release.</li> <li>2 days passed between releases.</li> <li>1 commit was understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>0 issues like '(#ID)' were seen in commit messages</li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong>Uncategorized</strong> <ul> <li>Merge pull request <a href="https://redirect.github.com/Byron/open-rs/issues/91">#91</a> from amrbashir/feat/windows/detachded-using-shellexecutew (b268647)</li> <li>Split into two functions for better readability (4506b2f)</li> <li>Use <code>ShellExecuteW</code> for detached spawning on Windows (191cb0e)</li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Byron/open-rs/blob/main/changelog.md">open's changelog</a>.</em></p> <blockquote> <h2>5.1.1 (2024-03-03)</h2> <h3>Bug Fixes</h3> <ul> <li><!-- raw HTML omitted --> add <code>shellexecute-on-windows</code> feature. That way, it's possible to toggle on a feature that might cause issues in some dependency trees that contain <code>flate2</code> with <code>zlib-ng</code> backend.</li> </ul> <h3>Commit Statistics</h3> <!-- raw HTML omitted --> <ul> <li>3 commits contributed to the release.</li> <li>2 days passed between releases.</li> <li>1 commit was understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>0 issues like '(#ID)' were seen in commit messages</li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong>Uncategorized</strong> <ul> <li>Merge branch 'validate-linkage' (<a href=" |
||
|
dependabot[bot]
|
85fb3e1ff3
|
Bump mockito from 1.3.0 to 1.4.0 (#12063) | ||
|
Stefan Holderbach
|
e5f086cfb4
|
Bump version to 0.91.1 (#12085)
|
||
|
Stefan Holderbach
|
3016d7a64c
|
Bump version for 0.91.0 release (#12070) | ||
|
dependabot[bot]
|
2697ea9a25
|
Bump mockito from 1.2.0 to 1.3.0 (#11976)
Bumps [mockito](https://github.com/lipanski/mockito) from 1.2.0 to 1.3.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/lipanski/mockito/releases">mockito's releases</a>.</em></p> <blockquote> <h2>1.3.0</h2> <ul> <li><a href=" |
||
nils-degroot
|
67a63162b2
|
Add date support in from xlsx (#11952)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> This PR add date support when using the `open` command on a xlsx file, and the using `from xlsx` on a xlsx file. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> Currently dates in xlsx files are read as nulls, after this PR this would be regular dates. # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Darren Schroeder
|
a2a1c1656f
|
remove unused dependencies (#11938)
# Description This PR removes unused dependencies. The `cargo machete --with-metadata` tool was used to determine what is unused and then I recompiled. Putting this up here to see what happens in MacOS and Linux in the CI and see if anything breaks. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
David Matos
|
123bf2d736
|
fix format date based on users locale (#11908)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description Hi, Fixes #10838, where before the `date` would be formatted incorrectly, and was not picking `LC_TIME` for time formatting, but it picked the first locale returned by the `sys-locale` crate instead. Now it will format time based on `LC_TIME`. For example, ``` // my locale `nl_NL.UTF-8` ❯ date now | format date '%x %X' 20-02-24 17:17:12 $env.LC_TIME = "en_US.UTF-8" ❯ date now | format date '%x %X' 02/20/2024 05:16:28 PM ``` Note that I also changed the `default_env.nu` as otherwise the Time will show AM/PM twice. Also reason for the `chrono` update is because this relies on a fix to upstream repo, which i initially submitted an [issue](https://github.com/chronotope/chrono/issues/1349#event-11765363286) <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - [X] `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - [X] `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - [X] `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - [X] `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
dependabot[bot]
|
68a3d7c430
|
Bump trash from 3.2.1 to 3.3.1 (#11894)
Bumps [trash](https://github.com/ArturKovacs/trash) from 3.2.1 to 3.3.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/ArturKovacs/trash/releases">trash's releases</a>.</em></p> <blockquote> <h2>v3.3.1</h2> <h3>Bug Fixes</h3> <ul> <li>Use <code>AtomicI32</code> instead of I64 for compatibility with <code>armel</code></li> </ul> <h3>Commit Statistics</h3> <ul> <li>1 commit contributed to the release.</li> <li>2 days passed between releases.</li> <li>1 commit was understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>1 unique issue was worked on: <a href="https://redirect.github.com/ArturKovacs/trash/issues/99">#99</a></li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong><a href="https://redirect.github.com/ArturKovacs/trash/issues/99">#99</a></strong> <ul> <li>Use <code>AtomicI32</code> instead of I64 for compatibility with <code>armel</code> (98049f1)</li> </ul> </li> </ul> <!-- raw HTML omitted --> <h2>v3.3.0</h2> <h3>New Features</h3> <ul> <li>improved error granularity Inform about operating-system specific errors more clearly, thus avoid degenerating error information.</li> </ul> <h3>Bug Fixes</h3> <ul> <li>Use <code>AtomicI32</code> in tests for compatibility with <code>armel</code> platform</li> </ul> <h3>Commit Statistics</h3> <ul> <li>5 commits contributed to the release over the course of 5 calendar days.</li> <li>25 days passed between releases.</li> <li>2 commits were understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>1 unique issue was worked on: <a href="https://redirect.github.com/ArturKovacs/trash/issues/99">#99</a></li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong><a href="https://redirect.github.com/ArturKovacs/trash/issues/99">#99</a></strong> <ul> <li>Use <code>AtomicI32</code> in tests for compatibility with <code>armel</code> platform (920ff0c)</li> </ul> </li> <li><strong>Uncategorized</strong> <ul> <li>Improved error granularity (452be83)</li> <li>Removed tracing. (2b1c9fa)</li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Byron/trash-rs/blob/master/CHANGELOG.md">trash's changelog</a>.</em></p> <blockquote> <h2>3.3.1 (2024-02-12)</h2> <h3>Bug Fixes</h3> <ul> <li><!-- raw HTML omitted --> Use <code>AtomicI32</code> instead of I64 for compatibility with <code>armel</code></li> </ul> <h3>Commit Statistics</h3> <!-- raw HTML omitted --> <ul> <li>1 commit contributed to the release.</li> <li>2 days passed between releases.</li> <li>1 commit was understood as <a href="https://www.conventionalcommits.org">conventional</a>.</li> <li>1 unique issue was worked on: <a href="https://redirect.github.com/Byron/trash-rs/issues/99">#99</a></li> </ul> <h3>Commit Details</h3> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <ul> <li><strong><a href="https://redirect.github.com/Byron/trash-rs/issues/99">#99</a></strong> <ul> <li>Use <code>AtomicI32</code> instead of I64 for compatibility with <code>armel</code> (<a href=" |
||
|
dependabot[bot]
|
a4ef7c1ac4
|
Bump fancy-regex from 0.12.0 to 0.13.0 (#11893)
[//]: # (dependabot-start) ⚠️ **Dependabot is rebasing this PR** ⚠️ Rebasing might not happen immediately, so don't worry if this takes some time. Note: if you make any changes to this PR yourself, they will take precedence over the rebase. --- [//]: # (dependabot-end) Bumps [fancy-regex](https://github.com/fancy-regex/fancy-regex) from 0.12.0 to 0.13.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/fancy-regex/fancy-regex/releases">fancy-regex's releases</a>.</em></p> <blockquote> <h2>0.13.0</h2> <h3>Added</h3> <ul> <li>Support for relative backreferences using <code>\k<-1></code> (-1 references the previous group) (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Add <code>try_replacen</code> to <code>Regex</code> which returns a <code>Result</code> instead of panicking when matching errors (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/130">#130</a>)</li> </ul> <h3>Changed</h3> <ul> <li>Switch from regex crate to regex-automata and regex-syntax (lower level APIs) to simplify internals (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Allow escaping some letters in character classes, e.g. <code>[\A]</code> used to error but now matches the same as <code>[A]</code> (for compatibility with Oniguruma)</li> <li>MSRV (minimum supported Rust version) is now 1.66.1 (from 1.61.0)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix index out of bounds panic when parsing unclosed <code>(?(</code> (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/125">#125</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/fancy-regex/fancy-regex/blob/main/CHANGELOG.md">fancy-regex's changelog</a>.</em></p> <blockquote> <h2>[0.13.0] - 2023-12-22</h2> <h3>Added</h3> <ul> <li>Support for relative backreferences using <code>\k<-1></code> (-1 references the previous group) (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Add <code>try_replacen</code> to <code>Regex</code> which returns a <code>Result</code> instead of panicking when matching errors (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/130">#130</a>)</li> </ul> <h3>Changed</h3> <ul> <li>Switch from regex crate to regex-automata and regex-syntax (lower level APIs) to simplify internals (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/121">#121</a>)</li> <li>Allow escaping some letters in character classes, e.g. <code>[\A]</code> used to error but now matches the same as <code>[A]</code> (for compatibility with Oniguruma)</li> <li>MSRV (minimum supported Rust version) is now 1.66.1 (from 1.61.0)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix index out of bounds panic when parsing unclosed <code>(?(</code> (<a href="https://redirect.github.com/fancy-regex/fancy-regex/issues/125">#125</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
6a1691f378
|
Bump miette from 7.0.0 to 7.1.0 (#11892)
Bumps [miette](https://github.com/zkat/miette) from 7.0.0 to 7.1.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/zkat/miette/releases">miette's releases</a>.</em></p> <blockquote> <h2>v7.1.0</h2> <h3>Features</h3> <ul> <li><strong>derive:</strong> enable more boxed types to be #[diagnostic_source] (<a href="https://redirect.github.com/zkat/miette/issues/338">#338</a>) (<a href=" |
||
|
dependabot[bot]
|
1350f1eff7
|
Bump unicode-segmentation from 1.10.1 to 1.11.0 (#11891)
Bumps [unicode-segmentation](https://github.com/unicode-rs/unicode-segmentation) from 1.10.1 to 1.11.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Darren Schroeder
|
360ebeb0bc
|
update to the latest reedline and rusqlite (#11878)
# Description This updates nushell to the latest reedline and rusqlite dependencies. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Stefan Holderbach
|
7f1e025cc9
|
Bump roxmltree to 0.19 (#11876)
Manual changes to adapt to new error variants. Error text adapted from docs. Supersedes #11605 |
||
|
Stefan Holderbach
|
1b220815b9
|
Bump calamine to 0.24 (#11874)
# Description Update `DataType` to `Data` enum due to breaking change Supersedes #11834 # User-Facing Changes None obvious |
||
|
dependabot[bot]
|
317653d5d2
|
Bump which from 5.0.0 to 6.0.0 (#11832) | ||
|
dependabot[bot]
|
e7f1bf8535
|
Bump indexmap from 2.1.0 to 2.2.2 (#11746) | ||
nibon7
|
84517138bc
|
Bump miette from 5.10.0 to 7.0.0 (#11788)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description Bump miette from 5.10.0 to 7.0.0 # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com> |
||
|
Darren Schroeder
|
08931e976e
|
bump to dev release of nushell 0.90.2 (#11793)
# Description Bump nushell version to the dev version of 0.90.2 # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Jakub Žádník
|
c2992d5d8b
|
Bump to 0.90.1 (#11787)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> Merge after https://github.com/nushell/nushell/pull/11786 # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jakub Žádník
|
f5f21aca2d
|
Bump to 0.90 (#11730)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Ian Manske
|
4e0a65c822
|
Strict JSON parsing (#11592)
# Description Adds the `--strict` flag for `from json` which will try to parse text while following the exact JSON specification (e.g., no comments or trailing commas allowed). Fixes issue #11548. |
||
Michael Angerman
|
f286286510
|
update nu-ansi-term to 0.50, lscolors to 0.17, and add the Style attribute to Suggestion (#11635)
* update nu-ansi-term to 0.50.0 * update lscolors to 0.17.0 * add the Style attribute to Suggestion * bump Reedline to the latest main |
||
|
Darren Schroeder
|
f12f590d82
|
update deps calamine and quick-xml (#11582)
# Description This PR updates a few outdated dependencies. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Michel Lind (né Salim)
|
5d63f47c85
|
Replace htmlescape with v_htmlescape (#11572)
# Description `htmlescape` is unmaintained: https://crates.io/crates/htmlescape while `v_htmlescape` is: https://crates.io/crates/v_htmlescape and is used by two popular crates (`actix-files` and `minijinja`) Let's use this instead - I'm packaging `nu` in Fedora and there is understandable reluctance in bringing in an unmaintained crate if we can avoid it. # User-Facing Changes Should not be any; drop-in replacement # Tests + Formatting Tested using: - `cargo build` in the root folder (needed by some `nu-command` tests) - `cargo test --features sqlite` in `crates/nu-command` (`tests/commands/database/into_sqlite.rs` needs `rusqlite`) - `cargo test` in `crates/nu-cmd-extra` # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> N/A Signed-off-by: Michel Lind <salimma@fedoraproject.org> |
||
|
David Matos
|
ee6547dbb7
|
Initial implementation of umv from uutils (#10822)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description Hi, This closes #10446 , wherein we start implementing `mv` from `uutils`. There are some stuff to iron out, particularly * Decide on behavior from ignored tests * Wait for release/PRs to be approved on `uutils` side, but still can be tested for now. See [PR approved](https://github.com/uutils/coreutils/pull/5428), and [pending](https://github.com/uutils/coreutils/pull/5429). * `--progress` does not seem to work on `uutils mv` either and have not checked whether certain `X` size has to be achieved in order for it to appear, thus something to investigate as well, but thought it wasnt important enough to not make the PR. See [issue comment](https://github.com/nushell/nushell/issues/10446#issuecomment-1764497988), on the possible strategy to follow, mainly copy what we did with `ucp`. I still left some comments on purpose particularly on tests, which of course would be removed before something is decided here. :) @fdncred <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - [X] `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - [X] `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - [X] `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - [X] `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library <!-- > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
Skyler Hawthorne
|
7ac3e97bfe
|
Fix memory consumption of into sqlite (#10232)
# Description Currently, the `into sqlite` command collects the entire input stream into a single Value, which soaks up the entire input into memory, before it ever tries to write anything to the DB. This is very problematic for large inputs; for example, I tried transforming a multi-gigabyte CSV file into SQLite, and before I knew what was happening, my system's memory was completely exhausted, and I had to hard reboot to recover. This PR fixes this problem by working directly with the pipeline stream, inserting into the DB as values are read from the stream. In order to facilitate working with the stream directly, I introduced a new `Table` struct to store the connection and a few configuration parameters, as well as to make it easier to lazily create the table on the first read value. In addition to the purely functional fixes, a few other changes were made to the serialization and user facing behavior. ### Serialization Much of the preexisting code was focused on generating the exact text needed for a SQL statement. This is unneeded and less safe than using the `rusqlite` crate's serialization for native Rust types along with prepared statements. ### User-Facing Changes Currently, the command is very liberal in the input types it accepts. The strategy is basically if it is a record, try to follow its structure and make an analogous SQL row, which is pretty reasonable. However, when it's not a record, it basically tries to guess what the user wanted and just makes a single column table and serializes the value into that one column, whatever type it may be. This has been changed so that it only accepts records as input. If the user wants to serialize non-record types into SQL, then they must explicitly opt into doing this by constructing a record or table with it first. For a utility for inserting data into SQL, I think it makes more sense to let the user choose how to convert their data, rather than make a choice for them that may surprise them. However, I understand this may be a controversial change. If the maintainers don't agree, I can change this back. #### Long switch names The `file_name` and `table_name` long form switches are currently snake_case and expect to be as such at the command line. These have been changed to kebab-case to be more conventional. # Tests + Formatting To test the memory consumption, I used [this publicly available index of all Wikipedia articles](https://dumps.wikimedia.org/enwiki/20230820/), using the first 10,000, 100,000, and 1,000,000 entries, in that order. I ran the following script to benchmark the changes against the current stable release: ```nu #!/usr/bin/nu # let shellbin = $"($env.HOME)/src/nushell/target/aarch64-linux-android/release/nu" let shellbin = "nu" const dbpath = 'enwiki-index.db' [10000, 100000, 1000000] | each {|rows| rm -f $dbpath; do { time -f '%M %e %U %S' $shellbin -c ( $"bzip2 -cdk ~/enwiki-20230820-pages-articles-multistream-index.txt.bz2 | head -n ($rows) | lines | parse '{offset}:{id}:{title}' | update cells -c [offset, id] { into int } | into sqlite ($dbpath)" ) } | complete | get stderr | str trim | parse '{rss_max} {real} {user} {kernel}' | update cells -c [rss_max] { $"($in)kb" | into filesize } | update cells -c [real, user, kernel] { $"($in)sec" | into duration } | insert rows $rows | roll right } | flatten | to nuon ``` This yields the following results Current stable release: |rows|rss_max|real|user|kernel| |-|-|-|-|-| |10000|53.6 MiB|770ms|460ms|420ms| |100000|209.6 MiB|6sec 940ms|3sec 740ms|4sec 380ms| |1000000|1.7 GiB|1min 8sec 810ms|38sec 690ms|42sec 550ms| This PR: |rows|rss_max|real|user|kernel| |-|-|-|-|-| |10000|38.2 MiB|780ms|440ms|410ms| |100000|39.8 MiB|6sec 450ms|3sec 530ms|4sec 160ms| |1000000|39.8 MiB|1min 3sec 230ms|37sec 440ms|40sec 180ms| # Note I started this branch kind of at the same time as my others, but I understand the feedback that smaller PRs are preferred. Let me know if it would be better to split this up. I do think the scope of the changes are on the bigger side even without the behavior changes I mentioned, so I'm not sure if that will help this particular PR very much, but I'm happy to oblige on request. |
||
|
Jakub Žádník
|
7bb9ee55c4
|
Bump to dev version 0.89.1 (#11513)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
Jakub Žádník
|
2c1560e281
|
Bump version for 0.89.0 release (#11511)
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> - [x] reedline - [x] released - [x] pinned - [ ] git dependency check - [ ] release notes # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> |
||
|
nibon7
|
7e26b4fcc2
|
Bump sysinfo from 0.29 to 0.30 (#11484)
# Description Bumps `sysinfo` to 0.30. * Changelog https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md#0304 # User-Facing Changes N/A |
||
|
nibon7
|
748d82cec1
|
Bump windows from 0.48.0 to 0.52.0 (#11325)
# Description Bump `windows` to 0.52.0 and fix `is_admin` https://github.com/microsoft/windows-rs/pull/2476 # User-Facing Changes N/A --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Stefan Holderbach
|
5e5d1ea81b
|
Bump fancy-regex to single 0.12.0 version (#11389)
Supersedes #11039 that was broken due to dependabot not correctly taking the workspace into account (this bug has been worked around in #11387) |