mirror of
https://github.com/nushell/nushell.git
synced 2024-11-08 01:24:38 +01:00
0d518bf813
<!-- if this PR closes one or more issues, you can automatically link the PR with them by using one of the [*linking keywords*](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), e.g. - this PR should close #xxxx - fixes #xxxx you can also mention related issues, PRs or discussions! --> # Description <!-- Thank you for improving Nushell. Please, check our [contributing guide](../CONTRIBUTING.md) and talk to the core team before making major changes. Description of your pull request goes here. **Provide examples and/or screenshots** if your changes affect the user experience. --> ## Problem I tried converting one of my Rust web scrapers to Nushell just to see how it would be done, but quickly ran into an issue that proved annoying to fix without diving into the source. For instance, let's say we have the following HTML ```html <p>Hello there, <span style="color: red;">World</span></p> ``` and we want to extract only the text within the `p` element, but not the `span`. With the current version of nu_plugin_query, if we run this code ```nushell echo `<p>Hello there, <span style="color: red;">World</span></p>` | query web -q "p" | get 0 # returns "Hello there, World" # but we want only "Hello there, " ``` we will get back a `list<string>` that contains 1 string `Hello there, World`. To avoid scraping the span, we would have to do something like this ```nushell const html = `<p>Hello there, <span style="color: red;">World</span></p>` $html | query web -q "p" | get 0 | str replace ($html | query web -q "p > span" | get 0) "" # returns "Hello there, " ``` In other words, we would have to make a sub scrape of the text we *don't* want in order to subtract it from the text we *do* want. ## Solution I didn't like this behavior, so I decided to change it. I modified the `execute_selector_query` function to collect all text nodes in the HTML element matching the query. Now `query web --query` will return a `list<list<string>>` ```nushell echo `<p>Hello there, <span style="color: red;">World</span></p>` | query web -q "p" | get 0 | to json --raw # returns ["Hello there, ","World"] ``` This also brings `query web --query`'s behavior more in line with [scraper's ElementRef::text()](https://docs.rs/scraper/latest/scraper/element_ref/struct.ElementRef.html#method.text) which "Returns an iterator over descendent text nodes", allowing you to choose how much of an element's text you want to scrape without resorting to string substitutions. ## Consequences As this is a user-facing change, the usage examples will produce different results than before. For example ```nushell http get https://phoronix.com | query web --query 'header' ``` will return a list of lists of 1 string each, whereas before it was just a list of strings. I only modified the 3rd example ```nushell # old http get https://www.nushell.sh | query web --query 'h2, h2 + p' | group 2 | each {rotate --ccw tagline description} | flatten # new http get https://www.nushell.sh | query web --query 'h2, h2 + p' | each {str join} | group 2 | each {rotate --ccw tagline description} | flatten ``` to make it behave like before because I thought this one ought to show the same results as before. However, the second reason I changed the 3rd example is because it otherwise panics! If we run the original 3rd example with my modifications, we get a panic ``` thread 'main' panicked at crates/nu-protocol/src/value/record.rs:34:9: assertion `left == right` failed left: 2 right: 17 ``` This happens because `rotate` receives a list of lists where the inner lists have a different number of elements. However this panic is unrelated to the changes I've made, because it can be triggered easily without using the plugin. For instance ```nushell # this is fine [[[one] [two]] [[three] [four]]] | each {rotate --ccw tagline description} # this panics! [[[one] [two]] [[three] [four five]]] | each {rotate --ccw tagline description} ``` Though beyond the scope of this PR, I thought I'd mention this bug since I found it while testing the usage examples. However, I intend to make a proper issue about it tomorrow. # User-Facing Changes <!-- List of all changes that impact the user experience here. This helps us keep track of breaking changes. --> `query web --query "css selector"` now returns a `list<list<string>>` instead of a `list<string>` to make it more in line with [scraper's ElementRef::text()](https://docs.rs/scraper/latest/scraper/element_ref/struct.ElementRef.html#method.text). # Tests + Formatting <!-- Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass (on Windows make sure to [enable developer mode](https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging)) - `cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library > **Note** > from `nushell` you can also use the `toolkit` as follows > ```bash > use toolkit.nu # or use an `env_change` hook to activate it automatically > toolkit check pr > ``` --> I ran `cargo fmt --all -- --check`, `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` and the tests in the plugin. # After Submitting <!-- If your PR had any user-facing changes, update [the documentation](https://github.com/nushell/nushell.github.io) after the PR is merged, if necessary. This will help us keep the docs up to date. --> PR that updates the documentation to match the new 3rd example: https://github.com/nushell/nushell.github.io/pull/1235 |
||
|---|---|---|
| .. | ||
| nu_plugin_custom_values | ||
| nu_plugin_example | ||
| nu_plugin_formats | ||
| nu_plugin_gstat | ||
| nu_plugin_inc | ||
| nu_plugin_python | ||
| nu_plugin_query | ||
| nu-cli | ||
| nu-cmd-base | ||
| nu-cmd-dataframe | ||
| nu-cmd-extra | ||
| nu-cmd-lang | ||
| nu-color-config | ||
| nu-command | ||
| nu-engine | ||
| nu-explore | ||
| nu-glob | ||
| nu-json | ||
| nu-lsp | ||
| nu-parser | ||
| nu-path | ||
| nu-plugin | ||
| nu-pretty-hex | ||
| nu-protocol | ||
| nu-std | ||
| nu-system | ||
| nu-table | ||
| nu-term-grid | ||
| nu-test-support | ||
| nu-utils | ||
| README.md | ||
Nushell core libraries and plugins
These sub-crates form both the foundation for Nu and a set of plugins which extend Nu with additional functionality.
Foundational libraries are split into two kinds of crates:
- Core crates - those crates that work together to build the Nushell language engine
- Support crates - a set of crates that support the engine with additional features like JSON support, ANSI support, and more.
Plugins are likewise also split into two types:
- Core plugins - plugins that provide part of the default experience of Nu, including access to the system properties, processes, and web-connectivity features.
- Extra plugins - these plugins run a wide range of different capabilities like working with different file types, charting, viewing binary data, and more.