Very often we need to work with tables (say extracted from unstructured data or some
kind of final report, timeseries, and the like).
It's inevitable we will be having columns that we can't know beforehand what their names
will be, or how many.
Also, we may end up with certain cells having values we may want to remove as we explore.
Here, `update cells` fundamentally goes over every cell in the table coming in and updates
the cell's contents with the output of the block passed. Basic example here:
```
> [
[ ty1, t2, ty];
[ 1, a, $nothing]
[(wrap), (0..<10), 1Mb]
[ 1s, ({}), 1000000]
[ $true, $false, ([[]])]
] | update cells { describe }
───┬───────────────────────┬───────────────────────────┬──────────
# │ ty1 │ t2 │ ty
───┼───────────────────────┼───────────────────────────┼──────────
0 │ integer │ string │ nothing
1 │ row Column(table of ) │ range[[integer, integer)] │ filesize
2 │ string │ nothing │ integer
3 │ boolean │ boolean │ table of
───┴───────────────────────┴───────────────────────────┴──────────
```
and another one (in the examples) for cases, say we have a timeseries table generated and
we want to remove the zeros and have empty strings and save it out to something like CSV.
```
> [
[2021-04-16, 2021-06-10, 2021-09-18, 2021-10-15, 2021-11-16, 2021-11-17, 2021-11-18];
[ 37, 0, 0, 0, 37, 0, 0]
] | update cells {|value| i
if ($value | into int) == 0 {
""
} {
$value
}
}
───┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────┬────────────
# │ 2021-04-16 │ 2021-06-10 │ 2021-09-18 │ 2021-10-15 │ 2021-11-16 │ 2021-11-17 │ 2021-11-18
───┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────┼────────────
0 │ 37 │ │ │ │ 37 │ │
───┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────┴────────────
```
* Change path join signature
* Appending now works without flag
* Column path operation is behind a -c flag
* Move column path arg retrieval to a function
Also improves errors
* Fix path join tests
* Propagate column path changes to all path commands
* Update path command examples with columns paths
* Modernize path command examples by removing "echo"
* Improve structured path error message
* Fix typo
* Expand path when converting value -> PathBuf
Also includes Tagged<PathBuf>.
Fixes#3605
* Expand path for PATH env. variable
Fixes#1834
* Remove leftover Cows after nu-path refactor
There were some unnecessary Cow conversions leftover from the old
nu-path implementation.
* Use canonicalize in source command; Improve errors
Previously, `source` used `expand_path()` which does not follow
symlinks.
As a follow up, I improved the source error messages so they now tell
why the source file could not be canonicalized or read into string.
```
> [
[ msg, labels, span];
["The message", "Helpful message here", ([[start, end]; [0, 141]])]
] | error make
error: The message
┌─ shell:1:1
│
1 │ ╭ [
2 │ │ [ msg, labels, span];
3 │ │ ["The message", "Helpful message here", ([[start, end]; [0, 141]])]
│ ╰─────────────────────────────────────────────────────────────────────^ Helpful message here
```
Adding a more flexible approach for creating error values. One use case, for instance is the
idea of a test framework. A failed assertion instead of printing to the screen it could create
tables with more details of the failed assertion and pass it to this command for making a full
fledge error that Nu can show. This can (and should) be extended for capturing error values as well
in the pipeline. One could also use it for inspection.
For example: `.... | error inspect { # inspection here }`
or "error handling" as well, like so: `.... | error capture { fix here }`
However, we start here only with `error make` that creates an error value for you with limited support for the time being.
* Add subcommand `into filesize`
It's currently not possible to convert a number or a string containing a number
into a filesize. The only way to create an instance of filesize type today is
with a literal in nushell syntax. This commit adds the `into filesize`
subcommand so that file sizes can be created from the outputs of programs
producing numbers or strings, like standard unix tools.
There is a limitation with this - it doesn't currently parse values like `10 MB`
or `10 MiB`, it can only look at the number itself. If the desire is there, more
flexible parsing can be added.
* fixup! Add subcommand `into filesize`
* fixup! Add subcommand `into filesize`
* feat: spawn the executables directly if possible
This pull request changes nu-command so that it spawns the process directly if:
- They are a `.exe` on Windows
- They are not a `.sh` or `.bash` on not windows.
Benefits:
- As I explained in [this comment](https://github.com/nushell/nushell/issues/3898#issuecomment-894000812), this is another step towards making Nushell a standalone shell, that doesn't need to shell out unless it is running a script for a particular shell (cmd, sh, ps1, etc.).
- Fixes the bug with multiline strings
- Better performance due to direct spawning.
For example, this script shows ~20 ms less latency.
After:
```nu
C:\> benchmark { node -e 'console.log("sssss")' }
───┬──────────────────
# │ real time
───┼──────────────────
0 │ 63ms 921us 600ns
───┴──────────────────
```
Before
```nu
C:\> benchmark { node -e 'console.log("sssss")' }
───┬──────────────────
# │ real time
───┼──────────────────
0 │ 79ms 136us 800ns
───┴──────────────────
```
Fixes#3898
* fix: make which dependency optional
Also fixes clippy warnings
* refactor: refactor spawn_exe, spawn_cmd, spawn_sh, and spawn_any
* fix: use which feature instead of which-support
* fix: use which_in to use the cwd of nu
* fix: use case insensitive comparison of the extensions
Sometimes the case of the extension is uppercased by the "which_in" function
Also use unix instead of not windows. Some os might not have sh support
* Resolve rebase artifacts
* Remove leftover dependencies on removed feature
* Remove unnecessary 'pub'
* Start taking notes and fooling around
* Split canonicalize to two versions; Add TODOs
One that takes `relative_to` and one that doesn't.
More TODO notes.
* Merge absolutize to and rename resolve_dots
* Add custom absolutize fn and use it in path expand
* Convert a couple of dunce::canonicalize to ours
* Update nu-path description
* Replace all canonicalize with nu-path version
* Remove leftover dunce dependencies
* Fix broken autocd with trailing slash
Trailing slash is preserved *only* in paths that do not contain "." or
"..". This should be fixed in the future to cover all paths but for now
it at least covers basic cases.
* Use dunce::canonicalize for canonicalizing
* Alow cd recovery from non-existent cwd
* Disable removed canonicalize functionality tests
Remove unused import
* Break down nu-path into separate modules
* Remove unused public imports
* Remove abundant cow mapping
* Fix clippy warning
* Reformulate old canonicalize tests to expand_path
They wouldn't work with the new canonicalize.
* Canonicalize also ~ and ndots; Unify path joining
Also, add doc comments in nu_path::expansions.
* Add comment
* Avoid expanding ndots if path is not valid UTF-8
With this change, no lossy path->string conversion should happen in the
nu-path crate.
* Fmt
* Slight expand_tilde refactor; Add doc comments
* Start nu-path integration tests
* Add tests TODO
* Fix docstring typo
* Fix some doc strings
* Add README for nu-path crate
* Add a couple of canonicalize tests
* Add nu-path integration tests
* Add trim trailing slashes tests
* Update nu-path dependency
* Remove unused import
* Regenerate lockfile
* Allow different names for ...rest
* Resolves#3945
* This change requires an explicit name for the rest argument in `WholeStreamCommand`,
which is why there are so many changed files.
* Remove redundant clone
* Add tests
* Allow environment variables to be hidden
This change allows environment variables in Nushell to have a value of
`Nothing`, which can be set by the user by passing `$nothing` to
`let-env` and friends.
Environment variables with a value of Nothing behave as if they are not
set at all. This allows a user to shadow the value of an environment
variable in a parent scope, effectively removing it from their current
scope. This was not possible before, because a scope can not affect its
parent scopes.
This is a workaround for issues like #3920.
Additionally, this allows a user to simultaneously set, change and
remove multiple environment variables via `load-env`. Any environment
variables set to $nothing will be hidden and thus act as if they are
removed. This simplifies working with virtual environments, which rely
on setting multiple environment variables, including PATH, to specific
values, and remove/change them on deactivation.
One surprising behavior is that an environment variable set to $nothing
will act as if it is not set when querying it (via $nu.env.X), but it is
still possible to remove it entirely via `unlet-env`. If the same
environment variable is present in the parent scope, the value in the
parent scope will be visible to the user. This might be surprising
behavior to users who are not familiar with the implementation details.
An additional corner case is the the shorthand form of `with-env` does
not work with this feature. Using `X=$nothing` will set $nu.env.X to the
string "$nothing". The long-form works as expected: `with-env [X
$nothing] {...}`.
* Remove unused import
* Allow all primitives to be convert to strings
Given we can write nu scripts. As the codebase grows, splitting into many smaller nu scripts is necessary.
In general, when we work with paths and files we seem to face quite a few difficulties. Here we just tackle one of them and it involves sourcing
files that also source other nu files and so forth. The current working directory becomes important here and being on a different directory
when sourcing scripts will not work. Mostly because we expand the path on the current working directory and parse the files when a source command
call is done.
For the moment, we introduce a `lib_dirs` configuration value and, unfortunately, introduce a new dependency in `nu-parser` (`nu-data`) to get
a handle of the configuration file to retrieve it. This should give clues and ideas as the new parser engine continues (introduce a way to also know paths)
With this PR we can do the following:
Let's assume we want to write a nu library called `my_library`. We will have the code in a directory called `project`: The file structure will looks like this:
```
project/my_library.nu
project/my_library/hello.nu
project/my_library/name.nu
```
This "pattern" works well, that is, when creating a library have a directory named `my_library` and next to it a `my_library.nu` file. Filling them like this:
```
source my_library/hello.nu
source my_library/name.nu
```
```
def hello [] {
"hello world"
}
```
```
def name [] {
"Nu"
end
```
Assuming this `project` directory is stored at `/path/to/lib/project`, we can do:
```
config set lib_dirs ['path/to/lib/project']
```
Given we have this `lib_dirs` configuration value, we can be anywhere while using Nu and do the following:
```
source my_library.nu
echo (hello) (name)
```
* Allow sourcing paths with emojis
* Add source command tests for emoji paths
* Fmt
* Disable source tests on Windows with illegal paths
* Test sourcing also ASCII and single-quoted paths
We introduce it here and allow it to work with regular lists (tables with no columns) as well as symmetric tables. Say we have two lists and wish to zip them, like so:
```
[0 2 4 6 8] | zip {
[1 3 5 7 9]
} | flatten
───┬───
0 │ 0
1 │ 1
2 │ 2
3 │ 3
4 │ 4
5 │ 5
6 │ 6
7 │ 7
8 │ 8
9 │ 9
───┴───
```
In the case for two tables instead:
```
[[symbol]; ['('] ['['] ['{']] | zip {
[[symbol]; [')'] [']'] ['}']]
} | each {
get symbol | $'($in.0)nushell($in.1)'
}
───┬───────────
0 │ (nushell)
1 │ [nushell]
2 │ {nushell}
───┴───────────
```
We very well support `nth 0 2 3 --skip 1 4` to select particular rows and skip some using a flag. However, in practice we deal with tables (whether they come from parsing or loading files and whatnot) where we don't know the size of the table up front (and everytime we have these, they may have different sizes). There are also other use cases when we use intermediate tables during processing and wish to always drop certain rows and **keep the rest**.
Usage:
```
... | drop nth 0
... | drop nth 3 8
```
The nu-serde crate allows us to become much more generic with respect to how we
convert output to `nu-protocol::Value`s. This allows us to remove a lot of the
special-case code that we wrote for deserializing JSON values.
Co-authored-by: Darren Schroeder <343840+fdncred@users.noreply.github.com>
* Refactor Hash code to simplify md5 and sha256 implementations
Md5 and Sha256 (and other future digests) require less boilerplate code
now. Error reporting includues the name of the hash again.
* Add missing hash sha256 test
Select used to ignore absent values resulting in "squashing" where
columns for multiple rows could be squashed into a single row.
This fixes the problem by inserting null/nothing into absent value, thus
preserving the structure of rows. This makes sure that all selected
columns are present in each row.
