Improves startup time when using std-lib (#13842)

Updated summary for commit
[612e0e2](612e0e2160)
- While folks are welcome to read through the entire comments, the core
information is summarized here.

# Description

This PR drastically improves startup times of Nushell by only parsing a
single submodule of the Standard Library that provides the `banner` and
`pwd` commands. All other Standard Library commands and submodules are
parsed when imported by the user. This cuts startup times by more than
60%.

At the moment, we have stopped adding to `std-lib` because every
addition adds a small amount to the Nushell startup time.
With this change, we should once again be able to allow new
functionality to be added to the Standard Library without it impacting
`nu` startup times.

# User-Facing Changes

* Nushell now starts about 60% faster
* Breaking change: The `dirs` (Shells) aliases will return a warning
message that it will not be auto-loaded in the following release, along
with instructions on how to restore it (and disable the message)
* The `use std <submodule> *` syntax is available for convenience, but
should be avoided in scripts as it parses the entire `std` module and
all other submodules and places it in scope. The correct syntax to
*just* load a submodule is `use std/<submodule> *` (asterisk optional).
The slash is important. This will be documented.
* `use std *` can be used for convenience to load all of the library but
still incurs the full loading-time.
* `std/dirs`: Semi-breaking change. The `dirs` command replaces the
`show` command. This is more in line with the directory-stack
functionality found in other shells. Existing users will not be impacted
by this as the alias (`shells`) remains the same.

* Breaking-change: Technically a breaking change, but probably only
impacts maintainers of `std`. The virtual path for the standard library
has changed. It could previously be imported using its virtual path (and
technically, this would have been the correct way to do it):

  ```nu
  use NU_STDLIB_VIRTUAL_DIR/std
  ```

  The path is now simply `std/`:

  ```nu
  use std
  ```

  All submodules have moved accordingly.
  

# Timings

Comparisons below were made:

* In a temporary, clean config directory using `$env.XDG_CONFIG_HOME =
(mktemp -d)`.
* `nu` was run with a release build
* `nu` was run one time to generate the default `config.nu` (etc.) files
- Otherwise timings would include the user-prompt
* The shell was exited and then restarted several times to get timing
samples

(Note: Old timings based on 0.97 rather than 0.98, but in the range of
being accurate)

| Scenario | `$nu.startup-time` |
| --- | --- |
| 0.97.2
([aaaab8e](aaaab8e070))
Without this PR | 23ms - 24ms |
| This PR with deprecated commands | 9ms - <11ms |
| This PR after deprecated commands are removed in following release |
8ms - <10ms |
| Final PR (remove deprecated), using `--no-std-lib` | 6.1ms to 6.4ms |
| Final PR (remove deprecated), using `--no-config-file` | 3.1ms - 3.6ms
|
| Final PR (remove deprecated), using `--no-config-file --no-std-lib` |
1ms - 1.5ms |

*These last two timings point to the opportunity for further
optimization (see comment in thread below (will link once I write it).*

# Implementation details for future maintenance

* `use std banner` is a ridiculously deceptive call. That call parses
and imports *all* of `std` into scope. Simply replacing it with `use
std/core *` is essentially what saves ~14-15ms. This *only* imports the
submodule with the `banner` and `pwd` commands.

* From the code-comments, the reason that `NU_STDLIB_VIRTUAL_DIR` was
used as a prefix was so that there wouldn't be an issue if a user had a
`./std/mod.nu` in the current directory. This does **not** appear to be
an issue. After removing the prefix, I tested with both a relative
module as well as one in the `$env.NU_LIB_DIRS` path, and in all cases
the *internal* `std` still took precedence.

* By removing the prefix, users can now `use std` (and variants) without
requiring that it already be parsed and in scope.

* In the next release, we'll stop autoloading the `dirs` (shells)
functionality. While this only costs an additional 1-1.5ms, I think it's
better moved to the `config.nu` where the user can optionally remove it.
The main reason is its use of aliases (which have also caused issues) -
The `n`, `p`, and `g` short-commands are valuable real-estate, and users
may want to map these to something else.
  
For this release, there's an `deprecated_dirs` module that is still
autoloaded. As with the top-level commands, use of these will give a
deprecation warning with instructions on how to handle going forward.

To help with this, moved the aliases to their own submodule inside the
`dirs` module.

* Also sneaks in a small change where the top-level `dirs` command is
now the replacement for `dirs show`

* Fixed a double-import of `assert` in `dirs.nu`
* The `show_banner` step is replaced with simply `banner` rather than
re-importing it.

* A `virtual_path` may now be referenced with either a forward-slash or
a backward-slash on Windows. This allows `use std/<submodule>` to work
on all platforms.

# Performance side-notes:

* Future parsing and/or IR improvements should improve performance even
further.
* While the existing load time penalty of `std-lib` was not noticeable
on many systems, Nushell runs on a wide-variety of hardware and OS
platforms. Slower platforms will naturally see a bigger jump in
performance here. For users starting multiple Nushell sessions
frequently (e.g., `tmux`, Zellij, `screen`, et. al.) it is recommended
to keep total startup time (including user configuration) under ~250ms.

# Tests + Formatting

* All tests are green

* Updated tests:
- Removed the test that confirmed that `std` was loaded (since we
don't).
- Removed the `shells` test since it is not autoloaded. Main `dirs.nu`
functionality is tested through `stdlib-test`.
- Many tests assumed that the library was fully loaded, because it was
(even though we didn't intend for it to be). Fixed those tests.
- Tests now import only the necessary submodules (e.g., `use
std/assert`, rather than `use std assert`)
- Some tests *thought* they were loading `std/log`, but were doing so
improperly. This was masked by the now-fixed "load-everything-into-scope
bug". Local CI would pass due the `$env.NU_LOG_<...>` variables being
inherited from the calling process, but would fail in the "clean" GitHub
CI environment. These tests have also been fixed.

 * Added additional tests for the changes

# After Submitting

Will update the Standard Library doc page
This commit is contained in:
Douglas 2024-10-03 07:28:22 -04:00 committed by GitHub
parent 157494e803
commit 00709fc5bd
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
28 changed files with 578 additions and 375 deletions

View File

@ -159,7 +159,7 @@ pub fn evaluate_repl(
eval_source(
engine_state,
&mut unique_stack,
r#"use std banner; banner"#.as_bytes(),
r#"banner"#.as_bytes(),
"show_banner",
PipelineData::empty(),
false,

View File

@ -1007,14 +1007,17 @@ impl<'a> StateWorkingSet<'a> {
}
pub fn find_virtual_path(&self, name: &str) -> Option<&VirtualPath> {
// Platform appropriate virtual path (slashes or backslashes)
let virtual_path_name = Path::new(name);
for (virtual_name, virtual_path) in self.delta.virtual_paths.iter().rev() {
if virtual_name == name {
if Path::new(virtual_name) == virtual_path_name {
return Some(virtual_path);
}
}
for (virtual_name, virtual_path) in self.permanent_state.virtual_paths.iter().rev() {
if virtual_name == name {
if Path::new(virtual_name) == virtual_path_name {
return Some(virtual_path);
}
}

View File

@ -9,30 +9,33 @@ use nu_protocol::{
};
use std::path::PathBuf;
// Virtual std directory unlikely to appear in user's file system
const NU_STDLIB_VIRTUAL_DIR: &str = "NU_STDLIB_VIRTUAL_DIR";
pub fn load_standard_library(
engine_state: &mut nu_protocol::engine::EngineState,
) -> Result<(), miette::ErrReport> {
trace!("load_standard_library");
let (block, delta) = {
// Using full virtual path to avoid potential conflicts with user having 'std' directory
// in their working directory.
let std_dir = PathBuf::from(NU_STDLIB_VIRTUAL_DIR).join("std");
let std_dir = PathBuf::from("std");
let mut std_files = vec![
// Loaded at startup
("core", include_str!("../std/core.nu")),
// std module - Loads all commands and submodules
("mod.nu", include_str!("../std/mod.nu")),
("dirs.nu", include_str!("../std/dirs.nu")),
("dt.nu", include_str!("../std/dt.nu")),
("help.nu", include_str!("../std/help.nu")),
("iter.nu", include_str!("../std/iter.nu")),
("log.nu", include_str!("../std/log.nu")),
("assert.nu", include_str!("../std/assert.nu")),
("xml.nu", include_str!("../std/xml.nu")),
("input.nu", include_str!("../std/input.nu")),
("math.nu", include_str!("../std/math.nu")),
("formats.nu", include_str!("../std/formats.nu")),
// std submodules
("assert", include_str!("../std/assert.nu")),
("bench", include_str!("../std/bench.nu")),
("dirs", include_str!("../std/dirs.nu")),
("dt", include_str!("../std/dt.nu")),
("formats", include_str!("../std/formats.nu")),
("help", include_str!("../std/help.nu")),
("input", include_str!("../std/input.nu")),
("iter", include_str!("../std/iter.nu")),
("log", include_str!("../std/log.nu")),
("math", include_str!("../std/math.nu")),
("lib", include_str!("../std/lib.nu")),
("xml", include_str!("../std/xml.nu")),
// Remove in following release
("deprecated_dirs", include_str!("../std/deprecated_dirs.nu")),
];
let mut working_set = StateWorkingSet::new(engine_state);
@ -52,11 +55,9 @@ pub fn load_standard_library(
let std_dir = std_dir.to_string_lossy().to_string();
let source = r#"
# Define the `std` module
module std
# Prelude
use std dirs [
use std/core *
use std/deprecated_dirs [
enter
shells
g
@ -64,14 +65,13 @@ use std dirs [
p
dexit
]
use std pwd
"#;
let _ = working_set.add_virtual_path(std_dir, VirtualPath::Dir(std_virt_paths));
// Add a placeholder file to the stack of files being evaluated.
// The name of this file doesn't matter; it's only there to set the current working directory to NU_STDLIB_VIRTUAL_DIR.
let placeholder = PathBuf::from(NU_STDLIB_VIRTUAL_DIR).join("loading stdlib");
let placeholder = PathBuf::from("load std/core");
working_set.files = FileStack::with_file(placeholder);
let block = parse(

View File

@ -0,0 +1,71 @@
# run a piece of `nushell` code multiple times and measure the time of execution.
#
# this command returns a benchmark report of the following form:
# ```
# record<
# mean: duration
# std: duration
# times: list<duration>
# >
# ```
#
# > **Note**
# > `std bench --pretty` will return a `string`.
#
# # Examples
# measure the performance of simple addition
# > std bench { 1 + 2 } -n 10 | table -e
# ╭───────┬────────────────────╮
# │ mean │ 4µs 956ns │
# │ std │ 4µs 831ns │
# │ │ ╭───┬────────────╮ │
# │ times │ │ 0 │ 19µs 402ns │ │
# │ │ │ 1 │ 4µs 322ns │ │
# │ │ │ 2 │ 3µs 352ns │ │
# │ │ │ 3 │ 2µs 966ns │ │
# │ │ │ 4 │ 3µs │ │
# │ │ │ 5 │ 3µs 86ns │ │
# │ │ │ 6 │ 3µs 84ns │ │
# │ │ │ 7 │ 3µs 604ns │ │
# │ │ │ 8 │ 3µs 98ns │ │
# │ │ │ 9 │ 3µs 653ns │ │
# │ │ ╰───┴────────────╯ │
# ╰───────┴────────────────────╯
#
# get a pretty benchmark report
# > std bench { 1 + 2 } --pretty
# 3µs 125ns +/- 2µs 408ns
export def main [
code: closure # the piece of `nushell` code to measure the performance of
--rounds (-n): int = 50 # the number of benchmark rounds (hopefully the more rounds the less variance)
--verbose (-v) # be more verbose (namely prints the progress)
--pretty # shows the results in human-readable format: "<mean> +/- <stddev>"
] {
let times = (
seq 1 $rounds | each {|i|
if $verbose { print -n $"($i) / ($rounds)\r" }
timeit { do $code } | into int | into float
}
)
if $verbose { print $"($rounds) / ($rounds)" }
let report = {
mean: ($times | math avg | from ns)
min: ($times | math min | from ns)
max: ($times | math max | from ns)
std: ($times | math stddev | from ns)
times: ($times | each { from ns })
}
if $pretty {
$"($report.mean) +/- ($report.std)"
} else {
$report
}
}
# convert an integer amount of nanoseconds to a real duration
def "from ns" [] {
[$in "ns"] | str join | into duration
}

33
crates/nu-std/std/core.nu Normal file
View File

@ -0,0 +1,33 @@
use dt [datetime-diff, pretty-print-duration]
# Print a banner for nushell with information about the project
export def banner [] {
let dt = (datetime-diff (date now) 2019-05-10T09:59:12-07:00)
$"(ansi green) __ ,(ansi reset)
(ansi green) .--\(\)°'.' (ansi reset)Welcome to (ansi green)Nushell(ansi reset),
(ansi green)'|, . ,' (ansi reset)based on the (ansi green)nu(ansi reset) language,
(ansi green) !_-\(_\\ (ansi reset)where all data is structured!
Please join our (ansi purple)Discord(ansi reset) community at (ansi purple)https://discord.gg/NtAbbGn(ansi reset)
Our (ansi green_bold)GitHub(ansi reset) repository is at (ansi green_bold)https://github.com/nushell/nushell(ansi reset)
Our (ansi green)Documentation(ansi reset) is located at (ansi green)https://nushell.sh(ansi reset)
(ansi cyan)Tweet(ansi reset) us at (ansi cyan_bold)@nu_shell(ansi reset)
Learn how to remove this at: (ansi green)https://nushell.sh/book/configuration.html#remove-welcome-message(ansi reset)
It's been this long since (ansi green)Nushell(ansi reset)'s first commit:
(pretty-print-duration $dt)
Startup Time: ($nu.startup-time)
"
}
# Return the current working directory
export def pwd [
--physical (-P) # resolve symbolic links
] {
if $physical {
$env.PWD | path expand
} else {
$env.PWD
}
}

View File

@ -0,0 +1,161 @@
# Maintain a list of working directories and navigate them
# The directory stack.
#
# Exception: the entry for the current directory contains an
# irrelevant value. Instead, the source of truth for the working
# directory is $env.PWD. It has to be this way because cd doesn't
# know about this module.
#
# Example: the following state represents a user-facing directory
# stack of [/a, /var/tmp, /c], and we are currently in /var/tmp .
#
# PWD = /var/tmp
# DIRS_POSITION = 1
# DIRS_LIST = [/a, /b, /c]
#
# This situation could arise if we started with [/a, /b, /c], then
# we changed directories from /b to /var/tmp.
export-env {
$env.DIRS_POSITION = 0
$env.DIRS_LIST = [($env.PWD | path expand)]
}
def deprecation_warning [ ] {
print -e $"
(ansi red)Warning:(ansi reset) The 'std dirs' module will no longer automatically
be loaded in the next release. To continue using the Shells
feature, and to remove this warning, please add the following
to your config.nu:
use std/dirs shells-aliases *
Or see the documentation for more options.
"
}
# Add one or more directories to the list.
# PWD becomes first of the newly added directories.
export def --env add [
...paths: string # directory or directories to add to working list
] {
deprecation_warning
mut abspaths = []
for p in $paths {
let exp = ($p | path expand)
if ($exp | path type) != 'dir' {
let span = (metadata $p).span
error make {msg: "not a directory", label: {text: "not a directory", span: $span } }
}
$abspaths = ($abspaths | append $exp)
}
$env.DIRS_LIST = ($env.DIRS_LIST | insert ($env.DIRS_POSITION + 1) $abspaths | flatten)
_fetch 1
}
export alias enter = add
# Advance to the next directory in the list or wrap to beginning.
export def --env next [
N:int = 1 # number of positions to move.
] {
deprecation_warning
_fetch $N
}
export alias n = next
# Back up to the previous directory or wrap to the end.
export def --env prev [
N:int = 1 # number of positions to move.
] {
deprecation_warning
_fetch (-1 * $N)
}
export alias p = prev
# Drop the current directory from the list, if it's not the only one.
# PWD becomes the next working directory
export def --env drop [] {
deprecation_warning
if ($env.DIRS_LIST | length) > 1 {
$env.DIRS_LIST = ($env.DIRS_LIST | reject $env.DIRS_POSITION)
if ($env.DIRS_POSITION >= ($env.DIRS_LIST | length)) {$env.DIRS_POSITION = 0}
}
# step to previous slot
_fetch -1 --forget_current --always_cd
}
export alias dexit = drop
# Display current working directories.
export def --env show [] {
deprecation_warning
mut out = []
for $p in ($env.DIRS_LIST | enumerate) {
let is_act_slot = $p.index == $env.DIRS_POSITION
$out = ($out | append [
[active, path];
[($is_act_slot),
(if $is_act_slot {$env.PWD} else {$p.item}) # show current PWD in lieu of active slot
]
])
}
$out
}
export alias shells = show
export def --env goto [shell?: int] {
deprecation_warning
if $shell == null {
return (show)
}
if $shell < 0 or $shell >= ($env.DIRS_LIST | length) {
let span = (metadata $shell | get span)
error make {
msg: $"(ansi red_bold)invalid_shell_index(ansi reset)"
label: {
text: $"`shell` should be between 0 and (($env.DIRS_LIST | length) - 1)"
span: $span
}
}
}
_fetch ($shell - $env.DIRS_POSITION)
}
export alias g = goto
# fetch item helper
def --env _fetch [
offset: int, # signed change to position
--forget_current # true to skip saving PWD
--always_cd # true to always cd
] {
if not ($forget_current) {
# first record current working dir in current slot of ring, to track what CD may have done.
$env.DIRS_LIST = ($env.DIRS_LIST | upsert $env.DIRS_POSITION $env.PWD)
}
# figure out which entry to move to
# nushell 'mod' operator is really 'remainder', can return negative values.
# see: https://stackoverflow.com/questions/13683563/whats-the-difference-between-mod-and-remainder
let len = ($env.DIRS_LIST | length)
mut pos = ($env.DIRS_POSITION + $offset) mod $len
if ($pos < 0) { $pos += $len}
# if using a different position in ring, CD there.
if ($always_cd or $pos != $env.DIRS_POSITION) {
$env.DIRS_POSITION = $pos
cd ($env.DIRS_LIST | get $pos )
}
}

View File

@ -22,7 +22,8 @@ export-env {
}
# Add one or more directories to the list.
# PWD becomes first of the newly added directories.
# The first directory listed becomes the new
# active directory.
export def --env add [
...paths: string # directory or directories to add to working list
] {
@ -38,32 +39,33 @@ export def --env add [
$env.DIRS_LIST = ($env.DIRS_LIST | insert ($env.DIRS_POSITION + 1) $abspaths | flatten)
_fetch 1
}
export alias enter = add
# Advance to the next directory in the list or wrap to beginning.
# Make the next directory on the list the active directory.
# If the currenta ctive directory is the last in the list,
# then cycle to the top of the list.
export def --env next [
N:int = 1 # number of positions to move.
] {
_fetch $N
}
export alias n = next
# Back up to the previous directory or wrap to the end.
# Make the previous directory on the list the active directory.
# If the current active directory is the first in the list,
# then cycle to the end of the list.
export def --env prev [
N:int = 1 # number of positions to move.
] {
_fetch (-1 * $N)
}
export alias p = prev
# Drop the current directory from the list, if it's not the only one.
# PWD becomes the next working directory
# Drop the current directory from the list.
# The previous directory in the list becomes
# the new active directory.
#
# If there is only one directory in the list,
# then this command has no effect.
export def --env drop [] {
if ($env.DIRS_LIST | length) > 1 {
$env.DIRS_LIST = ($env.DIRS_LIST | reject $env.DIRS_POSITION)
@ -75,10 +77,8 @@ export def --env drop [] {
}
export alias dexit = drop
# Display current working directories.
export def --env show [] {
# Display current working directories
export def --env main [] {
mut out = []
for $p in ($env.DIRS_LIST | enumerate) {
let is_act_slot = $p.index == $env.DIRS_POSITION
@ -93,29 +93,26 @@ export def --env show [] {
$out
}
export alias shells = show
export def --env goto [shell?: int] {
if $shell == null {
return (show)
# Jump to directory by index
export def --env goto [dir_idx?: int] {
if $dir_idx == null {
return (main)
}
if $shell < 0 or $shell >= ($env.DIRS_LIST | length) {
let span = (metadata $shell | get span)
if $dir_idx < 0 or $dir_idx >= ($env.DIRS_LIST | length) {
let span = (metadata $dir_idx | get span)
error make {
msg: $"(ansi red_bold)invalid_shell_index(ansi reset)"
msg: $"(ansi red_bold)invalid_dirs_index(ansi reset)"
label: {
text: $"`shell` should be between 0 and (($env.DIRS_LIST | length) - 1)"
text: $"`idx` should be between 0 and (($env.DIRS_LIST | length) - 1)"
span: $span
}
}
}
_fetch ($shell - $env.DIRS_POSITION)
_fetch ($dir_idx - $env.DIRS_POSITION)
}
export alias g = goto
# fetch item helper
def --env _fetch [
offset: int, # signed change to position
@ -140,3 +137,12 @@ def --env _fetch [
cd ($env.DIRS_LIST | get $pos )
}
}
export module shells-aliases {
export alias shells = main
export alias enter = add
export alias dexit = drop
export alias p = prev
export alias n = next
export alias g = goto
}

124
crates/nu-std/std/lib.nu Normal file
View File

@ -0,0 +1,124 @@
# Add the given paths to the PATH.
#
# # Example
# - adding some dummy paths to an empty PATH
# ```nushell
# >_ with-env { PATH: [] } {
# std path add "foo"
# std path add "bar" "baz"
# std path add "fooo" --append
#
# assert equal $env.PATH ["bar" "baz" "foo" "fooo"]
#
# print (std path add "returned" --ret)
# }
# ╭───┬──────────╮
# │ 0 │ returned │
# │ 1 │ bar │
# │ 2 │ baz │
# │ 3 │ foo │
# │ 4 │ fooo │
# ╰───┴──────────╯
# ```
# - adding paths based on the operating system
# ```nushell
# >_ std path add {linux: "foo", windows: "bar", darwin: "baz"}
# ```
export def --env "path add" [
--ret (-r) # return $env.PATH, useful in pipelines to avoid scoping.
--append (-a) # append to $env.PATH instead of prepending to.
...paths # the paths to add to $env.PATH.
] {
let span = (metadata $paths).span
let paths = $paths | flatten
if ($paths | is-empty) or ($paths | length) == 0 {
error make {msg: "Empty input", label: {
text: "Provide at least one string or a record",
span: $span
}}
}
let path_name = if "PATH" in $env { "PATH" } else { "Path" }
let paths = $paths | each {|p|
let p = match ($p | describe | str replace --regex '<.*' '') {
"string" => $p,
"record" => { $p | get --ignore-errors $nu.os-info.name },
}
$p | path expand --no-symlink
}
if null in $paths or ($paths | is-empty) {
error make {msg: "Empty input", label: {
text: $"Received a record, that does not contain a ($nu.os-info.name) key",
span: $span
}}
}
load-env {$path_name: (
$env
| get $path_name
| split row (char esep)
| if $append { append $paths } else { prepend $paths }
)}
if $ret {
$env | get $path_name
}
}
# the cute and friendly mascot of Nushell :)
export def ellie [] {
let ellie = [
" __ ,",
" .--()°'.'",
"'|, . ,'",
" !_-(_\\",
]
$ellie | str join "\n" | $"(ansi green)($in)(ansi reset)"
}
# repeat anything a bunch of times, yielding a list of *n* times the input
#
# # Examples
# repeat a string
# > "foo" | std repeat 3 | str join
# "foofoofoo"
export def repeat [
n: int # the number of repetitions, must be positive
]: any -> list<any> {
let item = $in
if $n < 0 {
let span = metadata $n | get span
error make {
msg: $"(ansi red_bold)invalid_argument(ansi reset)"
label: {
text: $"n should be a positive integer, found ($n)"
span: $span
}
}
}
if $n == 0 {
return []
}
1..$n | each { $item }
}
# return a null device file.
#
# # Examples
# run a command and ignore it's stderr output
# > cat xxx.txt e> (null-device)
export def null-device []: nothing -> path {
if ($nu.os-info.name | str downcase) == "windows" {
'\\.\NUL'
} else {
"/dev/null"
}
}

View File

@ -1,247 +1,28 @@
# std.nu, `used` to load all standard library components
export module assert.nu
export module dirs.nu
export module dt.nu
export module formats.nu
export module help.nu
export module input.nu
export module iter.nu
export module log.nu
export module math.nu
export module xml.nu
export-env {
use dirs.nu []
use log.nu []
}
# Top-level commands: ellie, repeat, null-device, and "path add"
export use lib *
use dt.nu [datetime-diff, pretty-print-duration]
# std submodules
export module assert
export module bench
export module dt
export module formats
export module help
export module input
export module iter
export module log
export module math
export module xml
# Add the given paths to the PATH.
#
# # Example
# - adding some dummy paths to an empty PATH
# ```nushell
# >_ with-env { PATH: [] } {
# std path add "foo"
# std path add "bar" "baz"
# std path add "fooo" --append
#
# assert equal $env.PATH ["bar" "baz" "foo" "fooo"]
#
# print (std path add "returned" --ret)
# }
# ╭───┬──────────╮
# │ 0 │ returned │
# │ 1 │ bar │
# │ 2 │ baz │
# │ 3 │ foo │
# │ 4 │ fooo │
# ╰───┴──────────╯
# ```
# - adding paths based on the operating system
# ```nushell
# >_ std path add {linux: "foo", windows: "bar", darwin: "baz"}
# ```
export def --env "path add" [
--ret (-r) # return $env.PATH, useful in pipelines to avoid scoping.
--append (-a) # append to $env.PATH instead of prepending to.
...paths # the paths to add to $env.PATH.
] {
let span = (metadata $paths).span
let paths = $paths | flatten
if ($paths | is-empty) or ($paths | length) == 0 {
error make {msg: "Empty input", label: {
text: "Provide at least one string or a record",
span: $span
}}
}
let path_name = if "PATH" in $env { "PATH" } else { "Path" }
let paths = $paths | each {|p|
let p = match ($p | describe | str replace --regex '<.*' '') {
"string" => $p,
"record" => { $p | get --ignore-errors $nu.os-info.name },
}
$p | path expand --no-symlink
}
if null in $paths or ($paths | is-empty) {
error make {msg: "Empty input", label: {
text: $"Received a record, that does not contain a ($nu.os-info.name) key",
span: $span
}}
}
load-env {$path_name: (
$env
| get $path_name
| split row (char esep)
| if $append { append $paths } else { prepend $paths }
)}
if $ret {
$env | get $path_name
}
}
# convert an integer amount of nanoseconds to a real duration
def "from ns" [] {
[$in "ns"] | str join | into duration
}
# run a piece of `nushell` code multiple times and measure the time of execution.
#
# this command returns a benchmark report of the following form:
# ```
# record<
# mean: duration
# std: duration
# times: list<duration>
# >
# ```
#
# > **Note**
# > `std bench --pretty` will return a `string`.
#
# # Examples
# measure the performance of simple addition
# > std bench { 1 + 2 } -n 10 | table -e
# ╭───────┬────────────────────╮
# │ mean │ 4µs 956ns │
# │ std │ 4µs 831ns │
# │ │ ╭───┬────────────╮ │
# │ times │ │ 0 │ 19µs 402ns │ │
# │ │ │ 1 │ 4µs 322ns │ │
# │ │ │ 2 │ 3µs 352ns │ │
# │ │ │ 3 │ 2µs 966ns │ │
# │ │ │ 4 │ 3µs │ │
# │ │ │ 5 │ 3µs 86ns │ │
# │ │ │ 6 │ 3µs 84ns │ │
# │ │ │ 7 │ 3µs 604ns │ │
# │ │ │ 8 │ 3µs 98ns │ │
# │ │ │ 9 │ 3µs 653ns │ │
# │ │ ╰───┴────────────╯ │
# ╰───────┴────────────────────╯
#
# get a pretty benchmark report
# > std bench { 1 + 2 } --pretty
# 3µs 125ns +/- 2µs 408ns
export def bench [
code: closure # the piece of `nushell` code to measure the performance of
--rounds (-n): int = 50 # the number of benchmark rounds (hopefully the more rounds the less variance)
--verbose (-v) # be more verbose (namely prints the progress)
--pretty # shows the results in human-readable format: "<mean> +/- <stddev>"
] {
let times = (
seq 1 $rounds | each {|i|
if $verbose { print -n $"($i) / ($rounds)\r" }
timeit { do $code } | into int | into float
}
)
if $verbose { print $"($rounds) / ($rounds)" }
let report = {
mean: ($times | math avg | from ns)
min: ($times | math min | from ns)
max: ($times | math max | from ns)
std: ($times | math stddev | from ns)
times: ($times | each { from ns })
}
if $pretty {
$"($report.mean) +/- ($report.std)"
} else {
$report
}
}
# Print a banner for nushell with information about the project
export def banner [] {
let dt = (datetime-diff (date now) 2019-05-10T09:59:12-07:00)
$"(ansi green) __ ,(ansi reset)
(ansi green) .--\(\)°'.' (ansi reset)Welcome to (ansi green)Nushell(ansi reset),
(ansi green)'|, . ,' (ansi reset)based on the (ansi green)nu(ansi reset) language,
(ansi green) !_-\(_\\ (ansi reset)where all data is structured!
Please join our (ansi purple)Discord(ansi reset) community at (ansi purple)https://discord.gg/NtAbbGn(ansi reset)
Our (ansi green_bold)GitHub(ansi reset) repository is at (ansi green_bold)https://github.com/nushell/nushell(ansi reset)
Our (ansi green)Documentation(ansi reset) is located at (ansi green)https://nushell.sh(ansi reset)
(ansi cyan)Tweet(ansi reset) us at (ansi cyan_bold)@nu_shell(ansi reset)
Learn how to remove this at: (ansi green)https://nushell.sh/book/configuration.html#remove-welcome-message(ansi reset)
It's been this long since (ansi green)Nushell(ansi reset)'s first commit:
(pretty-print-duration $dt)
Startup Time: ($nu.startup-time)
"
}
# the cute and friendly mascot of Nushell :)
export def ellie [] {
let ellie = [
" __ ,",
" .--()°'.'",
"'|, . ,'",
" !_-(_\\",
# Load main dirs command and all subcommands
export use dirs main
export module dirs {
export use dirs [
add
drop
next
prev
goto
]
$ellie | str join "\n" | $"(ansi green)($in)(ansi reset)"
}
# Return the current working directory
export def pwd [
--physical (-P) # resolve symbolic links
] {
if $physical {
$env.PWD | path expand
} else {
$env.PWD
}
}
# repeat anything a bunch of times, yielding a list of *n* times the input
#
# # Examples
# repeat a string
# > "foo" | std repeat 3 | str join
# "foofoofoo"
export def repeat [
n: int # the number of repetitions, must be positive
]: any -> list<any> {
let item = $in
if $n < 0 {
let span = metadata $n | get span
error make {
msg: $"(ansi red_bold)invalid_argument(ansi reset)"
label: {
text: $"n should be a positive integer, found ($n)"
span: $span
}
}
}
if $n == 0 {
return []
}
1..$n | each { $item }
}
# return a null device file.
#
# # Examples
# run a command and ignore it's stderr output
# > cat xxx.txt e> (null-device)
export def null-device []: nothing -> path {
if ($nu.os-info.name | str downcase) == "windows" {
'\\.\NUL'
} else {
"/dev/null"
}
}

View File

@ -1,4 +1,8 @@
use std log
use std/log
export-env {
# Place NU_FORMAT... environment variables in module-scope
export use std/log *
}
def "nu-complete threads" [] {
seq 1 (sys cpu | length)

View File

@ -1,4 +1,4 @@
use std *
use std/assert
def run [
system_level,
@ -6,9 +6,9 @@ def run [
--short
] {
if $short {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log ($message_level) --short "test message"'
^$nu.current-exe --no-config-file --commands $'use std; use std/log; NU_LOG_LEVEL=($system_level) log ($message_level) --short "test message"'
} else {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log ($message_level) "test message"'
^$nu.current-exe --no-config-file --commands $'use std; use std/log; NU_LOG_LEVEL=($system_level) log ($message_level) "test message"'
}
| complete | get --ignore-errors stderr
}

View File

@ -1,5 +1,4 @@
use std *
use std log *
use std/assert
use commons.nu *
def run-command [
@ -12,12 +11,12 @@ def run-command [
] {
if ($level_prefix | is-empty) {
if ($ansi | is-empty) {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log custom "($message)" "($format)" ($log_level)'
^$nu.current-exe --no-config-file --commands $'use std/log; NU_LOG_LEVEL=($system_level) log custom "($message)" "($format)" ($log_level)'
} else {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log custom "($message)" "($format)" ($log_level) --ansi "($ansi)"'
^$nu.current-exe --no-config-file --commands $'use std/log; NU_LOG_LEVEL=($system_level) log custom "($message)" "($format)" ($log_level) --ansi "($ansi)"'
}
} else {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log custom "($message)" "($format)" ($log_level) --level-prefix "($level_prefix)" --ansi "($ansi)"'
^$nu.current-exe --no-config-file --commands $'use std/log; NU_LOG_LEVEL=($system_level) log custom "($message)" "($format)" ($log_level) --level-prefix "($level_prefix)" --ansi "($ansi)"'
}
| complete | get --ignore-errors stderr
}
@ -31,6 +30,7 @@ def errors_during_deduction [] {
#[test]
def valid_calls [] {
use std/log *
assert equal (run-command "DEBUG" "msg" "%MSG%" 25 --level-prefix "abc" --ansi (ansi default) | str trim --right) "msg"
assert equal (run-command "DEBUG" "msg" "%LEVEL% %MSG%" 20 | str trim --right) $"((log-prefix).INFO) msg"
assert equal (run-command "DEBUG" "msg" "%LEVEL% %MSG%" --level-prefix "abc" 20 | str trim --right) "abc msg"
@ -39,6 +39,7 @@ def valid_calls [] {
#[test]
def log-level_handling [] {
use std/log *
assert equal (run-command "DEBUG" "msg" "%LEVEL% %MSG%" 20 | str trim --right) $"((log-prefix).INFO) msg"
assert equal (run-command "WARNING" "msg" "%LEVEL% %MSG%" 20 | str trim --right) ""
}

View File

@ -1,5 +1,6 @@
use std *
use std log *
use std/log *
use std/assert
use commons.nu *
def run-command [
@ -10,9 +11,9 @@ def run-command [
--short
] {
if $short {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log ($message_level) --format "($format)" --short "($message)"'
^$nu.current-exe --no-config-file --commands $'use std; use std/log; NU_LOG_LEVEL=($system_level) log ($message_level) --format "($format)" --short "($message)"'
} else {
^$nu.current-exe --no-config-file --commands $'use std; NU_LOG_LEVEL=($system_level) std log ($message_level) --format "($format)" "($message)"'
^$nu.current-exe --no-config-file --commands $'use std; use std/log; NU_LOG_LEVEL=($system_level) log ($message_level) --format "($format)" "($message)"'
}
| complete | get --ignore-errors stderr
}

View File

@ -1,5 +1,6 @@
use std *
use std log *
use std/assert
use std/log
use std/log *
#[test]
def env_log-ansi [] {

View File

@ -1,4 +1,5 @@
use std *
use std/assert
#[test]
def assert_basic [] {

View File

@ -0,0 +1,7 @@
use std/assert
#[test]
def banner [] {
use std/core
assert ((core banner | lines | length) == 15)
}

View File

@ -1,6 +1,5 @@
use std assert
use std assert
use std log
use std/assert
use std/log
# A couple of nuances to understand when testing module that exports environment:
# Each 'use' for that module in the test script will execute the def --env block.
@ -48,7 +47,7 @@ def dirs_command [] {
# must execute these uses for the UOT commands *after* the test and *not* just put them at top of test module.
# the def --env gets messed up
use std dirs
use std/dirs
# Stack: [BASE]
assert equal [$c.base_path] $env.DIRS_LIST "list is just pwd after initialization"
@ -80,7 +79,7 @@ def dirs_command [] {
assert length $env.DIRS_LIST 2 "drop removes from list"
assert equal $env.PWD $c.path_b "drop changes PWD to previous in list (before dropped element)"
assert equal (dirs show) [[active path]; [false $c.base_path] [true $c.path_b]] "show table contains expected information"
assert equal (dirs) [[active path]; [false $c.base_path] [true $c.path_b]] "show table contains expected information"
# Stack becomes: [BASE]
dirs drop
@ -96,7 +95,7 @@ def dirs_next [] {
cd $c.base_path
assert equal $env.PWD $c.base_path "test setup"
use std dirs
use std/dirs
cur_dir_check $c.base_path "use module test setup"
dirs add $c.path_a $c.path_b
@ -117,7 +116,7 @@ def dirs_cd [] {
# must set PWD *before* doing `use` that will run the def --env block in dirs module.
cd $c.base_path
use std dirs
use std/dirs
cur_dir_check $c.base_path "use module test setup"
@ -139,7 +138,7 @@ def dirs_cd [] {
def dirs_goto_bug10696 [] {
let $c = $in
cd $c.base_path
use std dirs
use std/dirs
dirs add $c.path_a
cd $c.path_b
@ -153,7 +152,7 @@ def dirs_goto_bug10696 [] {
def dirs_goto [] {
let $c = $in
cd $c.base_path
use std dirs
use std/dirs
# check that goto can move *from* any position in the ring *to* any other position (correctly)
@ -174,4 +173,7 @@ def dirs_goto [] {
assert equal $env.PWD ($exp_dir | get $other_pos) "goto changed working directory correctly"
}
}
# check that 'dirs goto' with no argument maps to `dirs` (main)
assert length (dirs goto) 3
}

View File

@ -1,5 +1,5 @@
use std assert
use std dt *
use std/assert
use std/dt *
#[test]
def equal_times [] {

View File

@ -1,4 +1,4 @@
use std assert
use std/assert
def test_data_multiline [] {
let lines = [
@ -19,7 +19,7 @@ def test_data_multiline [] {
#[test]
def from_ndjson_multiple_objects [] {
use std formats *
use std/formats *
let result = test_data_multiline | from ndjson
let expect = [{a:1},{a:2},{a:3},{a:4},{a:5},{a:6}]
assert equal $result $expect "could not convert from NDJSON"
@ -27,7 +27,7 @@ def from_ndjson_multiple_objects [] {
#[test]
def from_ndjson_single_object [] {
use std formats *
use std/formats *
let result = '{"a": 1}' | from ndjson
let expect = [{a:1}]
assert equal $result $expect "could not convert from NDJSON"
@ -35,13 +35,13 @@ def from_ndjson_single_object [] {
#[test]
def from_ndjson_invalid_object [] {
use std formats *
use std/formats *
assert error { '{"a":1' | from ndjson }
}
#[test]
def from_jsonl_multiple_objects [] {
use std formats *
use std/formats *
let result = test_data_multiline | from jsonl
let expect = [{a:1},{a:2},{a:3},{a:4},{a:5},{a:6}]
assert equal $result $expect "could not convert from JSONL"
@ -49,7 +49,7 @@ def from_jsonl_multiple_objects [] {
#[test]
def from_jsonl_single_object [] {
use std formats *
use std/formats *
let result = '{"a": 1}' | from jsonl
let expect = [{a:1}]
assert equal $result $expect "could not convert from JSONL"
@ -57,13 +57,13 @@ def from_jsonl_single_object [] {
#[test]
def from_jsonl_invalid_object [] {
use std formats *
use std/formats *
assert error { '{"a":1' | from jsonl }
}
#[test]
def to_ndjson_multiple_objects [] {
use std formats *
use std/formats *
let result = [{a:1},{a:2},{a:3},{a:4},{a:5},{a:6}] | to ndjson | str trim
let expect = test_data_multiline
assert equal $result $expect "could not convert to NDJSON"
@ -71,7 +71,7 @@ def to_ndjson_multiple_objects [] {
#[test]
def to_ndjson_single_object [] {
use std formats *
use std/formats *
let result = [{a:1}] | to ndjson | str trim
let expect = "{\"a\":1}"
assert equal $result $expect "could not convert to NDJSON"
@ -79,7 +79,7 @@ def to_ndjson_single_object [] {
#[test]
def to_jsonl_multiple_objects [] {
use std formats *
use std/formats *
let result = [{a:1},{a:2},{a:3},{a:4},{a:5},{a:6}] | to jsonl | str trim
let expect = test_data_multiline
assert equal $result $expect "could not convert to JSONL"
@ -87,7 +87,7 @@ def to_jsonl_multiple_objects [] {
#[test]
def to_jsonl_single_object [] {
use std formats *
use std/formats *
let result = [{a:1}] | to jsonl | str trim
let expect = "{\"a\":1}"
assert equal $result $expect "could not convert to JSONL"

View File

@ -1,5 +1,5 @@
use std assert
use std help
use std/assert
use std/help
#[test]
def show_help_on_commands [] {

View File

@ -1,4 +1,5 @@
use std *
use std/assert
#[test]
def iter_find [] {

View File

@ -1,8 +1,8 @@
use std
use std/lib
#[test]
def path_add [] {
use std assert
use std/assert
let path_name = if "PATH" in $env { "PATH" } else { "Path" }
@ -11,19 +11,19 @@ def path_add [] {
assert equal (get_path) []
std path add "/foo/"
lib path add "/foo/"
assert equal (get_path) (["/foo/"] | path expand)
std path add "/bar/" "/baz/"
lib path add "/bar/" "/baz/"
assert equal (get_path) (["/bar/", "/baz/", "/foo/"] | path expand)
load-env {$path_name: []}
std path add "foo"
std path add "bar" "baz" --append
lib path add "foo"
lib path add "bar" "baz" --append
assert equal (get_path) (["foo", "bar", "baz"] | path expand)
assert equal (std path add "fooooo" --ret) (["fooooo", "foo", "bar", "baz"] | path expand)
assert equal (lib path add "fooooo" --ret) (["fooooo", "foo", "bar", "baz"] | path expand)
assert equal (get_path) (["fooooo", "foo", "bar", "baz"] | path expand)
load-env {$path_name: []}
@ -35,18 +35,18 @@ def path_add [] {
android: "quux",
}
std path add $target_paths
lib path add $target_paths
assert equal (get_path) ([($target_paths | get $nu.os-info.name)] | path expand)
load-env {$path_name: [$"(["/foo", "/bar"] | path expand | str join (char esep))"]}
std path add "~/foo"
lib path add "~/foo"
assert equal (get_path) (["~/foo", "/foo", "/bar"] | path expand)
}
}
#[test]
def path_add_expand [] {
use std assert
use std/assert
# random paths to avoid collision, especially if left dangling on failure
let real_dir = $nu.temp-path | path join $"real-dir-(random chars)"
@ -63,25 +63,21 @@ def path_add_expand [] {
with-env {$path_name: []} {
def get_path [] { $env | get $path_name }
std path add $link_dir
lib path add $link_dir
assert equal (get_path) ([$link_dir])
}
rm $real_dir $link_dir
}
#[test]
def banner [] {
std assert ((std banner | lines | length) == 15)
}
#[test]
def repeat_things [] {
std assert error { "foo" | std repeat -1 }
use std/assert
assert error { "foo" | lib repeat -1 }
for x in ["foo", [1 2], {a: 1}] {
std assert equal ($x | std repeat 0) []
std assert equal ($x | std repeat 1) [$x]
std assert equal ($x | std repeat 2) [$x $x]
assert equal ($x | lib repeat 0) []
assert equal ($x | lib repeat 1) [$x]
assert equal ($x | lib repeat 2) [$x $x]
}
}

View File

@ -1,5 +1,5 @@
use std log
use std assert
use std/log
use std/assert
#[before-each]
def before-each [] {

View File

@ -0,0 +1,11 @@
use std/assert
export use std *
#[test]
def std_post_import [] {
assert length (scope commands | where name == "path add") 1
assert length (scope commands | where name == "ellie") 1
assert length (scope commands | where name == "repeat") 1
assert length (scope commands | where name == "formats from jsonl") 1
assert length (scope commands | where name == "dt datetime-diff") 1
}

View File

@ -0,0 +1,11 @@
use std/assert
#[test]
def std_pre_import [] {
# These commands shouldn't exist without an import
assert length (scope commands | where name == "path add") 0
assert length (scope commands | where name == "ellie") 0
assert length (scope commands | where name == "repeat") 0
assert length (scope commands | where name == "from jsonl") 0
assert length (scope commands | where name == "datetime-diff") 0
}

View File

@ -1,7 +1,5 @@
use std xml xaccess
use std xml xupdate
use std xml xinsert
use std assert
use std/xml *
use std/assert
#[before-each]
def before-each [] {

View File

@ -1,15 +1,5 @@
use crate::repl::tests::{fail_test, run_test_std, TestResult};
#[test]
fn library_loaded() -> TestResult {
run_test_std("scope modules | where name == 'std' | length", "1")
}
#[test]
fn prelude_loaded() -> TestResult {
run_test_std("shells | length", "1")
}
#[test]
fn not_loaded() -> TestResult {
fail_test("log info", "")
@ -17,5 +7,5 @@ fn not_loaded() -> TestResult {
#[test]
fn use_command() -> TestResult {
run_test_std("use std assert; assert true; print 'it works'", "it works")
run_test_std("use std/assert; assert true; print 'it works'", "it works")
}

View File

@ -224,10 +224,10 @@ fn std_log_env_vars_are_not_overridden() {
("NU_LOG_DATE_FORMAT".to_string(), "%Y".to_string()),
],
r#"
use std
use std/log
print -e $env.NU_LOG_FORMAT
print -e $env.NU_LOG_DATE_FORMAT
std log error "err"
log error "err"
"#
);
assert_eq!(actual.err, "%MSG%\n%Y\nerr\n");
@ -237,7 +237,7 @@ fn std_log_env_vars_are_not_overridden() {
fn std_log_env_vars_have_defaults() {
let actual = nu_with_std!(
r#"
use std
use std/log
print -e $env.NU_LOG_FORMAT
print -e $env.NU_LOG_DATE_FORMAT
"#