Commit Graph

243 Commits

Author SHA1 Message Date
Yehuda Katz
5fbea31d15 Remove unused Display implementations
After the previous commit, nushell uses PrettyDebug and
PrettyDebugWithSource for our pretty-printed display output.

PrettyDebug produces a structured `pretty.rs` document rather than
writing directly into a fmt::Formatter, and types that implement
`PrettyDebug` have a convenience `display` method that produces a string
(to be used in situations where `Display` is needed for compatibility
with other traits, or where simple rendering is appropriate).
2019-11-25 10:07:20 -08:00
Andrés N. Robalino
06857fbc52 Take all rows having the column present. 2019-11-24 04:35:36 -05:00
Andrés N. Robalino
1c830b5c95 default command introduced. 2019-11-24 04:20:08 -05:00
Andrés N. Robalino
bc2d65cd2e Remove raw data debugging. 2019-11-23 19:16:25 -05:00
Andrés N. Robalino
1a0b339897 compact command introduced. 2019-11-23 19:05:44 -05:00
Andrés N. Robalino
f4d9975dab Clean up feature build flags. 2019-11-22 03:11:36 -05:00
Andrés N. Robalino
6e8b768d79 Requiring at least one member is no longer necessary. 2019-11-22 01:18:06 -05:00
Yehuda Katz
cdb0eeafa2 --no-edit 2019-11-21 14:22:32 -08:00
David Mason
b3c021899c combine functions behind to/from-c/tsv commands
fixes #969, admittedly without a --delimiter alias

moves from_structured_data.rs to from_delimited_data.rs to better
identify its scope and adds to_delimited_data.rs. Now csv and tsv both
use the same code, tsv passes in a fixed '\t' argument where csv passes
in the value of --separator
2019-11-19 16:02:35 +00:00
Jonathan Turner
90aeb700ea Add from_xlsx for importing excel files 2019-11-17 16:18:41 +13:00
Thomas Hartmann
1060ba2206 Fixes --headerless functionality for from-ssv.
Squashed commit of the following:

commit fc59d47a2291461d84e0587fc0fe63af0dc26f9f
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Tue Nov 12 15:39:38 2019 +0100

    Fixes inconsistencies in output.

commit da4084e9fdd983557b101207b381e333a443e551
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Tue Nov 12 13:04:10 2019 +0100

    remove unused enum.

commit 7f6a105879c8746786b99fb19bb9f0860c41796a
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Tue Nov 12 12:58:41 2019 +0100

    Starts refactoring from_ssv.

commit b70ddd169ef0c900e03fb590cb171cc7181528db
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Tue Nov 12 11:34:06 2019 +0100

    Fixes --headerless for non-aligned columns.

commit 6332778dd26de8d07be77b291124115141479892
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Tue Nov 12 10:27:35 2019 +0100

    Fixes from-ssv headerless aligned-columns logic.

commit 747d8c812e06349b4a15b8c130721881d86fff98
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Mon Nov 11 23:53:59 2019 +0100

    fixes unit tests for ssv.

commit c77cb451623b37a7a9742c791a4fc38cad053d3d
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Mon Nov 11 22:49:21 2019 +0100

    it compiles! one broken test.

commit 08a05964f56cf92507c255057d0aaf2b6dbb6f45
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Mon Nov 11 18:52:54 2019 +0100

    Backed into a corner. Help.

commit c95ab683025a8007b8a6f8e1659f021a002df584
Author: Thomas Hartmann <thomas.o.hartmann@gmail.com>
Date:   Mon Nov 11 17:30:54 2019 +0100

    broken but on the way
2019-11-12 16:04:55 +01:00
Thomas Hartmann
a3ff5f1246 Updates tests for from tsv, csv, and ssv.
With the proposed changes, these tests now become invalid. If the first line is
to be counted as data, then converting the headers to ints will fail. Removing
the headers and instead treating the first line as data, however, reflects the
new, desired mode of operation.
2019-11-12 16:04:55 +01:00
Andrés N. Robalino
00b3c2036a This is part of on-going work with capabilities when working with
tables and able to work with them for data processing & viewing
purposes. At the moment, certain ways to process said tables we
are able to view a histogram of a given column.

As usage matures, we may find certain core commands that could
be used ergonomically when working with tables on Nu.
2019-11-12 03:39:30 -05:00
Jonathan Turner
62a5250554 Add format command 2019-11-10 13:14:59 +13:00
Jonathan Turner
9043970e97
Merge pull request #943 from drmason13/from_csv-add-separator-arg
Add --separator argument to from_csv
2019-11-09 15:09:32 -08:00
Jonathan Turner
bab58576b4 Rename read to parse 2019-11-10 11:26:44 +13:00
David Mason
15986c598a Add --separator command to from_csv
The command takes a string, checks it is a single character and then
passes it to csv::ReaderBuilder via .delimiter() method as a u8.
2019-11-08 15:06:33 +00:00
Andrés N. Robalino
f966394b63
Merge pull request #888 from andrasio/data-primitives
WIP [data processing]
2019-11-03 16:52:21 -05:00
Andrés N. Robalino
889d2bb378 Isolate feature. 2019-11-03 16:36:47 -05:00
Andrés N. Robalino
d7b768ee9f Fallback internally to String primitives until Member int serialization lands. 2019-11-03 05:38:47 -05:00
Andrés N. Robalino
6ea8e42331 Move column paths to support broader value types. 2019-11-03 05:38:47 -05:00
Andrés N. Robalino
f57489ed92 get command tests already present and move to their own. 2019-11-02 21:05:27 -05:00
Jonathan Turner
e92d4b2ccb Rename add to insert 2019-11-02 14:47:14 +13:00
Andrés N. Robalino
e31ed66610 get :: support fetching rows using numbers in column path. 2019-10-31 14:20:22 -05:00
Andrés N. Robalino
65ae24fbf1 suite in place. 2019-10-31 04:42:18 -05:00
Andrés N. Robalino
b54ce921dd Better error messages. 2019-10-31 04:36:08 -05:00
Jonathan Turner
81691e07c6 Add prepend and append commands 2019-10-30 19:54:06 +13:00
Jonathan Turner
3820fef801 Add a simple read/parse plugin to better handle text data 2019-10-30 11:33:36 +13:00
Andrés N. Robalino
392ff286b2 This commit is ongoing work for making Nu working with data processing
a joy. Fundamentally we embrace functional programming principles for
transforming the dataset from any format picked up by Nu. This table
processing "primitive" commands will build up and make pipelines
composable with data processing capabilities allowing us the valuate,
reduce, and map, the tables as far as even composing this declartively.

On this regard, `split-by` expects some table with grouped data and we
can use it further in interesting ways (Eg. collecting labels for
visualizing the data in charts and/or suit it for a particular chart
of our interest).
2019-10-29 16:04:31 -05:00
Jonathan Turner
571b33a11c
Merge pull request #857 from andrasio/group-by
Can group rows by given column name.
2019-10-23 18:25:52 +13:00
Andrés N. Robalino
f1630da2cc Suggest a column name in case one unknown column is supplied. 2019-10-22 20:10:42 -05:00
Yehuda Katz
6a7c00eaef Finish the job of moving shapes into the stream
This commit should finish the `coloring_in_tokens` feature, which moves
the shape accumulator into the token stream. This allows rollbacks of
the token stream to also roll back any shapes that were added.

This commit also adds a much nicer syntax highlighter trace, which shows
all of the paths the highlighter took to arrive at a particular coloring
output. This change is fairly substantial, but really improves the
understandability of the flow. I intend to update the normal parser with
a similar tracing view.

In general, this change also fleshes out the concept of "atomic" token
stream operations.

A good next step would be to try to make the parser more
error-correcting, using the coloring infrastructure. A follow-up step
would involve merging the parser and highlighter shapes themselves.
2019-10-22 16:19:22 -07:00
Andrés N. Robalino
a317072e4e Cover failure not found files cases. 2019-10-22 08:08:24 -05:00
Andrés N. Robalino
0611f56776 Can group cells by given column name. 2019-10-20 18:42:07 -05:00
notryanb@gmail.com
9eda573a43 filter out the files that have the same size on multiple operating systems 2019-10-18 20:43:37 -04:00
notryanb@gmail.com
4f91d2512a add a test to calculate average of bytes 2019-10-18 20:43:37 -04:00
notryanb@gmail.com
43fbf4345d remove comment and add test for averaging integers 2019-10-18 20:43:37 -04:00
Thomas Hartmann
0d2044e72e Changes flag to minimum-spaces. 2019-10-15 22:05:32 +02:00
Thomas Hartmann
e7b37bee08 Adds filter test for named param. 2019-10-15 20:58:46 +02:00
Andrés N. Robalino
ec2e35ad81 'last' gets last row if no amount desired given. 2019-10-15 05:41:34 -05:00
Andrés N. Robalino
5ed1ed54a6 Move off 'sum' to internal command 'count' for tests. 2019-10-15 05:16:47 -05:00
Andrés N. Robalino
3f60c9d416 'first' gets first row if no amount desired given. 2019-10-15 04:17:55 -05:00
Jonathan Turner
bd6d8189f8
Merge pull request #830 from t-hart/pull-req/from-master
[DRAFT] Adds `from-ssv` command.
2019-10-15 18:28:43 +13:00
Thomas Hartmann
65546646a7 Pull in upstream changes. 2019-10-14 23:05:52 +02:00
Jason Gedge
ee8cd671cb Fix bug with multiple input objects to an external command.
Previously, we would build a command that looked something like this:

  <ex_cmd> "$it" "&&" "<ex_cmd>" "$it"

So that the "&&" and "<ex_cmd>" would also be arguments to the command,
instead of a chained command. This commit builds up a command string
that can be passed to an external shell.
2019-10-14 16:47:12 -04:00
Thomas Hartmann
de1c4e6c88 Implements from-ssv 2019-10-13 22:50:45 +02:00
Thomas Hartmann
648d4865b1 Adds unimplemented module, tests. 2019-10-13 21:15:30 +02:00
Jonathan Turner
193b00764b
Stream support (#812)
* Moves off of draining between filters. Instead, the sink will pull on the stream, and will drain element-wise. This moves the whole stream to being lazy.
* Adds ctrl-c support and connects it into some of the key points where we pull on the stream. If a ctrl-c is detect, we immediately halt pulling on the stream and return to the prompt.
* Moves away from having a SourceMap where anchor locations are stored. Now AnchorLocation is kept directly in the Tag.
* To make this possible, split tag and span. Span is largely used in the parser and is copyable. Tag is now no longer copyable.
2019-10-13 17:12:43 +13:00
Yehuda Katz
1ad9d6f199 Overhaul the expansion system
The main thrust of this (very large) commit is an overhaul of the
expansion system.

The parsing pipeline is:

- Lightly parse the source file for atoms, basic delimiters and pipeline
  structure into a token tree
- Expand the token tree into a HIR (high-level intermediate
  representation) based upon the baseline syntax rules for expressions
  and the syntactic shape of commands.

Somewhat non-traditionally, nu doesn't have an AST at all. It goes
directly from the token tree, which doesn't represent many important
distinctions (like the difference between `hello` and `5KB`) directly
into a high-level representation that doesn't have a direct
correspondence to the source code.

At a high level, nu commands work like macros, in the sense that the
syntactic shape of the invocation of a command depends on the
definition of a command.

However, commands do not have the ability to perform unrestricted
expansions of the token tree. Instead, they describe their arguments in
terms of syntactic shapes, and the expander expands the token tree into
HIR based upon that definition.

For example, the `where` command says that it takes a block as its first
required argument, and the description of the block syntactic shape
expands the syntax `cpu > 10` into HIR that represents
`{ $it.cpu > 10 }`.

This commit overhauls that system so that the syntactic shapes are
described in terms of a few new traits (`ExpandSyntax` and
`ExpandExpression` are the primary ones) that are more composable than
the previous system.

The first big win of this new system is the addition of the `ColumnPath`
shape, which looks like `cpu."max ghz"` or `package.version`.
Previously, while a variable path could look like `$it.cpu."max ghz"`,
the tail of a variable path could not be easily reused in other
contexts. Now, that tail is its own syntactic shape, and it can be used
as part of a command's signature.

This cleans up commands like `inc`, `add` and `edit` as well as
shorthand blocks, which can now look like `| where cpu."max ghz" > 10`
2019-10-10 08:27:51 -07:00
Andrés N. Robalino
837d12decd Filesystem shell can't cd into files. Ever. 2019-09-24 15:34:30 -05:00
Jonathan Turner
5ff94004c6 Add urlencode/urldecode 2019-09-19 16:25:29 +12:00
Andrés N. Robalino
91bea7fb2a
Assert the column is unknown.
did you mean in error messages appear when `get`ing unknown columns. Here we know the column does not exist so we check the exact error message.
2019-09-14 14:53:31 -05:00
Yehuda Katz
19767ad551 Taking another stab at replacing Span with Tag 2019-09-14 11:48:45 -05:00
Jonathan Turner
074a76c9d4 Fix test 2019-09-13 15:48:32 +12:00
Andrés N. Robalino
7838dac689 first and get coverage. 2019-09-12 05:22:58 -05:00
Andrés N. Robalino
c57c0eb371 pass lint checks. 2019-09-12 01:49:01 -05:00
Andrés N. Robalino
b35549adac Removes regex crate dependency. 2019-09-11 22:20:42 -05:00
Maximilian Roos
127381497c
run rustfmt 2019-09-11 10:36:50 -04:00
Andrés N. Robalino
ba8383ae2f to-[csv/tsv] fixes. 2019-09-10 07:00:25 -05:00
Andrés N. Robalino
f61144006f config test harness. 2019-09-10 05:08:01 -05:00
Andrés N. Robalino
f770409a60 cd '-' valueshell implementation and valueshell refactorings. 2019-09-08 05:40:47 -05:00
Andrés N. Robalino
77c2e4200e Filesystem cd refactor/cleanup. 2019-09-08 04:55:49 -05:00
Andrés N. Robalino
df9ff44956 Filesystem change directory coverage. 2019-09-08 03:09:05 -05:00
Andrés N. Robalino
eecda3ecba
Merge pull request #363 from twe4ked/cd-back
Implement `cd -` to return to the previous directory
2019-09-08 01:00:32 -05:00
Odin Dutton
159cf27e39 Implement cd - to return to the last path for the FilesystemShell 2019-09-08 15:10:46 +10:00
Jonathan Turner
9da896ad4e Attempt so simplify classified 2019-09-08 14:00:04 +12:00
Jonathan Turner
84628f298d Finish fixing failing tests. 2019-09-08 13:35:02 +12:00
Patrick Meredith
39fce1191f Fix typo 2019-09-04 13:38:17 -04:00
Patrick Meredith
e8764911cb Add comments for sample.{bson,db} 2019-09-04 13:36:12 -04:00
Jonathan Turner
479f0a566e Covert to_* commands to work on whole table 2019-09-04 18:48:40 +12:00
Patrick Meredith
05e858fa94 Add test 2019-09-03 22:37:26 -04:00
est31
cf0efb811e Trim trailing whitespace and set it in editorconfig 2019-09-03 02:52:52 +02:00
Andrés N. Robalino
9488c41dcd can embed a new field to the table 2019-09-02 00:37:13 -05:00
Andrés N. Robalino
ca0c6eaf58 This commit introduces a basic help feature. We can go to it
with the `help` command to explore and list all commands available.

Enter will also try to see if the location to be entered is an existing
Nu command, if it is it will let you inspect the command under `help`.

This provides baseline needed so we can iterate on it.
2019-08-31 19:06:11 -05:00
Patrick Meredith
b514d93ffd Add Decimal to sample.bson 2019-08-31 19:15:48 +02:00
Patrick Meredith
3d147d1143 Add SQLite support 2019-08-30 20:54:45 -04:00
Andrés N. Robalino
f1e8c433c2 [from/to]tsv support. 2019-08-29 04:02:16 -05:00
Andrés N. Robalino
9810df25b4 Readability improvement. 2019-08-29 01:31:56 -05:00
Andrés N. Robalino
4d2b0f43f5 Fix test. 2019-08-28 21:05:23 -05:00
Andrés N. Robalino
a07817e0e0 cover pipeline helper. 2019-08-28 20:30:51 -05:00
Andrés N. Robalino
55fb1f8dda Yo quiero Playground taconushell. 2019-08-28 19:32:42 -05:00
Yehuda Katz
f82cc4291f Migrate commands_test 2019-08-28 10:58:00 -07:00
Yehuda Katz
abac7cf746 Migrate rm 2019-08-28 10:48:52 -07:00
Yehuda Katz
2c65b2fc2f Migrate mv 2019-08-28 10:28:58 -07:00
Yehuda Katz
21ad06b1e1 Remove unwraps and clean up playground
The original intent of this patch was to remove more unwraps to reduce
panics. I then lost a ton of time to the fact that the playground isn't
in a temp directory (because of permissions issues on Windows).

This commit improves the test facilities to:

- use a tempdir for the playground
- change the playground API so you instantiate it with a block that
  encloses the lifetime of the tempdir
- the block is called with a `dirs` argument that has `dirs.test()` and
  other important directories that we were computing by hand all the time
- the block is also called with a `playground` argument that you can use
  to construct files (it's the same `Playground` as before)
- change the nu! and nu_error! macros to produce output instead of
  taking a variable binding
- change the nu! and nu_error! macros to do the cwd() transformation
  internally
- change the nu! and nu_error! macros to take varargs at the end that
  get interpolated into the running command

I didn't manage to finish porting all of the tests, so a bunch of tests
are currently commented out. That will need to change before we land
this patch.
2019-08-28 10:01:16 -07:00
Andrés N. Robalino
162c8b4274 swept clean quotes 2019-08-27 06:20:22 -05:00
Andrés N. Robalino
b502954558 [from|to]json additions/refactoring. 2019-08-27 06:05:51 -05:00
Jonathan Turner
3750a04cfc
Merge branch 'master' into expand-tilde 2019-08-27 16:23:56 +12:00
Yehuda Katz
34292b282a Add support for ~ expansion
This ended up being a bit of a yak shave. The basic idea in this commit is to
expand `~` in paths, but only in paths.

The way this is accomplished is by doing the expansion inside of the code that
parses literal syntax for `SyntaxType::Path`.

As a quick refresher: every command is entitled to expand its arguments in a
custom way. While this could in theory be used for general-purpose macros,
today the expansion facility is limited to syntactic hints.

For example, the syntax `where cpu > 0` expands under the hood to
`where { $it.cpu > 0 }`. This happens because the first argument to `where`
is defined as a `SyntaxType::Block`, and the parser coerces binary expressions
whose left-hand-side looks like a member into a block when the command is
expecting one.

This is mildly more magical than what most programming languages would do,
but we believe that it makes sense to allow commands to fine-tune the syntax
because of the domain nushell is in (command-line shells).

The syntactic expansions supported by this facility are relatively limited.
For example, we don't allow `$it` to become a bare word, simply because the
command asks for a string in the relevant position. That would quickly
become more confusing than it's worth.

This PR adds a new `SyntaxType` rule: `SyntaxType::Path`. When a command
declares a parameter as a `SyntaxType::Path`, string literals and bare
words passed as an argument to that parameter are processed using the
path expansion rules. Right now, that only means that `~` is expanded into
the home directory, but additional rules are possible in the future.

By restricting this expansion to a syntactic expansion when passed as an
argument to a command expecting a path, we avoid making `~` a generally
reserved character. This will also allow us to give good tab completion
for paths with `~` characters in them when a command is expecting a path.

In order to accomplish the above, this commit changes the parsing functions
to take a `Context` instead of just a `CommandRegistry`. From the perspective
of macro expansion, you can think of the `CommandRegistry` as a dictionary
of in-scope macros, and the `Context` as the compile-time state used in
expansion. This could gain additional functionality over time as we find
more uses for the expansion system.
2019-08-26 21:03:24 -07:00
Patrick Meredith
738675259e Improve test so that it should work on Windows 2019-08-26 21:26:49 -04:00
Patrick Meredith
87a99bbabf Implement to-bson 2019-08-26 20:07:59 -04:00
Dirkjan Ochtman
b77effa434 Fix formatting with cargo fmt 2019-08-26 20:19:05 +02:00
Andrés N. Robalino
8940e57cf3 mark #[test] back. (deleted and reverting it from.. 12e38063) 2019-08-25 21:50:55 -05:00
Andrés N. Robalino
12e3806349
Merge pull request #415 from androbtech/fromto-refact
[from|to]csv additions/refactoring.
2019-08-25 12:56:28 -05:00
Jonathan Turner
dda4a707a7
Merge pull request #421 from ramonsnir/reverse
add reverse
2019-08-26 05:41:07 +12:00
Andrés N. Robalino
0e14ba86ae [from|to]csv additions/refactoring.
Introduced flag to tell `from-to` / `to-csv` whether we want headers parsed and/or written.
2019-08-25 12:32:08 -05:00
Patrick Meredith
3c89cb7e98 Remove test that refuses to pass on Windows (it's just a minor formatting issue) 2019-08-25 12:25:40 -04:00
Ramon Snir
9735c3fcea
add reverse 2019-08-25 12:14:17 -04:00
Patrick Meredith
376809aa2a Normalize strings for bson tests 2019-08-25 11:43:15 -04:00
Patrick Meredith
e9673c31ea Remove redundant test 2019-08-25 09:50:25 -04:00