Doru c602b5a1e8 special-case ExternalStream in bytes starts-with (#8203) ... # Description `bytes starts-with` converts the input into a `Value` before running .starts_with to find if the binary matches. This has two side effects: it makes the code simpler, only dealing in whole values, and simplifying a lot of input pipeline handling and value transforming it would otherwise have to do. _Especially_ in the presence of a cell path to drill into. It also makes buffers the entire input into memory, which can take up a lot of memory when dealing with large files, especially if you only want to check the first few bytes (like for a magic number). This PR adds a special branch on PipelineData::ExternalStream with a streaming version of starts_with. # User-Facing Changes Opening large files and running bytes starts-with on them will not take a long time. # Tests + Formatting Don't forget to add tests that cover your changes. Make sure you've run and fixed any issues with these commands: - `cargo fmt --all -- --check` to check standard code formatting (`cargo fmt --all` applies these changes) - `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect` to check that you're using the standard code style - `cargo test --workspace` to check that all tests pass # Drawbacks Streaming checking is more complicated, and there may be bugs. I tested it with multiple chunks with string data and binary data and it seems to work alright up to 8k and over bytes, though. The existing `operate` method still exists because the way it handles cell paths and values is complicated. This causes some "code duplication", or at least some intent duplication, between the value code and the streaming code. This might be worthwhile considering the performance gains (approaching infinity on larger inputs). Another thing to consider is that my ExternalStream branch considers string data as valid input. The operate branch only parses Binary values, so it would fail. `open` is kind of unpredictable on whether it returns string data or binary data, even when passing `--raw`. I think this can be a problem but not really one I'm trying to tackle in this PR, so, it's worth considering.		2023-02-26 15:17:44 +01:00
..
assignment	allow different types of lists to be appended (#8157)	2023-02-22 12:53:36 +00:00
bytes	special-case ExternalStream in bytes starts-with (#8203)	2023-02-26 15:17:44 +01:00
date	Standardise the use of ShellError::UnsupportedInput and ShellError::TypeMismatch and add spans to every instance of the former (#7217)	2022-12-23 01:48:53 -05:00
hash_	Deprecate hash base64, extend decode and add encode commands (#5863)	2022-06-26 00:35:23 +03:00
math	support multiplication operation on string and list values (#8061)	2023-02-13 16:35:53 +00:00
move_	Fix mv error message issues (arrows, Windows paths) (#7197)	2022-11-23 16:55:13 +13:00
network	Fix typos and use more idiomatic assertions (#7755)	2023-01-15 15:03:32 +13:00
path	Display empty records and lists (#7925)	2023-02-22 16:18:33 +00:00
platform	Validate input range of du flags (#7962)	2023-02-06 21:34:32 +01:00
query	Feature cleanup (#7182)	2022-11-22 16:58:11 -08:00
random	Use imported names in Command::run signatures (#7967)	2023-02-05 22:17:46 +01:00
skip	last, skip, drop, take until, take while, skip until, skip while, where, reverse, shuffle, append, prepend and sort-by raise error when given non-lists (#7623)	2022-12-31 13:35:12 +02:00
str_	Improve error when regex rejects pattern. Resolution of #8037 (#8050)	2023-02-12 13:25:40 +00:00
take	last, skip, drop, take until, take while, skip until, skip while, where, reverse, shuffle, append, prepend and sort-by raise error when given non-lists (#7623)	2022-12-31 13:35:12 +02:00
url	Feat/7725 url join (#7823)	2023-01-22 19:49:40 +01:00
alias.rs	Expand Nushell's help system (#7611)	2022-12-30 17:44:37 +02:00
all.rs	Use with-env to avoid calling external command on invalid command (#8209)	2023-02-25 09:36:51 -08:00
any.rs	Use with-env to avoid calling external command on invalid command (#8209)	2023-02-25 09:36:51 -08:00
append.rs	revert changes on prepend and append (#7660)	2023-01-02 17:09:55 -08:00
break_.rs	New commands: break, continue, return, and loop (#7230)	2022-11-25 09:39:16 +13:00
cal.rs	return Error if get meet nothing and without "i" (#7002)	2022-12-31 13:27:09 +02:00
cd.rs	Re-enable some good tests, remove some bad tests (#7875)	2023-01-28 14:56:47 -06:00
compact.rs	Remove old nushell/merge engine-q	2022-02-07 14:54:06 -05:00
continue_.rs	New commands: break, continue, return, and loop (#7230)	2022-11-25 09:39:16 +13:00
cp.rs	cp progress bar implementation (#8012)	2023-02-22 11:57:38 -08:00
def.rs	fixes a def parsing bug with a default list (#8096)	2023-02-22 12:53:11 +00:00
default.rs	Make get hole errors and cell path hole errors identical (improvement on #7002) (#7647)	2023-01-02 14:45:43 -08:00
do_.rs	Bare word improvements (#8066)	2023-02-16 02:30:56 +00:00
drop.rs	last, skip, drop, take until, take while, skip until, skip while, where, reverse, shuffle, append, prepend and sort-by raise error when given non-lists (#7623)	2022-12-31 13:35:12 +02:00
each.rs	Remove deprecated --numbered flag from four commands (#7777)	2023-02-02 16:59:58 -06:00
echo.rs	Require block params (#4505)	2022-02-17 06:40:24 -05:00
empty.rs	Fix $in in blocks given to any and all (#6951)	2022-11-01 11:36:54 -07:00
enter.rs	Remove old nushell/merge engine-q	2022-02-07 14:54:06 -05:00
error_make.rs	add unspanned flag to error make, add tests (#6017)	2022-07-12 06:03:50 -05:00
every.rs	Another batch of command tests (#4496)	2022-02-16 07:38:02 -05:00
exec.rs	Add "fall-through" signatures (#7527)	2022-12-22 00:33:26 +02:00
export_def.rs	allow -h flags for export subcommands (#6189)	2022-08-02 10:26:16 -05:00
fill.rs	string | fill counts clusters, not graphemes; and doesn't count ANSI escape codes (#8134)	2023-02-20 06:32:20 -06:00
find.rs	Display empty records and lists (#7925)	2023-02-22 16:18:33 +00:00
first.rs	Make stream info visible to users in describe (#7589)	2023-01-03 21:08:05 -08:00
flatten.rs	str collect => str join (#6531)	2022-09-11 11:48:27 +03:00
for_.rs	Fix typos and use more idiomatic assertions (#7755)	2023-01-15 15:03:32 +13:00
format.rs	prevent panic with format command (#7522)	2022-12-19 13:10:02 +01:00
g.rs	Make g - switch to the last used shell (#6249)	2022-08-06 10:11:03 -05:00
get.rs	Make get hole errors and cell path hole errors identical (improvement on #7002) (#7647)	2023-01-02 14:45:43 -08:00
glob.rs	Expose filtering by file type in glob (#7834)	2023-01-28 07:50:12 -06:00
group_by.rs	more test fixes (#4499)	2022-02-16 12:24:45 -06:00
headers.rs	Type validation for headers command (#6918) (#7047)	2022-11-09 16:43:24 -08:00
help.rs	Reorder help <keyword> priority (#7929)	2023-02-05 23:51:09 +01:00
histogram.rs	add quantile column (#5583)	2022-05-18 20:47:26 -05:00
insert.rs	Remove deprecated --numbered flag from four commands (#7777)	2023-02-02 16:59:58 -06:00
into_filesize.rs	fix overflow on negative bytes (#7070)	2022-11-10 22:33:15 +01:00
into_int.rs	Binary into int (#5941)	2022-07-04 06:31:50 +12:00
last.rs	Make stream info visible to users in describe (#7589)	2023-01-03 21:08:05 -08:00
length.rs	Remove old nushell/merge engine-q	2022-02-07 14:54:06 -05:00
let_.rs	check external failed in let assignment (#8164)	2023-02-22 17:35:09 +00:00
lines.rs	Replace row conditions with closures in commands (#7428)	2022-12-10 19:24:06 +02:00
loop_.rs	Fix typos and use more idiomatic assertions (#7755)	2023-01-15 15:03:32 +13:00
ls.rs	Fix typos and use more idiomatic assertions (#7755)	2023-01-15 15:03:32 +13:00
merge.rs	remove block input support in merge (#7177)	2022-11-23 17:01:27 +13:00
mkdir.rs	Filesystem commands print --verbose to stderr (#8014)	2023-02-09 10:29:34 -08:00
mod.rs	special-case ExternalStream in bytes starts-with (#8203)	2023-02-26 15:17:44 +01:00
mut_.rs	Allow $env and mutable records to be mutated by = (closes #7110) (#7318)	2022-12-06 19:51:55 +02:00
n.rs	Refactor shell switching related code (#6258)	2022-08-07 13:30:40 -05:00
nu_check.rs	A set of fixes for stderr redirect (#7219)	2022-11-24 16:58:15 +13:00
open.rs	Move some from xxx commands to plugin (#7942)	2023-02-13 12:42:08 +00:00
p.rs	Refactor shell switching related code (#6258)	2022-08-07 13:30:40 -05:00
parse.rs	allow parse to work better with streams (#7870)	2023-02-08 20:59:02 -06:00
prepend.rs	revert changes on prepend and append (#7660)	2023-01-02 17:09:55 -08:00
print.rs	add -e flag to print, to print the value to stderr (#5935)	2022-07-02 09:54:49 -05:00
range.rs	Use 'table' on scripts and -c commands (#4377)	2022-02-09 05:58:54 -05:00
redirection.rs	Let redirection keep exit code (#7848)	2023-01-30 16:49:31 +01:00
reduce.rs	Test return in reduce closure from #7961 (#7973)	2023-02-06 11:01:08 +01:00
reject.rs	Re-enable some good tests, remove some bad tests (#7875)	2023-01-28 14:56:47 -06:00
rename.rs	fix rename when it is passed an empty column list to rename (#8086)	2023-02-15 21:47:27 +00:00
return_.rs	ensure that when nu evaluates files, it allows early returns (#7415)	2023-02-02 00:02:27 +01:00
reverse.rs	last, skip, drop, take until, take while, skip until, skip while, where, reverse, shuffle, append, prepend and sort-by raise error when given non-lists (#7623)	2022-12-31 13:35:12 +02:00
rm.rs	make cd, cp, ls, mv, open and rm automatically strip ansi codes (#6220)	2022-08-04 06:59:20 -05:00
roll.rs	Remove deprecated --numbered flag from four commands (#7777)	2023-02-02 16:59:58 -06:00
rotate.rs	str collect => str join (#6531)	2022-09-11 11:48:27 +03:00
run_external.rs	pipe binary data to external commands (#8058)	2023-02-24 21:39:52 +01:00
save.rs	allow Range to expand to array-like when converting to json (#8047)	2023-02-24 15:31:33 -06:00
select.rs	Make get hole errors and cell path hole errors identical (improvement on #7002) (#7647)	2023-01-02 14:45:43 -08:00
semicolon.rs	Remove old nushell/merge engine-q	2022-02-07 14:54:06 -05:00
seq_char.rs	Simplify seq char (#7054)	2022-11-09 17:06:47 -08:00
seq.rs	Make stream info visible to users in describe (#7589)	2023-01-03 21:08:05 -08:00
shells.rs	Refactor shell listing related code (#6262)	2022-08-08 06:31:24 -05:00
sort_by.rs	last, skip, drop, take until, take while, skip until, skip while, where, reverse, shuffle, append, prepend and sort-by raise error when given non-lists (#7623)	2022-12-31 13:35:12 +02:00
sort.rs	Add -n flag to sort (formerly only available on sort-by) (#7293)	2022-12-01 07:11:30 -06:00
source_env.rs	ensure that when nu evaluates files, it allows early returns (#7415)	2023-02-02 00:02:27 +01:00
split_by.rs	Remove old nushell/merge engine-q	2022-02-07 14:54:06 -05:00
split_column.rs	refactor: change column names from 'Column*' to 'column*' (#4556)	2022-02-19 19:26:47 -05:00
split_row.rs	Remove old nushell/merge engine-q	2022-02-07 14:54:06 -05:00
table.rs	table --collapse dont do truncation return message instead (#8172)	2023-02-22 18:35:45 +00:00
to_text.rs	Make to text stream ListStreams (#7577)	2022-12-22 16:38:07 -08:00
touch.rs	chore: chrono_update (#7132)	2022-11-27 07:19:02 +13:00
transpose.rs	Prevents duplicate fields in transpose -r (#5840)	2022-06-22 19:19:06 -05:00
try_.rs	fix some typos (#7773)	2023-01-16 12:43:46 +01:00
uniq_by.rs	uniq-by command (#7295)	2022-12-02 11:36:01 +01:00
uniq.rs	uniq -i does not convert to lowercase (#7192) (#7209)	2022-11-23 15:46:20 -08:00
update.rs	Remove deprecated --numbered flag from four commands (#7777)	2023-02-02 16:59:58 -06:00
upsert.rs	Remove deprecated --numbered flag from four commands (#7777)	2023-02-02 16:59:58 -06:00
use_.rs	Allow main command to define top-level module command (#7764)	2023-01-22 21:34:15 +02:00
where_.rs	Remove deprecated --numbered flag from four commands (#7777)	2023-02-02 16:59:58 -06:00
which.rs	Don't assume external ls (#4925)	2022-03-24 16:42:41 +13:00
while_.rs	Fix typos and use more idiomatic assertions (#7755)	2023-01-15 15:03:32 +13:00
with_env.rs	Re-enable some good tests, remove some bad tests (#7875)	2023-01-28 14:56:47 -06:00
wrap.rs	Use 'table' on scripts and -c commands (#4377)	2022-02-09 05:58:54 -05:00
zip.rs	Use variable names directly in the format strings (#7906)	2023-01-29 19:37:54 -06:00