2 Commits

Author	SHA1	Message	Date
Doru	d1a8992590	Initial --params implementation (#12249) ... # Description This PR adds a `--params` param to `query db`. This closes #11643. You can't combine both named and positional parameters, I think this might be a limitation with rusqlite itself. I tried using named parameters with indices like `{ ':named': 123, '1': "positional" }` but that always failed with a rusqlite error. On the flip side, the other way around works: for something like `VALUES (:named, ?)`, you can treat both as positional: `-p [hello 123]`. This PR introduces some very gnarly code repetition in `prepared_statement_to_nu_list`. I tried, I swear; the compiler wasn't having any of it, it kept telling me to box my closures and then it said that the reference lifetimes were incompatible in the match arms. I gave up and put the mapping code in the match itself, but I'm still not happy. Another thing I'm unhappy about: I don't like how you have to put the `:colon` in named parameters. I think nushell should insert it if it's [missing](https://www.sqlite.org/lang_expr.html#parameters). But this is the way [rusqlite works](https://docs.rs/rusqlite/latest/rusqlite/trait.Params.html#example-named), so for now, I'll let it be consistent. Just know that it's not really a blocker, and it isn't a compatibility change to later make `{ colon: 123 }` work, without the quotes and `:`. This would require allocating and turning our pretty little `&str` into a `String`, though # User-Facing Changes Less incentive to leave yourself open to SQL injection with statements like `query db $"INSERT INTO x VALUES \($unsafe_user_input)"`. Additionally, the `$""` syntax being annoying with parentheses plays in our favor, making users even more likely to use ? with `--params`. # Tests + Formatting Hehe	2024-03-24 15:40:21 -05:00
Skyler Hawthorne	7ac3e97bfe	Fix memory consumption of into sqlite (#10232) ... # Description Currently, the `into sqlite` command collects the entire input stream into a single Value, which soaks up the entire input into memory, before it ever tries to write anything to the DB. This is very problematic for large inputs; for example, I tried transforming a multi-gigabyte CSV file into SQLite, and before I knew what was happening, my system's memory was completely exhausted, and I had to hard reboot to recover. This PR fixes this problem by working directly with the pipeline stream, inserting into the DB as values are read from the stream. In order to facilitate working with the stream directly, I introduced a new `Table` struct to store the connection and a few configuration parameters, as well as to make it easier to lazily create the table on the first read value. In addition to the purely functional fixes, a few other changes were made to the serialization and user facing behavior. ### Serialization Much of the preexisting code was focused on generating the exact text needed for a SQL statement. This is unneeded and less safe than using the `rusqlite` crate's serialization for native Rust types along with prepared statements. ### User-Facing Changes Currently, the command is very liberal in the input types it accepts. The strategy is basically if it is a record, try to follow its structure and make an analogous SQL row, which is pretty reasonable. However, when it's not a record, it basically tries to guess what the user wanted and just makes a single column table and serializes the value into that one column, whatever type it may be. This has been changed so that it only accepts records as input. If the user wants to serialize non-record types into SQL, then they must explicitly opt into doing this by constructing a record or table with it first. For a utility for inserting data into SQL, I think it makes more sense to let the user choose how to convert their data, rather than make a choice for them that may surprise them. However, I understand this may be a controversial change. If the maintainers don't agree, I can change this back. #### Long switch names The `file_name` and `table_name` long form switches are currently snake_case and expect to be as such at the command line. These have been changed to kebab-case to be more conventional. # Tests + Formatting To test the memory consumption, I used [this publicly available index of all Wikipedia articles](https://dumps.wikimedia.org/enwiki/20230820/), using the first 10,000, 100,000, and 1,000,000 entries, in that order. I ran the following script to benchmark the changes against the current stable release: ```nu #!/usr/bin/nu # let shellbin = $"($env.HOME)/src/nushell/target/aarch64-linux-android/release/nu" let shellbin = "nu" const dbpath = 'enwiki-index.db' [10000, 100000, 1000000] | each {|rows| rm -f $dbpath; do { time -f '%M %e %U %S' $shellbin -c ( $"bzip2 -cdk ~/enwiki-20230820-pages-articles-multistream-index.txt.bz2 | head -n ($rows) | lines | parse '{offset}:{id}:{title}' | update cells -c [offset, id] { into int } | into sqlite ($dbpath)" ) } | complete | get stderr | str trim | parse '{rss_max} {real} {user} {kernel}' | update cells -c [rss_max] { $"($in)kb" | into filesize } | update cells -c [real, user, kernel] { $"($in)sec" | into duration } | insert rows $rows | roll right } | flatten | to nuon ``` This yields the following results Current stable release: |rows|rss_max|real|user|kernel| |-|-|-|-|-| |10000|53.6 MiB|770ms|460ms|420ms| |100000|209.6 MiB|6sec 940ms|3sec 740ms|4sec 380ms| |1000000|1.7 GiB|1min 8sec 810ms|38sec 690ms|42sec 550ms| This PR: |rows|rss_max|real|user|kernel| |-|-|-|-|-| |10000|38.2 MiB|780ms|440ms|410ms| |100000|39.8 MiB|6sec 450ms|3sec 530ms|4sec 160ms| |1000000|39.8 MiB|1min 3sec 230ms|37sec 440ms|40sec 180ms| # Note I started this branch kind of at the same time as my others, but I understand the feedback that smaller PRs are preferred. Let me know if it would be better to split this up. I do think the scope of the changes are on the bigger side even without the behavior changes I mentioned, so I'm not sure if that will help this particular PR very much, but I'm happy to oblige on request.	2024-01-15 21:41:25 -06:00