* feat: rework record sync for improved reliability
So, to tell a story
1. We introduced the record sync, intended to be the new algorithm to
sync history.
2. On top of this, I added the KV store. This was intended as a simple
test of the record sync, and to see if people wanted that sort of
functionality
3. History remained syncing via the old means, as while it had issues it
worked more-or-less OK. And we are aware of its flaws
4. If KV syncing worked ok, history would be moved across
KV syncing ran ok for 6mo or so, so I started to move across history.
For several weeks, I ran a local fork of Atuin + the server that synced
via records instead.
The record store maintained ordering via a linked list, which was a
mistake. It performed well in testing, but was really difficult to debug
and reason about. So when a few small sync issues occured, they took an
extremely long time to debug.
This PR is huge, which I regret. It involves replacing the "parent"
relationship that records once had (pointing to the previous record)
with a simple index (generally referred to as idx). This also means we
had to change the recordindex, which referenced "tails". Tails were the
last item in the chain.
Now that we use an "array" vs linked list, that logic was also replaced.
And is much simpler :D
Same for the queries that act on this data.
----
This isn't final - we still need to add
1. Proper server/client error handling, which has been lacking for a
while
2. The actual history implementation on top
This exists in a branch, just without deletions. Won't be much to
add that, I just don't want to make this any larger than it already
is
The _only_ caveat here is that we basically lose data synced via the old
record store. This is the KV data from before.
It hasn't been deleted or anything, just no longer hooked up. So it's
totally possible to write a migration script. I just need to do that.
* update .gitignore
* use correct endpoint
* fix for stores with length of 1
* use create/delete enum for history store
* lint, remove unneeded host_id
* remove prints
* add command to import old history
* add enable/disable switch for record sync
* add record sync to auto sync
* satisfy the almighty clippy
* remove file that I did not mean to commit
* feedback
* Add record migration
* Add database functions for inserting history
No real tests yet :( I would like to avoid running postgres lol
* Add index handler, use UUIDs not strings
* Fix a bunch of tests, remove Option<Uuid>
* Add tests, all passing
* Working upload sync
* Record downloading works
* Sync download works
* Don't waste requests
* Use a page size for uploads, make it variable later
* Aaaaaand they're encrypted now too
* Add cek
* Allow reading tail across hosts
* Revert "Allow reading tail across hosts"
Not like that
This reverts commit 7b0c72e7e0.
* Handle multiple shards properly
* format
* Format and make clippy happy
* use some fancy types (#1098)
* use some fancy types
* fmt
* Goodbye horrible tuple
* Update atuin-server-postgres/migrations/20230623070418_records.sql
Co-authored-by: Conrad Ludgate <conradludgate@gmail.com>
* fmt
* Sort tests too because time sucks
* fix features
---------
Co-authored-by: Conrad Ludgate <conradludgate@gmail.com>
* [feature] store env variables in History records
WIP: remove `HistoryWithoutDelete`, add some docstrings, tests
* Create History objects through builders.
Assure in compile-time that all required fields
are set for the given construction scenario
* (from #882) split Cmd::run into subfns
* Update `History` doc
* remove rmp-serde from history
* update warning
---------
Co-authored-by: Conrad Ludgate <conrad.ludgate@truelayer.com>
* wip
* Start testing
* Store host IDs, not hostnames
Why? Hostnames can change a lot, and therefore host filtering can be
funky. Really, all we want is a unique ID per machine + do not care what
it might be.
* Mostly just write a fuckload of tests
* Add a v0 kv store I can push to
* Appending works
* Add next() and iterate, test the pointer chain
* Fix sig
* Make clippy happy and thaw the ICE
* Fix tests'
* Fix tests
* typed builder and cleaner db trait
---------
Co-authored-by: Conrad Ludgate <conrad.ludgate@truelayer.com>
* Add `delete_at` is null condition during search
Since entries are searched everytime a delete process ends, the
--delete-it-all command will enter an infinite loop if searching the
whole history.
* Remove command blanking
Command blanking may violate the `unique(timestamp, cwd, command)`
condition.
* Overwrite command with random string when deleting
* Add rand dependency to client crate
---------
Co-authored-by: Ellie Huxtable <ellie@elliehuxtable.com>
* Try different workspace structure
Move main crate (atuin) to be on the same level with other crates in
this workspace
* extract common dependencies to the workspace definition
* fix base64 v0.21 deprecation warning
* questionable: update deps & fix chrono deprecations
possible panic sites are unchanged, they're just more visible now
* Revert "questionable: update deps & fix chrono deprecations"
This reverts commit 993e60f8de.
Adds a new `history_filter` setting through which users can specify a
list of regular expressions that match commands that should not be
recorded in the history.
* Allow overriding filter and search modes from CLI arguments
* Use session filter mode for bash up key binding
* We precisely do not want to add quotes here so that all arguments are properly passed
* Add --shell-up-key-binding hidden command argument and filter_mode_shell_up_key_binding configuration option to allow customizing the filter mode used when atuin is invoked from a shell up-key binding
* Correct up binding for zsh
Co-authored-by: Ellie Huxtable <ellie@elliehuxtable.com>
06ac9587 Show current version on server index (#436)
706b1aff Disable ARM docker builds (#438)
f2abc23a Update README.md
3c2b0550 Noyez fix dir hostname utf8 (#430)
3f5350de [feature] Add scroll wheel support to interactive history search (#435)
dcdde225 Fix text outline for dark mode
9ac0c60c Implement cursor (#412)
119ab9e0 Adds password prompt for register and login (#424)
e5df809d Noyez zsh histdb import (#393)
b08e2543 Improve default fish keybindings (#420)
4096c6ee Update README.md
cd2a3ab7 Add fish shell to key binding docs (#418)
b2782110 Bump clap_complete from 3.1.3 to 3.1.4 (#397)
ee66c0a1 Bump axum from 0.5.5 to 0.5.6 (#415)
4297e263 Bump tokio from 1.18.1 to 1.18.2 (#396)
dbd9ca53 Bump clap from 3.1.16 to 3.1.18 (#401)
a7c9d19d Bump tower-http from 0.3.2 to 0.3.3 (#399)
3b79f686 Bump axum from 0.5.4 to 0.5.5 (#402)
f3407710 Cleanup dependencies – disable unnecessary or unused features (#407)
ab294cde Don't pollute shell environment - remove 'id' variable (#408)
14b30606 Allow to build atuin server without client (#404)
5e4e8d11 Don't create config dir for server in default location if not needed (#406)
b7946cc9 Update Chinese version README.md (#403)
e0291f67 Update README.md
301190e4 Build ARM docker image in GitHub Actions using QEMU (#400)
1d030b9d Importer V3 (#395)
d3a4ff95 Bump clap from 3.1.15 to 3.1.16 (#392)
e9d2ec4b Add ctrl-k and ctrl-j for up and down (#394)
25afb5b2 Bump serde_json from 1.0.80 to 1.0.81 (#387)
4a839dab Adds stats summary (#384)
7a394b01 Bump serde from 1.0.136 to 1.0.137 (#375)
edd3f812 Bump clap_complete from 3.1.2 to 3.1.3 (#377)
d85d03d9 Bump log from 0.4.16 to 0.4.17 (#382)
dc3b7ef5 Bump tokio from 1.18.0 to 1.18.1 (#383)
12440c1c Bump serde_json from 1.0.79 to 1.0.80 (#376)
731042f4 Bump tower-http from 0.3.1 to 0.3.2 (#378)
82505e61 Bump clap from 3.1.12 to 3.1.15 (#381)
e05c19d0 Add Chinese documentation translation & Fix spelling mistakes (#373)
6e280e25 Add Russian documentation translation (#365)
40efdd11 Bump http from 0.2.6 to 0.2.7 (#368)
8bc5becc Bump tower-http from 0.3.0 to 0.3.1 (#367)
172ac8db Create FUNDING.yml
7cdd00b5 Bump tokio from 1.17.0 to 1.18.0 (#357)
9d2e9ea1 Search: Allow specifiying the limited of returned entries (#364)
93ab4e78 ignore JetBrains IDEs, tidy-up imports (#348)
2cb4cb39 Bump axum from 0.5.3 to 0.5.4 (#355)
796644e2 Add created_at column to users (#354)
f8233bcb SQLx cannot run this migration OK (#353)
d8ef5dd9 fix db range query (#351)
5926ea64 fix import auto for bash (#352)
43d299fd bump tui (#346)
8ac6571b Remove all select * from the server queries (#347)
4030de4b Add btree index on history table (#345)
b692e0ce Bump tower-http from 0.2.5 to 0.3.0 (#343)
3680f4ac Bump clap from 3.1.11 to 3.1.12 (#342)
7f5310a1 history list (#340)
* Attempting to implement zsh-histdb import
Import compiles passes tests, but doesn't run b/c of async runtime.
zsh-histdb uses sqlite, and sqlx-rs is async, but import code is sync.
* More working on importing histdb
* Rewriting tests and using `Vec<u8>` instead of `String`
- Rewriting tests to eliminate depencency on local file system
- Using `Vec<u8>` for command strings instead of `String` to eliminate
the utf8 errors i was seeing previously. Seems to be working.
* Running fmt
Co-authored-by: Bradley Noyes <b@noyes.dev>
* Disable unnecessary features of config crate, keep only toml
The atuin documentation only mentions the TOML format for configuration
files, but the config crate supports many other formats by default,
which increases the binary size by about 300 kiB.
* Disable unused features of cli-table crate
This doesn't seem to have noticeable effect on the binary size, but
disabling them cleans up the dependency tree a bit.
a050faa Use `--locked` flag for CI builds (#337)
8737474 Update Cargo.lock (#336)
eab1dbf sql builder (#333)
Fixed to include an updated lockfile that was missed in the last release
Thank you for the rapid report @orhun!
