David Dworken
1d878195b2
Rework ff98a7907c to use the newly added EntryId column rather than deleting based on the start time
2023-09-22 14:03:41 -07:00
David Dworken
9b847c5e35
Further optimize client-server roundtrips by including deletion and dump requests in submit responses (follow up to 1e43de689f)
2023-09-22 13:49:29 -07:00
David Dworken
2a5a6d65c4
Roll-forward "Add preliminary support for persisting pre-saved history entries remotely"
...
This rolls-forward commit 66916c27cb
2023-09-22 13:13:46 -07:00
David Dworken
66916c27cb
Revert "Add preliminary support for persisting pre-saved history entries remotely"
...
This reverts commit ff98a7907c
2023-09-21 12:39:20 -07:00
David Dworken
ff98a7907c
Add preliminary support for persisting pre-saved history entries remotely
2023-09-21 12:39:04 -07:00
David Dworken
a3b865fa6b
Move all handlers into either api_handlers.go for client-exposed handlers, or debug_handlers.go for debug/admin functionality
2023-09-21 11:47:16 -07:00
David Dworken
9fc1140ab9
Swap json decoding to Decoder API for shorter/clearer code
2023-09-21 11:42:02 -07:00
David Dworken
1e43de689f
Optimize number of round-trip HTTP connections made by the client by having the submit handler return metadata about whether there are pending dump/deletion requests
...
For now, I'm still keeping the dedicated endpoints for those functionalities, but since most of the time there are no dump/deletion requests this should cut down the number of requests made by the client by 2/3.
2023-09-21 11:35:24 -07:00
David Dworken
f64362fbba
Fix metrics tag format so handler tags are properly exported to datadog
2023-09-17 22:46:31 -07:00
David Dworken
1ab61f1d7d
Fix missing err != nil branch
2023-09-17 13:07:38 -07:00
David Dworken
d94e1bfa22
Fix imports after moving backend/server code in 772aaf2453
2023-09-17 12:53:55 -07:00
David Dworken
772aaf2453
Move internal/ directory into backend/server since it is all server-specific internals
2023-09-17 12:51:21 -07:00
David Dworken
04a0fc40cd
Make errors from updateUsageData(...) crash when running in non-prod to ensure they're detected/handled before deployment
2023-09-17 12:03:42 -07:00
David Dworken
b733878878
Skip re-initializing tables when running in prod to make server startup faster
2023-09-15 17:51:14 -07:00
David Dworken
63741a0d14
Optimize docker build efficiency by running apk add early so that it is cached
2023-09-15 17:10:55 -07:00
David Dworken
1036e14c17
Add debug prints to debug server startup
2023-09-15 17:07:25 -07:00
David Dworken
a66ea1387d
Refactor server.go to remove two global variables
2023-09-13 21:47:06 -07:00
Sergio Moura
a8360efa67
revert main.ReleaseVersion changes
2023-09-12 15:55:41 -04:00
Sergio Moura
0d30011a33
break down release versions and fix server tests
2023-09-12 15:44:03 -04:00
Sergio Moura
60a0e20dd9
extract server object to its own package
2023-09-12 15:44:03 -04:00
Sergio Moura
02b1e8287d
isolate all server handlers into a single struct, without using global variables
2023-09-12 15:44:01 -04:00
David Dworken
50c74e5881
Rename a bunch of DB functions + add error checking for DB table creation (follow up to #104 )
2023-09-12 08:30:43 -07:00
Sergio Moura
eaa837d17d
fix check for empty array result (II)
2023-09-11 16:37:55 -04:00
Sergio Moura
ff2ac2194d
fix %w
2023-09-11 13:03:29 -04:00
Sergio Moura
f9ee8ed189
use DateOnly as local constant instead of using the imported time package
2023-09-11 10:15:42 -04:00
Sergio Moura
3c18f62d99
finishing removing direct DB instructions from http handlers
2023-09-08 10:58:01 -04:00
Sergio Moura
0d6aa081d8
remove direct db instructions from apiRegisterHandler and statsHandler
2023-09-08 10:58:01 -04:00
Sergio Moura
382f234c09
remove usage_data related queries from the main package into a database package
2023-09-08 10:58:01 -04:00
Sergio Moura
a3e6d5a940
docker improvements:
...
* use alpine for small docker images
* use docker compose to wait for the database to be ready
* use a single Dockerfile with parameters to determine the correct GOARCH
* update makefile to account for the new GOARCH parameter on the Dockerfile
* remove native-arch-Dockerfile
2023-09-07 11:06:32 -04:00
David Dworken
aa3c7fc41a
Follow up to #103 : assign err from fmt.Fprintf to _ to ensure go-errcheck doesn't flag it
2023-09-07 07:57:20 -07:00
David Dworken
68e3a813c9
Follow up to #103 : pull context from r.Context() when used rather than at the start of functions
2023-09-07 07:56:03 -07:00
Sergio Moura
e6d922709d
fix tests
2023-09-07 09:50:36 -04:00
Sergio Moura
37670230af
Merge remote-tracking branch 'origin/master' into sergio/handlers
2023-09-07 09:33:42 -04:00
Sergio Moura
589b99e500
do not use http.StatusNoContent
2023-09-07 08:34:21 -04:00
David Dworken
2a95553fdf
Fix install.py script for arm7 so installs work on raspberry pis
2023-09-06 21:35:28 -07:00
Sergio Moura
2b1ba7e3ba
use single context and always return a status to the client
...
api handlers do not need an extra context. http.Request already has a context that is being ignored, so we leverage it and stop creating a new one.
make the endpoints return http.StatusNoContent instead of just closing the connection from the client.
2023-09-06 11:37:14 -04:00
Sergio Moura
83ad8c7b1f
wrap errors with %w instead of using %v
2023-09-05 15:08:55 -04:00
David Dworken
f1aefdbf30
Fix download URL for arm7
2023-02-18 22:29:01 -08:00
David Dworken
27cd544524
Add support for arm7 for #63 to support raspberry pis
2023-02-18 22:00:39 -08:00
David Dworken
8dc5168c64
Set max open conns to 1 for test code to fix race conditions that lead to flaky tests
2023-02-13 20:50:33 -08:00
David Dworken
f502cbee1d
Fix memory leak caused by not closing resp.Body ( #62 ) and also fix the server to work when the DD integration is disabled
2022-12-23 17:17:44 -08:00
David Dworken
06a4bf0bcb
Add code to deep clean the DB to remove entries from people with 1 device that haven't been active in at least 90 days
2022-12-17 21:27:00 -08:00
David Dworken
956377d472
linux arm is called aarch64, see #48
2022-12-12 19:45:54 -08:00
David Dworken
83667a52f4
Fix #51 by changing if --> elif
2022-12-12 19:44:02 -08:00
David Dworken
7c86b812bf
Add support for linux arm64 for #48
2022-12-11 20:39:45 -08:00
David Dworken
857e423e10
Fix linter errors + some general clean up
2022-12-11 20:31:50 -08:00
David Dworken
9ed325e0a5
Add support for limiting the number of registrations to fix #46
2022-12-11 19:42:51 -08:00
David Dworken
35444bf56e
Swap ioutil to non-deprecated alternatives + clean up pre-commit errors
2022-11-27 11:59:06 -08:00
David Dworken
3985010a17
Add test for the healthcheck endpoint
2022-11-26 22:53:14 -08:00
David Dworken
22330f3ee1
Ensure that tests assert a lack of leaked connections
2022-11-26 22:40:43 -08:00
David Dworken
f77eb38618
Rename misleading var
2022-11-26 22:16:11 -08:00
David Dworken
2ba535b650
Call Close() on sql.Rows to prevent connection leak
2022-11-26 22:15:16 -08:00
David Dworken
4e84f49de9
Revert "Move usage data into tx to optimize updating usage data"
...
This reverts commit aa501262e6
2022-11-26 18:59:32 -08:00
David Dworken
aa501262e6
Move usage data into tx to optimize updating usage data
2022-11-26 18:46:20 -08:00
David Dworken
865ce06b95
Optimize healthcheck handler to avoid unnecessary ping in prod + add prod-only config for connection reuse
2022-11-26 18:33:54 -08:00
David Dworken
a98bff0db8
Optimize query latency by moving the read count incrementing to a background task
2022-11-26 12:10:18 -08:00
David Dworken
c603537137
Optimize the apiSubmit function to ensure that it doesn't lead to errors from large insertions
2022-11-26 10:31:43 -08:00
David Dworken
8d87110405
Optimize the healthcheck analyzer to avoid a full table scan
2022-11-25 20:35:17 -08:00
David Dworken
1ce20157c7
Add datadog integration for monitoring the server
2022-11-25 20:04:40 -08:00
David Dworken
8dd9c1d9e4
Create the feedback table in pg
2022-11-20 00:12:41 -08:00
David Dworken
97d7c7e310
Add last registration date to internal status page
2022-11-19 09:41:40 -08:00
David Dworken
0bd9249df3
Update the healthcheck endpoint so it also writes to the DB to detect future cases of #38
2022-11-17 23:43:25 -08:00
David Dworken
d4efc62df1
Add client-side code to ask for feedback when uninstalling
2022-11-16 21:54:05 -08:00
David Dworken
48c77d88f7
Add endpoint to handle feedback reports
2022-11-16 21:42:07 -08:00
David Dworken
78d33cf437
Make the healthcheck endpoint work for non-global production instance usages to fix #35
2022-11-16 20:58:19 -08:00
David Dworken
fd3d932e9d
Document that the server can use sqlite
2022-11-16 20:54:47 -08:00
David Dworken
4aacd8b5e2
Merge pull request #31 from matthewcmead/sqlite_for_server
...
allow for sqlite backend database for low-profile self-hosting deploy…
2022-11-16 20:50:34 -08:00
David Dworken
35208680d8
Improve debug logs in server.go
2022-11-16 20:26:44 -08:00
m
77af69de86
allow for sqlite backend database for low-profile self-hosting deployments
2022-11-14 11:35:05 -05:00
David Dworken
3e0ed67518
Serve a HTML redirect
2022-11-06 14:59:00 -08:00
David Dworken
9cc485f3b1
Remove the redirect since it breaks the install.py
2022-11-06 14:43:49 -08:00
David Dworken
7ceb303caf
Delete the landing page and instead serve a redirect to the github repo
2022-11-06 14:33:28 -08:00
David Dworken
c2802f40dc
Use wait-for instead of a sleep for the dockerfile for self hosting
2022-11-05 13:46:08 -07:00
David Dworken
9ddae45b03
Add fully tested instructions for self hosting
2022-11-05 13:30:37 -07:00
David Dworken
36f776053b
Add docker-compose file for self hosting to fix #7
2022-11-04 23:41:56 -07:00
David Dworken
846f256fa4
Support a TMPDIR variable in case /tmp/ is mounted noexec
2022-11-03 18:04:16 -07:00
David Dworken
436d432065
Add warning message for the buggy slow version of hishtory
2022-11-02 20:52:37 -07:00
David Dworken
cc2a6046d3
Update demo gif to show search atoms
2022-11-01 22:39:54 -07:00
David Dworken
31ef7a2fdd
Better timing for the demo gif
2022-11-01 22:33:48 -07:00
David Dworken
24be30d1e1
Slow down gif
2022-11-01 19:13:47 -07:00
David Dworken
67bb2e979f
Add demo gif showing the control-r integration
2022-11-01 15:39:46 -07:00
David Dworken
4f708aa58c
Remove hard coded pg password and pull the new rotated password from an environment variable
2022-11-01 12:14:20 -07:00
David Dworken
19fbeda539
Set a correct version number for builds containing slsa-verifier <1.3.1
2022-11-01 11:16:23 -07:00
David Dworken
2efa93762a
Another attempt at getting github actions to pass
2022-11-01 11:03:49 -07:00
David Dworken
d2558c8fb5
Skip slsa updates only for the orphaned versions
2022-11-01 10:51:08 -07:00
David Dworken
2183a63f8f
Add an endpoint that is hit to determine if slsa is currently broken
2022-10-31 17:32:27 -07:00
David Dworken
cdc5421a7b
Move testutils to a separate package so as to move test-only code out of the main binary
2022-10-27 21:53:47 -07:00
David Dworken
b4936d588c
Add a very basic healthcheck handler to the backend
2022-10-23 17:54:07 -07:00
David Dworken
70c02bf0ed
Remove basic auth since that is now done by the LB
2022-10-22 11:08:33 -07:00
David Dworken
6b2e9059a4
Optimize the submit endpoint to only use 1 query and 1 insert rather than N
2022-10-18 14:52:35 -07:00
David Dworken
87d3dc4d97
Log the actual remote addr in log lines
2022-10-18 14:35:03 -07:00
David Dworken
ad809efef6
Migrate hishtory from k8s to docker compose
2022-10-18 11:53:35 -07:00
David Dworken
ab95b16ea2
Update comment
2022-10-17 20:38:40 -07:00
David Dworken
0da8021ab7
HTML escape the forced banner to prevent XSS (though this domain has no cookies so there would have been very limited harm from an XSS)
2022-10-15 09:58:47 -07:00
David Dworken
a40e44c985
Add explanation of why expectedPasswordHash is ok
2022-10-15 09:26:54 -07:00
David Dworken
38c185c9f9
Add environ variable to configure the postgres DB
2022-10-15 09:21:10 -07:00
David Dworken
68416f9d0f
Better string formatting for the internal analytics endpoint
2022-10-04 12:58:00 -07:00
David Dworken
9d95b4ca70
Add comma separation in stats page
2022-10-02 20:42:10 -07:00
David Dworken
a85315d366
Log hishtory version in internal analytics
2022-10-02 20:39:52 -07:00
David Dworken
b7c64b61c8
Always check gorm interactions for errors
2022-10-02 19:41:00 -07:00
David Dworken
757ebb9547
Prompt people if they run hishtory init and already have a bunch of entries + fix tests + add TODOs + add hishtory version to requests
2022-10-01 09:50:06 -07:00
David Dworken
18ddbf2ca9
Track queries in usage data + better formatting in the internal usage page
2022-09-30 23:38:35 -07:00
David Dworken
98a4f002fa
Get RemoteAddr from X-Real-Ip header
2022-09-29 23:51:45 -07:00
David Dworken
f0a3caed1c
Add an internal only handler to expose basic usage stats via HTTP and keep a per-device counter of number of history entries for my own vanity stats
2022-09-29 23:32:13 -07:00
David Dworken
a108268925
Add TODOs
2022-09-27 22:15:31 -07:00
David Dworken
5fc3cf3ebc
Increase speed
2022-09-24 16:14:05 -07:00
David Dworken
aa4a7310dd
Add hishtory.svg from asciinema
2022-09-24 16:12:02 -07:00
David Dworken
b0c4780d48
hishtory --> hiSHtory
2022-09-24 15:30:51 -07:00
David Dworken
63625e94fe
Add prominent github link to the website
2022-09-24 01:16:13 -07:00
David Dworken
c18c2502cc
Add a complex test with many users
2022-09-22 21:45:20 -07:00
David Dworken
c6d4f1ef68
Fix missing newline in zshrc, add test for hishtory redact prompting, and tag TODOs with plans
2022-09-22 18:22:06 -07:00
David Dworken
ee6680f571
Remove debugging information and give up on that test, and add work around for weird zsh bug with importing
2022-09-22 18:09:51 -07:00
David Dworken
6db8044bec
Add server tests for redaction
2022-09-21 21:46:46 -07:00
David Dworken
27157e675d
Add TODOs
2022-09-21 19:59:31 -07:00
David Dworken
3e89accf97
Delete the tmp client before downloading to clear macos's signature cache
2022-09-21 19:01:34 -07:00
David Dworken
414d8926f6
Fix incorrect go version in dockerfile
2022-09-20 23:27:36 -07:00
David Dworken
80822c722e
Optimize SQL query so it only runs one query rather than N
2022-09-20 21:36:02 -07:00
David Dworken
19bbf27efd
More reliable handling for deletion requests
2022-09-20 21:28:49 -07:00
David Dworken
a2f95fdc4d
Fix typo in SQL query
2022-09-20 21:15:20 -07:00
David Dworken
41f82e8034
Add ReadCount field to deletion requests so that we can eventually delete them from the DB
2022-09-20 21:13:59 -07:00
David Dworken
5391ecd220
First version of working redaction with passing integration tests
2022-09-19 22:49:48 -07:00
David Dworken
1bf510ff8a
Better offline handling, implemented the local portion of delete, and maybe fixed the bug when running tests on github actions
2022-09-18 09:42:24 -07:00
David Dworken
158c2f2c57
Bump backend version to go1.18
2022-09-03 21:28:47 -07:00
David Dworken
d0aaef10ff
Update landing pages
2022-06-04 23:29:13 -07:00
David Dworken
e745941350
Stop looping the demo video + increase speed
2022-06-04 23:22:07 -07:00
David Dworken
0fac3b7286
Make query params required rather than having weird undefined behavior
2022-06-04 23:03:05 -07:00
David Dworken
72bd46d4e8
Detect failures to install hishtory in install script
2022-06-04 22:09:30 -07:00
David Dworken
b760db58cd
Update URL formats for backend
2022-06-04 20:36:34 -07:00
David Dworken
1da703e9c2
Working update code for macos
2022-05-26 23:45:08 -07:00
David Dworken
47d13a9c27
Passing the basic dump testss
2022-05-22 19:45:46 -07:00
David Dworken
feaa8b2bd1
Add a DB dump test that passes on zsh (is failing for an unknown reason on bash currently) + fix backup and restore for WAL files + better offline support
2022-05-01 22:37:26 -04:00
David Dworken
cbc4e70605
Integrated client-side with dump requests, haven't written any integration tests yet
2022-04-28 11:46:14 -07:00
David Dworken
46d7e9e013
Tests passing when being run offline, still not integrated with the new API endpoints yet
2022-04-28 11:26:55 -07:00
David Dworken
8a018b71b8
Add back in the bootstrap handler since I realized it is still useful to provide some amount of history at first install immediately
2022-04-28 11:05:56 -07:00
David Dworken
0499010981
Remove the 'e' prefix from api endpoints + implement backend APIs for clean loading of all data from other instances
2022-04-28 10:56:59 -07:00
David Dworken
c79a510b9c
Maybe working xattr support for code signing
2022-04-25 22:03:31 -07:00
David Dworken
be27d190c6
Improve the code for the /download API endpoint so that it auto-decrements the tag until it finds a valid version
2022-04-21 22:02:28 -07:00
David Dworken
7e4f8e060f
Add TODO to install.py, and explicitly set CGO_ENABLED=1
2022-04-19 22:22:04 -07:00
David Dworken
0a03ce3407
Add missing .py extension
2022-04-19 20:44:35 -07:00
David Dworken
6c9236785a
Add an easy to use cross-platform install script for that classic curl | bash goodness
2022-04-19 20:36:14 -07:00
David Dworken
ec4d64dedf
Delete test that was testing deleted HTTP handlers
2022-04-17 12:56:10 -07:00
David Dworken
f3cbe98091
Fix download URLs for the new git tag format
2022-04-17 12:07:32 -07:00
David Dworken
e98783d30f
Add server-side code to handle updates for non-linux
2022-04-17 12:02:56 -07:00
David Dworken
f22bb5b4d9
Fix TODOs in tests
2022-04-16 23:02:36 -07:00
David Dworken
735a98a611
Add API endpoint for getting information about the latest hishtory version and download URLs
2022-04-16 16:34:09 -07:00
David Dworken
05f1af8714
Add trigger-cron endpoint + optimize dockerfile for server
2022-04-16 16:28:53 -07:00
David Dworken
d7ad96906f
Add todo + remove pending features that are already done
2022-04-16 12:39:28 -07:00
David Dworken
a436edbd16
Add backend code to delete entries that have already been read + start collecitng data on usage data so we can eventually prune data from users that are no longer using hishtory
2022-04-16 11:37:43 -07:00
David Dworken
2b676a935b
Optimize server by only sending new-ish records to the client
2022-04-16 10:30:39 -07:00
David Dworken
c369661e53
More server-side logging + client-side logging
2022-04-16 00:44:47 -07:00
