Commit Graph

65 Commits

Author SHA1 Message Date
tobi
b143877995
[feature] Unused avatar and header cleanup (#574)
* rename + tidy up remote pruning

* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships

* update logging to new function name

* add get avatars and headers to bun

* add pruneallmeta function

* don't set uncached since we're deleting anyway

* fix totalPruned being in wrong place

* test pruning meta

* go fmt ./...

* rename mediaprune

* add meta pruning to routine mediaprune

* tidy up cleanup job scheduling

* rename adminmediaremoteprune

* update mediacleanup to use renamed prune func

* update swagger docs a little bit

* reuse cancel + context
2022-05-15 15:45:04 +01:00
kim
26b74aefaf
[bugfix] Fix existing bio text showing as HTML (#531)
* fix existing bio text showing as HTML

- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes

Signed-off-by: kim <grufwub@gmail.com>

* go away linter

Signed-off-by: kim <grufwub@gmail.com>

* change buf reset location, change html mention tags

Signed-off-by: kim <grufwub@gmail.com>

* reduce FindLinks code complexity

Signed-off-by: kim <grufwub@gmail.com>

* fix HTML to text conversion

Signed-off-by: kim <grufwub@gmail.com>

* Update internal/regexes/regexes.go

Co-authored-by: Mina Galić <mina.galic@puppet.com>

* use improved html2text lib with more options

Signed-off-by: kim <grufwub@gmail.com>

* fix to produce actual plaintext from html

Signed-off-by: kim <grufwub@gmail.com>

* fix span tags instead written as space

Signed-off-by: kim <grufwub@gmail.com>

* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future

Signed-off-by: kim <grufwub@gmail.com>

* fix tag/mention replacements to use input string, fix link replace to not include scheme

Signed-off-by: kim <grufwub@gmail.com>

* use matched input string for link replace href text

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code (to appease linter :sobs:)

Signed-off-by: kim <grufwub@gmail.com>

* improve hashtagFinger regex to be more compliant

Signed-off-by: kim <grufwub@gmail.com>

* update breakReplacer to include both unix and windows line endings

Signed-off-by: kim <grufwub@gmail.com>

* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts

Signed-off-by: kim <grufwub@gmail.com>

* drop unnecessary code

Signed-off-by: kim <grufwub@gmail.com>

* update text package tests to fix logic changes

Signed-off-by: kim <grufwub@gmail.com>

* add raw note content testing to account update and account verify

Signed-off-by: kim <grufwub@gmail.com>

* remove unused modules

Signed-off-by: kim <grufwub@gmail.com>

* fix emoji regex

Signed-off-by: kim <grufwub@gmail.com>

* fix replacement of hashtags

Signed-off-by: kim <grufwub@gmail.com>

* update code comment

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: Mina Galić <mina.galic@puppet.com>
2022-05-07 17:55:27 +02:00
tobi
37d310f981
[feature] Dereference remote mentions when the account is not already known (#442)
* remove mention util function from db

* add ParseMentionFunc to gtsmodel

* add parseMentionFunc to processor

* refactor search to simplify it a bit

* add parseMentionFunc to account

* add parseMentionFunc to status

* some renaming for clarity

* test dereference of unknown mentioned account
2022-03-29 11:54:56 +02:00
tobi
55ad6dee71
[feature] Admin account actions (#432)
* add accountAction to the admin API

* model admin account action

* add admin account action to the processor

* add migration for new AdminAccountActions table

* fix accounts admin path

* Update swagger docs
2022-03-19 12:01:40 +01:00
tobi
07727753b9
[feature] Clean up/uncache remote media (#407)
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tobi
1b36e85840
[feature] Rework timeline code to make it useful for more than just statuses (#373)
* add preparable and timelineable interfaces

* initialize timeline manager within the processor

* generic renaming

* move status-specific timeline logic into the processor

* refactor timeline to make it useful for more than statuses
2022-02-05 12:47:38 +01:00
tobi
ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi
e04b187702
Refactor/tidy (#261)
* tidy up streaming

* cut down code duplication

* test get followers/following

* test streaming processor

* fix some test models

* add TimeMustParse

* fix uri / url typo

* make trace logging less verbose

* make logging more consistent

* disable quote on logging

* remove context.Background

* remove many extraneous mastodon references

* regenerate swagger

* don't log query on no rows result

* log latency first for easier reading
2021-10-04 15:24:19 +02:00
tobi
c7c9fff730
use timestamptz instead of timestamp (#237) 2021-09-20 17:41:52 +02:00
kim
4c4a622088
Prune unnecessary nullzeros, fixup db tags (#200)
* prune gtsmodel.Account bun tags, add note to gtsmodel dir

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* further database tag fixes

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* more db tag fixups

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* fix removing nullzero for account timestamps...

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add nullzero back to accountid tag

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* rename gtsmodel readme

Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-10 10:08:21 +02:00
tobi
555ea8edfb
Import export (#194)
* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* update bun library -> v1.0.4

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library -> v4.3.1-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* handle oauth token scope, fix user.SigninCount + token.UserID

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update oauth library --> v4.3.2-SSB

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* update sqlite library -> v1.13.0

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* review changes

* start with export/import code

* messing about with decoding/encoding

* some more fiddling

* stuff is WORKING

* working pretty alright!

* go fmt

* fix up tests, add docs

* start backup/restore doc

* tweaks

* credits

* update advancedVisibility settings

* review changes

Co-authored-by: kim (grufwub) <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2021-09-09 16:15:25 +02:00
kim (grufwub)
d2a85f2f5f handle oauth token scope, fix user.SigninCount + token.UserID
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-08 20:46:08 +01:00
tsmethurst
a116676814 Review changes 2021-09-03 10:30:40 +02:00
tsmethurst
cc7a35ee92 token tests 2021-09-02 12:22:09 +02:00
tsmethurst
42892f91e1 client_test 2021-09-02 12:17:14 +02:00
tsmethurst
6f7da0ecdd account + application tests 2021-09-02 12:10:59 +02:00
tsmethurst
4696e1a7b3 moving stuff around 2021-09-01 18:29:25 +02:00
tsmethurst
684bd56528 move oauth models into gtsmodel 2021-09-01 11:45:01 +02:00
tsmethurst
7b01304dac more updates 2021-09-01 11:12:10 +02:00
tsmethurst
2786b5f887 change muchos things 2021-09-01 11:11:26 +02:00
tsmethurst
fffff93b75 go fmt 2021-09-01 11:11:26 +02:00
tsmethurst
356d28fef9 more work on struct validation 2021-09-01 11:11:26 +02:00
tsmethurst
dc2e1bf9ab more work on struct validation 2021-09-01 11:11:26 +02:00
tsmethurst
8ba816cd83 statusmute annotations 2021-09-01 11:11:26 +02:00
tsmethurst
d2276fc553 start working on struct validation for gtsmodel 2021-09-01 11:11:26 +02:00
tobi
53507ac2a3
Mention fixup (#167)
* rework mention creation a bit

* rework mention creation a bit

* tidy up status dereferencing

* start adding tests for dereferencing

* fixups

* fix

* review changes
2021-08-29 12:03:08 +02:00
tobi
1582cf8bad
fix null returned session (#159) 2021-08-26 18:55:39 +02:00
tobi
ddc120d5e6
fix public timeline bug (#150) 2021-08-26 11:28:16 +02:00
tobi
2dc9fc1626
Pg to bun (#148)
* start moving to bun

* changing more stuff

* more

* and yet more

* tests passing

* seems stable now

* more big changes

* small fix

* little fixes
2021-08-25 15:34:33 +02:00
tobi
071eca20ce
Manually approves followers (#146)
* update go-fed

* update go-fed

* manuallyapprovesfollowers

* serialize manuallyApprovesFollowers
2021-08-23 12:46:05 +02:00
tobi
4920229a3b
Database updates (#144)
* start moving some database stuff around

* continue moving db stuff around

* more fiddling

* more updates

* and some more

* and yet more

* i broke SOMETHING but what, it's a mystery

* tidy up

* vendor ttlcache

* use ttlcache

* fix up some tests

* rename some stuff

* little reminder

* some more updates
2021-08-20 12:26:56 +02:00
Tobi Smethurst
0f2de6394a
Dereference remote replies (#132)
* decided where to put reply dereferencing

* fiddling with dereferencing threads

* further adventures

* tidy up some stuff

* move dereferencing functionality

* a bunch of refactoring

* go fmt

* more refactoring

* bleep bloop

* docs and linting

* start implementing replies collection on gts side

* fiddling around

* allow dereferencing our replies

* lint, fmt
2021-08-10 13:32:39 +02:00
Tobi Smethurst
0386a28b5a
Frodo swaggins (#126)
* more swagger fun

* document a whole bunch more stuff

* more swagger yayyyyyyy

* progress + go fmt
2021-08-02 19:06:44 +02:00
Tobi Smethurst
846057f0d6
Block/unblock (#96)
* remote + local block logic, incl. federation

* improve blocking stuff

* fiddle with display of blocked profiles

* go fmt
2021-07-11 16:22:21 +02:00
Tobi Smethurst
c71e55ecc4
clean up some weirdness in the router (#80) 2021-07-07 15:46:42 +02:00
Tobi Smethurst
3568579218
Blocklist import (#77)
* first steps on importing blocklists

* unblock domains properly
2021-07-06 13:29:11 +02:00
Tobi Smethurst
d389e7b150
Domain block (#76)
* start work on admin domain blocking

* move stuff around + further work on domain blocks

* move + restructure processor

* prep work for deleting account

* tidy

* go fmt

* formatting

* domain blocking more work

* check domain blocks way earlier on

* progress on delete account

* delete more stuff when an account is gone

* and more...

* domain blocky block block

* get individual domain block, delete a block
2021-07-05 13:23:03 +02:00
Tobi Smethurst
87cf621e21
Remote instance dereferencing (#70)
Remote instances are now dereferenced when they post to an inbox on a GtS instance.

    Dereferencing will be done first by checking the /api/v1/instance endpoint of an instance.
    If that doesn't work, /.well-known/nodeinfo will be checked.
    If that doesn't work, only a minimal representation of the instance will be stored.

A new field was added to the Instance database model. To create it:

alter table instances add column contact_account_username text;
2021-06-27 16:52:18 +02:00
Tobi Smethurst
b6c62309f2
separate public key handler (#64) 2021-06-26 16:21:40 +02:00
Tobi Smethurst
8c9a853343
Instance settings updates (#59)
Allow admins to set instance settings through a PATCH to /api/v1/instance

Update templates to reflect some of the new fields
2021-06-23 16:35:57 +02:00
Tobi Smethurst
aa8a0d0850
Streaming (#49)
Add new status and notification websocket streaming capabilities
2021-06-19 11:18:55 +02:00
Tobi Smethurst
82d9f88e42
Timeline improvements (#41)
Tidying up.
Parent/child statuses now display correctly in status/id/context.
2021-06-17 18:02:33 +02:00
Tobi Smethurst
b4288f3c47
Timeline manager (#40)
* start messing about with timeline manager

* i have no idea what i'm doing

* i continue to not know what i'm doing

* it's coming along

* bit more progress

* update timeline with new posts as they come in

* lint and fmt

* Select accounts where empty string

* restructure a bunch, get unfaves working

* moving stuff around

* federate status deletes properly

* mention regex better but not 100% there

* fix regex

* some more hacking away at the timeline code phew

* fix up some little things

* i can't even

* more timeline stuff

* move to ulid

* fiddley

* some lil fixes for kibou compatibility

* timelines working pretty alright!

* tidy + lint
2021-06-13 18:42:28 +02:00
Tobi Smethurst
3d77f81c7f
Move a lot of stuff + tidy stuff (#37)
Lots of renaming and moving stuff, some bug fixes, more lenient parsing of notifications and home timeline.
2021-05-30 13:12:00 +02:00
Tobi Smethurst
87177d840b
Announce/boost (#35)
Remote boosts incoming/outgoing now working.
2021-05-28 19:57:04 +02:00
Tobi Smethurst
40add68691
Notifications (#34)
Notifications working for:

* Mentions
* Faves
* New follow requests
* New followers
2021-05-27 16:06:24 +02:00
Tobi Smethurst
e670c32a91
Faves (#31)
* start on federating faves

* outbound federation of likes working
2021-05-24 18:49:48 +02:00
tsmethurst
aeb665df55 small fiddling to allow whalebird to work (a bit) 2021-05-22 15:51:20 +02:00
Tobi Smethurst
d839f27c30
Follows and relationships (#27)
* Follows -- create and undo, both remote and local
* Statuses -- federate new posts, including media, attachments, CWs and image descriptions.
2021-05-21 15:48:26 +02:00
Tobi Smethurst
6cd033449f
Refine statuses (#26)
Remote media is now dereferenced and attached properly to incoming federated statuses.
    Mentions are now dereferenced and attached properly to incoming federated statuses.
    Small fixes to status visibility.
    Allow URL params for filtering statuses:

	// ExcludeRepliesKey is for specifying whether to exclude replies in a list of returned statuses by an account.
      	// PinnedKey is for specifying whether to include pinned statuses in a list of returned statuses by an account.
      	// MaxIDKey is for specifying the maximum ID of the status to retrieve.
      	// MediaOnlyKey is for specifying that only statuses with media should be returned in a list of returned statuses by an account.

    Add endpoint for fetching an account's statuses.
2021-05-17 19:06:58 +02:00