tangled.org / core

4

Monorepo for Tangled tangled.org

4

core / knotmirror /

at dwn/spindle-microvm-debugssh 8 folders 9 files

Lewis knotmirror/resyncer: always forward sha256 3d ago

knotmirror: add redis cache to listLanguages Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

knotmirror/db: use `DBTX` interface Signed-off-by: Seongmin Lee <git@boltless.me>

3 weeks ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

knotmirror: key repos by repo_did + disk migration Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

knotmirror/resyncer: always forward sha256 Lewis: May this revision serve well! <lewis@tangled.org>

3 days ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

knotmirror: use repoDid for resync identifier Signed-off-by: Seongmin Lee <git@boltless.me>

3 weeks ago

knotmirror/resyncer: always forward sha256 Lewis: May this revision serve well! <lewis@tangled.org>

3 days ago

knotmirror/resyncer: always forward sha256 Lewis: May this revision serve well! <lewis@tangled.org>

3 days ago

knotmirror: introduce knotmirror KnotMirror is an external service that is intended to be used by appview. It will ingest all known git repos and provide xrpc methods to inspect them, so appview won't need to fetch individual knots on every page render. Using postgres exclusively instead of sqlite to support A LOT of concurrent writes. Signed-off-by: Seongmin Lee <git@boltless.me>

4 months ago

knotmirror/git: sync HEAD seperately on resync Ideally this should be done by dedicated event like `gitHeadUpdate`, but as we have lots of self-hosted knots, doing all the work from knotmirror would be easier than requesting users to update their knot again. Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

*: fmt, remove a couple little frontend bits Lewis: May this revision serve well! <lewis@tangled.org>

1 month ago

knotmirror: introduce knotmirror KnotMirror is an external service that is intended to be used by appview. It will ingest all known git repos and provide xrpc methods to inspect them, so appview won't need to fetch individual knots on every page render. Using postgres exclusively instead of sqlite to support A LOT of concurrent writes. Signed-off-by: Seongmin Lee <git@boltless.me>

4 months ago

knotmirror: introduce knotmirror KnotMirror is an external service that is intended to be used by appview. It will ingest all known git repos and provide xrpc methods to inspect them, so appview won't need to fetch individual knots on every page render. Using postgres exclusively instead of sqlite to support A LOT of concurrent writes. Signed-off-by: Seongmin Lee <git@boltless.me>

4 months ago

knotmirror/resyncer: always forward sha256 Lewis: May this revision serve well! <lewis@tangled.org>

3 days ago

resyncer_test.go

knotmirror/resyncer: always forward sha256 Lewis: May this revision serve well! <lewis@tangled.org>

3 days ago

knotmirror: use repoDid for resync identifier Signed-off-by: Seongmin Lee <git@boltless.me>

3 weeks ago

readme.md

KnotMirror#

KnotMirror is a git mirror service for all known repos. Heavily inspired by indigo/relay and indigo/tap.

KnotMirror syncs repo list using tap and subscribe to all known knots as KnotStream.