Monorepo for Tangled
tangled.org
1---
2title: Tangled docs
3author: The Tangled Contributors
4date: 21 Sun, Dec 2025
5abstract: |
6 Tangled is a decentralized code hosting and collaboration
7 platform. Every component of Tangled is open-source and
8 self-hostable. [tangled.org](https://tangled.org) also
9 provides hosting and CI services that are free to use.
10
11 There are several models for decentralized code
12 collaboration platforms, ranging from ActivityPub’s
13 (Forgejo) federated model, to Radicle’s entirely P2P model.
14 Our approach attempts to be the best of both worlds by
15 adopting the AT Protocol—a protocol for building decentralized
16 social applications with a central identity
17
18 Our approach to this is the idea of “knots”. Knots are
19 lightweight, headless servers that enable users to host Git
20 repositories with ease. Knots are designed for either single
21 or multi-tenant use which is perfect for self-hosting on a
22 Raspberry Pi at home, or larger “community” servers. By
23 default, Tangled provides managed knots where you can host
24 your repositories for free.
25
26 The appview at tangled.org acts as a consolidated "view"
27 into the whole network, allowing users to access, clone and
28 contribute to repositories hosted across different knots
29 seamlessly.
30---
31
32# Quick start guide
33
34## Login or sign up
35
36You can [login](https://tangled.org) by using your AT Protocol
37account. If you are unclear on what that means, simply head
38to the [signup](https://tangled.org/signup) page and create
39an account. By doing so, you will be choosing Tangled as
40your account provider (you will be granted a handle of the
41form `user.tngl.sh`).
42
43In the AT Protocol network, users are free to choose their account
44provider (known as a "Personal Data Service", or PDS), and
45login to applications that support AT accounts.
46
47You can think of it as "one account for all of the atmosphere"!
48
49If you already have an AT account (you may have one if you
50signed up to Bluesky, for example), you can login with the
51same handle on Tangled (so just use `user.bsky.social` on
52the login page).
53
54## Add an SSH key
55
56Once you are logged in, you can start creating repositories
57and pushing code. Tangled supports pushing git repositories
58over SSH.
59
60First, you'll need to generate an SSH key if you don't
61already have one:
62
63```bash
64ssh-keygen -t ed25519 -C "foo@bar.com"
65```
66
67When prompted, save the key to the default location
68(`~/.ssh/id_ed25519`) and optionally set a passphrase.
69
70Copy your public key to your clipboard:
71
72```bash
73# on X11
74cat ~/.ssh/id_ed25519.pub | xclip -sel c
75
76# on wayland
77cat ~/.ssh/id_ed25519.pub | wl-copy
78
79# on macos
80cat ~/.ssh/id_ed25519.pub | pbcopy
81```
82
83Now, navigate to 'Settings' -> 'Keys' and hit 'Add Key',
84paste your public key, give it a descriptive name, and hit
85save.
86
87## Create a repository
88
89Once your SSH key is added, create your first repository:
90
911. Hit the green `+` icon on the topbar, and select
92 repository
932. Enter a repository name
943. Add a description
954. Choose a knotserver to host this repository on
965. Hit create
97
98Knots are self-hostable, lightweight Git servers that can
99host your repository. Unlike traditional code forges, your
100code can live on any server. Read the [Knots](TODO) section
101for more.
102
103## Configure SSH
104
105To ensure Git uses the correct SSH key and connects smoothly
106to Tangled, add this configuration to your `~/.ssh/config`
107file:
108
109```
110Host tangled.org
111 Hostname tangled.org
112 User git
113 IdentityFile ~/.ssh/id_ed25519
114 AddressFamily inet
115```
116
117This tells SSH to use your specific key when connecting to
118Tangled and prevents authentication issues if you have
119multiple SSH keys.
120
121Note that this configuration only works for knotservers that
122are hosted by tangled.org. If you use a custom knot, refer
123to the [Knots](TODO) section.
124
125## Push your first repository
126
127Initialize a new Git repository:
128
129```bash
130mkdir my-project
131cd my-project
132
133git init
134echo "# My Project" > README.md
135```
136
137Add some content and push!
138
139```bash
140git add README.md
141git commit -m "Initial commit"
142git remote add origin git@tangled.org:user.tngl.sh/my-project
143git push -u origin main
144```
145
146That's it! Your code is now hosted on Tangled.
147
148## Migrating an existing repository
149
150Moving your repositories from GitHub, GitLab, Bitbucket, or
151any other Git forge to Tangled is straightforward. You'll
152simply change your repository's remote URL. At the moment,
153Tangled does not have any tooling to migrate data such as
154GitHub issues or pull requests.
155
156First, create a new repository on tangled.org as described
157in the [Quick Start Guide](#create-a-repository).
158
159Navigate to your existing local repository:
160
161```bash
162cd /path/to/your/existing/repo
163```
164
165You can inspect your existing Git remote like so:
166
167```bash
168git remote -v
169```
170
171You'll see something like:
172
173```bash
174origin git@github.com:username/my-project.git (fetch)
175origin git@github.com:username/my-project.git (push)
176```
177
178Update the remote URL to point to tangled:
179
180```bash
181git remote set-url origin git@tangled.org:user.tngl.sh/my-project
182```
183
184Verify the change:
185
186```bash
187git remote -v
188```
189
190You should now see:
191
192```bash
193origin git@tangled.org:user.tngl.sh/my-project (fetch)
194origin git@tangled.org:user.tngl.sh/my-project (push)
195```
196
197Push all your branches and tags to Tangled:
198
199```bash
200git push -u origin --all
201git push -u origin --tags
202```
203
204Your repository is now migrated to Tangled! All commit
205history, branches, and tags have been preserved.
206
207## Mirroring a repository to Tangled
208
209If you want to maintain your repository on multiple forges
210simultaneously, for example, keeping your primary repository
211on GitHub while mirroring to Tangled for backup or
212redundancy, you can do so by adding [multiple remotes](https://git-scm.com/docs/git-push#_remotes).
213
214You can configure your local repository to push to both
215Tangled and, say, GitHub. You may already have the following
216setup:
217
218```bash
219$ git remote -v
220origin git@github.com:username/my-project.git (fetch)
221origin git@github.com:username/my-project.git (push)
222```
223
224Now add Tangled as an additional push URL to the same
225remote:
226
227```bash
228git remote set-url --add --push origin git@tangled.org:user.tngl.sh/my-project
229```
230
231You also need to re-add the original URL as a push
232destination (Git will now use the original URL to fetch only):
233
234```bash
235git remote set-url --add --push origin git@github.com:username/my-project.git
236```
237
238Verify your configuration:
239
240```bash
241$ git remote -v
242origin git@github.com:username/my-project.git (fetch)
243origin git@tangled.org:user.tngl.sh/my-project (push)
244origin git@github.com:username/my-project.git (push)
245```
246
247Notice that there's one fetch URL (the primary remote) and
248two push URLs. Now, whenever you push, Git will
249automatically push to both remotes:
250
251```bash
252git push origin main
253```
254
255This single command pushes your `main` branch to both GitHub
256and Tangled simultaneously.
257
258To push all branches and tags:
259
260```bash
261git push origin --all
262git push origin --tags
263```
264
265If you prefer more control over which remote you push to,
266you can maintain separate remotes:
267
268```bash
269git remote add github git@github.com:username/my-project.git
270git remote add tangled git@tangled.org:user.tngl.sh/my-project
271```
272
273Then push to each explicitly:
274
275```bash
276git push github main
277git push tangled main
278```
279
280# Hosting websites on Tangled
281
282You can serve static websites directly from your git repositories on
283Tangled. If you've used GitHub Pages or Codeberg Pages, this should feel
284familiar.
285
286## Overview
287
288Every user gets a sites domain. If you signed up through Tangled's own
289PDS (`tngl.sh`), your sites domain is automatically
290`<your-handle>.tngl.sh` no setup needed. Otherwise, you can claim a
291`<subdomain>.tngl.io` domain from your settings.
292
293You can serve multiple sites per domain:
294
295- One **index site** served at the root of your domain (e.g.
296 `alice.tngl.sh`)
297- Any number of **sub-path sites** served under the repository name
298 (e.g. `alice.tngl.sh/my-project`)
299
300## Claiming a domain
301
302If you don't have a `tngl.sh` handle, you need to claim a domain before
303publishing sites:
304
3051. Go to **Settings → Sites**
3062. Enter a subdomain (e.g. `alice` to claim `alice.tngl.io`)
3073. Click **claim**
308
309You can only hold one domain at a time. Releasing a domain puts it in a
31030-day cooldown before anyone else can claim it.
311
312## Configuring a site for a repository
313
3141. Navigate to your repository
3152. Go to **Settings → Sites**
3163. Choose a **branch** to deploy from
3174. Set the **deploy directory** — the path within the repository
318 containing your `index.html`. Use `/` for the root, or a subdirectory
319 like `/docs` or `/public`
3205. Choose the **site type**:
321 - **Index site** — served at the root of your domain (e.g.
322 `alice.tngl.sh`)
323 - **Sub-path site** — served under the repository name (e.g.
324 `alice.tngl.sh/my-project`)
3256. Click **save**
326
327The site will be deployed automatically. You can see the status of your
328previous deploys in the **Recent Deploys** section at the bottom of the
329page.
330
331Sites are redeployed automatically on every push to the configured
332branch.
333
334## Custom domains
335
336Tangled currently doesn't support custom domains for sites. This will be
337added in a future update.
338
339## Deploy directory
340
341The deploy directory is the path within your repository that Tangled
342serves as the site root. It must contain an `index.html`.
343
344| Deploy directory | Result |
345|---|---|
346| `/` | Serves the repository root |
347| `/docs` | Serves the `docs/` subdirectory |
348| `/public` | Serves the `public/` subdirectory |
349
350Directories are served with automatic `index.html` resolution -- a
351request to `/about` will serve `/about/index.html` if it exists.
352
353## Site types
354
355| Type | URL |
356|---|---|
357| Index site | `alice.tngl.sh` |
358| Sub-path site | `alice.tngl.sh/my-project` |
359
360Only one repository can be the index site for a given domain at a time.
361If another repository already holds the index site, you will see a
362notice in the settings and only the sub-path option will be available.
363
364## Deploy triggers
365
366A deployment is triggered automatically when:
367
368- You push to the configured branch
369- You change the site configuration (branch, deploy directory, or site
370 type)
371
372## Disabling a site
373
374To stop serving a site, go to **Settings → Sites** in your repository
375and click **Disable**. This removes the site configuration and stops
376serving the site. The deployed files are also deleted from storage.
377
378Releasing your domain from **Settings → Sites** at the account level
379will disable all sites associated with it and delete their files.
380
381
382# Knot self-hosting guide
383
384So you want to run your own knot server? Great! Here are a few prerequisites:
385
3861. A server of some kind (a VPS, a Raspberry Pi, etc.). Preferably running a Linux distribution of some kind.
3872. A (sub)domain name. People generally use `knot.example.com`.
3883. A valid SSL certificate for your domain.
389
390## NixOS
391
392Refer to the [knot
393module](https://tangled.org/tangled.org/core/blob/master/nix/modules/knot.nix)
394for a full list of options. Sample configurations:
395
396- [The test VM](https://tangled.org/tangled.org/core/blob/master/nix/vm.nix#L85)
397- [@pyrox.dev/nix](https://tangled.org/pyrox.dev/nix/blob/d19571cc1b5fe01035e1e6951ec8cf8a476b4dee/hosts/marvin/services/tangled.nix#L15-25)
398
399## Docker
400
401Refer to
402[@tangled.org/knot-docker](https://tangled.org/@tangled.org/knot-docker).
403Note that this is community maintained.
404
405## Manual setup
406
407First, clone this repository:
408
409```
410git clone https://tangled.org/@tangled.org/core
411```
412
413Then, build the `knot` CLI. This is the knot administration
414and operation tool. For the purpose of this guide, we're
415only concerned with these subcommands:
416
417- `knot server`: the main knot server process, typically
418 run as a supervised service
419- `knot guard`: handles role-based access control for git
420 over SSH (you'll never have to run this yourself)
421- `knot keys`: fetches SSH keys associated with your knot;
422 we'll use this to generate the SSH
423 `AuthorizedKeysCommand`
424
425```
426cd core
427export CGO_ENABLED=1
428go build -o knot ./cmd/knot
429```
430
431Next, move the `knot` binary to a location owned by `root` --
432`/usr/local/bin/` is a good choice. Make sure the binary itself is also owned by `root`:
433
434```
435sudo mv knot /usr/local/bin/knot
436sudo chown root:root /usr/local/bin/knot
437```
438
439This is necessary because SSH `AuthorizedKeysCommand` requires [really
440specific permissions](https://stackoverflow.com/a/27638306). The
441`AuthorizedKeysCommand` specifies a command that is run by `sshd` to
442retrieve a user's public SSH keys dynamically for authentication. Let's
443set that up.
444
445```
446sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF
447Match User git
448 AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys
449 AuthorizedKeysCommandUser nobody
450EOF
451```
452
453Then, reload `sshd`:
454
455```
456sudo systemctl reload ssh
457```
458
459Next, create the `git` user. We'll use the `git` user's home directory
460to store repositories:
461
462```
463sudo adduser git
464```
465
466Create `/home/git/.knot.env` with the following, updating the values as
467necessary. The `KNOT_SERVER_OWNER` should be set to your
468DID, you can find your DID in the [Settings](https://tangled.sh/settings) page.
469
470```
471KNOT_REPO_SCAN_PATH=/home/git
472KNOT_SERVER_HOSTNAME=knot.example.com
473APPVIEW_ENDPOINT=https://tangled.org
474KNOT_SERVER_OWNER=did:plc:foobar
475KNOT_SERVER_INTERNAL_LISTEN_ADDR=127.0.0.1:5444
476KNOT_SERVER_LISTEN_ADDR=127.0.0.1:5555
477```
478
479If you run a Linux distribution that uses systemd, you can
480use the provided service file to run the server. Copy
481[`knotserver.service`](https://tangled.org/tangled.org/core/blob/master/systemd/knotserver.service)
482to `/etc/systemd/system/`. Then, run:
483
484```
485systemctl enable knotserver
486systemctl start knotserver
487```
488
489The last step is to configure a reverse proxy like Nginx or Caddy to front your
490knot. Here's an example configuration for Nginx:
491
492```
493server {
494 listen 80;
495 listen [::]:80;
496 server_name knot.example.com;
497
498 location / {
499 proxy_pass http://localhost:5555;
500 proxy_set_header Host $host;
501 proxy_set_header X-Real-IP $remote_addr;
502 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
503 proxy_set_header X-Forwarded-Proto $scheme;
504 }
505
506 # wss endpoint for git events
507 location /events {
508 proxy_set_header X-Forwarded-For $remote_addr;
509 proxy_set_header Host $http_host;
510 proxy_set_header Upgrade websocket;
511 proxy_set_header Connection Upgrade;
512 proxy_pass http://localhost:5555;
513 }
514 # additional config for SSL/TLS go here.
515}
516
517```
518
519Remember to use Let's Encrypt or similar to procure a certificate for your
520knot domain.
521
522You should now have a running knot server! You can finalize
523your registration by hitting the `verify` button on the
524[/settings/knots](https://tangled.org/settings/knots) page. This simply creates
525a record on your PDS to announce the existence of the knot.
526
527### Custom paths
528
529(This section applies to manual setup only. Docker users should edit the mounts
530in `docker-compose.yml` instead.)
531
532Right now, the database and repositories of your knot lives in `/home/git`. You
533can move these paths if you'd like to store them in another folder. Be careful
534when adjusting these paths:
535
536- Stop your knot when moving data (e.g. `systemctl stop knotserver`) to prevent
537 any possible side effects. Remember to restart it once you're done.
538- Make backups before moving in case something goes wrong.
539- Make sure the `git` user can read and write from the new paths.
540
541#### Database
542
543As an example, let's say the current database is at `/home/git/knotserver.db`,
544and we want to move it to `/home/git/database/knotserver.db`.
545
546Copy the current database to the new location. Make sure to copy the `.db-shm`
547and `.db-wal` files if they exist.
548
549```
550mkdir /home/git/database
551cp /home/git/knotserver.db* /home/git/database
552```
553
554In the environment (e.g. `/home/git/.knot.env`), set `KNOT_SERVER_DB_PATH` to
555the new file path (_not_ the directory):
556
557```
558KNOT_SERVER_DB_PATH=/home/git/database/knotserver.db
559```
560
561#### Repositories
562
563As an example, let's say the repositories are currently in `/home/git`, and we
564want to move them into `/home/git/repositories`.
565
566Create the new folder, then move the existing repositories (if there are any):
567
568```
569mkdir /home/git/repositories
570# move all DIDs into the new folder; these will vary for you!
571mv /home/git/did:plc:wshs7t2adsemcrrd4snkeqli /home/git/repositories
572```
573
574In the environment (e.g. `/home/git/.knot.env`), update `KNOT_REPO_SCAN_PATH`
575to the new directory:
576
577```
578KNOT_REPO_SCAN_PATH=/home/git/repositories
579```
580
581Similarly, update your `sshd` `AuthorizedKeysCommand` to use the updated
582repository path:
583
584```
585sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF
586Match User git
587 AuthorizedKeysCommand /usr/local/bin/knot keys -o authorized-keys -git-dir /home/git/repositories
588 AuthorizedKeysCommandUser nobody
589EOF
590```
591
592Make sure to restart your SSH server!
593
594#### MOTD (message of the day)
595
596To configure the MOTD used ("Welcome to this knot!" by default), edit the
597`/home/git/motd` file:
598
599```
600printf "Hi from this knot!\n" > /home/git/motd
601```
602
603Note that you should add a newline at the end if setting a non-empty message
604since the knot won't do this for you.
605
606## Secure Mode
607
608Secure Mode isolates each `git` subprocess to the repository it is
609operating on, using two mechanisms:
610
611- **Linux Landlock** restricts the filesystem paths the subprocess
612 can access -- it can only read/write its own repository and the
613 system directories it needs to run.
614- **UID isolation** runs each subprocess as a virtual UID assigned
615 to the repository owner, so that repositories belonging to
616 different owners are isolated from each other at the OS level
617 even if Landlock were somehow bypassed.
618
619Secure Mode requires:
620
621- Linux kernel >= 5.19 (Landlock V2). This is the minimum needed
622 for `git push` to work, because receive-pack's quarantine
623 migration uses cross-directory rename which requires the
624 Landlock `REFER` access right (added in V2). Kernels 5.13-5.18
625 support Landlock V1 and clones will work, but pushes will fail
626 with cross-device link errors. On kernels without any Landlock
627 support (< 5.13), the sandbox call is a no-op: UID isolation
628 still applies but no filesystem restriction is enforced.
629- `CAP_SETUID`, `CAP_SETGID`, and `CAP_CHOWN` available to the
630 knot process. The NixOS module grants these automatically; for
631 manual setups see the `setcap` step below.
632
633### NixOS
634
635Add `server.secureMode = true;` to your knot module configuration:
636
637```nix
638services.tangled.knot = {
639 server.secureMode = true;
640 # ... other options
641};
642```
643
644The NixOS module handles everything else automatically:
645
646- Grants the required capabilities to the knot service via
647 `AmbientCapabilities` in the systemd unit.
648- Installs a capability-bearing wrapper at
649 `/run/wrappers/bin/knot` via `security.wrappers`, so that
650 SSH-invoked git operations (pushes) also run under the correct
651 UID without requiring the service to run as root.
652- Runs `knot migrate-isolation` at service start to chown
653 existing repositories to their virtual UIDs.
654
655### Manual setup
656
657**Step 1.** Grant the required capabilities to the knot binary.
658This allows the knot process to switch to virtual UIDs at runtime
659without running as root. You will need to repeat this step
660whenever the binary is updated.
661
662```
663sudo setcap cap_setuid,cap_setgid,cap_chown+eip /usr/local/bin/knot
664```
665
666**Step 2.** Run the migration tool to assign virtual UIDs to all
667existing repositories and set their filesystem permissions. This
668must be run as root:
669
670```
671sudo knot migrate-isolation \
672 --git-dir /home/git \
673 --db /home/git/knotserver.db \
674 --internal-api 127.0.0.1:5444
675```
676
677You can re-run this at any time with `--force` to reapply
678permissions (e.g. after a manual repair or after updating the
679binary).
680
681**Step 2a.** Ensure the home directory is traversable by
682non-group users. Git subprocesses run as virtual UIDs that are
683not in the git group, and they need to resolve
684`$HOME/.config/git/config` to load the global config:
685
686```
687sudo chmod o+x /home/git
688```
689
690This adds only the execute bit, not read -- the virtual UIDs can
691traverse to known paths but cannot list directory contents.
692
693**Step 3.** Enable Secure Mode in your environment file:
694
695```
696KNOT_SERVER_SECURE_MODE=true
697```
698
699Or pass it as a flag:
700
701```
702knot server --secure-mode
703```
704
705**Step 4.** Regenerate the `AuthorizedKeysCommand` with the
706`-secure-mode` flag. This causes `knot keys` to emit guard
707command lines that include `-secure-mode`, so SSH pushes also
708get UID isolation:
709
710```
711sudo tee /etc/ssh/sshd_config.d/authorized_keys_command.conf <<EOF
712Match User git
713 AuthorizedKeysCommand /usr/local/bin/knot keys \
714 -o authorized-keys -secure-mode
715 AuthorizedKeysCommandUser nobody
716EOF
717```
718
719Reload `sshd` after making this change.
720
721> **Note:** the server will refuse to start in Secure Mode if any
722> repositories have not yet been isolation-migrated. Re-run
723> `migrate-isolation` if you see this error.
724
725## Troubleshooting
726
727If you run your own knot, you may run into some of these
728common issues. You can always join the
729[IRC](https://web.libera.chat/#tangled) or
730[Discord](https://chat.tangled.org/) if this section does
731not help.
732
733### Unable to push
734
735If you are unable to push to your knot or repository:
736
7371. First, ensure that you have added your SSH public key to
738 your account
7392. Check to see that your knot has synced the key by running
740 `knot keys`
7413. Check to see if git is supplying the correct private key
742 when pushing: `GIT_SSH_COMMAND="ssh -v" git push ...`
7434. Check to see if `sshd` on the knot is rejecting the push
744 for some reason: `journalctl -xeu ssh` (or `sshd`,
745 depending on your machine). These logs are unavailable if
746 using docker.
7475. Check to see if the knot itself is rejecting the push,
748 depending on your setup, the logs might be in one of the
749 following paths:
750 - `/tmp/knotguard.log`
751 - `/home/git/log`
752 - `/home/git/guard.log`
753
754# Spindles
755
756## Pipelines
757
758Spindle workflows allow you to write CI/CD pipelines in a
759simple format. They're located in the `.tangled/workflows`
760directory at the root of your repository, and are defined
761using YAML.
762
763A workflow has a set of common fields that apply no matter
764which engine you pick:
765
766- [Trigger](#trigger): A **required** field that defines
767 when a workflow should be triggered.
768- [Engine](#engine): A **required** field that defines which
769 engine a workflow should run on.
770- [Clone options](#clone-options): An **optional** field
771 that defines how the repository should be cloned.
772- [Environment](#environment): An **optional** field that
773 allows you to define environment variables.
774- [Steps](#steps): An **optional** field that allows you to
775 define what steps should run in the workflow.
776
777On top of these, each engine has its own options for things
778like dependencies and images. See [Engines](#engines) for
779the per-engine fields.
780
781### Trigger
782
783The first thing to add to a workflow is the trigger, which
784defines when a workflow runs. This is defined using a `when`
785field, which takes in a list of conditions. Each condition
786has the following fields:
787
788- `event`: This is a **required** field that defines when
789 your workflow should run. It's a list that can take one or
790 more of the following values:
791 - `push`: The workflow should run every time a commit is
792 pushed to the repository.
793 - `pull_request`: The workflow should run every time a
794 pull request is made or updated.
795 - `manual`: The workflow can be triggered manually.
796- `branch`: Defines which branches the workflow should run
797 for. If used with the `push` event, commits to the
798 branch(es) listed here will trigger the workflow. If used
799 with the `pull_request` event, updates to pull requests
800 targeting the branch(es) listed here will trigger the
801 workflow. This field has no effect with the `manual`
802 event. Supports glob patterns using `*` and `**` (e.g.,
803 `main`, `develop`, `release-*`). Either `branch` or `tag`
804 (or both) must be specified for `push` events.
805- `tag`: Defines which tags the workflow should run for.
806 Only used with the `push` event - when tags matching the
807 pattern(s) listed here are pushed, the workflow will
808 trigger. This field has no effect with `pull_request` or
809 `manual` events. Supports glob patterns using `*` and `**`
810 (e.g., `v*`, `v1.*`, `release-**`). Either `branch` or
811 `tag` (or both) must be specified for `push` events.
812
813For example, if you'd like to define a workflow that runs
814when commits are pushed to the `main` and `develop`
815branches, or when pull requests that target the `main`
816branch are updated, or manually, you can do so with:
817
818```yaml
819when:
820 - event: ["push", "manual"]
821 branch: ["main", "develop"]
822 - event: ["pull_request"]
823 branch: ["main"]
824```
825
826You can also trigger workflows on tag pushes. For instance,
827to run a deployment workflow when tags matching `v*` are
828pushed:
829
830```yaml
831when:
832 - event: ["push"]
833 tag: ["v*"]
834```
835
836You can even combine branch and tag patterns in a single
837constraint (the workflow triggers if either matches):
838
839```yaml
840when:
841 - event: ["push"]
842 branch: ["main", "release-*"]
843 tag: ["v*", "stable"]
844```
845
846### Engine
847
848Next is the engine on which the workflow should run, defined
849using the **required** `engine` field. The currently
850supported engines are:
851
852- `nixery`: This uses an instance of
853 [Nixery](https://nixery.dev) to run steps, which allows
854 you to add [dependencies](#dependencies) from
855 Nixpkgs (https://github.com/NixOS/nixpkgs). You can
856 search for packages on https://search.nixos.org, and
857 there's a pretty good chance the package(s) you're looking
858 for will be there.
859 See [Nixery engine](#nixery-engine).
860- `microvm`: Runs the whole workflow inside its own
861 microVM. Has configuration features for NixOS images
862 that will let you enable services, do Docker-in-VM, etc.
863 See [microVM engine](#microvm-engine).
864
865Example:
866
867```yaml
868engine: "nixery"
869```
870
871Each engine also adds its own workflow fields (dependencies,
872images, services, and so on). These are documented under
873[Engines](#engines).
874
875### Clone options
876
877When a workflow starts, the first step is to clone the
878repository. You can customize this behavior using the
879**optional** `clone` field. It has the following fields:
880
881- `skip`: Setting this to `true` will skip cloning the
882 repository. This can be useful if your workflow is doing
883 something that doesn't require anything from the
884 repository itself. This is `false` by default.
885- `depth`: This sets the number of commits, or the "clone
886 depth", to fetch from the repository. For example, if you
887 set this to 2, the last 2 commits will be fetched. By
888 default, the depth is set to 1, meaning only the most
889 recent commit will be fetched, which is the commit that
890 triggered the workflow.
891- `submodules`: If you use Git submodules
892 (https://git-scm.com/book/en/v2/Git-Tools-Submodules)
893 in your repository, setting this field to `true` will
894 recursively fetch all submodules. This is `false` by
895 default.
896
897The default settings are:
898
899```yaml
900clone:
901 skip: false
902 depth: 1
903 submodules: false
904```
905
906### Environment
907
908The `environment` field allows you define environment
909variables that will be available throughout the entire
910workflow. **Do not put secrets here, these environment
911variables are visible to anyone viewing the repository. You
912can add secrets for pipelines in your repository's
913settings.**
914
915Example:
916
917```yaml
918environment:
919 GOOS: "linux"
920 GOARCH: "arm64"
921 NODE_ENV: "production"
922 MY_ENV_VAR: "MY_ENV_VALUE"
923```
924
925By default, the following environment variables are set:
926
927- `CI` - Always set to `true` to indicate a CI environment
928- `TANGLED_PIPELINE_ID` - The AT URI of the current pipeline
929- `TANGLED_PIPELINE_KIND` - One of `push`, `pull_request` or
930 `manual`
931- `TANGLED_REPO_KNOT` - The repository's knot hostname
932- `TANGLED_REPO_DID` - The DID of the repository owner
933- `TANGLED_REPO_NAME` - The name of the repository
934- `TANGLED_REPO_DEFAULT_BRANCH` - The default branch of the
935 repository
936- `TANGLED_REPO_URL` - The full URL to the repository
937
938These variables are only available when the pipeline is
939triggered by a push:
940
941- `TANGLED_REF` - The full git reference (e.g.,
942 `refs/heads/main` or `refs/tags/v1.0.0`)
943- `TANGLED_REF_NAME` - The short name of the reference
944 (e.g., `main` or `v1.0.0`)
945- `TANGLED_REF_TYPE` - The type of reference, either
946 `branch` or `tag`
947- `TANGLED_SHA` - The commit SHA that triggered the pipeline
948- `TANGLED_COMMIT_SHA` - Alias for `TANGLED_SHA`
949
950These variables are only available when the pipeline is
951triggered by a pull request:
952
953- `TANGLED_PR_SOURCE_BRANCH` - The source branch of the pull
954 request
955- `TANGLED_PR_TARGET_BRANCH` - The target branch of the pull
956 request
957- `TANGLED_PR_SOURCE_SHA` - The commit SHA of the source
958 branch
959
960### Steps
961
962The `steps` field allows you to define what steps should run
963in the workflow. It's a list of step objects, each with the
964following fields:
965
966- `name`: This field allows you to give your step a name.
967 This name is visible in your workflow runs, and is used to
968 describe what the step is doing.
969- `command`: This field allows you to define a command to
970 run in that step. The step is run in a Bash shell, and the
971 logs from the command will be visible in the pipelines
972 page on the Tangled website. Any dependencies you added in
973 your engine's section (see [Engines](#engines)) will be
974 available to use here.
975- `environment`: Similar to the global
976 [environment](#environment) config, this **optional**
977 field is a key-value map that allows you to set
978 environment variables for the step. **Do not put secrets
979 here, these environment variables are visible to anyone
980 viewing the repository. You can add secrets for pipelines
981 in your repository's settings.**
982
983Example:
984
985```yaml
986steps:
987 - name: "Build backend"
988 command: "go build"
989 environment:
990 GOOS: "darwin"
991 GOARCH: "arm64"
992 - name: "Build frontend"
993 command: "npm run build"
994 environment:
995 NODE_ENV: "production"
996```
997
998## Engines
999
1000The common fields above apply to every workflow. Each engine
1001then adds its own fields on top. Pick an engine with the
1002[`engine`](#engine) field and use the matching section below.
1003
1004### Nixery engine
1005
1006#### Dependencies
1007
1008When you're running a workflow you'll usually need additional
1009dependencies. The `dependencies` field lets you define which
1010dependencies to get, and from where. It's a key-value map,
1011with the key being the registry to fetch dependencies from,
1012and the value being the list of dependencies to fetch.
1013
1014The registry URL syntax can be found [on the nix
1015manual](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-registry-add).
1016
1017Say you want to fetch Node.js and Go from `nixpkgs`, and a
1018package called `my_pkg` you've made from your own registry
1019at your repository at
1020`https://tangled.org/@example.com/my_pkg`. You can define
1021those dependencies like so:
1022
1023```yaml
1024dependencies:
1025 # nixpkgs
1026 nixpkgs:
1027 - nodejs
1028 - go
1029 # unstable
1030 nixpkgs/nixpkgs-unstable:
1031 - bun
1032 # custom registry
1033 git+https://tangled.org/@example.com/my_pkg:
1034 - my_pkg
1035```
1036
1037Now these dependencies are available to use in your
1038workflow!
1039
1040#### Complete nixery workflow
1041
1042```yaml
1043# .tangled/workflows/build.yml
1044
1045when:
1046 - event: ["push", "manual"]
1047 branch: ["main", "develop"]
1048 - event: ["pull_request"]
1049 branch: ["main"]
1050
1051engine: "nixery"
1052
1053# using the default values
1054clone:
1055 skip: false
1056 depth: 1
1057 submodules: false
1058
1059dependencies:
1060 # nixpkgs
1061 nixpkgs:
1062 - nodejs
1063 - go
1064 # custom registry
1065 git+https://tangled.org/@example.com/my_pkg:
1066 - my_pkg
1067
1068environment:
1069 GOOS: "linux"
1070 GOARCH: "arm64"
1071 NODE_ENV: "production"
1072 MY_ENV_VAR: "MY_ENV_VALUE"
1073
1074steps:
1075 - name: "Build backend"
1076 command: "go build"
1077 environment:
1078 GOOS: "darwin"
1079 GOARCH: "arm64"
1080 - name: "Build frontend"
1081 command: "npm run build"
1082 environment:
1083 NODE_ENV: "production"
1084```
1085
1086If you want another example of a workflow, you can look at
1087the one [Tangled uses to build the
1088project](https://tangled.org/@tangled.org/core/blob/master/.tangled/workflows/build.yml).
1089
1090### microVM engine
1091
1092#### Image
1093
1094A workflow picks the image to boot with the top-level `image`
1095field:
1096
1097```yaml
1098engine: microvm
1099image: nixos
1100```
1101
1102There are two flavours of images:
1103
1104- **NixOS images** (e.g. `nixos`): the whole guest is built
1105 with Nix, so you can configure it from the workflow file
1106 itself. The `dependencies`, `services`, `virtualisation`,
1107 `registry` and `caches` fields below are all understood
1108 here, and the guest builds and activates that configuration
1109 before any of your steps run.
1110- **Non-NixOS images** (e.g. `alpine`): there's no NixOS to
1111 configure, so the workflow-level config fields above have
1112 no effect. You still get a full machine to run steps in.
1113
1114The available image names depend on what the spindle operator
1115has installed. `nixos` and `alpine` are examples. If `image`
1116is omitted, the spindle's configured default image is used.
1117
1118#### Dependencies
1119
1120On the microVM engine, `dependencies` is a flat list of
1121packages that are made available to every step. This field
1122only applies to **NixOS images**; for other images you can
1123use the package manager included in a step.
1124
1125The guest builds a [`nix develop`](https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-develop)-style
1126devshell from your dependencies and uses it for each step,
1127so you can, for example, add `pkg-config` and `openssl` and
1128have the `openssl-sys` crate while compiling a Rust project
1129just work.
1130
1131A bare name like `go` is looked up in nixpkgs. You can also
1132point at any flake with the `flakeref#attr` syntax, so
1133`github:nixos/nixpkgs#hello` pulls `hello` straight out of
1134that flake.
1135
1136```yaml
1137dependencies:
1138 - go
1139 - github:nixos/nixpkgs#hello
1140```
1141
1142#### Registry
1143
1144The `registry` field remaps flake references, the same way
1145`nix registry` does. This lets you pin or alias the flakes
1146used by `dependencies`.
1147
1148For example, pin `nixpkgs` to `nixos-unstable` so that the
1149bare `go` above resolves from unstable, and alias your own
1150flake so you can use `myflake#tool` in `dependencies`:
1151
1152```yaml
1153registry:
1154 nixpkgs: github:nixos/nixpkgs/nixos-unstable
1155 myflake: github:me/x
1156```
1157
1158#### Caches
1159
1160The `caches` field is a map of Nix binary cache URL to its
1161trusted public key. These are fed into the spindle's read
1162proxy, so the guest can substitute prebuilt paths from them
1163instead of building everything from scratch.
1164
1165```yaml
1166caches:
1167 https://nix-community.cachix.org: "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
1168```
1169
1170#### Services and virtualisation
1171
1172The `services` and `virtualisation` fields are passed straight
1173through to NixOS. Anything you could write under
1174`services.*` or `virtualisation.*` in a NixOS configuration,
1175you can write here, and it's brought up before any of your
1176steps run.
1177
1178As a convenience, `true` works as shorthand for
1179`.enable = true` anywhere an `enable` option exists (e.g.
1180`virtualisation.docker: true`).
1181
1182```yaml
1183services:
1184 postgresql:
1185 enable: true
1186 ensureDatabases: ["spindle-workflow"]
1187 ensureUsers:
1188 - name: spindle-workflow
1189 ensureDBOwnership: true
1190
1191virtualisation:
1192 docker: true
1193```
1194
1195#### Recipes
1196
1197##### Lint, test and build a Node project
1198
1199```yaml
1200when:
1201 - event: ["push", "pull_request"]
1202 branch: ["main"]
1203
1204engine: microvm
1205image: nixos
1206
1207dependencies:
1208 - pnpm
1209
1210steps:
1211 - name: "Install dependencies"
1212 command: pnpm install --frozen-lockfile
1213 - name: "Lint and test"
1214 command: |
1215 pnpm run lint
1216 pnpm test
1217 - name: "Build"
1218 command: pnpm run build
1219```
1220
1221##### Check formatting
1222
1223```yaml
1224when:
1225 - event: ["push", "pull_request"]
1226 branch: ["main"]
1227
1228engine: microvm
1229image: alpine # slimmer image for checking the formatting
1230
1231steps:
1232 - name: "Install go"
1233 command: apk add go
1234 - name: "Check formatting"
1235 command: test -z $(gofmt -l .)
1236```
1237
1238##### Build a Rust project that links OpenSSL
1239
1240```yaml
1241when:
1242 - event: ["push", "pull_request"]
1243 branch: ["main"]
1244
1245engine: microvm
1246image: nixos
1247
1248dependencies:
1249 - gcc
1250 - cargo
1251 - rustc
1252 - clippy
1253 - rustfmt
1254 - pkg-config # exports PKG_CONFIG_PATH for the libraries below
1255 - openssl # the C library + headers openssl-sys links against
1256
1257steps:
1258 - name: "Check formatting"
1259 command: cargo fmt --check
1260 - name: "Clippy"
1261 command: cargo clippy --all-targets -- -D warnings
1262 - name: "Test"
1263 command: cargo test --all
1264 - name: "Release build"
1265 command: cargo build --release
1266```
1267
1268##### Run migrations and integration tests against PostgreSQL
1269
1270```yaml
1271when:
1272 - event: ["push", "pull_request"]
1273 branch: ["main"]
1274
1275engine: microvm
1276image: nixos
1277
1278environment:
1279 DATABASE_URL: "postgresql:///spindle-workflow?host=/run/postgresql"
1280
1281dependencies:
1282 - gcc
1283 - cargo
1284 - rustc
1285 - pkg-config
1286 - openssl
1287 - sqlx-cli
1288
1289services:
1290 postgresql:
1291 enable: true
1292 # has to be same name as the user for peer auth to work automatically
1293 ensureDatabases: ["spindle-workflow"]
1294 ensureUsers:
1295 - name: spindle-workflow
1296 ensureDBOwnership: true
1297
1298steps:
1299 - name: "Run migrations"
1300 command: sqlx migrate run
1301 - name: "Integration tests"
1302 command: cargo test --all
1303```
1304
1305##### Build and push a Docker image on tag
1306
1307```yaml
1308when:
1309 - event: ["push"]
1310 tag: ["v*"]
1311
1312engine: microvm
1313image: nixos
1314
1315virtualisation:
1316 docker: true
1317
1318steps:
1319 - name: "Build and push to ghcr.io"
1320 command: |
1321 set -euo pipefail
1322
1323 echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
1324 image="ghcr.io/$REGISTRY_USER/myapp:$TANGLED_REF_NAME"
1325
1326 docker build -t "$image" -t "ghcr.io/$REGISTRY_USER/myapp:latest" .
1327 docker push "$image"
1328 docker push "ghcr.io/$REGISTRY_USER/myapp:latest"
1329```
1330
1331##### Deploy to Cloudflare Workers on tag
1332
1333```yaml
1334# .tangled/workflows/deploy.yml
1335when:
1336 - event: ["push"]
1337 tag: ["v*"]
1338
1339engine: microvm
1340image: nixos
1341
1342dependencies:
1343 - pnpm
1344
1345steps:
1346 - name: "Install dependencies"
1347 command: pnpm install --frozen-lockfile
1348 - name: "Deploy worker"
1349 # `wrangler` picks up `CLOUDFLARE_API_TOKEN` from the env.
1350 # set it under **Settings → Secrets**.
1351 command: pnpm exec wrangler deploy
1352```
1353
1354##### Publish a release artifact
1355
1356```yaml
1357when:
1358 - event: ["push"]
1359 tag: ["v*"] # trigger on versions
1360
1361engine: microvm
1362image: nixos
1363
1364dependencies:
1365 - go
1366
1367steps:
1368 - name: "Build release binary"
1369 command: |
1370 mkdir -p dist
1371 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w" -o dist/myapp ./cmd/myapp
1372
1373 - name: "Publish artifact record"
1374 command: |
1375 set -euo pipefail
1376 # change this if you're not on `tngl.sh`
1377 PDS="https://tngl.sh"
1378 # also update this to your handle or did
1379 ATP_IDENTIFIER="user.tngl.sh"
1380 ARTIFACT_PATH="dist/myapp"
1381 ARTIFACT_NAME="myapp"
1382
1383 # set `ATP_APP_PASSWORD` under **Settings → Secrets**
1384 session=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.server.createSession" \
1385 -H "Content-Type: application/json" \
1386 -d "{\"identifier\":\"$ATP_IDENTIFIER\",\"password\":\"$ATP_APP_PASSWORD\"}")
1387 jwt=$(echo "$session" | jq -r .accessJwt)
1388 did=$(echo "$session" | jq -r .did)
1389
1390 # upload the binary as a blob
1391 blob=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.uploadBlob" \
1392 -H "Authorization: Bearer $jwt" \
1393 -H "Content-Type: application/octet-stream" \
1394 --data-binary @"$ARTIFACT_PATH")
1395
1396 # note that this requires an annotated tag (`git tag -a v1.0.0 -m ...`)
1397 tag_hash=$(git rev-parse "$TANGLED_REF_NAME^{tag}")
1398 tag_bytes=$(printf '%s' "$tag_hash" | xxd -r -p | base64 | tr -d '=')
1399
1400 # the sh.tangled.repo.artifact record for your artifact
1401 record=$(jq -n \
1402 --arg did "$did" \
1403 --arg tag "$tag_bytes" \
1404 --arg name "$ARTIFACT_NAME" \
1405 --arg repo "$TANGLED_REPO_URL" \
1406 --arg created "$(date -Iseconds)" \
1407 --argjson blob "$(echo "$blob" | jq .blob)" '{
1408 repo: $did,
1409 collection: "sh.tangled.repo.artifact",
1410 validate: false,
1411 record: {
1412 "$type": "sh.tangled.repo.artifact",
1413 tag: {"$bytes": $tag},
1414 name: $name,
1415 repo: $repo,
1416 artifact: $blob,
1417 createdAt: $created
1418 }
1419 }')
1420
1421 # create the record on the PDS
1422 curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.createRecord" \
1423 -H "Authorization: Bearer $jwt" \
1424 -H "Content-Type: application/json" \
1425 -d "$record"
1426```
1427
1428## Self-hosting guide
1429
1430### Prerequisites
1431
1432- Go
1433- For the **nixery** engine: Docker (or Podman with Docker
1434 compatibility enabled).
1435- For the **microVM** engine: a Linux host with KVM, plus the
1436 microVM host dependencies described in [Running microVM
1437 workflows](#running-microvm-workflows).
1438
1439### Configuration
1440
1441Spindle is configured using environment variables. The following environment variables are available:
1442
1443- `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`).
1444- `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`).
1445- `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required).
1446- `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`).
1447- `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`).
1448- `SPINDLE_SERVER_OWNER`: The DID of the owner (required).
1449- `SPINDLE_SERVER_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`).
1450- `SPINDLE_SERVER_DOCKER_SOCKET`: Path to Docker socket to expose to invoked Spindle containers (default: `""`).
1451- `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`).
1452- `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`).
1453
1454For the microVM engine, the following are also available
1455(prefix `SPINDLE_MICROVM_PIPELINES_`):
1456
1457- `SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`: Directory containing
1458 microVM images (**required** to use the engine). See
1459 [Running microVM workflows](#running-microvm-workflows).
1460- `SPINDLE_MICROVM_PIPELINES_DEFAULT_IMAGE`: Image used when a
1461 workflow doesn't set `image` (default: `"nixos-x86_64"`).
1462- `SPINDLE_MICROVM_PIPELINES_OVERLAY_DIR`: Where per-workflow
1463 temporary disks are created (default: the system temp dir).
1464- `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM`: Use KVM hardware
1465 acceleration (default: `true`). Without KVM, guests fall
1466 back to slow software emulation.
1467- `SPINDLE_MICROVM_PIPELINES_WORKFLOW_TIMEOUT`: Default
1468 workflow timeout (default: `"5m"`).
1469
1470Optional resource limits (a value of `0` disables that
1471limit). The limits cap usage across all running microVM
1472workflows:
1473
1474- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_MEMORY_MIB`
1475- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_VCPUS`
1476- `SPINDLE_MICROVM_PIPELINES_MAX_TOTAL_DISK_MIB`
1477
1478Optional cgroup enforcement:
1479
1480- `SPINDLE_MICROVM_PIPELINES_ENABLE_CGROUPS`: Place each
1481 workflow's QEMU and slirp4netns in a per-workflow cgroup=
1482 (default: `false`).
1483- `SPINDLE_MICROVM_PIPELINES_CGROUP_PARENT`: Parent cgroup;
1484 `self` resolves the spindle service's own cgroup (default:
1485 `"self"`).
1486- `SPINDLE_MICROVM_PIPELINES_CGROUP_PIDS_MAX`: Max processes
1487 per workflow cgroup (default: `4096`).
1488- `SPINDLE_MICROVM_PIPELINES_CGROUP_SWAP_MAX_MIB`: Max swap
1489 per workflow cgroup (default: `0`, no swap).
1490- `SPINDLE_MICROVM_PIPELINES_CGROUP_SUPERVISOR_MEMORY_MIN_MIB`:
1491 Memory protected for spindle itself so it isn't OOM-killed
1492 before the workflows (default: `512`).
1493
1494To push paths built inside microVMs back to a shared Nix
1495cache (and read from it), configure the cache (prefix
1496`SPINDLE_NIX_CACHE_`):
1497
1498- `SPINDLE_NIX_CACHE_READ_URLS`: Comma-separated binary cache
1499 URLs the guest reads from.
1500- `SPINDLE_NIX_CACHE_TRUSTED_PUBLIC_KEYS`: Comma-separated
1501 trusted public keys for those caches.
1502- `SPINDLE_NIX_CACHE_UPLOAD_URL`: Cache URL that paths built
1503 in the guest are uploaded to.
1504
1505### Running spindle
1506
15071. **Set the environment variables.** For example:
1508
1509 ```shell
1510 export SPINDLE_SERVER_HOSTNAME="your-hostname"
1511 export SPINDLE_SERVER_OWNER="your-did"
1512 ```
1513
15142. **Build the Spindle binary.**
1515
1516 ```shell
1517 cd core
1518 go mod download
1519 go build -o cmd/spindle/spindle cmd/spindle/main.go
1520 ```
1521
15223. **Create the log directory.**
1523
1524 ```shell
1525 sudo mkdir -p /var/log/spindle
1526 sudo chown $USER:$USER -R /var/log/spindle
1527 ```
1528
15294. **Run the Spindle binary.**
1530
1531 ```shell
1532 ./cmd/spindle/spindle
1533 ```
1534
1535Spindle will now start, connect to the Jetstream server, and begin processing pipelines.
1536
1537### Running microVM workflows
1538
1539The microVM engine needs a few extra things on the host, and
1540it needs images to boot.
1541
1542#### Host dependencies
1543
1544microVM workflows depend on a handful of host tools and
1545devices. spindle checks for the ones an image needs right
1546before it launches, so a missing dependency surfaces as a
1547clear error. You'll need:
1548
1549- `qemu`: the runner. The QEMU binary for the image's arch
1550 must be present (e.g. `qemu-system-x86_64`).
1551- `mkfs.ext4` (from `e2fsprogs`): to format the per-workflow
1552 writable volumes.
1553- [`slirp4netns`](https://github.com/rootless-containers/slirp4netns#install),
1554 `ip` (from `iproute2`), `mount` and `unshare` (from `util-linux`):
1555 used to sandbox guest networking.
1556- `/dev/kvm`: for hardware acceleration (unless you disable
1557 KVM with `SPINDLE_MICROVM_PIPELINES_ENABLE_KVM=false`).
1558- `/dev/vhost-vsock`: the guest agent talks to spindle over
1559 vsock.
1560
1561On NixOS, the [spindle
1562module](https://tangled.org/tangled.org/core/blob/master/nix/modules/spindle.nix)
1563puts `qemu`, `e2fsprogs`, `slirp4netns`, `iproute2` and
1564`util-linux` on the service's `PATH` for you.
1565
1566#### Building images
1567
1568Images are built with Nix. The flake exposes packages for the
1569two stock images (use the `-tarball` prefixed ones for a gzipped
1570tarball you can copy to another host):
1571
1572```shell
1573# a NixOS image
1574nix build .#spindle-nixos-image
1575# an Alpine image
1576nix build .#spindle-alpine-image
1577```
1578
1579#### Installing images
1580
1581Spindle looks for images in
1582`SPINDLE_MICROVM_PIPELINES_IMAGE_DIR`. An image is resolved by
1583the name a workflow puts in its `image` field, matched
1584literally against what's on disk:
1585
15861. a directory `<name>/` containing a `spec.json` (next to the
1587 kernel/initrd/store-disk), or
15882. a flat `<name>.json` self-contained spec.
1589
1590Resolution depends only on the name and what's on disk, never
1591on the host doing the resolving, so the same workflow resolves
1592to the same image on every spindle. If you keep multiple
1593arches side by side, you can name them `<name>-<arch>` (e.g.
1594`nixos-x86_64`, `alpine-aarch64`); the suffix is just part of
1595the name. To make a name like `nixos` work if you are hosting
1596multiple arches, you can use symlinks.
1597
1598On NixOS, you'll most likely want to use `systemd.tmpfiles.rules`
1599to set these up declaratively.
1600
1601## Architecture
1602
1603Spindle is a small CI runner service. Here's a high-level overview of how it operates:
1604
1605- Listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and
1606 [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream.
1607- When a new repo record comes through (typically when you add a spindle to a
1608 repo from the settings), spindle then resolves the underlying knot and
1609 subscribes to repo events (see:
1610 [`sh.tangled.pipeline`](/lexicons/pipeline.json)).
1611- The spindle engine then handles execution of the pipeline, with results and
1612 logs beamed on the spindle event stream over WebSocket
1613
1614### The engines
1615
1616Spindle has two execution backends, picked per-workflow with
1617the [`engine`](#engine) field:
1618
1619- **nixery**: executes each step in a fresh Docker container
1620 (Podman works too, if Docker compatibility is enabled so
1621 that `/run/docker.sock` is created), with state persisted
1622 across steps within the `/tangled/workspace` directory. The
1623 base image for the container is constructed on the fly using
1624 [Nixery](https://nixery.dev), which is/rhandy for caching
1625 layers for frequently used packages.
1626- **microvm**: runs the whole workflow inside its own
1627 microVM, supporting different images, with extra
1628 configuration for NixOS images (e.g. services in workflow file)
1629 See the [engine
1630 README](https://tangled.org/tangled.org/core/blob/master/spindle/engines/microvm/README.md)
1631 for the architecture in depth.
1632
1633The pipeline manifest is [specified here](https://docs.tangled.org/spindles.html#pipelines).
1634
1635## Secrets with openbao
1636
1637This document covers setting up spindle to use OpenBao for secrets
1638management via OpenBao Proxy instead of the default SQLite backend.
1639
1640### Overview
1641
1642Spindle now uses OpenBao Proxy for secrets management. The proxy handles
1643authentication automatically using AppRole credentials, while spindle
1644connects to the local proxy instead of directly to the OpenBao server.
1645
1646This approach provides better security, automatic token renewal, and
1647simplified application code.
1648
1649### Installation
1650
1651Install OpenBao from Nixpkgs:
1652
1653```bash
1654nix shell nixpkgs#openbao # for a local server
1655```
1656
1657### Setup
1658
1659The setup process can is documented for both local development and production.
1660
1661#### Local development
1662
1663Start OpenBao in dev mode:
1664
1665```bash
1666bao server -dev -dev-root-token-id="root" -dev-listen-address=127.0.0.1:8201
1667```
1668
1669This starts OpenBao on `http://localhost:8201` with a root token.
1670
1671Set up environment for bao CLI:
1672
1673```bash
1674export BAO_ADDR=http://localhost:8200
1675export BAO_TOKEN=root
1676```
1677
1678#### Production
1679
1680You would typically use a systemd service with a
1681configuration file. Refer to
1682[@tangled.org/infra](https://tangled.org/@tangled.org/infra)
1683for how this can be achieved using Nix.
1684
1685Then, initialize the bao server:
1686
1687```bash
1688bao operator init -key-shares=1 -key-threshold=1
1689```
1690
1691This will print out an unseal key and a root key. Save them
1692somewhere (like a password manager). Then unseal the vault
1693to begin setting it up:
1694
1695```bash
1696bao operator unseal <unseal_key>
1697```
1698
1699All steps below remain the same across both dev and
1700production setups.
1701
1702#### Configure openbao server
1703
1704Create the spindle KV mount:
1705
1706```bash
1707bao secrets enable -path=spindle -version=2 kv
1708```
1709
1710Set up AppRole authentication and policy:
1711
1712Create a policy file `spindle-policy.hcl`:
1713
1714```hcl
1715# Full access to spindle KV v2 data
1716path "spindle/data/*" {
1717 capabilities = ["create", "read", "update", "delete"]
1718}
1719
1720# Access to metadata for listing and management
1721path "spindle/metadata/*" {
1722 capabilities = ["list", "read", "delete", "update"]
1723}
1724
1725# Allow listing at root level
1726path "spindle/" {
1727 capabilities = ["list"]
1728}
1729
1730# Required for connection testing and health checks
1731path "auth/token/lookup-self" {
1732 capabilities = ["read"]
1733}
1734```
1735
1736Apply the policy and create an AppRole:
1737
1738```bash
1739bao policy write spindle-policy spindle-policy.hcl
1740bao auth enable approle
1741bao write auth/approle/role/spindle \
1742 token_policies="spindle-policy" \
1743 token_ttl=1h \
1744 token_max_ttl=4h \
1745 bind_secret_id=true \
1746 secret_id_ttl=0 \
1747 secret_id_num_uses=0
1748```
1749
1750Get the credentials:
1751
1752```bash
1753# Get role ID (static)
1754ROLE_ID=$(bao read -field=role_id auth/approle/role/spindle/role-id)
1755
1756# Generate secret ID
1757SECRET_ID=$(bao write -f -field=secret_id auth/approle/role/spindle/secret-id)
1758
1759echo "Role ID: $ROLE_ID"
1760echo "Secret ID: $SECRET_ID"
1761```
1762
1763#### Create proxy configuration
1764
1765Create the credential files:
1766
1767```bash
1768# Create directory for OpenBao files
1769mkdir -p /tmp/openbao
1770
1771# Save credentials
1772echo "$ROLE_ID" > /tmp/openbao/role-id
1773echo "$SECRET_ID" > /tmp/openbao/secret-id
1774chmod 600 /tmp/openbao/role-id /tmp/openbao/secret-id
1775```
1776
1777Create a proxy configuration file `/tmp/openbao/proxy.hcl`:
1778
1779```hcl
1780# OpenBao server connection
1781vault {
1782 address = "http://localhost:8200"
1783}
1784
1785# Auto-Auth using AppRole
1786auto_auth {
1787 method "approle" {
1788 mount_path = "auth/approle"
1789 config = {
1790 role_id_file_path = "/tmp/openbao/role-id"
1791 secret_id_file_path = "/tmp/openbao/secret-id"
1792 }
1793 }
1794
1795 # Optional: write token to file for debugging
1796 sink "file" {
1797 config = {
1798 path = "/tmp/openbao/token"
1799 mode = 0640
1800 }
1801 }
1802}
1803
1804# Proxy listener for spindle
1805listener "tcp" {
1806 address = "127.0.0.1:8201"
1807 tls_disable = true
1808}
1809
1810# Enable API proxy with auto-auth token
1811api_proxy {
1812 use_auto_auth_token = true
1813}
1814
1815# Enable response caching
1816cache {
1817 use_auto_auth_token = true
1818}
1819
1820# Logging
1821log_level = "info"
1822```
1823
1824#### Start the proxy
1825
1826Start OpenBao Proxy:
1827
1828```bash
1829bao proxy -config=/tmp/openbao/proxy.hcl
1830```
1831
1832The proxy will authenticate with OpenBao and start listening on
1833`127.0.0.1:8201`.
1834
1835#### Configure spindle
1836
1837Set these environment variables for spindle:
1838
1839```bash
1840export SPINDLE_SERVER_SECRETS_PROVIDER=openbao
1841export SPINDLE_SERVER_SECRETS_OPENBAO_PROXY_ADDR=http://127.0.0.1:8201
1842export SPINDLE_SERVER_SECRETS_OPENBAO_MOUNT=spindle
1843```
1844
1845On startup, spindle will now connect to the local proxy,
1846which handles all authentication automatically.
1847
1848### Production setup for proxy
1849
1850For production, you'll want to run the proxy as a service:
1851
1852Place your production configuration in
1853`/etc/openbao/proxy.hcl` with proper TLS settings for the
1854vault connection.
1855
1856### Verifying setup
1857
1858Test the proxy directly:
1859
1860```bash
1861# Check proxy health
1862curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/sys/health
1863
1864# Test token lookup through proxy
1865curl -H "X-Vault-Request: true" http://127.0.0.1:8201/v1/auth/token/lookup-self
1866```
1867
1868Test OpenBao operations through the server:
1869
1870```bash
1871# List all secrets
1872bao kv list spindle/
1873
1874# Add a test secret via the spindle API, then check it exists
1875bao kv list spindle/repos/
1876
1877# Get a specific secret
1878bao kv get spindle/repos/your_repo_path/SECRET_NAME
1879```
1880
1881### How it works
1882
1883- Spindle connects to OpenBao Proxy on localhost (typically
1884 port 8200 or 8201)
1885- The proxy authenticates with OpenBao using AppRole
1886 credentials
1887- All spindle requests go through the proxy, which injects
1888 authentication tokens
1889- Secrets are stored at
1890 `spindle/repos/{sanitized_repo_path}/{secret_key}`
1891- Repository paths like `did:plc:alice/myrepo` become
1892 `did_plc_alice_myrepo`
1893- The proxy handles all token renewal automatically
1894- Spindle no longer manages tokens or authentication
1895 directly
1896
1897### Troubleshooting
1898
1899**Connection refused**: Check that the OpenBao Proxy is
1900running and listening on the configured address.
1901
1902**403 errors**: Verify the AppRole credentials are correct
1903and the policy has the necessary permissions.
1904
1905**404 route errors**: The spindle KV mount probably doesn't
1906exist—run the mount creation step again.
1907
1908**Proxy authentication failures**: Check the proxy logs and
1909verify the role-id and secret-id files are readable and
1910contain valid credentials.
1911
1912**Secret not found after writing**: This can indicate policy
1913permission issues. Verify the policy includes both
1914`spindle/data/*` and `spindle/metadata/*` paths with
1915appropriate capabilities.
1916
1917Check proxy logs:
1918
1919```bash
1920# If running as systemd service
1921journalctl -u openbao-proxy -f
1922
1923# If running directly, check the console output
1924```
1925
1926Test AppRole authentication manually:
1927
1928```bash
1929bao write auth/approle/login \
1930 role_id="$(cat /tmp/openbao/role-id)" \
1931 secret_id="$(cat /tmp/openbao/secret-id)"
1932```
1933
1934# Webhooks
1935
1936Webhooks allow you to receive HTTP POST notifications when events occur in your repositories. This enables you to integrate Tangled with external services, trigger CI/CD pipelines, send notifications, or automate workflows.
1937
1938## Overview
1939
1940Webhooks send HTTP POST requests to URLs you configure whenever specific events happen. Currently, Tangled supports push events, with more event types coming soon.
1941
1942## Configuring webhooks
1943
1944To set up a webhook for your repository:
1945
19461. Navigate to your repository
19472. Go to **Settings → Hooks**
19483. Click **new webhook**
19494. Configure your webhook:
1950 - **Payload URL**: The endpoint that will receive the webhook POST requests
1951 - **Secret**: An optional secret key for verifying webhook authenticity (leave blank to send unsigned webhooks)
1952 - **Events**: Select which events trigger the webhook (currently only push events)
1953 - **Active**: Toggle whether the webhook is enabled
1954
1955## Webhook payload
1956
1957### Push
1958
1959When a push event occurs, Tangled sends a POST request with a JSON payload of the format:
1960
1961```json
1962{
1963 "after": "7b320e5cbee2734071e4310c1d9ae401d8f6cab5",
1964 "before": "c04ddf64eddc90e4e2a9846ba3b43e67a0e2865e",
1965 "pusher": {
1966 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq"
1967 },
1968 "ref": "refs/heads/main",
1969 "repository": {
1970 "clone_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1971 "created_at": "2025-09-15T08:57:23Z",
1972 "description": "an example repository",
1973 "fork": false,
1974 "full_name": "did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1975 "html_url": "https://tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1976 "name": "some-repo",
1977 "open_issues_count": 5,
1978 "owner": {
1979 "did": "did:plc:hwevmowznbiukdf6uk5dwrrq"
1980 },
1981 "ssh_url": "ssh://git@tangled.org/did:plc:hwevmowznbiukdf6uk5dwrrq/some-repo",
1982 "stars_count": 1,
1983 "updated_at": "2025-09-15T08:57:23Z"
1984 }
1985}
1986```
1987
1988## HTTP headers
1989
1990Each webhook request includes the following headers:
1991
1992- `Content-Type: application/json`
1993- `User-Agent: Tangled-Hook/<short-sha>` — User agent with short SHA of the commit
1994- `X-Tangled-Event: push` — The event type
1995- `X-Tangled-Hook-ID: <webhook-id>` — The webhook ID
1996- `X-Tangled-Delivery: <uuid>` — Unique delivery ID
1997- `X-Tangled-Signature-256: sha256=<hmac>` — HMAC-SHA256 signature (if secret configured)
1998
1999## Verifying webhook signatures
2000
2001If you configured a secret, you should verify the webhook signature to ensure requests are authentic. For example, in Go:
2002
2003```go
2004package main
2005
2006import (
2007 "crypto/hmac"
2008 "crypto/sha256"
2009 "encoding/hex"
2010 "io"
2011 "net/http"
2012 "strings"
2013)
2014
2015func verifySignature(payload []byte, signatureHeader, secret string) bool {
2016 // Remove 'sha256=' prefix from signature header
2017 signature := strings.TrimPrefix(signatureHeader, "sha256=")
2018
2019 // Compute expected signature
2020 mac := hmac.New(sha256.New, []byte(secret))
2021 mac.Write(payload)
2022 expected := hex.EncodeToString(mac.Sum(nil))
2023
2024 // Use constant-time comparison to prevent timing attacks
2025 return hmac.Equal([]byte(signature), []byte(expected))
2026}
2027
2028func webhookHandler(w http.ResponseWriter, r *http.Request) {
2029 // Read the request body
2030 payload, err := io.ReadAll(r.Body)
2031 if err != nil {
2032 http.Error(w, "Bad request", http.StatusBadRequest)
2033 return
2034 }
2035
2036 // Get signature from header
2037 signatureHeader := r.Header.Get("X-Tangled-Signature-256")
2038
2039 // Verify signature
2040 if signatureHeader != "" && verifySignature(payload, signatureHeader, yourSecret) {
2041 // Webhook is authentic, process it
2042 processWebhook(payload)
2043 w.WriteHeader(http.StatusOK)
2044 } else {
2045 http.Error(w, "Invalid signature", http.StatusUnauthorized)
2046 }
2047}
2048```
2049
2050## Delivery retries
2051
2052Webhooks are automatically retried on failure:
2053
2054- **3 total attempts** (1 initial + 2 retries)
2055- **Exponential backoff** starting at 1 second, max 10 seconds
2056- **Retried on**:
2057 - Network errors
2058 - HTTP 5xx server errors
2059- **Not retried on**:
2060 - HTTP 4xx client errors (bad request, unauthorized, etc.)
2061
2062### Timeouts
2063
2064Webhook requests timeout after 30 seconds. If your endpoint needs more time:
2065
20661. Respond with 200 OK immediately
20672. Process the webhook asynchronously in the background
2068
2069## Example integrations
2070
2071### Discord notifications
2072
2073```javascript
2074app.post("/webhook", (req, res) => {
2075 const payload = req.body;
2076
2077 fetch("https://discord.com/api/webhooks/...", {
2078 method: "POST",
2079 headers: { "Content-Type": "application/json" },
2080 body: JSON.stringify({
2081 content: `New push to ${payload.repository.full_name}`,
2082 embeds: [
2083 {
2084 title: `${payload.pusher.did} pushed to ${payload.ref}`,
2085 url: payload.repository.html_url,
2086 color: 0x00ff00,
2087 },
2088 ],
2089 }),
2090 });
2091
2092 res.status(200).send("OK");
2093});
2094```
2095
2096# Migrating knots and spindles
2097
2098Sometimes, non-backwards compatible changes are made to the
2099knot/spindle XRPC APIs. If you host a knot or a spindle, you
2100will need to follow this guide to upgrade. Typically, this
2101only requires you to deploy the newest version.
2102
2103This document is laid out in reverse-chronological order.
2104Newer migration guides are listed first, and older guides
2105are further down the page.
2106
2107## Upgrading to v1.15.0-alpha
2108
2109With v1.15.0-alpha, a knot itself owns its members and
2110per-repo collaborators directly. Previously this data was sourced from
2111PDS records (`sh.tangled.knot.member` and `sh.tangled.repo.collaborator`)
2112that the appview and the knot both read off the firehose.
2113The knot is now the source of truth and serves them over XRPC instead:
2114
2115- `sh.tangled.knot.addMember`, `sh.tangled.knot.removeMember`, `sh.tangled.knot.listMembers`
2116- `sh.tangled.repo.addCollaborator`, `sh.tangled.repo.removeCollaborator`, `sh.tangled.repo.listCollaborators`
2117
2118Until your knot is upgraded, the appview keeps reading its
2119members and collaborators from the old firehose-sourced records.
2120Upgrade to move your knot onto knot-owned access control.
2121
2122- Upgrade to the latest tag (v1.15.0 or above)
2123- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2124 hit the "retry" button to verify your knot
2125
2126## Upgrading to v1.14.0-alpha
2127
2128Starting with v1.14.0-alpha, the fully knot uses the repoDID as its
2129canonical handle for repositories. This unlocks repository
2130renames from the appview UI and changes the wire format for
2131the following lexicons (`sh.tangled.repo.pull`, `sh.tangled.repo.collaborator`,
2132`sh.tangled.repo.issue`, `sh.tangled.git.refUpdate`).
2133
2134Knots that have not been upgraded may silently drop new push
2135events, pull requests, issues, and collaborator invites for
2136repositories they host until upgraded. So upgrade please!!!
2137
2138- Upgrade to the latest tag (v1.14.0 or above)
2139- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2140 hit the "retry" button to verify your knot
2141
2142## Upgrading to v1.13.0-alpha
2143
2144Starting with v1.13.0-alpha, every repository on a knot is
2145assigned a DID. This makes repositories stable across
2146renames and transfers.
2147
2148When you upgrade your knot to this version, the server will
2149automatically mint DIDs for all existing repositories on
2150startup. This is a one-time process and you may see
2151additional log output during the first boot as DIDs are
2152assigned.
2153
2154- Upgrade to the latest tag (v1.13.0 or above)
2155- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2156 hit the "retry" button to verify your knot
2157
2158## Upgrading from v1.8.x
2159
2160After v1.8.2, the HTTP API for knots and spindles has been
2161deprecated and replaced with XRPC. Repositories on outdated
2162knots will not be viewable from the appview. Upgrading is
2163straightforward however.
2164
2165For knots:
2166
2167- Upgrade to the latest tag (v1.9.0 or above)
2168- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2169 hit the "retry" button to verify your knot
2170
2171For spindles:
2172
2173- Upgrade to the latest tag (v1.9.0 or above)
2174- Head to the [spindle
2175 dashboard](https://tangled.org/settings/spindles) and hit the
2176 "retry" button to verify your spindle
2177
2178## Upgrading from v1.7.x
2179
2180After v1.7.0, knot secrets have been deprecated. You no
2181longer need a secret from the appview to run a knot. All
2182authorized commands to knots are managed via [Inter-Service
2183Authentication](https://atproto.com/specs/xrpc#inter-service-authentication-jwt).
2184Knots will be read-only until upgraded.
2185
2186Upgrading is quite easy, in essence:
2187
2188- `KNOT_SERVER_SECRET` is no more, you can remove this
2189 environment variable entirely
2190- `KNOT_SERVER_OWNER` is now required on boot, set this to
2191 your DID. You can find your DID in the
2192 [settings](https://tangled.org/settings) page.
2193- Restart your knot once you have replaced the environment
2194 variable
2195- Head to the [knot dashboard](https://tangled.org/settings/knots) and
2196 hit the "retry" button to verify your knot. This simply
2197 writes a `sh.tangled.knot` record to your PDS.
2198
2199If you use the nix module, simply bump the flake to the
2200latest revision, and change your config block like so:
2201
2202```diff
2203 services.tangled.knot = {
2204 enable = true;
2205 server = {
2206- secretFile = /path/to/secret;
2207+ owner = "did:plc:foo";
2208 };
2209 };
2210```
2211
2212# Bobbin
2213
2214Bobbin is an API appview for Tangled records. It serves XRPC
2215endpoints for `sh.tangled.*`, with it you can get repos,
2216issues, pulls, comments, follows, stars, labels, pipelines,
2217and profiles. It is read-only, there is no auth, since that
2218should all be handled direct-to-PDS and knot respectively.
2219
2220**Bobbin has no permanent storage**.
2221
2222It is only a glorified edge index, in the graph theory
2223sense. Additionally it has a record cache, re-filled on
2224demand. All other data that Bobbin serves comes live from
2225PDSes & knots.
2226
2227## What Bobbin needs
2228
2229The way that Bobbin is able to pull off being
2230so stateless is by moving state upstream.
2231Primarily it depends on an instance of
2232[Hydrant](https://tangled.org/did:plc:6v3ul2ptnqctyxwkz5ti4amn)
2233, which is the service that gives an event stream
2234for Bobbin to quickly backfill from on every restart.
2235Backfilling ought to take less than a couple of minutes
2236maximum. If the upstream instance of Hydrant fails
2237while Bobbin is live, its list/count endpoints stop
2238advancing and report a stale cursor. Single-lookups
2239will continue working, due to the second dependency:
2240[Slingshot](https://tangled.org/did:plc:c7mc2fn47ihdihul4vjwsuy3/tree/main/slingshot).
2241Slingshot fetches individual records & resolves identities.
2242If the upstream instance of Slingshot fails, single-lookups
2243will fail with a `502` error. There are some aggregation
2244endpoints that use Slingshot for hydrating, which will also
2245fail.
2246
2247A soft dependency that ought to exist for Bobbin to operate
2248correctly is simply the plethora of knots that are out
2249there, that Bobbin talks to directly for git data and, for
2250knots at v1.15+, members & collaborators.
2251
2252## Building Bobbin
2253
2254Bobbin is under [Tangled's core monorepo, under bobbin/](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/tree/master/bobbin).
2255Here's an easy local debug-build:
2256
2257```sh
2258cargo build -p bobbin
2259```
2260
2261Bobbin loves being in a container. When using
2262`bobbin/containerfiles/bobbin.Containerfile`, it runs `cargo
2263build --release --bin bobbin --package bobbin` within a
2264little Debian runtime, exposing port 8090.
2265
2266## Configuration
2267
2268The best way to configure Bobbin is via a toml config file.
2269There's an `example.toml` in [Bobbin's subdir](https://tangled.org/did:plc:j5hmlfdrwkvtxm7cjmu7j2is/blob/master/bobbin/example.toml).
2270Every value is overridable by a `BOBBIN_*` env var.
2271The load order is env, then `--config <path>`, then
2272`/etc/bobbin/config.toml`, then built-in defaults.
2273
2274Load and check a config without starting the server:
2275
2276```sh
2277bobbin --config config.toml validate
2278```
2279
2280Minimal config is the two upstream URLs. The hydrant URL
2281takes `ws://` or `wss://`. An `http://` or `https://`
2282URL is rewritten to the matching websocket scheme at
2283connection-time.
2284
2285```toml
2286[server]
2287binds = ["127.0.0.1:8090"]
2288
2289# Loopback-only & can leave empty to disable debug introspection.
2290debug_bind = "127.0.0.1:8091"
2291
2292[hydrant]
2293url = "https://hydrant.example.com"
2294
2295[slingshot]
2296url = "https://slingshot.example.com"
2297```
2298
2299> 🦪 Lewis
2300>
2301> At time of writing, we (Tangled) don't host public
2302> instances of Hydrant or Slingshot. You will have to
2303> find public instances or spin these up yourself! :P
2304
2305Take a gander in the project's example.toml for an
2306exhaustive list of things to configure.
2307
2308You will discover fun things such as a configurable adaptive
2309loop that watches the cgroup memory limit & throttles heavy
2310requests under pressure. It only works if it detects a
2311cgroup limit is present. The config for that is in the
2312`[backpressure]` block of the config template.
2313
2314## Running Bobbin
2315
2316Start the server using a config toml:
2317
2318```bash
2319bobbin --config config.toml
2320```
2321Bobbin wakes up in a cold sweat and immediately gets to
2322work:
23231. It binds its listeners, connects to the Hydrant stream
2324 in the background.
23252. It serves requests from the first
2326 moment it's alive, even before the Hydrant stream connects
2327 or finishes catching up. Having a cold Hydrant itself
2328 costs only latency and approximate counts.
2329
2330## The API
2331
2332**Single lookups** take a record's AT-URI.
2333
2334- `getRepo` takes the repo URI:
2335
2336```sh
2337curl "$BOBBIN/xrpc/sh.tangled.repo.getRepo?repo=at://did:plc:boltless/sh.tangled.repo/squid"
2338```
2339```json
2340{
2341 "uri": "at://did:plc:boltless/sh.tangled.repo/squid",
2342 "cid": "bafyrei...",
2343 "value": { "$type": "sh.tangled.repo", "knot": "knot1.tangled.sh", "description": "...", "createdAt": "..." }
2344}
2345```
2346
2347- `getProfile` takes the full profile record URI, so a bare
2348 handle or DID will not resolve:
2349
2350```sh
2351curl "$BOBBIN/xrpc/sh.tangled.actor.getProfile?actor=at://did:plc:boltless/sh.tangled.actor.profile/self"
2352```
2353
2354- If Slingshot cannot serve the record, the response is `502`:
2355
2356```json
2357{ "error": "UpstreamFailed", "message": "upstream unavailable: ..." }
2358```
2359
2360**Aggregation** endpoints come in `list*` and `count*` pairs,
2361each with a `*By` sibling, and require a `subject` query param.
2362
2363- `listRepos` and `countRepos` key on the owner DID:
2364
2365```sh
2366curl "$BOBBIN/xrpc/sh.tangled.repo.countRepos?subject=did:plc:boltless"
2367```
2368```json
2369{ "count": 7, "distinctAuthors": 1 }
2370```
2371
2372```sh
2373curl "$BOBBIN/xrpc/sh.tangled.repo.listRepos?subject=did:plc:boltless&limit=3"
2374```
2375```json
2376{ "items": [ { "uri": "at://did:plc:boltless/sh.tangled.repo/squid", "cid": "bafyrei...", "value": { } } ], "cursor": null }
2377```
2378
2379- Bobbin validates the subject per collection. Here a repo URI
2380 is passed where a bare DID is required, so the call returns a
2381 `400`:
2382
2383```sh
2384curl "$BOBBIN/xrpc/sh.tangled.graph.listFollows?subject=at://did:plc:boltless/sh.tangled.repo/squid"
2385```
2386```json
2387{ "error": "InvalidRequest", "message": "invalid request: subject must be a bare did, got at-uri with collection sh.tangled.repo" }
2388```
2389
2390**Search** is a single endpoint over an in-mem full-text
2391index:
2392
2393```sh
2394curl "$BOBBIN/xrpc/sh.tangled.search.query?q=tangled&limit=2"
2395```
2396```json
2397{ "hits": [ { "uri": "at://...", "cid": "...", "nsid": "sh.tangled.repo", "score": 27.1, "value": { } } ], "cursor": null }
2398```
2399
2400**Git data** such as blob, tree, diff, log, and archive proxies
2401straight to the repo's knot, streamed back without caching.
2402
2403## Coverage and warm-up
2404
2405- While the edge index is catching up from Hydrant,
2406 the aggregation count is a lower bound & may still climb.
2407- One endpoint reports how far along the backfill it is:
2408
2409```sh
2410curl "$BOBBIN/xrpc/sh.tangled.bobbin.getCoverage"
2411```
2412
2413While warming up:
2414
2415```json
2416{ "ready": false, "eventsProcessed": 45588, "lastCursor": 51658 }
2417```
2418
2419Once caught up, Bobbin flips to ready:
2420
2421```json
2422{ "ready": true, "eventsProcessed": 106085, "lastCursor": 116527 }
2423```
2424
2425If starting up Hydrant for the first time, Hydrant itself
2426will take a decent while (a couple of hours) to backfill
2427from PDSes. Hydrant stores its backfill on disk. Bobbin
2428restart reaches `ready` in minutes by replaying event from
2429an already-populated Hydrant. If your Hydrant is new, expect
2430Bobbin to backfill in that same couple of hours that Hydrant
2431takes.
2432
2433## Loose ends and not-gonna-impl
2434
2435- **No coverage signal for per-knot rosters yet.**
2436 Coverage tracks the hydrant stream only. A v1.15 knot
2437 that is unreachable serves a stale or empty member set
2438 with nothing to flag it.
2439- **Knot eventstream fan-out isn't pooled.**
2440 Bobbin opens one websocket per v1.15
2441 knot on top of the hydrant subscription. A network with
2442 thousands of knots wants pooling or a shared subscription.
2443- **No sequential issue or PR numbers.** bobbin returns rkeys,
2444 not `#42` style ids like the web appview. A client
2445 deriving a display number does it from creation order. But
2446 why bother? rkeys are the IDs.
2447
2448# Hacking on Tangled
2449
2450We highly recommend [installing
2451Nix](https://nixos.org/download/) (the package manager)
2452before working on the codebase. The Nix flake provides a lot
2453of helpers to get started and most importantly, builds and
2454dev shells are entirely deterministic.
2455
2456To set up your dev environment:
2457
2458```bash
2459nix develop
2460```
2461
2462Non-Nix users can look at the `devShell` attribute in the
2463`flake.nix` file to determine necessary dependencies.
2464
2465## Running the appview
2466
2467The appview requires Redis and OAuth JWKs. Start these
2468first, before launching the appview itself.
2469
2470```bash
2471# OAuth JWKs should already be set up by the Nix devshell:
2472echo $TANGLED_OAUTH_CLIENT_SECRET
2473z42ty4RT1ovnTopY8B8ekz9NuziF2CuMkZ7rbRFpAR9jBqMc
2474
2475echo $TANGLED_OAUTH_CLIENT_KID
24761761667908
2477
2478# if not, you can set it up yourself:
2479goat key generate -t P-256
2480Key Type: P-256 / secp256r1 / ES256 private key
2481Secret Key (Multibase Syntax): save this securely (eg, add to password manager)
2482 z42tuPDKRfM2mz2Kv953ARen2jmrPA8S9LX9tRq4RVcUMwwL
2483Public Key (DID Key Syntax): share or publish this (eg, in DID document)
2484 did:key:zDnaeUBxtG6Xuv3ATJE4GaWeyXM3jyamJsZw3bSPpxx4bNXDR
2485
2486# the secret key from above
2487export TANGLED_OAUTH_CLIENT_SECRET="z42tuP..."
2488
2489# Run Redis in a new shell to store OAuth sessions
2490redis-server
2491```
2492
2493The Nix flake exposes a few `app` attributes (run `nix
2494flake show` to see a full list of what the flake provides),
2495one of the apps runs the appview with the `air`
2496live-reloader:
2497
2498```bash
2499TANGLED_DEV=true nix run .#watch-appview
2500
2501# TANGLED_DB_PATH might be of interest to point to
2502# different sqlite DBs
2503
2504# in a separate shell, you can live-reload tailwind
2505nix run .#watch-tailwind
2506```
2507
2508## Running knots and spindles
2509
2510An end-to-end knot setup requires setting up a machine with
2511`sshd`, `AuthorizedKeysCommand`, and a Git user, which is
2512quite cumbersome. So the Nix flake provides a
2513`nixosConfiguration` to do so.
2514
2515<details>
2516 <summary><strong>macOS users will have to set up a Nix Builder first</strong></summary>
2517
2518In order to build Tangled's dev VM on macOS, you will
2519first need to set up a Linux Nix builder. The recommended
2520way to do so is to run a [`darwin.linux-builder`
2521VM](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder)
2522and to register it in `nix.conf` as a builder for Linux
2523with the same architecture as your Mac (`linux-aarch64` if
2524you are using Apple Silicon).
2525
2526If you're on nix-darwin, you can simply add
2527
2528```
2529nix.linux-builder.enable = true;
2530```
2531
2532to your host's `configuration.nix`.
2533
2534Alternatively, you can use any other method to set up a
2535Linux machine with Nix installed that you can `sudo ssh`
2536into (in other words, root user on your Mac has to be able
2537to ssh into the Linux machine without entering a password)
2538and that has the same architecture as your Mac. See
2539[remote builder
2540instructions](https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements)
2541for how to register such a builder in `nix.conf`.
2542
2543> WARNING: If you'd like to use
2544> [`nixos-lima`](https://github.com/nixos-lima/nixos-lima) or
2545> [Orbstack](https://orbstack.dev/), note that setting them up so that `sudo
2546ssh` works can be tricky. It seems to be [possible with
2547> Orbstack](https://github.com/orgs/orbstack/discussions/1669).
2548
2549</details>
2550
2551To begin, grab your DID from http://localhost:3000/settings.
2552Then, set `TANGLED_VM_KNOT_OWNER` and
2553`TANGLED_VM_SPINDLE_OWNER` to your DID. You can now start a
2554lightweight NixOS VM like so:
2555
2556```bash
2557nix run --impure .#vm
2558
2559# type `poweroff` at the shell to exit the VM
2560```
2561
2562This starts a knot on port 6444, a spindle on port 6555
2563with `ssh` exposed on port 2222.
2564
2565Once the services are running, head to
2566http://localhost:3000/settings/knots and hit "Verify". It should
2567verify the ownership of the services instantly if everything
2568went smoothly.
2569
2570You can push repositories to this VM with this ssh config
2571block on your main machine:
2572
2573```bash
2574Host nixos-shell
2575 Hostname localhost
2576 Port 2222
2577 User git
2578 IdentityFile ~/.ssh/my_tangled_key
2579```
2580
2581Set up a remote called `local-dev` on a git repo:
2582
2583```bash
2584git remote add local-dev git@nixos-shell:user/repo
2585git push local-dev main
2586```
2587
2588The above VM should already be running a spindle on
2589`localhost:6555`. Head to http://localhost:3000/settings/spindles and
2590hit "Verify". You can then configure each repository to use
2591this spindle and run CI jobs.
2592
2593Of interest when debugging spindles:
2594
2595```
2596# Service logs from journald:
2597journalctl -xeu spindle
2598
2599# CI job logs from disk:
2600ls /var/log/spindle
2601
2602# Debugging spindle database:
2603sqlite3 /var/lib/spindle/spindle.db
2604
2605# litecli has a nicer REPL interface:
2606litecli /var/lib/spindle/spindle.db
2607```
2608
2609If for any reason you wish to disable either one of the
2610services in the VM, modify [nix/vm.nix](/nix/vm.nix) and set
2611`services.tangled.spindle.enable` (or
2612`services.tangled.knot.enable`) to `false`.
2613
2614# Contribution guide
2615
2616## Commit guidelines
2617
2618We follow a commit style similar to the Go project. Please keep commits:
2619
2620- **atomic**: each commit should represent one logical change
2621- **descriptive**: the commit message should clearly describe what the
2622 change does and why it's needed
2623
2624### Message format
2625
2626```
2627<service/top-level directory>/<affected package/directory>: <short summary of change>
2628
2629Optional longer description can go here, if necessary. Explain what the
2630change does and why, especially if not obvious. Reference relevant
2631issues or PRs when applicable. These can be links for now since we don't
2632auto-link issues/PRs yet.
2633```
2634
2635Here are some examples:
2636
2637```
2638appview/state: fix token expiry check in middleware
2639
2640The previous check did not account for clock drift, leading to premature
2641token invalidation.
2642```
2643
2644```
2645knotserver/git/service: improve error checking in upload-pack
2646```
2647
2648### General notes
2649
2650- PRs get merged "as-is" (fast-forward)—like applying a patch-series
2651 using `git am`. At present, there is no squashing—so please author
2652 your commits as they would appear on `master`, following the above
2653 guidelines.
2654- If there is a lot of nesting, for example "appview:
2655 pages/templates/repo/fragments: ...", these can be truncated down to
2656 just "appview: repo/fragments: ...". If the change affects a lot of
2657 subdirectories, you may abbreviate to just the top-level names, e.g.
2658 "appview: ..." or "knotserver: ...".
2659- Keep commits lowercased with no trailing period.
2660- Use the imperative mood in the summary line (e.g., "fix bug" not
2661 "fixed bug" or "fixes bug").
2662- Try to keep the summary line under 72 characters, but we aren't too
2663 fussed about this.
2664- Follow the same formatting for PR titles if filled manually.
2665- Don't include unrelated changes in the same commit.
2666- Avoid noisy commit messages like "wip" or "final fix"—rewrite history
2667 before submitting if necessary.
2668
2669## Code formatting
2670
2671We use a variety of tools to format our code, and multiplex them with
2672[`treefmt`](https://treefmt.com). All you need to do to format your changes
2673is run `nix run .#fmt` (or just `treefmt` if you're in the devshell).
2674
2675## Proposals for bigger changes
2676
2677Small fixes like typos, minor bugs, or trivial refactors can be
2678submitted directly as PRs.
2679
2680For larger changes—especially those introducing new features, significant
2681refactoring, or altering system behavior—please open a proposal first. This
2682helps us evaluate the scope, design, and potential impact before implementation.
2683
2684Create a new issue titled:
2685
2686```
2687proposal: <affected scope>: <summary of change>
2688```
2689
2690In the description, explain:
2691
2692- What the change is
2693- Why it's needed
2694- How you plan to implement it (roughly)
2695- Any open questions or tradeoffs
2696
2697We'll use the issue thread to discuss and refine the idea before moving
2698forward.
2699
2700## Developer Certificate of Origin (DCO)
2701
2702We require all contributors to certify that they have the right to
2703submit the code they're contributing. To do this, we follow the
2704[Developer Certificate of Origin
2705(DCO)](https://developercertificate.org/).
2706
2707By signing your commits, you're stating that the contribution is your
2708own work, or that you have the right to submit it under the project's
2709license. This helps us keep things clean and legally sound.
2710
2711To sign your commit, just add the `-s` flag when committing:
2712
2713```sh
2714git commit -s -m "your commit message"
2715```
2716
2717This appends a line like:
2718
2719```
2720Signed-off-by: Your Name <your.email@example.com>
2721```
2722
2723We won't merge commits if they aren't signed off. If you forget, you can
2724amend the last commit like this:
2725
2726```sh
2727git commit --amend -s
2728```
2729
2730If you're submitting a PR with multiple commits, make sure each one is
2731signed.
2732
2733For [jj](https://jj-vcs.github.io/jj/latest/) users, you can run the following command
2734to make it sign off commits in the tangled repo:
2735
2736```shell
2737# Safety check, should say "No matching config key..."
2738jj config list templates.commit_trailers
2739# The command below may need to be adjusted if the command above returned something.
2740jj config set --repo templates.commit_trailers "format_signed_off_by_trailer(self)"
2741```
2742
2743Refer to the [jujutsu
2744documentation](https://jj-vcs.github.io/jj/latest/config/#commit-trailers)
2745for more information.
2746
2747# Troubleshooting guide
2748
2749## Login issues
2750
2751Owing to the distributed nature of OAuth on AT Protocol, you
2752may run into issues with logging in. If you run a
2753self-hosted PDS:
2754
2755- You may need to ensure that your PDS is timesynced using
2756 NTP:
2757 - Enable the `ntpd` service
2758 - Run `ntpd -qg` to synchronize your clock
2759- You may need to increase the default request timeout:
2760 `NODE_OPTIONS="--network-family-autoselection-attempt-timeout=500"`
2761
2762## Empty punchcard
2763
2764For Tangled to register commits that you make across the
2765network, you need to setup one of following:
2766
2767- The committer email should be a verified email associated
2768 to your account. You can add and verify emails on the
2769 settings page.
2770- Or, the committer email should be set to your account's
2771 DID: `git config user.email "did:plc:foobar"`. You can find
2772 your account's DID on the settings page
2773
2774## Commit is not marked as verified
2775
2776Presently, Tangled only supports SSH commit signatures.
2777
2778To sign commits using an SSH key with git:
2779
2780```
2781git config --global gpg.format ssh
2782git config --global user.signingkey ~/.ssh/tangled-key
2783```
2784
2785To sign commits using an SSH key with jj, add this to your
2786config:
2787
2788```
2789[signing]
2790behavior = "own"
2791backend = "ssh"
2792key = "~/.ssh/tangled-key"
2793```
2794
2795## Self-hosted knot issues
2796
2797If you need help troubleshooting a self-hosted knot, check
2798out the [knot troubleshooting
2799guide](/knot-self-hosting-guide.html#troubleshooting).