> For the complete documentation index, see [llms.txt](https://docs.onym.foundation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onym.foundation/operating/deployment.md). # Deployment [`onym-infra`](https://github.com/onymchat/onym-infra) brings the reference Courier, Stellar Notary, iOS Moderation, and Backup services up on one DigitalOcean droplet via Docker Compose. This page covers the services in the table below. The signed Discovery publisher is deployed by its own workflow, as described later, and the live Android Moderation backend's deployment is not documented in this repository. | Service | Host | Seat | | ---------- | --------------------- | ----------------------------------------------------------- | | Caddy | — | reverse proxy, automatic TLS | | strfry | `nostr.onym.app` | [courier — message](/seats/courier/courier-nostr.md) | | blossom | `blossom.onym.app` | [courier — blob](/seats/courier/courier-nostr.md) | | relayer | `relayer.onym.app` | [notary — Stellar](/seats/notary/notary-stellar.md) | | moderation | `moderation.onym.app` | [moderation](/seats/moderation.md) — enforcement | | authority | `authority.onym.app` | [moderation](/seats/moderation.md) — judgment | | backup | `backup.onym.app` | [backup — Object-HTTP](/seats/backup/backup-object-http.md) | One box is a cost decision and nothing depends on it. The authority delivers verdicts to the interface's **public** hostname rather than over the private network, so the day the interface moves to another operator that address points elsewhere and nothing else changes — and this deployment exercises the same TLS + token + signature path everyone else must use. The backup operator is the one service here that stores bytes it cannot read, and the one whose disk is a hazard to everything else. Its sealed snapshots sit on a **separate block volume**, not the droplet's root filesystem: they are the only thing on this box measured in gigabytes, and a full root disk would stop the authority recording a verdict and the relay accepting an event. The deploy refuses to *continue* if that volume is not mounted and prepared — it creates the droplet first, which is what makes the two-run sequence below work — and the container refuses to start without a sentinel file inside it — so a reboot where the mount does not return fails loudly instead of quietly writing snapshots to the root disk. The one thing that genuinely must stay private is the triage model container, if enabled: case evidence was disclosed for adjudication, and sending it to a third party is a further disclosure. ## First run ```sh git clone --recurse-submodules onym-infra && cd onym-infra cp .env.example .env # DO_API_KEY, CF_API_TOKEN, hosts, size cp relayer.env.example relayer.env # RELAYER_SECRET_KEY (required) cp moderation.env.example moderation.env # DeviceCheck key + ids, interface seed cp authority.env.example authority.env # signing seed + admin token (required) cp backup.env.example backup.env # BACKUP_SIGNING_SEED (required) ./deploy/digitalocean/deploy.sh # creates the droplet, then stops # at the backup volume gate gh workflow run "Provision backup volume" --repo onymchat/onym-infra ./deploy/digitalocean/deploy.sh # this one goes all the way ``` **Two runs on a genuinely fresh environment, and the order is forced rather than clumsy.** `deploy.sh` refuses to continue until `/mnt/onym-backup` is a mounted, prepared volume; the provisioning workflow attaches a volume, which needs a droplet to attach it to. So the first deploy exists to create the droplet — it stops at the gate — the workflow then provisions against that box, and the second deploy completes. On a box that already has its volume, one run is enough and the gate is a no-op. **A running operator is not yet a reachable one.** Backup has no release asset, so a client finds it through the signed catalog or not at all — see [Discovery provider](#discovery-provider) below. Until the operator's manifest is reviewed, pinned into `onym-services` and published, the seat exists and no client can see it. The entry pins the sha256 of the bytes the operator serves, so **any** later change to what the manifest carries — limits, offers, issuers, public URL — invalidates it and needs re-reviewing and re-publishing. Deploys and publishes are not independent operations. Provisioning is idempotent and safe to re-run. It creates and attaches the volume, mounts it, writes the sentinel files the container checks, and chowns them to the unprivileged uid the operator runs as. It never runs `mkfs` on a volume that already exists — the only formatting is at creation, when the volume is definitionally empty — because a provisioning script that can reformat a populated volume will eventually delete the only copy of someone's backup on a re-run that looked routine. `BACKUP_SIGNING_SEED` is not like the other seeds. Clients pin the public key derived from it, so regenerating it makes the operator a different operator to everyone already enrolled, with no repair path. Generate it once and keep it somewhere you would keep a private key. The script creates or adopts an `s-2vcpu-4gb` droplet by name, adds a 2 GB swapfile, upserts **DNS-only** Cloudflare A records, syncs and brings the stack up. Re-runs update the box — but **only the stack, never the droplet's size**. An existing droplet is adopted by name and never resized, so changing `DO_DROPLET_SIZE` moves what a *new* box would be created as and nothing else. Growing a running one is a `doctl` resize with a power-off; the cloud-init swapfile is not rewritten by it, because cloud-init runs at creation only. Two traps: * The swapfile is written by cloud-init, which runs only at droplet **creation**. Five Rust builds share the box — relayer, both moderation interfaces, the authority, and now backup — and the 2 GB swapfile is what makes them fit; adding swap later is a manual `ssh` job. (This trap said "three" until backup arrived; it had already missed the Android interface.) * The Cloudflare records must stay grey-cloud. Proxying breaks Caddy's ACME challenge and the Nostr `wss://` connection. ## Bootstrapping the moderation pair Only the interface countersigning key settles after a boot, so the first deploy is two passes with no unsigned enforcement window: 1. Generate both seeds, leave `AUTHORITY_INTERFACE_KEY` empty, keep `MODERATION_ENFORCE_SIGNATURES=true`, deploy. The script derives the authority operator key before startup and materializes a matching manifest, so every accepted verdict is signed from the first request. 2. Read the interface's public key and re-run: ```sh curl -fsS https://moderation.onym.app/health \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["interfaceKey"])' ``` Until pass two the authority refuses newly registered mandates. That is a closed bootstrap state, not permission to accept unsigned verdicts. ## Manifest and policy documents The authority's manifest is materialized at deploy time from the template in the `moderation` submodule, with this deployment's operator key and `AUTHORITY_HOST`. Changing any of those bytes after consent creates a new manifest hash — existing mandates stay bound to the old terms and fresh consent is required. Caddy serves the ten policy documents from `moderation/authority/published/` as `text/markdown; charset=utf-8`, one per term and per violation class, with no `#fragment` links. They are served verbatim rather than rendered: these are the bytes that were signed off. `manifest.json.sig` is the detached Ed25519 signature over the exact published bytes — **base64 of 64 raw bytes plus a trailing LF, 89 bytes total**. Consumers must trim whitespace before decoding. `deploy.sh` signs inside the authority image, retires the old signature, moves the new manifest into place, then writes the new signature last, so the published pair can never cover different bytes. A signing failure leaves the previous pair intact; a later interruption degrades to 404 → soft-verify. Order matters against the client: this asset must be live and verified before `ModerationTrust.enforceManifestSignatures` is flipped on in `onym-ios`, since under enforcement a 404 rejects the manifest and blocks consent outright. ## Rotating an authority's countersigning key The interface derives a separate key per authority from `MODERATION_INTERFACE_SIGNING_SEED` plus a per-authority epoch in `MODERATION_INTERFACE_KEY_EPOCHS`. Empty means epoch 0 — the seed used directly, the un-rotated state. This is about rotation, not containment: all derived keys live in the same process as the root. What it buys is burning one relationship without invalidating every countersignature ever issued to everyone. `AUTHORITY_INTERFACE_KEY` takes a comma-separated list, which is what makes the cutover gapless: 1. Derive the next key without deploying it (bump the epoch in a scratch environment, read `rotatedInterfaceKeys` from `/health`). 2. **Add** it beside the current one, deploy. Both verify; nothing changes. 3. Set the authority's entry in `MODERATION_INTERFACE_KEY_EPOCHS`, deploy. Earlier countersignatures still verify. 4. Remove the old key, deploy. Step 4 is the only irreversible one, and it is the point. Any other ordering — or a single-value swap — refuses every registration for that authority until both sides agree. A wrong component id parses fine, silently leaves that authority on epoch 0, and surfaces as a *signature* error rather than a configuration one. The interface names every configured id in its boot log beside the key it produced; compare that against what the mandates carry. ## CI `.github/workflows/deploy.yml` runs the same script from a manual `workflow_dispatch`, writing all five env files from Secrets and Variables. The two-run sequence above applies here too: on an environment with no volume yet, the first dispatch stops at the gate, `Provision backup volume` runs against the droplet it created, and a second dispatch completes. This is also the relayer's deployment path — `onym-relayer` releases now publish manifests only. `AUTHORITY_INTERFACE_KEY` is a Variable, not a Secret: it is a public key, and it does not exist until the interface has booted once. The relayer's signed [notary operator manifest](/seats/notary/notary-stellar.md) ships inside its image: `onym-relayer`'s `sign-manifest.yml` workflow signs in CI and commits the exact bytes under `manifest-signed/`, the Dockerfile copies them to `/srv/operator-manifest/`, and this repo's compose file points `RELAYER_OPERATOR_MANIFEST` at that copy. After a re-sign, bump the `relayer` submodule here and deploy; the signing workflow's byte-verification step then confirms `https://relayer.onym.app/manifest.json` serves exactly the committed bytes. ## Discovery provider `discovery.onym.app` is live, but nothing in this repository serves it directly — the signed [discovery](/seats/discovery/discovery-static-ed25519.md) provider is published by `onym-discovery`'s own manual deploy workflow ([#4](https://github.com/onymchat/onym-discovery/pull/4), merged; the genesis publish has run; the live catalog is at sequence 4 and lists five seats). The `workflow_dispatch` `deploy.yml` builds the reference CLI, then signs and chains the snapshot onto the previously **published** one. A genesis publish is an explicit input, not a guess. Before a byte leaves the runner, the workflow verifies everything exactly as a client would. It then rsyncs the static tree to `/var/www/discovery` on the **same droplet**, idempotently installs a Caddy vhost, and only then upserts the grey-cloud DNS record. A mid-run failure therefore never leaves a public name pointing at a half-configured host. Signing seeds (`DISCOVERY_OPERATOR_SEED` and the courier/Blossom seat seeds) live as Actions secrets, with a `skip_signing` path for operators who sign locally instead. The job runs in a `production` environment gated by required reviewers. Two operational couplings with this repository's deploy: * **A deploy from this repository sweeps the discovery vhost away.** `deploy.sh` rsyncs with `--delete`, which removes the Caddy compose override the discovery deploy installed. After an `onym-infra` deploy, re-dispatch the `onym-discovery` deploy workflow — it is self-healing on re-run and puts the vhost back. * **The vhost step restarts the shared proxy.** Installing or restoring the override recreates the Caddy container that fronts every `onym.app` vhost, so open Nostr `wss://` connections drop and clients must reconnect. The workflow announces the recreation in its log before issuing it. ## Operating ```sh ssh root@ && cd /opt/onym-infra docker compose ps docker compose logs -f caddy # cert issuance / renewal docker compose logs -f authority ``` Both moderation services answer `GET /health`; `deploy.sh` checks for 200 at the end of a run. So does backup, and deploy additionally checks its `/manifest.json` and `/manifest.json.sig`, because clients pin those bytes and a manifest that stops resolving takes the seat out of reach. Backup is the one service here with a startup precondition, so it is the one whose failure looks like a loop rather than an error: if the block volume did not come back after a reboot, `docker compose ps` shows the container restarting, and `docker compose logs backup` prints its refusal to write sealed snapshots to the root filesystem. That is the sentinel check working. Re-mount `/mnt/onym-backup` — the fstab entry uses `nofail`, so the box boots without it — and the container settles on its next restart. The human moderation queue is at `https://$AUTHORITY_HOST/admin`, behind `AUTHORITY_ADMIN_TOKEN`. The authority refuses to start without this human route, and deploy catches an empty token before touching the droplet. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.onym.foundation/operating/deployment.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.