The whole system in one picture

What is decided, tested, proposed, open

Counted from the tags in the five chapter files. Also 28 marked MISSING (not written anywhere) and 31 NOT TESTED or design only.

Read in this order

1. 2DeploymentRepos, Kargo, ArgoCD, order
2. 3Cluster failoverAddress lists, the 5 s loop, swaps
3. 4DNS failoverThe chain, and what survives
4. 5Client-side routerRetry, rescue router, address books
5. 6Frontend failoverWorkers, the Vercel standby
6. 7Sign-in (Logto)Deploy, tokens, many domains
7. 8GlobalRegions, home landscape, data
8. 9Mercury (webhooks)One intake, the 421 rule, 72 h
9. 10Must settle3 gaps + 48 decisions
10. 11Whole-system testReal clusters, drill by drill

How to read this site

• Tags: every statement carries a small chip. Tap it to see its source. DECIDED · example: owner D3
  • DECIDED = an owner ruling or the design papers. TESTED = a test ran it. PROPOSED = a suggestion; whose is named. OPEN = you decide.
  • MISSING = not written anywhere yet. NOT TESTED = designed, never run.
• Words: a dotted underline means a tool or a word we made up. Tap it for a one-line meaning.
• Steppers: every flow is a picture you step through: Next, Back, or Play.
• Colours: one colour per actor, the same everywhere. The name is always written too.

Actors

• Person
• CI
• Kargo
• ArgoCD
• T3
• DNS
• Client
• Vendor

Tags

• DECIDED
• TESTED
• PROPOSED
• OPEN
• MISSING
• NOT TESTED

The running example

• Platform nitroso, service tin.
• Landscapes pichu (dev) → pikachu (staging) → raichu (prod, Singapore) + ampharos (prod, US).
• Clusters raichu-opal and raichu-ruby. Virtual landscape mew = raichu + ampharos.

How the chapters connect

• Deployment puts things on clusters. Cluster failover sends traffic to them.
• Frontend and sign-in are where a dead DNS name hurts most: a browser's first request, and signing in.
• Global is cluster failover plus sign-in across regions, plus mercury.
• The DNS proposal and the Logto test (pass 2) feed the DNS and sign-in screens.

Who edits what

• fleet is one public repo with three folders. DECIDED · owner D2; T3:450
  • registry/: landscapes and clusters. People, by reviewed PR.
  • machinery/: the deploy rules and the compiler chart. Nobody, day to day.
  • platforms/: row files. Machines only: T3 writes empty rows, Kargo writes versions.
• nitroso.carbon (one per platform): platform.yaml and shared charts. The nitroso team. DECIDED · carbon.md:168
• nitroso.tin (one per service): code, app chart, primordial chart. The tin team. DECIDED · T3:1003
• Creating the repo is the registration. No fleet edit. DECIDED · fleet.md:125; T3:1003
• Carbon and service repos hold no fleet knowledge and no fleet credentials. DECIDED · T3:1063
• GitHub guard: main needs a reviewed PR; Kargo's bot may bypass; CODEOWNERS on registry/**. DECIDED · fleet.md:609
• Open: machine files move to their own repo, fleet-state, so people and machines never push to each other's repo. OPEN · picks #4, default yes, untested

The two charts every service ships: DECIDED · owner D3 #5

• App chart = its pods. Goes to every cluster of the landscape.
• Primordial chart = what it needs from outside, such as a database. Goes once to Primordial.

One-time setup

• Records park and wait on a status until T3 is up; nobody orders them by hand. DECIDED · T3:15-16
• T3's controllers turn on in this order: cluster → traffic → platform → dependency → webhook → cf-deploy. DECIDED · T3:3516

Register a landscape and a cluster

• Where a cluster's cloud account lives is open. Default: fleet/registry/accounts/<vendor>-<name>.yaml. OPEN · O6, T3:601

Add a platform

• Names shorter than 5 letters are refused. DECIDED · T3:988

Add a service

• Row layout is open. Tested: platforms/nitroso/landscapes/pichu/tin.yaml. jaeda's pick: platforms/nitroso/tin/pichu.yaml, one folder per service. OPEN · picks #3, default b
• The service list file services.yaml is dropped by default; the list comes from the row names. The old design was refused by ArgoCD. OPEN · O1, default dropped TESTED · spike A:61-73, 157-169

Release a version

• Each stage step can be a bare name (automatic, health-gated) or {landscape, gate: manual, soak: 1h, verification}. Only the bare form ran live. DECIDED · fleet.md:410-430 TESTED · spike A:91
• Frontends ride the same pointer; their primordial chart carries a CloudflareDeploy instead (see Frontend). DECIDED · ARCH:737-746
• No git webhooks were wired in the test. Production should use them; otherwise budget 30 s per git hop and up to 5 min for Freight. PROPOSED · spike A:189-190

The order of the two charts

Nothing makes the app wait for its database today. One Kargo pin moves the primordial and app deploys in the same promotion. TESTED · spike A:112

What exists instead:

• Every app chart has a db-init job that runs before the pods update: it checks the database, key-value store and file store, runs schema migrations, seeds data. DECIDED · rules:665-670
• Pod health checks never depend on the database. DECIDED · rules:689
• During a database move, schema migrations are refused. DECIDED · T3:2075

Settings, three layers (last wins)

1. Chart defaults in the service repo (values.yaml). DECIDED · goals/dotnet-api.md:171
2. Per-landscape file in the chart, e.g. values.raichu.yaml: type: planetscale in raichu, type: neon in pichu. DECIDED · dotnet-api:171 OPEN · picks #7, default a
3. Emergency override in the row file's values: block. Short-lived; a check flags any override older than 7 days; Kargo never touches it. Measured: live in 16 s. DECIDED · fleet.md:364-367 TESTED · spike A:114-116
4. Not tested: how the AppSet picks values.raichu.yaml out of the published chart. MISSING

Rollback

• Service rollback: promote an older Freight in Kargo. Written for Workers; for normal services it is implied, not written. DECIDED · ARCH:744 MISSING · for normal services
• Database schema after rollback: not written; a forward migration stays applied. MISSING
  • Proposed: migrations stay backward-compatible for one release ("expand, then contract"). PROPOSED · this chapter, decision 12
• Deploy rules (compiler chart): the machinery-stable tag never moves back; revert in a new commit and move the tag forward. DECIDED · fleet.md:376-393
• Break-glass: a person may commit straight to a row file, skipping Kargo, until the next promotion. DECIDED · fleet.md:964; T3:1076

Remove

• A cluster: traffic off → drain → Decommission. Removing its registry file deletes its ArgoCD entries but keeps its pods. DECIDED · T3:824 TESTED · spike A:136-137
• A landscape: re-home its users first; platforms drop it from platform.yaml; refused while anything uses it. What enforces that is not written. DECIDED · ARCH:914; fleet.md:162 MISSING

Secrets at deploy time

• T3 writes app secrets to Infisical at project = nitroso, env = raichu, folder /database/maindb. DECIDED · T3:227
• Each cluster reads them through a chain that narrows at every step: Primordial store → platform store → service. DECIDED · T3:254
• Infisical down: running apps keep working; a brand-new service or cluster cannot get secrets until it is back. TESTED · spike B:99

Everything else in this chapter

The address list per landscape

• Every service name is <module>.<service>.<platform>.<landscape>.cluster.atomi.cloud. DECIDED · ARCH:57-59
  • Example: api.tin.nitroso.raichu.cluster.atomi.cloud.
  • Warning: Another paper writes api.zinc.nitroso.serving.cluster.atomi.cloud instead. The two naming rules clash: gap G1 on Must settle. OPEN · GAPS G1
• Route 53 holds the main answers. Cloudflare holds a copy and the parent zone atomi.cloud. DECIDED · ARCH:227
• The controller writes all three: Route 53, the S8 spare, the Cloudflare copy. DECIDED · ARCH:304-306
• Cache time (TTL): real Cloudflare refused 30 and 59; 60 is the lowest. Route 53 allows 30 s, but that is not tested. TESTED · spike D:103 PROPOSED · jaeda, how:124

Try it: kill a cluster

One tick of the loop

• A person's change (traffic flip, new cluster) skips the wait and lands within one tick. DECIDED · T3:2943
• Memory in four layers: process memory → a local Redis/memcached → a snapshot in S3 → live DNS as the truth. On start it reads DNS before it writes. DECIDED · owner D7; T3:2894-2907
• The loop watches itself: a "last good tick" metric pages when stale. DECIDED · T3:3060
• Primordial down: DNS freezes at its last state and keeps serving. DECIDED · ARCH:308-309
• Two controllers at once: the lease let one write; the other wrote 0 times. TESTED · spike D:167-176

The traffic switch

• traffic: true / false is a field in the cluster's registry file, fleet/registry/clusters/raichu-ruby.yaml. DECIDED · ARCH:310
• A person changes it by PR; the traffic controller acts within one tick. DECIDED · MASTER:287
• false = leave the address list. The cluster keeps running, so old connections finish. DECIDED · ARCH:311-312
• true = join the list. This is the switch for a canary and a cutover. DECIDED · ARCH:312-313
• Destroying a cluster is a separate act (a Decommission record), never implied by false. DECIDED · ARCH:315-316

Swap a cluster (blue/green)

• Each address change was one all-or-nothing call; real Cloudflare served it in about 3.4 s. TESTED · spike D:87
• Users see nothing: at every moment at least one address is alive. PROPOSED · jaeda, how:176
• The whole swap has never been run end to end. NOT TESTED · design only, MASTER:503

Machine, cluster or region?

• An older paper says "~5–10 s". That cannot happen with 3 failed probes 5 s apart; jaeda proposes saying "13 to 19 s". PROPOSED · jaeda, G11
• No global load balancer: the 30–60 s DNS lag is accepted. DECIDED · owner D17

What our apps do on normal days

• The app holds one name: its home landscape's. DECIDED · ARCH:240-241
• On a network error it retries exactly once, over a fresh connection. Never on an HTTP error reply. DECIDED · lib-contracts:79
• Browsers try the next address of a name by themselves. DECIDED · ARCH:292 NOT TESTED · real browsers, how:240
• The rare case, where the name itself is dead, is the rescue router.

Outside callers

• Callers we do not control (another company's server, a script) have no rescue router. They rely on the short TTL and the 5 s loop: a stale window of up to about a minute. Fine if they retry. DECIDED · ARCH:42-47 PROPOSED · jaeda, how:238
• Nothing tells partners to retry. MISSING · GAPS G20

Everything else in this chapter

The chain

• Route 53 down: no answers for the main name. DECIDED · owner D36
• Cloudflare zone down: the pointer to Route 53 is lost once devices forget it, usually a day or two, so a short Cloudflare outage does not hurt. PROPOSED · jaeda, how.md
• The design's spare (S8) also lives on Route 53, so a Route 53 outage kills the main name and the spare together. PROPOSED · jaeda, how:195
• The controller also writes a copy into Cloudflare, but devices are likely never sent there, because the pointer sends them to Route 53. PROPOSED · reading, jaeda; OPEN
• The domain itself breaking, or a big mistake by us in the records, can also kill the name. Our own mistake is the likeliest; the loop's guards (never remove the last address, freeze) are tested. PROPOSED · jaeda, how.md TESTED · spike D:69-71

Try it: pick what fails

jaeda's position until the proposal lands

The proposal

registrar: Cloudflare Registrar        (TESTED RDAP)
   |
root (.)  --NS 172800 s-->  .cloud TLD (registry Identity Digital; servers trs-dns, Tucows)  (TESTED dig)
   |
.cloud    --NS 900 s----->  atomi.cloud on Cloudflare (tim/brianna)        (TESTED dig)
   |                        Cloudflare's own NS TTL: 86400 s
   |                        "does not exist" cached 1800 s (SOA minimum)
   |                        not signed (no DNSSEC)                          (TESTED RDAP)
   |
atomi.cloud --NS (TTL ours)--> cluster.atomi.cloud on Route 53             (DECIDED design, NOT BUILT)
                               Route 53 NS TTL usually 172800 s
                               |
                               +-- <l>.cluster.atomi.cloud    A  TTL 60  (LB IPs of healthy clusters)
                               +-- *.<l>.cluster.atomi.cloud  CNAME -> it
                               +-- api.lithium.aldehyde.<l>.cluster.atomi.cloud = the Logto issuer

spare (S8): <l>.cluster.<backup-base> on Route 53, same IPs            (DECIDED design, NOT BUILT)
            domain, TLD and registrar not chosen yet                    (OPEN)

Facts that shape any fix

• A name looked up before it exists is remembered as "no such name" for 1,800 s on our zone. TESTED · spike D:106
• DNS changes reach users within about TTL + 3 s (TTL 60 on Cloudflare: worst 57 s on 8.8.8.8). TESTED · spike D:93-102
• Cloudflare write → all its nameservers: p50 3.4 s, max 8.3 s. TESTED · spike D:87
• Route 53 writes, TTL 30 and geoproximity are not tested (no AWS token). NOT TESTED · how:127
• A third DNS host only helps if Route 53 and Cloudflare are both down. PROPOSED · jaeda, how.md

Two things share the word "failover"

• Source: jaeda's wording. PROPOSED · jaeda, how:184-188

The three edge files

• T3 writes them from existing records; nobody edits them. DECIDED · ARCH:256-283; T3:2863
• Each file is written twice: Cloudflare R2, and AWS S3 behind CloudFront. Also reachable on *.cloudfront.net, which needs none of our DNS. DECIDED · T3:2878; ARCH:261-264
• No signing; the baked allowlist replaces it. DECIDED · ARCH:282-283
• Each copy of Doc A lists the other places, so the set of places can grow without a new app release. DECIDED · ARCH, via how.md

Walk: a client when its address dies

What it survives

• Source: jaeda's reading of the design. PROPOSED · jaeda, how.md

What it does not survive: PROPOSED · jaeda, how.md

• Cloudflare and AWS down together.
• A dead home region: that is data to restore, not an address problem. DECIDED · ARCH:254
• An app unopened so long that every place on its old list is gone. Rule needed: keep at least one place alive forever.
• A wrong book published by us: limited to our own domains, and the sign-in address is never taken from a book.

Where it runs

• On: Flutter apps and browser code. Off: the nextjs server; its rescue is a redeploy. DECIDED · ARCH:244-246; nextjs:165-168
• Built by bun frontend-utils (browser) and dart-api-engine (Flutter). Both have test rows. DECIDED · bun-frontend-utils.md:280; dart-api-engine.md:187
• A scheduled test exercises the sleeping path so it does not rot. DECIDED · bfu:128
• Not written: where the browser stores the book, and whether spare names have valid certificates and CORS for the web page. MISSING

Four cases where it is not instant

1. The dead cluster goes silent instead of refusing: the client waits for its connect time limit first. No library sets one today. PROPOSED · jaeda, how.md MISSING · GAPS G10
2. Writes: a retry is only safe when the write carries a unique key. PROPOSED · jaeda, how.md
3. Only one cluster behind the name: no second address; the rescue router takes over, which is slower (seconds). PROPOSED · jaeda, how.md
4. Callers we do not control: no rescue router; they wait for DNS, 30 to 60 s, unless they retry. PROPOSED · jaeda, how.md

• "Gossip"? Not strictly: apps do not talk to each other. It is "address books that list where the address books are". PROPOSED · jaeda, how.md

Release of a web frontend

• One production path: OpenNext → Cloudflare Workers. Vercel is only a standby, never a second production path. DECIDED · nextjs:39, 43-44
• First ever deploy: the chart creates the Worker once with a "not deployed yet" placeholder, because Cloudflare's first upload goes live at 100 %. DECIDED · T3:3296-3299 TESTED · hole H1, spike C:90
• The landscape is chosen at runtime by a Worker setting, never guessed by the browser. DECIDED · nextjs:328-332
• The sign-in address (issuer) is built in at build time; a different landscape issuer needs a new build. DECIDED · nextjs:266-268, 332-335
  • That clashes with "one build promotes through all stages" (gap G8). OPEN · decision 27
• In-cluster frontend chart: only in the six dev/test/preview profiles, off by default. Never production. DECIDED · frontend-standalone-chart.md:57-61

The standby flip

What exists in the design: DECIDED · nextjs:180-184; ARCH:329-331

• CI deploys every release to both Cloudflare and Vercel.
• Vercel has the domain and certificates set up in advance.
• A weekly test request checks the Vercel copy works.
• Failover = repoint the CNAME (a DNS alias) to Vercel.

What is not written anywhere: MISSING

• Who or what flips the CNAME, and what triggers it.
• Who owns the weekly test: open owner decision OD-25, default a production-operations node under T3. OPEN · nextjs:413-415
• How the Vercel build gets its landscape settings and secrets.
• What replaces the Cloudflare-only parts on Vercel: R2 page cache, Durable Object queue, D1 tag cache, Cloudflare Images.

Proposed defaults: PROPOSED · this chapter, decisions 24-26

• A person flips it with a one-command runbook after the weekly test is green; no automatic flip in v1.
• Web name web.<service>.<platform>.<landscape>.cluster.atomi.cloud, written by the traffic controller.
• A second "standby" build for Vercel, page cache off.

What is lost when something dies

• Phone apps in a Cloudflare zone death: API calls move to the spare name, but sign-in cannot follow (the issuer is baked). PROPOSED · reading, this chapter

A browser's first request

• A browser's first request has no rescue router: the page that holds it has not loaded yet. If DNS cannot answer, nothing we ship can help. DECIDED · ARCH:333-334
• So for browsers only DNS fixes help: the CNAME flip, or the zone coming back.
• DNS changes reach users within about TTL + 3 s (worst 57 s on 8.8.8.8 with TTL 60). Measured on A records, not a CNAME flip. TESTED · spike D:93-102
• A name looked up before it exists is cached as "does not exist" for 30 minutes, so the Vercel name must exist before any flip. TESTED · spike D:106-107

Server side, browser side, phone

• nextjs server: rescue router off; it retries once on a network error; its rescue is a redeploy. What "redeploy" means in practice is not written. DECIDED · nextjs:165-168; bfu:83-87 MISSING
• Browser: rescue router on; reads the saved Doc C; checks the allowlist; offline page from a service worker. DECIDED · nextjs:165-166; frontend-ux-doctrine.md:119-120
• Flutter: each landscape is its own store app; issuer, sign-in page and key addresses are built in; the router keeps the last good book forever. DECIDED · flutter-base.md:362-369; dart-api-engine.md:57-71
• The spare base name (S8) gives phone apps a separate built-in root if atomi.cloud dies. DECIDED · ARCH:326-327

Error catalogs on the edge

• Problem catalog = each service's list of known errors, with a "can retry" flag. The UI uses it: retry, stay, or show a full error. DECIDED · frontend-ux-doctrine.md:112-118
• T3 publishes it as a static file next to the address books, 60 s cache; frontends never call Primordial for it. DECIDED · ARCH:284-287; portal:327-331
• Error portal (erbium) = a human-readable site of those errors, on Cloudflare only. If Cloudflare dies it is gone; accepted. DECIDED · ARCH:338-339
• Unit: per service (older) or per platform × tier (newer). OPEN · decision 28

Everything else in this chapter

How it is deployed

• Our code: a small fork of Logto (aldehyde.logto: one boot patch and a "sanitize preview copy" command) and logto-operator. DECIDED · alog:20-22, 85-88
• Database: one per landscape, an ordinary T3 Database (Neon in prod by default). Moving it never changes the issuer. DECIDED · T3:675-676, 2248-2249
• Upgrades: one landscape at a time, by Kargo. Test 1.45 → 1.46 with 3 copies: tokens, refresh, machine logins, keys 0 failures; interactive sign-ins 0–2 failed per roll. DECIDED · lop:699-702 TESTED · spike E:11
• Signing keys live in the database, so every copy has the same keys. TESTED · spike E:78
• Dev and preview: each instance has its own Logto and database. Preview copies of prod data must have every machine secret replaced first; that guard is not built yet. DECIDED · alog:136-137 OPEN · env:1486-1489
• No Logto version is pinned; default proposal v1.46.0. MISSING OPEN · decision 32

The issuer address

• Issuer = the exact address that signs tokens. Every token says "issued by <this>", and every service checks it.
• Fleet: api.lithium.aldehyde.<landscape>.cluster.atomi.cloud, made from the landscape, never typed in. DECIDED · alog:99-100; lib-contracts:150
  • Example: api.lithium.aldehyde.raichu.cluster.atomi.cloud.
• Permanent once live: changing it makes every token unverifiable. DECIDED · alog:102-105
• Built into apps; never from a file, setting or request. Apps carry a fixed landscape → issuer table. DECIDED · lib-contracts:160-169
• No mew-style name for Logto. DECIDED · alog:374-377

Tokens and lifetimes

• So 10 minutes holds only if logto-operator writes it on every API. That is its job. DECIDED · lop:735-736
• Services check tokens locally and never call Logto per request; they re-fetch keys on an unknown key id and keep the last good keys while Logto is down. DECIDED · env:484-485; T3:2361
• Reuse of an old refresh token being caught is assumed, but Logto does not document it. OPEN · study I:155
• Signing-key rotation policy: not written, not tested. MISSING NOT TESTED · spike E:93

When things fail

Can one Logto answer on two domains?

The sign-in standard (OIDC): one issuer = one exact address string. Several network addresses can reach it, but the string never changes. DECIDED · OpenID Discovery §4.3, Core §3.1.3.7, RFC 9207

The options for a dead name

• None is decided. OPEN · decision 31
• Proposed default: F if the DNS proposal adopts two hosts for the same zone; otherwise C, keeping A's behaviour. Avoid E. PROPOSED · this chapter
• jaeda's position (G5): accept A for v1 and put the issuer name on a base the DNS proposal makes survive one host outage. Do not run two issuers. PROPOSED · jaeda, G5

Test result: Logto on two names

• Recommendation: option (iii) = a spare address for Logto under the spare base name, the issuer string unchanged, libraries fall back to it. Tokens stay at 10 minutes. This is option C above. PROPOSED · levi, LOGTO-MULTIDOMAIN.md
• Today the design has no spare name for Logto, so if the issuer's name dies: signed-in users work up to 10 minutes, then wait; nobody new signs in. TESTED · LOGTO-MULTIDOMAIN.md

Machines (M2M)

• A machine gets its token from the target landscape's Logto and asks for the target service's audience. DECIDED · ARCH:376-377; env:1506-1507
  • Example: a nightly job calling tin in raichu signs in at raichu's Logto with audience "tin in raichu".
• Only the landscape picks the issuer, never the platform. DECIDED · lop:177-178

Everything else in this chapter

Landscapes and virtual landscapes

• mew = the production name over raichu + ampharos. celebi = the staging name; its hosts are only inferred ([pikachu]). DECIDED · fleet.md:231 OPEN · fleet.md:234
• The mew name answers with the addresses of every landscape row with serve: true. DECIDED · ARCH:224-225
• Region = an airport code from an allow-list. A landscape's region never changes. DECIDED · owner D1; fleet.md:213
• The codes for raichu and ampharos are not written. Proposed: SIN and IAD. MISSING PROPOSED · decision 47
• jaeda's position (G1): pichu, pikachu, raichu, ampharos are four landscapes, one sign-in issuer each. PROPOSED · jaeda, G1

How a new user gets a home

• Phones and browsers do the picking; server libraries just honour the claim. DECIDED · lib-contracts:199
• "Home per platform" (one paper) vs one home_landscape key in each landscape's Logto (another). Proposed: one home per user. OPEN · decision 41
• Re-homing a user is needed before a landscape is removed, but no mechanism exists. Proposed: a manual, audited runbook in v1. MISSING PROPOSED · decision 46

Where data lives

• A landscape is its data. DECIDED · ARCH:660
• Each platform's database, key-value store and files sit in the landscape's region by default. DECIDED · T3:1195
• Each landscape has its own Logto with its own database. DECIDED · ARCH:396-400
• Caches live inside each cluster and never cross regions. DECIDED · owner D16
• No live copying between regions in v1. A dead region = restore from backup, then point the record at the new place: hours. DECIDED · T3:2649; ARCH:293
• Backups land at a different vendor (hard rule). Recovery point: minutes; recovery time: hours. DECIDED · ARCH:664-666
• A standby database at a second vendor: default no (about 2× database cost). OPEN · O3

How a user reaches home

• The app holds one name, its home's, e.g. api.tin.nitroso.raichu.cluster.atomi.cloud. DECIDED · ARCH:240-241
• Sign-in for raichu is api.lithium.aldehyde.raichu.cluster.atomi.cloud, from a built-in table. DECIDED · lib-contracts:150, 169
• A raichu token is refused by ampharos. DECIDED · lib-contracts:158
  • Warning: One paper says the opposite: one "serving" landscape with one issuer and a raichu ↔ ampharos failover. This is gap G1. OPEN · lop:558-568
• If raichu's region dies, the rescue router does not help (it changes addresses, not landscapes). Ana waits for the restore, or is re-homed. DECIDED · ARCH:254

Machines across regions

• A machine gets its token from the target landscape's Logto. Example: a job calling tin in ampharos signs in at ampharos's Logto. DECIDED · env:1508; ARCH:376-378
• Audiences are per landscape, never reused across landscapes. DECIDED · env:1600

Moving across regions

• The owner wants cross-region moves. The Migration record is the default way. DECIDED · owner D8 OPEN · O2, default yes
• Editing a "move field" (provider, account, region, major version) does nothing by itself: T3 says MigrationRequired and keeps serving the old one. DECIDED · T3:1231
• Steps: build beside → copy → check → short write pause → switch → keep the old one 48 h → delete. DECIDED · T3:2000-2256

• Measured: Postgres moves between Neon, PlanetScale and DigitalOcean, all in Singapore, with the operator in the US (217 ms away): writes waited 3.2 s, 0 failed. TESTED · spike M:4, 22
• In one region it should be 0.1–0.3 s. PROPOSED · estimate, spike M
• Measured since, in one region (Singapore): a live move works up to 100,000 writes a second, 0 failed, 0 lost, with the library. TESTED · spike L

• The pause is set by how far the copy is behind when writes are held, not by the library. Up to about 50,000 writes a second the copy keeps up. TESTED · spike L
• What breaks first is the caller's own time limit: a 5 s limit at 100,000 writes a second → 1.54M failed. TESTED · spike L
• One Postgres took 300,000 writes a second when the library groups writes, about 20,000 one by one. TESTED · spike L
• The operator must run near the database, but Primordial's region is not written. Proposed: a small fence helper per region. DECIDED · T3:1676 MISSING PROPOSED · decision 48

Put off past v1

• Live copying between regions. DECIDED · T3:3641
• A global load balancer: ruled no. DNS movement stays; 30–60 s lag accepted. DECIDED · owner D17
• Cross-vendor backups are not put off: they are required. DECIDED · T3:3641

Everything else in this chapter

Where it runs

• Once for the whole fleet, as an ordinary service on its own mercury platform, in every serving landscape (raichu and ampharos). DECIDED · ARCH:431-437
• In-cluster, plain DNS, no Cloudflare in the path. DECIDED · ARCH:555-556
• Hot data (dedup, queue) in each landscape's Upstash; long-lived data in one Neon database for mew. DECIDED · mercury:176, 186
• Which region that Neon sits in is not named. Proposed: raichu. MISSING PROPOSED · decision 44

One webhook, step by step

• 421 ("Misdirected Request") means only "not mine". DECIDED · mercury:287
• A retry may land in the other landscape and be delivered twice; receivers must ignore repeats. DECIDED · mercury:257
• About 24 h of continuous failure to one receiver opens a breaker until a probe or a person re-enables it. DECIDED · ARCH:531-533
• A receiver in the same landscape that allows private calls is called in-cluster; others by public name. DECIDED · mercury:307

Who sets it up

• Each consumer platform has one internal tenant (WebhookEngine in its carbon chart, e.g. internal/nitroso). DECIDED · ARCH:466-484; T3:3171
• Each receiver service adds a WebhookRoute in its own chart (e.g. tin). DECIDED · ARCH:466-484
• T3's webhook controller turns those into calls to mercury's management API; only mercury writes its own Upstash config. DECIDED · T3:3171
• A tenant's home name is fixed for life; registered intake URLs are treated as permanent. DECIDED · mercury:141; T3:3201
• Previews: one fixed mercury endpoint per provider (the "callback rail"); the old per-preview tunnel is deleted. DECIDED · public-callback-tunnel.md:6, 155, 196

When things die

Still open

• Which issuer for mercury's management API, which sits behind the mew name while tokens are per landscape? Proposed: the caller's own landscape issuer; needs a security review. MISSING PROPOSED · decision 43
• Queued events in a dead region: proposed, the surviving region's mercury drains the dead region's Upstash (a vendor service, likely still up). PROPOSED · decision 45
• Older text puts WebhookEngine in mercury's own chart; newer text puts it in the consumer's carbon chart. OPEN · GAPS G9

Ticks and the reply line are also at the bottom of this screen.

The 3 gaps to settle first

Also proposed by jaeda

All 48 open decisions

Also open in the T3 goal but outside these chapters: O9–O14 (Garden rotation, Upstash alternative, Valkey operator licence, Aurora in v1, R2 as durable store, Aurora upgrades) – see T3:611-667. celebi's hosts default [pikachu] (fleet.md:234).

Every gap found while writing

Tools we use

ArgoCD

makes clusters match what git says.

ApplicationSet (AppSet)

an ArgoCD rule that stamps out one deploy per file, repo or cluster it finds.

Kargo

moves a version from stage to stage by editing one file per stage.

Helm chart

a package of Kubernetes files with settings.

Infisical

our secret store, on Primordial.

ESO (External Secrets)

copies secrets from Infisical into each cluster.

cert-manager

gets TLS certificates from Let's Encrypt.

Route 53

AWS's DNS host; holds our main cluster answers.

Cloudflare

holds the atomi.cloud zone, a copy of our DNS, Workers and R2.

Cloudflare Workers

Cloudflare runs small programs at its edge; our web frontends run there.

OpenNext

turns a nextjs app into a Worker.

Vercel

a nextjs host; our standby for web.

R2 / S3 / CloudFront

file hosting at Cloudflare / AWS / AWS's CDN.

Logto

open-source sign-in server (OIDC).

Neon, PlanetScale, DigitalOcean Postgres, Aurora

hosted Postgres vendors.

Upstash

hosted key-value store (Redis protocol), durable.

Dragonfly / Valkey

in-cluster caches.

Tigris

hosted file store.

Grafana Cloud

where metrics and alerts go.

k3s / k3d

small Kubernetes / k3s in docker (used in tests).

Gitea

a self-hosted git server (stood in for GitHub in tests).