The whole system, tested for real

Repos in the PoC: dpoc-fleet, dpoc-fleet-state, dpoc-nitroso.carbon, dpoc-nitroso.tin (bun API), dpoc-nitroso.web (frontend), dpoc-aldehyde.carbon, dpoc-aldehyde.lithium (Logto), and – created during the run – dpoc-argon.carbon, dpoc-argon.zinc.

Picture: see the diagram above.

Row file (contract), e.g. platforms/nitroso/tin/pichu.yaml:

platform: nitroso
service: tin
landscape: pichu
pin: {primordial: "1.0.0", chart: "1.0.0", image: "1.0.0"}   # Kargo; "" = not deployed
values: {}                                                       # human emergency override, Kargo never touches it

CHANGED: the machine rows live in their own repo (pick 4b) and are written to a bot git server first, then mirrored to GitHub (the org blocks deploy keys; CHALLENGES 1).

1. L0 seed (by hand, once): clusters exist; Argo CD + Kargo + cert-manager on Primordial; Secrets for vendor tokens, the CA, one credential per cluster; kubectl apply -f machinery/fleet-root.yaml.
2. fleet-root syncs the CRDs, T3, the platforms AppSet, the baseline AppSet and registry/ (Landscapes, ClusterRegistrations).
3. T3 cluster controller: each ClusterRegistration → Argo CD cluster Secret with labels landscape, mark + a gateway certificate in that cluster.
4. baseline AppSet: every labelled cluster gets the gateway health responder + default TLS store (seen: all 4 clusters in < 1 min, no human step).
5. platforms AppSet finds dpoc-*.carbon repos → one platform-<p> app → Kargo Project + the platform's AppSets.
6. T3 materializer finds dpoc-<p>.<svc> repos → stub rows for every landscape in platform.yaml.
7. Per service: a pipeline app renders the Kargo Warehouse + Stages from stages: [pichu, [raichu, ampharos]].
8. Logto first (platform aldehyde, service lithium): its Database record → T3 makes a Neon database in that landscape's region → Logto starts (2 replicas per cluster) → T3 boot-seeds its own admin app.
9. Services (tin, web): Database/KeyValue/Store/Email + LogtoApp records → T3 builds them → app pods start (they cannot start without the database secret) → Argo CD reports Healthy only when the records are Ready.
10. Traffic: T3 probes every cluster every 5 s and writes the A-sets; edge docs are published to two hosts.

Picture: see the diagram above.

• One version = image + app chart + primordial chart at the same semver; a half-published release makes no Freight.
• Kargo writes the pin into the first stage's row; the Stage is Healthy only when every Application of that row (Primordial + every cluster) is Healthy at that version; then the next stages get it.
• A broken version stops at the first stage (pichu); raichu and ampharos stay on the old one. BUILT (drill 3 numbers in drills.json).

• App name = <service>.<platform>.<landscape>.<root> (LPSM reversed), e.g. tin.nitroso.raichu.c.dpoc.specpenguin.dev.
• Logto issuer per landscape: https://lithium.aldehyde.<landscape>.c.dpoc.specpenguin.dev/oidc – baked into apps, never read from a doc.
• Roots (PROPOSAL-DNS option 4, with the hosts we have):

• Records per root R and landscape L: L.R A = healthy cluster IPs (TTL 60) · *.L.R CNAME → L.R · <mark>.L.R A = one cluster (dark-verify).

Picture: see the diagram above.

Hosts: Tigris public bucket dpoc-edge and AWS S3 + CloudFront. No signing: the client only accepts addresses ending in .dpoc.specpenguin.dev.

• One Logto issuer per landscape (pichu, raichu, ampharos), each with its own database in its own region (Neon Singapore / Oregon), at least 2 replicas per cluster; raichu runs on 2 clusters = 4 replicas sharing one database.
• A user's home landscape is a claim; the client always talks to its home's issuer and API. A dead home region = restore event, not a routing event (home-sharding).
• Access token 10 min, refresh token 14 days rotating. If the issuer's DNS name dies, signed-in users keep working until the pass ends; nobody new signs in (G5 position).

Mercury is the fleet's shared webhook service: an ordinary bun service on its own platform, one intake per serving landscape, behind the same gray DNS. It accepts a vendor's webhook, stores the event in that landscape's Upstash KV, then probes every candidate subscriber at delivery time: exactly one 200 (the owner) and every other 421 ("not mine") completes the delivery; anything else refuses loudly and retries the whole set later (72 h obligation, then dead letter). If every intake is down, the vendors' own retries cover the gap.

1. Bot writes to GitHub: no deploy keys in the org → bot git server + mirror (corpus: name a GitHub App).
2. Images: ghcr refused org CI → own registry.
3. GitHub discovery without a token: 60 calls/h → slow polls (corpus: GitHub App + webhooks).
4. Carbon chart names must follow dpoc-<p>-carbon.
5. Spare DNS name cannot be independent without a second domain.
6. TLS certs need one wildcard per platform (re-issued by T3).
7. Logto's own app records cannot live in Logto's chart (Argo sync would wait on itself).

• What happened: Kargo and T3 must push version pins and stub rows to dpoc-fleet-state. The ProtoDurian org disables deploy keys ("Deploy keys are disabled for this repository"). The only token on hand is the owner's gh login token (admin:org, repo on every repo).
• Example: putting that token on a droplet would let anything on it rewrite any repo of the owner.
• PoC fix: bots push to a small git server on the build machine (git.dpoc.specpenguin.dev, Gitea); a relay on the lead's box mirrors it to GitHub every 3 s. The GitHub copy is what Argo CD reads, so GitHub stays the source of truth people see.
• Corpus change: name the credential: one GitHub App on the org with contents: write on fleet-state only (Kargo + T3) and metadata: read on the org (discovery). The design says "the fleet-write credential lives here" but never says what it is.

• What happened: a workflow with packages: write pushing ghcr.io/protodurian/dpoc-probe got denied (org package policy; no API to change it).
• PoC fix: a registry on the build machine, reg.dpoc.specpenguin.dev (Let's Encrypt TLS, anyone can pull, push needs a password kept as a GitHub repo secret).
• Corpus change: the release CI section should say which registry and which org setting it needs ("allow GitHub Actions to create packages"), and that new ghcr packages start private, so clusters need a pull credential or a visibility step.

• What happened: the platforms AppSet (SCM generator) and the T3 materializer both list the org's repos. Without a token GitHub allows 60 calls an hour per IP; at 90 s + 60 s polling we were at 32 left after 10 minutes.
• PoC fix: SCM generator every 180 s, materializer every 120 s, ETag caching. Result: a new repo is noticed in up to 3 minutes.
• Corpus change: same GitHub App as item 1 (5,000 calls/h) + a GitHub webhook to the AppSet controller, so "new repo → pipeline" is seconds, not minutes.

• What happened: the compiler expects charts dpoc-<p>-<svc> and -primordial. The carbon repo first published dpoc-nitroso (no -carbon), so its pipeline waited forever with no error.
• PoC fix: carbon charts are dpoc-<p>-carbon / dpoc-<p>-carbon-primordial; carbon's Warehouse has no image subscription (spike A hole 12).
• Corpus change: state the carbon chart names and the image-less Warehouse in fleet.md.

• What happened: everything lives under specpenguin.dev, a Cloudflare zone. The proposal (PROPOSAL-DNS.md option 4) wants the spare base on another domain, TLD and registrar.
• PoC fix: main root c.dpoc.specpenguin.dev is one zone on two hosts (Route 53 + DigitalOcean, the proposal's two-host shape with DigitalOcean standing in for Google). Spare stand-in cf.dpoc.specpenguin.dev lives on Cloudflare.
• What this cannot show: "Cloudflare (the parent) is dead" – both names die with it. A truly independent spare needs a second domain on another host.

• What happened: tin.nitroso.pichu.c.dpoc… is two labels under the landscape; *.pichu.c.dpoc… does not cover it.
• PoC fix: each cluster's certificate carries *.<platform>.<landscape>.<root> for every platform; T3 re-issues it when a new platform appears (seen live: 9 → 12 → 15 names as nitroso and aldehyde arrived). DNS needs only *.<landscape>.<root> (DNS wildcards match any depth).
• Corpus change: say the cert SAN set grows per platform, and who re-issues it.

• What happened: the provisioning credential used the docker container name; Primordial pods could not resolve it (Was there a typo in the url or port?). Also the CA field was stored base64 twice. Both fixed in minutes.
• Corpus change: none; the cluster controller DoD should include "Primordial can reach every cluster API" as a status condition.

• What happened: logto db seed failed on Neon: permission denied to create role (42501, "Only roles with the CREATEROLE attribute may create roles"). T3's two-generation logins (app_a/app_b) are plain app logins.
• PoC fix: generation logins get CREATEROLE.
• Corpus change: the Database record needs a privilege field (e.g. privileges: app | admin), and the rotation section must say what happens to roles the app itself created (Logto creates logto_tenant_* roles owned by whichever generation ran the seed).

• What happened: the PoC keeps credentials only in T3's memory (no Infisical). After a restart T3 cannot read them back, so it issues a new generation for every module (seen: generation 1 → 4 after three image updates). Apps survive (they re-read the file and retry), but every restart is a fleet-wide rotation.
• Corpus: this is exactly why the design makes Infisical "the commit point". The PoC skipped Infisical; a real build must not.

• What happened: if dpoc-aldehyde.lithium's primordial chart held a LogtoApp, its Argo CD app could only be Healthy after Logto runs, and Logto cannot run before its Database (same app) is Ready → the stage waits on itself.
• PoC fix: Logto's chart has only its Database; the T3 logto controller boot-seeds its own admin app straight into Logto's database.

• What happened: Logto 1.0.0 could not start (item 8). Its Kargo promotion kept waiting for health (our argocd-update retry window is 15 min, needed for spike A hole 4). Logto 1.0.1 (the fix) was published 2 minutes later but could not promote until the first one ended.
• PoC fix: aborted the stuck promotion (kargo.akuity.io/abort); 1.0.1 promoted at once.
• Corpus change: say what aborts a waiting promotion when newer Freight arrives (T3 could do it: "newer Freight for the same stage + current promotion unhealthy > N min → abort"), or keep the health wait short and do the "apps not created yet" retry differently.