Secrets management

Audience

PLANA staff. The end-to-end story of where a secret lives, who can read it, and how to rotate it.

PLANA uses a two-store, dual-write approach to secrets:

Store	Format	Primary use	Lifecycle
SOPS	age-encrypted YAML in git	Source of truth for everything reconciled by Flux	Committed, versioned
Vaultwarden	Browser-accessible vault	Human reference + UI access	Mirror; manually maintained

Every secret created on the platform must end up in both stores. Skipping either is a stability bug — SOPS keeps the cluster reconcilable from git; Vaultwarden keeps secrets discoverable when an engineer is debugging at 02:00.

SOPS — the cluster's source of truth

SOPS encrypts YAML files in place. The encrypted file is safe to commit; the cleartext is only ever in memory during a controlled decryption.

Property	Value
Encryption	age (Curve25519 + ChaCha20-Poly1305) — not GPG
Public key	age1aq8ysk43qj8fqr5phrecukxu4xn59m5awmqvr083kwgfsrej7spsmmc4jv
Private keys	Per-machine, not shared. Located at ~/.config/sops/age/keys.txt on each staff member's laptop
Encrypted file	infra/secrets/plana-pulse.enc.yaml
Decryption tool	~/bin/sops (v3.9.4) — sops in PATH is a broken Python shim, do not use

A secret added to SOPS becomes available to the cluster via the sops-secrets-operator, which decrypts the matching subtree and produces a Kubernetes Secret in the right namespace.

How to add a secret

# DO NOT cat the file or dump it — sops set updates in place
~/bin/sops set 'infra/secrets/plana-pulse.enc.yaml' '["pulse_banking"]["new_bank_client_id"]' '"abc123"'
~/bin/sops set 'infra/secrets/plana-pulse.enc.yaml' '["pulse_banking"]["new_bank_client_secret"]' '"def456"'

sops set updates a single key without ever printing the rest of the file. Never sops --decrypt to stdout or to disk for an edit — that risks leaking cleartext into shell history or a swap file.

How to read a secret

~/bin/sops -d --extract '["pulse_banking"]["new_bank_client_id"]' \
  infra/secrets/plana-pulse.enc.yaml

--extract reads just one key. Avoid sops -d <file> (which decrypts everything to stdout) unless you genuinely need a full file dump (rare).

Categories in the SOPS file

The single encrypted YAML is organised as a tree:

age_key_info
ai_agents:           # anthropic, openrouter
authentik:           # bootstrap_token, db, google oauth
exoscale:            # api keys, dns
forgejo:             # registry, ssh keys, sync bot
kubernetes:          # any cluster-wide creds
load_balancer
mailu_hetzner:       # the Hetzner box
matomo:              # db, admin, oidc
matrix:              # admin tokens
monitoring:          # alertmanager webhooks
odoo_demo
penpot
pgadmin
plana:               # platform-wide
plana_erp:           # erp.planapulse.ai
postgresql:          # pg01 admin
pulse_account
pulse_events
pulse_website
ssh:                 # deploy keys
vaultwarden:         # admin token
redis:               # password

When adding a new category, follow the existing nesting style — flat is better than deep.

Vaultwarden — the human-friendly mirror

A self-hosted Bitwarden-compatible vault at https://vault.planapulse.com. PLANA staff use it for:

• Looking up a secret in a browser without sops -d
• Sharing a credential with a teammate (Bitwarden's collection model)
• Storing customer-specific credentials we hold (banking portal logins, third-party tool accounts)
• Personal passwords (each staff member has their own vault inside the same Vaultwarden)

Access:

• SSO-only via Authentik OIDC. No password-based login.
• TOTP mandatory (enforced by Authentik upstream)
• All staff are in the plana-staff collection
• A separate plana-admins collection holds higher-privilege creds (SOPS age keys for new staff, root passwords, etc.)

Rule: dual-write

When you create a secret, write it to both:

1. SOPS — the cluster needs to read it
2. Vaultwarden — humans need to find it later

A secret in only one place is a bug. The on-call runbook will fail without SOPS; the new engineer onboarding will fail without Vaultwarden.

This is captured in the secrets storage policy memory.

Per-namespace Kubernetes Secrets

Within the cluster, secrets are normal Kubernetes Secret resources mounted as env vars or files into pods. They are created by:

1. The sops-secrets-operator watches SopsSecret resources in infra/k8s/<namespace>/secrets.yaml (which reference SOPS-encrypted chunks)
2. Decrypts the relevant SOPS file using its in-pod age key
3. Produces the Secret resource the application consumes

Kubernetes Secrets are stored in etcd encrypted at rest (an Exoscale SKS feature). They are visible to anyone with get secrets RBAC in the namespace, so RBAC must be tight:

• Tenant namespaces grant get secrets only to the worker SA
• Tools namespaces grant it only to the tool's SA
• Customers never have any RBAC on PLANA-side namespaces

API-key-class secrets — the most-rotated

PSD2 client secrets, Anthropic API keys, Stripe webhook secrets — these are the ones we rotate most often. Each follows the same procedure:

1. Generate the new value (either externally or by openssl rand -hex 32)
2. Add to SOPS (sops set ...)
3. Add to Vaultwarden (paste into the relevant collection item)
4. Reconcile the secret to the cluster (Flux pulls automatically on next loop; or flux reconcile kustomization <name>)
5. Verify the consuming service has picked up the new value (rollout restart if needed)
6. Where applicable, revoke the old value at the source (Stripe, the bank's PSD2 portal, Anthropic console)

Long-lived secrets — the special cases

Secret	Special handling
SOPS age private keys	Per-staff-member. Never shared. To onboard a new staff member, add their public key to the SOPS recipients list and re-encrypt every file. To offboard, remove their public key + re-encrypt + ensure the laptop is wiped.
TPP qualified certificates (QWAC + QSEALC)	Annual renewal from a Bulgarian QTSP. Procedure in infra/docs/runbooks/tpp-cert-renewal.md
Let's Encrypt TLS private keys	Managed by cert-manager; rotated on each renewal automatically. No human handling.
PostgreSQL plana role password	Hardest to rotate — touches every service. Procedure in infra/docs/runbooks/postgres-role-rotation.md

What is NOT in SOPS

Some things look like secrets but should not be in SOPS:

• Kubernetes Service URLs (e.g. redis.redis:6379) — these are topology, not secrets
• Endpoint URLs in general — public if there's an Envoy route
• Customer email addresses — they're in the platform DB, treated as PII under GDPR but not as "secrets"

When in doubt, ask: "Would leaking this give an attacker an action they couldn't otherwise take?" If yes, it's a secret. If no, it's configuration.

Operational tasks

Onboard a new staff member with SOPS access

1. New staff member generates an age keypair on their laptop: age-keygen -o ~/.config/sops/age/keys.txt
2. They share the public key with the SOPS admin
3. SOPS admin adds the public key to .sops.yaml in the infra repo
4. SOPS admin re-encrypts the affected files: sops updatekeys infra/secrets/*.enc.yaml
5. Commit + merge
6. The new staff member can now decrypt SOPS files

Offboard a staff member

1. Remove their public key from .sops.yaml
2. sops updatekeys infra/secrets/*.enc.yaml
3. Commit + merge
4. Revoke their Forgejo access (which removes their git push)
5. Disable their Authentik account
6. Revoke any active sessions in Vaultwarden
7. If the laptop is owned by them and contained the SOPS age private key, ensure they wipe it (a signed letter is acceptable; we trust people)

Rotate the Anthropic API key

1. Generate a new key in the Anthropic console
2. sops set 'infra/secrets/plana-pulse.enc.yaml' '["ai_agents"]["anthropic_api_key"]' '"sk-ant-..."'
3. Update Vaultwarden item "Anthropic API key"
4. flux reconcile kustomization ai-bos-agent
5. kubectl -n ai-bos-agent rollout restart deploy/ai-agents
6. Verify a chat works in BOS
7. Revoke the old key in the Anthropic console

Where to read more

• Network policies — the who can talk to whom side of access
• Audit logging — what reads are captured
• Identity → API keys — workspace vs service keys
• Operations → Flux GitOps — how SOPS reconciles into the cluster