Developer DocsOperations & observability

Operations & observability

Running the platform day to day: health, logs, backups, and a short runbook. Pairs with Deployment.

Health checks

GET /api/v1/health is public (for load balancers / uptime monitors) and returns:

{ "status": "ok", "version": "1.0.0", "service": "playlogiq-api",
  "checks": { "database": { "status": "ok", "latency_ms": 3 } },
  "timestamp": "…" }

200 when healthy, 503 when a check fails (status: "degraded"). It verifies Supabase/Postgres connectivity. Point your uptime monitor at it per environment.

Logs

  • App/worker/service logs: docker compose logs -f <service> (e.g. web, api, worker, rest, db). The worker logs each queue’s activity and delivery outcomes; webhook attempts are also persisted in webhook_deliveries.
  • Error tracking (optional): Sentry is wired via NEXT_PUBLIC_SENTRY_DSN / SENTRY_*. Product analytics (optional): PostHog via NEXT_PUBLIC_POSTHOG_KEY / _HOST. Both are off if the env vars are unset.

Backups & restore

  • A daily job dumps the whole database and uploads it off-site to a Hetzner Storage Box, pruning beyond the retention window (infra/backup/backup.sh, scheduled via infra/backup/crontab.example). Config comes from infra/.env (STORAGEBOX_*, BACKUP_RETENTION_DAYS).
  • Restore with infra/backup/restore.sh. Test a restore periodically — an untested backup is not a backup.

Runbook

Roll an app to a new image

cd /opt/playlogiq/infra
docker compose pull <web|api|worker|docs>
docker compose up -d --force-recreate <service>
docker inspect playlogiq-<service>-1 --format '{{.Image}}'   # confirm digest changed

Wait for CI “Build & push images” to be green for the commit before pulling.

Apply a database migration

docker compose exec -T db psql -v ON_ERROR_STOP=1 -U postgres -d postgres \
  < ../supabase/migrations/NNN_description.sql
docker compose restart rest      # if it added a table/RPC PostgREST must expose

Apply only the new files in production (older migrations aren’t all idempotent). See Deployment.

PostgREST returns 404 for a new table/RPC

It hasn’t reloaded its schema cache: docker compose restart rest.

Inspect background jobs

The worker drains BullMQ queues over Redis. Check worker health with docker compose logs -f worker; inspect Redis with docker compose exec redis redis-cli. Queues are documented in Background jobs.

A service is unhealthy

docker compose ps shows status; docker compose restart <service>; check GET /health and the logs. db must be healthy before web/api will serve.

Ingestion rejects / rate limits

429 responses mean a tenant is over its per-minute budget (RATE_LIMIT_INGEST_PER_MIN, default 6000). The limiter fails open if Redis is down. See Event ingestion API → Rate limits.

Environments (staging + prod)

Run staging on a separate, dedicated server — never share a database or box with production. Staging runs the identical full stack (Caddy + the four apps + worker + the whole self-hosted Supabase), so RAM is the binding constraint.

  • Production (reference): Hetzner CPX42 — 8 vCPU / 16 GB / 320 GB.
  • Staging (recommended): Hetzner CPX32 — 4 vCPU / 8 GB / 160 GB. 8 GB comfortably runs the full stack at staging load; 4 GB risks OOM with all of Supabase + four Next.js apps + the worker.

Images are environment-baked

The NEXT_PUBLIC_* values (Supabase URL + anon key, and the app/API/player/docs URLs) are inlined into the browser bundle at build time. Production CI (images.yml) bakes the production URLs and tags images :latest on push to main. Staging CI (images-staging.yml) bakes the staging URLs and tags :staging on push to develop.

Consequence: a staging server cannot reuse :latest — its bundle would call production. It pulls :staging (built with crm.staging.logiqdesk.com, etc.).

Bring up a staging server

  1. Provision the server (Ubuntu) and add DNS A records for the staging hosts → the staging IP. The Caddyfile is {$DOMAIN}-parametrised, so with DOMAIN=staging.logiqdesk.com the hosts are crm.staging.logiqdesk.com, play.staging.logiqdesk.com, crm-api.staging.logiqdesk.com, crm-docs.staging.logiqdesk.com, crm-db.staging.logiqdesk.com.
  2. Clone the repo to /opt/playlogiq and cd infra.
  3. Generate fresh secrets: bash gen-secrets.sh — a staging environment must have its own secrets (never copy production’s).
  4. In infra/.env set DOMAIN=staging.logiqdesk.com and IMAGE_TAG=staging (so compose pulls the :staging images), plus any external creds you want staging to exercise.
  5. Add a repo variable NEXT_PUBLIC_SUPABASE_ANON_KEY_STAGING (the staging ANON_KEY from step 3) so images-staging.yml bakes the right anon key, then run that workflow (push to develop or dispatch it).
  6. docker login ghcr.io (if the images are private), docker compose up -d, apply migrations to the fresh DB (bash db-migrate.sh --seed), and bootstrap an admin (bash bootstrap-admin.sh).

Promotion model

develop ──images-staging.yml──► :staging ──► staging server (staging URLs, staging DB)
   │  validate (health, smoke test, dashboard, migrations)
   ▼
merge to main ──images.yml──► :latest ──► production server
  • Migrations are plain SQL (environment-agnostic): apply the new files to staging first, verify, then production — same files, same order.
  • Keep the two servers on separate databases and secrets; never point staging at production data.

Production builds are incremental — a commit rebuilds only the app images that changed (a shared packages/**/lockfile/Dockerfile change rebuilds all). A docs-only commit rebuilds just docs.

DeploymentSecurity & compliance