Skip to content

Production checklist ​

The Deploy page writes a values.yaml for the Helm chart. This page is the list to go through before real users arrive. Every key is described in Helm values.

The chart isn't published to a registry yet (#53). Install it from a checkout: helm install poiesis ./deployment/helm/poiesis -n poiesis -f values.yaml.

Install ​

  • Kubernetes 1.29+. The chart pins kubeVersion: ">=1.29.0-0".
  • Create the Secrets first: the chart only references them, it never creates credentials. The database Secret holds DATABASE_URL; the S3 Secret holds AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, plus AWS_SESSION_TOKEN for temporary credentials.
  • TaskPods read the same Secrets by reference (secretKeyRef). No credential value is written into a Job or Pod spec.
  • Pin image.tag (it defaults to the chart's appVersion).
  • Migrations run as a Helm hook before the API starts; give the hook database access.

Reachability and TLS ​

  • Enable api.ingress with TLS, through cert-manager or an existing TLS Secret.
  • Set auth.resource to the public URL, scheme included (https://…, even though Poiesis sees plain HTTP behind the ingress). Clients then discover your issuers from any 401; see Discovery.
  • With a path in the URL (https://example.org/poiesis), route /.well-known/oauth-protected-resource/poiesis to the API too.

Database ​

  • Managed Postgres or an HA operator.
  • sslmode=verify-full with sslrootcert=/etc/ssl/poiesis/ca.crt, and the CA in a ConfigMap named by database.caConfigmap (key ca.crt).

Storage ​

  • Object storage reachable from the task namespace, with credentials in a Secret or through IRSA. See Storage and staging.
  • taskpods.persistence.maxSizeGi set to the largest volume a task may ask for (the schema refuses an install without it).
  • A StorageClass with ReadWriteOnce volumes, and retainAfterSeconds chosen for how long you keep finished tasks for debugging.
  • Leave taskpods.outputGlobFallback off unless clients need it.
  • Leave taskpods.fileUrlRoots empty unless tasks read shared data you mount into TaskPods; list only those mounts.
  • taskpods.activeDeadlineSeconds set above your longest task. The default stops every task after one hour. See Time limit.
  • With a ResourceQuota on cpu or memory in the task namespace, every container needs requests and limits (or a LimitRange that sets them). The chart sets them on trec, tif, tof and ack through taskpods.systemResources. The containers run one after another next to trec, so with the defaults a TaskPod requests at least 110m CPU and 384Mi memory (trec with tif or tof), plus trec's share while executors run. Executors get theirs from the task's resources; check that such a quota admits them too, or add a LimitRange with defaults.
  • tctl.unschedulableTimeoutSeconds left null if tasks may queue for capacity (a full cluster, a ResourceQuota, an autoscaler adding nodes): they wait in QUEUED until the time limit. Set it to fail tasks that wait longer for a node.

Identity ​

  • At least one issuer, with an audience of its own per Poiesis instance and tokenTypes matching your IdP (Keycloak: [JWT]). See Keycloak.
  • clients set, so only your clients' tokens are accepted.
  • A requiredClaim if only some of the issuer's users may use the service.
  • auth.apiKeys.enabled only if engines need it, with keyMinting.clients limited to the client people log in with and a maxTtl you accept as the deprovisioning lag.

Authorization ​

  • Keep authz.adminRead off unless operators may see users' task data.
  • With a decision service: secure transport (loopback sidecar, mesh mTLS or HTTPS), a NetworkPolicy so only Poiesis reaches it, and sharing only if tasks should be visible to others. See Authorization.

Security ​

  • Serve Poiesis over TLS only; API keys are bearer credentials.

  • podSecurityEnforce: restricted (the default) for Poiesis-owned containers. Executors run user images and are exempt; constrain them with the namespace's PodSecurity admission level.

  • networkPolicy.enabled, and restrict egress from the task namespace: tof and executors can reach whatever host a URL names. The policies cover the API and TCtl. They allow DNS, TCP 443 and 6443 to any address, and Postgres on networkPolicy.databaseEgress.port (default 5432, to any address). Narrow Postgres with databaseEgress.to, and add an issuer or S3 endpoint on another port with networkPolicy.extraEgress.

  • With an ingress controller in another namespace, allow it to reach the API:

    yaml
    networkPolicy:
      enabled: true
      ingressFrom:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

    The migration hook isn't selected by the policies; under a default-deny policy of your own, allow it to reach Postgres too.

  • Rate-limit at the ingress or gateway: Poiesis doesn't.

  • Split tasks into their own namespace with taskpods.namespace if you want them apart from the control plane. The API and TCtl stay in the release namespace, and so does TCtl's leader Lease; the chart renders their Roles for Jobs, PVCs and Pods in the task namespace. Create the database and S3 Secrets in that namespace too: TaskPods read them there, and the chart does not copy them. See taskpods.databaseSecret.

Data-plane isolation (an input allowlist, per-task storage credentials, a default TaskPod NetworkPolicy) is planned (#126).

Availability ​

  • api.replicas above 1 and api.pdb.enabled, or api.autoscaling.
  • api.topologySpreadConstraints to spread API replicas across zones or nodes.
  • On rollouts the API sleeps api.preStopSleepSeconds (5 s) before SIGTERM, then gunicorn lets running requests finish for up to 30 s. Keep api.terminationGracePeriodSeconds (40 s) at least their sum.
  • tctl.replicas above 1 for faster failover (leader election means only one is active). The chart has no TCtl PodDisruptionBudget: an evicted TCtl only delays task state updates until a replica holds the lease again.
  • tctl.probes.liveness left on. TCtl touches a heartbeat file after each loop iteration; the probe restarts it when the file is older than maxHeartbeatAgeSeconds (10 minutes). Raise that if a pass can take longer, which needs thousands of tasks or a very slow API server.
  • What TCtl does in an outage:
    • API server down or slow: each call times out (30 seconds at most). A pass that can't list every Job and Pod does nothing; a leader that can't renew its Lease within 20 seconds stops acting and stands down. Tasks stay as they are and are finalised once the API server answers again.
    • Postgres down: each pass fails when it reads the task rows, and does nothing. A terminal write that fails rolls back with its logs and is tried again on the next pass.
    • In both, TCtl keeps running and keeps touching its heartbeat, so it isn't restarted. Tasks that finish meanwhile are recorded when the outage ends, as long as their Job hasn't been removed after retainAfterSeconds (#171).

Observability ​

  • Ship the API's stdout and stderr to a log store: the audit log lines are on stdout, and Poiesis keeps nothing.
  • logLevel: INFO (DEBUG while setting up).

Logs ​

The API, TCtl and each TaskPod's trec, tif and tof write their log to stderr, one JSON object per line (logging.format: json, the default). logging.format: text gives coloured lines for reading in a terminal; "" picks text when stderr is a terminal and JSON otherwise. The audit log keeps its own format on stdout.

Every line has timestamp (RFC 3339, UTC), level, logger and message, and exception when there is one. API lines written while serving a request have its request_id, the value returned in the X-Request-ID response header and used in audit lines. trec, tif and tof lines, and TCtl lines about one task, have its task_id.

The API writes one access line per request (logger poiesis.access): method, path, status, duration_ms, request_id and, when the request was authenticated, owner. Query strings, headers and bodies are never logged. Probe requests (/healthz, /readyz, /startupz) are logged at DEBUG.

Follow one request, from the request id a client got back:

sh
kubectl -n poiesis logs deploy/poiesis-api \
  | jq -cR 'fromjson? | select(.request_id == "3f2c9a1e-…")'

Follow one task in TCtl and in its TaskPod:

sh
kubectl -n poiesis logs deploy/poiesis-tctl \
  | jq -cR 'fromjson? | select(.task_id == "6d7e0c2a-…")'
# The TaskPod namespace (taskpods.namespace, default the release's):
kubectl -n poiesis logs job/task-6d7e0c2a-… --all-containers \
  | jq -cR 'fromjson? | select(.task_id)'

fromjson? skips lines that aren't JSON, such as the executors' output or the API command's first line.

Service info ​

GET /service-info tells registries and clients which deployment they talk to. Give each deployment its own id and, ideally, an environment:

yaml
serviceInfo:
  id: org.example.tes.prod
  name: Example TES
  description: TES for the Example genomics platform
  organization:
    name: Example
    url: https://example.org
  contactUrl: mailto:tes@example.org
  documentationUrl: https://docs.example.org/tes
  environment: prod
  createdAt: 2026-01-15T09:00:00Z

Without the chart, set the same as POIESIS_SERVICE_INFO_ID, _NAME, _DESCRIPTION, _ORGANIZATION_NAME, _ORGANIZATION_URL, _CONTACT_URL, _DOCUMENTATION_URL, _ENVIRONMENT and _CREATED_AT.

  • Unset values keep the built-in ones; environment is then left out and createdAt is the time of the request.
  • URLs must be http(s); contactUrl may also be mailto:. createdAt is ISO 8601 with a UTC offset. Invalid values stop the API from starting.
  • id is also sent to a decision service as context.tes.service_id (Authorization).
  • version is the Poiesis version; the TES version is type.version.

Upgrading ​

  • From auth.oidc: see Migrating an existing deployment.
  • Issuer hardening (token type, per-issuer claim rules): see Upgrading to issuer hardening.
  • From tes-authzen/1: see Migrating from tes-authzen/1.
  • Migration 000004 adds indexes on tasks for listing: owner with creation time, creation time with id, and a GIN index on tags. It builds them with plain CREATE INDEX, so writes to tasks (new tasks, state changes) wait until it finishes. On a large table, run the upgrade when few tasks are submitted.