Production checklist
The Deploy page writes a values.yaml for the Helm chart. This page is the list to go through before real users arrive. Every key is described in Helm values.
The chart isn't published to a registry yet (#53). Install it from a checkout: helm install poiesis ./deployment/helm/poiesis -n poiesis -f values.yaml.
Install
- Kubernetes 1.29+. The chart pins
kubeVersion: ">=1.29.0-0". - Create the Secrets first: the chart only references them, it never creates credentials. The database Secret holds
DATABASE_URL; the S3 Secret holdsAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY, plusAWS_SESSION_TOKENfor temporary credentials. - TaskPods read the same Secrets by reference (
secretKeyRef). No credential value is written into a Job or Pod spec. - Pin
image.tag(it defaults to the chart'sappVersion). - Migrations run as a Helm hook before the API starts; give the hook database access.
Reachability and TLS
- Enable
api.ingresswith TLS, through cert-manager or an existing TLS Secret. - Set
auth.resourceto the public URL, scheme included (https://…, even though Poiesis sees plain HTTP behind the ingress). Clients then discover your issuers from any401; see Discovery. - With a path in the URL (
https://example.org/poiesis), route/.well-known/oauth-protected-resource/poiesisto the API too.
Database
- Managed Postgres or an HA operator.
sslmode=verify-fullwithsslrootcert=/etc/ssl/poiesis/ca.crt, and the CA in a ConfigMap named bydatabase.caConfigmap(keyca.crt).
Storage
- Object storage reachable from the task namespace, with credentials in a Secret or through IRSA. See Storage and staging.
taskpods.persistence.maxSizeGiset to the largest volume a task may ask for (the schema refuses an install without it).- A StorageClass with
ReadWriteOncevolumes, andretainAfterSecondschosen for how long you keep finished tasks for debugging. - Leave
taskpods.outputGlobFallbackoff unless clients need it. - Leave
taskpods.fileUrlRootsempty unless tasks read shared data you mount into TaskPods; list only those mounts. taskpods.activeDeadlineSecondsset above your longest task. The default stops every task after one hour. See Time limit.- With a ResourceQuota on
cpuormemoryin the task namespace, every container needs requests and limits (or a LimitRange that sets them). The chart sets them ontrec,tif,tofandackthroughtaskpods.systemResources. The containers run one after another next totrec, so with the defaults a TaskPod requests at least 110m CPU and 384Mi memory (trecwithtifortof), plustrec's share while executors run. Executors get theirs from the task'sresources; check that such a quota admits them too, or add a LimitRange with defaults. tctl.unschedulableTimeoutSecondsleftnullif tasks may queue for capacity (a full cluster, a ResourceQuota, an autoscaler adding nodes): they wait inQUEUEDuntil the time limit. Set it to fail tasks that wait longer for a node.
Identity
- At least one issuer, with an audience of its own per Poiesis instance and
tokenTypesmatching your IdP (Keycloak:[JWT]). See Keycloak. clientsset, so only your clients' tokens are accepted.- A
requiredClaimif only some of the issuer's users may use the service. auth.apiKeys.enabledonly if engines need it, withkeyMinting.clientslimited to the client people log in with and amaxTtlyou accept as the deprovisioning lag.
Authorization
- Keep
authz.adminReadoff unless operators may see users' task data. - With a decision service: secure transport (loopback sidecar, mesh mTLS or HTTPS), a NetworkPolicy so only Poiesis reaches it, and
sharingonly if tasks should be visible to others. See Authorization.
Security
Serve Poiesis over TLS only; API keys are bearer credentials.
podSecurityEnforce: restricted(the default) for Poiesis-owned containers. Executors run user images and are exempt; constrain them with the namespace's PodSecurity admission level.networkPolicy.enabled, and restrict egress from the task namespace:tofand executors can reach whatever host a URL names. The policies cover the API and TCtl. They allow DNS, TCP 443 and 6443 to any address, and Postgres onnetworkPolicy.databaseEgress.port(default5432, to any address). Narrow Postgres withdatabaseEgress.to, and add an issuer or S3 endpoint on another port withnetworkPolicy.extraEgress.With an ingress controller in another namespace, allow it to reach the API:
yamlnetworkPolicy: enabled: true ingressFrom: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: ingress-nginxThe migration hook isn't selected by the policies; under a default-deny policy of your own, allow it to reach Postgres too.
Rate-limit at the ingress or gateway: Poiesis doesn't.
Split tasks into their own namespace with
taskpods.namespaceif you want them apart from the control plane. The API and TCtl stay in the release namespace, and so does TCtl's leader Lease; the chart renders their Roles for Jobs, PVCs and Pods in the task namespace. Create the database and S3 Secrets in that namespace too: TaskPods read them there, and the chart does not copy them. Seetaskpods.databaseSecret.
Data-plane isolation (an input allowlist, per-task storage credentials, a default TaskPod NetworkPolicy) is planned (#126).
Availability
api.replicasabove 1 andapi.pdb.enabled, orapi.autoscaling.api.topologySpreadConstraintsto spread API replicas across zones or nodes.- On rollouts the API sleeps
api.preStopSleepSeconds(5 s) before SIGTERM, then gunicorn lets running requests finish for up to 30 s. Keepapi.terminationGracePeriodSeconds(40 s) at least their sum. tctl.replicasabove 1 for faster failover (leader election means only one is active). The chart has no TCtl PodDisruptionBudget: an evicted TCtl only delays task state updates until a replica holds the lease again.tctl.probes.livenessleft on. TCtl touches a heartbeat file after each loop iteration; the probe restarts it when the file is older thanmaxHeartbeatAgeSeconds(10 minutes). Raise that if a pass can take longer, which needs thousands of tasks or a very slow API server.- What TCtl does in an outage:
- API server down or slow: each call times out (30 seconds at most). A pass that can't list every Job and Pod does nothing; a leader that can't renew its Lease within 20 seconds stops acting and stands down. Tasks stay as they are and are finalised once the API server answers again.
- Postgres down: each pass fails when it reads the task rows, and does nothing. A terminal write that fails rolls back with its logs and is tried again on the next pass.
- In both, TCtl keeps running and keeps touching its heartbeat, so it isn't restarted. Tasks that finish meanwhile are recorded when the outage ends, as long as their Job hasn't been removed after
retainAfterSeconds(#171).
Observability
- Ship the API's stdout and stderr to a log store: the audit log lines are on stdout, and Poiesis keeps nothing.
logLevel: INFO(DEBUGwhile setting up).
Logs
The API, TCtl and each TaskPod's trec, tif and tof write their log to stderr, one JSON object per line (logging.format: json, the default). logging.format: text gives coloured lines for reading in a terminal; "" picks text when stderr is a terminal and JSON otherwise. The audit log keeps its own format on stdout.
Every line has timestamp (RFC 3339, UTC), level, logger and message, and exception when there is one. API lines written while serving a request have its request_id, the value returned in the X-Request-ID response header and used in audit lines. trec, tif and tof lines, and TCtl lines about one task, have its task_id.
The API writes one access line per request (logger poiesis.access): method, path, status, duration_ms, request_id and, when the request was authenticated, owner. Query strings, headers and bodies are never logged. Probe requests (/healthz, /readyz, /startupz) are logged at DEBUG.
Follow one request, from the request id a client got back:
kubectl -n poiesis logs deploy/poiesis-api \
| jq -cR 'fromjson? | select(.request_id == "3f2c9a1e-…")'Follow one task in TCtl and in its TaskPod:
kubectl -n poiesis logs deploy/poiesis-tctl \
| jq -cR 'fromjson? | select(.task_id == "6d7e0c2a-…")'
# The TaskPod namespace (taskpods.namespace, default the release's):
kubectl -n poiesis logs job/task-6d7e0c2a-… --all-containers \
| jq -cR 'fromjson? | select(.task_id)'fromjson? skips lines that aren't JSON, such as the executors' output or the API command's first line.
Service info
GET /service-info tells registries and clients which deployment they talk to. Give each deployment its own id and, ideally, an environment:
serviceInfo:
id: org.example.tes.prod
name: Example TES
description: TES for the Example genomics platform
organization:
name: Example
url: https://example.org
contactUrl: mailto:tes@example.org
documentationUrl: https://docs.example.org/tes
environment: prod
createdAt: 2026-01-15T09:00:00ZWithout the chart, set the same as POIESIS_SERVICE_INFO_ID, _NAME, _DESCRIPTION, _ORGANIZATION_NAME, _ORGANIZATION_URL, _CONTACT_URL, _DOCUMENTATION_URL, _ENVIRONMENT and _CREATED_AT.
- Unset values keep the built-in ones;
environmentis then left out andcreatedAtis the time of the request. - URLs must be http(s);
contactUrlmay also bemailto:.createdAtis ISO 8601 with a UTC offset. Invalid values stop the API from starting. idis also sent to a decision service ascontext.tes.service_id(Authorization).versionis the Poiesis version; the TES version istype.version.
Upgrading
- From
auth.oidc: see Migrating an existing deployment. - Issuer hardening (token type, per-issuer claim rules): see Upgrading to issuer hardening.
- From
tes-authzen/1: see Migrating from tes-authzen/1. - Migration
000004adds indexes ontasksfor listing: owner with creation time, creation time with id, and a GIN index ontags. It builds them with plainCREATE INDEX, so writes totasks(new tasks, state changes) wait until it finishes. On a large table, run the upgrade when few tasks are submitted.
