Skip to content

Tasks and TaskPods ​

Every task runs as one Pod, the TaskPod, wrapped in a Kubernetes Job. Init containers do the work in order; a sidecar records what happens.

  • tif stages inputs, the executors run one after another, tof uploads outputs. All share the task's volume.
  • An executor that exits non-zero stops the task, unless it sets ignore_error.
  • trec marks the task INITIALIZING when it starts, RUNNING when the first executor starts, and writes executor logs to Postgres as the Pod runs; tctl writes every final state.

The containers ​

ContainerKindDoes
trecNative sidecarWatches its own Pod, marks the task INITIALIZING, then RUNNING when the first executor starts, and writes executor logs to Postgres.
tifInitStages inputs onto the volume: s3://, http(s)://, inline content, then creates the task's directories that have no mount of their own. Only if the task has inputs or such directories.
exec-0 … exec-NInitYour executors, in order, in your images.
tofInitUploads outputs from the volume: s3://, http(s)://. Only if the task has outputs.
ackMainExits 0 so the Pod reaches Succeeded. Reuses the Poiesis image, so no second image pull.

trec is a native sidecar: an init container with restartPolicy: Always, a Kubernetes 1.29 feature. It starts before the other init containers and stays up for the whole Pod. It never writes the final state: TCtl does, and backfills any executor logs TRec missed.

TaskPodinit containers, in ordertifstage inputs, if anyexec-0your first commandexec-…each reads what the last wroteexec-Nyour last commandtofupload outputs, if anyackmain container, exits 0trecnative sidecarrestartPolicy: Alwaysmarks INITIALIZING,RUNNING, writes logsvolumeshared by all but ack

Executors ​

Executors run strictly in order, as init containers named exec-{N}. If executor i fails, executors i+1…n never start and the task ends in EXECUTOR_ERROR. Data passes between executors only through the paths the task declares; see Task paths.

Executors run with no shell, exactly as submitted: image, command, working directory and environment come from the task. Only an executor that sets ignore_error or stdin, stdout or stderr runs under /bin/sh, which its image must then have.

The Pod doesn't mount its ServiceAccount token. Only trec mounts one, at /var/run/secrets/kubernetes.io/serviceaccount, so executors, which run user images, can't call the Kubernetes API.

Executors that may fail ​

By default the first executor that exits non-zero stops the task. An executor with "ignore_error": true doesn't: its exit code is recorded in the task logs and the next executor runs. If the rest succeed, the task ends in COMPLETE.

json
"executors": [
  {
    "image": "alpine:3.20",
    "command": ["sh", "-c", "exit 3"],
    "ignore_error": true
  },
  {"image": "alpine:3.20", "command": ["echo", "hello"]}
]

This task ends in COMPLETE, and logs[0].logs[0].exit_code is 3.

Kubernetes stops the Pod when an init container fails, so Poiesis runs an ignore_error executor under /bin/sh:

  • The shell runs the command as submitted, as positional arguments, so nothing is re-quoted or expanded. It writes the exit code to the container's termination message (/dev/termination-log) and exits 0.
  • The image must have /bin/sh. Without it the container can't start and the task ends in EXECUTOR_ERROR; its system log says the executor sets ignore_error and needs /bin/sh.
  • kubectl get pod shows the container exited 0. The real exit code is in its state.terminated.message, where TRec and TCtl read it, and in the task logs.

stdin, stdout and stderr ​

An executor's stdin is a file in its container to read standard input from; stdout and stderr are files to write standard output and error to. Kubernetes can't connect a container's streams to files, so an executor that sets any of them runs under /bin/sh, which opens the files and then execs the command:

  • The command is passed as positional arguments, so nothing is re-quoted or expanded, and exec keeps its PID, signals and exit code. With ignore_error too, one wrapper does both.
  • The directory of stdout and stderr is created if missing. The same path for both gets both streams, in order.
  • The paths must be absolute (400 otherwise). Put them under a path the task declares (an input, output or volume) to keep the files: like anything else an executor writes, a file elsewhere is lost with its container.
  • The image must have /bin/sh. Without it the container can't start and the task ends in EXECUTOR_ERROR; its system log says the executor sets stdout (or stdin, stderr), which needs /bin/sh in its image.
  • The container's own stdin stays closed: stdin comes from the file only.

Task logs ​

GET /tasks/{id}?view=FULL returns one entry per finished executor in logs[].logs[], with its start and end time, exit code and output:

  • stdout holds the last 64 KiB of the executor's container log. The log mixes the executor's stdout and stderr, so both end up here, and stderr stays empty.
  • An executor that writes its stdout or stderr to a file on the task volume gets the last 64 KiB of that file instead, as stdout or stderr; TRec reads it from the volume, without following symlinks. The container log then goes to the other field. A file elsewhere in the container can't be read, so that stream is missing from the task logs.
  • If the log can't be read (for example, the ServiceAccount lacks pods/log: get), stdout stays empty and TRec or TCtl logs a warning. The task itself is unaffected. TCtl, which backfills logs TRec missed, has no access to task volumes and stores only the container log.

logs[].outputs lists each file tof uploaded: its path in the executor, its size_bytes and the url it was uploaded to. A DIRECTORY or glob output gives one entry per file. See Output file logs.

The BASIC view leaves stdout, stderr and system_logs out. MINIMAL, the default of GetTask and ListTasks, holds only id and state, plus executors as an empty list because the schema requires it. system_logs holds lines from Poiesis itself, such as a non-zero exit of tif or tof with its cause. TRec stopping when the Pod ends isn't one: Kubernetes stops the sidecar last, and TRec exits 0 on SIGTERM.

Who records what ​

Final states go through a conditional UPDATE that requires the row to be non-terminal and gives CANCELING precedence, so a final state lands once and a racing cancel wins, without application locks. See Architecture for what TCtl reconciles.

Time limit ​

Each task's Job sets activeDeadlineSeconds from the chart value taskpods.activeDeadlineSeconds: 3600 seconds (one hour) by default. The limit is the same for every task; a TES request can't change it.

yaml
taskpods:
  activeDeadlineSeconds: 86400 # one day; null for no limit

The clock starts when the Job is created, so time spent waiting for a node, pulling images and staging inputs counts. When it runs out, Kubernetes stops the Pod and marks the Job failed with reason DeadlineExceeded. TCtl reads that from the Job and ends the task as SYSTEM_ERROR with reason DeadlineExceeded and "task exceeded its time limit of N seconds" in system_logs, whether the Pod is already gone, Failed or still stopping. The killed executor's exit code (137 or 143) is not reported as the cause.

A new value applies to tasks created after the API restarts with it; tasks that already have a Job keep the limit they were created with.

A task whose Pod can't be scheduled yet (no node with room, a full ResourceQuota) waits in QUEUED; by default this limit is what ends the wait. To fail such tasks sooner, set tctl.unschedulableTimeoutSeconds:

yaml
tctl:
  unschedulableTimeoutSeconds: 1800 # null (default): wait for the limit

With taskpods.activeDeadlineSeconds: null there is no limit, and a task that can't be scheduled waits for as long as it takes unless tctl.unschedulableTimeoutSeconds is set. A Pod that was scheduled but doesn't start (an image that can't be pulled) still fails after 5 minutes: TCtl's pending timeout doesn't depend on either setting.

Cancelling ​

POST /tasks/{id}:cancel writes CANCELING and deletes the Job. The Pod disappears before reaching a terminal phase; TCtl finds the task without a Job or Pod and writes the final state, which lands as CANCELED because CANCELING takes precedence.

If the Job delete fails, the call returns 500 and the task stays CANCELING. Cancel again to retry the delete; TCtl also deletes the Job of a CANCELING task on its next pass, within about 5 seconds. Cancelling a task that is already terminal changes nothing and returns 200.

Tags ​

TES tag values are strings. Poiesis drops tags whose value is null and accepts the task, as if the tag was never sent (in tag filters, sharing scopes and what a decision service sees, too). Cromwell sends such a tag, parent_workflow_id, for top-level workflows. Any other non-string value (a number, an object) gets 422.

Read-only fields ​

Poiesis assigns task IDs, state, creation time and logs itself and ignores client-supplied values (the TES API marks them read-only).