Skip to content

Storage and staging ​

Only inputs and outputs touch your storage. tif copies each input onto the task's volume before the executors run; tof uploads each output after they finish.

  • Inputs: s3://, http://, https:// and inline content.
  • Outputs: s3://, http:// and https://.
  • file:// for both, only where the operator turns it on (see Local paths).
  • Inline content as an output, or an unknown scheme, is refused with 400. GET /service-info lists the supported schemes.
  • Also refused with 400 at submit, since they would only fail in the Pod: an input with neither url nor content, a DIRECTORY input over http(s):// or as inline content, and an s3:// input or output when the deployment gives tif and tof no S3 credentials.

Common patterns ​

  • No IO: executors only. Useful to smoke-test image pulls and scheduling.
  • S3 in, S3 out: inputs[] and outputs[] point at object storage.
  • Inline content: inputs[].content carries the file body; no external storage needed. An input with non-empty content ignores its url, as TES requires.
  • Multi-executor: executors run in order; data persists between them only under the task's paths.

S3 ​

Point Poiesis at any S3-compatible store (AWS S3, Ceph, SeaweedFS, MinIO):

yaml
s3:
  endpoint: ""            # AWS_ENDPOINT_URL; empty for AWS
  region: eu-central-1    # AWS_REGION
  existingSecret: poiesis-s3

The Secret holds AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The chart never creates it:

bash
kubectl -n poiesis create secret generic poiesis-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=… \
  --from-literal=AWS_SECRET_ACCESS_KEY=…

For temporary credentials, add an AWS_SESSION_TOKEN key. It is optional.

trec, tif and tof read these keys from the Secret by reference (secretKeyRef), so the values are not in the Job or Pod spec. Executors do not get them. The endpoint and region are not secret and are set as plain values.

A Pod can only read Secrets in its own namespace. If taskpods.namespace is not the release namespace, create the Secret there as well; the chart does not copy it. Use taskpods.s3Secret if it has a different name in the task namespace:

yaml
taskpods:
  namespace: poiesis-tasks
  s3Secret: poiesis-s3 # default: s3.existingSecret

If the Secret is missing in the task namespace, no TaskPod can start, not only the ones that use s3://.

tif and tof need an access key: from existingSecret, from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the API's environment, or from taskpods.extraEnv. Without one, a task with an s3:// input or output is refused with 400.

On EKS, the service accounts can carry an IAM role through IRSA:

yaml
serviceAccount:
  api:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/poiesis
  taskpod:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/poiesis

An IRSA role alone isn't enough yet: tif and tof still require an access key, so without one s3:// URLs are refused at submit.

Poiesis talks to storage with AWS credentials. Google Cloud Storage works through its S3-compatible API with HMAC keys in the Secret. Native GCS, Azure Blob and FTP are planned (#29), as are FUSE mounts (#27).

Glob outputs ​

An output whose path has wildcards, such as /outputs/*.vcf, uploads every file and directory that matches, each under its path relative to path_prefix (inferred from the path when unset). Patterns follow Python's glob: *, ? and [...]; ** matches one level only, and {a,b} is not expanded.

If the glob matches no file, tof uploads nothing for that output and logs a warning naming it; the task still succeeds.

taskpods.outputGlobFallback: true makes tof upload every file under path_prefix instead, for clients whose patterns don't match the files they mean. That can export far more than the client asked for. It is off by default.

An output that is a symlink pointing outside the task volume, or a directory holding one, fails the task: tof would otherwise follow the link into its own container. Links that stay inside the volume are fine.

Local paths ​

file:// URLs are off by default; a task that uses one gets a 400. A file:// path is read or written inside tif and tof, not on the client's machine, so it is only useful for data the operator mounts into TaskPods, such as a shared reference volume.

yaml
taskpods:
  extraVolumes:
    - name: reference
      persistentVolumeClaim: { claimName: reference-data }
  extraVolumeMounts:
    - name: reference
      mountPath: /mnt/reference
      readOnly: true
  fileUrlRoots: ["/mnt/reference"]
  • A URL must name a path under one of fileUrlRoots; anything else is refused with 400.
  • tif and tof resolve symlinks before they read or write, and stop if the real path leaves those directories.
  • GET /service-info lists file:// only when it is on.

HTTP(S) inputs ​

tif downloads an http:// or https:// input with a GET request. It stages the input only from a 2xx response. The body goes to a temporary file next to the input's path and is renamed into place once it is complete, so a failed download leaves no file at the input's path.

ResponseWhat tif does
2xxStages the body as the input.
301, 302, 303, 307, 308Follows up to 10 redirects, unless the request has reached an https:// URL and the new one isn't.
429, 500, 502, 503, 504, connection error, timeout, body cut shortRetries twice, after 1 s and 2 s, each time from the input URL.
Anything elseFails the task; nothing is staged. An expired presigned URL answers 403.

A failed download ends the task in SYSTEM_ERROR (see Staging failures). Directories can't be downloaded over HTTP(S).

HTTP(S) outputs ​

An outputs[].url can be an http:// or https:// URL that grants access by itself, such as a presigned upload URL from Amazon S3 or Google Cloud Storage, so tof needs no credentials for the client's storage. tof streams each file from disk in a PUT request with a Content-Length header, as presigned upload URLs require.

  • A presigned URL is signed for one object, so use one FILE output per presigned URL.
  • For a DIRECTORY or glob output, tof derives each file's URL by appending the file's relative path, escaped, to the path of the output URL, keeping its query string. Those URLs carry no valid signature, so point the output URL at a server that signs each upload as it arrives and redirects it to the storage provider.
  • If a glob matches no file, tof uploads nothing (see Glob outputs). With outputGlobFallback on, it uploads every file under path_prefix instead, and behind a signing server all of those files are signed and stored.

Responses:

ResponseWhat tof does
2xxDone.
307, 308Sends the file again to the new URL, unless the request has reached an https:// URL and the new one isn't. The first request carries the file too, so up to the whole file reaches a redirecting server.
301, 302, 303Fails: following would turn the PUT into a GET.
429, 500, 502, 503, 504, connection error, timeoutRetries twice, after 1 s and 2 s, each time from the output URL.
Anything elseFails; the task's system_logs and the tof log show the start of the response body.
  • tof sets no size limit, but providers cap a single PUT: 5 GB on Amazon S3.
  • tof connects to whatever host the output URL or a redirect names, from the TaskPod's network, as executors can. To limit where tasks send data, restrict egress from the task namespace with a NetworkPolicy.
  • Over http://, the URL (including any signature) and the file travel unencrypted; use https:// outside a trusted network.
  • The URL is stored with the task, including any signature in its query string, so issue presigned URLs with a short expiry.

Output file logs ​

Once every output is uploaded, tof stores one entry per uploaded file in the task's logs[].outputs, shown in the BASIC and FULL views:

FieldValue
pathThe file's path in the executor, e.g. /outputs/dir/a.txt.
size_bytesThe file's size, as a string.
urlWhere the file was uploaded, without the URL's query string or fragment.
  • A DIRECTORY or glob output gives one entry per file under it, each with the URL that file went to.
  • The query of a presigned URL is its signature, so it is not stored in the entry. The output's own url in outputs[] keeps it, as submitted.
  • A tof that runs again replaces the entries; it doesn't add more.
  • If tof can't store the entries, it fails and the task ends in SYSTEM_ERROR, though its files are uploaded: a COMPLETE task always has its file list.

Staging failures ​

When tif or tof fails, the task's system_logs say why. The filer writes one line to its container's termination message, and TCtl copies it into the log:

text
container tif exited with code 1: input /inputs/reads.fq: HTTPError: GET https://bucket.example/reads.fq returned 403: <Error><Code>AccessDenied</Code>...

The line is at most 1 KiB. Query strings and the credentials of a URL are cut from it, since the query of a presigned URL is its signature.

What failedTask state
Storage unreachable, a 5xx, a timeoutSYSTEM_ERROR
Access denied, an expired or rejected URL, a missing inputSYSTEM_ERROR
A full task volume, or the task can't be read from or its output file logs written to the databaseSYSTEM_ERROR
A declared output the executors didn't produceEXECUTOR_ERROR

The rule: a failed filer is SYSTEM_ERROR, because no executor failed. The one exception is an output whose path isn't on the task's volume when tof runs, or is a file where a DIRECTORY was declared. The executors didn't produce what the task declared, so the task ends in EXECUTOR_ERROR and the log names the output:

text
container tof exited with code 1: output /outputs/sorted.bam was not produced by the executors

A glob that matches no file is not a failure (see Glob outputs).