Storage and staging
Only inputs and outputs touch your storage. tif copies each input onto the task's volume before the executors run; tof uploads each output after they finish.
- Inputs:
s3://,http://,https://and inlinecontent. - Outputs:
s3://,http://andhttps://. file://for both, only where the operator turns it on (see Local paths).- Inline content as an output, or an unknown scheme, is refused with
400.GET /service-infolists the supported schemes. - Also refused with
400at submit, since they would only fail in the Pod: an input with neitherurlnorcontent, aDIRECTORYinput overhttp(s)://or as inlinecontent, and ans3://input or output when the deployment givestifandtofno S3 credentials.
Common patterns
- No IO: executors only. Useful to smoke-test image pulls and scheduling.
- S3 in, S3 out:
inputs[]andoutputs[]point at object storage. - Inline content:
inputs[].contentcarries the file body; no external storage needed. An input with non-emptycontentignores itsurl, as TES requires. - Multi-executor: executors run in order; data persists between them only under the task's paths.
S3
Point Poiesis at any S3-compatible store (AWS S3, Ceph, SeaweedFS, MinIO):
s3:
endpoint: "" # AWS_ENDPOINT_URL; empty for AWS
region: eu-central-1 # AWS_REGION
existingSecret: poiesis-s3The Secret holds AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The chart never creates it:
kubectl -n poiesis create secret generic poiesis-s3 \
--from-literal=AWS_ACCESS_KEY_ID=… \
--from-literal=AWS_SECRET_ACCESS_KEY=…For temporary credentials, add an AWS_SESSION_TOKEN key. It is optional.
trec, tif and tof read these keys from the Secret by reference (secretKeyRef), so the values are not in the Job or Pod spec. Executors do not get them. The endpoint and region are not secret and are set as plain values.
A Pod can only read Secrets in its own namespace. If taskpods.namespace is not the release namespace, create the Secret there as well; the chart does not copy it. Use taskpods.s3Secret if it has a different name in the task namespace:
taskpods:
namespace: poiesis-tasks
s3Secret: poiesis-s3 # default: s3.existingSecretIf the Secret is missing in the task namespace, no TaskPod can start, not only the ones that use s3://.
tif and tof need an access key: from existingSecret, from AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the API's environment, or from taskpods.extraEnv. Without one, a task with an s3:// input or output is refused with 400.
On EKS, the service accounts can carry an IAM role through IRSA:
serviceAccount:
api:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/poiesis
taskpod:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/poiesisAn IRSA role alone isn't enough yet: tif and tof still require an access key, so without one s3:// URLs are refused at submit.
Glob outputs
An output whose path has wildcards, such as /outputs/*.vcf, uploads every file and directory that matches, each under its path relative to path_prefix (inferred from the path when unset). Patterns follow Python's glob: *, ? and [...]; ** matches one level only, and {a,b} is not expanded.
If the glob matches no file, tof uploads nothing for that output and logs a warning naming it; the task still succeeds.
taskpods.outputGlobFallback: true makes tof upload every file under path_prefix instead, for clients whose patterns don't match the files they mean. That can export far more than the client asked for. It is off by default.
An output that is a symlink pointing outside the task volume, or a directory holding one, fails the task: tof would otherwise follow the link into its own container. Links that stay inside the volume are fine.
Local paths
file:// URLs are off by default; a task that uses one gets a 400. A file:// path is read or written inside tif and tof, not on the client's machine, so it is only useful for data the operator mounts into TaskPods, such as a shared reference volume.
taskpods:
extraVolumes:
- name: reference
persistentVolumeClaim: { claimName: reference-data }
extraVolumeMounts:
- name: reference
mountPath: /mnt/reference
readOnly: true
fileUrlRoots: ["/mnt/reference"]- A URL must name a path under one of
fileUrlRoots; anything else is refused with400. tifandtofresolve symlinks before they read or write, and stop if the real path leaves those directories.GET /service-infolistsfile://only when it is on.
HTTP(S) inputs
tif downloads an http:// or https:// input with a GET request. It stages the input only from a 2xx response. The body goes to a temporary file next to the input's path and is renamed into place once it is complete, so a failed download leaves no file at the input's path.
| Response | What tif does |
|---|---|
2xx | Stages the body as the input. |
301, 302, 303, 307, 308 | Follows up to 10 redirects, unless the request has reached an https:// URL and the new one isn't. |
429, 500, 502, 503, 504, connection error, timeout, body cut short | Retries twice, after 1 s and 2 s, each time from the input URL. |
| Anything else | Fails the task; nothing is staged. An expired presigned URL answers 403. |
A failed download ends the task in SYSTEM_ERROR (see Staging failures). Directories can't be downloaded over HTTP(S).
HTTP(S) outputs
An outputs[].url can be an http:// or https:// URL that grants access by itself, such as a presigned upload URL from Amazon S3 or Google Cloud Storage, so tof needs no credentials for the client's storage. tof streams each file from disk in a PUT request with a Content-Length header, as presigned upload URLs require.
- A presigned URL is signed for one object, so use one
FILEoutput per presigned URL. - For a
DIRECTORYor glob output,tofderives each file's URL by appending the file's relative path, escaped, to the path of the output URL, keeping its query string. Those URLs carry no valid signature, so point the output URL at a server that signs each upload as it arrives and redirects it to the storage provider. - If a glob matches no file,
tofuploads nothing (see Glob outputs). WithoutputGlobFallbackon, it uploads every file underpath_prefixinstead, and behind a signing server all of those files are signed and stored.
Responses:
| Response | What tof does |
|---|---|
2xx | Done. |
307, 308 | Sends the file again to the new URL, unless the request has reached an https:// URL and the new one isn't. The first request carries the file too, so up to the whole file reaches a redirecting server. |
301, 302, 303 | Fails: following would turn the PUT into a GET. |
429, 500, 502, 503, 504, connection error, timeout | Retries twice, after 1 s and 2 s, each time from the output URL. |
| Anything else | Fails; the task's system_logs and the tof log show the start of the response body. |
tofsets no size limit, but providers cap a singlePUT: 5 GB on Amazon S3.tofconnects to whatever host the output URL or a redirect names, from the TaskPod's network, as executors can. To limit where tasks send data, restrict egress from the task namespace with a NetworkPolicy.- Over
http://, the URL (including any signature) and the file travel unencrypted; usehttps://outside a trusted network. - The URL is stored with the task, including any signature in its query string, so issue presigned URLs with a short expiry.
Output file logs
Once every output is uploaded, tof stores one entry per uploaded file in the task's logs[].outputs, shown in the BASIC and FULL views:
| Field | Value |
|---|---|
path | The file's path in the executor, e.g. /outputs/dir/a.txt. |
size_bytes | The file's size, as a string. |
url | Where the file was uploaded, without the URL's query string or fragment. |
- A
DIRECTORYor glob output gives one entry per file under it, each with the URL that file went to. - The query of a presigned URL is its signature, so it is not stored in the entry. The output's own
urlinoutputs[]keeps it, as submitted. - A
tofthat runs again replaces the entries; it doesn't add more. - If
tofcan't store the entries, it fails and the task ends inSYSTEM_ERROR, though its files are uploaded: aCOMPLETEtask always has its file list.
Staging failures
When tif or tof fails, the task's system_logs say why. The filer writes one line to its container's termination message, and TCtl copies it into the log:
container tif exited with code 1: input /inputs/reads.fq: HTTPError: GET https://bucket.example/reads.fq returned 403: <Error><Code>AccessDenied</Code>...The line is at most 1 KiB. Query strings and the credentials of a URL are cut from it, since the query of a presigned URL is its signature.
| What failed | Task state |
|---|---|
Storage unreachable, a 5xx, a timeout | SYSTEM_ERROR |
| Access denied, an expired or rejected URL, a missing input | SYSTEM_ERROR |
| A full task volume, or the task can't be read from or its output file logs written to the database | SYSTEM_ERROR |
| A declared output the executors didn't produce | EXECUTOR_ERROR |
The rule: a failed filer is SYSTEM_ERROR, because no executor failed. The one exception is an output whose path isn't on the task's volume when tof runs, or is a file where a DIRECTORY was declared. The executors didn't produce what the task declared, so the task ends in EXECUTOR_ERROR and the log names the output:
container tof exited with code 1: output /outputs/sorted.bam was not produced by the executorsA glob that matches no file is not a failure (see Glob outputs).
