Skip to content

Workflow engines ​

EngineStatusChecked with
Nextflow (nf-ga4gh)TestedNextflow 25.10.4, nf-ga4gh 1.5.0
Snakemake (TES executor)TestedSnakemake 9.27.2, plugin 0.1.3
py-tesTested1.1.4
SprocketTestedSprocket 0.34.1
cwl-tesPartly tested: API key as Basic password onlymain branch
CromwellNot tested yet
ToilNot tested yet

Tested means real workflows ran end to end. Using another engine? Open an issue or send a PR with the setup.

Credentials ​

Workflow engines read one credential when they start and use it for the whole run. Give them a Poiesis API key, not an access token: a token expires within the hour, a key lasts as long as you minted it for (7 days by default, if your operator turned keys on).

Mint a key first, from the Swagger UI at <API URL>/docs or with curl (see Minting a key), and keep it out of version control. The examples use http://localhost:8000, the port-forwarded dev stack.

EngineBearer (preferred)Basic (key as password)
Nextflow (nf-ga4gh)tes.oauthTokentes.basicUsername, tes.basicPassword
Snakemake (TES executor)--tes-token--tes-user, --tes-password
cwl-tesnot usable (see below)--user, --password
py-tesHTTPClient(token=…)HTTPClient(user=…, password=…)
Sprocketauth.type = "bearer"auth.type = "basic"

Nextflow, Snakemake, py-tes and Sprocket were checked with both forms; cwl-tes with Basic only.

Nextflow ​

The nf-ga4gh plugin sends tes.oauthToken as Authorization: Bearer <token> on every request:

groovy
plugins {
  id 'nf-ga4gh'
}

process.executor = 'tes'
tes.endpoint = 'http://localhost:8000/ga4gh/tes'
tes.oauthToken = 'poi_k7H2pQ…'

The plugin appends /v1 to tes.endpoint, so it ends in /ga4gh/tes.

Keep the key out of nextflow.config (it tends to get committed): store it as a Nextflow secret and refer to it.

bash
nextflow secrets set POIESIS_KEY poi_k7H2pQ…
groovy
tes.oauthToken = secrets.POIESIS_KEY

Basic works too, with any user name:

groovy
tes.basicUsername = 'nextflow'
tes.basicPassword = 'poi_k7H2pQ…'

Set only one of the two: both write the Authorization header.

TaskPods can't see your local file system, so processes that pass files need an object store as the workDir. With the dev stack's S3 store:

groovy
workDir = 's3://poiesis-test/nextflow'

aws {
  accessKey = 'admin'
  secretKey = 'password'
  client {
    endpoint = 'http://localhost:9000'
    s3PathStyleAccess = true
  }
}

Create the bucket once before the first run (see S3 inputs and outputs). Older S3-compatible stores (SeaweedFS 3.x, for one) reject the AWS SDK's default checksums with Content input stream does not support mark/reset; for those, run Nextflow with AWS_REQUEST_CHECKSUM_CALCULATION=when_required. The dev store accepts them.

An end-to-end run, with the two configs above in nextflow.config and this main.nf:

groovy
process SAY_HI {
  container 'ubuntu:latest'

  input:
  val person

  output:
  stdout

  script:
  """
  echo "Hello ${person} from the TES executor!"
  """
}

workflow {
  Channel.of('Alice', 'Bob', 'Charlie') | SAY_HI | view
}
bash
kubectl -n po port-forward svc/poiesis-api 8000:8000 &
kubectl -n deps port-forward svc/s3 9000:9000 &
nextflow run main.nf

Each process runs as its own TaskPod, and the three greetings print when they finish. Written against Nextflow 25.04.2; authentication checked with 25.10.4 and nf-ga4gh 1.5.0.

Snakemake ​

The TES executor plugin (pip install snakemake-executor-plugin-tes) takes the key as --tes-token and sends it as a bearer token:

bash
snakemake --executor tes \
  --tes-url http://localhost:8000/ga4gh/tes \
  --tes-token poi_k7H2pQ… \
  --default-storage-provider s3 \
  --default-storage-prefix s3://poiesis-test/snakemake \
  --jobs 10

The plugin pins py-tes 0.4, which appends /v1/tasks to --tes-url, so the URL ends in /ga4gh/tes, as for Nextflow. The token can also come from the environment, which keeps it out of the shell history: export SNAKEMAKE_TES_TOKEN=poi_k7H2pQ…. For Basic, use --tes-user (any name) and --tes-password poi_… (or SNAKEMAKE_TES_USER and SNAKEMAKE_TES_PASSWORD) instead.

TaskPods don't share a file system with the host running Snakemake, so the executor needs remote storage such as S3 (--default-storage-* and the matching storage plugin).

Things that trip runs up, none of them about the key:

  • At the start of every job the plugin runs pip install for the storage plugin inside the TaskPod, so TaskPods need to reach PyPI (or a mirror), or use an image where that install is a no-op.
  • When a job fails, plugin 0.1.3 tries to read $GITHUB_WORKSPACE/funnel.log and stops with KeyError: 'GITHUB_WORKSPACE'; the job's own error is in the TaskPod's logs.
  • The default snakemake/snakemake image is amd64-only; on arm64 nodes pass an arm64 image with --container-image.

cwl-tes ​

cwl-tes checks its --token locally as an RS256-signed JWT before sending anything, so it refuses an API key (Token is not valid). Send the key as a Basic password instead:

bash
cwl-tes --tes http://localhost:8000 \
  --user cwl-tes --password poi_k7H2pQ… \
  workflow.cwl inputs.yml

--user and --password are on cwl-tes's main branch but not in the 0.3.0 release on PyPI; install from the repository until a release includes them:

bash
pip install git+https://github.com/ohsu-comp-bio/cwl-tes.git

py-tes ​

py-tes, the Python TES client (also used by the Snakemake executor and cwl-tes), sends token as a bearer token. Since 1.1 it takes the server's base URL and finds /ga4gh/tes/v1 itself (older versions need <base URL>/ga4gh/tes):

python
import tes

client = tes.HTTPClient("http://localhost:8000", token="poi_k7H2pQ…")
task_id = client.create_task(
    tes.Task(executors=[tes.Executor(image="alpine", command=["echo", "hi"])])
)
print(client.get_task(task_id).state)

tes.HTTPClient(url, user="py-tes", password="poi_k7H2pQ…") sends Basic instead. Set either token or user and password, not both.

Sprocket ​

Sprocket (St. Jude's WDL engine) has a TES backend. Name it in sprocket.toml and give it the key as a bearer token:

toml
[run]
backend = "poiesis"

[run.backends.poiesis]
type = "tes"
url = "http://localhost:8000/ga4gh/tes/v1"
insecure = true          # needed for an http:// URL
inputs = "s3://poiesis-test/sprocket/inputs/"
outputs = "s3://poiesis-test/sprocket/outputs/"

[run.backends.poiesis.auth]
type = "bearer"
token = "poi_k7H2pQ…"

[run.storage.s3]
region = "us-east-1"
endpoint = "http://localhost:9000"

[run.storage.s3.auth]
access_key_id = "admin"
secret_access_key = "password"

Unlike Nextflow and Snakemake, Sprocket uses url as is, so it ends in /ga4gh/tes/v1 (with /ga4gh/tes, task creation gets 404). For Basic, use type = "basic", any username, and the key as password.

TaskPods don't share a file system with the host running Sprocket, so inputs and outputs must be S3 URLs (ending in /), and [run.storage.s3] must let Sprocket itself reach the same bucket: it uploads local File inputs under inputs and reads each task's stdout, stderr and output files from under outputs. s3://File inputs are passed to Poiesis as they are.

Sprocket sets each executor's stdout and stderr to /mnt/task/stdout and /mnt/task/stderr and uploads both as outputs; Poiesis writes them there (see stdin, stdout and stderr), and the task logs show their tails. With cgroup_usage on (the default), Sprocket also records each task's CPU and memory use in a task volume, /mnt/task/usage, which TIF creates.

Sprocket runs commands with bash by default; for images without it (alpine), set shell = "sh" under [run.task].

Sprocket sends ignore_error: true and reads the exit code from the task log, so a task with a non-zero exit is COMPLETE in Poiesis and failed in Sprocket.

Checked with Sprocket 0.34.1, bearer and Basic, cgroup_usage on.

Cromwell ​

Not tested end to end yet. One known incompatibility is handled: Cromwell's TES backend tags every task with workflow_id, root_workflow_id and parent_workflow_id, which is null for a top-level workflow. TES tag values are strings, so Poiesis drops tags whose value is null and accepts the task: the task keeps the other two tags, and parent_workflow_id is simply absent, as if never sent (in tag filters, sharing scopes and what the decision service sees too). Any other value that is not a string (a number, an object) still gets 422.

Toil ​

Not tested. Toil's batch systems (Slurm, Kubernetes, AWS Batch and others) don't include TES, so Toil can't send tasks to Poiesis today.

TES compliance suite ​

Poiesis passes the GA4GH TES compliance suite for TES 1.1.0; CI runs it on every pull request. To run it yourself, see Testing.

Other clients ​

Anything that can set a header works: send Authorization: Bearer poi_… on every request, or HTTP Basic with the key as password. A 401 with invalid token means the key is unknown, revoked or expired; mint a new one. A 403 on create or cancel with API key lacks scope tasks:write means a read-only key.

A client that logs in by itself (to mint a key, say) needn't be told where: the WWW-Authenticate header of any 401 points to the deployment's metadata, which lists its issuers; see Discovery.