Loop Templates: Iteration, Aggregation, and Concurrency

Loop templates let you run the same template body across many inputs without duplicating DAG nodes. Instead of writing one task node per file, region, or record, you declare a single loop template and let the engine fan out to as many iterations as the data demands. Aether supports three distinct iteration modes — static items known at definition time, dynamic items produced by an upstream task, and an expression-driven repeat-until pattern — each suited to different pipeline shapes. All three modes share the same concurrency, aggregation, and output wiring model.

Loop Template Structure

A loop template is declared with the loop key inside a templates entry:

{
  "loop": {
    "name": "run-loop",
    "items": ["jan.csv", "feb.csv", "mar.csv"],
    "concurrency": 2,
    "body": "process-file",
    "arguments": {
      "parameters": [
        { "name": "filename", "value": "{{iterator.item}}" },
        { "name": "index",    "value": "{{iterator.index}}" }
      ]
    },
    "aggregate": {
      "strategy": "list",
      "parameters": ["status", "rows"]
    },
    "outputs": {
      "parameters": [
        { "name": "status", "type": "array" },
        { "name": "rows",   "type": "array" }
      ]
    }
  }
}

Field reference

Field	Type	Description
`body`	string	Name of the template to run for each iteration. Required.
`items`	any[]	Static list of items to iterate over.
`itemsFrom`	string	Expression that resolves to a task output array at runtime.
`repeatCondition`	string	Boolean expression evaluated after each iteration.
`concurrency`	int	Max parallel iterations. `0` = unlimited.
`maxIterations`	int	Safety cap on total iteration count.
`arguments`	Arguments	Parameters passed to `body` on each iteration.
`aggregate`	Aggregate	How to combine iteration outputs into loop-level outputs.
`inputs`	Inputs	Parameters this loop accepts when called from a parent DAG.
`outputs`	Outputs	Parameters exported by this loop after all iterations.
`timeout`	string	Duration limit for the entire loop (e.g. `"1h"`).

Iteration Mode 1: Static Items

Use items when the full list of inputs is known when you write the workflow. The engine creates one iteration per element.

{
  "loop": {
    "name": "run-loop",
    "items": ["jan.csv", "feb.csv", "mar.csv"],
    "concurrency": 2,
    "body": "process-file",
    "arguments": {
      "parameters": [
        { "name": "filename", "value": "{{iterator.item}}" },
        { "name": "index",    "value": "{{iterator.index}}" }
      ]
    },
    "aggregate": {
      "strategy": "list",
      "parameters": ["status", "rows"]
    },
    "outputs": {
      "parameters": [
        { "name": "status", "type": "array" },
        { "name": "rows",   "type": "array" }
      ]
    }
  }
}

Inside arguments, the special interpolation variables are:

{{iterator.item}} — the current element from the items array
{{iterator.index}} — the zero-based index of the current iteration

With concurrency: 2, at most two iterations run simultaneously. Setting concurrency: 0 allows all iterations to run in parallel.

Iteration Mode 2: Dynamic Items from a Task Output

Use itemsFrom when the list of inputs is produced at runtime by an upstream task. The expression resolves to a JSON array stored in a task output parameter.

{
  "dag": {
    "name": "pipeline",
    "tasks": [
      { "name": "produce",  "template": "producer" },
      {
        "name": "process",
        "template": "dynamic-loop",
        "dependencies": ["produce"],
        "arguments": {
          "parameters": [
            {
              "name": "items-list",
              "valueFrom": { "parameter": "tasks.produce.outputs.parameters.files" }
            }
          ]
        }
      }
    ]
  }
}

The loop template receives the list via its declared inputs and references it in itemsFrom:

{
  "loop": {
    "name": "dynamic-loop",
    "inputs": {
      "parameters": [
        { "name": "items-list", "type": "array" }
      ]
    },
    "itemsFrom": "{{inputs.parameters.items-list}}",
    "body": "worker",
    "arguments": {
      "parameters": [
        { "name": "file", "value": "{{iterator.item}}" }
      ]
    },
    "aggregate": {
      "strategy": "list",
      "parameters": ["processed"]
    },
    "outputs": {
      "parameters": [
        { "name": "processed", "type": "array" }
      ]
    }
  }
}

The producer task emits a JSON array (e.g. ["data-1.csv", "data-2.csv"]) as an output parameter. The engine binds that array to itemsFrom at dispatch time and spawns one iteration per element.

Iteration Mode 3: Repeat Condition

Use repeatCondition for polling, retry-until, or counter-based loops where you don’t know the number of iterations up front. The engine evaluates the expression after each completed iteration and starts the next one only if it returns true.

{
  "loop": {
    "name": "repeat-loop",
    "repeatCondition": "loop_iter.index != 2",
    "maxIterations": 10,
    "body": "step",
    "outputs": {
      "parameters": [
        { "name": "done", "type": "bool" }
      ]
    }
  }
}

The expression context exposes loop_iter.index (the zero-based index of the iteration just completed). When loop_iter.index != 2 evaluates to false (i.e. after the third iteration, index 2), the loop stops. maxIterations acts as a hard cap — the loop terminates even if repeatCondition would continue beyond it.

Always set maxIterations when using repeatCondition. A condition that never becomes false would otherwise loop forever. The engine stops the loop and marks it failed once the iteration count reaches the cap.

Concurrency and Iteration Ordering

The concurrency field controls how many iterations the engine dispatches simultaneously:

Value	Behavior
`0`	Unlimited — all iterations dispatched immediately
`1`	Serial — one iteration at a time, in order
`N > 1`	At most N iterations running concurrently

For repeatCondition loops, iterations always run serially (concurrency is effectively 1) because the condition is evaluated against the result of the previous iteration.

Aggregation Strategies

After all iterations complete, the loop collects their outputs according to the aggregate configuration and exposes them as loop-level output parameters.

list (collect all)
last (default)
first

strategy: "list" gathers the specified output parameters from every iteration into a JSON array, sorted by iteration index:

{
  "loop": {
    "name": "score-loop",
    "items": ["item-1", "item-2", "item-3"],
    "concurrency": 3,
    "body": "score-item",
    "aggregate": {
      "strategy": "list",
      "parameters": ["score"]
    },
    "outputs": {
      "parameters": [
        { "name": "score", "type": "array" }
      ]
    }
  }
}

If each iteration produces score: 95, the loop output is score: [95, 95, 95]. A downstream task receives the full array and can compute aggregates (sum, average, etc.) over it.

strategy: "last" (or omitting aggregate entirely) uses the last iteration’s outputs as the loop-level outputs. This is appropriate when each iteration overwrites a shared resource or when only the final state matters.

{
  "aggregate": {
    "strategy": "last"
  }
}

strategy: "first" uses the first iteration’s outputs. This is useful for early-exit patterns where the first successful result is the one you want:

{
  "loop": {
    "name": "run-loop",
    "items": ["alpha", "beta", "gamma"],
    "concurrency": 1,
    "body": "process",
    "aggregate": {
      "strategy": "first",
      "parameters": ["result"]
    },
    "outputs": {
      "parameters": [
        { "name": "result", "type": "string" }
      ]
    }
  }
}

The parameters field inside aggregate lists which output parameter names to collect. Omit it to collect all declared output parameters.

Consuming Loop Outputs Downstream

Loop outputs are referenced from a parent DAG exactly like any other task output:

{
  "dag": {
    "name": "pipeline",
    "tasks": [
      { "name": "batch", "template": "score-loop" },
      {
        "name": "summarize",
        "template": "collect",
        "dependencies": ["batch"],
        "arguments": {
          "parameters": [
            {
              "name": "scores",
              "valueFrom": { "parameter": "tasks.batch.outputs.parameters.score" }
            }
          ]
        }
      }
    ]
  }
}

The summarize task receives scores as a JSON array — the aggregated result of all loop iterations.

Design Considerations

Use list aggregation for fan-in

When downstream tasks need to process all iteration results together, use strategy: "list" to collect outputs into an array that a single join task can consume.

Cap iterations defensively

Set maxIterations even on items-based loops. It prevents runaway execution if the items array is unexpectedly large from a dynamic source.

Tune concurrency to your workload

High concurrency is great for CPU-bound parallel tasks but can overwhelm downstream APIs or shared resources. Use concurrency: 1 for sequential processing and concurrency: N to throttle fan-out.

itemsFrom enables data-driven pipelines

Combine a producer task with itemsFrom to build fully data-driven loops where the number of iterations is determined by runtime data rather than the workflow definition.

Get Started

Core Concepts

Guides

Extension Points

Loop Templates: Iteration, Aggregation, and Concurrency

Loop Template Structure

Field reference

Iteration Mode 1: Static Items

Iteration Mode 2: Dynamic Items from a Task Output

Iteration Mode 3: Repeat Condition

Concurrency and Iteration Ordering

Aggregation Strategies

Consuming Loop Outputs Downstream

Design Considerations

Use list aggregation for fan-in

Cap iterations defensively

Tune concurrency to your workload

itemsFrom enables data-driven pipelines

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Extension Points

Documentation Index

​Loop Template Structure

​Field reference

​Iteration Mode 1: Static Items

​Iteration Mode 2: Dynamic Items from a Task Output

​Iteration Mode 3: Repeat Condition

​Concurrency and Iteration Ordering

​Aggregation Strategies

​Consuming Loop Outputs Downstream

​Design Considerations

Use list aggregation for fan-in

Cap iterations defensively

Tune concurrency to your workload

itemsFrom enables data-driven pipelines

Build docs developers (and LLMs) love

Loop Template Structure

Field reference

Iteration Mode 1: Static Items

Iteration Mode 2: Dynamic Items from a Task Output

Iteration Mode 3: Repeat Condition

Concurrency and Iteration Ordering

Aggregation Strategies

Consuming Loop Outputs Downstream

Design Considerations