Buffer

Accumulates events into batches. Flushes when the batch reaches the configured size or the timeout expires.

Configuration

- buffer:
    name: batch
    size: 100
    timeout: "30s"

Fields

FieldTypeDefaultDescription
namestringrequiredTask name.
sizeintrequiredNumber of events per batch.
timeoutduration30sFlush timeout — sends the batch even if not full.
partition_keystringTemplate for partitioned buffering. Events with the same key are batched together.
depends_onlistUpstream task names.
retryobjectRetry configuration.

Example: Partitioned buffering

- buffer:
    name: batch_by_region
    size: 50
    timeout: "10s"
    partition_key: "{{event.data.region}}"

Events are grouped by region. Each partition flushes independently when it reaches 50 events or 10 seconds.