OCI Sync

Pulls an OCI artifact (manifest + layers) from a registry and emits one event per file. Downstream tasks decide what to do with each file’s content — parse it, store it, transform it.

Works with any OCI-compliant registry: GHCR, ECR, GAR, ACR, Artifactory, Harbor, Docker Hub, Quay, self-hosted.

Each UTF-8 text event contains {path, content, digest, artifact_digest} on event.data. The shape mirrors Git Sync so the same downstream pipeline (buffer → diff → cache write) works with either source. Binary layers (or tar entries) that cannot be safely decoded as UTF-8 arrive as EventData::Bytes instead, with {path, digest, artifact_digest} moved to event.meta so downstream tasks such as object_store::write can still route by key.

Both oras push artifacts and Docker container images are supported. oras-style layers — identified by the presence of the org.opencontainers.image.title annotation — emit one event per layer using the annotation as the path. Docker image layers — layers without a title annotation — are merged in manifest order with overlay-fs semantics: later layers override earlier ones, .wh.<name> markers delete files from lower layers, and .wh..wh..opq markers hide entire subtrees. Only the surviving final state is emitted downstream.

Configuration

- oci_sync:
    name: pull_flows
    artifact: "{{env.OCI_FLOWS_ARTIFACT}}"
    credentials_path: /etc/flowgen/credentials/registry.json

Fields

Example: sync flows from a registry into the NATS KV cache

flow:
  name: oci_sync_flows
  tasks:
    - generate:
        name: trigger
        interval: "30s"

    - oci_sync:
        name: pull_repo
        artifact: "{{env.OCI_FLOWS_ARTIFACT}}"
        credentials_path: /etc/flowgen/credentials/registry.json

    - buffer:
        name: collect_layers
        size: 10000
        timeout: "5s"

    - script:
        name: write_cache_keys
        code: |
          let actions = [];
          for layer in event.data.batch {
              let parsed = parse_yaml(layer.content);
              let key = "flowgen.flows." + parsed.flow.name;
              actions.push(#{
                  action: "put",
                  key: key,
                  content: layer.content,
              });
          }
          actions

    - nats_kv_store:
        name: save_to_kv
        operation: put
        bucket: flowgen_system
        key: "{{event.data.key}}"
        credentials_path: /etc/nats/credentials.json
        url: "{{env.NATS_URL}}"

Output

Format: JSON for UTF-8 files, EventData::Bytes for binary files. Each file emitted produces one event.

For UTF-8 files, event.data contains:

For binary files, event.data carries the raw bytes and the same routing fields (path, digest, artifact_digest) land on event.meta instead. Downstream tasks match EventData::Bytes directly and read event.meta.path to know where the payload belongs.

Layer formats

The layer’s mediaType in the manifest dispatches the extractor:

Inside tar and tar+gzip layers, directories, symlinks, and hardlinks are skipped. Only regular files produce events. Docker whiteout markers (.wh.<name>, .wh..wh..opq) are consumed by the layer-merge pass rather than emitted downstream.

This means both push paths work:

• oras push {{env.OCI_FLOWS_ARTIFACT}} flow.yaml:application/yaml — one raw layer per file, path from the :mediaType annotation.
• docker push {{env.OCI_FLOWS_ARTIFACT}} (from a Dockerfile that COPYs YAML files across multiple stages) — layers are merged into a single overlay-fs snapshot; the final state emits one event per surviving file.

Bootstrap flows

Two end-to-end bootstrap flows reconcile an OCI artifact into the system cache. They tick on an interval, list existing cache entries, and emit one put per layer and one delete per orphaned key:

• examples/oci/system_sync_flows.yaml — keys each entry by flow.name parsed from the layer body so the filename is incidental. The reconciler reads from flowgen.flows.* and starts, stops, and hot-reloads flows accordingly.
• examples/oci/system_sync_resources.yaml — keys each entry by the layer’s relative path under flowgen.resources.*. The runtime ResourceLoader reads from the same keys when tasks reference resource: <path>. See Resources.

Both skip the rest of their pipeline when the artifact digest has not moved, so the only cost on a no-change tick is a manifest HEAD plus a list_keys round-trip.

Change detection

Each tick issues an HTTP HEAD against the artifact’s manifest URL to read Docker-Content-Digest from the response header. The digest is compared against the last successful pull, cached under flow.{flow_name}.oci_digest.{artifact} in the shared cache. On a match, the layer blobs are not fetched and the source emits only the upstream completion signal — one line per tick in the logs:

INFO flowgen_oci::sync::processor: OCI manifest digest unchanged since last pull, skipping layer fetch artifact=… digest=sha256:…

Works for both mutable tags (:prod) and immutable digests (@sha256:…) — the digest is authoritative in either case, so a re-tag of :prod to a new release still triggers a pull.

The cached digest is persisted only after every layer event was sent, so a mid-pull failure causes the next tick to re-emit the full batch.

Set force_pull: true to bypass the cache — use only to re-seed a downstream cache mutated out of band.