AI Completion

Run LLM completions from multiple providers within a flow. Supports custom OpenAI-compatible endpoints, tool calling via MCP servers, streaming, RAG via static context, and sandboxed agent execution.

Configuration

- ai_completion:
    name: analyze
    provider: google
    model: gemini-2.5-flash-lite
    credentials_path: /path/to/credentials.json
    prompt: "Analyze this data: {{event.data}}"

Fields

Providers

For providers not listed above (cloud-native services with non-api_key auth, on-prem model servers, multi-tenant routing), set provider: custom and point endpoint at any service speaking the OpenAI chat-completions wire format. The flowgen AI gateway (llm_proxy task) can itself act as that endpoint, letting you route, log, and rate-limit across upstream models from one place.

Credentials file format

For API-key providers, credentials file is a simple JSON:

{ "api_key": "sk-..." }

For vertexai, the JSON sets the project and region; authentication itself goes through Google Application Default Credentials (ADC) — the pod must have access to a service account via the GOOGLE_APPLICATION_CREDENTIALS env var or workload identity:

{ "project_id": "my-gcp-project", "region": "us-central1" }

Example: custom OpenAI-compatible endpoint

Point at any service speaking the OpenAI chat-completions wire format — an internal model server, a self-hosted gateway, or the flowgen AI gateway itself routing to one or more upstream models.

- ai_completion:
    name: internal_llm
    provider: custom
    model: my-internal-model
    endpoint: "https://llm-gateway.internal.example.com/v1"
    credentials_path: /etc/flowgen/credentials/internal_llm.json
    prompt: "{{event.data.question}}"

For endpoints without auth, omit credentials_path:

- ai_completion:
    name: unauth_endpoint
    provider: custom
    model: my-model
    endpoint: "http://llm.cluster.local/v1"
    prompt: "{{event.data}}"

Example: RAG with static context

static_context documents are injected into every completion. Mix inline strings and resource files.

- ai_completion:
    name: support_bot
    provider: openai
    model: gpt-4-turbo
    credentials_path: /etc/flowgen/credentials/openai.json
    system_prompt: "Answer using the provided context."
    prompt: "{{event.data.question}}"
    static_context:
      - resource: "docs/product_manual.md"
      - resource: "docs/pricing.md"
      - "Support contact: support@example.com"
    max_turns: 3

Example: MCP tool calling

The agent connects to MCP servers, discovers tools, and can call them during completion. Use flowgen’s own MCP server (mcp_tool task) or any external MCP-compatible server.

- ai_completion:
    name: agent
    provider: anthropic
    model: claude-3-5-sonnet-20241022
    credentials_path: /etc/flowgen/credentials/anthropic.json
    prompt: "{{event.data.task}}"
    max_turns: 5
    mcp_servers:
      - url: "http://localhost:3001/mcp"
      - url: "http://external-tools:8080/mcp"

Optional per-server auth uses the same credentials JSON format as http_request (bearer_auth or basic_auth):

    mcp_servers:
      - url: "https://tools.example.com/mcp"
        credentials_path: /etc/flowgen/credentials/mcp_tools.json

Tool-use passthrough

When tool_passthrough: true, the task bypasses rig’s agent loop and forwards caller-supplied tools and tool_choice (from the request body) directly to the upstream provider. Tool invocations returned by the model surface as tool_calls on the response instead of being executed in-process. The client — typically opencode, Claude Desktop, or any OpenAI-tool-use SDK — runs the tools locally and posts the results back as role: "tool" follow-up messages.

Use this mode when flowgen is the AI gateway for a tool-executing client. Leave it false for agent-mode flows that execute tools themselves via mcp_servers or inline definitions.

- llm_proxy:
    name: proxy

- ai_completion:
    name: forward
    provider: custom
    endpoint: "{{event.data.endpoint}}"
    model: "{{event.data.routed_model}}"
    credentials_path: "{{event.data.credentials_path}}"
    prompt: "{{event.data.prompt}}"
    system_prompt: "{{event.data.system_prompt}}"
    tool_passthrough: true
    stream: true

Passthrough is a no-op when the request omits tools: the flow follows the normal completion path so a single ai_completion task can serve both tool-using and plain chat clients without a second flow.

Works across every provider flowgen supports (OpenAI, Anthropic, Cohere, Gemini, Mistral, Groq, Together, xAI, OpenRouter, Perplexity, HuggingFace, VertexAI, and any OpenAI-compatible endpoint via provider: custom). Provider-native quirks in the tool schema — Anthropic’s tool_use block, Google’s parallel-call ordering — are handled by the underlying rig client without any per-provider config here.

Sandbox

Optional sandboxing for MCP tool execution via nsjail. Rhai scripts do not need sandboxing (safe by design).

- ai_completion:
    name: agent
    provider: google
    model: gemini-2.5-flash-lite
    prompt: "{{event.data}}"
    sandbox:
      memory_limit_mb: 512
      time_limit_seconds: 30
      max_pids: 10
      allow_network: false

Output

Format: JSON

In stream: true mode, each chunk is emitted as a separate event with partial text content.