Skip to content

Open-source LLM gatewayApache-2.0single binary

One endpoint in front of every LLM provider.

Relay takes OpenAI- and Anthropic-shaped requests, checks them against a policy, draws a healthy key from your pool and streams the answer back. Failover, rate limits and per-request metering move out of your app and into the gateway.

Open the consoleQuickstart
$docker run -p 8080:8080 -p 8081:8081 wyolet/relay:standalone

Routing simulation

#0001 claude-code

200

POST /claude-code/v1/messages model=claude-sonnet-4-6

  1. authsk-wr-…7a2e → policy eng
  2. policygrants claude-sonnet-4-6 → anthropic
  3. shapeanthropic, bytes passed through
  4. drawanthropic/k1 (round-robin)
  5. 200streaming from anthropic/k1
  6. settleusage + cost recorded async
    Breakers

    01Drop-in

    Change the base URL. Keep the SDK.

    Relay serves the OpenAI and Anthropic wire formats and translates between them, so the client code you ship today keeps working against any model in your catalog.
    base URL
    the only line that changes
    api key
    a relay key (sk-wr-…), never a provider key
    model
    picks the upstream; the policy decides if it may
    stream
    passes through; bytes are copied when shapes match
    app.py
    import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.r.wyolet.com/openai/v1",
    api_key=os.environ["RELAY_API_KEY"],
)

# a Claude model through the OpenAI SDK; Relay translates both ways
resp = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "hello"}],
    stream=True,
)

    02Request path

    Five stops between your code and the provider.

    Postgres holds the config but is never read while serving. Every pod routes from a snapshot rebuilt on NOTIFY, so a change in the console reaches the fleet in about a second.

    1. 01 · auth

      Authenticate

      The relay key is hashed and looked up in an in-memory snapshot. No match, 401.

      sha256 · no DB read

    2. 02 · route

      Route

      The key's policy must grant the model. The model resolves to a host binding: wire adapter plus upstream model name.

      403 if not granted

    3. 03 · draw

      Reserve and draw

      Rate-limit budget is reserved and a healthy host key drawn from the pool. Open breakers are skipped; failover happens here.

      one Redis Lua call

    4. 04 · stream

      Forward and stream

      Same shape in and out: bytes pass through. Different shapes: each chunk is translated through Relay's canonical protocol.

      passthrough | translate

    5. 05 · settle

      Settle

      After the response ends, a detached goroutine commits usage, records key health and emits the usage event.

      off the response path

    No mid-stream failover

    Every retry across keys and hosts happens before the first byte reaches the caller. Once bytes flow the request is committed, and an upstream error is surfaced as-is, with its original status and Retry-After.

    03Spec sheet

    What ships in the binary.

    Everything below runs in the open-source build. The hosted console is the same code.
    Wire formats
    OpenAI Chat Completions, Anthropic Messages and OpenAI Responses in; any host out. Streaming included.
    cross-shape translation · byte passthrough
    Key pools
    Many host keys per host behind one relay key. Effective limits add up instead of stopping at one key's ceiling.
    prioritized · round-robin · least-recently-used
    Circuit breakers
    One breaker per key, classified by failure. Open keys are skipped and heal after cooldown; auth failures re-resolve the secret out of band.
    auth · rate limit short/long · 5xx · network
    Relay keys & policies
    Bearer tokens stored as sha256 hashes and scoped by policy: which models, which providers, how fast. Leak one and the damage stops at its limits.
    model allowlists · rate limits · revocable
    Provider secrets
    Stored AES-GCM encrypted, referenced from env, or fetched at runtime from a secret manager and held in memory only.
    AWS · Azure · GCP · Bitwarden · 1Password
    Metering
    Every request is recorded with token counts, timing and cost priced from an open catalog of 400+ models. Payload capture is opt-in.
    Postgres or ClickHouse
    Observability
    Prometheus metrics and structured JSON logs. Coding-agent sessions are attributed per session and subagent.
    /metrics · JSON logs
    Proxy mode
    Callers keep their own provider credential. Relay forwards byte-for-byte and adds the endpoint, limits and logging.
    off by default
    Two planes
    Inference and control listen on separate ports, so only the inference plane has to face the internet.
    :8080 inference · :8081 console + API

    Upstreams

    OpenAI · Anthropic · Google AI Studio · AWS Bedrock · Vertex AI · Azure OpenAI · Groq · Mistral · xAI · Ollama · OpenRouter · anything speaking either wire format. Open catalog

    04Clients

    Coding agents, on any model you route.

    Each agent gets its own base path with the discovery endpoints it probes for, so its model picker lists what your key's policy grants, whichever host serves it. Session, agent and subagent headers land on every usage event.
    ~/.claude/settings.json
    {
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.r.wyolet.com/claude-code",
    "ANTHROPIC_AUTH_TOKEN": "sk-wr-your-relay-key",
    "CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY": "1",
    "CLAUDE_CODE_GATEWAY_HINT_HEADERS": "1"
  }
}
    Base paths
    Claude Code/claude-code
    Codex CLI/codex/v1
    OpenCode/opencode/v1
    OpenAI SDK/openai/v1
    Anthropic SDK/anthropic

    Claude Code can also run in proxy mode: it keeps its own Claude login and Relay only observes. Setup guide

    05Deploy

    Use ours, or run it on your own metal.

    The hosted console and the self-hosted binary are the same code. Pick by where you want the data to live.

    Hosted

    r.wyolet.com

    Sign in, add a provider key, mint a relay key. Multi-user workspaces, usage and cost dashboards, nothing to operate.

    console
    r.wyolet.com
    inference
    api.r.wyolet.com
    Open the console

    Self-hosted

    One container, or the Helm chart

    The standalone image bundles the API, admin UI, database and a seeded model catalog. The chart runs its own Postgres, ClickHouse and Valkey. Nothing phones home.

    $docker run -p 8080:8080 -p 8081:8081 wyolet/relay:standalone

    :8080 inference · OpenAI + Anthropic shapes

    :8081 admin UI + control API, setup wizard

    QuickstartSource

    06FAQ

    Questions.

    How are provider keys stored?

    Stored keys are AES-GCM encrypted with RELAY_MASTER_KEY before they reach Postgres. Self-hosted, you can instead reference an environment variable or fetch from AWS, Azure or GCP Secret Manager, Bitwarden or 1Password, in which case the secret is held in memory only.

    What happens when a key is rate-limited or a provider is down?

    The key's breaker records the failure class (auth, short or long rate limit, 5xx, network) and opens. Relay draws the next healthy key in the pool before any byte reaches you. Open breakers heal after their cooldown; a rotated key gets a fresh breaker automatically.

    How much latency does it add?

    Routing reads an in-memory snapshot, rate limits cost one Redis round-trip, and bytes are copied rather than re-serialised when shapes match. Added latency stays in single-digit milliseconds at p50 under real load; scale out by adding pods.

    Can I call a Claude model with the OpenAI SDK?

    Yes. The model string selects the host; if the host speaks a different wire format, Relay translates the request and every streamed chunk. Anthropic-only features such as cache_control are not carried across shapes.

    Is the hosted version different from the open-source one?

    Same code. r.wyolet.com is our deployment of it with multi-user workspaces. Self-host when you want the data on your own infrastructure.