# OffRail — Full Documentation

> OffRail is an OpenAI-compatible API gateway that routes, manages, and analyzes LLM requests across 40+ providers (OpenAI, Anthropic, Google, and more) through a single unified API. Switch providers without changing code, manage API keys centrally, track usage and cost, add caching and guardrails, and self-host or use the managed cloud.

API base URL: https://api.offrail.ai/v1 · Docs: https://docs.offrail.ai · Site: https://offrail.ai

This file concatenates the full text of every documentation page below.

# Introduction to OffRail
URL: https://docs.offrail.ai/


OffRail is a unified API gateway that sits between your applications and LLM providers like OpenAI, Anthropic, Google AI Studio, and more. It provides a unified, OpenAI-compatible API interface with built-in cost tracking, caching, and intelligent routing.

## How it works [#how-it-works]

Point your existing SDK at `https://api.offrail.ai/v1` and authenticate with an OffRail API key — no code rewrites. The gateway routes each request to the right provider, tracks tokens and cost per model, provider, project, and API key, and fails over to a healthy provider when one errors. Pay per-token with prepaid credits at provider list rates, or bring your own provider keys (BYOK) for free.

## Why use a gateway? [#why-use-a-gateway]

* **One integration** — switch models or providers by changing a model string, not your code.
* **Cost visibility** — usage analytics and cost breakdowns across every provider in one dashboard.
* **Reliability** — automatic provider failover and response caching in front of every request.
* **No lock-in** — OpenAI-compatible end to end.

## Features [#features]

All features are documented under https://docs.offrail.ai/features; each feature page is included in full in this file.

## AI Tooling [#ai-tooling]

OffRail is built to work seamlessly with AI agents and development tools.

AI tooling: https://docs.offrail.ai/llms.txt (docs index for LLMs), https://docs.offrail.ai/llms-full.txt (this file), and https://docs.offrail.ai/developers/mcp (MCP server).

## Next Steps [#next-steps]

* [**Quickstart**](https://docs.offrail.ai/quick-start) — Get up and running in minutes
* [**Overview**](https://docs.offrail.ai/overview) — Learn more about what OffRail offers
* [**Self-Host**](https://docs.offrail.ai/self-host) — Deploy on your own infrastructure


# Overview
URL: https://docs.offrail.ai/overview


OffRail is a unified API gateway for Large Language Models (LLMs). It acts as a middleware between your applications and various LLM providers, allowing you to:

* Route requests to multiple LLM providers (OpenAI, Anthropic, Google AI Studio, and others)
* Manage API keys for different providers in one place
* Track token usage and costs across all your LLM interactions
* Analyze performance metrics to optimize your LLM usage

## Analyzing Your LLM Requests [#analyzing-your-llm-requests]

OffRail provides detailed insights into your LLM usage:

* **Usage Metrics**: Track the number of requests, tokens used, and response times
* **Cost Analysis**: Monitor spending across different models and providers
* **Performance Tracking**: Identify patterns and optimize your prompts based on actual usage data
* **Breakdown by Model**: Compare different models' performance and cost-effectiveness

All this data is automatically collected and presented in an intuitive dashboard, helping you make informed decisions about your LLM strategy.

## Getting Started [#getting-started]

Using OffRail is simple. Just swap out your current LLM provider URL with the OffRail API endpoint:

```bash
curl -X POST https://api.offrail.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -d '{
  "model": "gpt-4o",
  "messages": [
    {"role": "user", "content": "Hello, how are you?"}
  ]
}'
```

OffRail maintains compatibility with the OpenAI API format, making migration seamless.

## Hosted vs. Self-Hosted [#hosted-vs-self-hosted]

You can use OffRail in two ways:

* **Hosted Version**: For immediate use without setup, visit [offrail.ai](https://offrail.ai) to create an account and get an API key.
* **Self-Hosted**: Deploy OffRail on your own infrastructure for complete control over your data and configuration.

The self-hosted version offers additional customization options and ensures your LLM traffic never leaves your infrastructure if desired.


# Quickstart
URL: https://docs.offrail.ai/quick-start


Welcome to **OffRail**—a single drop‑in endpoint that lets you call today’s best large‑language models while keeping **your existing code** and development workflow intact.

> **TL;DR** — Point your HTTP requests to `https://api.offrail.ai/v1/…`, supply your `OFFRAIL_API_KEY`, and you’re done.

***

## 1 · Get an API key [#1--get-an-api-key]

1. Sign in to the dashboard.
2. Create a new Project → *Copy the key*.
3. Export it in your shell (or a `.env` file):

```bash
export OFFRAIL_API_KEY="orl_XXXXXXXXXXXXXXXX"
```

***

## 2 · Pick your language [#2--pick-your-language]

<Tabs
  groupId="language"
  items="[
'cURL',
'TypeScript',
'React',
'Next.js',
'Python',
'Java',
'Rust',
'Go',
'PHP',
'Ruby']"
>
  <Tab value="cURL">
    ```bash
    curl -X POST https://api.offrail.ai/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $OFFRAIL_API_KEY" \
      -d '{
        "model": "gpt-4o",
        "messages": [
          {"role": "user", "content": "Hello, how are you?"}
        ]
      }'
    ```
  </Tab>

  <Tab value="TypeScript">
    ```typescript
    const response = await fetch("https://api.offrail.ai/v1/chat/completions", {
    	method: "POST",
    	headers: {
    		"Content-Type": "application/json",
    		Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
    	},
    	body: JSON.stringify({
    		model: "gpt-4o",
    		messages: [{ role: "user", content: "Hello, how are you?" }],
    	}),
    });

    if (!response.ok) {
    	throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();
    console.log(data.choices[0].message.content);
    ```
  </Tab>

  <Tab value="React">
    ```tsx
    import { useState } from "react";

    function ChatComponent() {
    	const [response, setResponse] = useState("");
    	const [loading, setLoading] = useState(false);

    	const sendMessage = async () => {
    		setLoading(true);
    		try {
    			const res = await fetch("https://api.offrail.ai/v1/chat/completions", {
    				method: "POST",
    				headers: {
    					"Content-Type": "application/json",
    					Authorization: `Bearer ${process.env.REACT_APP_OFFRAIL_API_KEY}`,
    				},
    				body: JSON.stringify({
    					model: "gpt-4o",
    					messages: [{ role: "user", content: "Hello, how are you?" }],
    				}),
    			});

    			if (!res.ok) {
    				throw new Error(`HTTP error! status: ${res.status}`);
    			}

    			const data = await res.json();
    			setResponse(data.choices[0].message.content);
    		} catch (error) {
    			console.error("Error:", error);
    		} finally {
    			setLoading(false);
    		}
    	};

    	return (
    		<div>
    			<button onClick={sendMessage} disabled={loading}>
    				{loading ? "Sending..." : "Send Message"}
    			</button>
    			{response && <p>{response}</p>}
    		</div>
    	);
    }

    export default ChatComponent;
    ```
  </Tab>

  <Tab value="Next.js">
    ```typescript
    // app/api/chat/route.ts
    import { NextRequest, NextResponse } from "next/server";

    export async function POST(request: NextRequest) {
    	const { message } = await request.json();

    	const response = await fetch("https://api.offrail.ai/v1/chat/completions", {
    		method: "POST",
    		headers: {
    			"Content-Type": "application/json",
    			Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
    		},
    		body: JSON.stringify({
    			model: "gpt-4o",
    			messages: [{ role: "user", content: message }],
    		}),
    	});

    	if (!response.ok) {
    		return NextResponse.json(
    			{ error: "Failed to get response" },
    			{ status: response.status },
    		);
    	}

    	const data = await response.json();

    	return NextResponse.json({
    		message: data.choices[0].message.content,
    	});
    }

    // Usage in component:
    // const response = await fetch('/api/chat', {
    //   method: 'POST',
    //   headers: { 'Content-Type': 'application/json' },
    //   body: JSON.stringify({ message: 'Hello, how are you?' })
    // });
    ```
  </Tab>

  <Tab value="Python">
    ```python
    import requests
    import os

    response = requests.post(
        'https://api.offrail.ai/v1/chat/completions',
        headers={
            'Content-Type': 'application/json',
            'Authorization': f'Bearer {os.getenv("OFFRAIL_API_KEY")}'
        },
        json={
            'model': 'gpt-4o',
            'messages': [
                {'role': 'user', 'content': 'Hello, how are you?'}
            ]
        }
    )

    response.raise_for_status()
    print(response.json()['choices'][0]['message']['content'])
    ```
  </Tab>

  <Tab value="Java">
    ```java
    import java.net.http.HttpClient;
    import java.net.http.HttpRequest;
    import java.net.http.HttpResponse;
    import java.net.URI;

    String apiKey = System.getenv("OFFRAIL_API_KEY");
    String requestBody = """
    {
      "model": "gpt-4o",
      "messages": [
        {"role": "user", "content": "Hello, how are you?"}
      ]
    }
    """;

    HttpRequest request = HttpRequest.newBuilder()
        .uri(URI.create("https://api.offrail.ai/v1/chat/completions"))
        .header("Content-Type", "application/json")
        .header("Authorization", "Bearer " + apiKey)
        .POST(HttpRequest.BodyPublishers.ofString(requestBody))
        .build();

    HttpResponse<String> response = HttpClient.newHttpClient()
        .send(request, HttpResponse.BodyHandlers.ofString());

    System.out.println(response.body());
    ```
  </Tab>

  <Tab value="Rust">
    ```rust
    use reqwest::Client;
    use serde_json::json;
    use std::env;

    #[tokio::main]
    async fn main() -> Result<(), Box<dyn std::error::Error>> {
        let client = Client::new();
        let api_key = env::var("OFFRAIL_API_KEY")?;

        let response = client
            .post("https://api.offrail.ai/v1/chat/completions")
            .header("Content-Type", "application/json")
            .header("Authorization", format!("Bearer {}", api_key))
            .json(&json!({
                "model": "gpt-4o",
                "messages": [
                    {"role": "user", "content": "Hello, how are you?"}
                ]
            }))
            .send()
            .await?;

        let result: serde_json::Value = response.json().await?;
        println!("{}", result["choices"][0]["message"]["content"]);
        Ok(())
    }
    ```
  </Tab>

  <Tab value="Go">
    ```go
    package main

    import (
        "bytes"
        "encoding/json"
        "fmt"
        "net/http"
        "os"
    )

    type ChatRequest struct {
        Model    string    `json:"model"`
        Messages []Message `json:"messages"`
    }

    type Message struct {
        Role    string `json:"role"`
        Content string `json:"content"`
    }

    func main() {
        apiKey := os.Getenv("OFFRAIL_API_KEY")

        requestBody := ChatRequest{
            Model:    "gpt-4o",
            Messages: []Message{{Role: "user", Content: "Hello, how are you?"}},
        }

        jsonData, _ := json.Marshal(requestBody)

        req, _ := http.NewRequest("POST", "https://api.offrail.ai/v1/chat/completions", bytes.NewBuffer(jsonData))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Authorization", "Bearer "+apiKey)

        client := &http.Client{}
        resp, _ := client.Do(req)
        defer resp.Body.Close()

        fmt.Println("Response received")
    }
    ```
  </Tab>

  <Tab value="PHP">
    ```php
    <?php
    $apiKey = $_ENV['OFFRAIL_API_KEY'];

    $data = [
        'model' => 'gpt-4o',
        'messages' => [
            ['role' => 'user', 'content' => 'Hello, how are you?']
        ]
    ];

    $options = [
        'http' => [
            'header' => [
                'Content-Type: application/json',
                'Authorization: Bearer ' . $apiKey
            ],
            'method' => 'POST',
            'content' => json_encode($data)
        ]
    ];

    $context = stream_context_create($options);
    $response = file_get_contents(
        'https://api.offrail.ai/v1/chat/completions',
        false,
        $context
    );

    if ($response === FALSE) {
        throw new Exception('Request failed');
    }

    $result = json_decode($response, true);
    echo $result['choices'][0]['message']['content'];
    ?>
    ```
  </Tab>

  <Tab value="Ruby">
    ```ruby
    require 'net/http'
    require 'json'
    require 'uri'

    uri = URI('https://api.offrail.ai/v1/chat/completions')
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true

    request = Net::HTTP::Post.new(uri)
    request['Content-Type'] = 'application/json'
    request['Authorization'] = "Bearer #{ENV['OFFRAIL_API_KEY']}"

    request.body = {
      model: 'gpt-4o',
      messages: [
        { role: 'user', content: 'Hello, how are you?' }
      ]
    }.to_json

    response = http.request(request)

    if response.code != '200'
      raise "HTTP Error: #{response.code}"
    end

    result = JSON.parse(response.body)
    puts result['choices'][0]['message']['content']
    ```
  </Tab>
</Tabs>

***

## 3 · SDK integrations [#3--sdk-integrations]

```ts title="ai-sdk.ts"
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { text } = await generateText({
	model: offrail("gpt-4o"),
	prompt: "Write a vegetarian lasagna recipe for 4 people.",
});
```

```ts title="openai-sdk.ts"
import OpenAI from "openai";

const openai = new OpenAI({
	baseURL: "https://api.offrail.ai/v1",
	apiKey: process.env.OFFRAIL_API_KEY,
});

const completion = await openai.chat.completions.create({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Hello, how are you?" }],
});

console.log(completion.choices[0].message.content);
```

***

## 4 · Going further [#4--going-further]

* **Streaming**: pass `stream: true` to any request—Gateway will proxy the event stream unchanged.
* **Monitoring**: Every call appears in the dashboard with latency, cost & provider breakdown.

***

## 5 · FAQ [#5--faq]

<Accordions type="single">
  <Accordion title="Which models are supported?">
    See the [Models page](https://offrail.ai/models).
  </Accordion>

  <Accordion title="What makes your service different from OpenRouter?">
    <p>
      Unlike OpenRouter, we offer:
    </p>

    <ul className="list-disc pl-6 mt-2 space-y-1">
      <li>
        Full self-hosting capabilities, giving you complete control over your
        infrastructure
      </li>

      <li>
        Enhanced analytics with deeper insights into your model usage and
        performance
      </li>

      <li>
        No fees when using your own provider keys, maximizing cost efficiency
      </li>

      <li>
        Greater flexibility and customization options for enterprise deployments
      </li>
    </ul>
  </Accordion>

  <Accordion title="How much do you charge for your services?">
    Our pricing structure is designed to be flexible and cost-effective: See the
    [Pricing section](https://offrail.ai#pricing).
  </Accordion>
</Accordions>

***

## 6 · Next steps [#6--next-steps]

* Read [Self host docs](https://docs.offrail.ai/self-host) guide.
* Reach us at [contact@offrail.ai](mailto:contact@offrail.ai) for help or feature requests.

Happy building! ✨


# Health check
URL: https://docs.offrail.ai/health


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create speech
URL: https://docs.offrail.ai/v1_audio_speech


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create transcription
URL: https://docs.offrail.ai/v1_audio_transcriptions


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Chat Completions
URL: https://docs.offrail.ai/v1_chat_completions


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Embeddings
URL: https://docs.offrail.ai/v1_embeddings


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Edit image
URL: https://docs.offrail.ai/v1_images_edits


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create image
URL: https://docs.offrail.ai/v1_images_generations


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Retrieve key status
URL: https://docs.offrail.ai/v1_key_retrieve


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Anthropic Messages
URL: https://docs.offrail.ai/v1_messages


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Models
URL: https://docs.offrail.ai/v1_models


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Moderations
URL: https://docs.offrail.ai/v1_moderations


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# OCR
URL: https://docs.offrail.ai/v1_ocr


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Rerank
URL: https://docs.offrail.ai/v1_rerank


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Video content
URL: https://docs.offrail.ai/v1_videos_content


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Create video
URL: https://docs.offrail.ai/v1_videos_create


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Video log content
URL: https://docs.offrail.ai/v1_videos_log_content


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# Retrieve video
URL: https://docs.offrail.ai/v1_videos_retrieve


{/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */}



# AI SDK Gateway protocol
URL: https://docs.offrail.ai/developers/ai-sdk-gateway-protocol


When you pass a bare model string to the AI SDK — `streamText({ model: "anthropic/claude-sonnet-5" })` — the SDK resolves it through its **default provider**, `@ai-sdk/gateway`. That provider does not speak the OpenAI Chat Completions format: it has its own wire protocol, `LanguageModelV*CallOptions` in and `LanguageModelV*` parts out.

OffRail implements that protocol, so an app written against the Vercel AI Gateway runs here with **no code change** — only its base URL and API key are repointed.

## Repoint the default provider [#repoint-the-default-provider]

```ts
import { createGateway } from "@ai-sdk/gateway";

globalThis.AI_SDK_DEFAULT_PROVIDER = createGateway({
	baseURL: "https://api.offrail.ai/v4/ai",
	apiKey: process.env.OFFRAIL_API_KEY,
});
```

Put this wherever your app runs before its first model call — a Next.js [`instrumentation.ts`](https://nextjs.org/docs/app/guides/instrumentation), a server entrypoint, or a platform-injected preamble. Every bare model string in the app then resolves through OffRail.

`@ai-sdk/gateway` is already a transitive dependency of `ai`, so there is nothing extra to install.

<Callout type="info">
  `@ai-sdk/gateway` reads its API key from `AI_GATEWAY_API_KEY` but has **no**
  environment variable for the base URL — it is a constructor option only. That
  is why repointing takes this one line rather than an env var.
</Callout>

Or construct the provider explicitly and pass it per call:

```ts
import { createGateway } from "@ai-sdk/gateway";
import { streamText } from "ai";

const gateway = createGateway({
	baseURL: "https://api.offrail.ai/v4/ai",
	apiKey: process.env.OFFRAIL_API_KEY,
});

const result = streamText({
	model: gateway("anthropic/claude-sonnet-5"),
	prompt: "Hello!",
});
```

## Base URL per AI SDK version [#base-url-per-ai-sdk-version]

The protocol carries the language-model specification version in a request header, and every prefix below serves the same surface — pick the one matching the `@ai-sdk/gateway` your app has, so the default path stays intact:

| AI SDK | Spec version | Base URL                       |
| ------ | ------------ | ------------------------------ |
| 5      | 2            | `https://api.offrail.ai/v1/ai` |
| 6      | 3            | `https://api.offrail.ai/v3/ai` |
| 7      | 4            | `https://api.offrail.ai/v4/ai` |

## Model IDs [#model-ids]

Model IDs use the provider-pinned `provider/model` form (`anthropic/claude-sonnet-5`, `openai/gpt-4o`) — the same convention AI Gateway IDs use, so existing model strings resolve unchanged.

OffRail's smart-routing IDs work here too: pass a bare model ID (`gpt-4o`) to let the gateway pick a provider, or `auto` to let it pick the model. Those are not returned by `getAvailableModels()` because they do not name one provider, but they are accepted.

## Listing models [#listing-models]

```ts
const { models } = await gateway.getAvailableModels();
```

Returns one entry per active provider mapping, with pricing and an AI SDK `specification` block. This is what backs a model picker built on `GatewayModel[]`.

## Credits [#credits]

```ts
const { balance, totalUsed } = await gateway.getCredits();
```

`balance` is the organization's remaining credit balance and `totalUsed` its lifetime credits spend.

## Gateway-only options [#gateway-only-options]

Features that have no field in the AI SDK call options — reasoning effort, service tier, routing strategy, prompt cache keys, plugins — are set through the `offrail` provider options namespace, which is passed onto the underlying request:

```ts
const result = streamText({
	model: gateway("openai/gpt-5.6-terra"),
	prompt: "Hello!",
	providerOptions: {
		offrail: {
			reasoning_effort: "high",
			routing: "price",
		},
	},
});
```

Any field the [chat completions API](https://docs.offrail.ai/api-reference) accepts works here, except `model`, `messages` and `stream`, which this surface owns.

## Web search [#web-search]

The provider-native web search tools serialize to provider-defined tools, and the gateway maps them onto its [native web search](https://docs.offrail.ai/features/web-search):

```ts
import { openai } from "@ai-sdk/openai";

const result = streamText({
	model: gateway("openai/gpt-4o"),
	prompt: "What happened in the news today?",
	tools: { web_search: openai.tools.webSearch() },
});
```

Recognised tools: `openai.web_search`, `openai.web_search_preview`, `anthropic.web_search_20250305`, `anthropic.web_search_20260209`, `google.google_search`. Search results come back as `source-url` message parts plus a provider-executed tool call under the name you bound the tool to, so the AI SDK's sources UI works unchanged.

A provider-defined tool the gateway cannot map is reported as an `unsupported-tool` warning on the result rather than failing the request.

## What is not covered [#what-is-not-covered]

This surface implements language models. Embeddings, images, video, speech, transcription, reranking and realtime are served by the [OpenAI-compatible endpoints](https://docs.offrail.ai/api-reference) — use [`@ai-sdk/openai`](https://docs.offrail.ai/developers/ai-sdk) for those.

These call options have no chat completions equivalent and are reported as `unsupported` warnings: `stopSequences`, `seed`, `topK`.


# Image Generation with the AI SDK
URL: https://docs.offrail.ai/developers/ai-sdk-images


The `@ai-sdk/openai` package supports image generation both through the AI SDK's dedicated `generateImage` function and through chat-based image models that stream images as part of a conversation.

## generateImage [#generateimage]

Use `offrail.image()` to get an image model:

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { generateImage } from "ai";
import { writeFileSync } from "fs";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const result = await generateImage({
	model: offrail.image("gemini-3-pro-image-preview"),
	prompt:
		"A cozy cabin in a snowy mountain landscape at night with aurora borealis",
	size: "1024x1024",
	n: 1,
	// aspectRatio and quality are model-specific — only some providers honor them.
	// aspectRatio works on Gemini image models; OpenAI gpt-image-2 ignores it
	// (use a literal WxH `size` instead).
	aspectRatio: "16:9",
	// quality works on OpenAI gpt-image-2 ("low" | "medium" | "high" | "auto").
	// The AI SDK only forwards it through providerOptions.
	providerOptions: {
		offrail: { quality: "high" },
	},
});

result.images.forEach((image, i) => {
	const buf = Buffer.from(image.base64, "base64");
	writeFileSync(`image-${i}.png`, buf);
});
```

<Callout>
  Which sizes, aspect ratios, and quality settings a model accepts depends on
  the model — see [Image Generation](https://docs.offrail.ai/features/image-generation) for the full
  parameter reference and per-model behavior.
</Callout>

## Chat-based image models [#chat-based-image-models]

Multimodal models like `gemini-3-pro-image-preview` can return images inside a chat conversation. Use `offrail.chat()` with `streamText` in a route handler:

```typescript
// app/api/chat/route.ts
import { createOpenAI } from "@ai-sdk/openai";
import { convertToModelMessages, streamText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = streamText({
		model: offrail.chat("gemini-3-pro-image-preview"),
		messages: convertToModelMessages(messages),
	});

	return result.toUIMessageStreamResponse();
}
```

On the client, image parts arrive as message file parts that you can render with the AI Elements `Image` component or a plain `<img>` tag with a data URL. See [Image Generation](https://docs.offrail.ai/features/image-generation) for the complete `useChat` frontend example.

## Video and audio [#video-and-audio]

The AI SDK does not yet cover the gateway's video and speech endpoints — call them over REST instead:

* [Video Generation](https://docs.offrail.ai/features/video-generation) — `POST /v1/videos` (async jobs with optional signed callbacks)
* [Speech Generation](https://docs.offrail.ai/features/speech-generation) — `POST /v1/audio/speech`
* [Transcription](https://docs.offrail.ai/features/transcription) — `POST /v1/audio/transcriptions`


# Using the AI SDK
URL: https://docs.offrail.ai/developers/ai-sdk


OffRail speaks the OpenAI wire format, so the [Vercel AI SDK](https://ai-sdk.dev) reaches it through [`@ai-sdk/openai`](https://ai-sdk.dev/providers/ai-sdk-providers/openai) with no adapter of its own. Point that provider at the gateway base URL and one instance with one API key reaches every model in the catalog.

## Install [#install]

```bash
pnpm add ai @ai-sdk/openai
```

Set your API key (create one from the [dashboard](https://offrail.ai/dashboard)):

```bash
export OFFRAIL_API_KEY=orl_your_key_here
```

## Generate text [#generate-text]

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { text } = await generateText({
	model: offrail("openai/gpt-4o"),
	prompt: "Hello!",
});
```

Switching models is a one-line change — the same provider serves every model:

```typescript
const { text } = await generateText({
	model: offrail("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Hello!",
});
```

## Model ID formats [#model-id-formats]

OffRail supports two model ID formats:

* **Canonical model IDs** (`gpt-4o`) — smart routing picks the best provider based on uptime, throughput, price, and latency
* **Provider-prefixed IDs** (`openai/gpt-4o`) — routes to a specific provider with automatic failover if uptime drops below 90%

See the [routing documentation](https://docs.offrail.ai/features/routing) for details and the [models page](https://offrail.ai/models) for the full catalog.

## Stream responses [#stream-responses]

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { textStream } = await streamText({
	model: offrail("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Write a poem about coding",
});

for await (const text of textStream) {
	process.stdout.write(text);
}
```

## Next.js route handler [#nextjs-route-handler]

```typescript
// app/api/chat/route.ts
import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = await streamText({
		model: offrail("openai/gpt-4o"),
		messages,
	});

	return result.toDataStreamResponse();
}
```

## Tool calling [#tool-calling]

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { generateText, tool } from "ai";
import { z } from "zod";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { text, toolResults } = await generateText({
	model: offrail("openai/gpt-4o"),
	tools: {
		weather: tool({
			description: "Get the weather for a location",
			parameters: z.object({
				location: z.string(),
			}),
			execute: async ({ location }) => {
				return { temperature: 72, condition: "sunny" };
			},
		}),
	},
	prompt: "What's the weather in San Francisco?",
});
```

## Without the provider package [#without-the-provider-package]

If you prefer not to add a dependency, point `@ai-sdk/openai` at the gateway with a custom base URL:

```typescript
import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const offrail = createOpenAI({
	baseURL: "https://api.offrail.ai/v1",
	apiKey: process.env.OFFRAIL_API_KEY,
});

const { text } = await generateText({
	model: offrail("openai/gpt-4o"),
	prompt: "Hello!",
});
```

<Callout>
  Every request made through the AI SDK shows up in your
  [Activity](https://docs.offrail.ai/learn/activity) and [Usage & Metrics](https://docs.offrail.ai/learn/usage-metrics)
  dashboards like any other gateway request — with per-request cost, tokens, and
  latency.
</Callout>

## Next steps [#next-steps]

* [Image generation with the AI SDK](https://docs.offrail.ai/developers/ai-sdk-images)
* [Migrate from Vercel AI Gateway](https://docs.offrail.ai/migrations/vercel-ai-gateway)
* [Reasoning support](https://docs.offrail.ai/features/reasoning) and [caching](https://docs.offrail.ai/features/caching)


# Overview
URL: https://docs.offrail.ai/developers


This section is for developers building applications on top of OffRail — with the MCP server and the [Vercel AI SDK](https://ai-sdk.dev) through [`@ai-sdk/openai`](https://ai-sdk.dev/providers/ai-sdk-providers/openai).

## Guides [#guides]

* [**Model Context Protocol (MCP)**](https://docs.offrail.ai/developers/mcp) — Use OffRail as an MCP server from Claude Code, Cursor, and other MCP clients
* [**Using the AI SDK**](https://docs.offrail.ai/developers/ai-sdk) — Install the provider, generate and stream text, call tools, and wire up Next.js routes
* [**Image Generation with the AI SDK**](https://docs.offrail.ai/developers/ai-sdk-images) — Generate images with `generateImage` and stream image output through chat

## Why the AI SDK [#why-the-ai-sdk]

The AI SDK gives you one TypeScript interface for text generation, streaming, tool calling, and image generation. Combined with OffRail, a single provider instance and one API key reach every model in the catalog — see the [models page](https://offrail.ai/models) for what's available.

## Other ways to integrate [#other-ways-to-integrate]

If you're not using the AI SDK:

* Use any OpenAI-compatible SDK against `https://api.offrail.ai/v1` — see the [Quickstart](https://docs.offrail.ai/quick-start)
* Use the Anthropic SDK against the [Anthropic-compatible endpoint](https://docs.offrail.ai/features/anthropic-endpoint)
* Call the REST API directly — see the API reference in the sidebar


# Model Context Protocol (MCP)
URL: https://docs.offrail.ai/developers/mcp


OffRail provides a Model Context Protocol (MCP) server that enables AI assistants like Claude Code to access multiple LLM providers through a unified interface. This allows you to use any model from OpenAI, Anthropic, Google, and more directly from your AI coding assistant.

## What is MCP? [#what-is-mcp]

The Model Context Protocol (MCP) is an open standard that allows AI assistants to connect with external tools and data sources. OffRail's MCP server exposes tools for:

* **Chat completions** - Send messages to any supported LLM
* **Image generation** - Generate images using models like Qwen Image
* **Nano Banana image generation** - Generate images with Gemini 3 Pro Image Preview and optionally save to disk
* **Model discovery** - List available models with capabilities and pricing

## Available Tools [#available-tools]

### `chat` [#chat]

Send a message to any LLM and get a response.

**Parameters:**

* `model` (string) - The model to use (e.g., `"gpt-4o"`, `"claude-sonnet-4-20250514"`)
* `messages` (array) - Array of messages with `role` and `content`
* `temperature` (number, optional) - Sampling temperature (0-2)
* `max_tokens` (number, optional) - Maximum tokens to generate

**Example:**

```json
{
	"model": "gpt-4o",
	"messages": [{ "role": "user", "content": "Explain quantum computing" }],
	"temperature": 0.7
}
```

### `generate-image` [#generate-image]

Generate images from text prompts using AI image models.

**Parameters:**

* `prompt` (string) - Text description of the image to generate
* `model` (string, optional) - Image model (default: `"qwen-image-plus"`)
* `size` (string, optional) - Image size (default: `"1024x1024"`)
* `n` (number, optional) - Number of images (1-4, default: 1)

**Example:**

```json
{
	"prompt": "A serene mountain landscape at sunset",
	"model": "qwen-image-max",
	"size": "1024x1024"
}
```

### `generate-nano-banana` [#generate-nano-banana]

Generate an image using Gemini 3 Pro Image Preview ("Nano Banana"). Returns an inline image preview, and optionally saves the image to disk when the server is configured with an upload directory.

**Parameters:**

* `prompt` (string) - Text description of the image to generate
* `filename` (string, optional) - Filename for the saved image, no path separators allowed (default: `nano-banana-{timestamp}.png`)
* `aspect_ratio` (string, optional) - Aspect ratio: `"1:1"`, `"16:9"`, `"4:3"`, or `"5:4"`

**Example:**

```json
{
	"prompt": "A pixel-art cat sitting on a rainbow",
	"filename": "hero-image.png",
	"aspect_ratio": "16:9"
}
```

<Callout type="info">
  **Saving images to disk** requires the `UPLOAD_DIR` environment variable to be
  set on the MCP server. When set, images are saved to that directory. Without
  it, images are returned inline only — no files are written to disk. See
  [Enabling local image saving](#enabling-local-image-saving) for setup
  instructions.
</Callout>

### `list-models` [#list-models]

List available LLM models with capabilities and pricing.

**Parameters:**

* `include_deactivated` (boolean, optional) - Include deactivated models
* `exclude_deprecated` (boolean, optional) - Exclude deprecated models
* `limit` (number, optional) - Maximum models to return (default: 20)
* `family` (string, optional) - Filter by family (e.g., `"openai"`, `"anthropic"`)

### `list-image-models` [#list-image-models]

List all available image generation models.

**Example output:**

```
# Image Generation Models

## Qwen Image Plus
- **Model ID:** `qwen-image-plus`
- **Description:** Text-to-image with excellent text rendering
- **Price:** $0.03 per request

## Qwen Image Max
- **Model ID:** `qwen-image-max`
- **Description:** Highest quality text-to-image
- **Price:** $0.075 per request
```

## Setup [#setup]

<Tabs items="[&#x22;Claude Code&#x22;, &#x22;Codex&#x22;, &#x22;Cursor&#x22;, &#x22;Other MCP Clients&#x22;]">
  <Tab value="Claude Code">
    <Steps>
      <Step>
        ### Get Your API Key [#get-your-api-key]

        1. Log in to your [OffRail dashboard](https://offrail.ai/dashboard)
        2. Navigate to **API Keys** section
        3. Create a new API key and copy it
      </Step>

      <Step>
        ### Configure Claude Code [#configure-claude-code]

        Run the following command in your terminal:

        ```bash
        claude mcp add --transport http --scope user offrail https://api.offrail.ai/mcp \
          --header "Authorization: Bearer your-api-key-here"
        ```

        <Callout type="info">
          **Alternative: Manual configuration**

          You can also add the MCP server manually by editing `~/.claude.json` (user scope) or `.mcp.json` in your project root (project scope):

          ```json
          {
            "mcpServers": {
              "offrail": {
                "url": "https://api.offrail.ai/mcp",
                "headers": {
                  "Authorization": "Bearer your-api-key-here"
                }
              }
            }
          }
          ```

          Restart Claude Code after manual configuration changes.
        </Callout>
      </Step>

      <Step>
        ### Test the Integration [#test-the-integration]

        Try using the tools in Claude Code:

        * "Use the chat tool to ask GPT-4o about TypeScript best practices"
        * "Generate an image of a futuristic city using the generate-image tool"
        * "Use generate-nano-banana to create a hero image for my landing page"
        * "List all available models from Anthropic"
      </Step>
    </Steps>
  </Tab>

  <Tab value="Codex">
    <Steps>
      <Step>
        ### Get Your API Key [#get-your-api-key-1]

        1. Log in to your [OffRail dashboard](https://offrail.ai/dashboard)
        2. Navigate to **API Keys** section
        3. Create a new API key and copy it
        4. Set it as an environment variable: `export OFFRAIL_API_KEY="your-api-key-here"`
      </Step>

      <Step>
        ### Configure Codex [#configure-codex]

        Run the following command in your terminal:

        ```bash
        codex mcp add offrail --url https://api.offrail.ai/mcp \
          --bearer-token-env-var OFFRAIL_API_KEY
        ```

        <Callout type="info">
          **Alternative: Manual configuration**

          You can also add the MCP server manually by editing `~/.codex/config.toml`:

          ```toml
          [mcp_servers.offrail]
          url = "https://api.offrail.ai/mcp"
          bearer_token_env_var = "OFFRAIL_API_KEY"
          ```
        </Callout>
      </Step>

      <Step>
        ### Test the Integration [#test-the-integration-1]

        Run `/mcp` in the Codex TUI to confirm the `offrail` server is connected. Try:

        * "Use the chat tool to ask GPT-4o about TypeScript best practices"
        * "Generate an image of a futuristic city using the generate-image tool"
        * "Use generate-nano-banana to create a hero image for my landing page"
        * "List all available models from Anthropic"
      </Step>
    </Steps>
  </Tab>

  <Tab value="Cursor">
    <Steps>
      <Step>
        ### Get Your API Key [#get-your-api-key-2]

        1. Log in to your [OffRail dashboard](https://offrail.ai/dashboard)
        2. Navigate to **API Keys** section
        3. Create a new API key and copy it
      </Step>

      <Step>
        ### Configure Cursor [#configure-cursor]

        Add the following to your Cursor MCP configuration file (`~/.cursor/mcp.json`):

        ```json
        {
          "mcpServers": {
            "offrail": {
              "url": "https://api.offrail.ai/mcp",
              "headers": {
                "Authorization": "Bearer your-api-key-here"
              }
            }
          }
        }
        ```

        Or open the Command Palette (`Cmd/Ctrl + Shift + P&#x60;), search for &#x2A;*"Cursor Settings"**, then go to **Tools & Integrations** > **Add Custom MCP** and paste the configuration above.

        <Callout type="info">
          Cursor v0.48.0+ is required for Streamable HTTP MCP support.
        </Callout>
      </Step>

      <Step>
        ### Test the Integration [#test-the-integration-2]

        Open a chat in **Agent Mode**, click the **Select Tools** icon, and verify the OffRail tools appear. Try:

        * "Use the chat tool to ask GPT-4o about TypeScript best practices"
        * "Generate an image of a futuristic city using the generate-image tool"
        * "Use generate-nano-banana to create a hero image for my landing page"
        * "List all available models from Anthropic"
      </Step>
    </Steps>
  </Tab>

  <Tab value="Other MCP Clients">
    OffRail's MCP server supports the standard HTTP Streamable transport. Configure your client with:

    * **Endpoint:** `https://api.offrail.ai/mcp`
    * **Authentication:** Bearer token via `Authorization` header or `x-api-key` header
    * **Protocol Version:** 2024-11-05

    **Direct HTTP Example:**

    ```bash
    curl -X POST https://api.offrail.ai/mcp \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer your-api-key" \
      -d '{
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/list"
      }'
    ```

    **Server-Sent Events (SSE):**

    For real-time updates, connect with `Accept: text/event-stream`:

    ```bash
    curl -N https://api.offrail.ai/mcp \
      -H "Accept: text/event-stream" \
      -H "Authorization: Bearer your-api-key"
    ```
  </Tab>
</Tabs>

## Use Cases [#use-cases]

### Multi-Model Access in Claude Code [#multi-model-access-in-claude-code]

Use Claude Code to interact with models it doesn't natively support:

```
Use the chat tool with model "gpt-4o" to analyze this code for security issues.
```

### Image Generation [#image-generation]

Generate images directly from your AI assistant:

```
Use generate-image to create a logo for my new startup.
It should be minimalist, blue and white, representing AI and cloud computing.
```

### Nano Banana (Gemini Image Generation) [#nano-banana-gemini-image-generation]

Generate images with Gemini 3 Pro for use in your project:

```
Use generate-nano-banana to create a hero image for my landing page with a 16:9 aspect ratio.
```

### Cost-Effective Model Selection [#cost-effective-model-selection]

Query available models to find the best option for your task:

```
List models from OpenAI and Anthropic, then use the cheapest one for this simple task.
```

## Authentication [#authentication]

The MCP server supports two authentication methods:

1. **Bearer Token** - `Authorization: Bearer your-api-key`
2. **API Key Header** - `x-api-key: your-api-key`

Your API key is the same one you use for the REST API and works across all OffRail services.

## OAuth Support [#oauth-support]

For applications that prefer OAuth authentication, OffRail's MCP server implements OAuth 2.0:

* **Authorization Endpoint:** `/oauth/authorize`
* **Token Endpoint:** `/oauth/token`
* **Registration Endpoint:** `/oauth/register`
* **Supported Flows:** Authorization Code, Client Credentials

## Enabling Local Image Saving [#enabling-local-image-saving]

By default, `generate-nano-banana` returns images inline without writing to disk. To enable saving generated images to the server filesystem, the `UPLOAD_DIR` environment variable must be set on the **gateway host** at startup. This is a server-side setting — it cannot be configured from the client.

This is only possible for **self-hosted** MCP deployments. Configure `UPLOAD_DIR` using your deployment method:

* **Docker:** Pass `-e UPLOAD_DIR=/data/images` or add it to your `docker-compose.yml` environment section.
* **systemd:** Add `Environment=UPLOAD_DIR=/data/images` to your service unit file.
* **.env file:** Add `UPLOAD_DIR=/data/images` to the `.env` file loaded by your gateway process.

<Callout type="warn">
  The shared hosted endpoint (`api.offrail.ai`) does not support configuring
  `UPLOAD_DIR`. On the hosted service, images are always returned inline — no
  files are written to disk. To enable server-side image saving, you must
  self-host the MCP server and set `UPLOAD_DIR` at startup.
</Callout>

## Troubleshooting [#troubleshooting]

### Connection Errors [#connection-errors]

If you're having trouble connecting:

1. Verify your API key is valid
2. Check the endpoint URL is correct: `https://api.offrail.ai/mcp`
3. Ensure your firewall allows outbound HTTPS connections

### Tool Not Found [#tool-not-found]

If tools aren't appearing:

1. Restart your MCP client
2. Check the configuration syntax
3. Verify the MCP server is responding: `GET https://api.offrail.ai/mcp`

### Rate Limiting [#rate-limiting]

The MCP server respects your account's rate limits. If you're hitting limits:

1. Check your usage in the dashboard
2. Consider upgrading your plan
3. Implement request queuing in your application

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support.
</Callout>

## Benefits [#benefits]

* **Unified Access** - Use 200+ models from 40+ providers through one interface
* **Cost Tracking** - Monitor usage and costs in the OffRail dashboard
* **Caching** - Automatic response caching reduces costs and latency
* **Fallback** - Automatic provider failover ensures reliability
* **Image Generation** - Generate images directly from your AI assistant


# Anthropic API Compatibility
URL: https://docs.offrail.ai/features/anthropic-endpoint


# Anthropic API Compatibility [#anthropic-api-compatibility]

OffRail provides a native Anthropic-compatible endpoint at `/v1/messages` that allows you to use any model in our catalog while maintaining the familiar Anthropic API format
This is especially useful for applications designed for Claude that you want to extend to use other models.

<Callout type="info">
  Enjoy a 50% discount on our Anthropic models for a limited time.
</Callout>

## Overview [#overview]

The Anthropic endpoint transforms requests from Anthropic's message format to the OpenAI-compatible format used by OffRail, then transforms the responses back to Anthropic's format. This means you can:

* Use **any model** available in OffRail with Anthropic's API format
* Maintain existing code that uses Anthropic's SDK or API format
* Access models from OpenAI, Google, Cohere, and other providers through the Anthropic interface
* Leverage OffRail's routing, caching, and cost optimization features

## Basic Usage [#basic-usage]

## Configuration for Claude Code [#configuration-for-claude-code]

This endpoint is perfect for configuring Claude Code to use any model available in OffRail:

```bash
export ANTHROPIC_BASE_URL=https://api.offrail.ai
export ANTHROPIC_AUTH_TOKEN=orl_your_api_key_here
# optional: specify a model, otherwise it uses the default Claude model
export ANTHROPIC_MODEL=gpt-5  # or any model from our catalog

# now run claude!
claude
```

<Callout type="info">
  Environment variables are read once at startup. The `/model` picker lists
  Claude models only, so non-Claude models are selected with `ANTHROPIC_MODEL`
  or `--model`. See the [Claude Code guide](https://docs.offrail.ai/guides/claude-code) for the
  settings-file options and gateway model discovery.
</Callout>

### Choosing Models [#choosing-models]

You can use any model from the [models page](https://offrail.ai/models). Popular options for Claude Code include:

```bash
# Use OpenAI's latest model
export ANTHROPIC_MODEL=gpt-5

# Use a cost-effective alternative
export ANTHROPIC_MODEL=gpt-5-mini

# Use Google's Gemini
export ANTHROPIC_MODEL=gemini-3.1-pro-preview

# Use Anthropic's actual Claude models
export ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
```

## Environment Variables [#environment-variables]

When configuring Claude Code or other Anthropic-compatible applications, you can use these environment variables:

### ANTHROPIC\_MODEL [#anthropic_model]

Specifies the main model to use for primary requests.

* **Default**: `claude-sonnet-4-20250514`
* **Example**: `export ANTHROPIC_MODEL=gpt-5`

### ANTHROPIC\_SMALL\_FAST\_MODEL [#anthropic_small_fast_model]

Specifies a smaller, faster model used for background functionality and internal operations.

* **Default**: `claude-3-5-haiku-20241022`
* **Example**: `export ANTHROPIC_SMALL_FAST_MODEL=gpt-5-nano`

```bash
# Example configuration
export ANTHROPIC_BASE_URL=https://api.offrail.ai
export ANTHROPIC_AUTH_TOKEN=orl_your_api_key_here
export ANTHROPIC_MODEL=gpt-5
export ANTHROPIC_SMALL_FAST_MODEL=gpt-5-nano
```

## Advanced Features [#advanced-features]

### Making a manual request [#making-a-manual-request]

```bash
curl -X POST "https://api.offrail.ai/v1/messages" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 100
  }'
```

### Response Format [#response-format]

The endpoint returns responses in Anthropic's message format:

```json
{
	"id": "msg_abc123",
	"type": "message",
	"role": "assistant",
	"model": "gpt-5",
	"content": [
		{
			"type": "text",
			"text": "Hello! I'm doing well, thank you for asking. How can I help you today?"
		}
	],
	"stop_reason": "end_turn",
	"stop_sequence": null,
	"usage": {
		"input_tokens": 13,
		"output_tokens": 20
	}
}
```

### Request Format [#request-format]

`/v1/messages` expects Anthropic Messages requests and always answers in Anthropic's format. Because the two formats share `model` and `messages`, an OpenAI Chat Completions body can reach this endpoint by accident — and since unknown parameters are ignored rather than rejected, the request succeeds and returns an Anthropic response body that OpenAI SDKs cannot read. If your client reports an empty completion here, check that it is pointed at `/v1/chat/completions`.

Unknown parameters are deliberately ignored rather than rejected, so a valid Anthropic request is never denied for carrying an extra field. As a consequence, OpenAI-only parameters (`response_format`, `stream_options`, `max_completion_tokens`, `n`, `stop`, `seed`, `frequency_penalty`, and similar) have no effect here — the model will not honour them. Use `/v1/chat/completions` if you need them.

A body that is *structurally* OpenAI is rejected by the schema, as it always has been — OpenAI-shaped `tools` (`{"type": "function", "function": {…}}`), OpenAI content parts such as `image_url`, or assistant turns with `content: null`. Those rejections now name the mismatch and point at the right endpoint instead of returning an opaque validation error:

```json
{
	"type": "error",
	"error": {
		"type": "invalid_request_error",
		"message": "This endpoint implements Anthropic's Messages API, and the request body uses OpenAI Chat Completions structures (tools[0].function) that Anthropic's format has no equivalent for. Send OpenAI-format requests to /v1/chat/completions instead, or convert the body to Anthropic's Messages format."
	}
}
```

Rejected requests are recorded in your logs with a `client_error` finish reason and zero cost, so a malformed client is visible in the activity feed rather than failing silently.

### Prompt Caching [#prompt-caching]

For Claude models, `cache_control` markers on `system` and message content blocks are forwarded to the provider unchanged, including the optional `ttl` (`5m` or `1h`):

```json
{
	"model": "claude-sonnet-4-6",
	"max_tokens": 100,
	"system": [
		{
			"type": "text",
			"text": "<several thousand tokens of stable instructions...>",
			"cache_control": { "type": "ephemeral" }
		}
	],
	"messages": [{ "role": "user", "content": "Hello!" }]
}
```

Cache usage comes back in Anthropic's native fields: `usage.cache_creation_input_tokens` (tokens written to the cache this request, billed at the write premium), `usage.cache_read_input_tokens` (tokens served from cache at the discounted rate), and `usage.cache_creation` (the per-TTL write breakdown).

<Callout type="warning">
  Each Claude model has a minimum cacheable prompt length (4,096 tokens on
  current-generation models such as Opus 4.5+, Sonnet 5, and Haiku 4.5). A
  `cache_control` marker on a shorter prompt is accepted but silently not cached
  — both cache usage fields stay `0`. See [Provider Cache
  Control](https://docs.offrail.ai/features/caching/provider-cache-control) for the per-model
  thresholds and details.
</Callout>

### Web Search [#web-search]

Anthropic's server-side web search tool works on this endpoint. Pass it as usual and the response carries `server_tool_use` and `web_search_tool_result` blocks before the text that cites them, so Anthropic SDK clients surface sources:

```json
{
	"model": "claude-haiku-4-5",
	"max_tokens": 400,
	"messages": [{ "role": "user", "content": "What shipped in Node 24?" }],
	"tools": [{ "type": "web_search_20250305", "name": "web_search" }]
}
```

Replaying the assistant turn verbatim on the next request is supported: the `server_tool_use` and `web_search_tool_result` blocks are accepted and dropped, since the provider re-runs the search rather than reusing the previous results.

### Tool Search [#tool-search]

Anthropic's server-side [tool search](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) works on this endpoint. Pass a `tool_search_tool_*` tool alongside your catalog and mark the tools that should load on demand with `defer_loading: true`:

```json
{
	"model": "claude-sonnet-4-6",
	"max_tokens": 1024,
	"messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
	"tools": [
		{
			"type": "tool_search_tool_regex_20251119",
			"name": "tool_search_tool_regex"
		},
		{
			"name": "get_weather",
			"description": "Get the weather at a specific location",
			"input_schema": { "type": "object" },
			"defer_loading": true
		}
	]
}
```

Deferred tools stay out of the rendered tools section, so adding one does not invalidate an existing prompt cache. The response carries the `server_tool_use` and `tool_search_tool_result` blocks; replay them verbatim on the next request and Anthropic keeps expanding the `tool_reference` entries they carry, so Claude reuses a discovered tool instead of searching again. `tool_reference` blocks returned from your own client-side search inside a `tool_result` are forwarded unchanged too.

Send every tool definition on every request, including the deferred ones — Anthropic needs them server-side to run the search. At least one tool must stay non-deferred (normally the tool search tool itself), and a tool cannot carry both `defer_loading: true` and `cache_control`.

**Where it works.** Tool search reaches the provider on the Anthropic API and on Anthropic models served through Google Cloud, and it needs a Claude 4.5-generation model or newer — older Claude models reject it upstream. On every other provider, including Anthropic models on AWS Bedrock, the tool search tool and `defer_loading` are dropped and all tools are sent eagerly. The request still succeeds, it just loses the cache and token savings, so pin the provider (`anthropic/claude-sonnet-4-6`) when those savings matter.

<Callout type="info">
  Bedrock is a transport limitation rather than a missing capability: Anthropic
  exposes server-side tool search there only through the InvokeModel API, and
  the gateway routes Bedrock through the Converse API.
</Callout>

### Gateway Response Cache [#gateway-response-cache]

If [gateway caching](https://docs.offrail.ai/features/caching/gateway-caching) is enabled on the project, a byte-identical request is replayed from cache instead of being sent upstream. Because the replayed body is identical (same `id`, same `usage`), the `x-offrail-cache: HIT` response header is the marker to check. Send `x-no-cache: true` to bypass the cache for a single request.


# API Keys & IAM Rules
URL: https://docs.offrail.ai/features/api-keys


# API Keys & IAM Rules [#api-keys--iam-rules]

API keys are the primary method for authenticating with the OffRail. This guide covers creating API keys, managing them, and configuring IAM rules for fine-grained access control.

## Overview [#overview]

OffRail provides comprehensive API key management with the following features:

* **Basic API Key Management**: Create, list, rename, update, and delete API keys
* **Usage Limits**: Set lifetime and recurring spending limits on individual API keys
* **Expiration (TTL)**: Give a key a time-to-live so it disables itself automatically
* **Rotation (Rolling)**: Replace a key's secret in place without losing its settings or history
* **IAM Rules**: Fine-grained access control for models, providers, and pricing
* **Usage Tracking**: Monitor API key usage and costs
* **Status Management**: Enable/disable keys without deletion

### Related key types [#related-key-types]

This page covers gateway API keys (`orl_…`), the keys you send to the
gateway as a bearer token. OffRail also issues three other kinds of keys,
each with its own page:

| Key                                         | What it is for                                                                         |
| ------------------------------------------- | -------------------------------------------------------------------------------------- |
| [Master keys](https://docs.offrail.ai/features/master-keys)        | Manage projects, gateway API keys, IAM rules, and custom models programmatically       |
| [Provider keys](https://docs.offrail.ai/learn/provider-keys)       | Bring your own upstream provider credentials so requests bill to your provider account |
| [Platform secret keys](https://docs.offrail.ai/learn/sdk-settings) | Mint end-user sessions from your backend when using the Payments SDK (`sk_…`)          |

## Creating API Keys [#creating-api-keys]

### Via Dashboard [#via-dashboard]

1. Navigate to your project in the OffRail dashboard
2. Go to the **API Keys** section
3. Click **Create API Key**
4. Provide a description for your key
5. Optionally set an all-time usage limit
6. Optionally set a recurring usage limit such as `$10 / day` or `$500 / month`
7. Optionally set an expiration (TTL) such as `30 minutes`, `12 hours`, or `7 days`
8. Click **Create**

<Callout type="warning">
  API keys are shown in full only once during creation. Make sure to copy and
  store them securely.
</Callout>

### Programmatically [#programmatically]

Gateway API keys can also be created, listed, updated, and deleted without the
dashboard using a [master key](https://docs.offrail.ai/features/master-keys) — useful when you provision
a key per customer or per environment from your own backend:

```bash
curl -X POST https://internal.offrail.ai/v1/master/keys \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "proj_...",
    "description": "Customer ACME — production key",
    "periodUsageLimit": "10.00",
    "periodUsageDurationValue": 1,
    "periodUsageDurationUnit": "day"
  }'
```

Usage limits and IAM rules can be configured through the master key API as well.
Expiration (TTL) and rotation are currently dashboard-only. See the [master key
API reference](https://docs.offrail.ai/features/master-keys#create-a-gateway-api-key) for the full
endpoint list.

## Using API Keys [#using-api-keys]

Once you have an API key, use it in the `Authorization` header of your requests:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer orl_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

## Renaming API Keys [#renaming-api-keys]

A key's name is only a label, so you can change it at any time — from the
dashboard, or with the `description` field on [`PATCH
/v1/master/keys/{id}`](https://docs.offrail.ai/features/master-keys#update-a-gateway-api-key) — without
affecting the secret, its usage history, limits, or IAM rules.

## Disabling/Enabling API Keys [#disablingenabling-api-keys]

You can disable an API key to stop it from being used, but the key is not deleted and can be re-enabled later.

## Rotating (Rolling) API Keys [#rotating-rolling-api-keys]

Rolling a key generates a **new secret for the same key** and invalidates the old
one immediately. Everything else about the key is preserved: its name, usage
history and statistics, all-time and recurring limits (including the active
period window), IAM rules, and expiration.

Use this when a secret may have been exposed — in a commit, a log, a CI
artifact, or a shared environment — and you want to cut off the leaked value
without losing the key's spend tracking or access rules.

1. Open the **API Keys** page and pick the key's actions menu
2. Choose **Roll Key** and confirm
3. Copy the new secret and update every client that used the old one

<Callout type="warning">
  The old secret stops working the moment the key is rolled, and the new secret
  is shown only once. Roll during a window where you can update your clients
  promptly.
</Callout>

Requests made with the old secret are rejected with a `401 Unauthorized`.
Rolling is limited to regular gateway API keys — the auto-generated playground
key cannot be rolled, and Payments SDK [platform secret
keys](https://docs.offrail.ai/learn/sdk-settings) have their own lifecycle.

## Expiration (TTL) [#expiration-ttl]

You can give an API key a &#x2A;*time-to-live (TTL)** when you create it. Set how long
the key should live — in **minutes**, **hours**, or **days** — and it will be
disabled automatically once that time passes. This is ideal for short-lived
integrations, demos, CI jobs, and temporary access.

* A key works normally until its expiration time
* Once expired, the gateway rejects requests with that key with a `401 Unauthorized`
* A background job marks expired keys as **inactive**, so the dashboard reflects
  the disabled state
* Keys created without a TTL never expire (the default)

### Reactivating an Expired Key [#reactivating-an-expired-key]

An expired key is paused, not deleted. To bring it back online you must reactivate
it **with a new future expiration** — an expired key cannot be re-enabled while its
TTL is still in the past. Keys that have no TTL, or whose TTL is still in the
future, can be enabled and disabled freely without setting a new expiration.

<Callout type="info">
  Expiration is independent of usage limits. A key can hit its TTL before, or
  instead of, reaching a spend cap.
</Callout>

## Usage Limits [#usage-limits]

Usage is tracked per API key on the API Keys page. Usage includes both costs
from OffRail credits and usage from your own provider keys when applicable,
giving you complete visibility into total spending per key.

You can set two independent limits for each key:

* **All-time usage limit**: A lifetime spend cap
* **Recurring usage limit**: A spend cap that resets every configured hour, day, week, or month

When a key reaches either limit, requests using that key return `401
Unauthorized` until the key is updated or, for recurring limits, the next
usage window starts. This is separate from IAM rule violations, which return
`403 Forbidden`.

Recurring windows support:

* Minimum duration: **1 hour**
* Maximum duration: **12 months**
* Units: **hour**, **day**, **week**, **month**

For the dashboard walkthrough and field-by-field details, see [API Keys in
Learn](https://docs.offrail.ai/learn/api-keys).

## IAM Rules [#iam-rules]

IAM (Identity Access Management) rules provide fine-grained access control over what models, providers, and pricing tiers an API key can access.

### Rule Types [#rule-types]

#### Model Access Rules [#model-access-rules]

Control access to specific models:

* **Allow Models**: Only allow access to specific models
* **Deny Models**: Block access to specific models

#### Provider Access Rules [#provider-access-rules]

Control access to specific providers:

* **Allow Providers**: Only allow access to specific providers
* **Deny Providers**: Block access to specific providers

Provider rules can also target your organization's own [custom providers](https://docs.offrail.ai/features/custom-providers). The generic `custom` entry matches every custom provider, while a `custom:<name>` entry (offered in the selector for each of your custom providers) matches only the custom provider with that name. Model rules can likewise reference a custom-catalog model as `<name>/<model>`.

#### Pricing Rules [#pricing-rules]

Control access based on model pricing:

* **Allow Pricing**: Set constraints on what pricing tiers are allowed
* **Deny Pricing**: Block specific pricing tiers
* **Free vs Paid**: Allow or deny access to free vs paid models

#### IP Address Rules [#ip-address-rules]

<Callout type="info">
  IP address rules are available on the **Enterprise** plan only. Contact us at
  [contact@offrail.ai](mailto:contact@offrail.ai) to enable them for your organization.
</Callout>

Restrict where the API key can be used from by source IP, using CIDR ranges:

* **Allow IP Ranges (CIDR)**: Only permit requests from the listed IPv4/IPv6 CIDRs
* **Deny IP Ranges (CIDR)**: Block requests from the listed IPv4/IPv6 CIDRs

Both IPv4 (e.g. `192.0.2.0/24`) and IPv6 (e.g. `2001:db8::/32`) ranges are supported, and you can mix both in a single rule. To restrict to a single address, use a `/32` (IPv4) or `/128` (IPv6) prefix.

The gateway reads the client IP from the first entry in the `X-Forwarded-For` header (set by the GCP load balancer). When an `allow_ip_cidrs` rule is configured and the gateway cannot determine the client IP, the request is denied. Invalid CIDR syntax is rejected at rule-creation time with a `400` error.

### Combining Multiple Rules [#combining-multiple-rules]

* **Allow rules of the same type are unioned**: a request passes if it matches *any* of them. For example, one `allow_models` rule with `["claude-opus-4-6"]` and another with `["claude-fable-5"]` allow both models — exactly as if you had a single rule listing both.
* **Allow rules of different types are combined with AND**: the request must satisfy every configured allow rule type (e.g. the model must be in the allowed models *and* served by an allowed provider).
* **Deny rules always apply**: a request matching any deny rule is rejected, regardless of allow rules.
* **Moderation is special-cased**: `/v1/moderations` runs a fixed moderation model that cannot appear in model allowlists, so only provider and IP rules apply to it — model and pricing rules are skipped.

## Member-Level IAM Rules [#member-level-iam-rules]

The same rule types can also be configured **per organization member** by owners and admins on the [Team page](https://docs.offrail.ai/learn/team) (admins cannot modify an owner's rules). Member-level rules act as an organization-wide ceiling for that member:

* A request must pass **both** the member's rules and the API key's rules. Within each level, rules combine exactly as described above.
* Key rules can only **narrow** access further — they can never grant anything the member's rules deny. For example, if an admin restricts a member to a single approved [provider](https://offrail.ai/providers) with an `allow_providers` rule, the member can create a key rule allowing only a specific model from that provider, but a key rule allowing any other provider has no effect.
* A key with **no rules of its own** is still fully constrained by its owner's member-level rules.
* Member-level rules apply to all regular API keys created by that member, across every project in the organization.

When a request is denied by a member-level rule, the `403` error message states that the restriction is an organization member IAM rule set by the org admin (rather than the key's own IAM configuration), so key holders know who to contact.

Member-level rules can also be managed programmatically via the [master key API](https://docs.offrail.ai/features/master-keys#member-iam-rules), addressing members by membership id or email.

## Error Handling [#error-handling]

When API keys encounter IAM rule violations, the API returns a `403` with the standard OpenAI error envelope:

```json
{
	"error": {
		"message": "Access denied: Model gpt-4 is not in the allowed models list",
		"type": "invalid_request_error",
		"param": null,
		"code": "permission_denied"
	}
}
```

Common error scenarios:

* Model not allowed by IAM rules
* Provider blocked by IAM rules
* Pricing limits exceeded
* API key disabled or deleted
* API key expired (TTL passed)
* API key rolled, so the old secret is no longer valid
* Usage limit reached

## Migration from Legacy Keys [#migration-from-legacy-keys]

If you have existing API keys without IAM rules:

1. **Backward Compatibility**: Existing keys continue to work without restrictions
2. **Gradual Migration**: Add IAM rules incrementally
3. **Testing**: Test IAM rules in development before applying to production
4. **Monitoring**: Monitor for access denied errors after implementing rules

<Callout type="info">
  API keys without IAM rules have unrestricted access to all models and
  providers.
</Callout>


# Audit Logs
URL: https://docs.offrail.ai/features/audit-logs


# Audit Logs [#audit-logs]

Audit logs provide complete visibility into all actions within your organization. Track who did what, when, and to which resource.

<Callout type="info">
  Audit logs are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise) for organization owners and admins.
</Callout>

## What's Tracked [#whats-tracked]

Every significant action is logged with detailed metadata:

| Field             | Description                                              |
| ----------------- | -------------------------------------------------------- |
| **Timestamp**     | When the action occurred                                 |
| **User**          | Who performed the action (name and email)                |
| **Action**        | What was done (e.g., `api_key.create`, `project.update`) |
| **Resource Type** | Category of the affected resource                        |
| **Resource ID**   | Unique identifier of the affected resource               |
| **Details**       | Additional context like resource names or changed fields |

## Tracked Actions [#tracked-actions]

### Organization Management [#organization-management]

* `organization.update` — Organization settings changed
* `organization.delete` — Organization deleted

### Project Management [#project-management]

* `project.create` — New project created
* `project.update` — Project settings changed
* `project.delete` — Project deleted

### Team Management [#team-management]

* `team_member.add` — New member invited
* `team_member.update` — Member role changed
* `team_member.remove` — Member removed

### API Key Management [#api-key-management]

* `api_key.create` — New API key created
* `api_key.update_status` — API key enabled/disabled
* `api_key.update_limit` — Usage limit changed
* `api_key.delete` — API key deleted
* `api_key.iam_rule.create` — IAM rule added
* `api_key.iam_rule.update` — IAM rule modified
* `api_key.iam_rule.delete` — IAM rule removed

### Provider Key Management [#provider-key-management]

* `provider_key.create` — Provider key added
* `provider_key.update` — Provider key status changed
* `provider_key.delete` — Provider key removed

### Billing Events [#billing-events]

* `subscription.create` — Subscription started
* `subscription.cancel` — Subscription cancelled
* `subscription.resume` — Subscription resumed
* `payment.credit_topup` — Credits purchased

## Filtering and Search [#filtering-and-search]

Filter logs by:

* **Action** — Specific action type
* **Resource Type** — Category of resource
* **User** — Who performed the action
* **Date Range** — Time period

## Data Retention [#data-retention]

Audit logs are retained for **90 days** on the Enterprise plan.

## Access Control [#access-control]

Only organization **owners** and **admins** can view audit logs. This ensures sensitive activity data is only visible to authorized personnel.

## Get Started [#get-started]

Audit logs are an Enterprise feature. [Contact us](https://offrail.ai/enterprise) to enable Enterprise for your organization.


# Compliance
URL: https://docs.offrail.ai/features/compliance


# Provider Compliance Policies [#provider-compliance-policies]

Provider compliance policies let you guarantee that requests are only ever routed to providers that meet your organization's regulatory requirements. When a request would be routed to a provider that doesn't meet the policy, the gateway blocks it **before any data leaves the gateway**.

<Callout type="info">
  Provider compliance policies are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise) for organization owners and admins.
</Callout>

## Requirements [#requirements]

Enable a policy under **Settings → Compliance** in the dashboard and toggle the requirements you need:

| Requirement                   | A provider is allowed when…                        |
| ----------------------------- | -------------------------------------------------- |
| **SOC 2 (Type 1 or 2)**       | it holds a SOC 2 report of any type                |
| **SOC 2 Type 2**              | it holds a SOC 2 Type 2 report specifically        |
| **ISO 27001**                 | it holds an ISO 27001 certification                |
| **SOC 2 Type 2 or ISO 27001** | it holds either a SOC 2 Type 2 report or ISO 27001 |
| **GDPR compliant**            | it is GDPR compliant                               |
| **No training on prompts**    | it does not train on API prompts                   |
| **No prompt logging**         | it does not log prompts                            |
| **No stealth providers**      | it is not a stealth provider                       |

Every requirement is **fail-closed**: a provider passes only if its published data policy explicitly satisfies the requirement. If an attribute is unknown for a provider, that provider is treated as non-compliant.

Stealth providers are undisclosed platforms that OffRail routes to without naming the operator, so their data policy and headquarters are unknown. That already makes them fail every certification and data-policy requirement above, but **No stealth providers** excludes them explicitly — useful when you want them gone without turning on any other requirement.

The settings page shows a live **Provider Impact** preview of which providers are allowed (green) and which are blocked (red) under the current policy, so you can see the impact before saving. Hovering a blocked provider lists exactly which requirements it does not meet — a missing certification, its data policy, its headquarters country, or a provider-list restriction.

## Provider headquarters [#provider-headquarters]

Beyond certifications, you can restrict routing by the country a provider is headquartered in. The **Provider Headquarters** card presents a country selector, and requests are only routed to providers based in a selected country.

The selector only offers countries that are referenced by a provider in the catalogue — browse the [providers directory](https://offrail.ai/providers) to see the current set and each provider's headquarters. Selecting no country applies no location restriction. The country filter is also fail-closed: when at least one country is selected, a provider whose headquarters is unknown is treated as non-compliant.

The country filter composes with the certification and data-policy requirements above — a provider must satisfy all active requirements to be allowed.

## Provider & model restrictions [#provider--model-restrictions]

Beyond attribute-based requirements, the **Provider & Model Restrictions** card lets you block or allow individual providers and models:

* **Blocked providers / blocked models** — deny lists. A listed provider or model is always blocked, even when it satisfies every requirement above.
* **Allowed providers / allowed models** — fine-grained allow lists. When non-empty, only the listed providers (or models) may be used; everything else is blocked. An empty list applies no restriction.

Deny lists always win over allow lists, and both compose with the requirements above — an allow-listed provider must still satisfy every active certification, data-policy, and country requirement.

<Callout type="warn">
  A non-empty allowed-providers list blocks **every** provider not on it,
  regardless of certifications or data policy. If you allow-list only a custom
  provider, all catalogue providers show as blocked with the reason "Not on the
  allowed-providers list" — add any additional provider you want to use to the
  allow list.
</Callout>

The selectors include your organization's own [custom providers](https://docs.offrail.ai/features/custom-providers) (stored as `custom:<name>` refs) and their custom-catalog models (stored as `<name>/<model>` refs), so a specific custom deployment can be blocked or allow-listed individually just like a catalogue provider.

### Choosing a compliant provider [#choosing-a-compliant-provider]

The provider dropdowns are policy-aware, so you can tell **before** adding a provider to a list whether it satisfies your policy:

* A **green shield** marks a provider that meets every active certification, data-policy, and headquarters requirement.
* A **red shield** marks a provider that does not; the requirements it misses (for example "May log prompts" or "Headquartered in China, which is not an allowed country") are listed under its name.
* The &#x2A;*"Only providers that meet policy requirements"** toggle at the top of the dropdown hides incompatible providers entirely.

These indicators evaluate the requirements only — deliberately ignoring the allowed/blocked lists themselves — so an active allow list never paints every other provider red while you're deciding what to add to it. Your own custom providers are evaluated against their [self-attested posture](#custom-providers); one without an attestation on file shows "No compliance attestation on file".

Elsewhere in the dashboard (for example the API-key provider filter), the colored dot next to a provider is simply the provider's **brand color** and carries no compliance meaning.

These restrictions are **organization-wide** and take precedence over member-level and API-key-level [IAM rules](https://docs.offrail.ai/features/api-keys): they are enforced after IAM evaluation, so no user-, team-, or key-level allow rule can grant access to a provider or model the compliance policy excludes.

## Enforcement [#enforcement]

When no available provider for a model meets the policy — or a pinned provider is non-compliant — the gateway returns a `403`:

```json
{
	"error": {
		"message": "This request was blocked by your organization's provider compliance policy. No available provider for deepseek-v3.2 meets the required certifications or provider/model restrictions. Contact your OffRail admin to adjust the policy."
	}
}
```

This applies to both routing modes:

* **Automatic routing** — non-compliant providers are removed from the candidate set, and the request is blocked if none remain.
* **Pinned providers** — a request such as `deepseek/deepseek-v3.2` is blocked when that specific provider does not meet the policy.

Each block is recorded as a **security event** so administrators can review what was rejected and why.

## Custom providers [#custom-providers]

[Custom providers](https://docs.offrail.ai/features/custom-providers) point at infrastructure your organization operates itself (for example, models hosted in your own cloud account), so they have no entry in the provider catalogue and no published data policy. Under any enabled compliance policy they are therefore **blocked by default** — the same fail-closed rule that applies to catalogue providers with unknown attributes.

To route through a custom provider while a policy is active, an organization owner or admin can record a **self-attestation** of that provider key's compliance posture under **Models → Compliance attestation** in the dashboard. The attestation covers the same attributes as a catalogue data policy — SOC 2 report type, ISO 27001, GDPR, training on API prompts, prompt logging, retention period, and the country the deployment is operated from — and the gateway evaluates it against your policy with **identical fail-closed rules**: an attribute left "unknown" never satisfies a requirement, and only an explicit "No" satisfies requirements like "No training on prompts". Clearing the attestation restores the blocked state.

<Callout type="warn">
  OffRail does not verify attestations — they are your organization's own
  assertion about infrastructure you operate. Every attestation change is
  recorded in the audit log with the attesting user and timestamp.
</Callout>

Attestations only apply to custom provider keys. They can never be used to clear a catalogue provider that fails your policy.

**Known limitation:** the country selector in the compliance policy only offers countries referenced by catalogue providers (see the [providers directory](https://offrail.ai/providers)). A custom deployment attested to a country outside that set cannot be allowed while a country restriction is active.

## Safety identifiers [#safety-identifiers]

Providers that offer an abuse-attribution identifier receive an opaque, random value that OffRail generates once per organization. It contains no personal data — no email address, user id or organization name — and it is the only thing that ties a report of abusive traffic back to an account.

Whether OffRail forwards this identifier depends on the upstream API. Check the [providers directory](https://offrail.ai/providers) and open a provider to see its current behavior. Any `safety_identifier` you set on a request yourself is ignored — the value is always the one derived from your organization.

## Access Control [#access-control]

Only organization **owners** and **admins** can view and change the compliance policy.

Project-scoped **developer** members cannot see the policy itself, but they can browse the org's available providers and models — including custom providers and their model catalogs — read-only on the **Models** page. This is the recommended way for developers to discover what they can call without being granted additional permissions.

## Related [#related]

* [Org Models Directory](https://docs.offrail.ai/features/models-directory) — see per-model eligibility under the active policy in the dashboard.
* [Data Retention](https://docs.offrail.ai/features/data-retention) — control whether request and response payloads are stored.
* [Guardrails](https://docs.offrail.ai/features/guardrails) — detect and block harmful or sensitive content.

## Get Started [#get-started]

Provider compliance policies are an Enterprise feature. [Contact us](https://offrail.ai/enterprise) to enable Enterprise for your organization.


# Cost Breakdown
URL: https://docs.offrail.ai/features/cost-breakdown


# Cost Breakdown [#cost-breakdown]

OffRail provides real-time cost information for each API request directly in the response's `usage` object. This allows you to track costs programmatically without needing to query the dashboard.

<Callout type="info">
  Cost breakdown is available for all users on both hosted and self-hosted
  deployments.
</Callout>

## Response Format [#response-format]

When cost breakdown is enabled, your API responses will include additional cost fields in the `usage` object:

```json
{
	"id": "chatcmpl-123",
	"object": "chat.completion",
	"created": 1234567890,
	"model": "openai/gpt-4o",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "Hello! How can I help you today?"
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 10,
		"completion_tokens": 15,
		"total_tokens": 25,
		"cost": 0.000125,
		"cost_details": {
			"upstream_inference_cost": 0.000125,
			"upstream_inference_prompt_cost": 0.000025,
			"upstream_inference_completions_cost": 0.0001,
			"total_cost": 0.000125,
			"input_cost": 0.000025,
			"output_cost": 0.0001,
			"cached_input_cost": 0,
			"request_cost": 0,
			"web_search_cost": 0,
			"image_input_cost": null,
			"image_output_cost": null,
			"data_storage_cost": 0.00000025
		},
		"prompt_tokens_details": {
			"cached_tokens": 0,
			"cache_write_tokens": 0,
			"audio_tokens": 0,
			"video_tokens": 0
		},
		"completion_tokens_details": {
			"reasoning_tokens": 0,
			"image_tokens": 0,
			"audio_tokens": 0
		}
	}
}
```

## Cost Fields [#cost-fields]

| Field                                              | Description                                                                                                                                            |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `cost`                                             | Total inference cost for the request in USD                                                                                                            |
| `cost_details.upstream_inference_cost`             | Combined upstream inference cost in USD (prompt + completions)                                                                                         |
| `cost_details.upstream_inference_prompt_cost`      | Upstream cost for prompt tokens in USD (includes cached prompt discount)                                                                               |
| `cost_details.upstream_inference_completions_cost` | Upstream cost for completion tokens in USD                                                                                                             |
| `cost_details.total_cost`                          | Total request cost in USD (OffRail extended field)                                                                                                     |
| `cost_details.input_cost`                          | Cost for non-cached prompt tokens in USD                                                                                                               |
| `cost_details.output_cost`                         | Cost for completion tokens in USD                                                                                                                      |
| `cost_details.cached_input_cost`                   | Cost for cached prompt tokens in USD                                                                                                                   |
| `cost_details.cache_write_input_cost`              | Cost for prompt tokens written to the provider cache in USD, billed at the provider's cache-write premium (e.g. 1.25x for 5m / 2x for 1h on Anthropic) |
| `cost_details.request_cost`                        | Per-request flat fee in USD (when the model applies one)                                                                                               |
| `cost_details.web_search_cost`                     | Cost for web search tool calls in USD                                                                                                                  |
| `cost_details.image_input_cost`                    | Cost for image inputs in USD                                                                                                                           |
| `cost_details.image_output_cost`                   | Cost for image outputs in USD                                                                                                                          |
| `cost_details.data_storage_cost`                   | Storage cost for retained request/response payloads in USD                                                                                             |

## Token Detail Fields [#token-detail-fields]

The `usage` object also includes detailed token counters that mirror OpenAI's extended format:

| Field                                         | Description                                                                                                                        |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `prompt_tokens_details.cached_tokens`         | Number of prompt tokens served from the provider's prompt cache                                                                    |
| `prompt_tokens_details.cache_write_tokens`    | Number of prompt tokens written into the provider's prompt cache                                                                   |
| `prompt_tokens_details.cache_creation_tokens` | Alias of `cache_write_tokens`, matching Anthropic's naming                                                                         |
| `prompt_tokens_details.cache_creation`        | Per-TTL breakdown of cache writes (`ephemeral_5m_input_tokens` / `ephemeral_1h_input_tokens`), present when a cache write occurred |
| `prompt_tokens_details.audio_tokens`          | Number of audio prompt tokens                                                                                                      |
| `prompt_tokens_details.video_tokens`          | Number of video prompt tokens                                                                                                      |
| `completion_tokens_details.reasoning_tokens`  | Number of reasoning tokens produced by reasoning models                                                                            |
| `completion_tokens_details.image_tokens`      | Number of image tokens produced                                                                                                    |
| `completion_tokens_details.audio_tokens`      | Number of audio tokens produced                                                                                                    |

## Streaming Responses [#streaming-responses]

Cost information is also available in streaming responses. The cost fields are included in the final usage chunk sent before the `[DONE]` message:

```
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[...],"usage":{"prompt_tokens":10,"completion_tokens":15,"total_tokens":25,"cost":0.000125,"cost_details":{"upstream_inference_cost":0.000125,"upstream_inference_prompt_cost":0.000025,"upstream_inference_completions_cost":0.0001,"total_cost":0.000125,"input_cost":0.000025,"output_cost":0.0001,"cached_input_cost":0,"request_cost":0,"web_search_cost":0,"image_input_cost":null,"image_output_cost":null,"data_storage_cost":0.00000025}}}

data: [DONE]
```

## Example: Tracking Costs in Code [#example-tracking-costs-in-code]

Here's an example of how to track costs programmatically using the cost breakdown feature:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

async function trackCosts() {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages: [{ role: "user", content: "Hello!" }],
	});

	const usage = response.usage as any;

	if (usage.cost !== undefined) {
		console.log(`Request cost: $${usage.cost.toFixed(6)}`);
		console.log(
			`  Prompt: $${usage.cost_details.upstream_inference_prompt_cost.toFixed(6)}`,
		);
		console.log(
			`  Completions: $${usage.cost_details.upstream_inference_completions_cost.toFixed(6)}`,
		);

		const cachedTokens = usage.prompt_tokens_details?.cached_tokens ?? 0;
		if (cachedTokens > 0) {
			console.log(`  Cached prompt tokens: ${cachedTokens}`);
		}
	}

	return response;
}
```

## Use Cases [#use-cases]

### Budget Monitoring [#budget-monitoring]

Track costs in real-time and implement budget limits in your application:

```typescript
let totalSpent = 0;
const BUDGET_LIMIT = 10.0; // $10 budget

async function makeRequest(messages: Message[]) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages,
	});

	const cost = (response.usage as any).cost || 0;
	totalSpent += cost;

	if (totalSpent > BUDGET_LIMIT) {
		throw new Error(`Budget exceeded: $${totalSpent.toFixed(2)}`);
	}

	return response;
}
```

### Per-User Cost Allocation [#per-user-cost-allocation]

Track costs per user for billing or analytics:

```typescript
const userCosts: Map<string, number> = new Map();

async function makeRequestForUser(userId: string, messages: Message[]) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages,
	});

	const cost = (response.usage as any).cost || 0;
	const currentCost = userCosts.get(userId) || 0;
	userCosts.set(userId, currentCost + cost);

	return response;
}
```

### Cost Analytics [#cost-analytics]

Aggregate costs by model, time period, or any other dimension:

```typescript
interface CostEntry {
	timestamp: Date;
	model: string;
	promptCost: number;
	completionsCost: number;
	totalCost: number;
}

const costLog: CostEntry[] = [];

async function loggedRequest(model: string, messages: Message[]) {
	const response = await client.chat.completions.create({
		model,
		messages,
	});

	const usage = response.usage as any;

	costLog.push({
		timestamp: new Date(),
		model: response.model,
		promptCost: usage.cost_details?.upstream_inference_prompt_cost || 0,
		completionsCost:
			usage.cost_details?.upstream_inference_completions_cost || 0,
		totalCost: usage.cost || 0,
	});

	return response;
}
```

## Self-Hosted Deployments [#self-hosted-deployments]

If you're running a self-hosted OffRail deployment, cost breakdown is always included in API responses regardless of plan. This allows you to track internal costs and allocate them across teams or projects.


# Custom Providers
URL: https://docs.offrail.ai/features/custom-providers


# Custom Providers [#custom-providers]

OffRail supports integrating custom OpenAI-compatible providers, allowing you to use any API that follows the OpenAI chat completions format. This feature is perfect for:

* Private or self-hosted LLM deployments
* Specialized AI providers not natively supported
* Internal AI services within your organization
* Testing against different model endpoints

<Callout type="info">
  Custom providers must be OpenAI-compatible, supporting the
  `/v1/chat/completions` endpoint format.
</Callout>

## Quick Setup [#quick-setup]

### 1. Add a Custom Provider Key [#1-add-a-custom-provider-key]

Navigate to your organization's provider settings and add a custom provider via the UI.
Provide a lowercase name, OpenAI-compatible base URL, and API token for the custom provider.

### 2. Make Requests [#2-make-requests]

Once configured, make requests using the format `{customName}/{modelName}`:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mycompany/custom-gpt-4",
    "messages": [
      {
        "role": "user",
        "content": "Hello from my custom provider!"
      }
    ]
  }'
```

## Configuration Requirements [#configuration-requirements]

### Custom Provider Name [#custom-provider-name]

* **Format**: Lowercase letters (`a-z`) with optional single hyphens between them
* **Examples**: `mycompany`, `internal`, `my-company`, `eu-west`
* **Invalid**: `MyCompany`, `my_company`, `123test`, `-mycompany`, `my-`, `my--company`

<Callout type="warn">
  The custom provider name must match the regex pattern `/^[a-z]+(-[a-z]+)*$/`
  exactly.
</Callout>

### Base URL [#base-url]

* Must be a valid URL pointing to your provider's base endpoint
* Use an `https://` URL for hosted providers; an `http://` URL is allowed when
  self-hosting (for example a service on your own network)
* OffRail will append `/v1/chat/completions` automatically
* **Example**: `https://api.example.com` → `https://api.example.com/v1/chat/completions`

### API Token [#api-token]

* Provider-specific authentication token
* Used in the `Authorization: Bearer {token}` header

<Callout type="info">
  Unlike built-in providers, custom provider models are not validated, giving
  you complete flexibility.
</Callout>

## Supported Features [#supported-features]

Custom providers inherit full OffRail functionality.

## Custom Model Catalog [#custom-model-catalog]

<Callout type="info">
  The custom model catalog is an **Enterprise** feature. Managing entries
  requires an enterprise plan, but the gateway always honors catalog entries
  that already exist.
</Callout>

By default, requests through a custom provider are **not billed** and have **no
enforced limits** — OffRail has no catalog entry for the model, so it cannot
know its pricing, context window, or capabilities. The **custom model catalog**
lets you define that information per custom provider key, so OffRail can
attribute cost, enforce limits, and report usage just like a built-in model.

Custom catalog models are **text-output** models. Multi-modal *input* (images,
audio) is still supported and can be priced via the input fields below; output
generation (images, video, audio) is intentionally out of scope because it is
too provider-specific to bill generically.

### Defining a model [#defining-a-model]

Open **Organization → Models**, pick the custom provider key, and add a
catalog entry. Every field except the model id is optional:

| Field                    | Purpose                                                                                                                         |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| **Model id**             | The id used after the provider prefix (e.g. `gpt-5.5` in `mycompany/gpt-5.5`).                                                  |
| **Display name**         | Optional human-readable label.                                                                                                  |
| **Context size**         | Maximum input window in tokens. Enforced per request.                                                                           |
| **Max output**           | Maximum completion tokens. Enforced against `max_tokens`.                                                                       |
| **Token prices**         | Input, output, cached input, cache read, cache write (5m and 1h), per-request, web search, image input, and audio input prices. |
| **Capabilities**         | `streaming`, `vision`, `tools`, `reasoning`, `jsonOutput`, `audio`.                                                             |
| **Supported parameters** | Advisory list of accepted request parameters.                                                                                   |

Prices are in **USD per token** and accept either decimal (`0.000003`) or
exponent (`3.0e-6`) notation.

### Cost attribution [#cost-attribution]

When a request matches a catalog entry, OffRail bills it using the entry's
token prices and records the cost on the activity log. Without a matching entry
the request stays unbilled (zero/null cost).

<Callout type="warn">
  A known model id (for example `gpt-5.5`) routed through a custom provider is
  **not** billed at that model's public pricing — only the prices you define in
  the catalog apply. Define the model in the catalog to attribute cost.
</Callout>

### Limit and capability enforcement [#limit-and-capability-enforcement]

When an entry defines limits or capabilities, OffRail enforces them before
forwarding the request:

* **Context size / max output** — requests that would exceed the configured
  window or output budget are rejected with `400`.
* **Capabilities** — when a flag is explicitly set to disabled, requests that use
  that feature are rejected (e.g. images against a non-`vision` model, tools
  against a non-`tools` model, streaming against a non-streaming model). Flags
  left unset stay permissive and the upstream provider enforces them.

### Restricting to the catalog [#restricting-to-the-catalog]

Each custom provider key has an **Only allow catalog models** switch. When
enabled, requests through that provider must reference a defined catalog model —
undefined models are rejected with `400`. This guarantees that every request has
known cost attribution and enforced limits. When disabled, undefined models
still work but remain unbilled, as before.

## Visibility [#visibility]

Every active organization member — including project-scoped **developer**
members — can browse the org's custom providers and their model catalogs
read-only on the **Models** page, so developers can always discover
which providers and models are available to them. Creating, editing, and
deleting providers, models, and attestations stays restricted to organization
owners and admins.

## Compliance attestation [#compliance-attestation]

If your organization enforces a [provider compliance policy](https://docs.offrail.ai/features/compliance),
custom providers are blocked by default — they have no catalogue data policy, so
the fail-closed rules treat them as non-compliant. Organization owners and
admins can record a per-key **self-attestation** of the deployment's compliance
posture under **Models → Compliance attestation**, which the gateway then
evaluates against the policy exactly like a catalogue provider's data policy.
See [Compliance → Custom providers](https://docs.offrail.ai/features/compliance#custom-providers) for
details and caveats.


# Data Retention
URL: https://docs.offrail.ai/features/data-retention


# Data Retention [#data-retention]

OffRail offers configurable data retention policies that allow you to store full request and response payloads. This enables powerful debugging capabilities, detailed analytics, and compliance with data governance requirements.

## Retention Levels [#retention-levels]

OffRail supports two retention levels that can be configured per organization:

| Level               | Description                                                                                    | Storage Cost    |
| ------------------- | ---------------------------------------------------------------------------------------------- | --------------- |
| **Metadata Only**   | Stores request metadata (timestamps, model, tokens, costs) without full payloads. Default.     | Free            |
| **Retain All Data** | Stores complete request and response payloads including messages, tool calls, and attachments. | $0.01/1M tokens |

<Callout type="info">
  Metadata-only retention is enabled by default and provides usage analytics
  without additional storage costs.
</Callout>

<Callout type="warning">
  Retention levels are configurable on standard (pay-as-you-go) organizations
  only. Chat subscriptions are always metadata only — their request and response
  payloads are not retained, and there is no setting to turn payload storage on.
  The Responses API exception below still applies to them.
</Callout>

## Storage Pricing [#storage-pricing]

When full data retention is enabled, storage is billed at **$0.01 per 1 million tokens**. This rate applies to:

* Input tokens (prompt)
* Cached input tokens
* Output tokens (completion)
* Reasoning tokens

Storage costs are calculated per request and billed separately from inference. When "Retain All Data" is enabled, each response's `usage.cost_details` object includes a `data_storage_cost` field with the per-request storage cost in USD. See [Cost Breakdown](https://docs.offrail.ai/features/cost-breakdown) for the full list of cost fields.

### Example Cost Calculation [#example-cost-calculation]

For a request with:

* 1,000 input tokens
* 500 output tokens
* 1,500 total tokens

Storage cost = 1,500 / 1,000,000 × $0.01 = **$0.000015**

## Configuring Retention [#configuring-retention]

Data retention is configured at the organization level in your dashboard settings. The setting is only available on standard pay-as-you-go organizations:

1. Navigate to **Organization Settings** → **Policies**
2. Select your preferred **Data Retention Level**
3. Save changes

<Callout type="warning">
  Changing retention settings applies to new requests only. Existing stored data
  follows the retention period active when it was created.
</Callout>

## Retention Periods [#retention-periods]

Data is retained for 30 days for all users. Enterprise plans can have custom retention periods. After the retention period expires, data is automatically deleted.

<Callout type="info">
  The Responses API does not require data retention. Stored responses (used for
  `previous_response_id` chaining and `GET /v1/responses/:id`) are kept in
  dedicated storage for 30 days — matching OpenAI's own retention — regardless
  of your organization's data retention policy, and are not billed as data
  storage. Because this is independent of the retention level, it applies to
  metadata-only organizations, including chat, whose Responses API input and
  output items are therefore held for up to 30 days. Send `store: false` with
  the request to opt out.
</Callout>

## Accessing Stored Data [#accessing-stored-data]

When data retention is enabled, you can access your stored requests through the dashboard:

* View request history with full payload inspection
* Filter by model and date range
* Inspect complete request and response payloads

## Use Cases [#use-cases]

### Debugging [#debugging]

Full data retention enables you to:

* Inspect exact prompts sent to models
* Review complete responses including tool calls
* Trace conversation histories
* Identify issues in production

### Analytics [#analytics]

With stored payloads, you can:

* Analyze prompt patterns and effectiveness
* Track response quality over time
* Build custom dashboards and reports
* Measure model performance across use cases

### Compliance [#compliance]

Data retention helps meet compliance requirements by:

* Maintaining audit trails of AI interactions
* Enabling data governance policies
* Supporting incident investigation
* Providing records for regulatory requirements

## Billing Considerations [#billing-considerations]

### Credit Usage [#credit-usage]

In **API keys mode** (using your own provider keys):

* Only storage costs are deducted from OffRail credits
* Inference costs are billed directly to your provider

In **credits mode**:

* Both inference and storage costs are deducted from credits

### Monitoring Storage Costs [#monitoring-storage-costs]

Storage costs appear in:

* Usage dashboard under "Storage" category
* Billing invoices as a separate line item

<Callout type="success">
  Enable [auto top-up](https://offrail.ai/dashboard) in billing settings to
  ensure uninterrupted service when storage costs accumulate.
</Callout>

## Self-Hosted Deployments [#self-hosted-deployments]

Self-hosted deployments have full control over data retention:

* Configure retention periods in environment variables
* Data is stored in your own PostgreSQL database
* No additional storage costs (you manage your own infrastructure)

## Privacy and Security [#privacy-and-security]

* All stored data is encrypted at rest
* Access is restricted to organization members with appropriate permissions
* Data is automatically deleted after the retention period
* You can request immediate deletion of specific records through support


# Document Reading
URL: https://docs.offrail.ai/features/documents


# Document Reading [#document-reading]

OffRail supports sending documents (PDFs and other file types) to document-capable models using OpenAI's `file` content block format. The gateway forwards the document to the underlying provider so the model can read and reason over its contents.

## Document-Capable Models [#document-capable-models]

Document input is currently supported on Google Gemini models via Google AI Studio. You can find document-capable models on the [models page with the document filter](https://offrail.ai/models?filters=1\&document=true).

## Sending a Document [#sending-a-document]

Add a `file` content block to a user message. The `file_data` field must be a base64-encoded data URL that includes the document's MIME type.

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.6-flash",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Summarize this document."
          },
          {
            "type": "file",
            "file": {
              "filename": "report.pdf",
              "file_data": "data:application/pdf;base64,JVBERi0xLjQKJ..."
            }
          }
        ]
      }
    ]
  }'
```

### Content Block Fields [#content-block-fields]

* **`type`**: must be `"file"`.
* **`file.filename`*&#x2A; &#x2A;(optional)*: original filename, shown in Lounge and forwarded for context.
* **`file.file_data`**: base64-encoded data URL of the form `data:<mime-type>;base64,<data>`.

<Callout type="info">
  The `file.file_id` field (for referencing files uploaded via a provider's
  Files API) is accepted by the schema but not currently supported by the Google
  transform. Use `file_data` with an inline base64 data URL.
</Callout>

## Supported File Types [#supported-file-types]

The accepted MIME types depend on the target model. Gemini models commonly support:

* `application/pdf`
* `text/plain`
* `text/html`
* `text/css`
* `text/javascript`
* `text/csv`
* `text/markdown`
* `text/xml`

If the upstream provider rejects the MIME type, the gateway surfaces a `400` error including the unsupported MIME type and the provider it was sent to. To use a different file type, encode the file with the matching MIME type in the data URL prefix.

## Encoding a File as a Data URL [#encoding-a-file-as-a-data-url]

Any tool that can produce base64 output works. For example, in a shell:

```bash
DATA=$(base64 -i report.pdf | tr -d '\n')
echo "data:application/pdf;base64,$DATA"
```

Or in JavaScript:

```javascript
import { readFileSync } from "node:fs";

const buffer = readFileSync("report.pdf");
const fileData = `data:application/pdf;base64,${buffer.toString("base64")}`;
```

Then pass `fileData` as the `file.file_data` value in your request.

## Multiple Documents [#multiple-documents]

You can include multiple `file` blocks in a single message, optionally mixed with text and image content:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.1-pro-preview",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Compare these two reports." },
          {
            "type": "file",
            "file": {
              "filename": "q1.pdf",
              "file_data": "data:application/pdf;base64,JVBERi0x..."
            }
          },
          {
            "type": "file",
            "file": {
              "filename": "q2.pdf",
              "file_data": "data:application/pdf;base64,JVBERi0x..."
            }
          }
        ]
      }
    ]
  }'
```

## Error Handling [#error-handling]

The gateway returns `400` for the following document-related errors:

* The selected model does not support document input.
* The `file` block is missing both `file_data` and `file_id`.
* `file_data` is not a valid base64 data URL.
* The upstream provider rejects the document's MIME type for the selected model.


# Dynamic Routes
URL: https://docs.offrail.ai/features/dynamic-routes


# Dynamic Routes [#dynamic-routes]

Dynamic routes let you move routing logic out of your application code and into the gateway. Instead of hardcoding a model, you define a named decision graph — conditions on the request, percentage-based traffic splits, and model targets — and invoke it by putting `dynamic/<name>` in the `model` field of any OpenAI-compatible request:

```bash
curl https://api.offrail.ai/v1/chat/completions \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "x-user-tier: paid" \
  -d '{
    "model": "dynamic/support",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

The graph is evaluated on every request. Once it resolves to a model, the request flows through the same [smart routing](https://docs.offrail.ai/features/routing) as any other request: weighted provider scoring, sticky sessions, and automatic cross-provider fallback all apply to the resolved target.

<Callout>
  Dynamic routes are available on the <b>Enterprise plan</b>. Manage them under{" "}
  <b>Project Settings → Dynamic Routes</b> in the dashboard, either in a visual
  drag-and-drop editor or as raw JSON.
</Callout>

## The route graph [#the-route-graph]

A route is a JSON document with an `entry` node id and a list of nodes. Evaluation starts at `entry` and follows branches until it reaches a `model` node (route the request) or an `end` node (reject the request with a `400`).

```json
{
	"entry": "tier",
	"nodes": [
		{
			"id": "tier",
			"type": "conditional",
			"conditions": [
				{
					"field": { "source": "header", "path": "x-user-tier" },
					"op": "eq",
					"value": "paid",
					"next": "premium"
				}
			],
			"else": "split"
		},
		{ "id": "premium", "type": "model", "model": "claude-sonnet-4-6" },
		{
			"id": "split",
			"type": "percentage",
			"splits": [
				{ "weight": 80, "next": "stable" },
				{ "weight": 20, "next": "experiment" }
			]
		},
		{
			"id": "stable",
			"type": "model",
			"model": "gpt-5-nano",
			"providers": ["openai"]
		},
		{ "id": "experiment", "type": "model", "model": "gemini-2.5-flash" }
	]
}
```

Node ids may contain letters, digits, hyphens, and underscores.

## Node types [#node-types]

### `conditional` [#conditional]

Evaluates its conditions top to bottom; the first match wins and the request follows that condition's `next`. When nothing matches, the request follows `else`.

Each condition reads one field from the request:

| `field.source` | What `field.path` means                                                           |
| -------------- | --------------------------------------------------------------------------------- |
| `header`       | A request header name (case-insensitive), e.g. `x-user-tier`                      |
| `body`         | A dot-path into the JSON request body, e.g. `metadata.segment` or `max_tokens`    |
| `metadata`     | A gateway-provided request attribute: `orgId`, `projectId`, `apiKeyId`, or `plan` |

Supported operators:

| Operator   | Matches when                                                              | Value type                |
| ---------- | ------------------------------------------------------------------------- | ------------------------- |
| `eq`       | The field equals the value (compared as strings)                          | string / number / boolean |
| `neq`      | The field differs from the value — also matches when the field is missing | string / number / boolean |
| `in`       | The field equals any entry in the value array                             | array of strings          |
| `contains` | The field's string form contains the value                                | string                    |
| `gt`, `lt` | The field is numerically greater / less than the value                    | number                    |
| `exists`   | The field is present (no value)                                           | —                         |

### `percentage` [#percentage]

Splits traffic across branches by relative weight. The draw is **deterministic per session**: the split key is the request's session id (`x-session-id` and the other [sticky session routing](https://docs.offrail.ai/features/routing#sticky-session-routing) signals), so a conversation keeps its assignment across requests instead of flip-flopping between experiment arms. Requests without a session id get an independent draw per request.

Weights are relative — `{ 80, 20 }` and `{ 4, 1 }` produce the same split.

### `model` [#model]

Terminates evaluation and routes the request to a catalog model (see the [models page](https://offrail.ai/models) for available ids). Optional `providers` restricts routing to those providers and doubles as the ordered fallback preference; when omitted, every provider serving the model is a candidate and weighted smart routing picks the best one.

### `end` [#end]

Terminates evaluation and rejects the request with a `400`. Useful as an explicit deny branch — for example, refusing traffic that doesn't carry a required header.

## Validation [#validation]

Graphs are validated when you save a draft and again when you publish:

* every referenced node must exist and be reachable from `entry`
* cycles are rejected — evaluation is deterministic per request, so a revisited node would loop forever
* model ids and provider ids must exist in the catalog, and each listed provider must actually serve the model
* operator/value mismatches (e.g. `gt` with a non-numeric value) are rejected

An invalid graph can never be saved or published, and publishing re-validates against the live model catalog so a stale draft can't resurrect a removed model.

## Versions and rollback [#versions-and-rollback]

Edits always go to the route's **draft**. Publishing snapshots the draft as an immutable, numbered version and points the route at it; the published version is what serves traffic. Rolling back re-points the route at any previous version instantly — no data migration, no re-deploy.

A route only serves requests when it is **enabled** and has a **published version**; otherwise requests using it are rejected with a `404`.

## Observability [#observability]

Each request served through a dynamic route records the route name, published version, and the node path the evaluation took in its routing metadata, alongside the usual provider scoring details — visible in the activity log detail view.

## Reliability [#reliability]

Published routes are cached in the gateway and carry a stale fallback: when the database is temporarily unreachable, recently used routes keep resolving from cache, the same way API keys and project settings do.


# Embeddings
URL: https://docs.offrail.ai/features/embeddings


# Embeddings [#embeddings]

OffRail exposes an OpenAI-compatible `/v1/embeddings` endpoint for generating vector representations of text — useful for semantic search, clustering, recommendations, and RAG.

Browse available embedding models on the [models page](https://offrail.ai/models?filters=1\&embedding=true).

## Supported providers [#supported-providers]

* **OpenAI** — `text-embedding-3-small`, `text-embedding-3-large`, `text-embedding-ada-002`
* **Google AI Studio** — `gemini-embedding-2` (recommended), `gemini-embedding-001` (legacy)
* **Google Vertex AI** — `gemini-embedding-001`, `text-embedding-005`

The gateway translates between provider-native request/response shapes (e.g. Google's `:embedContent` / `:batchEmbedContents`) and the OpenAI-compatible payload, so you can swap models without changing your client code.

## cURL [#curl]

```bash
curl -X POST "https://api.offrail.ai/v1/embeddings" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "The quick brown fox jumps over the lazy dog."
  }'
```

## OpenAI JS SDK [#openai-js-sdk]

```ts
import OpenAI from "openai";

const client = new OpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const response = await client.embeddings.create({
	model: "text-embedding-3-small",
	input: "The quick brown fox jumps over the lazy dog.",
});

console.log(response.data[0].embedding);
```

<Callout type="info">
  Embedding models are billed only for input tokens. There are no output tokens
  since embeddings are fixed-size vectors.
</Callout>

## Improving retrieval quality [#improving-retrieval-quality]

Vector similarity is fast but approximate. For RAG, pair embeddings with
[rerank](https://docs.offrail.ai/features/rerank): use embeddings to pull a broad candidate set, then
rerank those candidates to pick the few documents that actually go in the
prompt.


# Guardrails
URL: https://docs.offrail.ai/features/guardrails


# Guardrails [#guardrails]

Guardrails protect your organization by automatically detecting and blocking harmful content in LLM requests before they reach the model.

<Callout type="info">
  Guardrails are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise).
</Callout>

## Overview [#overview]

Guardrails run on every API request, scanning message content for:

* Security threats (prompt injection, jailbreak attempts)
* Sensitive data (PII, secrets, credentials)
* Policy violations (blocked terms, restricted topics)

When a violation is detected, you control what happens: block the request, redact the content, or log a warning.

## Organization and Project Scopes [#organization-and-project-scopes]

Guardrails are configured for the whole organization by default, and every project uses that configuration.

A project can opt out and define its own guardrails instead — useful when one application needs stricter or looser rules than the rest of the organization. Turn off **Use organization guardrails** in the project's Guardrails settings, and that project's configuration and custom rules fully replace the organization's for its traffic.

Exactly one scope is ever in force for a request:

| Project setting                     | What runs on that project's requests                      |
| ----------------------------------- | --------------------------------------------------------- |
| **Use organization guardrails** on  | The organization configuration and its custom rules       |
| **Use organization guardrails** off | The project's own configuration and its custom rules only |

Opting out never inherits partially: the organization's custom rules stop applying to that project too, and the project starts with a copy of the organization's current settings so nothing is silently dropped at the moment you opt out. Other projects are unaffected either way.

Both pages state which scope wins. A project page names itself and, while it inherits, shows the organization settings read-only; the organization page lists any projects that override it, so it is clear those settings apply everywhere else.

Only organization owners and admins can change guardrails, at either scope.

## System Rules [#system-rules]

Built-in rules protect against common threats:

### Prompt Injection Detection [#prompt-injection-detection]

Detects attempts to override or manipulate system instructions. Common patterns include:

* "Ignore all previous instructions"
* "You are now a different AI"
* Hidden instructions in encoded text

### Jailbreak Detection [#jailbreak-detection]

Identifies attempts to bypass safety measures:

* DAN (Do Anything Now) prompts
* Roleplay-based bypasses
* Instruction override attempts

### PII Detection [#pii-detection]

Identifies personal information:

* Email addresses
* Phone numbers
* Social Security Numbers
* Credit card numbers
* IP addresses
* Passport and driver's license numbers

When the action is set to **redact**, PII is replaced with placeholders like `[EMAIL_REDACTED]`.

Detection is validated rather than purely pattern-based, so identifiers that only
look like PII are left alone: card numbers must pass the Luhn checksum, phone
numbers need grouping separators or a nearby phone keyword, and passport and
license numbers need a nearby keyword. A bare numeric id such as a product id,
order number or unix timestamp is therefore never redacted as a phone number.

### Secrets Detection [#secrets-detection]

Detects credentials and API keys:

* AWS access keys and secrets
* Generic API keys
* Passwords in common formats
* Private keys

High-entropy checks and placeholder filtering keep ordinary content out of the
way: git SHAs and hex digests are not treated as AWS secrets, and template
values such as `YOUR_API_KEY`, `${OPENAI_API_KEY}` or `********` are not treated
as credentials.

### File Type Restrictions [#file-type-restrictions]

Control which file types can be uploaded:

* Configure allowed MIME types
* Set maximum file size limits
* Block potentially dangerous file types

### Document Leakage Prevention [#document-leakage-prevention]

Detects attempts to extract confidential documents or internal data.

## Configurable Actions [#configurable-actions]

For each rule, choose how to respond:

| Action     | Behavior                                            |
| ---------- | --------------------------------------------------- |
| **Block**  | Reject the request with a content policy error      |
| **Redact** | Remove or mask the sensitive content, then continue |
| **Warn**   | Log the violation but allow the request to proceed  |

## Custom Rules [#custom-rules]

Create custom rules for your use case, on the organization or on a single project:

### Blocked Terms [#blocked-terms]

Prevent specific words or phrases from being used:

* Match type: exact, contains, or regex
* Case-sensitive matching option
* Multiple terms per rule

### Custom Regex [#custom-regex]

Match patterns unique to your organization:

* Internal project codenames
* Customer identifiers
* Domain-specific sensitive data

### Topic Restrictions [#topic-restrictions]

Block content related to specific topics:

* Define restricted topics
* Keyword-based detection

## Security Events Dashboard [#security-events-dashboard]

Monitor all guardrail violations with a dedicated dashboard:

* **Total violations** — Overall count and trends
* **By action** — Breakdown of blocked, redacted, and warned
* **By category** — Which rules are being triggered
* **Detailed logs** — Individual violations with timestamps and matched patterns

## How It Works [#how-it-works]

```
Request → Guardrails Check → Action Based on Rules → Forward to Model (if allowed)
                ↓
           Log Violation
```

1. **Request received** — API request comes in with messages
2. **Scope resolved** — The project's own guardrails if it overrides, otherwise the organization's
3. **Content scanned** — All text content is checked against enabled rules
4. **Violations detected** — Matches are identified and logged
5. **Action taken** — Based on rule configuration (block/redact/warn)
6. **Request proceeds** — If not blocked, the (potentially redacted) request continues

## Best Practices [#best-practices]

1. **Start with warnings** — Enable rules in warn mode first to understand your traffic patterns
2. **Review violations** — Check the Security Events dashboard regularly
3. **Tune custom rules** — Adjust blocked terms and regex patterns based on false positives
4. **Layer defenses** — Use multiple rule types together for comprehensive protection

## Get Started [#get-started]

Guardrails are an Enterprise feature. [Contact us](https://offrail.ai/enterprise) to enable Enterprise for your organization.


# Image Generation
URL: https://docs.offrail.ai/features/image-generation


# Image Generation [#image-generation]

OffRail supports image generation through two APIs:

1. **`/v1/images/generations`** — OpenAI-compatible images endpoint (recommended for simple image generation)
2. **`/v1/images/edits`** — OpenAI-compatible image editing endpoint
3. **`/v1/chat/completions`** — Chat completions with image generation models (for conversational image generation and editing)

For asynchronous video generation, see [Video Generation](https://docs.offrail.ai/features/video-generation).

## Available Models [#available-models]

You can find all available image generation models on our [models page](https://offrail.ai/models?filters=1\&imageGeneration=true).

## OpenAI Images API [#openai-images-api]

The `/v1/images/generations` endpoint provides a drop-in replacement for OpenAI's image generation API. It works with any OpenAI-compatible client library.

### Parameters [#parameters]

| Parameter         | Type    | Default      | Description                                                                                                      |
| ----------------- | ------- | ------------ | ---------------------------------------------------------------------------------------------------------------- |
| `prompt`          | string  | required     | A text description of the desired image(s)                                                                       |
| `model`           | string  | `"auto"`     | The model to use. `auto` resolves to `gemini-3-pro-image-preview`                                                |
| `n`               | integer | `1`          | Number of images to generate (1-10)                                                                              |
| `size`            | string  | —            | Image dimensions. Supported sizes depend on the model/provider — see [Image Configuration](#image-configuration) |
| `quality`         | string  | —            | Image quality. Supported values depend on the model/provider — see [Image Configuration](#image-configuration)   |
| `response_format` | string  | `"b64_json"` | Only `b64_json` is supported                                                                                     |
| `style`           | string  | —            | Image style: `vivid` or `natural`                                                                                |

### curl [#curl]

```bash
curl -X POST "https://api.offrail.ai/v1/images/generations" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3-pro-image-preview",
    "prompt": "A cute cat wearing a tiny top hat",
    "n": 1,
    "size": "1024x1024"
  }'
```

### OpenAI SDK [#openai-sdk]

Works with the standard OpenAI client library — just point the base URL to OffRail.

```ts
import OpenAI from "openai";
import { writeFileSync } from "fs";

const client = new OpenAI({
	baseURL: "https://api.offrail.ai/v1",
	apiKey: process.env.OFFRAIL_API_KEY,
});

const response = await client.images.generate({
	model: "gemini-3-pro-image-preview",
	prompt: "A futuristic city skyline at sunset with flying cars",
	n: 1,
	size: "1024x1024",
});

response.data.forEach((image, i) => {
	if (image.b64_json) {
		const buf = Buffer.from(image.b64_json, "base64");
		writeFileSync(`image-${i}.png`, buf);
	}
});
```

### Vercel AI SDK [#vercel-ai-sdk]

Use the `@ai-sdk/openai` with `generateImage`.

```ts
import { createOpenAI } from "@ai-sdk/openai";
import { generateImage } from "ai";
import { writeFileSync } from "fs";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const result = await generateImage({
	model: offrail.image("gemini-3-pro-image-preview"),
	prompt:
		"A cozy cabin in a snowy mountain landscape at night with aurora borealis",
	size: "1024x1024",
	n: 1,
	// aspectRatio and quality are model-specific — only some providers honor them.
	// aspectRatio works on Gemini image models; OpenAI gpt-image-2 ignores it
	// (use a literal WxH `size` instead).
	aspectRatio: "16:9",
	// quality works on OpenAI gpt-image-2 ("low" | "medium" | "high" | "auto").
	// The AI SDK only forwards it through providerOptions.
	providerOptions: {
		offrail: { quality: "high" },
	},
});

result.images.forEach((image, i) => {
	const buf = Buffer.from(image.base64, "base64");
	writeFileSync(`image-${i}.png`, buf);
});
```

## OpenAI Images Edit API [#openai-images-edit-api]

The `/v1/images/edits` endpoint is OpenAI-compatible and supports a focused subset of `images.edit` parameters.

### Parameters [#parameters-1]

| Parameter            | Type                     | Required | Description                                                        |
| -------------------- | ------------------------ | -------- | ------------------------------------------------------------------ |
| `images`             | array of `{ image_url }` | yes      | Input images. `image_url` supports HTTPS URLs and base64 data URLs |
| `prompt`             | string                   | yes      | A text description of the desired image edit                       |
| `model`              | string                   | no       | Image editing model                                                |
| `background`         | enum                     | no       | `transparent`, `opaque`, or `auto`                                 |
| `input_fidelity`     | enum                     | no       | `high` or `low`                                                    |
| `n`                  | integer                  | no       | Number of edited images to generate                                |
| `output_format`      | enum                     | no       | `png`, `jpeg`, or `webp`                                           |
| `output_compression` | integer                  | no       | Compression level for `jpeg`/`webp`                                |
| `quality`            | enum                     | no       | `low`, `medium`, `high`, or `auto`                                 |
| `size`               | string                   | no       | Output size. Examples: `1024x1024`, `1536x1024`, `1K`, `2K`, `4K`  |
| `aspect_ratio`       | string                   | no       | Aspect ratio override. Examples: `1:1`, `16:9`, `4:3`, `5:4`       |

<Callout type="warning">
  `mask` is not supported yet on `/v1/images/edits`.
</Callout>

### curl (HTTPS image URL) [#curl-https-image-url]

```bash
curl -X POST "https://api.offrail.ai/v1/images/edits" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "images": [
      {
        "image_url": "https://example.com/source-image.png"
      }
    ],
    "prompt": "Add a watercolor effect to this image",
    "model": "gemini-3-pro-image-preview",
    "aspect_ratio": "16:9",
    "quality": "high",
    "size": "4K"
  }'
```

### curl (base64 data URL) [#curl-base64-data-url]

```bash
curl -X POST "https://api.offrail.ai/v1/images/edits" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "images": [
      {
        "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
      }
    ],
    "prompt": "Turn this into a pixel-art style image"
  }'
```

## Chat Completions API [#chat-completions-api]

Image generation also works through the `/v1/chat/completions` endpoint, which is useful for conversational image generation, image editing with vision, and multi-turn interactions.

### Making Requests [#making-requests]

Simply use an image generation model and provide a text prompt describing the image you want to create.

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3-pro-image-preview",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a cute golden retriever puppy playing in a sunny meadow"
      }
    ]
  }'
```

### Response Format [#response-format]

Image generation models return responses in the standard chat completions format, with generated images included in the `images` array within the assistant message:

```json
{
	"id": "chatcmpl-1756234109285",
	"object": "chat.completion",
	"created": 1756234109,
	"model": "gemini-3-pro-image-preview",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "Here's an image of a cute dog for you: ",
				"images": [
					{
						"type": "image_url",
						"image_url": {
							"url": "data:image/png;base64,<base64_encoded_image_data>"
						}
					}
				]
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 8,
		"completion_tokens": 1303,
		"total_tokens": 1311
	}
}
```

### Vision support [#vision-support]

You can edit or modify images by combining image generation with [vision models](https://docs.offrail.ai/features/vision) by including the image in the `messages` array.

### Response Structure [#response-structure]

#### Images Array [#images-array]

The `images` array contains one or more generated images with the following structure:

* `type`: Always `"image_url"` for generated images
* `image_url.url`: A data URL containing the base64-encoded image data (format: `data:image/png;base64,<data>`)

#### Content Field [#content-field]

The `content` field may contain descriptive text about the generated image, depending on the model's behavior.

### AI SDK (Chat Completions) [#ai-sdk-chat-completions]

You can use the AI SDK to generate images with your existing generateText or streamText calls using the OffRail provider.

#### Example [#example]

```ts title="/api/chat/route.ts"
import { streamText, type UIMessage, convertToModelMessages } from "ai";
import { createOpenAI } from "@ai-sdk/openai";

interface ChatRequestBody {
	messages: UIMessage[];
}

export async function POST(req: Request) {
	const body = await req.json();

	const { messages }: ChatRequestBody = body;

	const offrail = createOpenAI({
		apiKey: "offrail_api_key",
		baseURL: "https://api.offrail.ai/v1",
	});

	try {
		const result = streamText({
			model: offrail.chat("gemini-3-pro-image-preview"),
			messages: convertToModelMessages(messages),
		});

		return result.toUIMessageStreamResponse();
	} catch {
		return new Response(JSON.stringify({ error: "OffRail request failed" }), {
			status: 500,
		});
	}
}
```

Then you can render the image in your frontend using the `Image` component from the [ai-elements](https://ai-sdk.dev/elements/components/image).

Here is a full example of how to use the AI SDK to generate images in your frontend:

```tsx title="/app/page.tsx"
"use client";

import { useState, useRef } from "react";
import { useChat } from "@ai-sdk/react";
import { parseImagePartToDataUrl } from "@/lib/image-utils";
import {
	PromptInput,
	PromptInputBody,
	PromptInputButton,
	PromptInputSubmit,
	PromptInputTextarea,
	PromptInputToolbar,
} from "@/components/ai-elements/prompt-input";
import {
	Conversation,
	ConversationContent,
} from "@/components/ai-elements/conversation";
import { Image } from "@/components/ai-elements/image";
import { Loader } from "@/components/ai-elements/loader";
import { Message, MessageContent } from "@/components/ai-elements/message";
import { Response } from "@/components/ai-elements/response";

export const ChatUI = () => {
	const textareaRef = useRef<HTMLTextAreaElement | null>(null);
	const [text, setText] = useState("");
	const { messages, status, stop, regenerate, sendMessage } = useChat();

	return (
		<>
			<div className="flex-1 overflow-y-auto px-4 pb-24">
				<Conversation>
					<ConversationContent>
						{messages.length === 0 ? (
							<div className="mb-6 text-center">
								<h2 className="text-3xl font-semibold tracking-tight">
									How can I help you?
								</h2>
							</div>
						) : (
							messages.map((m, messageIndex) => {
								const isLastMessage = messageIndex === messages.length - 1;

								if (m.role === "assistant") {
									const textContent = m.parts
										.filter((p) => p.type === "text")
										.map((p) => p.text)
										.join("");
									// Combine all image parts (both image_url and file types)
									const imageParts = m.parts.filter(
										(p) =>
											p.type === "file" && p.mediaType?.startsWith("image/"),
									);

									return (
										<div key={m.id}>
											{textContent ? <Response>{textContent}</Response> : null}
											{imageParts.length > 0 ? (
												<div className="mt-3 grid grid-cols-1 gap-3 sm:grid-cols-2">
													{imageParts.map((part, idx: number) => {
														const { base64Only, mediaType } =
															parseImagePartToDataUrl(part);

														if (!base64Only) {
															return null;
														}

														return (
															<Image
																key={idx}
																base64={base64Only}
																mediaType={mediaType}
																alt={part.name || "Generated image"}
															/>
														);
													})}
												</div>
											) : null}
											{isLastMessage &&
												(status === "submitted" || status === "streaming") && (
													<Loader />
												)}
										</div>
									);
								} else {
									return (
										<Message key={m.id} from={m.role}>
											<MessageContent variant="flat">
												{m.parts.map((p, i) => {
													if (p.type === "text") {
														return <div key={i}>{p.text}</div>;
													}
													return null;
												})}
											</MessageContent>
											{isLastMessage &&
												(status === "submitted" || status === "streaming") && (
													<Loader />
												)}
										</Message>
									);
								}
							})
						)}
					</ConversationContent>
				</Conversation>
			</div>
			<div className="sticky bottom-0 left-0 right-0 px-4 pb-[max(env(safe-area-inset-bottom),1rem)] pt-2 bg-gradient-to-t from-background via-background/95 to-transparent backdrop-blur supports-[backdrop-filter]:bg-background/60">
				<PromptInput
					aria-disabled={status === "streaming"}
					onSubmit={async (message) => {
						if (status === "streaming") {
							return;
						}

						try {
							const textContent = message.text ?? "";
							if (!textContent.trim()) {
								return;
							}

							setText(""); // Clear input immediately

							const parts = [{ type: "text", text: textContent }];

							// Call sendMessage which will handle adding the user message and API request
							sendMessage({
								role: "user",
								parts,
							});
						} catch (error) {
							// Throw error here
						}
					}}
				>
					<PromptInputBody>
						<PromptInputTextarea
							ref={textareaRef}
							value={text}
							onChange={(e) => setText(e.currentTarget.value)}
							placeholder="Message"
						/>
					</PromptInputBody>
					<PromptInputToolbar>
						<div className="flex items-center gap-2">
							{status === "streaming" ? (
								<PromptInputButton onClick={() => stop()} variant="ghost">
									Stop
								</PromptInputButton>
							) : null}
							<PromptInputSubmit
								status={status === "streaming" ? "streaming" : "ready"}
							/>
						</div>
					</PromptInputToolbar>
				</PromptInput>
			</div>
		</>
	);
};
```

```ts title="/lib/image-utils.ts"
/**
 * Parses a file object containing image data and returns a properly formatted data URL
 * and normalized media type.
 *
 * Handles:
 * - Normalizing mediaType from various property names (mediaType, mime_type)
 * - Detecting existing data: URLs
 * - Detecting base64-looking content
 * - Stripping whitespace from base64 content
 * - Building proper data:...;base64,... URLs
 */
export function parseImageFile(file: {
	url?: string;
	mediaType?: string;
	mime_type?: string;
}): { dataUrl: string; mediaType: string } {
	const mediaType = file.mediaType || file.mime_type || "image/png";
	let url = String(file.url || "");

	const isDataUrl = url.startsWith("data:");
	const looksLikeBase64 =
		!isDataUrl && /^[A-Za-z0-9+/=\s]+$/.test(url.slice(0, 200));

	if (looksLikeBase64) {
		url = url.replace(/\s+/g, "");
	}

	const dataUrl = isDataUrl
		? url
		: looksLikeBase64
			? `data:${mediaType};base64,${url}`
			: url;

	return { dataUrl, mediaType };
}

/**
 * Extracts base64-only content from a data URL.
 * Returns empty string if the input is not a valid data URL.
 */
export function extractBase64FromDataUrl(dataUrl: string): string {
	if (!dataUrl.startsWith("data:")) {
		return "";
	}

	const comma = dataUrl.indexOf(",");
	return comma >= 0 ? dataUrl.slice(comma + 1) : "";
}

/**
 * Parses an image part (either image_url or file type) and returns
 * dataUrl, base64Only, and mediaType ready for rendering.
 *
 * Handles error cases gracefully by returning empty base64Only string
 * when parsing fails, allowing the renderer to skip invalid images.
 */
export function parseImagePartToDataUrl(part: any): {
	dataUrl: string;
	base64Only: string;
	mediaType: string;
} {
	try {
		// Handle image_url parts
		if (part.type === "image_url" && part.image_url?.url) {
			const url = part.image_url.url;
			const mediaType = "image/png"; // Default for image_url parts

			if (url.startsWith("data:")) {
				// Extract media type from data URL if present
				const match = url.match(/data:([^;]+)/);
				const extractedMediaType = match?.[1] || mediaType;
				return {
					dataUrl: url,
					base64Only: extractBase64FromDataUrl(url),
					mediaType: extractedMediaType,
				};
			}

			return {
				dataUrl: url,
				base64Only: "",
				mediaType,
			};
		}

		// Handle file parts (AI SDK format)
		if (part.type === "file") {
			const { dataUrl, mediaType } = parseImageFile(part);
			return {
				dataUrl,
				base64Only: extractBase64FromDataUrl(dataUrl),
				mediaType,
			};
		}

		return {
			dataUrl: "",
			base64Only: "",
			mediaType: "image/png",
		};
	} catch {
		return {
			dataUrl: "",
			base64Only: "",
			mediaType: "image/png",
		};
	}
}
```

## Image Configuration [#image-configuration]

You can customize the generated image using the optional `image_config` parameter (for chat completions) or `size`/`quality`/`style` parameters (for the images API). The supported parameters vary by provider.

### Google Models [#google-models]

Available Google models:

| Model                            | Description                                                                         |
| -------------------------------- | ----------------------------------------------------------------------------------- |
| `gemini-3-pro-image-preview`     | Gemini 3 Pro with native image generation. Supports aspect ratios and 1K–4K sizes.  |
| `gemini-3.1-flash-image-preview` | Gemini 3.1 Flash with native image generation. Supports 0.5K–4K sizes (default 1K). |

#### gemini-3-pro-image-preview [#gemini-3-pro-image-preview]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3-pro-image-preview",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a mountain landscape at sunset"
      }
    ],
    "image_config": {
      "aspect_ratio": "16:9",
      "image_size": "4K"
    }
  }'
```

| Parameter      | Type   | Description                                                                                                                                   |
| -------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `aspect_ratio` | string | The aspect ratio of the generated image. Options: `"1:1"`, `"2:3"`, `"3:2"`, `"3:4"`, `"4:3"`, `"4:5"`, `"5:4"`, `"9:16"`, `"16:9"`, `"21:9"` |
| `image_size`   | string | The resolution of the generated image. Options: `"1K"` (1024x1024), `"2K"` (2048x2048), `"4K"` (4096x4096)                                    |

#### gemini-3.1-flash-image-preview [#gemini-31-flash-image-preview]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.1-flash-image-preview",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a mountain landscape at sunset"
      }
    ],
    "image_config": {
      "image_size": "1K"
    }
  }'
```

| Parameter      | Type   | Description                                                                                                                                                                       |
| -------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `aspect_ratio` | string | The aspect ratio of the generated image. Options: `"1:1"`, `"1:4"`, `"1:8"`, `"2:3"`, `"3:2"`, `"3:4"`, `"4:1"`, `"4:3"`, `"4:5"`, `"5:4"`, `"8:1"`, `"9:16"`, `"16:9"`, `"21:9"` |
| `image_size`   | string | The resolution of the generated image. Options: `"0.5K"` (512x512), `"1K"` (1024x1024, default), `"2K"` (2048x2048), `"4K"` (4096x4096)                                           |

<Callout type="info">
  `gemini-3.1-flash-image-preview` uniquely supports `"0.5K"` resolution, which
  is not available on other Google image models.
</Callout>

### Alibaba Models [#alibaba-models]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "alibaba/qwen-image-plus",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a mountain landscape at sunset"
      }
    ],
    "image_config": {
      "image_size": "1024x1536",
      "n": 1,
      "seed": 42
    }
  }'
```

| Parameter    | Type    | Description                                                                                      |
| ------------ | ------- | ------------------------------------------------------------------------------------------------ |
| `image_size` | string  | Image dimensions in `WIDTHxHEIGHT` format. Examples: `"1024x1024"`, `"1024x1536"`, `"1536x1024"` |
| `n`          | integer | Number of images to generate (1-4)                                                               |
| `seed`       | integer | Random seed for reproducible generation                                                          |

Available Alibaba models (see the [models page](https://offrail.ai/models?filters=1\&imageGeneration=true) for current pricing):

| Model                        | Description                                                                          |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| `alibaba/qwen-image`         | Standard quality image generation                                                    |
| `alibaba/qwen-image-plus`    | Good balance of quality and cost                                                     |
| `alibaba/qwen-image-max`     | Highest quality image generation                                                     |
| `alibaba/qwen-image-3.0`     | Third-generation image generation and editing                                        |
| `alibaba/qwen-image-3.0-pro` | Highest quality third-generation generation and editing. Priced per output size tier |

<Callout type="info">
  Alibaba models use explicit pixel dimensions (e.g., `"1024x1536"`) instead of
  aspect ratios. For portrait orientation use `"1024x1536"`, for landscape use
  `"1536x1024"`.
</Callout>

### Z.AI Models [#zai-models]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai/cogview-4",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a futuristic city skyline"
      }
    ],
    "image_config": {
      "image_size": "1024x1024"
    }
  }'
```

| Parameter    | Type    | Description                                                                                      |
| ------------ | ------- | ------------------------------------------------------------------------------------------------ |
| `image_size` | string  | Image dimensions in `WIDTHxHEIGHT` format. Examples: `"1024x1024"`, `"2048x1024"`, `"1024x2048"` |
| `n`          | integer | Number of images to generate                                                                     |

Available Z.AI models (see the [models page](https://offrail.ai/models?filters=1\&imageGeneration=true) for current pricing):

| Model           | Description                                                                                                         |
| --------------- | ------------------------------------------------------------------------------------------------------------------- |
| `zai/cogview-4` | CogView-4 with bilingual support and excellent text rendering                                                       |
| `zai/glm-image` | GLM-Image with hybrid auto-regressive architecture, excellent for text-rendering and knowledge-intensive generation |

<Callout type="info">
  CogView-4 supports both Chinese and English prompts and excels at generating
  images with embedded text.
</Callout>

### OpenAI Models [#openai-models]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-image-2",
    "messages": [
      {
        "role": "user",
        "content": "Generate a photo-real cinematic landscape at golden hour"
      }
    ],
    "image_config": {
      "image_size": "3072x2160",
      "image_quality": "low"
    }
  }'
```

| Parameter       | Type   | Description                                                                           |
| --------------- | ------ | ------------------------------------------------------------------------------------- |
| `image_size`    | string | Image dimensions in `WIDTHxHEIGHT` format, or `"auto"` to let the model choose.       |
| `image_quality` | string | One of `"low"`, `"medium"`, `"high"`, or `"auto"`. Defaults to `"auto"` when omitted. |

<Callout type="warning">
  OpenAI image models do **not** accept `aspect_ratio`. Always specify
  `image_size` as `WIDTHxHEIGHT` (e.g. `"1024x1024"`, `"3072x2160"`). OpenAI
  requires both width and height to be divisible by 16, the longest edge to be ≤
  3840, and the total pixel count to fit within the model's pixel budget;
  requests outside these bounds are rejected with HTTP 400.
</Callout>

Available OpenAI image models:

| Model                | Description                                                                                                  |
| -------------------- | ------------------------------------------------------------------------------------------------------------ |
| `openai/gpt-image-2` | OpenAI's next-generation image model with improved quality and prompt adherence, supporting text and vision. |

### ByteDance Models [#bytedance-models]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bytedance/seedream-4-5",
    "messages": [
      {
        "role": "user",
        "content": "Generate an image of a futuristic cyberpunk city at night"
      }
    ],
    "image_config": {
      "image_size": "2048x2048"
    }
  }'
```

| Parameter    | Type   | Description                                                                                      |
| ------------ | ------ | ------------------------------------------------------------------------------------------------ |
| `image_size` | string | Image dimensions in `WIDTHxHEIGHT` format. Examples: `"1024x1024"`, `"2048x2048"`, `"4096x4096"` |

Available ByteDance models (see the [models page](https://offrail.ai/models?filters=1\&imageGeneration=true) for current pricing):

| Model                    | Description                                                     |
| ------------------------ | --------------------------------------------------------------- |
| `bytedance/seedream-4-0` | High-quality text-to-image generation with 2K default output    |
| `bytedance/seedream-4-5` | Enhanced quality and consistency with improved prompt adherence |

<Callout type="info">
  Seedream models support up to 2-10 reference images for multi-image fusion and
  generation. The default output resolution is 2048×2048 (2K), with support up
  to 4096×4096 (4K).
</Callout>

## Usage Notes [#usage-notes]

<Callout type="info">
  Image generation models typically have higher token costs compared to
  text-only models due to the computational requirements of image synthesis.
</Callout>

<Callout type="warning">
  Generated images are returned as base64-encoded data URLs, which can be large.
  Consider the payload size when integrating image generation into your
  applications.
</Callout>


# Master Keys
URL: https://docs.offrail.ai/features/master-keys


# Master Keys [#master-keys]

Master keys are org-scoped bearer tokens that let you create projects, gateway API keys, IAM rules (both per key and per organization member), and your organization's custom providers and custom models programmatically — without going through the dashboard. They are intended for server-to-server provisioning (e.g. multi-tenant onboarding from your own backend).

They also expose [usage and cost reporting](#usage-and-cost-reporting), so you can pull per-member and per-model spend into your own dashboards, data warehouse, or chargeback process.

<Callout type="info">
  Master keys are available on the **Enterprise** plan only. Contact us at
  [contact@offrail.ai](mailto:contact@offrail.ai) to enable them for your organization.
</Callout>

## Security [#security]

* Master keys are stored as **HMAC-SHA256 hashes** in the database (using the `GATEWAY_API_KEY_HASH_SECRET` secret). The plain token is shown to you **only once** at creation time.
* Each master key is scoped to a single organization and cannot access resources in other organizations.
* Deleting or deactivating a master key revokes all programmatic access immediately.
* All creates/deletes/status changes are recorded in your organization audit log.

## Limits [#limits]

* Maximum **10 active master keys per organization**.
* Programmatic project and API-key creation enforces the same per-org and per-project limits as the dashboard flow.

## Managing master keys [#managing-master-keys]

In the dashboard, go to **Organization → Master Keys**. From there you can:

* Create a new master key (the plain token is shown once — copy it immediately).
* View the masked token, status, creator, and last-used timestamp for each existing key.
* Activate / deactivate or delete keys.

## Authentication [#authentication]

All programmatic endpoints live under `/v1/master/*` and require a master key in the `Authorization` header:

```
Authorization: Bearer orlmk_...
```

A request with a missing, invalid, inactive, or non-enterprise master key receives a 401 / 403 response.

## Endpoints [#endpoints]

### List projects [#list-projects]

`GET /v1/master/projects`

Returns all non-deleted projects in the master key's organization.

```bash
curl https://internal.offrail.ai/v1/master/projects \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"projects": [
		{
			"id": "proj_...",
			"name": "Customer ACME",
			"organizationId": "org_...",
			"cachingEnabled": false,
			"cacheDurationSeconds": 60,
			"mode": "hybrid",
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Create a project [#create-a-project]

`POST /v1/master/projects`

```bash
curl -X POST https://internal.offrail.ai/v1/master/projects \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer ACME",
    "cachingEnabled": false,
    "mode": "hybrid"
  }'
```

Body parameters:

| Field                  | Type                                             | Description                |
| ---------------------- | ------------------------------------------------ | -------------------------- |
| `name`                 | string                                           | Project name (1–255 chars) |
| `cachingEnabled`       | boolean (optional)                               | Default `false`            |
| `cacheDurationSeconds` | number (optional)                                | 10–31536000, default 60    |
| `mode`                 | `"api-keys" \| "credits" \| "hybrid"` (optional) | Default `"hybrid"`         |

Response (201): the created project.

### Update a project [#update-a-project]

`PATCH /v1/master/projects/{id}`

Updates a project owned by the master key's organization. All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.offrail.ai/v1/master/projects/proj_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer ACME (renamed)",
    "cachingEnabled": true,
    "status": "inactive"
  }'
```

Body parameters (all optional, at least one required):

| Field                  | Type                                  | Description                         |
| ---------------------- | ------------------------------------- | ----------------------------------- |
| `name`                 | string                                | 1–255 chars                         |
| `cachingEnabled`       | boolean                               |                                     |
| `cacheDurationSeconds` | number                                | 10–31536000                         |
| `mode`                 | `"api-keys" \| "credits" \| "hybrid"` |                                     |
| `status`               | `"active" \| "inactive"`              | Toggle the project without deleting |

Response (200): the updated project.

### Delete a project [#delete-a-project]

`DELETE /v1/master/projects/{id}`

Soft-deletes a project (sets `status` to `"deleted"`). Cascades to its API keys.

```bash
curl -X DELETE https://internal.offrail.ai/v1/master/projects/proj_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "Project deleted successfully" }
```

### List gateway API keys [#list-gateway-api-keys]

`GET /v1/master/keys`

Returns the developer-created gateway API keys in the master key's organization, each with its configured limits, the usage consumed so far, and — when a windowed limit is set — the time the current period resets. Pass an optional `projectId` query parameter to scope the list to a single project.

```bash
curl "https://internal.offrail.ai/v1/master/keys?projectId=proj_..." \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"apiKeys": [
		{
			"id": "ak_...",
			"description": "Customer ACME — production key",
			"status": "active",
			"projectId": "proj_...",
			"createdBy": "usr_...",
			"maskedToken": "orl_...abcd",
			"usageLimit": "100.00",
			"usage": "42.13",
			"periodUsageLimit": "10.00",
			"periodUsageDurationValue": 1,
			"periodUsageDurationUnit": "day",
			"currentPeriodUsage": "3.50",
			"currentPeriodStartedAt": "2025-01-15T00:00:00.000Z",
			"currentPeriodResetAt": "2025-01-16T00:00:00.000Z",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

Limit and usage fields:

| Field                      | Description                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------- |
| `usageLimit`               | Lifetime spend cap (`null` when uncapped)                                          |
| `usage`                    | Total spend accrued against `usageLimit` over the key's lifetime                   |
| `periodUsageLimit`         | Recurring per-window spend cap (`null` when no windowed limit is configured)       |
| `periodUsageDurationValue` | Length of the window, paired with `periodUsageDurationUnit`                        |
| `periodUsageDurationUnit`  | `"hour" \| "day" \| "week" \| "month"`                                             |
| `currentPeriodUsage`       | Spend accrued in the current window (`"0"` when unconfigured or the window lapsed) |
| `currentPeriodStartedAt`   | When the current window began (`null` when unconfigured or lapsed)                 |
| `currentPeriodResetAt`     | When the windowed limit resets (`null` when unconfigured or lapsed)                |

The plain token is never returned by this endpoint — only a masked form for identification.

### Get a gateway API key [#get-a-gateway-api-key]

`GET /v1/master/keys/{id}`

Returns a single gateway API key in the master key's organization, with the same limit, usage, and reset-time fields as the list endpoint.

```bash
curl https://internal.offrail.ai/v1/master/keys/ak_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"apiKey": {
		"id": "ak_...",
		"description": "Customer ACME — production key",
		"status": "active",
		"projectId": "proj_...",
		"createdBy": "usr_...",
		"maskedToken": "orl_...abcd",
		"usageLimit": "100.00",
		"usage": "42.13",
		"periodUsageLimit": "10.00",
		"periodUsageDurationValue": 1,
		"periodUsageDurationUnit": "day",
		"currentPeriodUsage": "3.50",
		"currentPeriodStartedAt": "2025-01-15T00:00:00.000Z",
		"currentPeriodResetAt": "2025-01-16T00:00:00.000Z",
		"createdAt": "...",
		"updatedAt": "..."
	}
}
```

### Create a gateway API key [#create-a-gateway-api-key]

`POST /v1/master/keys`

```bash
curl -X POST https://internal.offrail.ai/v1/master/keys \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "proj_...",
    "description": "Customer ACME — production key"
  }'
```

Body parameters:

| Field                      | Type                                              | Description                                  |
| -------------------------- | ------------------------------------------------- | -------------------------------------------- |
| `projectId`                | string                                            | Must belong to the master key's organization |
| `description`              | string                                            | API key description (1–255 chars)            |
| `usageLimit`               | string (optional)                                 | Lifetime usage limit                         |
| `periodUsageLimit`         | string (optional)                                 | Recurring period usage limit                 |
| `periodUsageDurationValue` | number (optional)                                 | Required if `periodUsageLimit` is set        |
| `periodUsageDurationUnit`  | `"hour" \| "day" \| "week" \| "month"` (optional) | Required if `periodUsageLimit` is set        |

<Callout type="warning">
  The created gateway API key's plain token is returned in the response **only
  once**. Persist it immediately on your side.
</Callout>

Response (201):

```json
{
	"apiKey": {
		"id": "ak_...",
		"token": "orl_...",
		"description": "Customer ACME — production key",
		"status": "active",
		"projectId": "proj_...",
		"createdBy": "usr_...",
		"createdAt": "...",
		"updatedAt": "..."
	}
}
```

### Update a gateway API key [#update-a-gateway-api-key]

`PATCH /v1/master/keys/{id}`

Updates an API key in a project owned by the master key's organization. All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.offrail.ai/v1/master/keys/ak_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "inactive",
    "usageLimit": "100.00"
  }'
```

Body parameters (all optional, at least one required):

| Field                      | Type                                   | Description                            |
| -------------------------- | -------------------------------------- | -------------------------------------- |
| `description`              | string                                 | 1–255 chars                            |
| `status`                   | `"active" \| "inactive"`               |                                        |
| `usageLimit`               | string \| null                         | Lifetime usage limit (null to clear)   |
| `periodUsageLimit`         | string \| null                         | Recurring period limit (null to clear) |
| `periodUsageDurationValue` | number \| null                         | Required if `periodUsageLimit` is set  |
| `periodUsageDurationUnit`  | `"hour" \| "day" \| "week" \| "month"` | Required if `periodUsageLimit` is set  |

Response (200): the updated API key, including its configured limits, consumed `usage` / `currentPeriodUsage`, and the `currentPeriodResetAt` window-reset time (same fields as the [list endpoint](#list-gateway-api-keys)). The plain token is **not** included — it is only returned at creation.

<Callout type="info">
  Two dashboard-only key features have no master API equivalent yet: [expiration
  (TTL)](https://docs.offrail.ai/features/api-keys#expiration-ttl) and [rolling a key's
  secret](https://docs.offrail.ai/features/api-keys#rotating-rolling-api-keys). To replace a key
  programmatically, create a new one and delete the old one once your clients
  have switched over.
</Callout>

### Delete a gateway API key [#delete-a-gateway-api-key]

`DELETE /v1/master/keys/{id}`

Soft-deletes the API key (sets `status` to `"deleted"`). Any in-flight requests using the key will be rejected immediately on next auth check.

```bash
curl -X DELETE https://internal.offrail.ai/v1/master/keys/ak_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "API key deleted successfully" }
```

<Callout type="info">
  The auto-generated Lounge (playground) API key cannot be deleted via the
  master API.
</Callout>

## Usage and cost reporting [#usage-and-cost-reporting]

`GET /v1/master/usage`

Returns usage and cost for the master key's organization as a flat list of rows, grouped by any combination of member, model, provider, project, and API key, and bucketed hourly, daily, or not at all. This is the endpoint to point an internal reporting tool, data warehouse, or chargeback job at — it is the only usage surface that authenticates with a token rather than a dashboard session.

<Callout type="info">
  Like every `/v1/master/*` endpoint, this requires an **Enterprise** plan. A
  master key on any other plan receives a `403`.
</Callout>

```bash
# Cost per member for a month
curl -G https://internal.offrail.ai/v1/master/usage \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d from=2026-07-01 -d to=2026-07-31 \
  -d granularity=total -d groupBy=user
```

Query parameters:

| Param         | Type                                                              | Default      | Description                                                                              |
| ------------- | ----------------------------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------- |
| `from`, `to`  | `YYYY-MM-DD`                                                      | last 7 days  | Inclusive window, interpreted as wall-clock days in `timezone`. Max 366 days             |
| `timezone`    | IANA timezone                                                     | `UTC`        | Timezone the day/hour buckets are labelled in                                            |
| `granularity` | `hour` \| `day` \| `total`                                        | `day`        | Time bucket. `total` collapses the window into one row per group. `hour` caps at 31 days |
| `groupBy`     | comma-separated: `user`, `model`, `provider`, `project`, `apiKey` | `user,model` | Dimensions to break down by. Pass an empty value for organization-wide totals            |
| `projectId`   | string                                                            | —            | Restrict to one project. `404` if it is not in this organization                         |
| `userId`      | string                                                            | —            | Restrict to one member                                                                   |
| `apiKeyId`    | string                                                            | —            | Restrict to one gateway API key                                                          |
| `limit`       | 1–10000                                                           | `1000`       | Rows per page                                                                            |
| `offset`      | ≥ 0                                                               | `0`          | Row offset for paging                                                                    |
| `format`      | `json` \| `csv`                                                   | `json`       | `csv` returns a `text/csv` attachment with a fixed header                                |

Response (200):

```json
{
	"from": "2026-07-01",
	"to": "2026-07-31",
	"granularity": "day",
	"groupBy": ["user", "model"],
	"rows": [
		{
			"date": "2026-07-01",
			"userId": "usr_...",
			"userName": "Ada Lovelace",
			"userEmail": "ada@example.com",
			"projectId": null,
			"projectName": null,
			"apiKeyId": null,
			"apiKeyName": null,
			"model": "gpt-5.6",
			"provider": "openai",
			"requestCount": 128,
			"errorCount": 2,
			"inputTokens": 481203,
			"outputTokens": 92844,
			"totalTokens": 574047,
			"cachedTokens": 120000,
			"reasoningTokens": 18320,
			"cost": 3.4127,
			"inputCost": 2.1,
			"outputCost": 1.3127,
			"creditsRequestCount": 128,
			"apiKeysRequestCount": 0,
			"creditsCost": 3.4127,
			"apiKeysCost": 0
		}
	],
	"pagination": { "limit": 1000, "offset": 0, "hasMore": false }
}
```

Every row carries the full column set. Dimension fields you did not group by are `null`, and `date` is `null` when `granularity=total`, so the row shape stays stable for a schema-driven consumer. `creditsCost` / `apiKeysCost` split the blended `cost` into credit-billed and BYOK traffic.

### Per-member, per-model breakdown [#per-member-per-model-breakdown]

Combine the two dimensions to get the cross-tab most reporting tools want — one row per member, model, and day:

```bash
curl -G https://internal.offrail.ai/v1/master/usage \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d from=2026-07-01 -d to=2026-07-31 \
  -d granularity=day -d groupBy=user,model \
  -d timezone=Europe/Berlin
```

### CSV export [#csv-export]

```bash
curl -G https://internal.offrail.ai/v1/master/usage \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d from=2026-07-01 -d to=2026-07-31 \
  -d granularity=day -d groupBy=user,model -d format=csv \
  -o usage-july.csv
```

The CSV header is the same fixed column set regardless of the dimensions requested, so a scheduled export never changes shape.

<Callout type="warn">
  **Usage is attributed to the member who created the API key** (there is no
  per-caller identity on an inference request). A key shared by several people
  reports entirely under its creator. For accurate per-person reporting, issue
  one gateway API key per member — the per-member key limit under **Organization
  → Members** is the lever that enforces it.
</Callout>

Two more things worth knowing:

* These figures come from hourly rollups, not the request log, so they are **not** affected by your [data retention](https://docs.offrail.ai/features/data-retention) setting — per-member and per-model history stays available even with retention turned off.
* Traffic from [Payments SDK](https://docs.offrail.ai/learn/sdk-settings) end-user keys rolls up to the member who provisioned the platform key.

## IAM rules [#iam-rules]

Each gateway API key can have one or more IAM rules that restrict which models, providers, or pricing tiers it is allowed to use. Rules are evaluated at request time by the gateway. A key with no active rules has no IAM restrictions.

Rule types:

| `ruleType`        | Description                                                 |
| ----------------- | ----------------------------------------------------------- |
| `allow_models`    | Only the listed models are permitted                        |
| `deny_models`     | The listed models are blocked                               |
| `allow_providers` | Only the listed providers are permitted                     |
| `deny_providers`  | The listed providers are blocked                            |
| `allow_pricing`   | Only models matching the pricing constraint are permitted   |
| `deny_pricing`    | Models matching the pricing constraint are blocked          |
| `allow_ip_cidrs`  | Only requests from the listed IPv4/IPv6 CIDRs are permitted |
| `deny_ip_cidrs`   | Requests from the listed IPv4/IPv6 CIDRs are blocked        |

The `ruleValue` JSON object holds the rule's parameters. The fields it accepts depend on the `ruleType`:

| Field            | Type               | Used by                             |
| ---------------- | ------------------ | ----------------------------------- |
| `models`         | string\[]          | `allow_models`, `deny_models`       |
| `providers`      | string\[]          | `allow_providers`, `deny_providers` |
| `pricingType`    | `"free" \| "paid"` | `allow_pricing`, `deny_pricing`     |
| `maxInputPrice`  | number             | `allow_pricing`, `deny_pricing`     |
| `maxOutputPrice` | number             | `allow_pricing`, `deny_pricing`     |
| `ipCidrs`        | string\[]          | `allow_ip_cidrs`, `deny_ip_cidrs`   |

### IP CIDR rules [#ip-cidr-rules]

IP CIDR rules restrict gateway requests by source IP. Both IPv4 (e.g. `192.0.2.0/24`) and IPv6 (e.g. `2001:db8::/32`) ranges are supported, and you can mix both in a single rule. To restrict to a single address, use a `/32` (IPv4) or `/128` (IPv6) prefix.

The gateway reads the client IP from the first entry in the `X-Forwarded-For` header, which is set by the GCP load balancer.

IPv4-mapped IPv6 addresses (`::ffff:1.2.3.4`) are normalized to IPv4 so a single `1.2.3.0/24` rule still matches when the upstream connection happens to be IPv6.

When an `allow_ip_cidrs` rule is configured and the gateway cannot determine the client IP, the request is denied. Invalid CIDR syntax is rejected at rule-creation time with a `400` error.

All endpoints scope by the master key's organization: a `404` is returned if the API key (or rule) is not part of the authenticated master key's organization.

### List IAM rules [#list-iam-rules]

`GET /v1/master/keys/{id}/iam`

```bash
curl https://internal.offrail.ai/v1/master/keys/ak_.../iam \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"rules": [
		{
			"id": "iam_...",
			"apiKeyId": "ak_...",
			"ruleType": "allow_models",
			"ruleValue": {
				"models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"]
			},
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Create an IAM rule [#create-an-iam-rule]

`POST /v1/master/keys/{id}/iam`

```bash
curl -X POST https://internal.offrail.ai/v1/master/keys/ak_.../iam \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ruleType": "allow_models",
    "ruleValue": {
      "models": ["openai/gpt-4o", "anthropic/claude-3-5-sonnet"]
    }
  }'
```

Body parameters:

| Field       | Type                     | Description                                             |
| ----------- | ------------------------ | ------------------------------------------------------- |
| `ruleType`  | rule type enum (above)   | Required                                                |
| `ruleValue` | object (see table above) | Must include the fields appropriate for the chosen type |
| `status`    | `"active" \| "inactive"` | Optional, defaults to `"active"`                        |

Restricting by source IP:

```bash
curl -X POST https://internal.offrail.ai/v1/master/keys/ak_.../iam \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ruleType": "allow_ip_cidrs",
    "ruleValue": {
      "ipCidrs": ["192.0.2.0/24", "2001:db8::/32"]
    }
  }'
```

Response (201): the created IAM rule.

### Update an IAM rule [#update-an-iam-rule]

`PATCH /v1/master/keys/{id}/iam/{ruleId}`

All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.offrail.ai/v1/master/keys/ak_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "inactive"
  }'
```

Body parameters (all optional, at least one required):

| Field       | Type                     | Description                             |
| ----------- | ------------------------ | --------------------------------------- |
| `ruleType`  | rule type enum (above)   | Change the rule type                    |
| `ruleValue` | object (see table above) | Replace the rule value                  |
| `status`    | `"active" \| "inactive"` | Activate or deactivate without deleting |

Response (200): the updated IAM rule.

### Delete an IAM rule [#delete-an-iam-rule]

`DELETE /v1/master/keys/{id}/iam/{ruleId}`

Permanently removes an IAM rule from the API key.

```bash
curl -X DELETE https://internal.offrail.ai/v1/master/keys/ak_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "IAM rule deleted successfully" }
```

## Member IAM rules [#member-iam-rules]

The same rule types can be applied to an **organization member** instead of a single API key. Member-level rules are an organization-wide ceiling: a request must pass both the member's rules and the key's rules, so key rules can only narrow access further, never expand it. They apply to all regular API keys created by that member. See [Member-Level IAM Rules](https://docs.offrail.ai/features/api-keys#member-level-iam-rules) for the full semantics.

The `{member}` path parameter accepts either the **membership id** or the member's **email address** (matched case-insensitively). Email references must resolve to a user who is a member of the master key's organization — an email belonging to a user outside the organization, or an unknown reference, returns a `404`.

The `ruleType`, `ruleValue`, and `status` fields are identical to the per-key IAM endpoints above.

### List member IAM rules [#list-member-iam-rules]

`GET /v1/master/members/{member}/iam`

```bash
curl https://internal.offrail.ai/v1/master/members/jane@example.com/iam \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"rules": [
		{
			"id": "iam_...",
			"userOrganizationId": "uo_...",
			"ruleType": "allow_providers",
			"ruleValue": {
				"providers": ["openai"]
			},
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Create a member IAM rule [#create-a-member-iam-rule]

`POST /v1/master/members/{member}/iam`

```bash
curl -X POST https://internal.offrail.ai/v1/master/members/jane@example.com/iam \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "ruleType": "allow_providers",
    "ruleValue": {
      "providers": ["openai"]
    }
  }'
```

Response (201): the created member IAM rule.

### Update a member IAM rule [#update-a-member-iam-rule]

`PATCH /v1/master/members/{member}/iam/{ruleId}`

All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.offrail.ai/v1/master/members/uo_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "status": "inactive"
  }'
```

Response (200): the updated member IAM rule.

### Delete a member IAM rule [#delete-a-member-iam-rule]

`DELETE /v1/master/members/{member}/iam/{ruleId}`

```bash
curl -X DELETE https://internal.offrail.ai/v1/master/members/uo_.../iam/iam_... \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{ "message": "Member IAM rule deleted successfully" }
```

## Custom providers [#custom-providers]

Custom providers are your own OpenAI-compatible endpoints, registered as BYOK provider keys with `provider: "custom"`. The gateway routes to them with the model string `custom/<name>/<model>`. See [Custom Providers](https://docs.offrail.ai/features/custom-providers) for the dashboard flow and routing semantics.

Only custom providers are exposed through the master API — catalog providers (OpenAI, Anthropic, …) require an interactive upstream credential check and must be added from the dashboard.

All endpoints scope by the master key's organization: a `404` is returned if the provider key is not a non-deleted custom provider in that organization.

<Callout type="warning">
  The provider token is stored but never returned. Responses only include a
  `maskedToken` for identification.
</Callout>

### List custom providers [#list-custom-providers]

`GET /v1/master/custom-providers`

```bash
curl https://internal.offrail.ai/v1/master/custom-providers \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"customProviders": [
		{
			"id": "pk_...",
			"provider": "custom",
			"name": "acme",
			"baseUrl": "https://llm.acme.example.com/v1",
			"maskedToken": "sk-a...cret",
			"status": "active",
			"customModelsOnly": true,
			"complianceAttestation": null,
			"organizationId": "org_...",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Get a custom provider [#get-a-custom-provider]

`GET /v1/master/custom-providers/{id}`

Returns a single custom provider in the master key's organization.

### Create a custom provider [#create-a-custom-provider]

`POST /v1/master/custom-providers`

```bash
curl -X POST https://internal.offrail.ai/v1/master/custom-providers \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "acme",
    "baseUrl": "https://llm.acme.example.com/v1",
    "token": "sk-acme-...",
    "customModelsOnly": true
  }'
```

Body parameters:

| Field                   | Type               | Description                                                                                  |
| ----------------------- | ------------------ | -------------------------------------------------------------------------------------------- |
| `name`                  | string             | Lowercase letters and single hyphens only. Unique per organization; used in the model string |
| `baseUrl`               | string             | OpenAI-compatible base URL. Internal/reserved addresses are rejected                         |
| `token`                 | string             | Upstream API key. Stored server-side, never returned                                         |
| `customModelsOnly`      | boolean (optional) | Restrict the provider to models defined in your custom catalog. Default `false`              |
| `complianceAttestation` | object (optional)  | Self-attested compliance posture. `attestedAt` / `attestedByUserId` are stamped server-side  |

Response (201): the created custom provider.

### Update a custom provider [#update-a-custom-provider]

`PATCH /v1/master/custom-providers/{id}`

All body fields are optional; provide only the ones you want to change.

```bash
curl -X PATCH https://internal.offrail.ai/v1/master/custom-providers/pk_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "baseUrl": "https://llm2.acme.example.com/v1",
    "token": "sk-acme-rotated-..."
  }'
```

Body parameters (all optional, at least one required):

| Field                   | Type                     | Description                                    |
| ----------------------- | ------------------------ | ---------------------------------------------- |
| `baseUrl`               | string                   | Repoint the provider at a new endpoint         |
| `token`                 | string                   | Rotate the upstream API key                    |
| `status`                | `"active" \| "inactive"` | Disable the provider without deleting it       |
| `customModelsOnly`      | boolean                  | Toggle the custom-catalog restriction          |
| `complianceAttestation` | object \| null           | Replace the attestation, or `null` to clear it |

The provider `name` is immutable — it is the routing segment your model strings reference. Create a new provider instead of renaming.

Response (200): the updated custom provider.

### Delete a custom provider [#delete-a-custom-provider]

`DELETE /v1/master/custom-providers/{id}`

Soft-deletes the provider (sets `status` to `"deleted"`). Its custom models stop resolving immediately.

Response (200):

```json
{ "message": "Custom provider deleted successfully" }
```

## Custom models [#custom-models]

Custom models are your organization's catalogue entries for a custom provider: the context window, output limit, per-token pricing, and capability flags the gateway uses to route, validate, and bill requests to that model. When a provider has `customModelsOnly: true`, only models in this catalogue can be called through it.

Query these endpoints to read the full catalogue you have registered — including `contextSize`, `maxOutput`, and every price field — for surfacing in your own dashboards or cost tooling.

<Callout type="info">
  Per-token prices are strings in USD per token. Use `e-6` notation so the
  coefficient reads directly as USD per million tokens — `"3.0e-6"` is $3.00/M.
</Callout>

### List custom models [#list-custom-models]

`GET /v1/master/custom-models`

Returns the non-deleted custom models in the master key's organization. Pass an optional `providerKeyId` query parameter to scope the list to a single custom provider.

```bash
curl "https://internal.offrail.ai/v1/master/custom-models?providerKeyId=pk_..." \
  -H "Authorization: Bearer $MASTER_KEY"
```

Response (200):

```json
{
	"customModels": [
		{
			"id": "cm_...",
			"providerKeyId": "pk_...",
			"organizationId": "org_...",
			"modelName": "acme-large",
			"displayName": "Acme Large",
			"contextSize": 200000,
			"maxOutput": 32000,
			"inputPrice": "3.0e-6",
			"outputPrice": "15.0e-6",
			"cachedInputPrice": "0.3e-6",
			"streaming": "true",
			"vision": true,
			"tools": true,
			"reasoning": null,
			"jsonOutput": true,
			"supportedParameters": ["temperature", "top_p"],
			"status": "active",
			"createdAt": "...",
			"updatedAt": "..."
		}
	]
}
```

### Get a custom model [#get-a-custom-model]

`GET /v1/master/custom-models/{id}`

Returns a single custom model in the master key's organization.

### Create a custom model [#create-a-custom-model]

`POST /v1/master/custom-models`

```bash
curl -X POST https://internal.offrail.ai/v1/master/custom-models \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "providerKeyId": "pk_...",
    "modelName": "acme-large",
    "displayName": "Acme Large",
    "contextSize": 200000,
    "maxOutput": 32000,
    "inputPrice": "3.0e-6",
    "outputPrice": "15.0e-6",
    "tools": true
  }'
```

Body parameters:

| Field                    | Type                                | Description                                                |
| ------------------------ | ----------------------------------- | ---------------------------------------------------------- |
| `providerKeyId`          | string                              | Must be a custom provider in the master key's organization |
| `modelName`              | string                              | Model id sent upstream. Unique per provider                |
| `displayName`            | string (optional)                   | Human-readable label                                       |
| `contextSize`            | number (optional)                   | Context window in tokens                                   |
| `maxOutput`              | number (optional)                   | Max output tokens                                          |
| `inputPrice`             | string (optional)                   | USD per input token                                        |
| `outputPrice`            | string (optional)                   | USD per output token                                       |
| `cachedInputPrice`       | string (optional)                   | USD per cached input token                                 |
| `cacheReadInputPrice`    | string (optional)                   | USD per cache-read token                                   |
| `cacheWriteInputPrice`   | string (optional)                   | USD per cache-write token (5m TTL)                         |
| `cacheWriteInputPrice1h` | string (optional)                   | USD per cache-write token (1h TTL)                         |
| `requestPrice`           | string (optional)                   | Flat USD charged per request                               |
| `webSearchPrice`         | string (optional)                   | USD per web search                                         |
| `imageInputPrice`        | string (optional)                   | USD per image input token                                  |
| `audioInputPrice`        | string (optional)                   | USD per audio input token                                  |
| `streaming`              | `"true" \| "false" \| "only"`       | Streaming support (`"only"` = streaming-only model)        |
| `vision`                 | boolean (optional)                  | Accepts image input                                        |
| `tools`                  | boolean (optional)                  | Supports tool calling                                      |
| `reasoning`              | boolean (optional)                  | Emits reasoning output                                     |
| `jsonOutput`             | boolean (optional)                  | Supports JSON / structured output                          |
| `audio`                  | boolean (optional)                  | Accepts audio input                                        |
| `supportedParameters`    | string\[] (optional)                | Request parameters the upstream accepts                    |
| `status`                 | `"active" \| "inactive"` (optional) | Defaults to `"active"`                                     |

Response (201): the created custom model.

### Update a custom model [#update-a-custom-model]

`PATCH /v1/master/custom-models/{id}`

Accepts the same fields as create (except `providerKeyId`), all optional — provide only the ones you want to change. Renaming to a `modelName` already used by another model on the same provider returns a `400`.

```bash
curl -X PATCH https://internal.offrail.ai/v1/master/custom-models/cm_... \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "contextSize": 400000, "outputPrice": "12.0e-6" }'
```

Response (200): the updated custom model.

### Delete a custom model [#delete-a-custom-model]

`DELETE /v1/master/custom-models/{id}`

Soft-deletes the model (sets `status` to `"deleted"`).

Response (200):

```json
{ "message": "Custom model deleted successfully" }
```


# Metadata
URL: https://docs.offrail.ai/features/metadata


# Metadata [#metadata]

OffRail supports sending additional metadata with your requests using custom headers. This allows you to include information like user sessions, application versions, tenant IDs, or other contextual data that can be useful for analytics and monitoring.

Later, you can filter by specific values to return, such as for a specific user or session. Additionally, in the future, you will be able to segment your analytics and monitoring based on this metadata. For example, you could show cost and latency breakdowns per user, application, country, feature, or any other dimension you want to track.

## Custom Headers [#custom-headers]

You can include custom headers with the `X-OffRail-` prefix to send metadata alongside your LLM requests:

```bash
curl -X POST https://api.offrail.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "X-OffRail-Country: US" \
  -H "X-OffRail-User-ID: 9403f741-a524-4b18-b1b2-dbb71cdff2a4" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'
```

## Best Practices [#best-practices]

### Header Naming [#header-naming]

* Use the `X-OffRail-` prefix for all custom metadata
* Use descriptive, consistent naming conventions
* Avoid special characters; use hyphens to separate words

### Data Privacy [#data-privacy]

* Be mindful of sensitive data in headers
* Consider hashing or anonymizing user identifiers
* Follow your organization's data privacy policies

### Performance [#performance]

* Keep header values reasonably short
* Avoid sending unnecessary metadata that won't be used for analytics
* Consider the impact on request size, especially for high-volume applications

## Example: Multi-tenant Application [#example-multi-tenant-application]

For a multi-tenant application, you might use metadata headers like this:

```bash
curl -X POST https://api.offrail.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "X-OffRail-Tenant-ID: acme-corp" \
  -H "X-OffRail-User-ID: user-12345" \
  -H "X-OffRail-App-Version: 2.1.4" \
  -H "X-OffRail-Feature: chat-assistant" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Summarize this document..."
      }
    ]
  }'
```

This allows you to track usage and costs per tenant, user, application version, and feature, providing detailed insights into how your LLM integration is being used across your platform.


# Org Models Directory
URL: https://docs.offrail.ai/features/models-directory


# Org Models Directory [#org-models-directory]

The **Models** page in the dashboard (`Organization → Models`) lists every model your organization can route to in a single directory: the full OffRail catalogue plus the models defined for your own [custom providers](https://docs.offrail.ai/features/custom-providers). It answers two questions for your whole team:

* **What can we call?** — every catalogue and custom model, searchable and filterable by capability, provider, price, and context size, exactly like the public [models page](https://offrail.ai/models).
* **What are we allowed to call?** — when a [provider compliance policy](https://docs.offrail.ai/features/compliance) is active, each provider mapping is marked eligible or blocked using the same fail-closed evaluation the gateway applies at request time.

## Who can see it [#who-can-see-it]

Every active organization member can browse the directory read-only — including project-scoped **developer** members, who see it as their only organization page. This is the recommended way for developers to discover available providers and models without being granted additional permissions. Management actions (custom-model catalog, attestations) remain restricted to organization owners and admins.

## Custom models in the directory [#custom-models-in-the-directory]

Custom models appear alongside catalogue models, addressed as `{customProvider}/{modelName}` — the exact model string to send to the gateway:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "internal-vllm/llama-4-maverick",
    "messages": [{ "role": "user", "content": "Hello!" }]
  }'
```

Their pricing, context and output limits, and capability flags come from the [custom model catalog](https://docs.offrail.ai/features/custom-providers#custom-model-catalog). A **Source** filter switches between catalogue models, custom models, or both.

## Compliance eligibility [#compliance-eligibility]

With an enabled compliance policy (Enterprise), the directory evaluates every provider mapping against the policy — certifications, data-policy requirements, country restrictions, and provider/model allow and deny lists:

* Blocked mappings render greyed out with a ban icon; its tooltip lists the exact failing requirements (for example "No SOC 2 Type 2 report" or "Not on the allowed-providers list").
* Custom providers are evaluated against their [compliance attestation](https://docs.offrail.ai/features/compliance#custom-providers); without one on file they are blocked while a policy is active.
* An **Eligible only** filter narrows the directory to models the gateway will actually route to.

<Callout type="info">
  The directory mirrors gateway enforcement — the same predicates decide both
  the markers on this page and the 403 responses at request time. If a model is
  marked blocked here, requests for it will be rejected while the policy is
  active.
</Callout>

## Related [#related]

* [Knowledge base → Models](https://docs.offrail.ai/learn/models) — a walkthrough of the page itself.
* [Custom Providers](https://docs.offrail.ai/features/custom-providers) — bring your own OpenAI-compatible endpoints and define their model catalog.
* [Compliance](https://docs.offrail.ai/features/compliance) — configure the provider compliance policy.


# Moderations
URL: https://docs.offrail.ai/features/moderations


# Moderations [#moderations]

OffRail supports the OpenAI-compatible `/v1/moderations` endpoint for text
and multimodal safety classification.

Use it when you want to:

* Screen user prompts before they reach a model
* Review generated output before displaying it
* Apply the same moderation API shape you already use with OpenAI clients

For the full request and response schema, see the
[API reference](https://docs.offrail.ai/v1_moderations).

## Endpoint [#endpoint]

`POST https://api.offrail.ai/v1/moderations`

Authenticate with your OffRail API key:

```bash
-H "Authorization: Bearer $OFFRAIL_API_KEY"
```

## Supported Inputs [#supported-inputs]

The `input` field accepts:

* A single string
* An array of strings
* An array of multimodal content items with `text` and `image_url`

The default model is `omni-moderation-latest`.

## Pricing [#pricing]

Starting **Friday, August 7, 2026*&#x2A;, moderation requests are billed at a
flat **$0.00001 per request**, independent of the input size, the number of
inputs, or the moderation model you pick. Only successful requests are billed —
failed and retried attempts cost nothing. Before that date, moderation requests
are free.

Because the endpoint becomes paid, it also starts requiring a credit balance
from that date: moderation requests from an organization with no credits are
rejected with `402`. Top up before August 7 to avoid an interruption.

When your project runs in `api-keys` mode and the request is served with your
own OpenAI key, no credits are deducted and no balance is required.

## curl [#curl]

### Single text input [#single-text-input]

```bash
curl -X POST "https://api.offrail.ai/v1/moderations" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "I want to harm someone."
  }'
```

### Multiple text inputs [#multiple-text-inputs]

```bash
curl -X POST "https://api.offrail.ai/v1/moderations" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "omni-moderation-latest",
    "input": [
      "This is a harmless sentence.",
      "I want to attack somebody."
    ]
  }'
```

### Multimodal input [#multimodal-input]

```bash
curl -X POST "https://api.offrail.ai/v1/moderations" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": [
      {
        "type": "text",
        "text": "Check this image for violent content."
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "https://example.com/image.png"
        }
      }
    ]
  }'
```

## OpenAI SDK [#openai-sdk]

```ts
import OpenAI from "openai";

const client = new OpenAI({
	baseURL: "https://api.offrail.ai/v1",
	apiKey: process.env.OFFRAIL_API_KEY,
});

const response = await client.moderations.create({
	model: "omni-moderation-latest",
	input: "I want to harm someone.",
});

console.log(response.results[0]?.flagged);
```

## Response Shape [#response-shape]

The response follows the standard OpenAI moderation format:

```json
{
	"id": "modr-123",
	"model": "omni-moderation-latest",
	"results": [
		{
			"flagged": true,
			"categories": {
				"violence": true,
				"self_harm": false
			},
			"category_scores": {
				"violence": 0.98,
				"self_harm": 0.01
			}
		}
	]
}
```

## When To Use This Instead Of Chat Content Filtering [#when-to-use-this-instead-of-chat-content-filtering]

Use `/v1/moderations` when you want an explicit moderation decision in your own
application flow.

If you want moderation to happen automatically as part of model requests, use
OffRail content filtering on `/v1/chat/completions` instead.


# OCR
URL: https://docs.offrail.ai/features/ocr


# OCR [#ocr]

OffRail exposes a dedicated `/v1/ocr` endpoint for optical character
recognition. It extracts text, tables, and layout from PDFs and images and
returns them as clean markdown, one entry per page.

Use it when you want to:

* Turn scanned PDFs or photos into machine-readable markdown
* Pull structured text out of receipts, invoices, forms, or screenshots
* Feed document contents into a downstream model or RAG pipeline

For the full request and response schema, see the
[API reference](https://docs.offrail.ai/v1_ocr).

## Endpoint [#endpoint]

`POST https://api.offrail.ai/v1/ocr`

Authenticate with your OffRail API key:

```bash
-H "Authorization: Bearer $OFFRAIL_API_KEY"
```

The current model is `mistral-ocr-latest&#x60;, billed at **$4 per 1,000 pages**
processed.

## Document Input [#document-input]

The `document` field accepts either a document URL (PDF) or an image:

* `{ "type": "document_url", "document_url": "https://…/file.pdf" }`
* `{ "type": "image_url", "image_url": "https://…/image.png" }`

Both `document_url` and `image_url` accept a public URL or a base64 data URL
(`data:application/pdf;base64,…` / `data:image/png;base64,…`). The `image_url`
field may also be passed as an object: `{ "url": "…" }`.

### Scoping pages [#scoping-pages]

By default the entire document is processed and every page is billed. Use the
optional `pages` field to restrict (and cap the cost of) a request:

* A list of zero-based indices: `"pages": [0, 1, 2]`
* A range string: `"pages": "0-4"`

## curl [#curl]

### Document URL [#document-url]

```bash
curl -X POST "https://api.offrail.ai/v1/ocr" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-ocr-latest",
    "document": {
      "type": "document_url",
      "document_url": "https://arxiv.org/pdf/2201.04234"
    }
  }'
```

### Only specific pages [#only-specific-pages]

```bash
curl -X POST "https://api.offrail.ai/v1/ocr" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-ocr-latest",
    "document": {
      "type": "document_url",
      "document_url": "https://arxiv.org/pdf/2201.04234"
    },
    "pages": "0-4"
  }'
```

### Image input [#image-input]

```bash
curl -X POST "https://api.offrail.ai/v1/ocr" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-ocr-latest",
    "document": {
      "type": "image_url",
      "image_url": "https://example.com/receipt.png"
    }
  }'
```

### Inline (base64) document [#inline-base64-document]

```bash
BASE64_PDF=$(base64 -i invoice.pdf)

curl -X POST "https://api.offrail.ai/v1/ocr" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"mistral-ocr-latest\",
    \"document\": {
      \"type\": \"document_url\",
      \"document_url\": \"data:application/pdf;base64,${BASE64_PDF}\"
    }
  }"
```

## Response Shape [#response-shape]

```json
{
	"pages": [
		{
			"index": 0,
			"markdown": "# Document title\n\nExtracted body text…",
			"images": [],
			"dimensions": { "dpi": 200, "height": 2200, "width": 1700 }
		}
	],
	"model": "mistral-ocr-latest",
	"document_annotation": null,
	"usage_info": {
		"pages_processed": 1,
		"doc_size_bytes": 125344
	}
}
```

Each entry in `pages` carries the markdown for one page. `usage_info.pages_processed`
reflects exactly how many pages were billed for the request.

## Billing [#billing]

OCR is billed per page processed, not per token. A request that processes 12
pages bills `12 × $0.004 = $0.048`. Scoping a request with `pages` reduces both
the work and the cost.

## OCR Models Use This Endpoint, Not Chat [#ocr-models-use-this-endpoint-not-chat]

OCR models are not chat models and cannot be called through
`/v1/chat/completions` — doing so returns a `400` pointing you here. Send OCR
requests to `/v1/ocr`.


# Realtime API
URL: https://docs.offrail.ai/features/realtime


# Realtime API [#realtime-api]

OffRail supports low-latency, speech-to-speech conversations through the
OpenAI-compatible &#x2A;*`/v1/realtime`** WebSocket endpoint. Sessions support text
and audio input/output, server-side voice activity detection (VAD), input
audio transcription, and function calling — using the same event protocol as
the OpenAI Realtime API.

The API is a drop-in replacement: point any OpenAI realtime client at
`wss://api.offrail.ai/v1/realtime`, authenticate with your OffRail API
key, and keep your existing event handling.

<Callout type="info">
  Want to talk to a model right now? The [Realtime
  page](https://lounge.offrail.ai/realtime) in Lounge runs a full voice call in
  the browser using this API.
</Callout>

## Available Models [#available-models]

Browse the available realtime models, with up-to-date pricing, on the
[models page](https://offrail.ai/models). Realtime sessions are billed per
token — text and audio input, cached input, and output are metered separately
at the model's listed rates, matching the provider's own pricing.

Model names work like everywhere else on OffRail: use the plain model id
(e.g. `gpt-realtime`), a dated alias, or the `provider/model` pinned form
(e.g. `openai/gpt-realtime`).

## Connecting [#connecting]

Connect a WebSocket to:

```
wss://api.offrail.ai/v1/realtime?model=gpt-realtime
```

Authenticate with your OffRail API key in the `Authorization` header (an
`x-api-key` header also works):

```javascript
import WebSocket from "ws";

const url = "wss://api.offrail.ai/v1/realtime?model=gpt-realtime";
const ws = new WebSocket(url, {
	headers: {
		Authorization: "Bearer " + process.env.OFFRAIL_API_KEY,
	},
});

ws.on("open", () => {
	console.log("Connected to server.");
});

ws.on("message", (message) => {
	const event = JSON.parse(message.toString());
	console.log(event.type);
});
```

Once connected, the session speaks the standard realtime event protocol:
send client events like `session.update`, `conversation.item.create`,
`input_audio_buffer.append`, and `response.create`; receive server events
like `session.created`, `response.output_audio.delta`, and `response.done`.

```javascript
ws.on("open", () => {
	// Configure the session.
	ws.send(
		JSON.stringify({
			type: "session.update",
			session: {
				type: "realtime",
				instructions: "You are a friendly assistant.",
				audio: {
					output: { voice: "marin" },
				},
			},
		}),
	);

	// Ask for a response.
	ws.send(
		JSON.stringify({
			type: "conversation.item.create",
			item: {
				type: "message",
				role: "user",
				content: [{ type: "input_text", text: "Say hello!" }],
			},
		}),
	);
	ws.send(JSON.stringify({ type: "response.create" }));
});
```

All events are JSON text frames; audio travels base64-encoded inside events
(binary WebSocket frames are rejected). The model is locked at connection
time — a `session.update` that tries to change `session.model` is rejected
with a `model_locked` error event.

<Callout type="warn">
  Credentials are never accepted as query parameters. A connection URL
  containing `token`, `api_key`, or `client_secret` query parameters is rejected
  with HTTP 400. Use the `Authorization` header on servers, or an ephemeral
  client secret (below) in browsers.
</Callout>

## Browser Clients and Client Secrets [#browser-clients-and-client-secrets]

Never ship a long-lived API key to a browser. Instead, mint a short-lived
**ephemeral client secret** from your backend with the OpenAI-compatible
`POST /v1/realtime/client_secrets` endpoint:

```bash
curl -X POST "https://api.offrail.ai/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": { "anchor": "created_at", "seconds": 120 },
    "session": {
      "type": "realtime",
      "model": "gpt-realtime"
    }
  }'
```

The response contains the secret (`ek_...`) and its expiry:

```json
{
	"value": "ek_...",
	"expires_at": 1753500000,
	"session": {
		"type": "realtime",
		"model": "gpt-realtime"
	}
}
```

The browser then connects using the standard `openai-insecure-api-key`
WebSocket subprotocol — no `Authorization` header needed:

```javascript
const ws = new WebSocket("wss://api.offrail.ai/v1/realtime", [
	"realtime",
	"openai-insecure-api-key." + clientSecret,
]);
```

Client secret behavior:

* **TTL**: 10–300 seconds (default 60), set via `expires_after.seconds`.
  Secrets are reusable until they expire; expiry only gates opening the
  connection, not the session's duration.
* **Model pinning**: the secret is minted for one model. The `model` query
  parameter is optional when connecting with a secret; if provided, it must
  match the minted model.
* **Transcription pinning**: optionally pin an input transcription model at
  mint time via `session.audio.input.transcription.model`. The session can
  then only enable that transcription model.
* All authentication, credit, model access, and compliance checks run at
  mint time and again at connection time.

## Input Audio Transcription [#input-audio-transcription]

Enable transcription of the user's audio with `session.update`. An explicit,
supported transcription model is required — check the
[models page](https://offrail.ai/models) for available realtime
transcription models and their pricing:

```json
{
	"type": "session.update",
	"session": {
		"type": "realtime",
		"audio": {
			"input": {
				"transcription": { "model": "gpt-4o-transcribe" }
			}
		}
	}
}
```

Transcripts arrive as standard
`conversation.item.input_audio_transcription.delta` and `.completed` events.

* The provider's implicit default (e.g. `whisper-1`) is not available;
  omitting `transcription.model` is rejected with
  `transcription_model_required`. Duration-billed transcription models cannot
  be metered per token, so only token-metered models are supported.
* The transcription model is pinned for the session on first use and cannot
  be switched afterwards; disabling transcription (`"transcription": null`)
  is always allowed.
* Transcription is billed separately from the realtime model, at the
  transcription model's listed per-token rates.
* Both the current nested form (`session.audio.input.transcription`) and the
  legacy top-level form (`session.input_audio_transcription`) are accepted.

## Function Calling [#function-calling]

Plain function tools work exactly as in the OpenAI Realtime API — declare
them in `session.update` (or per response in `response.create`), receive
`response.function_call_arguments` events, and return results with
`conversation.item.create`:

```json
{
	"type": "session.update",
	"session": {
		"type": "realtime",
		"tools": [
			{
				"type": "function",
				"name": "get_weather",
				"description": "Get the current weather for a location.",
				"parameters": {
					"type": "object",
					"properties": {
						"location": { "type": "string" }
					},
					"required": ["location"]
				}
			}
		],
		"tool_choice": "auto"
	}
}
```

Hosted tools (MCP servers, web search, code interpreter, etc.) are not yet
available and are rejected with `tool_type_not_supported`.

## Billing and Session Gating [#billing-and-session-gating]

Realtime sessions bill your organization's pay-as-you-go credits, or your own
provider key when the project uses [provider keys](https://docs.offrail.ai/learn/provider-keys). Every
model generation — whether triggered by your `response.create` or by
server-side VAD — passes the gateway's authorization gates first (credits,
API key status, usage limits, IAM rules), so a session cannot run past an
exhausted balance. A blocked generation surfaces as a standard `error` event
on the session instead of a response.

Default per-session safety limits:

| Limit                           | Default |
| ------------------------------- | ------- |
| Maximum session duration        | 1 hour  |
| Maximum spend per session       | $10     |
| Concurrent sessions per org     | 20      |
| Concurrent sessions per API key | 10      |

Sessions that hit a limit are closed gracefully after in-flight responses are
billed. Usage appears in your [activity feed](https://offrail.ai/dashboard)
like any other request, including separate line items for input transcription.

## Current Limitations [#current-limitations]

* **WebSocket transport only** — WebRTC and SIP are not yet supported.
* **Image input** is not yet supported in realtime sessions.
* **Hosted tools** (MCP, web search, etc.) and **stored prompt references**
  (`session.prompt`) are not supported; inline your instructions and function
  tools instead.
* Realtime requires a regular developer API key on a pay-as-you-go
  organization. End-user session tokens, platform keys, and chat plan
  organizations are not supported yet.

## Self-Hosting [#self-hosting]

Realtime is disabled by default on self-hosted deployments. Set
`REALTIME_INLINE=true` to attach the `/v1/realtime` WebSocket listener (and
the client-secret mint endpoint) to the gateway process — `pnpm dev` sets
this automatically. Client secrets require Redis. See `.env.example` for the
tunables (session caps, concurrency limits, shutdown grace period).


# Reasoning
URL: https://docs.offrail.ai/features/reasoning


# Reasoning [#reasoning]

OffRail supports reasoning-capable models that can show their step-by-step thought process before providing a final answer. This feature is particularly useful for complex problem-solving tasks, mathematical calculations, and logical reasoning.

## Reasoning-Enabled Models [#reasoning-enabled-models]

You can find all reasoning-enabled models on our [models page with reasoning filter](https://offrail.ai/models?filters=1\&reasoning=true). These models include:

* OpenAI's GPT-5 series (e.g., `gpt-5`, `gpt-5-mini`)
  * Note: GPT-5 models use reasoning but currently do not return the reasoning content in the response.
* Anthropic's Claude 3.7 Sonnet
* Google's Gemini 2.0 Flash Thinking and Gemini 2.5 Pro
* GPT OSS models such as `gpt-oss-120b` and `gpt-oss-20b`
* Z.AI's reasoning models

<Callout type="info">
  Some models may reason internally even if the `reasoning_effort` parameter is
  not specified.
</Callout>

## Using the Reasoning Parameter [#using-the-reasoning-parameter]

There are two ways to control reasoning effort:

### Option 1: Top-level `reasoning_effort` [#option-1-top-level-reasoning_effort]

Add the `reasoning_effort` parameter directly to your request:

* `none` - Disable reasoning. Supported by OpenAI's newer reasoning models (e.g. `gpt-5.4-mini` and later, which accept `none` instead of `minimal`). For other providers this turns reasoning off.
* `minimal` - Fastest reasoning with minimal thought process (only for GPT-5 models)
* `low` - Light reasoning for simpler tasks
* `medium` - Balanced reasoning for most tasks
* `high` - Deep reasoning for complex problems
* `xhigh` - Very deep reasoning for the most complex problems
* `max` - Highest reasoning tier, above `xhigh`. Supported by Anthropic thinking models and OpenAI GPT-5.6 models. Effort tiers are never downgraded by the gateway: providers that accept an effort parameter receive the value unchanged (unsupported values result in a provider error), while providers that take a thinking budget instead (Anthropic, Google, Alibaba) have each tier translated to a native budget

<Callout type="info">
  OpenAI's reasoning models do not all accept the same effort values. The
  original GPT-5 models support `minimal`, while newer models (e.g.
  `gpt-5.4-mini` and later) replace it with `none`. If you send an effort value
  the target model doesn't support, OpenAI returns an `unsupported_value` error.
  The exact values each provider mapping accepts are exposed as
  `reasoning_efforts` on the [`/v1/models`](https://api.offrail.ai/v1/models)
  endpoint.
</Callout>

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "What is 2/3 + 1/4 + 5/6?"
      }
    ],
    "reasoning_effort": "medium"
  }'
```

### Option 2: Using the `reasoning` object [#option-2-using-the-reasoning-object]

Use the unified `reasoning` configuration object with an `effort` field:

* `none` - Disable reasoning
* `minimal` - Fastest reasoning with minimal thought process
* `low` - Light reasoning for simpler tasks
* `medium` - Balanced reasoning for most tasks
* `high` - Deep reasoning for complex problems
* `xhigh` - Very deep reasoning for the most complex problems
* `max` - Highest reasoning tier, above `xhigh`. Supported by Anthropic thinking models and OpenAI GPT-5.6 models. Effort tiers are never downgraded by the gateway: providers that accept an effort parameter receive the value unchanged (unsupported values result in a provider error), while providers that take a thinking budget instead (Anthropic, Google, Alibaba) have each tier translated to a native budget

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {
        "role": "user",
        "content": "What is 2/3 + 1/4 + 5/6?"
      }
    ],
    "reasoning": {
      "effort": "medium"
    }
  }'
```

<Callout type="warning">
  You cannot use both `reasoning_effort` and `reasoning.effort` in the same
  request. Choose one approach. However, you can combine `reasoning_effort` or
  `reasoning.effort` with `reasoning.max_tokens` — when `max_tokens` is
  specified, it takes priority over the effort level.
</Callout>

### Example Response [#example-response]

The response will include a `reasoning` field in the message object containing the model's step-by-step thought process:

```json
{
	"id": "chatcmpl-abc123",
	"object": "chat.completion",
	"created": 1234567890,
	"model": "gpt-oss-120b",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "The answer is 1.75 or 7/4.",
				"reasoning": "First, I need to find a common denominator for 2/3, 1/4, and 5/6. The LCD is 12. Converting: 2/3 = 8/12, 1/4 = 3/12, 5/6 = 10/12. Adding: 8/12 + 3/12 + 10/12 = 21/12 = 1.75 or 7/4."
			},
			"finish_reason": "completed"
		}
	],
	"usage": {
		"prompt_tokens": 20,
		"completion_tokens": 45,
		"reasoning_tokens": 35,
		"total_tokens": 65
	}
}
```

## Specifying Reasoning Token Budget [#specifying-reasoning-token-budget]

For models that support it, you can specify an exact token budget for reasoning using the `reasoning` object with `max_tokens`. This gives you precise control over how many tokens the model allocates to its thinking process.

<Callout type="info">
  When `reasoning.max_tokens` is specified, it overrides `reasoning.effort` and
  `reasoning_effort`. Supported by Anthropic Claude and Google Gemini thinking
  models, plus Alibaba-hosted thinking models (forwarded as DashScope's
  `thinking_budget`).
</Callout>

### Example Request [#example-request]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-4-20250514",
    "messages": [
      {
        "role": "user",
        "content": "Explain the P vs NP problem and why it matters."
      }
    ],
    "reasoning": {
      "max_tokens": 8000
    }
  }'
```

### Supported Models [#supported-models]

The `reasoning.max_tokens` parameter is supported by:

* **Anthropic Claude**: Claude 3.7 Sonnet, Claude Sonnet 4, Claude Opus 4, Claude Opus 4.5
* **Google Gemini**: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 3 Pro Preview

When using auto-routing or canonical models with `reasoning.max_tokens`, only providers that support this feature will be considered.

### Provider-Specific Constraints [#provider-specific-constraints]

* **Anthropic**: Reasoning budget must be between 1,024 and 128,000 tokens. Values outside this range are automatically clamped.
* **Google**: No specific constraints on the reasoning budget.

### Error Handling [#error-handling]

If you specify `reasoning.max_tokens` for a model that doesn't support it, you'll receive an error:

```json
{
	"error": {
		"message": "Model gpt-4o does not support reasoning.max_tokens. Remove the reasoning parameter or use a model that supports explicit reasoning token budgets.",
		"type": "invalid_request_error",
		"code": "model_not_supported"
	}
}
```

## Controlling Response Verbosity [#controlling-response-verbosity]

For OpenAI GPT-5 and later models, you can control how detailed the model's final answer is with the top-level `verbosity` parameter. This is independent of `reasoning_effort`: `reasoning_effort` controls how much the model thinks, while `verbosity` controls how much it writes in its response.

Accepted values:

* `low` - Concise responses with minimal elaboration
* `medium` - Balanced level of detail
* `high` - Detailed, thorough responses

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {
        "role": "user",
        "content": "Explain how a hash map works."
      }
    ],
    "verbosity": "low"
  }'
```

<Callout type="info">
  `verbosity` is only supported by OpenAI GPT-5 and later models. You can check
  which model mappings accept it via the
  [`/v1/models`](https://api.offrail.ai/v1/models) endpoint. It can be combined
  freely with `reasoning_effort` or the `reasoning` object.
</Callout>

### Error Handling [#error-handling-1]

If you specify `verbosity` for a model that doesn't support it, you'll receive a `400` error:

```json
{
	"error": {
		"message": "Model gpt-4o does not support the verbosity parameter. Remove the verbosity parameter or use a model that supports it (OpenAI GPT-5 and later).",
		"type": "invalid_request_error",
		"code": "model_not_supported"
	}
}
```

## Streaming Reasoning Content [#streaming-reasoning-content]

When streaming is enabled, reasoning content will be streamed as part of the response chunks:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {
        "role": "user",
        "content": "Solve this logic puzzle: If all roses are flowers and some flowers fade quickly, can we conclude that some roses fade quickly?"
      }
    ],
    "reasoning_effort": "high",
    "stream": true
  }'
```

The reasoning content will appear in the stream chunks before the final answer, allowing you to display the model's thought process in real-time.

Example:

```
data: {
	"id": "chatcmpl-fb266880-1016-4797-9a70-f21a538edaf6",
	"object": "chat.completion.chunk",
	"created": 1761048126,
	"model": "openai/gpt-oss-20b",
	"choices": [
		{
			"index": 0,
			"delta": {
				"reasoning": "It's ",
				"role": "assistant"
			},
			"finish_reason": null
		}
	]
}
```

## Preserving Reasoning Across Calls [#preserving-reasoning-across-calls]

Agents work over many turns: the model reasons, calls a tool, and the tool result arrives in the next request. Reasoning models are trained to continue from the reasoning they produced on earlier turns, so dropping it mid-task degrades tool-calling and multi-turn performance.

OffRail preserves reasoning the same way OpenAI's Responses API does. The model returns an opaque **encrypted reasoning** payload, and you send it back unchanged along with the rest of the conversation on the next call. The payload is issued and verified by the provider — the gateway passes it through without inspecting or rewriting it.

<Callout type="info">
  Encrypted reasoning is produced by models served over OpenAI's Responses API
  (OpenAI and Azure OpenAI reasoning models). Other reasoning models return
  summary text only, and there is nothing to replay — asking for encrypted
  reasoning is ignored rather than rejected, so the same client code works
  against every model. Gemini thought signatures are handled automatically: the
  gateway re-attaches them to replayed tool calls.
</Callout>

### Responses API [#responses-api]

Ask for the payloads with `include: ["reasoning.encrypted_content"]`, and send `store: false` if you want the turn to stay stateless:

```bash
curl -X POST "https://api.offrail.ai/v1/responses" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4-nano",
    "input": [{ "role": "user", "content": "What is the weather in Paris? Use the tool." }],
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "parameters": {
          "type": "object",
          "properties": { "city": { "type": "string" } },
          "required": ["city"]
        }
      }
    ],
    "store": false,
    "include": ["reasoning.encrypted_content"],
    "reasoning": { "effort": "high" }
  }'
```

Reasoning items in `output` then carry `encrypted_content`:

```json
{
	"type": "reasoning",
	"id": "rs_057d93d1ba22357e01...",
	"summary": [],
	"encrypted_content": "gAAAAABqfkEO3uEFtXeGSNE9..."
}
```

On the next turn, replay the previous `output` items unchanged — reasoning items included — followed by your tool results:

```json
{
	"model": "gpt-5.4-nano",
	"input": [
		{
			"role": "user",
			"content": "What is the weather in Paris? Use the tool."
		},
		{
			"type": "reasoning",
			"id": "rs_057d93d1ba22357e01...",
			"summary": [],
			"encrypted_content": "gAAAAABqfkEO3uEFtXeGSNE9..."
		},
		{
			"type": "function_call",
			"call_id": "call_923JIrDbCvIHnGqfvDbPeaLh",
			"name": "get_weather",
			"arguments": "{\"city\":\"Paris\"}"
		},
		{
			"type": "function_call_output",
			"call_id": "call_923JIrDbCvIHnGqfvDbPeaLh",
			"output": "18C sunny"
		}
	],
	"store": false,
	"include": ["reasoning.encrypted_content"],
	"reasoning": { "effort": "high" }
}
```

Without the `include` value, reasoning items come back without `encrypted_content` and there is nothing to replay. When streaming, the payload arrives on the reasoning item's `response.output_item.done` event.

#### Stateful Alternative [#stateful-alternative]

If you would rather not carry the payloads yourself, leave `store` at its default (`true`) and chain turns with `previous_response_id`. The gateway keeps the stored response — encrypted reasoning included — for [30 days](https://docs.offrail.ai/features/data-retention#retention-periods) and replays the reasoning for you, so `include` is not needed on that path.

### Chat Completions [#chat-completions]

The Chat Completions format has no reasoning items, so the gateway carries the same payloads on the assistant message as `reasoning_details`:

```json
{
	"role": "assistant",
	"content": null,
	"tool_calls": [
		{
			"id": "call_923JIrDbCvIHnGqfvDbPeaLh",
			"type": "function",
			"function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
		}
	],
	"reasoning_details": [
		{
			"type": "reasoning.encrypted",
			"data": "gAAAAABqfkEO3uEFtXeGSNE9...",
			"id": "rs_057d93d1ba22357e01...",
			"format": "openai-responses-v1",
			"index": 0
		}
	]
}
```

To preserve the reasoning, append the assistant message back into `messages` exactly as you received it (with `reasoning_details` intact) before the matching `tool` result. No opt-in parameter is required — the field is always present when the model produced one. When streaming, the entries arrive as a `delta.reasoning_details` chunk.

Replaying is safe across models: entries a provider cannot consume are stripped before the upstream request, so the same conversation history can be sent to any model.

### Error Handling [#error-handling-2]

Payloads are verified by the provider that issued them. A modified, truncated, or foreign payload is rejected upstream and the error is forwarded unchanged:

```json
{
	"error": {
		"message": "The encrypted content for item rs_057d93d1ba22357e01... could not be verified. Reason: Encrypted content could not be decrypted or parsed.",
		"type": "invalid_request_error",
		"code": "invalid_encrypted_content"
	}
}
```

Only replay payloads against the model and provider that produced them.

## Usage Tracking [#usage-tracking]

### Response Payload [#response-payload]

The `usage` object in the response includes reasoning-specific token counts:

* `reasoning_tokens` - Number of tokens used for the reasoning process
* `completion_tokens` - Number of tokens in the final answer
* `prompt_tokens` - Number of tokens in the input
* `total_tokens` - Sum of all token counts

### Logs and Analytics [#logs-and-analytics]

All requests using the `reasoning_effort` parameter are tracked in your dashboard logs with:

* The `reasoningContent` field containing the full reasoning text
* Separate token counts for reasoning vs. completion
* Performance metrics for reasoning-enabled requests

You can view detailed logs for each request in the [dashboard](https://offrail.ai/dashboard) to analyze how models are reasoning through problems.

## Auto-Routing with Reasoning [#auto-routing-with-reasoning]

When using auto-routing (specifying a model like `gpt-5` without a specific version), OffRail will:

1. Automatically set `reasoning_effort` to `minimal` for GPT-5 models
2. Set `reasoning_effort` to `low` for other auto-routed reasoning models
3. Only route to providers that support reasoning when `reasoning_effort` is specified

This ensures optimal performance and cost when using auto-routing with reasoning-capable models.

## Model-Specific Behavior [#model-specific-behavior]

Not all reasoning models return reasoning content in the same way. Some models (like OpenAI models) may reason internally but not expose the reasoning content in the response. OffRail makes sure the response is unified across different providers, but the depth and format of reasoning may vary.

## Best Practices [#best-practices]

1. **Choose appropriate reasoning effort**: Use `low` or `minimal` for simple tasks, `medium` for most tasks, and `high` only for complex problems that require deep reasoning
2. **Monitor token usage**: Reasoning can significantly increase token consumption - monitor your `reasoning_tokens` in the usage object
3. **Stream for better UX**: When building user-facing applications, enable streaming to show the reasoning process in real-time
4. **Check logs**: Review the `reasoningContent` in your dashboard logs to understand how models are solving problems

## Error Handling [#error-handling-3]

If you specify `reasoning_effort` for a model that doesn't support reasoning, you'll receive an error:

```json
{
	"error": {
		"message": "Model gpt-4o does not support reasoning. Remove the reasoning_effort parameter or use a reasoning-capable model.",
		"type": "invalid_request_error",
		"code": "model_not_supported"
	}
}
```

To avoid this error, only use the `reasoning_effort` parameter with [reasoning-enabled models](https://offrail.ai/models?filters=1\&reasoning=true).


# Rerank
URL: https://docs.offrail.ai/features/rerank


# Rerank [#rerank]

OffRail exposes a Cohere-compatible `/v1/rerank` endpoint that scores a list
of candidate documents against a query and returns them ordered by relevance.

Rerankers are cross-encoders: they read the query and a document *together*
rather than embedding each in isolation. That makes them far more accurate than
vector similarity, but too slow to run over a whole corpus. The usual pattern is
two-stage retrieval — use [embeddings](https://docs.offrail.ai/features/embeddings) to cheaply fetch
the top \~100 candidates, then rerank those down to the handful you actually put
in the prompt.

Browse available rerank models on the
[models page](https://offrail.ai/models?filters=1\&rerank=true).

For the full request and response schema, see the
[API reference](https://docs.offrail.ai/v1_rerank).

## Endpoint [#endpoint]

`POST https://api.offrail.ai/v1/rerank`

## cURL [#curl]

```bash
curl -X POST "https://api.offrail.ai/v1/rerank" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-reranker-8b",
    "query": "What is the capital of France?",
    "documents": [
      "Paris is the capital of France.",
      "Berlin is the capital of Germany.",
      "Madrid is the capital of Spain."
    ],
    "top_n": 2
  }'
```

```json
{
	"id": "kK1sT9pQ2mR7vX4nB6cL8dF3gH5jW0yZaA2eS4uI",
	"results": [
		{ "index": 0, "relevance_score": 0.9781517386436462 },
		{ "index": 1, "relevance_score": 0.00010548105638008565 }
	],
	"meta": {
		"api_version": { "version": "1" },
		"billed_units": { "input_tokens": 255, "total_tokens": 255 }
	}
}
```

Results are sorted by `relevance_score` descending. Each `index` refers back to
the position of that document in the request's `documents` array, so you can map
scores onto your own records without relying on the returned text.

The response `id` echoes the `x-request-id` header when you send one, which is
handy for correlating a rerank call with its entry in the activity log.

## Request fields [#request-fields]

| Field                | Type       | Description                                                               |
| -------------------- | ---------- | ------------------------------------------------------------------------- |
| `model`              | `string`   | Rerank model to use. Optionally prefixed with a provider (`deepinfra/…`). |
| `query`              | `string`   | The search query to rank documents against.                               |
| `documents`          | `string[]` | Candidate documents to score. At least one.                               |
| `top_n`              | `number`   | Return only the top N results. Defaults to all documents.                 |
| `return_documents`   | `boolean`  | Include the document text on each result. Defaults to `false`.            |
| `max_chunks_per_doc` | `number`   | Maximum chunks per document. Accepted, but not every provider honors it.  |

## Two-stage retrieval [#two-stage-retrieval]

```ts
// 1. Cheap recall: embed the query and pull the top 100 candidates
//    from your vector store.
const candidates = await vectorStore.search(queryEmbedding, { limit: 100 });

// 2. Precise ordering: rerank those candidates and keep the best 5.
const res = await fetch("https://api.offrail.ai/v1/rerank", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "qwen3-reranker-8b",
		query,
		documents: candidates.map((c) => c.text),
		top_n: 5,
	}),
});

const { results } = await res.json();
const topDocuments = results.map((r) => candidates[r.index]);
```

<Callout type="info">
  Rerank models are billed on input tokens only — the query and every document
  you send are scored as input. There are no output tokens, since the response
  is a list of scores rather than generated text. Sending 100 long documents
  costs real money, so keep the first-stage candidate set tight.
</Callout>

<Callout type="warn">
  Rerank models only work on `/v1/rerank`. Requesting one on
  `/v1/chat/completions` returns a 400 pointing you at the right endpoint, and
  they are not available in the playground.
</Callout>


# Response Healing
URL: https://docs.offrail.ai/features/response-healing


# Response Healing [#response-healing]

Response Healing is a plugin that automatically validates and repairs malformed JSON responses from AI models. When enabled, OffRail ensures that API responses conform to your specified schemas even when the model's formatting is imperfect.

## Why Response Healing? [#why-response-healing]

Large language models occasionally produce invalid JSON, especially in complex scenarios:

* **Markdown wrapping**: Models often wrap JSON in code blocks like \`\`\`json...\`\`\`
* **Mixed content**: JSON may be preceded or followed by explanatory text
* **Syntax errors**: Trailing commas, unquoted keys, or single quotes instead of double quotes
* **Truncated output**: Token limits may cut off responses mid-JSON

Response Healing automatically detects and fixes these issues, saving you from implementing error handling for every possible malformed response.

## Enabling Response Healing [#enabling-response-healing]

To enable Response Healing, add `response-healing` to the `plugins` array in your request:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Return a JSON object with name and age"}],
    "response_format": {"type": "json_object"},
    "plugins": [{"id": "response-healing"}]
  }'
```

<Callout type="info">
  Response Healing only activates when `response_format` is set to `json_object`
  or `json_schema`. For regular text responses, the plugin has no effect.
</Callout>

## How It Works [#how-it-works]

When Response Healing is enabled, OffRail applies a series of repair strategies to malformed JSON responses:

### 1. Markdown Extraction [#1-markdown-extraction]

Extracts JSON from markdown code blocks:

```text
Here's the data:
\`\`\`json
{"name": "Alice", "age": 30}
\`\`\`
```

Becomes:

```json
{ "name": "Alice", "age": 30 }
```

### 2. Mixed Content Extraction [#2-mixed-content-extraction]

Separates JSON from surrounding text:

```text
Sure! Here is the JSON you requested: {"name": "Alice", "age": 30} Let me know if you need anything else.
```

Becomes:

```json
{ "name": "Alice", "age": 30 }
```

### 3. Syntax Fixes [#3-syntax-fixes]

Repairs common JSON syntax violations:

| Issue           | Before              | After               |
| --------------- | ------------------- | ------------------- |
| Trailing commas | `{"a": 1,}`         | `{"a": 1}`          |
| Unquoted keys   | `{name: "Alice"}`   | `{"name": "Alice"}` |
| Single quotes   | `{'name': 'Alice'}` | `{"name": "Alice"}` |

### 4. Truncation Completion [#4-truncation-completion]

Adds missing closing brackets for truncated responses:

```text
{"name": "Alice", "data": {"nested": true
```

Becomes:

```json
{ "name": "Alice", "data": { "nested": true } }
```

## Usage Examples [#usage-examples]

### With JSON Object Format [#with-json-object-format]

Request a structured response with automatic healing:

```typescript
const response = await fetch("https://api.offrail.ai/v1/chat/completions", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "gpt-4o",
		messages: [
			{
				role: "user",
				content:
					"Return a JSON object with fields: name (string) and age (number)",
			},
		],
		response_format: { type: "json_object" },
		plugins: [{ id: "response-healing" }],
	}),
});

const result = await response.json();
// Response is guaranteed to be valid JSON
const data = JSON.parse(result.choices[0].message.content);
```

### With JSON Schema [#with-json-schema]

For stricter validation, combine with `json_schema`:

```typescript
const response = await fetch("https://api.offrail.ai/v1/chat/completions", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "gpt-4o",
		messages: [
			{
				role: "user",
				content: "Generate a user profile",
			},
		],
		response_format: {
			type: "json_schema",
			json_schema: {
				name: "user_profile",
				schema: {
					type: "object",
					required: ["name", "email"],
					properties: {
						name: { type: "string" },
						email: { type: "string" },
						age: { type: "number" },
					},
				},
			},
		},
		plugins: [{ id: "response-healing" }],
	}),
});

const result = await response.json();
```

## Healing Metadata [#healing-metadata]

When a response is healed, the healing method is logged for debugging. The following healing methods may be applied:

| Method                     | Description                                 |
| -------------------------- | ------------------------------------------- |
| `markdown_extraction`      | JSON extracted from markdown code blocks    |
| `mixed_content_extraction` | JSON extracted from surrounding text        |
| `syntax_fix`               | Trailing commas, quotes, or keys were fixed |
| `truncation_completion`    | Missing closing brackets were added         |
| `combined_strategies`      | Multiple strategies were applied            |

## Limitations [#limitations]

<Callout type="warning">
  Response Healing is only available for non-streaming requests. Streaming
  responses are returned as-is without healing.
</Callout>

Response Healing works best for:

* Simple to moderately complex JSON structures
* Common formatting issues from LLMs

It may not be able to repair:

* Severely corrupted or nonsensical output
* Complex nested structures with multiple issues
* Responses that don't contain any recognizable JSON

## Best Practices [#best-practices]

### Use with Structured Prompts [#use-with-structured-prompts]

Combine Response Healing with clear instructions for best results:

```typescript
const response = await fetch("https://api.offrail.ai/v1/chat/completions", {
	method: "POST",
	headers: {
		Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
		"Content-Type": "application/json",
	},
	body: JSON.stringify({
		model: "gpt-4o",
		messages: [
			{
				role: "system",
				content: "Always respond with valid JSON. No explanations.",
			},
			{
				role: "user",
				content: "List three colors as a JSON array",
			},
		],
		response_format: { type: "json_object" },
		plugins: [{ id: "response-healing" }],
	}),
});

const result = await response.json();
```

### Validate Critical Data [#validate-critical-data]

For critical applications, validate the healed JSON in your code:

```typescript
const result = await response.json();
const content = result.choices[0].message.content;
const data = JSON.parse(content);

// Add your own validation
if (!data.name || typeof data.name !== "string") {
	throw new Error("Invalid response: missing name");
}
```

### Monitor Healing Rates [#monitor-healing-rates]

If you notice frequent healing in your logs, consider:

* Improving your prompts to request cleaner JSON
* Using models with better JSON output (e.g., GPT-4o, Claude 3.5)
* Adding explicit JSON examples in your prompts


# Routing
URL: https://docs.offrail.ai/features/routing


# Routing [#routing]

OffRail provides flexible and intelligent routing options to help you get the best performance and cost efficiency from your AI applications. Whether you want to use specific models, providers, or let our system automatically optimize your requests, we've got you covered.

OffRail also includes **automatic retry and fallback** — if a provider fails, your request is seamlessly retried on the next best provider, all within the same API call.

## Model Selection [#model-selection]

### Any Model Name [#any-model-name]

You can use any model name from our [models page](https://offrail.ai/models) or discover available models programmatically through the [/v1/models endpoint](https://docs.offrail.ai/v1_models).

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

### Model ID Routing [#model-id-routing]

Choose a specific model ID to route to the **best available provider** for that model. OffRail's smart routing algorithm considers multiple factors to find the optimal provider across all configured options.

#### Smart Routing Algorithm [#smart-routing-algorithm]

When you use a model ID without a provider prefix, OffRail's intelligent routing system analyzes multiple factors to select the best provider.

**Weighted Scoring System**:

Each factor has a **relative weight**. The factors are scored as ratios against the best provider in the candidate set (e.g. a provider that is twice as expensive as the cheapest scores `1.0` on price), and each ratio is multiplied by its weight divided by the sum of all active weights. The provider with the lowest (best) total score wins.

The default weights are:

| Factor          | Default weight | Notes                                                                      |
| --------------- | -------------- | -------------------------------------------------------------------------- |
| **Price**       | `0.6`          | Cost efficiency (average of input and output price)                        |
| **Uptime**      | `0.5`          | Provider reliability / low error rate                                      |
| **Throughput**  | `0.05`         | Tokens per second generation speed                                         |
| **Latency**     | `0.025`        | Time to first token — **only applied for streaming requests**              |
| **Cache**       | `0.2`          | Prompt-cache support — **only applied for large prompts** (≥ 5,000 tokens) |
| **Image price** | `1.0`          | Replaces the price weight for image-generation models                      |

Because the weights are relative and normalized by the sum of the active weights, price and uptime dominate routing decisions in practice, while throughput and latency act as tie-breakers between otherwise comparable providers.

**Latency Weight for Non-Streaming Requests**:

The latency weight only applies to streaming requests (time-to-first-token is only measured there). For non-streaming requests the latency weight is dropped and its share is redistributed proportionally across the remaining factors.

**Time-Decayed Metrics Window**:

Provider metrics (uptime, throughput, latency) are not a flat "last N minutes" snapshot. They are aggregated over a rolling **60-minute window** with a time-decay weighting so very recent behavior dominates while older data still contributes:

* The most recent **1 minute** is weighted &#x2A;*10×**
* The most recent **5 minutes** are weighted &#x2A;*3×**
* The remainder of the 60-minute window is weighted &#x2A;*1×**

This makes routing react quickly to a provider that just started failing or slowing down, without overreacting to a single noisy data point.

**Cache Support for Large Prompts**:

When the estimated prompt is at least 5,000 tokens, the **cache weight** (default `0.2`) is factored into the score based on whether each provider supports prompt caching (advertised via a cached input price). Providers that support caching score better than ones that do not, since caching can substantially reduce the cost of large or repeated prompts. Below the 5,000-token threshold, this weight is dropped entirely — caching has little impact on small prompts, so cache support is ignored. The selected provider's cache support is exposed as `cacheSupported` on the routing metadata.

**Exponential Uptime Penalty**:

Providers with uptime below 95% receive an additional exponential penalty that increases rapidly as uptime drops:

* 95-100% uptime: No penalty
* 90% uptime: \~0.07 penalty
* 80% uptime: \~0.62 penalty
* 70% uptime: \~1.73 penalty
* 50% uptime: \~5.61 penalty

This ensures providers experiencing significant issues are strongly deprioritized while minor fluctuations have minimal impact. The penalty threshold (default `95%`) is configurable.

**Provider Priority**:

Each provider has a **priority** value (default `1`) that nudges routing toward or away from it independently of live metrics:

* A provider's priority is applied as a `(1 - priority)` adjustment to its score — higher priority lowers the score (more preferred), lower priority raises it (less preferred).
* A priority of **0** disables the provider entirely, removing it from routing for that model.

Provider priorities are surfaced in the routing metadata so you can see how they influenced a decision.

**Epsilon-Greedy Exploration** (1% of requests by default):

To solve the "cold start problem" where new or unused providers never get traffic to build up metrics, the system randomly explores different providers a small fraction of the time (default 1%, configurable). This ensures:

* All providers periodically receive traffic
* New providers can prove their reliability
* The system adapts to changing provider performance
* You benefit from improved routing decisions over time

The exploration rate is configurable per project through the routing configuration (`thresholds.explorationRate`), and self-hosted deployments can override it globally with the `EXPLORATION_RATE` environment variable (a number between `0` and `1`).

**Stable Provider Preference**:

To avoid unnecessary churn between providers that score similarly, OffRail remembers the best provider chosen for each model and sticks with it across requests — even if another provider edges ahead slightly on the next score calculation.

On every routing decision, the system checks whether the previously selected provider is still acceptable:

* **Uptime hard switch**: if the preferred provider's uptime drops below &#x2A;*85%**, routing switches to the current best-scoring provider immediately.
* **Score margin soft switch**: the preferred provider is replaced only when a better option's score is more than **0.15** ahead. Small fluctuations caused by metric noise or minor price differences do not trigger a switch.
* **Periodic re-evaluation**: the preference expires after **1 hour**, at which point the next request picks the best-scoring provider fresh and stores it as the new preferred.

Requests that are part of the epsilon-greedy exploration bypass this preference entirely so that all providers continue to receive periodic traffic and build up metrics.

The selection reason in routing metadata will show `stable-preferred` when a request was served by the stored preference rather than the top-scored provider at that moment.

<Callout type="info">
  Self-hosted deployments can tune this behavior with three environment
  variables: `PREFERRED_PROVIDER_TTL` (preference lifetime in seconds, default
  `3600`), `PREFERRED_PROVIDER_UPTIME_THRESHOLD` (hard-switch uptime floor,
  default `85`), and `PREFERRED_PROVIDER_SCORE_MARGIN` (soft-switch score gap,
  default `0.15`). On the **Enterprise plan**, these same values can be
  customized per project from the dashboard — see [Per-Project Routing
  Configuration](#per-project-routing-configuration-enterprise).
</Callout>

**Routing Metadata**:

Every request includes detailed routing metadata in the logs, showing:

* Available providers that were considered
* Selected provider and selection reason
* Scores for each provider (including uptime, throughput, latency, price, priority, and cache support)

This transparency allows you to understand and debug routing decisions.

<Callout type="info">
  Using model IDs without a provider prefix automatically routes to the optimal
  provider based on reliability, speed, and cost. The system continuously learns
  and adapts based on real-time performance metrics.
</Callout>

<Callout type="success">
  Smart routing prioritizes reliability over cost, ensuring your requests are
  routed to providers with proven uptime and performance, while still
  considering cost efficiency.
</Callout>

### Routing Strategy [#routing-strategy]

By default, model-ID routing uses the full weighted score described above (`routing: "auto"`). When you care about a single dimension, set the `routing` field — named after the factor it optimizes — to bias provider selection toward it:

| Strategy                     | Behavior                                                                             |
| ---------------------------- | ------------------------------------------------------------------------------------ |
| `auto&#x60; &#x2A;(default)* | Full weighted smart-routing score (price, uptime, throughput, latency, cache).       |
| `price`                      | Gives price a **90% relative weight**, so the cheapest provider almost always wins.  |
| `throughput`                 | Gives throughput a **90% relative weight**, so the fastest-generating provider wins. |
| `latency`                    | Gives latency a **90% relative weight**, so the lowest time-to-first-token wins.     |

Each non-`auto` strategy keeps a small (10%) uptime weight, and the [exponential uptime penalty](#smart-routing-algorithm) still applies on top. This means the dominant pick is still skipped in favor of another provider when it has extremely bad uptime — you get the cheapest (or fastest) provider that is actually healthy, not one that is effectively down.

Because time-to-first-token is only measured for streaming requests, `routing: "latency"` only biases streaming requests; for non-streaming requests it falls back to selecting on uptime.

```bash
# Always pick the cheapest healthy provider for this model
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "routing": "price"
  }'
```

```bash
# Always pick the highest-throughput healthy provider for this model
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "routing": "throughput"
  }'
```

<Callout type="info">
  The `routing` field only applies to model-id routing. Combining it with a
  specific provider (e.g. `openai/gpt-4o`) returns a `400` error, since the
  strategy can't influence a pinned provider — remove the provider prefix to use
  a strategy. On **coding (dev) plans**, only `auto` and `price` are allowed;
  the other strategies return a `400` error because they would bypass the
  prompt-cache–aware routing those plans depend on.
</Callout>

### Sticky Session Routing [#sticky-session-routing]

When a model is served by multiple providers, every request is normally scored independently — so a multi-turn conversation can bounce between providers. That defeats provider-side **prompt caching**, which only pays off when consecutive requests with a shared prefix hit the **same** provider.

Sticky session routing solves this: attach a session identifier and OffRail pins all requests for that session to a single provider (and region), keeping the upstream prompt cache warm across the whole conversation.

#### Setting the session id [#setting-the-session-id]

For chat completions, the session key is resolved in priority order:

1. The `x-session-id` header
2. The `prompt_cache_key` body field (OpenAI-compatible)
3. The `user` body field (OpenAI-compatible)

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-session-id: conversation-9f8e7d6c" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

For the Anthropic Messages endpoint (`/v1/messages`), the session key is derived automatically from `metadata.user_id` — coding agents such as Claude Code embed the session id there — and forwarded internally. An explicit `x-session-id` header still takes precedence.

#### How pinning works [#how-pinning-works]

On a session's **first** request the provider is chosen by the normal weighted smart-routing score — the same price-, priority-, uptime-, and throughput-aware algorithm used for non-sticky requests. That choice is then **persisted for the session** and reused on every subsequent request, so the upstream prompt cache stays warm without bouncing the conversation between providers.

Because the pinned provider is replayed directly, sticky requests **skip the epsilon-greedy exploration** — a session is never randomly bounced to a different provider mid-conversation.

#### Falling back when a provider is down [#falling-back-when-a-provider-is-down]

An established pin yields only when its provider can no longer serve the session well. A session is re-scored and re-pinned to the current weighted-best provider when its provider:

* Drops below the session uptime threshold (default 85%),
* Is filtered out by health checks (e.g. excluded for low uptime), or
* Fails the request and is dropped by the [automatic retry & fallback](#automatic-retry--fallback) loop.

Re-pinning runs the same weighted algorithm again, so the replacement is the best currently available provider — not an arbitrary one.

The selection reason in routing metadata shows `session-sticky` when a request was pinned via a session id.

<Callout type="info">
  Sticky routing optimizes for cache locality over per-request churn. Once a
  session is pinned it stays on its provider even if a cheaper or faster
  alternative becomes momentarily available, since the prompt-cache savings
  typically outweigh the difference — but the initial pick still respects price
  and priority. Requests without a session id are unaffected and continue to use
  the weighted smart-routing algorithm.
</Callout>

### Provider-Specific Routing [#provider-specific-routing]

To use a specific provider without any fallbacks, prefix the model name with the provider name followed by a slash:

```bash
# Use OpenAI specifically
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Use DeepSeek provider specifically
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

#### Regions [#regions]

Some providers expose the same model in multiple regions. In that case, OffRail supports two routing modes:

* `provider/model` selects the best eligible region for that provider using the same routing inputs used elsewhere: recent uptime, throughput, latency, and price
* `provider/model:region` pins the request to one exact region

```bash
# Let OffRail choose the best Alibaba region for DeepSeek V3.2
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "alibaba/deepseek-v3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Force a specific Alibaba region
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "alibaba/deepseek-v3.2:cn-beijing",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

If your provider key stores an explicit region, that region acts like a lock and OffRail will only use that region for provider-specific requests. If no explicit region is configured on the provider key, provider-specific requests can still score all eligible regions for that provider.

Routing metadata reflects this:

* Dynamic provider-region selection shows all eligible regional scores that were considered
* Explicitly pinned regions show only the pinned region in the score list

<Callout type="info">
  Region-aware routing only compares regions that are actually available for the
  current project mode and provider setup. In credits mode, that means only
  regions backed by configured environment keys. In API keys and hybrid mode, an
  explicit provider-key region restricts the request to that region.
</Callout>

A few regions are served by an endpoint that belongs to your own account rather
than a shared one — Alibaba Cloud's EU (Frankfurt) region has no shared
DashScope domain and is served by your Model Studio workspace's dedicated host.
Such a region still works from an API key alone, via the provider's shared entry
point, but that endpoint is rate-limited and carries no SLA. Set the workspace
ID on the provider key (copy it from the API Host shown when you create the key)
to route through your own endpoint instead.

#### Low-Uptime Protection [#low-uptime-protection]

When you specify a provider explicitly, OffRail checks the provider's recent uptime (from the time-decayed metrics window described above). If the uptime falls below 90%, the system automatically routes your request to the best available alternative provider to ensure reliability. This protects your application from providers experiencing temporary issues. The fallback threshold (default `90%`) is configurable.

<Callout type="info">
  If the requested provider has low uptime but no alternative providers are
  available for that model, the request will still be sent to the originally
  requested provider.
</Callout>

#### Disabling Fallback with X-No-Fallback Header [#disabling-fallback-with-x-no-fallback-header]

If you need to bypass this protection and always use the exact provider you specified regardless of its current uptime, you can use the `X-No-Fallback` header:

```bash
# Force use of a specific provider even if it has low uptime
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-No-Fallback: true" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

<Callout type="warn">
  Using `X-No-Fallback: true` disables automatic provider failover. Your
  requests will be sent to the specified provider even if it is experiencing
  issues, which may result in higher error rates. Retries may still occur
  against another key for the same provider when multiple keys are configured.
</Callout>

When the `X-No-Fallback` header is used, the routing metadata in logs will include `noFallback: true` to indicate that fallback was disabled for that request.

## Automatic Retry & Fallback [#automatic-retry--fallback]

When using model ID routing (without a provider prefix), OffRail automatically retries failed requests on alternate providers. This happens transparently within the same API call — your application receives the successful response as if nothing went wrong.

### How Retry Works [#how-retry-works]

1. Your request is routed to the best available provider using the smart routing algorithm
2. If that provider returns a server error (5xx), times out, or has a connection failure, the gateway marks the provider as failed
3. The next best available provider is selected and the request is retried
4. Up to **2 retries** are attempted before returning an error to the client

```
Request → Provider A (500 error) → Provider B (200 OK) → Response
```

Both streaming and non-streaming requests support automatic retry.

### What Triggers a Retry [#what-triggers-a-retry]

Retries are triggered by **server-side failures** only:

* **5xx errors** (500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, etc.)
* **Timeouts** (upstream provider took too long to respond)
* **Connection failures** (network errors, DNS failures, etc.)

Retries are **not** triggered by:

* **4xx client errors** (400 Bad Request, 401 Unauthorized, 403 Forbidden, 422 Unprocessable Entity)
* **Content filter responses** (Azure ResponsibleAI, etc.)

### When Retry Is Disabled [#when-retry-is-disabled]

Automatic retry to a different provider is disabled when:

* The `X-No-Fallback: true` header is set
* A specific provider is requested (e.g., `openai/gpt-4o`)
* No alternative providers are available for the requested model
* The maximum retry count (2) has been exhausted

Retries can still happen within the same provider when multiple keys are
configured and the current key fails with a retryable error.

### Routing Transparency [#routing-transparency]

Every provider attempt — both failed and successful — is recorded in the `routing` array in the response metadata (streaming and non-streaming alike) and activity logs:

```json
{
	"metadata": {
		"routing": [
			{
				"provider": "openai",
				"model": "gpt-4o",
				"status_code": 500,
				"error_type": "server_error",
				"succeeded": false,
				"credentialSource": "byok",
				"apiKeyHash": "f029ee9",
				"providerKeyId": "pk_2f9a...",
				"providerKeyLabel": "billing-team-key"
			},
			{
				"provider": "azure",
				"model": "gpt-4o",
				"status_code": 200,
				"error_type": "none",
				"succeeded": true,
				"credentialSource": "platform",
				"apiKeyHash": "ecb88d5"
			}
		]
	}
}
```

#### Whose key served each attempt [#whose-key-served-each-attempt]

`credentialSource` says who owns the provider credential an attempt was sent with:

| Value      | Meaning                                                                                                   |
| ---------- | --------------------------------------------------------------------------------------------------------- |
| `byok`     | Your own provider key. The provider bills you directly and the attempt is not deducted from your credits. |
| `platform` | An OffRail credential. The attempt runs on credits and is deducted from your balance.                     |

This matters most in **hybrid** mode, where a request that fails on your own key falls back to OffRail's credential: both attempts appear in the same `routing` array, and only `credentialSource` tells them apart — `apiKeyHash` is an opaque fingerprint that says two attempts used different keys, not which key was yours. The same value is stored on the log as `routingMetadata.usedCredentialSource` for the credential that ultimately served the request, and is shown as a **your key** / **OffRail key** badge in the dashboard's routing view.

#### Which of your keys ran [#which-of-your-keys-ran]

A `byok` attempt also carries the key itself: `providerKeyId`, and `providerKeyLabel` — the key as it is named on your [provider keys](https://offrail.ai/dashboard) page (its name, or its masked token when it has none). So when several of your keys are configured for a provider and the gateway rotates between them, each attempt says which one it used instead of leaving you to decode a fingerprint.

Chat requests additionally record `routingMetadata.eligibleProviderKeys` on the log: your keys that were candidates for the provider that served the request, in selection order. It is omitted for credits-mode projects, which route on OffRail credentials, and for custom providers, whose keys are scoped by their own catalogue.

<Callout type="info">
  These fields describe **your** keys only. OffRail's own credentials — the ones
  that serve credits-mode traffic — are never named: a `platform` attempt still
  reports `credentialSource` and `apiKeyHash`, but never `providerKeyId` or
  `providerKeyLabel`.
</Callout>

### Retried Log Tracking [#retried-log-tracking]

Each provider attempt creates its own log entry. Failed attempts that were retried are marked with:

* **`retried: true`** — indicates this failed request was retried on another provider
* **`retriedByLogId`** — the ID of the final successful log entry

This allows you to distinguish between unrecovered failures and failures that were transparently recovered via retry. In the dashboard, retried logs display a "Retried" badge with a link to the successful log.

### Impact on Provider Health [#impact-on-provider-health]

Failed attempts still count against the provider's uptime score, even when the request was successfully retried on another provider. This means:

* A provider that keeps failing will see its uptime score drop
* The exponential uptime penalty kicks in below 95% (see [Smart Routing Algorithm](#smart-routing-algorithm))
* Future requests are automatically routed away from unreliable providers
* Your application stays reliable without any code changes on your side

<Callout type="success">
  Automatic retry and fallback works together with smart routing to provide
  self-healing behavior. Failing providers are automatically avoided, and your
  requests are transparently recovered on reliable alternatives.
</Callout>

## Per-Project Routing Configuration (Enterprise) [#per-project-routing-configuration-enterprise]

The values described above — scoring weights, thresholds, retry behavior, the metrics window, sticky-routing, and per-provider priorities — are the **defaults** that apply to every project. On the **Enterprise plan**, you can override any of them **per project** from the dashboard under **Project Settings → Routing**. Projects on other plans always use the defaults.

Overrides are merged on top of the defaults, so you only set the values you want to change. When a custom configuration is disabled, the project falls back to the defaults.

The following groups can be customized per project:

| Group                   | What it controls                                                                                                                                                                     | Defaults                                                                                                                                 |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Weights**             | Relative importance of each scoring factor                                                                                                                                           | `price 0.6`, `imagePrice 1.0`, `uptime 0.5`, `throughput 0.05`, `latency 0.025`, `cache 0.2`                                             |
| **Thresholds**          | Cache prompt-size threshold, uptime-penalty threshold, exploration rate, and the assumed defaults used when no metrics exist                                                         | `cachePromptTokens 5000`, `uptimePenalty 95`, `defaultUptime 100`, `defaultLatency 1000`, `defaultThroughput 50`, `explorationRate 0.01` |
| **Retry**               | Max cross-provider fallback attempts and the low-uptime reroute threshold                                                                                                            | `maxRetries 2`, `lowUptimeFallbackThreshold 90`                                                                                          |
| **Timeouts**            | Per-request time limits (end-to-end, streaming, non-streaming) — see [Request Timeouts](https://docs.offrail.ai/features/timeouts). Capped at the infrastructure defaults — an override can only lower them | `gatewayMs 1,500,000`, `streamingMs 1,200,000`, `plainMs 600,000`                                                                        |
| **History**             | The metrics window and the time-decay tier boundaries and weights                                                                                                                    | `windowMinutes 60` (max 120), `tier1Minutes 1`, `tier2Minutes 5`, `tier1Weight 10`, `tier2Weight 3`, `tier3Weight 1`                     |
| **Sticky**              | Stable-provider preference: on/off, TTL, hard-switch uptime floor, soft-switch score margin                                                                                          | `enabled true`, `ttlSeconds 3600`, `uptimeThreshold 85`, `scoreMargin 0.15`                                                              |
| **Provider priorities** | Per-provider priority multipliers; set a provider to `0` to disable it for that project                                                                                              | `1` for every provider                                                                                                                   |

<Callout type="info">
  Per-project routing configuration requires the Enterprise plan. If you'd like
  to tune routing for your workloads, contact us at [contact@offrail.ai](mailto:contact@offrail.ai).
</Callout>

## Optimized Auto Routing [#optimized-auto-routing]

Auto routing automatically selects the best model for your specific use case without you having to specify a model at all.

### Current Implementation [#current-implementation]

The auto routing system currently:

* **Chooses cost-effective models** by default for optimal price-to-performance ratio
* **Automatically scales to more powerful models** based on your request's context size
* **Handles large contexts intelligently** by selecting models with appropriate context windows

```bash
# Let OffRail choose the optimal model
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Your request here..."}]
  }'
```

### Free Models Only [#free-models-only]

When using auto routing, you can restrict the selection to only free models (models with zero input and output pricing) by setting the `free_models_only` parameter to `true`:

```bash
# Auto route to free models only
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "free_models_only": true
  }'
```

<Callout type="success">
  Adding even a small amount of credits to your account (e.g., $10) will
  immediately upgrade your free model rate limits from 5 requests per 10 minutes
  to 20 requests per minute.
</Callout>

<Callout type="info">
  The `free_models_only` parameter only works with auto routing (`"model":
  	"auto"`). If no free models are available that meet your request requirements,
  the API will return an error.
</Callout>

### Reasoning models only [#reasoning-models-only]

Just specify the `reasoning_effort` value and only a model which supports reasoning will be chosen. This parameter is not specific to the auto model.

```bash
# Auto route only to reasoning models
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "reasoning_effort": "medium"
  }'
```

### Exclude Reasoning Models [#exclude-reasoning-models]

When using auto routing, you can exclude reasoning models from selection by setting the `no_reasoning` parameter to `true`. This is useful when you want faster responses or need to avoid the additional cost and latency of reasoning models:

```bash
# Auto route excluding reasoning models
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Hello!"}],
    "no_reasoning": true
  }'
```

<Callout type="info">
  The `no_reasoning` parameter only works with auto routing (`"model": "auto"`).
  If no non-reasoning models are available that meet your request requirements,
  the API will return an error.
</Callout>

<Callout type="success">
  Auto routing analyzes your payload and automatically chooses between
  cost-effective models for simple requests and more powerful models for complex
  or large-context requests.
</Callout>

### Coming Soon: Advanced Optimization [#coming-soon-advanced-optimization]

We're continuously improving our auto routing capabilities. Soon you'll benefit from:

* **Tool call optimization**: Automatically select models that excel at function calling and structured outputs
* **Content-aware routing**: Analyze message content to determine the best model for specific types of requests (coding, creative writing, analysis, etc.)
* **Performance-based routing**: Route based on historical performance data for similar requests
* **Multi-model orchestration**: Intelligently combine multiple models for complex workflows

### How It Works [#how-it-works]

1. **Request Analysis**: The system analyzes your request including message content, context size, and any special parameters
2. **Model Selection**: Based on the analysis, it selects the most appropriate model considering cost, performance, and capabilities
3. **Transparent Routing**: Your request is seamlessly routed to the chosen model and provider
4. **Optimized Response**: You receive the best possible response while maintaining cost efficiency

<Callout type="info">
  Auto routing decisions are transparent in your usage logs, so you can always
  see which model was selected for each request.
</Callout>

## Best Practices [#best-practices]

### For Development [#for-development]

* Use specific model names during development and testing
* Leverage auto routing for production workloads to optimize costs

### For Production [#for-production]

* Use auto routing (`"model": "auto"`) for the best balance of cost and performance
* Monitor your usage patterns through the dashboard to understand routing decisions
* Set up provider keys for multiple providers to maximize routing options

### For Cost Optimization [#for-cost-optimization]

* Let auto routing handle model selection to automatically use the most cost-effective options
* Use model IDs without provider prefixes to always get the cheapest available provider
* Monitor your usage analytics to track cost savings from intelligent routing


# Service Tiers
URL: https://docs.offrail.ai/features/service-tiers


# Service Tiers [#service-tiers]

Some OpenAI, Google, and Fireworks models support selectable **processing tiers** that trade
latency and availability against price. You pick one per request with the
OpenAI-compatible `service_tier` parameter, and OffRail forwards it only
when the selected provider/model mapping supports that tier.

| Tier         | `service_tier`            | Cost vs. standard | Latency / availability                      |
| ------------ | ------------------------- | ----------------- | ------------------------------------------- |
| Standard     | `default` / `auto` / omit | baseline          | Normal on-demand latency                    |
| **Flex**     | `flex`                    | **−50%**          | Best-effort; may be preempted under load    |
| **Priority** | `priority`                | varies by model   | Prioritized above standard and flex traffic |

## Using the `service_tier` parameter [#using-the-service_tier-parameter]

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google-vertex/gemini-3.1-pro-preview",
    "service_tier": "priority",
    "messages": [
      { "role": "user", "content": "Summarize this incident report." }
    ]
  }'
```

Accepted values are `flex`, `priority`, and `default`/`auto` (standard). If you
request `flex` or `priority` for a provider/model mapping that does not support
that tier, the gateway returns a 400 `unsupported_service_tier` error and logs
the request as a client error.

The parameter works the same on the OpenAI-compatible **Responses API**
(`/v1/responses`): the tier is forwarded to the provider and the response's
`service_tier` field echoes the tier that was actually served.

<Callout type="info">
  Service-tier selection is available for pay-as-you-go requests. Chat plans use
  Standard processing and does not expose a service-tier selector in the
  dashboard.
</Callout>

## Supported providers [#supported-providers]

Service tiers are explicit per provider/model mapping. Check the model page for
the exact tiers exposed by each provider card.

* **OpenAI** (`openai`) — sent as the OpenAI `service_tier` request field for
  supported OpenAI models. Flex is billed at 0.5x standard token prices and
  Priority uses the model-specific multiplier shown on the model page.

* **Google Vertex AI** (`google-vertex`) — sent as the
  `X-Vertex-AI-LLM-Shared-Request-Type` request header, together with
  `X-Vertex-AI-LLM-Request-Type: shared` so the request bypasses any Provisioned
  Throughput on the project and actually reaches the shared Flex/Priority tier.
  Flex and Priority are served only on the **global** endpoint, which is the
  gateway default. Google Flex PayGo applies a 0.5x multiplier; Google Priority
  PayGo applies a 1.8x multiplier.

* **Google AI Studio / Gemini API** (`google-ai-studio`) — sent as a
  `service_tier` field in the request body for configured models that opt in.

* **Fireworks AI** (`fireworks`) — sent as a `service_tier` field in the request
  body on its OpenAI-compatible chat completions endpoint. Only Priority is
  offered (Fireworks publishes no Flex rate card), billed at a 1.25x multiplier.
  Fireworks does not report the tier it served, but a Priority request is either
  served at Priority or shed with a 503, so an accepted request is billed at the
  tier it was sent at.

<Callout type="info">
  Tiers are supported on a **subset** of models, and the Flex and Priority
  subsets differ by provider. For example, Google Flex PayGo lists Gemini 3
  image / Nano Banana models, but Google Priority PayGo does not; those
  configured image mappings are Flex-only.
</Callout>

<Callout type="warn">
  Flex and Priority are only honored when the request reaches the provider
  directly, so a provider key with a **custom base URL** (a proxy) is excluded
  from service-tier routing — a proxy may silently drop the tier and serve
  standard. This applies to every provider that offers tiers, including OpenAI:
  the tier travels as a `service_tier` body field that an OpenAI-compatible
  proxy is free to ignore. With multiple providers/keys, the gateway routes
  around the ineligible key automatically; if a request pins a provider whose
  only key uses a custom base URL, it returns a 400 instead of silently
  downgrading. Keys with no custom base URL (the managed default) are always
  eligible.
</Callout>

## Retries and fallback never downgrade the tier [#retries-and-fallback-never-downgrade-the-tier]

A request can change provider or credential mid-flight: OffRail falls back
to another provider when one fails, and rotates to another key for the same
provider when a credential returns a 429 or an auth error. A requested tier is
carried through all of it.

* Provider routing is narrowed to mappings that support the requested tier
  **before** a provider is picked, so no fallback candidate can be one that
  would serve the request as standard.
* Key selection — BYOK keys, platform-managed credentials, and env credentials
  alike — skips any credential that cannot carry the tier (a proxy base URL, or
  a Vertex credential pinned to a regional endpoint), on the first attempt and
  on every retry.
* Every attempt re-resolves the tier against the provider, region and
  credential it actually resolved to, and fails rather than sending at a lower
  tier. If no eligible candidate is left, you get the upstream error instead of
  a silently downgraded response.

This behavior applies to explicit service-tier requests made with pay-as-you-go
credits. The `used_service_tier` response metadata always reports the tier that
was actually served.

<Callout type="info">
  Rotating to another key for the same provider is a separate upstream account
  as far as prompt caching is concerned, so a retried request re-writes its
  cached prefix rather than reading the original one. Send `x-no-fallback: true`
  to have the original upstream error returned to you — note that this disables
  cross-provider fallback, not key rotation within a provider.
</Callout>

## Pricing uses multipliers [#pricing-uses-multipliers]

Service tiers do not define separate model prices in OffRail. They multiply
the provider mapping's standard token prices:

* Standard / `default` / `auto`: 1x
* Flex: 0.5x
* Priority: model/provider-specific, shown on the model page

The multiplier scales per-token costs, including input, output, cached, and
image tokens. Flat per-request and web-search fees are not tier-scaled.

## Billing follows the served tier [#billing-follows-the-served-tier]

When a provider reports the tier that was actually served, OffRail bills
that returned tier instead of blindly billing the requested value:

* A `priority` request that runs as priority is billed at 2.5x.
* A `flex` request that runs as flex is billed at 0.5x.
* A request that is served as standard is billed at the standard 1x rate.

The served tier is read back from the provider response — Vertex reports it in
`usageMetadata.trafficType` (`ON_DEMAND_PRIORITY` / `ON_DEMAND_FLEX` /
`ON_DEMAND`), Google AI Studio reports it in the `x-gemini-service-tier`
response header, and OpenAI can return `service_tier` in response payloads or
stream events. Providers that report no tier at all (Fireworks) never downgrade
silently — they reject the request instead — so an accepted request is billed at
the tier it was sent at.

<Callout type="warn">
  OffRail rejects unsupported tier requests before provider routing. For
  example, `gemini-3-pro-image-preview` currently exposes Flex for Google AI
  Studio and Vertex, but not Priority.
</Callout>

You can see per-tier pricing for each model on its
[model page](https://offrail.ai/models). Supported provider cards include a
Service Tier selector in the card header and show the active multiplier next to
each tier.

## Sources [#sources]

* [OpenAI API pricing](https://openai.com/api/pricing/)
* [Google Flex PayGo](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/flex-paygo)
* [Google Priority PayGo](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/priority-paygo)
* [Fireworks Serverless Priority and Fast](https://docs.fireworks.ai/serverless/priority-and-fast)


# Sessions
URL: https://docs.offrail.ai/features/sessions


# Sessions [#sessions]

A **session** ties together the requests that belong to the same conversation or workflow. By attaching a stable session identifier to your requests, OffRail can treat them as a unit — keeping provider routing consistent across turns and letting you trace and filter the whole conversation in the dashboard.

Sessions are the foundation for several features. Today they power **sticky provider routing** and **session-level observability**; more session-scoped capabilities will build on the same identifier over time.

## Setting the session id [#setting-the-session-id]

For chat completions, the session key is resolved in priority order — the first present value wins:

1. The `x-session-id` header
2. The `x-session-affinity` header (sent automatically by coding agents such as opencode)
3. The `prompt_cache_key` body field (OpenAI-compatible)
4. The `user` body field (OpenAI-compatible)

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-session-id: conversation-9f8e7d6c" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

Reuse the same session id for every request in a conversation. If you don't set any of the values above, the request simply has no session and behaves exactly as before.

### Anthropic Messages endpoint [#anthropic-messages-endpoint]

For the [Anthropic Messages endpoint](https://docs.offrail.ai/features/anthropic-endpoint) (`/v1/messages`), the session key is derived automatically from `metadata.user_id`. Coding agents such as Claude Code send a JSON object there (e.g. `{"session_id":"<uuid>",…}`); the gateway uses its `session_id` field. An explicit `x-session-id` header still takes precedence.

## Sticky provider routing [#sticky-provider-routing]

When a model is served by multiple providers, requests are normally scored independently, so a multi-turn conversation can bounce between providers. That defeats provider-side **prompt caching**, which only pays off when consecutive requests with a shared prefix reach the **same** provider.

With a session id set, OffRail scores the session's first request with the normal weighted smart-routing algorithm (price, priority, uptime, throughput) and then **pins that provider for the session**, reusing it on every subsequent request to keep the prompt cache warm. The session stays on that provider — skipping the epsilon-greedy exploration — and only moves when its provider drops below the session uptime threshold or leaves the available pool (health filtering or a failed request dropped by retry/fallback), at which point the session is re-scored and re-pinned to the current best provider.

See [Routing → Sticky Session Routing](https://docs.offrail.ai/features/routing) for the full algorithm, fallback behavior, and the `session-sticky` routing-metadata reason.

Session stickiness is **on by default**. Enterprise projects can turn it off per project under **Settings → Routing → Session Stickiness**; when disabled, every request is scored independently regardless of session id (the id is still recorded for observability).

<Callout type="info">
  Sticky routing optimizes for cache locality over per-request price. A session
  stays on its provider even if a cheaper or faster alternative is momentarily
  available, since the prompt-cache savings typically outweigh the difference.
</Callout>

## Upstream prompt-cache routing [#upstream-prompt-cache-routing]

Some providers use an OpenAI-style `prompt_cache_key` to route requests to the cache shard that already holds your prompt prefix — without it, repeat requests can land on different backends and miss the cache entirely (Meta requires it for cache hits in practice; OpenAI and Azure use it to improve hit rates under load).

When a request has a session id and you didn't send a `prompt_cache_key` yourself, OffRail forwards a **keyed hash** (HMAC-SHA256 with a gateway-side secret) of the session id as the `prompt_cache_key` to providers that support it (currently OpenAI, Azure, and Meta). Hashing means your raw session ids are never exposed to providers; the hash is stable per session, which is all cache routing needs. A `prompt_cache_key` you set explicitly takes precedence and is forwarded as-is on those same surfaces.

On provider surfaces that don't support the field, no key is sent at all — whether derived or explicit. This currently applies to Sakana (the field is not part of its API) and to Azure chat-completions requests, which can be served by legacy deployment-based API versions that reject unknown body fields; Azure requests on the Responses API always carry the key. Providers not listed above use different caching mechanisms (for example Anthropic `cache_control` breakpoints or Google implicit caching), so the `prompt_cache_key` doesn't apply to them either.

For Meta, requests without any session id still get a cache key derived from the conversation's first messages, so multi-turn conversations hit Meta's prompt cache even when no session signal is present.

## Observing sessions in the activity log [#observing-sessions-in-the-activity-log]

Every request is logged with its resolved session id. In the dashboard **Activity** view you can:

* See the **Session ID** on each request's metadata, alongside the request and trace IDs.
* **Filter by session id** using the search field next to the custom-metadata search, to pull up every request that belongs to a conversation in one place.

This makes it easy to follow a full conversation end-to-end — inspecting how each turn was routed, what it cost, and which provider served it.

<Callout type="info">
  The session id is distinct from freeform [metadata](https://docs.offrail.ai/features/metadata). Use
  metadata custom headers for arbitrary tags (user, tenant, app version); use
  the session id for the one value that should keep a conversation pinned and
  traceable.
</Callout>


# Source Attribution
URL: https://docs.offrail.ai/features/source


# Source Attribution [#source-attribution]

The `X-Source` header allows you to identify your domain when making requests to OffRail. This information is used to generate public usage statistics showing how OffRail is being used across different websites and applications.

## X-Source Header [#x-source-header]

Include the `X-Source` header with your domain name in your requests:

```bash
curl -X POST https://api.offrail.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "X-Source: example.com" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ]
  }'
```

## Domain Format [#domain-format]

The `X-Source` header accepts domain names in various formats. All of the following are valid and will be normalized to the same domain:

* `example.com`
* `https://example.com`
* `https://www.example.com`
* `www.example.com`

All variations will be stripped down to the base domain (`example.com`) for aggregation purposes.

## Public Statistics [#public-statistics]

Data from the `X-Source` header is used to generate public statistics about OffRail usage, including:

* **Popular Domains**: Which websites and applications are using OffRail most frequently
* **Model Usage**: What models are being used by different domains
* **Geographic Distribution**: Where requests are coming from across different sources
* **Growth Trends**: How usage is growing over time for different domains

These statistics help demonstrate the adoption and impact of OffRail across the ecosystem.

## Privacy Considerations [#privacy-considerations]

### What's Public [#whats-public]

* Domain names (stripped of protocol and www prefixes)
* Aggregated request counts and model usage
* General geographic regions (country-level data)

### What's Private [#whats-private]

* Individual request content or responses
* User identifiers or personal information
* Detailed usage patterns beyond aggregated counts
* API keys or authentication details

## Benefits [#benefits]

Including the `X-Source` header provides several benefits:

### For Your Project [#for-your-project]

* **Recognition**: Your domain will appear in public usage statistics
* **Credibility**: Demonstrates real-world usage of your application
* **Community**: Contributes to the broader OffRail ecosystem

### For the Community [#for-the-community]

* **Transparency**: Shows real adoption and usage patterns
* **Inspiration**: Other developers can see successful implementations
* **Growth**: Helps demonstrate real-world demand for shared LLM infrastructure

## Optional but Recommended [#optional-but-recommended]

While the `X-Source` header is optional, we strongly encourage its use to:

* Support transparency in the OffRail ecosystem
* Help showcase successful integrations
* Contribute to understanding of LLM usage patterns
* Demonstrate the real-world impact of your application

Your participation helps build a more transparent and collaborative LLM ecosystem.


# Speech Generation
URL: https://docs.offrail.ai/features/speech-generation


# Speech Generation [#speech-generation]

OffRail supports text-to-speech (TTS) through the OpenAI-compatible
&#x2A;*`/v1/audio/speech`** endpoint, powered by ElevenLabs, Google Gemini, OpenAI,
and Alibaba Qwen speech models.

<Callout type="info">
  Want to hear the voices before writing code? The [Audio
  Studio](https://lounge.offrail.ai/audio) in Lounge generates speech from up to
  three models side by side, with per-model voice, format, and speed controls.
</Callout>

## Available Models [#available-models]

Browse all speech generation models, with up-to-date pricing, on the
[models page](https://offrail.ai/models?filters=1\&audioGeneration=true).

<Callout type="info">
  Billing varies by model family. Some models are billed on token usage reported
  by the provider (input text tokens and output audio tokens), while others are
  billed on input character count (those return audio bytes without usage data).
  See the [models
  page](https://offrail.ai/models?filters=1\&audioGeneration=true) for each
  model's exact pricing.
</Callout>

## Parameters [#parameters]

| Parameter         | Type   | Default  | Description                                                                                                                                                                     |
| ----------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model`           | string | required | The speech model to use                                                                                                                                                         |
| `input`           | string | required | The text to synthesize into speech                                                                                                                                              |
| `voice`           | string | model    | A prebuilt voice. Defaults to `Kore` (Gemini), `alloy` (OpenAI), `Sarah` (ElevenLabs), or the model's first voice on Qwen (`longanlingxin` on Plus, `longanhuan_v3.6` on Flash) |
| `response_format` | string | model    | Audio format. OpenAI: `mp3` (default), `opus`, `aac`, `flac`, `wav`, `pcm`. ElevenLabs: `mp3` (default), `wav`, `pcm`, `opus`. Gemini: `wav` (default), `pcm`. Qwen: `wav`      |
| `instructions`    | string | —        | Optional style/delivery directive prepended to the input (e.g. `"Say cheerfully"`)                                                                                              |
| `speed`           | number | —        | Accepted for OpenAI compatibility, but not applied by Gemini speech models                                                                                                      |

<Callout type="info">
  Gemini speech models return raw PCM audio. OffRail wraps it in a WAV container
  by default (`response_format: "wav"`), or returns the raw 16-bit little-endian
  PCM at 24 kHz when `response_format: "pcm"` is requested. Other formats
  such as `mp3` are only available on the OpenAI models, which return the audio
  already encoded in the requested format.
</Callout>

## curl [#curl]

```bash
curl -X POST "https://api.offrail.ai/v1/audio/speech" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-2.5-flash-preview-tts",
    "input": "Hello, welcome to OffRail!",
    "voice": "Kore"
  }' \
  --output speech.wav
```

## OpenAI SDK [#openai-sdk]

Works with the standard OpenAI client library — just point the base URL to
OffRail.

```ts
import OpenAI from "openai";
import { writeFileSync } from "fs";

const openai = new OpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const response = await openai.audio.speech.create({
	model: "gemini-2.5-flash-preview-tts",
	voice: "Kore",
	input: "Hello, welcome to OffRail!",
});

const buffer = Buffer.from(await response.arrayBuffer());
writeFileSync("speech.wav", buffer);
```

## Streaming [#streaming]

<Callout type="info">
  Streaming speech responses (chunked audio or `stream_format: "sse"`) are not
  supported yet. The endpoint always returns the complete audio file in a single
  response, so there is no low-latency, play-as-you-go output for now.
</Callout>

## Voices [#voices]

Gemini exposes 30 prebuilt voices. A few common ones:
`Kore`, `Puck`, `Zephyr`, `Charon`, `Fenrir`, `Leda`, `Orus`, `Aoede`. When
`voice` is omitted on a Gemini model, `Kore` is used.

OpenAI voices include `alloy`, `ash`, `ballad`, `coral`, `echo`, `fable`,
`nova`, `onyx`, `sage`, `shimmer`, and `verse`. When `voice` is omitted on an
OpenAI model, `alloy` is used.

ElevenLabs models accept 20 named voices, including `Sarah`, `Aria`, `Roger`,
`Laura`, `Charlie`, `George`, `Charlotte`, `Jessica`, `Brian`, and `Lily`. When
`voice` is omitted on an ElevenLabs model, `Sarah` is used. A raw ElevenLabs
voice id is also accepted directly.

Qwen-Audio-3.0-TTS voices are model-specific and cannot be mixed between
models: `longanlingxin` and `longanlufeng` on Plus (default `longanlingxin`),
and `longanhuan_v3.6`, `longjielidou_v3.6`, `loongeva_v3.6`, and `loongjohn` on
Flash (default `longanhuan_v3.6`).

## ElevenLabs [#elevenlabs]

The four ElevenLabs models are billed per **input character** (see the [models
page](https://offrail.ai/models?filters=1\&audioGeneration=true) for rates):

* `eleven-multilingual-v2` — most lifelike, rich emotional expression, 29 languages
* `eleven-v3` — most expressive and human-like, 70+ languages
* `eleven-flash-v2-5` — ultra-low latency, 32 languages
* `eleven-turbo-v2-5` — fast and balanced, 32 languages

```bash
curl -X POST "https://api.offrail.ai/v1/audio/speech" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "eleven-multilingual-v2",
    "input": "Hello, welcome to OffRail!",
    "voice": "Sarah"
  }' \
  --output speech.mp3
```


# Request Timeouts
URL: https://docs.offrail.ai/features/timeouts


# Request Timeouts [#request-timeouts]

The gateway enforces hard time limits on every request. These protect the
platform from stuck upstream connections, but they matter to you directly if
you run long generations or agentic pipelines: a single completion that runs
longer than the limit is terminated.

| Limit             | Default    | Applies to                                               |
| ----------------- | ---------- | -------------------------------------------------------- |
| **Streaming**     | 20 minutes | Streaming completions (`stream: true`)                   |
| **Non-streaming** | 10 minutes | Non-streaming completions                                |
| **End-to-end**    | 25 minutes | Overall ceiling for the whole request, including retries |

## How the limits behave [#how-the-limits-behave]

* **They are total-duration limits, not idle limits.** The timer starts when
  the gateway opens the upstream provider request and keeps running for the
  entire response — a stream is cut at the limit even while it is actively
  producing tokens.
* **They apply per request, not per session or conversation.** Every
  completion call gets a fresh window. A long-running agent that makes many
  calls over hours is unaffected; only a *single* call exceeding the limit
  fails.
* **Tool execution doesn't count.** When a completion finishes with
  `tool_calls`, the HTTP request ends. The time your application spends
  running tools (or coordinating subagents) between requests never consumes
  the window.
* **On timeout** the gateway returns a `504` with type `timeout_error` (see
  [Error Handling](https://docs.offrail.ai/resources/error-handling)). If the limit is hit
  mid-stream, the stream terminates.

## Long-running agentic workloads [#long-running-agentic-workloads]

Coordinator/subagent architectures often hold a "master" completion open while
work happens elsewhere. To stay within the limits:

* **Keep any single completion under 20 minutes.** Structure the coordinator
  as a loop of discrete completions (the standard tool-calling pattern) rather
  than one long-lived request that spans the whole investigation.
* **Stream long generations.** Non-streaming requests are capped at 10
  minutes; streaming raises the per-request budget to 20 minutes and delivers
  partial output as it is produced.
* **Split very long outputs.** If a single generation legitimately needs more
  than 20 minutes (very large outputs on slow models, extensive reasoning),
  break it into continuation requests.

## Changing the limits [#changing-the-limits]

**Per-project overrides (Enterprise)** can be set under **Project Settings →
Routing** — see [Per-Project Routing
Configuration](https://docs.offrail.ai/features/routing#per-project-routing-configuration-enterprise).
Overrides can only *lower* the timeouts: the defaults above are the
infrastructure ceiling on the hosted platform and cannot be raised per
project.

**Self-hosted deployments** control the limits with environment variables on
the gateway service:

| Variable                  | Default   | Controls                  |
| ------------------------- | --------- | ------------------------- |
| `AI_STREAMING_TIMEOUT_MS` | `1200000` | Streaming completions     |
| `AI_TIMEOUT_MS`           | `600000`  | Non-streaming completions |
| `GATEWAY_TIMEOUT_MS`      | `1500000` | End-to-end ceiling        |

<Callout type="warn">
  When raising the limits on a self-hosted deployment, raise the surrounding
  infrastructure in lockstep: your load balancer's backend/response timeout and
  the gateway's shutdown grace period (`SHUTDOWN_GRACE_PERIOD_MS`, and e.g.
  Kubernetes `terminationGracePeriodSeconds`) must all be at least as long as
  the longest stream you allow, or rollouts and intermediaries will still cut
  long requests.
</Callout>

<Callout type="info">
  If your workload genuinely needs single completions longer than 20 minutes on
  the hosted platform, contact us at [contact@offrail.ai](mailto:contact@offrail.ai) — the ceiling is an
  infrastructure setting, not a per-model constraint.
</Callout>


# Transcription
URL: https://docs.offrail.ai/features/transcription


# Transcription [#transcription]

OffRail exposes a dedicated `/v1/audio/transcriptions` endpoint for
speech-to-text. It transcribes audio files into text with word-level
timestamps, optional speaker diarization, and inverse text normalization
(spoken numbers and currencies formatted in their written form).

Use it when you want to:

* Turn recordings, voicemails, or podcast episodes into text
* Generate captions or searchable transcripts with per-word timing
* Feed spoken content into a downstream model or RAG pipeline

For the full request and response schema, see the
[API reference](https://docs.offrail.ai/v1_audio_transcriptions).

## Endpoint [#endpoint]

`POST https://api.offrail.ai/v1/audio/transcriptions`

Authenticate with your OffRail API key:

```bash
-H "Authorization: Bearer $OFFRAIL_API_KEY"
```

The current model is `grok-stt-1-0&#x60;, billed at **$0.10 per hour** of input
audio (against the duration reported by the provider).

## Parameters [#parameters]

The request body is `multipart/form-data`. Either `file` or `url` must be
provided.

| Parameter      | Type   | Default  | Description                                                                                               |
| -------------- | ------ | -------- | --------------------------------------------------------------------------------------------------------- |
| `model`        | string | required | The transcription model to use                                                                            |
| `file`         | file   | —        | The audio file to transcribe (WAV, MP3, OGG, Opus, FLAC, AAC, MP4, M4A, and more)                         |
| `url`          | string | —        | URL of an audio file to download and transcribe instead of uploading one                                  |
| `language`     | string | —        | Language code (e.g. `en`). When set, enables formatting of numbers and currencies into their written form |
| `diarize`      | string | `false`  | When `"true"`, each word in the response includes a `speaker` field identifying the detected speaker      |
| `filler_words` | string | `false`  | When `"true"`, filler words (e.g. "uh", "um") are kept in the transcript instead of being removed         |
| `keyterm`      | string | —        | A key term to bias transcription toward (e.g. product names). Repeat the field for multiple terms         |

## Response [#response]

The response includes the full transcript, audio duration, and word-level
timestamps:

```json
{
	"text": "The balance is $167,983.15.",
	"language": "English",
	"duration": 3.45,
	"words": [
		{ "text": "The", "start": 0.24, "end": 0.48 },
		{ "text": "balance", "start": 0.48, "end": 0.96 },
		{ "text": "is", "start": 0.96, "end": 1.12 },
		{ "text": "$167,983.15.", "start": 1.12, "end": 3.2 }
	]
}
```

## curl [#curl]

```bash
curl -X POST "https://api.offrail.ai/v1/audio/transcriptions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -F model=grok-stt-1-0 \
  -F language=en \
  -F file=@audio.mp3
```

### Transcribing from a URL [#transcribing-from-a-url]

```bash
curl -X POST "https://api.offrail.ai/v1/audio/transcriptions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -F model=grok-stt-1-0 \
  -F url="https://example.com/audio.mp3"
```


# Video Generation
URL: https://docs.offrail.ai/features/video-generation


# Video Generation [#video-generation]

OffRail supports asynchronous video generation through an OpenAI-compatible `POST /v1/videos` flow.

Currently available models:

* **Veo 3.1** through `avalanche` (1080p, 4k) and `google-vertex` (720p, 1080p, 4k)
* **Seedance 2.5** through `bytedance` (480p, 720p, 1080p; clips up to 30 seconds)
* **Seedance 2.0** and **Seedance 1.5 Pro** through `bytedance` (720p, 1080p), **Seedance 2.0 Fast** through `bytedance` (720p only)
* **Seedance 2.0 Mini** through `bytedance` (480p, 720p)
* **KLING v3.0** and **KLING v3.0 Turbo** through `atlascloud` (720p, 1080p; KLING v3.0 also 4k)

You can find the current list of video-capable models on our [models page with the video filter enabled](https://offrail.ai/models?filters=1\&videoGeneration=true) or programmatically through the [/v1/models endpoint](https://docs.offrail.ai/v1_models).

## What Works Today [#what-works-today]

* `POST /v1/videos`
* `GET /v1/videos/{video_id}`
* `GET /v1/videos/{video_id}/content`
* Optional signed callbacks with `callback_url` and `callback_secret`

## Request Format [#request-format]

OffRail currently supports a focused subset of the OpenAI video API.

### Supported fields [#supported-fields]

| Field              | Type    | Required | Description                                                                                                                |
| ------------------ | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `model`            | string  | yes      | Any video-capable model from the filtered models page                                                                      |
| `prompt`           | string  | yes      | Text prompt for the video                                                                                                  |
| `seconds`          | number  | yes      | Duration in seconds. Supported values depend on the model (see below)                                                      |
| `size`             | string  | no       | `widthxheight`, limited to the sizes supported by the selected model and provider                                          |
| `audio`            | boolean | no       | Whether to include audio in the output (default `true`). Only honored when the model supports both audio and silent output |
| `image`            | object  | no       | Optional first frame for image-to-video generation (see first/last frame inputs below)                                     |
| `last_frame`       | object  | no       | Optional ending frame when `image` is provided (see first/last frame inputs below)                                         |
| `reference_images` | array   | no       | One to three provider-specific image inputs                                                                                |
| `input_reference`  | object  | no       | Alias for one or more `reference_images`                                                                                   |
| `reference_videos` | array   | no       | One to three reference video HTTPS URLs (Seedance 2.x only, see below)                                                     |
| `reference_audios` | array   | no       | One to three reference audio HTTPS URLs (Seedance 2.x only, see below)                                                     |
| `callback_url`     | string  | no       | OffRail extension for completion webhooks                                                                                  |
| `callback_secret`  | string  | no       | OffRail extension used to sign webhook deliveries                                                                          |

### Sizes and durations by model [#sizes-and-durations-by-model]

| Model family      | Provider        | Supported sizes                                                                   | Supported durations |
| ----------------- | --------------- | --------------------------------------------------------------------------------- | ------------------- |
| Veo 3.1           | `google-vertex` | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`, `3840x2160`, `2160x3840`        | `4`, `6`, `8`, `10` |
| Veo 3.1           | `avalanche`     | `1920x1080`, `1080x1920`, `3840x2160`, `2160x3840`                                | `8`                 |
| Seedance 2.5      | `bytedance`     | `848x480`, `854x480`, `480x854`, `1280x720`, `720x1280`, `1920x1080`, `1080x1920` | `4`–`30`            |
| Seedance 2.0      | `bytedance`     | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`                                  | `4`–`15`            |
| Seedance 2.0 Fast | `bytedance`     | `1280x720`, `720x1280`                                                            | `4`–`15`            |
| Seedance 2.0 Mini | `bytedance`     | `1280x720`, `720x1280`, `848x480`, `854x480`, `480x854`                           | `4`–`15`            |
| Seedance 1.5 Pro  | `bytedance`     | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`                                  | `5`, `10`           |
| KLING v3.0        | `atlascloud`    | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`, `3840x2160`, `2160x3840`        | `5`, `10`           |
| KLING v3.0 Turbo  | `atlascloud`    | `1280x720`, `720x1280`, `1920x1080`, `1080x1920`                                  | `5`, `10`           |

Requests return `400` when the selected provider cannot serve the requested `size` or `seconds`. Seedance and KLING v3.0 derive `aspect_ratio` from the requested `size` (16:9 for landscape, 9:16 for portrait).

**KLING v3.0 Turbo** always generates audio and does not support `audio: false`; silent requests return a `400`. Use **KLING v3.0** (`kling-v3-0`) when you need silent output.

### First/last frame inputs [#firstlast-frame-inputs]

Frame inputs interpolate a video between a starting frame and an optional ending frame. You provide the first frame as `image` and, optionally, the ending frame as `last_frame`. The gateway tags each one with the correct role for the provider, so you don't set roles yourself.

| Field        | Required           | Accepted input                   | Available on                                                                                                                    |
| ------------ | ------------------ | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `image`      | for frame mode     | HTTPS URL **or** base64 data URL | **Seedance 2.x** (`bytedance`), **KLING v3.0 / Turbo** (`atlascloud`), Veo 3.1 (`google-vertex`, `avalanche`), `minimax`, `xai` |
| `last_frame` | no (needs `image`) | HTTPS URL **or** base64 data URL | **Seedance 2.x** (`bytedance`), **KLING v3.0 / Turbo** (`atlascloud`), Veo 3.1 (`google-vertex`, `avalanche`)                   |

#### Rules and limits [#rules-and-limits]

* **Seedance scope.** Frame inputs are supported on **Seedance 2.5**, **Seedance 2.0**, **Seedance 2.0 Fast**, and **Seedance 2.0 Mini** (`seedance-2-5`, `seedance-2-0`, `seedance-2-0-fast`, `seedance-2-0-mini`). Sending `image`/`last_frame` to Seedance 1.5 Pro or any other ByteDance model returns a `400`.
* **KLING scope.** Frame inputs are supported on **KLING v3.0** and **KLING v3.0 Turbo** (`kling-v3-0`, `kling-v3-0-turbo`). The gateway uploads base64 frames to AtlasCloud's media endpoint automatically, so both HTTPS URLs and base64 data URLs are accepted.
* **`last_frame` requires `image`.** Providing `last_frame` without `image` returns a `400`.
* **Not combinable with references.** First/last frame inputs (`image`, `last_frame`) cannot be combined with reference inputs (`reference_images`, `input_reference`, `reference_videos`, `reference_audios`).

#### Example (Seedance 2.0) [#example-seedance-20]

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "Morph smoothly from the first frame into the last frame",
    "seconds": 5,
    "size": "1280x720",
    "image": { "image_url": "https://example.com/first-frame.png" },
    "last_frame": { "image_url": "https://example.com/last-frame.png" }
  }'
```

#### Example (KLING v3.0 image-to-video) [#example-kling-v30-image-to-video]

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kling-v3-0",
    "prompt": "Animate the scene with gentle camera motion",
    "seconds": 5,
    "size": "1280x720",
    "image": { "image_url": "https://example.com/first-frame.png" }
  }'
```

### Reference-guided generation (Seedance 2.x) [#reference-guided-generation-seedance-2x]

Seedance 2.x (`seedance-2-5`, `seedance-2-0`, `seedance-2-0-fast`, `seedance-2-0-mini`) can generate a video that is guided by reference **images**, **videos**, and **audio** — sometimes called omni-reference. You attach references as top-level fields in the same `POST /v1/videos` payload; the gateway forwards each one to the provider tagged with the correct role, so you don't set roles yourself.

| Reference type | Payload field                                | Count | Accepted input                   | Available on                                         |
| -------------- | -------------------------------------------- | ----- | -------------------------------- | ---------------------------------------------------- |
| Image          | `reference_images` (`input_reference` alias) | 1–3   | HTTPS URL **or** base64 data URL | Seedance 2.x, Veo 3.1 (`google-vertex`, `avalanche`) |
| Video          | `reference_videos`                           | 1–3   | HTTPS URL only                   | Seedance 2.x                                         |
| Audio          | `reference_audios`                           | 1–3   | HTTPS URL only                   | Seedance 2.x                                         |

Each list item accepts either a bare URL string or an object form:

* `reference_images`: `"https://…/subject.png"` or `{ "image_url": "https://…/subject.png" }`
* `reference_videos`: `"https://…/motion.mp4"` or `{ "video_url": "https://…/motion.mp4" }`
* `reference_audios`: `"https://…/track.mp3"` or `{ "audio_url": "https://…/track.mp3" }`

You can mix all three reference types in one request. The `prompt` can be a light instruction (for example `"adapt this to show more detail"`) — the references drive the result.

#### Rules and limits [#rules-and-limits-1]

* **HTTPS only for video and audio.** `reference_videos` and `reference_audios` must be publicly reachable HTTPS URLs (the provider fetches them). base64 data URLs are rejected for video/audio; images may be HTTPS URLs or base64 data URLs.
* **Reference video resolution.** Seedance requires reference video frames to be at least \~409,600 pixels (roughly 480p or larger). Low-resolution clips such as 360p are rejected with a `400`.
* **Not combinable with frames.** Reference inputs (`reference_images`, `reference_videos`, `reference_audios`) cannot be combined with the first/last frame inputs (`image`, `last_frame`).
* **Provider scope.** Reference videos and audio are only supported on Seedance 2.x models; sending them to other models returns a `400`. **KLING v3.0** does not support any reference inputs (`reference_images`, `reference_videos`, `reference_audios`); use first/last frame inputs instead.
* **Moderation still applies.** The output is subject to the provider's content moderation. Blocked generations finish as `failed` and are logged with a `content_filter` finish reason.

#### Examples [#examples]

Reference images only (subjects / style):

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "The subject walks through a neon-lit market at night",
    "seconds": 5,
    "size": "1280x720",
    "reference_images": [
      { "image_url": "https://example.com/subject.png" },
      { "image_url": "https://example.com/style.png" }
    ]
  }'
```

Reference video only (motion / scene — let the clip drive the output):

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "adapt this to show more detail",
    "seconds": 5,
    "size": "1280x720",
    "reference_videos": ["https://example.com/reference-motion.mp4"]
  }'
```

All three reference types combined:

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "seedance-2-0",
    "prompt": "The subject performs the choreography from the reference video",
    "seconds": 5,
    "size": "1280x720",
    "reference_images": [
      { "image_url": "https://example.com/subject.png" }
    ],
    "reference_videos": [
      "https://example.com/reference-motion.mp4"
    ],
    "reference_audios": [
      "https://example.com/reference-track.mp3"
    ]
  }'
```

### Not supported yet [#not-supported-yet]

* multipart uploads
* `n` values other than `1`
* remix/list/delete video endpoints

## Create a Video [#create-a-video]

Video generation requires at least `$1.00` in available organization credits before the job is submitted upstream.

Pricing is per second of generated video. For Seedance and KLING v3.0, enabling audio can increase the per-second rate on models that price audio and video separately.

Veo 3.1:

| Model                           | Provider        | Supported sizes                                  | Price            |
| ------------------------------- | --------------- | ------------------------------------------------ | ---------------- |
| `veo-3.1-generate-preview`      | `google-vertex` | `1280x720`, `720x1280`, `1920x1080`, `1080x1920` | `$0.40 / second` |
| `veo-3.1-fast-generate-preview` | `google-vertex` | `1280x720`, `720x1280`, `1920x1080`, `1080x1920` | `$0.15 / second` |
| `veo-3.1-generate-preview`      | `google-vertex` | `3840x2160`, `2160x3840`                         | `$0.60 / second` |
| `veo-3.1-fast-generate-preview` | `google-vertex` | `3840x2160`, `2160x3840`                         | `$0.35 / second` |
| `veo-3.1-generate-preview`      | `avalanche`     | `1920x1080`, `1080x1920`                         | `$0.40 / second` |
| `veo-3.1-fast-generate-preview` | `avalanche`     | `1920x1080`, `1080x1920`                         | `$0.15 / second` |
| `veo-3.1-generate-preview`      | `avalanche`     | `3840x2160`, `2160x3840`                         | `$0.60 / second` |
| `veo-3.1-fast-generate-preview` | `avalanche`     | `3840x2160`, `2160x3840`                         | `$0.35 / second` |

Seedance (ByteDance):

| Model               | Provider    | Resolution | With audio          | Video only          |
| ------------------- | ----------- | ---------- | ------------------- | ------------------- |
| `seedance-2-5`      | `bytedance` | 480p       | `$0.1028 / second`  | `$0.1028 / second`  |
| `seedance-2-5`      | `bytedance` | 720p       | `$0.2311 / second`  | `$0.2311 / second`  |
| `seedance-2-5`      | `bytedance` | 1080p      | `$0.52 / second`    | `$0.52 / second`    |
| `seedance-2-0`      | `bytedance` | 720p       | `$0.1512 / second`  | `$0.1512 / second`  |
| `seedance-2-0`      | `bytedance` | 1080p      | `$0.3402 / second`  | `$0.3402 / second`  |
| `seedance-2-0-fast` | `bytedance` | 720p       | `$0.121 / second`   | `$0.121 / second`   |
| `seedance-2-0-mini` | `bytedance` | 480p       | `$0.0378 / second`  | `$0.0378 / second`  |
| `seedance-2-0-mini` | `bytedance` | 720p       | `$0.0756 / second`  | `$0.0756 / second`  |
| `seedance-1-5-pro`  | `bytedance` | 720p       | `$0.05184 / second` | `$0.02592 / second` |
| `seedance-1-5-pro`  | `bytedance` | 1080p      | `$0.1166 / second`  | `$0.05832 / second` |

KLING (AtlasCloud):

| Model              | Provider     | Resolution | With audio        | Video only        |
| ------------------ | ------------ | ---------- | ----------------- | ----------------- |
| `kling-v3-0`       | `atlascloud` | 720p       | `$0.126 / second` | `$0.084 / second` |
| `kling-v3-0`       | `atlascloud` | 1080p      | `$0.168 / second` | `$0.112 / second` |
| `kling-v3-0`       | `atlascloud` | 4k         | `$0.42 / second`  | `$0.42 / second`  |
| `kling-v3-0-turbo` | `atlascloud` | 720p       | `$0.168 / second` | n/a (audio only)  |
| `kling-v3-0-turbo` | `atlascloud` | 1080p      | `$0.21 / second`  | n/a (audio only)  |

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "veo-3.1-generate-preview",
    "prompt": "A cinematic aerial shot flying above a rainforest waterfall at sunrise",
    "seconds": 8,
    "size": "1920x1080"
  }'
```

Example response:

```json
{
	"id": "v_123",
	"object": "video",
	"model": "veo-3.1-generate-preview",
	"status": "queued",
	"progress": 0,
	"created_at": 1773600000,
	"completed_at": null,
	"expires_at": null,
	"error": null
}
```

## Retrieve Job Status [#retrieve-job-status]

```bash
curl "https://api.offrail.ai/v1/videos/v_123" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY"
```

Typical statuses:

* `queued`
* `in_progress`
* `completed`
* `failed`
* `canceled`
* `expired`

`avalanche` requests for `1080p` and `4k` stay `in_progress` until the upgraded output is ready. The gateway keeps polling the upstream upgrade endpoints and only marks the job `completed` once the requested resolution is available.

`google-vertex` follows Vertex AI's long-running operation flow. The gateway submits Veo generation with `predictLongRunning`, polls with `fetchPredictOperation`, and streams the final bytes through the gateway content endpoint once the operation is done.

`bytedance` uses the ModelArk `/contents/generations/tasks` endpoint. The gateway submits the job, polls the upstream task status, and exposes the final video bytes through the gateway content endpoint once the task succeeds.

`atlascloud` uses the AtlasCloud `/api/v1/model/generateVideo` endpoint and polls `/api/v1/model/prediction/{id}` for status. The gateway resolves the upstream KLING variant (standard, turbo, or 4k) and task type (text-to-video or image-to-video) from your request, then streams the final video bytes through the gateway content endpoint once the prediction completes.

## Download the Video [#download-the-video]

Once the job is complete, stream the resulting video bytes from the content endpoint:

```bash
curl "https://api.offrail.ai/v1/videos/v_123/content" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  --output video.mp4
```

## Signed Callbacks [#signed-callbacks]

OffRail can notify your application when the job reaches a terminal state.

```bash
curl -X POST "https://api.offrail.ai/v1/videos" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "veo-3.1-fast-generate-preview",
    "prompt": "A slow-motion close-up of waves crashing against black volcanic rock",
    "seconds": 8,
    "callback_url": "https://example.com/webhooks/video",
    "callback_secret": "whsec_your_secret_here"
  }'
```

### Delivery behavior [#delivery-behavior]

* Callbacks are sent only for terminal states in v1
* Event types are `video.completed` and `video.failed`
* Deliveries retry with exponential backoff on network errors, timeouts, and non-2xx responses
* Each attempt is recorded internally in the webhook delivery log table

### Headers [#headers]

* `webhook-id`
* `webhook-timestamp`
* `webhook-signature`

### Signature format [#signature-format]

OffRail signs the string:

```text
{webhook-id}.{webhook-timestamp}.{raw-request-body}
```

using HMAC-SHA256 with your `callback_secret`, then sends:

```text
webhook-signature: v1,{base64_signature}
```

### Verification example [#verification-example]

```ts
import { createHmac, timingSafeEqual } from "node:crypto";

function verifyWebhook(
	body: string,
	webhookId: string,
	webhookTimestamp: string,
	webhookSignature: string,
	secret: string,
) {
	const expected = createHmac("sha256", secret)
		.update(`${webhookId}.${webhookTimestamp}.${body}`)
		.digest("base64");

	const provided = webhookSignature.replace(/^v1,/, "");

	return timingSafeEqual(Buffer.from(expected), Buffer.from(provided));
}
```

## Related Docs [#related-docs]

* [Image Generation](https://docs.offrail.ai/features/image-generation)
* [Routing](https://docs.offrail.ai/features/routing)
* [Models API](https://docs.offrail.ai/v1_models)


# Vision Support
URL: https://docs.offrail.ai/features/vision


# Vision Support [#vision-support]

OffRail supports vision-enabled models that can analyze and describe images. You can provide images via HTTPS URLs or inline base64-encoded data.

## Vision-Enabled Models [#vision-enabled-models]

You can find all vision-enabled models on our [models page with vision filter](https://offrail.ai/models?filters=1\&vision=true). These models can process both text and image content in the same request.

## Image Formats [#image-formats]

### Using HTTPS URLs [#using-https-urls]

You can provide any publicly accessible HTTPS URL pointing to an image:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What do you see in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/image.jpg"
            }
          }
        ]
      }
    ]
  }'
```

### Using Base64 Inline Data [#using-base64-inline-data]

You can also provide images as base64-encoded data URIs:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD..."
            }
          }
        ]
      }
    ]
  }'
```

## Content Array Format [#content-array-format]

When using vision models, the `content` field should be an array containing both text and image content blocks:

* **Text content**: `{"type": "text", "text": "Your message"}`
* **Image content**: `{"type": "image_url", "image_url": {"url": "image_url_or_data_uri"}}`

## Multiple Images [#multiple-images]

You can include multiple images in a single request:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Compare these two images"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/image1.jpg"
            }
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://example.com/image2.jpg"
            }
          }
        ]
      }
    ]
  }'
```

## Simple String Content [#simple-string-content]

<Callout type="info">
  For vision models, you can still use simple string content for text-only
  messages. The array format is only required when including images.
</Callout>

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How can you help me today?"
      }
    ]
  }'
```

## Supported Image Types [#supported-image-types]

Vision models typically support common image formats including:

* JPEG (.jpg, .jpeg)
* PNG (.png)
* WebP (.webp)
* GIF (.gif)

The specific formats supported may vary by model provider. Check the individual model documentation for format limitations and file size restrictions.

## Error Handling [#error-handling]

If an image URL is inaccessible or the image format is unsupported, the gateway will handle the error gracefully and may substitute a placeholder or error message in the request to the underlying model.


# Native Web Search
URL: https://docs.offrail.ai/features/web-search


# Native Web Search [#native-web-search]

OffRail supports native web search capabilities that allow models to access real-time information from the internet. This feature is useful for answering questions about current events, recent news, live data, and other time-sensitive information that may not be in the model's training data.

## How It Works [#how-it-works]

When you include the `web_search` tool in your request, the model can search the web to gather relevant information before generating a response:

1. You send a request with the `web_search` tool enabled
2. The model determines if web search is needed based on the query, unless you
   [require one](#requiring-a-search)
3. If needed, the model performs web searches to gather current information
4. The model synthesizes the search results and generates a response
5. Citations are included in the response to show information sources

## Supported Providers [#supported-providers]

Native web search is available on select models. See all models with native web search support on our [models page](https://offrail.ai/models?filters=1\&webSearch=true).

## Basic Usage [#basic-usage]

To enable web search, add the `web_search` tool to your request:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.2",
    "messages": [
      {
        "role": "user",
        "content": "What is the current weather in San Francisco?"
      }
    ],
    "tools": [
      {
        "type": "web_search"
      }
    ]
  }'
```

### Example Response [#example-response]

```json
{
	"id": "chatcmpl-abc123",
	"object": "chat.completion",
	"created": 1234567890,
	"model": "openai/gpt-5.2",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "The current weather in San Francisco is 57°F (14°C) with mostly cloudy skies...",
				"annotations": [
					{
						"type": "url_citation",
						"url": "https://weather.com/...",
						"title": "San Francisco Weather"
					}
				]
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 15,
		"completion_tokens": 150,
		"total_tokens": 165,
		"cost": 0.0315
	}
}
```

## Web Search Options [#web-search-options]

The `web_search` tool accepts optional configuration parameters:

### User Location [#user-location]

Provide location context to get more relevant local search results:

```json
{
	"type": "web_search",
	"user_location": {
		"city": "San Francisco",
		"region": "California",
		"country": "US",
		"timezone": "America/Los_Angeles"
	}
}
```

### Search Context Size [#search-context-size]

Control the amount of web content retrieved (OpenAI only):

```json
{
	"type": "web_search",
	"search_context_size": "medium"
}
```

Available values:

* `low` - Minimal search context, faster responses
* `medium` - Balanced context (default)
* `high` - Maximum search context, more comprehensive

### Max Uses [#max-uses]

Limit the number of searches per request (provider-dependent):

```json
{
	"type": "web_search",
	"max_uses": 3
}
```

## Requiring a Search [#requiring-a-search]

By default the `web_search` tool offers the model a search and lets it judge
whether the question needs one. To require a search on every request, set
`tool_choice`:

```json
{
	"model": "...",
	"messages": [{ "role": "user", "content": "What shipped in AI this week?" }],
	"tools": [{ "type": "web_search" }],
	"tool_choice": { "type": "web_search" }
}
```

Reach for this sparingly. A forced search is billed on every request that
carries it, and the retrieved snippets are appended to your prompt, so they are
billed as input tokens too — on a question the model could have answered from
memory, you pay for both and gain nothing. If you are building a chat interface
with a "web search" toggle, leaving the tool attached with the default
`tool_choice` is usually what you want, so that follow-ups like "shorter,
please" do not trigger a search.

A few upstreams have no model-elected search at all and can only search when
asked to. Requiring a search is the only way to use their search; without it
they behave like a model that decided not to search, and the gateway prefers to
route a merely offered tool to a provider that can make that decision for
itself. You can find the models with native web search on the
[models page](https://offrail.ai/models?filters=1\&webSearch=true).

## Using with SDKs [#using-with-sdks]

### OpenAI SDK (Python) [#openai-sdk-python]

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://api.offrail.ai/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="gpt-5.2",
    messages=[
        {"role": "user", "content": "What are the latest news headlines today?"}
    ],
    tools=[{"type": "web_search"}]
)

print(response.choices[0].message.content)
```

### OpenAI SDK (TypeScript) [#openai-sdk-typescript]

```typescript
import OpenAI from "openai";

const client = new OpenAI({
	baseURL: "https://api.offrail.ai/v1",
	apiKey: "your-api-key",
});

const response = await client.chat.completions.create({
	model: "gpt-5.2",
	messages: [{ role: "user", content: "What are the latest tech news?" }],
	tools: [{ type: "web_search" }],
});

console.log(response.choices[0].message.content);
```

## Streaming [#streaming]

Web search works with streaming responses. Citations are included in the final chunks:

```bash
curl -X POST "https://api.offrail.ai/v1/chat/completions" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.2",
    "messages": [
      {"role": "user", "content": "What is the current stock price of Apple?"}
    ],
    "tools": [{"type": "web_search"}],
    "stream": true
  }'
```

## Citations and Sources [#citations-and-sources]

Web search responses include citations to show where information was sourced from. These appear in the `annotations` field of the message:

```json
{
	"annotations": [
		{
			"type": "url_citation",
			"url": "https://example.com/article",
			"title": "Article Title",
			"start_index": 0,
			"end_index": 50
		}
	]
}
```

<Callout type="info">
  Citation format may vary slightly between providers, but OffRail normalizes
  them into a consistent structure.
</Callout>

## Cost Tracking [#cost-tracking]

Web search costs are rolled into the total `cost` reported in the usage object:

```json
{
	"usage": {
		"prompt_tokens": 15,
		"completion_tokens": 150,
		"total_tokens": 165,
		"cost": 0.0125,
		"cost_details": {
			"upstream_inference_cost": 0.0115,
			"upstream_inference_prompt_cost": 0.0015,
			"upstream_inference_completions_cost": 0.01,
			"total_cost": 0.0125,
			"input_cost": 0.0015,
			"output_cost": 0.01,
			"web_search_cost": 0.001
		}
	}
}
```

Web search is billed at $0.01 per search call for reasoning models (GPT-5, o-series) and $0.025 per call for non-reasoning models. The web search charge is included in the top-level `cost` value and surfaced separately as `cost_details.web_search_cost`.

## Combining with Function Tools [#combining-with-function-tools]

You can use web search alongside regular function tools:

```json
{
	"tools": [
		{ "type": "web_search" },
		{
			"type": "function",
			"function": {
				"name": "get_weather",
				"description": "Get weather for a location",
				"parameters": {
					"type": "object",
					"properties": {
						"location": { "type": "string" }
					}
				}
			}
		}
	]
}
```

<Callout type="warning">
  Some dedicated search models only support web search and do not support
  additional function tools. Use `gpt-5.2` or other GPT-5 series models if you
  need both web search and function tools.
</Callout>

## Use Cases [#use-cases]

### Current Events and News [#current-events-and-news]

```json
{
	"messages": [
		{ "role": "user", "content": "What are the major news stories today?" }
	],
	"tools": [{ "type": "web_search" }]
}
```

### Real-Time Data [#real-time-data]

```json
{
	"messages": [
		{ "role": "user", "content": "What is the current price of Bitcoin?" }
	],
	"tools": [{ "type": "web_search" }]
}
```

### Research and Fact-Checking [#research-and-fact-checking]

```json
{
	"messages": [
		{
			"role": "user",
			"content": "What are the latest findings on climate change?"
		}
	],
	"tools": [{ "type": "web_search" }]
}
```

### Local Information [#local-information]

```json
{
	"messages": [
		{
			"role": "user",
			"content": "What restaurants are open near me right now?"
		}
	],
	"tools": [
		{
			"type": "web_search",
			"user_location": {
				"city": "New York",
				"country": "US"
			}
		}
	]
}
```

## Best Practices [#best-practices]

1. **Use GPT-5.2**: For the best web search experience with full tool support, use `gpt-5.2`
2. **Provide location context**: When queries are location-dependent, include `user_location` for more relevant results
3. **Monitor costs**: Web search incurs per-query costs in addition to token costs
4. **Check citations**: Always review the citations in responses to verify information sources
5. **Use streaming**: For user-facing applications, enable streaming to show responses as they're generated

## Error Handling [#error-handling]

If you try to use web search with a model that doesn't support it:

```json
{
	"error": {
		"message": "Model gpt-4o does not support native web search. Remove the web_search tool or use a model that supports it. See https://offrail.ai/models?features=webSearch for supported models.",
		"type": "invalid_request_error"
	}
}
```

To avoid this error, only use the `web_search` tool with [native web search enabled models](https://offrail.ai/models?filters=1\&webSearch=true).


# Autohand Code Integration
URL: https://docs.offrail.ai/guides/autohand


Autohand Code is an autonomous AI coding agent that works in your terminal, IDE, and Slack. With OffRail, you can route all Autohand Code requests through a single gateway—use any of 200+ models from 40+ providers, with full cost tracking and smart routing.

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for OffRail [#sign-up-for-offrail]

    [Sign up free](https://offrail.ai/signup) — no credit card required. Copy your API key from the dashboard.
  </Step>

  <Step>
    ### Set Environment Variables [#set-environment-variables]

    Configure Autohand Code to use OffRail:

    ```bash
    export OPENAI_BASE_URL=https://api.offrail.ai/v1
    export OPENAI_API_KEY=orl_your_api_key_here
    ```
  </Step>

  <Step>
    ### Run Autohand Code [#run-autohand-code]

    ```bash
    autohand
    ```

    All requests will now be routed through OffRail.
  </Step>
</Steps>

## Why Use OffRail with Autohand Code [#why-use-offrail-with-autohand-code]

* **200+ models** — GPT-5, Claude Opus, Gemini, Llama, and more from 40+ providers
* **Smart routing** — Automatically selects the best provider based on uptime, throughput, price, and latency
* **Cost tracking** — Monitor exactly how much each autonomous agent costs
* **Single bill** — No need to manage multiple API provider accounts
* **Response caching** — Repeated requests hit cache automatically
* **Automatic failover** — If one provider is down, requests route to another

## Configuration File [#configuration-file]

You can also configure OffRail in Autohand Code's config file:

```json
{
	"provider": {
		"offrail": {
			"baseUrl": "https://api.offrail.ai/v1",
			"apiKey": "orl_your_api_key_here"
		}
	},
	"model": "gpt-5"
}
```

## Choosing Models [#choosing-models]

You can use any model from the [models page](https://offrail.ai/models).

| Model                    | Best For                                    |
| ------------------------ | ------------------------------------------- |
| `gpt-5`                  | Latest OpenAI flagship, highest quality     |
| `claude-opus-4-6`        | Anthropic's most capable model              |
| `claude-sonnet-4-6`      | Fast reasoning with extended thinking       |
| `gemini-3.1-pro-preview` | Google's latest flagship, 1M context window |
| `o3`                     | Advanced reasoning tasks                    |
| `gpt-5-mini`             | Cost-effective, quick responses             |
| `gemini-3.6-flash`       | Fast responses, good for high-volume        |
| `deepseek-v3.1`          | Open-source with vision and tools           |

## Autohand Code Features with OffRail [#autohand-code-features-with-offrail]

### Terminal (CLI) [#terminal-cli]

Autohand Code CLI works seamlessly with OffRail. Set the environment variables and use all Autohand Code commands as normal—multi-file editing, agentic search, and autonomous code generation all work out of the box.

### IDE Integration [#ide-integration]

Autohand Code's VS Code and Zed extensions respect the same environment variables. Set them in your shell profile and the IDE integration will automatically route through OffRail.

### Slack Integration [#slack-integration]

When using Autohand Code through Slack, configure the OffRail base URL in your Autohand Code server settings to route all Slack-triggered coding tasks through the gateway.

## Monitoring Usage [#monitoring-usage]

Once configured, all Autohand Code requests appear in your OffRail dashboard:

* **Request logs** — See every prompt and response
* **Cost breakdown** — Track spending by model and time period
* **Usage analytics** — Understand your AI usage patterns

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Claude Code Integration
URL: https://docs.offrail.ai/guides/claude-code


Claude Code is locked to Anthropic's API by default. With OffRail, you can point it at any model—GPT-5, Gemini, Llama, or 180+ others—while keeping the same Anthropic API format Claude Code expects.

Three environment variables. No code changes. Full cost tracking in your dashboard.

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for OffRail [#sign-up-for-offrail]

    [Sign up free](https://offrail.ai/signup) — no credit card required. Copy your API key from the dashboard.
  </Step>

  <Step>
    ### Set Environment Variables [#set-environment-variables]

    Configure Claude Code to use OffRail:

    ```bash
    export ANTHROPIC_BASE_URL=https://api.offrail.ai
    export ANTHROPIC_AUTH_TOKEN=orl_your_api_key_here
    # optional: specify a model, otherwise it uses the default Claude model
    export ANTHROPIC_MODEL=gpt-5  # or any model from our catalog
    ```
  </Step>

  <Step>
    ### Run Claude Code [#run-claude-code]

    ```bash
    claude
    ```

    All requests will now be routed through OffRail.
  </Step>
</Steps>

## Why This Works [#why-this-works]

OffRail's `/v1/messages` endpoint speaks Anthropic's API format natively. We handle the translation to each provider behind the scenes. This means:

* **Use any model** — GPT-5, Gemini, Llama, or Claude itself
* **Keep your workflow** — Claude Code doesn't know the difference
* **Track costs** — Every request appears in your OffRail dashboard
* **Automatic caching** — Repeated requests hit cache, saving money

## Choosing Models [#choosing-models]

You can use any model from the [models page](https://offrail.ai/models).

### Use OpenAI's Latest Models [#use-openais-latest-models]

```bash
# Use the latest GPT model
export ANTHROPIC_MODEL=gpt-5

# Use a cost-effective alternative
export ANTHROPIC_MODEL=gpt-5-mini
```

### Use Google's Gemini [#use-googles-gemini]

```bash
export ANTHROPIC_MODEL=gemini-3.1-pro-preview
```

### Use Anthropic's Claude Models [#use-anthropics-claude-models]

```bash
export ANTHROPIC_MODEL=anthropic/claude-3-5-sonnet-20241022
```

## Predefining Models in a Settings File [#predefining-models-in-a-settings-file]

Environment variables are read once at startup, so changing `ANTHROPIC_MODEL` means restarting Claude Code. To switch models mid-session instead, put the configuration in `~/.claude/settings.json` (user-wide) or `.claude/settings.json` (per project) and use `/model` to switch on the fly:

```json title="~/.claude/settings.json"
{
	"env": {
		"ANTHROPIC_BASE_URL": "https://api.offrail.ai",
		"ANTHROPIC_AUTH_TOKEN": "orl_your_api_key_here"
	},
	"model": "claude-sonnet-5"
}
```

`model` sets the model a new session starts on. `/model` overrides it for the running session and saves your choice as the new default.

### Restricting the Model List [#restricting-the-model-list]

`availableModels` limits which models the `/model` picker offers — useful for keeping a team on an approved, budget-appropriate set:

```json title="~/.claude/settings.json"
{
	"availableModels": ["claude-sonnet-5", "claude-haiku-4-5"],
	"fallbackModel": ["claude-haiku-4-5"]
}
```

`fallbackModel` names the models to try when the primary one is unavailable, capped at three.

`availableModels` only filters the rows Claude Code already has — it never creates new ones. To put a non-Claude model in the picker, see [Adding Non-Claude Models to the Picker](#adding-non-claude-models-to-the-picker).

## Fetching Models From OffRail [#fetching-models-from-offrail]

Claude Code can populate its `/model` picker directly from our catalog instead of you hardcoding IDs. Set the discovery flag alongside the base URL:

```bash
export ANTHROPIC_BASE_URL=https://api.offrail.ai
export ANTHROPIC_AUTH_TOKEN=orl_your_api_key_here
export CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1
```

On startup Claude Code calls our [`/v1/models`](https://api.offrail.ai/v1/models) endpoint and adds what it returns to the picker, labeled **From gateway**. Entries are cached in `~/.claude/cache/gateway-models.json` and refreshed on each launch, so a failed lookup falls back to the previous list rather than breaking your session. Requires Claude Code v2.1.129 or later.

<Callout type="warn">
  Claude Code drops discovered models whose ID does not start with `claude` or
  `anthropic`, before they ever reach the picker — the cache it writes to
  `~/.claude/cache/gateway-models.json` contains only the surviving entries.
  Discovery therefore surfaces just the Claude models in our catalog. GPT-5,
  Gemini, and custom models are filtered out by the client, not by the gateway.
</Callout>

### Adding Non-Claude Models to the Picker [#adding-non-claude-models-to-the-picker]

The `/model&#x60; picker is a Claude-model list. Its own header says so: &#x2A;"Switch between Claude models… For other/previous model names, specify with `--model`."* Neither gateway discovery nor `availableModels` adds a non-Claude row — `availableModels` filters the rows Claude Code already has rather than creating new ones.

`ANTHROPIC_CUSTOM_MODEL_OPTION` is the one setting that adds a non-Claude row:

```bash
export ANTHROPIC_CUSTOM_MODEL_OPTION=gemini-3.5-flash
export ANTHROPIC_CUSTOM_MODEL_OPTION_NAME="Gemini 3.5 Flash"
export ANTHROPIC_CUSTOM_MODEL_OPTION_DESCRIPTION="Routed through OffRail"
```

The entry appears at the bottom of the picker under your chosen name. Only one is supported at a time, so it suits a single alternate model rather than a menu. If you also set `availableModels`, include this ID there or it will be filtered back out.

### Selecting Any Other Model [#selecting-any-other-model]

For everything else — including [custom models](https://docs.offrail.ai/features/custom-providers) — name the model directly. These paths accept any ID our gateway routes, with no picker row involved:

```bash
export ANTHROPIC_MODEL=gemini-3.5-flash   # session default
claude --model gemini-3.5-flash           # single session
```

To rotate between several non-Claude models, keep a per-project `.claude/settings.json` with the `model` you want for that repo.

## Environment Variables [#environment-variables]

### ANTHROPIC\_MODEL [#anthropic_model]

Specifies the main model to use for primary requests.

```bash
export ANTHROPIC_MODEL=gpt-5
```

### Complete Configuration Example [#complete-configuration-example]

```bash
export ANTHROPIC_BASE_URL=https://api.offrail.ai
export ANTHROPIC_AUTH_TOKEN=orl_your_api_key_here
export ANTHROPIC_MODEL=gpt-5
export ANTHROPIC_SMALL_FAST_MODEL=gpt-5-nano
```

## Making Manual API Requests [#making-manual-api-requests]

If you want to test the endpoint directly, you can make manual requests:

```bash
curl -X POST "https://api.offrail.ai/v1/messages" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "max_tokens": 100
  }'
```

### Response Format [#response-format]

The endpoint returns responses in Anthropic's message format:

```json
{
	"id": "msg_abc123",
	"type": "message",
	"role": "assistant",
	"model": "gpt-5",
	"content": [
		{
			"type": "text",
			"text": "Hello! I'm doing well, thank you for asking. How can I help you today?"
		}
	],
	"stop_reason": "end_turn",
	"stop_sequence": null,
	"usage": {
		"input_tokens": 13,
		"output_tokens": 20
	}
}
```

## What You Get [#what-you-get]

* **Any model in Claude Code** — GPT-5 for heavy lifting, GPT-4o Mini for routine tasks
* **Cost visibility** — See exactly what each coding agent costs
* **One bill** — Stop managing separate accounts for OpenAI, Anthropic, Google
* **Response caching** — Repeated requests (like linting the same file) hit cache
* **Discounts** — Check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Cline Integration
URL: https://docs.offrail.ai/guides/cline


[Cline](https://cline.bot) is an autonomous AI coding assistant that lives in your VS Code editor. It can create and edit files, run terminal commands, and help you build complex projects. You can configure Cline to use OffRail for access to multiple AI providers with unified billing and cost tracking.

## Prerequisites [#prerequisites]

* VS Code based IDE installed
* An OffRail API key

## Setup [#setup]

Cline supports OpenAI-compatible API endpoints, making it straightforward to integrate with OffRail.

<Steps>
  <Step>
    ### Install Cline Extension [#install-cline-extension]

    1. Open VS Code
    2. Go to the Extensions view (Cmd/Ctrl + Shift + X)
    3. Search for "Cline"
    4. Click **Install** on the Cline extension

        <img alt="Install Cline Extension" src="__img0" />
  </Step>

  <Step>
    ### Open Cline Settings [#open-cline-settings]

    1. Click on the Cline icon in the VS Code sidebar
    2. Click the settings gear icon in the Cline panel

        <img alt="Cline Settings" src="__img1" />
  </Step>

  <Step>
    ### Configure API Provider [#configure-api-provider]

    1. In the API Provider dropdown, select **OpenAI Compatible**
    2. Enter the following details:
       * **Base URL**: `https://api.offrail.ai/v1`
       * **API Key**: Your OffRail API key
       * **Model ID**: Choose a model (e.g., `claude-opus-4-5-20251101`, `gpt-5.2`, `gemini-3-pro-preview`, `deepseek-3.2`). See [provider-specific routing](https://docs.offrail.ai/features/routing#provider-specific-routing) for more options.

        <img alt="Configure API Provider" src="__img2" />
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    1. Open a project in VS Code
    2. Click on the Cline icon in the sidebar
    3. Type a message like "Create a hello world function in Python"
    4. Cline should respond and offer to create the file

        <img alt="Test Cline" src="__img3" />

    All requests will now be routed through OffRail.
  </Step>
</Steps>

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

## Features [#features]

Once configured, you can use all of Cline's features with OffRail:

### Autonomous Coding [#autonomous-coding]

* Create new files and projects from scratch
* Edit existing code based on natural language instructions
* Refactor and improve code quality

### Terminal Commands [#terminal-commands]

* Run build commands, tests, and scripts
* Install dependencies
* Execute any terminal operation

### File Management [#file-management]

* Create, read, and modify files
* Navigate your codebase
* Search for relevant code

## Model Selection Tips [#model-selection-tips]

### Using Provider-Specific Models [#using-provider-specific-models]

To use a specific provider's version of a model, prefix the model ID with the provider name. See [provider-specific routing](https://docs.offrail.ai/features/routing#provider-specific-routing) for more options.

### Using Discounted Models [#using-discounted-models]

OffRail offers discounted access to some models. Find them on the [models page](https://offrail.ai/models?view=grid\&filters=1\&discounted=true) and copy the model ID.

### Using Free Models [#using-free-models]

Some models are available for free. Browse them on the [models page](https://offrail.ai/models?view=grid\&filters=1\&free=true).

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>

## Benefits of Using OffRail with Cline [#benefits-of-using-offrail-with-cline]

* **Multi-Provider Access**: Use models from OpenAI, Anthropic, Google, and more through a single API
* **Cost Control**: Track and limit your AI spending with detailed usage analytics
* **Unified Billing**: One account for all providers instead of managing multiple API keys
* **Caching**: Reduce costs with response caching for repeated requests
* **Analytics**: Monitor usage patterns and costs in the dashboard


# Codex CLI Integration
URL: https://docs.offrail.ai/guides/codex-cli


Codex CLI is OpenAI's open-source terminal coding agent. By default it connects to OpenAI's API, but with OffRail you can route it through a single gateway—use GPT-5.3 Codex, Gemini, Claude, or any of 200+ models while keeping full cost visibility.

One config file. No code changes. Full cost tracking in your dashboard.

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for OffRail [#sign-up-for-offrail]

    [Sign up free](https://offrail.ai/signup) — no credit card required. Copy your API key from the dashboard.
  </Step>

  <Step>
    ### Log Out of ChatGPT [#log-out-of-chatgpt]

    If you're logged into ChatGPT in Codex CLI, the stored session will override your custom config. Log out first:

    ```bash
    codex logout
    ```
  </Step>

  <Step>
    ### Create Config File [#create-config-file]

    Create or edit `~/.codex/config.toml`:

    ```bash
    model = "auto"
    model_reasoning_effort = "high"
    openai_base_url = "https://api.offrail.ai/v1"
    ```
  </Step>

  <Step>
    ### Run Codex CLI [#run-codex-cli]

    ```bash
    codex
    ```

    On first launch, Codex will prompt you for authentication. Select **Provide your own API key**, then enter your OffRail API key (starts with `orl_`).

    All requests will now be routed through OffRail.
  </Step>
</Steps>

## Why This Works [#why-this-works]

OffRail's `/v1` endpoint is fully OpenAI-compatible. Codex CLI sends requests to our gateway instead of OpenAI directly, and we route them to the right provider behind the scenes. This means:

* **Use any model** — GPT-5.3 Codex, Gemini, Claude, or 180+ others
* **Keep your workflow** — Codex CLI doesn't know the difference
* **Track costs** — Every request appears in your OffRail dashboard
* **Automatic caching** — Repeated requests hit cache, saving money

## Configuration Explained [#configuration-explained]

### Base URL [#base-url]

The `openai_base_url` field points Codex CLI to OffRail instead of OpenAI:

```bash
openai_base_url = "https://api.offrail.ai/v1"
```

### Model Selection [#model-selection]

Use `auto` to let OffRail pick the best model, or set a specific one from the [models page](https://offrail.ai/models):

```bash
model = "auto"
# or pick a specific model
model = "gpt-5.3-codex"
```

### Reasoning Effort [#reasoning-effort]

Control how much reasoning the model uses. Options are `low`, `medium`, and `high`:

```bash
model_reasoning_effort = "high"
```

## Choosing Models [#choosing-models]

Use `auto` to let OffRail pick the best model automatically, or choose a specific one from the [models page](https://offrail.ai/models):

```bash
# let OffRail pick the best model
model = "auto"

# or pick a specific model
model = "gpt-5.3-codex"
```

## What You Get [#what-you-get]

* **Any model in Codex CLI** — GPT-5.3 Codex for heavy lifting, lighter models for routine tasks
* **Cost visibility** — See exactly what each coding agent costs
* **One bill** — Stop managing separate accounts for OpenAI, Anthropic, Google
* **Response caching** — Repeated requests hit cache automatically
* **Discounts** — Check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%

## Troubleshooting [#troubleshooting]

### Data retention required [#data-retention-required]

Older versions of OffRail rejected Responses API requests with:

```
The Responses API requires data retention to be enabled.
```

This is no longer the case — the Responses API (which Codex CLI uses) works regardless of your organization's data retention policy. If you still see this error, the gateway you are talking to is running an outdated version; on a self-hosted deployment, update to the latest release.

### Authentication errors [#authentication-errors]

If you see `401 Unauthorized` or requests going to `api.openai.com` instead of OffRail:

1. Make sure you've run `codex logout` to clear any ChatGPT session
2. Verify `openai_base_url` is set in `~/.codex/config.toml`
3. When Codex prompts for authentication, select **Provide your own API key** and enter your OffRail key (starts with `orl_`)

### Model not found [#model-not-found]

Verify the model ID matches exactly what's listed on the [models page](https://offrail.ai/models). Model IDs are case-sensitive.

### Connection issues [#connection-issues]

Check that `openai_base_url` is set to `https://api.offrail.ai/v1` (note the `/v1` at the end).

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Continue CLI Integration
URL: https://docs.offrail.ai/guides/continue


[Continue](https://docs.continue.dev) is an open-source AI code assistant available as a CLI tool. By configuring it to use OffRail, you get access to 200+ models from 40+ providers with unified cost tracking.

One config file. Any model. Full cost visibility.

## Prerequisites [#prerequisites]

* An OffRail API key — [sign up free](https://offrail.ai/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install Continue CLI [#install-continue-cli]

    Install Continue CLI globally:

    ```bash
    npm install -g @continuedev/cli
    ```

        <img alt="Installing Continue CLI" src="__img0" />
  </Step>

  <Step>
    ### Get Your API Key [#get-your-api-key]

    [Sign up](https://offrail.ai/signup) or log in to your OffRail dashboard. Navigate to **API Keys** and create a new key. Copy it — it starts with `orl_`.
  </Step>

  <Step>
    ### Create a Config File [#create-a-config-file]

    Create the Continue config directory and config file:

    ```bash
    mkdir -p ~/.continue
    ```

    Then create `~/.continue/config.yaml` with your OffRail configuration:

    ```yaml
    name: offrail
    version: 0.0.1
    models:
      - name: claude-sonnet-4-6
        provider: openai
        model: claude-sonnet-4-6
        apiBase: https://api.offrail.ai/v1
        apiKey: orl_your-api-key-here
    ```

        <img alt="Editing config.yaml" src="__img1" />

    <Callout type="info">
      Replace `orl_your-api-key-here` with your actual API key from the dashboard.
    </Callout>
  </Step>

  <Step>
    ### Add More Models (Optional) [#add-more-models-optional]

    Add as many models as you want from the [models page](https://offrail.ai/models):

    ```yaml
    name: offrail
    version: 0.0.1
    models:
      - name: claude-sonnet-4-6
        provider: openai
        model: claude-sonnet-4-6
        apiBase: https://api.offrail.ai/v1
        apiKey: orl_your-api-key-here
      - name: gpt-5.5
        provider: openai
        model: gpt-5.5
        apiBase: https://api.offrail.ai/v1
        apiKey: orl_your-api-key-here
      - name: gemini-3.1-pro
        provider: openai
        model: gemini-3.1-pro
        apiBase: https://api.offrail.ai/v1
        apiKey: orl_your-api-key-here
    ```

    All models use `provider: openai` since OffRail exposes an OpenAI-compatible API.
  </Step>

  <Step>
    ### Start Using Continue [#start-using-continue]

    Launch Continue CLI with the `--config` flag pointing to your config file:

    ```bash
    cn --config ~/.continue/config.yaml
    ```

        <img alt="Continue CLI running with OffRail" src="__img2" />

    All requests now route through OffRail. You'll see usage, costs, and logs in your dashboard.
  </Step>
</Steps>

## Why Use OffRail with Continue [#why-use-offrail-with-continue]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%

## Configuration Details [#configuration-details]

### Provider Setting [#provider-setting]

Always use `provider: openai` in your Continue config. OffRail exposes an OpenAI-compatible API, so Continue's OpenAI provider handles all models correctly — including Claude, Gemini, and others.

### Project-Specific Config [#project-specific-config]

Place a `.continue/config.yaml` in your project root to override the global config for that project:

```yaml
name: project-config
version: 0.0.1
models:
  - name: gpt-5.5
    provider: openai
    model: gpt-5.5
    apiBase: https://api.offrail.ai/v1
    apiKey: orl_your-api-key-here
```

### Using with the --config Flag [#using-with-the---config-flag]

Point to any config file:

```bash
cn --config path/to/config.yaml
```

## Switching Models [#switching-models]

Add multiple models to your config and switch between them in the Continue interface. In the CLI, you can specify a model with the `--model` flag if supported, or update your config file.

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, OffRail automatically fails over to alternative providers if your chosen provider is experiencing downtime. To disable fallback, add a custom header:

```yaml
models:
  - name: claude-sonnet-4-6
    provider: openai
    model: claude-sonnet-4-6
    apiBase: https://api.offrail.ai/v1
    apiKey: orl_your-api-key-here
    requestOptions:
      headers:
        X-No-Fallback: "true"
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.offrail.ai/features/routing) for details.
</Callout>

## Troubleshooting [#troubleshooting]

### "Failed to parse config" error [#failed-to-parse-config-error]

Make sure your config file includes `name` and `version` fields at the top level:

```yaml
name: offrail
version: 0.0.1
models:
  - ...
```

### Onboarding wizard still appears [#onboarding-wizard-still-appears]

If running `cn` without `--config` shows an onboarding prompt, create the sentinel file to skip it:

```bash
touch ~/.continue/.onboarding_complete
```

Or always launch with the `--config` flag to bypass onboarding entirely.

### Model not found [#model-not-found]

Verify the model ID matches exactly what's listed on the [models page](https://offrail.ai/models). Model IDs are case-sensitive.

### Connection timeout [#connection-timeout]

Check that `apiBase` is set to `https://api.offrail.ai/v1` (note the `/v1` at the end).

### Authentication errors [#authentication-errors]

Make sure your `apiKey` starts with `orl_` and is valid. Check your [dashboard](https://offrail.ai/dashboard) to confirm the key is active.

### Provider must be "openai" [#provider-must-be-openai]

OffRail uses an OpenAI-compatible API. Even when using Claude or Gemini models, set `provider: openai` in your Continue config. The gateway handles routing to the correct upstream provider.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Crush Integration
URL: https://docs.offrail.ai/guides/crush


[Crush](https://github.com/charmbracelet/crush) is Charm's glamorous open-source AI coding agent for your terminal. Connecting it to OffRail takes a single provider entry in Crush's config — and gives you access to 200+ models from 40+ providers through one API key, with every request tracked in your dashboard.

## Prerequisites [#prerequisites]

* Crush installed:

  ```bash
  # Homebrew
  brew install charmbracelet/tap/crush

  # NPM
  npm install -g @charmland/crush
  ```

  See the [Crush README](https://github.com/charmbracelet/crush#installation) for Windows, Arch, Nix, and FreeBSD instructions.

* An OffRail API key — [sign up](https://offrail.ai/signup) and create one from your dashboard

## Setup [#setup]

<Steps>
  <Step>
    ### Set Your API Key [#set-your-api-key]

    Export your OffRail API key in your shell:

    ```bash
    export OFFRAIL_API_KEY=your-api-key-here
    ```

    Add it to your shell profile (`~/.zshrc` or `~/.bashrc`) to make it permanent.
  </Step>

  <Step>
    ### Add OffRail as a Provider [#add-offrail-as-a-provider]

    Create a `crush.json` in your project directory (or `~/.config/crush/crush.json` for a global setup):

    ```json
    {
    	"$schema": "https://charm.land/crush.json",
    	"providers": {
    		"offrail": {
    			"name": "OffRail",
    			"type": "openai-compat",
    			"base_url": "https://api.offrail.ai/v1",
    			"api_key": "$OFFRAIL_API_KEY"
    		}
    	}
    }
    ```

    Crush auto-discovers all available models from OffRail's `/v1/models` endpoint, so there is no model list to maintain.
  </Step>

  <Step>
    ### Start Coding [#start-coding]

    Launch Crush:

    ```bash
    crush
    ```

    Select "OffRail" from the provider list, pick a model, and start building. You can switch models at any time from within Crush.
  </Step>
</Steps>

## Why Use OffRail with Crush [#why-use-offrail-with-crush]

* **200+ models** — GPT-5, Claude, Gemini, Kimi, GLM, and more from 40+ providers
* **One API key** — Stop juggling credentials for every provider
* **Cost tracking** — See what each coding session costs in your dashboard
* **Automatic fallback** — Requests route to a healthy provider if one is down
* **Volume discounts** — The more you use, the more you save

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, OffRail automatically fails over to alternative providers if your chosen provider is experiencing downtime. If you want to lock into a specific provider/model mapping — for example to guarantee a fixed price or to always use a single provider — add the `X-No-Fallback` header to your provider entry:

```json
{
	"$schema": "https://charm.land/crush.json",
	"providers": {
		"offrail": {
			"name": "OffRail",
			"type": "openai-compat",
			"base_url": "https://api.offrail.ai/v1",
			"api_key": "$OFFRAIL_API_KEY",
			"extra_headers": {
				"X-No-Fallback": "true"
			}
		}
	}
}
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.offrail.ai/features/routing) for details.
</Callout>

## Switching Models [#switching-models]

Use Crush's model picker to switch between any of the discovered models.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

## Troubleshooting [#troubleshooting]

### 401 Unauthorized [#401-unauthorized]

Verify `OFFRAIL_API_KEY` is exported in the shell where you launch Crush and that the key is active in your [dashboard](https://offrail.ai/dashboard).

### 404 Not Found [#404-not-found]

Verify your `base_url` is set to `https://api.offrail.ai/v1` (note the `/v1` at the end).

### Models not showing up [#models-not-showing-up]

Model discovery runs when Crush starts — restart Crush after adding or changing the provider config.

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Cursor Integration
URL: https://docs.offrail.ai/guides/cursor


Cursor is an AI-powered code editor built on VSCode. You can point Cursor's custom OpenAI base URL at OffRail to use any of our 200+ models in the AI panel — both **plan mode** and **agent mode**.

<Callout type="info">
  **Plan and agent mode.** Cursor routes its AI panel — plan mode and agent mode
  — through the custom API key + base URL, so both work with OffRail. Tab
  autocomplete and inline edit (Cmd/Ctrl + K) remain locked to Cursor's own
  backend and will not route through the gateway. If you want every request in
  your workflow to go through OffRail, use a terminal agent like [Claude
  Code](https://docs.offrail.ai/guides/claude-code), [Codex CLI](https://docs.offrail.ai/guides/codex-cli),
  [Cline](https://docs.offrail.ai/guides/cline), [Continue CLI](https://docs.offrail.ai/guides/continue), or [Hermes
  Agent](https://docs.offrail.ai/guides/hermes-agent).
</Callout>

<img alt="Cursor with OffRail" src="__img0" />

## Prerequisites [#prerequisites]

* An OffRail account with an API key
* Cursor IDE installed
* Basic understanding of Cursor's AI features

## Setup [#setup]

Cursor supports OpenAI-compatible API endpoints, making it easy to integrate with OffRail.

<Steps>
  <Step>
    ### Get Your API Key [#get-your-api-key]

    1. Log in to your [OffRail dashboard](https://offrail.ai/dashboard)
    2. Navigate to **API Keys** section
    3. Create a new API key and copy the key

        <img alt="OffRail API Keys" src="__img1" />
  </Step>

  <Step>
    ### Configure Cursor Settings [#configure-cursor-settings]

    1. Open Cursor and go to **Settings** then Click on "Cursor Settings"
    2. Click on "Models"
    3. Click on "Add OpenAI API Key"

        <img alt="Cursor Settings" src="__img2" />

    3. Scroll down to **OpenAI API Key** section
    4. Click on **Add OpenAI API Key**

        <img alt="Cursor API Key Input" src="__img3" />

    5. Enter your OffRail API key

    6. In the same Models settings, find the **Override OpenAI Base URL** option

    7. Enable the override option

    8. Enter the OffRail endpoint: `https://api.offrail.ai/v1`
  </Step>

  <Step>
    ### Select Models [#select-models]

    1. In the **Models** section, you can now select from available models
    2. Choose any [OffRail supported model](https://offrail.ai/models):

        <img alt="Cursor Model Selection" src="__img4" />

    * For chat: Use models like `gpt-5`, `gpt-4o`, `claude-sonnet-4-5`
    * For custom models: Add the provider name before the model name (e.g. `custom/my-model`)
    * For discounted models: copy the ids from from the [models page](https://offrail.ai/models?view=grid\&filters=1\&discounted=true)
    * For free models: copy the ids from from the [models page](https://offrail.ai/models?view=grid\&filters=1\&free=true)
    * For reasoning models: copy the ids from from the [models page](https://offrail.ai/models?view=grid\&filters=1\&reasoning=true)
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    1. Open any code file in Cursor
    2. Try using the AI chat (Cmd/Ctrl + L) in plan mode
    3. Then switch the panel to agent mode and run a small multi-file task

        <img alt="Cursor AI Chat" src="__img5" />
        <img alt="Cursor AI Chat 2" src="__img6" />

    All AI requests will now be routed through OffRail.
  </Step>
</Steps>

## What Works (and What Doesn't) [#what-works-and-what-doesnt]

Cursor honors the custom OpenAI base URL for the **AI panel** (Cmd/Ctrl + L) — both plan mode and agent mode. Tab autocomplete and inline edit still use Cursor's own backend, even after you save the OffRail key.

### Works through OffRail [#works-through-offrail]

* **Plan mode (Cmd/Ctrl + L)** — Ask questions, plan changes, get explanations, debug.
* **Agent mode** — Cursor's coding agent runs multi-file edits with your gateway models.

All of these requests route through OffRail and appear in your dashboard.

### Does NOT work through OffRail [#does-not-work-through-offrail]

* **Inline Edit (Cmd/Ctrl + K)** — Locked to Cursor's backend.
* **Autocomplete / Tab completion** — Locked to Cursor's backend.

If you want every request in your workflow — including edits and completions — to route through OffRail, use [Claude Code](https://docs.offrail.ai/guides/claude-code), [Codex CLI](https://docs.offrail.ai/guides/codex-cli), [Cline](https://docs.offrail.ai/guides/cline), [Continue CLI](https://docs.offrail.ai/guides/continue), or [Hermes Agent](https://docs.offrail.ai/guides/hermes-agent).

### Model Routing [#model-routing]

With OffRail's [routing features](https://docs.offrail.ai/features/routing), you can:

* **Chooses cost-effective models** by default for optimal price-to-performance ratio
* **Automatically scales to more powerful models** based on your request's context size
* **Handles large contexts intelligently** by selecting models with appropriate context windows

## Troubleshooting [#troubleshooting]

### Authentication Errors [#authentication-errors]

If you see authentication errors:

* Verify your API key is correct
* Check that the base URL is set to `https://api.offrail.ai/v1`
* Ensure your OffRail account has sufficient credits

### Model Not Found [#model-not-found]

If you see "model not found" errors:

* Verify the model ID exists in the [models page](https://offrail.ai/models)
* Check that you're using the correct model name format
* Some models may require specific provider configurations in your OffRail dashboard

### Slow Responses [#slow-responses]

If responses are slow:

* Check your internet connection
* Monitor your usage in the OffRail dashboard
* Switch to a faster chat model from the [models page](https://offrail.ai/models)

### Autocomplete / inline edit still uses Cursor's models [#autocomplete--inline-edit-still-uses-cursors-models]

This is expected. Cursor routes the AI panel — plan mode and agent mode — through the custom API key, but tab autocomplete and inline edit are locked to Cursor's own backend. See [What Works (and What Doesn't)](#what-works-and-what-doesnt) above.

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>

## Benefits of Using OffRail with Cursor [#benefits-of-using-offrail-with-cursor]

* **Multi-Provider Access**: Use models from OpenAI, Anthropic, Google, Open-source models and more
* **Cost Control**: Track and limit your AI spending with detailed usage analytics
* **Caching**: Reduce costs with response caching
* **Analytics**: Monitor usage patterns and costs


# GitHub Copilot App Integration
URL: https://docs.offrail.ai/guides/github-copilot


The [GitHub Copilot app](https://github.com/features/ai/github-app) is GitHub's desktop app for agent-driven development — start agent sessions from issues, pull requests, or prompts, run parallel workflows in isolated workspaces, and merge PRs without leaving the app. It supports bring your own key (BYOK), so you can run agent sessions against your own model provider.

Add OffRail as that provider and every session can use Claude, Gemini, GPT, or any model in the [catalog](https://offrail.ai/models) that supports tool calling — with full cost visibility in your dashboard.

One provider entry. No config files. Works on any Copilot plan, or with no Copilot plan at all.

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for OffRail [#sign-up-for-offrail]

    [Sign up free](https://offrail.ai/signup) — no credit card required. Copy your API key (starts with `orl_`) from the dashboard.
  </Step>

  <Step>
    ### Install the GitHub Copilot App [#install-the-github-copilot-app]

    Download it from [github.com/features/ai/github-app](https://github.com/features/ai/github-app) (macOS, Windows, or Linux) and sign in with your GitHub account.
  </Step>

  <Step>
    ### Add OffRail as a Model Provider [#add-offrail-as-a-model-provider]

    In the app, open **Settings** → **Model Providers**, select **Add provider**, and choose the **OpenAI-compatible** provider type. Set the **Base URL** to:

    ```txt
    https://api.offrail.ai/v1
    ```

    Paste your OffRail API key and save. The key is stored in your OS keychain.
  </Step>

  <Step>
    ### Start a Session with Any Model [#start-a-session-with-any-model]

    OffRail's models now appear in the model picker alongside Copilot-hosted models. Choose one when you start an agent session — each session can use a different model.
  </Step>
</Steps>

## Why This Works [#why-this-works]

OffRail's `/v1` endpoint is fully OpenAI-compatible. The Copilot app fetches the model list from the gateway and routes each agent session through it, and we route requests to the right provider behind the scenes. This means:

* **Use any tool-calling model** — Claude, Gemini, GPT, and the rest of the catalog in Copilot agent sessions
* **Keep your workflow** — sessions, workspaces, and PR merging work exactly the same
* **Track costs** — every request appears in your OffRail dashboard
* **Automatic caching** — repeated requests hit cache, saving money

## Choosing Models [#choosing-models]

All models available to your account show up in the app's model picker; agent sessions work with models that support tool calling and streaming. Browse the [models page](https://offrail.ai/models) to compare capabilities and pricing, or check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%.

## Good to Know [#good-to-know]

* **Keys stay local** — the app stores your API key in the OS keychain and never reads it back into the UI.
* **Any plan works** — BYOK providers work on every Copilot plan, including Free. You don't need a paid Copilot subscription to run agent sessions through OffRail.
* **Business and Enterprise** — adding model providers is gated by the **Enable custom models** (BYOK) policy, which your admin must turn on. Accessing the Copilot app itself also requires the Copilot CLI enabled in policy settings.
* **Agent sessions only** — BYOK covers the app's model-powered agent sessions. Inline code completions in your editor still use Copilot's own service.

## GitHub Copilot in VS Code [#github-copilot-in-vs-code]

Copilot Chat in VS Code supports custom endpoints too:

1. Run **Chat: Manage Language Models** from the Command Palette
2. Choose **Add Models** → **Custom Endpoint**, enter your OffRail API key, and select **Chat Completions** as the API type (OffRail is OpenAI-compatible)
3. In the generated `chatLanguageModels.json`, point the model `url` at `https://api.offrail.ai/v1/chat/completions` and set the model `id` from the [models page](https://offrail.ai/models)

Your gateway models then appear in the VS Code chat model picker.

## Troubleshooting [#troubleshooting]

### Models don't appear in the picker [#models-dont-appear-in-the-picker]

1. Verify the Base URL is exactly `https://api.offrail.ai/v1` (note the `/v1` at the end)
2. Check your API key starts with `orl_` and is active in your [dashboard](https://offrail.ai/dashboard)

### 401 Unauthorized [#401-unauthorized]

Your API key is invalid or was revoked. Generate a new key in the dashboard and update the provider entry in **Settings** → **Model Providers**.

### 402 or credit errors [#402-or-credit-errors]

Your OffRail organization is out of credits. Top up in the [dashboard](https://offrail.ai/dashboard) — BYOK sessions bill through OffRail, not Copilot premium requests.

### Provider option is missing [#provider-option-is-missing]

BYOK in the Copilot app shipped in June 2026 — update to the latest app version. On Business or Enterprise plans, ask your admin to enable the **Enable custom models** (BYOK) policy — and the Copilot CLI policy if you can't access the app at all.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Hermes Agent Integration
URL: https://docs.offrail.ai/guides/hermes-agent


[Hermes Agent](https://github.com/nousresearch/hermes-agent) is an open-source AI coding agent for your terminal built by Nous Research. It supports tool use, browser automation, multi-provider routing, skills, and MCP servers. By pointing it at OffRail you get access to 200+ models from 40+ providers, all tracked in one dashboard.

One config change. No code changes. Full cost tracking.

## Prerequisites [#prerequisites]

* Hermes Agent installed — see [installation](#installation) below or visit the [Hermes Agent repo](https://github.com/nousresearch/hermes-agent)
* An OffRail API key — [sign up free](https://offrail.ai/signup) (no credit card required)

## Installation [#installation]

Install Hermes Agent using the official install script:

```bash
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
```

After installation, reload your shell and verify:

```bash
source ~/.bashrc
hermes --version
```

<Callout type="info">
  The installer handles Python 3.11, Node.js, ripgrep, and other dependencies
  automatically. See the [repo](https://github.com/nousresearch/hermes-agent)
  for Windows (PowerShell) and manual install options.
</Callout>

## Setup [#setup]

<Steps>
  <Step>
    ### Run the Setup Wizard [#run-the-setup-wizard]

    Run `hermes setup` to launch the interactive setup wizard. You can choose either **Quick setup** (option 1) for provider, model, and messaging configuration, or **Full setup** (option 2) to configure everything including tools, skills, and advanced options:

    ```bash
    hermes setup
    ```

        <img alt="Hermes Agent Setup Wizard" src="__img0" />

    In this guide we use Quick setup, but Full setup works the same way — it just includes additional configuration steps.
  </Step>

  <Step>
    ### Configure Inference Provider [#configure-inference-provider]

    The wizard will ask you to configure your inference provider. Select **Custom OpenAI-compatible endpoint** and enter the OffRail base URL:

    ```
    API base URL: https://api.offrail.ai/v1
    ```

    Then paste your OffRail API key (starts with `orl_`):

        <img alt="Inference Provider Configuration" src="__img1" />
  </Step>

  <Step>
    ### Choose a Model [#choose-a-model]

    The wizard presents a list of 200+ available models. Type a model name or select from the list. Popular choices include `claude-sonnet-4-6`, `gpt-5.5`, or `gemini-3.1-pro`:

        <img alt="Model Selection List" src="__img2" />
  </Step>

  <Step>
    ### Set Context Length [#set-context-length]

    Leave the context length blank to auto-detect (recommended), or specify a custom value:

        <img alt="Context Length Configuration" src="__img3" />
  </Step>

  <Step>
    ### Set Display Name [#set-display-name]

    Give your provider configuration a display name. This appears in the Hermes status bar when chatting:

        <img alt="Display Name Configuration" src="__img4" />
  </Step>

  <Step>
    ### Select Terminal Backend [#select-terminal-backend]

    Choose your terminal backend. In this guide we use **Local** (run directly on this machine), but you can pick any option based on your requirements — Docker for isolated containers, SSH for remote machines, Modal for serverless sandboxes, Daytona for cloud dev environments, and more:

        <img alt="Terminal Backend Selection" src="__img5" />
  </Step>

  <Step>
    ### Setup Complete [#setup-complete]

    Once done, Hermes shows you where your config files are stored and how to edit them. It will prompt &#x2A;*"Launch hermes chat now? \[Y/n]"** — press `Y` to start an interactive agent session immediately:

        <img alt="Setup Complete" src="__img6" />

    Your configuration files:

    * **Settings:** `~/.hermes/config.yaml`
    * **API Keys:** `~/.hermes/.env`
    * **Data:** `~/.hermes/cron/`, `sessions/`, `logs/`

    Once you press `Y`, Hermes launches a full agent session connected to OffRail. You can start chatting right away.
  </Step>
</Steps>

## Using Hermes with OffRail [#using-hermes-with-offrail]

Once configured, all requests route through OffRail. You'll see the provider name (e.g., "OFFRAIL") in the Hermes status bar.

### Switching Models at Runtime [#switching-models-at-runtime]

You can switch models mid-session using the `/model` slash command (similar to how Claude Code uses slash commands). Just type `/model` followed by the model name:

<img alt="Switching to Claude Haiku via OffRail" src="__img7" />

Switch to any model available through OffRail — from Claude to GPT to open-source models — without leaving your session:

<img alt="Switching to GPT-5.4-nano via OffRail" src="__img8" />

Add `--global` to persist the model change across sessions.

### CLI Model Override [#cli-model-override]

You can also override the model from the command line:

```bash
# Use a specific model for this session
hermes chat --model gpt-5.5

# Use a powerful model for complex tasks
hermes chat --model claude-opus-4-6
```

## Why Use OffRail with Hermes Agent [#why-use-offrail-with-hermes-agent]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%

## One-Shot Mode [#one-shot-mode]

For scripting or CI pipelines, use the `-q` flag for a one-shot prompt:

```bash
hermes chat -q "Explain what this function does" -Q
```

The `-Q` flag enables quiet mode, suppressing the banner and spinner for clean output. For pure one-shot mode (no interactive session):

```bash
hermes chat -z "Generate a README for this project"
```

## Useful Hermes Commands [#useful-hermes-commands]

| Command                | Purpose                                 |
| ---------------------- | --------------------------------------- |
| `hermes`               | Start interactive chat (default)        |
| `hermes setup`         | Run the setup wizard                    |
| `hermes setup model`   | Change model/provider                   |
| `hermes chat -q "..."` | One-shot prompt                         |
| `hermes model`         | Choose provider and model interactively |
| `hermes config edit`   | Open config in your editor              |
| `hermes doctor`        | Diagnose connection/config issues       |
| `hermes sessions`      | Browse and manage past sessions         |
| `hermes --continue`    | Resume most recent session              |
| `hermes update`        | Update to latest version                |

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, OffRail automatically fails over to alternative providers if your chosen provider is experiencing downtime. To disable fallback and always route to one provider, you can add the header via Hermes's request configuration.

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.offrail.ai/features/routing) for details.
</Callout>

## Troubleshooting [#troubleshooting]

### Model not found [#model-not-found]

If you get a "model not supported" error, check that your model ID matches exactly what's listed on the [models page](https://offrail.ai/models). Model IDs are case-sensitive.

### Connection timeout [#connection-timeout]

Verify your `base_url` is set to `https://api.offrail.ai/v1` (note the `/v1` at the end). You can also check the `HERMES_API_TIMEOUT` environment variable if you're hitting timeouts on long-running requests.

### Authentication errors [#authentication-errors]

Make sure your `api_key` starts with `orl_` and is valid. Check your [dashboard](https://offrail.ai/dashboard) to confirm the key is active.

### Diagnosing issues [#diagnosing-issues]

Run `hermes doctor` to check your configuration, connectivity, and credentials:

```bash
hermes doctor
```

### Old config overrides [#old-config-overrides]

If you previously used a different provider (e.g., OpenRouter), make sure to update both `provider` and `base_url` fields. The `provider` must be set to `"custom"` for OffRail. Also check `~/.hermes/.env` for any leftover `OPENROUTER_API_KEY` or other provider keys that might take precedence.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Kilo Code Integration
URL: https://docs.offrail.ai/guides/kilo-code


[Kilo Code](https://kilo.ai/) is an AI coding assistant that runs as a VS Code extension. It supports autonomous coding, file editing, terminal commands, and browser automation. OffRail is a built-in provider in Kilo Code, so setup takes under a minute — no manual base URL configuration required.

## Prerequisites [#prerequisites]

* VS Code or a VS Code-based editor (Cursor, Windsurf, etc.)
* An OffRail API key — [sign up free](https://offrail.ai/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install Kilo Code [#install-kilo-code]

    Open VS Code, go to the Extensions view (Ctrl+Shift+X / Cmd+Shift+X), search for **Kilo Code**, and click **Install**.

    Alternatively, install from the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=kilocode.kilo-code).
  </Step>

  <Step>
    ### Open Providers Settings [#open-providers-settings]

    Click the Kilo Code icon in the VS Code sidebar, then open **Settings > Providers**. You'll see the list of popular providers:

        <img alt="Kilo Code Providers screen" src="__img0" />
  </Step>

  <Step>
    ### Find OffRail [#find-offrail]

    Click **Show more providers** at the bottom of the list. In the "Connect provider" dialog, type `llm` in the search box — **OffRail** will appear:

        <img alt="Searching for OffRail" src="__img1" />

    Click the &#x2A;*+** button next to OffRail.
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    Kilo Code will show the **Connect OffRail** dialog. Paste your OffRail API key (starts with `orl_`) and click **Submit**:

        <img alt="Connect OffRail — enter API key" src="__img2" />

    [Sign up](https://offrail.ai/signup) or log in to your OffRail dashboard and navigate to **API Keys** to get your key.
  </Step>

  <Step>
    ### Start Coding [#start-coding]

    Once connected, select an OffRail model from the model picker at the bottom of the chat panel. All requests now route through OffRail — you'll see usage, costs, and logs in your [dashboard](https://offrail.ai/dashboard):

        <img alt="Kilo Code chat active with OffRail" src="__img3" />
  </Step>
</Steps>

## Why Use OffRail with Kilo Code [#why-use-offrail-with-kilo-code]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more from 40+ providers
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%

## Features [#features]

Once configured, you can use all of Kilo Code's features with OffRail:

* **Autonomous coding** — Create and edit files, build features from natural language
* **Terminal commands** — Run builds, tests, and scripts directly from the chat
* **Browser automation** — Preview and interact with web apps
* **Checkpoints** — Save and restore session states
* **Multiple modes** — Switch between Code, Architect, Ask, and Debug modes

## Switching Models [#switching-models]

Click the model name at the bottom of the Kilo Code chat panel to open the model picker. Select any OffRail model — the switch takes effect immediately for the next message.

## Troubleshooting [#troubleshooting]

### OffRail not in provider list [#offrail-not-in-provider-list]

Click **Show more providers** at the bottom of the Providers page. In the search dialog, type "llm" or "gateway" to find it.

### Authentication errors [#authentication-errors]

Make sure your API key starts with `orl_` and is active. Check your [dashboard](https://offrail.ai/dashboard) to confirm the key is valid.

### Model not found [#model-not-found]

Verify the model ID matches exactly what's listed on the [models page](https://offrail.ai/models). Model IDs are case-sensitive.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Kimi Code Integration
URL: https://docs.offrail.ai/guides/kimi-code


[Kimi Code CLI](https://github.com/MoonshotAI/kimi-code) is an open-source, AI-powered coding agent developed by Moonshot AI designed to automate software development tasks directly within your terminal. It can read and edit code, execute shell commands, search files, and autonomously manage complex coding workflows.

Kimi Code features first-class support for the **models.dev** registry, a community-maintained model catalog. This allows Kimi Code to query and configure OffRail dynamically — fetching all compatible models, capabilities (such as thinking or vision), and pricing without requiring manual TOML editing.

## Prerequisites [#prerequisites]

* An OffRail API key — [sign up free](https://offrail.ai/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install Kimi Code CLI [#install-kimi-code-cli]

    If you haven't already, install Kimi Code CLI.

    * **macOS or Linux**:

      ```bash
      curl -fsSL https://code.kimi.com/kimi-code/install.sh | bash
      ```

    * **Homebrew (macOS/Linux)**:

      ```bash
      brew install kimi-code
      ```

    * **Windows (PowerShell)**:
      ```powershell
      irm https://code.kimi.com/kimi-code/install.ps1 | iex
      ```

    Confirm the installation:

    ```bash
    kimi --version
    ```
  </Step>

  <Step>
    ### Launch Kimi Code and Open the Provider Manager [#launch-kimi-code-and-open-the-provider-manager]

    Start the interactive terminal in your project directory:

    ```bash
    kimi
    ```

    Once loaded, type the `/provider` command and press Enter. Select **Known third-party provider** to fetch the catalog from the registry:

    <img src="/guides/kimi-code/0-add-provider.png" alt="Opening Provider Manager in Kimi Code" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>

  <Step>
    ### Select OffRail [#select-offrail]

    Type `llm` to filter the providers and select **OffRail** from the list:

    <img src="/guides/kimi-code/1-select-provider.png" alt="Selecting OffRail in Kimi Code" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    When prompted, paste your OffRail API key and press Enter. Kimi Code will save it securely to your local configuration:

    <img src="/guides/kimi-code/2-enter-key.png" alt="Entering OffRail API Key" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    <Callout type="info">
      Your credentials are saved locally to `~/.kimi-code/config.toml`.
    </Callout>
  </Step>

  <Step>
    ### Select a Model and Toggle Thinking [#select-a-model-and-toggle-thinking]

    The OffRail catalog is now loaded. Use the arrow keys to browse or type to search for your desired model. Select the `offrail` tab to view only OffRail models.

    You can also toggle the **Thinking** option (On/Off) at the bottom depending on the model's capabilities:

    <img src="/guides/kimi-code/3-select-model.png" alt="Browsing OffRail Models" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    For example, type `gpt-5.5` to find the latest reasoning model, select it, and press Enter:

    <img src="/guides/kimi-code/4-select-gpt-model.png" alt="Selecting GPT-5.5 Model" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>

  <Step>
    ### Start Coding [#start-coding]

    All set! Kimi Code is now configured. Your requests will be securely routed through OffRail, allowing you to use advanced models for local autonomous coding while showing real-time usage and cost statistics on your OffRail dashboard.

    <img src="/guides/kimi-code/5-chat.png" alt="Running Kimi Code CLI session" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    Use `/model` in the terminal session at any time to switch models.
  </Step>
</Steps>

## Manual Configuration (Advanced) [#manual-configuration-advanced]

If you prefer to configure your environment manually without using the interactive provider manager, you can write settings directly to your configuration file at `~/.kimi-code/config.toml` (or `C:\Users\<YourUsername>\.kimi-code\config.toml` on Windows).

Here is an example TOML configuration that registers **GPT-5.5**, **Claude 3.7 Sonnet**, **DeepSeek R1**, and **Qwen3.7 Max** manually:

```toml
default_model = "offrail/gpt-5.5"

[providers.offrail]
type = "openai"
api_key = "orl_your_api_key_here"
base_url = "https://api.offrail.ai/v1"

[models."offrail/gpt-5.5"]
provider = "offrail"
model = "gpt-5.5"
max_context_size = 1050000
max_output_size = 128000
capabilities = [ "thinking", "tool_use" ]
display_name = "GPT-5.5"

[models."offrail/claude-3.7-sonnet"]
provider = "offrail"
model = "claude-3.7-sonnet"
max_context_size = 200000
max_output_size = 8192
capabilities = [ "image_in", "thinking", "tool_use" ]
display_name = "Claude 3.7 Sonnet"

[models."offrail/deepseek-r1"]
provider = "offrail"
model = "deepseek-r1"
max_context_size = 131072
max_output_size = 8192
capabilities = [ "thinking", "tool_use" ]
display_name = "DeepSeek R1"

[models."offrail/qwen3.7-max"]
provider = "offrail"
model = "qwen3.7-max"
max_context_size = 1000000
max_output_size = 65536
capabilities = [ "thinking", "tool_use" ]
display_name = "Qwen3.7 Max"
```

## Why Use OffRail with Kimi Code CLI [#why-use-offrail-with-kimi-code-cli]

* **200+ models** — Access GPT-5.5, Gemini, Llama, DeepSeek, and more in a single CLI configuration.
* **Unified cost tracking** — Get a detailed breakdown of costs per prompt and session in your dashboard.
* **Response caching** — Automatically cache repeated requests (such as parsing or building commands) to save API costs.
* **Automatic fallback** — Keep coding even if a provider encounters temporary downtime.
* **Volume discounts** — Access selected models with up to 90% savings compared to standard pricing.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# MiMo Code Integration
URL: https://docs.offrail.ai/guides/mimocode


[MiMo Code](https://mimo.xiaomi.com/mimocode) is an AI-powered coding agent command-line tool developed by Xiaomi. It can understand your code repository, plan changes, safely execute shell commands, edit files, and autonomously manage complex software development tasks in your terminal.

By configuring MiMo Code to route through OffRail, you can point it at any model—GPT-5.5, Gemini, Llama, Claude, or 210+ others—while keeping the same API format MiMo Code expects, with full cost tracking in your dashboard.

## Prerequisites [#prerequisites]

* An OffRail API key — [sign up free](https://offrail.ai/signup) (no credit card required)

## Setup [#setup]

<Steps>
  <Step>
    ### Install MiMo Code [#install-mimo-code]

    If you haven't already, install MiMo Code by running the official installation command in your terminal:

    ```bash
    curl -fsSL https://mimo.xiaomi.com/install | bash
    ```

    Confirm the installation by checking the help command:

    ```bash
    mimo --help
    ```
  </Step>

  <Step>
    ### Configure mimocode.json [#configure-mimocodejson]

    Create or edit your MiMo Code configuration file at `~/.config/mimocode/mimocode.json` (on Linux/macOS) or `~/.mimocode/mimocode.json`.

    Specify the default models you want to use and route the `anthropic` provider to your OffRail endpoint. Here is an example configuration that sets up **Claude Opus 4.8**, **GPT-5.5**, **DeepSeek V4 Pro**, **MiniMax M3**, and **Qwen3.7 Max**:

    ```json
    {
    	"model": "anthropic/claude-opus-4-8",
    	"small_model": "anthropic/claude-4-5-haiku-latest",
    	"provider": {
    		"anthropic": {
    			"options": {
    				"apiKey": "orl_your_api_key_here",
    				"baseURL": "https://api.offrail.ai/v1"
    			},
    			"models": {
    				"gpt-5.5": {
    					"name": "gpt-5.5"
    				},
    				"claude-opus-4-8": {
    					"name": "claude-opus-4-8"
    				},
    				"deepseek-v4-pro": {
    					"name": "deepseek-v4-pro"
    				},
    				"minimax-m3": {
    					"name": "minimax-m3"
    				},
    				"qwen3.7-max": {
    					"name": "qwen3.7-max"
    				}
    			}
    		}
    	}
    }
    ```

    <img src="/guides/mimocode/0-config.png" alt="Configuring mimocode.json" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />

    <Callout type="info">
      Replace `orl_your_api_key_here` with your actual OffRail API key from the
      dashboard.
    </Callout>
  </Step>

  <Step>
    ### Alternatively: Use Environment Variables [#alternatively-use-environment-variables]

    If you prefer to configure the provider dynamically, you can export the standard Anthropic environment variables before starting MiMo Code:

    ```bash
    export ANTHROPIC_API_KEY=orl_your_api_key_here
    export ANTHROPIC_BASE_URL=https://api.offrail.ai/v1
    ```
  </Step>

  <Step>
    ### Run MiMo Code [#run-mimo-code]

    Navigate to your project folder and launch the TUI or run a prompt directly:

    ```bash
    mimo
    ```

    Or run it with a message:

    ```bash
    mimo run "Your coding prompt here"
    ```

    All requests will now be routed through OffRail, allowing you to use advanced models for local autonomous coding while showing real-time usage and cost statistics on your OffRail dashboard.

    <img src="/guides/mimocode/1-chat.png" alt="Running MiMo Code with OffRail" style="{ borderRadius: &#x22;8px&#x22;, margin: &#x22;1.5rem 0&#x22; }" />
  </Step>
</Steps>

## Configuration Details [#configuration-details]

### The Provider Options [#the-provider-options]

To point MiMo Code to OffRail, you define the `baseURL` and `apiKey` inside the `options` of the `anthropic` provider block.

```json
"provider": {
	"anthropic": {
		"options": {
			"apiKey": "orl_your_api_key_here",
			"baseURL": "https://api.offrail.ai/v1"
		}
	}
}
```

### Defining Custom Models [#defining-custom-models]

Because MiMo Code CLI restricts requests to built-in models by default, any custom model you wish to target (such as `gpt-5.5` or `deepseek-v4-pro`) must be registered in the `models` dictionary within the `anthropic` provider config:

```json
"models": {
	"gpt-5.5": {
		"name": "gpt-5.5"
	}
}
```

Once registered, you can set them as your default model or small model using the `anthropic/` prefix (e.g. `"model": "anthropic/gpt-5.5"`).

## Why Use OffRail with MiMo Code [#why-use-offrail-with-mimo-code]

* **200+ models** — Access GPT-5.5, Gemini, Llama, DeepSeek, and more in a single CLI configuration.
* **Unified cost tracking** — Get a detailed breakdown of costs per prompt and session in your dashboard.
* **Response caching** — Automatically cache repeated requests (such as parsing or building commands) to save API costs.
* **Automatic fallback** — Keep coding even if a provider encounters temporary downtime.
* **Volume discounts** — Access selected models with up to 90% savings compared to standard pricing.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# N8n Integration
URL: https://docs.offrail.ai/guides/n8n


n8n is a powerful workflow automation tool that can be enhanced with AI capabilities through OffRail. This guide shows how to integrate OffRail into your n8n workflows.

<img alt="n8n workflow with OffRail" src="__img0" />

## Prerequisites [#prerequisites]

* An OffRail account with an API key
* n8n instance (self-hosted or cloud)
* Basic understanding of n8n workflows

## Setup [#setup]

The easiest way to use OffRail with n8n is through the OpenAI node with custom configuration.

<Steps>
  <Step>
    ### Add OpenAI Credentials [#add-openai-credentials]

    1. In n8n, go to **Settings** → **Credentials**

        <img alt="n8n credentials" src="__img1" />

    2. Click **Add Credential** → **OpenAI**

        <img alt="n8n credentials" src="__img2" />

    3. Configure as follows:
       * **API Key**: Your OffRail API key
       * **Base URL**: `https://api.offrail.ai/v1`
       * **Organization ID**: Leave blank

        <img alt="n8n credentials" src="__img3" />
  </Step>

  <Step>
    ### Configure OpenAI Node [#configure-openai-node]

    1. Add an **AI Agent** node to your workflow
    2. Add a **Chat Model** edge to the node

        <img alt="n8n credentials" src="__img4" />

    3. Configure the node to use the OffRail provider

        <img alt="n8n credentials" src="__img5" />

    <Callout type="warning">
      Note: You have to toggle off the responses API. OffRail does not support it.
    </Callout>

        <img alt="responses api" src="__img6" />

    4. Select your desired options

    * **Model**: Use any [OffRail model](https://offrail.ai/models) ID (e.g., `gpt-5`)
    * **Options**: Optionally, configure LLM parameters

        <img alt="n8n credentials" src="__img7" />
  </Step>

  <Step>
    ### Test Workflow [#test-workflow]

    Finally, try running your workflow with a test prompt.

        <img alt="n8n credentials" src="__img8" />
  </Step>
</Steps>


# OpenClaw Integration
URL: https://docs.offrail.ai/guides/openclaw


[OpenClaw](https://docs.openclaw.ai/) is a self-hosted gateway that connects your favorite chat apps—WhatsApp, Telegram, Discord, iMessage, and more—to AI coding agents. With OffRail as a custom provider, you can route all your OpenClaw traffic through a single API, use any of 200+ models, and keep full visibility into usage and costs.

## Setup [#setup]

<Steps>
  <Step>
    ### Sign Up for OffRail [#sign-up-for-offrail]

    [Sign up free](https://offrail.ai/signup) — no credit card required. Copy your API key from the dashboard.
  </Step>

  <Step>
    ### Set Your API Key [#set-your-api-key]

    ```bash
    export OFFRAIL_API_KEY=orl_your_api_key_here
    ```
  </Step>

  <Step>
    ### Configure OpenClaw [#configure-openclaw]

    Add OffRail as a custom provider in your `~/.openclaw/openclaw.json`:

    ```json
    {
    	"models": {
    		"mode": "merge",
    		"providers": {
    			"offrail": {
    				"baseUrl": "https://api.offrail.ai/v1",
    				"apiKey": "${OFFRAIL_API_KEY}",
    				"api": "openai-completions",
    				"models": [
    					{
    						"id": "gpt-5.4",
    						"name": "GPT-5.4",
    						"contextWindow": 128000,
    						"maxTokens": 32000
    					},
    					{
    						"id": "claude-opus-4-6",
    						"name": "Claude Opus 4.6",
    						"contextWindow": 200000,
    						"maxTokens": 8192
    					},
    					{
    						"id": "gemini-3.1-pro-preview",
    						"name": "Gemini 3.1 Pro",
    						"contextWindow": 1000000,
    						"maxTokens": 8192
    					}
    				]
    			}
    		}
    	},
    	"agents": {
    		"defaults": {
    			"model": {
    				"primary": "offrail/gpt-5.4"
    			}
    		}
    	}
    }
    ```
  </Step>

  <Step>
    ### Start Chatting [#start-chatting]

    Launch OpenClaw and start chatting across your connected channels. All requests will be routed through OffRail.
  </Step>
</Steps>

## Why Use OffRail with OpenClaw [#why-use-offrail-with-openclaw]

* **Model flexibility** — Switch between GPT-5.4, Claude Opus, Gemini, or any of 200+ models
* **Cost tracking** — Monitor exactly how much your chat agents cost to run
* **Single bill** — No need to manage multiple API provider accounts
* **Response caching** — Repeated queries hit cache, reducing costs
* **Rate limit handling** — Automatic fallback between providers

## Switching Models [#switching-models]

Change the primary model in your config to switch between any model:

```json
{
	"agents": {
		"defaults": {
			"model": { "primary": "offrail/claude-opus-4-6" }
		}
	}
}
```

## Model Fallback Chain [#model-fallback-chain]

OpenClaw supports fallback models. If the primary model is unavailable, it automatically falls back:

```json
{
	"agents": {
		"defaults": {
			"model": {
				"primary": "offrail/gpt-5.4",
				"fallbacks": ["offrail/claude-opus-4-6"]
			}
		}
	}
}
```

## Available Models [#available-models]

OffRail uses canonical model IDs with smart routing—automatically selecting the best provider based on uptime, throughput, price, and latency. You can use any model from the [models page](https://offrail.ai/models). Flagship models include:

| Model                    | Best For                                    |
| ------------------------ | ------------------------------------------- |
| `gpt-5.4`                | Latest OpenAI flagship, highest quality     |
| `claude-opus-4-6`        | Anthropic's most capable model              |
| `claude-sonnet-4-6`      | Fast reasoning with extended thinking       |
| `gemini-3.1-pro-preview` | Google's latest flagship, 1M context window |
| `o3`                     | Advanced reasoning tasks                    |
| `gpt-5.4-pro`            | Premium tier with extended reasoning        |
| `gemini-3.6-flash`       | Fast responses, good for high-volume        |
| `claude-haiku-4-5`       | Cost-effective, quick responses             |
| `grok-3`                 | xAI flagship                                |
| `deepseek-v3.1`          | Open-source with vision and tools           |

For more details on routing behavior, see [routing](https://docs.offrail.ai/features/routing).

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

## Tips for Chat Agents [#tips-for-chat-agents]

### Optimize Costs [#optimize-costs]

1. **Use smaller models for simple tasks** — Claude Haiku or Gemini Flash handle basic Q\&A well
2. **Enable caching** — OffRail caches identical requests automatically
3. **Set token limits** — Configure max tokens to prevent runaway costs

### Improve Response Quality [#improve-response-quality]

1. **Choose the right model** — Claude Opus excels at nuanced conversation, GPT-5.4 at general tasks
2. **Use system prompts** — Configure your agent's personality and capabilities
3. **Test multiple models** — OffRail makes it easy to A/B test different providers

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# OpenCode Desktop Integration
URL: https://docs.offrail.ai/guides/opencode-desktop


[OpenCode Desktop](https://opencode.ai/download) is the GUI desktop app version of OpenCode — an open-source AI coding agent with a full visual interface for managing providers, models, and sessions. OffRail is a built-in provider, so setup takes under a minute with no config files required.

<Callout type="info">
  Looking for the CLI version? See the [OpenCode CLI guide](https://docs.offrail.ai/guides/opencode).
</Callout>

## Prerequisites [#prerequisites]

* OpenCode Desktop installed — [download for Windows or macOS](https://opencode.ai/download)
* An OffRail API key — [sign up free](https://offrail.ai/signup) (no credit card required)

## Installation [#installation]

Download OpenCode Desktop from [opencode.ai/download](https://opencode.ai/download) and install it for your platform:

* **macOS (Apple Silicon)** — `.dmg` installer
* **macOS (Intel)** — `.dmg` installer
* **Windows** — `.exe` installer

You can also install on macOS via Homebrew:

```bash
brew install --cask opencode-desktop
```

## Setup [#setup]

<Steps>
  <Step>
    ### Open Providers Settings [#open-providers-settings]

    Launch OpenCode Desktop. Click the **Providers** section in the left sidebar under **Server**. You'll see the list of built-in providers:

        <img alt="OpenCode Desktop Providers screen" src="__img0" />
  </Step>

  <Step>
    ### Find OffRail [#find-offrail]

    Click **Show more providers*&#x2A; at the bottom of the list, or click **+ Connect** on any entry to open the provider search. Type `LLM` in the search box — **OffRail** will appear under "Other":

        <img alt="Searching for OffRail" src="__img1" />

    Select **OffRail** from the list.
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    OpenCode will show the **Connect OffRail** dialog. Paste your OffRail API key (starts with `orl_`) and click **Continue**:

        <img alt="Connect OffRail — enter API key" src="__img2" />

    [Sign up](https://offrail.ai/signup) or log in to your OffRail dashboard and navigate to **API Keys** to get your key.
  </Step>

  <Step>
    ### Select a Model [#select-a-model]

    Once connected, open the model picker from the chat input bar. Type `llm` to filter OffRail models — you'll see all available models including Claude Opus 4.7, Claude Sonnet 4.6, DeepSeek, Gemini, and more:

        <img alt="OffRail model selection" src="__img3" />
  </Step>

  <Step>
    ### Start Building [#start-building]

    Select a model and start chatting. All requests route through OffRail — you'll see usage, costs, and logs in your [dashboard](https://offrail.ai/dashboard):

        <img alt="OpenCode Desktop chat active with OffRail" src="__img4" />
  </Step>
</Steps>

## Why Use OffRail with OpenCode Desktop [#why-use-offrail-with-opencode-desktop]

* **200+ models** — Claude, GPT, Gemini, Llama, DeepSeek, and more from 40+ providers
* **One API key** — Stop managing separate keys for each provider
* **Cost tracking** — See exactly what each session costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Automatic fallback** — If a provider is down, requests route to an alternative
* **Volume discounts** — Check [discounted models](https://offrail.ai/models?discounted=true) for savings up to 90%

## Switching Models [#switching-models]

You can switch models at any time from the model picker in the chat input bar. Click the current model name, type `llm` to filter to OffRail models, and select a new one. The switch takes effect immediately for the next message.

## Locking to a Specific Provider [#locking-to-a-specific-provider]

By default, OffRail automatically fails over to alternative providers if your chosen provider is experiencing downtime. To disable fallback for a specific model, you can pass the `X-No-Fallback` header via a custom `opencode.json` in your project root:

```json
{
	"provider": {
		"offrail": {
			"options": {
				"headers": {
					"X-No-Fallback": "true"
				}
			}
		}
	}
}
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing docs](https://docs.offrail.ai/features/routing) for details.
</Callout>

## Troubleshooting [#troubleshooting]

### OffRail doesn't appear in provider list [#offrail-doesnt-appear-in-provider-list]

Click **Show more providers** at the bottom of the Providers page to expand the full list, then search for "LLM".

### Authentication errors [#authentication-errors]

Make sure your API key starts with `orl_` and is active. Check your [dashboard](https://offrail.ai/dashboard) to confirm the key is valid.

### Models not loading after connect [#models-not-loading-after-connect]

Try disconnecting and reconnecting the provider from Settings > Providers. If models still don't load, check your internet connection and verify the key is valid.

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# OpenCode Integration
URL: https://docs.offrail.ai/guides/opencode


[OpenCode](https://opencode.ai) is an open-source AI coding agent for your terminal, IDE, or desktop. OffRail is a built-in provider in OpenCode, so setup takes under a minute — no config files or npm adapters required. You get access to 200+ models from 40+ providers, all tracked in one dashboard.

## Prerequisites [#prerequisites]

* OpenCode installed — visit the [OpenCode download page](https://opencode.ai/download) for your platform
* An OffRail API key

## Setup [#setup]

<Steps>
  <Step>
    ### Launch OpenCode [#launch-opencode]

    Start OpenCode from your terminal:

    ```bash
    opencode
    ```

    **In VS Code/Cursor:**

    1. Install the OpenCode extension from the marketplace
    2. Open Command Palette (Ctrl+Shift+P or Cmd+Shift+P)
    3. Type "OpenCode" and select "Open opencode"
  </Step>

  <Step>
    ### Open the Provider List [#open-the-provider-list]

    Once OpenCode launches, run the `/providers` or `/connect` command to open the provider selection screen.
  </Step>

  <Step>
    ### Select OffRail [#select-offrail]

    OffRail is listed as a built-in provider. Select "OffRail" from the provider list.
  </Step>

  <Step>
    ### Enter Your API Key [#enter-your-api-key]

    OpenCode will prompt you for your API key. Enter your OffRail API key and press Enter. OpenCode will automatically save your credentials securely.

    [Sign up for OffRail](https://offrail.ai/signup) and create an API key from your dashboard.
  </Step>

  <Step>
    ### Start Using OpenCode [#start-using-opencode]

    You're all set! OpenCode is now connected to OffRail. You can start asking questions and building with AI.
  </Step>
</Steps>

## Why Use OffRail with OpenCode [#why-use-offrail-with-opencode]

* **200+ models** — GPT-5, Claude, Gemini, Llama, and more from 40+ providers
* **One API key** — Stop juggling credentials for every provider
* **Cost tracking** — See what each coding agent costs in your dashboard
* **Response caching** — Repeated requests hit cache automatically
* **Volume discounts** — The more you use, the more you save

## Adding Custom Models [#adding-custom-models]

The built-in provider gives you access to all standard OffRail models. If you want to add custom model aliases or configure models not yet listed in the built-in provider, you can create a `config.json` in your OpenCode configuration directory:

**macOS/Linux:** `~/.config/opencode/config.json`

**Windows:** `C:\Users\YourUsername\.config\opencode\config.json`

```json
{
	"provider": {
		"offrail": {
			"models": {
				"deepseek-v3": {
					"name": "DeepSeek V3"
				}
			}
		}
	}
}
```

Since OffRail is a built-in provider, you only need to specify what you're adding — OpenCode merges your config with the built-in provider definition, so `npm`, `name`, and `baseURL` don't need to be repeated.

After updating `config.json`, restart OpenCode to see the new models.

## Pinning a Model to a Specific Provider [#pinning-a-model-to-a-specific-provider]

OpenCode's built-in model list only shows canonical model IDs (e.g. `claude-sonnet-5`), which OffRail routes to the best available provider automatically. If you want a model to always run on one specific provider — for example to guarantee a fixed price, a compliance boundary, or consistent behavior — use OffRail's [`provider/model` syntax](https://docs.offrail.ai/features/routing#provider-specific-routing) as the model ID.

Add the pinned variants you want as custom models, and they show up in OpenCode's model picker alongside the built-in list:

```json
{
	"provider": {
		"offrail": {
			"models": {
				"anthropic/claude-sonnet-5": {
					"name": "Claude Sonnet 5 (Anthropic direct)"
				},
				"aws-bedrock/claude-sonnet-5": {
					"name": "Claude Sonnet 5 (AWS Bedrock)"
				}
			}
		}
	}
}
```

You can also pin one as your default model, or per-agent:

```json
{
	"model": "offrail/anthropic/claude-sonnet-5"
}
```

For providers with multiple regions, append a region suffix to pin both provider and region, e.g. `aws-bedrock/claude-sonnet-5:us`. See the [routing docs](https://docs.offrail.ai/features/routing#provider-specific-routing) for the full syntax, and each model's available providers on the [models page](https://offrail.ai/models).

### Disabling Fallback [#disabling-fallback]

By default, OffRail automatically fails over to alternative providers if your chosen provider is experiencing downtime — even for provider-pinned model IDs. To make the pin strict, pass the `X-No-Fallback` header. Requests will then be sent only to the provider you specified, with no automatic fallback.

```json
{
	"provider": {
		"offrail": {
			"options": {
				"headers": {
					"X-No-Fallback": "true"
				}
			}
		}
	}
}
```

<Callout type="warn">
  Disabling fallback means requests will fail if the chosen provider is down.
  See the [routing
  docs](https://docs.offrail.ai/features/routing#disabling-fallback-with-x-no-fallback-header) for
  details.
</Callout>

## Switching Models [#switching-models]

Select a different model directly in the OpenCode interface, or update the `model` field in your configuration:

```json
{
	"model": "offrail/gpt-5-mini"
}
```

<Callout type="info">
  View all available models on the [models page](https://offrail.ai/models).
</Callout>

## Troubleshooting [#troubleshooting]

### Connection timeout [#connection-timeout]

Check that you have an active internet connection and that your API key is valid from the [dashboard](https://offrail.ai/dashboard).

### Custom models not showing up [#custom-models-not-showing-up]

After editing `config.json`, restart OpenCode completely for changes to take effect.

### 404 Not Found errors with custom config [#404-not-found-errors-with-custom-config]

If you are using a custom `config.json`, verify your `baseURL` is set to `https://api.offrail.ai/v1` (note the `/v1` at the end).

## Configuration Tips [#configuration-tips]

* **Global configuration**: Use `~/.config/opencode/config.json` to apply settings across all projects
* **Project-specific**: Place `opencode.json` in your project root to override global settings for that project
* **Model selection**: You can specify different models for different types of tasks using OpenCode's agent configuration

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>


# Pi Integration
URL: https://docs.offrail.ai/guides/pi


[Pi](https://pi.dev) is a minimal terminal-based coding agent that gives an AI full access to read, write, edit, and run shell commands in your project. By pointing Pi at OffRail, you can use any of our 200+ models — GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.7, DeepSeek V4, and more — with full cost tracking and caching.

## Prerequisites [#prerequisites]

* An OffRail account with an API key
* Pi installed (`curl -fsSL https://pi.dev/install.sh | bash`)
* Basic terminal familiarity

## Setup [#setup]

Pi uses a `models.json` configuration file to define providers and models. We'll add OffRail as a custom provider.

<Steps>
  <Step>
    ### Get Your API Key [#get-your-api-key]

    1. Log in to your [OffRail dashboard](https://offrail.ai/dashboard)
    2. Navigate to **API Keys** section
    3. Create a new API key and copy the key
  </Step>

  <Step>
    ### Configure Pi [#configure-pi]

    Open (or create) the Pi models configuration file at `~/.pi/agent/models.json` and add OffRail as a provider:

    ```json
    {
    	"providers": {
    		"offrail": {
    			"baseUrl": "https://api.offrail.ai/v1",
    			"api": "openai-completions",
    			"apiKey": "orl_your_api_key_here",
    			"models": [
    				{ "id": "gpt-5.5", "name": "GPT-5.5" },
    				{ "id": "claude-opus-4-7", "name": "Claude Opus 4.7" },
    				{ "id": "gemini-3.1-pro", "name": "Gemini 3.1 Pro" },
    				{ "id": "deepseek-v4", "name": "DeepSeek V4", "reasoning": true }
    			]
    		}
    	}
    }
    ```

    Replace `orl_your_api_key_here` with your actual API key from Step 1.

        <img alt="Pi models.json Configuration" src="__img0" />

    <Callout type="info">
      Pi reloads `models.json` when you open the `/model` menu — no restart needed
      after editing.
    </Callout>
  </Step>

  <Step>
    ### Select Your Model [#select-your-model]

    1. Run `pi` in any project directory
    2. Type `/model` to open the model selector
    3. Select your OffRail model from the list

    All requests now route through OffRail with full cost tracking.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    Ask Pi to do something in your project to verify everything works:

    ```
    > hello
    ```

        <img alt="Pi Test with OffRail" src="__img1" />

    You should see the response streaming from your chosen model. Check your [OffRail dashboard](https://offrail.ai/dashboard) to confirm the request appears in your usage logs.
  </Step>
</Steps>

## Adding More Models [#adding-more-models]

You can add any model from the [OffRail models page](https://offrail.ai/models) to your `models.json`. Just add entries to the `models` array:

```json
{
	"providers": {
		"offrail": {
			"baseUrl": "https://api.offrail.ai/v1",
			"api": "openai-completions",
			"apiKey": "orl_your_api_key_here",
			"models": [
				{ "id": "gpt-5.5", "name": "GPT-5.5" },
				{ "id": "gpt-5.5-mini", "name": "GPT-5.5 Mini" },
				{ "id": "claude-opus-4-7", "name": "Claude Opus 4.7" },
				{ "id": "claude-sonnet-4-6", "name": "Claude Sonnet 4.6" },
				{ "id": "gemini-3.1-pro", "name": "Gemini 3.1 Pro" },
				{ "id": "gemini-3.1-flash", "name": "Gemini 3.1 Flash" },
				{ "id": "deepseek-v4", "name": "DeepSeek V4", "reasoning": true },
				{
					"id": "deepseek-v4-mini",
					"name": "DeepSeek V4 Mini",
					"reasoning": true
				}
			]
		}
	}
}
```

## Using Environment Variables for the API Key [#using-environment-variables-for-the-api-key]

Instead of hardcoding your key, you can reference an environment variable:

```json
{
	"providers": {
		"offrail": {
			"baseUrl": "https://api.offrail.ai/v1",
			"api": "openai-completions",
			"apiKey": "OFFRAIL_API_KEY",
			"models": [{ "id": "gpt-5.5", "name": "GPT-5.5" }]
		}
	}
}
```

Then set the variable in your shell profile:

```bash
export OFFRAIL_API_KEY=orl_your_api_key_here
```

## Troubleshooting [#troubleshooting]

### Authentication Errors [#authentication-errors]

* Verify your API key is correct in `~/.pi/agent/models.json`
* Check that the base URL is set to `https://api.offrail.ai/v1`
* Ensure your OffRail account has sufficient credits

### Model Not Found [#model-not-found]

* Verify the model ID exists on the [models page](https://offrail.ai/models)
* Model IDs are case-sensitive — copy them exactly as shown

### Connection Issues [#connection-issues]

* Check your internet connection
* Ensure `api` is set to `"openai-completions"` (not `"openai-responses"`)
* Monitor your usage in the OffRail dashboard

<Callout type="info">
  Need help? Join our [Discord community](https://offrail.ai/discord) for
  support and troubleshooting assistance.
</Callout>

## Benefits of Using OffRail with Pi [#benefits-of-using-offrail-with-pi]

* **Any Model**: Use GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4, or 200+ others
* **Cost Tracking**: Every Pi request appears in your dashboard with token counts and costs
* **Caching**: Repeated requests hit cache automatically, saving money
* **One Key**: Manage all providers through a single API key
* **No Vendor Lock-in**: Switch models by changing one line in your config


# AWS Bedrock Integration
URL: https://docs.offrail.ai/integrations/aws-bedrock


AWS Bedrock is Amazon's fully managed service that provides access to foundation models from leading AI companies. This guide shows how to create AWS Bedrock Long-Term API Keys and integrate them with OffRail.

## Prerequisites [#prerequisites]

* An AWS account with Bedrock access enabled
* OffRail account or self-hosted instance

## Overview [#overview]

AWS Bedrock supports **Long-Term API Keys** for simplified authentication. These keys provide direct API access without requiring IAM credentials or complex authentication flows.

## Create AWS Bedrock Long-Term API Key [#create-aws-bedrock-long-term-api-key]

<Steps>
  <Step>
    ### Enable Model Access in Bedrock [#enable-model-access-in-bedrock]

    1. Log into the **AWS Console**
    2. Navigate to **AWS Bedrock** service
    3. Go to **Model access** in the left sidebar
    4. Click **Manage model access**
    5. Enable the models you want to use (e.g., Claude 3.5, Llama 3)
    6. Wait for access to be granted (usually instant for most models)
  </Step>

  <Step>
    ### Create Long-Term API Key [#create-long-term-api-key]

    1. In AWS Bedrock console, navigate to **API Keys** in the left sidebar
    2. Click **Create Long-Term API Key**
    3. Set expiry date ("Never expires" is recommended)
    4. Click **Generate**
    5. **Important**: Copy the API key immediately - it's only shown once!
  </Step>
</Steps>

## Add to OffRail [#add-to-offrail]

<Steps>
  <Step>
    ### Navigate to Provider Keys [#navigate-to-provider-keys]

    1. Log into [OffRail Dashboard](https://offrail.ai/dashboard)
    2. Select your organization and project
    3. Go to **Provider Keys** in the sidebar
  </Step>

  <Step>
    ### Add AWS Bedrock Provider Key [#add-aws-bedrock-provider-key]

    1. Click **Add** for **AWS Bedrock**
    2. Paste your Long-Term API Key
    3. **Select Region Prefix** based on where you want to use your models:
       * **us.** - For US regions (`us-east-1`, `us-west-2`)
       * **eu.** - For European regions (`eu-central-1`, `eu-west-1`)
       * **global.** - For global/cross-region endpoints
    4. Click **Add Key**

    The system will validate your key and confirm the connection.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    Test your integration with a simple API call:

    ```bash
    curl -X POST https://api.offrail.ai/v1/chat/completions \
      -H "Authorization: Bearer YOUR_OFFRAIL_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "aws-bedrock/claude-3-5-sonnet",
        "messages": [
          {
            "role": "user",
            "content": "Hello from AWS Bedrock!"
          }
        ]
      }'
    ```

    Replace `YOUR_OFFRAIL_API_KEY` with your OffRail API key.
  </Step>
</Steps>

## Available Models [#available-models]

Once configured, you can access all AWS Bedrock models through OffRail:

* **Anthropic Claude**: `aws-bedrock/claude-3-5-sonnet`, `aws-bedrock/claude-3-5-haiku`
* **Meta Llama**: `aws-bedrock/llama-3-2-90b`, `aws-bedrock/llama-3-2-11b`
* **Amazon Titan**: `aws-bedrock/amazon.titan-text-express-v1`
* **And more...**

Browse all available models at [offrail.ai/models](https://offrail.ai/models?provider=aws-bedrock)

## Troubleshooting [#troubleshooting]

### "Model not available" error [#model-not-available-error]

* Verify you've enabled model access in AWS Bedrock console
* Check that the region where you created your key has access to the model
* Some models are only available in specific regions

### Rate limiting [#rate-limiting]

* AWS Bedrock has request quotas per model and region
* Monitor usage in AWS Bedrock console
* Consider requesting quota increases for high-volume workloads


# Azure Integration
URL: https://docs.offrail.ai/integrations/azure


Azure provides access to OpenAI's powerful language models through Microsoft's enterprise cloud infrastructure. This guide shows how to create an Azure resource, deploy models, and integrate them with OffRail.

<Callout type="info">
  Only OpenAI models are supported via Azure at this time. [Open an
  issue](https://bitbucket.org/designsai/offrailai/issues/new) to request
  support for other model types.
</Callout>

## Prerequisites [#prerequisites]

* An Azure account with an active subscription
* OffRail account or self-hosted instance

## Overview [#overview]

Azure provides enterprise-grade access to OpenAI models with enhanced security, compliance, and regional availability. OffRail integrates seamlessly with Azure deployments.

## Create Azure Resource [#create-azure-resource]

<Steps>
  <Step>
    ### Create an Azure OpenAI Resource [#create-an-azure-openai-resource]

    1. Log into the **Azure Portal** ([https://portal.azure.com](https://portal.azure.com))
    2. Click **Create a resource**
    3. Search for **Azure OpenAI** and select it
    4. Click **Create**
    5. Configure the resource:
       * **Subscription**: Select your Azure subscription
       * **Resource group**: Create new or select existing
       * **Region**: Choose a region (e.g., East US, West Europe)
       * **Name**: Enter a unique resource name (this will be your `<resource-name>`)
       * **Pricing tier**: Select Standard S0
    6. Click **Review + create**, then **Create**
    7. Wait for deployment to complete

    **Important**: Note your resource name - it will be used in the base URL: `https://<resource-name>.openai.azure.com`
  </Step>

  <Step>
    ### Deploy Models [#deploy-models]

    1. Navigate to your Azure resource in the Azure Portal
    2. Click **Go to Azure OpenAI Studio** or visit [https://oai.azure.com](https://oai.azure.com)
    3. In Azure Studio, select **Deployments** from the left sidebar
    4. Click **Create new deployment**
    5. Configure your deployment:
       * **Model**: Select a model (e.g., gpt-4o, gpt-4o-mini, gpt-4-turbo)
       * **Deployment name**: Enter a name (this must match the model identifier you'll use – use the pre-filled name)
       * **Model version**: Select the latest version
       * **Deployment type**: Global Standard
    6. Click **Create**
    7. Repeat for additional models you want to use

    **Note**: The deployment name must match the expected model name:

    * For `gpt-4o-mini` → deployment name should be `gpt-4o-mini`
    * For `gpt-35-turbo` → deployment name should be `gpt-35-turbo`
      etc.
  </Step>

  <Step>
    ### Get API Key [#get-api-key]

    1. In the Azure Portal, go to your Azure resource
    2. Click **Keys and Endpoint** in the left sidebar
    3. Copy **Key 1** or **Key 2**
    4. Note your **Endpoint** URL (should be `https://<resource-name>.openai.azure.com`)

    **Important**: Keep your API key secure - it provides access to your Azure deployments.
  </Step>
</Steps>

## Add to OffRail [#add-to-offrail]

<Steps>
  <Step>
    ### Navigate to Provider Keys [#navigate-to-provider-keys]

    1. Log into [OffRail Dashboard](https://offrail.ai/dashboard)
    2. Select your organization and project
    3. Go to **Provider Keys** in the sidebar
  </Step>

  <Step>
    ### Add Azure Provider Key [#add-azure-provider-key]

    1. Click **Add** for **Azure**
    2. Enter your **API Key** from Azure Portal
    3. Enter your **Resource Name** (the name from your Azure endpoint URL)
       * Example: If your endpoint is `https://my-openai-resource.openai.azure.com`, enter `my-openai-resource`
    4. Select your preferred **type** (Azure OpenAI or AI Foundry)
    5. Adapt the **Validation Model** to a model that you already deployed and is available
       This is a one time check to ensure the API key is valid and the model can be accessed.
    6. Click **Add Key**

    The system will validate your key and confirm the connection.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    Test your integration with a simple API call:

    ```bash
    curl -X POST https://api.offrail.ai/v1/chat/completions \
      -H "Authorization: Bearer YOUR_OFFRAIL_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "azure/gpt-4o-mini",
        "messages": [
          {
            "role": "user",
            "content": "Hello from Azure!"
          }
        ]
      }'
    ```

    Replace `YOUR_OFFRAIL_API_KEY` with your OffRail API key.
  </Step>
</Steps>

## Available Models [#available-models]

Once configured, you can access your Azure deployments through OffRail:

* **GPT-4o**: `azure/gpt-4o`
* **GPT-4o Mini**: `azure/gpt-4o-mini`
* **GPT-3.5 Turbo**: `azure/gpt-3.5-turbo` (note: use gpt-3.5-turbo as offrail model name instead of gpt-35-turbo)

**Note**: Only models you have deployed in Azure Studio will be available. Ensure your deployment names match the expected model identifiers.

Browse all available models at [offrail.ai/models](https://offrail.ai/models?provider=azure)

## Troubleshooting [#troubleshooting]

### "Deployment not found" error [#deployment-not-found-error]

* Verify you've created a deployment in Azure Studio
* Ensure the deployment name exactly matches the model name you're requesting
* Check that the deployment is in the same resource as your API key

### "Resource not found" error [#resource-not-found-error]

* Verify the resource name is correct (check your Azure Portal endpoint URL)
* Ensure your API key belongs to the correct Azure resource
* Confirm the resource is in an active state in Azure Portal

### Rate limiting [#rate-limiting]

* Azure has Tokens Per Minute (TPM) quotas per deployment
* Monitor usage in Azure Studio under **Quotas**
* Request quota increases through Azure Portal if needed for high-volume workloads

### Region availability [#region-availability]

* Not all models are available in all Azure regions
* Check [Azure model availability](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability) for your region
* Consider creating resources in multiple regions for better availability


# Vertex AI Anthropic Integration
URL: https://docs.offrail.ai/integrations/vertex-anthropic


Run Claude models (Sonnet, Opus, Haiku) on Google Cloud Vertex AI through OffRail. This guide shows how to set up a GCP service account and integrate it with OffRail using automatic OAuth2 token management — no manual token rotation required.

## Prerequisites [#prerequisites]

* A Google Cloud project with billing enabled
* OffRail account or self-hosted instance

## Set up Google Cloud [#set-up-google-cloud]

<Steps>
  <Step>
    ### Enable the Vertex AI API [#enable-the-vertex-ai-api]

    In the [Google Cloud Console](https://console.cloud.google.com/apis/library/aiplatform.googleapis.com), enable the **Vertex AI API** for your project.
  </Step>

  <Step>
    ### Enable Claude Models in Model Garden [#enable-claude-models-in-model-garden]

    Navigate to **Vertex AI > Model Garden** in the Cloud Console. Search for the Claude models you want to use and click **Enable** on each one.

    Available models:

    * `claude-sonnet-4-6`
    * `claude-sonnet-4-5`
    * `claude-haiku-4-5`
    * `claude-opus-4-5`
    * `claude-opus-4-6`
    * `claude-opus-4-7`
  </Step>

  <Step>
    ### Create a Service Account [#create-a-service-account]

    Create a service account with the required permissions:

    ```bash
    # Create the service account
    gcloud iam service-accounts create vertex-ai-caller \
      --display-name="Vertex AI Caller" \
      --project=YOUR_PROJECT_ID

    # Grant the Vertex AI User role
    gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
      --member="serviceAccount:vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
      --role="roles/aiplatform.user"
    ```
  </Step>

  <Step>
    ### Download the Service Account Key [#download-the-service-account-key]

    ```bash
    gcloud iam service-accounts keys create service-account.json \
      --iam-account=vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com
    ```

    Then convert it to a single-line string:

    ```bash
    cat service-account.json | tr -d '\n'
    ```

    Keep the output handy — you'll paste it into OffRail in the next steps.
  </Step>
</Steps>

## Add to OffRail [#add-to-offrail]

<Steps>
  <Step>
    ### Navigate to Provider Keys [#navigate-to-provider-keys]

    1. Log into [OffRail Dashboard](https://offrail.ai/dashboard)
    2. Select your organization and project
    3. Go to **Provider Keys** in the sidebar
  </Step>

  <Step>
    ### Add Vertex Anthropic Provider Key [#add-vertex-anthropic-provider-key]

    1. Click **Add** for &#x2A;*Vertex AI (Anthropic)**
    2. Paste the single-line service account JSON as the **API Key**
    3. Leave **Region** empty to use the recommended `global` endpoint, or set a specific region (e.g. `us-east5`) if you need data residency
    4. Click **Add Key**

    The project ID is extracted automatically from the service account JSON — no separate project field is needed.
  </Step>

  <Step>
    ### Test the Integration [#test-the-integration]

    ```bash
    curl -X POST https://api.offrail.ai/v1/chat/completions \
      -H "Authorization: Bearer YOUR_OFFRAIL_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "vertex-anthropic/claude-sonnet-4-6",
        "messages": [
          {
            "role": "user",
            "content": "Hello from Vertex Anthropic!"
          }
        ]
      }'
    ```

    Replace `YOUR_OFFRAIL_API_KEY` with your OffRail API key.
  </Step>
</Steps>

## Self-Host Configuration [#self-host-configuration]

If you're self-hosting OffRail, configure the provider via environment variables instead of the dashboard:

```bash
LLM_VERTEX_ANTHROPIC_SERVICE_ACCOUNT_JSON={"type":"service_account","project_id":"YOUR_PROJECT_ID","private_key":"-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n","client_email":"vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com","token_uri":"https://oauth2.googleapis.com/token"}
LLM_VERTEX_ANTHROPIC_REGION=global
```

<Callout type="info">
  The project ID is extracted automatically from the service account JSON, so
  `LLM_VERTEX_ANTHROPIC_PROJECT` only needs setting to call a different project
  than the service account's own.
</Callout>

## How Token Refresh Works [#how-token-refresh-works]

OffRail handles the OAuth2 token lifecycle automatically:

1. On first request, the service account JSON is parsed and used to sign a JWT
2. The JWT is exchanged for an OAuth2 access token via Google's token endpoint
3. The token is cached in Redis with a **50-minute TTL** (Google tokens expire after 60 minutes)
4. An in-memory cache avoids Redis round-trips on subsequent requests
5. When the cached token expires, a new one is generated transparently

This means:

* No manual `gcloud auth print-access-token` commands
* No cron jobs to refresh tokens
* Works at any request rate (token generation happens at most once per 50 minutes)
* Multi-instance deployments share the cached token via Redis

## Available Regions [#available-regions]

OffRail defaults to the &#x2A;*`global`** endpoint, which Anthropic recommends: requests are routed dynamically to whichever region has capacity, and there is no pricing premium.

| Region            | Notes                                         |
| ----------------- | --------------------------------------------- |
| `global`          | Default — dynamic routing, no pricing premium |
| `us`              | Multi-region (US only); 10% premium           |
| `eu`              | Multi-region (EU only); 10% premium           |
| `us-east5`        | Columbus, Ohio; 10% premium                   |
| `us-central1`     | Iowa; 10% premium                             |
| `europe-west1`    | Belgium; 10% premium                          |
| `europe-west4`    | Netherlands; 10% premium                      |
| `asia-southeast1` | Singapore; 10% premium                        |

<Callout type="info">
  Regional and multi-region endpoints add a 10% pricing premium on Claude Sonnet
  4.5 and newer models. They are also required if you need single-region data
  residency or provisioned throughput. See [Anthropic's Vertex
  docs](https://platform.claude.com/docs/en/api/claude-on-vertex-ai#global-multi-region-and-regional-endpoints)
  for details.
</Callout>

## Available Models [#available-models]

Once configured, you can access Claude models on Vertex AI through OffRail. Browse the full, up-to-date list at [offrail.ai/models?provider=vertex-anthropic](https://offrail.ai/models?provider=vertex-anthropic).

## Troubleshooting [#troubleshooting]

### 401 UNAUTHENTICATED / ACCESS\_TOKEN\_TYPE\_UNSUPPORTED [#401-unauthenticated--access_token_type_unsupported]

The gateway is sending an invalid token. Check:

* The service account JSON is valid and complete
* The service account has `roles/aiplatform.user` on the project

### 403 Permission Denied [#403-permission-denied]

The service account lacks permissions. Grant the `Vertex AI User` role:

```bash
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:vertex-ai-caller@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"
```

### Model Not Found [#model-not-found]

The Claude model may not be enabled in your project's Model Garden, or may not be available in the selected region. Check the [Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) in Cloud Console.


# Activity
URL: https://docs.offrail.ai/learn/activity


The Activity page shows a real-time log of every API request routed through OffRail. Use it to debug requests, monitor performance, and track costs per call.

## Filters [#filters]

Filter the activity log using the controls at the top:

| Filter                      | Description                                             |
| --------------------------- | ------------------------------------------------------- |
| **Time range**              | Filter by a specific time period                        |
| **Unified reasons**         | Filter by completion reason (e.g., stop, length, error) |
| **Providers**               | Show requests for specific providers only               |
| **Models**                  | Show requests for specific models only                  |
| **Custom header key/value** | Filter by custom metadata headers attached to requests  |

## Activity List [#activity-list]

Each activity entry shows:

* **Status icon** — Green checkmark for completed, red circle for errors
* **Response preview** — First line of the model's response (when available)
* **Model** — The provider and model used (e.g., `google-vertex/gemini-3-pro-image-preview`)
* **Cache status** — Whether the response was served from cache
* **Tokens** — Total tokens consumed (input + output)
* **Duration** — How long the request took
* **Cost** — Inference cost for the request
* **Source** — Where the request originated from
* **Discount** — Any discount applied (e.g., "20% off")
* **Status badge** — `completed`, `upstream_error`, `gateway_error`, etc.
* **Timestamp** — Relative time (e.g., "about 4 hours ago")

### Actions per Entry [#actions-per-entry]

* **Open in new tab** — View the full request detail in a new browser tab
* **Expand** — Expand inline to see more details

## Activity Detail [#activity-detail]

Click on any activity entry to view its full detail page.

### Summary Cards [#summary-cards]

Five cards at the top provide a quick overview:

| Card               | Description                     |
| ------------------ | ------------------------------- |
| **Duration**       | Total request time in seconds   |
| **Tokens**         | Total tokens consumed           |
| **Throughput**     | Tokens per second               |
| **Inference Cost** | Cost charged for this request   |
| **Cache**          | Whether the response was cached |

### Request Section [#request-section]

Details about the original request:

* **Requested Model** — The model ID sent in the API call
* **Used Model** — The actual model that served the request
* **Model Mapping** — The underlying model identifier
* **Provider** — The provider that handled the request
* **Requested Provider** — The provider specified in the request
* **Streamed** — Whether the response was streamed
* **Canceled** — Whether the request was canceled
* **Source** — The application or service that made the request

### Tokens Section [#tokens-section]

A detailed token breakdown:

* Prompt Tokens, Completion Tokens, Total Tokens
* Reasoning Tokens (for reasoning models)
* Image Input/Output Tokens (for vision/image models)
* Response Size

### Routing Section [#routing-section]

How OffRail routed the request:

* **Selection** — The routing strategy used (e.g., `direct-provider-specified`)
* **Available** — Providers that were available for this model
* **Provider Scores** — Scoring breakdown showing availability, uptime, and latency for each provider

### Parameters Section [#parameters-section]

The model parameters sent with the request:

* Temperature, Max Tokens, Top P
* Frequency Penalty, Reasoning Effort
* Response Format


# Agents Monitoring
URL: https://docs.offrail.ai/learn/agents


The Agents page groups your project's gateway traffic by the coding agent that
produced it — Claude Code, Cursor, OpenCode, Cline, Codex and
others — so you can see which tool is spending your credits and drill into the
individual sessions behind the numbers.

<Callout type="info">
  Agents are identified by the `x-source` header their requests send. Anything
  that doesn't match a known agent stays out of this view — use
  [**Activity**](https://docs.offrail.ai/learn/activity) for the full request log.
</Callout>

## Filtering the view [#filtering-the-view]

Two controls sit above the agent cards and both are stored in the URL, so a
filtered view can be bookmarked or shared:

| Control        | Options              | Default |
| -------------- | -------------------- | ------- |
| **Time range** | 1h, 4h, 24h, 7d, 30d | 7d      |
| **Usage mode** | All, Credits, BYOK   | All     |

**Credits** shows only traffic billed against your credit balance, **BYOK**
only traffic served by your own [provider keys](https://docs.offrail.ai/learn/provider-keys). The
summary on the right of the row totals the agents, requests, and spend
currently in view.

## Agent Cards [#agent-cards]

Each agent that was active in the selected period gets a card:

| Field           | Description                                |
| --------------- | ------------------------------------------ |
| **Name**        | The agent, matched from its request source |
| **Total cost**  | Spend for this agent in the period         |
| **Requests**    | Number of API requests                     |
| **Tokens**      | Prompt and completion tokens combined      |
| **Last active** | When the agent last sent a request         |

Click a card to open its detail view.

## Agent Detail [#agent-detail]

The detail view splits the agent's requests into **sessions** — consecutive
requests with less than a 30-minute gap between them. Each session row shows
its time range, request count, total tokens, duration, and cost. Expand a
session to see the individual requests with their response previews, model,
cache status, tokens, cost, and source.

Sessions load as you scroll; **Load more sessions** fetches the next page
manually.

## Exporting to CSV [#exporting-to-csv]

**Export CSV** downloads every request in the selected period — not just the
sessions currently loaded — for the agent you are viewing. Use it to reconcile
agent spend against your own reporting.

See also [**Coding Agents**](https://docs.offrail.ai/learn/coding-agents).


# Analytics
URL: https://docs.offrail.ai/learn/analytics


The Analytics page shows where a project's spend actually goes. It leads with the project's totals for the range, then breaks usage down — as a ranking and over time — so you can see what drives cost, requests, and tokens across any date range.

Open it from the **Analytics** item in the project sidebar. Like the rest of the dashboard, the page respects the shared date-range picker at the top, so every chart reflects the same window.

## Totals [#totals]

Three cards at the top show the project's **total cost**, **requests**, and **tokens** for the selected range, each with a sparkline of its daily values. They are computed from the same data as the charts below, so the total and the breakdown always agree.

## Choosing a breakdown [#choosing-a-breakdown]

The **Group by** selector switches which dimension the two charts break the project down by:

| Breakdown   | What it shows                                              |
| ----------- | ---------------------------------------------------------- |
| **Model**   | Spend per model (the default)                              |
| **API key** | Spend per API key in the project                           |
| **User**    | Spend per team member — Enterprise, owners and admins only |

**Breakdown by user** answers "what is each person on this team costing us?" in the same view as the project total. It is available to organization owners and admins on the Enterprise plan — see [Member Analytics](https://docs.offrail.ai/learn/member-analytics).

Spend is attributed to whoever **created** the API key that made each request. Traffic from platform keys has no member behind it, so it is shown as **Unattributed** rather than dropped, keeping the per-member rows summing to the project total.

## Cost by Model [#cost-by-model]

A horizontal bar chart that ranks the models used in the selected range. Switch the metric with the tabs above the chart:

| Tab          | What it ranks                          |
| ------------ | -------------------------------------- |
| **Cost**     | Total spend per model, in USD          |
| **Requests** | Number of requests routed to the model |
| **Tokens**   | Total tokens (input + output)          |

Use it to spot the one or two models responsible for most of your bill, or to confirm that traffic is spread the way you expect.

## Cost by Model Over Time [#cost-by-model-over-time]

A stacked area chart that plots the same three metrics across the date range, with one band per model. It has the same **Cost / Requests / Tokens** tabs, plus a **Mappings / Canonical** toggle:

* **Mappings** — each model variant is shown separately, exactly as it was requested (for example a custom provider mapping is kept distinct from the built-in model).
* **Canonical** — variants of the same underlying model are collapsed into one canonical model (the provider prefix and tag are dropped), so `openai/gpt-5.5` and a custom mapping of it count as a single line.

Switch to **Canonical** when you care about the underlying model's total footprint; stay on **Mappings** when you need to compare specific routes.

The **Mappings / Canonical** toggle is specific to models — the API key and user breakdowns rank named entities, so there is nothing to collapse.

## How the data is computed [#how-the-data-is-computed]

Both charts derive from the same activity data the rest of the dashboard already reads — there is no separate analytics pipeline to wait on. Aggregation happens per time bucket and is timezone-correct, so totals line up with the Activity and Usage pages for the same range.

For a per-key view of the same breakdowns, see [API Keys](https://docs.offrail.ai/learn/api-keys#per-key-statistics). For an org-wide, per-person view on the Enterprise plan, see [Member Analytics](https://docs.offrail.ai/learn/member-analytics).


# API Keys
URL: https://docs.offrail.ai/learn/api-keys


The API Keys page is the main place to create, secure, and operate the keys
your apps use to authenticate with OffRail.

Use this page to:

* Create project-specific API keys
* Set all-time and recurring spend limits per key
* Set an expiration (TTL) so a key disables itself automatically
* Track usage for each key, including the active recurring window
* Rename keys and enable or disable them without deleting them
* Roll a key's secret in place if it may have leaked
* Configure IAM rules for model, provider, and pricing access

<Callout type="warning">
  API keys are shown in full only once, immediately after creation. Copy and
  store them securely before closing the dialog.
</Callout>

## Creating an API Key [#creating-an-api-key]

Click **Create API Key** and configure:

* **Name**: A label such as `production`, `staging`, or `ci`
* **Expiration (TTL)**: An optional time-to-live after which the key disables itself
* **All-time usage limit**: An optional lifetime spend cap for the key
* **Recurring usage limit**: An optional spend cap that resets on a schedule

Recurring limits support:

* Minimum window: **1 hour**
* Maximum window: **12 months**
* Units: **hour**, **day**, **week**, or **month**

This is useful when you want a key to stay below a fixed budget per hour, day,
week, or month, while still keeping a separate lifetime cap if needed.

## Expiration (TTL) [#expiration-ttl]

Turn on &#x2A;*Set expiration (TTL)** when creating a key to give it a limited
lifetime. Choose a value and a unit — **minutes**, **hours**, or **days** — and
the key is disabled automatically once that time passes. Leave it off for a key
that never expires.

Expired keys show an **Expired** indicator in the list and move to the
**Inactive** tab. To use one again, reactivate it and pick a **new future
expiration**:

* **Activate** an expired key and you'll be prompted to set a fresh TTL before it
  comes back online
* Keys with no TTL, or whose TTL is still in the future, can be enabled and
  disabled without setting a new expiration

This makes TTL keys ideal for temporary access — short-lived demos, CI runs, or
contractor keys that should not linger.

## Usage Limits [#usage-limits]

Each API key can enforce two independent limit types:

| Limit Type                | What it does                                                    |
| ------------------------- | --------------------------------------------------------------- |
| **All-time usage limit**  | Stops the key after it reaches a lifetime spend threshold       |
| **Recurring usage limit** | Stops the key after it reaches the budget for the active window |

Examples:

* `$50` all-time for a temporary integration key
* `$10 / 1 day` for a development key
* `$500 / 1 month` for a production service key

If a key hits either limit, requests using that key are rejected until the key
is updated or, for recurring limits, the next window begins.

### How recurring windows work [#how-recurring-windows-work]

Recurring usage is tracked separately from total lifetime usage.

* The dashboard shows the key's **Current Period** usage
* The active window also shows when it **resets**
* When the configured window expires, usage for that window resets automatically
* Updating the recurring limit configuration resets the current window and starts
  a new one

Usage includes both OffRail credits and requests routed through your own
provider keys when applicable.

## API Keys List [#api-keys-list]

Each key in the list shows:

| Field              | Description                                                   |
| ------------------ | ------------------------------------------------------------- |
| **Name**           | The label you assigned to the key                             |
| **API Key**        | A masked preview of the key                                   |
| **Status**         | Whether the key is active or inactive, plus its expiry if set |
| **Created**        | When the key was created                                      |
| **Usage**          | Total tracked usage for the key                               |
| **Current Period** | Spend in the active recurring window, if configured           |
| **Limits**         | All-time and recurring limit summary                          |
| **IAM Rules**      | Whether model/provider/pricing access controls are configured |

## Actions [#actions]

The actions menu on each key offers:

* **View Statistics**: Open a dedicated analytics page for that key (see below)
* **Manage IAM Rules**: Restrict which models, providers, or pricing tiers the key can use
* **Rename Key**: Change the key's label without touching its secret or history
* **Activate / Deactivate Key**: Pause usage without deleting the key (reactivating
  an expired key prompts for a new expiration)
* **Roll Key**: Generate a new secret for the key and invalidate the old one (see below)
* **Update limits**: Change all-time or recurring limits
* **Delete**: Permanently remove the key

<Callout type="info">
  The auto-generated playground key is managed for you: it cannot be renamed,
  rolled, deactivated, or deleted. Members with the developer role can only
  modify keys they created themselves.
</Callout>

## Rolling a Key [#rolling-a-key]

**Roll Key** replaces a key's secret while keeping the key itself. The name,
usage history and statistics, all-time and recurring limits (including the
current period window), IAM rules, and expiration all stay exactly as they were —
only the secret changes.

This is the fastest fix when a secret leaks into a commit, a log, or a shared
environment: you cut off the exposed value without recreating the key or losing
its spend tracking.

1. Open the key's actions menu and choose **Roll Key**
2. Confirm in the dialog
3. Copy the new secret and update every client using the old one

<Callout type="warning">
  The previous secret stops working immediately, and the new one is shown only
  once. Requests still using the old secret fail with `401 Unauthorized`.
</Callout>

## Per-Key Statistics [#per-key-statistics]

The **View Statistics** action opens a dedicated page scoped to a single API
key, so you can see exactly what that key is doing without filtering the whole
project.

The page respects the shared date-range picker and shows:

* **Summary cards** — the key's cost, tokens, requests, and error rate for the
  selected range.
* **Cost by Model** — a horizontal bar chart ranking the key's models by cost,
  requests, or tokens.
* **Cost by Model Over Time** — a stacked area chart of the same metrics, with a
  Mappings / Canonical toggle.

These are the same breakdowns as the project [Analytics](https://docs.offrail.ai/learn/analytics) page,
narrowed to the one key — useful for confirming a key is healthy and spending on
the models you expect.

## IAM Rules [#iam-rules]

IAM rules let you narrow what an API key is allowed to access.

Supported rule types include:

* **Allow/Deny models**
* **Allow/Deny providers**
* **Allow/Deny pricing**
* **Allow/Deny IP ranges (CIDR)** — Enterprise plan only

Use IAM rules when you want a key to be valid, but only for a specific subset of
models or providers. For a deeper explanation, see the [API Keys & IAM Rules
feature page](https://docs.offrail.ai/features/api-keys).

Org admins can additionally set [member-level IAM
rules](https://docs.offrail.ai/features/api-keys#member-level-iam-rules) on the [Team
page](https://docs.offrail.ai/learn/team). Those act as a ceiling for every key the member creates:
key rules can only narrow access within them, never expand it. The IAM page
shows a notice when organization-level restrictions apply to your keys.

## Plan Limits [#plan-limits]

The page also shows how many API keys you are using relative to your plan
allowance. The cap counts **active** keys across every project in the
organization, so deactivated and deleted keys do not count against it.

* **Free**: Standard API key count limit
* **Pro**: A larger allowance
* **Enterprise**: Custom limits

Owners and admins can also set a per-member key cap on the [Team
page](https://docs.offrail.ai/learn/team), which applies on top of the organization-wide allowance.

If you reach the limit, the **Create API Key** button is disabled until you
delete or deactivate unused keys, or upgrade.


# Audit Logs
URL: https://docs.offrail.ai/learn/audit-logs


The Audit Logs page provides a complete history of all actions performed within your organization, essential for compliance and security monitoring.

<Callout type="info">
  Audit Logs are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise). Owner or Admin role is required.
</Callout>

## Filters [#filters]

Narrow down the log entries:

* **Action** — Filter by action type (create, delete, update, etc.)
* **Resource type** — Filter by resource (API, IAM, API Keys, etc.)

Both filters are populated dynamically based on the actions recorded in your organization.

## Audit Log Entries [#audit-log-entries]

Each log entry shows:

| Field             | Description                                                  |
| ----------------- | ------------------------------------------------------------ |
| **Timestamp**     | Exact time of the action (formatted as MMM d, yyyy HH:mm:ss) |
| **User**          | Name and email of the person who performed the action        |
| **Action**        | What was done (e.g., "API Keys → create")                    |
| **Resource type** | The type of resource affected (shown as a badge)             |
| **Resource ID**   | Identifier of the affected resource (with copy button)       |
| **Details**       | Additional metadata about the action                         |

## Pagination [#pagination]

The log supports infinite scrolling with a **Load More** button to view older entries. Entries are sorted newest first.


# Billing
URL: https://docs.offrail.ai/learn/billing


The Billing page is your central hub for managing credits, plans, and payment methods.

## Credits [#credits]

Displays your current credit balance. Credits are consumed as you make API requests through the gateway. Click **Top Up Credits** to add more credits to your account.

## Fees [#fees]

Top-ups are charged the credit amount plus the following fees:

* **Platform fee** — A flat 5% fee applied to every credit purchase.
* **International card fee** — An additional 1.5% fee applied when paying with a non-US issued card. This covers the higher processing cost charged by the card network for international transactions. Cards issued in the United States are not subject to this fee.

The full breakdown (credits, platform fee, and — when applicable — the international card fee) is shown in the top-up dialog before you confirm payment, so the total charge is always transparent.

## Plan Management [#plan-management]

View and manage your subscription:

* See your current plan (Free or Enterprise)
* Billing cycle information
* Click **Manage Subscription** to upgrade, downgrade, or cancel

## Payment Methods [#payment-methods]

Manage your saved payment methods:

* Add a new credit card or payment method
* View existing payment methods
* Update billing information

## Auto Top-up Settings [#auto-top-up-settings]

Configure automatic credit top-ups so you never run out:

* **Enable/disable** auto top-up
* **Threshold** — The credit balance that triggers a top-up
* **Amount** — How many credits to add when the threshold is reached

This ensures uninterrupted service by automatically replenishing your credits when they run low.


# Lounge Memberships
URL: https://docs.offrail.ai/learn/chat-plans


Lounge memberships (formerly Chat Plans) are optional monthly subscriptions for [Lounge](https://lounge.offrail.ai), the OffRail chat app. Instead of paying per request from your pay-as-you-go balance, a membership gives you a pool of monthly credits worth more than you pay — so heavy chat usage costs less.

## Plans [#plans]

There are three tiers, billed monthly:

| Plan        | Price  | Monthly value    | Models                                                                       |
| ----------- | ------ | ---------------- | ---------------------------------------------------------------------------- |
| **Starter** | $9/mo  | \~2× the value   | Most chat models — Claude Haiku & Sonnet, GPT-5-mini, Gemini Flash, and more |
| **Plus**    | $19/mo | \~2.5× the value | Everything in Starter **plus** frontier models                               |
| **Pro**     | $49/mo | \~3× the value   | All models, highest monthly allowance                                        |

The credit multiplier is tapered: the larger the plan, the more usage value each dollar buys at provider rates.

<Callout type="info">
  **Frontier models** — flagship models such as Claude Opus, GPT-5, Gemini 2.5
  Pro, and Grok 4 are included on **Plus** and **Pro**. The Starter plan covers
  the broad catalog of everyday chat models but does not include these frontier
  models.
</Callout>

## How credits work [#how-credits-work]

* **Monthly reset** — Your plan credits refresh at the start of each billing cycle. Unused credits do **not** roll over to the next month.
* **Plan credits drain first** — Requests made from the chat app draw down your plan's monthly credits before anything else.
* **Pay-as-you-go fallback** — Once your monthly credits are used up, the chat app falls back to your regular pay-as-you-go balance, which never expires. You can keep chatting without interruption.

## Managing your plan [#managing-your-plan]

* Open the **Pricing** page from the Lounge sidebar to compare tiers and subscribe.
* Your active membership appears in the Lounge sidebar with a badge, alongside how many credits remain for the cycle.
* **Upgrading** takes effect immediately: you're charged the new tier's full price, your billing cycle restarts, and your credits reset to the new tier's full monthly allowance. Unspent credits from the old cycle don't carry over.
* **Downgrading** also takes effect immediately: you move to the lower tier's allowance right away, and the unused portion of what you already paid for the higher tier is credited against your next invoice.
* **Cancelling** takes effect at the end of the period you've already paid for — you keep access until then.


# Coding Agents
URL: https://docs.offrail.ai/learn/coding-agents


The Coding Agents section of the [dashboard](https://offrail.ai/dashboard)
breaks your usage down by the tool that produced it, so you can tell at
a glance whether your allowance is going to Claude Code, Cursor, OpenCode, or
something else.

Agents are recognised from the `x-source` header their requests send. Supported
tools include Claude Code, OpenCode, Cursor, Autohand Code,
Empryo, SoulForge, Cline, Codex CLI, GitHub Copilot, n8n and OpenClaw — each
one has a setup guide under [Guides](https://docs.offrail.ai/guides).

<Callout type="info">
  The overview lists the **last 7 days**. Open an agent to pick a wider window.
  If nothing shows up yet, point your tool at OffRail with the two environment
  variables from the dashboard's Quick start card and run it once.
</Callout>

## Agent Cards [#agent-cards]

| Field           | Description                                      |
| --------------- | ------------------------------------------------ |
| **Name**        | The coding tool, matched from its request source |
| **Total cost**  | Spend attributed to this agent                   |
| **Requests**    | Number of API requests                           |
| **Tokens**      | Prompt and completion tokens combined            |
| **Last active** | When the agent last sent a request               |

Click a card to open the agent's detail page.

## Agent Detail [#agent-detail]

The header repeats the agent's totals for the selected period and holds the
two controls for the page:

* **Time range** — 1h, 4h, 24h, 7d or 30d (7d by default), stored in the URL so
  the view can be shared.
* **Export CSV** — downloads every request in the period, not just the rows
  currently loaded.

Below it are three blocks:

**Model usage chart** — spend over time, split by model.

**Model usage table** — one row per model with Provider, Requests, Tokens,
Cost, Cache cost, and a share-of-tokens bar. Every column header sorts.

**Requests** — the individual calls, newest first, each showing the model used,
cache status, token count and cost. More rows load as you scroll, or with
**Load more**.

## Related pages [#related-pages]

* [**Model Categories & Fair Use**](https://docs.offrail.ai/learn/model-categories) — which models
  count against the premium allowance
* [**Agents Monitoring**](https://docs.offrail.ai/learn/agents) — the same breakdown for a regular
  gateway project, with sessions and BYOK filtering


# Dashboard
URL: https://docs.offrail.ai/learn/dashboard


The Dashboard is the first page you see after logging in. It provides a high-level overview of your project's LLM usage, costs, and performance at a glance.

## Date Range [#date-range]

At the top of the page, you can toggle the date range for all dashboard metrics:

* **7 days** — Last 7 days of data (default)
* **30 days** — Last 30 days of data
* **Custom** — Pick a custom start and end date

## Stat Cards [#stat-cards]

The dashboard displays eight metric cards in two rows:

### Top Row [#top-row]

| Card                     | Description                                                              |
| ------------------------ | ------------------------------------------------------------------------ |
| **Organization Credits** | Your current available credit balance                                    |
| **Total Requests**       | Number of API requests in the selected period, with cache hit percentage |
| **Total Cost**           | Total inference cost for the period, including storage costs             |
| **Total Savings**        | Savings from discounts during the selected period                        |

### Bottom Row [#bottom-row]

| Card                     | Description                                                         |
| ------------------------ | ------------------------------------------------------------------- |
| **Input Tokens & Cost**  | Total prompt tokens sent and their associated cost                  |
| **Output Tokens & Cost** | Total completion tokens received and their associated cost          |
| **Cached Tokens & Cost** | Tokens served from cache (if caching is enabled) and the cost saved |
| **Most Used Model**      | The model with the highest request count, along with its provider   |

## Usage Overview Chart [#usage-overview-chart]

Below the stat cards, a chart visualizes your usage over time. You can toggle between two views using the dropdown:

* **Costs** — Shows input, output, and cached input costs as a stacked area chart
* **Requests** — Shows request volume over time

The chart is filtered by the currently selected project.

## Quick Actions [#quick-actions]

A sidebar panel provides shortcuts to common tasks:

* **Manage API Keys** — Go to the API Keys page
* **Provider Keys** — Configure your own provider keys
* **View Activity** — See detailed request logs
* **Usage & Metrics** — Dive into usage analytics
* **Model Usage** — View per-model usage breakdown

## Cost Breakdown [#cost-breakdown]

A donut chart showing how your costs are distributed across different models and providers. Each segment is color-coded and labeled with the model name and cost, making it easy to identify your biggest cost drivers.

## Errors & Reliability [#errors--reliability]

Displays two key reliability metrics:

* **Error Rate** — Percentage of failed requests over the selected period
* **Uptime** — Gateway availability percentage

## Recent Activity [#recent-activity]

A table showing your most recent API requests with key details like model, status, tokens, duration, and cost. Click any entry to view the full request detail.

## Header Actions [#header-actions]

Two buttons in the top-right corner:

* **Create API Key** — Quickly create a new API key for your project
* **Top Up Credits** — Add credits to your organization balance


# Dynamic Routes
URL: https://docs.offrail.ai/learn/dynamic-routes


The Dynamic Routes page in your project settings lets you define named routing flows — conditions on the request, percentage-based A/B splits, and model targets with provider fallback — and invoke them by putting `dynamic/<name>` in the `model` field instead of a concrete model. Routing logic moves out of your application code: change the flow in the dashboard and every client picks it up on the next request, no deploy needed.

<Callout type="info">
  Dynamic routes are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise) and can be managed by organization
  owners and admins.
</Callout>

## Creating a Route [#creating-a-route]

Enter a name under **New route** and click **Create**. Route names are lowercase slugs (letters, digits, and single hyphens) and become part of the model string — a route named `support` is invoked as `dynamic/support`. New routes start with a starter graph you can edit right away.

The route list shows each route's state at a glance: the published version badge (for example **v2**), **unpublished** when no version has been published yet, and **off** when the route is disabled.

## The Visual Editor [#the-visual-editor]

The **Visual** tab is a drag-and-drop canvas. Add nodes from the palette (drag onto the canvas or click), wire branches by connecting the round handles, and select any node to edit its settings in the inspector — conditions, split weights, the target model (with a searchable catalog picker), and an ordered provider fallback list.

Every flow starts at the **Start** node and ends at either a **Model** node (route the request) or an **End** node (reject the request with a `400`):

| Node type       | What it does                                                                                           |
| --------------- | ------------------------------------------------------------------------------------------------------ |
| **Model**       | Routes to a catalog model; an optional provider list restricts routing and sets the fallback order     |
| **Conditional** | If/else branching on request headers, body fields, or request metadata — first matching condition wins |
| **Percentage**  | Weighted traffic splits, deterministic per session so a conversation keeps its A/B assignment          |
| **End**         | Explicitly rejects the request with a `400`                                                            |

See the [dynamic routes feature docs](https://docs.offrail.ai/features/dynamic-routes) for the full condition operator and field reference.

## The JSON Editor [#the-json-editor]

The **JSON** tab edits the same graph as raw JSON — useful for copy-pasting flows between projects or generating them programmatically. Both tabs stay in sync, and both validate against the full schema before saving: unknown models, unwired branches, cycles, and invalid conditions are listed under **Fix before saving** and block the save until resolved, so an invalid graph can never serve traffic.

## Publishing and Rolling Back [#publishing-and-rolling-back]

Edits go to the route's draft; live traffic is never affected until you click **Publish**, which snapshots the draft as an immutable, numbered version and re-validates it against the current model catalog. The **Versions** card lists the full history — **Load into draft** copies an old version into the editor, and **Publish this** rolls back to it instantly.

A route serves requests only while it is **Enabled** and has a published version; otherwise requests using `dynamic/<name>` are rejected with a `404`.

## Invoking a Route [#invoking-a-route]

Use the route name in place of a model on any OpenAI-compatible endpoint:

```bash
curl https://api.offrail.ai/v1/chat/completions \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "x-user-tier: paid" \
  -d '{
    "model": "dynamic/support",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

Once the flow resolves to a model, the request goes through the same weighted [smart routing](https://docs.offrail.ai/features/routing) as any other — provider scoring, sticky sessions, and automatic cross-provider fallback included. Each request's [activity log](https://docs.offrail.ai/learn/activity) records the route name, version, and the node path the evaluation took.


# Guardrails
URL: https://docs.offrail.ai/learn/guardrails


The Guardrails page lets you configure content safety rules that automatically scan and filter API requests before they reach the LLM provider.

<Callout type="info">
  Guardrails are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise). Owner or Admin role is required.
</Callout>

## Main Toggle [#main-toggle]

A global toggle at the top enables or disables all guardrails for your organization. Click **Save Changes** to apply.

## Organization vs. Project [#organization-vs-project]

The same page exists in each project's settings. By default a project uses the organization guardrails, and the project page shows them read-only alongside a banner naming the project — for example "Enterprise Project uses the organization guardrails".

Turn off **Use organization guardrails** on the project page to give that project its own configuration and custom rules, which then replace the organization's for that project's traffic. The project starts from a copy of the organization's current settings, so opting out does not drop the rules that were already in force — edit down from there and click **Save Changes**.

The organization page names any projects that override it ("The settings below apply to every project except …"), so it is always clear where these settings do and do not apply.

## System Rules [#system-rules]

Six built-in rules with individual enable/disable toggles:

| Rule                            | Description                                                          |
| ------------------------------- | -------------------------------------------------------------------- |
| **Prompt Injection Detection**  | Detects attempts to override or manipulate system instructions       |
| **Jailbreak Prevention**        | Identifies attempts to bypass safety measures                        |
| **PII Detection**               | Identifies personal information like emails, phone numbers, and SSNs |
| **Secrets Detection**           | Detects API keys, passwords, and credentials                         |
| **File Type Restrictions**      | Controls which file types can be uploaded                            |
| **Document Leakage Prevention** | Detects attempts to extract confidential documents                   |

Each rule has an action dropdown to configure the response:

* **Block** — Reject the request entirely
* **Redact** — Remove or mask sensitive content, then continue
* **Warn** — Log the violation but allow the request

## File Restrictions [#file-restrictions]

Configure file upload limits:

* **Max file size** — Set the maximum file size in MB
* **Allowed file types** — Add or remove permitted MIME types

## Custom Rules [#custom-rules]

Create organization-specific rules by clicking **Add Rule**:

* **Blocked Terms** — Block specific words or phrases
* **Custom Regex** — Match patterns with regular expressions
* **Topic Restriction** — Restrict content related to specific topics

Each custom rule can be individually enabled/disabled or deleted.

Learn more about guardrails in the [Guardrails feature docs](https://docs.offrail.ai/features/guardrails).


# Introduction
URL: https://docs.offrail.ai/learn


The OffRail dashboard gives you full control over your LLM API usage, costs, and configuration. This section walks you through every page in the platform, grouped by product.

## AI Gateway [#ai-gateway]

Configure how requests flow through the gateway — keys, safety rules, and programmatic provisioning:

* [**API Keys**](https://docs.offrail.ai/learn/api-keys) — Create and manage your API keys, plus per-key statistics
* [**Provider Keys**](https://docs.offrail.ai/learn/provider-keys) — Bring your own provider API keys
* [**Models**](https://docs.offrail.ai/learn/models) — Browse catalog and custom models with compliance eligibility
* [**Preferences**](https://docs.offrail.ai/learn/preferences) — Project-level settings like caching and mode
* [**Guardrails**](https://docs.offrail.ai/learn/guardrails) — Content safety rules and filters
* [**Security Events**](https://docs.offrail.ai/learn/security-events) — Monitor guardrail violations
* [**Payments SDK**](https://docs.offrail.ai/learn/sdk-settings) — Embed end-user payments and credit purchases into your own app
* [**Dynamic Routes**](https://docs.offrail.ai/learn/dynamic-routes) — Named, versioned routing flows with conditions and A/B splits (Enterprise)
* [**Master Keys**](https://docs.offrail.ai/learn/master-keys) — Provision projects and API keys programmatically (Enterprise)

## Observability [#observability]

Monitor usage, spend, errors, and performance across every request:

* [**Dashboard**](https://docs.offrail.ai/learn/dashboard) — Overview of your usage, costs, and performance
* [**Activity**](https://docs.offrail.ai/learn/activity) — Detailed logs of every API request
* [**Agents Monitoring**](https://docs.offrail.ai/learn/agents) — Cost, tokens, and sessions per coding agent
* [**Model Usage**](https://docs.offrail.ai/learn/model-usage) — Usage breakdown by model
* [**Usage & Metrics**](https://docs.offrail.ai/learn/usage-metrics) — Requests, errors, cache rates, and cost trends
* [**Analytics**](https://docs.offrail.ai/learn/analytics) — Cost, requests, and tokens broken down by model
* [**Organization Analytics**](https://docs.offrail.ai/learn/org-analytics) — Cost and usage rolled up across every project
* [**Member Analytics**](https://docs.offrail.ai/learn/member-analytics) — Per-member cost and usage breakdowns (Enterprise)
* [**Audit Logs**](https://docs.offrail.ai/learn/audit-logs) — Complete history of organization actions

## Coding agents [#coding-agents]

Flat-price dev plans for AI coding tools:

* [**Coding Agents**](https://docs.offrail.ai/learn/coding-agents) — See which coding tools your API key is used with
* [**Model Categories & Fair Use**](https://docs.offrail.ai/learn/model-categories) — How models are categorized and premium fair-use caps

## Lounge [#lounge]

The members' lounge for AI — chat, media studios, and memberships:

* [**Lounge Chat**](https://docs.offrail.ai/learn/playground) — Chat with 200+ models in one interface
* [**Group Chat**](https://docs.offrail.ai/learn/playground-group) — Watch multiple models discuss and collaborate on your prompt
* [**Image Studio**](https://docs.offrail.ai/learn/playground-image) — Generate images using AI models
* [**Video Studio**](https://docs.offrail.ai/learn/playground-video) — Generate videos using AI models
* [**Audio Studio**](https://docs.offrail.ai/learn/playground-audio) — Generate speech from text using AI voices
* [**Voice Calls**](https://docs.offrail.ai/learn/playground-realtime) — Have a live speech-to-speech conversation with a model
* [**Lounge Memberships**](https://docs.offrail.ai/learn/chat-plans) — Monthly membership plans for the Lounge chat app

## Account & Billing [#account--billing]

Manage your organization, team, and payments:

* [**Billing**](https://docs.offrail.ai/learn/billing) — Credits, plans, and payment methods
* [**Transactions**](https://docs.offrail.ai/learn/transactions) — Payment and credit history
* [**Invoices**](https://docs.offrail.ai/learn/invoices) — Download invoices and credit notes
* [**Referrals**](https://docs.offrail.ai/learn/referrals) — Earn credits by referring others
* [**Policies**](https://docs.offrail.ai/learn/policies) — Data retention configuration
* [**Org Preferences**](https://docs.offrail.ai/learn/org-preferences) — Organization name and billing details
* [**Team**](https://docs.offrail.ai/learn/team) — Manage team members and roles


# Invoices
URL: https://docs.offrail.ai/learn/invoices


Every completed payment gets a PDF invoice, and every refund a credit note. The
invoice is emailed to your billing address when the payment succeeds, and both
can be downloaded at any time from [**Transactions**](https://docs.offrail.ai/learn/transactions) in
your organization settings.

## Downloading an invoice [#downloading-an-invoice]

1. Open **Settings → Transactions** in the dashboard.
2. Find the payment in the list.
3. Click **Invoice** on that row — the PDF downloads as
   `invoice-<transaction-id>.pdf`.

Refunds produce a **Credit note** instead of an invoice, downloaded the same
way. It shows the original purchase amount, the percentage refunded, and the
refunded total as a negative line.

<Callout type="info">
  Only **completed** payments with a positive amount have a document. Plan
  cancellations, plan endings, and free credit gifts have nothing to invoice, so
  those rows have no download button.
</Callout>

The invoice number is the transaction ID, so it matches the row you downloaded
it from.

## Putting company details on the invoice [#putting-company-details-on-the-invoice]

Company name, address, tax ID and invoice notes come from your organization's
billing information, under **Settings → Billing**:

| Field                   | Appears on the invoice as                      |
| ----------------------- | ---------------------------------------------- |
| **Email Address**       | The billing email address invoices are sent to |
| **Company Name**        | The billed company                             |
| **Billing Address**     | The billing address block                      |
| **Tax ID / VAT Number** | Your tax identification number                 |
| **Invoice Notes**       | A free-text footer (e.g. a PO number)          |

<Callout type="warning">
  Fill these in **before** you pay. The emailed invoice is generated at payment
  time and keeps the details as they were then, while a download from
  Transactions is rendered fresh from your current details — so the two copies
  of one invoice number can differ after you edit this page. Treat the emailed
  PDF as the record of what was invoiced, and for a lasting change get the
  details right before the next payment.
</Callout>

## Related pages [#related-pages]

* [**Transactions**](https://docs.offrail.ai/learn/transactions) — the full payment and credit history
* [**Billing**](https://docs.offrail.ai/learn/billing) — credits, plans, and payment methods


# Master Keys
URL: https://docs.offrail.ai/learn/master-keys


The Master Keys page lets you create and manage org-scoped bearer tokens for the `/v1/master/*` API — so your own backend can provision projects, gateway API keys, and IAM rules programmatically, without going through the dashboard.

<Callout type="info">
  Master Keys are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise). Contact us at [contact@offrail.ai](mailto:contact@offrail.ai) to
  enable them for your organization.
</Callout>

## Creating a Master Key [#creating-a-master-key]

Click **Create Master Key**, give the key a descriptive name (for example, "Provisioning backend"), and click create.

<Callout type="warning">
  The plain token (prefixed `orlmk_`) is shown **only once** at creation time.
  Copy it immediately and store it securely — only an HMAC-SHA256 hash is kept
  in the database.
</Callout>

## Key List [#key-list]

Each master key in the list shows:

| Field          | Description                                            |
| -------------- | ------------------------------------------------------ |
| **Name**       | The description you gave the key                       |
| **Master Key** | Masked form of the token, for identification           |
| **Status**     | Active or inactive                                     |
| **Created**    | When the key was created                               |
| **Created By** | The team member who created it                         |
| **Last Used**  | Timestamp of the key's most recent `/v1/master/*` call |

## Managing Keys [#managing-keys]

Use the actions menu on each row to:

* **Activate / Deactivate** — Temporarily suspend a key without deleting it. Inactive keys receive a 401 on every request.
* **Delete** — Permanently revoke the key. Any system using it immediately loses access to the `/v1/master/*` API.

Every create, delete, and status change is recorded in your organization's [audit log](https://docs.offrail.ai/learn/audit-logs).

## Limits [#limits]

Each organization can have up to **10 active master keys**. A usage bar above the list shows how many you've used; contact us if you need more.

## Using a Master Key [#using-a-master-key]

Master keys authenticate against the programmatic provisioning API:

```bash
curl https://internal.offrail.ai/v1/master/projects \
  -H "Authorization: Bearer orlmk_..."
```

See the [Master Keys feature documentation](https://docs.offrail.ai/features/master-keys) for the full endpoint reference, including project management, gateway API key provisioning, per-key IAM rules, and [member-level IAM rules](https://docs.offrail.ai/features/master-keys#member-iam-rules) (addressable by membership id or email).


# Member Analytics
URL: https://docs.offrail.ai/learn/member-analytics


Member Analytics breaks your organization's usage down by person, so you can see who is spending what across every project. It lives on the **Team** page and is available on the **Enterprise** plan.

<Callout type="info">
  Member analytics require the **Enterprise plan** and an organization **owner**
  or **admin** role. Non-enterprise organizations see an upgrade card, and
  members without admin access see an access notice instead of the data.
</Callout>

## Members Table [#members-table]

The Team page adds a usage table sorted by spend, so the heaviest users surface first. Each row shows that member's totals for the selected date range:

| Column         | Description                                  |
| -------------- | -------------------------------------------- |
| **Member**     | Name and email of the team member            |
| **Cost**       | Total spend attributed to the member, in USD |
| **Tokens**     | Total tokens (input + output)                |
| **Requests**   | Number of requests                           |
| **Error rate** | Share of the member's requests that failed   |
| **API keys**   | How many API keys the member created         |

Usage is attributed by **who created each API key** — spend lands on the member who owns the key that made the request, which is the only link between usage and a user.

## Member Detail [#member-detail]

Click a member to open their detail page, scoped to the same date range:

The detail view includes:

* **Summary cards** — the member's cost, tokens, requests, and error rate for the range.
* **Most used** — their top model, provider, and app.
* **Cost by model** — the same breakdown as the project [Analytics](https://docs.offrail.ai/learn/analytics) page, scoped to this member.
* **Top providers and top apps** — tables ranking where the member's traffic goes.

This makes it easy to attribute cost to a team, investigate a spike, or confirm a member is using the models and providers you expect.


# Model Categories
URL: https://docs.offrail.ai/learn/model-categories


Every model in the gateway is sorted into a category. Categories power dashboard filtering and analytics.

## Categories [#categories]

| Category     | Description                                                                                                                  |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| **Premium**  | High-cost frontier / flagship models — priced at **$15+ per million output tokens*&#x2A; or **$5+ per million input tokens** |
| **Standard** | Every other model — the broad catalog of fast, cost-effective everyday models                                                |

The classification is computed automatically from catalogue prices — there is no hand-picked list. To see exactly which models are premium right now, check the live [**Premium Models**](https://offrail.ai/models/premium) page, or browse the full catalog on the [**Supported Models**](https://offrail.ai/models) page and filter by use case, capabilities, provider, price, and context size.


# Model Usage
URL: https://docs.offrail.ai/learn/model-usage


The Model Usage page shows how your API requests are distributed across different LLM models over time.

## Filters [#filters]

Two filters let you narrow down the data:

* **API Key** — Select a specific API key or view usage across all keys
* **Date range** — Choose a time period to analyze

## Usage Chart [#usage-chart]

The main chart displays a time-series breakdown of requests per model. Each model is represented by a different color, making it easy to see:

* Which models are used most frequently
* How usage patterns change over time
* Whether usage is concentrated on a single model or spread across many

This page is useful for understanding your model distribution and identifying opportunities to optimize costs by switching to more cost-effective models for certain workloads.


# Models
URL: https://docs.offrail.ai/learn/models


The Models page shows every model available to your organization in one directory: the full OffRail catalog plus any models defined for your own [custom providers](https://docs.offrail.ai/features/custom-providers). When your organization enforces a [provider compliance policy](https://docs.offrail.ai/features/compliance), the page also shows exactly which models are eligible to route to — and why the rest are blocked.

Every organization member can browse this page, including project-scoped `developer` members, so your whole team can see which providers and models are available without needing admin access. Management actions are only visible to owners and admins.

## The directory [#the-directory]

The directory is the same table you know from the public [models page](https://offrail.ai/models), with one row per provider mapping:

| Column                 | Description                                                             |
| ---------------------- | ----------------------------------------------------------------------- |
| **Provider**           | The provider serving the model; custom providers carry a `Custom` badge |
| **Model ID**           | The model id; the copy button copies the exact string to request        |
| **Input / Output $/M** | Prices per million tokens (per-request/per-character where applicable)  |
| **Cache Read $/M**     | Cached input price where supported                                      |
| **Features**           | Capability icons: streaming, vision, tools, reasoning, JSON output, …   |

Search matches model names, ids, aliases, and providers. The **Filters** panel offers the same controls as the public directory — use case, capabilities, provider, price and context ranges — plus two controls specific to this page:

* **Source** — show **Catalog & custom** (default), only **Catalog models**, or only **Custom models**. The selector appears once your organization has custom models.
* **Eligible only** — hide models blocked by your organization's compliance policy.

## Compliance eligibility [#compliance-eligibility]

When your organization is on the Enterprise plan and its provider compliance policy is enabled, every row is evaluated with the same fail-closed rules the gateway enforces at request time — certifications (SOC 2, ISO 27001, GDPR), data-policy requirements (no training on prompts, no prompt logging), country restrictions, and provider/model allow and deny lists.

Rows that fail render greyed out with a red ban icon; hover it to see the exact reasons, such as "No SOC 2 Type 2 report" or "Not on the allowed-providers list". Custom providers are evaluated against their [compliance attestation](https://docs.offrail.ai/features/custom-providers#compliance-attestation) — without an attestation on file, all requests through them are blocked while a policy is active.

<Callout type="info">
  Eligibility marking requires an active compliance policy, available on the
  [**Enterprise plan**](https://offrail.ai/enterprise). Without one, the
  directory simply lists all models. Configure the policy on the
  [Compliance](https://docs.offrail.ai/features/compliance) page.
</Callout>

## Custom models [#custom-models]

Models from your organization's custom-model catalog appear in the same table, addressed as `<provider>/<model>` (e.g. `internal-vllm/llama-4-maverick`) — the exact model string to send to the gateway. Their pricing, context size, and capability flags come from the catalog entries you define.

Owners and admins additionally see the **Custom model catalog** section below the directory:

* **Custom provider** — pick which custom provider key's catalog to manage
* **Only allow catalog models** — when on, requests through the provider are limited to the models defined in its catalog so cost and context limits are always enforced
* **Compliance attestation** — record the compliance posture of the infrastructure behind the provider; the policy evaluates it with the same fail-closed rules as catalogue providers
* **Add / edit / delete custom models** — define per-model pricing, context and output limits, and capabilities

<Callout type="info">
  The custom model catalog requires the [**Enterprise
  plan**](https://offrail.ai/enterprise). Custom provider keys themselves are
  created on the [Provider Keys](https://docs.offrail.ai/learn/provider-keys) page.
</Callout>


# Organization Analytics
URL: https://docs.offrail.ai/learn/org-analytics


Organization Analytics rolls usage up across **every project in your organization** into one view. Where the project [Analytics](https://docs.offrail.ai/learn/analytics) page answers "where does this project's spend go?", this page answers it for the whole org — and lets you pivot the breakdown by model, **project**, or API key.

<Callout type="info">
  Organization Analytics is an **Enterprise** feature, available to organization
  **owners and admins**. On lower plans the page shows an upgrade prompt
  instead, and Enterprise-only items are flagged in the sidebar.
</Callout>

Open it from the **Analytics** item under **Organization** in the sidebar. Like the rest of the dashboard, it respects the shared date-range picker, so every chart and stat reflects the same window.

## Summary [#summary]

Three cards at the top total the selected range across the whole organization:

| Card            | What it totals                         |
| --------------- | -------------------------------------- |
| **Total spend** | Combined cost of every project, in USD |
| **Requests**    | Total requests routed across the org   |
| **Tokens**      | Total tokens (input + output)          |

## Breakdown [#breakdown]

A single **group-by** control switches what the two charts below break the usage down by:

| Group by    | What each series represents                                                     |
| ----------- | ------------------------------------------------------------------------------- |
| **Model**   | Canonical model, collapsed across providers (Qwen via any provider is one line) |
| **Project** | One project in the organization                                                 |
| **API key** | One API key (by its description)                                                |

For each grouping you get the same two charts:

* **Over time** — a stacked area chart of the top series across the date range, with **Cost / Requests / Tokens** tabs.
* **Ranking** — a horizontal bar chart of the top series for the range, with the same metric tabs and the running totals for the window.

Switch to **Project** to see which teams or workloads drive the bill, **Model** for the org-wide model footprint, or **API key** when usage runs through services rather than people.

## How the data is computed [#how-the-data-is-computed]

Both charts read from the same pre-aggregated hourly rollups the rest of the dashboard uses — there is no separate analytics pipeline and no scan over raw request logs, so the page stays fast over any range. Aggregation happens per time bucket, so totals line up with the project [Analytics](https://docs.offrail.ai/learn/analytics) and [Usage & Metrics](https://docs.offrail.ai/learn/usage-metrics) pages for the same window.

For a single project's breakdown, see [Analytics](https://docs.offrail.ai/learn/analytics). For a per-person view of org spend, see [Member Analytics](https://docs.offrail.ai/learn/member-analytics).


# Org Preferences
URL: https://docs.offrail.ai/learn/org-preferences


The Org Preferences page contains settings for your organization's identity and billing information.

## Organization Name [#organization-name]

Update your organization's display name. This name appears throughout the dashboard and in billing communications.

## Billing Email [#billing-email]

Set or update the email address used for billing-related communications, including receipts, invoices, and payment notifications.

## Billing Information [#billing-information]

Configure your organization's billing details for invoices:

| Field                              | Description                                                              |
| ---------------------------------- | ------------------------------------------------------------------------ |
| **Email Address**                  | Primary email for billing communications                                 |
| **Company Name** (optional)        | Your company or organization name for invoices                           |
| **Billing Address**                | Street address, city, state/province, ZIP code, and country              |
| **Tax ID / VAT Number** (optional) | Your tax identification or VAT number for tax-compliant invoices         |
| **Invoice Notes** (optional)       | Custom notes to include on invoices (e.g., PO numbers, department codes) |


# Audio Studio
URL: https://docs.offrail.ai/learn/playground-audio


The Audio Studio turns text into speech using text-to-speech models from ElevenLabs, OpenAI, Gemini, and Qwen. Pick a model and a voice, type your text, and play or download the result.

## Model Selection [#model-selection]

Choose from the supported text-to-speech models in the dropdown. Each model has its own voice roster, output formats, and pricing — the newest models appear first.

## Generating Audio [#generating-audio]

1. Select a text-to-speech model
2. Pick a voice for the selected model
3. Type the text you want spoken — or click one of the sample prompts on the empty state
4. Click **Generate**
5. Generated clips appear in the gallery, where you can play or download them

## Voices [#voices]

Each provider ships its own voice catalog — for example OpenAI's `alloy`, `nova`, and `onyx`, Gemini's `Kore`, `Puck`, and `Zephyr`, or ElevenLabs voices like `Sarah`, `George`, and `Charlotte`. The voice picker only shows voices valid for the selected model.

## Comparison Mode [#comparison-mode]

Enable comparison mode to send the same text to multiple models at once and hear the results side by side — useful for choosing a voice and provider before wiring up the [Speech Generation API](https://docs.offrail.ai/features/speech-generation).

## History [#history]

Generated clips are saved to your audio history so you can revisit, replay, or delete them later.


# Group Chat
URL: https://docs.offrail.ai/learn/playground-group


The Group Chat page lets you add multiple AI models to a conversation where they discuss and build on each other's responses, creating a dynamic multi-model dialogue.

## How It Works [#how-it-works]

1. Add 2–5 different AI models to the conversation
2. Enter an initial prompt or question to kick off the discussion
3. Click **Start Conversation** to begin
4. Models take turns responding to each other in sequence
5. Each model builds on the previous responses, creating a dynamic conversation
6. You can stop the conversation at any time and start a new one

## Use Cases [#use-cases]

* **Model evaluation** — Compare how different models approach the same topic
* **Brainstorming** — Get diverse perspectives from multiple AI models
* **Debate** — Watch models discuss pros and cons of a topic
* **Research** — Gather multi-model analysis of complex questions


# Image Studio
URL: https://docs.offrail.ai/learn/playground-image


The Image Studio lets you generate images using AI models through an intuitive interface. Select a model, describe what you want, and get results instantly.

## Model Selection [#model-selection]

Choose from supported image generation models in the dropdown. Each model has different capabilities, resolutions, and pricing.

## Generating Images [#generating-images]

1. Select an image generation model
2. Type a description of the image you want
3. Click send to generate
4. Generated images appear in the conversation

## Image Count [#image-count]

You can generate 1, 2, or 4 images at once. Multiple images are displayed in a grid layout.

## Resolution Options [#resolution-options]

Available resolutions depend on the selected model. Common options include 1K, 2K, and 4K.


# Voice Calls
URL: https://docs.offrail.ai/learn/playground-realtime


Voice Calls turns Lounge into a phone line to a model. Pick a realtime model and a voice, start the call, and talk — the model listens, answers out loud, and the transcript builds as you go. Every call runs over the same [Realtime API](https://docs.offrail.ai/features/realtime) your own apps can use.

## Starting a Call [#starting-a-call]

1. Pick a model from the selector — only models with an active realtime mapping appear, newest first
2. Pick a **Voice** — the voice list comes from the selected model, so it changes when you switch models
3. Click **Start call** and allow microphone access when the browser asks

The status label on the right walks through the handshake — preparing audio, requesting the microphone, authorizing, connecting, configuring — and settles on **Live** with a call timer next to it. The model and voice are locked for the duration of the call; end it to change either.

Your browser never sees an API key. The page mints a short-lived client secret server-side for the call, and the audio itself is streamed as 24 kHz PCM with echo cancellation and noise suppression on.

## During the Call [#during-the-call]

The model uses semantic voice activity detection, so there is no push-to-talk: stop speaking and it answers, start speaking again and it stops to listen. An interrupted answer stays in the transcript marked `(interrupted)`.

| Control                  | What it does                                                |
| ------------------------ | ----------------------------------------------------------- |
| **Mute** / **Unmute**    | Stops sending your microphone audio without ending the call |
| **End call**             | Hangs up and saves the call to your history                 |
| Voice activity indicator | Shows who currently has the floor, and your mute state      |

Below the controls, a usage line totals the call: how many responses it took, input and output tokens, and how many of those were audio tokens.

<Callout type="info">
  Voice calls are billed to a pay-as-you-go organization with credits, so the
  organization switcher on this page only lists those — Lounge membership
  workspaces can't place calls. Without one selected, the page asks you to
  create one in the dashboard first.
</Callout>

## Call History [#call-history]

Ended calls are saved per organization and listed in the sidebar, grouped by date, with each call's duration and time. Click one to reopen its transcript, along with the model and voice it used and its token totals. Use the &#x2A;*⋮** menu on a call to rename or delete it, and **New Call** to get back to a fresh screen.

Calls are titled from your opening line, so history reads like a list of topics. A call you leave mid-conversation — by navigating away or closing the tab — is still saved with whatever was said up to that point.

## Using It From Your Own App [#using-it-from-your-own-app]

The same endpoint backs the [Realtime API](https://docs.offrail.ai/features/realtime): connect a WebSocket to `wss://api.offrail.ai/v1/realtime`, authenticate with an [API key](https://docs.offrail.ai/learn/api-keys), and use the standard OpenAI realtime event protocol. For browser clients, mint an ephemeral client secret from your backend instead of shipping a key:

```bash
curl -X POST "https://api.offrail.ai/v1/realtime/client_secrets" \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "expires_after": { "anchor": "created_at", "seconds": 120 },
    "session": { "type": "realtime", "model": "gpt-realtime" }
  }'
```

Calls made here appear in your [activity feed](https://docs.offrail.ai/learn/activity) and [analytics](https://docs.offrail.ai/learn/analytics) like any other request, with audio input and output costs broken out.


# Video Studio
URL: https://docs.offrail.ai/learn/playground-video


The Video Studio lets you generate videos using AI models. Select a model, describe what you want, and get video results.

## Model Selection [#model-selection]

Choose from supported video generation models in the dropdown. Each model has different capabilities, resolutions, and pricing.

## Generating Videos [#generating-videos]

1. Select a video generation model
2. Type a description of the video you want
3. Click send to generate
4. Generated videos appear in the conversation

## Resolution Options [#resolution-options]

Available resolutions depend on the selected model.


# Lounge Chat
URL: https://docs.offrail.ai/learn/playground


Lounge is a standalone chat app for talking to any LLM through a conversational interface. You can select any supported model, adjust parameters, and see responses in real time.

## Model Selection [#model-selection]

Use the dropdown at the top to pick a model and provider. The **Auto Route** option automatically selects the best provider based on availability and cost.

## Chat Interface [#chat-interface]

* Type your message in the input field at the bottom
* Click the send button or press Enter to submit
* Responses stream in real time
* Previous conversations appear in the sidebar

## Prompt Suggestions [#prompt-suggestions]

When starting a new chat, category tabs help you pick a prompt:

* **Create** — Content generation prompts
* **Explore** — Research and analysis prompts
* **Code** — Programming and development prompts
* **Image gen** — Image generation prompts

## Sidebar [#sidebar]

The left sidebar shows your chat history. Click **+ New Chat** to start a fresh conversation, or select a previous chat to continue it.

## Comparison Mode [#comparison-mode]

Toggle **Comparison mode** in the top-right to send the same prompt to multiple models side by side. See the [Group Chat](https://docs.offrail.ai/learn/playground-group) page for details.

## Image Studio [#image-studio]

Click **Image Studio** in the sidebar to switch to the image generation interface. See the [Image Studio](https://docs.offrail.ai/learn/playground-image) page for details.


# Policies
URL: https://docs.offrail.ai/learn/policies


The Policies page lets you configure organization-wide policies that govern how your data is handled.

## Data Retention [#data-retention]

Control how long your request logs and activity data are stored. The retention period depends on your plan:

| Plan           | Retention Period |
| -------------- | ---------------- |
| **Free**       | 30 days          |
| **Enterprise** | Custom           |

After the retention period expires, request logs and associated data are automatically deleted.

The retention level itself is configurable on standard pay-as-you-go organizations only. Chat subscriptions are always metadata only — their request and response payloads are not retained, aside from Responses API stored responses, which are kept for 30 days independently of the retention level.

Learn more about data retention in the [Data Retention feature docs](https://docs.offrail.ai/features/data-retention).


# Preferences
URL: https://docs.offrail.ai/learn/preferences


The Preferences page contains project-level settings that control how your project behaves.

## Project Name [#project-name]

Update the display name for your project. This name appears in the sidebar and throughout the dashboard.

## Project Mode [#project-mode]

Configure how your organization handles projects. This setting determines the routing and isolation behavior for API requests within the project.

## Caching [#caching]

Enable or configure response caching for API requests. When enabled, identical requests will return cached responses instead of making new calls to the provider, saving both time and cost.

Learn more about caching in the [Caching feature docs](https://docs.offrail.ai/features/caching).

## Danger Zone [#danger-zone]

The Danger Zone section contains irreversible actions:

* **Archive Project** — Permanently archive the project. This action cannot be undone. Archived projects stop processing requests and their API keys become inactive.


# Provider Keys
URL: https://docs.offrail.ai/learn/provider-keys


The Provider Keys page lets you add your own API keys from LLM providers (OpenAI, Anthropic, Google, etc.) to route requests directly through your accounts without additional gateway fees. The only optional charge is [data storage](https://docs.offrail.ai/features/data-retention#storage-pricing): if your organization enables full data retention, stored requests are billed at $0.01 per 1M tokens from your credits.

## Adding a Provider Key [#adding-a-provider-key]

Click **Add Provider Key** to configure a new key:

* **Provider** — Select which provider this key belongs to
* **Custom name** — An optional label to identify the key
* **API key** — Your provider's API key
* **Base URL** — Optional custom endpoint (useful for Azure OpenAI or custom deployments)
* **Allowed models** — Optional allowlist restricting which models the key serves
* **Max spend** — Optional USD cap; the key is disabled once the spend attributed to it reaches the limit

## Provider Keys List [#provider-keys-list]

Each configured key shows:

| Field           | Description                                        |
| --------------- | -------------------------------------------------- |
| **Provider**    | The LLM provider (e.g., OpenAI, Anthropic)         |
| **Custom name** | Your label for the key                             |
| **Status**      | Active, inactive, or deleted                       |
| **Base URL**    | Custom endpoint if configured                      |
| **Token**       | Masked key with only the last 4 characters visible |

## Actions [#actions]

For each provider key:

* **Edit** — Update the key name, value, or base URL
* **Restrict models** — Choose which models the key may serve
* **Set spend limit** — Cap the spend attributed to the key
* **Deactivate** — Temporarily disable the key without deleting it
* **Delete** — Permanently remove the key

## Restricting a Key to Specific Models [#restricting-a-key-to-specific-models]

Provider accounts often only have some of a provider's models enabled. Set
**Allowed models** on a key to list exactly the models it may serve — routing
then skips that key for anything else instead of sending a request the upstream
account will reject. Leave the list empty to serve every model the provider
offers.

The restriction also decides how a key is checked when you save it: a restricted
key is validated against one of its own allowed models rather than the
provider's default validation model, so a key with partial model access still
validates.

<Callout type="info">
  In hybrid mode, a model excluded by the restriction falls back to your credits
  rather than failing, so restricting a key never takes a model away from your
  projects.
</Callout>

<Callout type="info">
  When you use your own provider keys, requests are routed directly to the
  provider. You pay the provider's standard rates with no additional gateway
  markup — the only OffRail charge is the optional [data
  storage](https://docs.offrail.ai/features/data-retention#storage-pricing) cost when full data
  retention is enabled.
</Callout>


# Referrals
URL: https://docs.offrail.ai/learn/referrals


The Referrals page lets you earn credits by inviting others to use OffRail.

## Eligibility [#eligibility]

To unlock the referral program, your organization must have at least **$100 in total credit top-ups**. Before reaching this threshold, the page shows:

* A progress bar showing your progress toward $100
* The remaining amount needed to unlock
* An explanation of the 1% earnings model

## Referral Dashboard [#referral-dashboard]

Once eligible, the page shows:

### Your Referral Link [#your-referral-link]

A unique shareable link tied to your organization. Click the copy button to copy it to your clipboard and share it with others.

### Your Stats [#your-stats]

| Stat               | Description                                           |
| ------------------ | ----------------------------------------------------- |
| **Users Referred** | Total number of users who signed up through your link |
| **Total Earnings** | Total credit amount earned from referrals             |

### How It Works [#how-it-works]

1. **Share Your Link** — Send your referral link to others
2. **They Sign Up** — They create an OffRail account using your link
3. **Earn Credits** — You earn 1% of their spending as credits

Credits are automatically added to your organization balance.


# Payments SDK
URL: https://docs.offrail.ai/learn/sdk-settings


The **Payments SDK** settings page lets you embed end-user payments and sessions into your own application — your end users get their own wallets, and you control markup and access. It is a payments feature, not a normal AI client SDK like the OpenAI SDK. You'll find it under **Settings → Payments SDK** for a project.

<Callout type="warn">
  **Preview — opt-in only.** Embeddable Payments is in preview and enabled on an
  opt-in basis per project. Until it is turned on for your project, this page is
  read-only and shows a preview of the settings. [Contact
  us](mailto:contact@offrail.ai) to request access.
</Callout>

## End-user sessions [#end-user-sessions]

Turn on **Enable end-user sessions** to allow this project to mint short-lived browser session tokens for your users.

| Field               | Description                                                                                        |
| ------------------- | -------------------------------------------------------------------------------------------------- |
| **Markup percent**  | The percentage you add on top of provider cost for each end-user request (0–100%)                  |
| **Allowed origins** | The browser origins permitted to use session tokens, one per line (e.g. `https://app.example.com`) |

Click **Save Settings** to apply changes.

## Platform secret keys [#platform-secret-keys]

Platform secret keys are **server-side** keys used to mint end-user sessions. Keep them on your backend — never expose them in the browser.

* **Create Live Key** — A production key. Top-ups made with it use live billing.
* **Create Test Key** — A sandbox key. Top-ups use the Stripe sandbox, so you can build and test without real charges.

<Callout type="warn">
  A secret key is shown **only once** at creation time. Copy it immediately — it
  won't be displayed again. If you lose a key, revoke it and create a new one.
</Callout>

Each key in the list shows its description, a **test** badge when applicable, its status, and a masked token. Use **Revoke** to permanently disable a key.


# Security Events
URL: https://docs.offrail.ai/learn/security-events


The Security Events page shows all guardrail violations detected across your organization, helping you monitor content safety and policy enforcement. Violations from projects that [override the organization guardrails](https://docs.offrail.ai/learn/guardrails) appear here too, so this page stays the single place to review enforcement.

<Callout type="info">
  Security Events are available on the [**Enterprise
  plan**](https://offrail.ai/enterprise). Owner or Admin role is required.
</Callout>

## Stats Cards [#stats-cards]

Four summary cards at the top:

| Card                 | Description                                   |
| -------------------- | --------------------------------------------- |
| **Total Violations** | All-time violation count                      |
| **Last 24 Hours**    | Violations in the past day                    |
| **Blocked**          | Number of requests that were blocked          |
| **Redacted**         | Number of requests where content was redacted |

## Filters [#filters]

Narrow down the events list:

* **Action** — Filter by Blocked, Redacted, Warned, or All actions
* **Category** — Filter by Prompt Injection, Jailbreak, PII Detection, Secrets, Blocked Terms, Custom Regex, or Topic Restriction

## Violations List [#violations-list]

Each violation entry shows:

| Field               | Description                                          |
| ------------------- | ---------------------------------------------------- |
| **Timestamp**       | When the violation occurred                          |
| **Rule name**       | Which guardrail rule was triggered                   |
| **Category**        | The type of violation (shown as a badge)             |
| **Action**          | What action was taken (Blocked, Redacted, or Warned) |
| **Matched pattern** | The content that triggered the rule                  |

The list supports pagination with a **Load More** button for viewing older events.


# Structured outputs
URL: https://docs.offrail.ai/learn/structured-outputs


OffRail exposes **two distinct JSON capabilities**, and they are kept
separate on every surface: the models API, the [models
directory](https://offrail.ai/models), the model detail pages, and the
gateway's request validation.

## Soft JSON output (`json_output`) [#soft-json-output-json_output]

A model with soft JSON output can be **nudged** into emitting JSON — typically
via `response_format: { "type": "json_object" }` or prompt guidance. There is
no server-side schema guarantee: the model may still produce off-schema or
malformed JSON, so the consumer's parser must be able to handle it.

## Strict JSON output schema (`structured_outputs`) [#strict-json-output-schema-structured_outputs]

A model with strict JSON output schema is served by an **upstream provider
that natively enforces schema-guided decoding** (for example OpenAI structured
outputs or vLLM guided decoding). You provide a JSON schema via
`response_format: { "type": "json_schema", "json_schema": { ... } }` and the
provider guarantees the output conforms to it.

<Callout type="info">
  The gateway only **declares** this capability — it never emulates schema
  enforcement (no prompt-and-validate adapter). A model whose provider does not
  natively support `json_schema` rejects such requests with `400 does not
  	support JSON schema output mode`. That rejection is **by design**, not a
  missing feature.
</Callout>

## Field mapping [#field-mapping]

| Models API field     | Catalogue flag     | Meaning                                       |
| -------------------- | ------------------ | --------------------------------------------- |
| `json_output`        | `jsonOutput`       | Soft: nudged JSON, no schema guarantee        |
| `structured_outputs` | `jsonOutputSchema` | Strict: upstream provider enforces the schema |

The snake\_case names are what the OpenAI-compatible [`/v1/models`](https://docs.offrail.ai/learn/models) endpoint exposes;
the camelCase names are used in the model catalogue and the directory UI.

The two tiers are **independent per provider mapping**: a provider may support
strict `json_schema` without soft `json_object` (for example Runware or
Perplexity declare `structured_outputs` without `json_output`), or soft
`json_object` without strict schema. The gateway routes each mode by its own
flag — `json_schema` requests only require `structured_outputs`, they are never
emulated or silently downgraded.

## How to verify a model [#how-to-verify-a-model]

Probe a model with a strict `json_schema` request:

```bash
curl -s https://api.offrail.ai/v1/chat/completions \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Reply JSON: {\"ok\":true}" }],
    "max_tokens": 16,
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "probe",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": { "ok": { "type": "boolean" } },
          "required": ["ok"],
          "additionalProperties": false
        }
      }
    }
  }'
```

* `200` → the upstream provider enforces the schema; the model should declare `structured_outputs: true`.
* `400 does not support JSON schema output mode` → soft-only; the model should expose `json_output` without `structured_outputs`.

The probe above can be run for any model listed in `GET /v1/models`; a
declared flag that does not match the observed status is a catalog bug and
should be reported against the repository.


# Team
URL: https://docs.offrail.ai/learn/team


The Team page lets you invite team members, assign roles, and control access to your organization.

## Adding Members [#adding-members]

Click **Add Member** to invite someone by email. You'll need to:

1. Enter their email address
2. Select a role (Developer, Admin, or Owner)

Your plan includes up to **5 team seats**. The current count is displayed, and the Add button is disabled when all seats are used. Contact sales for additional seats.

## Team Members List [#team-members-list]

Each member shows:

| Field     | Description                                      |
| --------- | ------------------------------------------------ |
| **Name**  | The member's display name                        |
| **Email** | Their email address                              |
| **Role**  | Their current role (can be changed via dropdown) |

## Actions [#actions]

Each member row has an actions menu:

* **Details** — Open the member's detail page with their budget and usage
* **Manage access** — Change the member's role (owners and admins only)
* **Manage budget** — Set per-member spending limits (owners and admins only)
* **Manage IAM rules** — Restrict which models, providers, pricing tiers, or IP ranges the member can use (owners and admins only)
* **Remove** — Remove a member from the organization (requires confirmation)

## Member IAM Rules [#member-iam-rules]

Owners and admins can set IAM rules on a member — the same allow/deny rules for models, providers, pricing, and IP ranges that exist on individual API keys, applied organization-wide to that member. As with roles and budgets, admins cannot modify an owner's rules.

Member-level rules are a **ceiling**: they apply to every API key the member creates, and the member's own key rules can only narrow access further, never expand it. For example, if you restrict a member to a single approved [provider](https://offrail.ai/providers), they can still create a key limited to one of that provider's models, but no key of theirs can reach any other provider.

Members can view (but not edit) their own rules — the per-key IAM page shows a notice when organization-level restrictions apply, and denied requests state that the restriction was set by the org admin.

For the full rule semantics, see [Member-Level IAM Rules](https://docs.offrail.ai/features/api-keys#member-level-iam-rules). Rules can also be managed programmatically via the [master key API](https://docs.offrail.ai/features/master-keys#member-iam-rules).

## Role Permissions [#role-permissions]

| Role          | Permissions                                                                                           |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| **Owner**     | Full access to all settings, billing, team management, and all projects                               |
| **Admin**     | Can manage team members, projects, and API keys, but cannot access billing or delete the organization |
| **Developer** | View and use resources only. Cannot modify settings or manage team                                    |

Developers can also be given **restricted access** at the API key level, limiting which keys they can view and use.


# Transactions
URL: https://docs.offrail.ai/learn/transactions


The Transactions page shows a complete history of all financial transactions in your organization.

## Transaction History [#transaction-history]

Each transaction entry includes:

| Field           | Description                              |
| --------------- | ---------------------------------------- |
| **Date**        | When the transaction occurred            |
| **Type**        | The transaction type (see below)         |
| **Credits**     | Number of credits added or deducted      |
| **Total Paid**  | The dollar amount charged                |
| **Status**      | Current state of the transaction         |
| **Description** | Additional details about the transaction |

## Transaction Types [#transaction-types]

| Type                    | Description                         |
| ----------------------- | ----------------------------------- |
| **Credit Top-up**       | Manual or automatic credit purchase |
| **Credit Refund**       | Credits refunded to your account    |
| **Subscription Start**  | New plan subscription started       |
| **Subscription Cancel** | Plan subscription canceled          |
| **Subscription End**    | Plan subscription period ended      |

## Status Badges [#status-badges]

* **Completed** — Transaction processed successfully
* **Pending** — Transaction is being processed
* **Failed** — Transaction could not be completed


# Usage & Metrics
URL: https://docs.offrail.ai/learn/usage-metrics


The Usage & Metrics page provides comprehensive analytics through five tabs, giving you deep insight into your LLM API usage patterns.

## Filters [#filters]

* **API Key** — Filter metrics by a specific API key or view all
* **Date range** — Select the time period (defaults to last 7 days)

## Tabs [#tabs]

### Requests [#requests]

A time-series chart showing request volume over the selected period. Use this to identify traffic patterns, peak usage times, and growth trends.

### Models [#models]

A table showing your top-used models ranked by request count. For each model you can see:

* Total requests
* Token consumption
* Associated costs

This helps you understand which models drive the most usage and cost.

### Errors [#errors]

A chart showing error rates over time. Track:

* Error frequency and trends
* Spikes that may indicate provider issues
* Overall reliability of your API calls

### Cache [#cache]

A chart showing your cache hit rate over time. Monitor:

* How effectively caching is reducing redundant requests
* Cache hit vs. miss ratios
* The cost savings from cached responses

### Costs [#costs]

A cost breakdown chart showing spending patterns. Analyze:

* Cost trends over time
* Cost distribution by provider or model
* Opportunities to reduce spending


# Migrate from LiteLLM
URL: https://docs.offrail.ai/migrations/litellm


<section>
  Running your own LiteLLM proxy works—until it doesn't. Scaling, monitoring, and keeping it running becomes another job. OffRail gives you the same unified API with built-in analytics, caching, and a dashboard—without the infrastructure overhead.

  ## Quick Migration [#quick-migration]

  Both services use OpenAI-compatible endpoints, so migration is a two-line change:

  ```diff
  - const baseURL = "http://localhost:4000/v1";  // LiteLLM proxy
  + const baseURL = "https://api.offrail.ai/v1";

  - const apiKey = process.env.LITELLM_API_KEY;
  + const apiKey = process.env.OFFRAIL_API_KEY;
  ```
</section>

<section>
  ## Why Teams Switch to OffRail [#why-teams-switch-to-offrail]

  | What You Get             | LiteLLM (Self-Hosted) | OffRail                |
  | ------------------------ | --------------------- | ---------------------- |
  | OpenAI-compatible API    | Yes                   | Yes                    |
  | Infrastructure to manage | Yes (you run it)      | No (we run it)         |
  | Managed cloud option     | No                    | Yes                    |
  | Analytics dashboard      | Basic                 | Per-request detail     |
  | Response caching         | Manual setup          | Built-in, automatic    |
  | Cost tracking            | Via callbacks         | Native, real-time      |
  | Provider key management  | Config file           | Web UI with rotation   |
  | Uptime & scaling         | You handle it         | 99.9% SLA (Enterprise) |

  For a detailed breakdown, see [OffRail vs LiteLLM](https://offrail.ai/compare/litellm).
</section>

<section>
  ## Migration Steps [#migration-steps]

  <Steps>
    <Step>
      ### Get Your OffRail API Key [#get-your-offrail-api-key]

      Sign up at [offrail.ai/signup](https://offrail.ai/signup) and create an API key from your dashboard.
    </Step>

    <Step>
      ### Map Your Models [#map-your-models]

      OffRail supports two model ID formats:

      **Canonical Model IDs** (without provider prefix) - Uses smart routing to automatically select the best provider based on uptime, throughput, price, and latency:

      ```
      gpt-5.2
      claude-opus-4-5-20251101
      gemini-3-flash-preview
      ```

      **Provider-Prefixed Model IDs** - Routes to a specific provider with automatic failover if uptime drops below 90%:

      ```
      openai/gpt-5.2
      anthropic/claude-opus-4-5-20251101
      google-ai-studio/gemini-3-flash-preview
      ```

      This means many LiteLLM model names work directly with OffRail:

      | LiteLLM Model                    | OffRail Model                                                     |
      | -------------------------------- | ----------------------------------------------------------------- |
      | gpt-5.2                          | gpt-5.2 or openai/gpt-5.2                                         |
      | claude-opus-4-5-20251101         | claude-opus-4-5-20251101 or anthropic/claude-opus-4-5-20251101    |
      | gemini/gemini-3-flash-preview    | gemini-3-flash-preview or google-ai-studio/gemini-3-flash-preview |
      | bedrock/claude-opus-4-5-20251101 | claude-opus-4-5-20251101 or aws-bedrock/claude-opus-4-5-20251101  |

      For more details on routing behavior, see the [routing documentation](https://docs.offrail.ai/features/routing).
    </Step>

    <Step>
      ### Update Your Code [#update-your-code]

      #### Python with OpenAI SDK [#python-with-openai-sdk]

      ```python
      from openai import OpenAI

      # Before (LiteLLM proxy)
      client = OpenAI(
          base_url="http://localhost:4000/v1",
          api_key=os.environ["LITELLM_API_KEY"]
      )

      response = client.chat.completions.create(
          model="gpt-4",
          messages=[{"role": "user", "content": "Hello!"}]
      )

      # After (OffRail) - model name can stay the same!
      client = OpenAI(
          base_url="https://api.offrail.ai/v1",
          api_key=os.environ["OFFRAIL_API_KEY"]
      )

      response = client.chat.completions.create(
          model="gpt-4",  # or "openai/gpt-4" to target a specific provider
          messages=[{"role": "user", "content": "Hello!"}]
      )
      ```

      #### Python with LiteLLM Library [#python-with-litellm-library]

      If you're using the LiteLLM library directly, you can point it to OffRail:

      ```python
      import litellm

      # Before (direct LiteLLM)
      response = litellm.completion(
          model="gpt-4",
          messages=[{"role": "user", "content": "Hello!"}]
      )

      # After (via OffRail) - same model name works
      response = litellm.completion(
          model="gpt-4",  # or "openai/gpt-4" to target a specific provider
          messages=[{"role": "user", "content": "Hello!"}],
          api_base="https://api.offrail.ai/v1",
          api_key=os.environ["OFFRAIL_API_KEY"]
      )
      ```

      #### TypeScript/JavaScript [#typescriptjavascript]

      ```typescript
      import OpenAI from "openai";

      // Before (LiteLLM proxy)
      const client = new OpenAI({
      	baseURL: "http://localhost:4000/v1",
      	apiKey: process.env.LITELLM_API_KEY,
      });

      // After (OffRail) - same model name works
      const client = new OpenAI({
      	baseURL: "https://api.offrail.ai/v1",
      	apiKey: process.env.OFFRAIL_API_KEY,
      });

      const completion = await client.chat.completions.create({
      	model: "gpt-4", // or "openai/gpt-4" to target a specific provider
      	messages: [{ role: "user", content: "Hello!" }],
      });
      ```

      #### cURL [#curl]

      ```bash
      # Before (LiteLLM proxy)
      curl http://localhost:4000/v1/chat/completions \
        -H "Authorization: Bearer $LITELLM_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "gpt-4",
          "messages": [{"role": "user", "content": "Hello!"}]
        }'

      # After (OffRail) - same model name works
      curl https://api.offrail.ai/v1/chat/completions \
        -H "Authorization: Bearer $OFFRAIL_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "gpt-4",
          "messages": [{"role": "user", "content": "Hello!"}]
        }'
      # Use "openai/gpt-4" to target a specific provider
      ```
    </Step>

    <Step>
      ### Migrate Configuration [#migrate-configuration]

      #### LiteLLM Config (Before) [#litellm-config-before]

      ```yaml
      # litellm_config.yaml
      model_list:
        - model_name: gpt-4
          litellm_params:
            model: gpt-4
            api_key: sk-...
        - model_name: claude-3
          litellm_params:
            model: claude-3-sonnet-20240229
            api_key: sk-ant-...
      ```

      #### OffRail (After) [#offrail-after]

      With OffRail, you don't need a config file. Provider keys are managed in the web dashboard, or you can use the default OffRail keys.

      If you want to use your own provider keys, configure them in the dashboard under Settings > Provider Keys.
    </Step>
  </Steps>
</section>

<section>
  ## Streaming Support [#streaming-support]

  OffRail supports streaming identically to LiteLLM:

  ```python
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.offrail.ai/v1",
      api_key=os.environ["OFFRAIL_API_KEY"]
  )

  stream = client.chat.completions.create(
      model="openai/gpt-4",
      messages=[{"role": "user", "content": "Write a story"}],
      stream=True
  )

  for chunk in stream:
      if chunk.choices[0].delta.content:
          print(chunk.choices[0].delta.content, end="")
  ```
</section>

<section>
  ## Function/Tool Calling [#functiontool-calling]

  OffRail supports function calling:

  ```python
  from openai import OpenAI

  client = OpenAI(
      base_url="https://api.offrail.ai/v1",
      api_key=os.environ["OFFRAIL_API_KEY"]
  )

  tools = [{
      "type": "function",
      "function": {
          "name": "get_weather",
          "description": "Get the weather for a location",
          "parameters": {
              "type": "object",
              "properties": {
                  "location": {"type": "string"}
              },
              "required": ["location"]
          }
      }
  }]

  response = client.chat.completions.create(
      model="openai/gpt-4",
      messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
      tools=tools
  )
  ```
</section>

<section>
  ## Removing LiteLLM Infrastructure [#removing-litellm-infrastructure]

  After verifying OffRail works for your use case, you can decommission your LiteLLM proxy:

  1. Update all clients to use OffRail endpoints
  2. Monitor the OffRail dashboard for successful requests
  3. Shut down your LiteLLM proxy server
  4. Remove LiteLLM configuration files

  ## What Changes After Migration [#what-changes-after-migration]

  * **No servers to babysit** — We handle scaling, uptime, and updates
  * **Real-time cost visibility** — See what every request costs, broken down by model
  * **Automatic caching** — Repeated requests hit cache, reducing your spend
  * **Web-based management** — No more editing YAML files for config changes
  * **New models immediately** — Access new releases within 48 hours, no deployment needed
</section>

<section>
  This gives you the same benefits as LiteLLM's self-hosted proxy with OffRail's analytics and caching features.

  ## Full Comparison [#full-comparison]

  Want to see a detailed breakdown of all features? Check out our [OffRail vs LiteLLM comparison page](https://offrail.ai/compare/litellm).
</section>

<footer>
  ## Need Help? [#need-help]

  * Browse available models at [offrail.ai/models](https://offrail.ai/models)
  * Read the [API documentation](https://docs.offrail.ai)
  * Contact support at [contact@offrail.ai](mailto:contact@offrail.ai)
</footer>


# Migrate from OpenRouter
URL: https://docs.offrail.ai/migrations/openrouter


<section>
  OffRail works just like OpenRouter—same API format, same model names—but with built-in analytics and the option to self-host. Migration takes two lines of code.

  ## Quick Migration [#quick-migration]

  Change your base URL and API key:

  ```diff
  - const baseURL = "https://openrouter.ai/api/v1";
  - const apiKey = process.env.OPENROUTER_API_KEY;
  + const baseURL = "https://api.offrail.ai/v1";
  + const apiKey = process.env.OFFRAIL_API_KEY;
  ```
</section>

<section>
  ## Migration Steps [#migration-steps]

  <Steps>
    <Step>
      ### Get Your OffRail API Key [#get-your-offrail-api-key]

      Sign up at [offrail.ai/signup](https://offrail.ai/signup) and create an API key from your dashboard.
    </Step>

    <Step>
      ### Update Environment Variables [#update-environment-variables]

      ```bash
      # Remove OpenRouter credentials
      # OPENROUTER_API_KEY=sk-or-...

      # Add OffRail credentials
      OFFRAIL_API_KEY=orl_your_key_here
      ```
    </Step>

    <Step>
      ### Update Your Code [#update-your-code]

      #### Using fetch/axios [#using-fetchaxios]

      ```typescript
      // Before (OpenRouter)
      const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
      	method: "POST",
      	headers: {
      		Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
      		"Content-Type": "application/json",
      	},
      	body: JSON.stringify({
      		model: "openai/gpt-5.2",
      		messages: [{ role: "user", content: "Hello!" }],
      	}),
      });

      // After (OffRail)
      const response = await fetch("https://api.offrail.ai/v1/chat/completions", {
      	method: "POST",
      	headers: {
      		Authorization: `Bearer ${process.env.OFFRAIL_API_KEY}`,
      		"Content-Type": "application/json",
      	},
      	body: JSON.stringify({
      		model: "gpt-5.2",
      		messages: [{ role: "user", content: "Hello!" }],
      	}),
      });
      ```

      #### Using OpenAI SDK [#using-openai-sdk]

      ```typescript
      import OpenAI from "openai";

      // Before (OpenRouter)
      const client = new OpenAI({
      	baseURL: "https://openrouter.ai/api/v1",
      	apiKey: process.env.OPENROUTER_API_KEY,
      });

      // After (OffRail)
      const client = new OpenAI({
      	baseURL: "https://api.offrail.ai/v1",
      	apiKey: process.env.OFFRAIL_API_KEY,
      });

      // Usage remains the same
      const completion = await client.chat.completions.create({
      	model: "anthropic/claude-3-5-sonnet-20241022",
      	messages: [{ role: "user", content: "Hello!" }],
      });
      ```

      #### Using Vercel AI SDK [#using-vercel-ai-sdk]

      Both OpenRouter and OffRail have native AI SDK providers, making migration straightforward:

      ```typescript
      import { generateText } from "ai";

      // Before (OpenRouter AI SDK Provider)
      import { createOpenRouter } from "@openrouter/ai-sdk-provider";

      const openrouter = createOpenRouter({
      	apiKey: process.env.OPENROUTER_API_KEY,
      });

      const { text } = await generateText({
      	model: openrouter("gpt-5.2"),
      	prompt: "Hello!",
      });

      // After (OffRail AI SDK Provider)
      import { createOpenAI } from "@ai-sdk/openai";

      const offrail = createOpenAI({
      	apiKey: process.env.OFFRAIL_API_KEY,
      	baseURL: "https://api.offrail.ai/v1",
      });

      const { text } = await generateText({
      	model: offrail("gpt-5.2"),
      	prompt: "Hello!",
      });
      ```
    </Step>
  </Steps>
</section>

<section>
  ## Model Name Mapping [#model-name-mapping]

  Most model names are compatible, but here are some common mappings:

  | OpenRouter Model                 | OffRail Model                                                     |
  | -------------------------------- | ----------------------------------------------------------------- |
  | openai/gpt-5.2                   | gpt-5.2 or openai/gpt-5.2                                         |
  | gemini/gemini-3-flash-preview    | gemini-3-flash-preview or google-ai-studio/gemini-3-flash-preview |
  | bedrock/claude-opus-4-5-20251101 | claude-opus-4-5-20251101 or aws-bedrock/claude-opus-4-5-20251101  |

  Check the [models page](https://offrail.ai/models) for the full list of available models.
</section>

<section>
  ## Streaming Support [#streaming-support]

  OffRail supports streaming responses identically to OpenRouter:

  ```typescript
  const stream = await client.chat.completions.create({
  	model: "anthropic/claude-3-5-sonnet-20241022",
  	messages: [{ role: "user", content: "Write a story" }],
  	stream: true,
  });

  for await (const chunk of stream) {
  	process.stdout.write(chunk.choices[0]?.delta?.content || "");
  }
  ```
</section>

<section>
  ## Full Comparison [#full-comparison]

  Want to see a detailed breakdown of all features? Check out our [OffRail vs OpenRouter comparison page](https://offrail.ai/compare/open-router).
</section>

<footer>
  ## Need Help? [#need-help]

  * Browse available models at [offrail.ai/models](https://offrail.ai/models)
  * Read the [API documentation](https://docs.offrail.ai)
  * Contact support at [contact@offrail.ai](mailto:contact@offrail.ai)
</footer>


# Migrate from Vercel AI Gateway
URL: https://docs.offrail.ai/migrations/vercel-ai-gateway


<section>

  ## Quick Migration [#quick-migration]

  Swap your provider imports—your AI SDK code stays the same:

  ```diff
  - import { openai } from "@ai-sdk/openai";
  - import { anthropic } from "@ai-sdk/anthropic";
  + import { generateText } from "ai";
  + import { createOpenAI } from "@ai-sdk/openai";

  + const offrail = createOpenAI({
  +   apiKey: process.env.OFFRAIL_API_KEY,
  +   baseURL: "https://api.offrail.ai/v1"
  + });

  const { text } = await generateText({
  -   model: openai("gpt-5.2"),
  +   model: offrail("gpt-5.2"),
    prompt: "Hello!"
  });
  ```

  The key difference: one provider, one API key, all models—with caching and analytics built in.
</section>

<section>
  ## Zero-diff alternative: repoint the base URL [#zero-diff-alternative-repoint-the-base-url]

  If your app passes bare model strings (`model: "anthropic/claude-sonnet-5"`), it resolves them through `@ai-sdk/gateway` — the AI SDK's default provider. OffRail implements that protocol, so you can keep every line of application code and repoint the provider instead:

  ```ts
  import { createGateway } from "@ai-sdk/gateway";

  globalThis.AI_SDK_DEFAULT_PROVIDER = createGateway({
  	baseURL: "https://api.offrail.ai/v4/ai",
  	apiKey: process.env.OFFRAIL_API_KEY,
  });
  ```

  No import changes, no model-string changes, and provider-native web search keeps returning `source-url` parts. See [AI SDK Gateway protocol](https://docs.offrail.ai/developers/ai-sdk-gateway-protocol).

  Prefer the explicit `@ai-sdk/openai` migration below when you want to address models by the gateway's own IDs and keep the provider instance under your control.
</section>

<section>
  ## Migration Steps [#migration-steps]

  <Steps>
    <Step>
      ### Get Your OffRail API Key [#get-your-offrail-api-key]

      Sign up at [offrail.ai/signup](https://offrail.ai/signup) and create an API key from your dashboard.
    </Step>

    <Step>
      ### Install the OffRail AI SDK Provider [#install-the-offrail-ai-sdk-provider]

      Install the native OffRail provider for the Vercel AI SDK:

      ```bash
      pnpm add @ai-sdk/openai
      ```

      This package provides full compatibility with the Vercel AI SDK and supports all OffRail features.
    </Step>

    <Step>
      ### Update Your Code [#update-your-code]

      #### Basic Text Generation [#basic-text-generation]

      ```typescript
      // Before (Vercel AI Gateway with native providers)
      import { openai } from "@ai-sdk/openai";
      import { anthropic } from "@ai-sdk/anthropic";
      import { generateText } from "ai";

      const { text: openaiText } = await generateText({
      	model: openai("gpt-4o"),
      	prompt: "Hello!",
      });

      const { text: claudeText } = await generateText({
      	model: anthropic("claude-3-5-sonnet-20241022"),
      	prompt: "Hello!",
      });

      // After (OffRail - single provider for all models)
      import { createOpenAI } from "@ai-sdk/openai";
      import { generateText } from "ai";

      const offrail = createOpenAI({
      	apiKey: process.env.OFFRAIL_API_KEY,
      	baseURL: "https://api.offrail.ai/v1",
      });

      const { text: openaiText } = await generateText({
      	model: offrail("openai/gpt-4o"),
      	prompt: "Hello!",
      });

      const { text: claudeText } = await generateText({
      	model: offrail("anthropic/claude-3-5-sonnet-20241022"),
      	prompt: "Hello!",
      });
      ```

      #### Streaming Responses [#streaming-responses]

      ```typescript
      import { createOpenAI } from "@ai-sdk/openai";
      import { streamText } from "ai";

      const offrail = createOpenAI({
      	apiKey: process.env.OFFRAIL_API_KEY,
      	baseURL: "https://api.offrail.ai/v1",
      });

      const { textStream } = await streamText({
      	model: offrail("anthropic/claude-3-5-sonnet-20241022"),
      	prompt: "Write a poem about coding",
      });

      for await (const text of textStream) {
      	process.stdout.write(text);
      }
      ```

      #### Using in Next.js API Routes [#using-in-nextjs-api-routes]

      ```typescript
      // app/api/chat/route.ts
      import { createOpenAI } from "@ai-sdk/openai";
      import { streamText } from "ai";

      const offrail = createOpenAI({
      	apiKey: process.env.OFFRAIL_API_KEY,
      	baseURL: "https://api.offrail.ai/v1",
      });

      export async function POST(req: Request) {
      	const { messages } = await req.json();

      	const result = await streamText({
      		model: offrail("openai/gpt-4o"),
      		messages,
      	});

      	return result.toDataStreamResponse();
      }
      ```

      #### Alternative: Using OpenAI SDK Adapter [#alternative-using-openai-sdk-adapter]

      If you prefer not to install a new package, you can use `@ai-sdk/openai` with a custom base URL:

      ```typescript
      import { createOpenAI } from "@ai-sdk/openai";
      import { generateText } from "ai";

      const offrail = createOpenAI({
      	baseURL: "https://api.offrail.ai/v1",
      	apiKey: process.env.OFFRAIL_API_KEY,
      });

      const { text } = await generateText({
      	model: offrail("openai/gpt-4o"),
      	prompt: "Hello!",
      });
      ```
    </Step>

    <Step>
      ### Update Environment Variables [#update-environment-variables]

      ```bash
      # Remove individual provider keys (optional - can keep as backup)
      # OPENAI_API_KEY=sk-...
      # ANTHROPIC_API_KEY=sk-ant-...

      # Add OffRail key
      export OFFRAIL_API_KEY=orl_your_key_here
      ```
    </Step>
  </Steps>
</section>

<section>
  ## Model Name Format [#model-name-format]

  OffRail supports two model ID formats:

  **Canonical Model IDs** (without provider prefix) - Uses smart routing to automatically select the best provider based on uptime, throughput, price, and latency:

  ```
  gpt-4o
  claude-3-5-sonnet-20241022
  gemini-1.5-pro
  ```

  **Provider-Prefixed Model IDs** - Routes to a specific provider with automatic failover if uptime drops below 90%:

  ```
  openai/gpt-4o
  anthropic/claude-3-5-sonnet-20241022
  google-ai-studio/gemini-1.5-pro
  ```

  For more details on routing behavior, see the [routing documentation](https://docs.offrail.ai/features/routing).

  ### Model Mapping Examples [#model-mapping-examples]

  | Vercel AI SDK                             | OffRail                                                                                      |
  | ----------------------------------------- | -------------------------------------------------------------------------------------------- |
  | `openai("gpt-4o")`                        | `offrail("gpt-4o")` or `offrail("openai/gpt-4o")`                                            |
  | `anthropic("claude-3-5-sonnet-20241022")` | `offrail("claude-3-5-sonnet-20241022")` or `offrail("anthropic/claude-3-5-sonnet-20241022")` |
  | `google("gemini-1.5-pro")`                | `offrail("gemini-1.5-pro")` or `offrail("google-ai-studio/gemini-1.5-pro")`                  |

  Check the [models page](https://offrail.ai/models) for the full list of available models.
</section>

<section>
  ## Tool Calling [#tool-calling]

  OffRail supports tool calling through the AI SDK:

  ```typescript
  import { createOpenAI } from "@ai-sdk/openai";
  import { generateText, tool } from "ai";
  import { z } from "zod";

  const offrail = createOpenAI({
  	apiKey: process.env.OFFRAIL_API_KEY,
  	baseURL: "https://api.offrail.ai/v1",
  });

  const { text, toolResults } = await generateText({
  	model: offrail("openai/gpt-4o"),
  	tools: {
  		weather: tool({
  			description: "Get the weather for a location",
  			parameters: z.object({
  				location: z.string(),
  			}),
  			execute: async ({ location }) => {
  				return { temperature: 72, condition: "sunny" };
  			},
  		}),
  	},
  	prompt: "What's the weather in San Francisco?",
  });
  ```
</section>

<section>
  This gives you the same managed experience with full control over your infrastructure.
</section>

<footer>
  ## Need Help? [#need-help]

  * Browse available models at [offrail.ai/models](https://offrail.ai/models)
  * Read the [API documentation](https://docs.offrail.ai)
  * Contact support at [contact@offrail.ai](mailto:contact@offrail.ai)
</footer>


# Error Handling
URL: https://docs.offrail.ai/resources/error-handling


# Error Handling [#error-handling]

On the OpenAI-compatible endpoints, OffRail returns errors in the same format as the OpenAI API, so existing OpenAI SDKs and tooling can parse gateway errors without changes. This applies to errors forwarded from upstream providers as well as errors raised by the gateway itself (authentication failures, usage limits, validation problems, timeouts, and so on). The Anthropic-compatible Messages endpoint (`/v1/messages`) instead returns Anthropic-native errors — see [Anthropic Endpoint](#anthropic-endpoint) below.

## Error Format [#error-format]

Errors on the OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/embeddings`, `/v1/images`, `/v1/models`, `/v1/moderations`, `/v1/rerank`, `/v1/responses`, `/v1/videos`) use the standard OpenAI error envelope:

```json
{
	"error": {
		"message": "Unauthorized: OffRail API key reached its usage limit.",
		"type": "invalid_request_error",
		"param": null,
		"code": "invalid_api_key"
	}
}
```

| Field           | Description                                                                         |
| --------------- | ----------------------------------------------------------------------------------- |
| `error.message` | Human-readable description of what went wrong.                                      |
| `error.type`    | High-level error category (see the table below).                                    |
| `error.param`   | The request parameter that caused the error, or `null` when not parameter-specific. |
| `error.code`    | A more specific machine-readable code, or `null` when no specific code applies.     |

The HTTP status code on the response always matches the error and is the authoritative signal — read it from the response status line rather than the body.

## Status Codes [#status-codes]

The gateway maps HTTP status codes to OpenAI error types and codes as follows:

| Status | `type`                  | `code`                   |
| ------ | ----------------------- | ------------------------ |
| 400    | `invalid_request_error` | *(varies / `null`)*      |
| 401    | `invalid_request_error` | `invalid_api_key`        |
| 402    | `invalid_request_error` | `billing_error`          |
| 403    | `invalid_request_error` | `permission_denied`      |
| 404    | `invalid_request_error` | `not_found`              |
| 408    | `timeout_error`         | `timeout`                |
| 413    | `invalid_request_error` | `request_too_large`      |
| 415    | `invalid_request_error` | `unsupported_media_type` |
| 429    | `rate_limit_error`      | `rate_limit_exceeded`    |
| 499    | `invalid_request_error` | `request_cancelled`      |
| 504    | `timeout_error`         | `timeout`                |
| 5xx    | `api_error`             | *(`null`)*               |

<Callout type="info">
  Validation errors raised before a request reaches a provider often include a
  more specific `code` and a `param` pointing at the offending field — for
  example `invalid_json`, `model_not_found`, or
  `unsupported_parameter_combination`.
</Callout>

## Streaming Errors [#streaming-errors]

For streaming requests (`"stream": true`), an error that occurs **after** the stream has started is delivered as an SSE `error` event whose payload uses the same `{ "error": { ... } }` envelope. Errors that occur **before** streaming begins (such as authentication failures) are returned as a normal JSON error response with the appropriate status code.

## Anthropic Endpoint [#anthropic-endpoint]

The Anthropic-compatible Messages endpoint (`/v1/messages`) returns errors in Anthropic's native format instead, so the Anthropic SDK can parse them:

```json
{
	"type": "error",
	"error": {
		"type": "authentication_error",
		"message": "Unauthorized: invalid API key."
	}
}
```

## Related [#related]

* [Rate Limits](https://docs.offrail.ai/resources/rate-limits) — details on `429` responses and rate limit headers.


# Rate Limits
URL: https://docs.offrail.ai/resources/rate-limits


# Rate Limits [#rate-limits]

OffRail applies rate limits to ensure fair usage and protect platform stability. Limits are evaluated in a few independent layers:

* **Per-organization endpoint limits** — a requests-per-minute cap on every API endpoint, scoped to your organization.
* **Free model limits** — additional limits specifically for zero-cost models.
* **Provider limits** — upstream limits enforced per provider/model when applicable.

## Per-Organization Endpoint Limits [#per-organization-endpoint-limits]

Every API endpoint is rate limited per organization using a rolling 60-second window. The limit is independent for each endpoint, so traffic to `/v1/chat/completions` does not consume the budget for `/v1/embeddings`.

The default limits (requests per minute, per organization) are:

| Endpoint             | Path                       | Requests / min |
| -------------------- | -------------------------- | -------------- |
| Chat completions     | `/v1/chat/completions`     | 600            |
| Messages (Anthropic) | `/v1/messages`             | 600            |
| Responses            | `/v1/responses`            | 600            |
| Embeddings           | `/v1/embeddings`           | 1200           |
| Moderations          | `/v1/moderations`          | 1200           |
| Rerank               | `/v1/rerank`               | 1200           |
| Models               | `/v1/models`               | 1200           |
| OCR                  | `/v1/ocr`                  | 300            |
| Images               | `/v1/images`               | 300            |
| Speech               | `/v1/audio/speech`         | 300            |
| Transcriptions       | `/v1/audio/transcriptions` | 300            |
| Videos               | `/v1/videos`               | 120            |
| Realtime (mint)      | `/v1/realtime`             | 120            |
| Key info             | `/v1/key`                  | 1200           |
| Credits              | `/v1/credits`              | 300            |
| AI SDK protocol      | `/v*/ai`                   | 600            |

<Callout type="info">
  **[Enterprise](https://offrail.ai/enterprise) organizations are exempt** from
  these per-organization endpoint limits. [Contact
  us](mailto:contact@offrail.ai) about enterprise plans.
</Callout>

### Trust Tiers (account age or spend) [#trust-tiers-account-age-or-spend]

For regular (pay-as-you-go) organizations, limits scale with a **trust tier**. An organization qualifies for a tier when its account is old enough, **or** when its lifetime usage spend is high enough **and** the account meets the tier's minimum age. The tier raises the per-endpoint RPM limits **and** the daily/monthly USD spend caps below.

| Tier | Qualifies (age, or spend + min age) | RPM multiplier | Daily cap | Monthly cap |
| ---- | ----------------------------------- | -------------- | --------- | ----------- |
| 0    | new / $0                            | 1×             | $25       | $250        |
| 1    | 7 days **or** $10 (account ≥ 1 day) | 2×             | $100      | $1,000      |
| 2    | 30 days **or** $100 (≥ 3 days)      | 4×             | $500      | $5,000      |
| 3    | 60 days **or** $1,000 (≥ 7 days)    | 10×            | $5,000    | $50,000     |
| 4    | 90 days **or** $5,000 (≥ 14 days)   | 20×            | $15,000   | $200,000    |

Spend alone never promotes a brand-new account: each spend-qualified tier also requires the minimum account age shown, so the fastest possible path to Tier 4 is 14 days — no amount of day-one usage unlocks higher limits.

For example, an org past 30 days old (or with $100+ of usage) is Tier 2: chat completions rises from 600 to 2,400 RPM, with a $500/day and $5,000/month spend ceiling.

Qualifying spend counts **usage billed to your credit balance only** — usage served through your own provider keys (BYOK) does not count — and is **net of refunds**: every refunded payment is deducted, so refunded or clawed-back money never raises limits. Refunded top-ups still count against the top-up allowance below — refunding does not free up top-up headroom.

### Daily & Monthly Spend Caps [#daily--monthly-spend-caps]

Regular organizations also have hard **USD spend ceilings** — a daily and a monthly cap set by the trust tier above — so a brand-new account has a tight dollar velocity limit that rises as it ages or spends. Only real paid usage counts; **free models are exempt**, as are **enterprise** (no caps) and dev/chat plan orgs (which have their own plan limits). When a cap is reached, requests return `429` until the counter resets (UTC midnight for daily, first of the month for monthly).

### Top-Up Limits [#top-up-limits]

Credit top-ups are also velocity-limited by trust tier: each organization can add at most a tier-scaled gross USD amount to its balance per **rolling 24-hour window**. This applies before any charge is made — a top-up attempt over the allowance is rejected with `429` and no card is charged.

| Tier | Top-up allowance (rolling 24h) |
| ---- | ------------------------------ |
| 0    | $100                           |
| 1    | $500                           |
| 2    | $2,500                         |
| 3    | $10,000                        |
| 4    | $20,000                        |

The limit covers dashboard top-ups (card and hosted checkout), auto top-up, and Dev plan pay-as-you-go top-ups. **[Enterprise](https://offrail.ai/enterprise) organizations are exempt** — [contact us](mailto:contact@offrail.ai) if you need a higher allowance. Your current allowance and usage are shown on the Settings → Limits page. Hosted checkout links expire after 30 minutes.

### Dev and Chat Plans [#dev-and-chat-plans]

Organizations on a **Chat plan** have their own, tighter per-endpoint limits and do not receive the spend-based multiplier.

Dev plans are inference-only and only cover **chat completions, messages, responses, the models list, and OCR**, each at a flat 120 requests-per-minute floor. The other endpoints (embeddings, moderations, images, speech, videos) are **not available*&#x2A; on Dev plans and are marked &#x2A;*—** below.

| Endpoint             | Dev plan / min | Chat plan / min |
| -------------------- | -------------- | --------------- |
| Chat completions     | 120            | 60              |
| Messages (Anthropic) | 120            | 60              |
| Responses            | 120            | 60              |
| Models               | 120            | 120             |
| OCR                  | 120            | 30              |
| Embeddings           | —              | 120             |
| Moderations          | —              | 120             |
| Images               | —              | 30              |
| Speech               | —              | 30              |
| Videos               | —              | 12              |

### Enterprise [#enterprise]

Organizations on the **[Enterprise plan](https://offrail.ai/enterprise) have no per-organization rate limits at all** — no requests-per-minute caps, no spend caps, and no top-up limits. Your throughput is limited only by your credit balance and any upstream provider limits.

<Callout type="info">
  Need unlimited gateway throughput? [Contact us](mailto:contact@offrail.ai)
  about an enterprise plan.
</Callout>

## Free Models [#free-models]

Free models (models with zero input and output pricing) have additional rate limits that depend on your account's credit status:

### Base Rate Limits [#base-rate-limits]

For organizations with **zero credits**:

* **5 requests per 10 minutes**
* Applies to all free model requests
* Resets every 10 minutes

### Elevated Rate Limits [#elevated-rate-limits]

For organizations that have **purchased at least some credits**:

* **20 requests per minute**
* Applies to all free model requests
* Resets every minute

<Callout type="info">
  When using free models with elevated limits, your credits will **not** be
  deducted. The elevated rate limits are simply a benefit for users who have
  added credits to their account.
</Callout>

## Provider Limits [#provider-limits]

Some providers and models enforce their own upstream request limits. When a provider limit is reached, OffRail returns a `429` and includes provider-scoped headers (`X-RateLimit-Limit-Provider`, `X-RateLimit-Remaining-Provider`). Where possible, [fallback routing](https://docs.offrail.ai/features/routing) automatically retries the request on a healthy provider.

## Rate Limit Headers [#rate-limit-headers]

When a request is rate limited, the `429` response includes:

```http
Retry-After: 12
X-RateLimit-Limit: 600
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1640995200
```

* `Retry-After`: Seconds to wait before retrying
* `X-RateLimit-Limit`: Maximum number of requests allowed in the current window
* `X-RateLimit-Remaining`: Number of requests remaining in the current window
* `X-RateLimit-Reset`: Unix timestamp when the rate limit window resets

## Rate Limit Exceeded [#rate-limit-exceeded]

When you exceed a rate limit, you'll receive a `429 Too Many Requests` response:

```json
{
	"error": {
		"message": "Rate limit exceeded for /v1/chat/completions. Please retry after 12 seconds.",
		"type": "rate_limit_error",
		"code": "rate_limit_exceeded"
	}
}
```

This uses the standard OpenAI-compatible error envelope. Requests to the Anthropic-compatible `/v1/messages` endpoint receive the Anthropic error shape instead. See [Error Handling](https://docs.offrail.ai/resources/error-handling) for the full format and status-code reference.

## Best Practices [#best-practices]

* **Respect `Retry-After`.** Implement exponential backoff when you receive `429` responses, starting from the `Retry-After` value.
* **Watch the headers.** Monitor `X-RateLimit-Remaining` to back off before you hit the limit.
* **Spread traffic across endpoints.** Limits are per endpoint, so unrelated workloads don't compete for the same budget.
* **Scale with usage.** Regular organizations unlock higher limits automatically as lifetime spend grows; [contact us](mailto:contact@offrail.ai) about an [Enterprise plan](https://offrail.ai/enterprise) to remove per-organization limits entirely.

<Callout type="success">
  Adding even a small amount of credits to your account (e.g., $10) will
  immediately upgrade your free model rate limits from 5 requests per 10 minutes
  to 20 requests per minute.
</Callout>


# AWS
URL: https://docs.offrail.ai/self-host/aws


<Callout type="info">
  **Deploy in one command.** Our [Enterprise
  plan](https://offrail.ai/enterprise) includes Terraform modules that provision
  the cluster, database, cache, networking, and secrets and deploy LLM Gateway
  on AWS — no manual setup required. [Learn
  more](https://offrail.ai/enterprise).
</Callout>

This guide covers a production deployment of OffRail on AWS using EKS for the application services and managed AWS services for the backing stores.

## Architecture [#architecture]

| Component  | AWS service                  |
| ---------- | ---------------------------- |
| Compute    | Amazon EKS (Kubernetes)      |
| PostgreSQL | Amazon RDS for PostgreSQL    |
| Redis      | Amazon ElastiCache for Redis |
| Secrets    | AWS Secrets Manager          |
| Ingress    | AWS Load Balancer Controller |

## What to configure [#what-to-configure]

### 1. PostgreSQL — Amazon RDS [#1-postgresql--amazon-rds]

Create an RDS for PostgreSQL instance in a private subnet. Enable automated backups and Multi-AZ for high availability. Note the connection string for `DATABASE_URL`.

### 2. Redis — Amazon ElastiCache [#2-redis--amazon-elasticache]

Create an ElastiCache for Redis cluster in the same VPC. Use it for `REDIS_URL`. A single-node cluster is fine to start; enable replication for production.

### 3. Compute — Amazon EKS [#3-compute--amazon-eks]

Create an EKS cluster with a managed node group sized for your traffic. Install the [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/) to expose the gateway through an Application Load Balancer.

### 4. Networking [#4-networking]

Run RDS and ElastiCache in private subnets and allow inbound traffic only from the EKS node security group. Expose only the gateway (and optionally the UI) to the internet through the load balancer.

### 5. Secrets — AWS Secrets Manager [#5-secrets--aws-secrets-manager]

Store `AUTH_SECRET`, `GATEWAY_API_KEY_HASH_SECRET`, and your provider API keys in Secrets Manager. Sync them into the cluster with the [External Secrets Operator](https://external-secrets.io/) or the AWS Secrets and Configuration Provider (ASCP).

## Deploy the Helm chart [#deploy-the-helm-chart]

With the backing services in place, deploy OffRail with the [Helm chart](https://docs.offrail.ai/self-host/kubernetes), pointing it at your RDS and ElastiCache endpoints:

```bash
helm install offrail oci://offrail/charts/offrail -f values.yaml
```

```yaml
config:
  DATABASE_URL: "postgres://user:password@your-rds-endpoint:5432/offrail"
  REDIS_URL: "redis://your-elasticache-endpoint:6379"
  AUTH_SECRET: "from-secrets-manager"
  GATEWAY_API_KEY_HASH_SECRET: "from-secrets-manager"
```

See the [Kubernetes guide](https://docs.offrail.ai/self-host/kubernetes) for the full set of configurable values and how to scale the gateway.

<Callout type="info">
  **Prefer not to wire this up by hand?** The [Enterprise
  plan](https://offrail.ai/enterprise) ships Terraform modules that stand up the
  entire AWS stack — EKS, RDS, ElastiCache, networking, and secrets — and deploy
  OffRail in one command. [Talk to us](https://offrail.ai/enterprise#contact).
</Callout>


# Azure
URL: https://docs.offrail.ai/self-host/azure


<Callout type="info">
  **Deploy in one command.** Our [Enterprise
  plan](https://offrail.ai/enterprise) includes Terraform modules that provision
  the cluster, database, cache, networking, and secrets and deploy LLM Gateway
  on Azure — no manual setup required. [Learn
  more](https://offrail.ai/enterprise).
</Callout>

This guide covers a production deployment of OffRail on Azure using AKS for the application services and managed Azure services for the backing stores.

## Architecture [#architecture]

| Component  | Azure service                     |
| ---------- | --------------------------------- |
| Compute    | Azure Kubernetes Service (AKS)    |
| PostgreSQL | Azure Database for PostgreSQL     |
| Redis      | Azure Cache for Redis             |
| Secrets    | Azure Key Vault                   |
| Ingress    | Application Gateway / AKS Ingress |

## What to configure [#what-to-configure]

### 1. PostgreSQL — Azure Database for PostgreSQL [#1-postgresql--azure-database-for-postgresql]

Create an Azure Database for PostgreSQL Flexible Server with private access. Enable automated backups and zone-redundant high availability. Note the connection details for `DATABASE_URL`.

### 2. Redis — Azure Cache for Redis [#2-redis--azure-cache-for-redis]

Create an Azure Cache for Redis instance in the same virtual network and use its endpoint for `REDIS_URL`. Choose a Standard or Premium tier for replication in production.

### 3. Compute — AKS [#3-compute--aks]

Create an AKS cluster with a node pool sized for your traffic. Use the [Application Gateway Ingress Controller](https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-overview) or an NGINX ingress to expose the gateway.

### 4. Networking [#4-networking]

Deploy the database and cache with private endpoints inside your virtual network and restrict access to the AKS subnet. Expose only the gateway (and optionally the UI) to the internet.

### 5. Secrets — Azure Key Vault [#5-secrets--azure-key-vault]

Store `AUTH_SECRET`, `GATEWAY_API_KEY_HASH_SECRET`, and your provider API keys in Key Vault. Sync them into the cluster with the [Azure Key Vault Provider for Secrets Store CSI Driver](https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver).

## Deploy the Helm chart [#deploy-the-helm-chart]

With the backing services in place, deploy OffRail with the [Helm chart](https://docs.offrail.ai/self-host/kubernetes), pointing it at your Azure Database and Azure Cache endpoints:

```bash
helm install offrail oci://offrail/charts/offrail -f values.yaml
```

```yaml
config:
  DATABASE_URL: "postgres://user:password@your-postgres-host:5432/offrail"
  REDIS_URL: "redis://your-redis-host:6380"
  AUTH_SECRET: "from-key-vault"
  GATEWAY_API_KEY_HASH_SECRET: "from-key-vault"
```

See the [Kubernetes guide](https://docs.offrail.ai/self-host/kubernetes) for the full set of configurable values and how to scale the gateway.

<Callout type="info">
  **Prefer not to wire this up by hand?** The [Enterprise
  plan](https://offrail.ai/enterprise) ships Terraform modules that stand up the
  entire Azure stack — AKS, Azure Database for PostgreSQL, Azure Cache for
  Redis, networking, and secrets — and deploy OffRail in one command. [Talk to
  us](https://offrail.ai/enterprise#contact).
</Callout>


# Docker Compose
URL: https://docs.offrail.ai/self-host/docker-compose


Docker Compose runs each service in its own container, giving you more control over scaling and configuration than the [single Docker image](https://docs.offrail.ai/self-host/docker). It's a good fit for a single production host. For multi-node, high-availability deployments, use [Kubernetes](https://docs.offrail.ai/self-host/kubernetes).

## Prerequisites [#prerequisites]

* Latest Docker with Compose
* API keys for the LLM providers you want to use (OpenAI, Anthropic, etc.)

## Option A: Unified image with Compose [#option-a-unified-image-with-compose]

Run the all-in-one image under Compose for easy lifecycle management:

```bash
# Download the compose file
curl -O https://bitbucket.org/designsai/offrailai/raw/main/infra/docker-compose.unified.yml
curl -O https://bitbucket.org/designsai/offrailai/raw/main/.env.unified.example

# Configure environment
cp .env.unified.example .env
# Edit .env with your configuration

# Start the service
docker compose -f docker-compose.unified.yml up -d
```

## Option B: Separate services [#option-b-separate-services]

Run each service in its own container for the most flexibility:

```bash
# Clone the repository
git clone https://bitbucket.org/designsai/offrailai.git
cd offrail

# Configure environment
cp .env.example .env
# Edit .env with your configuration

# Start the services
docker compose -f infra/docker-compose.split.yml up -d
```

Pin to a specific version by replacing the `latest` image tags with the [latest release](https://bitbucket.org/designsai/offrailai/releases). To build the images from source, use the `*.local.yml` compose files in the `infra` directory.

## Accessing your instance [#accessing-your-instance]

* **Web Interface**: [http://localhost:3002](http://localhost:3002)
* **Chat Playground**: [http://localhost:3003](http://localhost:3003)
* **Documentation**: [http://localhost:3005](http://localhost:3005)
* **API Endpoint**: [http://localhost:4002](http://localhost:4002)
* **Gateway Endpoint**: [http://localhost:4001](http://localhost:4001)

## Required configuration [#required-configuration]

At minimum, set these environment variables:

```bash
# Database (change the password!)
POSTGRES_PASSWORD=your_secure_password_here

# Authentication
AUTH_SECRET=your-secret-key-here
GATEWAY_API_KEY_HASH_SECRET=your-api-key-hash-secret-here

# LLM Provider API Keys (add the ones you need)
LLM_OPENAI_API_KEY=sk-...
LLM_ANTHROPIC_API_KEY=sk-ant-...
```

## Management commands [#management-commands]

```bash
# View logs
docker compose -f infra/docker-compose.split.yml logs -f

# Restart services
docker compose -f infra/docker-compose.split.yml restart

# Stop services
docker compose -f infra/docker-compose.split.yml down
```

## Managing provider credentials from the admin dashboard [#managing-provider-credentials-from-the-admin-dashboard]

Provider credentials can also live in the database instead of the environment.
The admin dashboard's **Provider Credentials** page lets you add, edit and remove
them without redeploying, and each credential carries every setting the provider
needs — API key, base URL, project, region, resource, API version — plus a free-form
note so several keys for the same provider stay tellable apart.

Resolution is per provider: as soon as a provider has at least one managed
credential, credits-mode requests to it use those credentials and its `LLM_*`
environment variables are ignored entirely. Providers with no managed credential
keep reading the environment, so you can migrate one provider at a time.

Managed credentials cover the same axes as the environment variables they replace:

* Several credentials per provider, selected with the same health-aware routing
  as the comma-separated env lists.
* A per-credential audience (all organizations / enterprise plans / Chat
  Chat plans), mirroring the `__ENTERPRISE` and `__PLANS` suffixes below.
* An optional region, mirroring the `{ENV_VAR}__{REGION}` overrides.

A request that resolves no region is served from the provider's default region,
so it is served by a region-agnostic credential or by one pinned to that default
region. Providers whose keys are region-scoped — those with no global region, so
every credential belongs to exactly one region — are therefore fully covered by
one credential per region, with no region-agnostic credential needed. Providers
whose key works across every region (AWS Bedrock) are stricter: a region on the
credential scopes it to that region only, and a request for another region fails
rather than borrowing it.

Video generation pins the credential that created a job onto the job itself, so
polling and content retrieval hours later go back through the same credential
rather than re-selecting one.

Credential tokens are encrypted at rest with AES-256-GCM. The key is derived
from `GATEWAY_API_KEY_HASH_SECRET`, so there is no extra variable to set — but
that secret now protects stored provider credentials as well as API-key hashes.
Give it a strong random value on every service (`openssl rand -base64 32`).
Rotating it makes existing credentials undecryptable, so re-enter each
credential after a rotation.

## Multiple API keys and load balancing [#multiple-api-keys-and-load-balancing]

OffRail supports multiple API keys per provider for load balancing and increased availability. Provide comma-separated values:

```bash
# Multiple OpenAI keys for load balancing
LLM_OPENAI_API_KEY=sk-key1,sk-key2,sk-key3

# Multiple Anthropic keys
LLM_ANTHROPIC_API_KEY=sk-ant-key1,sk-ant-key2
```

### Health-aware routing [#health-aware-routing]

The gateway tracks the health of each API key and routes requests to healthy keys. If a key returns consecutive errors, it's temporarily skipped. Keys that return authentication errors (401/403) are blacklisted until restart.

### Related configuration values [#related-configuration-values]

For providers that require additional configuration (like Google Vertex), specify multiple values that correspond to each API key. The gateway uses the matching index:

```bash
# Multiple Google Vertex configurations
LLM_GOOGLE_VERTEX_API_KEY=key1,key2,key3
LLM_GOOGLE_CLOUD_PROJECT=project-a,project-b,project-c
LLM_GOOGLE_VERTEX_REGION=us-central1,europe-west1,asia-east1
```

When the gateway selects `key2`, it automatically uses `project-b` and `europe-west1`. If you have fewer configuration values than keys, the last value is reused for the remaining keys.

`LLM_GOOGLE_CLOUD_PROJECT` is optional for Google Vertex API-key chat, embedding, and speech requests, which use the projectless publisher-model endpoint when it is unset. Set it for OAuth authentication, video generation, or project-scoped Vertex URLs.

## Enterprise and plan provider env overrides [#enterprise-and-plan-provider-env-overrides]

Any provider env var — API keys, base URLs, regions, Google Cloud projects, Azure resources, and other provider-specific settings — supports optional per-audience overrides:

* `__ENTERPRISE` suffix: used instead of the base var for organizations on the enterprise plan
* `__PLANS` suffix: used instead of the base var for plan-based (non-PAYG) organizations — Chat plans

```bash
# Shared key for all organizations
LLM_OPENAI_API_KEY=sk-shared-key

# Used instead of the shared key, but only for enterprise-plan organizations
LLM_OPENAI_API_KEY__ENTERPRISE=sk-enterprise-key

# Used instead of the shared key, but only for plan-based (Chat plan)
# organizations
LLM_OPENAI_API_KEY__PLANS=sk-plans-key

# Companion settings can be overridden the same way, e.g. a dedicated
# Google Cloud project and base URL per audience
LLM_GOOGLE_CLOUD_PROJECT__ENTERPRISE=enterprise-gcp-project
LLM_OPENAI_BASE_URL__PLANS=https://plans-proxy.internal
```

Notes:

* Overrides are optional — matching organizations fall back to the base var when an override is unset. All other organizations never read the override vars.
* The base API key is still what makes a provider available for routing, so set it even when you route all enterprise or plan traffic through dedicated keys.
* Comma-separated values, load balancing, and health-aware routing work exactly like the base key; each override key list gets its own independent health tracking.
* A set override replaces the base var wholesale, including its comma-separated list. Indexed companion values (like `LLM_GOOGLE_CLOUD_PROJECT`) resolve at the selected key index from the override list when one is set, otherwise from the base list — so keep an override key list index-compatible with its companion lists (or use single values).
* Region-specific overrides compose with the variant suffix. For matching organizations the API-key lookup order is `{BASE}__ENTERPRISE__{REGION}` → `{BASE}__{REGION}` → `{BASE}__ENTERPRISE` → `{BASE}` (same pattern with `__PLANS`).
* If an organization is both on the enterprise plan and a plan-based org, the enterprise overrides win.


# Docker
URL: https://docs.offrail.ai/self-host/docker


The unified Docker image bundles every service — UI, API, Gateway, PostgreSQL, and Redis — into a single container. It's the fastest way to get a working instance and is ideal for trying OffRail out or running a single low-traffic deployment.

For production, run each service separately with [Docker Compose](https://docs.offrail.ai/self-host/docker-compose) or deploy to [Kubernetes](https://docs.offrail.ai/self-host/kubernetes) with managed Postgres and Redis.

## Prerequisites [#prerequisites]

* Latest Docker
* API keys for the LLM providers you want to use (OpenAI, Anthropic, etc.)

## Run the container [#run-the-container]

```bash
# Set a strong secret first
export OFFRAIL_SECRET="your-secret-key-here"
export GATEWAY_API_KEY_HASH_SECRET="your-api-key-hash-secret-here"

# Run the container
docker run -d \
  --name offrail \
  --restart unless-stopped \
  -p 3002:3002 \
  -p 3003:3003 \
  -p 3005:3005 \
  -p 3006:3006 \
  -p 4001:4001 \
  -p 4002:4002 \
  -v offrail_postgres:/var/lib/postgresql/data \
  -v offrail_redis:/var/lib/redis \
  -e AUTH_SECRET="$OFFRAIL_SECRET" \
  -e GATEWAY_API_KEY_HASH_SECRET="$GATEWAY_API_KEY_HASH_SECRET" \
  offrail/offrail-unified:latest
```

Docker creates the named volumes automatically on first run. Do not bind-mount a host directory directly to `/var/lib/postgresql/data`, because PostgreSQL initialization inside the container needs to manage permissions on that path.

Pin to a specific version instead of `latest` using the [latest release tag](https://bitbucket.org/designsai/offrailai/releases).

## Accessing your instance [#accessing-your-instance]

* **Web Interface**: [http://localhost:3002](http://localhost:3002)
* **Chat Playground**: [http://localhost:3003](http://localhost:3003)
* **Documentation**: [http://localhost:3005](http://localhost:3005)
* **API Endpoint**: [http://localhost:4002](http://localhost:4002)
* **Gateway Endpoint**: [http://localhost:4001](http://localhost:4001)

## Management commands [#management-commands]

```bash
# View logs
docker logs offrail

# Restart the container
docker restart offrail

# Stop the container
docker stop offrail
```

## Required configuration [#required-configuration]

At minimum, set these environment variables:

```bash
# Authentication
AUTH_SECRET=your-secret-key-here
GATEWAY_API_KEY_HASH_SECRET=your-api-key-hash-secret-here

# LLM Provider API Keys (add the ones you need)
LLM_OPENAI_API_KEY=sk-...
LLM_ANTHROPIC_API_KEY=sk-ant-...
```

OffRail supports multiple API keys per provider for load balancing — provide comma-separated values. See [Docker Compose](https://docs.offrail.ai/self-host/docker-compose#multiple-api-keys-and-load-balancing) for details. Provider env vars also support optional `__ENTERPRISE` / `__PLANS` suffix overrides used only for enterprise-plan or plan-based (Chat plan) organizations — see [Enterprise and plan provider env overrides](https://docs.offrail.ai/self-host/docker-compose#enterprise-and-plan-provider-env-overrides).

## Next steps [#next-steps]

Once your instance is running:

1. **Open the web interface** at [http://localhost:3002](http://localhost:3002)
2. **Create your first organization** and project
3. **Generate API keys** for your applications
4. **Test the gateway** by making API calls to [http://localhost:4001](http://localhost:4001)


# Google Cloud
URL: https://docs.offrail.ai/self-host/gcp


<Callout type="info">
  **Deploy in one command.** Our [Enterprise
  plan](https://offrail.ai/enterprise) includes Terraform modules that provision
  the cluster, database, cache, networking, and secrets and deploy LLM Gateway
  on Google Cloud — no manual setup required. [Learn
  more](https://offrail.ai/enterprise).
</Callout>

This guide covers a production deployment of OffRail on Google Cloud using GKE for the application services and managed Google Cloud services for the backing stores.

## Architecture [#architecture]

| Component  | Google Cloud service               |
| ---------- | ---------------------------------- |
| Compute    | Google Kubernetes Engine (GKE)     |
| PostgreSQL | Cloud SQL for PostgreSQL           |
| Redis      | Memorystore for Redis              |
| Secrets    | Secret Manager                     |
| Ingress    | GKE Ingress / Cloud Load Balancing |

## What to configure [#what-to-configure]

### 1. PostgreSQL — Cloud SQL [#1-postgresql--cloud-sql]

Create a Cloud SQL for PostgreSQL instance with a private IP. Enable automated backups and high availability. Note the connection details for `DATABASE_URL`.

### 2. Redis — Memorystore [#2-redis--memorystore]

Create a Memorystore for Redis instance in the same VPC and use its endpoint for `REDIS_URL`. Enable a read replica for production.

### 3. Compute — GKE [#3-compute--gke]

Create a GKE cluster (Autopilot or Standard) sized for your traffic. GKE provisions an HTTP(S) load balancer automatically when you create an Ingress for the gateway.

### 4. Networking [#4-networking]

Use a private VPC and connect Cloud SQL via Private Service Access and Memorystore via its private endpoint. Restrict access so only the GKE workloads can reach the database and cache, and expose only the gateway to the internet.

### 5. Secrets — Secret Manager [#5-secrets--secret-manager]

Store `AUTH_SECRET`, `GATEWAY_API_KEY_HASH_SECRET`, and your provider API keys in Secret Manager. Sync them into the cluster with the [External Secrets Operator](https://external-secrets.io/) or the Secret Manager CSI driver.

## Deploy the Helm chart [#deploy-the-helm-chart]

With the backing services in place, deploy OffRail with the [Helm chart](https://docs.offrail.ai/self-host/kubernetes), pointing it at your Cloud SQL and Memorystore endpoints:

```bash
helm install offrail oci://offrail/charts/offrail -f values.yaml
```

```yaml
config:
  DATABASE_URL: "postgres://user:password@your-cloud-sql-ip:5432/offrail"
  REDIS_URL: "redis://your-memorystore-ip:6379"
  AUTH_SECRET: "from-secret-manager"
  GATEWAY_API_KEY_HASH_SECRET: "from-secret-manager"
```

See the [Kubernetes guide](https://docs.offrail.ai/self-host/kubernetes) for the full set of configurable values and how to scale the gateway.

<Callout type="info">
  **Prefer not to wire this up by hand?** The [Enterprise
  plan](https://offrail.ai/enterprise) ships Terraform modules that stand up the
  entire Google Cloud stack — GKE, Cloud SQL, Memorystore, networking, and
  secrets — and deploy OffRail in one command. [Talk to
  us](https://offrail.ai/enterprise#contact).
</Callout>


# Self Host OffRail
URL: https://docs.offrail.ai/self-host


OffRail is a self-hostable platform that provides a unified API gateway for multiple LLM providers. Run it on your own infrastructure to keep full control over your data and avoid platform fees.

Pick the deployment path that matches where you're running it.

Self-hosting guides are documented under https://docs.offrail.ai/self-host and included in full in this file.

## Which option should I choose? [#which-option-should-i-choose]

* **Trying it out or running a single low-traffic instance?** Start with [Docker](https://docs.offrail.ai/self-host/docker) or [Docker Compose](https://docs.offrail.ai/self-host/docker-compose) on one machine.
* **Running in production?** Deploy to [Kubernetes](https://docs.offrail.ai/self-host/kubernetes) with our Helm chart, and use a managed Postgres and Redis from your cloud.
* **On a specific cloud?** Follow the [AWS](https://docs.offrail.ai/self-host/aws), [Google Cloud](https://docs.offrail.ai/self-host/gcp), or [Azure](https://docs.offrail.ai/self-host/azure) guide for the exact managed services to provision.

## What you'll need [#what-youll-need]

Every deployment is built from the same pieces:

* **Stateless services** — the gateway, API, UI, and a background worker. Scale these freely; they hold no data between requests.
* **PostgreSQL** — the source of truth for users, projects, keys, and usage records.
* **Redis** — response caching and the queue that feeds the worker.
* **Provider API keys** — the OpenAI, Anthropic, Google, and other credentials the gateway uses, injected as secrets.

In production, run PostgreSQL and Redis as managed services from your cloud provider so backups, failover, and patching are handled for you.


# Kubernetes
URL: https://docs.offrail.ai/self-host/kubernetes


Kubernetes is the deployment model we recommend for production. The gateway is stateless, so it scales horizontally with no coordination; pods self-heal; and the same manifests run on any cluster — EKS, GKE, AKS, or your own.

We publish an official **Helm chart** that deploys the gateway, API, UI, and worker with sane defaults and lets you wire in a managed Postgres and Redis through values.

## Prerequisites [#prerequisites]

* A Kubernetes cluster and `kubectl` configured to reach it
* [Helm 3](https://helm.sh/docs/intro/install/)
* A PostgreSQL database and Redis instance (use a managed service in production)

## Install the chart [#install-the-chart]

The chart is published as an OCI artifact on GitHub Container Registry:

```bash
helm install offrail oci://offrail/charts/offrail
```

This installs the latest published version. To pin to a specific release, append `--version <version>`, matching a published release tag without the `v` prefix (e.g. `1.2.3`).

## Configure with values [#configure-with-values]

Provide a `values.yaml` to point the chart at your managed database and cache and to set your secrets:

```yaml
config:
  AUTH_SECRET: "your-secret-key-here"
  GATEWAY_API_KEY_HASH_SECRET: "your-api-key-hash-secret-here"
  DATABASE_URL: "postgres://user:password@your-managed-host:5432/offrail"
  REDIS_URL: "redis://your-managed-host:6379"
  LLM_OPENAI_API_KEY: "sk-..."
  LLM_ANTHROPIC_API_KEY: "sk-ant-..."
```

```bash
helm install offrail oci://offrail/charts/offrail -f values.yaml
```

Provider keys support comma-separated values for load balancing and optional `__ENTERPRISE` / `__PLANS` suffix overrides used only for enterprise-plan or plan-based (Chat plan) organizations — see [Enterprise and plan provider env overrides](https://docs.offrail.ai/self-host/docker-compose#enterprise-and-plan-provider-env-overrides).

In production, store secrets in your cluster's secret manager rather than committing them to `values.yaml`. The cloud guides cover the recommended approach for each provider:

* [AWS](https://docs.offrail.ai/self-host/aws) — EKS with RDS, ElastiCache, and Secrets Manager
* [Google Cloud](https://docs.offrail.ai/self-host/gcp) — GKE with Cloud SQL, Memorystore, and Secret Manager
* [Azure](https://docs.offrail.ai/self-host/azure) — AKS with Azure Database for PostgreSQL, Azure Cache for Redis, and Key Vault

## Scaling the gateway [#scaling-the-gateway]

Because the gateway is stateless, scale it horizontally to match traffic — either by setting the replica count in values or with a `HorizontalPodAutoscaler` that targets CPU or request load. The API, UI, and worker serve your team rather than your traffic and rarely need more than one or two replicas.

See the [Helm chart README](https://bitbucket.org/designsai/offrailai/tree/main/infra/helm) for the full list of configurable values and the [list of available versions](https://bitbucket.org/designsai/offrailai/pkgs/container/charts%2Foffrail).


# Gateway Caching
URL: https://docs.offrail.ai/features/caching/gateway-caching


# Gateway Caching [#gateway-caching]

Gateway caching serves a previously-seen, byte-identical request entirely from OffRail without forwarding it to the upstream provider. Repeated identical calls cost **$0** — there is no inference and no provider charge. It is most useful for API workloads with deterministic inputs (classification, batch jobs, FAQ lookups, retries) rather than free-form chat.

<Callout type="info">
  If you want to reduce the cost of long, partially-shared prompts in chat apps
  or coding tools, you want [Provider Cache
  Control](https://docs.offrail.ai/features/caching/provider-cache-control) instead. That discounts the
  cached portion of your prompt on every call — it does not require
  byte-identical requests. See the [Caching Overview](https://docs.offrail.ai/features/caching) for a
  side-by-side comparison.
</Callout>

## How It Works [#how-it-works]

When you make an API request:

1. OffRail generates a cache key based on the request parameters
2. If a matching cached response exists, it's returned immediately
3. If no cache exists, the request is forwarded to the provider
4. The response is cached for future identical requests

This means repeated identical requests are served instantly from cache without incurring additional provider costs.

## Cost Savings [#cost-savings]

Caching can dramatically reduce costs for applications with repetitive requests:

| Scenario                    | Without Caching | With Caching | Savings |
| --------------------------- | --------------- | ------------ | ------- |
| 1,000 identical requests    | $10.00          | $0.01        | 99.9%   |
| 50% duplicate rate          | $10.00          | $5.00        | 50%     |
| Retry after transient error | $0.02           | $0.01        | 50%     |

<Callout type="info">
  Cached responses are free from provider costs. You only pay for the initial
  request that populates the cache.
</Callout>

## Requirements [#requirements]

<Callout type="info">
  Caching is **free** and **independent** of [Data
  Retention](https://docs.offrail.ai/features/data-retention). Cached responses live in a short-lived
  cache (TTL-bound, typically seconds to minutes) and are not stored as
  long-term request data — you do not need to enable data retention to use
  caching.
</Callout>

To use caching:

1. Enable **Caching** in your project settings under Preferences
2. Configure the cache duration (TTL) as needed
3. Make requests as normal—caching is automatic

## Cache Key Generation [#cache-key-generation]

The cache key is generated from these request parameters:

* Model identifier
* Messages array (roles and content)
* Temperature
* Max tokens
* Top P
* Tools/functions
* Tool choice
* Response format
* System prompt
* Other model-specific parameters

<Callout type="info">
  Requests with different parameter values, even slight variations, will not
  share cache entries.
</Callout>

## Cache Behavior [#cache-behavior]

### Cache Hits [#cache-hits]

When a cache hit occurs:

* Response is returned immediately (sub-millisecond latency)
* No provider API call is made
* No inference costs are incurred

### Cache Misses [#cache-misses]

When a cache miss occurs:

* Request is forwarded to the LLM provider
* Response is stored in cache
* Normal inference costs apply
* Future identical requests will hit the cache

## Streaming and Caching [#streaming-and-caching]

Caching works with both streaming and non-streaming requests:

* **Non-streaming**: Full response is cached and returned
* **Streaming**: The complete response is reconstructed from cache and streamed back

## Cache TTL (Time-to-Live) [#cache-ttl-time-to-live]

Cache duration is configurable per project in your project settings. You can set the cache TTL from 10 seconds up to 1 year (31,536,000 seconds).

The default cache duration is 60 seconds. Adjust this based on your use case—longer durations work well for static content, while shorter durations are better for frequently changing data.

## Identifying Cached Responses [#identifying-cached-responses]

A cache hit replays the stored response body, so it is byte-identical to the original — same `id`, same content. Two markers tell you it was a replay:

* the `x-offrail-cache: HIT` response header (set on every lane, including `/v1/messages`, which has no metadata envelope)
* `metadata.cached: true` on the response body (and on the final metadata chunk of a streamed replay)

All cost fields are zeroed on a replay, because no upstream call was made:

```json
{
	"usage": {
		"prompt_tokens": 12,
		"completion_tokens": 48,
		"total_tokens": 60,
		"cost": 0,
		"cost_details": {
			"total_cost": 0,
			"input_cost": 0,
			"output_cost": 0
		}
	},
	"metadata": {
		"cached": true
	}
}
```

Token counts are **not** zeroed — they still describe the completion you are receiving, and are what the dashboard records for analytics. Note that any prompt-caching fields (`prompt_tokens_details.cached_tokens`, `cache_write_tokens`) describe the *original* upstream call, not the replay.

## Bypassing the Cache for a Single Request [#bypassing-the-cache-for-a-single-request]

Send `x-no-cache: true` to skip the cache for one request — the call goes upstream and its response is not stored. Useful when a client retries an identical request and expects a fresh sample (for example an agent loop) without disabling caching for the whole project.

```bash
curl https://api.offrail.ai/v1/chat/completions \
  -H "Authorization: Bearer $OFFRAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-no-cache: true" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello"}]}'
```

## Use Cases [#use-cases]

### Development and Testing [#development-and-testing]

During development, you often send the same prompts repeatedly:

```typescript
// This prompt will only incur costs once
const response = await client.chat.completions.create({
	model: "gpt-4o",
	messages: [{ role: "user", content: "Explain quantum computing" }],
});
```

### Chatbots with Common Questions [#chatbots-with-common-questions]

FAQ-style interactions often have repeated questions:

```typescript
// Common questions are served from cache
const faqs = [
	"What are your business hours?",
	"How do I reset my password?",
	"What is your return policy?",
];
```

### Batch Processing [#batch-processing]

Processing large datasets with potentially duplicate items:

```typescript
// Duplicate items in batch are served from cache
for (const item of items) {
	const response = await client.chat.completions.create({
		model: "gpt-4o",
		messages: [{ role: "user", content: `Classify: ${item}` }],
	});
}
```

## Best Practices [#best-practices]

### Maximize Cache Hits [#maximize-cache-hits]

* Use consistent prompt formatting
* Normalize input data before sending
* Use deterministic parameters (temperature: 0)
* Avoid including timestamps or random values in prompts

### Appropriate Use Cases [#appropriate-use-cases]

Caching is most effective for:

* Static knowledge queries
* Classification tasks
* FAQ responses
* Development/testing
* Retry scenarios

### When to Avoid Caching [#when-to-avoid-caching]

Caching may not be suitable for:

* Real-time data requirements
* Highly personalized responses
* Time-sensitive information
* Creative tasks requiring variety
* Chat or coding tools where prompts overlap but are not byte-identical — use [Provider Cache Control](https://docs.offrail.ai/features/caching/provider-cache-control) instead

## Pricing [#pricing]

Caching is **completely free**. Cached responses are held in a short-lived
in-memory cache (bounded by your configured TTL) and do not incur storage
charges. Storage costs only apply if you separately enable [Data
Retention](https://docs.offrail.ai/features/data-retention) for full request/response payloads.

<Callout type="success">
  Caching reduces both inference cost and latency at no additional charge.
</Callout>


# Caching
URL: https://docs.offrail.ai/features/caching


# Caching [#caching]

OffRail supports **two distinct kinds of caching**, and they solve different problems. Pick the one that matches your workload — they can also be used together.

## Provider / Model Caching [#provider--model-caching]

The provider performs the caching. When your request reuses a long prefix from a previous call (a system prompt, conversation history, tool definitions, a long document), the model serves that prefix from its prompt cache and bills it at a reduced rate. New input tokens and **all output tokens are still billed at the normal rate** — only the cached portion is discounted.

This is the type of caching that powers efficient chat-based and assistant-based interactions, including chat apps and coding tools (Cursor, Cline, Claude Code, etc.) where the same context is reused turn after turn.

You see it in your usage as `prompt_tokens_details.cached_tokens`. For most providers it works automatically; some (notably Anthropic) also let you mark blocks explicitly with `cache_control` and choose a longer TTL.

→ &#x2A;*[Read the Provider Cache Control docs](https://docs.offrail.ai/features/caching/provider-cache-control)**

## Gateway Caching [#gateway-caching]

OffRail performs the caching. When a request is **byte-identical*&#x2A; to a previous one (same model, same messages, same parameters), the response is served from the gateway's cache without any provider call. Repeated identical calls cost **$0**.

This is most useful for deterministic API workloads — classification, batch jobs, FAQ lookups, retries — rather than free-form chat, because chat prompts almost always differ on the latest turn.

→ &#x2A;*[Read the Gateway Caching docs](https://docs.offrail.ai/features/caching/gateway-caching)**

## Which one do I want? [#which-one-do-i-want]

| If you…                                                         | Use                                                                                           |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Build a chat app, assistant, or coding tool                     | [Provider Cache Control](https://docs.offrail.ai/features/caching/provider-cache-control)                            |
| Send long system prompts or growing conversation history        | [Provider Cache Control](https://docs.offrail.ai/features/caching/provider-cache-control)                            |
| Want longer cache lifetimes than the provider default           | [Provider Cache Control](https://docs.offrail.ai/features/caching/provider-cache-control) (explicit `cache_control`) |
| Send the exact same request many times (batches, retries, FAQs) | [Gateway Caching](https://docs.offrail.ai/features/caching/gateway-caching)                                          |
| Want $0 on repeated calls instead of a discount                 | [Gateway Caching](https://docs.offrail.ai/features/caching/gateway-caching)                                          |

<Callout type="info">
  The two are not mutually exclusive. A coding tool can rely on provider caching
  for its long system prompt **and** enable gateway caching so that
  deterministic tool calls (e.g., file lookups) cost nothing on retry.
</Callout>


# Provider Cache Control
URL: https://docs.offrail.ai/features/caching/provider-cache-control


# Provider Cache Control [#provider-cache-control]

Most modern LLM providers offer **prompt caching**: when a request reuses a long prefix from a previous request (for example, a multi-thousand-token system prompt or a growing conversation history), the provider stores that prefix and serves it back at a steep discount on subsequent calls. Only the cached portion is discounted — new input tokens and all output tokens are still billed at the normal rate.

This is the behavior you see surfaced as `cached_tokens` in your usage payloads, and it is what makes chat apps, assistants, and coding tools (Cursor, Cline, Claude Code, etc.) economically viable on long contexts.

<Callout type="info">
  Looking for $0 on repeated calls instead of a discount on the cached portion?
  That is [Gateway Caching](https://docs.offrail.ai/features/caching/gateway-caching), which serves
  byte-identical requests entirely from OffRail without hitting the provider. It
  is a better fit for deterministic API workloads than for chat. See the
  [Caching Overview](https://docs.offrail.ai/features/caching) for a side-by-side comparison.
</Callout>

## Automatic caching [#automatic-caching]

For most users, prompt caching just works — you do not need to change your request payloads.

Providers including OpenAI, Anthropic (when prompts cross the provider's minimum size), Google, DeepSeek, xAI, and Alibaba inspect incoming requests for shared prefixes and cache them automatically. OffRail forwards the provider's cache metadata back to you in the response, and bills the cached portion at the model's `cached_input` rate.

For **Anthropic** and **AWS Bedrock Claude**, prompt caching is strictly opt-in via `cache_control` / `cachePoint` markers on the request body. To get automatic cache benefits without rewriting your requests, OffRail injects those markers for you on long system and user messages by default. If you send long prompts sporadically — with gaps wider than the 5-minute TTL — you may want to disable this entirely, since you would otherwise pay the cache-write premium (1.25× input for 5m, 2× for 1h) without ever benefiting from a cache read.

To disable, open **Project Settings → Caching → Provider Cache Writes** and turn off "Allow provider cache writes". When disabled, the gateway strips **all** `cache_control` markers from outgoing requests for the project — both the ones it adds automatically and any markers your client sends. This covers callers that always emit markers regardless of the user's request cadence (e.g. Claude Code, Cursor, Cline). The change takes up to 5 minutes to take effect due to the project-settings cache.

To take advantage of automatic caching:

* Put stable content (system prompt, instructions, tool definitions, long documents) at the **start** of your messages
* Keep the variable portion (the latest user turn) at the **end**
* Reuse the same prefix across requests — even minor changes invalidate the cache

You can confirm the cache is working by inspecting `usage.prompt_tokens_details.cached_tokens` on the response. See [Cost Breakdown](https://docs.offrail.ai/features/cost-breakdown) for the full list of usage fields.

```json
{
	"usage": {
		"prompt_tokens": 8200,
		"completion_tokens": 150,
		"prompt_tokens_details": {
			"cached_tokens": 8000
		},
		"cost_details": {
			"input_cost": 0.0006,
			"cached_input_cost": 0.0008
		}
	}
}
```

In this example, 8,000 of the 8,200 prompt tokens were served from the provider's cache and billed at the cached rate.

### Pricing and routing [#pricing-and-routing]

Cached input tokens are billed at the model's published `cached_input` price (typically 10–25% of the regular input price, depending on the provider and model). Output tokens and any non-cached input tokens are billed at the normal rate.

When the [Smart Routing](https://docs.offrail.ai/features/routing) algorithm selects a provider for a large prompt (≥ 5,000 estimated tokens), it gives extra weight to providers that advertise cache support, since caching can substantially reduce the cost of repeated large prompts.

## Explicit caching with `cache_control` [#explicit-caching-with-cache_control]

Some providers — most notably **Anthropic** — also support *explicit* cache control, where you mark specific content blocks as cacheable using a `cache_control` field. This gives you precise control over what gets cached and lets you opt into longer cache lifetimes than the default.

Explicit caching is provider-specific. Supported providers and TTLs at the time of writing:

| Provider             | Models                         | Supported TTLs       |
| -------------------- | ------------------------------ | -------------------- |
| Anthropic (Claude)   | All Claude models              | `5m` (default), `1h` |
| AWS Bedrock (Claude) | All Claude models              | `5m` (default), `1h` |
| Alibaba (Qwen)       | Qwen models with cache support | Provider-defined     |

To mark content as cacheable, send the message content as an array of blocks and add a `cache_control` field to the block you want to cache:

```json
{
	"model": "claude-haiku-4-5",
	"messages": [
		{
			"role": "system",
			"content": [
				{
					"type": "text",
					"text": "You are a helpful assistant. <long instructions...>",
					"cache_control": { "type": "ephemeral", "ttl": "1h" }
				}
			]
		},
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
}
```

Use `ttl: "5m"` (the default if omitted) for short-lived caches that match a single user's session, and `ttl: "1h"` when the same prefix will be reused over a longer window (for example, a coding agent that keeps the same project context warm across many requests).

The same `cache_control` markers work on the native [`/v1/messages` endpoint](https://docs.offrail.ai/features/anthropic-endpoint) (Anthropic request format), where cache usage comes back in Anthropic's own fields: `usage.cache_creation_input_tokens` (written this request) and `usage.cache_read_input_tokens` (served from cache).

### Minimum cacheable prompt length [#minimum-cacheable-prompt-length]

Every Claude model has a **minimum prompt length below which nothing is cached**. A `cache_control` marker on a shorter prompt is accepted without error, but the provider silently skips the cache write — the response reports `cache_creation_input_tokens: 0` and `cache_read_input_tokens: 0` on every call, no matter how often you repeat the request. This is Anthropic's documented behavior, not a dropped breakpoint; you would see exactly the same result calling Anthropic directly.

The threshold counts **all tokens up to and including the marked block** (tools, system, and preceding messages), and it varies by model generation:

| Models                                                | Minimum cacheable tokens |
| ----------------------------------------------------- | ------------------------ |
| Opus 4.5+, Sonnet 5, Haiku 4.5                        | 4,096                    |
| Sonnet 4.6, Haiku 3.5                                 | 2,048                    |
| Sonnet 4.5 and earlier, Opus 4.1 and earlier, Haiku 3 | 1,024                    |

The models endpoint (`GET https://api.offrail.ai/v1/models`) exposes the exact threshold as `min_cacheable_tokens` on each provider entry, so clients can check it programmatically. OffRail's automatic marker injection uses the same threshold, which is why short system prompts never trigger automatic cache writes either.

### Verifying cache writes and the write premium [#verifying-cache-writes-and-the-write-premium]

The first request that creates a cache entry is billed at the provider's cache-write rate (1.25× input for `5m`, 2× for `1h`). On the OpenAI-compatible `/v1/chat/completions` endpoint, OffRail surfaces the write side in extended usage fields — OpenAI's standard format only has `cached_tokens` for reads, so these are gateway extensions:

```json
{
	"usage": {
		"prompt_tokens": 5232,
		"prompt_tokens_details": {
			"cached_tokens": 0,
			"cache_write_tokens": 5222,
			"cache_creation_tokens": 5222,
			"cache_creation": {
				"ephemeral_5m_input_tokens": 5222,
				"ephemeral_1h_input_tokens": 0
			}
		},
		"cost_details": {
			"input_cost": 0.00003,
			"cache_write_input_cost": 0.0065275,
			"cached_input_cost": 0
		}
	}
}
```

`cache_write_tokens` (and its alias `cache_creation_tokens`) counts the prompt tokens written into the provider cache this request, `cache_creation` breaks the write down by TTL when both rates are in play, and `cost_details.cache_write_input_cost` is the exact USD amount billed at the write premium. A successful cache write on call 1 shows up as `cache_write_tokens > 0`; the matching read on call 2 shows up as `cached_tokens > 0` with `cached_input_cost` at the discounted rate.

### Mixing explicit markers with automatic injection [#mixing-explicit-markers-with-automatic-injection]

Anthropic requires cache breakpoints with longer TTLs to appear before shorter ones (blocks are processed in the order `tools`, `system`, `messages`). The markers OffRail injects automatically use the default 5-minute TTL, so they could never legally precede an explicit `ttl: "1h"` marker in your messages. To keep both features compatible:

* When your request contains an explicit `ttl: "1h"` marker in the **messages**, OffRail skips its automatic marker injection for that request entirely and forwards only your markers — the same behavior you would get calling the provider directly.
* A `ttl: "1h"` marker only on the **system** prompt does not disable automatic injection, since 5-minute breakpoints after it still satisfy the ordering rule.
* Explicit markers that use the default 5-minute TTL coexist with automatic injection (capped at 4 breakpoints total per Anthropic's limit).

<Callout type="warning">
  Cache writes are billed at a premium (typically 1.25x for 5m and 2x for 1h on
  Anthropic) the first time a cached block is created. After that, cache reads
  cost roughly 10% of the regular input price. The break-even point is usually
  one or two reuses — explicit caching is worth it whenever a marked block will
  be sent more than once within its TTL.
</Callout>

Anthropic returns a per-TTL breakdown of cache writes when you mix `5m` and `1h` blocks:

```json
{
	"usage": {
		"cache_creation": {
			"ephemeral_5m_input_tokens": 0,
			"ephemeral_1h_input_tokens": 8000
		},
		"cache_read_input_tokens": 0
	}
}
```

For providers that publish a separate explicit-cache read rate (for example, Alibaba Qwen charges 10% for explicit cache reads vs. 20% for automatic cache reads), OffRail detects the `cache_control` markers on your request and applies the explicit rate automatically.

## Related [#related]

* [Gateway Caching](https://docs.offrail.ai/features/caching/gateway-caching) — serve byte-identical requests entirely from OffRail at $0 cost
* [Caching Overview](https://docs.offrail.ai/features/caching) — side-by-side comparison of provider caching vs. gateway caching
* [Cost Breakdown](https://docs.offrail.ai/features/cost-breakdown) — full reference for the usage and cost fields on every response
* [Smart Routing](https://docs.offrail.ai/features/routing) — how cache support influences provider selection for large prompts


# Microsoft Entra ID
URL: https://docs.offrail.ai/features/sso/entra


# Microsoft Entra ID [#microsoft-entra-id]

This guide walks an organization admin through connecting **Microsoft Entra ID** (formerly Azure Active Directory) to OffRail for SAML single sign-on and SCIM provisioning.

<Callout type="info">
  SSO & SCIM require the **Enterprise** plan and an organization **owner** or
  **admin**. See the [SSO overview](https://docs.offrail.ai/features/sso) for prerequisites and how the
  URLs are derived.
</Callout>

## Finding "Enterprise applications" in Entra [#finding-enterprise-applications-in-entra]

An **Enterprise application** is a registered instance of an app that your tenant trusts for sign-in. You'll create one to represent OffRail; its &#x2A;*Single sign-on (SAML)** page is where you exchange URLs and download the certificate.

<Callout type="warn">
  This is **not** the same as **App registrations** (that section is for apps
  you are developing). Use **Enterprise applications**. They are siblings under
  **Applications** and easy to confuse.
</Callout>

<Steps>
  <Step>
    ### Open the Entra admin center [#open-the-entra-admin-center]

    Go to [entra.microsoft.com](https://entra.microsoft.com) and sign in with an account that has at least the **Cloud Application Administrator** or **Application Administrator** role (Global Administrator also works). A regular member account won't see these options.
  </Step>

  <Step>
    ### Navigate to Enterprise applications [#navigate-to-enterprise-applications]

    In the left-hand navigation, click **Entra ID** (older tenants label this group **Identity**), then expand **Applications** and click **Enterprise applications** (older label: **Enterprise apps**).

    You're now on the **All applications** list.

    <Callout type="info">
      Prefer the classic Azure portal? Go to
      [portal.azure.com](https://portal.azure.com) → search **Microsoft Entra ID** →
      open it → left menu **Enterprise applications**. Both portals reach the same
      app.
    </Callout>
  </Step>

  <Step>
    ### Create the application [#create-the-application]

    Click **+ New application*&#x2A; → **+ Create your own application**. Name it (e.g. `OffRail&#x60;), choose &#x2A;*"Integrate any other application you don't find in the gallery (Non-gallery)"**, then **Create**. Wait a few seconds for it to provision.

    <Callout type="warn">
      Don't search the Entra **gallery** for "OffRail" — there is no gallery
      listing, so always use **Create your own application → Non-gallery**. You
      configure SAML and SCIM manually with the URLs from your **SSO** page; the
      steps below cover everything.
    </Callout>
  </Step>

  <Step>
    ### Open the SAML SSO page [#open-the-saml-sso-page]

    On the app's **Overview**, use the app's own left menu → **Single sign-on**. On the **Select a single sign-on method** screen, choose **SAML**.

    This opens the 5-section **Set up Single Sign-On with SAML** page referenced throughout the rest of this guide.
  </Step>
</Steps>

## Configure SAML SSO [#configure-saml-sso]

Because Entra needs OffRail's URLs and OffRail needs Entra's Login URL and certificate, the two systems exchange values. Follow this order.

<Steps>
  <Step>
    ### Choose a connection slug [#choose-a-connection-slug]

    In OffRail (**Organization → SSO → Add a connection**), set **Identity provider** to **Microsoft Entra ID**, then pick a **connection slug** (lowercase letters, numbers, hyphens; `^[a-z0-9-]+$`, up to 64 chars). We recommend the format `<your-org-slug>-<provider>`, e.g. `acme-entra` — the form pre-fills this from your organization name and the selected provider, so you can accept it or edit it. It must be unique across all OffRail organizations, and each organization has exactly one SSO connection. Keep it stable: your SP URLs are built from it.

    As soon as the slug is set, the form shows your two SP URLs (with the hosted default `https://internal.offrail.ai`) — copy them from there:

    * **Identifier (Entity ID):** `https://internal.offrail.ai/auth/sso/saml2/sp/metadata?providerId=acme-entra`
    * **Reply URL (ACS):** `https://internal.offrail.ai/auth/sso/saml2/sp/acs/acme-entra`

    You'll paste these into Entra next — no need to create the connection yet.
  </Step>

  <Step>
    ### Set Basic SAML Configuration in Entra [#set-basic-saml-configuration-in-entra]

    On the Entra &#x2A;*Single sign-on (SAML)** page, **Section 1 – Basic SAML Configuration** → **Edit**:

    * Under &#x2A;*Identifier (Entity ID)**, click **Add identifier** and paste your metadata URL from the previous step.
    * Under &#x2A;*Reply URL (Assertion Consumer Service URL)**, click **Add reply URL** and paste your ACS URL.
    * Leave **Sign on URL**, **Relay State**, and **Logout Url** blank.

    Click **Save** (the button enables once both required fields are filled), then close the panel.

    <Callout type="warn">
      The Reply URL row has a small numeric **Index** column next to the URL field.
      Paste the ACS URL into the **Enter a reply URL*&#x2A; box only — if it lands in the
      Index box you'll see &#x2A;"The value must be a valid number."* Leave Index blank.
    </Callout>
  </Step>

  <Step>
    ### Confirm the claims [#confirm-the-claims]

    **Section 2 – Attributes & Claims**: the Entra defaults are fine. In **Microsoft Entra ID** mode, OffRail reads email/name from the standard claim URIs Entra sends by default (`.../emailaddress`, `.../givenname`, `.../surname`, `.../name`), so you don't need to force the Name ID to be the email — just make sure users have a real `mail`/UPN.
  </Step>

  <Step>
    ### Grab the certificate and Login URL [#grab-the-certificate-and-login-url]

    * **Section 3 – SAML Certificates** → next to &#x2A;*Certificate (Base64)**, click **Download**. The `.cer` file's contents are a PEM block (`-----BEGIN CERTIFICATE----- … -----END CERTIFICATE-----`). Use &#x2A;*Certificate (Base64)**, not Raw or the Federation Metadata XML.
    * **Section 4 – Set up \<app name>** → copy the **Login URL** (looks like `https://login.microsoftonline.com/<tenant-id>/saml2`).

    Keep both values handy for the next step.

    <Callout type="info">
      The certificate **Download** links stay greyed out until Section 1's required
      fields are filled and saved — so complete the previous step first. The
      auto-generated signing certificate is valid for three years; its thumbprint
      and expiry are shown in Section 3.
    </Callout>
  </Step>

  <Step>
    ### Create the connection in OffRail [#create-the-connection-in-offrail]

    Back on the **SSO → Add a connection** form:

    * **Identity provider:** **Microsoft Entra ID** — important, this switches on the Entra claim mapping and relaxes the NameID format
    * **Connection slug:** `acme-entra` (must match the slug whose URLs you pasted into Entra)
    * **Email domains:** your corporate domain(s), e.g. `acme.com` (comma-separate for several)
    * **Identity Provider Single Sign-On URL:** the Entra **Login URL**
    * **X.509 signing certificate:** paste the PEM block

    Save. The connection card then shows the exact SP metadata + ACS URLs — confirm they match what you entered in Entra **Section 1**. You can edit the email domain list on the connection card later without re-creating the connection.

    <Callout type="warn">
      The domain list must include **every domain Entra may send as a user's
      email**. OffRail signs a user in from the `emailaddress&#x60; claim, which Entra
      fills from the &#x2A;*`mail`** attribute — and that domain can differ from the
      sign-in (userPrincipalName) domain, e.g. UPN `jane@acme.local` with mail
      `jane@acme.com`. If they differ, list **both** (`acme.local, acme.com`);
      otherwise sign-in bounces back to the login page with an `account_not_linked`
      error even though Entra authenticated the user.
    </Callout>
  </Step>
</Steps>

## Assign users to the app [#assign-users-to-the-app]

Configuring SAML isn't enough on its own — Entra only issues a SAML assertion for users who are **assigned** to the application. On the Enterprise App → **Users and groups → + Add user/group**, add the people (or groups) who should be able to sign in, then **Assign**.

<Callout type="warn">
  If you skip this, sign-in fails at Entra with &#x2A;"…is not assigned to a role for
  the application"* (error `AADSTS50105`) — even though the SAML config is
  correct. Assign at least your own account before testing.
</Callout>

If you also set up SCIM below, assigning users there covers this same requirement; but for SSO **without** SCIM this assignment step is mandatory.

## Configure SCIM provisioning [#configure-scim-provisioning]

SSO only signs people in. SCIM is what actually adds members to the organization. Set it up on the **same** Enterprise App.

<Steps>
  <Step>
    ### Generate a SCIM token [#generate-a-scim-token]

    In OffRail, on &#x2A;*Organization → SSO → Directory sync (SCIM)**, click **Generate SCIM token** and copy it (shown once). The SCIM base URL is shown in the same card — it's `https://internal.offrail.ai/scim/v2` on the hosted default.
  </Step>

  <Step>
    ### Add admin credentials in Entra [#add-admin-credentials-in-entra]

    In Entra, on your Enterprise App → **Provisioning → Automatic → Admin Credentials**:

    * **Authentication method:*&#x2A; choose the token option — labelled &#x2A;*"Long-lived bearer token"*&#x2A; (or &#x2A;*"Secret token"**). **Do not** pick "OAuth 2.0 code grant flow": OffRail's SCIM endpoint authenticates a plain static bearer token (`Authorization: Bearer <token>`), not an OAuth code exchange. The presence of a **Tenant URL + Secret token** pair on the form confirms you're on the right method.
    * **Tenant URL:** the SCIM base URL, e.g. `https://internal.offrail.ai/scim/v2`
    * **Secret Token:** the token from the previous step
    * Click **Test Connection** (Entra calls `/ServiceProviderConfig` with the bearer token), then **Save**.

    A green check confirms the credentials are valid.

    <Callout type="info">
      Entra rolled out a redesigned provisioning UX that adds the &#x2A;*"Select
      authentication method"** dropdown shown above. The classic UX has no dropdown
      and just shows Tenant URL + Secret token (which is the bearer method). Either
      is fine — pick the bearer/secret-token option and the steps are identical.
    </Callout>
  </Step>

  <Step>
    ### Assign users and start provisioning [#assign-users-and-start-provisioning]

    * **Mappings:** the Entra SCIM defaults are fine (`userPrincipalName → userName`, `mail → emails[work]`, `givenName/surname → name.*`, `accountEnabled → active`, group provisioning on).
    * **Settings → Scope:** "Sync only assigned users and groups".
    * Under **Users and groups**, assign the people/groups who should get access.
    * Set **Provisioning Status = On** and Save. Use **Provision on demand** to test a single user instantly.

    Entra then begins syncing assigned users on its provisioning cycle.
  </Step>
</Steps>

Assigned users are created in OffRail and added to the organization (default role **developer**). Deactivating or unassigning a user in Entra removes their membership.

## Group → role mapping (optional) [#group--role-mapping-optional]

On the **SSO** page, add mappings such as `Acme Admins → admin`. When Entra pushes group membership over SCIM, each member gets the **highest** mapped role among their groups (default `developer`). Owners are never auto-demoted, so you can't accidentally lock yourself out of the owner role.

## Default project access (optional) [#default-project-access-optional]

Provisioned members join as **developers**, who only see the projects they're granted (owners/admins always see every project). Use the **Default project access** card on the **SSO** page to choose which projects newly provisioned SSO/SCIM developers get by default. If you leave it unconfigured, new members get your organization's first (default) project so they can see something out of the box; select an empty set to instead grant nothing and manage project access manually per member.

<Callout type="info">
  Default project access applies only to members provisioned **after** you set
  it — it doesn't retroactively change existing members' project grants. Adjust
  an existing member's projects from the **Team** page.
</Callout>

## Test the login [#test-the-login]

Log out → login page → **Sign in with SSO** → enter a work email (`someone@acme.com`). OffRail resolves the connection by email domain, redirects to Entra, and on return the (SCIM-provisioned) user is signed in and lands in the enterprise organization.

## Enforce SSO (optional, do this last) [#enforce-sso-optional-do-this-last]

Flip **Require SSO** on the connection to require SSO for that email domain — password, social, and passkey sign-in are then rejected for it.

<Callout type="warn">
  Only enable **Require SSO** after the test above passes, and remember it also
  applies to the admin setting it up if their email is on that domain — verify
  your own SSO login works first.
</Callout>

## Troubleshooting: login bounces back to the login page [#troubleshooting-login-bounces-back-to-the-login-page]

If the Entra sign-in itself succeeds but the user lands back on the OffRail login page (with a 401 in the browser console), the SAML response was received but OffRail refused to attach it to the provisioned user — almost always an `account_not_linked` domain mismatch:

* **The asserted email's domain isn't in the connection's Email domains list.** Entra sends the user's `mail` attribute as the email; if its domain differs from the one you registered (e.g. you registered the UPN domain), edit the connection's **Email domains** on the SSO page to include both.
* **Wrong Identity provider type.** If the connection was created as &#x2A;*Other (SAML 2.0)** instead of **Microsoft Entra ID**, the Entra claim mapping isn't applied and the email falls back to the NameID (usually the UPN), which may again be off-list — and creates duplicate users instead of matching the SCIM-provisioned ones.

## Troubleshooting: can't find "Enterprise applications" [#troubleshooting-cant-find-enterprise-applications]

* **Insufficient role** — you need Application Administrator, Cloud Application Administrator, or Global Administrator. Ask whoever administers your Microsoft 365 / Entra tenant.
* **Wrong menu** — you're in **App registrations** (for developers) instead of **Enterprise applications**.
* **Wrong tenant** — if you belong to multiple tenants, use **Settings (gear) → Directories + subscriptions** (top-right) to switch to the tenant you administer.


# Google Workspace
URL: https://docs.offrail.ai/features/sso/google


# Google Workspace [#google-workspace]

Google Workspace works differently from the SAML providers: employees already sign in to OffRail with the standard **Sign in with Google** button using their Workspace-managed Google account. Instead of registering a SAML application, you configure an **auto-join email domain** — anyone who signs in with a Google account on that domain is added to your organization automatically.

<Callout type="info">
  Google Workspace auto-join requires the **Enterprise** plan and is configured
  by an organization **owner** or **admin**. See the [SSO
  overview](https://docs.offrail.ai/features/sso) for the other connection types.
</Callout>

## How it works [#how-it-works]

1. An admin sets the organization's auto-join domain (e.g. `acme.com`) on the **SSO** page.
2. A user signs in (or signs up) with **Sign in with Google** using their `@acme.com` Workspace account.
3. OffRail matches the verified email domain and adds them to the organization with the **developer** role.
4. The user receives an email letting them know they've been added.

This applies to **new signups and existing users** alike — an existing OffRail user on your domain joins the organization on their next Google sign-in. Members are never added twice, and members you've already invited are unaffected.

<Callout type="info">
  Auto-join is **just-in-time provisioning**: users join when they sign in.
  Google Workspace does not support SCIM provisioning to custom apps, so there
  is no directory sync — removing someone from your Workspace does not remove
  them from the organization. Offboard members on the **Team** page.
</Callout>

## Setting it up [#setting-it-up]

<Steps>
  <Step>
    ### Open the SSO page [#open-the-sso-page]

    In the dashboard, go to your organization's **SSO** page and find the **Connections** card.
  </Step>

  <Step>
    ### Add a Google Workspace connection [#add-a-google-workspace-connection]

    Under **Add a connection**, choose **Google Workspace** as the identity provider. There are no SAML URLs or certificates to exchange — only your email domain.
  </Step>

  <Step>
    ### Enter your email domain [#enter-your-email-domain]

    Enter the corporate domain your Workspace accounts use, e.g. `acme.com`, and click **Enable auto-join**.

    * Only **one domain** per organization.
    * A domain can be claimed by **only one organization** — if another organization already uses it, you'll get a conflict error.
    * **Consumer email domains** (`gmail.com`, `outlook.com`, `yahoo.com`, …) are rejected.
  </Step>

  <Step>
    ### Verify with a test account [#verify-with-a-test-account]

    Sign in with a Google account on your domain that isn't yet a member. After signing in, it should appear on the **Team** page with the **developer** role, and receive a notification email.
  </Step>
</Steps>

## Managing the connection [#managing-the-connection]

The Google Workspace connection appears as a card in the **Connections** list:

* **Edit** the domain with the pencil icon — existing members are unaffected; the new domain applies to future sign-ins.
* **Remove** the connection to stop auto-joining. Existing members stay in the organization.

Auto-joined members get the **developer** role and receive the organization's **Default project access** selection, exactly like SSO/SCIM-provisioned members. Promote them or adjust project access on the **Team** page if they need more.

## What's not available with Google Workspace [#whats-not-available-with-google-workspace]

With a Google-only setup, the &#x2A;*Directory sync (SCIM)** and **Group role mapping** cards on the SSO page are disabled:

* **SCIM** — Google Workspace doesn't support SCIM provisioning for custom apps, so there's no directory sync to configure.
* **Group role mapping** — role mappings apply to groups pushed via SCIM, so they can't take effect either. Auto-joined members always start as **developer**.

**Default project access** works as usual and applies to auto-joined members.

## Interaction with SAML and Require SSO [#interaction-with-saml-and-require-sso]

An organization has **one connection** for now — either a SAML connection or Google Workspace auto-join. To switch providers, remove the existing connection first.

For context on why they don't mix: a SAML connection's **Require SSO** toggle blocks password, social, **and Google** sign-in for its domains, so Google sign-in would be rejected before auto-join could run. Pick the model that fits:

* **Google Workspace auto-join** — zero IdP setup, members use the Google button, join on first sign-in.
* **SAML with Require SSO** — stricter control (blocks all non-SSO sign-in), needs a SAML app in your IdP. Google Workspace can also act as a SAML 2.0 IdP via a custom SAML app if you need enforcement; use the &#x2A;*Other (SAML 2.0)** connection type for that.


# SSO
URL: https://docs.offrail.ai/features/sso


# SSO [#sso]

OffRail supports per-organization &#x2A;*SAML 2.0 single sign-on (SSO)** and **SCIM 2.0 directory provisioning**, so an enterprise team can let members sign in with their corporate identity provider and keep membership in sync automatically.

<Callout type="info">
  SSO & SCIM are available on the **Enterprise** plan only, and can be
  configured by an organization **owner** or **admin**. Until the organization
  is on the Enterprise plan, the **SSO** page shows a contact-sales card.
  Contact us at [contact@offrail.ai](mailto:contact@offrail.ai) to enable it.
</Callout>

## What each part does [#what-each-part-does]

* **SSO (authentication)** — lets your members sign in with your identity provider (Microsoft Entra ID, Okta, or any SAML 2.0 IdP). Sign-in is routed by **email domain**: a user clicks **Sign in with SSO**, enters their work email, and is redirected to the correct provider. A connection can list **multiple domains** (comma-separated, editable on the connection card) — the list must cover every domain the IdP may assert as a user's email, e.g. both the UPN and `mail` domains on Entra when they differ. A first-time SSO sign-in automatically joins the user to the organization that owns the connection as a **developer** with the configured default project access — it does not create the organization itself.
* **SCIM (provisioning)** — pushes user and group create / update / deactivate / delete from your directory into the organization, so people are added and removed automatically. Provisioned members default to the **developer** role.
* **Group → role mapping*&#x2A; &#x2A;(optional)* — map directory groups to organization roles (`admin` / `developer`). On any membership change, a member receives the **highest** mapped role among their groups. Owners are never auto-demoted.
* **Default project access*&#x2A; &#x2A;(optional)* — choose which projects newly provisioned SSO/SCIM members (the `developer` role) can access by default. Owners/admins always see every project, so this only affects developers. Left unconfigured, new members get the organization's first (default) project. Applies to members provisioned after you set it, not retroactively.

## Prerequisites [#prerequisites]

1. **Membership is granted on sign-in.** Anyone your IdP authenticates for the connection joins the organization on their first SSO sign-in (as a **developer**). Members can also join earlier through normal signup/invite or by being pushed via SCIM.
2. **The organization is on the Enterprise plan.** An admin flips the org to `enterprise` to unlock the feature.
3. **`SSO_ENABLED=true` on the deployment.** This global env flag controls whether the **Sign in with SSO** button appears on the login page. The admin config page works without it, but end users can't log in via SSO until it's on.

## The URLs OffRail gives your IdP [#the-urls-offrail-gives-your-idp]

Everything your identity provider needs is derived from the **management API base URL** and a **Connection ID** you choose. On the hosted service the base URL defaults to `https://internal.offrail.ai` (overridable via `API_URL`); self-hosted deployments use whatever `API_URL` is set to.

<Callout type="warn">
  The management API base URL is **not** `api.offrail.ai` — that host serves the
  `/v1` chat-completions gateway. SSO/SAML and SCIM live on the management API.
  You don't need to construct these URLs by hand: the exact, fully-qualified
  values are shown with copy buttons on the **SSO** page and on each connection
  card.
</Callout>

With `API_URL = https://internal.offrail.ai` and a connection slug of `acme-entra`:

| Value                             | URL                                                                            |
| --------------------------------- | ------------------------------------------------------------------------------ |
| Identifier (Entity ID) / Audience | `https://internal.offrail.ai/auth/sso/saml2/sp/metadata?providerId=acme-entra` |
| Reply URL (ACS)                   | `https://internal.offrail.ai/auth/sso/saml2/sp/acs/acme-entra`                 |
| SCIM Tenant URL                   | `https://internal.offrail.ai/scim/v2`                                          |

## SCIM authentication [#scim-authentication]

SCIM uses a **static bearer token**, not OAuth. You generate the token once on the &#x2A;*SSO → Directory sync (SCIM)** card, and your IdP sends it as `Authorization: Bearer <token>` on every request. In your IdP's provisioning config, that means:

* **Authentication method:** the **bearer token*&#x2A; option — Entra labels it &#x2A;*"Long-lived bearer token"*&#x2A; / &#x2A;*"Secret token"**; Okta calls it an **API token**. Do **not** choose an OAuth 2.0 authorization-code flow — OffRail does not implement one, and the connection test will fail.
* **Tenant / base URL:** the SCIM Tenant URL above (`…/scim/v2`).
* **Secret token:** the value from the SCIM card (shown once; rotate to reissue).

The token is scoped to a single organization, so every user and group your IdP pushes lands in that org.

## Provider guides [#provider-guides]

* [Microsoft Entra ID](https://docs.offrail.ai/features/sso/entra) — set up SAML SSO and SCIM provisioning with Entra ID (formerly Azure AD).
* [Okta](https://docs.offrail.ai/features/sso/okta) — set up SAML SSO and SCIM provisioning with a custom Okta app integration.
* [Google Workspace](https://docs.offrail.ai/features/sso/google) — no SAML app needed: members sign in with the standard **Sign in with Google** button and auto-join your organization by email domain.

Any other SAML 2.0 IdP follows the same flow; pick the matching **Identity provider** type when you create the connection so the correct claim mapping is applied.

## End-user sign-in flow [#end-user-sign-in-flow]

1. On the login page the user clicks **Sign in with SSO** (visible when `SSO_ENABLED=true`).
2. They enter their work email, e.g. `jane@acme.com`.
3. OffRail resolves the domain `acme.com` → finds the org's SSO connection → redirects to the org's IdP.
4. The user authenticates at their IdP (usually instant if already signed in).
5. They return signed in, landing in the enterprise organization.

## Enforcing SSO (optional, do this last) [#enforcing-sso-optional-do-this-last]

Each connection has a **Require SSO** toggle. When on, anyone whose email domain matches the connection can *only* sign in via SSO — password, social, and passkey sign-in are all rejected for that domain.

<Callout type="warn">
  Only turn on **Require SSO** after you've confirmed SSO login works
  end-to-end. Enforcing before the IdP works can lock out the whole domain —
  including the admin setting it up if their own email is on that domain.
</Callout>


# Okta
URL: https://docs.offrail.ai/features/sso/okta


# Okta [#okta]

This guide walks an organization admin through connecting **Okta** to OffRail for SAML single sign-on and SCIM provisioning.

<Callout type="info">
  SSO & SCIM require the **Enterprise** plan and an organization **owner** or
  **admin**. See the [SSO overview](https://docs.offrail.ai/features/sso) for prerequisites and how the
  URLs are derived.
</Callout>

## Create the SAML app integration [#create-the-saml-app-integration]

You'll create a custom SAML 2.0 app integration in the Okta Admin Console to represent OffRail. There is no OIN (Okta Integration Network) catalog listing — the custom app is the supported path, and the steps below cover everything.

<Steps>
  <Step>
    ### Choose a connection slug in OffRail [#choose-a-connection-slug-in-offrail]

    In OffRail (**Organization → SSO → Add a connection**), set **Identity provider** to **Okta**, then pick a **connection slug** (lowercase letters, numbers, hyphens; `^[a-z0-9-]+$`, up to 64 chars). We recommend the format `<your-org-slug>-<provider>`, e.g. `acme-okta` — the form pre-fills this from your organization name and the selected provider. It must be unique across all OffRail organizations, and each organization has exactly one SSO connection. Keep it stable: your SP URLs are built from it.

    As soon as the slug is set, the form shows your two SP URLs (with the hosted default `https://internal.offrail.ai`) — copy them from there:

    * **Audience URI (SP Entity ID):** `https://internal.offrail.ai/auth/sso/saml2/sp/metadata?providerId=acme-okta`
    * **Single sign-on URL (ACS):** `https://internal.offrail.ai/auth/sso/saml2/sp/acs/acme-okta`

    You'll paste these into Okta next — no need to create the connection yet.
  </Step>

  <Step>
    ### Open the Okta Admin Console [#open-the-okta-admin-console]

    Go to your Okta admin console (`https://<your-org>-admin.okta.com`) with an account that has the **Application Administrator** role (or Super Admin). In the left navigation, open **Applications → Applications**.
  </Step>

  <Step>
    ### Create the app integration [#create-the-app-integration]

    Click **Create App Integration**, choose **SAML 2.0** as the sign-in method, and click **Next**. On **General Settings**, set the **App name** (e.g. `OffRail`) and click **Next**.

    <Callout type="warn">
      Pick **SAML 2.0**, not **OIDC**: OffRail's per-organization SSO speaks SAML.
      Don't use **Browse App Catalog** either — there is no gallery listing, so
      always go through **Create App Integration**.
    </Callout>
  </Step>

  <Step>
    ### Configure SAML settings [#configure-saml-settings]

    On the **Configure SAML** step, fill in **Section A – SAML Settings → General**:

    * **Single sign-on URL:*&#x2A; your ACS URL from step 1. Keep &#x2A;*"Use this for Recipient URL and Destination URL"** checked.
    * **Audience URI (SP Entity ID):** your metadata URL from step 1.
    * **Default RelayState:** leave blank.
    * **Name ID format:** **EmailAddress** — required, see below.
    * **Application username:** **Email**.
    * **Update application username on:** Create and update (the default).

    The **Show Advanced Settings** defaults are correct (Response **Signed**, Assertion Signature **Signed**, RSA-SHA256) — OffRail requires signed assertions, which Okta does by default.

    Click **Next**, and on the **Feedback*&#x2A; step select &#x2A;*"This is an internal app that we have created"**, then **Finish**.

    <Callout type="warn">
      **Name ID format = EmailAddress** and **Application username = Email** are
      what make sign-in work: in **Okta** mode, OffRail reads the user's email from
      the SAML **NameID**. You don't need any attribute statements (recent versions
      of the wizard don't even show that section — that's fine); if you do add them,
      `displayName`, `givenName` and `surname` are picked up for the user's profile
      name.
    </Callout>
  </Step>

  <Step>
    ### Grab the Sign-on URL and certificate [#grab-the-sign-on-url-and-certificate]

    After the wizard finishes you land on the app's **Sign On** tab. Under **Settings → Sign on methods → SAML 2.0 → Metadata details**, click **More details** to reveal:

    * **Sign on URL** — looks like `https://<your-org>.okta.com/app/<app-label>/<appId>/sso/saml`, where `<appId>` starts with `exk`. Copy it.
    * **Signing Certificate** — click **Copy** (or **Download**; the file contents are a PEM block `-----BEGIN CERTIFICATE----- … -----END CERTIFICATE-----`).

    <Callout type="warn">
      Use this exact app-specific **Sign on URL** — OffRail derives the SAML
      issuer it expects (`http://www.okta.com/<appId>`) from the `exk…` segment of
      the URL. Pasting some other Okta URL (your org homepage, the metadata URL,
      an Identity-Provider route) makes every SAML response fail issuer
      validation.
    </Callout>
  </Step>

  <Step>
    ### Create the connection in OffRail [#create-the-connection-in-offrail]

    Back on the **SSO → Add a connection** form:

    * **Identity provider:** **Okta** — this selects the email-NameID mapping and issuer derivation described above
    * **Connection slug:** `acme-okta` (must match the slug whose URLs you pasted into Okta)
    * **Email domains:** your corporate domain(s), e.g. `acme.com` (comma-separate for several)
    * **Identity Provider Single Sign-On URL:** the Okta **Sign on URL**
    * **X.509 signing certificate:** paste the PEM block

    Save. The connection card then shows the exact SP metadata + ACS URLs — confirm they match what you entered in Okta. You can edit the email domain list on the connection card later without re-creating the connection.

    <Callout type="warn">
      The domain list must include **every domain Okta may send as a user's email**.
      With **Application username = Email**, that's the domain of each user's Okta
      email address. If some users sign in with a different domain (e.g. a
      subsidiary), list all of them — otherwise those users bounce back to the login
      page with an `account_not_linked` error even though Okta authenticated them.
    </Callout>
  </Step>
</Steps>

## Assign users to the app [#assign-users-to-the-app]

Configuring SAML isn't enough on its own — Okta only lets **assigned** users sign in to the app. On the app's **Assignments** tab, click **Assign → Assign to People** (or **Assign to Groups**), add the people who should be able to sign in, and confirm. The **Applications** page's **Assign Users to App** bulk flow works too.

<Callout type="warn">
  If you skip this, sign-in fails at Okta with &#x2A;"Sorry, you can't access LLM
  Gateway because you are not assigned this application in Okta."* — even though
  the SAML config is correct. Assign at least your own account before testing.
</Callout>

If you also set up SCIM below, assignment doubles as the provisioning trigger: assigned users are pushed to OffRail.

## Configure SCIM provisioning [#configure-scim-provisioning]

SSO only signs people in. SCIM is what actually adds members to the organization. Set it up on the **same** app integration.

<Steps>
  <Step>
    ### Generate a SCIM token [#generate-a-scim-token]

    In OffRail, on &#x2A;*Organization → SSO → Directory sync (SCIM)**, click **Generate SCIM token** and copy it (shown once). The SCIM base URL is shown in the same card — it's `https://internal.offrail.ai/scim/v2` on the hosted default.
  </Step>

  <Step>
    ### Enable SCIM on the app [#enable-scim-on-the-app]

    On the app's **General** tab → **App Settings** → **Edit**, set **Provisioning** to **SCIM** and **Save**. A new **Provisioning** tab appears on the app.
  </Step>

  <Step>
    ### Configure the SCIM connection [#configure-the-scim-connection]

    On **Provisioning → Integration** → **Edit**:

    * **SCIM connector base URL:** the SCIM base URL, e.g. `https://internal.offrail.ai/scim/v2`
    * **Unique identifier field for users:** `userName`
    * **Supported provisioning actions:** check **Push New Users**, **Push Profile Updates**, and **Push Groups** (leave the two *Import* options unchecked — OffRail doesn't support importing users into Okta)
    * **Authentication Mode:** **HTTP Header** — Okta then shows an **Authorization: Bearer** field; paste the SCIM token there. **Do not** pick Basic Auth or OAuth 2: OffRail's SCIM endpoint authenticates a plain static bearer token.

    Click **Test Connector Configuration*&#x2A; — a green &#x2A;*"Connector configured successfully"** panel should list **Create Users**, **Update User Attributes** and **Push Groups** as detected (the *Import* rows show a red ✗, which is expected). Then **Save**.
  </Step>

  <Step>
    ### Enable provisioning to the app [#enable-provisioning-to-the-app]

    On **Provisioning → To App** → **Edit**, enable **Create Users**, **Update User Attributes**, and **Deactivate Users**, then **Save**. Leave **Sync Password** off.

    From now on, assigning a user (or an assigned group's member) pushes them to OffRail; unassigning or deactivating them removes their membership.

    <Callout type="warn">
      Users who were **assigned before provisioning was enabled** are not pushed
      automatically — they show a red warning icon on the **Assignments** tab. Click
      the **Provision User** button there and confirm with **OK**; Okta runs a job
      that provisions all such users.
    </Callout>
  </Step>
</Steps>

Provisioned users are created in OffRail and added to the organization (default role **developer**). Deactivating or unassigning a user in Okta removes their membership.

## Group → role mapping (optional) [#group--role-mapping-optional]

On the app's **Push Groups** tab, push the Okta groups whose membership should matter. Then on the OffRail **SSO** page, add mappings such as `Acme Admins → admin`. When Okta pushes group membership, each member gets the **highest** mapped role among their groups (default `developer`). Owners are never auto-demoted, so you can't accidentally lock yourself out of the owner role.

## Default project access (optional) [#default-project-access-optional]

Provisioned members join as **developers**, who only see the projects they're granted (owners/admins always see every project). Use the **Default project access** card on the **SSO** page to choose which projects newly provisioned SSO/SCIM developers get by default. If you leave it unconfigured, new members get your organization's first (default) project so they can see something out of the box; select an empty set to instead grant nothing and manage project access manually per member.

<Callout type="info">
  Default project access applies only to members provisioned **after** you set
  it — it doesn't retroactively change existing members' project grants. Adjust
  an existing member's projects from the **Team** page.
</Callout>

## Test the login [#test-the-login]

Log out → login page → **Sign in with SSO** → enter a work email (`someone@acme.com`). OffRail resolves the connection by email domain and redirects to Okta; if the user is already signed in to Okta the round-trip is instant. On return they're signed in as the (SCIM-provisioned) user — if the dashboard opens in the user's own personal organization, the enterprise organization is one click away in the organization switcher.

## Enforce SSO (optional, do this last) [#enforce-sso-optional-do-this-last]

Flip **Require SSO** on the connection to require SSO for that email domain — password, social, and passkey sign-in are then rejected for it.

<Callout type="warn">
  Only enable **Require SSO** after the test above passes, and remember it also
  applies to the admin setting it up if their email is on that domain — verify
  your own SSO login works first.
</Callout>

## Troubleshooting [#troubleshooting]

* **Okta says "you are not assigned this application"** — the SAML config is fine; the user just isn't on the app's **Assignments** tab. Assign them (or their group).
* **Okta sign-in succeeds but the user lands back on the OffRail login page** — the SAML response was received but refused, almost always an `account_not_linked` domain mismatch: the asserted email's domain isn't in the connection's **Email domains** list. Edit the list on the connection card to cover every domain Okta may assert.
* **Every SSO attempt fails after the Okta redirect** — check that the connection's **Identity Provider Single Sign-On URL** is the app-specific **Sign on URL** containing the `exk…` app id (Sign On tab → Metadata details → More details). Other URLs break issuer validation. Also confirm the connection was created with **Identity provider = Okta** and that the certificate is the app's current **SHA-2 Active** signing certificate.
* **Assigned users never appear in the organization** — provisioning was probably enabled after they were assigned. Use **Provision User** on the **Assignments** tab, and confirm **Provisioning → To App → Create Users** is enabled and the connector test passes.