Using the AI SDK

Generate text, stream responses, and call tools with the Vercel AI SDK and OffRail

OffRail speaks the OpenAI wire format, so the Vercel AI SDK reaches it through @ai-sdk/openai with no adapter of its own. Point that provider at the gateway base URL and one instance with one API key reaches every model in the catalog.

Install

pnpm add ai @ai-sdk/openai

Set your API key (create one from the dashboard):

export OFFRAIL_API_KEY=orl_your_key_here

Generate text

import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { text } = await generateText({
	model: offrail("openai/gpt-4o"),
	prompt: "Hello!",
});

Switching models is a one-line change — the same provider serves every model:

const { text } = await generateText({
	model: offrail("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Hello!",
});

Model ID formats

OffRail supports two model ID formats:

Canonical model IDs (gpt-4o) — smart routing picks the best provider based on uptime, throughput, price, and latency
Provider-prefixed IDs (openai/gpt-4o) — routes to a specific provider with automatic failover if uptime drops below 90%

See the routing documentation for details and the models page for the full catalog.

Stream responses

import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { textStream } = await streamText({
	model: offrail("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Write a poem about coding",
});

for await (const text of textStream) {
	process.stdout.write(text);
}

Next.js route handler

// app/api/chat/route.ts
import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = await streamText({
		model: offrail("openai/gpt-4o"),
		messages,
	});

	return result.toDataStreamResponse();
}

Tool calling

import { createOpenAI } from "@ai-sdk/openai";
import { generateText, tool } from "ai";
import { z } from "zod";

const offrail = createOpenAI({
	apiKey: process.env.OFFRAIL_API_KEY,
	baseURL: "https://api.offrail.ai/v1",
});

const { text, toolResults } = await generateText({
	model: offrail("openai/gpt-4o"),
	tools: {
		weather: tool({
			description: "Get the weather for a location",
			parameters: z.object({
				location: z.string(),
			}),
			execute: async ({ location }) => {
				return { temperature: 72, condition: "sunny" };
			},
		}),
	},
	prompt: "What's the weather in San Francisco?",
});

Without the provider package

If you prefer not to add a dependency, point @ai-sdk/openai at the gateway with a custom base URL:

import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const offrail = createOpenAI({
	baseURL: "https://api.offrail.ai/v1",
	apiKey: process.env.OFFRAIL_API_KEY,
});

const { text } = await generateText({
	model: offrail("openai/gpt-4o"),
	prompt: "Hello!",
});

Every request made through the AI SDK shows up in your Activity and Usage & Metrics dashboards like any other gateway request — with per-request cost, tokens, and latency.