Custom providers

Braintrust supports custom AI providers, allowing you to integrate any AI model or endpoint into your evaluation and tracing workflows. This includes custom models from existing providers, self-hosted models, or proprietary AI services. Custom providers appear alongside standard providers in model selection dropdowns.

Setup

If you have custom models as part of your OpenAI or other accounts, or if you’re running your own AI endpoints, you can add them to Braintrust by configuring a custom provider.

Go to Settings > AI providers.
Add a new organization or project provider and choose Custom providers > New.
On Setup, choose the Provider type, enter a unique Name, and configure the credentials and endpoint required by your provider.
Under Models, click Custom model and enter the model identifier your endpoint accepts. Click the settings icon beside the model to configure its format, capabilities, and pricing.
Click Create. On Models, you can browse models and copy a Gateway test command.

Configuration options

Specify the following for your custom provider.

Name: A unique name for your custom provider
Model name: The name of your custom model (e.g., gpt-3.5-acme, my-custom-llama)
API base URL: The base URL for your provider (labeled Endpoint URL for Bedrock)
Format: The API format (openai, anthropic, google, window, or js). When an endpoint URL is configured, only openai, window, and js are available — the anthropic and google formats route requests directly to their respective first-party APIs rather than to the configured endpoint, so they are incompatible with a custom endpoint URL.
Flavor: Whether it’s a chat or completion model (default: chat)
Headers: Any custom headers required for authentication or configuration

Endpoint URLs must use http or https. To protect against server-side request forgery (SSRF), Braintrust validates the resolved target of each request.

Custom headers and templating

Custom headers let you send additional authentication or configuration values to your endpoint. Braintrust includes these headers in requests to the custom endpoint. To add a custom header:

Open the provider’s Setup tab.
Under Request configuration, click Additional header and enter its name and value.
Click Create or Update to save the provider.

Header values can also use Mustache templates to include information about the caller or requested model. The supported variables are:

{{email}}: Email of the user associated with the Braintrust API key
{{model}}: The model name being requested

Example header configuration:

Authorization: Bearer {{api_key}}
X-User-Email: {{email}}
X-Model: {{model}}

API keys are stored as one-way cryptographic hashes, never in plaintext.

Streaming support

If your endpoint supports streaming natively, enable This endpoint supports streaming. Braintrust will automatically convert the response to streaming format, allowing your models to work in the playground and other streaming contexts.

Model metadata

Model metadata describes a custom model’s capabilities and pricing. You can specify whether the model supports multimodal inputs and provide token prices so Braintrust can estimate usage costs. To configure model metadata:

Open the provider’s Setup tab.
In the Models section, click the settings icon beside the custom model.
Configure the fields below, then click Create or Update to save the provider.

The available metadata fields include:

Multimodal: Whether the model supports multimodal inputs
Input cost: Cost per million input tokens (for experiment cost estimation)
Output cost: Cost per million output tokens (for experiment cost estimation)
Input cache read cost: Cost per million cached input tokens read (for experiment cost estimation)
Input cache write cost: Cost per million input tokens written to cache (for experiment cost estimation)

Cache cost fields are relevant for models that support prompt caching (e.g., Anthropic’s Claude models, OpenAI’s GPT models with cache_control). These fields help Braintrust accurately estimate costs when using cached prompts in trace logs and evaluations.

Provider failover

If a custom model’s provider fails, Braintrust can retry the request against a fallback model on another provider. To set this up:

In your custom provider’s list of models, click the settings button () next to a model to open its options. Under Fallback models, add one or more models to try if this one fails.
Turn on Enable failover by default to apply failover to every request, or pass the x-bt-fallback-providers header to enable it per request.

Fallback models come from your other configured providers, and failover only reaches a provider whose credentials are configured. See Configure failover for custom models for the full reference, including how the gateway selects and retries providers.

Trace logs with custom providers

Trace custom provider LLM calls for observability and monitoring.

Automatic tracing

Once your custom provider is configured, tracing works automatically.

In the examples below, organizations on the EU data plane should replace api.braintrust.dev with api-eu.braintrust.dev.

import { OpenAI } from "openai";
import { initLogger } from "braintrust";

initLogger({
  projectName: "My Project",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

const client = new OpenAI({
  baseURL: "https://api.braintrust.dev/v1/proxy",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

// All API calls are automatically logged
const result = await client.chat.completions.create({
  model: "my-custom-model", // Your custom model name
  messages: [{ role: "user", content: "What is machine learning?" }],
});

import os

from braintrust import init_logger
from openai import OpenAI

init_logger(project="My Project")

client = OpenAI(
    base_url="https://api.braintrust.dev/v1/proxy",
    api_key=os.environ["BRAINTRUST_API_KEY"],
)

# All API calls are automatically logged
result = client.chat.completions.create(
    model="my-custom-model",  # Your custom model name
    messages=[{"role": "user", "content": "What is machine learning?"}],
)

Manual tracing

For more control over tracing, you can manually log calls to your custom provider.

import { trace } from "braintrust";

// Use your custom provider directly
const customResponse = await fetch("https://your-custom-endpoint.com/v1/chat", {
  method: "POST",
  headers: {
    Authorization: "Bearer your-custom-api-key",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "my-custom-model",
    messages: [{ role: "user", content: "Hello!" }],
  }),
});

// Manually trace the call
trace({
  name: "custom-model-call",
  input: { message: "Hello!" },
  output: await customResponse.json(),
  metadata: {
    model: "my-custom-model",
    provider: "custom",
  },
});

import requests
from braintrust import trace

# Use your custom provider directly
response = requests.post(
    "https://your-custom-endpoint.com/v1/chat",
    headers={
        "Authorization": "Bearer your-custom-api-key",
        "Content-Type": "application/json",
    },
    json={
        "model": "my-custom-model",
        "messages": [{"role": "user", "content": "Hello!"}],
    },
)

# Manually trace the call
trace(
    name="custom-model-call",
    input={"message": "Hello!"},
    output=response.json(),
    metadata={
        "model": "my-custom-model",
        "provider": "custom",
    },
)

Evaluations

Evaluations distill the non-deterministic outputs of custom models into an effective feedback loop that enables you to ship more reliable, higher quality products. Braintrust Eval is a simple function composed of a dataset of user inputs, a task, and a set of scorers. To learn more about evaluations, see the Experiments guide.

Basic evaluation setup

Use your custom models as evaluators in Braintrust experiments.

import { Eval } from "braintrust";
import { OpenAI } from "openai";

const client = new OpenAI({
  baseURL: "https://api.braintrust.dev/v1/proxy",
  apiKey: process.env.BRAINTRUST_API_KEY,
});

Eval("Custom Model Evaluation", {
  data: () => [
    { input: "What is 2+2?", expected: "4" },
    { input: "What is the capital of France?", expected: "Paris" },
  ],
  task: async (input) => {
    const response = await client.chat.completions.create({
      model: "my-custom-model",
      messages: [{ role: "user", content: input }],
    });
    return response.choices[0].message.content;
  },
  scores: [
    {
      name: "accuracy",
      scorer: (args) => (args.output === args.expected ? 1 : 0),
    },
  ],
});

import os

from braintrust import Eval
from openai import OpenAI

client = OpenAI(
    base_url="https://api.braintrust.dev/v1/proxy",
    api_key=os.environ["BRAINTRUST_API_KEY"],
)

def task(input):
    response = client.chat.completions.create(
        model="my-custom-model",
        messages=[{"role": "user", "content": input}],
    )
    return response.choices[0].message.content

def accuracy_scorer(output, expected, **kwargs):
    return 1 if output == expected else 0

Eval(
    "Custom Model Evaluation",
    data=[
        {"input": "What is 2+2?", "expected": "4"},
        {"input": "What is the capital of France?", "expected": "Paris"},
    ],
    task=task,
    scores=[accuracy_scorer],
)

Use custom providers for LLM-as-a-judge

Custom models can serve as evaluators for other AI systems.

import { LLMClassifierFromSpec } from "autoevals";

const relevanceScorer = LLMClassifierFromSpec("Relevance", {
  choice_scores: { Relevant: 1, Irrelevant: 0 },
  model: "my-custom-model",
  use_cot: true,
});

from autoevals import LLMClassifierFromSpec

relevance_scorer = LLMClassifierFromSpec(
    "Relevance",
    choice_scores={"Relevant": 1, "Irrelevant": 0},
    model="my-custom-model",
    use_cot=True,
)

Compare custom models

You can run experiments comparing your custom models against standard providers.

import { Eval } from "braintrust";

const models = [
  "gpt-4o-mini", // Standard OpenAI model
  "claude-sonnet-4-5-20250929", // Standard Anthropic model
  "my-custom-model", // Your custom model
];

for (const model of models) {
  Eval(`Model Comparison - ${model}`, {
    data: () => [
      { input: "Explain quantum computing", expected: "technical_explanation" },
      { input: "Write a haiku about code", expected: "creative_poetry" },
    ],
    task: async (input) => {
      const response = await client.chat.completions.create({
        model,
        messages: [{ role: "user", content: input }],
      });
      return response.choices[0].message.content;
    },
    scores: [
      {
        name: "quality",
        scorer: LLMClassifierFromSpec("Quality", {
          choice_scores: { High: 1, Medium: 0.5, Low: 0 },
          model: "gpt-4o", // Use a standard model for evaluation
        }),
      },
    ],
  });
}

from autoevals import LLMClassifierFromSpec
from braintrust import Eval

models = [
    "gpt-4o-mini",  # Standard OpenAI model
    "claude-sonnet-4-5-20250929",  # Standard Anthropic model
    "my-custom-model",  # Your custom model
]

quality_scorer = LLMClassifierFromSpec(
    "Quality",
    choice_scores={"High": 1, "Medium": 0.5, "Low": 0},
    model="gpt-4o",  # Use a standard model for evaluation
)

for model in models:

    def task(input):
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": input}],
        )
        return response.choices[0].message.content

    Eval(
        f"Model Comparison - {model}",
        data=[
            {"input": "Explain quantum computing", "expected": "technical_explanation"},
            {"input": "Write a haiku about code", "expected": "creative_poetry"},
        ],
        task=task,
        scores=[quality_scorer],
    )

Common use cases

Self-hosted models

For self-hosted models (e.g. using Ollama, vLLM, or custom deployments):

Set the endpoint URL to your self-hosted service
Choose the appropriate format based on your API compatibility
Configure any required authentication headers
Set streaming support based on your implementation

Fine-tuned models

For fine-tuned versions of existing models:

Use the same format as the base model
Set the model name to your fine-tuned model identifier
Configure the endpoint URL if using a custom deployment
Add any provider-specific headers for accessing fine-tuned models

Proprietary AI services

For proprietary or enterprise AI services:

Configure the endpoint URL provided by your AI service
Set up authentication headers as required
Choose the format that best matches your service’s API
Enable or disable streaming based on service capabilities

Before running large-scale evaluations, test a request from the provider’s Models tab or use a Braintrust playground to verify the configuration.

AI providers

SDKs

Developer tools

Runtime environments

Setup

Configuration options

Custom headers and templating

Streaming support

Model metadata

Provider failover

Trace logs with custom providers

Automatic tracing

Manual tracing

Evaluations

Basic evaluation setup

Use custom providers for LLM-as-a-judge

Compare custom models

Common use cases

Self-hosted models

Fine-tuned models

Proprietary AI services

​Setup

​Configuration options

​Custom headers and templating

​Streaming support

​Model metadata

​Provider failover

​Trace logs with custom providers

​Automatic tracing

​Manual tracing

​Evaluations

​Basic evaluation setup

​Use custom providers for LLM-as-a-judge

​Compare custom models

​Common use cases

​Self-hosted models

​Fine-tuned models

​Proprietary AI services

Setup

Configuration options

Custom headers and templating

Streaming support

Model metadata

Provider failover

Trace logs with custom providers

Automatic tracing

Manual tracing

Evaluations

Basic evaluation setup

Use custom providers for LLM-as-a-judge

Compare custom models

Common use cases

Self-hosted models

Fine-tuned models

Proprietary AI services