LLM-as-a-judge evaluators

LLM-as-a-judge evaluators use a language model to evaluate outputs based on natural language criteria. A scorer returns a numeric score, while a classifier returns a categorical label. They are best for subjective judgments like tone, helpfulness, or creativity that are difficult to encode in deterministic code. You can define LLM-as-a-judge scorers in three places:

Inline in SDK code: Define scorers directly in your evaluation scripts for local development or application-specific logic.
Pushed via CLI: Define scorers in TypeScript or Python files and push them to Braintrust for team-wide sharing and automatic evaluation of production logs.
Created in UI: Build scorers in the Braintrust web interface for rapid prototyping and simple configurations.

Most teams prototype in the UI, then push production-ready scorers via the CLI. See Scorers overview for guidance.

Choose a judge model

The right judge depends on the judgment you need, the context it must read, and how often you will run the evaluator. The same considerations apply to span, trace, and group scoring.

General-purpose LLMs handle broad or nuanced criteria, such as helpfulness, tone, or adherence to detailed instructions. Compare candidates on how well they agree with human judgments across the criteria in your rubric.
Reasoning models suit judgments that take several steps, such as assessing a complex solution. Check whether the extra reasoning improves that agreement enough to justify its latency and cost.
Structured decision models such as TypeSafe Jev select from a fixed set of answers rather than generating prose. Use them for frequent, focused checks such as whether a support reply addresses a request.

Treat these as starting points. Compare candidate judges on representative examples with human-assigned scores, including ambiguous and failing cases. Check that each model supports your input format and context length, and measure latency and cost alongside score quality.

Score spans

Span-level scorers evaluate individual operations or outputs. Use them for measuring single LLM responses, checking specific tool calls, or validating individual outputs. Each matching span receives an independent score. See Choose a judge model for guidance on matching a model to your evaluation criteria. Your prompt template can reference these variables:

{{input}}: The input to your task
{{output}}: The output from your task
{{expected}}: The expected output (optional)
{{metadata}}: Custom metadata from the test case

Use scorers inline in your evaluation code:

llm_scorer.eval.ts

import { Eval } from "braintrust";
import { LLMClassifierFromTemplate } from "autoevals";
import OpenAI from "openai";

const client = new OpenAI();

const MOVIE_DATASET = [
  {
    input:
      "A detective investigates a series of murders based on the seven deadly sins.",
    expected: "Se7en",
  },
  {
    input:
      "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.",
    expected: "Inception",
  },
];

async function task(input: string): Promise<string> {
  const response = await client.responses.create({
    model: "gpt-5-mini",
    input: [
      {
        role: "system",
        content:
          "Based on the following description, identify the movie. Reply with only the movie title.",
      },
      { role: "user", content: input },
    ],
  });
  return response.output_text ?? "";
}

const correctnessScorer = LLMClassifierFromTemplate({
  name: "Correctness",
  promptTemplate: `You are evaluating a movie-identification task.

Output (model's answer): {{output}}
Expected (correct movie): {{expected}}

Does the output correctly identify the same movie as the expected answer?
Consider alternate titles (e.g. "Harry Potter 1" vs "Harry Potter and the Sorcerer's Stone") as correct.

Return only "correct" if the output is the right movie (exact or equivalent title).
Return only "incorrect" otherwise.`,
  choiceScores: {
    correct: 1,
    incorrect: 0,
  },
  useCoT: true,
});

Eval("Movie Matcher", {
  data: MOVIE_DATASET,
  task,
  scores: [correctnessScorer],
});

from braintrust import Eval
from autoevals import LLMClassifier
from openai import OpenAI

client = OpenAI()

MOVIE_DATASET = [
    {
        "input": "A detective investigates a series of murders based on the seven deadly sins.",
        "expected": "Se7en",
    },
    {
        "input": "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.",
        "expected": "Inception",
    },
]


def task(input):
    response = client.responses.create(
        model="gpt-5-mini",
        input=[
            {
                "role": "system",
                "content": "Based on the following description, identify the movie. Reply with only the movie title.",
            },
            {"role": "user", "content": input},
        ],
    )
    return response.output_text


correctness_scorer = LLMClassifier(
    name="Correctness",
    prompt_template="""You are evaluating a movie-identification task.

Output (model's answer): {{output}}
Expected (correct movie): {{expected}}

Does the output correctly identify the same movie as the expected answer?
Consider alternate titles (e.g. "Harry Potter 1" vs "Harry Potter and the Sorcerer's Stone") as correct.

Return only "correct" if the output is the right movie (exact or equivalent title).
Return only "incorrect" otherwise.""",
    choice_scores={
        "correct": 1,
        "incorrect": 0,
    },
    model="gpt-5-mini",
)

Eval(
    "Movie Matcher",
    data=MOVIE_DATASET,
    task=task,
    scores=[correctness_scorer],
)

require "braintrust"
require "openai"

Braintrust.init

client = OpenAI::Client.new(api_key: ENV.fetch("OPENAI_API_KEY", nil))
judge_client = OpenAI::Client.new(api_key: ENV.fetch("OPENAI_API_KEY", nil))

MOVIE_DATASET = [
  {
    input: "A detective investigates a series of murders based on the seven deadly sins.",
    expected: "Se7en",
  },
  {
    input: "A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O.",
    expected: "Inception",
  },
]

correctness_scorer = Braintrust::Scorer.new("correctness") do |output:, expected:|
  response = judge_client.chat.completions.create(
    model: "gpt-5-mini",
    messages: [{
      role: "user",
      content: <<~PROMPT
        You are evaluating a movie-identification task.

        Output (model's answer): #{output}
        Expected (correct movie): #{expected}

        Does the output correctly identify the same movie as the expected answer?
        Consider alternate titles (e.g. "Harry Potter 1" vs "Harry Potter and the Sorcerer's Stone") as correct.

        Return only "correct" if the output is the right movie (exact or equivalent title).
        Return only "incorrect" otherwise.
      PROMPT
    }]
  )

  verdict = response.choices.first.message.content.to_s.strip.downcase
  {name: "Correctness", score: verdict == "correct" ? 1.0 : 0.0}
end

Braintrust::Eval.run(
  project: "Movie Matcher",
  cases: MOVIE_DATASET,
  task: lambda do |input:|
    response = client.chat.completions.create(
      model: "gpt-5-mini",
      messages: [
        {role: "system", content: "Based on the following description, identify the movie. Reply with only the movie title."},
        {role: "user", content: input}
      ]
    )
    response.choices.first.message.content || ""
  end,
  scorers: [correctness_scorer]
)

OpenTelemetry.tracer_provider.shutdown

using System.Text.Json;
using Braintrust.Sdk;
using Braintrust.Sdk.Eval;
using Braintrust.Sdk.OpenAI;
using OpenAI;
using OpenAI.Chat;

sealed class CorrectnessScorer(string openAIApiKey) : IScorer<string, string>
{
    public string Name => "Correctness";

    private readonly ChatClient _chatClient =
        new OpenAIClient(openAIApiKey).GetChatClient("gpt-5-mini");

    public async Task<IReadOnlyList<Score>> Score(TaskResult<string, string> taskResult)
    {
        var prompt = $$"""
            You are evaluating a movie-identification task.

            Output (model's answer): {{taskResult.Result}}
            Expected (correct movie): {{taskResult.DatasetCase.Expected}}

            Does the output correctly identify the same movie as the expected answer?
            Consider alternate titles (e.g. "Harry Potter 1" vs "Harry Potter and the Sorcerer's Stone") as correct.

            Reply with JSON only: {"score": 1, "reasoning": "..."} for correct, or {"score": 0, "reasoning": "..."} for incorrect.
            """;

        var completion = await _chatClient.CompleteChatAsync([new UserChatMessage(prompt)]);
        using var json = JsonDocument.Parse(completion.Value.Content[0].Text);
        var score = json.RootElement.GetProperty("score").GetDouble();
        var reasoning = json.RootElement.GetProperty("reasoning").GetString() ?? "";

        return [new Score(Name, score, new Dictionary<string, object> { { "reasoning", reasoning } })];
    }
}

class Program
{
    static readonly DatasetCase<string, string>[] MovieDataset =
    [
        DatasetCase.Of("A detective investigates a series of murders based on the seven deadly sins.", "Se7en"),
        DatasetCase.Of("A thief plants an idea into a CEO's mind through dream-sharing technology.", "Inception"),
    ];

    static async Task Main(string[] args)
    {
        var openAIApiKey = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!;
        var braintrust = Braintrust.Sdk.Braintrust.Get();
        var activitySource = braintrust.GetActivitySource();
        var openAIClient = BraintrustOpenAI.WrapOpenAI(activitySource, openAIApiKey);

        async Task<string> MovieTask(string input)
        {
            var response = await openAIClient.GetChatClient("gpt-5-mini").CompleteChatAsync(
                new SystemChatMessage("Identify the movie from the description. Reply with the title only."),
                new UserChatMessage(input));
            return response.Value.Content[0].Text;
        }

        var eval = await braintrust
            .EvalBuilder<string, string>()
            .Name("Movie Matcher")
            .Cases(MovieDataset)
            .TaskFunction(MovieTask)
            .Scorers(new CorrectnessScorer(openAIApiKey))
            .BuildAsync();

        var result = await eval.RunAsync();
        Console.WriteLine(result.CreateReportString());
    }
}

Define TypeScript or Python scorers in code and push to Braintrust:

llm_scorer.ts

import braintrust from "braintrust";

const project = braintrust.projects.create({ name: "my-project" });

project.scorers.create({
  name: "Helpfulness scorer",
  slug: "helpfulness-scorer",
  description: "Evaluate helpfulness of response",
  tags: ["quality"],
  messages: [
    {
      role: "user",
      content:
        'Rate the helpfulness of this response: {{output}}\n\nReturn "A" for very helpful, "B" for somewhat helpful, "C" for not helpful.',
    },
  ],
  model: "gpt-5-mini",
  useCot: true,
  choiceScores: {
    A: 1,
    B: 0.5,
    C: 0,
  },
  metadata: {
    __pass_threshold: 0.7,
  },
});

import braintrust

project = braintrust.projects.create(name="My project")

project.scorers.create(
    name="Helpfulness scorer",
    slug="helpfulness-scorer",
    description="Evaluate helpfulness of response",
    tags=["quality"],
    messages=[
        {
            "role": "user",
            "content": 'Rate the helpfulness of this response: {{output}}\n\nReturn "A" for very helpful, "B" for somewhat helpful, "C" for not helpful.',
        }
    ],
    model="gpt-5-mini",
    use_cot=True,
    choice_scores={
        "A": 1,
        "B": 0.5,
        "C": 0,
    },
    metadata={"__pass_threshold": 0.7},
)

Push to Braintrust using the bt CLI:

bt functions push llm_scorer.ts

bt functions push llm_scorer.py

Go to Scorers > + Scorer.
Enter a scorer name and slug.
Select LLM-as-a-judge.
Configure:
- Prompt: Instructions for evaluating the output
- Model: Which model to use as judge
- Choice scores: Map model choices (A, B, C) to numeric scores
- Use CoT: Enable chain-of-thought reasoning for complex evaluations
- Allow skip: Let the judge return a null score for non-applicable cases (see Skip non-applicable cases)
Click Save as custom scorer.

Score traces

Trace-level scorers evaluate entire execution traces including all spans and conversation history. Use these for assessing multi-turn conversation quality, overall workflow completion, or when your scorer needs access to the full execution context. The scorer runs once per trace. In an experiment, a scorer evaluates the trace whenever its prompt uses thread variables such as {{thread}}, since the full trace is available to every scorer. For online scoring, you also set the rule’s Scope field to Trace, which controls whether the scorer runs once per trace or per matching span. Prompt templates for trace-level scorers support the following reserved variables:

Variable	Type	Description
`{{input}}`	any	Input from the root span
`{{output}}`	any	Output from the root span
`{{expected}}`	any	Expected output from the root span (optional)
`{{metadata}}`	object	Metadata from the root span
`{{thread}}`	text	Full conversation rendered as human-readable text (excludes system messages for scorers)
`{{thread_with_system}}`	text	Full conversation including system messages, rendered as human-readable text
`{{thread_count}}`	number	Total number of messages in the thread
`{{first_message}}`	object	First message in the thread
`{{last_message}}`	object	Last message in the thread
`{{user_messages}}`	array	All user/human messages only
`{{assistant_messages}}`	array	All assistant messages only
`{{human_ai_pairs}}`	array	Turn pairs — each item has `{human, assistant}`
`{{preprocessed}}`	array	Messages returned by the scorer’s preprocessor for the trace, including system messages

Use {{thread}} to pass the full conversation to a judge model as formatted text. For scorers, {{thread}} omits system messages so the rubric isn’t polluted by your application’s system prompt. Use {{thread_with_system}} when the judge needs the system prompt as context. For non-scorer prompts, the two variables are equivalent because both include system messages. {{input}}, {{output}}, {{expected}}, and {{metadata}} come from the root span of the trace. {{preprocessed}} holds the messages the scorer’s preprocessor returned for the trace, including system messages. Because it’s an array, iterate it with a section tag such as {{#preprocessed}}{{role}}: {{content}}{{/preprocessed}}. Interpolating it directly renders the array as JSON.

Trace-level scoring requires TypeScript SDK v2.2.1+, Python SDK v0.5.6+, or Ruby SDK v0.2.1+.

Use scorers inline in your evaluation code:

trace_llm_scorer.eval.ts

import { Eval, wrapOpenAI, wrapTraced, type Scorer } from "braintrust";
import OpenAI from "openai";

const client = new OpenAI();
const wrappedClient = wrapOpenAI(new OpenAI());

const SUPPORT_DATASET = [
  { input: "My order hasn't arrived yet. Order #12345." },
  { input: "I need help resetting my password." },
];

const callLLM = wrapTraced(async function callLLM(messages: Array<{ role: string; content: string }>) {
  const response = await wrappedClient.chat.completions.create({
    model: "gpt-5-mini",
    messages,
  });
  return response.choices[0].message.content || "";
});

async function supportTask(input: string): Promise<string> {
  const messages: Array<{ role: string; content: string }> = [
    { role: "system", content: "You are a helpful customer support agent." }
  ];

  messages.push({ role: "user", content: input });
  const response1 = await callLLM(messages);
  messages.push({ role: "assistant", content: response1 });

  messages.push({ role: "user", content: "Can you provide more details?" });
  const response2 = await callLLM(messages);
  messages.push({ role: "assistant", content: response2 });

  messages.push({ role: "user", content: "Thank you for your help!" });
  const response3 = await callLLM(messages);

  return response3;
}

const conversationCoherence: Scorer = async ({ trace }) => {
  if (!trace) return null;

  const thread = await trace.getThread();
  const threadText = thread
    .map(msg => `${msg.role}: ${msg.content}`)
    .join("\n\n");

  const response = await client.responses.create({
    model: "gpt-5-mini",
    input: [
      {
        role: "user",
        content: `Evaluate the coherence of this customer support conversation:

${threadText}

Rate the conversation coherence:
- "A" for highly coherent with natural flow and consistent context
- "B" for mostly coherent with minor gaps or context issues
- "C" for incoherent, disjointed, or lost context

Return only the letter (A, B, or C).`,
      },
    ],
  });

  const rating = response.output_text?.trim().toUpperCase() || "C";
  const choiceScores = { A: 1, B: 0.6, C: 0 };
  const score = choiceScores[rating as keyof typeof choiceScores] ?? 0;

  return {
    name: "Conversation coherence",
    score,
    metadata: { rating, thread_length: thread.length },
  };
};

Eval("Support Conversation Quality", {
  data: SUPPORT_DATASET,
  task: supportTask,
  scores: [conversationCoherence],
});

from braintrust import Eval, wrap_openai, traced
from openai import AsyncOpenAI, OpenAI

client = OpenAI()
wrapped_client = wrap_openai(AsyncOpenAI())

SUPPORT_DATASET = [
    {"input": "My order hasn't arrived yet. Order #12345."},
    {"input": "I need help resetting my password."},
]


@traced
async def call_llm(messages):
    response = await wrapped_client.chat.completions.create(
        model="gpt-5-mini",
        messages=messages,
    )
    return response.choices[0].message.content or ""


async def support_task(input):
    messages = [
        {"role": "system", "content": "You are a helpful customer support agent."}
    ]

    messages.append({"role": "user", "content": input})
    response1 = await call_llm(messages)
    messages.append({"role": "assistant", "content": response1})

    messages.append({"role": "user", "content": "Can you provide more details?"})
    response2 = await call_llm(messages)
    messages.append({"role": "assistant", "content": response2})

    messages.append({"role": "user", "content": "Thank you for your help!"})
    response3 = await call_llm(messages)

    return response3


async def conversation_coherence(input, output, expected, trace=None):
    if not trace:
        return None

    thread = await trace.get_thread()
    thread_text = "\n\n".join([f"{msg['role']}: {msg['content']}" for msg in thread])

    response = client.responses.create(
        model="gpt-5-mini",
        input=[
            {
                "role": "user",
                "content": f"""Evaluate the coherence of this customer support conversation:

{thread_text}

Rate the conversation coherence:
- "A" for highly coherent with natural flow and consistent context
- "B" for mostly coherent with minor gaps or context issues
- "C" for incoherent, disjointed, or lost context

Return only the letter (A, B, or C).""",
            }
        ],
    )

    rating = (response.output_text or "C").strip().upper()
    choice_scores = {"A": 1, "B": 0.6, "C": 0}
    score = choice_scores.get(rating, 0)

    return {
        "name": "Conversation coherence",
        "score": score,
        "metadata": {"rating": rating, "thread_length": len(thread)},
    }


Eval(
    "Support Conversation Quality",
    data=SUPPORT_DATASET,
    task=support_task,
    scores=[conversation_coherence],
)

require "braintrust"
require "openai"

Braintrust.init

client = OpenAI::Client.new(api_key: ENV.fetch("OPENAI_API_KEY", nil))
judge_client = OpenAI::Client.new(api_key: ENV.fetch("OPENAI_API_KEY", nil))

SUPPORT_DATASET = [
  {input: "My order hasn't arrived yet. Order #12345."},
  {input: "I need help resetting my password."},
]

def chat(client, messages)
  client.chat.completions.create(model: "gpt-5-mini", messages: messages)
    .choices.first.message.content || ""
end

support_task = Braintrust::Task.new("support") do |input:|
  messages = [{role: "system", content: "You are a helpful customer support agent."}]

  messages << {role: "user", content: input}
  messages << {role: "assistant", content: chat(client, messages)}

  messages << {role: "user", content: "Can you provide more details?"}
  messages << {role: "assistant", content: chat(client, messages)}

  messages << {role: "user", content: "Thank you for your help!"}
  chat(client, messages)
end

conversation_coherence = Braintrust::Scorer.new("conversation_coherence") do |trace:|
  next nil unless trace

  thread = trace.thread
  thread_text = thread.map { |msg| "#{msg["role"]}: #{msg["content"]}" }.join("\n\n")

  response = judge_client.chat.completions.create(
    model: "gpt-5-mini",
    messages: [{
      role: "user",
      content: <<~PROMPT
        Evaluate the coherence of this customer support conversation:

        #{thread_text}

        Rate the conversation coherence:
        - "A" for highly coherent with natural flow and consistent context
        - "B" for mostly coherent with minor gaps or context issues
        - "C" for incoherent, disjointed, or lost context

        Return only the letter (A, B, or C).
      PROMPT
    }]
  )

  rating = (response.choices.first.message.content || "C").strip.upcase
  scores = {"A" => 1.0, "B" => 0.6, "C" => 0.0}

  {
    name: "Conversation coherence",
    score: scores.fetch(rating, 0.0),
    metadata: {rating: rating, thread_length: thread.length}
  }
end

Braintrust::Eval.run(
  project: "Support Conversation Quality",
  cases: SUPPORT_DATASET,
  task: support_task,
  scorers: [conversation_coherence]
)

OpenTelemetry.tracer_provider.shutdown

Define TypeScript or Python scorers in code and push to Braintrust:

trace_llm_scorer.ts

import braintrust from "braintrust";
import { z } from "zod";

const project = braintrust.projects.create({ name: "my-project" });

project.scorers.create({
  name: "Conversation coherence",
  slug: "conversation-coherence",
  description: "Evaluate multi-turn conversation coherence",
  parameters: z.object({
    trace: z.any(),
  }),
  messages: [
    {
      role: "user",
      content: `Evaluate the coherence of this conversation:

{{thread}}

Rate the coherence:
- "A" for highly coherent with natural flow
- "B" for mostly coherent with minor gaps
- "C" for incoherent or disjointed`,
    },
  ],
  model: "gpt-5-mini",
  useCot: true,
  choiceScores: {
    A: 1,
    B: 0.6,
    C: 0,
  },
});

import braintrust
from pydantic import BaseModel

project = braintrust.projects.create(name="my-project")

class TraceParams(BaseModel):
    trace: dict

project.scorers.create(
    name="Conversation coherence",
    slug="conversation-coherence",
    description="Evaluate multi-turn conversation coherence",
    parameters=TraceParams,
    messages=[
        {
            "role": "user",
            "content": """Evaluate the coherence of this conversation:

{{thread}}

Rate the coherence:
- "A" for highly coherent with natural flow
- "B" for mostly coherent with minor gaps
- "C" for incoherent or disjointed""",
        }
    ],
    model="gpt-5-mini",
    use_cot=True,
    choice_scores={
        "A": 1,
        "B": 0.6,
        "C": 0,
    },
)

Push to Braintrust:

bt functions push trace_llm_scorer.ts

bt functions push trace_llm_scorer.py

Go to Scorers > + Scorer.
Enter a scorer name and slug.
Select LLM-as-a-judge.
Configure:
- Prompt: Use the {{thread}} variable to reference the conversation thread.
- Model: Which model to use as judge
- Choice scores: Map model choices (A, B, C) to numeric scores
- Use CoT: Enable chain-of-thought reasoning for complex evaluations
- Allow skip: Let the judge return a null score for non-applicable cases (see Skip non-applicable cases)
Click Save as custom scorer.

Score groups

Group scope is available only for online scoring rules.

To judge a session or multi-turn conversation that spans multiple traces, use group scope on an online scoring rule and build the judge prompt around thread variables. Braintrust renders those variables from the traces included in the group for that scoring run, so the judge can evaluate the grouped conversation without looping over spans. For example, this LLM-as-a-judge prompt uses {{thread}} to evaluate the grouped conversation:

Evaluate whether the assistant resolved the user's problem over the course
of this conversation.

{{thread}}

Answer Y if the user's problem was resolved by the end, and N if it was not.

Template variables available at group scope:

Variable	At group scope
`{{thread}}`	Renders the conversation across the grouped traces
`{{thread_with_system}}`	Renders the conversation across the grouped traces, including system messages
`{{thread_count}}`	Counts messages across the grouped traces
`{{first_message}}`	First message across the grouped traces
`{{last_message}}`	Last message across the grouped traces
`{{user_messages}}`	User or human messages across the grouped traces
`{{assistant_messages}}`	Assistant messages across the grouped traces
`{{human_ai_pairs}}`	Turn pairs across the grouped traces
`{{preprocessed}}`	Preprocessor output across the grouped traces, including system messages

At group scope, use thread variables for grouped conversation context. Do not use {{input}}, {{output}}, {{expected}}, or {{metadata}} when the judge needs to read the grouped traces.

Set pass thresholds

Define minimum acceptable scores to automatically mark results as passing or failing. When configured, scores that meet or exceed the threshold are marked as passing (green highlighting with checkmark), while scores below are marked as failing (red highlighting).

Pass thresholds apply only to scorers that output numeric scores. Classifiers, which output labels, don’t use them.

Add __pass_threshold to the scorer’s metadata (value between 0 and 1):

project.scorers.create({
  name: "Helpfulness scorer",
  slug: "helpfulness-scorer",
  messages: [
    {
      role: "user",
      content: 'Rate the helpfulness of this response: {{output}}\n\nReturn "A" for very helpful, "B" for somewhat helpful, "C" for not helpful.',
    },
  ],
  model: "gpt-5-mini",
  choiceScores: { A: 1, B: 0.5, C: 0 },
  metadata: {
    __pass_threshold: 0.7,
  },
});

project.scorers.create(
    name="Helpfulness scorer",
    slug="helpfulness-scorer",
    messages=[
        {
            "role": "user",
            "content": 'Rate the helpfulness of this response: {{output}}\n\nReturn "A" for very helpful, "B" for somewhat helpful, "C" for not helpful.',
        }
    ],
    model="gpt-5-mini",
    choice_scores={"A": 1, "B": 0.5, "C": 0},
    metadata={"__pass_threshold": 0.7},
)

Skip non-applicable cases

Enable Allow skip so the judge can decline to score a test case that its rubric doesn’t apply to. When enabled, the judge can select a reserved Skip choice. When the model selects Skip, the scorer returns a null score, and skipped cases are excluded from score aggregates instead of counting against the average. Because Skip is reserved, you can’t name one of your own choices “Skip” while this setting is on. When creating or editing an LLM-as-a-judge scorer in the UI:

Turn on Allow skip in the scorer configuration. It applies to numeric judges at both span and trace scope. Classifiers use Allow “No match” instead.

Reference the skip criteria in your prompt so the judge knows when to use it. For example:

Prompt excerpt

Score the response's faithfulness to the source document:
- A: Fully faithful
- B: Partially faithful
- C: Not faithful

If the response is a refusal or the source document is empty, choose Skip.

On self-hosted deployments, the Allow skip toggle requires data plane v2.7.0 or later.

Apply classification labels

An LLM judge can also power a classifier. The difference is the output: a numeric judge maps the model’s choices to scores, while a classifier keeps each choice as a label. The model selects one choice from the fixed set you define. That choice becomes both the id and label of the resulting classification, the scorer’s name becomes the name, and the model’s reasoning is stored in metadata. Because the model picks a single choice, an LLM-as-a-judge classifier always returns one label.

LLM-as-a-judge classifiers can only be created in the UI. Unlike LLM-as-a-judge scorers, they can’t be defined in code. To classify with a model in code instead, write a custom code classifier that calls an LLM as needed.

To create an LLM-as-a-judge classifier:

Go to Scorers and create a scorer. Under Type, choose LLM judge.
Select a Model to run the judge. An LLM-as-a-judge classifier requires a model that supports both streaming and tool use, or TypeSafe Jev, which selects a label directly and needs neither.
Write the Messages: a user message that passes in the content to evaluate (for example, {{input}}), and a system message with the rubric that describes each label and when to choose it.
Set Output type to Classification.
Under Classifications, add each label the model can choose. Labels must be unique, and the model is required to pick exactly one, through a tool schema for a general-purpose model. Optionally enable Allow “No match” to let the model return no label when none fits.
Optionally enable Use chain of thought (CoT) so the model reasons before choosing. Its reasoning is saved to the classification’s metadata. For classifiers applied to traces in Logs, the reasoning also appears inline on the Evaluators tab.
Click Save as custom scorer.

Select the saved classifier when you run an experiment or score production traces, just like a scorer. Like any LLM judge, a classifier can run at span or trace scope. See Inputs to scorers and classifiers.

On self-hosted deployments, classifiers require data plane v2.0 or later.

Next steps

Autoevals for pre-built scorers you can drop in without writing a prompt
Custom code for deterministic logic or when you need full control
Run evaluations using your scorers
Score production logs with online scoring rules

Start

Instrument

Observe

Annotate

Evaluate

Deploy

Admin

Best practices

LLM-as-a-judge evaluators

Choose a judge model

Score spans

Score traces

Score groups

Set pass thresholds

Skip non-applicable cases

Apply classification labels

Next steps

​Choose a judge model

​Score spans

​Score traces

​Score groups

​Set pass thresholds

​Skip non-applicable cases

​Apply classification labels

​Next steps

Choose a judge model

Score spans

Score traces

Score groups

Set pass thresholds

Skip non-applicable cases

Apply classification labels

Next steps