Two refinements, one judge

In the previous article we added a ModelJudgeEvaluator to the TripPlanner evaluation, scoring NeighborhoodCoherence through a ScoreDimension and a 1–4 scale. It works, and it’s the shape you’ll find in most sample code.

It’s also more ceremony than that judge needs, and it feeds the judge more text than it should.

This is a short follow-up with two refinements to the same evaluator. The first is cosmetic but removes a whole type from the file: name the metric directly on the evaluator instead of routing it through a ScoreDimension. The second actually changes the scores you get: use evaluationTarget and reference to control exactly what the judge reads. No new concepts, just the same judge, cleaner and sharper.

Naming the metric directly

The version we shipped in part two declares a ScoreDimension, passes it into the evaluator’s dimensions: array, then reaches through .metric twice, once to aggregate and once to assert:

let neighborhoodCoherence = ScoreDimension(
    "NeighborhoodCoherence",
    description: "Are each day's activities geographically coherent with its stated neighborhood?",
    scale: .numeric([
        4: "Every day's activities clearly fit within the stated neighborhood.",
        3: "Most days fit, but one activity feels out of place for its neighborhood.",
        2: "About half the days have activities that don't match their stated neighborhood.",
        1: "The activities bear little relation to the stated neighborhoods.",
    ])
)

// aggregation
aggregator.computeMean(of: neighborhoodCoherence.metric)

// test
let neighborhoodCoherenceRate = result.aggregateValue(
    .mean(of: Self.evaluation.neighborhoodCoherence.metric)
)

For a judge that scores exactly one thing, that dimension is a wrapper around a name and a scale. ModelJudgeEvaluator will take both directly, which lets the dimension collapse into a plain Metric:

// A plain Metric instead of a ScoreDimension
let neighborhoodCoherence = Metric("NeighborhoodCoherence")

// The name and scale go straight onto the evaluator
ModelJudgeEvaluator(
    "NeighborhoodCoherence",
    scale: .numeric([
        4: "Every day's activities clearly fit within the stated neighborhood.",
        3: "Most days fit, but one activity feels out of place for its neighborhood.",
        2: "About half the days have activities that don't match their stated neighborhood.",
        1: "The activities bear little relation to the stated neighborhoods.",
    ]),
    judge: SystemLanguageModel.default,
    prompt: ModelJudgePrompt(instructions: "...")   // full prompt shown in section 4
)

// aggregation — no more .metric
aggregator.computeMean(of: neighborhoodCoherence)

// test — no more .metric
let neighborhoodCoherenceRate = result.aggregateValue(
    .mean(of: Self.evaluation.neighborhoodCoherence)
)

The knock-on effect is the nice part. Because neighborhoodCoherence is now a real Metric, both the aggregation and the test drop the .metric accessor and read exactly like the code-based metrics from the first article. The judge stops looking like a special case in your evaluation file.

One rule to respect, and it’s the kind that costs an afternoon. The string you pass to ModelJudgeEvaluator must match the Metric name character for character. Metric("NeighborhoodCoherence") with ModelJudgeEvaluator("Neighborhood Coherence") compiles perfectly and then fails at runtime with the metricsNotFound error we hit in part two. That string is the only link between the score the judge produces and the metric you aggregate. There is no compiler check on it.

So which form should you use? For a single dimension, the shorthand wins on every count. The named ScoreDimension earns its place when you’re scoring several qualities and reusing those dimensions across aggregation and assertions, the way Apple’s Book Tracker sample does with relevance and usefulness. One name, referenced from four places, is worth a type. One name, referenced from two, isn’t.

Sharper prompts: evaluationTarget and reference

The second refinement is about input. A ModelJudgePrompt carries three distinct things, and part two only used the first:

  • instructions: how to judge
  • evaluationTarget: what to judge
  • reference: what to compare against

evaluationTarget: what the judge actually reads

Easy thing to forget: the judge is a language model. It reads text, not your Swift Itinerary. Leave evaluationTarget out and the framework hands it the default representation of the whole object, pace, totalEstimatedCost, every field, whether or not any of it bears on the question you asked.

evaluationTarget is a closure that picks the part you care about and returns it as text. For neighborhood coherence, that’s one line per day:

evaluationTarget: { itinerary in
    itinerary.days
        .map { day in "\(day.neighborhood): \(day.activities.joined(separator: ", "))" }
        .joined(separator: "\n")
}

Which means the judge now sees this:

Tiergarten: Visit the Reichstag, Walk through the park
Kreuzberg: Street art tour, Turkish market, Dinner on Oranienstrasse

Exactly the neighborhood and activity pairs it needs to answer the question, and nothing else. No budget figures to get distracted by, no pace enum to reason about. Less noise in, less noise out.

reference: context, not an assertion

reference is optional labeled context you hand the judge, as a dictionary of label to value. Here we give it the original trip request, so it can tell an odd-looking pairing from one the traveller actually asked for:

reference: { sample, _ in
    ["Trip request": sample.input.promptDescription]
}

Now the nuance that trips people up, and it’s worth being explicit about. reference is informational only. The framework does not compare it against the target for you. It passes both to the judge, and the judge decides what to do with them.

Contrast that with the budget evaluator from the first article, where we pulled expected off the sample and computed the error percentage ourselves, in Swift, deterministically. That was an assertion. A judge’s reference is context. If you need a guaranteed comparison, write it in code. If you need judgement about how two pieces of text relate, that’s what the reference is for.

The full refined evaluator

Both refinements together, which is the version worth keeping:

let neighborhoodCoherence = Metric("NeighborhoodCoherence")

// inside evaluators:
ModelJudgeEvaluator(
    "NeighborhoodCoherence",
    scale: .numeric([
        4: "Every day's activities clearly fit within the stated neighborhood.",
        3: "Most days fit, but one activity feels out of place for its neighborhood.",
        2: "About half the days have activities that don't match their stated neighborhood.",
        1: "The activities bear little relation to the stated neighborhoods.",
    ]),
    judge: SystemLanguageModel.default,
    prompt: ModelJudgePrompt(
        instructions: """
            You are evaluating a day-by-day travel itinerary. For each day, \
            check whether the listed activities are genuinely coherent with \
            the stated neighborhood — i.e. they could realistically be done \
            there without excessive travel. Penalize itineraries that label a \
            day with one neighborhood but describe activities typical of a \
            different part of the city.
            """,
        evaluationTarget: { itinerary in
            itinerary.days
                .map { day in "\(day.neighborhood): \(day.activities.joined(separator: ", "))" }
                .joined(separator: "\n")
        },
        reference: { sample, _ in
            ["Trip request": sample.input.promptDescription]
        }
    )
)

And the two lines that consume it, both now free of .metric:

// aggregation
aggregator.computeMean(of: neighborhoodCoherence)

// test
let neighborhoodCoherenceRate = result.aggregateValue(.mean(of: Self.evaluation.neighborhoodCoherence))
#expect(neighborhoodCoherenceRate >= 3)

Same judge, same scale, same threshold as part two. One type fewer in the file, and a judge that reads a focused two-line summary with the original request beside it instead of a serialized struct. This is the version the rest of the series builds on, so it’s the one worth copying into your own evaluation.

Key takeaways

  • For a judge scoring a single quality, pass the name and scale: straight to ModelJudgeEvaluator and declare a plain Metric. Aggregation and assertions lose the .metric accessor.
  • Keep a named ScoreDimension when several dimensions are reused across aggregation and tests. That’s when the type pays for itself.
  • The evaluator’s string and the Metric name must match exactly. A mismatch compiles fine and fails at runtime with metricsNotFound.
  • Use evaluationTarget to hand the judge only the text the question needs. By default it receives the whole serialized object.
  • reference is context for the judge to reason with, not an automatic comparison. Deterministic checks still belong in a code-based evaluator.

Watch the series

This article is part of an ongoing AI Evaluations series. If you’d prefer the highlights on video, there’s a summary of the whole series on the Swift with Walid YouTube channel.