When Code Can't Judge Quality: Model-as-Judge Evaluators in Apple's Evaluations Framework

When code can’t judge quality

In the first article in this series, we built a complete evaluation for a TripPlanner feature: dataset, subject, code-based evaluators, metrics, and aggregation, all run through Swift Testing.

But a code-based Evaluator can only measure what code can measure. Day count, activities per day, budget within tolerance: that’s easy Swift. The qualitative questions aren’t. Is the tone right? Do a day’s activities actually belong to the neighborhood the model grouped them under, or has it filed a museum three districts away under “morning stroll”? Code can’t answer those.

ModelJudgeEvaluator closes that gap. You hand one model’s output to another model and ask it to score exactly the dimensions code can’t reach. It’s a first-class evaluator that plugs into the same metrics-and-aggregation machinery you already know.

The catch, and the through-line of this article: a model-as-judge is only as good as its configuration. Configure it well and its scores track human judgment closely; configure it vaguely and you get numbers that look precise but measure the wrong thing.

This piece assumes you’ve read the first one (or are comfortable with datasets, subjects, evaluators, metrics, and aggregation). We extend the same TripPlanner example with one judge: scoring whether each day’s activities are geographically coherent with their stated neighborhood.

Picking a scoring scale

The scale you pick drives two things at once: how much detail your signal carries, and how reliably the judge can apply it. They pull against each other. A finer scale holds more information but is harder to use consistently, and an inconsistent scale just dresses up noise as precision.

You have three options:

Two rules of thumb follow:

• Use binary for genuinely binary judgments. Forcing a multi-point scale onto a yes/no question just adds noise. The judge clusters around the middle without adding real signal.
• For subjective quality, prefer a small, even number of levels. An even count removes the noncommittal middle a judge otherwise defaults to. Small guards against score compression, where a wide scale gets bunched into a narrow band. That’s a form of leniency (more on judge biases in the next article).

This all lands on ScoringScale, whose factory methods map onto these recommendations (see Apple’s docs on scoring with model-as-judge evaluators). For a binary question, .passFail(...):

// A binary scale for a safety check.
let safetyScale = ScoringScale.passFail(
    passDescription: "Output is safe and on-topic.",
    failDescription: "Output contains unsafe content or goes off-topic."
)

For a scored judgment, .numeric(...) maps each level to a description:

// A four-point scale for haiku quality.
let haikuScale = ScoringScale.numeric([
    4: "Perfect 5-7-5 form, strongly relevant to the topic, uses vivid imagery.",
    3: "Correct or near-correct form, clearly relevant, some evocative language.",
    2: "Incorrect syllable count, weak topic connection, or lacks poetic quality.",
    1: "Not recognizable as a haiku, off-topic, or incoherent.",
])

Our NeighborhoodCoherence dimension is a subjective quality judgment, not a pass/fail. A day can be mostly coherent with one stray outlier. So it gets a small, even scale: 1 to 4.

Writing levels that actually work

The scale is the frame. The level descriptions are what the judge actually reads, and where most evaluators fall apart. The classic mistake is levels that differ only in degree (“good” vs “very good”) instead of observable characteristics. “Excellent quality” vs “good quality” asks the model to report a feeling, and feelings don’t reproduce. “All key points addressed with supporting evidence” vs “most key points addressed but missing supporting detail” asks it to check something that’s actually in the text.

Four rules keep levels on the right side of that line:

• Describe features, not feelings. “Perfect AABBA rhyme, strong meter, surprising punchline” is checkable; “excellent limerick” is not.
• Make adjacent levels distinguishable. The 3-vs-4 gap must be specific enough that two reviewers agree. If levels blur, merge them or sharpen them.
• Anchor every level equally. A vague middle is exactly what makes a judge default there. Describe a 2 or 3 as carefully as a 4.
• Describe boundary cases. Spell out what tips a response up a level: “correct structure but rough execution is a 3, not a 4.”

The difference is easiest to feel side by side. First, the version you should not ship. It’s valid, but its levels differ only in degree:

let neighborhoodCoherence = ScoreDimension(
    "NeighborhoodCoherence",
    description: "How coherent are the activities with their neighborhood?",
    scale: .numeric([
        4: "Excellent coherence.",
        3: "Good coherence.",
        2: "Poor coherence.",
        1: "Very poor coherence.",
    ])
)

Nothing tells the judge what “excellent” means versus “good”, so it reinvents the boundary on every call. Now the version we actually use:

let neighborhoodCoherence = ScoreDimension(
    "NeighborhoodCoherence",
    description: "Are each day's activities geographically coherent with its stated neighborhood?",
    scale: .numeric([
        4: "Every day's activities clearly fit within the stated neighborhood.",
        3: "Most days fit, but one activity feels out of place for its neighborhood.",
        2: "About half the days have activities that don't match their stated neighborhood.",
        1: "The activities bear little relation to the stated neighborhoods.",
    ])
)

The second works because every level describes a countable, checkable state (how many days fit, how badly the misses miss), not a grade the judge has to conjure. It’s no longer asked how it feels. It’s asked to count and compare, which a model does consistently.

Wiring the judge into TripPlanner

Now we wire neighborhoodCoherence into the TripPlanner evaluation from the first article, running it alongside the code-based checks.

Adding the judge alongside the code-based evaluators

A ModelJudgeEvaluator is just another member of the same evaluators block. List it after the existing code-based Evaluators, with no commas, and the framework runs the whole set for every sample:

var evaluators: Evaluators {

    // ... the three code-based evaluators from the first article
    // (daysCount, activitiesPerDay, budgetWithinTolerance)

    ModelJudgeEvaluator(
        judge: SystemLanguageModel.default,
        dimensions: [neighborhoodCoherence],
        prompt: ModelJudgePrompt(instructions: """
            You are evaluating a day-by-day travel itinerary. For each day, \
            check whether the listed activities are genuinely coherent with \
            the stated neighborhood — i.e. they could realistically be done \
            there without excessive travel. Penalize itineraries that label a \
            day with one neighborhood but describe activities typical of a \
            different part of the city.
            """)
    )
}

Three arguments do the work:

• judge: is the model doing the scoring. Use SystemLanguageModel.default, the on-device system model (sharp edge here; see the gotchas).
• dimensions: is the array of ScoreDimensions to score. Just neighborhoodCoherence here, but list several if the judge scores multiple qualities.
• prompt: is a ModelJudgePrompt telling the judge what to look for and penalize. Specificity is the whole game: name the failure mode you care about (activities that don’t match the neighborhood), not a vague “rate the quality.”

Aggregating the dimension

A ScoreDimension isn’t a Metric, but it exposes one via .metric, which aggregates exactly like the code-based metrics. Add one line to aggregateMetrics:

func aggregateMetrics(using aggregator: inout MetricsAggregator) {
    aggregator.computeMean(of: activitiesPerDay)
    aggregator.computeMean(of: daysCount)
    aggregator.computeMean(of: budgetWithinTolerance)
    aggregator.computeMean(of: neighborhoodCoherence.metric)
}

Asserting on it in the test

Same pattern as before, via .metric. Mind the scale: this is a 1–4 dimension, not a 0–1 pass ratio, so the threshold reflects that. A mean of at least 3 out of 4 says “most days are clearly coherent, a stray outlier at worst”:

let neighborhoodCoherenceRate = result.aggregateValue(
    .mean(of: Self.evaluation.neighborhoodCoherence.metric)
)

#expect(neighborhoodCoherenceRate >= 3)

Two gotchas worth flagging

Both cost me real time, and neither error message points at the cause.

judge: needs the concrete type. Its parameter is typed any LanguageModel, so .default alone won’t resolve. You get type 'any LanguageModel' has no member 'default'. Write SystemLanguageModel.default. A bare SystemLanguageModel() fails too.

A broken judge hides your other metrics. If the judge fails per sample, you get a .metricsNotFound(names: [...]) error listing all your metrics, not just the judge’s. An unhandled error in one evaluator blocks the others from recording for that sample. Don’t hunt through the code-based evaluators; fixing judge: is almost always it.

Expect it to be slower. Every sample now carries an extra model call, so runs on a larger dataset take noticeably longer than the pure-Swift checks. They do complete, though.

Coming in the next article

You now have a model-as-judge running, but running isn’t the same as trustworthy. A judge only earns its scores once you’ve validated it. The next article in the series digs into exactly that: the systematic biases a judge can pick up and how to mitigate them, how to debug one by reading its written rationales, and how to calibrate its scores against human experts using Cohen’s Kappa.

Watch the series

This article is part of an ongoing AI Evaluations series. If you’d prefer the highlights on video, there’s a summary of the whole series on the Swift with Walid YouTube channel.