Scoring What the Model Read: Metrics, a Judge, and 21 Receipts

Is a half-right answer a pass?

In part four we sent a photo of a receipt to Apple’s on-device model and got a Receipt value back. Store right, date right, and one of the two extracted items was really a product code printed on its own line.

Two fields out of three, one item out of two. Pass, fail, or something in between? Until that answer lives in code, you have anecdotes rather than an evaluation. This article turns it into numbers: a text comparison that forgives formatting, an F1 score for lists, a judge for what code can’t see, and the results on all 21 receipts.

When are two texts the same?

Comparing store names with == would fail almost every receipt. On receipt 000 the model wrote BOOK TA.K (TAMAN DAYA) SDN BHD where the reference says BOOK TA .K (TAMAN DAYA) SDN BHD. Neither is wrong.

private static func normalized(_ text: String) -> String {
    text.folding(options: [.caseInsensitive, .diacriticInsensitive, .widthInsensitive], locale: nil)
        .filter { $0.isLetter || $0.isNumber }
}

static let minimumContainedRatio = 0.6

static func sameText(_ lhs: String, _ rhs: String) -> Bool {
    let a = normalized(lhs), b = normalized(rhs)
    guard !a.isEmpty, !b.isEmpty else { return a == b }
    if a == b { return true }
    let (short, long) = a.count < b.count ? (a, b) : (b, a)
    return Double(short.count) >= minimumContainedRatio * Double(long.count) && long.contains(short)
}

Normalization folds case, accents, and width, then keeps only letters and digits. Digits stay on purpose: 600G and 800G are different products.

Equality after normalization handles formatting. Labels also gain and lose whole pieces, though. The model prefixes an item with its quantity (2 ChicMcMuffin for ChicMcMuffin), or drops a product code (PLUG CHAMPION for EY20 PLUG CHAMPION), and our own references sometimes keep a menu code the model leaves out. Hence the second rule, containment, with a guard: the shorter label must cover at least 60% of the longer one, because OIL sits inside both ENGINE OIL and OIL FILTER.

sameText("PLUG CHAMPION", "EY20 PLUG CHAMPION")   // true, covers 75%
sameText("OIL", "ENGINE OIL")                     // false, covers 33%
sameText("S.H.H. MOTOR", "S.H.H. MOTOR (SUNGAI RENGIT) SDN. BHD.") // false, covers 31%
sameText("WINDSHIELD CLEANER 120ML", "WAXCO WINDSHILED CLEANER 120ML") // false

Eleven matches in our run depended on containment, with ratios between 0.75 and 0.96, and every one was a harmless prefix or suffix. The 60% figure is a judgment call, not a law, but a ratio at least scales with the label where a flat character minimum doesn’t.

The last line is the instructive one. The model helpfully corrected a typo printed on the receipt, and we count that as a failure, because what we measure is fidelity to the paper rather than spelling. The rule is loose in the other direction too: AKODAA still matches AKODA. Knowing those edges is what lets you read the score correctly. A store that passes means the name was copied faithfully, not merely that the model identified the shop.

The date needs none of this. Its @Guide regex already forces YYYY-MM-DD, so == is both correct and strict.

Scoring a list that can be partly right

Four items found out of five deserves a better score than one out of five, and a better score than five out of five plus three invented. Pass or fail can’t say that. F1 can.

static func itemsF1(_ extracted: [ReceiptItem], expected: [ReceiptItem]) -> Double {
    if extracted.isEmpty && expected.isEmpty { return 1 }
    var unmatched = extracted
    var matches = 0
    for item in expected {
        if let index = unmatched.firstIndex(where: {
            sameText($0.name, item.name) && abs($0.price - item.price) < 0.01
        }) {
            unmatched.remove(at: index)
            matches += 1
        }
    }
    guard matches > 0 else { return 0 }
    let precision = Double(matches) / Double(extracted.count)
    let recall = Double(matches) / Double(expected.count)
    return 2 * precision * recall / (precision + recall)
}

An item counts when its normalized label and its price (within a cent) both match. A matched item leaves the pool, so nothing counts twice, which matters on the receipt listing the same snack twice: two lines on the paper need two lines in the answer. Precision says how much of the answer is real, recall says how much of the receipt was found, and F1 is only high when both are.

Receipt 007 lands at 0.67: one expected item, two extracted, one correct, so the stray product code costs a third of the score. Receipt 000 is the harshest. The price is right, but the model read RE MODELLING CLAY KIDDY FISH instead of KF, and a two-letter misreading loses the only item, for a score of zero.

There is no partial credit for near misses. That is a deliberate choice, and an edit distance would be the natural next iteration: it would separate “the model misread a letter” from “the model picked the wrong line”.

Wiring it into the evaluation

The subject calls the exact same code the app calls, which is the whole point of keeping the feature in a shippable package:

func subject(from sample: ReceiptSample) async throws -> ModelSubject<Receipt> {
    let (image, orientation) = try ReceiptImages.load(named: sample.key, in: dataset.images)
    let receipt = try await ReceiptExtractor.extract(
        from: image,
        orientation: orientation,
        prompt: sample.input.prompt
    )
    return ModelSubject(value: receipt)
}

Each comparison then becomes one Evaluator, falling back to ignore when a sample has no reference, which as we saw in the first article is a first-class outcome rather than an error:

Evaluator { sample, subject in
    guard let expected = sample.expected else { return store.ignore(rationale: "No reference") }
    return ReceiptMatching.sameText(subject.value.store, expected.store)
        ? store.passing()
        : store.failing(rationale: "Got \"\(subject.value.store)\", expected \"\(expected.store)\"")
}

The date evaluator is the same shape with ==, and the items one calls itemsF1 and reports it with items.scoring(score, rationale:) instead of passing or failing. Write real rationales. The Xcode report shows them next to each failing sample, and that is how we found most of the model’s mistakes and a few of our own. As in the rest of the series the evaluation runs from Swift Testing through the .evaluates trait, prints a Markdown report, attaches it to the test report, and asserts exactly one thing: every receipt produced an answer.

A judge that reads, but doesn’t see

Code can check whether the store is spelled as printed and whether each item is really on the paper. It can’t say whether the answer, taken as a whole, is faithful to the receipt. That is the judge’s job, following the recipe from part three: a metric named directly on the evaluator, a 1–4 scale anchored in observable facts, and a prompt that hands it only what it needs.

ModelJudgeEvaluator(
    "Faithfulness",
    scale: .numeric([
        4: "Store, date, and every item match the reference, and nothing is invented.",
        3: "Store and date match; at most one item is missing, extra, or mispriced.",
        2: "Store or date is wrong, or several items are missing, extra, or mispriced.",
        1: "The extraction doesn't describe the reference receipt.",
    ]),
    // The judge restates receipt text it's given; default guardrails refused some runs.
    judge: SystemLanguageModel(guardrails: .permissiveContentTransformations),
    prompt: ModelJudgePrompt(
        instructions: """
            You are grading fields that an app extracted from a photo of a store receipt, \
            against a human transcription of the same receipt. Treat differences in case, \
            accents, spacing, and item order as equivalent. Penalize invented items, items \
            copied from subtotal, tax, or payment lines, translated labels, and wrong prices.
            """,
        evaluationTarget: { receipt in receipt.judgeDescription },
        reference: { sample, _ in
            ["Reference receipt": sample.expected?.judgeDescription ?? "No reference"]
        }
    )
)

What the judge never receives is the photo. It compares two texts, so it grades faithfulness to the reference, not reading ability. We could hand it the image, since the model is multimodal now, but then one unverified reader would be grading another and we would need a third evaluation to trust the second.

Two things surprised us. First, guardrails. The default ones made the judge refuse to grade some receipts, apparently wary of restating their content, so it runs with .permissiveContentTransformations. Even then it still refuses the occasional sample, which shows as a dash in the report and counts as an evaluator failure rather than an inference failure.

Second, its temperament. In an early run against placeholder references, where every store was literally PLACEHOLDER STORE, the judge still averaged about two out of four and handed out a few threes. And while greedy sampling made the extraction repeat itself exactly, the judge’s average drifted from 2.80 to 2.75 between two runs of identical code. A judge is a model too. Calibrate it against human scores the way Apple’s BookTracker sample does with Cohen’s kappa, or give the job to a stronger model such as Private Cloud Compute, before you trust that column.

What the numbers actually said

A full run, 21 extractions plus 21 judge scores, takes about 1 min 45 s on a Mac.

The interesting part isn’t the averages, it’s the pattern behind them. The failures cluster:

• Overlaid text wins. On three receipts the model returned a handwritten name scribbled across the top (“tan woon yann”) as the store.
• Columns shift. On one receipt it paired each description with the price of the neighbouring line, so every item was individually plausible and collectively wrong.
• Dates get reinterpreted. One read month first (2019-11-01 for 11 January), one was invented outright (2024).
• Unreadable means invented. On the A4 scan with a tiny receipt in the corner, the model produced a store and a full item list out of nothing.
• Non-items become items. Product codes, the litres and pump line at a petrol station, a discount row.
• Unit price for line total. A recurring, quiet, off-by-a-multiplier error.

Every one of those is actionable. Item codes on their own line could be handled with one more sentence in the instructions, and the evaluation is right there to tell you whether that sentence helped or hurt. That loop, change one thing and re-measure, is what the whole series has been building toward. It only works because the extraction is reproducible: greedy sampling means a moving score is your doing.

Key takeaways

• Match text on purpose. Normalize case, accents, and punctuation, then allow containment with a ratio guard so OIL never matches ENGINE OIL. Document which near misses your rule refuses, because that is what the score actually means.
• Score lists with F1 so a partly right answer scores partly right. Require label and price to match, and consume matched items so duplicates on the paper need duplicates in the answer.
• Rationales are the debugging tool. They surface in the Xcode report next to each failing sample, and they catch mistakes in your references as well as in the model.
• Judge the judge before you trust it. Ours scored 2 out of 4 against placeholder references and drifted between identical runs. Calibrate against human scores, or promote the judge to a stronger model.
• Read the pattern, not the average. 71% on stores is a number. “It picks up handwriting overlaid on the header” is a fix.

The series

• Testing the Untestable: A Deep Dive into Apple’s Evaluations Framework
• When Code Can’t Judge Quality: Model-as-Judge Evaluators
• Two Refinements That Sharpen a Model-as-Judge Evaluator
• When the Model Can See: Reading Receipts with Multimodal Prompting
• This article, which closes the series.

If you’d rather have the highlights on video, there’s a summary of the series on the Swift with Walid YouTube channel.

Receipt photos come from the ICDAR 2019 SROIE dataset, as distributed in zzzDavid/ICDAR-2019-SROIE.