Issue 11 · Tools & Software · Jul 28, 2026

Where the 2.5% lands

Ten rate confirmations through two AI models, sixty runs, code published. The cheap one got 97.5% of fields right — and every miss landed on the rate, the MC number, or the load number.

Last week I wrote that pointing a cheap model at a rate confirmation costs about half a cent a document. I labelled it arithmetic rather than a measurement, because that is what it was, and said I would go find out.

Here is what it actually costs, what it actually gets wrong, and the part I did not expect.

The setup

I built ten rate confirmations, each in a layout designed to break something different:

The bordered carrier packet — all-caps labels, fax banner across the top, SHIPPER and CONSIGNEE instead of pickup and delivery.
The modern SaaS confirmation — clean, minimal, Order # rather than Load #, and the rate split across linehaul, fuel surcharge, reefer differential, and a volume discount that has to be summed.
The legal document — no rate field at all; the number lives inside a paragraph, spelled out in words and then in figures.
The multi-stop — two pickups and one delivery, so “the pickup date” is a question rather than a lookup.
The dot-matrix printout — monospace, fixed columns, dates as 29-JUL-26, city abbreviated to INDPLS.
The cross-border move — military time, ISO dates, weight given in kilograms and pounds.
The spot load — a posted range, an agreed rate, and a not-to-exceed ceiling, all three on the page as dollar figures.
The email forward — headers, a quoted reply chain, two superseded rates and three dates in the thread below.
The team expedite — a conditional on-time bonus listed under the confirmed rate, and both the broker’s and the carrier’s MC numbers on the page.
The handwritten override — printed rate struck through, a new one written beside it in pen.

Then I made them look real. Each was rendered to PDF, rasterised at 150 DPI, rotated slightly off-square, blurred, contrast-shifted, speckled, and saved as a low-quality JPEG before being converted back. The result looks like it came off a fax or a phone camera, because that is what arrives in a real inbox.

Each document went through the model three times — thirty runs — scored against known answers on twelve fields: load number, broker name, broker MC, pickup date, pickup city, delivery date, delivery city, total rate, equipment, weight, commodity, detention terms.

The limitation, stated up front

I wrote these documents. I also graded them. They mirror real broker layouts, but they are not ten real rate cons pulled from a carrier’s inbox, and I am not going to pretend the accuracy figure forecasts what your paperwork will do. A real inbox has more repetition than this — the same five brokers, over and over, which probably helps — and worse scans than I simulated, which probably hurts.

What the test measures honestly: the cost, the latency, and how a model handles specific document structures. That is worth knowing. It is not the same as a field study, and the difference matters enough to say twice.

What it cost

$0.00337 per document. A third of a cent.

My estimate was $0.0055. I was 63% too high — in the reader’s favour, which is the direction you want to be wrong in, but wrong is wrong and the correction belongs in print.

At roughly 200 rate confirmations a month, a seven-truck carrier would spend sixty-eight cents. Not sixty-eight dollars.

Whatever is keeping operators typing these by hand, it is not the API bill.

What it got right

97.5% of fields — 351 of 360.

Every pickup date. Every delivery date. Every city on all but one document. Every weight, including the one stated in kilograms that had to be converted. Every commodity. Every equipment type, including the reefer with a continuous temperature requirement. Every broker name. Every detention clause, including the one buried in a paragraph of legal boilerplate and the one that only applies at the first stop.

It correctly ignored the posted range and the not-to-exceed ceiling on document 7 and took the agreed rate. It correctly excluded the conditional on-time bonus on document 9. It correctly summed four line items into one total on document 2. It correctly picked the first of two pickups on document 4.

That is genuinely good, and it is worth saying plainly before the next section, because the next section is going to sound harsh.

What it got wrong

Nine of 360 reads. All nine sat in three fields, on three documents.

The rate

Document 10. Printed: $1,475.00, struck through. Handwritten beside it: $1,650 — per Mike 7/27.

A rate confirmation row. The printed figure $1,475.00 is struck through, and $1,650 — per Mike 7/27 is handwritten beside it with a check mark. — Document 10 of 10. The model read $1,475, three times out of three.

The model took $1,475. Every time.

That is $175 wrong, in the broker’s favour, on the field that becomes your invoice. And note what it is not: it is not an OCR failure. The handwriting is legible — a human reads that page correctly in about a second. The model saw a printed number and a handwritten number and preferred the printed one.

The prompt explicitly told it not to. The instruction “if a printed value is struck through and a handwritten value replaces it, the handwritten value governs” was in the request. It did it anyway, three times out of three.

The broker’s MC number

Document 9. Two MC numbers on the page: the broker’s MC-1455301 in the letterhead, the carrier’s MC-1622077 in the table below.

The model returned the carrier’s. Every time.

This is not a reading error either — both numbers were transcribed accurately. It is a role confusion: it knew what an MC number looked like and picked the wrong party’s. That is the field you would use to vet a broker, check their authority, or demonstrate later that you checked.

The load number

Document 8. TQ-441907 came back as TO-441907.

The most boring failure and, in some ways, the most expensive one. A wrong load reference does not look wrong. It flows into your invoice, the broker’s accounts-payable system rejects it, and you find out three weeks later when the payment does not arrive.

The finding that changes how you use this

Every one of those errors repeated on all three passes.

They are not random. They are not a bad-luck sample. The same document produces the same wrong answer, reliably. You cannot retry your way out of a layout the model reads wrong, and you cannot catch it by spot-checking a sample — if a broker’s format triggers one of these, it triggers it on every document that broker sends you, silently, forever.

That is a fundamentally different risk than “the model is 97.5% accurate” implies. A 2.5% random error rate is noise you can average out. A 2.5% deterministic error rate concentrated in one broker’s layout is a systematic leak.

Where the errors land is the whole story

Twelve fields. The three that failed:

Field	What it becomes
The rate	Your invoice
The broker’s MC	Your vetting record
The load number	Your payment reference

The nine that never failed are the descriptive ones — what, where, when, how heavy. Facts about the freight.

The model is close to flawless at everything that costs you nothing, and unreliable exactly where being wrong costs you money.

That is not a coincidence, and once you see why, it stops being surprising. Those three fields are the contested ones — the places where the document disagrees with itself. A struck-through number and a handwritten one. Two MC numbers for two different parties. A character that looks like another character. Everywhere the page says one thing clearly, the model reads it correctly. Everywhere the page requires a judgement about which of two things is true, it can go wrong.

Last week’s rule was automate the typing, never the deciding. The measurement sharpens it: some of the typing is deciding, and it is the typing that turns into money.

The part I did not expect

I ran the identical corpus — same documents, same prompt, same scoring — through a stronger model, Claude Sonnet 5.

It got all three right. The handwritten $1,650. The broker’s MC. The TQ.

Zero factual errors across all 360 field-reads.

The price of that: $14.95 per thousand documents, against $3.37 for the cheap one. About 4.4 times more.

Which is where the arithmetic gets faintly ridiculous. At 200 rate confirmations a month, the difference between the model that got the rate wrong and the model that did not is about two dollars and thirty cents.

I spent last week’s issue and most of this one carefully measuring a cost that turns out not to matter. Both options round to free. The entire question was never how cheap — it was which one gets the invoice right, and I would have found that out faster by asking the second question first.

Two honest caveats. Ten documents and thirty runs is a small sample, and a clean sweep at that size is decent evidence of a gap between two models — not evidence that the better one does not make mistakes. And its two deviations from my answer key were not errors at all: it returned Indpls, IN where the printout literally says INDPLS, and Sparks/Reno, NV where the page says Sparks/Reno, NV. Faithful reads, both. Though Indpls is also a lousy TMS key, which is its own small lesson — accurate and useful are not the same thing.

What I got wrong in my own test

Two of the failures I first recorded were mine, not the model’s.

My scorer required the detention numbers to match as digits, so a model that wrote “$60/hour after two hours free” was marked wrong against “after 2 hours.” And my answer key insisted on Reno, NV for a facility the document lists as Sparks/Reno, NV, where both answers are defensible.

Fixing those moved the cheap model from 95.8% to 97.5% — I had been grading my own ambiguity as its mistake. The corpus now declares alternates explicitly, and the rule was written before I knew which model it would end up helping.

I mention it because a benchmark whose author never reports scoring bugs is a benchmark whose author was not looking for them.

The counter-argument worth taking seriously

Michael Mignano at Union Square Ventures made the case last week that the best work goes into companies “not just building software to automate existing markets… but to fully obliterate them instead.” Automation optimises a broken process; obliteration removes the need for it.

Applied here, he is right. The only reason a rate confirmation needs transcribing is that our industry moves structured data around as a PDF attachment. Extracting it with a model is automating a process that arguably should not exist. EDI and APIs are the obliteration path.

Except that path is jammed. Hirschbach’s CTO, on why: “The biggest bottleneck is not usually the EDI standard itself. It is the partner-specific variations.” One large carrier maintains hundreds of integrations.

And here is the part that decides it for our reader: a ten-truck carrier has no leverage to obliterate anything. Hirschbach can push a vendor onto a modern integration platform. You cannot make a broker change the PDF it sends you. Automation is what is available to you, and that is a statement about market power, not about which architecture is better.

What to actually do

If you are building the rate-con pipeline from last week’s Tool of the Week, two changes:

1. Use the better model. The cheap-model instinct is wrong here. Not because the cheap one is bad — it is genuinely good — but because the saving is rounding error and the mistakes are not.

2. Highlight three columns in your sheet: Rate. Broker MC. Load number. Those get human eyes before anything is invoiced. The other nine go through unread.

Do the second one regardless of which model you run. The reason to check is not that the machine is unreliable; it is that those three fields are where a wrong answer costs you money instead of a correction. That is a property of the field, not of the model, and it will still be true when the models are better than they are today.

Two minutes of spreadsheet formatting. It is the cheapest control in this entire workflow, and it is the one the measurement paid for.

Run it yourself

The corpus builder, the extraction runner, the scorer, the answer key, and all sixty raw runs are published. It is about four dollars of API spend to reproduce end to end.

If you want the number that actually matters — what this does on real broker paperwork — send me ten rate confirmations, redacted however you like. The pipeline takes real documents without modification. I will publish what it finds, including if it embarrasses the conclusion above.

Reply to the Tuesday email, or write to editor@freightsignal.ai.

Issue #11 of Freight/Signal · back to all issues · subscribe to the Tuesday newsletter

Sources

Read this in full each Tuesday