We replaced an LLM with a model that answers in under 80ms, and it got more accurate

A client of ours has a narrow image-to-text task: turn a small image into a short code. A model we trained ourselves now does that job. On a held-out, human-verified test set it reads 97.4% of the images correctly, with a median answer time of 76ms on a serverless instance with one vCPU and 512 MiB of memory. GPT-4o, the LLM it replaced, read 78.2% of the same set in about 700ms.

We expected the small model to be faster, not more accurate.

OCR was not enough

OCR is the obvious first tool. A general-purpose cloud OCR service reads 19.4% of the test set correctly, because OCR engines are tuned for document text, and this is a different kind of input.

A dedicated model, until the LLMs read more

So we trained a dedicated model. It worked, but once multimodal LLMs came along they read more of the images correctly, and the model needed labelled examples that a person typed one by one. Each time the input changed we needed thousands of new ones.

An LLM got it running

With a multimodal LLM the service ran after a prompt and no training at all. We also logged every answer next to its image, and over time that log grew into a labelled data set we had not planned for.

Where the LLM fell short

GPT-4o answered in about 700ms. The most accurate model in our bench, Gemini 4 Argon, read 97.6% but took 3.9s at the median and 42.0s at the 95th percentile, and inside a 2s budget no LLM, alone or combined with others, got past about 87%.

Capacity also moved under us. Providers shift it to their newest model versions: a Gemini Flash model that was current a year earlier took 52s per image at the median when we measured it, and newer models queued our requests for up to two minutes. OpenAI shuts GPT-4o down in its API on January 14, 2027, so the model we had tuned our prompt for has an end date.

Buying guaranteed capacity does not fix this for a task this size. Provisioned throughput is billed per hour whether requests arrive or not, every model has a minimum deployment size, and the vendor documentation warns that capacity might not be available when you scale back up.

Back to a dedicated model

This time the labels were already there: we trained on the logged LLM answers. Against every model we benchmarked, the result lands within a point of the most accurate LLM and answers faster than any of them.

How we know the labels hold up

A model trained on LLM labels can learn the LLM's mistakes and still look good if it is graded against the same labels. So the test set never came from the LLM: for each of the 454 test images, two people answered independently and agreed, and the model never trained on any of them.

Training cleaned up the rest. Wherever the model and its label disagreed, the image went back to a person. Out of about 4,300 training images, 54 labels turned out wrong and 7 images were unreadable, so about 1.4% of the labels needed fixing.

One test set can still be lucky, so we also rotated it. Cross-validation splits the training images into five parts. The model trains on four and is scored on the fifth, and each part takes a turn as the unseen test set. Across the five rounds the score lands within a point of the result on the 454 people-checked images, and retraining with different random seeds moves it by about a point.

We also checked that the test set does not leak into training. Every test image was compared against the whole training set by perceptual hash, and even the closest match had a different answer. None of the 454 test answers appears anywhere in the training data.

What owning the model changes

One vCPU handles eight concurrent requests with a 95th percentile under 0.5s, and nothing waits in a queue at a provider. The model has no retirement date either: it stays as it is until we retrain it, and scaling up means another small instance instead of a capacity reservation.

The LLMs cost between $0.001 and $0.005 per correct read. At this client's volume that was a small bill, but it grows with every request, while the dedicated model costs close to nothing per request.

The model needs one CPU thread and a few hundred MiB of memory and answers in 34ms on a laptop, so it would fit on a small machine at the edge.

What still uses an LLM

The model does not answer everything. Each answer carries a confidence, calibrated on the cross-validation predictions, and the few images below the cutoff still go to an LLM as a fallback.

An LLM was the right way to start: it got the service running quickly, and because we logged its answers, it produced the training data for the model that replaced it.