Benchmarking open-weight models for regulated back-office work

There is a stubborn myth that only a frontier cloud model can do useful work, and that anything you can run yourself is a toy. In 2026 that is simply out of date for the tasks most businesses actually care about. But the counter-myth, that open models are now just as good at everything, is also wrong. The truth is more useful than either slogan, and this paper is about finding it for your own work rather than arguing about leaderboards.

Where open models are genuinely good now

The enterprise sweet spot is drafting, summarising, classifying, extracting and answering questions over your own documents. On these tasks, leading open-weight models like Llama 4, Qwen 3.5, Mistral and DeepSeek score within a few points of closed frontier models on standard knowledge benchmarks, according to public leaderboards, and in practice are widely judged production-ready. If your use case is a private assistant that reads your files and helps people write, the model is not your bottleneck any more.

Where open models still trail is the hardest end: multi-step agentic coding, the trickiest reasoning, the frontier of the frontier. If that is your core workload, weigh it honestly. For the back office, it rarely is.

Why leaderboards will mislead you

Treat public benchmark scores with suspicion, for two reasons. First, the classic benchmarks are saturated. MMLU, the one everyone quotes, is so close to maxed out that a two-point difference between models is noise, not signal. The field has moved to harder tests of coding, reasoning and long-context retrieval precisely because the old ones stopped discriminating. Second, and more important, none of these tests are your work. A model that tops a leaderboard on general knowledge can still be mediocre at summarising your specific contracts in your specific house style. The only benchmark that predicts your results is your tasks.

Provenance and licence are part of the score

For regulated work, raw capability is not the only axis. Two others matter as much. The first is provenance: where the weights came from, whether they are signed, whether you can pin the exact version you evaluated so it cannot change under you. The second is licence, which varies a lot between open models and decides whether you may actually use a given model commercially. It is worth noting that several of the strongest open models in 2026 come from Chinese labs, which is not a problem in itself when you run the weights entirely on your own hardware and nothing phones home, but it is a fact a careful buyer should know and reason about rather than discover later.

The reassuring part of running open models locally is that provenance becomes something you can actually control. You choose the model, you verify it, you host it, and it stays exactly as you left it. There is no silent upgrade that changes its behaviour mid-quarter, which is a real problem with rented models and a real headache for anyone who has to keep an evaluation valid over time.

A short evaluation protocol

Put together, a defensible model choice looks like this. Assemble a small suite of your real tasks with a known good answer for each. Shortlist two or three open models whose licence fits your use. Run the suite, blind if you can, and have a domain expert grade the outputs against your standard, not a generic one. Check provenance and pin versions. Re-run the suite when you change anything. It is not glamorous, and it is far more reliable than trusting a number someone else produced on a test that has nothing to do with your business.

Benchmark figures cited here come from public leaderboards rather than primary lab reports, and model capabilities move quickly. Treat this as a method, not a ranking. The method outlives the models.