A Model Is Not a Model

I hear the same sentence in planning meetings, said as if it settles the question. A model is a model. If the hosted price bothers us, we will run an open weight model ourselves and the problem goes away.
It does not. Not yet. That last word matters, and I will come back to it. First I need to name the two errors packed into the premise, because each one costs money on its own.
Models are not fungible
Capability is task shaped. A model that writes clean summaries can fail at tool use. A model that codes well on easy tasks can fall apart on the hard tail where the actual value sits. Swapping the model changes behavior, failure modes, and the verification load that follows the work. It is not a commodity substitution.
EvalShift’s analysis of what breaks when you switch LLMs puts it plainly: a model swap is not a config change, it is a behavior change delivered through a config file, with no diff for anyone to review. Which tools an agent calls, the order of those calls, the arguments, the structured output, the refusal behavior, the verbosity, the latency, and the cost can each move on their own. The worst case is quiet. The output text still looks correct while a tool call that was supposed to happen silently stops, so output only scoring cannot see the regression.
Prompts make it worse. A prompt tuned against one model is coupled to that model. Swap the model and I usually drag a prompt edit in as a second uncontrolled change. Now I have two changes and no clean way to say which one moved the result.
The numbers back this up. In Natum’s paired replay study, a candidate model served 2.5 times the throughput of the current model on 175 LiveCodeBench tasks and came close on easy tasks, 93 percent against 100 percent. On hard tasks it fell from 50 percent to 12 percent. Overall it solved 38.3 percent against 71.2 percent. Aggregate scores hid the hard tail. Most enterprise workloads earn their keep in that tail. The same study sizes the replay needed to detect a five point drop at roughly 538 paired tasks. That is the verification bill for a swap, and it arrives whether I budgeted for it or not.
I made a version of this point in The Agent You Approved Is Not the Agent Running: a model swap reopens approval because it changes the thing I approved. The same logic applies to procurement. If swapping changes behavior, then models are not interchangeable parts. Treating them as interchangeable is a planning error, not a sourcing strategy.
Open weight removes the license line, not the cost
The second error is accounting. Open weight removes a license line and a per token API price. It does not remove serving infrastructure, capacity planning, evaluations, update and patch cadence, monitoring, redundancy, or the team that runs all of it at 2 a.m. when it breaks. Per token price is the smallest line in the ledger. The run is the bill.
LeanLM’s self-hosting cost analysis works the arithmetic with October 2026 rates. At full saturation, a single H100 serving gpt-oss-120b reaches roughly $0.10 per million tokens in raw GPU cost. That sounds like a rout until I set it next to the hosted price for the identical model: $0.60 per million output tokens, and as low as $0.17 at one provider. The margin at perfect utilization is a few times, not the hundred times the hourly arithmetic suggests. At 10 percent utilization every self hosted number gets ten times worse. All in cost runs 3 to 5 times the raw GPU bill once engineering, redundancy, monitoring, and idle time are counted. For a DeepSeek class 671B model, hosted pricing beats raw self host GPU cost at every utilization level.
OpsMeter makes the companion point from the operations side: a lower token price does not guarantee a lower total cost, because a cheaper model that raises failure and retry rates erases the saving. The unit it recommends is cost per successful outcome. That is the same unit I argued for in Route by Outcome, Not by Token.
None of this says open weight never wins. LeanLM is explicit that self hosting beats frontier API pricing at sustained volume. It loses mainly against hosted open weight endpoints serving the same model. That distinction matters. The honest comparison is not open weight against the most expensive frontier API. It is owning the run against every other way to get that same model served, including someone else serving the same weights for me.
The yet, stated as tests
I said not yet, and I mean it as a testable claim, not a hedge. The strategy has arrived when four conditions hold on my own work, not on a public benchmark chart:
- A serving stack runs open weights at production reliability without a platform team standing behind it. Today the cost sources all assume that team exists. LeanLM works with a figure of one mid level MLOps engineer covering only 4 to 6 GPUs. Until the stack absorbs that role, the other tests sit downstream of it.
- Evaluations show parity on my own task classes, including the hard tail, at a replay size large enough to detect a real drop.
- An update path ships a new version of the weights without reopening verification from zero.
- A routing record shows cost per accepted outcome beating the hosted tier on real work, over time, with failures and retries counted.
When those four hold, I will change my answer. I will be able to point to the record that changed it. Until then, calling open weight the low cost answer is asserting the conclusion before the evidence exists. I covered the benefit side of the same ledger in Production Is Not a Result: production is not a result, and a deployed model is not a saved dollar until the outcome record says so.
Put it in the route table, not the strategy deck
Open weight belongs in my route table as a tier with an outcome record, not in the strategy deck as an answer. As a tier, it earns traffic the way every other tier does: task class by task class, measured on cost per accepted outcome, demoted when the record sours. As a deck answer, it bypasses all of that. It decides in advance, for every task, a question that only measurement can settle per task.
So the two questions I ask when the sentence comes up in a meeting are simple. First, what did the candidate model do on our hard tasks, in a paired replay large enough to matter? Second, who owns the run when the weights are ours, and what does that ownership cost at our actual utilization, not at saturation?
If nobody can answer those two, we do not have a model strategy. We have a price preference wearing a strategy label.
Thinx-Tank