How Volume and Intent Change How We’re Using It
The first era of enterprise AI is ending: You picked a vendor, e.g. OpenAI, Anthropic, Google – and their best model. You wrote better prompts, layered retrieval on top to feed your data into it, and paid by the token. It was, structurally, a subscription.
Why did it work? Due to capability, you’d only choose the best model anyway, and the volume was low enough for it not to hurt your wallet (well, until now, I’ve talked about this in my last article…)
Both conditions have now reversed. And the second era, the one being built right now, looks fundamentally different. Not one model behind one API, but a distributed estate of models at different capability tiers, running in different places, under different governance regimes, with a routing layer deciding what goes where.
Here's what's driving that, why the technology finally supports it, and where I think it increases the risk for the companies that approach it the wrong way.
The Unpredictability of Geopolitics
I’ve talked about this in one of my previous articles: On the evening of Friday 12 June 2026, the US Department of Commerce issued an export control directive covering two of Anthropic's frontier models. The order restricted access by any foreign national, including foreign nationals inside the United States, and the company's own overseas employees.
No provider can verify nationality in real time across hundreds of millions of accounts. The only compliant response was a global shutoff. The models went dark for everyone, with no notice. Controls were lifted on 30 June and access restored on 1 July; re-enablement across AWS, Google Cloud and Microsoft Foundry lagged further behind.
Eighteen days. Nothing to do with uptime, capacity, billing, or contract terms.
I want to be careful with this example, because the point isn't the politics, and it isn't a criticism of the vendor. Anthropic disputed the rationale publicly and the restriction was withdrawn. The point is that a risk nobody had modelled turned out to be real, and it originated in policy rather than in a boardroom. It could land on any provider, in any jurisdiction, next time.
What that fortnight demonstrated is that AI has stopped being software and started being infrastructure with geopolitics added-on.
Era two starts from that premise.
Force One: Volume Drives Up the Price
The International Data Corporation's projection is that by 2027, token and API call volumes among the Global 2000 will grow roughly a thousandfold, with the number of deployed agents growing tenfold.
The problem? The per-token cost base. Because what’s next? AI agents – and they do more than just one thing: while a single model call may be cheap, an agent handling one claim may classify the document, extract twenty fields, validate each against policy, check for duplicates, decide a routing path, draft a summary and write an audit entry. That's eight+ inference (that’s what the part is called that means the trained AI model is putting its input data to work) calls for one unit of work. Under a single-model architecture, every single one bills at flagship rates, including the six that a small, well-tuned model would handle indistinguishably.
Gartner expects at least 15% of day-to-day work decisions to be made autonomously through agentic AI by 2028, up from effectively zero in 2024, and around a third of enterprise software applications to include agentic AI by the same year. Who’s paying for it? You – if you’re not careful that is.
It's why Gartner also predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027 – citing escalating costs, unclear business value and inadequate risk controls. Ironically, we have seen Microsoft itself cancelling Claude Code licences for its own employees on cost grounds, shortly after encouraging staff to use it for their daily work.
Is this a solvable problem? Yes, the evidence is fairly strong. A study published on arXiv in May compared frontier models against a domain-trained small model on structured contract extraction: the self-hosted specialist matched or beat the frontier baselines on aggregate accuracy at roughly £0.01 per document, against £0.11–£0.34 for the frontier APIs. Gartner expects organisations to use small, task-specific models at least three times more than general-purpose large language models by 2027, for the simple reason that most real-world tasks never needed frontier scale to begin with.
While frontier models won’t go anywhere, they will be reserved for the work that warrants it in the future.
Force Two: Jurisdiction Becomes an Architecture Constraint
The second force is regulatory.
EU AI Act obligations for high-risk systems has just landed, including Article 12 requirements for automatic event logging across a system's entire lifetime.
If your audit trail runs through infrastructure you don't control (e.g. Microsoft’s, Open AI’s or Anthropic’s), in a jurisdiction subject to extraterritorial legal process, that trail is not only structurally incomplete but also hard to defend in review.
That's the near term. The medium term is more structural. Gartner expects fragmented AI laws to cover half the world's economies by 2027, driving something like £3.7 billion in compliance spending, and around 35% of countries being “forced” to use region-specific AI platforms on grounds of compliance and linguistic constraints. You can see where this could become a challenge for a multinational.
We’re already seeing self-sufficiency out in the wild. Sovereignty has moved from procurement preference to procurement requirement in regulated sectors. HSBC for example is already running self-hosted Mistral models for credit assessment and compliance review. This will spread outward from finance, healthcare, legal, defence and government, as these patterns always do.
The design implication is straightforward: where all of this logical reasoning takes place determines how you design your architecture and where your data is stored. But that’s currently not the case for many organisations.
Force Three: The Hardware Is Already Available
The good news is: none of this matters if you can run those capable models yourself without a data centre.
Does the current hardware market have what it takes for the average organisation to seriously consider local LLMs? Yes, many manufacturers have heavily invested in hardware that are specifically designed for running autonomous agents like NVIDIA’s DGX Spark, Acer's Veriton GN100, ASUS’ Ascent GX10, Dell’s Pro Max GB10 and many other manufacturers like MSI, Gigabyte, HP, Lenovo, Apple – you name it, there’s a solution for the type of workload you’re using it for.
I recommend that you carry one important technical point into any buying conversation: inference has two phases with opposite hardware appetites. Prefill, which means reading the prompt, is compute-bound. Decode, which generates the answer, is memory-bandwidth-bound, roughly bandwidth divided by model size.
In buying terms that comes down to two numbers. The compute figure – for example NVIDIA’s Spark's petaflop of FP4 – is engine size: the bigger it is, the faster the machine reads a long prompt. Memory capacity is desk space, because the prompt has to sit somewhere while it's being read; 128GB is the going rate in this class, and below that, I can tell you that speed stops mattering. Two further things are worth knowing. Software maturity is one: you'll have seen NVIDIA mentioned as one of the main players in the AI game. For this reason, NVIDIA's CUDA is well established where AMD's ROCm is less so, so the same specification sheet yields less in practice. Sustained cooling and power are the other, since a small box can hit its advertised number in a burst and then throttle under load because it becomes too hot. The rule of thumb: long prompts and short answers – buy compute; short prompts and long answers – buy bandwidth.
IDC expects 80% of CIOs to turn to edge services which means inference running out where the work happens – in a branch office, on a factory floor or on the laptop itself, rather than in a central data centre – to meet AI inference demand by 2027. The direction of travel on inference is outward from the centre, even as training stays firmly in the hyperscalers' hands.
The Question Stands: What Will Era 2 Look Like?
Put the three forces together and the destination is reasonably clear. We’re looking at a model estate – tiered by capability and spread across locations – governed as a portfolio.
A tiered model layer. High-volume, narrow, well-defined work – classification, extraction, routing, summarisation, translation, grounded Q&A – served by small open-weight models running on infrastructure you control. Genuinely open-ended reasoning, novel generation and high-stakes judgement escalated to the frontier. Confidence and data privacy thresholds decide the boundary.
A routing layer as core infrastructure. Gateways like LiteLLM, OpenRouter and Portkey already do this commercially. The strategic value lies in swapping a model becoming merely a configuration change.
An inference platform as a named layer in the stack, with unified control over placement, policy and lifecycle across cloud, on-premises and edge.
Graceful degradation as a design principle. No single model's disappearance should cascade into total pipeline failure. That should now be a standard architectural review question, in the same category as "what happens if this region goes down."
Evaluation as an institutional asset. This is what I consider most important. In a world of interchangeable models, your private evaluation set – based on your own data, reflecting your own quality bar – allows you to switch suppliers in a week rather than a quarter. While public leaderboards give you a shortlist, only your own evaluations give you a decision. In my opinion, these will become the durable competitive assets a company can build in this period, and currently almost nobody is resourcing it properly.
What Has to Go Right, and Probably Won’t Everywhere
I'd be doing you a disservice if I presented this as inevitable. It isn't, and the frictions are substantial.
The economics only work above a threshold. Genuine break-even estimates cluster around sustained volumes of roughly 10 million tokens a day, somewhere near £3k–£4k a month in equivalent API spend. That is a lower bar than it sounds: a handful of agentic workloads, or a few heavy users (see my last article), can clear it inside a month. Below it, APIs win comfortably once you price in deployment engineering, monitoring, patching and incident response. If you want to be on the safe side, allocate 20–30% of a senior engineer to keep a production inference stack healthy.
You'd be buying into a memory shortage. With many data centres being built around the world and the long-term memory supply contracts that go along with them, there’s been an ongoing memory shortage. While the price rises have slowed down and we even experienced a brief dip in 2026 thanks to new memory compression techniques, manufacturing capacity still hasn’t opened up, not helped by the fact only three companies provide about 95% of the world’s DRAM supply. As such, forecasts indicate 2027 or even 2028 at the earliest for prices to fall. That means local AI hardware is – for the moment at least – only getting more expensive.
Idle capacity is pure waste. A box at 20% utilisation has an effective cost per token five times the spreadsheet figure. The reminder here is that cloud scales to zero, but your hardware doesn't.
The capability ceiling is real. Novel code generation, genuinely long-context analysis, hard multi-step reasoning, low-resource languages – that’s where you’d use cloud-based models, and the gap on the hardest work is not closing as fast as leaderboard charts imply.
Safety work becomes your work. Content filtering, prompt injection defence, output sanitisation, CVE monitoring on model repositories, jailbreak resistance. Hosted providers are already doing all of this – and you'd be taking that on.
And the governance maturity isn't there yet. Roughly 21% of organisations report a mature governance model for autonomous agents. McKinsey finds 62% experimenting with agents and fewer than a quarter scaled to production. So if you think back of Gartner’s 40% cancellation forecast – this is why. Era two rewards deliberateness rather than speed.
What Changes Inside the Business
The architecture shift drags an organisational one behind it.
Procurement changes shape. Model supply chain becomes a diligence question: what's the vendor's export control exposure, what are the notice periods, what's the exit path, and – the new one – can we obtain weights or an equivalent fallback if hosted access stops?
A model estate function appears. Someone owns the inventory: which models run where, on what licences, at what versions, against which evals, with what fallback. Today this is scattered across three teams and documented nowhere.
The CFO conversation changes. AI spend will become a mix of fixed infrastructure and variable frontier consumption, with a routing ratio that finance can forecast against, which is, frankly, much preferred compared to “we can scale indefinitely – with the financial consequences”.
Skills shift. Prompt engineering was era one's headline skill. Era two rewards evaluation design, routing logic, orchestration and model governance. Gartner expects at least half of knowledge workers to be developing skills to work with, govern or create agents on demand for complex tasks by 2029.
Five Things I Expect To See by 2029
1. The model estate becomes a standard artefact – as normal to maintain as a data catalogue or service registry, and audited as such.
2. "Where does inference execute" becomes a contractual term in enterprise agreements, the way data residency did after GDPR.
3. Private evaluation suites become a recognised asset class internally – resourced, versioned, owned, and treated as the switching mechanism they are.
4. The routing ratio becomes a reported metric. Some percentage of calls served locally, tracked quarterly, discussed at board level as both a cost and a resilience number.
5. Most organisations end up hybrid – no clean architecture, a frontier dependency they can't remove, and a local tier handling most of the work that keeps the lights on when something upstream breaks.
Realistically, we’ll see the last one come true. But it’s okay because the goal was never independence.
What You Can Do in the Next 90 Days
If you want to move to a more sustainable, advanced AI model that keeps your costs in control, I suggest you do the below.
1. Inventory AI traffic by token volume. Find the 20% of tasks driving 80% of spend. It won’t take long and you may find the results surprising.
2. Build a private evaluation set from your own data for the top three workloads. This is the asset; everything else is downstream of it.
3. Price each workload four ways – frontier flagship, provider small model, private cloud open weights, local hardware. The migration order tends to declare itself.
4. Put a routing layer in front of everything, even if it routes 100% to one provider today. This is the cheapest insurance available, and it takes days.
5. Pilot one workload end-to-end on rented GPUs before buying anything. This will give you real-world insights if what you intend to use it for works.
6. Run the June exercise. Ask which workflows stop if your primary model becomes suddenly unavailable, write the answer down and show it to someone senior.
In a Nutshell
Era one asked: which model should we use?
Era two asks a harder and more useful set of questions. What does our model estate look like? Who governs it? What runs where, and why? What still works when a supplier, a regulator or a government removes an option we'd assumed was permanent?
The companies that will compound through this period will be the ones building the routing, the evaluations and the optionality with foresight rather than a crisis forcing their decision. So, I recommend that you seize the chance while it’s there.
