Long-term, I think we're going to end this year at maybe 50/50 background and real-time workloads, but I see this going to 90/10 in favor of background.
Neil Movva, CEO Sail Research
There have been three eras in the post-ChatGPT world: pre-training, reasoning, and agentic inference.
We’re firmly in the era of agentic inference. OpenAI’s recent agent-driven reacceleration, Grok Bot, Instinct and many recent releases give us a glimpse of a future of fleets of long-horizon background agents across the economy.
Last week I listened to Neil Movva, CEO of Sail Research, articulate how the workload properties of long horizon agents fundamentally differ from chat or training. The differences in properties have bearings on all of compute, memory and power, and if long horizon agents soon became the dominant workload, the ripple effects will reach far and wide.
We’ll walk through:
Compute: Impact on CPU:GPU ratios
Memory: HBM’s role (or lack thereof)
Power: SMRs
Compute: CPU:GPU Reach Parity, Cores Increase
This is one of the most consensus takes at the moment.
Long-horizon agents use GPUs to think and CPUs to execute. The ratio of thinking to execution is expected to eventually result in CPU:GPU parity in datacentres, a steep rise from the 1:8 ratio in the first generations of AI factories.
As AI expands from training to inference and increasingly to agentic and multiagent systems, general purpose server CPU density continue to increase.
Lip-Bu Tan, Intel CEO
We now believe we're almost in parity at this point and could eventually even skew more to CPUs on a unit basis.
David Zinsner, Intel CFO
With the number of cores directly driving agent performance, as Rene Haas described on Arm’s earnings call.
The agentic workloads are essentially capacity constrained in terms of throughput by the number of CPUs you have.
For running agentic workloads, more cores is a better outcome because the software overhead is simpler. You can run virtual machines or virtual jobs on single cores, and they're going to be much more efficient in terms of throughput.
Unlike embedded CPUs inside GPU nodes, standalone CPU servers will run isolated sandbox environments to execute code, crawl the web, and fetch real-time APIs. For example, Microsoft's Fairwater Atlanta campus features a ~300MW GPU building supported by a separate ~48MW CPU and storage building dedicated to running these execution environments.
SemiAnalysis published a great essay on the various CPU efforts by x86 vendors, Accelerator companies with CPUs, Hyperscaler-designed Arm CPUs, Merchant Arm vendors and Chinese CPU suppliers.
Sail Research itself is attacking long horizon agent workloads as a systems problem.
Instead of optimizing inference in isolation, Sail couples inference and execution together. Sailboxes are persistent execution sandbox environments where agents actually run. When an agent blocks on inference, the sandbox automatically pauses and resumes when the response arrives. You’re no longer paying for allocated compute while the agent is simply waiting.
As agent runs extend to hours and days, the onus will be on compute providers to optimise for the nature of this workload, oscillating between thinking and executing, suspending and resuming compute as state accumulates.
Memory: HBM & The Capacity Trade
Fundamentally, inference is all about dataflow.
Deep learning gave hardware an unusually regular workload: large tensor operations, known dependencies, repeated layers, and enough reuse to justify specialized hardware and expensive compilation. Inference pushes this further. The same model may execute billions of times, while latency increasingly depends on where weights and KV cache live, how activations move between stages, and whether communication can overlap with computation.
The closer memory moves to the arithmetic units, the faster each useful access can become. But capacity usually falls and cost per stored bit rises. Move farther away and the system can hold more data at lower cost, but it must wait longer and spend more effort moving that data back to computation.
This is why computers have a memory hierarchy rather than one universal memory. NAND provides persistent capacity. DDR provides a large host working space. HBM supplies high aggregate bandwidth close to an accelerator. SRAM and registers hold much smaller pieces of data close enough for immediate reuse.
GPU tensor cores perform matrix multiplications on weights and KV caches stored in memory - the attention mechanism in transformers relies on a lookup across all KV pairs computed during prefill for decode.
Transformers offer much better accuracy over larger context windows than RNNs but necessarily consumes more memory - decode grows the KV cache and needs to read the model weights with every new token. This is why decode is memory-bound and prefill is compute-bound - with agents, utilisation of GPUs drops significantly due to intensive memory consumption.
Model sizes and approximate layers:
1-3B parameters ~24-32 layers
7-13B parameters ~32-40 layers
70B parameters ~80 layers
100B+ parameters ~100+ layers
Of course, frontier models are trillions of parameters nowadays, with Grok 5 estimated to be more than 6 trillion parameters in size. These weights are easily 1 TB+ in file size. Kimi K3, at 2.8T parameters, is c. 1.4 TB at MXFP4.
Memory bandwidth is the limiting factor for decode, which is why Nvidia has become the biggest customer of HBM, and why HBM is sold out for the next few years across all three major providers (Samsung, SK Hynix, Micron).
HBM uses 3D stacking to increase bandwidth with each generation, which is ultimately a packaging innovation.
Scaling HBM through 3D stacking presents two challenges.
Firstly, HBM achieves high bandwidth largely through brute-force parallelism: thousands of relatively slow electrical connections running through stacked dies, rather than a smaller number of genuinely high-speed links. Those through-silicon vias and micro-bumps consume scarce chip “shoreline,” introduce capacitance that degrades signals, and become increasingly costly in power, heat and packaging complexity as speeds rise
HBM faces challenges related to cost, power, and heat. Its intricate 3D stacking is expensive to manufacture and package. While HBM is efficient per byte of transferred data, the overall power consumption still increases with increased memory capacity and bandwidth. More power means more heat, which can cause thermal issues and impact reliability and performance.
Austin Lyons
Secondly, the fabs producing HBM are not inclined to ramp up production to meet future demand, having endured the cyclicality of the memory market in the past.
This is why memory bandwidth hasn’t kept up with compute.
It's about sidestepping the HBM shortage and focusing on more extreme offload to other forms of memory such as flash. I'm quite passionate about that idea. Everyone on my team knows that I keep banging the drum around what would we have to change about the model architecture to make offloading KV cache to flash work at a much greater level.
The underlying premise for using HBM is to prioritise latency over capacity.
If long horizon agents become the dominant workload, what would the memory hierarchy look like? SRAM or HBM for active decoding, DDR for warmer state, and flash for the much larger body of inactive KV caches, checkpoints and agent history.
Instead of attacking the memory wall through reducing distance or compression, long horizon agents would suit pooling, i.e. leveraging another memory tier.
Whereas chat inference keeps almost the entire live request in accelerator memory, a mature long-horizon system could invert the ratio: perhaps 5% of agent state in HBM and 95% on flash. HBM would still carry nearly all active decode traffic, but it would no longer need to hold the accumulated state of every agent waiting to run.
A long-running agent accumulates tool outputs, documents, code, browser state, messages from subagents and checkpoints needed for recovery. This state continues growing throughout the task. Keeping it in HBM while an agent waits on an API, another agent or human approval would amount to reserving the datacentre’s most expensive memory for an idle process. Flash becomes the parking layer: HBM holds the state required for the current burst of decoding, while the much larger record needed to resume dormant agents sits in cold storage.
Chinese fabs like YMTC and CXMT have started shipping competitive NAND and DRAM products respectively, and will eventually commoditise this market, in turn making storage of state for long horizon agents orders of magnitude cheaper.
Power: Trading Concentration For Aggregate Capacity
The idea is, in both cases, I do not want to be bidding against Anthropic or OpenAI for compute capacity. I’m not going to win against them and I don’t want to. I want to be more creative and use the supply that they don’t find legible today. And over time I amass enough aggregate supply. I’m never going to get concentrated supply. I will only get aggregate supply. And over time I build my aggregate factory that is unbeatable on economics.
We are building a factory. We’re trying to build the best steel factory in the world. But it will come through mini mills, not through large monolithic steel plants.
Just as latency is traded for capacity when it comes to memory, uptime is traded for capacity when it comes to power.
Training requires hyper concentration of compute. Chat inference is the next rung down on the ladder, still favouring compute density to deliver fast decode.
Long horizon agents can leverage a fleet of small compute pools, as Neil describes:
I think that the market still assumes that we have to go shake down those 100-megawatt and ten-megawatt data centers wherever we can find them. It's still the attitude I hear from a lot of data center developers. But increasingly we're seeing a few new thinkers realize that inference is going to be suitable for these distributed one-megawatt data centers. We're quite in agreement with that, and we are very happy to buy small pools of compute across the United States and use that as our inference fleet.
Ten megawatts is probably on the edge of what's possible today. And one megawatt, I would argue, is plentiful. So there's this incredible floor on the market where you can find lots of aggregate power, but it will not be concentrated.
The role of a long horizon agent runtime is to route agents across this distributed compute to make a thousand one-megawatt pools match the performance of a gigawatt campus.
Furthermore, once an agent’s state lives outside the accelerator, the agent is no longer tied to a particular server. An agent can be suspended at one site, its state persisted to storage, and resumed wherever power and accelerators become available. Flash does more than reduce HBM demand: it converts fragmented compute pools into a fungible fleet.
Gigawatt-scale campuses remain extremely difficult to build, and it’s very likely that OpenAI and Anthropic capture the majority of new incremental compute built over the next few years.
Whilst the labs compete for larger pools, smaller brownfield and greenfield pools will be up for grabs.
To that end:
Neoclouds like Crusoe are building modular data centers scalable from hundreds of kilowatts to tens or hundreds of megawatts, whilst others like Megaport build distributed inference capacity in increments of single-digital MW building blocks.
Small Modular Reactors (<300 MW per module) moving from demos to production, e.g. Valar Atomics
Growth in GPU marketplaces like SF Compute, Akash Network, Prime Intellect, Andromeda
Bloom Energy packages its fuel cells in compact energy servers, where 4 to 6 server modules are combined to form 200-300 kW modules, whilst FuelCell Energy launches 12.50 MW utility-grade power blocks that can run on pipeline natural gas today and transition to green hydrogen or biogas later
Conclusion
The common thread is a change in the unit of optimisation.
Training optimised for FLOPs and chat for time-to-token; long-horizon agents will increasingly optimise for cost per completed task. Resources previously suboptimal become economically useful: CPUs execute, flash warehouses state, and small pools of compute and power are stitched together by an agent runtime. HBM and gigawatt campuses will continue to serve the fast inference market, but that may no longer be the dominant workload.