The Inference First Architecture Shift

By Nekud October 16, 2026 18 min read

The most consequential single architectural insight of the late-2020s AI industry is one that is now obvious in retrospect but that took several years of practical experience with deployed large language models to articulate cleanly. The inference workload and the training workload, although they share an underlying model and many of the same fundamental mathematical operations, are not the same workload. They have different bottlenecks. They have different runtime distributions. They have different optimization targets. And they have, as a consequence, different silicon ideals. The realization has produced one of the largest coordinated architectural shifts in the modern semiconductor industry: the migration of an increasing fraction of AI compute investment from general-purpose training-optimized silicon to specialized inference-optimized silicon, and the bifurcation of the AI chip market into a two-track industry in which the silicon for training and the silicon for inference are different chips designed for different purposes.

The IEEE Spectrum framing of the shift, in the magazine’s October 2026 cover story, was unusually direct: “The AI Inference Revolution Is Here.” The IEEE piece argued that 2026 is the year in which inference overtook training in the strategic priorities of the largest AI compute buyers, and that the architectural response to that change — the proliferation of inference-optimized silicon — was the dominant industry trend in AI chips for the year. The pieces of evidence supporting the framing were, by the time of the article’s publication, already substantial: Nvidia’s acquisition of Groq at the end of 2025 and the unveiling of the Groq 3 LPU at GTC 2026; Microsoft’s third-generation Maia 300 chip optimized for inference; Google’s split of the TPU family into TPU 8t for training and TPU 8i for inference; Intel’s rename of the Crescent Island inference accelerator to GPU D-Series; the emergence of several independent inference-specialized silicon companies including Tensordyne, d-Matrix, Cerebras (in its inference mode), and Majestic Labs. The architectural shift is, in substance, a confirmation that the inference workload has become large enough to justify its own silicon program.

This essay traces the inference-first architecture shift through the engineering reasons why inference and training workloads demand different silicon, the design patterns that have emerged for inference-optimized accelerators, the major commercial programs that are executing on the shift, and the strategic implications of the shift for the AI infrastructure industry. It focuses on three threads. The first is the engineering thread: why does the inference workload benefit from a different architectural point than the training workload, and what does the inference-optimized silicon actually look like. The second is the commercial thread: what are the principal inference-optimized silicon programs, and what are the architectural choices that distinguish them. The third is the strategic thread: what does the architectural shift imply for Nvidia’s merchant-GPU position, for the custom-silicon programs of the hyperscalers, and for the longer-term structure of the AI infrastructure industry. The thesis the essay develops is the following. The inference-first architectural shift is not a fashionable pivot; it is a structural response to a workload that has grown large enough to justify its own silicon program, and it is going to be a permanent feature of the AI infrastructure industry.

Why inference and training want different silicon

The architectural difference between inference and training starts with the workload pattern. Training a large transformer is, computationally, a single very large graph that runs for weeks and that consumes the bulk of a large data-centre cluster for the duration of the run. The training workload is a sequential workload: the optimizer steps, the checkpoint writes, the gradient reductions, and the all-reduce operations are all tied to the cadence of a single optimizer step, and the silicon for that workload has to be optimized for the dense, sequential nature of the run. Inference, in contrast, is a fundamentally parallel workload. Each user request is independent of each other user request, and the data-centre has to serve thousands or millions of independent requests per second. The inference workload is, accordingly, optimized for throughput-per-watt across many independent requests, not for throughput-per-watt across a single large graph.

The second architectural difference is between the two phases of an individual inference request. The prefill phase processes the user’s prompt into a context cache that contains the keys and values of the prompt tokens. The prefill phase is computationally dense: it is a single forward pass through the model on a relatively short input, and it is dominated by floating-point matrix multiplication. The prefill phase wants the same kind of dense, floating-point-optimized silicon that training wants. The decode phase generates one token at a time, attends to all of the accumulated context keys and values, and proceeds sequentially through the response until the model decides to stop. The decode phase is memory-bound: it does a small amount of floating-point work per token but reads a large amount of model weights and a large amount of context per token. The memory bandwidth limit on the decode phase is, on most production workloads, the dominant constraint on the inference performance per dollar.

The third architectural difference is between the latency requirements of inference and the throughput requirements of training. Training wants high floating-point throughput per accelerator for weeks at a time, and the silicon has to be optimized for sustained floating-point performance. Inference wants low latency per user request, and the silicon has to be optimized for the latency of the slowest phase of the request (which is, in practice, the decode phase). Low latency and high throughput are competing optimization targets: a chip optimized for low latency typically has a small amount of fast on-chip memory and a modest amount of floating-point throughput, while a chip optimized for high throughput has a large amount of on-package memory and very high floating-point throughput. The two designs point in opposite directions.

The architectural response to these three differences is what the industry calls a memory-centric inference accelerator: a chip that has a substantial amount of on-chip SRAM, a large amount of on-package or on-board memory, modest floating-point throughput tuned to the rate at which memory can supply data, and a low-latency interconnect between the floating-point units and the memory. The architectural goal of the memory-centric design is to push the memory bandwidth close to the theoretical maximum available from the chosen memory technology, while keeping enough floating-point throughput to do useful work per byte of memory. The architecture is, on the published engineering record, substantially different from the dense-floating-point GPU that is optimal for training. The two architectures point at different parts of the same silicon-design space.

How memory-centric silicon works

The memory-centric inference accelerator is, in design, a relatively old idea. The early systolic-array designs from the late 2010s were attempts to push close to the memory-bandwidth ceiling, and several of the early AI accelerator startups of the late 2010s and early 2020s had memory-centric architectures at their core. What is new in the late 2020s is the maturation of memory technology — HBM3e, HBM4, advanced on-package SRAM, advanced packaging — to the point at which a memory-centric design can be substantially more cost-effective than a general-purpose GPU at the same inference workload. The maturation of memory has been the technical condition that has made the late-2020s shift possible.

The Nvidia Groq 3 LPU is the cleanest example of the memory-centric design point. The chip has, on the published record, 500 megabytes of on-die SRAM connected directly to its floating-point units. The amount of on-die SRAM is, in the absolute terms of any single chip, a substantial fraction of the working memory that a production-sized language model needs at inference time, and it is enough to keep a substantial fraction of the model weights in a low-latency memory. The floating-point throughput of the LPU is much smaller than that of a contemporary Nvidia GPU — the chip has, on Nvidia’s published disclosure, “far less raw computing power than a standard GPU” — but the floating-point throughput is sized to the rate at which the on-die SRAM can supply data, so the throughput is rarely the bottleneck. The result, on Nvidia’s published workload performance, is that the LPU substantially outperforms the GPU on decode-heavy inference workloads.

The Microsoft Maia 300 chip is a different memory-centric design point. The chip does not have the LPU’s 500 megabytes of on-die SRAM. Instead, the chip carries 288 gigabytes of HBM4 memory per accelerator and runs at 18 terabytes per second of memory bandwidth. The choice is, in architecture, a tradeoff between the latency-advantage of on-die SRAM and the capacity advantage of on-package HBM. The on-package HBM is too slow for the absolute lowest-latency decode workloads, but the much larger capacity allows each accelerator to hold a substantially larger fraction of a frontier model’s weights. The hybrid of HBM and on-die SRAM, in the chip’s published design description, is engineered by Microsoft to give a balanced latency-and-throughput profile across the full range of inference workloads that Copilot, GitHub Copilot, and Microsoft Foundry will need to serve.

The Google TPU 8i chip is the third major memory-centric design point. The chip’s design philosophy is the same as the Maia 300 philosophy — large on-package HBM, modest floating-point throughput, an on-chip fabric optimized for memory-bandwidth-heavy decode — but with Google’s distinct design language in the on-chip organization and in the per-chip interconnect. Broadcom, which is Google’s principal manufacturing and design partner for the TPU family, indicated in the third quarter of 2026 that production shipments of TPU 8i had begun, with mass production targeted for the fourth quarter. The Google TPU 8i is, on the basis of the published order book, already on track to be one of the largest-deployed inference-optimized silicon parts in the world, with Anthropic’s planned ramp from 1 gigawatt in 2026 to 5 gigawatts in 2027 being the largest single customer commitment.

The independent inference-optimized silicon programs are also worth noting. Tensordyne’s Napier chip is, on the published product description, a memory-centric inference accelerator in the same design family as the LPU. d-Matrix’s Raptor chip is a memory-centric accelerator designed for low-latency LLM serving and intended to integrate into Nvidia’s data-centre systems through NVLink Fusion. Majestic Labs’ architecture is a memory-aggregation design in which the working memory is distributed across a set of memory dice connected by a high-bandwidth fabric to a small number of compute dice. Cerebras, which is best known for its wafer-scale training engine, has also developed a wafer-scale inference mode for serving very large language models with very low latency; the Cerebras inference mode has been adopted by Amazon Web Services for the memory-intensive portion of the AWS-Anthropic inference stack. The architectural diversity across these independent programs is substantial, and it is a marker that the inference-specialized silicon market is, in late 2026, still in the early phase of design exploration.

The prefill-and-decode split

The cleanest architectural pattern for combining memory-centric silicon with training-optimized silicon is the prefill-and-decode split. In this pattern, a single inference request is split into two pieces that are handled on two different kinds of silicon. The prefill phase, which is computationally dense and runs once per request, is handled on a training-optimized GPU or on a generic floating-point accelerator. The decode phase, which is memory-bound and runs many times per request, is handled on a memory-centric inference accelerator. The prefill phase prepares the context cache on the GPU; the decode phase streams the response tokens from the memory-centric accelerator; the two phases are stitched together by a router that directs each user request to the appropriate accelerator for each phase.

The prefill-and-decode split is, in the late 2020s, the most architecturally distinctive design pattern in the inference-chip market. It is the design pattern that Nvidia has chosen for the Vera Rubin platform, in which the Rubin GPU handles the prefill phase and the Groq 3 LPU handles the decode phase. The pattern allows the customer to buy a heterogeneous system in which each phase of the inference request is handled by the silicon that is best for that phase, and the overall system has a per-watt performance that no single-chip design can match. The pattern has, on Nvidia’s published workload performance, produced a 30x improvement in AI factory throughput on agentic workloads relative to the previous-generation Grace Blackwell design, with up to 60x improvement at some points on the Pareto frontier of the workload profile.

The prefill-and-decode split is also the design pattern that AWS has chosen for the Anthropic stack. In the AWS-Anthropic inference stack, the Trainium training-optimized accelerators handle the prefill phase for Anthropic’s Claude models, and the Cerebras wafer-scale inference engine handles the decode phase for the same models. The split is, in AWS’s published design description, motivated by the same architectural logic that motivates the Vera Rubin split: each phase of the request is handled by the silicon that is best for that phase, and the silicon that is best for one phase is different from the silicon that is best for the other phase. AWS does not have a Groq-equivalent in its proprietary stack and has chosen instead to partner with Cerebras to provide the memory-centric inference silicon that the Anthropic workload demands.

The prefill-and-decode split is, however, not the only design pattern for combining memory-centric and floating-point-centric silicon. Microsoft has chosen, with Maia 300, to combine the prefill and decode phases on a single chip, with a design in which the on-chip fabric is engineered to allow the same chip to handle both phases efficiently. The architectural benefit of Microsoft’s monolithic approach is that the customer does not have to orchestrate a handoff between two different kinds of silicon for each request, and the per-request latency is therefore lower than it would be in a heterogeneous system. The architectural cost is that the chip is not as specialized as a memory-centric chip would be on the decode phase, and not as specialized as a floating-point-centric chip would be on the prefill phase. Microsoft has, on the basis of its published design rationale, decided that the latency benefit of the monolithic approach is worth the loss of specialization.

The TPU 8i chip is also a single-chip design in the same style as Microsoft’s Maia 300. Google has, in the TPU v8 generation, chosen to maintain a single-chip architecture for inference rather than to bifurcate the silicon for prefill and decode. The architectural reasoning is, on Google’s published design description, that the per-chip performance of a well-engineered single-chip inference design is competitive with the per-chip performance of a heterogeneous system, and that the operational simplicity of a single-chip design is worth some loss of specialized performance. The reasoning is plausible, and the choice reflects Google’s distinctive design philosophy, which has historically preferred single-chip architectural simplicity over heterogeneous-system architectural specialization.

The strategic implications for Nvidia and the broader market

The architectural shift toward memory-centric inference silicon is, by late 2026, the largest single strategic challenge that Nvidia has faced in the merchant GPU market since the company entered the AI compute business. The challenge is not that Nvidia is losing its position in training: the Vera Rubin platform is, on every published benchmark, a substantial improvement over the previous-generation Grace Blackwell design for training workloads, and Nvidia’s training-floating-point throughput is still the industry standard. The challenge is that the inference workload has become large enough to justify its own architectural point, and that the architectural point is sufficiently different from the training-optimized GPU that Nvidia’s existing silicon is not the best for the inference workload even when it is competitive on the training workload. The challenge forces Nvidia to do what it had been trying not to have to do: design a second architectural point, in the form of the Groq 3 LPU, that is intentionally different from the GPU.

Nvidia’s response to the strategic challenge is the Groq 3 acquisition and the integration of the LPU into the Vera Rubin platform. The acquisition was the largest AI infrastructure M&A; deal of 2025, and it was, on the strategic logic of the deal, an explicit recognition that the inference-first architectural shift was large enough to justify buying an existing memory-centric silicon program rather than building one from scratch. The integration of the LPU into the Vera Rubin platform is, in turn, an admission that the heterogeneous-system pattern (the prefill-and-decode split) is the right pattern for some inference workloads, and that Nvidia needs to be present at both architectural points. The strategic prize for Nvidia is that the Vera Rubin platform can serve the inference workload in a way that the merchant GPU alone could not, and that the architectural shift has not eroded Nvidia’s market position in absolute terms even though it has changed Nvidia’s market position in relative terms. The strategic risk is that the LPU is, on its own, less competitive than the memory-centric accelerators of the hyperscaler custom-silicon programs, and that the integration of the LPU into the Vera Rubin platform is, accordingly, a defensive move rather than an offensive one.

The strategic implications for the hyperscaler custom-silicon programs are also substantial. The Microsoft, Google, and Amazon programs are all, by late 2026, investing substantially in inference-optimized silicon, and the architectural shift has validated the strategic logic of those investments. The shift has, however, also raised the bar for what counts as a competitive inference design. The first-generation inference-optimized chips of the early 2020s were good enough to demonstrate the architectural idea, but they were not good enough to be the principal silicon of a hyperscale inference fleet. The second-generation inference-optimized chips of the mid-2020s — the TPU v8, the Maia 200, the Trainium — were good enough to handle a fraction of the inference fleet but not good enough to displace the GPU entirely. The third-generation inference-optimized chips of the late 2020s — the TPU 8i, the Maia 300, the Groq 3 — are, on the published performance record, good enough to displace the GPU on a substantial fraction of the inference fleet. The bar is, in other words, rising with each generation, and the architectural shift is producing a continual escalation in the sophistication of the silicon designs.

The strategic implications for the broader market are that the AI infrastructure industry of the late 2020s is going to be substantially more architecturally diverse than the AI infrastructure industry of the early 2020s. The early 2020s were characterized by a near-monoculture of Nvidia GPUs at the silicon level, with the diversification happening at the system level (different rack designs, different cooling systems, different networking topologies). The late 2020s are characterized by a polyculture of architecturally distinct silicon programs, with the diversification happening at the silicon level. The polyculture is good for the AI infrastructure industry: the architecturally diverse silicon produces more efficient inference economics than the GPU monoculture, and the more efficient inference economics are what allow the AI service market to continue to scale at the rate that user demand requires. The polyculture is challenging for Nvidia and for the GPU-centric parts of the merchant chip market: it requires that Nvidia (and the GPU-centric chip companies) compete on a substantially different playing field than the one they had dominated in the early 2020s.

The longer-term trajectory of inference silicon

The inference-first architectural shift is unlikely to be the last major architectural shift in the AI silicon market. The architectural target of the late 2020s — memory-centric inference silicon — is itself a moving target, and the moving target is the difference between the workload of an LLM-based AI service in 2026 and the workload of an LLM-based AI service in 2028 or 2030. The workload of an LLM-based AI service is being shaped, in real time, by the rise of agentic AI: AI systems that take multi-step actions, use tools, retrieve external information, and act over minutes or hours rather than over the hundreds of milliseconds of a single chat response. The agentic workload has its own architectural implications, and the inference silicon market is going to evolve to address them.

The agentic workload has, on the AI Infra Summit 2026 panel descriptions, three architectural implications for the inference silicon of the late 2020s and early 2030s. The first is the long-context workload: agentic systems accumulate long context as they work through multi-step tasks, and the accumulated context can run to hundreds of thousands of tokens. Long context means more memory pressure on the decode phase, and the inference silicon of the late 2020s is going to need substantially more on-package memory than the inference silicon of the early 2020s. The second is the tool-and-retrieval workload: agentic systems interleave the model generation with tool calls and with retrieval-augmented-generation steps, and the tool and retrieval steps happen on different silicon than the model generation. The inference silicon is, accordingly, going to need to interoperate cleanly with the tool-execution silicon and with the retrieval silicon, and the system-level design problem becomes a substantial fraction of the inference-stack design problem. The third is the multi-user workload: agentic systems are typically deployed in such a way that one user request triggers many parallel agent calls, and the inference silicon has to handle the parallel agent calls efficiently. The parallel workload is a different shape than the single-chat workload, and the inference silicon of the late 2020s is going to need to be optimized for the parallel workload.

The response of the major inference silicon programs to the agentic workload is, in late 2026, beginning to take shape. Microsoft’s Maia 300, with its 12,288-accelerator superpod and its 96-accelerator racks, is positioned to handle the multi-user workload at scale. Google’s TPU 8i, in its 9,216-chip superpod configuration, is similarly positioned. Nvidia’s Vera Rubin platform, with the Groq 3 LPU handling the decode phase of each agent call, is positioned to handle the agentic workload through its heterogeneity. The architectural response across the major programs is, accordingly, broadly similar in direction, even though the specific design choices differ from program to program.

The inference-first architectural shift is, in summary, the dominant silicon-side trend in the AI infrastructure industry of the late 2020s. The shift is producing a polyculture of architecturally distinct inference silicon programs, each of which is designed around a different point in the architectural trade-off space between floating-point throughput, memory bandwidth, on-chip SRAM, and memory capacity. The polyculture is producing more efficient inference economics than the GPU monoculture of the early 2020s, and it is, accordingly, enabling the scaling of the AI service market at the rate that user demand requires. The polyculture is also challenging the merchant GPU market to evolve in ways that the GPU monoculture had not demanded, and it is, accordingly, producing a structural shift in the competitive landscape of the AI chip industry. The shift is large enough, and durable enough, that it is going to be a permanent feature of the AI infrastructure industry for the rest of the decade, and probably for the decade beyond that.