For two years the story of AI hardware was a single word: GPUs. That era is ending. As racks push past 120 kilowatts and clusters stretch to hundreds of thousands of accelerators, the constraints have migrated to everything around the compute die—the memory stacked next to it, the light that moves data between chips, and the power and cooling needed to keep it all alive.
Memory is the new bottleneck
A modern AI accelerator is only as useful as the memory bandwidth feeding it. That is why high-bandwidth memory—DRAM stacked vertically and bonded to the processor—has become the most supply-constrained part of the chain. HBM demand grew on the order of 130% in a single year, and the market is expected to keep compounding at rates rarely seen in semiconductors.
Supply is concentrated: one Korean manufacturer holds well over half the market, with two others splitting the rest. The next generation, HBM4, is already sampling at bandwidths above 2.5 terabytes per second per stack. For memory makers this is a structural shift from a commoditized, cyclical business to a scarce one priced for scarcity.
Moving data with light
As models are split across thousands of chips, more of the energy and delay is spent not computing but moving data between chips. Traditional pluggable transceivers sit at the edge of the board; co-packaged optics (CPO) move the optical engine right next to the switch or accelerator silicon, cutting the power per bit and improving reliability.
The largest silicon and foundry players are racing to productize CPO because networking—not raw compute—increasingly limits how large a cluster can practically grow. It is a quiet but pivotal shift: photonics moving from the cable into the package.
800 volts and the return of DC
Power delivery has become a first-order design problem. A current flagship rack already draws roughly 120–130 kilowatts; next-generation designs target 600 kilowatts and up to a megawatt per rack. At those levels the incumbent low-voltage AC distribution wastes too much copper and too much energy.
The answer is 800-volt DC distributed from the facility to the rack—an architecture that carries far more power through the same copper and cuts conductor mass by nearly half. It pulls in a new set of suppliers: high-voltage power electronics, gallium-nitride and silicon-carbide devices, busbars, and on-rack energy storage to smooth enormous, spiky loads.
Heat is no longer optional
Air cannot remove 120 kilowatts from a rack. Direct-to-chip liquid cooling has moved from exotic to mandatory, with cold plates, liquid-cooled busbars, and warm-water loops now designed in from the start; immersion cooling waits in the wings. Thermal management has become a constraint that shapes the entire building.
The takeaway is that "AI infrastructure" is no longer a chip story. It is a systems story spanning memory, optics, power, and thermals—each with its own specialized suppliers, and each now on the critical path.
Sources
Disclaimer. This report is produced by Dendra Capital for informational purposes only. It reflects views as of the date of publication, draws on sources believed to be reliable but not guaranteed, and is subject to change without notice. It does not constitute investment research, or financial, legal, or tax advice, nor an offer or solicitation to buy or sell any security. Past performance is not indicative of future results.