Chips 2026

Every answer you get from an AI system was computed by a physical object. Somewhere in a building the size of a warehouse, a piece of silicon roughly the size of a postage stamp did several quadrillion arithmetic operations, pulled the results out of memory, sent them across a wire to a few dozen identical pieces of silicon, and handed you a sentence. It's incredible.

Most writing about AI stops at the software, because truth be told there are very few who fully understand both the software and the hardware. Myself included. But, I've managed to get this far by asking the right questions, sitting down with the people on the frontlines of this technology, and being forthcoming about what I don't know. I find that this humility has gotten me far. This primer is about the hardware.

Once a year, the people who design those objects present their work at a conference called Hot Chips. The presenters are the engineers themselves rather than the marketing departments, and they show real numbers. Imagine if the people who designed the engine turned up at the car show instead of the salespeople. In August 2026, that conference included the first public description of a chip built by OpenAI, two new processors from Google, an Arm instruction set running inside an IBM mainframe, and a self-driving car chip from Waymo that is already carrying passengers.

This primer walks through what was presented there, along with the history behind it and the industry around it, organized by the kind of chip rather than the order of the talks. You do not need an engineering background. Every term gets defined the first time it appears.

By the end you should be able to follow a conversation about AI hardware, understand why the industry is spending hundreds of billions of dollars on it, and know which claims to be skeptical of.

If you retain five things from this primer, make them these.

1. Memory is the story, not compute

Memory has grown from roughly half of what an AI chip costs to build to nearly two-thirds. Processors have got much faster than memory for twenty years running. Almost every design decision in this book is an attempt to move less data, move it a shorter distance, or move the computation to where the data already is. When you hear a claim about a new chip, ask how much data it has to move to do the work, not how fast it calculates.

2. Power is the real ceiling

Chips can be manufactured faster than electrical substations can be built. That is why NVIDIA now measures itself in words of output per megawatt, why a 45-degree coolant temperature is a headline feature, and why flattening the shape of a rack's power demand can yield 40% more capacity without any chip getting faster.

3. Nobody sells a chip

Every serious vendor now sells a rack: chips, cooling, power distribution, wiring and management software as one object. That is not marketing. A model too large for one chip forces the chips to depend on each other, and once they do, the system is the product.

4. Building your own chip became normal

OpenAI, Google, Meta, Microsoft, Amazon and Waymo all design their own silicon. OpenAI did it in nine months. If your workload is enormous, stable and known, buying a general-purpose part means paying for circuitry you never use.

5. The open-versus-closed fight is the one to watch

NVIDIA's advantage rests on controlling every layer, including the proprietary wiring between chips. AMD, Microsoft, Broadcom and the RISC-V community are all betting that customers eventually refuse single-supplier lock-in on the most critical part of the system. That disagreement, not any performance number, is the thing that will decide the shape of this industry.

A note on skepticism

Every performance figure in this primer is the vendor's own. They are measured on benchmarks the vendor selected, against comparisons the vendor chose. That does not make them false; these are engineers presenting to other engineers, and they get challenged. But when you see a number like "30 times better," always ask three questions: better than what, measured how, and at what operating point. Most of the enormous multiples in this book are true only at one specific corner of the performance curve, and usually it is the corner the competitor is worst at.

In December 1947, three physicists at Bell Labs pressed two gold contacts into a sliver of germanium and got it to amplify an electric current. That was the first transistor.

Before it, computers switched electricity using vacuum tubes. A tube was about the size of a small lightbulb, ran hot, and burned out. A machine with seventeen thousand of them spent a good part of its life being repaired. The transistor did the same job in a fraction of the space, cold, and without dying.

That created a new problem. A useful circuit needs hundreds of transistors, and every one of them had to be wired by hand. The wiring became the limit.

In 1958 Jack Kilby at Texas Instruments built several components onto a single piece of material. A few months later Robert Noyce at Fairchild worked out how to print the connections onto it too. That is the integrated circuit, and it is the reason a chip is one object instead of a bag of parts. Noyce and a colleague named Gordon Moore left Fairchild in 1968 and started Intel.

Moore's Law is not a law

In 1965 Moore wrote a short piece for a trade magazine. He had noticed the number of components on a chip was roughly doubling every year, and he guessed it would keep going for another decade. He revised it to every two years in 1975.

It held for about fifty years, which is remarkable for a guess made over a weekend. But it was never a law of physics. It was an observation that turned into a business plan. Once the whole industry believed the doubling would happen, every company scheduled its products around it and funded the research to deliver it, and the belief made itself true.

The split that created the modern industry

Into the 1980s, if you designed chips you owned a factory. Intel designed and built. So did Motorola, Texas Instruments and everyone else. Designing without manufacturing was not really an option, because nobody would build your design for you.

In 1987 a man named Morris Chang started a company in Taiwan on an odd premise. TSMC would build chips for other companies and design none of its own. No competing with your own customers. Just manufacturing.

That broke the industry in half. You could now design a chip without spending billions on a factory. Companies that design without manufacturing are called fabless, and almost every chip company you can name is one. NVIDIA, AMD, Apple, Qualcomm and Broadcom all are. Intel is the famous holdout, and it has not gone well for them.

How Intel lost the lead

Intel led chip manufacturing for roughly thirty years. Two things ended that.

It missed mobile. Intel was offered the chip in the first iPhone and turned it down, because the price Apple wanted did not clear the margin Intel expected. Its chief executive later said the volume forecasts turned out to be off by a factor he had not imagined. That decision handed the biggest chip market of the next fifteen years to Arm.

Then it stalled. Intel's move to its 10-nanometer generation ran years behind schedule while TSMC kept delivering on time. Somewhere around 2018, the most advanced chips in the world stopped coming from Intel and started coming from Taiwan, and they have not gone back.

How NVIDIA won

NVIDIA sold graphics cards to people who play video games. In 2006 it released CUDA, software that let anyone use a graphics chip for general mathematics instead of only for drawing pictures.

Researchers picked it up. In 2012 a neural network called AlexNet, trained on two NVIDIA gaming cards bought off the shelf, won an image recognition contest by a margin nobody had seen before. The field moved to GPUs almost immediately.

CUDA is the reason NVIDIA is hard to displace, and it matters more than any chip in this book. Nearly all AI software is written for it. A competitor has to beat the chip and replace twenty years of software that every engineer in the field already knows how to use. The second problem is much harder than the first.

Start with sand. Silicon is refined out of it, melted, grown into a cylinder of near-perfect crystal, and sliced into mirror-flat discs about the width of a dinner plate. Each disc is called a wafer.

Onto that wafer, a machine prints a pattern using light. It is the same basic idea as photographic film, but at a scale where the features are a few dozen atoms across. The pattern is built up in layers, and it forms hundreds of millions or billions of tiny electrical switches called transistors. A transistor does one thing. It lets current through, or it does not. Everything a computer does is built out of enormous numbers of those switches turning on and off.

When the printing is done, the wafer is cut into rectangles. Each rectangle is a die, one chip. The die is then mounted into a protective package with metal contacts on the bottom, and that package is what gets joined together into a computer.

Why "smaller" mattered so much

For about sixty years, the industry got better mainly by making transistors smaller. Smaller transistors mean more of them fit in the same space, they switch faster, and each one uses less power. That steady shrinking is what people mean by Moore's Law, and it is why a phone today outruns a supercomputer from the 1990s.

The generations of shrinking are named by a number: "5 nanometer," "3 nanometer," "2 nanometer." Those numbers stopped describing anything you could actually measure about fifteen years ago. Nothing on a 3-nanometer chip is three nanometers across. They are generation names now, the way a car model year is not a measurement. The only thing you have to remember is that a smaller number means a newer, denser, more expensive process.

What broke

Two things went wrong at once. First, the shrinking slowed down. Transistors are now small enough that they are running into the size of atoms, and at that scale electricity starts leaking through switches that are supposed to be closed. Each new generation delivers less improvement than the last. Second, the cost went up sharply, so the newest process is dramatically more expensive per unit of area than the one before it. Past a certain size the features are far smaller than the wavelength of light used to print them, so the industry moved to machines that use extreme ultraviolet light, and those are among the most complicated machines ever built. They are referred to as EUV machines. Layers those machines cannot resolve in one pass get printed several times over with slightly offset patterns, each pass carrying its own coating and etching. The denser the features, the more likely a single speck of contamination ruins a chip, so more of every wafer gets thrown away. More expensive machines, more steps, more waste, all at once.

There is also a hard physical ceiling. The machine that prints the pattern can only expose a rectangle of a certain size in one shot, called the reticle limit. You cannot simply make one enormous chip; past a certain size, you physically cannot print it.

The industry's answer is the subject of most of this primer. Rather than build one big chip on the newest, most expensive process, build several smaller ones, put each on whatever process suits it, and join them together inside a single package. Those smaller pieces are called chiplets. Fujitsu's presentation at Hot Chips had the clearest illustration. Their processor is three separate pieces of silicon as opposed to just one.

The calculating cores sit on the expensive 2-nanometer process, where the extra speed and the lower power are worth paying for. The memory cache sits on a second piece directly underneath them, built on an older 5-nanometer process. The input and output circuitry sits on a third piece alongside, also 5-nanometer.

Almost nothing is lost by doing it this way, because those parts would barely have improved on the newer process anyway. A single memory cell needs six transistors arranged so they reliably hold their value, and packing them tighter makes them less stable, which is why the industry has left that cell at roughly the same size for three generations running. Analog circuits have a related problem. They work in continuously varying voltages rather than ones and zeros, and smaller devices make those voltages noisier and harder to read.

The result is that less than a third of the silicon in the chip is built on the process that costs the most.

That single decision, repeated across the industry, is why chips have stopped being one thing and started being assemblies.

NVIDIA does not have a factory. Neither does AMD, Apple, Qualcomm or Broadcom. They design chips and send the design somewhere else to be built, and that somewhere else is almost always TSMC.

Three companies in the world can build at the leading edge. TSMC in Taiwan, Samsung in South Korea, and Intel in the United States. That is the entire list, and TSMC does the large majority of it, something close to nine out of every ten of the most advanced chips.

The reason the list is three names long instead of thirty is cost. A factory that can build at that level runs upward of twenty billion dollars and takes years to bring into production. Very few organizations on earth can write that check, and the ones that can mostly decided not to.

The machine all of them depend on

Every one of those factories buys its most important tool from the same supplier.

ASML, in the Netherlands, is the only company that makes the extreme ultraviolet machines from Chapter 04. Not the largest maker. The only one. A single machine costs a few hundred million dollars and ships in pieces across several cargo planes.

There is no second source. If you want to build the most advanced chips, you buy from ASML or you do not build them. It is the tightest chokepoint in any major industry I can think of.

Why Taiwan is in the news

Put those two facts next to each other. Almost all of the world's most advanced chips are manufactured on one island, and that island is claimed by China.

A serious disruption there would not be a shortage in the ordinary sense. There is no other factory to switch to, and building one takes the better part of a decade. That is why chips stopped being a trade question and became a defense question.

What the export controls actually do

Since 2022 the United States has restricted what can be sold to China, and it is worth knowing what is actually on the list. Two things. The most advanced AI chips themselves, and the equipment needed to make advanced chips at all. ASML cannot ship its EUV machines to China.

The logic is that you cannot catch up if you cannot buy the tools. The counter-argument is that it gives China every reason to build its own tools, and there is evidence pointing both ways. Nobody serious claims to know how this ends.

The CHIPS and Science Act, passed the same year, put roughly fifty-two billion dollars toward building factories inside the United States. TSMC, Samsung and Intel are all constructing American plants with it. Those plants will help. They will not move the leading edge out of Taiwan this decade.

The two chokepoints nobody mentions

Design software. Nobody designs a modern chip by hand. Three companies make the tools that do it: Cadence, Synopsys and Siemens. Every chip in this book was designed using software from one of them.

Arm. It makes no chips and no software. It licenses the instruction set from Chapter 06 and the core designs that go with it, and it sits inside almost every phone on earth.

So an industry that looks like hundreds of companies is actually a short chain with a single supplier at several links. That is the real shape of it.

A processor does not understand a programming language. It understands a fixed list of very simple commands, such as: add these two numbers, fetch this value from memory, jump to that point in the program if this condition is true. That list is called an instruction set, and it is the contract between software and hardware. Write a program, and a compiler translates it into instructions from that list.

Here is why this matters commercially. If your software is compiled for one instruction set, it will not run on a processor that speaks a different one. Whoever controls the instruction set therefore controls which chips your software can run on. That is one of the most valuable positions in technology.

The three that exist

x86 is the one inside almost every laptop and traditional server. It is controlled by Intel and AMD. If you want to build an x86 processor, you essentially cannot; there is no path to a license.

Arm is the one inside every phone, and increasingly inside data centers. Arm the company does not usually make chips. It licenses the design to Apple, Qualcomm, Amazon, NVIDIA and hundreds of others, who build their own. You pay Arm, but you get to build.

RISC-V (pronounced "risk five") is the third, and this one is a bit different. It is an open standard, governed by a non-profit that is deliberately based in Switzerland rather than the United States, so that no single government can restrict who is allowed to use it. Nobody owns it. Anyone can build a processor that uses it, without permission or payment. It has been used in small embedded controllers for years, the things inside a hard drive or a sensor.

The problem with free

An open standard has an obvious failure mode. If anyone can add features, every chip ends up slightly different, and software has to be rewritten for each one. That would make the whole thing useless for ordinary computers.

RISC-V's answer is a mechanism called a profile. A profile is a published list saying any chip claiming this label must include exactly these features. Software written for the label runs on any chip carrying it. The profile for serious 64-bit computers is called RVA23, and it was finalized in October 2024.

Hot Chips 2026 was, in effect, RVA23's coming-out party. Krste Asanović, the standard's chief architect and one of its original creators at Berkeley, gave the status update: RISC-V is sixteen years old, and 2026 is the first year that server-grade chips built to the standard actually exist and are shipping.

The part that is not finished

Canonical, the company behind Ubuntu, put a number on how far along the software really is. About 95% of Ubuntu's packages have been made available for RISC-V. That sounds like the job is nearly done.

It isn't. "Available" only means the code was translated for the new chip and the translation ran without errors. Nobody checked the result. It is the difference between having a recipe translated into French and having someone actually cook it.

The checking is done by test suites. Most serious software ships with one. It is a batch of automatic checks the authors wrote to catch their own mistakes. Run the suite and you find out whether the program really works on that chip, or merely compiled on it.

On Intel and AMD chips, 24,815 Ubuntu packages have test suites, and 92.3% of them pass. On RISC-V, only 3,443 packages have been tested at all, and barely half of those pass.

That gap is not really about RISC-V being broken. Almost all RISC-V software is still tested on simulated chips instead of real ones, because until this year there were hardly any real ones to test on. Simulation is slow. A job that takes an hour on real hardware takes a day, and anything running too long gets logged as a failure. Canonical wants real RISC-V servers in its build farm, and that is exactly what 2026 finally makes possible.

The most commercially significant signal came from NVIDIA, which announced it is porting the software stack behind its GPUs to RISC-V, and that customers can now build their own RISC-V processor and drop it into an NVIDIA server rack.

The CPU, or central processing unit, is the general-purpose processor. It is the part of a computer that reads a program one step at a time, does whatever each step says, and decides what to do next based on the result. It is very fast at this and very flexible, but it fundamentally works in sequence.

A modern server CPU contains many independent copies of this machinery, each called a core. More cores means more separate tasks can run at the same time. For roughly the last fifteen years, the competition between Intel, AMD and Arm has largely been about core count: whoever could pack more cores into a socket won cloud business, because a cloud provider's job is to run thousands of unrelated customer workloads simultaneously.

Why that changed

An AI agent runs one task with a lot of steps in a row, not a thousand unrelated ones. The agent decides what to do, calls a tool, waits for the answer, reads it, decides what to do next, calls another tool. Each step depends on the one before it, so you cannot run them at the same time. Every one of those steps runs on the CPU.

So what decides how long a user waits is how fast a single core finishes one step, not how many cores you own. NVIDIA put this bluntly: the goal is agent performance per core, not core count.

What each company built

NVIDIA's Vera is a deliberate rejection of the core-count race. 88 custom cores rather than the 96 or 128 its competitors offer, but designed so each one is fast and, crucially, predictable. Most processors let two tasks share one core's machinery, which improves total throughput but means each task's speed depends on what the other task happens to be doing. NVIDIA split the resources so each task gets its own, which gives up a little peak speed for a lot of consistency. They also used a type of memory normally found in phones, which delivers roughly five times more bandwidth per watt than the server standard.

Intel's Diamond Rapids went the other way and scaled up: 256 cores, 1.28 gigabytes of on-chip fast memory, and a new internal layout that pushes the compute out to the edges and centralizes memory and connections in the middle.

Arm did something structurally new. Arm normally sells designs to other companies who build the actual chips. This year it presented a finished server chip of its own, up to 136 cores, along with reference servers and named partners. Arm reported that over 60% of the CPUs paired with AI accelerators are now Arm-based.

Fujitsu's MONAKA optimized for electricity. Because power consumption rises with the square of voltage, running about 30% below the normal voltage roughly halves the power. Doing that reliably requires custom circuit design, which Fujitsu did. The result is 144 cores running heavy mathematics inside a 350-watt air-cooled envelope.

IBM presented the strangest chip at the conference, and the easiest to explain. IBM mainframes still process an estimated 70% of the world's transactions by value, and they run on an instruction set nothing else uses, which means very little modern software runs on them. IBM's new processor solves this by understanding two instruction sets in hardware: its own, and Arm's. Ordinary Arm Linux software runs natively on a mainframe, alongside mainframe software, on the same chip.

A GPU (graphics processing unit) was invented to draw video games. Drawing a frame means calculating the color of two million pixels, and the calculation for each pixel is roughly the same. So a GPU was built the opposite way to a CPU. Instead of a few very clever units that handle one complex thing at a time, thousands of simple units that all do the same simple thing simultaneously.

It turned out that training and running an AI model is almost entirely the same operation. Multiply a huge grid of numbers by another huge grid of numbers, over and over. That is exactly what thousands of identical simple units are good at. This is the whole reason NVIDIA became one of the most valuable companies on earth.

The thing that changed in 2026

Nearly every GPU talk at Hot Chips spent most of its time not on the chip, but on the rack, which is the metal cabinet holding 72 of them, plus the cooling, the power distribution and the wiring. That is steadily becoming the product now. You no longer just buy a chip. You buy a cabinet that arrives on a pallet.

The reason is that the chips became too dependent on each other to be sold separately. A modern AI model does not fit in one GPU's memory. It has to be split across dozens, which means those dozens have to talk to each other constantly and extremely fast. Whether the whole thing works depends more on the wiring between the chips than on the chips.

The new scoreboard

NVIDIA was explicit about this. They are no longer selling raw calculation speed. The number they compete on is tokens per second per megawatt, which is how many words of AI output a rack produces for a given amount of electricity, measured at a specific level of responsiveness for each user. This matters because electricity is the one thing they cannot get more of quickly, so this number sets a hard ceiling on how many people a company can serve. Double it and you serve twice the customers on the same grid connection, which is why it now drives pricing across the whole industry. Their claim for the new Rubin generation is up to 30 times the output per megawatt of their previous generation on an agent workload.

They get there two ways.

Lower precision. Numbers in a computer are stored with a certain number of digits. More digits means more accuracy but more memory and more electricity. It turns out AI models mostly do not need many digits. The industry has moved from 16 digits' worth of precision, to 8, and now to 4. Each halving roughly doubles speed and halves memory, because the same circuit can chew through twice as many half-size numbers in a go, and half-size numbers take up half the room.

Sparsity. Inside an AI model, an enormous number of the values being multiplied are zero or near-zero. That is not an accident. Models are built with steps that deliberately flatten anything negative to zero, and the attention machinery that decides which words matter ends up scoring most of them at almost nothing. Multiplying all of that by zero is wasted work. Rubin adds hardware that detects and skips those. NVIDIA claims this now works without retraining the model.

The three players

NVIDIA's Rubin is the frontier part, sold as a 72-GPU liquid-cooled rack.

AMD's MI455X and Helios rack match it closely on paper, 432 gigabytes of memory per chip and 72 chips per rack, but they differ on one strategic point. NVIDIA connects its GPUs using a proprietary system called NVLink that only NVIDIA makes. AMD built its equivalent on Ethernet, the ordinary open networking standard, using switches from another company. If you are worried about being locked into one supplier, that difference is the entire pitch.

Intel's Crescent Island is not competing at the frontier, and Intel said so out loud. The part is a 350-watt air-cooled card that plugs into an ordinary server, with up to 480 gigabytes of cheaper, slower memory. The bet is that a lot of AI work needs to hold an enormous model more than it needs to read it quickly.

If you run one enormous workload over and over, and you know exactly what it looks like, a general-purpose chip is wasteful. It contains circuitry for jobs you never do. A chip designed for your exact job can be dramatically cheaper to run.

For most of computing history that logic did not apply, because designing a chip cost hundreds of millions of dollars and took years, and by the time it arrived the workload had changed. That calculation has flipped for the largest AI companies, and Hot Chips 2026 was where it became undeniable.

OpenAI built a chip

The headline announcement came from OpenAI's VP of Hardware, and the timeline is the story. Design concept in October 2024. First working design files in February 2025. Design frozen in July 2025. Sent to the factory in November 2025, which is nine months start to finish. First silicon back in February 2026. Running production ChatGPT traffic by July 2026.

For comparison, a chip like this normally takes two to three years. The chip is called Jalapeño, and it was built with Broadcom and Celestica.

How good it is depends on how you use it. Run it flat out, caring only about total volume and not about how long any one person waits, and it comes in 1.5 to 1.9 times better per unit of electricity than NVIDIA's parts. That is a substantial gain but not a startling one.

Now insist that every user gets their words back almost instantly. A GPU pushed that way can only work on a handful of requests at a time, so most of the chip sits idle and its efficiency collapses. Jalapeño holds up, and that is where OpenAI claims between 9 and 104 times more output per unit of electricity.

Speed normally costs you efficiency. Their claim is that on this chip it doesn't.

Why one chip instead of several

OpenAI's design principle was "dark silicon is cheaper than idle accelerators." An AI request goes through phases that need completely different things: reading your prompt needs raw arithmetic, drafting needs tiny fast steps, checking the draft needs memory bandwidth. The obvious move is to build a specialized chip for each phase. OpenAI argues that is wrong, because an idle specialized chip still burns power for its packaging, its memory and its cooling, and because the model's working state has to be shipped across a network every time the phase changes. Better to build one balanced chip that switches off the parts it is not using.

They also used AI to help design the chip itself, reporting a 56% improvement on one arithmetic unit and around 10% area savings on major blocks against what their own engineers had already optimized.

The others

Google presented two chips at once for the first time, one tuned for training models and one for running them. The training pod holds 9,600 chips sharing two petabytes of memory. Google has been doing this since 2015 and reports a million-fold increase in system performance since.

Meta's MTIA 400 exists because GPUs are bad at Meta's biggest workload, which is recommending posts. That job is limited by memory rather than arithmetic, so GPUs sit idle waiting for data.

Microsoft's Maia 200 and SambaNova's SN50 both make the same architectural bet: let software decide exactly when every piece of data moves, rather than letting hardware guess.

Cerebras takes the most extreme approach in the industry. Rather than cutting the wafer into chips, they use the entire wafer as one chip. No wiring between chips, because there are no separate chips. They claim 2,000 times the memory bandwidth of NVIDIA's Rubin.

A processor cannot calculate with data it does not have. Every number it works on has to be fetched from memory first. If the memory cannot supply numbers as fast as the processor can consume them, the processor sits idle. That situation is called the memory wall, and it has been getting worse for twenty years.

The numbers Micron presented make it concrete. The arithmetic speed of AI accelerators has been roughly tripling every two years. The speed at which memory can supply data has been improving by less than double every two years. Those two lines diverge, and the gap compounds.

What HBM is

The fix is HBM, High Bandwidth Memory. Ordinary computer memory sits in slots on the motherboard, a few centimeters from the processor, connected by a modest number of wires. HBM does two things differently. It stacks the memory chips vertically, up to sixteen high, connected by thousands of microscopic vertical wires drilled straight through the silicon. It sits directly next to the processor on the same slab, connected by an enormous number of very short wires.

The result is roughly an order of magnitude more data per second. A typical server processor with ordinary memory manages around 300 gigabytes per second. An accelerator with eight HBM stacks manages several terabytes per second.

Why it is so expensive

Three things compound. Stacking requires drilling and bonding operations that ordinary memory does not need, so yields are lower. Making the same amount of storage in HBM consumes roughly three times as much silicon as ordinary memory does, because it carries a control chip underneath, thousands of vertical wires, and four times as many separately addressable sections so that all those wires have somewhere to go. And the world's memory production capacity has been essentially flat for over a decade. Memory makers spent the 2010s repeatedly building too much, crashing the price and losing money, so the three that survived became extremely reluctant to add capacity. New factories also take more than two years to build.

The consequence is a genuine shortage, record profits for the three companies who make it, and a roughly sevenfold rise in the spot price of memory. Most of that growth is price, not volume. Volumes have barely moved. Prices did all the work.

Samsung put the bottom line on a slide. Memory has grown from 52% of what an AI chip costs to build in early 2024 to 63% by late 2025. If memory is two-thirds of your cost, redesigning around memory stops being an engineering preference and becomes the business.

Where it is going

Smarter stacks. Every HBM stack sits on a chip at the bottom that has never done anything except pass signals through. Samsung wants to build that chip the way processors are built, so it can start doing real work: managing the memory, testing it, repairing it when a cell fails. Every job it takes over is a job the processor no longer needs room for, which hands silicon back to the part actually computing. Further out, Samsung wants to drop the slab altogether and stack the memory straight onto the processor. They claim that would cut the power spent shifting data by 70%.

Cheap capacity. A company called Oxmiq proposed stacking the slow, cheap memory used in storage drives instead. You get eight to sixteen times the capacity for the same money, at a fraction of the speed. They were unusually honest about where that helps, which is only the parts of a model that get read rarely. Modern models carry huge sets of specialist weights that are called on occasionally, and those can sit in slow memory without anyone noticing. Anything read constantly still needs HBM.

Compute inside the memory. Samsung's other talk went further and put small multiplier circuits inside the memory chips themselves, right beside the data, so the numbers never travel to the processor at all. On real hardware running a Llama model it produced 3.01 times more output per second than the same system with ordinary memory. It also looks like ordinary memory from the outside, so a chip can use it without being redesigned.

Chapter 04 explained why the industry stopped building one big chip. This chapter is about the consequence: if your product is now five or ten separate pieces of silicon, how you join them together determines how well it works.

There are three broad approaches, and they are usually described by dimension.

2D is the old way. Separate chips sit on a circuit board, connected by copper traces. The distance is centimeters, which in electrical terms is very far. Sending data that far costs meaningful energy and time.

2.5D puts the chips side by side on a shared slab of silicon called an interposer, which carries far denser and shorter wiring than a circuit board can. This is how HBM is attached to every AI accelerator today. TSMC's version is called CoWoS, and you will see that acronym constantly.

3D stacks the chips directly on top of each other. The distance becomes microns rather than centimeters. This is the direction everything is heading, and it is why the conference talked so much about heat.

Why the distance matters so much

d-Matrix, a startup selling chips designed around this exact problem, measured what it costs to move one bit of data. The answer depends almost entirely on how far that bit has to travel.

Call moving a bit inside the chip's own memory one unit of energy. Sending it straight up into a chip stacked directly on top costs six to eight units. Sending it sideways out to an HBM stack and back costs about 110 units.

It is the same bit of data in every case. The only thing that changed is the distance, and that costs more than a hundred times the electricity. In a system moving petabytes every second, that difference is megawatts.

Bonding without solder

Chips are normally joined with tiny balls of solder that melt to form the connection. There is a limit to how small a solder ball can be, and once you space them closer than roughly their own width, the molten balls touch and short together. That sets a floor on how tightly connections can be packed.

Hybrid bonding gets rid of the solder. Two silicon surfaces are polished until they are flat to within a few atoms, then pressed together at room temperature, where they bond on contact. Heating them afterwards makes the copper pads on each side grow into one another. What you end up with is solid copper running straight from one chip into the next, with no gap in between.

SK hynix showed the spacing between connections dropping from 30 micrometers to under 18. A micrometer is a thousandth of a millimeter. That allows stacks of more than 20 memory layers instead of 16, and the stacks run cooler too, because solid copper carries heat away far better than a solder ball with a gap around it. Every plan in this book for stacking memory straight onto a processor depends on this working. Without it the short paths stay theoretical, and the data keeps taking the expensive route.

The standard, and the thing the standard forgot

UCIe is the standard that lets chiplets from different companies plug into each other, roughly the way USB lets any device plug into any computer. Without something like it, a chiplet only works alongside its own maker's parts.

It isn't free. Intel showed what it cost them on a cheap laptop chip. The expensive kind of packaging puts a slab of silicon under the chiplets, the interposer from earlier, and that allows very fine, closely spaced contacts. Drop it to save money and the contacts have to sit three times further apart. Intel's connection block ended up 70% bigger for the same job, drew more power, and could not run as fast.

BOS Semiconductors, a Korean company, pointed at a bigger hole. UCIe says how data moves between chiplets. It says nothing about how they find each other when the power comes on, how the system boots, how they keep the same time, or how one reports a fault to another. A newer standard called the Open Chiplet Atlas covers that, and BOS built the first automotive chip to use it.

Why memory suppliers now carry the risk

SK hynix made one more point worth keeping. Memory used to be installed at the very end of assembly. In modern AI packaging, HBM is assembled first. A memory defect found late now scrapes an extremely expensive finished package rather than a cheap module. That inverts the risk model for memory suppliers, and it is part of why HBM is priced the way it is.

A frontier AI model is too large to fit inside one accelerator's memory. It has to be split across dozens, sometimes thousands. That means every time the model produces a single word, those chips have to exchange intermediate results with each other, and they have to do it before the next word can start.

So the wiring is not plumbing. It sits directly in the path of every answer, and if it is slow the whole system is slow no matter how fast the chips are.

Five networks, not one

NVIDIA split it into five, which helps, because the word "network" had been doing five different jobs at once:

Scale-up connects the accelerators inside a single rack so they behave like one big chip. This needs extreme bandwidth over a short distance. NVIDIA's version is NVLink.

Scale-out connects racks to each other across the building. This is Ethernet, the same technology as ordinary networking, tuned heavily.

Scale-across connects one building to another, potentially in a different city, because a single site cannot get enough electricity.

Scale-in is a small processor in every server that handles security, storage and network management, keeping that work off the main chips and in a separate trust boundary. These are called DPUs.

Context scale is storage that holds what a model remembers about long conversations, so it does not have to recompute it.

The strategic split

NVIDIA's NVLink is proprietary. Only NVIDIA makes it, and if you use it you are in NVIDIA's ecosystem. The argument for it is that controlling both ends lets you make it much faster: NVIDIA quotes three times lower latency and ten times the packet rate against Ethernet.

AMD, Microsoft and Broadcom all went the other way and built their equivalent on Ethernet, which any company can make switches for. AMD's version gives every accelerator in a rack the ability to read and write directly into any other accelerator's memory, as if it were local, over ordinary Ethernet cabling.

This is the clearest strategic disagreement at the conference and it is about whether customers accept a single supplier for the most performance-critical part of the system.

Two numbers worth remembering

NVIDIA said that in an AI facility, the optical components, meaning the lasers and fiber that carry data between racks, consume electricity equal to about 10% of the compute itself. That is a startling fraction for what is essentially cabling.

And on resilience: in a conventional network, one failed link takes down all the bandwidth for about 1,080 milliseconds while the system notices and reroutes. NVIDIA's new topology loses 10% of bandwidth, notices in 2.68 milliseconds, and recovers in 100. At the scale of a training run that runs for weeks, that difference is the difference between finishing and not.

A modern AI accelerator draws between 700 and 1,400 watts. A rack of 72 of them, plus networking and cooling, can draw well over 100 kilowatts. A large AI facility is measured in megawatts, and increasingly in hundreds of megawatts, which is the draw of a small city.

That is the actual constraint. Chips can be manufactured faster than electrical substations can be built and connected. Which means the question stopped being "how many chips can I buy" and became "how much useful work can I extract from the electricity I can actually get."

This is why NVIDIA measures itself in tokens per megawatt, and why several of the most interesting engineering results at the conference were about electricity rather than computation.

Cooling

Air cannot carry heat away fast enough at these densities. A given volume of water absorbs roughly four times the heat that the same volume of air does, so the industry has moved to liquid, pumped directly over the chips. The detail that matters is the temperature of the liquid. If your coolant needs to arrive at 30 °C, you need refrigeration, which costs electricity and often water. NVIDIA designed Rubin to work with coolant arriving at 45 °C, which is warm enough that you can cool it with outside air in most climates. No chillers, no evaporative water loss.

Voltage

Pushing electricity through a wire loses some of it as heat, and the loss grows sharply as the current rises. You reduce current by raising voltage. The industry is moving to 800 volts of direct current for rack power distribution, roughly like an electric vehicle, for exactly this reason.

Cerebras made the same argument at a much smaller scale and it is more vivid. In a normal system, the components that convert power to the voltage a chip needs sit about 50 millimeters away on the circuit board, and the electricity loses energy traveling that distance. Cerebras puts those converters directly on the silicon, about half a millimeter away. They claim a hundredfold improvement.

Smoothing the spikes

A training run does not draw power evenly. It ramps up hard, runs, then drops as the machines synchronize. Those swings are difficult for an electricity grid to absorb, so the utility makes you provision for the peak even though your average is much lower.

NVIDIA added energy storage that fills in the troughs and absorbs the peaks, flattening the profile. That gives a 13% reduction in peak power, and combined with other work, up to 40% more chips per unit of electricity you have contracted for. The chips did not get more efficient. The shape of the demand did.

NVIDIA's language-processing accelerator does a version of this inside the chip. Because it knows its own schedule exactly, cycle by cycle, it can order current from the regulator before it needs it, so the supply arrives exactly when the demand does. Over 60% less voltage sag, over 70% less overshoot.

Heat as a reliability problem

Google made the point most directly, citing basic chemistry: the rate of degradation in semiconductors roughly doubles for every 10 to 15 °C increase in operating temperature. Running hot does not just throttle performance today. It shortens the life of the machine.

NVIDIA's LPU turns determinism into thermal headroom. A normal processor cannot predict what it will be doing, so when any part gets hot it throttles the whole chip. A processor whose schedule is fixed at compile time can tune each block individually so every block runs right at its limit and none exceeds it, which turns wasted thermal margin into extra work done.

Two years ago TeraWulf mined bitcoin. Today it has more than $12.8 billion of contracted revenue for renting computers to AI companies.

Companies doing that have a name now. A neocloud rents out AI computing power by the hour. It takes a building with electricity running into it, fills the building with NVIDIA chips, and sells access to companies that would rather not build their own. That is the entire business.

Why everyone piled in

The field filled up fast because the trade is easy to enter. Standing up compute costs roughly $10 to $15 million per megawatt. That sounds enormous until you see what the compute earns. Anthropic's revenue per megawatt has reached as high as $50 million. When the barrier is that low and the spread is that wide, everyone does it, which is why the price of renting a chip has started climbing.

Converted bitcoin miners moved first. They already owned land with power running to it, they already had the electrical connections, and they had spent the crypto years learning how to turn megawatts into money. Cipher walked away from most of its bitcoin business and now has a $9.3 billion order book anchored by Amazon. Applied Digital signed fifteen-year leases with CoreWeave for 400 megawatts in North Dakota. Across the listed miners, as much as 70% of revenue could come from AI by the end of 2026, up from roughly 30% at the start of the year.

Then everyone else. a16z runs its own pool of chips and rents it to portfolio companies in exchange for equity. SoftBank is launching SB Neo, with its $65 billion in OpenAI supplying a ready customer. Groq, after NVIDIA licensed its technology and hired away its chief executive, raised $350 million explicitly to build a neocloud, which means the company NVIDIA took apart is now renting out NVIDIA's chips. In August, Roundhill launched an ETF for the whole category.

The part that gets skipped

The neocloud owns almost none of it.

CoreWeave's order book is approaching $100 billion across forty-nine data centers, every one of them leased. It tried to buy its landlord and failed. Fluidstack owns none of its five American buildings, and Google guarantees every lease. Nebius is building its own campuses, but its flagship $19.4 billion Microsoft contract sits in a rented building in New Jersey. Every owned campus at gigawatt scale delivers its first power in 2027 or later, while essentially every floor operating today is rented.

That matters because of a mismatch in the paperwork. Customers sign for chips for two to five years. Buildings are leased for ten to fifteen. So a neocloud has to refill the same building with newly bought chips three times or more before the rent stops. The rent is fixed. The chips are not, and nobody has priced what happens to the margin on the second and third refill.

Underneath all of it sits a layer of guarantees. NVIDIA has promised to cover the rent at Nscale, CoreWeave and Lambda, including a commitment to CoreWeave that runs to 2032. Google has done the same at three more sites and taken warrants for up to about 14% of one landlord's equity along the way. The customers are just as concentrated. Microsoft anchors five of the six largest neoclouds and was two thirds of CoreWeave's revenue last year.

So the chip company guarantees the rent, the search company guarantees the rent, and three customers sit inside almost everyone's revenue line.

The landlords noticed

IREN now sells compute directly, with Fluidstack as its customer rather than its tenant. Anthropic is reported to have signed straight with TeraWulf, around $19 billion over twenty years, though that comes from a single source. In late August, Anthropic signed $45 billion over six years with Nscale at a campus in West Virginia that Nscale actually owns.

The bear case has been wrong all year, and it is worth saying so plainly. CoreWeave's second quarter showed revenue more than doubling, and an order book that grew by about $30 billion in six weeks. Against that, the loss more than doubled and margins narrowed.

Nikesh Arora, the chief executive of Palo Alto Networks, put the skeptical case better than the short sellers have. In two years, he said in August, you will be able to buy a neocloud for less than they raise at today. He separates the generic ones from what he calls neoscalers, the handful with scale, contracted customers, their own power and their own software.

Everyone says they want to be a neocloud. What they appear to want is to be the landlord, and the landlords have started cutting out the middle.

Every chip so far leaves the factory with its wiring permanent. An FPGA does not. The name stands for field-programmable gate array, and the useful half is "field-programmable": you can rewire it after it has been built and installed. Inside is a grid of simple logic blocks and a mesh of switches between them. Send it a new configuration, the switches flip, and it becomes a different piece of hardware.

There is a catch. A chip wired permanently for one job always beats a rewireable one at that job, on speed, power and cost, because the rewireable chip has to carry all those switches around whether it uses them or not. But when the job changes, the permanent chip is scrap and the FPGA just gets reconfigured.

That makes them worth it in three cases.

The standard is still moving. A new radio protocol or radar technique might be revised twice before anyone is willing to commit it to permanent silicon.

The volume is too small. Defense, medical imaging and industrial equipment sell in thousands, not millions, so a custom chip would never earn back what it cost to design.

The timing has to be guaranteed. Some jobs need an answer within a fixed number of microseconds, every single time. An ordinary processor cannot promise that, because its operating system can interrupt whatever it is doing at any moment. An FPGA has no operating system and does only what it is wired to do.

What AMD presented

AMD gave two talks on its Versal line, which is really a hybrid. It puts reconfigurable logic alongside conventional processor cores and dedicated fixed-function blocks for the mathematics that never changes.

Versal RF handles radio. The problem they describe as "drinking from the RF firehose": you capture an enormous, wide slice of radio spectrum, and you must reduce it to something useful in real time, within tight limits on size, weight and power, because the equipment might be on an aircraft. This generation captures up to 18 gigahertz of bandwidth and converts 32 billion samples per second, with 19 times the mathematical throughput of the previous generation.

Versal Premium Gen 2 is about security and integration. Two things stand out.

First, the memory is inside the package rather than on the board. That cuts the board area by about 60% and total board size by roughly 2.5 times, which matters when the device has to fit an aircraft slot. But it also shrinks the attack surface: an attacker can no longer physically tap the wires between chip and memory, because there are no external wires to tap.

Second, post-quantum cryptography. The concern is called "harvest now, decrypt later": an adversary records your encrypted traffic today, stores it, and decrypts it years from now when quantum computers can break today's encryption. For data that must stay secret for decades, that is a present-day problem. AMD's position is that no single replacement algorithm works everywhere, because different governments mandate different ones, so the chip supports several plus a "crypto agility layer" to switch between them.

Why this matters for AI

Their list of the attacks they design against is worth reading for anyone thinking about AI outside a data center: bus snooping (physically tapping the wires between chips), firmware replay (loading an old, vulnerable software image), device substitution (swapping a genuine component for a malicious one), and side-channel extraction (deducing secret keys by measuring power draw, electromagnetic emissions or timing).

None of these are possible against a chip in a locked data center. All of them are possible against a chip in a car, a factory, or a piece of equipment in the field. As AI moves out of the building, that threat model comes with it.

Everything so far has assumed a data center: unlimited power, liquid cooling, and a user who will wait a second for an answer. Move AI into a car, a factory or a robot and all three assumptions break.

The critical difference is that a data center optimizes for average speed across many users, while a car needs a guaranteed maximum response time for one. A system that is usually fast and occasionally slow is fine for a chatbot and unacceptable for a vehicle.

Waymo's chip

Waymo used its keynote to reveal its first purpose-built chip for self-driving, already shipping in its sixth-generation vehicles. You can ride in one today.

The specifications are modest by data center standards and that is the point: 208 square millimeters of silicon, memory inside the package, and under 75 watts, against 700 to 1,400 for a data center accelerator. It contains a Waymo-designed AI engine they call the carTPU, plus their own image processing, plus licensed graphics circuitry for handling lidar and radar.

Look at what they left out, because it shows what you do when predictability matters more than peak speed. There are no caches, no branch prediction, and no hardware register allocation. Those features all exist to make an unpredictable program run faster on average, at the cost of making its timing variable. Waymo removed them. Everything is decided when the software is compiled, so the chip's timing is knowable in advance.

Their stated design goal is "first pixel in to embedding out," meaning the time from light hitting the camera sensor to a usable understanding of the scene, for one frame, with nothing batched up behind it.

Their closing line is the one to remember: real-world performance with true co-design matters over raw TOPS. A company that controls its own sensors, its own models and its own silicon can beat a much larger general-purpose part.

Cars are consolidating

Infineon described the broader shift. A car used to contain a hundred or more small computers, one per function, connected by an enormous and heavy wiring harness. That is collapsing into a few central computers plus a handful of zone controllers, one per region of the car, which cuts wiring weight and cost substantially.

The difficulty is that one zone controller has to do several incompatible things at once: hard real-time control where a response is guaranteed within 50 microseconds, signal processing, AI inference, power management, and audio. That mix is why the industry wants a scalable instruction set, and why RISC-V is being seriously evaluated.

But Infineon's honest conclusion cuts against the enthusiasm. In their words, the detailed aspects of the instruction set mostly do not matter; what decides an automotive design win is memory, connectivity, safety certification, availability of certified development tools, and cost.

The chiplet question, for cars

BOS Semiconductors, a Korean company, showed a chiplet-based automotive AI accelerator using licensed technology from Tenstorrent. It is a small but real signal: chip design intellectual property is now being licensed and assembled the way software libraries are.