Micron held its Technology and Product Webinar last week. I watched it and took notes. Here’s my analyst take for investors and tech executives, along with some feedback for Micron.
1. Does AI memory demand hold up as inference gets specialized?
Vivek Arya asked an important question, and Micron needs a better answer to it.
Vivek pointed out that memory is now half or more of an accelerator’s bill of materials. Customers are talking about “despeccing”, i.e. using less or cheaper memory. So could inference accelerators end up with less DRAM or different DRAM? That would mean fewer bits for Micron...
Micron’s Scott DeBoer answered that systems perform better with more memory capacity and bandwidth. Customers will pick the minimum spec that fits their product and their business. Micron’s job is to add supply and push bandwidth higher with products like vertical HBM.
He closed with “Inherently, more memory does equal better system performance”.
My take: That answers the wrong question…
Vivek is right that accelerators now ship in such volume that customers can build hardware for each workload. Look at Google’s training and inference chips:
TPU 8i, the inference chip, has less compute and more memory.
TPU 8t, the training chip, has more compute, but less memory bandwidth since training is mostly compute-bound.
The important point: Customers are right-sizing memory.
For a given workload, beyond a certain point, more memory adds cost faster than it adds throughput.
So DeBoer is right that more memory helps performance, but that’s not what Vivek was asking. The question Vivek was asking is what are the implications for memory vendors of “right-sizing” the memory to the given workload.
I’d take Vivek’s question a step further.
Inference itself splits into two phases. Prefill processes the prompt, and it’s compute-bound. Decode generates the output one token at a time, and it’s memory-bound. Disaggregated inference runs the two phases on separate chips.
So are some accelerators carrying more HBM than their job needs? If chips get sized separately for training, prefill and decode, does total memory demand fall? Or do they offset? A prefill chip might need less HBM. But a decode chip wants more, and racks often have more decode chips than prefill chips...
There’s a lot to unpack here.
Start with the fact that chipmakers already differentiate on how much memory they give an accelerator. AMD’s MI455X carries 432 GB of HBM4. Nvidia’s Rubin carries 288 GB.
And some designs swap HBM for cheaper DRAM. Remember when Nvidia announced Rubin CPX, a prefill chip with 128 GB of GDDR7 and no HBM?
Startups are making different bets here too. For example, Positron’s Asimov chip skips HBM entirely for LPDDR5X, configurable from 288 GB up to 2.3 TB per chip, so the customer picks the capacity for the workload. That could lead to extreme right sizing…
Yes, GDDR7 and LPDDR are still DRAM, and Micron sells both. But they have different margins, and that changes over time.
So beyond just asking about how right-sizing impacts total bit volume, we also want to know about mix.
Recall the “trade ratio”, where HBM takes three times the wafers per bit of DDR5. Micron says “this trade ratio only increases with future generations of HBM”. So every bit that moves from HBM to LPDDR or GDDR due to architectural right-sizing frees up many DRAM wafers.
But again, HBM earns lower margins than Micron’s other DRAM today, so the bits that move could be more profitable, not less...
The point I’m trying to make is that memory demand now depends on how each accelerator gets sized for its workload.
Think about future XPU and GPU roadmaps:
- How much HBM does a training chip carry?
- Does prefill run on HBM or GDDR?
- Does the KV cache live in HBM, SRAM or LPDDR?
Architectural choices decide how many bits the industry needs, and which kind.
So forecasting demand means tracking where accelerator architecture is going.
And surely memory vendors are taking this into consideration when forecasting their supply plans, right?
Micron sees many of these designs early, too, especially now that it co-designs custom base dies with customers starting at HBM4E.
Micron should have a much more thoughtful answer than the one it gave. I
know they have one; it’s just a matter of better articulating it.
2. Does custom HBM help or hurt Micron’s margins?
Who captures the value when customers put their own logic in the base die?