Things we cover:

• Why the CUDA moat shrank once LLM inference became the biggest market

• Intel Foundry’s culture change, and EMIB as an alternative when CoWoS capacity is short

• Advanced packaging, from OSATs to 3D stacking

• Prefill and decode disaggregation, and why it becomes a rack and data center design problem

• Edge compute for physical AI, including Rivian’s RAP1

• AI-native EDA startups vs. Cadence, Synopsys, and Siemens

• Rapid fire on TSMC, Anthropic, OpenAI, Hock Tan, and quantum

Also, Primary is hiring:

This podcast is lightly edited for clarity.

Introducing Austin

Gaby: Today’s guest is Austin Lyons. He is the writer behind Chipstrat. When I was getting up to speed on the compute market, one of the first places I looked for information was Substacks. All these independent analysts were writing incredible pieces at the intersection of technology and business, and Chipstrat was one of the earliest ones that I found. So getting to talk with him and hear his journey, not only building the media engine around Chipstrat, but also being in conversation with some of the most senior leaders in compute, has been really fun. What did you think?

Brian: One of the things about Austin, one, he’s just a really nice guy, but he has this set of experiences, from his MBA to startups to working in industry, that has now given him some credibility to predict what are the most important themes coming, whether it’s emerging ASICs or the reemergence of Intel as a really exciting force in the semiconductor industry. Just an exciting conversation to hear what he’s thinking about these days of what’s coming.

The Compute 100 is a podcast about the most consequential market in AI.

Gaby: These are the conversations with builders and investors about the technology and the capital powering the greatest infrastructure buildout ever.

Brian: I’m Brian Schechter.

Gaby: I’m Gaby Lorenzi.

Brian: We’re investors at Primary.

Gaby: Let’s dive in to The Compute 100. This podcast is for informational purposes only and does not constitute investment advice or an offer to buy or sell any securities.

From Intel Engineer to Chipstrat

Gaby: Austin Lyons, thank you so much for joining us on The Compute 100 podcast. We’re super excited for the conversation.

Austin: Awesome. Yes, thank you for having me.

Gaby: You sit at one of probably the most interesting places in the content-meets-semiconductor world over the past five years. I’m sure your career has gone through a complete evolution as this market has caught up to the world that you’re really interested in. I’d love to maybe start 10 years back, even your first career post-grad, and hear your story up to what you’re working on today at Chipstrat.

Austin: Sure, totally. I’m older than I look, so it might be longer than people will think. I got trained as an electrical engineer in undergrad and in grad school. Did my master’s at the University of Illinois. And interestingly, when I was at the University of Illinois, I was actually in the PhD program, but I started a startup with some friends, some other grad students. This was back in the Kickstarter days.

We launched something on Kickstarter, had a lot of fun with that, and it really opened my aperture for what’s possible. So I was like, oh man, I’ve got to go get into industry and maybe get into startups. I actually went straight to Intel out of grad school, down in Austin, Texas, and was working on a startup at night, not in semiconductors. This was more of the IoT, iPhone apps, B2B SaaS early days era.

So I was doing hardware engineering by day, I was doing software engineering by night. And that set me down a path of thinking beyond just engineering, but more about who is the customer? What do they really want? Is this useful? Ultimately, we raised some money, we built a product, and it turned out customers didn’t really want it. So you learn early on the hard way. Fast forward, I ended up eventually moving into product management and getting an MBA.

I was really impacted by those entrepreneurial experiences. So I sit at this nexus of, I understand the deep technology, I can think like a product manager and ask the questions about roadmap, value prop, pricing, that kind of thing. And then I have a bit of an entrepreneurial spirit myself. So of course, I always love to study startups as well.

Gaby: And so all of this was phenomenally well-timed with what is today the compute revolution. Take us through the last five years of deciding to go out on your own with Chipstrat and really build around the ecosystem that is education and technical deep dives around the compute ecosystem.

Austin: Yes. So I was working as a product manager by day and of course the world was changing. ChatGPT was launched and all of a sudden semiconductors, which was my original first love, became super interesting. Early on, of course, it was just from the perspective of logic, but now it’s so much more. It’s logic, memory, optics, power, foundry, all the things. I was spending a lot of time consuming media.

I was training for marathons, so consuming podcasts, reading people’s things, trying to pay attention. And I just saw a gap in the market from an education content perspective. There’s either very technical things or very non-technical, thin, FinTwit-type things. And I just thought, we’re missing a Ben Thompson for the semiconductor industry here.

I felt like my background gave me a particular take on certain things. So I just decided to start writing, because it was as accessible as ever. And then it snowballed from there. You start writing and essentially you start putting your public thoughts out there in the court of public opinion. It’s a little intimidating at first, but it turns out most people are really nice and really interested, and you find the other nerds like you who want to talk about this stuff.

And then, as you guys know how it goes, it just snowballs from there.

Early Takes: AI Accelerator Startups and AMD

Brian: So what is your particular take? I know that there’s a lot in that, but walk us through some of it. Ben Thompson has some core ideas that animate his writing. Curious to hear what are some of your core ideas right now. I’m sure they’re constantly evolving, but as some of them become solidified, what are they?

Austin: Sure. So some of my early takes. Very early on, there’s an AI accelerator startup, Etched, and I started looking at them. Early on, it was just the Nvidia gravy train. Everyone realized, holy cow, GPUs are the future of AI, and Nvidia is obviously dominating in training, and we weren’t even into the inference era yet. And I saw that Etched was out there and I started asking questions like, could there be companies in this space other than Nvidia?

So Etched was one of them that got me into thinking about, what would it take for a startup to compete with Nvidia? Would the model labs trust and buy from a startup? Would the hyperscalers trust and buy from a startup? And so looking at first principles of, what’s wrong with the GPU? And what’s right with the GPU? I started pulling on threads and realizing, hey, GPUs are super flexible.

That’s great for training, that’s great for inference. But at the same time, once you start to fast forward, if transformer-based inference is the defining workload of our time, which has turned out to be true, then it would make a lot of sense for hyperscalers who actually know their workloads to invest in something that might be a little bit less flexible, but as a trade-off, it’s hyper-designed for the workload at hand, for attention and LLMs and KV cache.

So early on I started saying, hey, I think there’s something here. I think these AI accelerator companies can compete. And then similarly, early on, I was actually pretty bullish AMD. I think that was one of my first posts, about a year in, that started to get a lot of traction. Everyone at the time was talking about this CUDA moat.

Part of the CUDA moat early on was the fact that Nvidia had written software so that anyone in almost any domain could accelerate their workload, from scientific computing to oil and gas to quantitative finance and so on and so forth. So anyone who’s trying to compete, say AMD was trying to compete, people would say, oh, well, you have to write libraries in all of these different domains and it’s going to take you hundreds of thousands of man-years.

But once it became that the TAM for LLM inference was going to be way bigger than any of these other high-performance computing things, all of a sudden that CUDA moat kind of shrank. To compete for all these dollars, maybe AMD doesn’t have to even worry about all those other markets. Maybe if they just focus on making their Instinct chip and their ROCm software competitive for LLM inference and start there, they could actually compete.

So those are actually two very similar takes, which is, I think startups can compete in this market and I think that AMD can compete. I’ll just pause there, but it was very focused on logic early on.

Intel: Fix CPUs and Foundry First

Brian: I also enjoyed some of your articulation around Intel and where you see their place in the market right now. Would love to hear you go on that. Maybe we could set the table of Austin’s takes on various dimensions of the compute world and then we’ll double click wherever seems most interesting.

Gaby: I think, Austin, we’ve had probably the most ex-Intel people on this podcast. So we’re happy to have another one and just excited to get your take as someone who was there.

Austin: Yes. Well, I’m wearing my Intel blue today. So, Intel. Early on, once Nvidia started selling all the GPUs and making a ton of money, everyone was saying, where’s Intel? Where’s Intel’s GPU? At the time, I said, hold the phone, Intel is a CPU company. And frankly, at the time, this was back in the Pat Gelsinger days, when they were even trying to figure out, are we going to have an external foundry or not? What is our strategy? And Intel was behind.

Basically, the advantage that Intel always had was they could do their own manufacturing. For many, many years, that gave them a performance benefit compared to everyone else, because they could get to the next node from a manufacturing process faster, and therefore they could reap that and bring more performant, more power-performant products to the market. Again, mostly CPUs, both for client and for server.

Then there was a period of time around 10 nanometer, 7 nanometer, where Intel actually fell behind for various reasons. One being not getting on EUV fast enough. Basically you can think of Intel as manufacturing and as product. They’re an integrated device manufacturer. They do both. They don’t use TSMC. Manufacturing used to really benefit the product team. Once manufacturing fell behind, all of a sudden that really started to drag down the product team and expose, if you have the same technology as everyone else or worse transistor technology, are your designs actually as performant and do they compete? So manufacturing started to fall down and then the product team started to fall down.

Meanwhile, everyone’s starting to say, okay, Intel’s struggling, but oh, they just need to make GPUs to catch up. And my question was, should they make GPUs? Or should they not make GPUs, and not make all these other networking chips and stuff, and actually just focus on fixing both of those things? Make better CPUs like they used to. Of course, AMD was starting to eat their lunch, Arm was starting to eat their lunch. And figure out the manufacturing thing.

Not only get back to being competitive, but then of course the problem with manufacturing. Everyone used to manufacture their own chips, because there’s actually a benefit if you can co-design and design for manufacturing. If you can do that together, not just design the chip but also design the manufacturing, you can optimize up and down the stack. But with every generation of foundry, it becomes more and more expensive, to the point where most companies didn’t have the volume to fully utilize and amortize all of those huge CapEx investments. So people kept falling off, but Intel had enough volume. Even they were starting to get to the limits. So it started to beg the question, if their manufacturing could catch back up and if they could take on external companies, could they get back into the game and compete with TSMC, both on performance but also financially? Could it actually be competitive from a cost perspective?

So early on, my Intel hot take was simply, they shouldn’t focus on GPUs. They should actually just right the ship, make sure their CPUs are performant, and double down on foundry. And now, I will say, that has paid off for Intel. CPUs have exploded with agentic AI. I didn’t necessarily expect that early on, but it has worked out for them in that their CPUs have been in high demand and can help continue to fund the business as they’re starting to bring foundry up.

And of course, within the past six months, we’ve started to hear a lot about Intel Foundry and external customers. So they are on stable ground with CPUs and with manufacturing. It still begs the question, what is their AI strategy? What is their GPU strategy? But I think now Lip-Bu has put enough sturdy ground underneath their business that they have earned the right to expand into GPUs.

Gaby: Was your belief that investing in manufacturing is what will enable Intel to be a better partner to external customers, hence using that spare capacity to deliver for whoever they might be selling to in the next decade?

Austin: Yes. So the question for Intel is, why didn’t people just raise their hand when everyone wanted to make GPUs and TSMC said, we are supply limited, and it depends on where you are in line, and you get allocated so many wafers? And Intel said, hey, we’re going to get into the game. Why didn’t people rush over to Intel?

Obviously, there’s a big cost in saying, now we have to redesign everything for a different foundry. But ultimately it was cultural. When you’re an IDM, an integrated device manufacturer, your foundry ultimately serves your product team, and you do have this internal groupthink of, this is the way we do things and this is the best. That is not a customer-centric mindset. Your customer is the product team, but you’re all the same company. So when Intel said that they were going to become an external foundry and they started working even with small customers, it did give them an opportunity to start trying to build that muscle.

And probably the best thing that has happened at Intel is they’ve brought in external hires who’ve worked at other foundries, whether it was GlobalFoundries or even Micron or Samsung, memory companies, who already have that correct orientation of, I work in a service business. Because that’s what manufacturing is.

So yes, I think Intel has had to undergo a lot of cultural change, but they seem to have worked through this. And ultimately it will end up benefiting Intel products, because the Intel manufacturing team now has to make the decision, who should get the capacity? Who should get the wafers? Should it be our internal products team or should it be our external customers? And that’s actually going to be great for both of them, because now the products team can’t be sloppy and just be like, ah, we screwed up here, do a hot run and run a couple more wafers for us. It’s like, no, no, no, you have to behave like an external customer. You need to plan in advance, and there might be times where you have to go to the end of the line and wait your turn.

Gaby: Yeah, it’s interesting. It’s almost similar to what’s playing out in some of the hyperscalers that are designing their own chips now, and the way that they have to think about resource allocation between their own internal teams using their systems for the development of AI versus selling those chips, which are ultimately becoming really valuable product lines. So it is going to be interesting to see how that ultimately affects Intel and what their business looks like in the future. Very, very curious to see.

Austin: Yes, totally. It’s a brand new world. And of course, again, none of us could have predicted all the geopolitical stuff that’s going on, which also indirectly benefits Intel Foundry as customers are incentivized to say, I’ll give you a chance.

Advanced Packaging

Gaby: Totally. So one area that you spoke to that Intel has done a lot of really great innovation in is packaging. Something that we’ve been tracking from a startup perspective is how much value is going to be accruing to the fabs themselves. As we start to see systems like power become more integrated into the package, cooling becoming more integrated, so much value is going to be going to packaging and who’s doing that. So I’d love to hear a little bit more about what you saw and those learnings from your deep dives into the Intel space, and then do a little bit of a segue into what you’re tracking in packaging these days.

Austin: So packaging used to be a fairly boring space, low value, low margin. If you go way back in time, packaging is literally taking discrete transistors and wiring them together. And then it was taking discrete chips on a board, printed circuit boards, and just big copper traces in between them.

But nowadays, we’ve moved into advanced packaging, where we’re taking chiplets or dies and trying to connect them in three dimensions. And that’s way more sophisticated from the perspective of how do you route everything electrically. It’s complicated from both a design and a manufacturing perspective. From a design perspective, you’re thinking about crosstalk and noise and electrically routing all these signals to the right places.

Brian: Austin, maybe if I could break it down even more simply and just check for understanding, because the 3D thing was confusing for me for a while. One way that I think about advanced packaging right now is, we used to have city streets where every building had at least alleys between them, if not roads.

And we now have begun building the buildings to be touching each other and have shared plumbing and electrical connectivity between the two of them underneath. And so that is now the 3D element, where it’s no longer just everything laid out on a board assuming that it’s its own piece. Is that what we mean by 3D?

Austin: Yeah, that’s a good analogy. Historically it was just, you had a chip and that was it. You’ve got a house and it’s in the country, whatever. We’re not even dealing with roads. And then eventually there’s this 2.5D, where it was, yes, let’s start to put buildings near each other, and now we have to put roads and alleys and connections between them.

And then as we’re headed into 3D, for example, high bandwidth memory is 3D-stacked DRAM chips. You’re literally stacking them up and you have to think about the internal plumbing. If you’re talking about a high-rise building, how do I get heating and cooling to all of these, get the waste out, electricity?

And we will continue. In the future, there are actually companies who are working on stacking logic chips. You could either have logic chips with memory on top or logic chips stacked on logic chips. So yes, now we do have to think about the infrastructure in three dimensions.

It used to be OSATs that did this packaging, and that stands for outsourced semiconductor assembly and test. It says outsourced. Right there you can tell it’s low value. But now it’s obviously getting higher and higher value and getting more and more complicated. At the end of the day, it’s very similar to what foundries do when they’re fabricating logic chips or memory chips. It’s that level of deposition and etch and patterning.

Now the OSATs are trying to move into that space, because they want to accrue the value at the OSAT layer and they want to say, no, no, we can do CoWoS now, so if you can’t get it from TSMC, we can do that too. But of course it’s also an opportunity for TSMC and Intel to expand from just making logic chips to doing more of the full-on logic, dicing, packaging, everything.

Gaby: And so when you think about it, maybe selfishly from a startup perspective, thinking about where there’s new technologies that are interesting to invest in, as so much more value is going to be accruing to people who are doing packaging or TSMC as a fab, how do you think about the startup landscape evolving? Is it largely going to be startups that are building around new processes or startups that are enabling new types of packaging techniques?

We’ve seen companies that want to be a next-gen Amkor or a next-gen OSAT more broadly. But would love to just hear how you think this affects the newcomers and how they build something that’s really valuable.

Austin: Yeah, this is tough. One of the hard parts when you first look at the semiconductor industry is it feels like everywhere in the value chain, there’s two or three or four companies and they’re huge. So you have to ask, okay, if I am a PhD student and I’m studying some really interesting new material or something, how do I take on Applied Materials or someone like that?

I don’t have scale, but what could I do? Obviously, one thing that you’re thinking about right away is, could I build some interesting technology that they would want to acquire? But if you want to be a full-on company, let’s talk advanced packaging. As this is getting smaller and smaller and the power consumed by these dies continues to increase, there’s really interesting questions around how do you deliver power there?

Power used to come in from way off to the side. Are there ways to integrate it much closer on the package, or even as a die, as part of this 3D stack? So you can think about, are there power delivery opportunities? Are there power dissipation opportunities? Interestingly, I think what probably doesn’t get appreciated enough is from an EDA or a design tool perspective.

There used to be people that would design and think about the electrical characteristics, and maybe different mechanical engineer type people thinking about the thermal characteristics. And now we live in a world where you have to always be doing both and thinking about both, because it’s just such small places with such hot, hot dies that are interconnected in three dimensions.

So I think there’s opportunities to ask, are there ways that we could help solve this or create a new user experience from a design perspective, where the electrical design and the thermal design is taken into consideration? There’s definitely opportunities. It’s just about that seam of how do you take your innovation and bring it to market and get in here.

EUV, EMIB, and CoWoS

Brian: Can we actually spend another minute on that? Austin, take us further into that mistake of not investing in EUV, and then also the technical bets that they did make that people are now starting to say look like they’re starting to be really promising.

Austin: Both of those, because both are important. From the logic perspective, one thing that people don’t appreciate about Intel is they actually do a lot of the transistor research and development that enables future generations that the whole industry uses, TSMC included.

Intel’s always been really good at that early R&D. They have innovations from the early days like strained silicon and using different materials as oxides and things like this. So they’ve always really pushed forward on stuff. But there was a period where they took some bets on some materials that didn’t work, and they ended up delaying jumping to EUV. I can’t remember off the top of my head exactly why, if it was a cost concern.

Brian: I think it was a cost concern. I think the cost was too big.

Austin: That actually aligns with the idea that they had some more financial-oriented folks who were driving the ship during those times and not engineering-centric folks. So Intel, yeah, they slowed down there. Basically they were stuck at a particular node, 10, 7, while TSMC was able to catch up and keep going. And then of course everyone, AMD for example, is using TSMC and they’re getting on smaller and smaller transistors, and Intel was essentially stuck.

At the same time, Intel has always been innovating on packaging, and EMIB is one of their packaging processes that they use. The idea is, once you start to make chips that are really big and you hit the reticle limit, you might want to make two of them and tie them together.

TSMC, their approach is CoWoS, chip on wafer on substrate. And right there you can hear it. There’s three levels there, chip, wafer, substrate. So there’s this interposer where the connections happen. Intel has historically had EMIB, where basically just in the substrate itself, they embed these little bridges. So they just put the connections in between the chips right where you need them. The rest of it is a substrate.

And it hasn’t gotten much attention at all until very recently. But one of the benefits of being an integrated device manufacturer is, when your technology works and you have the scale of Intel’s product team making CPUs for tons of laptops and every company worldwide, you have the opportunity to take your packaging and put it in production and run it at scale. So now Intel Foundry is talking to potential customers who either just can’t get enough CoWoS capacity, ultimately they might be gated by the amount of wafers they can produce at TSMC, or, there’s different flavors of CoWoS, but some of the early flavors that were adopted have particular trade-offs.

And even as TSMC has come up with something called CoWoS-L, which is local silicon interconnect, and it’s fairly similar to EMIB, there’s still the three layers versus two. But there are some scaling problems with the way that CoWoS is implemented. And Intel Foundry can essentially stand on the sideline and say, hey guys, we’ve got this thing called EMIB, it scales really well. We are already making it in volume.

We’re servicing all of our Intel products team and all of their products that they’re shipping. So we know that it works and we have spare capacity. So yes, as we’ve moved to these AI accelerators that want to just be as big as freaking possible and have all this HBM that needs to be interconnected, it’s the perfect timing for EMIB and Intel Foundry.

Information Sources and the Master Plan

Brian: Where are you getting your information from? We listen to various thought leaders in the compute space and they all have their information edge. Where do you get your information? And also, to the extent that you’re open to it, what are your goals? What’s the Austin Lyons master plan?

Austin: Yeah, these are awesome questions. So where do I get my information? I of course try to consume all the public information that I can, earnings reports, whatever. When I can get my hands on expert interviews, those are interesting. You always have to take those with a grain of salt. And then I have the opportunity in my day job with Creative Strategies as an analyst to talk to a lot of these merchant silicon companies and talk with their executives.

Sometimes I’ll bring them on and publicly interview them, or other times I’ll get to talk to them behind closed doors and just ask them what’s top of mind. So that is an awesome privilege. But I also am always trying to talk to startups, see what people are saying on X. Again, a lot of it is public conversation. If there’s one thing I could do more of, it’s continue to try to talk to people who are in the trenches as much as possible.

Going to OCP or conferences is a really good place to meet people and have the opportunity to just ask them. I try to balance, okay, what is the market narrative around something? What is the FinTwit narrative? What are the thought leaders saying? What is Gavin Baker saying? But what am I hearing from companies? What are they saying?

And then thinking about questions where it’s like, well, the company said this, but did they actually mean this? Or they specifically didn’t talk about this. So trying to find those gaps and noodle on some of those. And then I will say, one of my advantages, I think, is that I don’t live in Silicon Valley. I’m outside of the thought bubble of, everyone’s thinking this or saying this. I live in the Midwest, and so we have a particular perspective on how the world works. And as you guys are experienced with as well, I’ve worked in industries outside of this.

That lived experience also gives you an ability to think critically and differently about certain things. And then to Brian’s question of the master plan. When I was a product manager, what was awesome was, ah, I like to think about strategy, I like to think about technology, this is so fun. I get to make a plan. What’s our one-year plan? What’s our three-year plan?

And then you think about it for a little bit and then you just spend the rest of your life implementing it. So it’s a little bit of strategy and a lot of execution. What I like about my current role, working as an analyst, writing Chipstrat, and then I also host a podcast called Semi Doped with Vik Sekar, is I just like to be on the forefront as new information comes out and think strategically. Oh, what does this mean?

China’s going to shut down access to indium phosphide. Oh no, is that real? What does it mean? So a lot of just thinking strategically. And then fortunately I’ve been able to build up relationships with executives and people at these companies to start to ask, I think this means this for your company. What is your answer to that?

Brian: Yeah, it’s really interesting, because what it points to, for the hardware designer, whether it’s a startup or an incumbent or a hyperscaler, is having a point of view around the TAM associated with different types of value that AI can create. Because ultimately the TAM needs to be there, and the margin profile will need to be there as well.

Disaggregated Compute

Gaby: I’d love to spend more time on disaggregated compute. One of the areas that people are spending time thinking about is, in response to disaggregation, a lot of new areas emerge, or maybe not new, but the downfalls of disaggregation emerge, like networking or how close your power supply is. So walk through your framework. If we move to this disaggregated world where a single vendor has all of these different compute offerings for different parts of workloads, how do you actually bring that together in a way that you can still enable efficient serving of inference, or maybe it ends up being for training, but largely for inference, when you have to deal more with networking, power, memory, et cetera?

Austin: Yes. So when it comes to disaggregation, we’ll start with prefill-decode disaggregation, because everyone knows that. It actually makes a lot of sense. There’s these two subtasks and they have two different memory constraints or profiles, how much memory capacity and memory bandwidth they need. Prefill doesn’t need much and decode needs a lot.

So right there you’re like, wait a minute, we could have two different hardware SKUs, and then we could run part of the workload on one and part of the workload on the other. But then to your point, Gaby, now that means you do add some complexity. You need to orchestrate across them, you need to send information across them. So right away that starts to point to, we’re not talking about chip design, we’re talking about rack design and ultimately data center design. You start to ask questions about, what is the most power-efficient, most cost-effective, fastest way to send information back and forth?

You could go so far as to ask, how should we even architect the data center? Do you have racks of prefill chips, racks of decode chips, or do you have a rack that has prefill and decode in it so they can talk back and forth? And that even goes to co-design with the model companies. How are they taking their mixture of experts and splitting it across? We’ve got these layers on this GPU and these layers on that GPU, this expert on this GPU, this expert on that GPU.

I do think that’s why it pushes Nvidia to just say, oh, I want to try to get 576 GPUs in the same rack, even if the power’s crazy, if the cooling’s crazy, if the manufacturing’s crazy. You’re essentially trying to keep as much of the compute as close as possible to make it easier to send everything over copper if you can. But it does start to become a problem, and it opens up opportunities again to rethink. Oh yeah, but what about long context size and KV caching, and a cache miss versus a cache hit?

Should we get clever and interesting with having a rack right next door that’s just a ton of storage, so you can store the KV caches as close as possible? So we’re obviously disaggregating things and saying, how can we have the proper SKU for the proper workload need? But we’re also realizing, oh crap, there’s trade-offs, and how can we also aggregate it as much as possible and get them as close as possible?

Time will tell where other interesting innovations happen, from networking to memory to cooling, to continue to let us get more done faster but not waste as much power sending bits all over the place.

Physical AI and Edge Compute

Brian: As we’re talking about this, a lot of the reason for all of this comes back to the size of the models and the size of the context windows that they benefit from in terms of developing more and more useful forms of intelligence. This is a change of topic some, but what are you most excited about in terms of AI and what capabilities are coming online? And what are you most hungry to see the semiconductor industry do to help enable those types of possibilities?

Austin: I’m most excited about physical AI, about embodied AI. If you think about it, yes, in the last 30 years, an increasing amount of our world’s GDP is impacted by digital things, Netflix and Facebook and the internet and Riverside.fm and what have you. But so much of the world is still physical. Everything that you buy, everything that’s manufactured, Apple, Nvidia, it’s all supply chains, it’s all physical hardware, it’s all getting moved, it’s all getting shipped.

Of course, I live in places where people are growing corn and soybeans and hogs and things. Anyone who eats anything, that had to get physically moved. And most of that stuff is still not touched by AI. It’s barely touched by digital. There’s some logistics and things that go on. But literally, I was at a large grocery retailer and we were using mainframes still, and this was not a decade ago.

And they will continue to. There’s places where people are still using pen and paper. So it’s crazy to think, obviously just getting digital is one thing, but man, how can physical AI just unlock even more productivity? And it’s not just making what you’re doing more productive, like saying, here’s the way we work today, how can we be more productive? One of the cool things with anything digital that’s AI-enabled is it enables some solopreneur to do so much more than they ever could before.

Oh, hey, I’m a domain expert. I’m a dentist and I’ve always thought our software sucks, but now with AI, I can just write better dental software, because I’m the domain expert and I can just do it, send Fable off for a week. My oldest is in middle school and he’s been spending this summer coding an indie game and it’s super cool. He knows how to code but not that good, but Fable can just go off and do it.

And then he can spend a lot of time thinking about the story and the graphics and the art and stuff like this. Now imagine if we could take that and put it in the physical world. What if I could realize that I hate this particular part of my car? Of course I could 3D print it and stuff, but what if I could actually manufacture and assemble it and be like, hey, I’m going to make this bespoke add-on piece to cars and start selling it Etsy-style or something?

But it’s actually something physical. In Iowa where I live, lots of small towns are supported by a local manufacturer. It might be snowmobiles or basketball hoops or whatever. But they suffer from brain drain like everyone else does, which is everyone going off to the big city, and they don’t want to come back and live there and work on it. What if you could use robotics and physical AI innovations somehow to make it so the same amount of people could get more physical work done, or they could graduate to managing fleets of humanoids instead of being the person doing the thing?

It’s obviously very fuzzy, but what I’m excited about is, man, it feels like there’s so much human productivity that’s not unlocked yet, and there’s so many interesting, capable people that have very interesting mechanical or physical ideas, but we just don’t have the tools for them to just build.

Brian: And so what hardware, compute-specific unlocks do you think will lead to breakthroughs in physical AI? Or do you think that’s part of the equation?

Austin: Yeah, good question. Historically, you’ve just got embedded compute, and it’s little CPUs and not much memory and it’s all pretty wimpy. Nvidia has had Orin and Thor. They have a small portfolio, and it has certain power requirements, and it fits perfectly for certain use cases. But if you go talk to people who are doing AI at the edge, you’ll find out there are customers who are like, oh yeah, we just have to take three Nvidia SOMs and bolt them together so that we have enough compute to do what we want to do.

But that requires a ton of power. There’s all these trade-offs today. So I just don’t think we’re yet to a point where we’ve got enough different compute options for the edge, whether it’s more FLOPs or more memory or less power or just more different configurations. We are in the early days and it is a little bit of, you get what you get. Think about it. With a humanoid, it’s got a brain, it’s got sensing, it’s got all these little sensors at the fingertips, and you could have distributed computing within it.

I don’t think we even know what is the right hardware architecture there. There’s CPUs, there’s GPUs, there’s FPGAs. But are we designing the systems we build today based on the hardware that’s given to us, or are we building the right hardware for where these workloads are going?

Gaby: We talk a good bit about the hardware lottery, and this idea that what is working and the research ideas that win are just what is suitable to what exists at the time. If you think about it, cloud was enabled by CPUs, AI enabled by GPUs, and there’s likely another iteration of this going to happen in the physical AI world, a little bit more than just what a Jetson offers, if we use that as the best edge compute that exists today. But it almost makes it hard to think about what are the use cases that a next-gen Jetson, or the next best piece of edge compute, could enable.

We’ve spent most of our time thinking about how edge will impact personal computers, or the way that we’re running more models locally, and how that eventually will trickle down into the economics of the AI economy. But of course the physical AI world is going to be, hopefully, an explosive customer of edge compute. In terms of the conversations you’re having with people around this market, where are the real pull and pain points of what exists today in the physical AI space? We hear a lot about power, a lot about weight, about latency. But I’d be curious, in the conversations that you’re in, where you feel like physical AI’s compute is not stacking up today.

Austin: I think what Rivian is doing with their RAP1, the Rivian Autonomy Processor, is pretty interesting, because they are designing their own platform. They said what they needed off the shelf wasn’t there. And it was very specific. For personal vehicle autonomy, we need a real-time operating system and some real-time safety that has hard latency budgets and everything. But we also have cameras, we have lidar, so all this sensor data input, lots of data, it’s moving around. They actually created their own communication protocol. They called it RivLink, which is like NVLink or something.

They made a lot of trade-offs for their particular portfolio of vehicles, and I think that they’re also extending it to Mind Robotics potentially, which is, hey, can we also have humanoids that help make our vehicles someday? I thought it was just an interesting example of needing certain amounts of compute, a certain amount of interconnect bandwidth to send all this information around, and yet still having particular power constraints. Now, of course, at the end of the day, they’re connected to a big battery in your car. They don’t want to drain your battery, but they have more power.

Whether it’s humanoids or, what’s interesting is what Meta’s doing with those Orion glasses. Those were going to be crazy expensive, of course for some of the optics reasons, but still at the end of the day, it’s battery, compute. And if you think about it, most of the edge compute that we’re using today was probably designed two or three or four or five years ago. What was the AI that they were designing for at the time? It was convolutional neural networks and more traditional vision AI and ML.

So I actually don’t know if anyone has designed edge compute around end-to-end transformers yet, other than, again, maybe Rivian, maybe Tesla. That would probably be a big opportunity. If you started from first principles and you said, I want to run a transformer end-to-end at the edge, what is the right compute to support that?

Where Startups Can Get a Foothold

Brian: What are you most excited about from a startup or early-stage company creation standpoint?

Austin: I always love hearing about the technology, but what I’m most interested in is, where can a startup actually gain a foothold against a very large incumbent? In networking, what always seems to happen over the last 20 years is a startup, it’s usually people with a lot of experience, spin out of a company and then they go start something and then they get acquired by Cisco or Marvell or Broadcom. And it’s amazing, it works. If you want to start a startup and get acquired, go to networking. But it’s a little bit less interesting, because you’re like, oh, I know the playbook here.

What’s more interesting is the people who are playing in the space of ASML or foundry, like we talked about, where everyone on the internet’s just like, oh, you can never do that. That’s the dumbest thing I’ve ever heard of. How could you ever compete with ASML? There’s some startups, Substrate, xLight, where they’re saying, hey, what if we use X-ray lithography, or what if we use electron beams? What if we just think really crazy from first principles? What could that unlock?

I’m excited, from a strategy and entrepreneur perspective, to see people with chutzpah who say, no, no, no, we actually think there’s a way to pull this off. And I just like to watch it. One, I want to understand their crazy. Tell me more. You think this is possible. Why is it possible? I want to learn from you. And then two, I just want to see how it plays out.

Brian: One of my passions as a VC is thinking about people and what they want to do and where they’re going, and then the questions associated, like, how do you get there? One thing that leads me to wonder about for you is, what is the question that you think people are asking that maybe they don’t even know how to fully frame, that comes up again and again, maybe in the context of consulting or maybe in the context of people who are engaging with your content that are hungry to come to you? What are people really trying to find out?

Gaby: Can I give a guess and then you can tell us if it’s right? My guess is that people want to know how to trade in the stock market and are trying to get some intel.

Austin: Yeah, totally. Ultimately that’s what it boils down to, and I was trying to abstract it in my head. The simplest thing people ask is essentially, is there an interesting insight that you have that I could profit off of that is not obvious to other people? If I abstract that, even when I work with tech companies, I think what the executives want to know is, what am I not taking seriously enough that I should be? It could be a new idea.

One of the things that I put out into the world before a lot of people saw it coming was, hey, I do think CPU demand is going to shoot up. I didn’t see it as early as I should have, but once you start to think it through, you’re like, oh, wait, I’m just going to have a bunch of interns doing my bidding. What are they going to do? They’re going to use Microsoft Excel or whatever, Google Drive, whatever. And then I had tech executives say, oh, we read your piece and we hadn’t really thought this through, but this could impact our supply chain decisions if we’re going to need more CPUs than we realized.

So abstracted a little bit, it’s, because I’m afforded the time and I have the experience, how can I traverse the frontier of knowledge and just think through interesting things and find one that has an implication that’s going to really matter? Could matter for institutional investors, could matter for tech executives.

And actually, I was in a PhD program. I just stopped after my master’s. But when I was doing research, it was actually very, very similar. I was doing research in graphene and carbon nanotubes, and what you would do is you would survey the landscape. I’d work with my professor and he’d be like, go read all these papers as a new student and then see what people are doing, and what haven’t they done that should be done?

It’s that same muscle. What is Nvidia doing? What is AMD doing? What are the problems that startups are working on? Because that can tip you off into problems that the big companies aren’t solving. And then, what’s an interesting space that people just haven’t really thought through much? Let me quick go think about it. Are there any interesting implications? And then I could publish it or I could sell it to clients or what have you.

That’s what also gets me up, because you don’t know what that’s going to be tomorrow. You’re like, well, I did this yesterday and I came up with something interesting, but what about next week? I should go put pen to paper and come up with, what are my theories, and try to name them and come back to them.

But for me, it does usually end up at, okay, this is the way the industry works. There’s various incentives for two or three players to exist. Yet there’s always technological revolutions, even on a cadence. If you look at optics and networking, it’s 400G, then it’s 800G, then it’s 1.6T. So there’s the status quo, and then there’s inflection points that are going to happen.

So there’s questions of, can incumbents take advantage of those inflection points, or could startups come in and take advantage? I guess some of my rules of thumb are, just because you’re the incumbent doesn’t mean you will continue to be the incumbent. And just because you’re a startup doesn’t mean you can’t compete here. Usually you just have to be clever about routes to market and whatnot.

Even with transistors, there’s a company, imec, that’s over in Belgium, and they research this stuff and they publish a roadmap out to 2040. So it’s not crazy for a grad student to just be like, well, let me go think about some of that 2040 stuff. Is there a company to build there?

AI-Native Chip Design

Gaby: It’s fascinating, because as they’re publishing roadmaps till 2040, we’re also living in this world of just accelerated, compressed development times, research periods, all of this. What Dario calls the compressed 21st century, where all of research, whether it be across pharma, semis, whatever, is just going to be compressed. I imagine you’re an avid user, and it sounds like your son is an avid user of AI. I’m curious how you’re seeing the strongest teams actually applying AI to this industry.

We’re investors in a company that’s doing AI for materials science with a focus on compute. We see that as a really valuable way to be identifying areas where there’s new opportunities for materials to drive innovation. But obviously chip design has been a big area, verification, manufacturing. Would love to hear about where you’re actually seeing AI applied to this industry from a software perspective.

Austin: I think this is going to be the Harvard case studies of our time, when we look at this period where you’ve got companies that weren’t built AI-native, and then there’s going to be startups that are built AI-native. So even AI-native chip design, let’s talk it through. Right now there’s three big EDA companies, Cadence, Synopsys, Siemens.

They have all the customers. They have been qualified with TSMC with their PDKs, process design kits. So there’s all of this momentum, institutional knowledge, and everything that’s built up. On the other hand, you’ve got startups who have the freedom to think from first principles and build AI-native chip design. So the question is going to be, who’s going to win?

Is it the big guys, who have actually solved a lot of really hard non-technical problems, but have to overcome their old way of thinking and figure out, do they bolt AI into EDA? Or the startups, who are going to build EDA for AI from the start, but then they have to go convince TSMC to work with them and they have to go convince customers to use them?

I think that’s what’s going to be fascinating to watch play out. As far as companies using AI, I’m surprised when I go talk to even some of my friends in Iowa. They’ll be working at power companies, traditional power companies. And even they are saying, Austin, all we talk about is AI, and are we doing enough and are we thinking enough? And hey, we’re doing on-premises. We’re buying huge racks of GPUs.

It’s totally new to us, but we have to either sink or swim, and we’ve got to learn how to swim quickly.

Gaby: Yeah, it’s interesting. On the AI EDA side, we saw, not Chipstrat, we saw ChipStack get acquired by Cadence. And I think a lot of people are interested to see how Synopsys is going to evolve. But like you said, a lot of these duopolies, or areas where there’s two to three major players, whether it be in EDA or other areas, are thinking about, how can we integrate AI into our stack?

And a lot of the question, like you mentioned, is going to be, do we bolt it on or does someone new come in entirely? Part of the question is going to be how relationship-driven and vendor-connected these industries are. Beyond EDA, and the power thing you mentioned around everybody thinking about AI at the forefront, are there other areas in the semiconductor world or compute landscape more broadly where you feel like AI hasn’t been applied enough? One area I’ve been learning a little bit more about has been the defect detection world.

If you think about defect detection, maybe a company like KLA or Lam Research is building technology that can use computer vision and really powerful AI models to detect whether a chip has been produced correctly. That’s one area I’m spending time in and learning more about, but I’d be curious if there’s areas you feel like are underserved.

Austin: Interesting. Yeah, that one’s cool. AMD just bought a company, I think they’re called MEXT, and they were doing something really cool, which is applying AI, it is more traditional AI, but to memory. Basically trying to predict cache hits and cache misses and trying to predict access patterns. I thought that was a pretty cool use case. Even something as simple as what you study in computer science 201 or something, when do you write something to a cache, and when there’s a cache miss, there’s a write-through, some of this very simple stuff.

It’s probably actually still some very simple heuristics or algorithms. What if you applied AI to that? So I do think there’s probably an opportunity to literally go up and down the stack, at the operating system level, down to how we access memory, and there’s probably a lot at the networking and data sending layer. Could you predict how you send information even more intelligently than we do?

I think there’s a lot of interesting opportunity to explore where to apply AI next.

What’s Next at Chipstrat

Gaby: Perfect. So Austin, tell us about what’s on the docket. What are you working on? What’s coming out soon in the Chipstrat world?

Austin: What I’ve been starting to do lately is zoom out from individual companies and look at the markets as a whole, because there’s a lot of new dynamics happening because of the huge demand-supply imbalance. So memory, zooming out and saying, oh, there’s all these LTAs, these long-term agreements. How should we think about the dynamics of memory markets, which were traditionally more of a commodity, when they have an LTA in place? They’ve got agreed commitments from people to buy capacity for five years.

How does that impact pricing? How does it impact whether they should spin up more capacity or not? There’s a lot of interesting game theory things where it’s like, oh, the rules just changed a little bit here. Trying to look at that across optics and networking. And then of course, I think foundry is super interesting again. For all intents and purposes, TSMC was an earned monopolist, which means if they wanted to, they could charge whatever prices they want, but they hadn’t historically. Then they started leaning into selling their value.

But now all of a sudden, as Intel Foundry and Samsung Foundry come into play, what does that mean for how quickly they raise their prices, and also how quickly they expand capacity or don’t? Because expanding capacity is super expensive and risky for someone like TSMC. Again, there’s these dynamics I’m trying to dive into, which is, how are they calculating how much they could spend and how much capacity they are leaving on the table?

And this is also so interesting because if you don’t build it today, that impacts you in four years. And if you don’t build it today, you know that it’s going to go to Intel or Samsung potentially. So are you lifting them up? Or is TSMC believing, no, we don’t think they’ll be able to fulfill it? There’s a lot of gamesmanship. It’s fun to look across memory, look across foundry, look across all these markets and figure out, how are people thinking about this world where demand is so crazy?

And then we still don’t know. Companies are ramping up capacity for 2028, 2029. And even just predicting, is the rate of growth going to slow or is it going to stay? Because of course we see free cash flows going towards zero and there’s all these financing questions. So I’m just trying to also project a couple years and think three, four years down the road.

Gaby: Awesome. Very excited to be reading whatever market you’re digging into next. Brian, what are your rapid fire questions for today?

Rapid Fire

Brian: So this is a new experiment for us on The Compute 100 podcast. Gaby, we’ll ping pong. And Austin, you just need to answer quickly. All right. You have to put your entire net worth into one company in the semiconductor industry. Which one would it be?

Austin: Oh man. TSMC.

Gaby: What pending IPO are you most excited for?

Austin: Anthropic.

Brian: If one marquee name in the AI revolution, public or private, were to take a true nosedive within the next six months, which would you guess it would be and why?

Austin: Off the cuff, my brain goes straight to OpenAI. They’re awesome, but it’s just their path dependency of how they got to where they were, essentially at the core being researchers and then thinking, oh yeah, we will go to consumer and we’ll make a bunch of money. And then it’s like, dude, people only want to pay $20 a month. Meanwhile, Anthropic’s like, I know where it’s at. It’s charging enterprises hundreds of thousands of dollars for tokens.

Now, of course, to Sam Altman’s credit and the OpenAI team, they have done great with Codex and they’re chasing that, so I think actually they’re fine. But if they fell off, it could be them. Obviously there’s distractions around hardware and lots of other vectors it could be tempting to go into. So if they’re trying to prevent a nosedive, I would just say, dude, focus on enterprises and big customers.

Gaby: Who is your favorite CEO on earnings calls?

Austin: Favorite CEO on earnings calls. Oh, Hock Tan. Hock Tan from Broadcom. I love him because he is very wise, very experienced, very old, also has a lot of power in the industry, and sometimes he doesn’t really give a you-know-what. And I love how that comes through.

Brian: Shoot, I had one for you. That was a great one, Gaby. Who would you like to hear us interview on a podcast who is not someone you typically hear?

Austin: Interview Richard Ho from OpenAI. He’s their head of hardware. Of course, they’re building their own hardware, which we’ve heard about recently with the Jalapeño chip and Broadcom stuff. But they’re also building the AI that helps them to build the hardware. So what a cool perspective that he has, building the AI, building the hardware. Amazing.

Gaby: If you weren’t doing this, what would you be doing?

Austin: Oh man. If I wasn’t doing this, I would be trying to create the best lifestyle I could, and using my brain. How I make career decisions is just, am I going to get to use my brain more or less? Of course, you’re balancing the trade-offs there too. When I was in grad school, you have total intellectual freedom and you also don’t make any money. So at some point you’re like, well, I have to feed kids.

Gaby: The thing that is exciting to your brain is also the biggest market ever, so you’ve coincided in a good way.

Brian: All right, let’s keep going. Again, I had one, but this is fun. Gaby, your questions are good and I want to get Austin’s answers.

Gaby: What is your favorite book in the semiconductor world?

Austin: Focus: The ASML Way is really good. It’s about ASML. I thought that one was cool. I thought Tae Kim’s book on Nvidia was good too when it came out. And of course everyone’s going to say Chip War, Chris Miller. That’s a great one too. There’s actually lots of good ones out there.

Brian: All right, maybe one more. What’s a question for us that you’d be curious to hear about before we wrap up?

Austin: What are you guys most excited about, looking three or four years out, not three months out, into the future about compute?

Brian: That reminds me of one of my questions for you, which was actually, quantum relevant within the next five years, yes or no?

Austin: Quantum companies, please convince me yes, but I say no.

Brian: Cool, great. Most excited. Gaby and I spend a lot of time thinking about the rising generation of entrepreneurs and operators who are AI-native and want to build the future of compute for AI. Watching them do seemingly impossible things, and transforming the industry along the lines of what we’ve been talking about, is a thing of constant fascination.

Austin: Yes, I love that. It’s like, dude, AI is magic. It’s like Harry Potter before he went to Hogwarts and after he went to Hogwarts. Think about these kids that just have this magic wand. It’s amazing.

Gaby: It’s crazy. It is a different world spending time on college campuses. I feel like they are learning in their freshman year what I learned in all of college. And that’s not actually a curriculum thing. That is just the speed at which they can keep up with the world around them and get exposure to things, and a lot of that is obviously because of AI. It’s just so fascinating. So the new generation of talent here is just really exciting.

Austin: Love it. That’s exciting.

Gaby: Amazing. Well, Austin, thank you so much for joining us. We really appreciate the conversation, and thank you for coming on the podcast.

As Stephen Nellis reported for Reuters:

Synopsys, a maker of software used in designing computing chips, said on Wednesday that it has struck a deal to share revenue with OpenAI as the two develop an AI model for the chip business.

GPT-Synopsys, as the model will be called, will be specifically trained and tuned to take on chip design tasks, which start with a description of the chip’s circuits in a code-like language and span to determining how to lay out billions of transistors on a small square of silicon...

Ghazi told Reuters in an interview that OpenAI will pay Synopsys a training subscription fee for learning to use Synopsys tools. When Synopsys customers use the product, Synopsys and OpenAI will share revenue based on how well the model improves the design of a chip.

“We structured the agreement in a way that it will not be cannibalizing our business,” Ghazi said. “It will be an upside to our business given we’re delivering more value to the customer.”

Hmmm... is EDA not dead? I could have sworn everyone was saying the model labs were going to eat EDA’s lunch, because EDA is “just software” and OpenAI’s Jalapeño showed the death of EDA? Right?

Enough FUD, let’s work through this ourselves.

First, let’s discuss how Synopsys positions agentic AI with respect to Synopsys’ EDA tools. Here’s the portfolio slide from Investor Day:

Importantly, the foundation is the tool layer, i.e. the existing EDA tools like VCS/Verdi, RTL-to-GDS, SPICE, Mask/TCAD, and Ansys physics simulation tools on the S&A side.

One layer above this are task agents, i.e. narrow jobs like Coverage Closure agent, PPA Closure agent, Analog Design agent, and so on.

Atop this are long-horizon agents which take an objective, decompose it, and orchestrate task agents + tools.

“A verification agent engineer is able to take an outcome or an objective, like here is a spec 200, 300, 500 pages long, and give me a verified RTL and test benches compared to the spec. Or here is a design that I’m trying to improve the coverage on, and I have about 30%, 40% coverage. Take this and make this into a 90% coverage situation.”

Shankar Krishnamoorthy, Chief Product Development Officer

As an analogy, if you use Claude Code or Codex for software vibe-coding, the tool layer is using git for version control or curl to hit an API; these are the lower level tools.

One layer up are skills that package the instructions and scripts for a particular job, like a skill to “bump this dependency and fix whatever breaks” or “write a database migration”. The skill knows which commands to run and how to check the output.

And at the top is the long-running agent that takes an outcome like “upgrade this app to the new version of its web framework”, and it plans the steps, calls the skills and tools, runs the test suite, fixes what fails, and keeps going until the tests pass.

Synopsys’ stack is the hardware equivalent concept.

Of course there are differences… including the business model. git and curl are free, whereas Synopsys’ tools are licensed. Are there incentives to create free open-source tool alternatives in that foundation layer? Hold that thought.

Synopsys pointed out that agents enable parallel exploration. Why not have several task agents each using a tool to explore the state space in parallel? Better outcomes for the customer (explore more solutions), and more tool use (license revenue) for Synopsys:

“The total work that can be accomplished is gated by the amount of human cycles that are available... you’re only gated by the compute on the right-hand side. The more compute you have, the more exploration, the more agents that can run in parallel”

Shankar Krishnamoorthy

That’s pretty awesome. Solution space exploration is definitely limited by the number of engineers on a team, which can’t easily scale up; but agents can!

But wait... what about OpenAI and Jalapeño using AI? Recall that OpenAI says AI helped take Jalapeño, its first custom inference chip, “from initial design to tapeout in nine months”… was that replacing Synopsys? Or was OpenAI scaling up it’s use of Synopsys?

At Investor Day, Synopsys CEO Sassine Ghazi pointed out Jalapeño still ran through Synopsys EDA, then Broadcom as back-end ASIC partner:

“when Jalapeño was announced, there was this simplistic extrapolation that if a model can build software, therefore the model can build a Fusion Compiler or a PrimeSim or VCS, and EDA is doomed... That simplistic extrapolation cannot be more far off than reality” — Sassine Ghazi

“The model needs these guardrails in order to check the physics”
— Sassine Ghazi, to Reuters

Ah ok, so OpenAI was running AI during front-end design, and mostly at the task/agent layer, not the tool layer and seemingly not during back-end design (Broadcom).

But could the tool layer be replaced by AI too? If not, why not? Hold that thought too, we’ll get there.

OK then, EDA vendors are the tool layer.

But how much of that agentic exploration layer value will accrue to EDA vendors vs the model labs?

The bear case is model labs will own the agent layers circled above, and EDA vendors are limited to the tool layer.

Synopsys acknowledged these agent layers are something customers are exploring owning themselves, but suggested seen three different paths forward with customers so far:

1) Use Synopsys agentic layers

2) Roll your own

3) Use a frontier lab offering

Digging into each:

1) Synopsys full stack

The customer takes everything from Synopsys. The value prop of this turnkey offering is speed-to-market for the customer, and trust that Synopsys is the domain expert here and can unlock token efficiency that others can’t:

“We have access to the guts of the tool. We have deep API that they’re not available in lane two or three.” — Sassine Ghazi

Obviously, this provides the most revenue opportunities for Synopsys:

2) Roll your own (customer-owned platform + agents)

Synopys said most early adopters are exploring this lane today; their agents still use Synopsys tool licenses, but they want to own the agentic layer.

“I have my own special sauce. I want to build my own agent... I do not want to reinvent a debug agent.” —Sassine Ghazi, describing lane-two customers

I’m not surprised most customers would land here, as customers retain optionality (can explore using different models, innovate or optimize the harness, etc), and can limit dependence on Synopsys (and reduce spend with Synopsys). Granted, you’re paying your own engineer salaries for development and maintenance instead. No free lunch.

OK, so now the surprising announcement, which I was there for :)

Paid subscribers get my thoughts on

• GPT-Synopsys

• Labs owning the agentic layer. Bear case? Or not?

• Outcome-based pricing

• Can AI replace the tool layer?

• Retraining

• Open-source EDA

• AI simulation shortcuts

• OpenAI’s commitment

Is EDA dead?

Let’s get into it.

Read more

But actually using LLMs within software.. well, that still sucks.

Which is too bad, because LLMs are great at making sense of unstructured data. They can make judgment calls like “read this customer complaint and figure out what to do about it.”

That use case is so freaking valuable. Just ask Sierra, with its $15B valuation and long list of well-known customers.

Why do LLM calls embedded within deterministic software suck? It’s not just that it’s slow and expensive. which is true...

The big concern is the type signature. An LLM call is a function typed String -> String. If you’ve ever worked with a statically typed language, this is like the least informative signature in software. For example, if the LLM is expected to return a number, you’d rather the output signature be Integer.

When the function just says “I’ll take anything and will return anything”… lol. It’s very difficult for the caller to reason about it, so they end up having to write a runtime validator to parse the output and check the schema.

Sure, you can use Structured Outputs, which makes sure model-generated outputs match JSON schemas provided by the developer. That’s an improvement. Before, asking “Is this customer asking for a refund? Reply yes or no” to the LLM might give back “Yes, it looks like they want a refund, although...” And your code then has to parse that string and hope to extract the correct “yes”.

But with structured outputs, you send a schema like

{ “refund_requested”: “boolean”, “category”: “billing | technical | account” }

and the reply is guaranteed to be valid JSON in that shape, e.g.

{”refund_requested”: true, “category”: “billing”}

This is obviously much better for the developer.

Yet under the hood it works through constrained decoding, which is still slow. The model still generates the answer token by token, and at each step the server blocks any token that would break the schema. So if the next valid character has to be t or f for a boolean, every other token gets zeroed out. This works, but it’s still slow and expensive...

There’s also no confidence signal. The input might be a long, rambling paragraph from a customer. How confident is the LLM that the customer actually asked for a refund? Shouldn’t the developer act differently if the confidence is 95% versus 52%?

Yes, you can ask the LLM to include a confidence field in its response, but that number is just another generated token.

Then there’s hallucination. What if the answer that comes back was made up? It looks fine to the engineer, and it may even parse cleanly. But it’s not real! A trustworthy confidence level would help here too.

So what do we need to call an LLM over and over, deep in the bowels of a program, with confidence? A fixed set of possible answers, a guaranteed response time, and an honest confidence attached to every answer.

In other words, intelligence with a function signature.

With Jev from Typesafe AI, we get just that. A frontier-intelligence function call. Unstructured state in, typed probabilistic decisions out. AND FAST:

Same 27 questions, one request each, started at the same time. Jev returns all 27 typed answers with probabilities in 0.114 seconds for $0.000081. GPT-5.6 Terra is still generating its answer token by token. Source: TypeSafe.

With Jev, you send the state and a set of questions, each with a type. A Noul is a yes/no question, and it returns the probability the answer is yes. A Choice picks one option from a list you define. A Score places the input on a scale you define, like calm to very angry:a

respon se = client.system_one(
    state={
        "ticket": "My flight was cancelled. Can I get a refund?",
        "refund_policy": "Cancelled flights are eligible for a full refund.",
    },
    questions={
        "refund_requested": Noul(instructions="Does the customer request a refund?"),
        "request_type": Choice(
            instructions="What is the main request?",
            criteria={"refund": "...", "rebooking": "...", "information": "..."},
        ),
        "frustration": Score(
            instructions="How frustrated does the customer appear?",
            criteria=["Calm and neutral", "Concerned but civil", "Very angry"],
        ),
    },
)

What comes back from the model is typed values. refund_requested comes back as a probability, say 0.97. request_type comes back as “refund”, along with a probability for every option and a confidence score. frustration comes back as a position on your scale, say 0.4.

Then your code decides what to do. You focus on the business logic:

a = response.answers

if a["refund_requested"].noul > 0.9 and a["request_type"].choice == "refund":
    issue_refund(ticket_id)

elif a["frustration"].score > 1.5:
    route_to_human(ticket_id)

Dude! So simple!

TypeSafe calls it a “smart if-statement”, and in their manifesto they say

“Computers can do so much by just branching on bits, imagine if they could also branch on common sense, understanding, and intent.”

Jev also can’t hallucinate an answer, because it can only answer with the options you gave it! Jev can still be wrong, of course. It can pick rebooking when the customer wanted a refund. But it comes with a confidence signal too, which should help clue you in.

TypeSafe trains Jev with a new post-training method it calls RLCD, reinforcement learning for calibrated decisions. RLHF trained models to write answers people prefer. RLVR trained models to get checkable answers right, like math. RLCD trains the model’s probabilities to be honest.

Calibrated means that when Jev says 0.8, it’s right about 80% of the time. That’s across many answers, not a promise about any one of them.

Calibrated probabilities are something you can program against.

And Jev is fast.

Why?

It answers every question in a single pass through the model. An LLM writes its answer one token at a time, and each token waits on the one before it. Jev has no token-by-token loop! That’s why output tokens are free btw.

What is Jev good for? Quick judgment calls. Classifying, routing, scoring, extracting, verifying, and ranking.

Just think of Jev as a tool for developers. Soo… for agents? :)

It’s meant to put fast and cheap intelligence into software.

And it has implications for AI infrastructure!

Let’s dig into some questions you might have:

• Decode is the reason AI accelerators have HBM. What does a model with no decode phase need instead?

• Is this a slice of today’s LLM inference market, or a new one?

• What happens to fast decode solutions like Groq and Cerebras?

• Who wins?

• Can a humanoid or a car run frontier-class judgment on an edge SoC?

• What about a phone?

Let’s get into it.

Read more

We debated things like:

What if you could get 10x more tokens, at a fixed interactivity, for a particular model, out of your megawatts than you could buying something off the shelf?

It was a great conversation; here’s the video and transcript.

Highly recommend reading / listening / AI’ing.

This transcript is lightly edited for clarity.

Why AI sells as systems

David Goldman: Austin, welcome to the show.

Austin Lyons: Thank you for having me.

David Goldman: AI infrastructure is in the middle of one of the biggest capital buildouts in history, but under the surface, even though the numbers are going up, there’s been a pretty big change as the dollar spend on inference has gotten bigger than the spending on training. This is a trend that you’ve covered a lot in your writing and something you have a very unique perspective on. So I want to spend a little bit of time unpacking the implications of this shift.

Let’s start at the top with how these systems get sold and what system selling is. Historically, when cloud buyers went out to the semiconductor industry, they were trying to commoditize server hardware as much as possible. They would work with industry groups, they would try to get standard designs done so that they could drive down pricing and negotiate with each part of that design. But increasingly over the last few years, they’ve been buying whole systems. So maybe let’s start at the top and just say what goes in a system, and why is this trend happening?

Austin Lyons: Yes. I like the idea of starting with systems when we’re talking about AI, because that’s really what it’s all about these days. It’s rack scale, all the way out to full data center clusters. The compute at the heart of these systems is ultimately GPUs, and that’s what gets the airtime — or AI accelerators — but it’s actually a full system.

So stepping back, the question is how did we get here, and why are people buying a full system from one vendor and not commoditizing it? The workload that we all care about today is LLM inference. If you’re running at the frontier and you’re wanting to use the latest and greatest OpenAI model or Anthropic model, these are models that are two trillion or more parameters. What that means is, even as you quantize these and try to run them without using as much memory as possible, you might still need something like a terabyte, or two terabytes, or several terabytes of memory just for the model. Because with these models, as we saw with the scaling laws over late 2022, 2023, 2024 — the bigger the models and the more compute you have, the better answers you get.

Way back in the day, we might have taken an AI model and put it on just one GPU. But if you think about today’s frontier models, you need maybe terabytes of memory, and yet a single GPU can only have, for example, 288 gigabytes of HBM on it to store these model weights. So quickly you say, wait a minute, one model can’t fit in a single GPU’s memory. So what do you have to do? You need several GPUs. You might say, okay, take that model and split it across four GPUs or eight GPUs. And that’s how we started to work our way into “this isn’t just a single chip anymore, this is a whole system.” That’s how we got to, for example, the Grace Blackwell NVL72 rack — a rack with 72 GPUs in it.

We’ve gotten to a place where we’re talking about AI systems even just for inference. Even if you were only to have one rack, you’ve got GPUs, CPUs, networking, power, and cooling. So back to the question of why the industry isn’t commoditizing it: a big piece of where we’re at today is that Nvidia did a really good job of getting to this rack scale first. They said, hey, we can help design the whole system and make sure it works across 72 GPUs and 36 CPUs, and that they can all talk to each other. And by the way, it also takes software and compilers so that you, the model developer, can write your model and actually run it across all this. So even though the industry always wants commoditization, competition, and multiple suppliers, Nvidia did a good job of frankly getting there first and making this complicated system turnkey, so that for the software developer it’s as easy as writing their model, deploying it, and it “ just works”.

David Goldman: So you need a very big, complex system to make these LLMs work in a cloud. But if I’m a cloud buyer — Nvidia has famously high margins, and they charge that on all of the different parts of the system, not just on the GPU. If you go out in the Valley, there are all sorts of companies offering one piece of this puzzle. If I’m a cloud buyer, why would I be willing to give so much margin to Nvidia instead of trying to piece each piece together — buy my GPU from Nvidia maybe, my networking from someone else, my software from someone else, my cooling from someone else? What kind of value do you get from getting it all at once? Is it around speed? Is it around simplicity? What are the trade-offs that people think about?

Austin Lyons: Yeah. We are not yet in the era where people are trying to squeeze down their costs as much. We’ve still been in the era of speed to market. We saw that with Elon Musk and xAI, where he’s been able to stand up data centers very quickly using Nvidia systems. I don’t think Elon’s goal was to say “how can we do this as cheaply as possible,” but more “how can we stand up this compute and get productive tokens out of it as quickly as possible?” So even though a company can potentially pull the components from many different vendors, there’s just a desire to buy it and stand it up quickly.

And to your other point, there’s definitely simplicity there. We see even AMD, with their Helios rack that they’re bringing to market, seeing that same pull from customers: I want to be able to buy the full rack, get it, stand it up as quickly as possible, and get software running as quickly as possible.

David Goldman: Each of Nvidia and AMD have had decades to put together this puzzle, and tons of financial firepower to do M&A. Nvidia acquired Mellanox; AMD has done a bunch of acquisitions to build out these system-selling strategies. Is it possible for a startup to compete now that you have to sell systems?

Austin Lyons: That is a tough and very interesting question. To your point, if you’re a startup now, you can’t necessarily just come in and say “I’m going to build a better GPU, a better AI ASIC.” People are going to say, okay, great — now what do I do with that? Are you going to make your end customers piece together the whole system? We just said they want to stand it up and deploy it as quickly as possible. Or are you, as the startup, now going to have to take on building the rest of the system?

I think there are probably different approaches here. And by the way, this is why we see that it takes hundreds of millions of dollars now for a chip startup, when maybe back in the day you used to do several rounds of just a couple million dollars to prove out your little proof of concept. So can a startup compete here? Yes, but they have to be very smart. It could also be death by a thousand paper cuts if you’re trying to invent a new AI accelerator and you’re also trying to invent your own proprietary networking and maybe your own proprietary cooling.

Actually, for example, Cerebras took a very interesting approach. They’re an AI accelerator startup — they’ve gone public — and they make what they call wafer-scale engines. For those listeners who aren’t as familiar: instead of taking a whole wafer of chips and dicing them up into individual chips, like two GPUs and a CPU that you package, they said, why don’t we just leave it at the wafer scale and make the whole wafer our compute engine? But then they had to invent how you make them all communicate, how you cool all of that, how you deliver power to all of that. They basically had to reinvent all of it, and that takes a lot of time. So I think any other AI ASIC company that followed could look at an example like Cerebras and say: wow, they did very innovative technical things here, but they had to invent everything, and that’s time-consuming and expensive. How should we do this differently?

David Goldman: Yeah. I think you see some parallels with the buildout of internet infrastructure in the ‘90s and early 2000s. You start with proprietary, then as open standards proliferate, there’s more capability for more people to offer things, and you can get a little bit more mix and match. I know you’ve seen SambaNova, for example, has partnered with Intel to offer x86 CPUs alongside their system. So there seem to be some trends, even though incumbents clearly have an advantage in system selling.

Prefill, decode, and the split inference workload

David Goldman: One of those trends, which I think you’ve touched on a lot in your writing: as we move into this inference era, you’re starting to see the inference workload itself get split, into what’s called prefill and decode. Can you tell us what those two things are, and why you might want to split that workload onto different types of silicon?

Austin Lyons: Sure. At the end of the day, if you just think of simply talking to an AI chatbot — you write a paragraph of explanation, “hey, I want you to go research this and give me an answer” — the first thing the AI system, the model under the hood, needs to do is read through the whole prompt that you gave it. That’s the prefill stage.

What’s happening is, in parallel — one of the key innovations of the transformer model is this idea of attention — you work through everything in the long English paragraph, and you ask which of these words are connected to other words in the sentence. That way you can piece together the context of the whole paragraph and what each word is referring to. And you can do all of this in parallel. If there are a hundred words, you can look at all hundred words in parallel and compare them to the other hundred words and do these calculations to figure out whether they’re related or not. That’s prefill. You’re ultimately doing a bunch of linear algebra, a bunch of matrix multiplication.

Now, when it’s time to give you an answer — especially if you remember back to when ChatGPT first launched and was a lot slower — you’d see the answer come out word by word. That’s because every word that I’m about to say depends on the words that I just said. And when I say that word, the next word also depends on those other words. This is decode. This is where you’re predicting the next tokens, or the next words if we’re talking about English. And that is sequential. You can’t do it in parallel.

People running this inference at scale started to realize: hey, wait a minute. In prefill, I’ve got all these GPUs doing all of this parallel computation, and it actually doesn’t require a huge amount of memory. I’ve paid for this high-bandwidth memory — which is very expensive, and getting more expensive by the day — and in prefill, that HBM is just sitting there being underutilized. Interesting. Okay, now I’ve got these other GPUs that are running decode, and they’re actually not fully utilizing all of their FLOPS, all their units of compute, because decode is what’s called memory-bound. It’s a lot of “I made a prediction, and I need to get the weights and whatever I need from the KV cache, then do a little calculation, then go back and forth and back and forth,” one at a time. You’re not doing things in parallel; you’re really just waiting on memory.

So at the highest level — even Nvidia led the way with this — they said: we’ve got a bunch of GPUs in the prefill phase being heavily utilized for compute with their memory underutilized, and a bunch of GPUs in the decode phase basically underutilizing their compute and totally utilizing their memory. Naturally, any engineer is going to look at that and say: huh, maybe we should disaggregate these. Maybe prefill should run on systems that have lots of compute but don’t necessarily need all that memory. And maybe on the decode side, we should really emphasize memory bandwidth — how quickly information can get shuttled around — and maybe it doesn’t even need quite as much compute; we should essentially overindex on the memory part.

So naturally, we started to move into a world where Nvidia shipped a software layer for their systems called Dynamo, which helps orchestrate this across Nvidia GPUs. And that actually gave rise to the Groqs and the Cerebrases — these AI ASIC startups that were actually started even before transformer-based LLMs were the defining workload of our era. They had made architectural choices where they used a lot of this really fast on-chip memory called SRAM — where you use transistors to store the memory, instead of DRAM, which is capacitors and transistors, which is what HBM is made of. We won’t go way down into those memory details, but basically these early startups had made a bet on having really high memory bandwidth. And once the workload got separated into prefill and decode, they could raise their hand and go: oh wait, we’re actually really good at decode. In fact, we can go even faster than GPUs.

This took us from the inference era where everything was on GPUs to saying: what if you could slot in one of these AI ASICs, heavily built on SRAM, that can maybe unlock a thousand tokens a second? Whereas a GPU running decode — just the way GPUs are more general-purpose in their design and their memory hierarchy decisions — maybe could only run at a fraction of that.

David Goldman: Is that something you should always do? Do we always need to split up prefill and decode? Or are there just certain applications where it’s really good to have speed — I’m willing to pay a premium for speed, therefore I’m willing to go through the hassle of splitting these things up, having orchestration software, having different types of silicon and all the things associated with that — versus just running it on the GPUs or the system I’ve already bought?

Austin Lyons: Yeah, there are so many different nuances here. Take the hyperscalers deploying tens of billions of dollars of GPUs. Early on, GPUs are very general-purpose, very flexible — they give you the freedom to change your workloads. But let’s say you’re OpenAI and you say: no, these are our specific models that we know we’re iterating on — the frontier one, the medium-sized one, the small one. You start to say, hey, we should really cater to this workload’s needs, and therefore it would make sense to deal with the complexities you pointed out — splitting up prefill and decode and orchestrating that. It might be worth it. Maybe that unlocks, for example, being able to sell an ultra-premium tier where you get really, really fast inference, and maybe there’s a small subset of users who would pay 10x more to get tokens that are 5x faster. So I think there are certain model labs and hyperscalers at scale that are saying the complexity is totally worth it — we’re willing to deal with different SKUs, different chips.

On the other hand, I think there are going to be tons of enterprises, and arguably the long tail of consumers, who of course want inference, and they’re thinking about cost and speed, but they aren’t going to want to manage all of that complexity. For them, where we’re at today, they’re not going to need it. Of course, everyone always wants faster inference — so if there’s a way for them to get that outcome without dealing with all the complexity, they’re going to want it.

When I’m trying to think about this space and ask whether it’s one-size-fits-all GPU or many different chips — where is this going to go in the end — I look to CPUs. When you look at any CPU vendor, or any cloud like Google Cloud, they don’t just deploy one CPU, even for customers who want to rent them from their cloud. They have a portfolio: hey, this one has a lot of memory in case you’re running a database; oh, this one’s actually pretty vanilla and it’s nice and cheap if you’re just running an API server. There are still shapes — this family of chip, that family of chip. I do think there’s a world where we get to a couple of different shapes of AI accelerators.

Where we are now: we had this training era, then this inference era — those were all on GPUs. Then we’ve got this next era, which is a GPU plus a specific decode chip that’s made a good memory-hierarchy trade-off so you can have high interactivity, as they call it — really fast tokens. But where we are today, those are multi-vendor: Nvidia plus Groq — of course, Nvidia acquired Groq, so they’re trying to bring that all in-house — or someone else plus Cerebras — AMD plus Cerebras, or Trainium plus Cerebras, or whatever. My thought is that we will ultimately go to those SKUs living inside the same silicon vendor, because it starts to get complicated when your route to market depends on another company — I sell a GPU and they sell a decode-specific thing. But it is working right now. There’s definitely demand for it.

David Goldman: It feels like there’s this inherent tension in the market. You want to buy systems — that’s what customers are saying, they want someone to do the work of putting all of these things together. But at the same time, they also want the right silicon for the right job. They want specialized things for decode, and they want the right proportion of CPUs for the workload. So it’s not a one-size-fits-all system. Ultimately, do you think you get more and more fragmentation here? Or are customers going to say: “Nvidia, please solve this problem for me — buy Groq, buy the next company, buy the next company, keep selling me systems. AMD — buy Talis [sp?], work with Cerebras, figure it out for me.”

Because, to give you the counter side: there are separate companies for CPUs and GPUs. We’ve decided those are separate enough markets that they can have standalone companies. We can figure out the system with CPUs and GPUs from different vendors — that’s been a solved problem for a long time. So maybe that could be an end state, where you have decode silicon that is a completely different market, and we give it a catchy name like DPU or something. Well, not that one, because it’s already been used.

Austin Lyons: Right, totally. Zooming way out, when you look at the semiconductor industry in the long run, there always seem to be three or four vendors in a certain thing. Whether you look at wafer fab equipment, foundries — it’s complicated, but we’re getting back to maybe having two or three — CPU vendors, GPU vendors. We are of course in an era, as happens whenever there’s drastic innovation, where a ton of competitors have popped up. So I think it would not be crazy to zoom out and say: in the grand arc, maybe only three or four people will shake out, and therefore there’ll be some sort of consolidation.

But I personally think this isn’t as simple as “Nvidia buys [that startup], AMD buys them,” even though we’re seeing some of that — and of course there are regulatory things to talk about there. If you look up and down the stack, there’s actually a lot of competition.

For example, at the neocloud layer: a neocloud might be incentivized to say, hey, I can tell the customer wants the right silicon for the right job, but they also don’t want to deal with the complexity. I could buy a bunch of different silicon, deal with all the complexity myself, sell them tokens as a service, and try to give them the speed or the cost they’re looking for — and differentiate from other neoclouds that way.

So I do think there are still routes to market where startups can come in today and say: hey, I am the best decode solution, you should try me. Or maybe I’m a prefill solution. And figure out a way to get to customers while letting the end developer not deal with all that complexity. Then you keep pulling on that thread and play it forward: could an AI ASIC startup ever merge with a neocloud? I don’t know. Maybe it won’t just be Nvidia or AMD buying all of these companies.

The rise of the neoclouds

David Goldman: You touch on an interesting thing here, because I think neoclouds are really under-discussed when people talk about AI infrastructure. We tend to focus a lot on semiconductor companies and systems companies, and not as much on the cloud layer. And I sort of have a pet theory that it’s because most of the traditional VCs missed out on those as investments, and so we don’t like to spend too much time giving credit where we don’t get to claim any.

Austin Lyons: Yeah.

David Goldman: But the reality is that if you look at the handful of neoclouds that came up in this first wave, they’ve created a lot more equity value. The top three are worth something like $125 billion in public markets, which is a lot more than what we’ve been talking about with some of these chip startups. So why do you think investors had a lot of trouble understanding the first wave of neoclouds and missed out on those investments?

Austin Lyons: Yeah. For listeners, by “neocloud” we mean a cloud company that started by just renting GPUs. I’ll talk through my hesitation when I first saw the idea of neoclouds, and maybe I’m a fair proxy for some investors — maybe they thought this way too.

Let’s back up. Who’s renting from these neoclouds? Well, it turns out a lot of it is hyperscalers, which is ultimately driven by demand from the biggest model labs. So the question is: wait a minute, you’re saying that OpenAI is running on GPUs that Microsoft is renting from some neocloud? Doesn’t Microsoft just have their own data centers? And the answer is: of course they do, and they’re trying to build more, but at the end of the day we are limited by access to power. There are financial reasons to rent versus buy — maybe even go off balance sheet as the capex has increased year over year over year. So there are very legitimate reasons why even someone as sophisticated as Microsoft Azure might say: actually, I want to rent GPUs from someone, and I’m also building other data centers, but maybe it’s a stopgap.

Looking at that, I said: okay, so you’re going to have a neocloud company — maybe they have access to power, and maybe they have a good way to raise capital against assets like this. Bitcoin miners, for example, historically have had this experience, and maybe they’re shifting into GPUs. And so I thought: huh, okay. They were doing Bitcoin mining, now GPUs are hot, so they’re going to do GPUs, but they’re just going to rent them to Azure. But Azure is also standing up their own data centers. So how is this sustainable? They’ve got huge customer concentration — literally maybe one customer. But to their credit, that’s how you get the financing: you say, I’ve got Microsoft who’s going to rent these from me, and so people will lend against that and believe in that.

That’s part of why I missed it as these companies were popping up. There’s a legitimate need in the marketplace for people who have access to power, who can get financing, who can manage this, and who can move quickly — to stand up GPUs, run models on them, and rent it out as bare metal or maybe at a higher level of abstraction. And even though they have serious customer concentration risk, so does everyone else in the semiconductor industry right now. Who are Nvidia’s end customers? Even Nvidia, the biggest and best, is selling a lot of GPUs to a small set of customers.

David Goldman: Yeah, it’s not uncommon to see someone go public with 90% customer concentration these days.

Austin Lyons: Totally. That is the name of the game. Look at any component supplier — even in the interconnect space or the switching space. Look at Credo, who made these active electrical cables, a really awesome invention. Same thing: even when they went public, they were selling to a handful of customers. The name of the game for this era is watching how this unfolds and seeing how some of the people who win one big customer ultimately win a couple more, build out more of a portfolio, and reduce a little bit of that risk. But that’s just the way the industry works right now.

David Goldman: If you wind the clock back five years, before these neoclouds got big — AWS is an incredible business; Amazon and Microsoft have fortress balance sheets, great relationships with Nvidia and all the semiconductor companies, expertise in setting up data centers, software, customer relationships. It seems to me they could have done this, and certainly most people thought they would, which is why so many missed out on these investments. Was it a strategic decision, where they thought “maybe this isn’t going to be a big enough market, I’m not sure I want to spend all the money and take the risk”? Or was there some special sauce in what these neoclouds were able to do — setting up quicker, being more creative in financing, converting old Bitcoin data centers? Were they doing something different from what Microsoft might have done, or was Microsoft ceding market share to them, or some other factor? Not to pick on Microsoft — insert anyone here.

Austin Lyons: Maybe a little bit of everything. At the end of the day, this is risky business, because the investments might be $40 billion this year, $80 billion next year, a hundred billion the year after that. I don’t think it’s that the existing players didn’t believe the future we’re in now would manifest — I think it’s all about timing. And they’re probably not incentivized to just sprint out and stand up all these data centers. It’s a huge investment.

The neoclouds were able to move faster and take on more risk. They can kind of go for broke: all right, cool, let’s convert, let’s buy a bunch of GPUs, let’s get moving. It’s a bit like the innovator’s dilemma. There are already three big clouds, they already have all the customers, and I think they were probably looking at each other — AWS and Google and Azure — saying: are we all believing this future is coming and investing in GPUs at the same rate as each other? But that could still not be enough supply to meet demand.

Now, I do think demand went higher faster than everyone expected, especially once we got reasoning models, and now of course the agentic age. So even if they all looked around and said, “we think if we grow supply like this — it’s a little risky and these numbers feel really big, but we think the demand will be there” — demand skyrocketed past that, and it gave neoclouds an opportunity to come in and say: yeah, we’ll fill that gap.

David Goldman: I think there was also maybe a difference in interest level. You heard some of the hyperscalers make comments about how these bare-metal GPU instances are low-margin and sort of commodity, and therefore they didn’t want to support them as much. Whereas the neoclouds read them as revenue, and therefore good.

Austin Lyons: Yeah, you make a fair point. Anyone whose business was renting CPUs, or selling services on top of CPUs — the cost structure there is much better than GPUs. So you could see CFOs saying: wait a minute, we’re going to spend a ton of money and our margins are going to go down, even if our margin dollars go up. There are still conversations to be had that might make you slow down or hesitate a little bit.

David Goldman: So is this a durable state of affairs? Obviously this is a very fast-moving market, but “first mover,” “more willing to take risk,” “willing to take on lower margin” — those are not necessarily durable advantages that will last a decade or more. What do you see playing out with these neoclouds? Do they get acquired by hyperscalers? Do they consolidate into a neo-hyperscaler or something?

Austin Lyons: Yeah, it’s very interesting. I’m not sure there will always be a need for a hundred neoclouds, but I do think it’s real demand, and it will remain real. I don’t think the big three clouds will always meet everyone’s needs indefinitely. Not only that — there’s always going to be innovation. Take World Labs, Fei-Fei Li’s company. They’re coming out with world models. Who’s going to make the bet there? What if those don’t run well on the exact shape of hardware all these big clouds have invested in? I think there will always be new workloads or new demands popping into existence where a neocloud can pop up quickly and say: I can meet your need, and I can innovate there. That will always exist. There may not be enough cutting-edge frontier demand for a hundred neoclouds to hop on, but I definitely think there will always be a need for these more nimble, smaller GPU and AI ASIC rental companies that can innovate a lot closer to where the model labs and AI-enabled software companies are innovating.

Circular financing

David Goldman: I feel like we can’t talk about neoclouds without at least discussing circular financing risk. This is probably the thing I hear most from people who are skeptical about the durability of AI infrastructure. The argument goes: the neoclouds get equity investment from Nvidia in many cases. They use that equity to buy GPUs. They then use those GPUs as collateral to take on debt. A lot of that debt is sometimes backstopped by either their hyperscaler or by Nvidia itself. It all sort of perpetuates, and the revenue from the neocloud buying GPUs goes back to Nvidia, who invests in more neoclouds. People see a circular financing issue, like the vendor financing that happened in the dot-com bubble and was revealed to be an inflator of that bubble. On their last earnings call, Nvidia addressed this directly and said they see it differently. Do you agree with them, or do you think there are some concerns here?

Austin Lyons: You know, I was definitely the type of person where, right away, I said: okay, this is different, this feels funny, I need to dig in and understand it and try to understand both sides. I can definitely see why it looks like circular financing — call it what you will.

But going back to thinking through demand, supply, and the cost of capital: it is a fact that there’s just insatiable demand, especially now with agentic AI, where the barrier to entry for software development has gone as close to zero as possible. I’ve got a son who made a 70,000-line video game this summer, and he didn’t write those lines himself — he used Codex to do it. That’s amazing, it’s unreal. And just wait until he’s in high school and college and beyond. He’s going to use AI so much more intelligently than me. He’s going to use way more tokens than me. I can’t even believe what it’s going to be like in the future. You can look at every industry and see lots of people writing software, doing interesting things they couldn’t do before. So the demand is totally real.

The supply is very fixed. At the end of the day, you might look at someone like TSMC — there are only so many wafers that come out, only so much CoWoS capacity. But even as GPUs get built, the question is who has the capital to buy them. Today we might be talking $5–10 million a rack, or more. Who has that kind of money lying around? So there is this cost-of-capital, financing thing that comes into play. If customers are saying “I just want inference, as fast as possible, as soon as possible, please make it happen” — who in the supply chain has the money to invest in standing up all these data centers and running them? It could be neoclouds. Okay — do they have access to the capital they need to make tens of billions of dollars of investment, or even a few billion? A lot of these might be early companies, or former Bitcoin miners. They might have some access to capital, but not enough.

If you’re Nvidia, sitting there with all this money as the world’s biggest company, and you’re saying demand is incredible and supply is what it is — but it’s not just building the GPUs, it’s getting them powered up, financed, stood up — of course it makes sense from their perspective to help get this stood up. The banks aren’t so sure they want to lend to the neoclouds, but if Microsoft or Google or AWS says “I’ll be the offtaker — don’t think about the neocloud when you’re lending, think about me,” and if Nvidia can also come in and say “we want to help make this happen, can we put our brand behind it, backstop it, whatever” — I can see why Nvidia would want to do that. Now, does that mean they also benefit from it? Of course, totally. It’s customers, and maybe even revenue sharing — a new source of revenue for Nvidia.

David Goldman: Well, that’s capitalism. They’re not going to do it if they don’t benefit.

Austin Lyons: Exactly. But as sort of a techno-optimist: if it’s going to feel funny, but there are ways to get more compute stood up faster so that more people around the world can do the awesome things they’re trying to do, then I’d say, all right, I can get behind that.

David Goldman: I think it ultimately boils down to differences of opinion on the durability of the cash flow that comes from these assets. If you went out and said “I’m going to build a toll road,” you can get a lot of financing for that. You don’t have to put in a lot of equity, and you can get a lot of debt, because people know this road is going to have X number of cars, we know the traffic patterns, you’ll collect this amount of money. It’s very safe. You can raise lots of debt at very low rates for projects like that, even if you’re a new company. Obviously there’s a big difference between a toll road and a GPU-based AI factory, as some people are calling them. But from Nvidia’s perspective, and from some of the hyperscalers’ perspectives, these are fairly safe assets that will be completely paid back in two or three years, and then there’s a stream of cash flows coming out of them. Even if demand goes down a little or doesn’t grow at the same rate, you’ll still get value out of them, and they’re not going to depreciate super quickly. Nvidia has that view and is willing to put their balance sheet behind it, the hyperscalers feel the same way — and not everyone else has the same view. That’s where the rubber is going to meet the road.

The flip side, though, is that you have the existing neocloud set who are now very embroiled with Nvidia — they’re in lockstep, they rely on them for financing. To the extent that customers are going to demand more and different silicon, that creates an opportunity for new neoclouds who can figure out how to make it all work together and choose the right silicon for what customers want. Do you think that’s effective counter-positioning? Will we see another wave of neoclouds like the first time, or will the existing set — the CoreWeaves, the Nebiuses, companies like that — figure this out and just start using Nvidia plus Groq, or AMD plus whoever?

Austin Lyons: Right. I think both will exist, but you make a good point about the trade-off. It’s kind of like golden handcuffs. The trade-off for a neocloud is: hey, they’re backstopped by Nvidia, maybe they got financing, and they’re obviously getting allocation for GPUs. So those particular neoclouds might feel like, if there’s different silicon out there that’s very competitive, even for a subset of workloads, we may not feel like we can go buy it and offer it — because what if we don’t get as much allocation in the future? Don’t bite the hand that feeds you. So I do think there will be opportunities for other neoclouds to come in and say: we’ve got a bunch of different silicon, and maybe we can abstract it and run your workloads across it so you don’t need to worry about it. There will be opportunities for someone to come in and counter-position.

Four conditions for the next trillion-dollar chip company

David Goldman: In the last couple of years there have really been two big outcomes — we’ve touched on both — in the semiconductor space: Groq, which sold to Nvidia, and Cerebras, which went public. But neither one really won a hyperscaler before they were able to do this, and in the case of Groq, they sold to Nvidia. You wrote an article about what you think the conditions are for the next trillion-dollar chip company, and you had four conditions: capable of running trillion-plus-parameter models; rack-scale chips, which we’ve touched on; beating an incumbent on a KPI; and landing a frontier anchor. Points one and four — running trillion-parameter models and landing a frontier anchor — are correlated around the idea that the frontier matters most. Why do you think that’s the case, and a prerequisite to being the next big breakout company in silicon?

Austin Lyons: Very interesting question. Ultimately, today, it’s the best models — especially if you can run them at fast enough speeds — where my belief is the outsized value will accrue. Yes, there are lots of use cases where you can use older models, smaller models, and you don’t have to run them as fast. I think that pie will continue to expand. I just don’t think people will pay a premium for it.

So if you’re an AI accelerator company and you’re trying to put as much muscle behind a few arrows as possible, you’d want to compete at the frontier. Today it’s software developers saying: yes, I will pay — not $200 a month; we’ll pay for tokens, thousands of dollars a month, tens of thousands a month — if we can get it fast, if we can get Claude Fable, for example, or the latest OpenAI model. And of course the frontier will always keep getting better.

It also feels like the frontier is where there will be the least competition, because GPUs, for example, can’t get there today on speed, and we know that. If you’re aiming at a 70-billion-parameter Llama 3 and going after all those workloads that are valuable but don’t need the highest intelligence, there’s going to be a lot of competition there too — and it could literally be old Hoppers or old Amperes from Nvidia.

When you look at that Pareto frontier curve we always see — I’ll describe it for people who are just listening: the x-axis is interactivity, which is how fast the tokens come, tokens per second per user; the y-axis is throughput. The slower you go, the more tokens you can generate concurrently; but way out there on the far right, going fast, even if you can’t serve as many users — that’s where the value is accruing today. And it’s hard to see a world where that changes.

David Goldman: That graph gets shown a lot, particularly when Jensen or people from Nvidia talk about what they’re going to be able to do with Groq plus Nvidia. But is tokens per second per user — that interactivity KPI — still the right one for startups to think about? Or are there changing needs because of power constraints, cost constraints, new workloads like agentic coding? Is it still all about speed?

Austin Lyons: It’s not all about speed. In my opinion, the way to compare people is at a fixed interactivity, for a given unit of power. We are power-constrained. Ultimately, if a neocloud gets access to 100 megawatts, they’re going to have to ask themselves: how can I get as much revenue as possible out of this 100 megawatts? They might say, I want to bet on allocating some of my megawatts to really fast tokens, because I think we can charge more for them. So if you’re going way to the right on the interactivity curve — let’s say they’re aiming for 800 tokens per second or higher, because they feel they can charge a premium for that, and they’ve only got so many megawatts — then they’re going to ask: how many concurrent users can I serve? What is my token throughput? So I think it’s token throughput at a fixed interactivity, normalized by power. But again, not every workload needs that.

David Goldman: I didn’t hear you say the word “cost.” One of the ways you get better interactivity is by using more expensive memory, using SRAM. So how much does cost play into that equation?

Austin Lyons: Cost plays into it. If you’re in that use case — you’re a neocloud, you’ve got 100 megawatts, you’re trying to generate as many tokens at a fixed interactivity as you can — you also have a fixed budget to spend on compute. If you buy an Nvidia Vera Rubin rack, the latest and greatest, plus nine accompanying Groq LPU racks, you might get really high on that interactivity, and it might be pretty good power-normalized — but you might have spent half your budget, or all of your budget, right there. So cost absolutely comes into play. When I’m thinking about the user experience, I’m thinking about interactivity and how many people can be served. But if you’re a neocloud or any buyer of compute, you’re definitely thinking about cost. And cost could mean: I can get the same performance out of two racks from this vendor versus eleven racks from that one. Or, for the same fixed cost, what if I could get eleven racks from this new competitor — and therefore five or ten times more tokens at that interactivity?

Enterprise and on-prem AI

David Goldman: This kind of buyer thinking is very emblematic of a cloud or hyperscaler with a huge instance they’re trying to spread over lots of users. I personally always struggle with holding two ideas in my head at the same time. On the one hand, all of the initial demand and value has been going to the frontier labs, served by a combination of hyperscalers and neoclouds. On the other hand, we and many other people believe AI is going to be something as big as the internet. It’s going to diffuse into businesses all over the world. Every company is going to have some AI element, in the same way every company has a website now — there are no more dot-com versions of companies; everyone’s got a website, everyone’s got an app. Soon everyone will have some AI element in their business. And in today’s internet world, more workloads exist on-prem than in the cloud. If I’m a coffee shop, I might have a server in my coffee shop; I’m probably not going to have an AWS account. So ultimately, does enterprise actually become the big market here instead of cloud?

Austin Lyons: That is a very good question. I think they’re both going to be massive. Let’s talk enterprise, because I don’t think people appreciate that enough. Even Nvidia is trying to get ahead of it — they changed their reporting and their business units so that for data center it’s essentially hyperscaler and non-hyperscaler. It gets a little fuzzy, but they’re saying that, by the way, the non-hyperscaler segment is growing faster than the hyperscaler one, and right now the revenue is pretty close on both — a little fuzzy because they put neoclouds in there.

So the question is: what workloads are going to go to the enterprise, and why? You’re totally right that there is going to be diffusion of generative AI across every industry, and I definitely don’t think we’re there yet. Marc Andreessen said fifteen years ago that software is eating the world — every company is going to be a software company. And to some extent that was right: logistics, manufacturing, healthcare — they’re all using software, even if it’s just internal tools. I think we’re going to a world where agentic AI eats the world, and every company is going to be an agentic AI company. Like I said, pointing at my son: fast-forward fifteen years, and yes, they’re all going to be agentic.

Okay — so if agentic AI is core to how businesses run, are they all going to have hundreds of millions of dollars to spend on tokens every year? Totally not. There are going to be all sorts of reasons for companies to want to deploy workloads on premises. One: cost. Two: owning your own data, and figuring out how you even differentiate in a world like that. It won’t be all-or-nothing — go to the cloud for the frontier workloads, and do as much as you can of the older, smaller models on premises.

By the way, you could also read Nvidia’s hyperscaler/non-hyperscaler split as a proxy for frontier closed models versus open-source models, because if you’re running enterprise AI locally today — on premises, on a server, on your desktop — it’s got to be an open-source model. That’s a little bit why we’re in the world we’re in today: if you want the best model, you have to go to the cloud. And for everything else — do you want to pay for tokens, or for token generators? I think a lot of people, if they can afford it, would rather own token generators.

But fast-forward a little: what happens if open-source frontier models can keep up, or be good enough? I think there will continue to be a rise in the amount of workloads run on premises. There are all sorts of reasons. Of course you can point to regulated industries that will have to run on premises. But even if I’m not regulated: maybe I think the labeled data my humans generate is valuable. Say I’m an insurance company. We have all this agentic stuff doing claims processing, and my humans are going in and correcting it — that was good, that was good, that was wrong. Let’s keep that data internally and fine-tune our own model so it gets it right in the future. Maybe I want to run that locally because I’m in charge of the model, I’m in charge of the data, I keep it, it’s all my intellectual property. And maybe I see that as how I differentiate in the future — I’ve got better agents than my insurance competitor.

Can you do all this stuff in the cloud and feel like it’s secure? You totally can. But at the end of the day, when we’re talking about diffusion, we want every engineer at every company to be able to tinker with it, touch it, play with it, use it themselves. Sometimes when stuff’s in the cloud, you get the convenience, but you lose the ability to get under the hood — whether it’s a closed model or even an open model in the cloud.

David Goldman: It’s funny you say that — I almost feel the opposite way about enterprise. To me it seems gated a little bit by software. So many of these companies haven’t moved workloads even to the cloud, for data sovereignty, regulatory, privacy, IP protection reasons. They’re very concerned about stuff leaving their IT premises, and they would gladly do more in AI, but they don’t have engineers in-house who know how to post-train a model or tinker with this stuff. Those people are expensive, and they don’t want to hire them. So they want to do it on-prem in the enterprise, but no one’s quite figured out how to help them do that yet. That, to me, feels like the missing piece that would unlock a lot of enterprise hardware sales.

Austin Lyons: I definitely agree with you — that’s where we are today. Back up eight years: everyone wanted to be a software company, but they didn’t have software engineers. I live in Iowa, so if you were a software engineer there and willing to work in insurance or ag or retail, you were a rock star. You could walk in and they’d say: yes, thank you, we need you, we didn’t have this capability before. Fast-forward to now, and anyone can vibe-code, which is actually pretty awesome, because now the domain experts — the person in insurance who knows insurance really well — can actually build the solution they want.

I think we’re now where we were eight years ago, but for fine-tuning: I’m an insurance expert and I can vibe-code a thing, and I’ve got some software people here, but none of us know how to fine-tune yet. That’s the education piece. If my children had to go to college today and pick a major, I’d say: pick anything, plus machine learning, and learn how to fine-tune, because then you can understand a domain and also understand how to actually apply AI. So I think there will probably be a rise of AI engineers, if you will. Maybe agents will do this for you and bring that cost down to zero faster than agents brought software down to zero — which took, you know, forty years. But it is a pain point today that companies don’t have generative AI familiarity yet. I don’t think that pain point will be there forever. Maybe it’s five years, maybe more, I don’t know. But I don’t think it will always be a blocker. If every company became a software company with software-literate people on staff, I think eventually everyone will have LLM-fine-tuning-literate people on staff too.

David Goldman: So when that day comes, is this just a huge unlock for Nvidia, and they get that much more revenue? Or do you think anyone else has a chance at that market?

Austin Lyons: That’s a great question. What happened to IBM, you know? There are giants, and they’re first, and they ride a huge wave. And then, to all the points we’ve talked about — zooming out, there are three or four winners, and people want competition. The more people can tinker, the more this diffuses, the more opportunity there is. One company cannot meet everyone’s needs at the right price point, at the right speed. They can meet lots of people’s needs, but there will always be people trying to do some interesting bespoke thing who say the off-the-shelf stuff uses too much power — I know I’ve got this crazy setup, but I can’t do 130 kilowatts, I can only do 50. There will always be emerging workloads where the off-the-shelf stuff just doesn’t fit.

And the tough part is, when you’re Nvidia and you’ve got these huge hyperscalers, you’re not necessarily incentivized to go find those little people. You’re not interested in picking up pennies; you’re interested in picking up billion-dollar bills. So I think Nvidia will be totally fine — they have great solutions, and they’re always going to have customers coming to them for the latest and greatest, deployed quickly. But I do think there will continue to be new opportunities popping up, especially as this diffuses, where people can compete.

Clean-sheet silicon for LLMs

David Goldman: Circling back to this idea of the next trillion-dollar company. There’s an explosion of opportunities in different workloads, and also, as we’re hearing, different markets. Does that mean there’s an opportunity for a trillion-dollar company? Or are we going to get $500 billion companies? AMD is still not even a trillion-dollar company, and they’ve been around for a really long time and have a lot of pieces of this puzzle.

Austin Lyons: I just think about the size of the market. Here’s the deal, and part of why I came to that conclusion. GPUs obviously have a history in graphics and doing things in parallel, and that has been shifting toward AI-centric. Nvidia is saying: yes, these data center GPUs — you’re not going to run Doom on them. They used to support FP64, and they still do, but we’re going to spend all of our transistors, as we do a node shrink, on FP4 and FP8, the lower precision that AI models really want. But at the same time, it was still a general-purpose GPU.

That’s why I said we went from training with GPUs, to inference with GPUs, to this next era where LLMs are the workload — and we still haven’t had silicon designed specifically for LLMs. We had GPUs that morphed from their early roots to fit the shape of what we’re doing. Let’s take a GPU and slap on this SRAM thing. But no one yet has really brought a chip to market that was designed specifically for LLMs. So the question is: what if you can be the first one that can stand up a gigawatt’s worth? Which is not simple.

David Goldman: Well, to some degree, the Nvidia GPUs of the last couple of cycles are chips designed specifically for LLMs. It’s not like you can take Grace Blackwell and play a video game on it easily. It’s highly specialized for this type of workload. And in particular, when you get into these combination GPU-LPU systems, or the AMD Helios, these are really designed specifically for LLM workloads.

Austin Lyons: They are morphed specifically for LLM workloads. They have legacy ways of doing the networking, legacy ways of thinking about the memory hierarchy. I definitely agree that they are iterating toward what is best for the inference workloads they serve. But the question is: what if you started with a blank sheet?

We actually saw this from OpenAI with their Jalapeño chip, which they presented at Hot Chips recently. They said: we started with a blank design, and we’re thinking very differently about it, making very different architectural decisions. They said, instead of shipping all this KV cache around and having all this shared memory and all this contention — the data is never in the right place when we want it, and our compute is always sitting around — what if every accelerator had its own little HBM slice? What if we mapped the workload and rethought it so we don’t have all this contention and shipping data around?

Tensordyne is another example. They said: should it be matrix multiplication, or could we do log math? That would turn multiplies into adds, which are really fast in silicon. So I actually do think there are architectural knobs.

Nvidia’s next GPU still has to have backward support for the software — not exactly, but to some extent; they want to support all the workloads, so if it ran on Hopper they want it to mostly run on Blackwell. But what if you could start with a clean sheet and make different architectural decisions — about the way you do the compute, the memory hierarchy, the way you network it up?

Etched said: let’s do low-voltage inference. What would happen if we ran this at lower voltage, instead of putting in a lot of oomph so it can go really fast? If power is fixed, maybe there are benefits, even with trade-offs — it doesn’t run as fast, but it’s significantly lower power.

So I do think there’s opportunity to make clean-sheet designs, make different architectural decisions, and therefore unlock that KPI I talked about. What if you could get 10x more tokens, at that fixed interactivity, for that particular model, out of your 100 megawatts, than you could buying something off the shelf from Nvidia or AMD?

David Goldman: You hit on another interesting tension. On the one hand, you have someone like OpenAI, who is a large customer of Nvidia but is also now designing their own chip. When you hear them talk about how they did that design, they claim it was very AI-optimized — that they were able to do it much quicker because of AI acceleration. And then you also have companies like Tensordyne, who have an entirely new paradigm, a whole new way of approaching this problem, that a customer probably wouldn’t have thought of on their own, because they’re more focused on their specific workload than on new ideas from first principles. How do you think about the trade-off between custom silicon companies using AI to create something specific for what they need — maybe working with a Broadcom or Marvell to help them finish and get it into production — versus these companies with entirely new approaches?

Austin Lyons: It’s so interesting to think about OpenAI buying compute from the big vendors, mostly GPUs, while also building their own silicon — with the advantage of their chip designers working hand-in-hand with their software team, co-designing for specific workloads. Off-the-shelf silicon vendors are not inventing in a vacuum; they’re talking with their biggest customers, asking where their roadmaps are going and how to make sure their silicon meets those needs. But that’s different from OpenAI internally having their ML team and their chip team working closely together and co-designing.

That said, they’re going to land on a particular set of trade-offs. Everything in engineering is about trade-offs. Do you want more HBM or more SRAM? It’s going to cost you something either way. Do you want more die size, or do you stack it higher and take the thermal trade-offs? The merchant vendors made a particular set of trade-offs — and they need to sell their chips to as many customers as possible, even if it’s only a handful these days. The internal teams will make their own set of trade-offs, given what they know about the workloads they’re running.

But to think those two sets of trade-offs are all you’ll ever need? I think there’s opportunity for someone like a Tensordyne to say: what if there are these totally crazy trade-offs, and we take the risk of doing the R&D on them? The log math stuff — surely it’s crossed the minds of people at the merchant silicon vendors or the internal XPU teams; they’ve seen a paper. But they may not be incentivized to take the risk. If you’re OpenAI’s XPU team making your first chip, are you going to play around with log math? Or are you going to say: no, no, let’s make these other interesting but proven decisions, like HBM slices that have been used elsewhere in industry?

David Goldman: Yeah. People forget there are people involved in these decisions who have career risk.

Austin Lyons: Exactly.

David Goldman: If you can make a decision that’s high-probability and still works, but maybe isn’t the 10x — that’s probably better for you if you’re at a big company, particularly delivering your first version of something. You don’t want to screw up.

Austin Lyons: And it’s expensive, too. You’re going to tape it out, and you might be all-in a billion dollars or something. You don’t want to get that wrong, or have it get cancelled before you can stand it up. So yes, there’s this whole human side, these incentives. And again, that continues to create opportunities for startups to say: we’re going to take that risk. Or, in Tensordyne’s case: we’ve been taking that risk — we were trying the log math in a different market, and now we’re ready to bring it to this one. Could an OpenAI say “that’s super interesting, let’s try a couple racks”? Absolutely — these are very sophisticated buyers, and they’re always watching what else is out there, because they completely understand that designs made at a particular point in time may not fit exactly what they need a couple of years from now.

AI-designed chips and the falling barrier to custom silicon

David Goldman: Are more companies looking at custom chips? OpenAI has a lot of money and a lot of really talented engineers — even without AI, they probably could have designed their own chip. But as AI makes designing a chip easier — it’s not at the level of your son creating a video game with Codex, but you talk to teams and they’re seeing lots of improvements — there’s some floor to how cheap it can get, because you ultimately need to tape out and do things in the physical world. Do you think we’ll see a big proliferation of companies that never would have tried custom silicon giving it a shot?

Austin Lyons: I definitely think so, and you can already look at examples where it’s happening. Why would you design your own silicon? If you know your workload really well, what you bought off the shelf — where design decisions were already made — might not map to your workload perfectly. That might be okay at first. But take Rivian, for example. They went through a couple of different merchant silicon vendors, mapping their workload to the parts, and it was fine enough. But if you’re running an electric vehicle and trying to do autonomous driving, you have very specific needs. You want to use as little power as possible, because otherwise you’re taking battery away from the customer being able to drive another couple of miles. On the other hand, you need to run as fast as possible, because if it takes too long to make a decision, that’s another 20 meters of braking distance while you were thinking. Now all of a sudden they’ve got lidar and cameras and all this data flowing around that they need to do inference on as fast as possible. And by the way, it used to be convolutional neural networks; now they’re doing end-to-end VLA — vision-language-action — models. The workload has been changing, and in this world of real-time inference on big models with power under control, plus a particular interconnect bandwidth and memory capacity and memory bandwidth — they’re saying: man, this stuff off the shelf just doesn’t fit our needs and our cost profile.

So the answer could be: design your own chip. Well, there’s a cost to that. You need engineers familiar with front-end design, back-end — maybe you can partner for testing — and at the end of the day, that might take you three years of development. You’re paying those engineers for years, there’s a roadmap, it’s a very big investment, and then there’s the cost to tape out and actually get it built. But what if, with the help of AI, it still takes a hundred people but instead of three years it takes one? Maybe your cost is cut by two-thirds. Where before, even though the performance and the headroom would let you do interesting things, your CFO was saying “we just can’t add another $2,000 to the bill of materials” — maybe now you come back and say it’s only going to be $700 on the BOM, and they say: okay, that’s really interesting, we think we can hack it.

So I think adding AI to chip design — speeding up time to market, doing more with the same number of people — will ultimately be net good, and will reduce that cost barrier to entry. Or it could be a talent thing: how do I go find a hundred people? Maybe you only need fifty. I just think it will reduce barriers to entry, and we’ll see all sorts of cases where people who never thought about making their own chip will say: that’s actually something we could do.

David Goldman: It’s an interesting dynamic, because the same speed-up available to these companies is also available to the merchant silicon teams — who are probably even better positioned to use these tools, because an experienced engineer who understands the trade-offs is going to use the tool better than someone approaching their first chip design. So maybe they can start proliferating their number of SKUs and serve more customers with more semi-custom things within what is somewhat merchant. I just think it’s really interesting. I don’t know how it’s going to work.

Austin Lyons: Yeah. I would hope that merchant silicon companies — the thing is, when you’re a merchant silicon company making a particular product, there has to be a big enough market to capture enough customers for it to be worth your time and investment. There’s probably a point where the ROI didn’t make sense: the market’s only this big, the SKU costs this much to develop — not worth it. But if the cost can come down by half, maybe it clears the internal rate of return you needed. So yes, I’d expect big companies to be able to create more innovations as well.

David Goldman: Awesome. Well, I don’t think anyone really knows how it’s going to play out. This is such an exciting time. Thank you so much — this has been an incredible conversation. I’ve really enjoyed it.

Austin Lyons: Yes, thank you. This was fun. Let’s do it again.

David Goldman: Absolutely.

My hypothesis is that we’re on the verge of agentic AI crossing the chasm from early adopters to the early majority because product companies are turning these capabilities into simple, turnkey, yet useful services. And the upfront cost is low or zero. For now…

Grok Bot showed the way:

Free. Simple.

From the video:

There wasn't anything to learn. It was just like bringing on a co-worker to my team

The same video hints at the impact this could on the AI infra supply chain:

It's not using your computer. So you can keep doing work in parallel while bot is working on another instance.

Meta shipped Muse recently; a personal agent with its own cloud computer, accessible through an app or WhatsApp:

WhatsApp!!!

Meta’s distribution (3.6B daily active users across the family of apps) is where adoption could inflect. If Meta does it right, they have a direct connection to billions of active users!! And AI infra demand too. Good thing they are building out all those datacenters, even if they keep getting punished by investors because of it…

Of course right now Muse is starting with a US rollout so those billions of users are potential reach, not necessarily day one Muse users. But obviously downloading an app is a familiar reflex for nearly everyone in the world; setting up a Mac Mini etc is not.

And btw, if you’re feeling brave, there’s also Instinct:

Consumer cloud agents will continue to accelerate demand for server CPUs.

Yes we have a lot of ChatGPT and Claude adoption already amongst normies, but not necessarily heavily agentic use from most that aren’t coding/vibe-coding. Although Astra is showing more normie use cases like driving DaVinci for editing.

But Muse, Grok Bot, Instinct show simple user experiences have the potential to quickly drive adoption to the masses. And each person can generate more than a single agent run per day; Grok Bot’s docs say:

Multiple Bots share one user-scoped computer and can run in parallel.

Could 100 million people run 1 billion daily agents? Seems plausible.

Of course, a billion daily agentic tasks doesn’t imply needing a billion CPU cores because it’s not like we’ll have a billion simultaneously active agents. That said, there will still be a heck of a lot of CPU cores running simultaneous agents, and those agents will each kick off some LLM inference and quite often also stress the existing cloud infra (fetching data from the web and so on). And the question is whether Grok and Meta and Instinct have enough CPUs for this growth?

And these are all ADDITIVE to OpenAI and Anthropic, who will continue to innovate and make it easier for folks to use agents to do the things they find useful at work or home.

Subscribe now

What does this mean for the ecosystem?

For paid subscribers:

• CPUs: We obviously need more CPUs. But which ones?

• Accelerators: Which accelerators? Nvidia? AMD? XPUs? Startups?

• HBM and DRAM: Agents need memory for the model and for the computer it drives. Which demand grows faster?

• NAND: Agents leave a lot of work behind. What does that mean for flash demand?

• CoWoS and EMIB: More accelerators means more advanced packaging. Who captures the growth?

Read more

• EDA: Synopsys, Cadence, Siemens

• Memory: Samsung, SK Hynix, Micron

• Leading-edge foundry: TSMC, Samsung, Intel

• GPU: Nvidia, AMD

• CPU: Intel, AMD, Arm

• AECs: Credo, Marvell, Astera

• Scale-up interconnect: NVLink, UALink, ESUN

• Litho: ASML alone at EUV, only ASML and Nikon at immersion

Fixed costs are the main driver. The buy-in to compete in nearly all semiconductor markets is large and rises every generation. This isn’t SaaS! Interestingly, model lab dynamics look similar; the costs to play the game are enormous and rising every generation.

Take leading-edge foundry as an illustration. Chip companies used to own their fabs. But the cost of a leading-edge fab never stops rising... $100M, then $1B, then $10B:

Only companies with serious volume can amortize that.

Most chip companies didn’t sell enough chips to keep a fab running at full capacity. So each generation, more of them sold off their fabs. That’s where the fabless design industry and merchant foundries came from.

So the number of players in leading-edge fabs keeps falling because the market share needed to break even keeps rising. We can turn that into a rough rule. Take the market size and divide it by the revenue one player needs to break even, and that gives us how many players the market can support. As a first-order toy model…

Let’s do some napkin math as an illustration. We can approximate the size of the leading-edge foundry market; TSMC did $122B in revenue in 2025, and 74% of that was from “advanced nodes” of 7nm and below (call it ~$90B), so if TSMC has 90%+ of that market, we can call the market $100B+. Then let’s say a single sub-2nm fab is ~$20B-$30B. A $100B+ annual market with a $20-$30B buy-in supports about three players? Seems reasonable. Of course if the market share is even 60%, 25%, 15%, you’re looking at less revenue in a year than the cost of a single fab…

Yes, you amortize a fab and don’t pay for the whole thing up front; TSMC depreciates equipment over five years, so a $20B-30B fab is roughly $6B a year of cost. So that’s how one company can build and own many fabs at once. But the CapEx treadmill never stops; you’re building the next fab while depreciating the last one, hence the $41B of annual CapEx. And that doesn’t account for R&D costs. Staying on the roadmap cost TSMC $41B of CapEx plus about $8B of R&D in 2025. And I think they could spend even more.

We can visualize the break-even volume for a credible program at each node, meaning a fab or two of capacity plus the process R&D to stay on the roadmap. At 2nm that’s roughly two $28B fabs depreciating plus a few billion a year of R&D, call it ~$14B a year of fixed cost:

Look at the hypothetical 2nm break-even of ~97K wafer starts per month, nearly 1.2M wafers a year. That’s an entire gigafab worth of demand (TSMC’s own term for a 100K wspm site), running essentially full, all year, just to pay for the program.

Of course, fabs don’t start at that kind of volume. A new fab ramps for months at lower wafer starts while yield matures. So the buy-in isn’t just the fab. It’s the fab plus the underutilized ramp, losing money on every wafer until volume and yield catch up.

So why do most semiconductor markets hold ~3 main players?

Why not fewer?

Why not more?

Do growing markets allow more entrants?

And sure, foundry has the highest fixed costs so it’s a great illustration, but why does this extend to other markets?

Read more

Nvidia is planning to use a $6 billion deal it struck this week to build one of the world’s most powerful open-weight AI models, one that would compete with Chinese heavyweights like DeepSeek and Kimi K3, according to people familiar with the matter.

The chip giant’s licensing deal with the AI startup Poolside is also set to present a direct challenge to frontier U.S. AI companies including OpenAI and Anthropic, since open-weight models are generally far cheaper to operate and allow easy customization.

Nvidia has long been shipping the Nemotron family of open models, with the stated goal of giving every company a trusted base model to specialize for their bespoke needs:

“Our goal with Nemotron is to raise the bar for general intelligence, but provide that intelligence in a way that allows businesses around the world to specialize it for their own problems”. —Bryan Catanzaro, Nvidia VP

Nemotron is open and customizable. Artificial Analysis called Ultra “the leading US open weights model”, while noting it sits behind “the Chinese-led open weights frontier”. For now.

Nvidia is working on it. Back in March, it launched the Nemotron Coalition with partners like Mistral, Cursor, Perplexity, and Thinking Machines Lab to build competitive frontier open models. Can a coalition move at the speed of light though?

As we’ll see below, Poolside can help with iteration speed…

But first and more importantly, why do competitive frontier open models even matter?

Diffusion matters as much as innovation

A technology that can transform every sector is, by definition, a general-purpose technology. General-purpose technology is also abbreviated GPT, but not that GPT... I know, slightly confusing.

Check out Jeff Ding’s book: Technology and the Rise of Great Powers: How Diffusion Shapes Economic Competition.

The important takeaway is that the countries that win with GPTs aren’t the countries that invent the GPT, but the countries that diffuse the GPT.

We idolize invention; diffusion sounds boring. But it’s super important!

Actually, let’s back up a bit. Marc Andreessen’s 15-year-old "Software Is Eating The World" saw the diffusion of another general-purpose technology (software) and predicted it would upend every industry. Even the software industry! Every company would become a software company. Of course software wasn’t new, but thanks to various innovations like cloud computing and higher-level programming languages & frameworks, the rate at which it could diffuse was increasing.

Notably, Andreesen said the constraint preventing companies and industries from software-enabled productivity gains was people.

“Many people in the U.S. and around the world lack the education and skills required to participate in the great new companies coming out of the software revolution... There’s no way through this problem other than education, and we have a long way to go.”
—Marc Andreesen

This is a diffusion argument. For software to eat the world, it needs to diffuse across every industry, and that’s predicated on having enough software-engineering-literate people in every industry.

Crazy enough, fifteen years after Andreesen’s article, we see traditional software development diffusing rapidly across every industry thanks to generative AI! Now, suddenly, domain experts can be literate software engineers.

Hey Marc, let me update it 15 years later:

… There’s no way through this problem other than education vibe coding.

Jeff Ding’s interview with Microsoft’s Brad Smith makes the same skilling argument for Generative AI. Professor Ding argues that countries that see expanded GDP and increased productivity are those that can properly diffuse the general-purpose technology to the “average engineer”.

Should that occur, industries will get rebuilt not just around software, but around generative AI.

After all, agentic AI is a general-purpose technology that has already transformed software engineering and will eventually transform every industry.

The question that follows, then, is what will it take for agentic AI to diffuse broadly? Not just the diffusion of software engineering thanks to agentic AI, but the actual application of agentic AI across every industry.

Jensen argues that one important ingredient is the existence of frontier-class, competitive open-weight models.

And I wholeheartedly agree.

From Jensen’s “Open Weights and American AI Leadership” published on July 24, 2026:

“Our AI leadership will be judged not by one frontier AI model, but by whether the United States builds a strong, open ecosystem that diffuses into every sector”

Yes. Diffusion. The fastest diffusion is predicated on the existence of competitive frontier opens weights models. Again from Jensen:

“Startups, established businesses, universities, and public institutions can build on advanced models without training one from scratch or paying frontier-model prices for every task. Open weights let every organization match the right model to the right job at the right cost, reserving frontier-scale capability for genuine frontier problems and running efficient, specialized models everywhere else.”

Preach it Brother Huang!

And ideally fully open source, i.e. open model, data, and weights — see OpenMDW. Also a nice explainer table from opensource.org:

Why are open models what diffusion needs?

First, open weights means the average engineer across any industry can tinker with it. Quantize it to fit the hardware you use. Prune it. Fine-tune it on your proprietary data. Want more control over costs and performance? Host it yourself, choose batch size vs interactivity tradeoffs, tier the KV cache to improve the hit rate, and so on. All the while keeping your data private, with full transparency into the methods powering your models.

Don’t want to host it? Then don’t. The weights are portable, so you rent one host this quarter and a cheaper one next quarter. Free market.

Second, open models spur competition on price. A handful of closed vendors who all monetize tokens have little reason to race each other toward pennies-per-task. But diffusion needs lower prices. Jevon’s paradox, right? Open models mean many hosts can serve the same model, pressuring token prices down toward the cost of compute.

One more. Open models help companies retain and make use of their proprietary data. There’s the obvious sovereignty-and-regulated-industries angle, but that’s well-trodden. What’s also interesting here is having more control over the feedback loop: fine-tune a model, use it, generate more labeled data as you use it, feed that back into the post-training, and so on. Easiest and fastest when you’re in full control.

Diffusion already happening, but can be sped up with competitive frontier open model

Diffusion of GenAI across American industries is already happening, thanks in part to open models.

It’s there in Nvidia’s numbers, although a bit fuzzy. From the Q2 FY27 CFO commentary earlier this week, ACIE (AI clouds, industrial, and enterprise) was at about 45% of data center revenue, growing roughly twice as fast as hyperscale.

Of course, ACIE includes AI clouds (neoclouds), which mostly serve the model labs. So ACIE alone isn’t a direct measure of pure enterprise diffusion, but it’s in there.

I turn to Dell for enterprise read-through. On the Q4 FY26 call, Dell said

“the enterprise portion of our five-quarter pipeline grew and actually was the fastest-growing portion of the five-quarter pipeline.”

Yes, it’s easier to grow small numbers. But the Q1 FY27 call said the AI customer count “surpassed five thousand” and up over 50% in six months.

And note what those customers buy. Dell’s mix is shifting to “x86 + Blackwell... driven primarily by enterprise deployment, air cooling being the number one consideration”.

Dell is referring to products like the air-cooled PowerEdge server on HGX B200/B300 baseboards with x86 CPUs, not the Arm-based GB200/GB300 NVL72 racks hyperscalers buy. One can argue this is a “diffusion SKU”.

Yet to accelerate the diffusion of GenAI across healthcare, insurance, agriculture, retail, manufacturing, defense, and so on, we need to close the gap between proprietary, frontier-leading models and American open-source models.

I previously thought that a frontier-capable open model was simply not possible given the amount of recurring investment needed to stay at the frontier. Nvidia is one of the few companies with that kind of money, and I assumed Nvidia was incentivized to train only “big enough” Nemotron models to test their hardware end-to-end. i.e. eating your own dogfood.

Catanzaro said as much:

“Nemotron is not just a model, it’s all of our AI technology that helps us build the GPUs and systems for the future. There’s a lot of questions about how networking or the GPU should work in order to make AI progress, and we are able to answer those questions better because of building Nemotron”.

Moreover, on the business strategy side, wouldn’t a competitive open frontier model compete with the closed labs, the biggest consumers of Nvidia compute?

Well, Nvidia is buying Poolside, and I’ve changed my mind. I think Jensen wants to build competitive frontier open models. And it’s a good thing.

Sure, Jensen told us analysts at GTC in March something to the effect of, “we don’t have to be absolutely the world’s best model, but we have to be near it”. But with Poolside and the rumored Hugging Face acquisition, Nvidia is spending ~$20B (allegedly) on the talent and distribution needed to succeed. And that $20B will pale in the long run compared to what Nvidia will spend on compute to train models.

You don’t spend that kind of money if you aren’t trying to compete. Jensen Huang is a winner. I think he now wants to be as close to the frontier as possible, especially on the jagged edges that matter, like coding.

My read on the acquisition of Poolside is that it gives Nvidia the iteration speed needed to catch up.

Yes, via talent, but also via a bunch of internal tools and IP. According to Poolside’s blogs, training experiments that “used to take weeks to organize and schedule are now handled by the orchestrator in under an hour, often as few as ten minutes”. Batch size and learning rate sweeps run in “fewer than ten minutes, as opposed to days”. Deduplication went from “over a week’s worth of engineering time to schedule and run” to “the click of a button”.

Jensen is adding speed to his team through tools and people.

So this is somewhat like Meta buying rockstars for Meta Superintelligence Lab, except Poolside is 100+ people that already work well together and have already built a bunch of tooling.

This makes sense for Nvidia and for the diffusion of agentic AI across America and the world.

Why this is a good thing

Competitive frontier open models mean GenAI diffuses more quickly into every industry.

Open models enable fungibility, which unlocks competition. Competition and portability give the end customer control. Control unlocks innovation and cost efficiency.

American and global productivity and GDP benefit accordingly.

Yes, this aligns with Nvidia’s business model. As it should! A lower token price means more tokens. Cheaper tokens mean more demand. More demand means more compute. Open models pulling forward diffusion also pulls forward Nvidia revenue.

Give the model away, monetize the hardware.

This can be done. Existence proof one is autonomous driving with Alpamayo, a family of open VLA models. Nvidia monetizes via three computers: “training (NVIDIA DGX), simulation and validation (NVIDIA Omniverse and NVIDIA Cosmos on NVIDIA RTX), and in-vehicle computing (NVIDIA DRIVE AGX)”.

Existence proof two, robotics. Cosmos world foundation model weights are open too. Monetize via same three computers.

Nemotron + enterprise AI looks similar. Nvidia trains the model, releases the weights, and sells the hardware needed for fine-tuning and deployment. DGX Station and DGX Spark on the desk if you want, too. An important part of enterprise GenAI tinkering will be on local compute.

Zooming out, competitive open frontier models also lead to a healthier market structure. Dylan Patel on the Dwarkesh Podcast recently said “Anthropic and OpenAI are taking as much as 40% to 50% of compute next year”. Is a world where two labs buy half the world’s compute the healthiest outcome? Or do we want the long tail of enterprises buying compute or renting from neoclouds and CSPs, so one lab missing on a quarter doesn’t punish the entire AI infrastructure ecosystem? Like a local or state government putting the brakes on a datacenter buildout...

Put differently, more compute buyers is good for anyone selling optics, copper, power, cooling, and memory. And everyone investing in it.

Now the big question: is this in direct competition with Nvidia’s biggest end-customers, OpenAI and Anthropic? Yes and no. It’s nuanced.

We have to ask on what plane of competition are we talking about frontier? It’s jagged after all, right? Having open models catch up at different points of different jagged spikes doesn’t mean death to Anthropic and OpenAI, and in fact it may help them focus on which spikes matter.

Whatever happened to Sora? Oh yeah, Codex. Competition pushed OAI to be better, and that’s a good thing. More competition is good.

This is technology expansion. It’s not zero-sum. We can have a healthy OAI and Anthropic, and competitive frontier open models too.

It’s coming up a lot on earnings calls. From GlobalFoundries:

Karl Ackerman, BNP Paribas: For my follow-up, if I may, could you discuss what portion of those 7 customers on your SCALE platform are working on NPO or near-packaged optics -- and I guess how should we think about the timing of your NPO opportunity?

Tim Breen, GF CEO: Yes. Maybe just to take a step back, and I think there’s obviously a year ago, the industry wasn’t talking a lot about NPO -- now it’s talking a lot about it.

Let me remind newer folks quick about NPO.

Today’s pluggable transceiver is the industry’s workhorse, but it burns a lot of power. The module connects at the switch faceplate (far from the switch chip), so a long copper trace on the board is needed. High-frequency signals over copper degrade, so a power-hungry digital signal processor (DSP) sits in the transceiver, cleaning up and retiming signals in both directions, transmit and receive.

Linear optics such as LRO above give that cleanup work to the switch ASIC’s own SerDes. The transceiver can shed the DSP silicon (LRO sheds the receive half; LPO sheds all of it), and the power spent per transmitted bit drops.

Co-packaged optics take this to the logical conclusion and move the optical engine onto the chip’s own package.

NPO is the middle step. The optical engine moves onto the board a few centimeters from the chip, into a socket, but remains outside the package:

A nice two min video about this from my chat with Tom Barber:

So “What is NPO?” is one of the five Ws, and now you might now be wondering about the other four Ws:

• WHEN NPO?

• WHERE NPO?

• WHO NPO?

• WHY NPO?

Well, more context.

Remember that the CPO switches in the news like Nvidia’s Quantum-X and Broadcom’s Davisson are CPO used in scale-out. And a quick reminder that scale-out doesn’t simply mean “rack-to-rack” and scale-up doesn’t simply mean “within a rack” as lots of folks tend to say online.

GPUs/XPUs/AI ASICs want to access each other’s memory so fast that the remote memory feels like their own. Accelerators on the same “scale-up domain” can do just that. They wish they had more HBM, but for now it’s just you get whatever is attached to your board and that’s it, so the hack is to access your neighbors as if it was your own. This landscape will change with HBM pooling btw, but that’s a topic for another day.

If accelerators aren’t on the same scale-up domain, then they communicate over the scale-out network.

Scale-up has historically lived within a server node first (e.g. just 8 GPUs in a scale-up domain) then within a rack (72 GPU domain). But the scale-up domain can span neighboring racks too! NVL576 connects eight racks of 72 Rubin Ultra GPUs into a single NVLink domain. We even see Kyber rack NVL1152 taking it to eight racks of 144 GPUs, with optical links between racks.

So the first CPO products are switches on the scale-out network. Not GPUs. The optical engine moves onto the switch substrate, and nothing changes on the GPU side; the endpoints still run pluggables and copper. This is just like the images above where the O/E is moving closer to switch ASIC.

OK, with that in mind, back to what the industry is saying about when, where, who and why.

We can see a shift in language from various industry participants; for example, back in February, Macom CEO Stephen Daly positioned Macom’s products as “optimized for co-packaged and highly integrated architectures, like CPO and NPO”. Ah ok, both NPO and CPO are a-comin’

In May the prepared remarks carried the same script line in different words, and it was the only CPO mention in the call. By August, the line read “highly integrated architectures like NPO and XPO”. What? Where is CPO? And XPO?! And that call also mentioned NPO nine times. Macom now talks about NPO and XPO, not CPO.

This raises follow-on questions.

• When is NPO ramping? And why didn’t this NPO-before-CPO come up sooner?

• Does NPO push CPO back entirely, or only in certain sockets?

• Who benefits from NPO? Who doesn’t?

• How much of the “CPO” shipping today is actually NPO? Is marketing at hand here?

• Is NPO accretive or cannibalistic? For whom?

We’ll answer those below for subscribers.

We’ll also unpack the related MSAs and what they tell us.

We’ll touch on many companies including GlobalFoundries, TSMC, Tower, Lumentum, Coherent, AAOI, Macom, Semtech, Credo, Astera Labs, Marvell, Nvidia, AMD, Broadcom, Arista, Ciena, ASE, Celestica, Fabrinet, InnoLight, Eoptolink, HG Genuine.

Read more

Then we can ask whether LTAs change how we model demand, how we think about supply and price, and whether the incentives to bring new capacity online sooner change.

So two overarching questions we’ll look to address

1. How is today’s memory supply responding to demand, and on what timeline?

2. Do incentives like LTAs change that in any direction?

This article will begin by building our own simple supply model of the memory market. Not aiming for the best forecast out there, rather a simple enough one to be useful and teach us some important details along the way.

So a basic tutorial to get you started. Teach a man to fish...

Let’s model supply. In subsequent articles we’ll look at demand.

Today we’ll address

• Wafer starts and total bits

• Cell size and process nodes

• Node generations

• Node adoption across the industry

• HBM impact on bit density

• Industry totals

  • Checking the number two ways

• Supply growth model

  • Node migration puts and takes

  • HBM wafer absorption

  • The conventional mix die-size penalty

• Capacity coming online

• Checking the toy model

Let’s dig in.

One might start with the market-level question “how many bits the industry will produce?”

We can estimate this on a company-by-company basis. If you know the number of wafer starts per month, and the die size, you can figure out how many die per wafer and how many gigabytes of DRAM per die. Then you can make assumptions about yield to get the number of good die per wafer.

bits per wafer = usable wafer area ÷ die area × yield × bits per die

Or more directly, if you have the bit density:

bits per wafer = usable wafer area × bit density × yield

For reference, TechInsights measured Micron’s 1α DRAM at a die size of 25.41 mm² for an 8Gb DDR4 part with a bit density of 0.315 Gb/mm² and a cell size of 1,672 nm².

Plugging that in,

• 300mm wafer = π × 150² = 70,686 mm²

• With a 3mm edge exclusion, and subtracting the partial die that fall off the edge, about 63,800 mm² is usable

• 63,800 ÷ 25.41 = 2,511 total die

• At 85% yield, 2,134 good die

• 2,134 × 8 Gb ÷ 8 = 2,134 GB, so about 2.1 TB per wafer at 1α

2.1 TB per wafer at the 1α node. Yes, one alpha is actually its name.

Note that not every company can squeeze the same amount of bits out of a given area. It depends on the process node that the company and the fab are on. A DRAM bit consists of one transistor and one capacitor. The area it takes on the die is roughly 6F², where F is the feature size set by the process node.

With each process node, the transistor and capacitor shrink linearly. Because the cell is 6F², the area is reduced by the square of the shrink. i.e., going from say 16.5nm to 11.5nm is a 1.43x reduction in feature size and a 2.06x reduction in cell area. So process shrinks do matter.

The Greek letters make it obvious that memory process nodes are different than logic nodes. But note that Micron uses Greek letters, while Samsung and SK hynix use Latin letters. Hence 1-gamma and 1c are the same generation. Micron explains why they switched:

“We started with 1x, but as we continued to shrink and name the next nodes, we hit the end of the Roman alphabet. That’s why we switched to the Greek alphabet alpha, beta, gamma and so on.” — Micron

Logic nodes are just marketing today, as there are no transistor dimensions that are actually 2 nanometers or 18 Angstroms. But DRAM generations do somewhat attempt to tie back to an actual dimension, the “half-pitch”.

SK hynix sometimes writes it “1cnm”, “1c-nanometer” and “1c nm”. I don’t know why they include the nanometers bit. The most explicit name is “the sixth-generation of the 10-nanometer technology”.

Here’s the memory process size and naming by generation:

1γ is the leading edge in production, two generations on from the 1α we used in the example above. Bit density is roughly 0.55 Gb/mm², so a wafer capacity is ~3.7 TB. That’s a lot, and is obviously more than the 2.1 TB per wafer we estimated above. So process shrinks matter.

The big three are roughly on the same roadmap, but they don’t get there at the same time. Sort of like TSMC vs Intel vs Samsung Foundry for 18A and 14A logic nodes.

CXMT is behind on process nodes. Nomura’s CXMT initiation puts its mainstream node at 1x-1y, four generations back from the 1γ/1c leading edge. This is an important detail! You can’t just compare wafer starts per month.

Anyway, we’re starting to build some understanding. We can work out the bits on one wafer. 2.1 TB at 1α, 3.7 TB at 1γ.

What else do we need? Well, we aren’t accounting for HBM and how it decreases bits per wafer. Then we need to blend the process mix (not all fabs are on the same process), multiply by each producer’s wafer starts, and compare this total of our bottoms-up approach against industry estimates.

Note that gives us a good understanding of today’s supply, but doesn’t tell us about future supply. And recall we want to understand whether supply can respond to the increase in demand, and how LTAs impact that.

For that, we need to account for growth: what’s getting built, when does it actually produce bits, and any gains/losses from node transitions and HBM mix along the way. All of that is below.

And we have to check all that work and see if we’re on the right track with industry estimates. If not, we should think about why.

Oh yeah, and we’ll pencil out how far CXMT is behind and whether it matters.

Read more

His main argument: there is no one-size-fits-all for agentic infrastructure. The workloads under that label are too different from each other to land on one machine.

The industry has spent this year announcing agentic racks. Madhu thinks the label hides how different those machines actually are. Some of what gets sold as one is “just dense liquid-cooled racks that take advantage of what was provisioned in the data center for GPUs already”. But a gateway, a RAG database, a storage node, and a sandbox server each need different memory, storage, and CPU.

What is AMD’s agentic Venice part is tuned for? With 256 cores at roughly 400 watts, his framing for it is “maximizes threads per megawatt”. When your power constraint is a fixed amount of MWs, threads is the number that decides how much agent capacity you get.

The takeaways:

• Agents turn tokens into CPU work. Tool calls, sandboxes, verification, then a loop back around when the answer isn’t good enough. The GPU produces the tokens. The CPU runs the code those tokens describe.

• CPU to GPU has reached about one to one, and Madhu expects it to keep shifting toward CPUs. Every gain in GPU token output creates more tool calls and more sandbox runs, and those land on CPUs.

• AMD now cuts the server market three ways. Host nodes feeding GPUs, general purpose, and a new tier for agent orchestration and sandboxes.

• Concurrency beats per-thread speed, for now. Coding agents are coarse-grained and want every thread they can get. “What’s happening today with agents may not be what’s happening in two years.”

• Operators deploy as few SKUs as they can. Every extra SKU is another load-balancing layer and more stranded capacity. One trick is to buy the 256-core part and run 128 cores for the turbo headroom if needed.

• To date, agents mostly don’t run beside the GPUs. They run on laptops, on-prem boxes, or general-purpose scale-out, decoupled from the nodes serving the LLM.

• Tokenomics is a routing problem. Not every task needs the PhD-grade model when an intern-grade one will do. AMD’s own IT is testing whether an MI350P on-prem can cut token spend by around 40%.

• Openness wins the orchestration layer eventually. Bespoke now, standardized later, with Kubernetes as the template.

Full transcript below (lightly edited for clarity).

All right, hello everyone. We have a special guest today, Madhu Rangarajan from AMD. Welcome, Madhu. Tell us about your role at AMD.

MR: Hi. So, I’ve been at AMD for about three years now, and I run the compute and enterprise AI product team. So, that’s all of the EPYC CPUs and all of the GPUs that are not rack scale. So, think the PCIe card, think GPUs that are eight-way and so on. And I’ve been in the server industry for about 26 years now. Prior to this, I was at Ampere, prior to that at Intel, prior to that at Dell, predominantly in engineering and architecture roles, but more of a product role for the last few years.

Okay, awesome, interesting. So, perfect. You’re the perfect guest today because I want to talk CPUs for agentic AI, and then I also want to hit on enterprise AI, especially like sub rack scale. So, we’ll try to get to those both. So, let’s get into the fun stuff. So, let’s talk about CPUs. So, like maybe to lay the groundwork for listeners, at advancing AI recently, Lisa’s kind of called out these three different workloads in to explain sort of how the server CPU market is shaking out. Can you walk us through those and maybe tell us like why AMD sees the world this particular way?

MR: Yeah. So, stepping back a little bit, right? The transition from the chatbot era to the agentic era, basically drives a lot more CPUs because instead of being a chatbot just answering questions that you ask it, you now have an entire workflow where based on the token outputs from the GPUs, the real work is being done, whether it’s coding agents or any other sort of agents. And what that’s driving is a much more complex end-to-end workflow that involves a lot of CPU. And this is where I think we have been decomposing this workflow all the way from there’s a gateway. There’s a orchestration step. There is rag in order to inject enterprise data into your results. There’s the reasoning itself, which has been the interesting part for the last few years and all the attention has been there, but now making the reasoning do some useful work is driving CPU demand again. And after the reasoning, you’re going to have tool use because, well, at the end of the day, you want to do real work, and that means you have to run some tools and the tools tend to run on the CPUs. Then you have to do some verification to see, hey, did the agent do what I asked it to correctly? And then you loop around until the verification model says, yeah, that’s satisfactory. And then a user gets a response saying, okay, the agent did its thing. And at that point, the user might go, well, I don’t like the answer. Can you go do it again and this time fix this? So, you now have a series of steps that are CPU bound and some that are GPU bound running in a loop, and that’s what’s driving this crazy CPU demand we are seeing in the market right now. And the workload itself is not one size fits all. A lot of this workload is very familiar. All of the things that we know and love from the past like databases and CRM and ERP and all of the stuff that agents calling those workloads instead of humans calling those workloads. And in addition to that, you have some new emerging tiers like agentic orchestration and agentic tools running, where you now have agents creating code on the fly and executing those at extreme concurrency. So that’s creating a new tier of servers. So based on all of that analysis, we kind of broke it up into three large buckets, right? There is the host node itself, which is the CPU that connects up to the GPU, where you want a lot of single threaded performance, where you want a lot of memory bandwidth and IO bandwidth. And the goal is to make sure you maximize GPU throughput and the GPU is never left waiting for the CPU. Right? So that’s one tier of servers that we’ve been focused on. You’ve seen us have the Turin high frequency part serving that and we’re going to continue that with Venice and Verona. The second bucket of CPUs is the tried and true general purpose because agentic drives demand for databases and all and agents may run EDA tools and agents are going to do ERP. So all of the CPUs across enterprises and cloud that have been doing those workloads, you’re going to need a lot more of them now that agents are trying to make use of those CPUs. And the third bucket is this agentic orchestration and tools, where now you have swarms of agents trying to do a lot of work. So, you have to now think about it at a fleet scale level, where you want to balance how many agents you can run concurrently and how quickly any given agent is able to complete its task. And that’s what data center operators would be optimizing all day long and that’s that third tier that we introduced a new lower power Venice for in advancing AI recently.

Nice. Okay, this is helpful history. And I think like what I’m hearing you say if we kind of like go back in time was obviously like back in the cloud era, there’s these general purpose CPUs. You guys had different SKUs there to fit the different workload needs there, whether it’s a database or an API server, whatever. Then of course, fast forward to the early chatbot days 2023, 2022. All of a sudden there’s the need for the head node to feed the GPUs. And then of course with reasoning models and now agentic AI, it’s suddenly like, oh, we need this third bucket, the place where the agent’s going to live, do work, and a lot of that is CPUs and maybe ping-ponging back and forth with the GPU, but a lot is actually happening on those CPUs.

MR: Yeah, for example, right, the you ask it to you feed in a big piece of code and ask the agent to optimize it and make it more performant. You’re then going to eventually feed it into the GPU and the GPU is going to output a bunch of tokens, which is basically the modified code. And then you want to run that code and test it and see if it needs further modification. Or you could get more sophisticated and say, you know what, output five versions of this code for me, and then I’m going to run all five versions in a fan out manner and figure out which one works the best and then come back with the response. And all of that’s going to drive more CPU usage once the tokens are output.

Sure. Yes, that makes sense. So, okay, then if I’m thinking about like a one-word kind of priority for these different sockets, if you will, like what’s the priority for the host node versus the agent server versus general purpose?

MR: So, the general purpose, of course, it depends because general purpose is such a broad category, but in general, when you’re running general purpose at scale, you are trying to hit some kind of latency SLA while maximizing capacity, right? So, you don’t want to over index on just single threaded performance, but you don’t want to over index on throughput either because if you over index on throughput, then each individual user may be unhappy because their responses are slower. If you over index on single threaded performance, a single user might be happy, but then you end up with not being able to serve enough users, right? So there is always this fine balance that hyperscalers have been doing for years to get the exact right SKU in their data center, meeting the balancing latency and throughput. Right? So that’s kind of how I think about the general purpose. Of course, enterprises have different constraints, right? For them, they need to run their workload, but they may have constraints such as, I’m in a data center with only 10 kilowatts of rack power, or I don’t have any more space in my IT closet. So there are other constraints that make enterprises pick different SKUs, or there’s software licensing involved. I have a core-based licensing model, so therefore I may pick a lower core count SKU with a very high frequency. So, I think there isn’t a single answer in general purpose, but that’s also the reason why we have a lot of SKUs there. Now going to the host node, And you can see us advancing PCIe generations in the new Helios. You might have seen us have an actual coherent interconnect between the CPU and the GPU. So those are the areas we’re innovating there. And on the agentic, it’s going to be a mix and I think anyone that says we know exactly what agents are going to do now, will probably have a surprise because that’s going to change. But what we’re largely seeing today with the coding agent is they’re very coarse grain. They want extreme concurrency. They want as many threads as possible. They aren’t necessarily valuing per thread performance as much today because they want to run as many tools as possible and then they’re waiting for an LLM call, which happens in latencies that are pretty large. So, concurrency seems to be the primary metric today for most of the deployments we’re seeing, but at the same time, there could be more fine-grained agents. There could be some kind of agents and orchestration tiers where you need that more performance. But that’s the reason we have a portfolio, right? We have parts that have high single threaded performance and parts that have extreme concurrency because what’s happening today with agents may not be what’s happening in two years. Today, for example, agentic AI has a lot of Python code running. They create lots of Python code on the fly. That drives a certain kind of compute need. Two years from now, the agents could decide they’re going to run something else. They could come up with their own language that’s more efficient, right? And that’s going to need a different type of compute. So I think the best way to make sure you’re the best agentic CPU is to make sure you have a portfolio of CPUs that serve a fairly broad set of your usages effectively.

Sure. Yes, that makes sense. Oh, that’s helpful. So it feels like general purpose, it depends, but some sort of balance between throughput and latency. The host node, definitely it’s kind of what I hear you saying is speed. So things like clock speed, IPC, memory bandwidth, IO bandwidth and so on. And then, if what I felt like you were saying right now is for the agent server, it’s kind of it’s like about density and concurrency, but maybe, I felt like you were saying have an open mind because maybe they’ll maybe there’ll be some bifurcation where some of the workloads may need performance, some may So, do you foresee like a world where there’s like two different classes of parts for agentic AI?

MR: Yeah, I would never say never. I think data center operators try to keep as few SKUs as possible deployed at the data center level. And they usually try to pick the SKU that balances across multiple vectors the best. But if there is demand for any given workload that gets large enough to demand a certain set of CPU characteristics, they absolutely will demand unique will deploy unique infrastructure for that.

Sure. And I guess, but I so tell me a little bit more like why try to minimize the number of SKUs? Because I’m thinking out loud like, I guess you would have to orchestrate the different agentic workloads to the different SKUs. So that already is complex.

MR: Correct. So because think about it this way, right? You get a let’s say you get a thousand jobs that come in. If all of the servers you have are the exact same type of server, you just have to figure out which server is free and load balance your workload there. If you now have some servers with more memory and some servers with less memory and maybe your different storage configs, now you have the added layer of, well, this workload needs more storage, so it’s going to go here. This workload needs more memory, so it needs to go here. And then you add the third layer of, well, now different CPU SKUs optimized for different workloads. That’s another layer of how you load balance and orchestrate it. And the more you fragment things, the more you risk having stranded capacity. So if you’re operating at scale, you end up figuring out what’s the baseline demand for the different types of workloads and you can afford to kind of deploy different infrastructure for those different workloads. But until you can figure out what that baseline is, it’s a hard choice to make because you may make a bet on the wrong side.

Mhm. Mhm. I see. Yeah, that’s very interesting. Yeah, having stranded.

MR: And maybe just to add on to that, sorry. Yeah, go ahead. One more example I can think of is, for example, you could buy a 256 core Venice. And if there’s a workload that needs more memory bandwidth and a little higher single threaded performance, well, you could just keep 128 cores out of the 256 utilized and you can get that dynamic range to get that performance. So that’s one way to mitigate that is to say, well, I’ll deploy the higher core count CPUs, but if there are cases where I need to get a little more performance, a little more turbo frequency, that’s one way to say, you know what? I’ll modulate the utilization on that server to get more performance per core. So, there are different ways to handle this. Build a super set SKU and modulate the utilization, use two different SKUs. So, there are multiple ways I envision data center operators could is to could handle this situation.

Yeah, okay, that’s really interesting. I like the idea of sort of dynamically taking a particular SKU like that has lots of cores and kind of making it flex and behave differently for a different workload. Interesting. Okay, so now tell me, in the life cycle of hyperscalers and kind of the people on the bleeding edge who are needing agentic racks because they are orchestrating lots of agents, like where are we as far as like where are agents actually running? So I assume, and correct me if I’m wrong, that early on it was like, okay, we’re just getting into this agentic world. There’s no such thing as an agentic rack yet. Are they were they running those on the head node or were they actually not so concerned with like the IO latency and just sending them out to a CPU rack somewhere? And then how is that kind of changing? Like where are we today?

MR: Yeah, so I think there are a few different models. I think of how I use agents every day for myself. And at work, I use Claude Code a lot. And in that case, my agents are running right on my laptop. And the API calls are being made somewhere in the cloud to run the large frontier models to get tokens, right? So in that model, I got about four or five different agents running on my laptop. All of the API calls to the LLMs are happening in the cloud. So it’s that’s one model. The other model could be, well, the agents on my laptop, I can’t share them. So I want to share them with a bunch of my colleagues because I want that agent to be useful to all of them. So one option is, well, I could run it on a on-prem server in behind the firewall, well access controlled and everything that my colleagues can access. So now you start having those shared agents. Or you can run them on any virtual machine in the cloud anywhere, right? So that’s the other way to run it. So I think most of the these types of agents tend to run a little decoupled from the LLM inference call itself. On the other hand, I could have like a Strix Halo where I have one box where I’m doing the agents and I’m doing the LLM and I’m doing everything else. So I think multiple models exist. But for at scale deployments today, I’m predominantly seeing these agents run on largely general purpose servers in a scale out manner, quite decoupled from the LLM, the GPU nodes that are running the LLMs themselves because when you’re running, The GPU could be serving another LLM inference call and that’s a much better use of the GPU resources. So, at least my view is I’d be hard pressed to see why you would run it that way at scale unless there’s a very specific use case where it’s extremely tightly coupled.

Mm, okay. Yes. So, and I think you make a good point, which is there’s a lot of agentic use case today that’s happening that’s running on our laptop CPUs. So it’s not always even running in a VM on a server CPU. And so when it is, what I feel like I hear you saying is today it’s probably running on racks that are you mentioned scale out, but these would not be in the GPU data hall on the east-west network. They would be in a traditional data center somewhere.

MR: Yes.

Yeah. Gotcha. Okay. Yeah, that’s helpful. So then, maybe just extending this train of thought, when we hear about like, oh, so and so’s going to deploy a gigawatt of AI compute, of course a person can kind of like calculate how many head node CPUs that is. And we’ve talked about going from four to one ratio in the head node world to now it’s more closer to one. But how should people think about like, is there kind of like a rule of thumb of like, oh, a gigawatt’s getting deployed, so how many agentic CPUs that is? And I’m sure it’s it obviously is the answer is it depends. But should we think like people are buying agentic CPUs greenfield when they’re deploying a gigawatt?

MR: So, I think when they’re deploying a gigawatt, they have to think about the predominant use cases and the ratios, right? We’ve heard multiple ratios in the industry, 20 to 30% on compute nodes, maybe 70% on GPU nodes. We’ve heard ratios that are more even 50/50. So we’ve seen numbers that range all over the place. We’ve done some of our own internal testing with coding agents and how long they spend on the CPU nodes and how long they are actually spending on the LLM GPU nodes. So we got some coarse grain numbers from those as well. Now, some use cases stress the CPUs a lot more, some use cases stress the GPUs a lot more, so there’s going to be a range here. But generally we are seeing this one is to one becoming much more common based on those power ratios I just spoke about. But what we expect to happen over time is that ratio shifts even more in favor of the CPU because if you just plot what has been happening to let’s use SPECint_rate as a metric for CPU performance and plot what’s going on at SPECint_rate say over the last five years. And maybe even project forward what it looks like for the next five years. Project GPU tops and how many tokens GPUs put out over the last five years and project forward for the next five years, you’ll see that one line looks like this, the GPU line kind of goes significantly higher. And the more tokens that are put out by the GPU, the more CPU resource you need resources you need to do useful work with those tokens. So we actually think one is to one will climb to even more on the CPU side as we move forward in time driven by the sheer capability of the GPUs and all of the software innovations happening on quantization and so on.

Mm, that’s fascinating. So now, at the same time, you guys are always introducing like more and more cores per CPU, which maybe can kind of offset that a little bit. Like, but is there room to run with the number of cores that goes on a CPU or is like 256, 512, is that kind of going to be like the max?

MR: I have not seen that slowdown. We’ve always figured out how to put more cores and more importantly feed those cores, right? Because it’s not important it’s not enough to just have a lot of cores. You need enough memory bandwidth for those, you need enough fabric bandwidth, you need enough IO and networking. And you can see how quickly memory ecosystems been evolving, DDR5 from 6400 to 8,000 to MRDIMM 12800 and even faster speeds in the future. You can see the PCIe specs evolving fast with Gen 6 now and faster speeds coming soon. So we have managed to figure out how to squeeze in more cores and still feed the cores generation over generation and I envision that continues.

Okay, fascinating. Okay, so advancing AI, you guys had a lot of like, Venice SKUs that you launched and you were kind of tying it into this these SKUs could be good for host node versus agentic versus general purpose. Like, high level in case anyone didn’t watch it, like, what should we have in our head when we think AMD head node, do we think Venice HF and what do we think for each socket?

MR: Yeah. So, maybe I’ll talk about both Turin and Venice. Turin’s out now. Venice is going to be available in volume really soon. If I think head node host CPU, for Turin we had the 64 core 5 GHz high frequency part. And in Venice, we’ll have a 96 core 5 GHz high frequency part. For general purpose, big range of SKUs, right? Going all the way up to 192 cores, 500 watts in Turin, going up to 256 cores, 600 watts in Venice. And for these agentic tools and sandboxes, we have a power optimized 256 core, which is going to be closer to 400 watts instead of 600 watts. Optimized for sandbox concurrency. And again, the rationale behind that part is you’re going to have a lot of sandboxes provisioned on a server, but sandboxes also tend to have very spurty, bursty, branchy code running on it. So they’re not all going to be running like a throughput workload on a server at any given time. So, this 256 cores at around 400 watts maximizes how many threads you can provision in any given data center. And each thread still has the dynamic range to run as fast as possible when a given sandbox is active. So that’s kind of the rationale. It maximizes threads per megawatt.

Ah. Yeah. Maximizes threads per megawatt. That’s pretty that’s really interesting. Yeah, do you think, I mean, do you think there’s a point where people are talking about threads per megawatt as opposed to like CPUs per megawatt?

MR: I think they’re starting to get there sandboxes per megawatt, cores per megawatt because that is the unit of deployment and as you very well know, there’s a lot of power constraint in data centers. So if I have a 10 megawatt facility, how can I maximize how much sandbox capacity I can get there, right? So that is a critical metric and cores is a pretty good proxy for that as long as your cores perform well.

Yeah. All right, super interesting. Okay, so now let’s talk kind of competitive positioning. I for anyone who’s been following this closely knows there’s been a lot of companies in the space saying they’ve got an agentic CPU rack now. And I maybe to remind listeners, Nvidia has Vera, which was originally a head node and now they also have a standalone CPU rack. That was 88 cores and 176 threads. Arm launched their AGI CPU. I think it was up to 136 cores and 136 threads, so no SMT or multi-threading. And interestingly, which set us up nicely for this conversation, on Arm’s most recent earnings call, there was a sell side analyst who specifically used the AMD framing and he said head node,

MR: Yeah. So, the one SKU fits all, I think, anyone in this industry long enough to know that the customer environments are diverse, the workloads are diverse, the constraints are diverse, that I don’t necessarily buy into that view of the world. Sometimes you’re different driven to different SKUs based on workload, and sometimes you’re driven to different SKUs based on constraints. Sometimes you’re driven to SKUs based on, hey, I already plumbed this data center for liquid cooling because I have a bunch of racks and GPUs. And now I’d like to slide in a compute rack that also takes advantage of the liquid cooling, right? Some of what are being called agentic racks, in my view are just dense liquid cooled racks that take advantage of what was provisioned in the data center for GPUs already. I don’t think there’s anything that makes them agentic specifically because again, as I mentioned, agentic is a diverse set of steps. Many of those steps are like gateways running nginx, which are things that we know and love for many years now. And some of them are like these agentic tools and orchestration. So you’re going to have an entire data center that’s running a end-to-end agentic workflow, and you’re going to have multiple parts to save those to serve those workloads, right? Your rag databases are going to need something different from what your storage nodes are going to need, or are going to be something different from what your gateway needs and what the tools need, right? So, I think you will continue to see those types of diverse deployments, and even if you consolidate down to say two or three SKUs for your entire data center, you’ll still have multiple configurations in the rack when it comes to memory provisioning and how much storage you put in it and so on.

Yeah, that makes a lot of sense.

MR: Yeah, there’s no such thing as an agentic rack.

Yeah, well, no, and I think that’s very fair based on what we talked about, where you’ve outlined there are different workloads and they have different characteristics and therefore they need something different from the compute that they’re deployed on. And I actually like the way at the end saying there’s no such thing as an agentic rack, stripping away the marketing and just saying like, a dense liquid cooled rack, a dense air cooled rack, a maybe less dense but high performance liquid cooled rack that’s really what we’re talking about at the end of the day.

MR: Yes. Exactly. 100% agree with that. That’s how I think about the world.

Yeah, no, that is helpful. So thank you for sharing that. Okay, maybe one last question on the CPU front and agentic CPU front. A lot of hyperscalers have designed their own arm-based CPUs, Google Axion and Microsoft Cobalt and so forth. Traditionally those have been for general purpose and actually even a lot of them are for first-party workloads, but some are for third-party workloads as CSPs. Do you see those fitting into serving some of there’s obviously general purpose. Do you see them fitting into the agentic workloads at all?

MR: Yeah, I mean, they are general purpose CPUs and they can run a broad set of workloads, right? I mean, within the constraints of the Arm ecosystem, but if they wanted to run some Python code on a single thread, they could obviously do that really well. But our job is to make sure we run it even better than that and we give the concurrency and the PCO and the for what benefits that make it hard for them to choose their own CPU versus what we have to offer. So, that’s how I think about it is a competitive environment. We have other merchant silicon vendors that are really smart that we compete against, and then we have all of the hyperscalers who are building their own silicon with really smart people there, and we just have to do better than them and be generationally ahead, and that keeps justifying why our deployments continue to grow.

Nice. Okay, that’s a good answer. All right, let’s move into the enterprise AI stuff. So, can you walk us through listeners in case they forgot. Obviously, there’s a lot of talk about Helios lately, which is rack scale, but talk to us about the sub rack scale options.

MR: Yeah, and maybe we should talk, we can start talking about tokenomics and then we can talk about some other motivations like regulatory reasons and some other things like that. But if you think tokenomics, I think every corporation is starting to use AI far more extensively, and that has put a lot of pressure on token budgets, right? Every everyone I know is using a lot more tokens and the bills are growing dramatically. Now, the question that we have to ask is not every AI related task needs to go to a PhD scientist equivalent AI. Some of them could go to an intern equivalent AI, right? So, I think that’s really where tokenomics comes in is route the highest value requests to the highest cost AI resource, and things that can be done with smaller models in lower cost hardware, route them to that and create a more balanced environment in terms of token spend. I think that’s something that every single IT department across enterprises is looking at. Our own IT department is looking at that. We are doing experiments with the MI350P PCIe card, which can go into a broad set of on-prem enterprise servers, air cooled, slotted into a PCIe slot and get it going. And we are looking to see if we can serve save like 40% on token costs by load balancing the right tokens to that on-prem resource, and then the right set of token requests to the really large frontier models in the cloud. So, I think that’s what’s motivating customers to think about it is minimizing cloud token cost spend while still not compromising on the output of the AI. So, where you need the frontier models, you don’t want to run it on a smaller model because the quality of the output will be wrong and you’ll just run it five times to get the right answer. So, you don’t want to do that. You have to know what to balance where, and I think that’s where a lot of work is going on to have intelligent model routers and so on.

Yeah, okay, interesting. I mean, I can relate to this even from my own personal experience, which is I’ve got my $200 Claude plan and when I need Fable and I’m doing frontier model for coding, but then the stuff that I’m coding, I’m ultimately running and maybe have some lighter LLMs in it where I’m like, oh, dude, I’m totally not burning Fable tokens on some of this stuff. And in fact, and this is probably is actually very related to our conversation here. I have an AMD Ryzen AI Max, I think it’s 395 plus, it’s like the framework version, here in my And so even I have been thinking about like, how do I reduce my spend and orchestrate my tokens to running sort of from my own token generator, so free tokens if you will, versus paying for frontier tokens. And so you’re saying, a lot of enterprises are thinking the same way and MI350P is enough for their medium-sized models and maybe even the eight-way server nodes are enough to run even like 1 trillion plus parameter models.

MR: Yes. I think that’s what we’re seeing. You can do some medium to large reasoning on prem with those servers and then use the frontier models for the highest value stuff. And in some cases, they don’t even use those smaller GPUs on prem. They rent the smaller GPUs in the cloud. So that’s another model too.

Oh, there you go.

MR: Yeah. So I think both models are valid. You I think the whole world is was going hybrid even before this happened. And now it’s a much more complicated hybrid.

Sure, totally. So what about what are you seeing with customers like, are there concerns that to be able to run on premise or even on GPUs that you rent, you have to use an open source model, generally a little bit behind the frontier, versus the frontier models. Like is the open weight model that might be kind of six months behind a hindrance or not really?

MR: I think that’s where the model routers come in. If it’s a simple summarization task, I don’t think it’s necessarily as much of a hindrance. But if you’re asking it to do write the most complex piece of code, then it could be because it takes a few months for those models to catch up. So I think this is where how you route the models becomes important, right? Simpler use cases or use cases those models are good at, you route to those devices and the rest goes to the frontier models and that’s going to be a constant it’s a constantly shifting thing.

Yes. Indeed. Indeed. So and then what about desk side inference, whether it’s a laptop or a small kind of mini PC that are kind of gaining steam. Do you see those also just plugging into it sits behind the orchestration and or how are you thinking about it?

MR: Oh, I think you’re going to see all kinds of models, right? I’m setting up one of those at home to run some personal agents and I’m not necessarily and right now I’m still at a point where I don’t let the agents go unrestrained. So I would like to control the access policies and what they can access and so on. So I think there’s going to be a whole bunch of people doing things like that. I already spoke about running a bunch of agents on my laptop. I think that model will continue as well. I can envision a day when you even have a couple of smaller agents running on your phone. Right? So I think there is going to be agents deployed across multiple places and maybe even talking to each other and doing the right thing in the right place. I think this has always been the panacea of edge computing for a while is, hey, let’s have a far edge and a near edge and a regional data center and a central data center and the problem was always orchestrating it. But when AI can figure out how to do that, maybe that actually becomes more reality.

Sure, totally. Do you it feels like there’s probably opportunity for startups or companies to create this orchestration. I mean, is everyone doing their own bespoke orchestration or are there companies out there that can tackle this for customers?

MR: I think we are seeing some open source orchestration stacks pop up, but if I think about a lot of the biggest customers out there, it’s still a little bespoke. I think just like any rapidly evolving industry, it’ll start with things that are bespoke and people are modifying and then eventually become a little more standardized and open source. I think Kubernetes is a great example of that and I think we will see the same evolution here. I think eventually openness wins.

Sure, totally. And what about security? This just popped in my mind. We see the Anthropic’s got models that are very advanced and they’re very good at cyber security, either defense or offense. If I’m buying my own token generator, running stuff on prem, do I need to get more sophisticated from a security perspective in like now my on premise models need to be able to prevent someone’s other models from attacking them or something or is this just like a networking firewall issue?

MR: My view on AI and security is especially the most advanced AI models are extremely useful tools to make your entire system more secure, right? So I think you have to leverage regardless of what how you are splitting your workload between on prem GPUs and advanced frontier models. I think you’d use the most advanced models to find all of the security attack surfaces and close them. So I think of this as an useful tool to secure the system, no matter how you compose the system.

I like it. Yeah, that makes sense. AI is yet another tool to help you level up and do even better from a security perspective. Yes. Okay. So this has been great, Madhu. Let’s see, last question. I so I feel like what I’m hearing the theme for this whole conversation is AMD has a portfolio of compute for your different workloads, for your different size customers. We talked about CPUs and now we’ve talked about essentially sub rack scale GPUs. Anything else is that like the correct maybe vision and the way we should think about AMD going forward and oh, and is there anything that we’re under appreciating with that sort of portfolio and you’ve mentioned open before the open and the portfolio value.

MR: Yeah, and I think I should also add, right? There’s the CPUs, there’s the sub rack scale GPUs, there’s the Helios to run the most advanced frontier models. We have our Pensando DPUs to connect all of these together. Then we have the laptops and the we have the smaller AI boxes. So we got a portfolio that spans all of those and I think that’s important in this era because people are deploying this so many different ways.

Yes. So the TAM will continue to grow from the desktop to the edge, to the far edge, to the cloud and AMD has offerings for all of it.

MR: Yes. And we will continue on the server CPU side to make sure we have a portfolio because anyone that says they know exactly what agents are going to do say three years from now, well, I would love them to give me some stock advice as well. But I think having a portfolio that serves a broad set of workloads is what is going to win the win at the end of the day.

Yes, I agree with you. Yes, like you said earlier, maybe it runs Python today, maybe the agents decide to run rust or go tomorrow. Who knows? But

MR: Or they come up with their own new language too.

Right? Yes, there’s probably something even more efficient out there. There you go. Awesome. Well, this was a great conversation. We covered a lot of ground. Thank you for your time and we’ll have to stay in touch.

MR: Yeah, absolutely. Thank you so much.

His main argument: robots, humanoids, and autonomous vehicles add up to one of the hardest systems-engineering problems in computing. And the physical half of the global economy still runs on a fraction of the compute the digital half already gets.

Physical AI comes down to two numbers: latency from a sensed photon to torque on an actuator, and system weight in grams. No bolting on a bigger cooling system the way a data center would.

And Global GDP is about $115T. Roughly $70T is physical, $40T is digital. Almost all compute spend goes to the smaller, digital half. And yet the physical economy runs on a tenth of the compute per dollar of output.

That’s pretty mind-blowing! More things stood out:

• The vaue prop of Physical AI is time, not automation. An autonomous vehicle doesn’t remove the drive to your kid’s game. It gives back your attention during it.

• Every physical industry breaks into four verbs: grown or mined, manufactured, moved, operated. AI gains there flow into two-thirds of global GDP.

• A humanoid hand may be harder to build than an AI data center. Every joint shares the same latency and torque constraints, zero tolerance for error. “The thing you never want to do is hallucinate during the middle of surgery.”

• A robot isn’t one compute problem, it’s four: locomotion (real-time), conversation (LLM-based), actuation orchestration, and cloud sync for retraining.

• Memory and model size at the edge have no settled answer yet. Henry expects a decade of trading precision for footprint before it converges.

• Braking stays local, coordination goes to the cloud. No cell-tower latency on a meters-scale decision, but a warehouse of robots needs central coordination.

• Fleet orchestration, not the chip, is becoming the real edge. Design the robot and the conveyor together, don’t bolt one onto the other.

• Arm co-designed Graviton with Amazon, once Amazon knew its workloads well enough to spec exactly what it needed. Same pattern now starting in AVs and robotics: go vertical on custom silicon, built on Arm.

• No ceiling on compute demand, every robotics company tells Henry the same thing. His comparison: PC modding culture from his GeForce days.

Full transcript below (lightly edited for clarity).

Hello listeners. We have a special guest with us today, Drew Henry, EVP of Arm’s Physical AI Business Unit. Welcome, Drew.

DH: Thank you, Austin. I’ve been looking forward to this all day.

Awesome, good, me too, man. And of course, nice studio you’re in. That looks awesome.

DH: Isn’t it nice? I know. I really like it.

Yeah, it’s good. Everyone who’s watching take notes. This is how you do it for these Silicon companies out there.

Okay, so Drew, before we get into talking about physical AI, which is the topic here, I wanted to go through your background because you had an amazing journey. And so I’m just going to kind of read your LinkedIn profile for the listeners in case they aren’t familiar with you. And I’d love afterward for you to give me some color on maybe the through line of all your interesting experiences and how it helps you in the role that you’re in now.

So what I thought was awesome, you co-founded Animated Technologies in the early days of computer graphics and that got acquired. Then you went to Silicon Graphics, which I think most folks have heard of, that’s Jim Clark’s company during its heyday. And from there, in the dotcom boom, you went to a streaming media startup where you were an exec through the IPO. So all already right there, very interesting.

And then you went to this little company called Nvidia, which I think most people are familiar with in 2001, and you were a GM there for 11 years. And then you were at SanDisk, which is also a very hot company right now. A couple more startups, and you’ve been at Arm almost nine years now, where first you were the founding GM of the infrastructure business and launched Neoverse. Again, very awesome.

Quite a career there, but now you’re taking Arm into physical AI. And also, I noticed you’re on the board of Teradyne. So quite the expansive background from startups to Fortune 500 companies.

So, yes, tell us, what’s the through line and how does it apply to what you’re doing now in physical AI?

DH: Yeah, thanks. The through line for me is just I just always enjoyed working on really interesting things. And so from early days in studying engineering, both my undergrad and graduate degrees. I just was you just there’s just a curiosity that you have and that you build from something like that. And so that’s kind of the thing. I just always wanted to work on very interesting things through my career and I had the opportunity to do that being here in Silicon Valley. I’m one of the few people, by the way, that’s actually born in Silicon Valley and works in Silicon Valley. The but some of those things have been really informative to me.

So for instance, working at Silicon Graphics helped me deeply appreciate what it means to be disrupted and not being a disruptor. So I was through, I went through that entire cycle with that company. I was there in the very early days and when I kind of finished that cycle, I realized, man, I just I really I want to be on where technology is disrupting, where technology is on it on its growth side, where it’s where the company is really well positioned and stuff like that. And so every decision I made from that is based upon being on the disruptor side of things and not being on the disrupted side of things.

And when I joined Nvidia was literally a few hundred people at the time that I joined. Very early days. And so that was the fun time of trying to figure out what you’re going to do to create markets and stuff like that. So that’s that through line. The through line was learning painfully early on in my career, some of the lessons of being disrupted and then just figuring out how to be on the side of being a disruptor.

Sure, interesting. Really fascinating. I love that chasing your curiosity and then yes, trying to think how can you be on the front line of the disruption. But I think very cool for you as you’re a GM to think there’ll be a natural evolution to what you’re working on and how can you probably prevent the disruption in the long run.

DH: Yes.

That I wake up and go to bed at night thinking about.

DH: Great, that’s great.

Okay, so now you’ve moved into physical AI. So tell us more. I’m curious, like, how does Arm even define physical AI? I think it has a broad definition out there. People think of many different things. How do you define physical AI?

DH: Yeah, for us, it is we think of it as how AI, the latest in AI technologies are being embodied into physical devices. Devices that can sense what’s going on in the world, can take can make a decision about what to do based upon what it’s sensing and then can go off and take action in it. But do that safely in a way in which you’re really helping your helping solve problems. And the and so that’s how we think about it. When I turn it into kind of technical terms, I think about it as the key metric being the latency between a sensed photon and torque being applied to an actuator. That metric is key. And then the second metric that we always think about is that in this physical AI space, every gram matters. So you can’t have a massively huge computing system with a massively huge cooling system. Like in my GeForce days, we make huge cooling systems we put on those things. These things are everything is about a total system design. So that’s how we think about it. That’s how we kind of think about what’s most important for it.

Hm, I like it. Sense, decide and act. I like that at a high level because it’s not focused on a particular embodiment, humanoids or automotive or something, but it’s a little bit more abstract, higher level. And then, yeah, the engineering definitions there are interesting. Photon and torque applied that does helps get people in the mindset of this is sensing the world and taking action in the world. So yeah, very cool. Now, okay, so why should a normal person care about physical AI? Is this about productivity? What is it?

DH: Yeah, I’ll put it in terms I think that a lot of people can appreciate. One of the greatest benefits of what we’re trying to work on in physical AI is actually to give people time back. So, for instance, when my I’ve I have two daughters and the and of course, my wife and I, like anyone, very involved in our daughter’s lives and they were volleyball players. So we carting them around to volleyball tournaments and stuff like that. And I was myself or my wife were driving all the time. And the and there’s this amazing amount of time that you have with your kid when you’re driving them around all over the place, but you’re driving, right? So there’s only so much presence or interaction that you get with your kid at during that time. And being able to build an autonomous vehicle where, yeah, I want to be with my kid as they go to the this volleyball tournament or something like that. But man, there’s an opportunity to be more present with them, right? We can tell stories together, we can watch movie together, we can do whatever during that particular time. And so to me, that’s what this benefit of physical AI is about giving people time back or really at the core, just creating productivity. And there and applying that across this whole world of the of physical oriented industries is going to probably result in one of the biggest growth in global GDP in the history of the world.

Sure, totally. Really cool. Yeah, I like the framing around time as the KPI, if you will, or the value prop. Obviously, time is such a precious resource for everyone. And then, of course, when you think about enterprises, time matters and you can usually quantify that. But I think even for consumers, framing things around time is really interesting. Of course, we see robots dancing or robots making a cup of coffee, and it’s not often a problem that I can relate to, but the problem of like, I don’t have enough time. I think that is pretty universal and very sort of user-centric. So that as a recovering product manager, that resonates with me.

DH: I agree. It’s funny when you I was listening to something recently and somebody framed something in a really interesting way about how to think about the physical AI. And if you just kind of look around the world, things are either grown or mined. They’re then manufactured. They are then moved and then they’re operated in some fashion, right? Yeah. So grown or mined, manufactured and then moved in some fashion or other operated. And the and when you look across all of those things, you think about, wow, if I can improve the productivity of any of those things or whatever metrics matter, that is what the benefit of physical AI actually is to all of those different industries. So that’s transportation, that’s logistics, that’s manufacturing, that’s mining, that’s agriculture. All those things are things that will benefit substantially, if we appropriately apply

Yeah. And that, makes your argument that this could be sort of the biggest impact to GDP and obviously a huge TAM because really when you talk about it’s grown or mined, it’s manufactured, it’s moved. I think supply chains and just anything you can think about from fishing to farming to making silicon wafers and chips you can see to making tractors, you can see how physical AI could be a substrate underneath all of that.

DH: And it’s interesting. I and I agree with you and I like the way that you described it, Austin. The interesting thing for us when I really think about this is I if you take a step back and you just kind of look at global GDP, gross domestic product. This is just what the world spends on creating valued goods. It’s about $115 trillion last year, ish, right? $70 trillion, are related to things that are manufacturing, transportation, construction, these things that are very physically oriented. A lot of physical labor, all those things I was just talking about. And then the other $40 trillion, approximately speaking, is digital, kind of the digital side of the economy. It is things like finance and entertainment, communications, things like that are all kind of digitally oriented. And the very interesting thing when you look at both those things, is you actually recognize the vast majority that we spend today as a world on compute is actually applied to that digital side of the economy.

Yeah.

DH: And matter of fact, if you really dig into it’s about a 10x difference between the right the way that compute is used on the digital side of the economy than it is on the physical side of the economy. And so on the physical side of the economy, if we can bring AI the benefit is really coming from the world of AI. If we can bring AI into the physical economy, into transportation, logistics manufacturing, all the areas I just talked about, and just increase productivity small amounts, that’s a massive benefit to what is the largest portion, two-thirds of global GDP.

Fascinating. Yeah, I’ve never thought about that way of just how such a percentage of GDP is really physical. And I can think back to previous jobs working like for a grocery retailer and thinking about what was spent on compute and obviously a lot of it was just on like the let’s make a web app or a mobile app or the IT side of things. But really very little on anything supply chain, physical movement of goods and everything. And so yeah, I can totally see how we’re in 40% of that GDP, we’re investing in compute, but what about that other 60% or whatever the numbers were, what it’s just like untouched. What could happen? And of course, not only if we apply compute, but now, obviously, in this era, there’s very interesting AI that can help and we can get into that. Not make interesting value happen physically, not just how do you apply AI to solving the problem of how should we, restock our groceries, but what about something more physical? Could you have something restocking groceries at night, for example?

DH: Exactly. Exactly. And do it in a way where there’s an intelligent system that’s behind it that is constantly working on optimizing it and then transferring the optimizations that just recognized there right in the place where the goods are transacting people are walking into a market to buy a to buy something and then up and then upstreaming all that intelligence going, hey, listen, this is what’s hot right now because of whatever demand is interesting in the marketplace. Let’s make sure that we are keeping this well stocked and let’s make sure the entire supply chain is prepared for that. That’s the interesting thing that we’re on the verge of being able to add to this physical side of the economy.

Nice, amazing. So now going back, you mentioned every gram matters and that got me thinking about constraints. So talk to me about constraints in physical AI because it’s obviously it’s different than data centers or a laptop on your desk. We’re talking you’ve got batteries because things are moving, but yet there’s actuators, you’re like applying torque somewhere in the system. So there’s like, there’s definitely power constraints, there’s weights, there’s probably thermals, there’s probably connectivity if it’s mobile. So I’m wondering how physical AI system designers balance all of those tradeoffs. They’re a little bit different. But then at the same time, I’m thinking like, well, this does feel a lot like mobile in some respects, weight, thermals, connectivity. So how are you seeing, the similarities and differences in designing around the constraints of physical AI?

DH: It’s such an interesting question, Austin, and I really like the way you framed it because it is a it’s a total systems engineering problem. And I actually believe that it is the most complicated computing problem ever. This issue on physical AI, just particularly if you look at like a humanoid. And the and this issue, if you just spend a little bit of time thinking about it, you can appreciate the complexity of it. So, imagine you’re trying to build a humanoid robot and what you care about is the precise movement of this little tiny joint at the end of a finger, okay? Just trying to move that thing. And you want to move that thing and you and it’s got to grab something safely, it can’t crush it’s but it’s got to grip it appropriately. It has to actually get in touch. It’s got to make touch with it.

Yeah.

DH: The computing system and then the communication, the computing system through all the actuation that exists through that robotic arm, all the way out to this little tiny actuator that’s moving around the very end of the appendage. All of that has to be coordinated extraordinarily well. And if you’re and if each one of those joints is off by even the smallest amount of tolerances, you’re never going to grab what you want to grab. So you’ve got to think about it from the from that endpoint and bring that design thinking all the way through. This is what I’m trying to accomplish and can I accomplish this in a way in which I actually get this task done. And so these issues of latency I see something, I sense something, I touch something to act to informing actuation, the action that you have to take. And then you have to do that all safely because you don’t want to do anything wrong. I tell people all the time, the thing you never want to do is hallucinate during the middle of surgery.

Right. Totally. Yes.

DH: So these are the constraints, which is what makes it a systems problem and with all due respect to my friends that are building these incredibly massive AI data centers for AI factory kind of data centers. I think the humanoid robotic platform is more complicated because of all those things.

Sure. Yeah. No, that is fascinating of thinking about the systems approach. Lots of actuators, lots of sensors, lots of communication, lots of control, latency budgets, to your point. And then I guess maybe even more broadly, because there is an AI model that’s running on the humanoid in this example, it is even bigger systems problem, which is you got the humanoid, but then you do have the like training of the AI model and the deployment of the AI model. So yeah, maybe say a little bit about that briefly, just it doesn’t feel like just embedded computing anymore. It’s got that AI model piece applied to it.

DH: Yeah, embedded computing for those that aren’t familiar with it. You’re literally writing exactly what you want everything to do. Literally coding it. You’re just writing, please do this, then do that. Whereas AI is very different. AI is about, hey, I’m creating a model that kind of understands all the different things that this device could do and how do I get that device to do the things that I really want it to do by exploiting AI. The interesting thing to me about this is as we have just gotten continuously deeper and deeper into us. And by the way, we’ve been working in these industries at from arm for a very long time. I remind people that just in the last 12 months, we’ve shipped two billion devices in this in these areas that relate to these physical AI application areas. So it’s not we’re not new to us. But the computing has changed quite a bit. And to your point, that question that you’re asking about Austin about different AI models, there if you look at a autonomous any type of an autonomous system, there’s actually four different compute that compute layers that you need to think about. There’s the compute layer that is the layer of motion. In robotics, it’s referred to as locomotion; in autonomous vehicles, it’s about the autonomous motion through the world. And that is very real-time based. I’m driving down the road, I’ve got to break in that in a very fixed period of time, so there’s actually latency matters. There’s a very fast clock that’s going, so there’s a real-time nature to the way these things work. It’s actually real-time system. That is different than when you’re interacting. So I’m sitting in an autonomous vehicle and I’m having a conversation with that vehicle, right? That is a that conversation with that vehicle or I might be asking that vehicle, hey, change my destination, I want to go somewhere else. That’s not quite the same type of real-time system. It’s more LLM-based than what the other one is based. And so there’s even different AI models that you run inside these different types of systems. And then all of that has to hit the compute plane that manages all the actuation. And actually orchestrates all the actuation. It’s got to do that safely, it’s got to do that reliably and repeatably. And then eventually, the thing does connect to that to connect to the cloud because you want to update it, train it bring up new models and stuff like that. So there’s all these different layers of compute. So when you when we talk to people about how we the computing platforms that you want to design that are built on top of arm, a lot of questions come up on, well, exactly which plane of this computer are you trying to solve for? Because models are different, training is different.

Yes. Oh man, that’s that is complex. I can see your argument for it’s the most complex system because yeah, real-time systems, latency budgets, you have to your point of like, you got to get photon and make a decision, hit the brakes if you need to. You’re going 60 miles an hour, it’s got to be so fast because otherwise it’s literally turning into meters until maybe you strike something. But then on the other hand, yeah, I can see like talking to the vehicle is a totally different ball game. So which gets me thinking like, okay, GPUs, CPUs, NPUs, FPGA, like what real-time operating systems, like what are all like the building blocks here?

DH: Yeah, those all of the above. I guess is the best way to describe it. We as a as I tell people, we don’t have specifically dog in the hunt on that. Meaning that you’ve got to design a system, the system has to be able to accomplish something. It’s it and it’s an engineering problem and it’s got to do things in a certain amount of time and there are portions of that workload that you execute on the CPU subsystem. There’s portions of that workload that you actually execute on the real-time safety subsystem that underlies this whole thing. So if the thing goes haywire, it has a safe space to fall back to. There’s workload that you, hey, I’ve got this workload. I’ve got to do it in a fraction of a second and so I need a dedicated acceleration unit to do that could be any one of the things that you talked about. It’s a system design and our experience is and as we all the customers that we talk to in this area, and we talk to the world’s leading, there’s just not a robotics or autonomous vehicle company that of that I haven’t talked to in the last 90 days. All of them have just different views about how they want to go about doing this stuff. And that’s the power of what we offer is that we don’t walk in and say, here you go boom, follow this thing go to our the to this specific conference that we’re going to do and we’re going to tell you exactly what you need to build, you’ve got to use this XX platform to use it. Our approach is to enable people to build what is the right solution for the market, while also bringing the benefit of arm, which is that if you just if you do it on top of arm, you’re just going to get the software compatibility across all the stuff that you’re trying to design. So you get the benefit out of open source contributions and the like because there’s just people are just building on top of the arm ecosystem today. So the we offer that flexibility people because it is just a systems engineering problem as you’re talking about earlier, Austin, and you’ve got to have that design flexibility to design across all of it. And we just like I said, we don’t have a dog in a hunt, you just decide what you want.

Yeah. So we talked about compute and how your goal at arm is really to support customers’ needs, however they see fit with whatever compute blocks they need, and of course, you’ve got the software compatibility on top of that. I’m curious, the entire system, also memory is like a big thing that’s different these days than AI of the past, before maybe it was convolutional neural networks, working on some of that perception, not as big of models. And now we’re in a world where some folks at the frontier are using like vision language action models and things like this that are these end-to-end models and it’s like taking perception, planning and control and to just like a big one big model and it has like big memory requirements. So obviously, compute is part of it, but how are you guys seeing memory constraints changing at the edge? I know at the same time there’s innovations to like make KD cache smaller and stuff like this. But yeah, what do you see when you talk to all these companies? Are they concerned about memory? Do they have enough?

DH: Well, I think the world wish it had more memory right now. Well, true. The simple thing. You can’t listen to anybody’s earning reports today and not know that there’s questions about memory access and the like. But those are constraints that exist at the moment. The so to get to your question, this is what I think this is what I just like the physical AI space and it’s and what’s so interesting to me about it with the kind of the background that I’ve had and things I’ve worked on in my career, is it is all new, right? This isn’t you’re really it’s not a disruption market. This is we’re actually going in and bringing into really well understood areas of manufacturing, mining, all the things I talked about before. And we’re bringing the power of this into that area. And so it’s all new. And so designs are varying. The specific issue about memory is fascinating because it’s this constant tradeoff that people have about, well, how big do I want my model to be based upon the constraints that you talked about and how do I distill that into as small as possible? So that again, I can trade off that this the robot can’t walk around with something a compute system that weighs a ton, right? It’s got to it’s measured in grams, how much it can weigh. And including the memory system subsystem and the like. So you’re constantly making those tradeoffs. So what we find is that what people care about, they really want, they want absolutely as the maximize the value, the performance that they have. Your system has a certain amount of performance in it. How do I maximize that? How do I make it so super fast, which is just architectures that we’re good at. And if memory is a perceived problem for me, well, how do I then reduce my need for it, right? So you can actually you can change data types. Some people use FP floating point data types with lots of bits and those are relatively big for a model. But if you can distill that into another lower resolution data type, but that gives you still sufficient precision to what you want to do, that can substantially change the size of the models that you have. But there’s tradeoffs if you do that. So what we find is that there is no right answer today. This isn’t like this is a world where 20 years of optimizations have existed. This is a world where new models are being invented, new architectures are being explored, new inventions are being decided upon. And there and we will eventually find what are the most efficient of those over the next decade, but it’s going to be a decade to figure all that out.

Mhm. Mhm. Nice. I see. Yeah, so it sounds much more nuanced than just like, you got to have 32 gigs of RAM, otherwise you’re not serious. But it’s much it sounds much more like, this is the system architecture that we think makes sense and so how can we do more if memory is a constraint? How can we distill down into a point where we feel like we have enough headroom, it’s not a constraint or whatever. It does seem like I can see some back and forth between like the ML engineers and maybe like the system designers a little ping-pong of like, hey, I want a little bit higher precision versus like, okay, all the tradeoffs. That’s engineering, right? Like, okay, fine, it’s going to the bomb’s going to go off.

DH: That’s my day.

I bet. I bet. Every day, that’s awesome. Okay, I have another question for you, Drew. So, when we’re talking about these systems, so take the car, for example, there’s obviously some real-time stuff that has to happen. It has to happen there at the edge. And then there’s some LLM inference happening. Like, how do you see like edge to cloud kind of hybrid playing out across physical AI? Like, will we see some stuff runs in the cloud, some stuff runs locally? Like, how are companies thinking about this?

DH: Yeah, I think it is going to eventually it’ll distill into kind of both those things. It’ll be very use case dependent. So, let’s go back to the car example. I don’t see anytime soon or ever that a the braking system of a car is going to be something that’s figured out in the cloud.

Mm, right.

DH: Right? It’s just it’s too there the needs are too instantaneous. And so why would why in the world would you say, hey, listen, I’m going to add a requirement of latency, ping time to a cell tower and then the actual time back to the computer. I’m not you’re just not going to do that. So there are things that without doubt will remain local. Then you kind of get into, well, but every really the most advanced autonomous cars today that we see on the road, everything from what Tesla’s doing to what Waymo’s doing, to what the Wave guys are doing, the Rivian team these are all just world-class teams. They are looking at, okay, there’s a portion of this work that I’m going to do in the cloud. Like, for instance, I’m training all my models in the cloud. So I’ve got a new model. I’m going to download that to the car off offline. So that there’s a connection that exists. And then when you start thinking about robotic systems, you start thinking about, well a humanoid robot will be doing work, but it’s probably let’s project it into the work in into enterprise and into businesses. It is probably part of a fleet. So that fleet then has fleet management that’s associated. Okay, well, this robot needs to go is going to go over there. This robot’s going to go over there. It’s going to do these different tasks. And so there’s a coordination that ends up happening where you want to actually have the system be able to do all the autonomous work that it needs to be able to do, but then recognize that, hey, I want to there’s coordination that needs to happen. And so there’s intercommunication that needs to happen. That intercommunication is going to be cloud-based. And so that’s where I think it becomes a tradeoff. The interesting is when you actually start thinking about logistics centers and factories and how they evolve because in a if you’ve got if I’m in a logistics center like the center that’s shipping packages, goods in, goods out kind of thing. That’s the four walls that make up that building are under the control of the systems designer that’s designed that whole system. And they’ve got robots that are all over the place inside those things today. But you may make decisions that, hey, listen, because I’ve got a whole bunch of robots that are there and I might have central compute that’s available to you. There are some things that I might actually allow the decisions to be done in a compute system because it’s close enough from a latency standpoint. And that’s where that’s the thing that just boggles my mind about this world of physical AI is it’s not just about robots. It’s not just about autonomous vehicles. It’s about how all of that then interacts in those things I talked about before, growing crops, mining, manufacturing, moving, logistics centers, all that stuff. There’s a level of designing it as a at a systems level that is as complicated as just dividing the designing the system the robotic platform itself.

Yeah, fascinating.

DH: Right?

Really interesting to think through. I know it’s like kind of mind-blowing to think through like the fleet level orchestration. Obviously, LLMs and AI will be very helpful there. And then you yeah, you start to get to this like, how do you define the area of responsibility? Like, what is the robot allowed to do for itself? Even making that work is awesome. It can walk over and pick up stuff. And what decisions can it make versus what decisions does it need to send up to like the central planning brain? And then of course, and then this is interesting because then now some organizations their like differentiation and how they compete with others could be not just the platform that they’re using, but that higher level orchestration. Like, are they really sophisticated about getting goods in and out moving, making interesting decisions?

DH: Yeah. Yeah. For instance it’s you can just think about a conveyor system, right? A conveyor system is moving goods down the line. And then maybe there’s a robot there that’s sorting goods and things like that. At some point, someone’s going to go, hey, you know what? Let’s design those two things together as a system. Let’s not just stick a robot in front of a conveyor system that’s there. Let’s actually, let’s take a step back and think about all the coordination of those things that happen. That’s this orchestration that occurs.

Interesting. So interesting. Okay, so let me ask you a question then. In where my mind is going is like, okay, let’s say we’re in this world where there’s robots, but there’s also like a data center based model, maybe doing some orchestration. Like, is there a benefit? Does architectural consistency matter? Is there a benefit if I’m running arm in the robot, running arm in the cloud? Or how do you guys think about that?

DH: Yeah. So first off, the answer to the question is, do we see that happening? The simple answer is yes, we see it happening all over the place. And it’s following the same pattern that we saw even in the as people build large cloud data centers if you take a step back and just kind of look, just squint and look at that large cloud data center, it’s just a big computing system, right? And so the Amazon guys when they when we worked with them to build Graviton, they just knew the workloads that they were building that they were running. They knew the power constraints that they had. They knew the IO requirements that they had. Everything when you’re at designing at that level, not just trying to piece something out of what’s available through a bunch of supply chain points. When you’re really designing that, you really care then about how your compute plane looks. So you actually start looking at, well, how do I customize that to my particular needs? And so they were they all voted with their feet. Every major cloud company’s already decided to do that. We’re seeing the same thing now with the happening with the autonomous drive space. Tesla early first mover on that, right? Tesla is super famous for the fact that they build everything themselves. Rivian if you’re a AI first company like Rivian is, you go, listen, I care about latency. I care about the software load that I have. I care about the data types that I care about. I really want to have something that’s specific to my designs. And so RJ and that team, they just said, we’re going to go after it. We’re going to build our own platform. And now as we see this in the just as the physical AI space into other types of form factors, not just cars, autonomous cars, which are robots on wheels. Other things are happening that other seeing the same thing in other areas around the world, other companies around the world, just going, I just I need to make a decision about when something is so unique to what I’m doing, I want to make sure it’s part of my vertical supply chain. Now, there’s a lot of people that are building really powerful things using technologies that from folks like Nvidia, which is amazing. Their technologies are amazing. And they’re really satisfied with that because it solves the problem that they need. Nvidia does great work. And then there’s people like the Rivian guys who are going, listen, I just I want to do something that’s a little more specific to what I need and I’m going to design my own thing. So we’re seeing it all over the place.

Yeah, interesting. Okay, so maybe last thing, as you’re talking to all these leading robotics players, is there something that comes up in every conversation that maybe you’re surprised by or that we haven’t been thinking about us in the audience?

DH: What’s interesting to me is and surprising and I guess somewhat validating is that everyone I’ve talked to has said, Drew, I’m convinced that we will have you will not be able to satisfy the amount of compute that I need at any time. So please just keep building more and more compute systems. Because they just take a step back and they go, okay, well, maybe we solve the problem about how a car drives around the world, but then I got to put people in the car and they and we want them to have this natural experience being able to talk to the car and give the car direction, and that it could take action on and have it actually infer through hand gestures and the like. So there’s these everyone comes back and going, it’s a decade or more of high performance. Matter of fact this is I was joking recently and I think this is serious. I was talking to a friend of mine. I was thinking back to the days when I was doing the GeForce business and we had this world of computer enthusiasts, right? The modders, the ones that are building their own computing platforms and the like and tweaking it and overclocking and all that stuff. Austin, I’m sure you were one of them. That is I think it’s going to happen to robotics. Sure. I think we’re going to have people that are going to get robotic platforms and they’re going to want to overclock it. They’re going to want to tweak it. They’re going to want to do their own optimizations on it. We’re in that type of a world where people are just going to want more computers than they can possibly get it and they’re going to even try to figure out how to hack it for themselves.

Yeah, that makes a lot of sense. And speaking of GeForce, it does kind of remind me of like computer graphics where it’s like never enough. It’s like, wow, this looks amazing. Can we make it look even more real? Can you give us even more compute? And the same thing. I don’t think people are going to be like, okay, this robotics platform is useful enough you can stop here.

DH: Yeah, right. No, I think the analogy is perfect. And where are we now in that? Everything’s neural graphics based, right? We’re using AI technologies now to in to through AI generate frames that don’t exist. It’s not rendered, it’s actually completely AI generated the frames that insert between other rendered frames. And so the so you’re absolutely right. This is one of those spaces where people will thirst for as much computing as they can possibly get.

Awesome. I love it. Well, Drew, thank you for this. This got me fired up about physical AI. I look forward to seeing what you guys at Arm do over the next couple years and I’m sure the listeners walked away learning something. So thank you.

DH: Austin, it’s always good to see you, man.

I was pumped to see it. This is right up my alley! It’s AI accelerator startups + a strategy puzzle of sorts... and requires us to get into the technical nitty-gritty a bit. Does it matter to AMD’s competitiveness? Stock price?

Taalas builds what it calls model-specific silicon. Rather than loading weights from memory into a programmable accelerator, it hard-codes a specific model’s weights directly into the chip as read-only memory. The result is fast, cheap, and completely inflexible. Which is the point! Its first chip, the HC1, runs Llama 3.1 8B and nothing else. And that’s the point...

Also note the HC1 is referred to as just a tech demonstration.

Btw, Bajic spent 10 years at AMD, then later co-founded Tenstorrent and ran it as CEO and CTO before starting Taalas in 2023.

So AMD paid an undisclosed amount for a company with no revenue, no named customers, and one chip that runs one open-weight model. But weight wait! The idea has merit once you dig into it!!!

Let’s unpack it.

If you’re new here, skim this useful background from the Chipstrat archive so you can follow the narrative arc:

• The Rise of Groq: Slow, then Fast (Jun 2024). How an SRAM-based accelerator found its market, and why ultra-fast inference turned out to be a business rather than a demo.

• Right Tool for the Job (Dec 2025). Why Nvidia bought Groq to fill an ultra-low-latency gap in its portfolio.

• The Multi-Silicon Era Is Here (Mar 2026). Nvidia formally embracing disaggregation, and what it opens up for multi-vendor datacenters.

• Right Systems for Agentic Workloads (Jan 2026). Why low-latency accelerators and long-context accelerators are different machines.

• Right-Sized AI Infrastructure. Marvell XPUs. (Oct 2025). Right-sizing infrastructure per workload instead of buying one monolithic platform.

The overarching question we’ll attack: does this change how we should perceive AMD’s competitiveness? And it ultimately comes down to whether you believe ultra-high interactivity is a real and profitable product tier. Jensen does....

Of course the whole story is a bit juicy too. Recently at Advancing AI (July 23rd), AMD announced a partnership with Cerebras that is also a very-fast-tokens play. But then AMD agreed to buy Taalas fourteen days later!? What to make of that?

Let’s talk through:

• Why agents want fast tokens

• The interactivity revenue curve

• Why GPUs can’t get to high interactivity

• What Taalas is

• 🔐 Speculative ways to use Taalas.

• 🔐 The IP AMD is buying

• 🔐 The semi-custom inference cluster

• 🔐 How much context actually fits in HC1

• 🔐 Enterprise and edge?

• 🔐 What this means for Cerebras

• 🔐 So is AMD cheap here?

Interactivity is a product tier, and it has a premium price

Remember when Groq CEO Jonathan Ross did the real-time LLM demo on CNN on Valentine’s Day in 2024? That was awesome. Demo is like 2 minutes into the video. Watch to the end.

Real-time voice was a fun demo and shows the power of super-low-latency tokens. But agents are the reason the fast token market is blossoming.

In the chatbot days, humans were the bottleneck; we can only read and interact so fast. But agents operate much faster! And as a result, they generate way more tokens. And even though the machine will run for days, of course we humans still want to see all that agentic work done in a reasonable time.

To illustrate, a 50-step agent chain generating 500 tokens per step is 25,000 tokens of sequential work. At 200 tokens per second per user, that’s roughly two minutes. But at 1,200 tps that’s only ~20 seconds. And at 12,000 tps, it’s ~2 seconds. Which one commands a premium? Not the slow two minutes of tokens...

To pique your interest, Taalas claims 16,960 tokens per second for a particular use case!

Nvidia says customers will pay a lot for fast tokens

Look at Nvidia’s modeling of the value of fast tokens:

The x-axis is interactivity aka tokens per second per user. Faster is better.

Notice Nvidia shows five service tiers of sorts in dollars per million tokens.
Free $0. Medium $3. High $6. Premium $45. Ultra $150.

So the same million tokens sells for 25x more at the Ultra tier than at High.
That’s the premium for speed!

So why isn’t everyone selling Ultra tier tokens? Because GPUs can’t get there.

Blackwell NVL72, in all its liquid-cooled, state-of-the-art scale-up domain glory, can’t hit 600+ tokens per second per user. It. just. can’t.

GPUs can’t do super fast tokens

Why? Communication latency. Collectives latency.

Recall that decode is memory-bound; every token means streaming the activated weights plus the KV cache out of HBM before doing a little math on it. Of course, useful models don’t generally fit on a single GPU’s HBM, so we shard it across more GPUs.

Tensor parallelism helps here, since each GPU reads a slice of every matrix and you get that bandwidth back in parallel. The tradeoff is latency, because now the GPUs have to talk back and forth for every layer, every token.

Take a big MoE like DeepSeek-V3 at 671B parameters. Its config tells us it’s 61 layers, with the first 3 dense, so 58 MoE layers, 256 routed experts, 8 active per token.

Tensor parallelism requires at least one all-reduce per layer. Then each MoE layer adds two all-to-alls on top; one to dispatch each token to its experts and one to gather the results back. That’s something like 116 all-to-alls plus 61 all-reduces, so call it ~175 collectives per token. Don’t forget that a collective means all the GPUs wait for the slowest one to respond...hence needing low jitter…

So how long does it take a collective to process? Measured small-message collectives over NVLink land in the single-digit microseconds, roughly 4 to 9 µs depending on the protocol.

So 175 collectives at 4 µs is ~700 µs per token. And that’s just the GPUs talking to each other… before any math happens.

Granted, engines overlap some of this with compute, but a collective requires synchronization between all GPUs so there’s a floor on how much you can overlap.

To be fair, this is single-user decode napkin math and tricks like speculative decode improve interactivity. But still, there are limits to a GPU-only path unlocking the “Ultra” tier above.

Well then, what to do?

Nvidia showed the way. Prefill on Vera Rubin NVL72 and part of decode on Groq’s SRAM-based LPUs.

(See more in Right Tool for the Job and The Multi-Silicon Era Is Here. Disaggregation makes this possible.)

Which unlocks that blue circle:

Yet AMD’s pareto curve would be like the Blackwell NVL72 curve. It would stop short of the blue circle.

So AMD addressed that. But not with a $20B deal like Nvidia. Rather, a partnership, a disaggregated system with Cerebras: Helios does prefill, the Wafer-Scale Engine does decode.

Why not buy Cerebras? AMD doesn’t have that kind of money, and Cerebras is public and too expensive anyway. And maybe there’s better long term solutions anyway…

But then fourteen days later AMD announced an agreement to buy Taalas!

Subscribe now

What Taalas is

Taalas came out of stealth in February 2026 with the HC1.

TSMC N6, 815 mm², 53 billion transistors, and no HBM or external DRAM. Everything the chip needs sits on the die. Sounds good when memory prices are so dang high right?

The model weights live in a mask-ROM recall fabric burned into the silicon. A smaller SRAM holds the KV cache and any fine-tuned weights. Hmm, what about long contexts?

No advanced packaging either. No CoWoS bottlenecks. Cheaper.

About 250W in a PCIe card, in racks that draw 12 to 15 kW and run on air. Deploy anywhere.

It runs Llama 3.1 8B. Hmm, edge model? Draft model? Hmm…

Changing models means changing two mask layers, which carry both the weights and the dataflow, on a two-month turnaround.

The idea stems from earlier tech. From Sally’s article:

Taalas is borrowing some ideas from the structured ASICs of the early 2000s to make its hardwired model-specific chips. Structured ASICs used gate arrays and hardened IP blocks, changing only the interconnect layers to adapt the chip to a specific workload. At the time, this was seen as a more cost-effective alternative to a full-custom ASIC that was more performant than an FPGA.

“There are definitely parallels,” Bajic said. “It’s a similar idea to eASIC and gate arrays, but the underlying technology looks really different.”

Of course, the eye-candy: 16,960 tokens per second per user at 0.75 cents per million tokens. OK! That’s FAST!

Of course the figure is self-reported, the model is quantized “aggressively” by Taalas’s own description, and the benchmark point is 1k/1k input/output. That’s small input/output, an important detail, we’ll dig into it below.

The customers? Bajic from EETimes:

“The big thing at the root of this idea is the assumption that the customer is willing to commit to this [chip/model] for a year. There will definitely be a lot of people who won’t, but some people will.”

Did you notice how Nvidia’s chart with our blue circle, it topped out at 1,000 tokens per second per user and put $150 on that tier? Taalas claims 17K tokens per second on an 8B model at 0.75 cents per million tokens. How much is that worth? Well, there are lots of asterisks.

So is Taalas a Cerebras replacement, or something else?

Is there a job an 8B chip does better than anything else on the market?

What about long contexts? Is this dead-on-arrival?

What about AFD?

Hmm, did AMD buy a product, or a hiring pipeline?

And does any of this change what AMD is worth today?

Let’s get into it.

So what do I make of it?

Read more

That makes pricing complicated; it’s not simply “charge whatever you want”. You have to know what your product is worth to the customer (that alone can be hard to measure) and then decide how much of that to take. Take too little, and you’re leaving shareholders’ money on the table. Take too much, and you’ll anger customers and incentivize competition. And regulators start paying attention...

TSMC historically played it safe, erring on the side of undercharging. In return, they had happy customers with no reason to seek an alternative. That’s changing.

TSMC is raising prices. It’s part of their strategy; colloquially, “selling our value”. This “selling our value” is listed among their six profitability factors. Of course, TSMC prefers not to explicitly talk about raising prices; for example, the last call pointed to improved utilization and reduced costs for margin gains. We are better at manufacturing, and that’s why margins are up. Of course, gross margin rose 9.1 points, and manufacturing performance explained 3.2 of those points. The other 5.9 points? Pricing. Highly recommend you read more here.

So what about ASML?

As an earned monopolist, do they approach pricing like TSMC?

Yes and no. Both call it value-based pricing, and ASML says so in those words. But ASML measures that value with a much narrower yardstick, and unlike TSMC, it’s happy to discuss the whole thing on an earnings call.

So, buy or sell?

Well, will ASML start charging for value?

Management spent this quarter hinting that it will. Consensus assumes it won’t — and the way the street has 2027 modeled, ASML’s customers end up paying less per wafer of capacity than they do today. That has never happened…

Here’s what this article will cover.

1. How to model ASML.

2. Does ASML charge for its value?

3. What’s changing. April: “not the way we do business.” July: “more flexibility for pricing.”

4. ASML vs TSMC.

5. Four things that lift 2027 revenue.

6. Price per unit of capacity. The 2027 decline consensus is implicitly forecasting.

7. So is it cheap? What 2027 looks like if the ASP moves, what it looks like if it doesn’t, and what today’s price already assumes.

8. What breaks it. Three risks.

So: does ASML start charging for that value, or doesn’t it? Below is how I’d answer that, and what each answer is worth at today’s price.

Read more

In the pandemic-era supercycle, TSMC spent 53% of revenue on CapEx. In the AI supercycle, with roughly $100 billion a year of operating cash flow, they’re spending only 34-36%:

Matching the 2021 playbook today would mean another $30 billion a year. What to make of that?

Maybe TSMC got burned by 2021-2023 and ~35% is the preferred comfort level for management. Or it could be a lack of faith in the long-term demand?

TSMC has arguably the best ability to ascertain true demand given the breadth of customers, and talking to customers’ customers... So why aren’t they investing more? What did Wei see?! ... jk

Let’s pull out a few interesting questions from TSMC’s great quarter.

What’s in this piece

• N2 has arrived

• Gross margin peaks, gross profit keeps growing

• 🔐 Digging into the CapEx gap

• 🔐 What happens if they underinvest

• 🔐 Pricing… no talk, but it’s in the margins

• 🔐 What to watch going forward

N2 has arrived!

N2 debuted at 3% of wafer revenue this quarter:

Only 3%? Look at that baby little red bar in 2Q26... But interestingly every node debuts at about a billion dollars of revenue:

Mix share shrinks because TSMC’s revenue keeps growing.

Given that wafer prices rise every generation, flat debut dollars implies shrinking debut volume (wafer prices are rough, just for illustration):

Volume shrinks because each new node has more layers and longer cycle times, so the same capacity ships fewer wafers in its first ninety days.

So N2’s debut is the fewest, most expensive wafers TSMC has ever sold. Customers pay the highest price per wafer ever, and these wafers cost TSMC the most to make, with yields still climbing and a new fab’s depreciation spread over small volume.

Interestingly, N5 is still the biggest node in the portfolio (33%), even though it’s six years old:

Why is such an old node still so dominant? AI. The answer is always AI. 😂

N5 is the AI node. Chips ship on whatever node was mature when they were designed two or three years earlier, and today’s AI fleet (Blackwell on N4P, Google’s TPUs, Trainium, most custom XPUs) was designed in 2022-24, when N5 was the obvious choice for big dies needing proven yield.

It’s interesting how long each node family hangs on:

Of course, TSMC’s “5nm” node isn’t one process, it’s a node family. N5, then N5P, N4, N4P, N4X. So what’s six years old and 33% is the family, not the original process.

On this earnings call, C.C. said A14 will be “an even larger and long-lasting node for TSMC than N2, just like a 2-nanometer technology is a larger and longer-lasting node than 3-nanometer”. Each node is a decade-long product line, each bigger than the last.

Rubin-class accelerators move to N3 through 2027, hence N3 getting three new fabs. TSMC even approved three greenfield fabs for N3, a four-year-old node. One in Taiwan, one in Arizona, one in Japan. New fabs for kind of old nodes.

The longer a node family hangs on, the better the margins. N3 is “very tight,” its gross margin set to “cross over to the corporate average in second half 2026”, and its 2022 tools depreciate off in 2027 (equipment runs on a five-year schedule). Sold out, above average, depreciation-free.

Given that fabs are a ten-year+ asset, today’s CapEx is TSMC’s opinion on demand for the next decade...

Gross margin peaks, gross profit keeps growing

Gross margin printed 67.7%, a fifth straight record. But Q3 is guided down to 66.0%.

Reminds me of an MBA professor that I had, who told our class you bank profit dollars, not margin percentages. i.e. don’t sweat the margins too much

And the dollars keep climbing. Gross profit was roughly $27.2 billion this quarter; the Q3 guide implies about $29.8 billion. Annualized, that’s about $120 billion a year of gross profit.

Why the margin decline? N2 ramp. TSMC guided Q3 margin down “primarily as we expect the steep ramp-up of our 2-nanometer technology to dilute our gross margin by about 3 to 4 percentage points”. That’s worse than the 2 to 3 points guided in April, because the ramp got steeper. But steep ramp and more dilution sooner means more premium wafers sooner! So it’s fine.

The 67.7% gross margin might be the high-water mark until roughly 2028. Through 2027 there’s more margin drag as N2 dilutes for about eight quarters, A16 ramps behind it, and Arizona Fab 2 arrives 2H27. But if Q3 prints margins above 66.5%, well then, it would mean N2 pricing is covering its own ramp. One way to get pricing hints...

The last margin peak, 62.2% in 4Q22, was largely an FX bump at a cyclical top, and it stood for eleven quarters because demand then collapsed and margins fell. This peak is different! It comes from loading a new node as fast as physically possible while demand is spiking. So we’ll see another little mountaintop on the margins chart, but for an opposite and much better cause.

What paid subscribers get in the rest of this piece

First we’ll digging into the CapEx gap to figure out how much they could invest but aren’t. Then we’ll talk through the implications of TSMC underinvesting.

Also, pricing… TSMC doesn’t credit it to margins verbally, but it’s there. We dig in.

And what to watch going forward.

Thoughts, numbers and charts. Carry on:

Read more

Things we cover:

• Inventing the active optical cable with Mellanox

• Why hyperscale AI’s error-free requirement changed the game

• GaAs (unconstrained) vs. InP (substrate-constrained)

• The three 1.6T flavors: 8×200, 16×100 LPO, 32×50 NRZ

• The roadmap to 3.2T and 12.8T via bi-di and 2-D arrays

• The fabless model and WIN Semiconductors capacity

This podcast is lightly edited for clarity.

Subscribe now

From HP Labs to the First 10 Gigabit Ethernet

Austin: Hello everyone. Today we have a special guest, Al Yuen, CEO of PicoJool. PicoJool is an optical connectivity company and we’ll get into all the interesting details. But first I wanted to introduce you guys to Al. So Al, tell us about you and your background — I know you’ve been in the industry for a long time.

Al: Yes. So after grad school at UC Santa Barbara — where a lot of the photonics folks have originated — I went into HP, HP Labs, where we focused on photonics research. And then around ‘99 I left HP and started my first company, called Alvesta, and we created the world’s first 10 gigabit Ethernet. At the time, 10 gigabit — which is 10 billion bits per second — we were actually trying to figure out applications, how people would use this in ‘99. We would make up things like, people will want to stream video someday in all the rooms in their house. But today, of course, we’re doing 1600 gigabit, or 1.6 terabits. So since then I’ve gone to various companies. I ran a division for Coherent, then started some other solar and clean-tech companies, and finally ended up at Lumentum in my last gig. And about two years ago, through Playground Global — which is our funder — we started PicoJool. So far so good on what we’re doing, which is basically in the interconnect space, and we’ll talk more about that today.

Austin: Awesome. Wow, what a great background. I love that you guys invented early 10 gigabit Ethernet and then had to create ideas to sell people — to convince people that yes, this is useful, people will want to use it. That’s awesome. Now, remind me — you also helped invent the active optical cable. Is that right?

Al: Sure. Very interesting story. Back in Alvesta, my first startup — another small startup at the time called Mellanox, which of course now is inside Nvidia and created this whole hyperscale and the whole InfiniBand ecosystem — they approached us. They had these very bulky copper cables, even back then, 25 years ago. They said, the copper cables are very bulky, they could only reach tens of meters — now it’s even shorter. We’d like an optical option, but we don’t really want to commit to a full optical solution. So could you put the optics inside the connector? And we said, why not? So we took that same transceiver that’s typically on a board and literally embedded it right into the connector. Then we thought, well, that’s not very efficient and clean — so we embedded the whole thing inside; the optics went inside the connector. And this is the world’s first demo of an active optical cable, meaning electrical to electrical: electrical comes in, electrical goes out, but inside there’s the electrical-to-optical transition. Electrons come in, photons carry the information, and electrons go back out at point B. So that’s how the whole active optical cable concept came through Mellanox. And since then, for the last 25 years, the AOC has been the standard workhorse in many, many data centers.

The Engineer’s Mindset and Why Optics Must Exit the Rack

Austin: Amazing. So having invented that and then watched it become widely adopted — produced in the millions of cables — how has that impacted the way you think about what’s possible as an entrepreneur in this space?

Al: I like to think of ourselves more as engineers. There’s a difference between scientists or researchers — what people call R&D — and engineering, or product development. I always tell people I’m more of an engineer. Engineers solve problems and want to create products that meet specifications. If you need a Prius, you certainly don’t design a Ferrari — that’s overkill. The product fits the need: spec, cost, reliability, and in our case reach, or amount of power. So what we do is really look at solving the specific problem. Today, copper has shrunk to about 3 to 4 meters of reach at 200 gigabits per second per lane. And therefore they can’t get the information off each rack. Racks and racks of GPUs, CPUs — lots of compute power — and these racks are getting very hot because they’re packing more and more GPUs per rack, because they can’t exit the rack. To exit the rack, you need an optical solution. There are many technologies, and VCSEL-based active optical cables are one of them — that’s what we’re focused on. Low cost, highly reliable — going way back to our roots, replacing a copper cable with an active optical cable. So the problem really hasn’t changed; it’s just that the speed and aggregate bandwidth is now 1600 times more than the old 1 gigabit Ethernet. It’s exciting. It’s been a long journey, but we’re still at it, and we see a long future for VCSEL-based technology going forward.

Austin: Nice. So you have this history of thinking like an engineer — what is the problem right ahead of us in the industry? Not “we need to invent new physics,” but how can we be thoughtful about it. Same cable form factor in the AOC case, electrons in, electrons out, but we could use optics here — could we put the transceiver on the end of the cable? Thinking very pragmatically. And now, fast forward 25 years, in 2024 you started PicoJool. It sounds like you’re tackling the problem again of communicating lots of information — this time GPUs talking to each other, rack to rack, where it’s such high bandwidth that copper is shrinking to 3 meters or so. So you’re again thinking about how to use active optical cables, but this time with VCSELs. So tell us — why VCSELs, and what about the problem made you want to start PicoJool and actually get in the game?

Why VCSELs — 25 Years of Shipping in the Millions

Al: Great question. VCSELs, for one thing, have been around since ‘96 — a technology that’s been in product, in the field, in data centers since 1996, starting with the first gigabit Ethernet. I’m more of a historian today: here’s a 1 gigabit Ethernet that HP created, and at the time Honeywell — very early companies that did 1 gigabit. And from 1 gigabit all the way to today — 1600 gigabit, what they call 1.6 terabits — it’s all been VCSEL-based. The practicality of these solutions has to meet capacity, demand, cost — all of it. If you’re trying to get into a hyperscaler today, you need to meet the whole checklist. You can’t say, I meet everything but it’s three times the cost of your target; or everything’s great but I can’t get the reach. They want all of it. And right now the only thing that meets it is copper — mainly driven by cost. Cost for copper is very minimal. Even active copper, where you have some signal integrity and signal processing built in, like an AEC — active electrical cable — the cost is still quite minimal compared to more elegant, longer-reach technologies.

Now, what’s important in a data center: when you say “fiber optic communication,” everybody hears it and goes, oh, isn’t that the over-the-ocean stuff, the transatlantic subsea cables? Absolutely — there are many flavors of optical communication; it’s a very big umbrella. What we’re talking about today for data centers is a very short reach — today they call it scale-up. It’s basically a row, typically 25 to 30 meters, or in English terms 75 feet. Very short — within your house, from one end to the other, or potentially even shorter. And in that short reach, the shorter the reach, typically the higher the volume. In the data center you’re connecting all these GPUs, CPUs, and ASICs together — millions of interconnections, miles of fiber inside a data center that may be a football-field length. But when you leave that data center and go between cities, buildings, countries, you have fewer fibers but need longer distance. So back to the data center: you have to have millions-per-month capacity to meet the volume demand.

That’s one of the boxes — you’ve got to check off all the boxes. If you demonstrate one or two racks, or go to a show and demonstrate just the technology — technology, science, R&D show the capability — that’s a demonstration. But to ship in volume, millions per month, you need this whole ecosystem: the connector, the transceiver, the sockets — everything has to be millions per month. And any one of those bill-of-material parts, if it’s missing, then you can’t ship in millions per month. That’s what’s happening with a lot of these new technologies that are fantastic from a demonstration and capability standpoint going forward. But VCSEL’s domination is that for the last 25 years we’ve been shipping in the millions per month. So there’s no invention of a technology, capacity, or foundries needed — we already have that. So we design the latest VCSEL — today it’s 200 gigabit, which we just announced — put it into the whole ecosystem, they package it together, and voila, we can build millions per month without waiting for machines, or even buildings, to be built up, then new machines, then new processes. All of that has existed for the last 25 years to service this short-reach data center application.

Austin: Okay, let me reflect it back to you. The problem you’re solving is scale-up with optical interconnects. But the key insight — with your pragmatic hat on — is: how can we use technology and a supply chain that already exists and can already ship millions of cables, components, and parts per month? That’s one reason VCSELs are so attractive — they’re not a new technology, the supply chain is not new. So even though we hear about all these other interesting things — the Broadcoms, Coherents, Lumentums; silicon photonics, EMLs, micro-LEDs — you’re saying, hey, don’t rule out VCSELs. Even though they’re not new, there are advantages. But if VCSELs have been around so long, why aren’t other folks trying to take them to 200 gig per lane, times eight lanes, 1.6T and beyond?

The Hyperscale Shift to Error-Free

Al: Yeah, like you said, there are multiple technologies — silicon photonics, EML, even micro-LEDs — all vying for this 200 gigabit electrical signal coming in. From the ASIC, GPU, or CPU you have 200G per lane of electrical signal. When you go to the electrical-to-photonic (E-to-O) transition, it can go to different lanes through a different IC that may be a gearbox. You don’t necessarily have to run at 200G straight through — which we can — and that’s the most elegant: straight through without an intermediary gearbox saves power, cost, and latency. So 200 gigabit would be ideal.

But what’s changed in hyperscale is different from Ethernet. Ethernet information is sent in packets. Whatever information — “where should I go in Italy, I’m going on vacation” — gets divided up by your search engine, sent in packets, and comes back to you. We’re not very cognizant if there’s an error drop or some delay, because those are in the hundreds of nanoseconds or milliseconds — to us it’s a thousandth of a second, we don’t notice. But for hyperscale systems, which are literally thousands of GPUs acting as one brain — one supercomputer, one high-performance cluster — that latency is super critical, because the GPU notices anything in the tens or hundreds of nanoseconds. So the typical Ethernet bit error rate needs to drop from 10⁻⁶ to below 10⁻¹⁰, what we call error-free, because any errors slow down the whole training and inference — all the AI infrastructure. That’s changed, and it’s allowed these higher-cost single-mode solutions — single-mode being longer distance, longer reach, very high performance — to come into the data center and hyperscale systems, because of this requirement for very low bit error rate. And now VCSELs have to raise the bar from 10⁻⁶ (typical Ethernet the last 25 years) to 10⁻¹⁰, 10⁻¹². And we’ve done that — we’ve pushed our VCSELs to 200 gigabit, and even used at 100 or 50 gigabit NRZ you can leverage that to very low bit error rate. So the answer is: things changed for hyperscalers to require error-free, and that’s allowed these high-end single-mode solutions to compete directly with VCSELs, because of that additional spec that’s new to hyperscale AI systems.

Austin: Okay. So because the capacity for sustaining errors is much lower — we don’t want all these GPUs just waiting — that changes the game from the cloud/SaaS days to now, when everything’s acting as one big computer. So the bit error rate has to be much lower. And VCSELs, which are short reach, have to come down to meet that; or, what you’re saying is, these other technologies that are longer reach and higher power were already closer to the necessary bit error rate, so people say, why don’t we bring those into short reach? But surely that has power and cost tradeoffs — taking something that talks at low error rate over a long distance and trying to bring it in. So you guys must be taking a different tack: no, no, let’s just make VCSELs error-free. And presumably that’s a cost or power tradeoff you’d rather make — or is it back to the manufacturing supply chain capacity?

One Pizza Oven vs. 5,000 Pizzas — The Capacity Argument

Al: Yeah, it’s again a complicated matrix of items you have to check off. Those shipping silicon photonics and EML for long reach — or DFBs — single-mode, high-performance devices, have been around a similar time, 25 years. They’ve been used for long reach because of that super-high performance, very few fibers, and what we call WDM — wavelength division multiplexing. Fibers are very expensive when you’re going hundreds of kilometers, so you want to use very few fibers but pass more information through more wavelengths, more colors, in that same fiber. For short reach — tens of meters — you’re not as locked into the cost of the fiber; it comes down, because you’re only 10 meters versus 10 kilometers. So the volume supply chain of the high-performance, low-bit-rate device would seem a natural fit: hyperscalers want low bit rate, let’s go with the Ferrari — the super-high-speed, high-performance single mode. But those players have been used to building in maybe 100K — 100,000 — volumes, because you don’t need as many between cities and countries. And all of a sudden they come into the data center. Even though their performance is excellent, the cost is a little higher because of the single-mode packaging. And the infrastructure to build millions per month is 10, 20, 50× what exists. That’s brick and mortar. It’s like: I only have one pizza oven, and I’ve been used to a small clientele — maybe 20 an hour. Someone comes in and says, I’d like to order 5,000 pizzas, and I need them in an hour. You’re going, I need 100 pizza ovens. That’s the exact same problem the high-performance single-mode long-reach players are coming into. Silicon photonics, EML — excellent technology, very good bit rate — but the infrastructure needs to be built up. That’s what you’re seeing: a lot of announcements with people holding shovels, saying we’re investing in the next supply chain, the buildings. That’s great — bringing more manufacturing to the world, and back to the US, all great for the industry. But VCSELs have been around for 25 years and shipping in the millions. So we’re not building, we’re not putting shovel to ground — we’re just changing the actual VCSEL.

We’re just changing the actual VCSEL performance — the chip — and leveraging the existing infrastructure. And so for VCSEL capacity, we call it unconstrained. Constrained means they’re sold out: we’re sold out through next year; if you’re going to place an order, it’s going to be an eight-month, 18-month lead time — a year and a half from now we can get it to you. For us, unconstrained just means we have a certain lead time that’s limited only by our cycle time of building through the factory. A VCSEL run and then a packaging run may be four, eight, twelve weeks — but that’s limited just by the fact that we have to build it and ship it, not by the constraint of the supply and ecosystem of machines, or pizza ovens. We have plenty of pizza ovens: place the order, and we’ll get you your order in the cycle time we commit to.

Austin: Gotcha. Okay, that’s very interesting, and a great competitive advantage for you. So on the constrained side — is that what listeners hear about when they hear about indium phosphide being a bottleneck? Where in the supply chain is it constrained? And for you, what’s different about VCSELs that makes them unconstrained?

InP vs. GaAs and the Fabless Split

Al: Yeah. So a lot of silicon photonics and EMLs are based on this material, indium phosphide. For VCSELs, our technology has always been gallium arsenide. For the listeners it may be, okay, one III-V compound versus another — what’s the difference? Indium phosphide is material-constrained from the very beginning. You can’t even get a base substrate — even before you process it, just the substrates for indium phosphide are limited, before you get it made into a product, whether that’s EML, silicon photonics, or VCSELs. The indium phosphide bare material is already limited. Gallium arsenide: unconstrained. So we start with that. And then you go through the fabs — fabrication depends on foundries, usually very large companies. Companies like PicoJool and others don’t own large clean-room factories. We design the individual VCSEL chip device, and then we use foundries to manufacture it. Those foundries are available — but they can’t get enough indium phosphide starting material. Now, after that, once you get to the chip level — you dice it up, okay, great, I’ve got the laser, I’m ready to go — to get from the chip to a pluggable device, an actual optical engine, there’s a ton of stuff that happens. You’ve got laser drivers, you’ve got boards, and for single mode you have to have all the machines that align that silicon photonics or EML to a very, very small-core single-mode fiber. Those machines have to be readily available. VCSELs, again, have that millions-per-month volume infrastructure — it doesn’t need to be built up. So from the very beginning: indium phosphide material constraint, then you have to build it into lasers, then finally package it into transceivers — and all along that supply chain, it’s not used to building millions per month. All of that has to be built up: hardware, alignment machines, testers. VCSELs have all of that infrastructure existing already.

Austin: I see — yeah, that makes a ton of sense. And I love your props, by the way — I liked the little VCSEL you held up. So tell us more: what are we looking at, and walk us through exactly what you design, and where it gets handed off, built, and packaged — where your responsibilities end.

The VCSEL Tree of Knowledge and the Handoff to WIN

Al: Yeah — so background is super important here. There’s what we call a tree of knowledge of VCSELs. You have this line from Honeywell through Finisar, and Finisar goes into II-VI, which goes into Coherent — so there’s the Coherent line. Then for us, obviously, HP went into Avago, went into Broadcom — the HP–Broadcom line. And finally there’s Picolight and E2O — I’m throwing in some old names from 25, 30 years ago — which go into JDSU, another big name from the dot-com time, and JDSU spins off Lumentum and Viavi. We’re from the Lumentum arm. All three of these major arms have a lot of VCSEL knowledge — and the strength of PicoJool is that we’ve tapped into designers from all three branches. Imagine all that know-how — again, not patented; know-how, recipes. I use this example: you can hand three different chefs a recipe for a soufflé and you’ll most likely get three different soufflés, because it’s really difficult to get a perfect one. It’s not just crack the eggs, beat the eggs. Same thing in the VCSEL.

So to answer your question — what do we do? We take all that know-how and design the epi layers. All these little lines you see — these are epi layers designing this vertical cavity. It’s not edge-emitting, where the light comes out of the edge of a flat chip; VCSELs are vertical-cavity, surface-emitting — the light comes out of the surface. We design the internal cavity of the laser, all the dopings, all the process. After we design, we hand it to an epi foundry that grows the material and gives us back an unprocessed epi wafer. We’ve been working with WIN Semiconductors in Taiwan — that’s our foundry. Once the epi wafer is ready, we hand it to WIN, they run it through their clean-room process, and they make the final VCSEL device. And here’s another beauty of VCSELs versus an edge emitter: at the wafer level you can start testing and probing each one — 100% tested, what they call known good die — before you have to singulate and dice it into arrays. That advantage is huge, because the more work you add before you yield the device, the more value you lose downstream. You always want to yield upstream. Wafer-level testing for VCSELs is a real advantage versus edge-emitting technologies.

And WIN is only the wafer processing. The wafer comes out and is diced into individual VCSELs, and we work with our partners to build those into either active optical cables or transceivers. That’s another foundry, if you will, but a packaging company. That said — companies like TSMC are now going into co-packaging: after they make their silicon wafer, they’ll package the optics directly on top of it. There’s a whole emerging field called CPO, where the traditional wafer-processing foundries are stacking technologies together — 3-D wafer-level packaging. For us: we only do the wafer at WIN, and then that WIN VCSEL goes to our module-integrator partners, who build it up into the active optical cables or transceivers.

Austin: Gotcha, that’s helpful. So take us back to your roadmap. I know you mentioned a 50G version, 100G, 200G — and a recent launch. Tell us more about your roadmap, what you launched, what you announced.

200G/Lane and the Three Flavors of 1.6T

Al: As I mentioned early on, 200 gigabit per lane is the benchmark — the bar you have to clear. EMLs and silicon photonics have all done that, and now VCSELs have reached it. PicoJool just announced our 200 gigabit; we’ll start sampling next quarter. That’s for a very simple transceiver where you have eight channels of 200 gigabit coming in — the aggregate bandwidth of 8×200 is 1600 gigabit, or 1.6 terabits.

So that’s the standard ramping today. There are 800 gigabit transceivers as well, that’s also shipping — that’s 8×100. And the next generation, today’s generation, is 8×200 — the 200 gigabit VCSEL we announced. But there are many flavors of that because of the specification. Aggregate bandwidth 1.6T — check. But there are different ways to get there if you want very low power or very low bit error rate.

Running 200G, I liken it to a Ferrari — it can give you 200 miles per hour, but it’s very high-end, relatively expensive, because you need certain signal integrity and signal processing. So now I say, I want very low cost, low power, but I still want low bit error rate. And what people have done is: let’s slow it down. Let’s use the 200G-performance VCSEL, but run it at 100 gigabit.

So now you have an excellent VCSEL that gives you a lot more performance, and if you use it at half the speed, you get much better bit error rates — the signal-to-noise, or relative intensity noise, drops as well. That’s 100G. But you need more lanes — to get to 1600 you need 16 lanes of 100. And recently something came out called micro-VCSELs.

And that goes even slower, down to 50G — they call it NRZ. So instead of PAM4, which has four levels (0, 1, 2, 3), we go back to the original NRZ, which is 0 and 1. Now you use the entire 0-to-1 signal-to-noise, which again reduces your bit error rate. But you need more channels — 32 channels at 50G to get to 1.6T.

But we’re shipping all three, and all three are in demand depending on whether customers want what they call fast and narrow (8×200); or an LPO — linear drive, no DSP, low power — which is 16×100G; or, if they want really low bit error rate and very low power, the 32×50G NRZ, which they call slow and wide.

We tend to call it “fast and wide” and “faster and narrow.” When you’re in high-speed interconnect, we try not to use “slow” in any of our marketing.

Austin: That’s good, that’s good. Okay, interesting — this is really cool. So you’re saying there are many ways to get to 1.6T. You could have 8×200, which uses PAM4 and requires a lot of DSP and power, but it’s definitely possible. Or 16×100 or 32×50, and you need less DSP for each of those — the 32×50 has much less because it’s NRZ. Yeah, this is all fascinating. So will that approach still hold once you move to 3.2T? Is it going to be that same combination of possibilities?

The Roadmap to 3.2T and 12.8T

Al: Great question — people always ask, what’s the roadmap ahead, what’s the future, is this the end of the road? Okay, 1.6T, VCSELs can do it — but is there a 3.2T? A 6.4T? A 12.8T? I mean, Andy Bechtolsheim — notorious — he’s created an XPO that’s literally going to give you 12.8T in a big pluggable today.

So they’re planning way ahead, because no one has ever told us in the last 30 years, “whoa, we have way too much bandwidth.” We have to have that bandwidth. And exactly like you said — what’s the future? Number one, we can use something called bi-di, bidirectional. Meaning we just add another wavelength — not the complexity of WDM where you have eight or 16 wavelengths like single mode.

We basically just add another wavelength to our existing one. So two wavelengths, passing them in both directions — bidirectional. That doubles the bandwidth without changing anything else, except you add another laser at a different wavelength, and you leverage the entire ecosystem. So from 1.6 to 3.2, we could add another wavelength. The other way, of course, is to double the speed. Can we do 100G NRZ?

That’s in the works. We’re developing 100G NRZ — today is 50G NRZ, but we’re developing 100G NRZ, leveraging our 200 gigabit VCSEL running at 100G NRZ. And in the future, we can go to more channels. The beauty of VCSELs again — surface emitting. Edge emitters can only have a one-dimensional array — a 1×4, 1×8, 1×12 — it just makes a long bar.

But for surface emitting, we can have a two-dimensional array — 2×4, 2×12, 2×16 — and couple all the light very elegantly with an optical fiber bundle. In that case I’m kind of unlimited. I can go up to 64 channels today in a 4×16 connector that’s the size of — let me show you. A 4×16 fiber is this size — here’s my finger. And that has 64 channels in it. If I run them at 200, that gets me to 12.8T. So in essence, the technology of today — without having to go to 400G per lane, which we’re also looking at — but at 200G, with more channels, with more colors (another color for bi-di), you double, you triple, by size. So that roadmap to 12.8T, we believe, is very solid and very clear — without even having to invent any new technology. And then with new technology, it just gets easier, if you can do 400G per lane.

Austin: Sure. Fascinating. So it’s just the same 200G VCSEL over and over — put it in an array and you get more of those. Or are you having to invent a new VCSEL to get the 100G version to run at NRZ?

Al: We’re just starting tests. We believe the 200G VCSEL has the capability to go to 100G NRZ. So it’s not a new VCSEL — it’s just a different coding on the signal coming in, running at NRZ instead of PAM4, to take advantage of the 0-to-1, using the whole signal-to-noise ratio as one bit as opposed to four levels. So that’s the difference.

Lead Times, Yields, and WIN’s Capacity

Austin: Gotcha. Yeah, this goes back to your engineering, pragmatic mindset — taking a Lego block and figuring out different ways to place it, different ways to use it, to unlock this whole roadmap. That’s pretty cool. So you talked about unconstrained gallium arsenide and working with WIN — they’re used to making this stuff, they’ve got all the pizza ovens they need. So if a big hyperscaler comes to PicoJool and says, we want a million of your pizzas — what does that look like? How does that actually happen?

Al: Right. If they want a million VCSELs, we’d typically give them an eight- to ten-week lead time. That’s the basic. We can accelerate it — push and have engineering carry certain wafers — but typically eight to ten weeks on the VCSEL side. You’ll get a wafer, or individually diced VCSELs. If you want transceivers, then that eight weeks tags on a certain number of weeks to package it all into the transceiver.

Those are constrained strictly by the packaging process — not by ordering equipment or building capacity. That ecosystem works, and it leverages the typical cycle time of building out. Typical cycle time: eight weeks for the VCSEL device, then four to six weeks for the module after that. So you’re looking at anywhere from 12 to 16 weeks to get to the full module, starting from an epi reactor growing epi and going all the way through.

And we’ll continue to drive that lower — lead time or cycle time. But also yields, which I didn’t mention much. Yields are: if I make a VCSEL, how many known-good die can I get out of a wafer? The higher the yield — up to 100% — the fewer wafers I have to run, and the higher the capacity of the factory. If every wafer goes through and I get 100%, I need fewer wafers, and therefore less capacity for the demand. Obviously getting to 100% is very hard, but VCSELs have been perfecting that process for many years, for many decades — and now we’re leveraging all of that. It’s not new, it’s not something that has to be established, it’s not based on new technology. That’s very important. WIN has been doing it for about 10 years, since we transferred that for a consumer electronics application back in 2016. They have a capacity of up to a thousand of these wafers per week. And there are about 240,000 VCSELs on each wafer, because they’re really tiny. So that adds up to a million yielded from maybe 10 wafers. The capacity is huge for datacom. So I think we have no worries — once we get to the 200G, the product specs are met with the customer, and reliability qualification is done, then we just ramp readily with WIN, and they’re ready to go.

The Fabless Model — Stay Small, Leverage the Supply Chain

Austin: Nice. Amazing. It’s quite compelling. Normally when people hear “there’s a startup trying to compete in a space with these huge incumbents,” the question is: how’s this startup going to compete? How do they get to market, find customers, build up supply chain? But what I hear you saying is that you’re taking industry veterans with process know-how — similar ways of thinking as competitors, you’ve been in the game a long time — and tapping into an existing supply chain. And at the end of the day, it’s not like you have to win 50 customers; there’s a handful of big customers that would really make a difference for PicoJool if they said yes. But the most important point is the unconstrained gallium arsenide — being able to make a million, 10 wafers with 240,000 on each, whatever you yield, we’re talking millions of VCSELs very quickly. Because across all of semiconductors — memory, CPUs, AI accelerators — there’s so much demand and such constrained supply that, yes, you want to compete on cost and engineering performance, but there’s also a bit of: if it’s good enough and it’s in production and you can install it into my data center, game on. So it feels like you have a strategy that lets you deliver shipped VCSELs as soon as possible.

Al: Yeah, the model has been around for decades in silicon. I’m in Palo Alto, in Silicon Valley. Most chip companies designing integrated circuits, CPUs, GPUs, don’t have their own foundry. A lot of people use Intel; even AMD uses TSMC. AMD is a huge chip company and they don’t have their own foundries today. TSMC, Intel, Global Foundries — many foundries are the factory floor, the clean rooms, of all these startups. So just because of our small startup size doesn’t mean we can’t ship in the millions per month and compete directly with the very large presence of other optical suppliers and competitors. That’s the beauty — we can stay very lean. Our motto is “stay small,” and that has to do with many things — our name is PicoJool, right? Very, very low power, and that’s one of the things driving us. We’re a well-experienced small team, but we get huge benefit by leveraging WIN Semiconductors and working with our supply chain. They’ve got the factories and the clean rooms, and we can ramp very quickly by providing our designs and our unique specialty, then partnering with these large companies that are already shipping — and they just drop-ship. So we don’t need a big company to ship in the millions per month.

Austin: That’s amazing. Definitely punching above your weight — that is awesome. So when is your high-volume ramp targeted for? If I recall, the press release said something —

Ramp Timing — Sampling Now, HVM in Early 2027

Al: Yes, the press release said we’re starting to sample next quarter. And that’s in different flavors — we’ve got customers for the 50G NRZ, the 100G LPO, and the 200G. So all of those begin to sample. And the process from our device to actual product shipment — revenue, in our case — is a period they call qualification, or reliability testing. Everything has to not only meet spec at zero hour, but be predicted to last 10 years, or a number of years, through what they call accelerated aging — they test it at higher temperature and higher bias conditions, then estimate back to normal operating conditions: can it last 10 years in the field? For very small, tier-two customers, that can be as short as three months. But for tier one — because they have much more to lose if there’s an issue with their connection — it takes more than six months to get up and running. So ramping at WIN is available today; but we have to go through this qualification cycle with our customers before they give the orders and everything is approved. So we’ve also said we’ll most likely start ramping in early 2027, next year.

Austin: Okay, got it. Thank you for the education here. So — sampling, then the qualification process, then ramping. And probably, as you’ve been saying throughout, you’re not concerned about the ramping. You had to build the product and let customers kick the tires, and once they say let’s go, it’s off to the races. Okay, awesome. Well, we’ve covered so much — this has been amazing. I’ve learned a lot, and I know the listeners will have too. Is there anything else, any last things about PicoJool, or anything we didn’t talk about that you were hoping to cover?

Mentoring the Next Generation of Photonics Engineers

Al: One thing that’s very interesting to me — as you can see from the image, I’m a very experienced, elderly startup person. One thing to point out: very few people went into hardware and photonics in our space, because young people over the last 25 years — literally since dot-com — came into the workforce more on the application side, the software side. People wanted to go into computer science. So the aging, experienced hardware folks in photonics need to transfer all this knowledge. That’s one of my passions — to bring on the next generation, and the generation after that, for photonics. Because just like VCSELs and other technology, we see many decades ahead, and as I said, no one’s saying “way too much bandwidth.” We see bandwidth demand increasing with robotics and autonomous vehicles and what have you — everything is going to be bit- and connectivity-constrained. So we want to spend time to educate and train. We’re trying to hire hardware engineers, and train young folks — maybe without the experience — to be the VCSEL designers and transceiver designers of the future. That’s really exciting for us, because we’ve got all this knowledge, 10, 20, 30, 40 years, and it feels wonderful to have the hardware excitement again — not only in the markets, but the investment community. Silicon Valley is booming with photonics and hardware. So we don’t take it for granted. It’s a great opportunity, and we definitely want to take young entrepreneurs, young engineers, folks interested in this space, along for the ride — and then they take it from there.

Austin: I love it. Very inspiring, very cool. It’s never been a better time for interconnects, photonics, and optics folks. I love that you industry veterans want to bring up the next generation and give them the opportunity to learn from folks like you — to revitalize, rebuild, make sure we have a reinvigorated workforce, so that for my generation and the generation of my children, they can keep having more and more data moved around, faster and faster.

Al: That’s right. Yeah, really enjoyed our conversation, Austin. Thank you.

Austin: Awesome. Thank you, Al. Appreciate it.

But Groq and Cerebras run architectures you can poke holes in. Both were designed before LLMs went mainstream, and their early design choices don’t suit today’s models. I call them “preGPT” accelerators.

The famous example (well-covered ground already) is that they’re SRAM-only. It takes many Groq racks to serve even one smallish model, and even Cerebras’ wafer-scale marvel (900,000 cores on one piece of silicon) can’t hold a frontier model’s weights on a single wafer, so you have to expand to many wafer-level systems. Tough economics, a KV cache that doesn’t scale gracefully, and so on.

Admittedly, I looked at these engineering details and figured these companies were dead in the water when it came to LLMs. Well, to their credit, who could have predicted such thicc models 10 years ago? But no HBM to support long-context... how will they survive? More context is better…. failure to thrive?

What’s important is that they both had production silicon available, and, regardless of the shortcomings of SRAM-only architecture, they outperformed GPUs on a certain KPI (interactivity). As Jensen nicely drew for us, SRAM chips for decode unlocked new possibilities on the Pareto frontier for 2026:

And that’s why they had eleven figure outcomes. $XX Billion, kind of nuts right!

I was so wrong about architectural shortcomings preventing success.

Looking back, what mattered most? Timing.

Groq was in production right when Nvidia came calling. Cerebras, right when OpenAI did. Forget the architectural shortcomings. If you can ship and unlock a new Pareto frontier, good things happen. In fact, I thought the fundamental problem for Groq and Cerebras was being too early, making design decisions before transformers took off. Being too early is indistinguishable from being wrong...

And yes, starting that early nearly killed them; both came close to running out of cash. But starting early was also the whole advantage, because when the unforeseen inference wave hit, they had silicon in production. Well, they were both still default dead until Meta released Llama, the first useful open weights LLM, which allowed Groq to go show the world how awesome a super fast chatbot experience was.

Again, the combination of production silicon at the right moment and a Pareto frontier the GPU can’t reach is all that mattered. And then Nvidia’s Dynamo was icing on the cake, disaggregating the workload so the memory-bound decode could run on SRAM chips.

The postGPT era

If the first era of inference was GPUs, Groq and Nvidia ushered in the next era with the GPU+LPU inference. But another era is incoming. The postGPT accelerators. Rack-scale systems specifically designed for LLM inference from day one. From matmuls to interconnects to memory hierarchy.

There are many startups in this space, not to mention custom silicon XPUs (Maia, MTIA, Trainium, etc). So which of these could be the next Groq/Cerebras with a whale of a customer and an acquisition/IPO?

What it takes to be the next trillion-dollar chip company

If we’ve learned anything so far, it’s that we shouldn’t overindex on engineering architectures. Sure they matter in the long run, but in this demand environment, contenders simply need to hit these four criteria:

• Runs frontier 1T+ models

• Ships rack-scale

• Beats the incumbent on one KPI

• Lands a frontier anchor

Runs frontier 1T+ models. Frontier models are where the value gets captured and where scale happens, so frontier customers are the ones most willing and incentivized to deploy a postGPT accelerator at volume. A system that can’t run a 1T-param model is playing a sub-frontier game. I’m bullish on the usefulness of sub-frontier, especially in the enterprise, but the SAM is small. They won’t mint the next Groq/Cerebras.

Ships rack-scale. The race to gigawatt-scale inference runs on racks. To win, one must design and deploy racks of accelerators, quickly and reliably.

Beats the incumbent, clearly, on one KPI. The challenger doesn’t need to be best at everything. But they must be an order of magnitude better at something.

Lands a frontier anchor. The challenger needs someone serious about gigawatt-scale and buying into a roadmap and building a long-term relationship. A model lab or hyperscaler is best. Could be a big neocloud. Probably not a sovereign.

Of course, software support is table stakes.

But otherwise, I think it’s that simple.

Yes, the technical details matter; quantization, memory hierarchy, scale-up domain size, etc. But truly, right now they matter only insofar as they lead to differentiated performance (”beats the incumbent on one KPI”). The architecture behind each contender is in the teardowns; up here, judge them by the KPI it buys.

This is a unique time. Token demand FAR exceeds supply, and the shortage is worst for low-latency frontier tokens, where GPUs can’t keep up or can only do so at uneconomical throughput. This high-interactivity token scarcity turned “good enough architecture in production” into a winning hand for Groq and Cerebras.

But it’s time-bound; there was a window, and Groq/Cerebras grabbed it. And being first matters.

Which raises the next question: what’s the next window, and who will get there soonest?

Picking the next winner could be as simple as asking who can deploy a gigawatt of postGPT inference accelerators soonest?

To answer that, here’s the rest of the piece.

• The frontier race, all eleven ranked by when their frontier rack ships, as a scoreboard and a timeline

• A full teardown of Tenstorrent, the only one serving a frontier-class rack today, a free sample of the diligence

• [Paid] The other ten teardowns, each scored and sourced, the frontier contenders, the decode add-in, and the sub-frontier plays

• [Paid] The receipts, only two of the eight frontier names have a clean third-party number

• [Paid] Who actually looks like the next trillion-dollar chip company, and the argument that reframes the whole list

The frontier race: who reaches rack-scale, soonest

Well then: “Which startups clear those four hurdles, sorted by ship date soonest?”

Let’s take a look. I’ve sorted them by when they claim their frontier-class generation hits rack scale. Take it with a grain of salt, and remember this isn’t yet anything about performance, but just when they say they’ll have silicon deployed.

Also note that many of these companies have shipped products already, but the earlier products were sub-frontier as they pivoted their way toward rack scale for modern frontier LLMs. So I’m only focusing on when they are shipping frontier-capable products according to their roadmap

If we scour all the public sources (press releases, blogs, podcasts), we get the following:

Visualized:

Caveat though. Every entry above is a first-rack or first-sample milestone, not a gigawatt figure. IMO first rack matters, but time to first gigawatt matters most.

First rack proves the whole shebang works. But gigawatt scale proves the company works.

Who is shipping in 2026, and who isn’t?

Tenstorrent, Etched, and SambaNova ship production racks in 2026.

A year back sit Positron, Tensordyne, and Rebellions. Rebellions claims 2026, but its frontier REBEL is still only sampling, so its real rack is more like 2027.

Fractile and MatX are two years back. Both have 2027 milestones on their roadmaps, but those are samples and footholds, not scaled production shipments. Production racks at customers are realistically a 2028 story.

In this supply-starved environment, a year is expensive. That’s not to say the 2027+ companies aren’t technically competitive, nor does it imply they won’t win serious customer share in the grand arc of time. After all, it only takes one of the few large customers to get to scale. But you’re not in the game until you’re on the field! No matter how technically sound your architecture is, if you’re not on the field, you’re not in the game.

Who will be on the field first?

Tenstorrent claims to be “shipping production silicon now”. Promising! But what about customers? Tenstorrent has neoclouds and sovereigns; no frontier lab or hyperscaler has been announced yet. Could those early customers accelerate Tenstorrent to gigawatt scale? We dig into that in its teardown below.

An acquisition could help Tenstorrent get to market, as big silicon incumbents already have relationships with hyperscalers. Qualcomm was reported in mid-June to be circling a roughly $8–10B deal. But Keller supposedly denied any Qualcomm talks recently, so that path looks cold for now.

Etched is a bigger contender than it looked just months ago. It claims rack-scale shipments this summer, well ahead of the other popular names like MatX and Fractile. I think Etched was left for dead, but it turns out they are alive and claim to be right at the front of the line.

Notably, Etched is talking gigawatts. It claims “a path to gigawatt-scale in 2027,” and it’s talking about building the company infrastructure to pull it off with a Taiwan factory, plus a data center, test house, and NPI prototyping lab at its San Jose HQ. From the press release, co-founder Rob Wachen said “Our approach from the beginning has been to build for gigawatt-scale... Production is the product.”

Etched hasn’t disclosed specific customers but mentioned $1B+ in contracts booked. Etched’s talk about a “path to gigawatt-scale” and “building for gigawatt-scale” seems to imply they are targeting the type of companies that can support that scale.

Whether Etched actually hits gigawatt scale in 2027 remains to be seen, but it’s at least aimed at the right target if they want to be in the trillion-dollar race.

SambaNova also claims a 2026 deployment on its frontier SN50 landing in the second half. This is going to make for an exciting 2H26 and early 2027 if we’ve got SambaNova, Etched, and Tenstorrent all shipping competitive racks!

So far, SambaNova’s named customers are enterprises JPMorgan Chase and SoftBank, and these are explicitly NOT gigawatt installations. From Sally Ward-Foxton

SambaNova has a multi-year agreement in place with JPMorganChase.

“This is a breakthrough because most suppliers haven’t figured out how to get into the enterprise,” Liang said. “The enterprise market is going to be significant; it won’t be 100%, but it’s going to be a meaningful player in the overall market, and we’re starting to see that.”

Enterprise customers are looking to host private models and private data in their own environment, which is unlikely to be a gigawatt-scale data center, Liang said.

Of course, if you have tens of 50-100MW deployments, well, that counts!

Maybe SambaNova’s plan is to demonstrate feasibility with reasonably sized enterprises and use that success to try to win hyperscalers? Or just win with enterprises, which should turn into a very large market on its own.

Behind the 2026 crew are five more competitors including Rebellions, Positron, and Tensordyne in 2027, with Fractile and MatX not reaching production racks until 2028.

Rebellions reaches a rack soonest of this 2027+ tier, with REBEL sampling now. But its named customers, mostly Korean enterprises plus Saudi Aramco, sit on the sub-frontier ATOM, not the frontier REBEL, and none is a gigawatt buyer.

Positron has a promising customer, Oracle.

Some consider OCI (Oracle Cloud Infrastructure) a hyperscaler and others call it a neocloud (ClusterMax rates it as a gold tier neocloud). Regardless, OCI can support GW scale installations.

So is a promising path to GW scale for Positron, and they are the first on this list who can publicly say they’re already deployed in a hyperscaler. The catch is that Oracle is currently running Atlas, Positron’s sub-frontier first-gen server. It’s not rackscale. The frontier system, Asimov, doesn’t tape out until late 2026.

So Positron already has a customer who could scale to a gigawatt, just not yet on a frontier product.

Tensordyne recently announced its Napier system has been taped out on TSMC 3nm and production is underway. And Tensordyne’s logarithmic math approach is super interesting.

But its customer book looks a lot like Tenstorrent’s. Cirrascale and BlueSky Compute are named, plus a dozen-plus letters of intent to evaluate beta systems, and roughly $200M of forecast, not booked, demand. Real interest, but neoclouds and LOIs, not a frontier lab ordering a gigawatt.

Fractile is further out. Its roadmap targets a late-2026 commercial tape-out, customer samples in Q2 2027, and production silicon only at the very end of 2027, so racks in customer data centers are realistically a 2028 story. But it’s courting exactly the right customer. In May 2026, The Information reported Anthropic was in early talks to buy Fractile chips once they ship, which would make a frontier lab its anchor, the kind that scales fastest. The talks were preliminary and the deal size wasn’t disclosed though.

MatX is pre-silicon, on roughly the same clock as Fractile. It targets tape-out within a year of its $500M Feb 2026 raise, first chips a few months after that, and a 2027 foothold inside a frontier lab, routing ~1% of production traffic to its silicon as an A/B test.

That foothold is a validation experiment, not production racks; by Pope’s own tape-out-to-production math of one to two years, volume racks are a 2028-at-earliest story.

It hasn’t dated real rack scale, only that CEO Reiner Pope “would like to be... shipping multiple gigawatts a year” (Cheeky Pint). So it’s got the right goal, just behind on the timeline.

Yet it may have the best customer list of anyone here. MatX says it sells only to frontier labs, “maybe five different customers in the world” and its 1% trial would be a foothold with such a customer. If that little trial is a success, that’s the type of customer who would scale quickly to a gigawatt faster than any neocloud or enterprise mentioned.

Deeper Teardown

Of course, details matter. Here’s the due diligence on eleven companies with one question for each.

Can it put a frontier-class rack in front of a whale of a customer before the window closes?

Below is the summarized diligence on all eleven, the eight frontier contenders first, then the add-in and the sub-frontier plays.

For each we’ll hit on

• KPI is the one outcome the chip beats a GPU on.

• Anchor is who is actually deploying it.

• Architecture

• Verified?

Then a bunch of extra sourced details.

Tenstorrent (Q2 2026)

Jim Keller is arguing a contrarian narrative. One chip runs everything, no disaggregation. Keller believes that, in the long run, general-purpose can perform as well as specialized + disaggregation.

• KPI: low cost at high interactivity. Runs ~any model (~90% of Hugging Face), prefill and decode on the same silicon.

• Anchor: five-plus neocloud colos serving live, plus sovereign and IP licensing (Japan, LG). No frontier lab, no hyperscaler.

• Architecture: GDDR6, no HBM; RISC-V Tensix cores; Ethernet scale-out on-chip; 100% open-source software stack. One chip runs both prefill and decode, the deliberate counter to disaggregation.

• Verified? ⚠️ DeepSeek ~308 tok/s/user is a served-model number, but relayed by Keller, not an independently-published AA figure. The Prodia video claim is similar: AA independently ranks Prodia’s model at the top of its leaderboard, but that predates the Tenstorrent partnership, and the 10× speed-on-Tenstorrent-hardware number is Tenstorrent/Prodia’s own benchmark, not AA-reproduced. No MLPerf; the 350→500 tok/s headline is partly forward-looking.

DETAILS

The anti-thesis, on record.

• At TT-Deploy (May 2026), Jim Keller bet against specialization and disaggregation: “the big fad is disaggregation, special purpose hardware, SRAM. Do you know how many people are going to be talking about that in 2 years? None.”

• And: “you don’t just run part of an LLM… that’s not a good business plan long run. You could be one-shotted by one model.”

• Stan Sokorac’s (Sr. Fellow, Software) proof point: a pipeline that downloads random Hugging Face models achieves a ~90% pass rate, so “we can run ~2.5M of the 2.8M models on Hugging Face.” (TT-Deploy keynote; Sokorac talk)

Architecture.

• Tensix core = matrix engine + vector unit + 5 “baby” RISC-V cores per tile, each with local L1 SRAM, over a 2D NoC mesh (no shared global memory); Blackhole adds 16 “big” 64-bit RISC-V CPU cores and can run host-less (The Register).

• Anti-disaggregation by design: each chip has SRAM + DRAM + networking, so it runs prefill + decode + video on the same silicon (Vasiljevic talk).

• Scale-out: standard Ethernet, 10×400 Gbps = 1 TB/s chip-to-chip, the anti-NVLink bet. Blackhole = TSMC 6nm.

• Memory: large on-chip SRAM + GDDR6, explicitly no HBM. Official Blackhole spec: 180 MB on-chip SRAM, 28–32 GB GDDR6 at 448–512 GB/s (tenstorrent.com/hardware/blackhole).

Shipping status.

• Three taped-out generations (Grayskull → Wormhole → Blackhole).

• Shipping Blackhole cards: 120 Tensix / 664 TFLOPS BLOCKFP8 / 300 W; p100a $999, p150a/b $1,399.

• Blackhole Galaxy production servers reached GA Apr 28 2026 (6U, 32 chips, 23 PFLOPS dense FP8, ~$110K) (The Register GA).

• Yield caveat, now in the official spec: launched at 140 Tensix, a Jan 2026 firmware cut to 120 (≈1–2% perf hit on typical workloads; new cards ship Bin-3 parts with two disabled Tensix columns) (Tom’s Hardware).

• “Installed in ≥5 neocloud colos outside Tenstorrent” goal hit (TT-Deploy).

Deployments / anchors.

• Infra/colo/neocloud + finance, not hyperscalers: Equinix, Cirrascale, ai&, Prodia Labs, Virtu Financial, Turiyam, OrionVM (Tenstorrent newsroom).

• The other half is RISC-V/chiplet IP licensing + sovereign design: Japan/LSTC (Ascalon IP, Rapidus), a ~$50M Japan program to train 200 designers, LG co-developing SoCs.

• The gigawatt question. The July 2026 slate widened the book (Galaxy superclusters now anchor Equinix’s Distributed AI Hub, launched with OrionVM and BetterBrain, plus new ai&, Virtu Financial, and Cirrascale deployments), but none of it is gigawatt-class. The headline neocloud, Cirrascale, is ClusterMAX Silver, running in the thousands of GPUs, and already lost OpenAI to Azure (ClusterMAX). Every name here buys racks, yes, but not gigawatts. Aggregating many mid-size buyers may get there, but slower.

Performance / verification.

• Absent from MLPerf v6.0.

• ~308 tok/s/user on DeepSeek live is a served-model measurement, but per Keller’s own telling, not an independently published Artificial Analysis number; the roadmap to 500 tok/s/user at $6/M TCO is forward-looking.

• The Prodia video record (Wan 2.2, 2.4s, 10× a GPU) needs the same unpacking: Prodia’s model already led the Artificial Analysis leaderboard before the Tenstorrent partnership, so that ranking is genuinely AA-verified — but the 10× speedup on Tenstorrent hardware (33.8 fps vs. 5.5 fps on Nvidia) is Tenstorrent and Prodia’s own collaboration benchmark, not a number AA independently re-ran (Tenstorrent newsroom; Vasiljevic talk).

• Stack is 100% open source (TT-Forge, TT-Lang), the anti-CUDA-moat argument.

Team / funding.

• Founded 2016 (Toronto) by Ljubisa Bajic, Ivan Hamer, and Milos Trajkovic; Jim Keller CEO since ~2023.

• Series D $693M+ closed Dec 2024 at ~$2.6–2.7B post (Samsung Securities + AFW lead; Bezos, Hyundai, LG) (Series D).

• A reported ~$800M / ~$3.2B Fidelity round (Nov 2025) is not company-confirmed.

That’s the free read. Behind the paywall:

• The other ten teardowns, each scored on the same four questions as Tenstorrent, its KPI, its anchor, its architecture, and what’s actually verified, then the full sourced details behind every line. The seven remaining frontier contenders (Etched, Rebellions, SambaNova, Positron, Tensordyne, Fractile, MatX), the decode add-in (d-Matrix), and the two sub-frontier plays (Furiosa, Taalas).

• The verification scorecard, which two of the eight frontier contenders have a clean independent number, which one has only a partner-run result, and which remain vendor-stated.

• Which “frontier” chips can’t yet run a frontier model, the sub-1T ceiling sitting under every verified number, and whose only MLPerf result is for the wrong chip.

• The memory split, company by company, who leans on HBM for raw bandwidth and who drops it for commodity LPDDR or GDDR to dodge the supply crunch, and what each choice costs them.

• Where the field lands on disaggregation, one chip running prefill and decode versus a split tier, and which way it’s converging.

• Who actually looks like the next trillion-dollar chip company

If you’re deciding who to watch, partner with, invest in, or compete against, this is the half that matters.

Etched (H2 2026)

A frontier inference system, sold as a full rack, has its real edge in low-voltage inference and cluster-scale memory. And more flexible than it’s remembered for: the system that launched in June runs DeepSeek, Qwen, Llama, and Mamba. Mamba is an SSM, not a transformer… so clearly it’s more flexible than previous talk of “transformer-only ASIC” made it out to be…

Read more

If you’re not up to speed, the basic idea is that HBF is a stack of NAND dies built the way an HBM stack is built; dies stacked vertically, wired together with through-silicon vias (TSVs), and sitting right next to the GPU on the package interposer:

What’s interesting is that it has the same read bandwidth as an HBM4 stack, but with roughly 10x the capacity. And it’s made of NAND, not DRAM like HBM. (NAND is the cheap stuff.)

The first samples of the memory itself are expected very soon from Sandisk, sometime in the second half of 2026. Samples of the first AI inference devices built with HBF follow in early 2027.

Everything has tradeoffs, flash too. We’ll look at why those tradeoffs aren’t so bad for inference decode workloads, and the impact HBF will have on the memory market.

Table of contents

• Flash is storage. But could it work for memory?

• Weight memory is what drives GPU count

• DRAM is no longer scaling well. But NAND is

• CMOS directly Bonded to Array: how the HBF stack is built (for the device physics layer, see Vik’s High Bandwidth Flash: NAND’s Bid for AI Memory)

• [Paid] Inference decode fits this profile; training does not

• [Paid] Four ways HBF attaches to a GPU, each with a different supply chain winner

• [Paid] The disaggregated inference cost model

• [Paid] What HBF does to the memory markets

• [Paid] Nvidia is the certification gate; custom silicon moves first

• [Paid] Competitive landscape: Sandisk, SK Hynix, Samsung, YMTC

• [Paid] The patent: a processor bonded to NAND

• [Paid] HBM is infeasible for edge devices; HBF is not

• [Paid] Timeline and risks

Flash is storage. But could it work for memory?

NAND flash has traditionally been used for information storage, i.e. where data lives when it’s not being used. Cheap, dense, and non-volatile. But slooooooowwwww....

But memory is where the accelerator keeps information handy during computation, and it has to be fast enough that compute never waits. SRAM is used for memory and can be read very quickly, on the order of about a nanosecond. Next in the memory hierarchy is DRAM, which is slower to read at ~100 nanoseconds.

But storage like NAND takes about a hundred microseconds, roughly 1,000x slower than DRAM. By definition that’s storage and not memory, right? It takes way too long to read, but good for long term storage. That won’t work for computation... way too slow! Compute would just be sitting idle, waiting forever for data.

So how could NAND flash possibly be used as memory?

Well, could there be a way to get the right data to the accelerator in time for computation, even with NAND’s slow reads? If you start the read it WAY before the accelerator needs the data, it could work?

In addition to intelligently scheduling when data is requested, one can also try to achieve high bandwidth from NAND. Bandwidth is the amount of data that arrives per unit of time. So if you know the read is going to take a long time, well, might as well have a bunch of reads running in parallel, if possible, right?

HBF unlocks high bandwidth by stacking NAND dies to increase the “width” or the parallelism. Each die is divided into many sub-arrays, which are small blocks of NAND that can each be read at the same time, independently of one another. An HBF stack places 16 of these dies behind a single interface, thousands of bits wide, so a huge number of sub-arrays are available to read at once. Any single read still takes thousands of nanoseconds, but thousands of reads can run in parallel, so a lot of data can be moved simultaneously.

The 16 flash dies are stacked and connected with through-silicon vias (TSVs). A controller logic die is then bonded directly onto the NAND array. That logic die schedules the parallel sub-array reads and drives the results out over the wide interface to the GPU. The bonded die is called “CMOS directly Bonded to Array” or CBA:

The logic + HBF stack is on a high-bandwidth interposer. The resulting HBF delivers 1.6 TB/s of read bandwidth, which is the same as an HBM4 stack at the JEDEC spec’s 6.4 Gb/s operating point:

To underscore the importance of what the packaging is doing here, compare against the NAND you can buy today. Conventional flash ships about 14 GB/s behind a PCIe 5.0 NVMe controller. Packaged as HBF, the same material delivers 1.6 TB/s. Roughly 100x the bandwidth, from packaging alone. Bullish advanced packaging!

Again, HBF’s latency is still 10-100x slower than HBM. But if the accelerator knows what data it needs in advance, it can prefetch it and avoid waiting on any single slow read. The argument for HBF is that inference decode is the perfect workload. I’ll explain in more detail later for paid subscribers.

Weight memory is what drives GPU count

What are the economic implications of HBF?

Inference cost scales with GPU count, and for today’s massive frontier models, GPU count is often driven by memory capacity per GPU.

Why?

Well, you need enough HBM to hold all the weights, but HBM is co-packaged with the accelerator; a fixed amount of HBM is bonded into each GPU package. So you can’t add memory without adding GPUs. Hence, bigger models mean more GPUs.

Of course, weights aren’t the only thing the accelerator needs to store in memory and acccess quickly and often. The KV cache and activations sit in memory too, and both stay in HBM or DRAM.

But the KV cache takes new writes every token; NAND’s endurance can’t handle that. NAND has much lower write endurace.

And activations need low-latency random access that NAND is too slow to give.

So HBF is for storing model weights.

And frontier weights are huge and always wanting to be even bigger. Wouldn’t it be nice to hold the model in significantly cheaper memory than HBM?

Recall that a 70B parameter model at fp16 (2 bytes per parameter) requires 70 × 10⁹ × 2 = 140 GB just for weights. A 1T parameter model thus needs 1,000 × 10⁹ × 2 = 2 TB. But that’s a lot of HBM; today’s shipping HBM4 stacks hold 36 GB (12-Hi); 16-Hi parts push 48 GB, and the JEDEC spec tops out at 64 GB. So a large model needs many stacks, which means many GPUs. Expensive!

That also means more interconnect (to move data around all those GPUs), which requires more power and, ultimately, a higher cost per output token (watts and $).

But HBF can provide 512 GB of capacity per stack!

So 512 GB per stack versus 48 GB for HBM4 is roughly 10x the capacity at the same bandwidth. Sandisk states up to 8-16x across the HBM family, depending on which generation you compare against.

Interesting! Super promising. Consider the implications of storing frontier models in a few HBF stacks rather than spread across many HBM stacks across many GPUs in a server or rack.

New opportunities arise too! Could on-premises, air-cooled deployments at enterprises have frontier-model weights capacity without needing so many expensive GPUs?

CMOS directly Bonded to Array (CBA)

Let’s dig into the manufacturing of HBF a bit more. Dang, I seriously didn’t mean to be punny with that bit.

Sandisk stacks 16 BiCS NAND dies (BiCS, or Bit Cost Scalable, is just its brand of 3D NAND) with TSVs (the same way HBM is built) and bonds the controller onto the array:

The result matches HBM4’s footprint, height, and power envelope, so it can drop into the same interposer slot an HBM stack sits in today. It speaks nearly the same electrical interface, too, though the host memory controller has to change, so it isn’t quite drop-in. And being NAND, it’s non-volatile (it holds data with the power off), so unlike DRAM it needs no constant refresh power.

One open question is the NAND cell type. NAND can pack more than one bit into each memory cell:

Highly recommend this explainer from Vik:

Vik's Newsletter

High Bandwidth Flash: NAND’s Bid for AI Memory

Welcome to a 🔒 subscriber-only deep-dive edition 🔒 of my weekly newsletter. Each week, I help investors, professionals and students stay up-to-date on complex topics, and navigate the semiconductor industry…

Read more

a year ago · 26 likes · 4 comments · Vikram Sekar

Note that the fewer bits each NAND cell packs, the faster and more durable the cell is, but at the tradeoff of density.

QLC (quad-level cell) holds four bits, cheap and dense, but is the quickest to wear out. SLC (single-level cell) holds one, yet is far more durable.

Sandisk hasn’t said which HBF uses, but Irrational Analysis reported recently that industry sources tell him it will be SLC, which would improve HBF’s write endurance by an order of magnitude, but at the cost of density.

Sandisk makes the NAND, but turning 16 dies into a stack that matches HBM’s footprint, thermals, and interface is advanced packaging work... the same thing the big 3 HBM folks have already perfected. So SK Hynix is the stacking partner.

Sandisk brings the flash, SK Hynix brings the stack.

The two began an OCP standardization effort in February 2026.

Standardization?

You might expect Sandisk to want a monopoly on HBF, right? Actually, it wants other suppliers, so long as Sandisk is the biggest. After all, hyperscalers won’t design a sole-source part into a billion-dollar platform, so being the only supplier is a risk to HBF adoption.

A ratified standard with multiple suppliers removes that risk and gets HBF designed in. Again, Sandisk would rather own a big share of a market that exists than all of one that never does. It reminds me of with Credo and active electrical cables. Grow the category into a real market, and win the largest slice of it.

So how does HBF compare to HBM?

HBF has the edge when it comes to capacity vs HBM4, but trails in most other respects:

Note the 512 GB capacity is 14x the 36 GB 12-Hi HBM4 shipping today, ~10.7x the 48 GB 16-Hi flagship, and 8x the 64 GB JEDEC max.

For the same bandwidth, HBF has up to 8-16x the capacity at roughly 2x the power.

Yes, HBF has worse bandwidth per watt, but for model weight storage one can argue the metrics that matter most are capacity and capacity per watt, where HBF wins.

SanDisk also claims a similar cost to an HBM stack despite 8-16x the capacity, which works out to roughly 10x lower cost per GB.

I thought NAND was way cheaper?!

The raw NAND is cheap per bit, but an HBF stack isn’t cheap. Most of its cost is the same advanced packaging that makes HBM expensive like the 16-high TSV stacking, the CBA logic die, and the interposer. So you get HBM-stack pricing with ~10x the bits, not SSD pricing.

Of course, HBF has weaknesses compared to HBM. It’s all tradeoffs.

Microsecond latency against HBM’s ~100 ns, a minimum read 128x larger, and write endurance that rules out anything write-heavy.

Objections to HBF are primarily those three issues. But the painfulness of those weaknesses depends on the workload. And for decode inference, HBF isn’t so bad! We’ll cover it more below.

Interestingly, John Carmack has been thinking about this too; yesterday’s X post has over 1M views:

Click on it and read the whole thing! Here’s an important snippet:

“model inference can have a deterministic memory access pattern. You don’t need ‘random access memory’ at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.

Clearly, there’s merit to flash for weights for inference.

There are lots of nuances too.

And many big questions. How does HBF impact HBM demand? How does HBF impact the already hot NAND market? I’ll address those and more.

What paid subscribers get in the rest of this piece:

• Why decode doesn’t care about NAND’s weaknesses

• The four ways HBF attaches to a GPU: direct HBM replacement, mixed HBM+HBF slots, HBM-as-cache, and disaggregated prefill/decode. Each implies a different GPU attachment rate and a different set of supply chain winners, and the earliest route to production isn’t the obvious one.

• The disaggregated cost model, worked: a table pricing the model weight tier for Llama 405B, DeepSeek-V3, and a 1T-class model. HBM vs HBF

• What HBF does to the DRAM and NAND markets

• Route to market: Nvidia, hyperscaler custom silicon, what about ASIC startups and other merchant GPU vendors?

• A Sandisk patent that hints at the long term roadmap for HBF

• Timeline and risks

And more!

Read more

Just look at the past four quarters:

Revenue went from $11.3 billion to $41.5 billion. Gross margin went from 45.7% to 84.9% (!!!)

Each quarter beat the guidance Micron gave the quarter before, and the magnitude of the beat grew each time.

And here’s a mind-bender. Micron’s fiscal Q3 revenue of $41.5 billion was larger than its revenue in any previous full year in company history. Bigger than the $37.4 billion it booked in all of FY2025. Bigger than the prior record of $30.8 billion in FY2022.

Of course, what does that margin chart scream for folks who’ve been around the block? Peak. Memory is the most cyclical business in semiconductors. It’s like a sine wave:

That nice little rolling wave there is the semiconductor cycle.

I love how Doug O’Laughlin visualized the cycle here with various markets moving around on a merry-go-round:

Think of memory’s revenue wave as the weighted average of all the memory-consuming semis markets. Each one sits somewhere on the merry-go-round and carries weight equal to how much memory it buys, so the cycle you see in Micron’s revenues is really just the combined center of mass.

Which raises the question… which quadrant is that center of mass in now, and what would tip it into the next one (top right)?

Well, the merry-go-round still exists, it’s just that the mass of datacenter AI is sooo big that all the other kids on the merry-go-round don’t meaningfully contribute to the center of gravity as much anymore. AI is the dad who jumped on the merry-go-round with a couple of three-year-olds. Dad’s weight wins.

And yeah, I cropped that earlier Micron revenue chart for dramatic effect. Here’s what it looks like now:

YO! The amplitude of that sine wave is much bigger!

So let’s dig into that demand more and figure out if and when more supply is coming online. Can supply counterbalance, or will this memory supercycle persist?

Behind the paywall:

• Demand has two drivers… and don’t forget NAND: HBM is stacked DRAM, so it pulls many wafers off the DRAM market, right as server CPUs pull from the same pool. And AI is eating NAND too.

• The shortage goes out through 2027: CapEx up by half while the shortage horizon slid out a full year, and the construction-clock, node-shrink, and cleanroom limits behind it.

• What the contracts change: sixteen take-or-pay deals lock ~20% of DRAM and a third of NAND to 2030

• SK Hynix and Samsung say it on their own calls: the supply-demand gap widening into 2027, demand for the next three years above capacity, DRAM ASPs up ~60% in a quarter, HBM “sold out”.

• What still cuts the other way: ~80% of DRAM still reprices with the market, and Micron’s own guide already flags a “moderation in the rate of price increases”.

• The scorecard: the handful of variables that decide persist versus revert, each with a read and a date.

Read more

The coherent host is the most valuable. The traditional cloud CPU is the least.

Many readers asked if it matters whether the CPU is x86 or Arm.

Honestly, not as much as made out to be. But let’s go socket by socket.

Quick context

The ISA is the language a CPU speaks. Software gets compiled into that language, and a chip can only run code written for its dialect.

x86 has been the server default for decades. Yet Arm has been gaining in servers, first slowly, then quickly as Graviton, Axion, and Cobalt took hold in cloud, and now inside AI infrastructure as hyperscalers build Arm into their GPU server stacks.

Naturally, everyone asks which ISA is “better” for agentic AI; they’re both just fine.

The more interesting question at each socket is whether the software running there cares which ISA it runs on? Specifically, is the ISA a “moat” at any of the agentic sockets? Let’s see:

1. Coherent host: ISA is irrelevant

The coherent host’s moat is the coherent link to the GPU, not its ISA.

NVLink-C2C connects Nvidia’s Grace CPU to the Blackwell GPU at 900 GB/s, providing a shared address space in which the GPU reads CPU DRAM as if it were local. Vera doubles that to 1.8 TB/s with Rubin. Infinity Fabric ties AMD’s EPYC to the Instinct MI455X at comparable bandwidth. The coherent link is what makes this socket valuable. It’s what no other CPU can replicate without a bilateral design agreement with the GPU vendor... like NVLink Fusion...

Before Grace, Nvidia GPU servers shipped with standard x86 hosts (Intel Xeon or AMD EPYC) connected over PCIe. Grace Hopper (2023) was Nvidia’s first coherent superchip: Grace CPU (Arm, Neoverse V2) connected to the Hopper GPU via NVLink-C2C at 900 GB/s — and Nvidia’s first deployment of the full datacenter CUDA stack on an Arm server CPU. CUDA already ran on Arm through the Jetson embedded line, but this was the server-grade debut.

Grace Blackwell carried that forward; Vera Rubin extends it with a custom Arm CPU (88 Nvidia-designed cores) at 1.8 TB/s to Rubin.

So clearly, ISA isn’t a differentiator for the 800-lb gorilla. Host software runs on either.

What about AMD? ROCm is effectively x86-native. AMD’s coherent platform is built around EPYC, so an Arm port has naturally never been a priority.

The main takeaway is that the ISA is baked into the accelerator platform choice.

NVLink Fusion is Nvidia’s move to open the coherent-host socket to third-party CPUs. Previously, the only CPU that could claim a coherent seat on Nvidia’s backend was the one Nvidia built (Grace/Vera). NVLink Fusion allows other vendors to couple their processors to Blackwell GPUs over the same high-bandwidth coherent link Grace uses. Note that no NVLink Fusion product has actually shipped yet, these are simply announced partnerships. But the partner list includes Qualcomm (Arm), Fujitsu, Intel (x86), and SiFive (RISC-V).

If and when these ship, the coherent-host socket will be accessible to any ISA, so the moat is most definitely not the ISA. RISC-V even... although lots of software porting required.

2. Standard host: ISA nearly irrelevant, and eroding

The standard host’s job is to keep the GPU fed: tokenize inputs, batch requests, stage data over PCIe, manage memory. The CPU needs to work as fast as possible and also move a lot of data. PCIe can become a bottleneck here… hence the coherent host.

The hyperscalers started with x86 standard hosts paired with their XPUs, but that has moved toward Arm. AWS pairs Graviton with Trainium. Google pairs Axion with its gen 8 TPUs.

The feed-the-XPU stack runs on x86 or Arm interchangeably; ISA is not the moat.

Note that there is still an x86 standard-host business in smaller deployments, specifically enterprises and small neoclouds running DGX, Instinct MI355X, RTX Pro 6000 servers, and so on.

In these setups, the host often runs double duty with GPU feeding and application-tier workloads on the same box. That brings legacy x86 software dependencies back into the picture, and ISA does matter. Lower volume, but will grow.

Takeaway: if the host is doing double duty as application processor, then ISA matters. Otherwise, nope.

The two orbits closest to the GPU give the same answer: ISA does not matter there. The three that remain do not all agree. One has a real x86 lock-in story. One has a wrinkle. One… not so much.

3. Thinker: ISA irrelevant, and the category is still forming

The thinker is the GPU-coupled agent: reasoning-heavy work that runs close to the GPU on the scale-out backend, passes large contexts to the model, and needs per-core performance and low round-trip latency above all else. Real-time world model controllers, agents driving tight perception-action loops, anything where a single network hop eats the frame budget.

The thinker commands a higher ASP than the doer. That premium is earned on per-core performance and bandwidth, not protected by a proprietary link. The scale-out backend runs over open InfiniBand or Ethernet, reachable by any NIC-equipped CPU. AMD Venice Classic (Zen 6, high-frequency cores), Intel Diamond Rapids (P-cores, 2027), Nvidia Vera, Qualcomm Oryon, and Arm AGI in its thinker-flex configuration can all reach it. Every major ISA in the same fight.

The software stack for thinker agents consists of Python, orchestration frameworks, and custom agent runtimes built over the last two years, running in containers and targeting both architectures from day one. There is no installed base of thinker software to be locked into, because the thinker socket barely exists at volume yet. Most agentic work today is coding agents, and that is a doer workload. So when the thinker software stack gets written at scale, it will be ISA-agnostic by default.

Hence, the ISA question is already settled before the volume arrives.

4. Doer: ISA matters, with a specific wrinkle

The doer is the most valuable orbit where ISA matters.

The doer is the action agent: coding, tool use, web search, API calls, compiling, running tests, filing pull requests. The metric is threads per watt — run as many agents as possible per rack. The agent’s own orchestration code is ISA-agnostic. Python runs anywhere. The frameworks that drive agents are all multi-architecture.

Yet the ISA matters in the execution environment the agent spins up.

If the target codebase compiles x86 binaries, e.g. a C++ service, a Go application, a Rust binary targeting x86 Linux and runs x86 test suites, you need x86. Yes technically you could use an x86 CPU emulator on Arm but that would kill performance.

So the doer’s ISA dependency is a dependency on what the target codebase compiles to, not on the agent runtime itself. A coding agent deployed to assist a Python web shop has no meaningful ISA dependency, but a coding agent deployed against a legacy enterprise C++ codebase does!

This matters for companies running agents against their own internal codebases. Their internal build pipelines, CI systems, and test suites were likely written assuming x86. Years of accumulated toolchain dependencies. The long tail of enterprise repos still carries enough x86-specific build logic that moving to Arm would require real migration work. That’s where ISA creates genuine friction today. LLMs make it easier than ever, but it’s still work. Maybe Fable can one shot it though...

So doer agentic CPUs, especially in bigger enterprise deployments, have an ISA dependence.

5. Traditional cloud: the strongest moat

Agentic AI is also driving traditional cloud workloads. Agents ultimately query ERPs, hit databases, and call APIs, all of which run on general-purpose cloud CPUs.

This is the orbit with real x86 legacy lock-in. Proprietary enterprise software from the 2000s and 2010s shipped as x86 binaries, sometimes with no source code and no porting path.

Definitely an ISA moat here. But the least value capture in the agentic AI path. Moreover, this ISA moat is shrinking over time.

But it’s real, especially for enterprises.

Where it lands

x86’s moat lives at the outer orbits: the traditional cloud socket, where old software licenses and ISV certifications built a real moat, and the doer socket, where it depends on what the agent actually executes.

The playing field is level at the inner orbits where the software is new, the frameworks were built for containers, and competing on power efficiency matters more than installed base.

The most valuable socket, the coherent host, does not care about the ISA. It cares about which accelerator agreed to share a coherent address space with you.

The closer you get to the GPU, the newer the software and the less the ISA matters.

So, who wins?

Every one of them will tell you it’s them. I’ve spent a lot of time listening to executives at all of these companies, and each frames the comparison around the socket where its own chip happens to win. The market, meanwhile, sees the crowd of launches and weighs them all the same. Up and to the right! But these are really several different CPUs competing for different jobs, and they don’t all carry the same ASP or capture equal value; some sockets are near-monopolies, others are headed for a price war.

To cut through the positioning, I needed a mental model to help answer my questions. Which sockets matter? Which specs matter? Who competes where?

That model is what follows.

Framing: The CPU Sockets that Orbit the GPU

Let’s start with the GPU at the center of our model. Don’t let anyone tell you otherwise. CPUs are not the center of the AI universe; GPUs are. Also, I’m using GPU interchangeably for AI accelerator / XPU / GPU.

In this model, the orbits represent the CPU’s jobs to be done. Each job creates a socket a CPU can fill, and the closer the job is to the GPU, the more valuable that socket is. I know you have lots of questions. But aren’t CPUs general-purpose? Does “closeness” to GPU truly matter? We’ll get to those.

1) First orbit: the Host CPU

Every GPU server is built around a host CPU, usually one or two sockets, that the GPUs attach to. It’s also called the head node, and its job is to run the GPU driver, launch kernels, tokenize and stage data, manage memory, and keep the accelerator fed. Really, it’s just “do whatever’s necessary to keep the accelerator fed.” Some people joke the head node is a glorified memory controller.

The host CPU sits in the token path. Thus, the host CPU must NEVER be the bottleneck. Stalled GPUs are insanely expensive, and the host exists to make sure that never happens.

The specs that matter for a host CPU follow from that fact: high per-core performance (so it can keep up with the GPU and decide what to do next quickly), high bandwidth to the GPU (so it can move data without stalling it), and enough memory bandwidth and capacity to stage what the GPU needs. Core count is secondary.

Nuance: coherent host vs. standard host

Let’s get a bit nuanced.

The host orbit splits in two on the question, “Does the GPU need to use the CPU’s memory as an extension of its own?”

Early in the training era, the answer was no, and the link was PCIe. The host CPU stages data, the GPU copies it across the bus, the two keep separate memory pools. PCIe Gen5 moves about 64 GB/s per direction, Gen6 about 128 unidirectional. That’s fine when the host is just feeding tokens.

Reasoning models changed the calculus. When a model thinks before it answers, its KV cache (the attention state it holds while generating) balloons, and when it spills out of GPU memory, the GPU has to tier it into CPU DRAM and read it back fast, inside the generation loop. PCIe bandwidth is no longer enough to keep up.

So Nvidia built a coherent link, NVLink-C2C, that presents a shared address space, letting the GPU read CPU memory as if it were local. That is Grace Blackwell: a Grace CPU and a Blackwell GPU fused into one coherent module, sharing data at 900 GB/s, about seven times what PCIe Gen5 could move (Vera doubles it to 1.8 TB/s).

The coherent host is the head node rebuilt for the reasoning era, and as context grows, more of its value moves across that coherence line.

So if we split the host orbit to account for the nuance, it looks like this:

Closest in is the coherent host, tied to the GPU over a coherent link and built for KV-cache tiering and long-context reasoning. It is the most valuable seat and the most proprietary because that link exists only between an accelerator and the CPU it was designed with.

A step further out sits the standard host, tied to the GPU over ordinary PCIe and handling the tokenizing, batching, and feeding. It is less tightly coupled, but it has one thing the coherent host does not. It can sit in front of any accelerator, since PCIe is universal. That makes the standard host open, modular, multi-vendor territory.

2) Second orbit: the Agents CPU

Agents are the new workload, and today’s agents are mostly not a GPU job. An agent is given a task, and it loops. It asks the model what to do, gets back an instruction, executes it (run some code, query a database, search the web, call an API), observes the result, and asks again.

The model’s thinking is one step in that loop (on the GPU).

Everything around it, the parsing, the tool calls, the code execution, the state management, runs on a CPU. That work would overwhelm the host CPU if you let it.

The host is in the token path and cannot afford to be busy doing anything else, and there are now thousands or millions of agents pinging the model, not a handful of humans. So the agent work gets offloaded to standalone CPUs dedicated to running agents. The goal there is different from the host, namely to run as many agents as possible in a given power footprint.

The metric is threads per watt, not raw per-core speed. Lots of cores, lots of threads, low power, enough cache to keep each agent’s small working set on-chip.

This orbit is where the CPU demand is exploding. It is already most of the reason the CPU-to-GPU ratio in AI infrastructure has moved from roughly 1:4 toward 1:1.

Nuance: thinkers vs. doers (GPU-coupled vs. CPU-bound).

Not all agent work is the same; some should run in the same datacenter as the GPUs, but much needn’t. Vik’s Yellow Pages splits CPUs on exactly this axis, reasoning vs. action, and the same split applies to the work itself.

The doers (CPU-bound, action agents). Here, the heavy lifting is on the CPU, and the GPU sits idle while it waits for the CPU. Coding is the most popular and a great example. Agents write code, compile it, run it, open a pull request, etc. Compiling and running can takes seconds, which dwarfs any network latency. So the extra hop for the GPU to talk over the front-end network to a CPU rack in another datacenter is no big deal. Hence, the doers can leave the GPU data hall and run on cheaper CPUs somewhere else.

The thinkers (GPU-coupled, reasoning agents). Here, the heavy lifting is the model’s thinking, and per step the agent is mostly waiting on the GPU. A reasoning-heavy task that ships a large context to the model every step, then waits, is GPU-coupled. This work aims to stay close to the GPU, on the scale-out backend network inside the data hall, so round trips and large context transfers remain fast. And because each agent mostly waits on the GPU, one fast core can host many of them, so this socket prizes per-core speed first and core count only second.

Take, for example, agents that drive real-time generation and world models. Fei-Fei Li’s World Labs frames a world model as a renderer, a planner, and the real-time loop connecting them, the same perception-action cycle that drives an embodied agent. The GPU renders a frame, a CPU reads the user’s or agent’s input and conditions the next step, and around again, all inside a frame budget of 16 to 33 milliseconds (30 to 60 FPS). A single network hop eats that budget, so the controlling CPU has to sit on the backend, next to the GPU. It is a reasoning job, not an action one, so the CPU does little computing but has to be fast and close. Thus, we need high per-core performance and low latency, not necessarily high thread count.

And as the world’s persistent state grows the way a long-context KV cache does, it pushes toward the coherent host. As AI moves from words to worlds, more of the agentic workload lands in this tightest, most GPU-coupled corner.

Side note: The strongest case for keeping even the doers on the backend is the agent’s growing context. A long session accumulates a large context, and every turn the model attends over all of it through the KV cache, the attention state it builds from those tokens. Recomputing that cache each turn is wasteful, so the inference system keeps and reuses it, tiering it out of GPU memory into a dedicated context-memory rack as it grows (Nvidia builds this with BlueField DPUs, fast storage, and KV-aware routing in Dynamo). But that pulls the KV cache onto the backend, not the doer. The agent’s memory is really just tokens, and its durable memory (files, a vector database, retrieval stores) never touches the GPU at all. The doer ships a prompt and a tool result, text in and text out, and the inference system turns that into KV and caches it. So the context-memory rack is real backend infrastructure, and the doer’s compute can still leave.

One caveat: if the work produces a very large artifact the model then has to reason over, you keep it near the backend so the big payload does not crawl across a slow link to reach the GPU.

3) Third orbit: traditional cloud CPUs

The outer orbit is the CPU as we have always known it, the general-purpose server CPU running web services, databases, microservices, and VMs.

Cloud CPUs are not in the AI loop directly. But agents often reach out to all of it; the ERPs, the CRMs, the databases, the web servers, and so on. And the more that agents multiply, the more ordinary cloud-CPU demand they pull along behind them.

Nuance: there is a portfolio here too

I know you all know this, but to be pedantic:

Traditional cloud buyers have always chosen cloud CPU SKUs by workload. General-purpose, compute-optimized, memory-optimized, storage-optimized, etc. A per-core-licensed database wants fewer, faster cores, because the license is priced per core. A web tier wants density and throughput per watt. That same split, fast cores vs. dense cores, runs straight through every vendor’s lineup, and it is why “the cloud CPU” was never one SKU.

Where is the Value and Who Competes at Each Socket

Put it together and you have a spectrum of CPUs that all benefit from agentic AI, but not in the same place or to the same degree. Map them onto the orbits above, and each lands in a different socket: coherent host, standard host, thinker, doer, or traditional cloud, and some in more than one.

Those sockets are not worth the same. One is close to a monopoly; the rest will be in a price war.

This is what current head-to-head comparisons miss. Every vendor counter-positions, framing the whole “agentic CPU” question around the socket where its own part wins. Nvidia leans on the coherent host and thinker, AMD and Arm on the doer’s rack-scale density, and so on. The marketing then reads as if everyone leads, because each is measuring a different socket.

To know who actually captures value, you have to ask two things of each chip. Which socket is it really built for, and what is that socket worth?

Behind the paywall, I answer both.

First, the money. What each of the five sockets is worth, which one is a near-monopoly, and which are headed for a price war.

Then the field. Where Nvidia’s Vera, AMD’s EPYC, Intel’s Xeons, Arm’s AGI rack, and Qualcomm’s CPU each land, who owns the moat, who is the most complete, who is shut out of the one socket that actually pays, and who is renting a way in.

Read more