SemiAnalysis: The Real 100x in AI Comes from Hardware-Software Co-Design, Not Faster Chips
Dylan Patel explains why optimizing models, kernels and silicon together delivers breakthrough gains, and why the CUDA moat was never really about CUDA
Dylan Patel, founder of SemiAnalysis, is making a provocative argument that contradicts the conventional narrative about AI progress. The biggest gains in artificial intelligence don't come from faster chips alone. They come from co-designing software and hardware together across the entire stack, turning what could be a 2x improvement here and a 2x there into a 100x leap forward. In a wide-ranging conversation with Sequoia Capital partners Shaun Maguire and Sonya Huang, Patel laid out why DeepSeek's expert layers were optimized specifically for Nvidia's Hopper architecture, why Google's TPUs struggle to run certain models efficiently, and why the so-called CUDA moat was never really about CUDA at all.
Hardware-Software Co-Design Drives Exponential Gains
Patel fundamentally disagrees with the view that AI efficiency gains over the past three years have primarily come from hardware improvements with some model-level algorithmic advances sprinkled in. His research shows something quite different. On the most optimized deployment of DeepSeek, there has been roughly a 30x improvement from Hopper to Blackwell over the last three years. But the total intelligence per watt has improved closer to 60x for equivalent quality, with some measurements showing cost decreases of similar magnitude. The delta comes from co-optimization across layers, not from any single improvement.
The shapes of all the expert layers in DeepSeek V3 were optimized specifically for Hopper architecture, Patel explained. For V4, they are being optimized for Blackwell and Huawei's chips. This creates an interesting dynamic where Google's TPUs, despite being objectively amazing chips that run all of DeepMind and handle pre-training for Anthropic, actually perform poorly when running DeepSeek models. The model was simply not designed for that hardware. Conversely, other models run exceptionally well on TPUs but struggle on Nvidia GPUs.
The level of optimization goes deeper than most observers realize. Patel noted that considerations include expert shapes, network input-output patterns, how collectives are handled, and the arithmetic intensity of attention mechanisms. Everything is co-optimized between the model, the hardware, and the infrastructure software in between. It becomes nearly impossible to disentangle where the gains are coming from because the real breakthrough happens when you optimize across all three layers simultaneously.
Model Architectures Diverge Based on Hardware Choices
OpenAI and Anthropic are converging toward meaningfully different model architectures, according to Patel's analysis. OpenAI's models are much more sparse, which brings certain benefits and pushes them toward different hardware optimization. Anthropic's models are still sparse but more dense in general, creating different trade-offs. These architectural choices are not made in isolation from hardware. The network topology matters significantly as well. All of Nvidia's chips connect to NVLink switches, which can only connect 72 GPUs. Google's interconnect has no switch but can connect 8,000 chips at super high bandwidth by passing through other chips. These fundamental infrastructure differences influence what kinds of model architectures work best on each platform.
The implications extend to which labs will prefer which hardware. The way OpenAI's models are headed makes TPUs potentially a terrible choice for them, Patel suggested. Meanwhile, the direction Anthropic and Google's models are taking could make GPUs a poor fit for training. He emphasized this is not about one architecture being objectively better than another. It comes down to co-design. You cannot measure them in isolation when the optimization extends all the way up to the model layer.
The CUDA Moat Is Actually an Ecosystem Moat
The narrative around Nvidia's CUDA moat is misunderstood, in Patel's view. The moat was never really about CUDA itself. What happened instead is that the downstream products became more optimized for Nvidia hardware. DeepSeek, Qwen, Alibaba, Tencent, and Xiaomi all released models co-designed for GPUs. When users try to run these models on TPUs, they often do not perform well. Google would need to create their own open source model ecosystem, which they have started with the Gemma models, to establish similar network effects for TPU deployment.
The traditional CUDA moat argument assumed tens of thousands of customers who each need programmability and compatibility. That premise is changing rapidly. There are only on the order of tens of major model companies, not thousands. These labs are willing to write custom kernels for different chips. Anthropic uses TPUs extensively for training and Trainium plus GPUs for inference. Model companies like Claude and newer code generation models have become quite capable at optimization work, commoditizing much of the software stack. The real lock-in comes from the fact that if you are an inference API provider or a reinforcement learning company trying to customize open models for business use cases, you are downstream of an ecosystem that predominantly uses Nvidia. The expert dimensions and hidden dimensions are structured for GPUs, so you need GPUs to run them efficiently, even if you personally do not care about writing CUDA kernels.
InferenceX Tracks 60x Annual Cost Improvements
SemiAnalysis launched InferenceX, a living benchmark that runs the latest models on over $50 million of donated hardware every single day. The initiative became possible because Patel had built enough credibility in the ecosystem to secure contributions from CoreWeave, Crusoe, Nebius, Oracle, Microsoft, Amazon, Google and OpenAI. The benchmark collaborates with SG Lang, vLLM, and now Radix Arc and Infract, which lead open source inference optimization efforts. Once TPUs and Trainium get added, the total donated hardware value should exceed $100 million.
The benchmark runs approximately 15 different chip types on all the latest models every day, including the best models from Chinese labs like Moonshot, Alibaba, DeepSeek, and Qwen, as well as top US open source models. The system sweeps across many different configurations and optimization types, then publishes all results and configurations publicly. This creates a Pareto optimal curve, solving a major problem in inference benchmarking where people compare suboptimal configurations to make their own hardware look better. Anyone who wants the optimal deployment can download open source containers from InferenceX and run that configuration, or even auto-download the most optimal point for each model.
The throughput versus interactivity curve that InferenceX measures has become the most important curve for the industry, Patel argued. Different workloads fall at different points on this curve. Some applications need super low latency and very fast response for individual users, where batch size is low and techniques like speculative decoding are valuable. Other workloads involve batch processing many documents overnight where cost per token matters far more than speed. The infrastructure currently treats AI as one-size-fits-all, but over time the market will segment across this curve. Anthropic's Claude fast mode costs substantially more than regular mode. OpenAI has priority queuing. The data shows model cost for equivalent quality has dropped roughly 60x annually, an incredible pace driven by software optimization, hardware improvements, and their co-design.
Memory Bandwidth and Power Density Are Key Bottlenecks
When asked about the biggest bottlenecks he is tracking across any level of the stack, Patel highlighted memory capacity and bandwidth. The NAND cell was invented roughly 25 years ago. The DRAM cell was invented about 40 years ago. There has been no major breakthrough in the fundamental cell structure during that entire period. Progress has come from stacking more HBM and running it faster, but new innovations are coming where instead of stacking HBM separately from the chip, memory gets stacked directly on the chip. That makes bandwidth explode, and several companies are working on this approach.
Power density represents another critical constraint. For the last two decades, data center and desktop chips have peaked at roughly one watt per square millimeter of silicon. If you look at a chip that is 100 square millimeters, power consumption generally runs around 100 watts or slightly less. The newest Nvidia silicon and newest TPU silicon still fall in that one watt per square millimeter range. Chips are now reaching 1,400 watts, with the next generation Nvidia Rubin targeting 2,000 watts and Rubin Ultra potentially hitting 4,000 watts. But these increases come from adding more silicon area, not increasing power density.
The exciting development is work now in progress to pump substantially more than one watt per square millimeter into silicon. This would mean needing less silicon area to achieve the same compute, though running at higher power. Obviously it creates thermal challenges and electrical interference issues, which is why it remains a hard engineering problem and why the industry has been stuck at roughly one watt per square millimeter. But solving this constraint could unlock significant gains.
The Compute Crunch Will Persist as Models Expand TAM Faster Than Supply Grows
Every quarter substantially more compute gets deployed than the prior quarter, with more data centers built. This year will see 20 gigawatts deployed even accounting for delays, and next year over 30 gigawatts. Yet Patel expects the compute crunch to continue for the foreseeable future. The reason comes down to a fundamental dynamic where the total addressable market for useful AI work expands faster than compute capacity increases.
The TAM for models like Mythos 5 and Fable is not just 2x that of Opus, he explained. The model is so much better and can handle so many more tasks that the addressable market has grown far more than 2x. Yet the world's compute did not double in the six to eight months between when Opus 4.5 launched and when Fable and Mythos arrived. Models are improving in capability faster than compute is scaling. Anthropic's margins on Opus 4.8 tokens run north of 80 percent at API pricing, though total corporate gross margins get pulled down somewhat by deals through Bedrock and Vertex. With margins that high, Anthropic has the financial capability to pay above market rates for every GPU they acquire. The same logic applies to other leading labs with strong unit economics.
The question becomes what happens if model progress stalls. Patel's conversations with engineers at Anthropic and OpenAI suggest they remain highly confident that progress will continue at a rapid pace, perhaps even accelerating because models themselves now help write the infrastructure and optimize the code to launch the next generation sooner. This creates a kind of pseudo-recursive self-improvement loop. If that assessment is correct, the compute crunch persists. If model capabilities plateau, then the tide could turn as supply catches up to demand.
Neoclouds Exist Because Hyperscaler Expertise Did Not Transfer to AI Workloads
The emergence of neoclouds like CoreWeave and Crusoe might seem surprising given the hyperscalers' advantages in scale and infrastructure. Patel wrote a report in 2023 that upset Amazon, called Amazon Cloud Crisis, explaining why Amazon had been the best cloud provider for traditional workloads. Their Nitro NICs offered tenant isolation by running the hypervisor on the NIC and selling all cores to customers. They bought raw NAND and built custom SSDs for lower storage costs. Their custom Graviton CPUs drove down per-core costs. All of these innovations worked brilliantly for the traditional CPU-based cloud world.
But in the AI cloud, much of this expertise became irrelevant or even detrimental. The Nitro NICs hurt performance and still deliver worse performance today, though they have improved with iterations. Security features designed for multi-tenant timeslicing do not matter when customers rent entire 72-GPU racks under long-term contracts rather than single GPUs for a few hours. Custom networks optimized for traditional workloads at Google and Amazon actually worked against AI performance. Microsoft's cost savings from building their own data centers backfired when their data center teams could not handle the need to rapidly double forecasts, forcing them to lease neocloud capacity.
Performance advantages and time to market explain much of the neocloud opportunity. The people building these companies are highly leveraged equity owners who get rich by delivering compute faster. They came from backgrounds like Bitcoin mining that taught them to operate in high-fluctuation markets. Meanwhile, massive hyperscaler organizations offer no individual wealth creation incentive for building a data center six months faster. Nvidia CEO Jensen Huang actively supports the neocloud ecosystem because he desperately wants a multipolar world. A world where only the hyperscalers build compute and only OpenAI, Anthropic and Google have leading models would leave Nvidia with drastically reduced leverage. Five years from now, Crusoe and CoreWeave existing means Google's TPU will be weaker and Amazon's Trainium will be weaker, preserving GPU market share.
Data Center Utilization Varies Wildly Based on Operational Sophistication
Not all gigawatts are created equal. Trainium sells at sub-$10 billion per gigawatt rental rate to Anthropic and OpenAI. GPUs before the recent compute shortage typically went for $12 billion to $13 billion per gigawatt from neoclouds and Amazon. The SpaceX deal with Google reportedly reached around $25 billion per gigawatt annually, a massive premium. Data center colocation pricing used to run $60 per kilowatt per month and now transacts anywhere from $120 to $160, with some deals hitting $200 for customers with weaker credit ratings in premium facilities, or as low as $80 in India where grid reliability and connectivity are inferior.
Operational sophistication creates enormous value differences. Google will deploy 1.5 gigawatts of hardware in a one gigawatt data center because they understand workloads so deeply they can slosh power around. Instead of hardware running at 60 percent to 70 percent power consumption, they use the full gigawatt continuously. Some operators do deals with utilities where the grid can sustainably provide one gigawatt, but for all but three days per year can deliver two gigawatts. They deploy two gigawatts of capacity and use backup generators and batteries to handle those peak constraint days. This requires supreme management of workload, backup power, and onsite generation capability. Companies that execute on these strategies can effectively sell twice as many gigawatts from the same grid connection, or sell capacity in locations where others cannot build at all.
A gigawatt given to Anthropic generates objectively more revenue than a gigawatt given to OpenAI based on current monetization, though both companies could sell every gigawatt they currently have given rate limits and token maximums. CoreWeave delivers objectively better GPU compute performance than Amazon, Google or Microsoft based on InferenceX testing of performance and reliability. But CoreWeave must sell capacity six months before it comes online and use those contracts to secure debt financing for purchase orders they have already issued. SpaceX by contrast tells customers the capacity is running now and you can buy it immediately, commanding premium pricing because of the balance sheet strength to carry inventory.
Space Data Centers Unlikely to Matter Within Five Years
Asked what percentage of inference compute will happen in space on a 10 to 15 year time frame, Patel offered a nuanced view that cuts against some of the hype. He does not think space data centers will matter much in the next three to five years. The real factor comes down to the cost of building power on terrestrial land and how much power can be deployed terrestrially. But by 2040, he expects the vast majority of compute will operate in space. By 2030, just OpenAI and Anthropic combined will likely command over 100 gigawatts. Adding Meta, Google and other players brings the total to a humongous deployment dedicated to inference. By 2040, the industry will be operating at terawatt scale.
The curve of productivity gains from AI will drive inference deployment to become one of the biggest markets in the world, much larger than oil in Patel's assessment. If that forecast proves correct, terrestrial power constraints will eventually force migration to space. But in 2030, he expects sub-1 percent of incremental compute going to orbital facilities. The technology and economics need another decade to mature before space becomes the primary deployment environment.