DeepMind's Yao Shunyu: Pre-Training Isn't Over, Heroism Has Passed, and the Next Frontier Is Long-Horizon Tasks
Zhang Xiaojun Podcast interview recorded March 2026
Yao Shunyu, a research scientist at Google DeepMind who previously spent time at Anthropic training Claude 3.7 and 4.5, offers a brutally honest assessment of where AI stands today: the era of individual heroism has ended, pre-training still has significant runway, and contrary to widespread belief, models haven't hit any fundamental walls. Speaking from his position working on Gemini 3, Yao delivers one of the more candid insider perspectives on what's actually happening inside the world's leading AI labs.
Pre-Training Is Far From Dead
Perhaps Yao's most contrarian view challenges the prevailing narrative that pre-training has plateaued. "My experience is that it hasn't," he states flatly when asked about Scaling Laws reaching their limits. "And my feeling is in the next four months I don't see any signs of it ending either." This directly contradicts what many observers believed after the initial excitement around reasoning models and post-training improvements.
Yao argues that most people who claim to have hit a wall are actually encountering bugs, not fundamental limits. "From my observation, I think probably the vast majority of people who hit a wall, it's because of the third reason—there's a bug somewhere in their work that they haven't noticed themselves. So they think it's reached its limit." He points to incorrect scientific assumptions about token horizons, data choices, or simple implementation errors as the real culprits.
The technical nuance matters. When pressed on what's actually improving, Yao explains that "the model's ability to learn things is getting stronger and stronger. It used to take a lot of effort to get the model to learn to do something. But now it probably doesn't require that much effort. The most important thing is you need to clearly define the problem and figure out how to build the right data."
Why Coding Exploded First
On the question of why agentic coding became the breakout application, Yao provides unusual clarity on what actually happened at Anthropic. The advantage coding had was structural, not arbitrary. "The coding scenario has two biggest advantages. The first advantage is its reward signal—its feedback signal—is very well-defined," he explains. When implementing a feature, inputs and outputs are testable. Success or failure is binary and immediate.
The second advantage was data infrastructure. "GitHub has aggregated over the past few decades a lot of high-quality code written by many excellent programmers. And starting from that code you can build a tremendous number of environments." This combination of clear feedback and abundant high-quality data made coding the ideal first domain for large-scale reinforcement learning.
But there was also randomness involved. Yao reveals that Claude 3.5's coding strength emerged somewhat accidentally from technical choices that happened to favor code quality in the pre-training data. "If you just randomly collect from the web without any data filtering, naturally the quality of code data would be a bit higher than others because if you look at web pages you'll find GitHub's quality is significantly higher than other normal web pages." Once Anthropic noticed this advantage in the wild through user feedback, the company went all-in—a decision enabled by its unique organizational structure.
Anthropic's Structural Advantage and Its Limits
Yao describes Anthropic as having "very strong execution" enabled by top-down decision-making that most companies can't replicate. The secret? Technical credibility concentrated in the founding team. "Its technical leader is actually a cofounder of the company," specifically pointing to Jared Kaplan and Sam McCandlish. "They make this decision themselves. It's their company. So they have the authority to do this top-down."
This structure allowed Anthropic to make aggressive bets on coding when the opportunity appeared. "I think when I joined, everyone had definitely already seen this thing could be done and was important. But didn't quite know how to do it. And when I went in, I was researching with everyone how to do it." The company could rapidly mobilize resources around large-scale reinforcement learning in ways that organizations with separated technical and business leadership cannot.
However, Yao left Anthropic after about a year, citing both cultural shifts as the company scaled rapidly and disagreement with CEO Dario Amodei's "anti-China stance," which he describes as accounting for roughly 40% of his decision to leave. "I actually didn't really agree with Dario's anti-China stance. I think as a company CEO, for him personally whatever views he holds, I think it's fine. But as a company CEO, I think pushing this view to such an extreme was a very emotional reaction."
Google's Different Playbook
At Google DeepMind, Yao found a completely different organizational model that nonetheless works. Unlike Anthropic's top-down betting, Google operates with a portfolio approach. "Big companies and startups—their playbooks are fundamentally different. Because for startups, what's important is to make bets. I have to bet on something. If I want to bet, it means there's risk."
Google's advantage is different: "A big company's mindset might be not only can I minimize the gambling aspect, but I can also have reserves in every area. And then if anything succeeds I can catch up. And if I succeed at something myself I might even take the lead." This means Google doesn't need the same kind of concentrated technical leadership that Anthropic requires.
Pre-training at Google has become "a very, very controllable thing, a predictable thing. You can know the next generation won't be bad." The organization has clarified who owns what, turning model development into an engineering project that plays to Google's traditional strengths. "Google's engineering management ability is very strong. So it can slowly do it well."
The inflection point for Gemini wasn't just Gemini 3, according to Yao—it was the one-two punch of Nano Banana going viral and building user volume, immediately followed by Gemini 3 retaining those users through superior model quality. "I think if there were only Gemini 3, it probably wouldn't have had such great results. Because when your market share is less than even 10%, whether your model is slightly better or worse, it just spreads too slowly."
The Next Paradigm: Long-Horizon Tasks
When asked about the next major breakthrough, Yao points to long-horizon tasks as the critical frontier, not multimodal generation or continue learning. His vision is captured in a slogan he uses internally: "train with finite context, use as infinite context." The goal is to let models handle very long tasks within limited context windows through intelligent context management—essentially teaching models to selectively remember and forget like humans do.
"Human context is actually very, very short. If you ask me now what I ate last night, I can't remember at all," Yao says. "Because it's not critical to my current scenario. So I choose to forget it. So human context is essentially very short. But they can selectively forget and selectively retrieve to bring back these important information relevant to the current scenario."
This direction is intimately connected with ML coding—using AI to automate AI research itself. Yao believes that within six to twelve months, "AI will do experiments itself" in the sense of completing entire research projects from start to finish: designing experiments, running them, analyzing results, proposing new hypotheses, and iterating. "The chain is not yet complete. But I think this chain might be the next thing to gradually become complete."
Distillation, Benchmarks, and What Actually Matters
On the topic of Chinese labs distilling from frontier models, Yao makes a subtle but important distinction between "hard distillation" and "soft distillation." Hard distillation—taking tokens generated by Claude and training directly on them—is both unethical and "intellectually rather foolish" because it shows companies "don't even know what they want to do." But soft distillation, using other models as assistants in your data pipeline or as evaluators, "is actually, I think, commercially a bit of a gray area. But from a technical perspective, it's quite interesting."
He argues that Chinese labs using this approach "may have become pioneers in multi-agent training. And it's true multi-agent. Because if they use models from different companies with these smarter approaches and integrate them into a single training system, each model's distribution might be very different."
As for the current benchmark race, Yao is dismissive. "I think in a sense, none are that worth paying attention to." The public benchmarks everyone competes on have essentially saturated. "Everyone would compete for first place on various benchmarks to prove their model's basic capability is strong. This thing, actually by now it has reached public attention—those benchmarks are somewhat maxed out."
The Death of Individual Heroism
Perhaps Yao's most striking thesis is that AI research has fundamentally shifted from individual brilliance to collective execution. "I think everyone right now is basically—everyone is a surfer. Fundamentally it's a wave, not the surfer," he says. "The era of individual heroism for language models has probably passed."
When did heroism end? "It was the Transformer moment. At that point when the technology hadn't yet reached the scale-up stage, the person who discovered that technology might be a hero. After that technology was found, for probably a long time from the model side, it's all been more about collectivism."
This isn't false modesty—Yao genuinely believes the field has become simple enough that individual contributions matter far less than systematic execution. "AI as a field is fundamentally simple. I don't think there's any—except maybe that leap moment where the idea might require some really deep insights—in the process after that many ideas are actually very trivial. Very stupid, basically. Anyone could think of them. Anyone could do them. It's just that you got lucky and happened to seize the opportunity to do it."
What matters now isn't brilliance but reliability. "I think in this industry, the most important trait is being reliable, doing things carefully, and being responsible for what you do. This is the most important trait." He goes further: "It just feels like this AI thing doesn't really require much brains. Doesn't require much brains. Really doesn't require much brains."
The Wrapper Problem and Product Moats
On why application companies like OpenClaw and Manus sold to model labs despite strong initial traction, Yao identifies the fundamental issue: insufficient escape velocity from the model companies' gravity. "I think for wrappers to survive in the current environment there are two approaches. One approach is escape fast enough—my growth is fast enough that by the time model companies catch on I've already captured significant user mindshare."
Even Cursor, "pretty much the fastest-growing startup I can think of" in the AI-native coding space, now finds itself in "a fairly competitive relationship" with Anthropic after Claude Code launched. The core problem is winner-takes-all dynamics in productivity tools. "When it comes to coding, at its core it's essentially a professional need serving professional users. It's a productivity tool. A common scenario with productivity tools is winner takes all."
The other survival path is staying small enough that model companies can't be bothered to compete—which Yao suggests Midjourney has successfully done. "The market is so small that perhaps even though you could say Gemini could make an effort to replicate what Midjourney does, it might take some effort, some money, some data to pull it off, but it's small enough to the point where Gemini probably wouldn't want to spend much time on that."
Why Programmers Should Worry
Yao doesn't sugarcoat the future for software engineers. When asked when programmers will be completely replaced, he responds: "I think that day will come. But it won't come all at once." He estimates that eventually "now 1/1000 of the people do the work of everyone in the past, earning 100 times the current salary."
His assessment of his own coding productivity is stark. Conservatively, "the model wrote over 90%. If not conservative, maybe 99% or 100%." The remaining portion he writes himself "is becoming less and less." As a researcher, his experimental velocity has increased "20 or even 50 times faster" compared to a year and a half ago.
The advice for programmers who want to survive? "Embrace new things. I think that's very important." Future programmers will need to be technically strong, understand how their work fits into larger organizations, and have strong planning ability to break complex tasks into pieces that AI can handle. But Yao cautions that these requirements may shrink further: "Maybe in six months you come ask me, I find that the last thing AI can already do. Then only two things remain."
What's Actually Not Working
When asked about his PhD work in theoretical high-energy physics, Yao reveals the experience that shaped his views on what makes good research. "I think for myself personally, I learned a lot, grew a lot. But for this world, it didn't produce any contribution." The problem wasn't the quality of the work—"No one would say my PhD papers were bad"—but rather the fundamental structure of the field.
High-energy theory has progressed beyond what experiments can verify, leaving only mathematical self-consistency and the subjective judgments of senior researchers as evaluation criteria. "When this field completely lacks experiments and objective standards, there definitely won't be just one framework that appears. At this time who does well, who doesn't do well, actually depends on the subjective judgments of some old-timers in the field."
This experience taught Yao to "do things with relatively objective evaluation criteria, or from another perspective, do things that can have an impact on this world." It's why he's so allergic to vague claims in AI that can't be falsified—like certain prominent figures whose statements are "not even wrong because it's not well-defined."
The China-US Divergence
Yao sees the AI competition between China and the US heading in fundamentally different directions, shaped by each region's historical strengths. The US focus on enterprise and productivity software reflects easy profit margins and straightforward business models—"I help you write code, my cost is 150 a month, I sell it to you for 200, I make 50."
China's consumer-focused approach draws on a decade of experience with sophisticated indirect monetization. "What China is really strong at is the consumer side. It can come up with really, really complex product features or structures and in a way that seems very indirect to you, in a very unnatural way, to snowball that profit." The example he gives is Douyin, which doesn't charge per video but quietly monetizes through ads, live streaming, and e-commerce.
This divergence means the super-app narrative dominating Chinese AI discussions makes little sense for the US market. "If you ask me which of OpenAI and Anthropic would die first—of course they won't really die, just which would become less important first—at that time I would think maybe Anthropic would become less important first. But later first OpenAI got punched by Google, then Anthropic itself got on track. So now it seems Anthropic has more advantage."
On whether American companies understand ByteDance, Yao is blunt: "My feeling is no. Not yet." While everyone recognizes ByteDance as severely undervalued and dominant in consumer markets, true comprehension of how the company operates remains elusive.
The Real Bottleneck
Despite all the technical progress, Yao identifies product definition as the current limiting factor. "I think now it's the latter—the things you want to do have reached a plateau. Right now we know there's a chatbot, you can teach it to do this. And then there's coding, you can teach it to do this. And then we don't know what else to teach it."
This isn't a model capability problem. "This model is still a very smart kid. You can actually teach it many things. But we humans as teachers now don't know what the next thing to teach is, or how to reasonably teach it using current paradigms." The implication is that product managers who can identify the right problems and define clear training objectives have become more valuable than researchers who can implement solutions.
Looking at OpenAI's current position, Yao notes the irony that by not fully replacing search before Google caught up on chatbots, "OpenAI saved Google's life." Had OpenAI taken search all the way before Google responded, "then it would be very tough. But fortunately, OpenAI did this thing first, then made Google realize this thing is important. But it didn't take this thing all the way."
The final assessment is matter-of-fact: "I think no one's position is secure right now. The form of AI still has a long way to go. We're not at any endgame yet." For a field supposedly racing toward AGI, that's a sobering reminder that the real challenges may be less about scaling compute and more about figuring out what to build.