Yao Shunyu: Pre-Training Hasn't Hit the Wall, AI Research Is Fundamentally Simple, and the Era of Individual Heroism Is Over
Former Anthropic and Google DeepMind researcher discusses training Claude 3.7, 4.5, and Gemini 3 — Recorded March 2026
Yao Shunyu arrived at the interview wearing amber-tinted glasses that shift color with the light. There are two Yao Shunyus in Silicon Valley's AI circles, both graduated from Tsinghua in the same class. The first jumped from OpenAI to Tencent in 2025. The second, today's guest, moved from Anthropic to Google DeepMind the same year. Over the past two years, he has contributed to key models including Claude 3.7, 4.5, and Gemini 3. His career trajectory tells a bigger story about how AI research has evolved from an era of individual brilliance into one of collective execution.
The Most Controversial Take: Pre-Training Progress Hasn't Slowed
Standing in Q1 2026 as a frontline researcher, Yao offers a perspective that contradicts the prevailing narrative. "I think not at all," he says flatly when asked if model improvement is slowing down. "My experience is that it hasn't. And my feeling is in the next four months I don't see any signs of it ending either."
This view is somewhat controversial. A few months ago, many people were discussing whether Scaling Law had reached its limit. Yao's take is unequivocal: it hasn't. "My guess would be when someone thinks a pattern has reached its limit, it's basically one of these two situations. One situation is they feel the applicable range of this pattern has reached its limit. Another possibility is this person feels that one of its conditions can no longer be met, like data has hit a wall. But actually there's a third possibility—there's a bug somewhere in their work that they haven't noticed themselves."
From Yao's observation, he believes the vast majority of people who hit a wall do so because of the third reason. "For example, one possibility is when you're working on Scaling Laws, some scientific assumptions weren't quite right. Or there's simply a bug. I don't think this is surprising in the industry. Many times fixing a single bug brings far greater progress than some fancy tricks."
Pre-training has actually gotten stronger in the past few months, Yao reveals. "Model pre-training has actually gotten stronger in the past few months. My feeling is in the next four months I don't see any signs of it ending either." When asked why people think it's reaching its limit, he says: "Obviously I don't know why people think it's reached its limit, because I myself don't feel it's reached its limit."
What Drives Progress: Data, Compute, or Algorithms
On the question of what drives model capabilities, Yao believes all three contribute but data and compute are strongly correlated. "When your compute goes up you'll naturally attract more data. When data goes up you'll naturally need more compute." For algorithms, he argues algorithmic progress often has phase transitions—there's a phase where you haven't figured out what to do at all, and algorithms are extremely critical. But after discovering the most important algorithmic insight, improvements become gradual.
"For example, from the perspective of language model pre-training, that algorithmic leap was the development of the Transformer. But after the Transformer was discovered, it's been mostly gradual and smooth," he explains. Within relatively clear frameworks like pre-training and post-training, the main drivers are indeed compute and data. But in other directions like multimodal generation, it's still a scientific problem that hasn't been fully figured out.
Yao pushes back against the notion that AI is hitting fundamental limits. "I feel AI doesn't give people the sense that it's hitting a wall because first, you can try many things. Second, it's not that everyone has run out of ideas with no ideas left to try. More often it's that there are too many ideas, need to try them one by one." He predicts that very soon, within six to twelve months, AI will begin doing experiments itself—not just writing code and running experiments, but also analyzing results, proposing new hypotheses, designing new code, and running new experiments in a complete loop.
How Anthropic Made the Coding Bet
Yao joined Anthropic in September 2024, just as the company was gearing up for its most important pivot. At that time, the goal was clear: large-scale reinforcement learning applied to coding. "I think when I joined, everyone had definitely already seen this thing could be done and was important. But didn't quite know how to do it. And when I went in, I was researching with everyone how to do it."
The origin of this bet traces back to Claude 3. After that model was released, many people on Twitter discussed that Claude 3 seemed to write code better than GPT-4. "So being able to do one important thing better than GPT-4 was quite impressive," Yao recalls. This was market feedback that Anthropic took very seriously.
But why coding specifically? Yao points to two major advantages. "The first advantage is its reward signal—that is, its feedback signal—is very well-defined. A software engineering task is often: I need to write some code to implement a feature. This is something very easy to test. Your input and output match up, then it means your implementation is successful." The second advantage is data: GitHub has aggregated decades of high-quality code from excellent programmers, providing a tremendous foundation for building environments.
The resulting model, Claude 3.7, became a watershed for Anthropic's post-training. "Before 3.7, post-training was in a relatively small-scale, more like patching things up, that kind of state for the model," Yao explains. During that era, no one really figured out how post-training should scale up. But during that period, whether OpenAI, Anthropic, or even China's DeepSeek, they realized how to scale this up: "You have to find the right environment where the feedback signal is clear enough and the environment itself is a strong data source. And then on top of that you can make the training very stable. Then it can work."
The development from 3.7 to 4.5 involved both pre-training and post-training at much larger scale, though Yao notes that in terms of paradigm, there wasn't anything particularly major that changed. The key was systematic execution at scale.
Anthropic's Unique Organizational Advantage
What enabled Anthropic to execute so decisively on coding? Yao points to the company's unusual organizational structure. "Anthropic as a company, it can implement this relatively top-down mechanism, is a very unique thing." The crucial factor is that the person making technical decisions must also be the company's decision maker. "You have to be technically convincing, then you can convince the researchers below to do this thing. On the other hand, you have to be the decision-maker at the company, you have to be able to take responsibility for the company."
Anthropic has this going for it: its technical leaders, Jared Kaplan and Sam McCandlish, are both cofounders of the company. "They make this decision themselves, it's their company. So they have the authority to do this top-down." Yao contrasts this with OpenAI: "When Ilya was there, it might have been possible. But Ilya later, on one hand I don't know for what reason, he seemed to have lost the ability to make decisions. And then he left."
When Yao joined Anthropic, the company had around seven to eight hundred people total. His first impression was of very strong execution and a relatively top-down structure. "It's just that it's actually a relatively top-down company. So after many things are decided they go all in." The atmosphere between employees was excellent—everyone knew each other, and there was no hiding of information.
The company's cofounding team has remained intact, which Yao sees as crucial. "Not a single person has left the company. If you look at their past, they are a group of people who have truly fought battles together. In the past they originated from, they were all former OpenAI employees." Many were co-authors on key papers like Scaling Law and GPT-3. "They are people who have been in the trenches together. I think mutual trust is still very key."
Why Coding Took Off and What Comes Next
The coding scenario has been in rapid development since Claude 3.5 (or 3.6, as some called the updated version) was released. By Yao's conservative estimate, 90% of the code he writes today is model-generated. "If not conservative, maybe 99% or 100%. The remaining 10% is what it can't write or why I didn't let it write."
What it can't write is becoming less and less. About a year and a half ago, only Claude could actually write software engineering code, and even then the model had many flaws. "Sometimes when it wrote code it would only focus on this one file, it wouldn't pay much attention to the relationships between multiple files. And if a class definition was buried many layers deep, the model probably couldn't find it. Now I think this is happening less and less."
The efficiency boost is dramatic. Compared to a year or even a year and a half ago, the efficiency of running experiments and implementing ideas could be 20 or even 50 times faster. "It can be pretty insane. You can open several at the same time and you have several ideas and test them simultaneously. And sometimes even the model can help you monitor some experiments, monitor some results and stuff."
But this hasn't made working hours shorter—it's made them longer. "Because development speed has increased, the more you try, the more you want to try. There are more and more ideas to try." Yao typically arrives at the office around 10 AM and stays until 10 or 11 PM when alone in the US. "In the GenAI field, no one can just coast along."
Looking ahead, Yao believes coding excellence will require more than technical strength. "Good programmers in the future, first, technically speaking they will definitely be very strong. Because if you're technically weak there's no reason why AI can't replace you. But being technically strong might not be the only thing—it won't be a necessary condition, it might be a sufficient condition." What will matter is understanding how your work fits into a large organization, strong planning ability to break down complex things into smaller tasks for AI, and perhaps most importantly, the ability to adapt as AI capabilities continue to evolve.
From Physics to AI: Rejecting Individual Heroism
Yao's path to AI was unconventional. He studied condensed matter theory as an undergraduate at Tsinghua, focusing on non-Hermitian systems and open quantum mechanics. His research on topological phenomena in open quantum systems led to important work that established a new framework for describing such systems. But despite this success, he switched directions for his PhD at Stanford to pursue high-energy physics.
"This is the weakness of human nature. I feel like I always love challenging myself with things I don't know," he explains. Looking back, he views his PhD as a period where he learned a lot personally but "for this world, it didn't produce any contribution." High-energy theory had developed to the point where experiments completely couldn't catch up. Progress depended on mathematical self-consistency rather than experimental validation, and "when this field completely lacks experiments and objective standards, there definitely won't be just one self-consistent framework that appears. At this time who does well, who doesn't do well, actually depends on the subjective judgments of some old-timers in the field."
This realization led him to seek work with "relatively objective evaluation criteria" or "things that can have an impact on this world." Between quantum computing and AI, he chose AI because quantum computing's main bottleneck was experimental implementation, not algorithmic design. "AI is more about having an idea and then you can use some numbers to verify it. This numerical aspect in AI might be training a model or something like that. And this is actually quite similar to doing physics."
What he found in AI was something fundamental: "I think AI as a field is fundamentally simple." Unlike physics, where you can't understand theory at a certain energy scale without experimental data at that scale, "AI isn't bound by this constraint. It doesn't matter if you don't understand it, it can still move forward. And also right now the fact is I can do any experiment I can think of. It's just that possibly I need some time to scale up the compute or get the infrastructure ready, but there's no fundamental difficulty."
This simplicity has profound implications. "I think in this era, the most important trait is being reliable, being detail-oriented, and taking responsibility for what you do. These are the most important traits." When pressed on whether AI requires brains, Yao is blunt: "Doesn't require much brains. Really doesn't require much brains."
This worldview shapes his most controversial stance: "I think the era of individual heroism for language models has probably passed." He dates that era to the Transformer moment. "After that technology was found, for probably a long time from the model side, it's all been more about collectivism—whether this group can work together, whether they can toward a common goal spend their own time together and their own energy. That's the most important thing."
When asked about his personal contributions to models like Claude 3.7 or Gemini 3, Yao consistently deflects. "I think my personal contribution, I personally, my contribution to any model, my statement is always I feel like I'm not that important to that thing. I think more importantly I was very lucky to have the opportunity to join an important project at that time and did some things." He elaborates: "I think everyone now is, everyone is a surfer. Essentially it's a wave, not you the surfer. It's AI, this thing itself is this wave. It will move forward whether you surf this wave or not."
Google's Comeback: From Chaos to Engineering Excellence
Yao left Anthropic in September 2025 for Google DeepMind, drawn not by Gemini 2.5's public success but by the people and the opportunity to learn different things. "I didn't really agree with Dario's anti-China stance," he notes as one factor, accounting for perhaps 40% of his decision. But more importantly, he wanted to work on things Anthropic didn't do—multimodal generation, lower-level engineering infrastructure, and learning from a broader set of humanity.
What he found at Google was an organization emerging from chaos. In the earliest days, according to colleagues' descriptions, pre-training was "very chaotic." But by the time Yao arrived, things had changed. "The organization has more clarity now. Once the organization is clear, especially pre-training has become very, very clear now. That is who is responsible for what, and every point, who is the responsible person at every node. These things are very clear."
This clarity, combined with Google's strong technical background and systematic approach, made pre-training "a very, very controllable thing, a predictable thing. You can know the next generation won't be bad." Google's strength lies in finding an extremely simple product form where everyone looks the same, then competing relentlessly on technology. "That's exactly what Google is good at. Because search engines are exactly like that. Everyone has the same search box, one button, but it just searches faster than you and more accurately than you. There's nothing you can do about it."
The organizational model differs fundamentally from Anthropic's. While Anthropic operates top-down for startups betting on specific directions, Google operates more bottom-up with reserves in every area. "Big companies more say, like this kind of relatively deterministic thing like pre-training is already a relatively deterministic paradigm, then maybe Google will be more like making it into an engineering project. Google's engineering management ability is very strong."
Two events created Gemini's market breakthrough: Nano Banana and Gemini 3, back to back. "If there were only Gemini 3 it probably wouldn't have had such great results. Because when your market share is less than even 10%, whether your model is slightly better or worse, it just spreads too slowly. But what Nano Banana did was first, it went viral in the market, it was a huge hit. Then a ton of people downloaded the Gemini app. And then Gemini 3 was released right after, retaining those users."
Yao sees this as Google capitalizing on a strategic error by OpenAI. "OpenAI saved Google's life," he argues. "Everyone used to worry this chatbot would completely replace search. If this really happened Google would actually be in a tough spot. But fortunately OpenAI did this thing first, then made Google realize this thing is important, but it didn't take this thing all the way, didn't completely kill off search. Maybe just ate some market share. As a result, let Google itself catch up on chatbots too. Now the one in a tough spot is them."
The Technical Reality of Long Context and Post-Training
On Gemini's long-context capabilities, Yao offers a revealing comment: "There are some tricks." When pressed, he admits "there are some tricks that really surprised me" related to pre-training, though he can't disclose details. The principle he articulates is that doing long context well requires both pre-training and post-training approaches.
Looking forward, Yao is working on two main directions: ML coding and long horizon. ML coding aims to achieve the complete AI-training-itself process, where AI can complete an entire research project from start to finish—not only writing and running code but also seeing results, analyzing them, knowing where it went wrong, proposing new hypotheses, designing new code, and running new experiments. "This chain is not yet complete. But I think this chain might be the next thing to gradually become complete."
Long horizon work pursues his mantra: "Train with finite context, use as infinite context." The solution isn't making single training segments longer but rather teaching models to selectively forget and retrieve information, much like humans do. "Human context is actually very very short. If you ask me now what I ate last night, I can't remember at all, because it's not critical to my current scenario. So I choose to forget it." The approaches being explored include both pre-training solutions like sparse attention and post-training solutions involving context management.
On the distinction between pre-training and post-training, Yao offers nuance often missing from public discourse. "From a pure technical perspective, it's hard to say the essential difference between SFT and RL. Pre-training and SFT are essentially not that different—you just take the data you get as your ground truth, treat that as your expert output, then align toward that distribution. Reinforcement learning might be a broader level where the original output is not a given expert but something I produced myself, and among them there are good results and bad results."
For him, the biggest difference lies in data. "For pre-training data what matters more is having a good distribution—the distribution needs to be broad enough, or aligned well enough with the scope you want to cover. But data quality doesn't need to be extremely high. But for post-training it's the opposite. In terms of distribution, it may be much narrower, but for the data it does have, the quality requirements are very high."
Why Wrappers Can't Escape Model Companies
On the question of AI products and wrappers, Yao is characteristically blunt. Both Manus and OpenClaw chose to sell to model companies (Meta and OpenAI respectively, though Meta's Manus acquisition was later revoked). "I think for wrappers to survive in the current environment there are two approaches. One approach is escape fast enough—that is, my growth is fast enough that by the time model companies catch on, I've already captured significant user mindshare."
Cursor is trying to take this path, but even they feel strong crisis pressure. "Its relationship with Anthropic right now has entered a very delicate phase. They used to be close, seamless partners. Anthropic provided the model, Cursor provided the product. Later Anthropic developed Claude Code itself. Claude Code has become very successful. And then Cursor is now trying to build its own model. So they're already in a fairly competitive relationship."
The fundamental issue is that "when it comes to coding, at its core it's essentially a professional need serving professional users. It's a productivity tool. A common scenario with productivity tools is winner takes all." The other survival path is to be small enough that model companies can't be bothered—Midjourney might be an example, though Yao notes the market is "so small that perhaps even though Gemini could make an effort to replicate what Midjourney does, Gemini probably wouldn't want to spend much time on that."
On OpenAI's acquisition of OpenClaw, Yao speculates: "I think for them the biggest benefit was if aside from how much they spent, the biggest benefit was gaining a really strong product team in Asia. Obviously there are many talented AI people in China. Although perhaps currently in terms of technology, purely from a technical standpoint, Chinese AI hasn't really caught up with the US yet. But obviously there are many talented AI people in China, whether in pure technology or in product. In terms of product, I think China essentially has better talent than the US."
The Distillation Debate and Chinese Labs
Yao introduces a useful distinction when discussing Chinese model companies: "hard distillation" versus "soft distillation." Hard distillation is "taking a bunch of tokens generated by Claude and forcibly training on them. If you do something like this, I feel first, it's not very ethical from a business standpoint. And intellectually, it's rather foolish, because the companies doing this essentially demonstrate one thing: they don't even know what they want to do. The only thing they can do is copy others."
Soft distillation involves smarter approaches, such as using other models as assistants in generating your own training data pipeline or as evaluators for answers generated by your own model. "This is actually, I think, commercially a bit of a gray area. But from a technical perspective, it's quite interesting. Because if you think about it, in a sense Chinese labs may have become pioneers in multi-agent training."
He declines to name specific companies on record but suggests the pattern is clear in the industry. What's distinctive about ByteDance, in his view, is that "Doubao is definitely not as smart as Gemini or Claude, but Doubao's voice generation is extremely good. And I find that for life questions I just want to ask Doubao because it's so fast."
Why don't US companies optimize this product feature? "In the US I think people are more focused on how to improve work efficiency. The fact is if you ask about these daily topics, actually you'll find that Gemini from generation to generation does better and better. But for now these American companies are all still in the stage of working hard to push the upper limits of intelligence forward. And ByteDance, of course it's also pushing the upper limits, but I think it might just be doing very well in user optimization too."
What Makes a Good AI Researcher
Yao has developed his own hiring methodology that reflects his philosophy about AI research. He gives candidates 24 hours to complete a reinforcement learning project from scratch using whatever resources they want. "This thing isn't that hard in the AI era. Without AI this would be impossible. No one could do it in 24 hours. But with AI, it's actually quite easy because AI can do the whole thing for you."
The test evaluates two things. "In this era, evaluating someone, like whether they write good code is actually useless, because most people don't need to write code themselves anymore. What's more important is whether they can effectively leverage AI." But there's a trap: "If you let AI do everything but you don't really try to understand what AI did for you, you'll be exposed during that one-hour discussion."
There's also what Yao calls "some rather dark cleverness" in the design. "Why it was designed as 24 hours is to see how much this person values this opportunity. Can they stay up all night? If they're willing to pull an all-nighter they can survive these 24 hours. If they can't make it then it just means they probably don't value this opportunity that much."
For Yao, the most important traits in AI research are reliability, attention to detail, and taking responsibility for what you do. "I think in this era, the most important trait is being reliable, being detail-oriented, and taking responsibility for what you do. These are the most important traits." He elaborates that modern AI training is a large system requiring holistic understanding. "You actually need to understand all aspects of this system to have a holistic understanding of what makes something useful because of what, rather than saying the thing itself is useful."
This requires going against human nature. "Every individual's human nature might be to make their own things show up better. But for a company or an organization, the most beneficial thing is to make the entire company's system very solid systematically. This is actually the best for you personally, because once this system is solid you can leverage this system to produce more output. But the bad thing is this system will make your individual heroism not shine. But you can rest assured that others' individual heroism also won't shine."
The Future Landscape
Looking at the current state of the field, Yao sees models as increasingly commoditized at the capability level. "From the actual user experience, you can feel the differences between these three companies' models. But the hard part is in the past, you could see this difference on paper too. Now, on paper, everyone is actually pretty close. And when you look at the numbers on paper, like looking at SWE-bench, you'll find the best is only maybe one percentage point or two percentage points better than the not-so-good ones, but actually everyone is around 80%."
The competition has shifted from raw capabilities to product execution and market positioning. "Now it feels like just relying on hitting these publicly recognized model capabilities actually doesn't have much meaning anymore. Releasing fast also shows actually this problem has become easy for everyone. Everyone knows the know-how now. There are no secrets anymore."
On OpenAI's position, Yao offers a measured assessment. "I think the trends in China and the US have already diverged. What China is really strong at is the consumer side. It can come up with really, really complex product features or structures. But that doesn't work for productivity software. Productivity software is very straightforward." He believes OpenAI's advantage lies in its product execution, though even that is being challenged as Google catches up.
For the future, Yao sees no position as secure. "I don't think anyone's position is secure right now. The form of AI still has a long way to go. We're not at any endgame yet." He's puzzled by the sentiment in China that companies are fighting over a super app like it's zero-sum. "Is this form the super app form? What if someone else comes out with a completely different form one day, and your functionality becomes a subset of that thing? That's quite possible, right?"
The next paradigm-level changes might come from areas still in early exploration. "ML coding and long horizon" are valuable but not quite paradigm-shifting. "Paradigm-level might still be those more uncertain things, like multimodal generation. I think there might be a hero or a group of heroes." But he's dismissive of vague concepts like world models, where "ten thousand people have ten thousand world models" and definitions remain unclear.
For young people considering AI research, his advice is stark. "I think purely working on language models is no longer a blue ocean. I think it's too late—the last train has already left." He considers himself fortunate to have caught that last train. "There are many other things. Like the multimodal generation we just mentioned, there may still be many opportunities there. Robotics probably has even more opportunities. And even more extreme, there's like whether you can use AI to help with real scientific problems."
A Philosophy of Work
Throughout the conversation, Yao returns to themes that seem at odds with the cult of genius surrounding AI. He has no mentors in the industry, no old friends he needs to protect, allowing him to "criticize whoever I want." He's openly dismissive of researchers who spend time on Slack "talking about grand principles" rather than doing actual work. He believes "ideas are cheap" and that implementation is everything.
His personal trajectory reflects this philosophy. From choosing Gezhi High School's competition class over Shanghai's four top schools ("I felt this competition class, in today's terms, it's an underdog. I felt like the barefoot aren't afraid of those wearing shoes. Worth a shot"), to texting Tsinghua's admissions office to demand equal treatment for Shanghai students, to switching from successful condensed matter research to high-energy physics because he "always love challenging things I'm not good at," Yao has consistently chosen the harder, less obvious path.
But he's also learned from disappointment. His PhD in high-energy physics taught him the value of objective evaluation criteria. "I think the biggest difference between empirical laws and scientific laws is quite blurry," he reflects, but in AI "you can run experiments. It doesn't matter if you don't understand it. It can still move forward."
As for his own future, Yao is characteristically uncertain. "I think probably not" when asked if he'll stay at Google long-term. "I will still try to challenge myself. Need to torture myself. But I just might need to find something worth torturing myself for." What's clear is that individual achievement matters less to him than being part of systems that work. "I think in this era, the most important trait is being reliable, being detail-oriented, and taking responsibility for what you do. These are the most important traits."
It's a philosophy that fits the moment. As AI research matures from a field driven by individual brilliance into one requiring massive coordination, systematic rigor, and collective execution, researchers like Yao represent a new archetype. Not heroes, but highly capable operators who understand that in the age of AI, the wave matters more than the surfer, the system matters more than the star, and reliability matters more than genius. Whether this marks progress or loss depends on your perspective, but Yao is clear about his: the era of individual heroism is over, and good riddance.