Google Transcript: Jeff Dean on Inference Hardware Specialization and Self-Improving AI Systems
Y Combinator Startup School 2026, September 20, 2026
Are AI Models Already Junior Engineers?
Diana Hu: All right. Should we get started, Jeff?
Jeff Dean: Sure. Sounds great.
Diana Hu: All right. Jeff, welcome. And again, thank you so much for being here. Especially, I just got a cold, and thank you for being here.
Jeff Dean: Yeah, I am afraid I have lost my voice. I do not normally sound quite like this, but we will do what we can.
Diana Hu: So, you built MapReduce, Bigtable, TensorFlow, the TPU, Gemini. We could spend a whole hour on all the things you have done, but what I love is that you are still making bold predictions in public. Last year in May 2025 at AI Ascent, you said that AI is at the level of a junior engineer. That was about a year ago. How close are we to that prediction?
Jeff Dean: Yeah, I mean, I feel like the models have been getting a lot better at sort of agent-based, longer-running coding tasks. It seems pretty clear that they are now actually pretty capable. Depending on exactly your definition of junior engineer, it seems pretty spot-on, I would say.
Diana Hu: What did you underestimate from that prediction?
Jeff Dean: I think the ability to do more and more complex tasks has been growing faster than I thought. I also think outside of coding, these agent-based systems are really starting to shine in other domains. I think that is going to be an important trend in the future.
Diana Hu: Give us another bold prediction. What do you think is going to be the 2027 edition?
AI Systems That Improve Themselves
Jeff Dean: I think you will see a lot more automation of ML systems themselves. Basically getting ML systems to improve their capabilities by running lots of experiments, breaking things down into subproblems, running those subproblems in a tight automatic experimentation loop, putting the results together, and being able to then get some improved system out from that sort of fully automated problem decomposition and automated experimentation. I think that is going to be really exciting. I think that also applies not just to ML, but also to other fields of science and engineering. Basically anything where you can have a measurable objective, I think you can actually make a lot of progress these days.
Diana Hu: Now let us go back a little bit in history. Way back in 2001, Google Search used to run on hard drives.
Jeff Dean: Yep.
Diana Hu: And you and Sanjay did the math and realized that at some point, the whole search index would finally fit in all of the RAM of all the computers you had running. You made that radical realization, and you basically, in a few days with Sanjay, shipped in production a whole new search version that worked in RAM rather than hard drives. That was the thing that got Google searches to be so fast. History tends to remix. What is the "it fits in memory" moment right now in 2026 that everyone in this room should still be thinking about and designing?
The Google Search Breakthrough That Changed Everything
Jeff Dean: Yeah, I mean, it is a little different, but I think you are going to see more and more high-performance and low-energy inference hardware systems. I think everyone is now realizing that inference is the key to making these agent-based systems be available to more and more people, that latency is really important, and that specialization of the hardware is a really key way you can make things that are more energy-efficient and lower latency than more general-purpose computational devices like, say, GPUs or TPUs. Everyone here is used to waiting for responses on models. Waiting is no fun.
Diana Hu: Master speed. So you are saying, what if we do not have to wait anymore?
Jeff Dean: Yeah. Imagine what you could do with something where the latency is 50x better.
AI Agents Will Run for Weeks
Diana Hu: Interesting thought. Now, what is one assumption that perhaps 6,000 people in this room hold that is already false about AI?
Jeff Dean: That is a good question. I think probably one thing is people do not quite realize how possible it is to have agent-based systems that can run not just for an hour or two hours on a problem you care about, but for some problem domains and with highly capable models underlying them, you can get them to run for days or weeks and do really, really complicated tasks. Some people are starting to see inklings of this, but I do not think everyone has really internalized this. That is going to be a pretty big deal.
Diana Hu: What is a particular task that you have run that has run for weeks? What was it? What did you tell the agents to solve?
Jeff Dean: You can tell agents to go off and implement completely new versions of software in different programming languages that might have better safety properties or better performance properties, and then they can go off and actually do that in a pretty serious way.
The Napkin Math That Led to TPUs
Diana Hu: That is pretty cool. Now, one thing that you have been very well known for is you are really good at napkin math. Sounds funny. One of the stories about you is that back in 2013, when speech recognition started to work at Google, you did the napkin math where if every Google user used their phone and talked to it and used the speech recognition system for just 3 minutes a day, you found that you would have to double the Google server fleet, which would be really, really expensive just to do speech translation.
Jeff Dean: Yeah.
Diana Hu: And instead, you basically built a custom chip, and that was the origin story of the TPU.
Jeff Dean: Yeah. We were starting to see really good quality results on the deep learning-based speech models we were training, but they were computationally expensive compared to the old speech system. But they halved the error rate. That was like the equivalent of 20 years of advances in speech recognition in just a few months of fiddling with the model, scaling it up a bit, and getting better data. So we started to get worried that if speech worked a lot better, people would use it more. That back-of-the-envelope calculation was really about that. What if people start to use speech recognition more to dictate emails or to talk to their phone or whatever? It turned out that we realized we needed some better solution than running on CPUs at the time. So we came up with TPUs, which are sort of very specialized for essentially low-precision dense linear algebra, which is at the heart of nearly all of the modern machine learning algorithms we use today. If you build a specialized chip for low-precision dense linear algebra and cannot do anything else, that turns out to be really useful for machine learning inference, even though it cannot run Chrome or Word or whatever. That produced a chip a couple years later that was 30 to 80 times more energy-efficient than CPUs and GPUs of the day, and also much, much lower latency, like 20 to 30x lower latency.
Diana Hu: Which is incredible, what the foundation that TPU has become today. No way you would have predicted that TPU would be so foundational now with transformer architecture, which was invented way later after you actually invented the TPU.
Jeff Dean: Yeah, that is sort of why we built a general-purpose linear algebra system, which is what a TPU is really. We knew ML algorithms were still evolving and you did not want to overspecialize, but you wanted to specialize enough that you got the dramatic performance benefits of having very big multiplier units. We could have high-speed memory. We could have high-speed interconnect for later TPUs that brought many, many chips to bear on the same problem efficiently. We have continued to scale those up and improve their performance over many, many generations now.
How to Find Breakthrough Ideas
Diana Hu: Incredible napkin math. Napkins are good. So actually, what is good napkin math that everyone here who wants to be a future founder should run tonight to potentially build something as consequential as the TPU?
Jeff Dean: It is always hard to say. Think about what problems you see in whatever it is you are thinking about, what bottlenecks you see, and are there very different ways of thinking of the solutions to some of those problems that would get you an order of magnitude or two orders of magnitude better performance or capability or whatever it is. Because sometimes if you just squint at a problem and you think about not necessarily being anchored on exactly how that problem is solved today, but how you would solve it from first principles, you can come up with really good ideas that are maybe not what other people are thinking about.
The AI Engineer’s New Mental Model
Diana Hu: That is a good tip. For everyone here who does not know, years ago, Jeff wrote a very famous list called "Latency Numbers Every Computer Scientist Should Know." These are numbers around, for example, how long a cache miss takes, a disk seek, a network packet traveling, let us say, from California to the Netherlands, lots of numbers like this about distributed systems and systems engineering. It has been taped and become the bible for a lot of distributed systems engineers. Now fast forward, that list is up for an update. Give us the AI edition for now in 2026.
Jeff Dean: If you looked at what is important in AI systems these days, you would want to know things like the bandwidth between your main memory system on your accelerator to the on-chip memory to the multiplier unit. You want to know how much energy does it take to do a single multiplier operation. What is the interconnect bandwidth between chips, and how many chips can you connect with that bandwidth? Then if you go beyond that domain, what is the fall-off in network bandwidth when you need to talk to 10,000 chips instead of 500? I think these are all really important numbers to learn, and they really affect how you think about solving particular kinds of problems.
Diana Hu: One interesting thing that I have heard you talk about is that nowadays, the unit that you measure everything in is energy.
Jeff Dean: Yeah.
Diana Hu: You pointed out that doing a calculation or math costs about 1 picojoule, but moving the data and doing data I/O costs a thousand times that.
Jeff Dean: Yeah. Just bringing it in from HBM on an accelerator into the processor so it can actually compute on it.
Why AI Is Really an Energy Problem
Diana Hu: Yep. That gap kind of quietly decides what products are possible and how these algorithms in AI are built. What are the kinds of problems that founders keep calling model problems, but are in fact actually energy or data I/O problems?
Jeff Dean: The example you raised of a 1,000x difference in moving data versus actually computing on it in terms of energy is a pretty significant one, and it shapes a lot of aspects of what we do in machine learning. Because if you did not have that 1,000x difference, then you would not have to do batching. But you have to do batching of many examples or many tokens at once in order to amortize that data movement, so that you can not pay a 1,000x slowdown, but pay a 1,000x divided by batch size energy cost. For really low latency, batching is not really very good. These kinds of things and the energy behind various decisions in the computer hardware we use really affect a lot of decisions we make in building higher-level systems.
Diana Hu: A very concrete example is just how training models is done. There is this whole concept of batching the datasets and running epochs. People perhaps may confuse that as a model problem, but it is really a systems data I/O problem, right?
Jeff Dean: Yeah. You have to assemble batches to get better efficiency in your hardware. Ideally, you might do batch size 1 training, but it is not as good in terms of efficiency. So people use pretty large batches these days.
Diana Hu: You are well known for taking off on a long week or weekend and coming up with a brilliant solution. Is there such a thing as Jeff going and working on it for a couple weeks and getting batch size equals 1 training done?
Jeff Dean: I have been thinking more about inference, actually. Inference is a pretty interesting problem because you do want very low latency. Training, you do not necessarily need incredibly low latency. I think there is a lot of room for specializing hardware more for inference than we are today.
Diana Hu: What are some of those interesting things on inference that you are really thinking a lot about?
Jeff Dean: Just trying to minimize data movement, trying to think about incredibly low-precision operations, and maybe not supporting lots and lots of different kinds of precisions. If you feel like you have a good answer for what kinds of precision you need, maybe just build that into the hardware and not much else.
Diana Hu: Which brings down to a core analogy I heard from famous computer scientists, that really the whole process of AI is a big compression problem. In order to have the data be lossy and compress it and then restore it, you basically need to understand it.
Jeff Dean: If you truly understand the data, you should be able to compress it really well.
Diana Hu: And now transformer architecture is basically one of the ways that has turned out to work really well.
Jeff Dean: Yeah, working pretty well so far. Good work by my colleagues.
Context Engineering Is the Next Frontier
Diana Hu: Yes. Now let us zoom out a bit. AI progress used to mean just better models. You had more data, trained models with bigger parameters. But increasingly in the last year or so, it is everything around the model. Not just the model size and number of parameters or more data, it is everything around things like retrieval, tools, memory, agent tools, and it might get consolidated into what people call context engineering, right?
Jeff Dean: Yeah. The model is really only one piece of what you are trying to do, which is build an overall system that can solve really interesting problems. That involves a model that knows how to use various tools. It maybe knows how to retrieve relevant information, maybe has a history of other information that it has retrieved for past problems, and it can put information into the context of the model. The nice thing about that is that information is really clear to the model, unlike the training data the model was trained on, where it is all kind of like trillions of tokens stirred together into a soup of hundreds of billions or trillions of parameters, but it is all less clear than the actual context that the model sees directly for this particular problem or use case.
Jeff Dean: Then being able to understand what tools are available, which ones are going to help the model solve this next phase of the problem, how to decompose a problem into a sequence of tool calls, maybe trying multiple approaches to solve the problem and seeing which ones work and being able to evaluate that. This is the whole orchestration of complex agent and multi-agent systems that I think is going to be more and more important. Super exciting times, I would say.
Diana Hu: The fun thing about this particular problem domain set is actually something that everyone in this room can do. Before, to train a model, you needed an incredible amount of resources, an incredible amount of access to GPUs and data. But for context engineering, everyone here can do it. You just need the API to something like Gemini and then work on your own setup for your own retrieval, your own tool calls, and so forth. What are some tips for everyone here? How does everyone get better at and become exceptional at context engineering?
Jeff Dean: A really good way to do it is to use these models and harnesses and tools to try to solve problems, and then sometimes you can actually see where the models are failing. Often, you can actually make the model work better and succeed at that kind of problem by not just adjusting the model parameters, which is hard to do from the outside, but by creating better guidelines for the model, writing skills for the model to know how to use different tools that would be incredibly useful for solving this particular class of problem. As you do that, you end up on this kind of self-improving setup that you are trying to use to solve things. That is a really good way to get better at understanding what additional information the model would want in order to become more capable.
The Skill That Made AI Better at Optimization
Diana Hu: Can you give an example of some context engineering you personally have done, skills you wrote or tools that really made a huge difference in your workflow?
Jeff Dean: Sanjay and I were working a few weeks ago, and we often do some amount of performance improvement for very low-level libraries. We have a microbenchmark library we have written at Google where you can write microbenchmarks of how long different kinds of operations take or how long does it take to populate this data structure. Sometimes those data structures are used on millions of processes across Google, so it is actually pretty important to make sure they are high performance. You can write microbenchmarks, but without an agent-based system, what you usually do is measure what the current performance is on some benchmarks you care about, make some modifications to improve the performance, then rerun the benchmarks to see where things improved, run a broader set of benchmarks, and measure the cache footprint of things.
Jeff Dean: We wrote a skill that basically taught the model how to do most of those things in various sequences so that it could actually do self-improving benchmark measurement, code changes, measure the performance improvement, and then iterate on that. That seemed to work pretty well for some kinds of problems. It really just is us giving the approach we would use as people to the model in a form that it could use.
Diana Hu: Wow, that seems very impressive. So you are saying you have this skill that if someone got access to it, it could perform optimizations like Jeff Dean. Seems like the world would love this and it is worth an infinite amount of money to someone to have access to this.
Jeff Dean: We actually published a document maybe a few months ago called "Performance Hints" that Sanjay and I wrote. It is like a 30-page document about various kinds of performance tricks. Some people have taken that and given it in summarized form to various models and seen that the model can now get better at reasoning about performance issues in code.
Diana Hu: You heard it all here. You could actually optimize your own code like Jeff Dean if you take this paper that you published, "Performance Hints."
Jeff Dean: Yep. It is freely available, so you should all try it.
Why Long-Running Agents Fail
Diana Hu: Very cool. Now, you are talking about agents. Everyone here is probably building one or built one at some point. I am sure everyone has seen your agent go off the rails at perhaps step 30 or 40. Agents are great for up to step 10 or something, and then get shaky at step 50. What do you think is the constraint today? Is it context evaluators, or just errors that compound because it is basically an open-loop system?
Jeff Dean: Obviously, we want agents to be able to run for very long periods of time because that is how they are going to solve more and more complicated problems. But as you observe today, they sometimes stop working after 10 interactions with the tools. Sometimes that is because the model is trying to do something it does not have a lot of experience doing. It has been trained on a whole set of things, and as soon as you get a little bit off the distribution of things it knows how to do, like most machine learning models, its performance will start to degrade. The farther you get off the comfort zone of what it knows how to do, the more likely it is to not work as well.
Jeff Dean: There is a bunch of things you can do. One is to give the model skills and hints that tend to keep it on the more brightly lit path of things it does know how to do. Having multi-agent systems where you have multiple agents trying different approaches, and you have maybe another model or another agent evaluating which ones of those seem promising, is another way to search the space of possible solutions, stick to the ones that seem most promising, and discard the ones that did not seem to work or went off the rails. That is a very useful general technique: inference-time compute to perform search over plausible ways of solving the problem, which can get much higher performance or much more reliability in long-running agent flows.
Diana Hu: What are some ways you implemented this particular workflow for your agents internally?
Jeff Dean: We have harnesses, and then we have a whole set of skills, particularly in the internal Google development environment. We have skills so that the agents know how to use lots of our internal tooling for coding, code reviews, measuring performance, or fetching log files. Those are just skills that you can add to make the base model more capable, even though it has not necessarily been trained on exactly the way that Google internal engineers would fetch log files from our proprietary system. With the right kind of skill definition, you can actually get it to work, and that improves the usefulness of the agents.
Where Startups Can Still Beat Google
Diana Hu: Now let us talk about where startups can win. This section is one that I personally care a lot about because everyone in this room needs to decide what to build in the future if you are a future founder. The thing about Google is you codesign everything on the system, from the processors to the products. Which are the layers that someone like Google will keep building and compounding and being better at, and where can a two- or three-person team still win?
Jeff Dean: Obviously, Google and our Gemini models and our hardware infrastructure are really trying to build very general models that can do almost anything. But in a lot of cases, that means that we do not have a lot of attention on particular domains where perhaps a really well-designed surface, and maybe a model and set of skills, or a specialized model that is not in the general mix of things that our models do well, can actually have a significant advantage. You can build something delightful, really high accuracy, and really high quality for a domain that you are really passionate about. That is where the two or three people in a room building that can have an advantage.
Jeff Dean: But I would also caution that the general models are definitely getting better at a broader and broader range of things. So you have to figure out: is that thing you are working on going to be durable, or do you think the models at the forefront are going to get better at that in the next 6 months or 12 months, or is it something they are not going to be able to do for 2 or 3 years? You want to weigh that as you are deciding what to work on.
Diana Hu: The general models, of course, you are going to keep working on and keep making better. How should the audience reason about what areas to pick and work on?
Jeff Dean: The most important thing is to pick something you are super excited about and want to build and think would be useful in the world. If you do that, you are already way ahead compared to if you wake up and think, "I do not really want to do this," or if you build something that is actually not that useful to the world or to many people. That is the number one selection criterion I try to apply for what problem I should work on next.
Jeff Dean: Second, I think you want to look at what the current, more general models can do in that problem domain. You can test them: are they able to do this thing very well? If they are completely failing, that is probably a good sign. If they are kind of able to do some of it, but not very well, that is maybe not a great sign. That is probably a sign that the capability is starting to be present in those models, and with more training data, larger-scale models, or whatever, it is likely to get better. Look for something where the model succeeds 0% or 1% of the time, not 20%.
Diana Hu: How do you find those? Are those things effectively out of distribution from the training set, and what exactly is the problem shape that fits that?
Jeff Dean: Sometimes it is a product that you build that might have access to a particular kind of data that a general model might not have. It might be that you are building something to help users organize all their own personal information, and the general model will not necessarily have access to that. There, you can have a big advantage because all of a sudden your product has visibility into important data.
Jeff Dean: It could be some incredibly hard problem where if you get the right training data and you can train a more specific model than a general-purpose one, you can actually do that in a very affordable way. Maybe it does not take that much compute to train a niche model for this particular problem, but you can get something that is highly accurate. That can sometimes be a really good building block for solving an important problem that is maybe not handled very well by the general model.
Diana Hu: There are basically two paths. The first path is a little bit funny: you guys are organizing the world's information, that is well covered, but organizing personal information is open. The second path: you talked about more specialized models in certain domains. Can you tell us more about what some of these domains are?
Jeff Dean: If you look at my colleagues' work on AlphaFold, that was a very specific model for protein folding. It was highly successful and was able to really handle that domain quite well, so that all of a sudden you now have this amazing tool and model that can give you answers to questions about proteins and their structure really effectively. But it is not a general model; it is a very specific one. There are other domains where that kind of approach can work really well, maybe in materials science or chip design, that will enable you to leverage the capabilities of a very accurate but niche model to do things that are hard today.
How to Become an AI-Native Founder
Diana Hu: If some of you find a problem of similar shape like AlphaFold, it could be a good problem to work on. Now let us assume you found a problem to work on. We are going to talk a bit about how you become an AI-native founder. In the past, you said that managing a fleet of 50 or 100 agents is all about writing really good, crisp design docs or specs. How do people get good at that? What do those look like?
Jeff Dean: You will have a lot more success when working with your virtual agents if you can clearly specify what it is you want. The clearer you are on what it is you want, the more the agent will have guidelines and an outline of what it is trying to accomplish. Whereas if you do not specify very much stuff, the agent has to infer what it is you meant. In many cases, it might infer things that are different than what you imagined. We have always told computer scientists from the very beginning that it is really important to specify what the software you are writing is trying to accomplish before going and writing it. Now we actually have agent-based systems that can do the writing, but the importance of specifying what it is you want has actually gone up. Before, you would be handing it off to a very intelligent human who maybe has context or can ask you follow-up questions. Agents can sometimes do that, but clear specifications are a really good idea.
Jeff Dean: To give you an example of a use of a coding agent that works extremely well: you can ask today's models to translate software from one computer language to another very effectively. In that case, you actually have an incredibly detailed specification: you have the whole software that says what the system is supposed to do. If you have a Python implementation of something and you want a Go implementation of it, that is something that the models seem incredibly capable at doing these days. It can take all the tests that are in Python, translate the tests to Go, make sure they pass in the Go version, compare behavioral differences between the implementations until there are not any, and be highly effective because that spec is so clear.
Diana Hu: Now let us assume every founder gets good at running hundreds of agents at the same time and all the code is written for them by the agents. What becomes the scarce skill?
Jeff Dean: Having incredibly good taste in what you ask your agents to work on. That is the crux of a research problem from my background. A researcher can have all the tools and all the techniques, but often most of the battle is what problem are you going to spend your time on. If you pick the problem well and you succeed in solving it, that is way better than if you delightfully execute a research investigation into a rather boring problem. That high-level wisdom of what to work on is incredibly important, and I think models are not necessarily going to be that good at it. You are going to have people steering a lot of AI-assisted computation in order to accomplish great things and more quickly. That essence of what it is you want your models to do is the key thing you should focus on.
Diana Hu: Let us talk a bit more about taste, because it gets talked about a lot right now in this current era with agent coding. How do you build taste? How do you make it concrete?
Jeff Dean: It is a difficult thing. It is not like there is a measurable objective of taste in a lot of cases. Some of it is from experience. Working on a lot of different problems in the past kind of teaches you about what kinds of problems might be interesting in the future, or what kinds of things might be just barely possible by cobbling together previous approaches, and then what open problems you might have to work on in order to get to something magical or highly useful.
Jeff Dean: Another way you can get more experience for yourself is to write down a bunch of things you think might be important in the next 12 months. Maybe you pick one of them to work on, but go back and evaluate in 12 months which of those other things actually seemed important, which ones other people in the world went out and created, and which ones they did not seem to do yet. That can give you a lot more samples for your own taste creation capability. That is an important skill to have.
Question Your Biggest Assumptions
Diana Hu: A third way we were talking about earlier was doing very crazy thought experiments.
Jeff Dean: That is another good way. Sometimes it is good to not take as a given things that most people seem to take as a given. I was doing a crazy thought experiment with some colleagues the other day. For 60 years, the whole silicon chip design and fabrication industry has done tremendous work to make smaller- and smaller-scale transistors that are very low error rate. The assumption that we want is that every chip we manufacture of the same design should be identical to every other chip. You do not want any bits to flip; no bits should flip. There are all kinds of error margins built in, and memories have ECC memory these days.
Jeff Dean: At the macro scale, we do not make that assumption when we are building large-scale distributed systems. We build reliable, large-scale distributed file systems out of unreliable parts. Individual disks can fail, but your data should be safe. We have mechanisms at a higher level to enable us to have three copies of the data on three different machines and three different racks, so that if any rack switch or individual machine or disk fails, you still have your data. We have Reed-Solomon encoding techniques. But we do not seem to do this at a really extreme level in the transistor-level scale of the technology we are working on.
Jeff Dean: An interesting thought experiment is: what would happen if you tried to build a system out of transistors that might have 20 errors per day rather than one every million years? That would be a very different design point and might enable you to do really interesting things on the fabrication side. You would have very different design methodologies, because if you want to get a signal from here to there and you have these super unreliable transistors, you might have very different ways of signaling. You might send it along multiple redundant paths in order to make sure that it gets along one of them. I think that would be a pretty interesting set of thought experiments. I am not saying we should go do this, but that is the kind of thing where you do want to occasionally question assumptions. Oftentimes these thought experiments do not work out because there are very good reasons that for the last 50 years we have done this thing this way and not that way, but it is good to revisit those every so often.
Diana Hu: That is so wild. It is starting to rhyme a lot with neuromorphic computing, or the human brain and how nature works.
Jeff Dean: Signals in our brain are not especially reliable at getting from one place to another. In brains, when there are really important things you need to get from one place to another, there are multiple pathways that enable you to do that.
Diana Hu: What is one of these crazy assumptions that you threw out the window that actually built a consequential system in the past?
Jeff Dean: TPUs is a good example: being able to specialize hardware for a very niche problem domain before that problem domain seemed as important as it is today. The origin of MapReduce is another good example. Sanjay and myself and a number of other colleagues had worked on various iterations of the crawling and indexing system at Google. We had written lots of hand-parallelized code with lots of checkpointing to make sure it would be robust and reliable if it was running on 100 computers or 1,000 computers and some of those died. But that code tended to be intermixed with the relatively simple thing you often were trying to do, like looking at all the contents of all the web pages and computing on the side a mapping from URL to what language the page is in. It would get obscured by all this other code for parallelization and reliability.
Jeff Dean: We remembered our training in functional languages and realized we could squint at those problems and develop this MapReduce abstraction above the implementation. Below the implementation, you could put all the checkpointing and reliability mechanisms into that lower-level library that everything could then build on. That became a hugely successful way of dealing with very large-scale computations at Google in a robust and reliable way, from that thought experiment of: if we squint at it, could we find lots of problems that fit into this abstraction?
AI That Builds Better AI
Diana Hu: You talked a bit about your interest right now working on customized hardware. Right now, AlphaChip lays out chips. You also have AlphaEvolve that proposes solutions, evaluates them, and keeps all the ones that work. It seems like you are starting to build systems that can compound and build AI that builds AI.
Jeff Dean: More generally, there is this foundation of the scientific method: you propose an experiment, you implement what you need to run the experiment, you evaluate the experiment, and then you get results from that. There are more and more problems that are now possible to implement where that whole loop of running not just a few experiments, but running many, many experiments—because you are able to automate that loop and make the latency of that loop extremely low—is going to be really important. It is going to enable us to tackle lots of different problem domains in science and engineering, in machine learning model design itself, and in engineering tasks like designing chips.
Jeff Dean: If you can actually do those things in an automated way and have an orchestration framework that can take very high-level objectives and break them down into subproblems, each of those subproblems can run one of these automated loops exploring the best way to solve that subproblem. Then the orchestration framework can put together subproblem solutions into the overall solution for the higher-level problem. That is going to accelerate machine learning progress, accelerate science, and accelerate engineering. That is going to be amazing.
Diana Hu: A lot of fields where you can have very good evaluators, adjacent to things that can be formally verified, are ripe for AI systems that can self-improve.
Jeff Dean: In a lot of cases, your evaluators need to be made much faster. As an example, my colleagues did some work maybe a decade ago on quantum chemistry, where you are trying to understand the properties of a particular molecule. You can generate some molecule configuration and you want to understand what properties it has. You can run a very computationally intensive density functional theory simulator, which is something that might take a night of computation to tell you the answer for one thing. My colleagues took a bunch of output from those simulation runs—the input molecule configurations and the outputs of the expensive simulator—and used it to train a neural approximation to the simulator. Instead of it taking a night, they made something that was 300,000 times faster and nearly as accurate as running the full-scale simulator.
Jeff Dean: That completely changes how you do science. If you have 10 million things to screen, you could do that while you go to lunch, rather than it being a 6-month endeavor where you try to scrape together enough compute to run all these simulations. There is a lot of room in many domains for much faster validation models, possibly learned validation models that can get you an approximation to the true answer much more rapidly. That changes how those experimental loops can be thought of and how quickly you can go around them.
Diana Hu: What are some of the spaces and problems that you are super excited that this sped-up scientific method is going to solve or achieve?
Jeff Dean: Machine learning itself is one. Can we have a model that is able to recursively self-improve itself by running lots of experiments? If you think about how models are improved today in large research teams, people think of some ideas, run a bunch of small-scale experiments, see if those worked out well, take the most promising ones to try at larger scale, evaluate that, and integrate the results into a new recipe for your model.
Jeff Dean: There is no real impediment to making that a much more automated loop where the model itself decides it is going to explore, maybe with a nudge from people at the highest level like, "Why don't you try some new ideas around model architectures that incorporate this?" It will go run lots of experiments, see which ones work, and incorporate them at a much more rapid rate. Effectively, you want to optimize your discoveries per unit of compute input.
Diana Hu: As everyone in the room becomes founders or starts their careers, they will probably collect lots of rejections. It has happened to you too, Jeff. In 2014, you, Geoffrey Hinton, and Oriol Vinyals wrote a paper on distillation, which is taking a big teacher model to train a much smaller and more efficient model that is cheaper to compute with fewer parameters. It has become a trick that everyone is using right now in industry, and that paper got rejected at NeurIPS.
Jeff Dean: I do not fault the program committee. A lot of times a paper gets three reviews, and one of the reviewers looked at it and said it was unlikely to have significant impact. When we wrote the paper, we saw this was a super important problem because we knew making cheaper, highly capable models from larger-scale models was something we desperately wanted to do to serve models to more people in domains like speech or vision. But sometimes the reviewer does not have that experience because maybe they are not thinking about large-scale AI services and are thinking about whether this is a fundamental advance. It gets rejected every so often, and that is fine. We put it on arXiv, people read it, people use it, and it is all good. We do use it in making our Flash models from our larger-scale Pro model. That is partly why our Flash models in Gemini are so capable relative to their size and speed, some of the best in the benchmark for their model size class.
Diana Hu: Even if you get rejected, keep going.
Jeff Dean: That is the lesson I would distill from that.
Build Something That Truly Matters
Diana Hu: You joined Google as a 20-person startup back in 1999. If you were to take the 25-year-old Jeff Dean from back then and teleport him to today in this era with your skills, what would you do? Do you join a frontier lab, or start a company?
Jeff Dean: It is always hard to say, and it is a very personal choice of what you want to spend your time on. To me, some of the most important questions are: are you going to work on something you really care about, and if you are able to make progress on it with colleagues you like working with, will that make a difference in the world in some positive way? Will you be able to offer that service to help biochemists, or programmers, or all consumers on the internet? What you should strive to do is have impact in the world that is positive, work with people you enjoy working with, work hard, and do your best.
Jeff Dean: In terms of joining a frontier lab versus starting a company with two or three of your close friends, those are different experiences. In a large, established organization, you have structure, lots of amazing colleagues who know things you do not, lots of interesting problems to work on, and an existing platform for impact where your work influences lots of people in the world. As a very small startup, you have to have something you are passionate about, and there is a lot of risk in taking on that problem in a way that allows you to succeed and grow an endeavor. But that can also be incredibly rewarding. At the very least, regardless of what path you take, ask yourself: if I work on this problem and the best possible outcome happens, will the world be a lot better in some way, or will the world go, "Eh, that is kind of cool, but whatever"? That is not the kind of thing you should spend your time on.
Diana Hu: You have been able to be an incredible mentor and manager to many engineers and build huge systems. What are some lessons for everyone here on how to work with smart people or find smart people?
Jeff Dean: You always want to find people who have really good skills in some area that is needed in a team you are trying to form, whether inside a company or starting a company. But you also want to find people that you delight being around, because you are going to spend a lot of time around people working on really hard problems. You want people who are low ego, team players, and have complementary skills to your own.
Jeff Dean: I always find working in a small team where people know things that I do not know, and where maybe I have some skills that other people do not have as much of, is super fun. You are collectively building something that none of you could do individually, but in the process you actually gain a lot of new knowledge and skills for yourself, and so do they. View your engineering or research career as having an amazing tool belt of techniques. You always want to be adding new tools to that tool belt, because you never know when you might come across a problem where you need these four specialized tools rather than these three. Adding more tools makes it more likely that the problems you encounter in the future will be solvable by you.
Diana Hu: Multiple people in this room will eventually build something as consequential as you have done with MapReduce, TPU, distillation, and so forth. What problem do you hope they will be working on?
Jeff Dean: The world is a very big place and full of problems. I am particularly excited about new approaches to hardware, like much more efficient inference hardware. I think there are radically different kinds of algorithms for machine learning that might be much more data-efficient than the approaches we are using today. If you think about our large-scale models today, they probably see 1,000 times as much data as a human does by the age of 18. Yet the human by the age of 18 is better in a lot of things and on par with frontier models that have seen way more data. Could you come up with much more data-efficient systems that can continuously learn from their own actions? Continual learning is really interesting. Multi-agent interactions is an interesting thing. Creating ways of having better discourse among people in the world, having much more civil conversations, and helping people meet other people all over the world that they should know based on their interests—these are all interesting things. There are lots of cool things in the world, and we should all go and strive to make even cooler things occur.
Diana Hu: That sounds wonderful. Thank you so much, Jeff Dean. That is all we have today.
Jeff Dean: Appreciate it. Thank you all.