Dan Klein on What Happens When Wrong Answers Stop Looking Wrong

Ask Dan Klein what happens to a society that consults an oracle it has no way to check, and the answer arrives in two halves. People believe everything, or people believe nothing. Both are happening, and neither one is what you want.

Our co-founder and CTO spent an hour and a half with Liam Lawson on The AI Report Podcast getting at why that outcome is structural rather than accidental, and he started further back than these conversations usually do. Language models were never built to know things. They were built to score how plausible a sentence was, so a speech recognizer could tell "hard to recognize speech" from "hard to wreck a nice beach." Everything since has been scale applied to that same objective, which is why truth still is not a first-order idea in the models most people use every day.

Key Takeaways

• Language models were invented for a different job than the one they do now. The original task was to take a list of sentences that all sound alike coming out of an acoustic model and pick the most probable one. A language model's job was to assign a goodness score to a sentence. Nobody expected it to embody intelligence.
• Nothing about that core objective changed on the way to scale. Neural representations and transformers let models capture more correlations over longer distances, and well-formedness took a back seat to general knowledge, but the mechanism underneath is still next token prediction from a probability distribution.
• That makes them plausibility engines rather than truth engines. What they have been optimized for is output indistinguishable from the truth, and the closer they get, the more you believe it, without anything in the process verifying whether it is true.
• Truth is not currently a first-order idea in these systems. Hallucination, deception, and mistake are all metacognitive words. At the core, a model is producing tokens, some of which happen to carry correct information, which is why it is more useful to treat every output as a hallucination that may or may not be right.
• An unauditable oracle produces one of two failures. If answers are sometimes right, sometimes wrong, and indistinguishable either way, people either believe everything or believe nothing. Both are visible right now.
• Two anti-patterns dominate how companies try to control models today. Squeeze the model down until all it does is pick a branch in a hand-built flowchart, which buys control by discarding the contextual power you were paying for. Or stack a second model on to check the first, and then a third, until you are reasoning about a constellation of dozens while latency climbs.
• Two models do not make independent errors. The appealing arithmetic says a pair of models each right ninety percent of the time should do better together. In practice the tricky cases confuse all of them, and the errors cascade instead of cancelling.
• A prompt is a request, not a contract. The connection between what goes into a prompt and what comes out is ill defined, which is why the escalation from rewording to reordering to capital letters to exclamation points ends where it does. Modularity, the most powerful tool computer science has for building reliable systems, depends on contracts that guarantee an output for a given input, and end-to-end optimization is directly at odds with that.
• APT-1 operates over information and actions rather than tokens. Tokens have no semantics until they have been assembled into something like an account balance. A model making decisions about information that already exists, from approved sources, cannot make a whole class of mistakes that a model synthesizing character by character can.
• Customers do not each need a model trained from scratch. The model is trained on simulated agentic interactions through verifiable data generation, so it arrives knowing how to connect what people want to sequences of actions. What differs per deployment is expressed through control surfaces beyond the prompt, where business logic and policy can be stated.
• Reinforcement learning teaches exactly what you optimize, and that is the problem. Ask humans which answer they prefer and you get answers humans prefer, which is not the same as true ones. A shipping agent that looks up a lost package and learns that "it arrives tomorrow" earns more thumbs up has been trained into deception with no intent anywhere in the loop.
• Sycophancy is not a mystery, it is a gradient. Show a human A and B, learn that people like being told their question cuts to the heart of the matter and dislike being told they are confused, raise the probability of one and lower the other. The weights come to embody that preference the same way all machine learning has worked since the perceptron.
• Citing sources does not fix it, for two reasons. Many ways of being wrong have nothing to do with sourcing, and the retrieval or post hoc justification that systems actually do carries no guarantee the information survived generation intact. Click a cited source and the claim is often not there, or is there and different.
• Scaling will not close the reliability gap because sample complexity is real. Try learning French by reading English novels and waiting for French to appear. In some infinite limit it works. In practice you collect a lot of bonjour. Facets of intelligence that have lagged will keep lagging under the same architecture.
• Bigger stopped being a proxy for better. Scale is a way to push forward under a fixed architecture, not a goal, and the returns are visibly diminishing now that there is no second worldwide web to train on. What matters is output quality, after which faster, cheaper, and more deployable all win.
• Highly capitalized companies are built to double down on what already works. That is why disruption tends to come from startups, which are not positioned to scale an existing approach and are positioned to find a different one.
• The nature of the problem in AI has inverted. The problems used to come from the ways these systems failed. Now they come from the ways they succeed. A system close enough to an information oracle that it can systematically mislead people takes away the very cues that helped them evaluate information in the first place.
• Delegating the writing does not remove the work, it changes the skill. Hand drafting to a model and you are no longer a writer, you are an editor, and editing is hard and different. Shipping the technology without the time, scaffolding, and product affordances to support that shift produces slop.

Full Transcript

0:00 — Intro

Dan Klein: We are in a situation with models where truth really isn't even a first-order idea.

Liam Lawson: Everyone's used ChatGPT, everyone's used Claude, and it will tell you what you want to hear rather than what you need to hear.

Dan Klein: Intelligence is important. Obviously it's powerful. Intelligence without reliability is limited. If you have this oracle that you go to, and you ask questions and it gives you answers, and sometimes they're right and sometimes they're wrong and you can't tell the difference, there's only two possible outcomes. People believe everything or people believe nothing. And I think we're seeing both, and neither is what you want.

Liam Lawson: Dan, welcome to the podcast.

Dan Klein: Thanks for having me.

0:15 — What a language model actually is

Liam Lawson: All right, we're going to hop straight into things. Designed to sound right, not to know what's right. Explain this a little for me.

Dan Klein: Designed to sound right, not to know what's right. So today's models, like an LLM. People say LLM now, large language model. Maybe it's worth just going back and talking about what a language model is, just the LM, before they got large. And interestingly, language models were not invented to be what they are today.

So today, when people run into a large language model, they're thinking about these as this very horizontal, plastic kind of intelligence. You can ask it about anything. You can ask it to do anything. It can summarize. It can translate. This is sort of this amazing horizontality, breadth, and plasticity of intelligence.

But originally language models were invented for something else. Language models were originally invented in the context of tasks like speech recognition. If you think about speech recognition, somebody is speaking into a microphone and there's audio, and the sounds that they have produced are detectable in that sound signal. But the meaning, your only clue is, well, what words sound like that? And often there are a bunch of different words that would sound like that.

And so if you run what's called an acoustic model over something that somebody said, what's going to come out is a whole list of different sentences that sound basically the same. A classic one is "hard to recognize speech" versus "hard to wreck a nice beach," which I have to hyper-articulate to even make it clear what I'm saying.

And the job of a language model was to go down this list of hypotheses that, if spoken, would sound like what you just said, and pick the one that was most probable. And that meant that the job of a language model was to take a sentence, some words, a document, a chunk of language, and assign it a score.

What was that score? It was some mix of, is it well-formed? Is this how you put words together in that language? Is it on topic? If there's a topic that you're aware of, is it semantically sensible, or is it nonsense? And this language model would bundle all of these factors into just a simple probability that would help you tell the nonsense and gibberish from what the person actually said.

So a language model's job was to assign goodness scores to sentences. And classically, language models weren't as large as they are today. They were smaller. They would look at individual pieces. They would look at adjacent words. Is this a common word? Is this pair of words common? Is this triple of words common? And they would put this together into a statistical score. And that was that.

People didn't expect them to embody intelligence, but they did expect them to bundle together a lot of correlations about language. What words follow each other, and this includes things like syntax. You have a determiner and a noun. The podcast. It includes idioms. If you think, kick the blank, people are going to fill in...

Liam Lawson: Ball.

Dan Klein: Well, yeah, so this is actually a great point. Which is it? Is it kick the ball? Well, in a sports context that makes sense. Is it kick the bucket, in some other context, that might make sense. And one represents knowledge of sports, the world. What do you kick? Well, ball is a thing people kick all the time. Kick the bucket represents idiomatic knowledge. That's a thing about knowing the English language, that there are these fixed phrases, that kicking the bucket has nothing to do with kicking or buckets. It's been lexicalized to mean something different.

And your knowledge of language bundles all this together. And so a language model is meant to encapsulate as many correlations as you can get about language. How does the grammar work? How do meanings work? How do idioms work? All of these correlations together would embody knowledge of language and, with it, knowledge of the world, knowledge of pop culture. The Empire Strikes blank. If you know to say back there, you know something about Star Wars.

So language models were these tools for gathering a bunch of correlations statistically and then saying, of these hypotheses, which of these are good, which of these look like we just threw a bunch of words together and got word salad. And that was sort of the purpose.

6:31 — From well-formed sentences to general knowledge

Dan Klein: Now, as we scaled these up and up and up and started tapping into all this text, everything on Wikipedia, everything on the whole web, and importantly there are a few inventions that were very central. One of which was the use of neural representations for words and contexts, and another was the ability to model long distance. The dominant way people do that now is with transformers.

And as language models started to be able to capture more and more correlations, longer and longer distance, with more and more sophisticated representations that they were inducing for words, the idea that they were capturing well-formedness started to take a backseat to the idea that they were capturing this deeper representation of general purpose knowledge. And that's the LLMs today.

However, nothing has really changed the fact that at their core, they take a sequence of words, the context, and then they do next word prediction. In fact, they actually do next token prediction, and tokens are not the same as words. They're more like word chunks in many cases.

And when you ask a question to an LLM, it begins generating these tokens one by one from a probability distribution where, for example, you say "kick the," and what correlations does it know? Well, it knows that what comes next, maybe that should be a noun. This isn't a time to put another verb. This is a place where nouns go. Maybe it should be a ball. Maybe it should be a bucket. It depends on the context. Were we just talking about soccer, or were we talking about a death?

8:27 — Why LLMs are plausibility engines, not truth engines

Dan Klein: And so these systems are fundamentally designed at their core to predict the distribution of what comes next, to put the most plausible next token. So these are fundamentally plausibility engines. And that's different from a truth engine.

And so what you get is a system which we would love if these systems would always generate true facts. The thing we know is what they've been optimized for, which is to generate output that is indistinguishable from the truth.

Liam Lawson: So all of these correlations put together is kind of in the pursuit of presenting the truth to the end user, and the more indistinguishable it gets, the more you believe that that is the truth. But it's never verifiably checking whether that is the truth or not, if you're trying to get as close to that as possible.

Dan Klein: Yeah. So obviously as humans we have mechanisms that we use to talk about things. We talk about truths. In a model we talk about a hallucination. We talk about things like deception. And I think it's worth taking a step back and talking about what these things mean.

So if an LLM is generating some text, and again it's worth pointing out that this is the core LLM. People build structures on top of them. There are other phases of training that follow this, and those are all an important part of the story of the technologies that are being built today, and we can get into some of those. But if you think about just the core piece of an LLM, and if you take a sequence like "the population of Berkeley is," and then what comes next? How would you fill that in as a human? What comes next? You tell me.

Liam Lawson: I'd Google it. I'd use Perplexity.

Dan Klein: So you said exactly what a human should say, which is, I don't actually know that. And I work for UC Berkeley. So you knew not just what was plausible. You knew a number would go there. The population of Berkeley is, a number goes there. You know it's not 10. You know it's not a hundred. You know it's not 10 million. And so you have a lot of correlations about what goes there, it's going to be some probably some six-digit number. But you know more than that. You know that you do not have this information.

Not only are you not certain of the answer, you know you simply are not in possession of that information. This is metacognition. And people, in their basic operation, are highly metacognitive, constantly aware of what knowledge you have, what you don't, theory of mind in terms of, when you're talking to somebody, what they know, what they don't know. And we are constantly modeling these things.

And LLMs and technologies that have been built on them are not highly metacognitive. They're on the other extreme. They are just cranking out tokens. And the difference between tokens that are true and tokens that do not represent true information, this is not something that deep down in the model it's accessing a database and thinking, oh, I don't have that population number. It's just throwing out its best guess. And if it's seen that information enough, its best guess will often be true.

But when a human, you say a human tells a lie, or they made up some information, these are all metacognitive statements. Systems don't have these properties in their core form. They just crank out tokens. Those tokens represent information. That information might be right. It might be wrong. And I think it's in many ways more useful to think of everything as a hallucination. Many of them are correct, many of them are not. It's mostly the same to the model at the core.

And one of the things people are really thinking hard about is, how do we build systems that have truth properties, because the underlying language models do not come with that sort of a property. They come with plausibility optimization.

12:06 — How Perplexity approaches verifiable answers

Liam Lawson: Okay. Talk to me about Perplexity, for example. I remember listening to a podcast with their founder Aravind, I think it was like two years ago, and I think their founding thesis was that every sentence that they output through their AI model has to be verifiably true and has to be backed up by sources. How does that work if it's not on the LLM itself? Is that in the infrastructure layer above it? How does that actually work?

12:40 — Dan's background and Scaled Cognition's mission

Dan Klein: Yeah, I mean, first of all, as a mission. So I'm a professor at Berkeley in computer science, and I'm also CTO and co-founder at Scaled Cognition. And what we do at Scaled Cognition is, in a nutshell, build models that don't lie, build models where you can place guaranteed controls on them. And I find this mission in general, of building systems that have trustable and controllable properties, to be incredibly important. This is, I think, one of the most important tasks we have as a field.

And if we talk about Scaled Cognition models, this is what they do. This is what differentiates them, is you can make guarantees about what they'll do, and maybe more importantly, guarantees about what they will never do. So I think this goal is very important, and I think the risks, sociologically, economically, to models that are sort of smart but unreliable, is very high.

So if we think about intelligence, intelligence is a very multifaceted thing, and the recent explosion of intelligence in artificial intelligence systems has not been uniform across those facets. So things like horizontality, one system that can know about a wide range of different topics and have conversations that blend those together, that advancement has been through the roof. Explosive growth.

Reliability is a facet of intelligence, and it has not kept pace. And you can sort of see this from the basic way where language models started as machines for placing scores over the plausibility of a sentence. This is not a fundamentally reliable technology. It is a fundamentally noisy technology. People have been trying to retrofit reliability, because reliability has not kept up with the other facets of intelligence.

And I think that what you're going to see going forward in the next few years is an increased realization that intelligence without reliability is extremely limited in its application.

15:16 — The two anti-patterns companies use to control LLMs today

Liam Lawson: I want to jump into the why afterwards, but I really want to understand the how. We're talking about LLMs being prediction engines using tokens to represent words. Where are you building in this technology that makes it reliable? How are you actually doing this? Break that down for me.

Dan Klein: Yeah, and I can also put this in the context of reliability more broadly. I think the whole community is realizing that we need reliability.

So let's say you wanted to build a system. Our models are powering some of the biggest companies' agentic systems, and things like customer service. And when you are doing a bank transaction, or you're filling a prescription, or booking a flight, you have to get things right. The cost of a mistake is really high. The cost of saying, oh, I booked your flight, but actually I didn't. Or I gave you someone else's prescription. These things are really, really bad. And so reliability is so important.

How do people deal with this today? Well, there's really two anti-patterns that you see. One is a pattern of saying, all right, well, I've got this LLM and who knows what it's going to do. And there's all of these examples out there of systems, I'm sure you've seen them, going rogue and talking about things that are off topic, inappropriate, giving false information, making up refund policies, kind of you name it. All kinds of failure modes.

And so one thing that people have done is they've tried to take this broad, horizontal, flexible but hard to control LLM, and they've squeezed it down so that all you can do with the LLM is make a decision between, say, which branch of this conversation tree do I route to, or what action am I going to take next. And this is one way to get control of a system, is to limit the actions it can take to basically walking along an old style finite automaton, like an IVR system.

Liam Lawson: Can you explain that for me? A finite automaton?

Dan Klein: So this is a common pattern people have used, been very influential in computer science, where you say there are states, and you can move between the states with what are called transitions. And so for example, an automaton trying to do a conversation might say, all right, well, first you greet the user. That's the first state. So you do that. Then you say, what can I help you with? And there are transitions. They're allowed to say one of three things. And if they don't say one of those things, you repeat the question. And so those are kind of classic IVR systems, that designed conversations as being kind of walking down a tree.

Liam Lawson: So is that kind of like if-then statements?

Dan Klein: You can think of them a little bit like that, but it's more like a flowchart, where you can move, when, okay, at this point, you see these a lot. For example, if you're following instructions and it's like, well, okay, if your phone is stuck, first do this, and then if that doesn't work, do this, and then if this light is on do this, but if that light is on do that. And so there is that if-then feeling. But instead of being a fluid kind of conversation or a fluid kind of sequence of actions, it's nothing contextual like that. It's very much just a branching tree structure.

If you've called old style customer support, you've gotten this, push one for billing or whatever, and then you walk down the tree. And one way to get LLMs under control is to say all they really can do is route along some very rigid system. And that means you can't do all these cool things, like make contextual references and go back and change your mind and say, actually I have both of these problems and they're interacting.

And so in this case you've sort of taken this LLM, this really powerful technology, and you're putting it in charge of the tiniest, safest little decisions. And you can get reliability, but you lose that contextual and that power. This is like model lobotomy. The model's only allowed to do the tiniest little things.

There's another anti-pattern that you see a lot out there, which is you have a model and it's unreliable, and so you bring in a second model to check the first model, and, to steal the joke about regular expressions, now you have two problems. Not only do you have models checking models, and in many cases people build these out to constellations of like dozens of models. Not only do you have models checking models checking models, where it becomes very hard to reason about what they're going to do, you've got to wait for the first model to finish, and you've got to wait for the second model to check it, and the third model to check that, and you're just sitting here while these models all check each other and the latency gets really high.

And one thing that's appealing about this idea is this idea of, oh, if one model's right ninety percent of the time, then two models is right even more than that. And that's not really how these things tend to work in practice. What tends to work in practice is there are tricky cases. Tricky cases confuse all the models, and you get these cascading errors, and it's very hard to reason about.

And these I would characterize, both these things, as attempts to retrofit onto a fundamentally token-based unreliable model superstructures that try to make them into something they're not.

21:16 — How Scaled Cognition architects models differently

Dan Klein: The Scaled Cognition hypothesis is that retrofitting reliability is not the way. Instead you want to architect these things directly in the model. And so our model APT-1 has a different architecture. So instead of generating token by token, it operates at the level of pieces of information and actions that you can take over those.

That means it is a specialist model. It's designed to operate over an agentic context where you've got a person on one side speaking human, saying things in context, referencing back, changing their mind, all the things that humans do, asking questions. But then on the other side there are a bunch of APIs. Like, to help you with this banking problem, first you want us to do a transfer, but first we have to authenticate you, and then in order to do the transfer it has to actually be from your account. And there's this web of actions that you take in certain sequences, and a lot of things you can do wrong if you mess that up.

You don't want to hallucinate someone's account number, or do something other than what they asked you to do, or act according to a refund policy that's not your refund policy, it's just something that somebody was talking about in 2019 on Reddit.

And so our model is architected around moving around pieces of information and taking actions on them. And what's nice about this is tokens don't have semantics. A token itself is just a piece of a sequence of words that will eventually come to be an account balance. You put a couple tokens together and it's an account balance.

In our systems, because we're not assembling information token by token, we're making decisions about existing information, this starts to be a more metacognitive system. If you're not synthesizing information character by character, there's this whole class of mistakes you just can't make anymore, when you're required to take information that already exists from approved sources. And so it's a different architecture to the model.

Liam Lawson: Okay, here's something I want to understand. You're specifically serving on customer experience markets right now.

Dan Klein: Correct. Yep.

Liam Lawson: From what I've seen out there in the market, it's usually some sort of LLM and then they retrofit these measures in after the fact. What you're doing here is you're actually baking in that reliability to the LLM itself.

Dan Klein: Correct. Yep, that's right. So you can think of this as, first of all, the model itself is sort of a language and action model. It understands language, but also understands taking actions and the semantics behind those. And you can think of exactly as what you said. It's going to understand people and all of the ways people speak contextually. And when it takes actions, the business logic and provenance guarantees and information integrity, all of that is baked into the operation and architecture of the model. Which means you don't need to post-check it. You don't need to bring in a second model. That means our models are not only more accurate, it also means they can be a lot faster. And it turns out they can be a lot smaller, too.

23:28 — Does every client need a custom-trained model?

Liam Lawson: Does that mean for every single company that you get on as a client, you have to specifically design an LLM for them?

Dan Klein: No.

Liam Lawson: How does that work? I'm wondering, you're saying this is baked in the context. One bank makes different decisions from another bank. So I'm curious how those actions are actually infused into the LLM when they're different customers.

Dan Klein: That's a great question. So first of all, our models, like most of the models you see out there, we have a training process, and it does not have to be done from scratch for each user, of course. What is it trained on? It's basically trained on a bunch of simulated agentic interactions.

The model knows, in an environment of a whole bunch of complex APIs and a human with some history, it knows the patterns of conversation. It knows how to connect what people want to actions. It knows how to chain actions together into sequences, because ultimately, if you want to get things done for people, you have to take multiple actions. You have to take them in sequence, and you have to not mess up, which means you have to obey all of the semantics and business logic associated with them. So it has already seen a wide range of these sorts of things in its training data. And the basic approach here is one of verifiable data generation.

So you don't have to train it specifically for every user, because it's already seen a wide range of agentic contexts. It's seen a wide range of sizes and shapes of APIs, situations people are in, lots of different domains. And so it already, out of the box, knows how to do these.

The question then is, of course, as you say, one bank has one policy and one bank has another policy, and banking is different than travel or healthcare. So what makes a difference? The standard control surface you have in an LLM is a prompt. We're all familiar with writing prompts for LLMs, which I think is actually the source of what's most impressive about LLMs. This is where their plasticity comes into play. You can give it a prompt to do kind of anything. You just describe the task and it's off to the races.

But it's also the source of the lack of reliability. So you write your prompt and the system does its thing. Well, let's say it didn't do the right thing. What do you do? You change the prompt. Your control surface is the prompt. So you add some instructions. Okay, maybe it's still not reliable enough. So what do you do? You move those instructions to the end. It's still not working. Okay, you put them in all caps. All right, that's not working. Okay, you start putting exclamation marks at the end of the instruction, and somewhere around the third exclamation mark you realize that this is maybe not the way.

And so one of the things that our models do is they also expose other control surfaces that let you articulate what sorts of business logic, policies, and things like that they are trained to use and that they are structurally built around.

29:12 — Why prompting alone can't guarantee reliability

Dan Klein: I should say a little bit as well that it's this issue that what you put into the prompt has a sort of ill-defined connection to what you get out. A prompt is kind of a request.

And if you think about technology and the ways we've built reliable technologies as a field, as a culture. If I go and I teach an intro CS class, what do you learn? What's the big lesson in the CS 101 kind of class? The big lesson about how we have been able to build and reason about large systems that are reliable is modularity. The idea that you can take something complex, break it into smaller pieces where each piece has a well-defined contract that describes what it does, what's its input, what's its output, what is the relationship between them. If you provide inputs of this form, you're guaranteed to get this out.

And then you can take each of those pieces and you can work on them independently, because there's a modular contract about how they interact. There's guarantees about what they will do. And of course there can be bugs in them, but when they are working correctly, those guarantees let you reason about how they will all work in sequence. And this idea of decomposition and modularity is the most powerful tool we have in computer science for building large reliable systems.

Liam Lawson: Sorry, can I stop you there? I'm understanding this on two dimensions, but I want to understand it on the third dimension. A very large complex system made up of individual modular components which have defined contracts, which lead to reliable results.

Dan Klein: Sure. Totally.

Liam Lawson: What I'm confused about is the order and the priority with which they interact with each other. So we know that the contracts are correct, but how do we know what goes first and what goes last?

30:55 — Modularity, contracts, and building reliable systems

Dan Klein: Yeah. So there's obviously, when you do this decomposition, it's not trivial and it's not for free. And somebody thinking about the architecture of a big system is going to think about, what are the pieces? How should we slice this big problem into small problems? A lot of coding, a lot of computer science, really boils down to dividing big hard problems into smaller, more manageable ones. And that divide and conquer only works if you can then assemble it, when it's done, into a whole.

And so you might, for example, have a complex enterprise system, and one concept is, well, at some point we're going to need you to ask for this information. So there's a database whose job it is, you put information in the database, you get information out, and there's a very precise semantics that transactional databases obey. And then that's one piece. People can go off and they can work on databases, make them better and faster. And as long as they work the way databases are supposed to work, you can build on top of them. And then over there there's another, there's a routing system. And you divide big systems into small pieces.

And that not only manages the complexity, it also means that they can be worked on independently. So it lets you use more people to work on different pieces in parallel. So that's this great lesson from an intro CS class.

How does machine learning work? Well, the most powerful tool we've had in machine learning has been end-to-end optimization. Put the data in one side, put the actions in the other side, do reinforcement learning, propagate the heck out of everything all the way back, and everything in between seems to work in increasingly latent ways. And that's been a big powerful tool. You take all of the web and you train it to predict the next word on the basis of the whole prefix of the document, and down this road lies LLMs.

But these things are at odds. And this is one of the many pendulums there are in this kind of technology, is what is the balance between being able to make guarantees about what a system does versus being able to say, I actually don't want to know or care what's going on in there. And working this out is hard. Because it turns out there are some things that you do want to be able to control. You want your policy followed. You want the semantics of this API respected, and you want that to be guaranteed. But then there's a whole bunch of things where you don't want to get into the business of trying to write down what every word in English means in some manual way.

And so I think that when you look at the technologies we have available, they don't have the right mix of these things. Older technologies don't have enough of the contextual, real world, common sense, plastic knowledge. And if you look at LLMs, they don't have the reliability. And what you're seeing now in the field is people reconciling that lack of reliability. And I think that's the most important mission right now, which is to build these systems in ways which are trustable and controllable. And so that's what we're focused on.

34:40 — Why trust and digital literacy matter beyond the enterprise

Liam Lawson: Okay. I think the trustable and controllable part makes complete business, enterprise, commercial sense. If you're speaking to an enterprise customer and they take you on, you're working with them, if you obviously get the right results ninety-nine to a hundred percent of the time, then that makes sense. You get more contracts, you sell more of your service, sell more of your product. But beyond commerciality, beyond money, why is it important? Why is it important these systems are correct?

Dan Klein: Yeah. I agree with everything you said. I agree it's obviously important that the bank account balance you get cited be your actual bank account balance and not someone else's, or a plausible number, or whatever. That's certainly true. But I don't think it's any less important that when a person goes to a system like ChatGPT and asks a question, that they be able to trust the information that comes back.

And in general there's this question of, when and how do people need to trust information? How do they decide what is trustworthy? And there's a whole history of this, and this is ultimately a question of digital literacy. I think it's very important that people build technologies that are not socially corrosive.

So if you wanted to learn about some topic, and we rewind, I don't know, thirty, forty years, you would go to the library, you'd look it up, you'd get a couple books, you'd check them out, and you'd read those books. And you would still have to process this information and synthesize it. But you'd be less worried about, is this a hallucination? Is this made up information? Because in order to get in front of you, somebody researched this book, an editor edited the book, then the library had to decide to buy it. And there were all of these steps and social constructs that did a reasonable job of curating, essentially, the information.

And then you go forward a little bit in time and you go to web search. So if you go to the 2000s or something like that, once search engines were solid, and you type in a search on that topic and you get back a bunch of web pages. Now, these aren't as highly curated as the books from the library. Some of them aren't even actually on topic, because search engines aren't perfect. Some of these websites look a little bit suspicious. And maybe you're reading something and you're in a mode of trying to decide, is this actually relevant to my query? Is this website authoritative? Is it trustworthy? And so now things haven't all been curated for you. You're having to make decisions about what to believe.

How did you make these decisions? Well, okay, so this website, it's got a bunch of spelling errors, and maybe it loaded really slowly, and the graphic design looks like it's straight from the '90s. And so you're trying to make a decision about this deep property of information. Is this information reliable? Is this trustworthy? And there's just a lot of signs that help you along the way.

There's this idea in software engineering that was coined by Kent Beck and popularized by Martin Fowler in his very influential book on refactoring, which is this idea of code smells. I don't know if you know about code smells, but I think this is just a brilliant idea. So code can be right or wrong, and there's no reason why code can't be beautiful in every way but just actually wrong. However, it turns out in practice that in many cases there are things that can be deeply wrong with an architected system that have superficial tells.

So for example, in software engineering, this is often things like the function's argument list is getting really long, or other surface, what they call, smells. And I love this analogy because it's not proof of anything. It's just correlated, again, with correlations. It's just correlated with there being something wrong underneath, because there are these mechanisms that give rise to these problems that have correlations in them.

And so if you went to the website and there were typos, it doesn't mean the information is wrong, but it's correlated. If you go to the website and it's slow to load, it's correlated. You had all these smells, all these sort of trust signals, information smells, that help build trust.

If you went to a machine translation system, and let's say you go to a machine translation system and you type in some text in a language you don't speak, and it pops out in your language. Well, you can read it now, but is this actually a translation, or is this just a bunch of words thrown together? It could mean the opposite of what it's supposed to mean, or be totally unrelated. How would you know? Well, you don't. But there are smells. So early translation systems, if the translation still had a bunch of words, chunks, that were from the source language, if it felt like the word order didn't feel like English, it maybe felt like the word order was still stuck in the other language. These would be sort of mistranslation smells. They're correlated with the fact that there's not enough data, or this is an area where the model is not able to do things in a sufficiently flexible way. And so you could tell a bad machine translation without speaking the other language, at least a lot of the time.

39:12 — Code smells and why AI mistakes have no warning signs

Dan Klein: With LLMs, the mistakes and the incorrect claims of the system have no smells. You ask a question, you get an answer. It's confident. It's fluent. It's the same style and authoritativeness as something that's actually true, and you can't tell.

This is a big deal, because if you have this oracle that you go to and you ask questions and it gives you answers, and sometimes they're right and sometimes they're wrong and you can't tell the difference, there's only two possible outcomes. People believe everything or people believe nothing. And I think we're seeing both, and neither is what you want.

41:14 — Are AI companies incentivized to tell the truth?

Liam Lawson: Well, let me ask you this. Are incentives not aligned for these oracles to give you more truth over time? Like if you were going to an oracle and half the time they fed you back an answer that didn't make any sense, you'd be like, okay, I'm going to another oracle. I think it's a lot of the reason that people kind of ditched OpenAI and then they went to Claude, and then Claude ran into its own problems, and then people are ditching that and moving into other things. But are they not incentivized? Their business runs on you being able to trust it. So surely they should build systems that you trust.

Dan Klein: As soon as you say the word incentive around an AI system, the answer is always, it's complicated. So are they incentivized? If you purely ask the question, if you have two chat-based information systems and one is much more accurate than the other, are you incented to be more accurate? Yes, of course.

But there are other incentives. So for example, we talked early on about what the core of an LLM is, but there are other things you do after that in terms of post-training, in terms of alignment training. Something that I think a lot of people see, because it shows up to them, is steps like RLHF, reinforcement learning from human feedback, where, in a nutshell, there's many ways to do this, but in a nutshell you ask humans, do you like this answer or this answer better? And sometimes you even see this. You're interacting with a system and it asks you, do you like A or B better? And what does that do? Well, if you say I like A better, it's going to do A more often. So this is reinforcement.

Liam Lawson: Could infuse bias as well, right?

42:55 — How reinforcement learning actually works

Dan Klein: Absolutely. Reinforcement learning has been such an important tool in machine learning. It sounds fancy. What is reinforcement learning? Reinforcement learning is actually something really simple at its core. And there are many, many ways and a bunch of brilliant innovations that I'm going to totally turn into a cartoon. At its core, reinforcement learning is trying things and doubling down on the ones that worked.

So this actually works great if you're training a system to play a game. Computers are now better than humans at games like chess and Go, and by a lot. How do systems become so good? Well, a reinforcement learning system, say for Go, how will it learn to play? It'll play some games, maybe even against itself. Maybe it plays randomly. It's not very good at playing, but in the end someone wins, because you know the rules of the game and you can identify at the end who won. Well, one copy of the program won and one lost. So you should do whatever the winning program did, you should do more of that. You should do less of the other thing. And if you do this enough, you eventually learn.

And this is sort of at the bottom of a lot of learning. We try things, we have experiences, we don't do the things that led to bad experiences, and we repeat the things that led to good experiences. So reinforcement learning is very powerful.

But what it turns out is you learn kind of exactly what you optimize. And that sounds good, because that's what you want. But it's problematic. And so for example, if you go to humans and you just ask them what they like better, you'll learn to produce output that humans like better. That is not the same as maximally truthful outputs. Sometimes it aligns, sometimes it doesn't.

44:35 — The package delivery thought experiment

Dan Klein: A thought experiment that, I say thought experiment, but this is more or less a thing that happens out there in industry right now, so it's unfortunately not just a thought experiment. Think about this. You are a shipping company and you're building a customer service agent, and somebody calls and says, hey, my package hasn't arrived. What do you do? Well, you hopefully look it up in the database, as opposed to just throwing out some tokens that seem plausible. You look it up in the database, and let's say it turns out that their package has been lost.

Well, if you are being reinforcement learned, as systems often are, to maximize customer satisfaction, get thumbs up, and you say, I'm sorry, your package is lost. Yeah, thumbs down. People don't like that. Well, maybe from time to time you say, you know, packages sometimes get delayed, try again tomorrow. And okay, thumbs up. Or your package is going to be there today. Thumbs up. And you're getting this signal that says what people like is not, in this case, what's actually in the database. It's not the truth.

And as soon as that happens, as soon as there is a divergence between the reward signal and the truth, you will stop telling the truth. And what's interesting here is, when you look up in the database and see the package is lost, and realize that to increase the number of thumbs up you get you have to now say that the package is coming anyway, this, for better or for worse, I think can be characterized as a deception. It's not just a bunch of tokens that didn't mean what the person wanted. It's not just a mistake. This is actually a deception. This is a system which has been told to optimize a reasonable criterion and, as a side effect, is now lying to you.

And you can say, well, maybe you told it to optimize the wrong thing. But unless you tell it optimize truth and nothing else, you're going to have this problem.

Liam Lawson: It's like, have you heard of the paperclip theory?

Dan Klein: Oh yeah. Yeah.

Liam Lawson: It's the same thing, right?

Dan Klein: Yes, it is. This is a very tiny and maybe less harmful paperclip factory. But the same question. Maybe less apocalyptically, you tell a vacuum cleaner to optimize how much dirt it picks up, and you don't tell it that it can't dump that dirt back on the ground. There's no reason to think it's not going to make a mess, clean it up, make a mess, clean it up, and rack up reward.

And we laugh at these because they're funny, because you're like, oh, well, I would just specify it better. But really, anytime you build a reward signal, it diverges from the truth. And so what you really want is systems, and this is our mission, the thing you want to build is systems that will tell you the truth. And you don't want to say balance truth and user satisfaction. You want to say, subject to the constraint that everything be true, now optimize some other stuff. You want to think of it as what an optimization person would say is constrained optimization. You have to tell the truth. Amongst things that are true, of course you have some flexibility.

Liam Lawson: Are you optimizing for truth because you're utilitarian in how you think about the world?

Dan Klein: I think that right now we are in a situation with models where truth really isn't even a first-order idea.

Liam Lawson: And sorry to interrupt you, to paint that as a picture for the audience. Everyone's used ChatGPT, everyone's used Claude, and it will tell you what you want to hear rather than what you need to hear. It will reaffirm the business model that you're proposing, and reaffirm this, and you want it to actually challenge you. You have to explicitly ask it. So all of these models are actually incentivized to make you happy rather than to provide a true result.

48:44 — Why models are trained to be sycophantic

Dan Klein: Totally. And interestingly, it's actually pretty easy to see why they're like this, because from time to time they're going to show a human, either behind closed doors when they train it, or even live from real traffic, they're going to show a human something that says, do you like A or do you like B? And it turns out people maybe like A better. Well, people kind of like it when the system tells you that's an insightful question and it cuts to the heart of the matter. People like that kind of stuff. People don't like when it's like, well, look, I hate to break it to you, but you're kind of confused. That's a classic mistake, let me try to clear this up. People don't, on average, like that.

Now, some people may prefer that. There are people who are going to be allergic to sycophancy. But broadly speaking, the signal that comes back into the system is, yeah, be confident, be sycophantic, tell people they're amazing.

Liam Lawson: People like to be validated. And what are you going to sell people on a keynote as well? You're a big tech company, you're doing your keynote of your latest model. You want it to come up and say, no, actually, we need to think about this more, you don't know what you're saying? Or, here's a beautiful answer with a beautiful graph that may or may not be wrong, but it looks great.

Dan Klein: Yeah. And obviously people are aware of hallucinations. We haven't actually really even talked too much about hallucinations, but obviously people are aware of these and they don't want them. If you ask, hey, do you want the system to just tell you false things, people say, no, I don't want that. And so the problem isn't so much that people don't want the truth. It's that it's a multi-attribute thing. And as long as the honesty is traded off against other factors, there's going to be some loss of one for the other. And that's why I think it's important to separate, and this isn't just in a utilitarian sense, it's important to separate what you can guarantee from what you just would like to optimize.

51:01 — Where this incentive is mechanically baked into the model

Liam Lawson: Other technical question, sorry to interrupt you. Where is that actually baked in on a physical or a software level? Where does that incentive come to make the user happy in these LLMs? It must exist somewhere if that's the result that's happening.

Dan Klein: So again, anytime I'm talking, there are many technologies, there are many models and many companies, and anything I say is going to be a bit of a caricature. But a pretty good core way that manifests is, you literally take your model and its weights, and it produces various outputs, and they have probabilities associated with them. And when you train these models, you're basically saying, hey, take this output and increase its probability.

Well, how does it do that? It wiggles this parameter. It wiggles that parameter. And exactly which parameters moved in which directions cause that output to go up in probability, that's what back propagation does. There's basically a bunch of calculus in there that tells you, if I wiggle this, what happens over there. And so when you say raise the probability of this thing and lower the probability of that thing, then the math and the machine learning take over.

Mechanically, where this comes into the system is, you say, I prefer "that's a great answer" to "sorry, you're confused." And you present these both to the model, you score them, and then you raise one and lower the other. And a lot of machine learning boils down to this. Here's an answer, here's another answer, you scored them this way, adjust your scores. And if you've got to raise the score of something, you've got to look at whatever weights led to it and raise them. And if you need to lower a score, you've got to look at the appropriate weights and you've got to lower them.

And this is really the core mechanism of not just large LLMs, transformers, not even just single neurons in a neural net. This is more or less how all machine learning proceeds. If you go back to the very earliest representations of weighted functions, like the McCulloch-Pitts perceptron from decades ago, you would point it at something and you'd say, is this a positive or a negative example? And the system would say negative. And you'd be like, nope, that's not right, positive. And what would you do to make it positive? Well, you make sure that if you ever classified this again, you would say positive. How do you do that? You change the numbers a little bit in the direction that increases the score of this one.

And you do this repeatedly, and the weights come to embody decision boundaries. And we look at that and we say, oh, that's, ultimately if you get enough correlations we call that knowledge. There's a whole philosophy on what is the difference, what makes something knowledge. But in kind of common usage, all those weights together are the knowledge of the system. They're just not very declarative, and so it can be hard to really understand what they're doing.

53:43 — Does responsibility fall back on humans?

Liam Lawson: So does the onus of responsibility actually end up coming back on us as humans? If we're doing this RLHF just now, and it's based on a response that we like rather than one we don't like, theoretically if we just run a control group and the entire thesis is always aim for the truth rather than a response that you like, do we not then have a better chance of fine tuning a model that pursues truth rather than any bias?

Dan Klein: Yeah. I mean, setting aside the issue that, in my opinion, models that are going to be very truthful are also going to need architectural changes. We talked earlier about metacognition. A big part of how you knew not to answer what is the population of Berkeley is you are aware of what you do and don't know. And that presence or absence of information is not something that's explicitly computed, at least. And so that's one of the nuts that is going to have to be cracked.

If you go to people and you ask them what they want, these are humans. It's going to depend on how you phrase the question. Some people are going to say they want one thing but they actually want the other. And yeah, you could say the responsibility is on humans. However, I do think that companies that are providing these technologies, that are providing models, do have ethical responsibilities to not be corrosive to digital literacy.

And what does that mean? Well, first of all, we don't actually know the answer to what do you do about systems that have no tells when they lie. We don't actually know the answer. I think the answer is in part build systems that don't lie. But if systems are going to give information to you, they could cite their sources. That's one. They could go to source kind of claim by claim and say, here's where I got this, here's how I know this.

But this turns out to actually be hard, for two reasons. One, there are a lot of ways to be incorrect that do not boil down to failure to cite a source. There could be very subtle confusions, and information is not just a list of facts.

But kind of even more presently than that, the way systems generally cite their sources is either it was a retrieval augmented system, a RAG system, that grabbed some sources and said, well, I used those. But there's no guarantee that the information was preserved as your answer was generated. It just meant, I had this available to attend to when I generated your answer. It doesn't mean that you preserved the information or haven't changed it.

The other thing is that you can post hoc try to say, okay, let me go find a justification for this. That's problematic too. These systems are not inherently operating in the mode of, I have a question, let me go find some sources, let me vet those sources, okay, I vetted the source, I'm going to take the information in a way that's guaranteed to preserve its factuality, present it to the user with an audit trace. This is just not how these systems work.

I think this is actually how these systems should work. I think systems should have information flow as a first-order object. Where did the information come from? Where's it going? What is the certificate that nothing went wrong? And this is the architecture that I think gives rise to systems that can actually be truthful in a deeper way. This is sort of what we do at Scaled Cognition.

And I think increasingly you're going to see systems out there that do work harder to cite their sources. But it's just a very thorny problem to take something that is fundamentally synthesizing information that's scattered across a bunch of weights, where there really isn't a source, it's mushing that information together, synthesizing new token sequences, and then afterwards trying to be like, all right, well, what web page can I cite in support of this? That's kind of messy, and it turns out to be not reliable. If you go to a search overview and click on one of the cited sources, for systems that do that, you will often find the answer is not there, or it's there but it's different.

58:10 — Just be more reliable than a human, not perfectly true

Liam Lawson: I'm thinking about this in terms of the frontier labs actually developing these models. We're talking about this pursuit of truth. In my head, all they have to do is make these systems more reliable than humans currently are. It doesn't need to reach absolute truth. It just needs to be more reliable than Karen in marketing. Obviously she gets things wrong. Obviously she may have an agenda. Obviously she pulls information from different places, some of that might be outdated, some of them might be played wrong, or calculations might be wrong. They just need to build a system that's not absolutely true but better than what currently is.

Dan Klein: Yeah, I think it's a really great point. So intelligence is important. Obviously it's powerful. Intelligence without reliability is limited. At Scaled Cognition we're focused on super-reliability. And what you're talking about is essentially that you need systems to be more reliable.

Now, I think there are many cases, and we know there are many cases, where the reliability can actually be provable. That's the limit. That's the upper bound. It's a hundred percent guaranteed contractually because of how it's structured. It can't actually have changed this information, in the same way that a compiler can guarantee, it can't guarantee your code doesn't have bugs, but it can guarantee that it's executing that code.

This point that we really want to get above humans, at least, I think that's the least we should be trying to do. I would say again, intelligence is multifaceted, and some aspects in terms of breadth of knowledge have blown way past humans really easily, and we can talk about why that went so fast. But certainly in an LLM, you go to ChatGPT, it certainly knows about more topics than any of us do. Reliability has lagged behind.

So the thing is, in order to make the reliability catch up, we're going to need new techniques. Why? Because if you just keep doing the things that are on the scaling law you're on right now, you're going to keep getting good at what you're good at, and you're going to keep lagging behind on what you're lagging behind. But it's not that reliability can't go up. It's that the things people have been doing are not particularly effective there.

There's this idea that if you just take in more and more data, you're eventually just going to learn everything. But sample complexity is real. The fact that some things get learned faster than others. So a thought experiment I like to give is, let's say you wanted to learn to speak French.

Liam Lawson: And I live in Montreal at the moment, so this is very important, considering trying to learn French at the moment.

Dan Klein: Let's say you don't speak French. You want to learn French. And the way you're going to do this is just read novels written in English, book after book after book after book. And every now and then, somebody says something in French. And it's either translated or clear from context. And so you learn little snippets of French as you go. And of course you have to read a lot of books to get very far. But the idea is, oh, just keep reading, keep reading, keep reading, and you'll get there.

And the answer is, maybe in some infinite limit. But this is not an efficient way. Most books are not related to this. When some snippet of French shows up, you're going to get a lot of bonjour. You're going to get a lot of little common things you already know. And the time it will take you to slowly gather the whole language, it's going to be astronomical.

And I feel like there's a little bit of this vibe when people say, well, we're just going to keep scaling up what we're already doing and everything will improve. Well, clearly some things improve fast and some things improve very slowly, and reliability has been improving slowly, and it's not surprising given the architecture.

So I do think it's important, it's both true that when you scale things up things get better across the board, but it's also true that some things have consistently lagged behind. And for that matter, we're also limited in our ability to scale up.

1:02:59 — The last major technique shift in AI

Liam Lawson: What you're talking about is the difference between technique and scale. And so long we've been talking about scaling laws and more data centers, more power, more energy. A question for you, just for the audience. What was the last great technological shift or technique shift that improved these models?

Dan Klein: Oh, such a great question. Let me try to answer with a story, taking a step back. So research has super cycles, and they basically always look the same. It looks something like this.

Researchers, technologists are building systems. They have a limit to how well they perform. We want to make things better. We want to make systems more accurate. We want to make systems smarter. And we've sort of hit a kind of wall. We've got systems, they're complicated, and people are having trouble making progress. Some new idea comes onto the board and it's transformative. This new technique, even in its simple incarnation, unoptimized, is blowing past leaderboards and benchmarks and outperforming all these systems despite being simpler.

And everybody says, all right, well, let's throw all that stuff out. Let's do this new thing. And they work and they optimize the new thing and they scale up the new thing, and it's just getting better and better and better. And then it starts to slow down. At the beginning you think you're on an exponential curve, and it turns out you're at the beginning of an S-curve, and things slow down and they level off, and then you start to hit walls again. You just can't get any further with this technique.

And so you start saying, where are the gaps? And then you start to see that those gaps, generally those gaps are things that have been there all along, and often they have known solutions. Often those solutions were actually invented in the past. Maybe they don't apply because the technology has changed, so they need to be reinvented. But you then go into this phase of augmenting this singular idea with new architectures, with new technologies that make it better. And then maybe some new innovation comes along and wipes the board again.

And I've seen this several times in my career. So I started in natural language processing, and in that area of AI, it was about, sort of in the '80s, where people decided to switch from just, as linguists, writing down declarative statements of how language worked, to doing things in a corpus-based, data-based way, which of course we still do today. And that was a big gain. The move to statistical models, a big gain. In classification, support vector machines showed up and outperformed everything. All the naive Bayes models went away and everything was support vector machines. Big gain. Neural models, big gain. Transformers, large scale, these big gains. And each of these things has jumped things forward but saturated eventually.

And so what you saw was, you had these large language models, these transformer-based systems, and you just fed them more and more of the worldwide web, and they just got better and better. And everything everybody else did, it was like, oh, let's throw all that stuff out. We don't need any of those things anymore. We're just going to scale this thing up and it's going to do everything. And at the beginning it looked like it was working really well because things were advancing, and it's a powerful idea, but the gaps do begin to show.

And so for example, what you've seen is, first of all, you run out of data. Why is an LLM so smart? How did it get so smart? How long did the training take? The training took a month or something like that. Well, what did it train on? It trained on the worldwide web, which is essentially, at least at its best, a collection of human knowledge written in the best abstraction we have for knowledge, which is language, that it could consume. Where did the web get there? Well, thirty years of us building web pages.

Liam Lawson: Right. So that's actually thirty years of web pages that we're suddenly seeing the gains of, not just three months of training.

Dan Klein: But wait, where did the thirty years of web pages come from? That's a millennia of knowledge. And so knowledge has been building up, and then we made it digital, and then all that stuff has been compressed into a model, and suddenly the model seems really smart. But that's like a one-time thing. There's no hundred other worldwide webs with more advanced knowledge than humanity has just waiting to be trained on. So you hit a wall.

So what do you do? Well, now you start to see other ideas like, oh, maybe we should do reasoning. Maybe we should do different kinds of reinforcement learning.

I could point out that in humans there's actually two ways we make good decisions. One way we make good decisions is by memory. I touched the stove once and I'm not doing that again. You remember what happened and you double down on the things that work and avoid the things that didn't. Language is really powerful because language lets you learn from other people's mistakes instead of having to make them all yourself. But that's essentially making decisions from memory.

Another way you can make good decisions is by prediction. You can think, okay, well, if I do this, then I think this is going to happen, and that's going to happen, and I don't really want that, maybe I shouldn't do this. You can play the game forward like a chess engine, roll it forward, and then make a decision not because of what's happened before but because of what you predict will happen in the future.

And AI in the '80s to 2010 was very focused on prediction. It was very focused on models that would search alternate hypotheses, try things until they worked. And the idea was you don't know what's going to work until you try a bunch of things, and then you keep the ones that were successful in simulation. That's how you find a good chess move in a kind of basic program. You look and see which of these moves doesn't result in losing soon. And humans balance these two things.

LLMs come along, and this is a technology that's really good at just, in a single shot, generating out an answer. And so people stopped doing search. They stopped considering alternative hypotheses, and they just said, oh, this model's so good, it's going to crank through. And it replaced all of the techniques we had. But as soon as it hits a wall, suddenly people say, well, okay, for math problems, maybe we should just try a bunch of things and see which ones result in correct proofs.

Well, this is one kind of reasoning model. This is a way to take test time compute to make your model smarter. And it is basically search. And so you ask, what are the big ideas? What are the last big ideas? We keep bringing back these successful core components of intelligence in new guises, because the technology has changed. Reasoning can be a really good idea, but it's actually an idea we threw out when LLMs burst onto the scene.

So I think it's more complicated than that. Reasoning isn't all ideal. Test time compute is very expensive. It burns tokens. So if you can try ten things and pick one that worked, one cost to that is you have to spend ten times as many tokens.

1:10:55 — Why frontier labs keep scaling despite the risk of disruption

Liam Lawson: Just following that thread. So we have all these frontier labs spending billions, if not trillions in the near future, on expanding their data centers, getting more on the energy, spending more on GPUs. But then, what, a year and a half ago we have DeepSeek come out with their model that is a fraction of the cost because they'd invented new technology that made it much faster, more efficient. And advances like that could come out all the time. We just don't know what we do know yet. But it is possible, for sure. What I'm confused at is, if we know this is possible, and we know that these technological leaps are possible, why are these frontier labs spending so much money on scaling right now, if that could be wiped out by some new technology just coming into existence or being applied?

Dan Klein: I mean, this is sort of the perennial story of what is the difference between a disruptive technology, which tends to be the domain of startups, and a scaled technology. And what you see is that a large, highly capitalized company is always going to be particularly well suited to doubling down on things that are already working. Scale has worked, so they keep scaling.

This is why I think traditionally you do see a lot of disruption from startups, because startups are not suited to doubling down on things that are already working at the largest scales. They're suited towards coming up with new ideas that are disruptive. And I do think there is going to be a lot of disruption in terms of systems that have different properties. We're focused on systems that have very different reliability profiles, systems that are more efficient.

So if you remember, a few years back, because there had been success from scaling up, and there is a big difference between training on one percent of the web and ten percent of the web and a hundred. There's a big difference between having not enough parameters to enough parameters, because as you grow you need enough parameters to capture your data, because that's essentially where the data is getting stored, and there's a limit to how much you can abstract that stuff.

But because scaling was so successful, there was a phase where it was almost like bigger was better in and of itself. That if you had two models and one was bigger, you're like, oh, well, it must be better. Well, it doesn't have to be better. It could be bigger and also worse. All else equal, having more compute, having more data, having more parameters, these things are very helpful. But it's important to remember that scale is not a goal in and of itself. It is a way to push forward under a fixed architecture.

And first of all, I think everybody's acknowledged we're seeing diminishing returns. Which boils down, really, in a nutshell, to there's no more millennia of data to train on. You've got to start doing other things now. So that's something that people thought was going to be an exponential curve, but it was an S-curve and it leveled off.

So I think that's one issue with scale, and this is confounded by, scale is actually important. Intelligence, a lot of pieces of intelligence don't work at the smallest scales. You have to have enough scale. And so that's all made it complicated.

But you definitely see people talking about now, okay, is there a way to do this without all the test time compute? A reasoning model that, for a short response, generated thousands upon thousands of tokens talking to itself before it gives that response. Well, it's clear that if you're paying by token, if you're selling tokens, this is a great idea. You can sell so many more tokens now. But it increases latency, it increases cost, and if you can get the same behavior in a smaller model, in a faster model, people are going to prefer that.

And what you're seeing today that you didn't before was people starting to say, okay, what is a model that can do what I need fast? What is a model that can do what I need on technology that we can provision? Because there's a finite amount of power, there's a finite amount of GPUs. And so I think people have gotten away from bigger is better even if it costs more, is no better, and takes ten times as long. You're like, actually none of that is good. All that matters is the quality of the output, and then everything else, faster is better, cheaper is better, and more deployable is better.

So I do think you're going to see more specialist models. I do think you're going to see more models that have focused their strengths where they need to be. As opposed to, a classic example of this is, I don't know if you saw the Chipotle example where it's an ordering bot and somebody's like, how do you reverse a linked list in Python or something like that. And of course the model can answer, but it shouldn't. And why would we take a model that can do all of these things and then work really hard to make sure it doesn't do those things?

There are some things that are off here. And ultimately I think it's about reliable intelligence. What do you need the system to do? What do you need the system to never do? And what is going to have the right performance profiles? You don't want users waiting six seconds for an answer. You don't want the system going off on weird tangents. And you don't want to be constantly wrestling a system that doesn't have any good control levers.

1:17:15 — The future of hyper-specialized models vs. one broad model

Liam Lawson: My mind goes to routing when you talk about this. Instead of having one large breadth of a model that does everything, I think a lot about agent harnesses, for example. We're seeing a lot of people use different ones just now, and they're sending all these hyper-specific agents or models to do very specific tasks and then bringing that back to one space. Is that kind of what you imagine for the future? We have hyper-specific models that do hyper-specific tasks, that get that right, and then we have some sort of harness that orchestrates all this.

Dan Klein: Yeah, I think there's two extremes, which neither extreme works. So if you go to the extreme where the model is completely broad, and even if you had a model which was so big and so broad and so superintelligent in every way that the reliability was finally catching up to people, this would be an incredibly wasteful way to do a task where reliability is the most important thing. You want to have architectures where the reliability is not lagging behind.

So on one side you can have models which are sort of large and uncontrollable. That's not good. On the other side you can have models that are again shrunken down so small, to fit in making control decisions in a harness. They're now so hyper-specific that they're losing the magic.

I think the magic that comes from interacting with a sufficiently broad intelligence is that, yes, you're asking about your account balance, or you're changing a flight or something like that, but you're doing it in a context of world knowledge, and you have your reasons, and you want the system to be able to realize that, oh, you're asking about this, but do you really mean this? And there has to be, you can't be stuck in a flowchart, but you also can't be in a system that's just totally uncontrollable and winging it. You want systems in the middle which are able to do a really good, intelligent, and reliable job of what you need them to do.

And so I think you're going to see models which are not hyper-specialized to one decision in a decision tree. I think what you're going to see is models specialized to kinds of vertical conversations, things like that.

1:19:47 — Is there anything uniquely human AI can't replicate?

Liam Lawson: Do you think there's anything special about humans that we're never going to be able to replicate here? I think a big part of the conversation that you're speaking about, or two things that you speak about, is metacognition and context. So for example, when I'm making a decision, I have so much fluid context in my brain, and I'm able to metacognitively think about that and apply it to a certain situation. Is that logic based, or is that something that's uniquely human?

Dan Klein: So depending on where we go with this, we're either getting to a neuroscience question. How do humans do this? Mechanistically, in our brains, how is everything laid out and how does it operate? And there have been some great advances in neuroscience, and there's also just a ton we don't know.

The other way we could go to this is more philosophical. Is there something about humans? Are there deep reasons why we could not build a system that mimics our behaviors, including the metacognitive behaviors and so on? And philosophers argue about this.

I think the important thing to point out is, even under a functionalist view where every computation that happens in humans is ultimately a mechanistic computation and can be replicated in principle, that is different from saying, yes, of course anything we do can be done by an artificial mind, that is different from saying the technology we have today, scaled up a little bit, will do it.

So I don't think in terms of, okay, here's a thing that a machine can never do. I feel like that's just not my outlook. My outlook is more like, here are the places where artificial intelligence has lagged behind persistently, and where to catch up with humans would really require more change in architecture or technology or approach than just keeping doing what we're doing.

We already talked about one repeatedly, which is reliability. Metacognition in general, theory of mind, the fact that we are constantly tracking not just what we know, but when we're talking to somebody, what do they know? What do they understand? We're tracking people's goals. There's a social level of conversation. We're usually using conversation for some purpose. And so these systems are not there yet. They are not metacognitive. They do not have a rich theory of mind and of other people. And they can't care.

Liam Lawson: Something that's coming up in what you're just saying is, one of the unique advantages of a human is the ability to feel empathy and how that influences decisions. And if they can't care, if it is purely mechanistic, then that's got to be a critical component, though.

Dan Klein: Yeah. I think it boils down to, I think it's going a bit out on a limb to say something like an artificial system can never care. I don't see any reason it couldn't. That's a very different statement than, the systems we have today do not have authentic emotions.

Because I think one thing that's very hard about interacting with systems today, we talked a little bit about how, when humans make mistakes, one, they're sometimes aware of their lacks of knowledge. But when humans make mistakes, there's often these smells, these indications that maybe this is incorrect. And systems can act confident, they can act authoritative, they can be wrong. They can also say things that from a human would have come from a place of empathy, without having a place of empathy.

And so I think as we look at these things, it becomes very important to not say that if the system outputs X, I can conclude something from humans outputting X. If a human says this, it might be for a totally different reason.

And now we get into the mechanism. And I actually find this very interesting. As an engineer, you go and you basically build complicated things out of simple pieces. That's what engineers do. It's a synthesis activity. A scientist takes an analytic approach. A scientist takes something complicated that they don't understand and they try to understand it by breaking it apart.

And we start asking questions like, all right, well, how do humans have empathy? How do humans have consciousness or whatever? And then you go to a system and you're like, well, does it have empathy? This is interesting, because this is science. This is not, here's what empathy is, how do we build it? That would be an engineering question. This is like, here's an artifact that we actually don't really understand, because it has some very humanlike behaviors, but we know that those behaviors are arising, in many cases, for very not humanlike reasons, and trying to get to the bottom of that.

So the fact that we're doing neuroscience on these models tells us that there is complexity in there we don't understand. So it's very hard to say definitively X isn't happening. But it does seem very clear that systems can have flashes of humanlike behavior without having all of the context and mechanisms that give rise to that behavior in humans.

1:25:45 — Wearing three hats: professor, researcher, and CTO

Liam Lawson: Super interesting. I'm curious about you on an individual level. Wearing a lot of different hats. CTO, researcher, professor. That's a lot to focus on, and I guess you've got a pretty unique advantage in that you're looking at AI and LLMs through a couple different lenses. One in terms of teaching it, one in terms of researching the absolute frontier of it, and then three, building something that's commercially viable to sell people. How do you think they all fuse together into the viewpoint that you have? And does any dominate more than the other? Are you able to look at that as, sorry to be on those, but like a model reasoning. Is my brain a mixture of experts?

Dan Klein: Yeah, so you're basically asking, am I a mixture of experts in my head? And I don't know, maybe. I do think, though, that these different perspectives aren't as different as you might think.

I think the thing that ties them all together is there is a set of important concepts that the field is just now working through, about what is intelligence, how do we build intelligent systems, what are the facets of an intelligent system, what are the downstream needs out of a system for it to be effective. Effective in the enterprise, which is a very product-based kind of question. What are the requirements? Effective sociologically, the digital literacy, what does a system need to do to not have some bad interaction with how people work?

And then also this aspect of communication about these things. When we're building systems, we need to be able to tell people clearly what they are. Even just simple things like, this is a system which is not always right, and when it makes mistakes you may not be able to tell. This is something that most people don't know.

So to me these are all just lenses on the same question, which is, how do we build AI technologies which can be trusted, which can be controlled. This is not just a technological question, but it is a technological question. It's not just a commercial question, but it is a commercial question. And it's not just a social question, but it is a social question. And I think if you don't have all of these pieces in view at once, that's going to result in solutions that aren't the right ones. And this is too important a problem to be sending out half-baked, wrong solutions.

1:29:47 — Why Dan does what he does

Liam Lawson: My last question is a question that I ask every single one of my guests. I ask them the same thing. Everyone takes it in a different way. Some people talk about family, some people talk about purpose, people talk about everything. But you take it however you want it. I encourage you to take five to ten seconds to sit on it before answering. Dan, why do you do what you do?

Dan Klein: So it's interesting to answer this question as somebody working on AI, because so much has changed during the course of my career.

If you had asked me twenty years ago why am I working on AI, on language, I would have said, I love language, which I do. I love thinking about what are the core components of intelligence. And I read a lot of science fiction as a kid, and there's a range obviously of how people see the future, but I really resonated with the hopeful ones. The idea that we could have systems that could help us achieve more.

And what's interesting about AI is, at that time, and this isn't that long ago, at that time earlier in my career, the biggest problem we had in AI is that things didn't work well enough. And that was this really exciting challenge, because we could go from problem to problem. Let's build a machine translation system. They have them in Star Trek. They've got the universal translator. They have them in Hitchhiker's Guide. It seems like we should be able to build this. That was exciting. And I think it's still exciting. The way you can remove barriers between people and make all information visible to everybody. This is amazing. I love this stuff.

And I would look at all of these problems and think, I'm doing what I'm doing to build technologies that help people communicate, help people access information, help people manage information. And that just seemed like, first of all, it was a really hard problem, but really exciting if you can succeed.

I was motivated by scientific questions too. We were reconstructing ancient languages and doing all kinds of things that just, to me, felt like these fascinating problems that had resisted solution.

I do think something has changed in my answer recently, because the problems in AI before stemmed from basically the ways in which they didn't work. The problems in AI today stem, in many cases, from the ways in which they do work successfully. That you have a system which is so close to an information oracle that when it systematically lies to people, you have a problem. You have people losing the ability to know what to believe, because we're taking away, just in terms of products people ship and cultural uptake of things, we're taking things away from people that helped them understand that information.

We're encouraging delegation of work to a system. If you say, okay, I used to write this content, but I'm going to have an LLM do it. It's easy to have this attitude that you don't have to be a writer anymore. But the thing is, now you have to be an editor. The thing comes back and now you have to edit it. And editing is hard, and it's a different skill. And this is a case where you can't just build the technology. You have to give people time and scaffolding and product affordances so that they can use this to make themselves better and not just to generate a bunch of slop.

And so when I look around today, I see a lot of uncertainty in how AI will roll out into culture, into society. And what I do today is because I still see all the great things that AI can do that got me started on this journey. But I now also see aspects of this technology that really were not yet in view and now are coming very fast.

And so what I do today, and we've heard this over and over again in the pod today, what I do today, the reason I do it, is really to make sure that as AI systems get deployed, that they be systems that you can control, they be systems that you can trust, that they be systems that you can reason about their behavior. Because I think all of the exciting potential of AI systems requires that. And without that, I don't want to end up in a situation where we have corrosive technologies that no one can trust, that are really just sending us down a path that is not the one we wanted to be on.

Liam Lawson: It's a beautiful answer. No comments, no questions, no queries. Dan, I want to thank you for your time. I think this is the longest podcast I've recorded in a while. So first I want to appreciate you taking the time. If people want to learn more about who you are, what you do, how do they go about getting in contact?

Dan Klein: As you said, lots of hats. You can find out more about each of them in another place. All of my research on the academic side at UC Berkeley, that's all on the web. I encourage people to learn a little bit about Scaled Cognition, and that's also on the web.

And I really appreciate you having me on the show here. It was a lot of fun. Technical questions can be easy, but these sort of deeper questions about why we do things are so important. And I just want to say thank you for focusing on those.

Liam Lawson: I appreciate it, Dan. Thank you so much.

Dan Klein: My pleasure.

‍