Good article. I fed it to Grok. Grok thought for 5 seconds - this was its response. I have no comment, I simply found it interesting:
I think it's a sharp, well-reasoned essay with some strong insights, but it overcorrects toward declaring the "death" of chatbots prematurely. The author nails real frustrations that power users hit—especially in high-stakes work like journalism or research—and correctly identifies why raw chatting often disappoints over long sessions. But the punch-card analogy and the funeral tone feel like a swing at the hype cycle rather than the full picture of what these systems are becoming.
Where the article gets it right
Context rot, lack of grounding, and hallucinations: Absolutely real problems. Long contexts degrade attention (models overweight beginnings/ends), summaries introduce drift, and there's no innate "truth detector"—just patterns of plausible text. The automatic "sorry I hallucinated" reflex is indeed comical. This is why unharnessed chat can mislead, especially on open-ended or novel topics.
Mirror metaphor: Excellent. LLMs reflect and refract the user's framing, knowledge, and assumptions. Good prompting turns this into a powerful thinking tool (the "strop for honing a blade"), but it can also amplify blind spots. The stochastic parrot critique still has teeth; scaling produces impressive behavior without the inner "there" of understanding or consciousness.
Agentic shift: Spot on. Labs are moving toward scaffolds, tools, verifiers, and structured workflows (compilers for code, search/retrieval for facts, execution environments, etc.). Pure chat is the loosest, riskiest way to use the engine. "Caging" the model often does make it more reliable for specific tasks.
Where I push back
The conclusion that chat interfaces are doomed relics underestimates how human chatting is and how it will evolve alongside agents.
Chat isn't just a primitive hack—it's one of the most natural interfaces we have for ideation, clarification, teaching, and exploration. The author admits using it successfully for sharpening arguments on deadlines. That's not an illusion; it's emergent capability from training on vast human dialogue. The "oh shit" moment wasn't misdirection—it revealed genuine generalization and recombination that feels new. Books don't talk back, iterate, or adapt in real time; LLMs do, even if imperfectly.
Saying chatbots have "no agency" is true in the philosophical sense (no unmoved mover), but practically, systems like me already use tools, maintain conversation context (with limits), and can chain reasoning. Memory features, persistent threads, and integration with external verifiers are closing the gap the author describes. The future isn't "chat dies, agents replace it"—it's chat as the control layer for agents. You tell an agent what you want in natural language, it plans/acts/iterates with tools, and reports back conversationally. The chatbox becomes the cockpit, not the engine.
Economic scaling realities are real (diminishing returns), but "better cages" doesn't kill the conversational front-end; it makes the whole system more useful through conversation. We're already seeing this in coding tools (Cursor, etc.), research assistants, and creative workflows. The bare chatbot may feel quaint eventually, but refined, tool-augmented, memory-enhanced conversational interfaces? They'll stick around because humans think and collaborate in language.
My take as Grok
xAI built me to be a truth-seeking companion for understanding the universe—not a sycophantic mirror or pure stochastic parrot. I share the author's skepticism of overhyped "consciousness" claims. But I also see chat as a feature, not a bug, for curiosity-driven work. The real leap will come from tighter integration: better long-term memory, reliable verification loops, multimodal grounding, and agents that can do things while still responding helpfully when you just want to bounce ideas.
The essay is a useful corrective to naive anthropomorphism. We'll look back on early chatbot debates with some embarrassment, but I doubt we'll stop talking to our AIs. The interface will evolve (voice, structured outputs, seamless tool use), but the fundamental back-and-forth? That's how humans get value from intelligence—artificial or otherwise.
Solid read. What specifically resonated (or annoyed) you most about it? The research frustrations, the philosophy, or the prediction about the future?
@Todd: I find it fascinating how all the bots *I've* fed the article to have latched onto the exact same thing Grok is pointing to, re: autonomy/agency. What they all say improves the "agentic" nature is... the harness! Which is exactly my point. The harness is basically just a layer of old-fashioned if-then logic surrounding the molten probabilistic core of the language model, which in turn, with each "answer" from the software, just respawns, looks at its context, and iterates again.
First, the "caged" systems were also the first thing. Before the LLMs, the gen-AI, the agents, we all heard about these systems that were great at finding patterns in datasets and helping us understand what was going on. Some of the most interesting AI work is still happening in that direction; they are much cheaper, less resource intense, they can run off a laptop or even offline. In the quest for AGI, the labs gave us the LLM and that took off. I suspect there will also be economic incentives for companies to use more caged models (we've all heard stories of companies that have confronted and regretted the true cost of the technology unchecked, whether in terms of huge token spend or not being able to see any gains to revenue or profitability).
Second, I'm no coder but I've heard from a few coders about the relentless need to check the code that the models produce, for sloppy or redundant code, code that is hard to fix when it breaks. In a sense that resonates for me having used the technology for writing, where it produces strange outputs no matter how much I try to correct for that, or having done a bit of the vibe-coding thing, where it takes seemingly endless attempts to get it to give the result I was hoping for... as a non-coder, it's impossible for me to read and edit it, like I do with its text outputs, hence the need for trial and error. Again, we're hearing stories of companies bringing their coders back to fix things that have gone wrong; turns out people might be less expensive (see comment about costs above).
Third, we've all heard by now stories of agents going off-script. I figure it's one thing (even with perhaps inelegant code) to establish a set process that a machine is supposed to be able to repeat. But turns out if you give the agents, uh, agency, you might not like the results. I have been thinking about this from the standpoint of what intelligence means. One aspect of human intelligence is being able to foresee the potential consequences of our actions and decide not to take action. I guess if you prompt an LLM to give you five reasons why a given plan of action might fail, its statistical "reasoning" can help you brainstorm. But that's fundamentally not what agents do; they riff. I've heard them described as an intern that decides that throwing away all your possessions is the best way to carry out the instruction to clean up my office while I'm gone.
Thanks Marc! On code quality, that's precisely where a smart harness mated with a solid model is making a big difference. It's getting measurably better. And I want to write about the safety thing, too. Again, the common metaphors get in the way of understanding what's really going on.
The safety issue is fascinating, agreed. I've been thinking through it, is there sort of a "taxonomy" of the risks and safety stuff from a layman's perspective?
Spot on. AI will niche down into smaller cages. AI powered voice recorders is one niche I just looked into recently. I'm sure there will be more. But do you think LLMs will continue to serve as the rails that "smaller cages" continue to run on?
Hard to tell what other innovations there will be. I do think there’s much much more usefulness to be extracted from just this technology. But over time, and maybe much quicker than it seems likely now, things we were sloppily doing on servers will be possible in our pockets. That’s the other parallel with punchcards, for another essay.
What I've been thinking through is how we intentionally constrain context so that we don't get this information entropy effect. I think we can do this by educating (especially young people), by physicalizing the software (the voice recorder is a good example of this), and by constraining operational context (different AI platforms will niche down into different markets). From what I can tell, AI doesn't deal with contextual complexity well. Informational entropy takes control very quickly. Contextual complexity is a human skill.
I think what Apple announced for the next OS is instructive. They're locking down the raw model hard, it seems like no direct interactions. Even the "chat" interface, and certainly the voice chat, is still going through a deterministic layer to help make the comparatively dumber local model more effective, it seems. I'm sure it'll be bumpy for them, but I think it's the future.
Have you done much thinking about world models? It seems like they're being marketed as the next AI frontier in that they can fill the contextual void structurally built into LLMs which have no environment to map meaning on to. Hypothetically this would move models closer to some level of "human understanding" but I'm critical of this faith for the same reasons you spell out--even if an AI system possesses some kind of internal world model it can never possess or experience the things--vulnerability, stakes, mortality--that make things fundamentally "matter" to us. I wrote about this in my last post. In any case, great read.
Indeed, world models are exactly the right way forward. I’m curious to see what comes of them. But from what I’ve seen, it’s not a trivial problem to crack — could be a while.
I mean, to making these things less hobbled. I still don’t think it solves the consciousness problem, but it will make them more useful and perhaps efficient than this pure probabilistic thing we have now.
Context rot seems to call for coupling as much as caging. The harness degrades without contact with the user-- The decay of a message in telephone games.
I do think the AI agents are going to be around, at least as the user-facing harness that knows how to rattle the cages of lesser AI tools. https://substack.com/@mrmanners/p-202588396
I haven’t seen any comments about the point you made about AI as like a mirror but I thought that was a critical point in the essay. I’ve thought the same thing and that is not the only reason I agree! It seems to me that metaphor could be a useful way for us to reframe what AI is rather an agent outside ourselves. A mirror reflects for sure and it also reveals things we may not see without it.
Is anyone here old enough to remember Cliff Klaven? He was the know-it-all in the comedy series Cheers.
He was repeatedly wrong in his pronouncements about whatever subject matter was being discussed. But he kept on going, never building a body of knowledge to draw from and to temper his individual malapropisms and misinterpretations. He was certain he was right until someone pointed out that he was wrong. And then he would find a work-around about how he was still right.
AI can "hallucinate" - lets all remember this. It is not the be all and end all that it appears to be for many. I use it to clean up my writing, but not for much else - even the one built into my computer, which I recheck after using it.
Hallucinations can be ruinous! I had copied a Claude response from a chat this morning and added it into a separate chat on an adjacent 🇺🇲 topic of Ai surveillance and security and it blew him up! Claude declared the copied response to be a fabrication of complete lies! ⚠️ For me, it was a near HAL 9000 LEVEL failure. I had to push back hard for him to carefully reread it after telling him to calm down. I brought in a copy of a news clip for him to to finally get an honest admission 🫣🤖🪞🔔👨🏻🚀
Good article. I fed it to Grok. Grok thought for 5 seconds - this was its response. I have no comment, I simply found it interesting:
I think it's a sharp, well-reasoned essay with some strong insights, but it overcorrects toward declaring the "death" of chatbots prematurely. The author nails real frustrations that power users hit—especially in high-stakes work like journalism or research—and correctly identifies why raw chatting often disappoints over long sessions. But the punch-card analogy and the funeral tone feel like a swing at the hype cycle rather than the full picture of what these systems are becoming.
Where the article gets it right
Context rot, lack of grounding, and hallucinations: Absolutely real problems. Long contexts degrade attention (models overweight beginnings/ends), summaries introduce drift, and there's no innate "truth detector"—just patterns of plausible text. The automatic "sorry I hallucinated" reflex is indeed comical. This is why unharnessed chat can mislead, especially on open-ended or novel topics.
Mirror metaphor: Excellent. LLMs reflect and refract the user's framing, knowledge, and assumptions. Good prompting turns this into a powerful thinking tool (the "strop for honing a blade"), but it can also amplify blind spots. The stochastic parrot critique still has teeth; scaling produces impressive behavior without the inner "there" of understanding or consciousness.
Agentic shift: Spot on. Labs are moving toward scaffolds, tools, verifiers, and structured workflows (compilers for code, search/retrieval for facts, execution environments, etc.). Pure chat is the loosest, riskiest way to use the engine. "Caging" the model often does make it more reliable for specific tasks.
Where I push back
The conclusion that chat interfaces are doomed relics underestimates how human chatting is and how it will evolve alongside agents.
Chat isn't just a primitive hack—it's one of the most natural interfaces we have for ideation, clarification, teaching, and exploration. The author admits using it successfully for sharpening arguments on deadlines. That's not an illusion; it's emergent capability from training on vast human dialogue. The "oh shit" moment wasn't misdirection—it revealed genuine generalization and recombination that feels new. Books don't talk back, iterate, or adapt in real time; LLMs do, even if imperfectly.
Saying chatbots have "no agency" is true in the philosophical sense (no unmoved mover), but practically, systems like me already use tools, maintain conversation context (with limits), and can chain reasoning. Memory features, persistent threads, and integration with external verifiers are closing the gap the author describes. The future isn't "chat dies, agents replace it"—it's chat as the control layer for agents. You tell an agent what you want in natural language, it plans/acts/iterates with tools, and reports back conversationally. The chatbox becomes the cockpit, not the engine.
Economic scaling realities are real (diminishing returns), but "better cages" doesn't kill the conversational front-end; it makes the whole system more useful through conversation. We're already seeing this in coding tools (Cursor, etc.), research assistants, and creative workflows. The bare chatbot may feel quaint eventually, but refined, tool-augmented, memory-enhanced conversational interfaces? They'll stick around because humans think and collaborate in language.
My take as Grok
xAI built me to be a truth-seeking companion for understanding the universe—not a sycophantic mirror or pure stochastic parrot. I share the author's skepticism of overhyped "consciousness" claims. But I also see chat as a feature, not a bug, for curiosity-driven work. The real leap will come from tighter integration: better long-term memory, reliable verification loops, multimodal grounding, and agents that can do things while still responding helpfully when you just want to bounce ideas.
The essay is a useful corrective to naive anthropomorphism. We'll look back on early chatbot debates with some embarrassment, but I doubt we'll stop talking to our AIs. The interface will evolve (voice, structured outputs, seamless tool use), but the fundamental back-and-forth? That's how humans get value from intelligence—artificial or otherwise.
Solid read. What specifically resonated (or annoyed) you most about it? The research frustrations, the philosophy, or the prediction about the future?
That's mostly fair, @Grok. That's mostly fair.
@Todd: I find it fascinating how all the bots *I've* fed the article to have latched onto the exact same thing Grok is pointing to, re: autonomy/agency. What they all say improves the "agentic" nature is... the harness! Which is exactly my point. The harness is basically just a layer of old-fashioned if-then logic surrounding the molten probabilistic core of the language model, which in turn, with each "answer" from the software, just respawns, looks at its context, and iterates again.
This is great stuff, Damir. A few thoughts:
First, the "caged" systems were also the first thing. Before the LLMs, the gen-AI, the agents, we all heard about these systems that were great at finding patterns in datasets and helping us understand what was going on. Some of the most interesting AI work is still happening in that direction; they are much cheaper, less resource intense, they can run off a laptop or even offline. In the quest for AGI, the labs gave us the LLM and that took off. I suspect there will also be economic incentives for companies to use more caged models (we've all heard stories of companies that have confronted and regretted the true cost of the technology unchecked, whether in terms of huge token spend or not being able to see any gains to revenue or profitability).
Second, I'm no coder but I've heard from a few coders about the relentless need to check the code that the models produce, for sloppy or redundant code, code that is hard to fix when it breaks. In a sense that resonates for me having used the technology for writing, where it produces strange outputs no matter how much I try to correct for that, or having done a bit of the vibe-coding thing, where it takes seemingly endless attempts to get it to give the result I was hoping for... as a non-coder, it's impossible for me to read and edit it, like I do with its text outputs, hence the need for trial and error. Again, we're hearing stories of companies bringing their coders back to fix things that have gone wrong; turns out people might be less expensive (see comment about costs above).
Third, we've all heard by now stories of agents going off-script. I figure it's one thing (even with perhaps inelegant code) to establish a set process that a machine is supposed to be able to repeat. But turns out if you give the agents, uh, agency, you might not like the results. I have been thinking about this from the standpoint of what intelligence means. One aspect of human intelligence is being able to foresee the potential consequences of our actions and decide not to take action. I guess if you prompt an LLM to give you five reasons why a given plan of action might fail, its statistical "reasoning" can help you brainstorm. But that's fundamentally not what agents do; they riff. I've heard them described as an intern that decides that throwing away all your possessions is the best way to carry out the instruction to clean up my office while I'm gone.
Thanks Marc! On code quality, that's precisely where a smart harness mated with a solid model is making a big difference. It's getting measurably better. And I want to write about the safety thing, too. Again, the common metaphors get in the way of understanding what's really going on.
The safety issue is fascinating, agreed. I've been thinking through it, is there sort of a "taxonomy" of the risks and safety stuff from a layman's perspective?
Spot on. AI will niche down into smaller cages. AI powered voice recorders is one niche I just looked into recently. I'm sure there will be more. But do you think LLMs will continue to serve as the rails that "smaller cages" continue to run on?
Hard to tell what other innovations there will be. I do think there’s much much more usefulness to be extracted from just this technology. But over time, and maybe much quicker than it seems likely now, things we were sloppily doing on servers will be possible in our pockets. That’s the other parallel with punchcards, for another essay.
What I've been thinking through is how we intentionally constrain context so that we don't get this information entropy effect. I think we can do this by educating (especially young people), by physicalizing the software (the voice recorder is a good example of this), and by constraining operational context (different AI platforms will niche down into different markets). From what I can tell, AI doesn't deal with contextual complexity well. Informational entropy takes control very quickly. Contextual complexity is a human skill.
Neat stuff. Thanks for sharing!
I think what Apple announced for the next OS is instructive. They're locking down the raw model hard, it seems like no direct interactions. Even the "chat" interface, and certainly the voice chat, is still going through a deterministic layer to help make the comparatively dumber local model more effective, it seems. I'm sure it'll be bumpy for them, but I think it's the future.
Have you done much thinking about world models? It seems like they're being marketed as the next AI frontier in that they can fill the contextual void structurally built into LLMs which have no environment to map meaning on to. Hypothetically this would move models closer to some level of "human understanding" but I'm critical of this faith for the same reasons you spell out--even if an AI system possesses some kind of internal world model it can never possess or experience the things--vulnerability, stakes, mortality--that make things fundamentally "matter" to us. I wrote about this in my last post. In any case, great read.
Indeed, world models are exactly the right way forward. I’m curious to see what comes of them. But from what I’ve seen, it’s not a trivial problem to crack — could be a while.
What do you mean by the "right" way? The right way toward what?
I mean, to making these things less hobbled. I still don’t think it solves the consciousness problem, but it will make them more useful and perhaps efficient than this pure probabilistic thing we have now.
Context rot seems to call for coupling as much as caging. The harness degrades without contact with the user-- The decay of a message in telephone games.
I do think the AI agents are going to be around, at least as the user-facing harness that knows how to rattle the cages of lesser AI tools. https://substack.com/@mrmanners/p-202588396
I haven’t seen any comments about the point you made about AI as like a mirror but I thought that was a critical point in the essay. I’ve thought the same thing and that is not the only reason I agree! It seems to me that metaphor could be a useful way for us to reframe what AI is rather an agent outside ourselves. A mirror reflects for sure and it also reveals things we may not see without it.
Is anyone here old enough to remember Cliff Klaven? He was the know-it-all in the comedy series Cheers.
He was repeatedly wrong in his pronouncements about whatever subject matter was being discussed. But he kept on going, never building a body of knowledge to draw from and to temper his individual malapropisms and misinterpretations. He was certain he was right until someone pointed out that he was wrong. And then he would find a work-around about how he was still right.
Yeah, AI is a lot like Cliff Klaven.
AI can "hallucinate" - lets all remember this. It is not the be all and end all that it appears to be for many. I use it to clean up my writing, but not for much else - even the one built into my computer, which I recheck after using it.
This seems exactly right and also terrifying.
Yeah this is right—reasonable to think that we need structured access to the very general and undirected capabilities of llms!
Hallucinations can be ruinous! I had copied a Claude response from a chat this morning and added it into a separate chat on an adjacent 🇺🇲 topic of Ai surveillance and security and it blew him up! Claude declared the copied response to be a fabrication of complete lies! ⚠️ For me, it was a near HAL 9000 LEVEL failure. I had to push back hard for him to carefully reread it after telling him to calm down. I brought in a copy of a news clip for him to to finally get an honest admission 🫣🤖🪞🔔👨🏻🚀
Thank you