One way to pump the brakes on the AI hype cycle may be to begin ignoring pundits who use the term 'AGI'. There's no agreement as to what AGI is, so evaluations of progress towards AGI or assessments of whether it is near or far away are, as far as I can tell, largely nonsense. What counts are the capabilities of actual (and fallible) AI systems vs those of actual (and fallible) humans. On that front, there is much that it interesting and important to report.
Just as a note, exactly a year ago, Blili-Hamelin and 15 other "AI researchers", put out a paper called "stop treating AGI as the north-star goal of AI research. I don't necessarily endorse all the arguments in the paper, and I can't vouch for the qualifications of the authors (I try to keep in mind that even anti-AGI writers are benefitting from the hype around it), but I couldn't agree more with the sentiments in the title. ( Link: https://arxiv.org/pdf/2502.03689 )
Tom, just one link to AGI in there, and it's interesting not because of claims about AGI, but about the limits of LLMs and how those point to them not being close to anything close to "conscious".
I should have been clearer -- we need to ignore the "AGI is coming (or not)" hype, which means we need to ignore the "limits to LLMs" hype -- because that's what it is. Gary Marcus may not be in the same league as Sam Altman, but he's a shill nonetheless: a "cognitive scientist" with no time for science.
We don't (and almost always can't) know the limits of either scientific exploration or technological achievement. Just as seriously, there is no generally-accepted account of "consciousness" (any more than there is of intelligence) against which to measure LLM progress or accomplishment. What we get from the skeptics is ta kind of mystical holism: the cherry-picking of some supposedly-unassailable feature of human mentality and a concomitant refusal to decompose that feature into the operational components (such as predicting the next word) at which AI development takes place.
I can't predict that AI scientists will isolate and develop enough such components or combine them in such a way as to replicate human intelligence from the ground up. I also can't predict that they won't and I definitely can't prove that it's impossible. What I say with confidence is that AI is important and will be transformational. That's where the focus needs to be.
I think the debate matters, in no small part because these charlatans from the frontier labs keep bringing it up.
I think AI will be immensely consequential, but if it's not close to becoming truly autonomous — that is to say unable to properly reason — we're looking at a tool, not a god.
Yes, programmers are being replaced as we speak by a quantum leap innovation that is able to do their jobs more efficiently by orders of magnitude. And that's coming to many other fields as well. The steady middle class white collar job is going the way of the factory job. The politics of this will be nasty enough.
Why are these otherwise smart-seeming people doing it? Because they need to justify their spend, feeding the beast. They've spun up a mad race — against each other, against China, against time — at the end of which they're promising their investors something like a monopoly on all labor, and endless wealth. To talk about these technologies as mere tools that will transform our lives — often for the better! — is against their interest. So they spin up dreams of building superintelligence to keep the money coming, sending observers into tizzies every single time.
You're right, it's impossible to rule out where all this will end up. But it's equally impossible to rule out far more humble outcomes. Why should we opt for the maximalism? Especially if it helps these reckless people keep inflating a catastrophic bubble if it turns out they're wrong?
There no disagreement here. Containing the hype supporting the "mad race" is what I took your original post to be about. To the degree that "AGI" is an industry-supported theme, essential to prolonging an ultimately unrealistic flow of investment, then I have to agree that "the debate matters". The problem is that if the questions debated aren't well-posed, the debate just feeds the hype. The remedy is to focus, as you do in much of your response, on the actual systems, their capabilities, their directions, their evident (and, yes, their likely) consequences (social, cultural, financial), and the available options for policy. That's the sober version of my tirade.
One way to pump the brakes on the AI hype cycle may be to begin ignoring pundits who use the term 'AGI'. There's no agreement as to what AGI is, so evaluations of progress towards AGI or assessments of whether it is near or far away are, as far as I can tell, largely nonsense. What counts are the capabilities of actual (and fallible) AI systems vs those of actual (and fallible) humans. On that front, there is much that it interesting and important to report.
Just as a note, exactly a year ago, Blili-Hamelin and 15 other "AI researchers", put out a paper called "stop treating AGI as the north-star goal of AI research. I don't necessarily endorse all the arguments in the paper, and I can't vouch for the qualifications of the authors (I try to keep in mind that even anti-AGI writers are benefitting from the hype around it), but I couldn't agree more with the sentiments in the title. ( Link: https://arxiv.org/pdf/2502.03689 )
Tom, just one link to AGI in there, and it's interesting not because of claims about AGI, but about the limits of LLMs and how those point to them not being close to anything close to "conscious".
I should have been clearer -- we need to ignore the "AGI is coming (or not)" hype, which means we need to ignore the "limits to LLMs" hype -- because that's what it is. Gary Marcus may not be in the same league as Sam Altman, but he's a shill nonetheless: a "cognitive scientist" with no time for science.
We don't (and almost always can't) know the limits of either scientific exploration or technological achievement. Just as seriously, there is no generally-accepted account of "consciousness" (any more than there is of intelligence) against which to measure LLM progress or accomplishment. What we get from the skeptics is ta kind of mystical holism: the cherry-picking of some supposedly-unassailable feature of human mentality and a concomitant refusal to decompose that feature into the operational components (such as predicting the next word) at which AI development takes place.
I can't predict that AI scientists will isolate and develop enough such components or combine them in such a way as to replicate human intelligence from the ground up. I also can't predict that they won't and I definitely can't prove that it's impossible. What I say with confidence is that AI is important and will be transformational. That's where the focus needs to be.
I think the debate matters, in no small part because these charlatans from the frontier labs keep bringing it up.
I think AI will be immensely consequential, but if it's not close to becoming truly autonomous — that is to say unable to properly reason — we're looking at a tool, not a god.
Yes, programmers are being replaced as we speak by a quantum leap innovation that is able to do their jobs more efficiently by orders of magnitude. And that's coming to many other fields as well. The steady middle class white collar job is going the way of the factory job. The politics of this will be nasty enough.
Why are these otherwise smart-seeming people doing it? Because they need to justify their spend, feeding the beast. They've spun up a mad race — against each other, against China, against time — at the end of which they're promising their investors something like a monopoly on all labor, and endless wealth. To talk about these technologies as mere tools that will transform our lives — often for the better! — is against their interest. So they spin up dreams of building superintelligence to keep the money coming, sending observers into tizzies every single time.
You're right, it's impossible to rule out where all this will end up. But it's equally impossible to rule out far more humble outcomes. Why should we opt for the maximalism? Especially if it helps these reckless people keep inflating a catastrophic bubble if it turns out they're wrong?
There no disagreement here. Containing the hype supporting the "mad race" is what I took your original post to be about. To the degree that "AGI" is an industry-supported theme, essential to prolonging an ultimately unrealistic flow of investment, then I have to agree that "the debate matters". The problem is that if the questions debated aren't well-posed, the debate just feeds the hype. The remedy is to focus, as you do in much of your response, on the actual systems, their capabilities, their directions, their evident (and, yes, their likely) consequences (social, cultural, financial), and the available options for policy. That's the sober version of my tirade.
Your link to Jason Willick is broken. The other links were very interesting, thank you for some food for thought.
Fixed!