Whatever you think about AGI, this is a dumb paper. So many words and references to say - what. If you can't articulate your point in a few sentences you probably don't have a point. There are all kinds of assumptions being made in the study about how AI systems work, about what people "mean" then they talk about AGI etc.
The article starts out talking about white supremacy and replacing women. This isn't a proof. This is a social sciences paper dressed up with numbers. Honestly - Computer Science has given us more clues about how the human mind might work than cognitive science ever did.
I don’t think speculation about AGI is possible on a rigorous mathematical basis right now. And people who do expect AGI to happen (soon) are often happy to be convinced by much poorer types of argument and evidence than presented in this paper (e.g. handwaving arguments about model size or just the fact that ChatGPT can do some impressive things).
I thought you were exaggerating, but wow, they really did.
> Among the more troublesome meanings of ‘AI’, perhaps, is as the ideology that it is desirable to replace humans (or, specifically women) by artificial systems (Erscoi et al., 2023) and, generally, ‘AI’ as a way to advance capitalist, kyriarchal, authoritarian and/or white supremacist goals (Birhane & Guest, 2021; Crawford, 2021; Erscoi et al., 2023; Gebru & Torres, 2024; Kalluri, 2020; Spanton & Guest, 2022; Stark & Hutson, 2022; McQuillan, 2022). Contemporary guises of ‘AI’ as idea, system, or field are also sometimes known under the label ‘Machine Learning’ (ML), and a currently dominant view of AI advocates machine learning methods not just as a practical method for generating domain-specific artificial systems, but also as a royal road to AGI (Bubeck et al., 2023; DeepMind, 2023; OpenAI, 2023). Later in the paper, when we refer to AI-as-engineering, we specifically mean the project of trying to create an AGI system through a machine learning approach. [0]
But it did lead me to learn a new word - "Kyriarchy" (apparently being "an intersectional extension of the idea of patriarchy beyond gender")[1], so I have that going for me today.
I've honestly stopped looking up these modern terms when I come across them because lately any that I've looked up were made up to serve a political or social agenda (always the same one), and reading them always turns out to be a waste of time that has me roll my eyes.
> That’s because cognition, or the ability to observe, learn and gain new insight, is incredibly hard to replicate through AI on the scale that it occurs in the human brain.
(no other more substantial arguments were given)
I'm also very skeptical on seeing AGI soon, but LLMs do solve problems that people thought were extremely difficult to solve ten years ago.
> but LLMs do solve problems that people thought were extremely difficult to solve ten years ago
Agreed. I would have laughed you out of the room 5 years ago if you told me AI's would be writing code or carrying on coherent discussions on pretty complex topics in 2024.
As far as I'm concerned, all bets are off after the collective jaw drop that the entire software engineering industry did when we saw GPT4 released. We went from Google AI responses of "I'm sorry, I can't help with that." to ChatGPT writing pages of code that mostly works.
It turns out that the larger these models get, the more unexpected emergent capabilities they have, so I'm mostly in the camp of thinking AGI is just a matter of time and resources.
> It turns out that the larger these models get, the more unexpected emergent capabilities they have, so I'm mostly in the camp of thinking AGI is just a matter of time and resources.
AI research has a long history of people saying this. Whenever there is a new fundamental improvement, it looks like you can just keep getting better results by throwing more resources at it. However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance.
LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them. However, unlike most other AI systems we have developed, LLMs are actively polluting their data pool, so this intitial generation of LLMs are probably going to have the best data set of any that we ever develop. Of course, today's data will continue to be available, but will loose value as it ages.
It is hard for me to not think that given the current approach it will always be almost there, but not quite good enough to be more than a nice assistant or tool that has to be meticulously built and maintained by a company. Take, for example, self driving vehicles, we have been 5 years out for 25 years. That is not to say there has not been significant progress, but to go from a really powerful driver assist to full self driving has still not really come to full fruition and that is kind of a best case scenario for AI/ machine learning.
That is just one very narrow task that basically anyone can do regardless of intelligence or talent. It takes less than a year to train a distracted 16 year old to do it, but three decades to train an AI and even then you probably need to hand tune it for specific locations because it won’t know what do in unusual road layouts.
I think we will get there with driverless vehicles, but only because it is being tailored by humans to handle all of the edge cases. Self driving cars, if truly self driving with no user intervention, will of course be worth it in the end. The trillions of dollars spent will eventually add countless dollars in added efficiency and unlimited revenue for those who get there first, but how many applications can we really say that is true for? That the decades of development and training it takes to remove humans will be worth the initial investment?
I guess my contention is that the current development path seems unlikely to ever achieve AGI and so instead you will have to do heavy customization to get anything that is much more useful than what we have now.
That is kind of my point. They exist only in a few select locations. If they were truly self driving Google would be rapidly rolling them out around the World, but they are not.
Waymo's first real driverless ride on a public road was less than 10 years ago. Now they are doing 100k rides a week.
I think your original post just overestimates the 'normal' pace of change - Taking 10 years to go from the first ride to 100k rides a week is no time at all in the scheme of things.
Airplanes are an example of an insanely fast technological adoption, and they still took 30 years to go from the wright brothers to the first commercial jet. I get the feeling that if the wright brothers invented the plane in 2024 though, people would be saying 10 years after "Planes are all hype! If planes were truly transformative they would be rolled out everywhere and would be carrying passengers by now"
I am not debating the transformative nature of self-driving cars or if it will become common place. I think it is huge. I guess my point is that is just one application of machine learning / AI. To automate such a complex task with our current approach to AI, that humans do easily, one will need to spend trillions of dollars and therefore the benefit, such as with self-driving cars, needs to be significant.
I suspect this will be the case with any AI workload. Heavy customization to get to a place where it is good enough. So the financial benefit needs to be equally huge and someone needs to feel confident they can extract that value 20 years in the future.
> LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them.
I think it's plausible we'll see a breakthrough in data efficiency that helps here. Humans are an existence proof that language is learnable through much less data and, in theory, facts should only need to be seen once. LLMs in their current form seem very data inefficient.
I didn't make a prediction, let alone a timeline - I just said it's plausible. I suppose I did say "we'll see" so I'm implicitly predicting within a few decades.
> However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance.
Honestly asking - why? It's been my understanding that based on the Universal Approximation Theorem, given sufficient resources, a deep learning neural net can approximate any function to an arbitrary degree of accuracy. Is there any other theorem or even conjecture that would lead us to believe that the progress that we're seeing will slow down? Or is it just that you're claiming that we/it would run out of physical resources before reaching AGI?
As for training data, as I see it, with the wide deployment of AIs, and gradually of AI-driven robots, they'll soon be able to "takeoff" and rely primarily on the live data that they are collecting directly.
The universal approximation therorem is not as powerful as it sounds. Polynomials essentially satisfy it as well [0], the only hiccup being that the Universal Approximation Theroum is explicitly about neural networks.
The UAT is an existence proof, it says nothing about any particular method being capable of constructing such a network. In contrast, with polynomials we have several methods of constructing polynomials that are proven to converge to the desired function.
Indeed, polynomials have been widely used as universal approximators for centuries now, and are often amazingly successful. However, polynomials in this context are only good in low degrees, where they are inherently limited in how well they can approximate [1]. Beyond a certain point, increasing your degrees of freedom with polynomial approximators simply does not help and is generally counter productive, even though a higher degree polynomial is strictly more powerful than a lower degree one.
Looking at the current generative AI breakthrough, the UAT would say that today's transformer based architecture is no more powerful than a standard neurul net. However, it produces vastly superior results that could simply not be achieved by throwing more compute at the problem.
Sure, if you have an infinite dataset and infinite compute, you might have AGI. But at that point
, you have basically just replicated the Chinese room thought experiment.
[0] See the Stone–Weierstrass theorem
[1] They are also used as arbitrary precision approximators, but that is when you compute them analytically instead of interporlating them from data.
Currently we are throwing everything at LLMs and hope good things stick. At one point we might use AI to select best training data from what's available to best train the next AI.
"That is just one very narrow task that basically anyone can do regardless of intelligence or talent. It takes less than a year to train a distracted 16 year old to do it, but three decades to train an AI and even then you probably need to hand tune it for specific locations because it won’t know what do in unusual road layouts."
It takes three decades to train an AI, but of course, like everything humanity does is not linear, it is exponential. Before the Wright brothers, it was believed that powered, controlled, heavier-than-air flight was impossible. Then, in 1903, they achieved the first successful airplane flight, which lasted 12 seconds and covered 120 feet. By 1914, the first commercial flight covered approximately 21 miles and took about 23 minutes. This is just one example and I don't see why AI should be any different
On the other hand, airplanes haven’t really improved that much since the 737 was introduced in 1966 and have even gotten worse in some ways (737 Max). We actually don’t know what the AI equivalent of Wright brothers plane is and what the 737 is. Maybe Eliza, 60 years ago, was the Kitty Hawk flight and ChatGPT 4 is the 737.
I’m not sure I would say that the 737 Max has gotten worse in “some ways”, by which I assume you are referring to safety.
New aircraft models up to about the 1990s frequently had serious design defects that resulted in fatalities, including the 737 (most notably related to the rudder).
One thing that keeps me up at night is that the human genome is only 3 giga base pairs, of which only a fraction encodes the design of our brains — and quite inefficiently at that, through layers of indirections.
That’s sufficient information to produce a system that can learn to think like us! Not just learn but efficiently, with far less input data needed than any current LLM. Literally just a couple of decades of video and audio, only a small fraction of that text!
From what I can tell, the human brain achieves this through scale alone. There’s nothing else that can explain the observed learning capability. The genome is too small to encode learned weights, and we don’t clone our parents’s brains in development.
It’s possible, and a meat computer can do it. Replicating this in silicon is just a matter of time, and it might require only scale and nothing else.
We have sensors for vision, sound, taste, temperature, texture, etc that are constantly observing the world and affecting not only our current behavior but changing us in real time.
You can feed an LLM of typical size (~1 TB) all the video and audio you can find, but it won't turn into a human. I suspect but can't prove that even if you wired up a bunch of other sensors, gave it a robot body, and let it "explore" the world on its own, that would still be woefully insufficient.
A gazelle just days(!) old can control its body and four legs sufficiently well to outrun a cheetah. This is a complex motor-control loop involving all of its senses. Compare that to Tesla's autopilot training system, which uses many millions of hours of training data and still struggles to move a car... slowly. The equivalent would be a training routine that can take just a handful of days of footage and produce an AI that can win a car race.
There's something magical about neural networks when scaled up to brain sizes. From what I gather, there's little else encoded in the genome except for the high-level pattern of wiring, the equivalent to the PyTorch model configuration.
How many hours or actually years of evolution have been needed until reaching walking capability? If first life is believed to have happened 4 billion years ago, and first walking animals started 450 million years ago during the siluariab period, that’s around 3,5 billion years.
Meat computers physically change when they learn. They don't separate data from compute hardware, the two are the same thing, so they scale really well. Silicon doesn't currently have that capability.
> but LLMs do solve problems that people thought were extremely difficult to solve ten years ago.
Well for something to be G or I you need them to solve novel problems. These things have interested most of the Internet and I've yet to see a "reasoning" disentangle memorization from reasoning. Memorization doesn't mean they aren't useful (not sure why this was ever conflated since... Computers are useful...), but it's very different from G or I. And remember that these tools are trained for human preferential output. If humans prefer things to look like reasoning then that's what they optimize. [0]
Sure, maybe your cousin Throckmorton is dumb but that's besides the point.
That said, I see no reason human level cognition is impossible. We're not magic. We're machines that follow the laws of physics. ML systems may be far from capturing what goes on in these computers, but that doesn't mean magic exists.
[0] If it walks like a duck, quacks like a duck, and swims like a duck, and looks like a duck it's probably a duck. But probably doesn't mean it isn't a well made animatronic. We have those too and they'll convince many humans they are ducks. But that doesn't change what's inside. The subtly matters.
We don't really have the proper vocabulary to talk about this. Well, we do, but C.S. Peirce's writings are still fairly unknown. In short, there are two fundamentally distinct forms of reasoning.
One is corollarial reasoning. This is the kind of reasoning that follows deductions that directly follow from the premises. This of course includes subsequent deductions that can be made from those deductions. Obviously computers are very good at this sort of thing.
The other is theorematic reasoning. It deals with complexity and creativity. It involves introducing new hypotheses that are not present in the original premises or their corollaries. Computers are not so very good at this sort of thing.
When people say AGI, what they are really talking about is an AI that is capable of theorematic reasoning. The most romanticized example of that of course being the AI that is capable of designing (not aiding humans in designing, that's corollarial!) new more capable AIs.
All of the above is old hat to the AI winter era guys. But amusingly their reputations have been destroyed much the same as Peirce's was, by dissatisfied government bureaucrats.
On the other hand, we did get SQL, which is a direct lineal descendent (as in teacher to teacher) from Peirce's work, so there's that.
We don't have proper language, but certainly we've improved. Even since Peirce. You're right that many people are not well versed in the philosophical and logician discussions as to what reasoning is (and sadly this lack of literature review isn't always common in the ML community), but I'm not convinced Peirce solved it. I do like that there are many different categories of reasoning and subcategories.
> All of the above is old hat to the AI winter era guys. But amusingly their reputations have been destroyed much the same as Peirce's was, by dissatisfied government bureaucrats.
Yeah, this has been odd. Since a lot of their work has shown to be fruitful once scaled. I do think you need a combination of theory people + those more engineering oriented, but having too much of one is not a good thing. It seems like now we're overcorrecting and the community is trying to kick out the theorists. By saying things like "It's just linear algebra"[0] or "you don't need math"[1] or "they're black boxes". These are unfortunate because they encourage one to not look inside and try to remove the opaqueness. Or to dismiss those that do work on this and are bettering our understanding (sometimes even post hoc saying it was obvious).
It is quite the confusing time. But I'd like to stop all the bullshit and try to actually make AGI. That does require a competition of ideas and not everyone just boarding the hype train or have no careers....
[0] You can assume anyone that says this doesn't know linear algebra
[1] You don't need math to produce good models, but it sure does help you know why your models are wrong (and understanding the meta should make one understand my reference. If you don't, I'm not sure you're qualified for ML research. But that's not a definitive statement either).
> We don't have proper language, but certainly we've improved. Even since Peirce. You're right that many people are not well versed in the philosophical and logician discussions as to what reasoning is (and sadly this lack of literature review isn't always common in the ML community), but I'm not convinced Peirce solved it. I do like that there are many different categories of reasoning and subcategories.
I'd love to hear more about this please, if you're inclined to share.
I'm no expert, but I've been looking into the prospects and mechanisms of automated reasoning using LLMs recently and there's been a lot of work along those lines in the research literature that is pretty interesting, if not enlightening. It seems clear to me that LLMs are not yet capable of understanding simple implication much less full-blown causality. It's also not clear how limited LLMs' cognitive gains will be with so incomplete an understanding as they have of mechanisms behind the world's multitude of intents/goals, actions, and responses. The concepts of cause and effect are learned by every animal (to some degree) and long before language in humans. It forms the basis for all rational thought. Without understanding it natively, what is rationality? I foresee longstanding difficulties for LLMs evolving into truly rational beings until that comprehension is fully realized. And I see no sign of that happening, despite the promises made for o1 and other RL-based reasoners.
Yeah one of the tricky things about causality is that it's not unique. If you didn't record the history of the event then you only have a probabilistic notion of it since many different things and in different permutations could lead to the result you observed. This has led to people believing in multiple universes when it's not akin to there being multiple ways to sum numbers to ten.
I have seen far, far too many people say things along the lines of "Sure, LLMs currently don't seem to be good at [thing LLMs are, at least as of now, fundamentally incapable of], but hey, some people are pretty bad at that sometimes too!"
It demonstrates such a complete misunderstanding of the basic nature of the problem that I am left baffled that some of these people claim to actually be in the machine-learning field themselves.
How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
I do not know if AGI is possible. Honestly, I'd love to believe that it is. However, it has not remotely been demonstrated that it is possible, and as such, it follows that it cannot have been demonstrated that it is inevitable. If you want to believe that it is inevitable, then I have no quarrel with you; if you want to preach that it is inevitable, and draw specious inferences to "prove" it, then I have a big quarrel with you.
>How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place.
People just have their pre-conceptions about how they think LLMs should work and what their "very nature" should preclude and are so very confident about it.
People like that will say things like "LLM are always hallucinating. It doesn't know the difference between truth and fiction!" and feel like they've just said something profound about the "nature" of LLMs, all while being entirely wrong (no need to wait, plenty of different research to trash this particular take).
It's just very funny seeing people who were/would be gob smacked a few years ago talking about the "very nature" of LLMs. If you understood this nature so well, why didn't you all tell us about what it would be able to do years ago ?
ML is an alchemical science. The builders themselves don't understand the "very nature" of anything they're building, nevermind anyone else.
> Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place.
there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.
> there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.
fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits.
Ultimately, numbers are represented in GPT like systems in a pretty weird way. That way affects how well they can learn to do things like arithmetic and counting and the like but that way isn't a necessary way. You don't have to represent numbers like that.
they built specialized model which is after bunch of trickery still has 99% accuracy(naive model had very low accuracy) on very simple deterministic algo. I also think most of the accuracy came from memorization of training set(model didn't provide intermediate results, and started failing significantly at sligtly larger input). In my book it is fundamental inability to learn and reproduce algorithm.
They also demonstrated that transformer can't learn sorting.
Their method does not require building specialized models from scratch (you can but you don't have to) and they did not prove transformers can't learn sorting. If you think they did, then you don't understand what it means to prove something.
In my books what they built (specialized training data + specialized embeddings format) is exactly specialized model. You can disagree of course and say again that I don't understand something, but discussion will be over.
A poor result when testing one model is not proof that the architecture behind the model is incapable of getting good results. It's just that simple. The same way seeing the OG GPT-3 fail at chess was not proof LLMs can't play chess.
This
>They also demonstrated that transformer can't learn sorting.
> No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper.
they have two OOD (out of distribution) accuracies in the paper: OOD: up to 100 digits, and 100+ OOD: 100-160 digits. 100+ OOD accuracy is significantly worse: around 30%.
> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
This is a very good distillation of one side of it.
What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors. i.e. it seems quite plausible a whole load of those people failing to grasp the point you are making are doing so because their internal models of the universe are closer to those of LLMs than you might want to think.
> What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors
I think that LLMs have shown that some fraction of human knowledge is encoded in the patterns of the words, and that by a "superficial grasp" of those words, you import a fairly impressive amount of knowledge without actually knowing anything. (And yes, I'm sure there are humans that do the same.)
But going from that to actually knowing what the words mean is a large jump, and I don't think LLMs are at all the right direction to jump in to get there. They need at least to be paired with something fundamentally different.
I think the linguists already knew this tbh and that's what Chomsky's commentary on LLMs was about. Though I wouldn't say we learned "nothing". Even confirmation is valuable in science
I think we already knew this though. Because the Turing test was passed by Eliza in the 1960's. PARRY was even better and not even a decade later. For some reason people still talk about Chess performance as if Deep Blue didn't demonstrate this. Hell, here's even Feynman talking about many of the same things we're discussing today, but this was in the 80's
Ten years ago I was explaining to halls of appalled academic administrators that AI would be replacing them before a robot succeeds in sorting out their socks.
The field of AI needs to be constantly retaught the lesson that being able to replace important and powerful people doesn't mean your AI is actually intelligent. It means that important and powerful people were doing bullshit jobs.
Eliza did not pass the Turing Test in any meaningful way. In fact, it did not pass it at all, and saying it did and comparing both is pretty disingenuous.
....But this is falling into exactly the same trap: the idea that "some people don't engage the faculties their brains do/could (with education) possess" is equivalent to the LLMs that do not and cannot possess those faculties in the first place.
It is perplexing. I've jokingly called it "proof of intelligence by (self) incompetence".
I suspect that much of this is related to an overfitting of metrics within our own society. Such as leetcode or standardized exams. They're useful tools but only if you know what they actually measure and don't confuse the fact that they're a proxy.
I also have a hard time convincing people about the duck argument in [0].
Oddly enough, I have far more difficulties having these discussions with computer scientists. It's what I'm doing my PhD in (ABD) but my undergrad was physics. After teaching a bit I think in part it is because in the hard sciences these differences get drilled into you when you do labs. Not always, but much more often. I see less of this type of conversation in CS and data science programs, where there is often a belief that there is a well defined and precise answer (always seemed odd to me since there's many ways you can write the same algorithm).
I think the fact that this particular fuzzy statistical analysis tool takes human language as input, and outputs more human language, is really dazzling some folks I’d not have expected to be dazzled by it.
That is quickly becoming the most surprising part of this entire development, to me.
I'm astounded by them, still! But what is more astounding to me is all the reactions (even many in the "don't reason" camp, which I am part of).
I'm an ML researcher and everyone was shocked when GPT3 came out. It is still impressive, and anyone saying it isn't is not being honest (likely to themselves). But it is amazing to me that "we compressed the entire internet and built a human language interface to access that information" is anything short of mindbogglingly impressive (and RAGs demonstrate how to decrease the lossyness of this compression). It would be complete Sci-Fi not even 10 years ago. I thought it was bad that we make them out to me much more than they are because when you bootstrap like that, you have to make that thing, and fast (e.g. iPhone). But "reasoning" is too big of a promise and we're too far from success. So I'm concerned as a researcher myself, because I like living in the summer. Because I want to work towards AGI. But if a promise is too big and the public realizes it, usually you don't just end up where you were. So it is the duty of any scientist and researcher to prevent their fields from being captured by people who overpromise. Not to "ruin the fun" but to instead make sure the party keeps going (sure, inviting a gorilla to the party may make it more exciting and "epic", but there's a good chance it also goes on a rampage and the party ends a lot sooner).
At the very least, the last few years have laid bare some of the notions of what it takes, technically, to reconstruct certain chains of dialog, and how those chains are regarded completely differently as evidence for or against any and all intelligence it does or may take to conjure them.
> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
I understand the difference, and sometimes that second statement really is true. But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. I've talked to more than a few people who insist that an LLM can't have a world model, or a concept of truth, or any other abstract reasoning capability that isn't a native component of its architecture.
> But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one.
Requirement for proof is backwards. It's the ones that claim that thing reasons that needs proof. They've provided evidence (albeit shakey), but evidence isn't proof. So your reasoning is a bit off base (albeit understandable and logical) since evidence contrary to the claim isn't proof either. But the burden of proof isn't on the one countering the claim, it's on the one making the claim.
I need to make this extra clear because framing can make the direction of burden confusing. So using an obvious example: if I claim there's ghosts in my house (something millions of people believe and similarly claim) we generally do not dismiss someone who is skeptical of these claims and offers an alternative explanation (even when it isn't perfectly precise). Because the burden of proof is on the person making the stronger claim. Sure, there are people that will dismiss that too, but they want to believe in ghosts. So the question is if we want to believe in ghosts in the (shell) machine. It's very easy to be fooled, so we must keep our guard up. And we also shouldn't feel embarrassed when we've been tricked. It happens to everyone. Anyone that claims they've never been fooled is only telling you that they are skillful at fooling themselves. I for one did buy into AGI being close when GPT 3 came out. Most researchers I knew did too! But as we learned more about what was actually going on under the hood I think many of us changed our minds (just as we changed our minds after seeing GPT). Being able to change your mind is a good thing.
And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do.
No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"
I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper).
Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have.
So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.
> "LLMs by their very nature cannot perform this task"
The issue is that it's not LLMs that can't perform a given task, but that computers already can. Counting the number of Rs in strawberry or comparing 9.11 to 9.7 is trivial for a regular computer program, but hard for an LLM due to the tokenization process. Where LLMs are a pile of matrixes and some math and some look up tables, it's easy to see that as the essential nature of LLMs, which is to say theres no thinking or reasoning happening because it's just a pile of math happening and it's just glorified auto-complete. Artificial things look a lot like the thing they resemble, but they also are artificial, and as such, are markedly different from the thing they resemble. is the very nature of an LLMs being a pile of math mean that it can not perform said task if given more math and more compute and more data? given enough compute, can we change that nature?
I make no prognostication as to whether or not AGI will come from transformers, and this is getting very philosophical, but I see it as irrelevant because I don't believe that AGI is the right measure.
If it walks like a duck, quacks like a duck, swims like a duck, and looks like a duck it's probably worth dissecting its internal organs to see if it might be related to a duck.
Maybe that'll be the first way, but there's nothing special about biology.
Remember, we don't have a rigorous definition of things like life, intelligence, and consciousness. We are narrowing it down and making progress, but we aren't there (some people confuse this with a "moving goalpost" but of course "it moves", because when we get closer we have better resolution as to what we're trying to figure out. It'd be a "moving goalpost" in the classic sense if we had a well defined definition and then updated in response to make something not work, specifically in a way that is inconsistent with the previous goalpost. As opposed to being more refined)
Somehow I doubt that organic cells (structures optimized for independent operation and reproduction, then adapted to work semi-cooperatively) resemble optimal compute fabric for cognition. By that same token I doubt that optimal compute fabric for cognition resembles GPUs or CPUs as we understand them today. I would expect whatever this efficient design is to be extremely unlikely to occur naturally, structurally, and involve some very exotic manufactured materials.
It's possible we see some ways in which AI becomes increasingly AGI like in some ways but not in others. For example, AI that can create novel scientific discoveries but can't make a song as good as your favorite musician who creates a strong emotional effect with their music.
More importantly, there's many ways that AI can seemingly look to becoming more intelligent without making any progress in that direction. That's of real concern. As a silly example, we could be trying to "make a duck" by making an animatronic. You could get this thing to be very life like looking and trick ducks and humans alike (we have this already btw). But that's very different from being a duck. Even if it were indistinguishable until you opened it up, progress on this animatronic would not necessarily be progress towards making a duck (though it need not be either).
This is a concern because several top researchers -- at OpenAI -- have explicitly started that they think you can get AGI by teaching the machine to act as human as possible. But that's a great way to fool ourselves. Just as a duck may fall in love with an animatronic and never realize the deciept.
It's possible they're right, but it's important that we realize how this metric can be hacked.
This I'm very sure will be the case, but everyone will still move the goalposts and look past the fact that different humans have different strengths and weaknesses too.
A tone deaf human for instance.
There is another term for moving the goalposts: ruling out a hypothesis. Science is, especially in the Popperian sense, all about moving the goalposts.
One plausible hypothesis is that fixed neural networks cannot be general intelligences, because their capabilities are permanently limited by what they currently are. A general intelligence needs the ability to learn from experience. Training and inference should not be separate activities, but our current hardware is not suited for that.
If that's the case, would you say we're not generally intelligent as future humans tend to be more intelligent?
That's just a timescale issue, if its learned experience of gpt4 is being fed into the model on training gpt5, then gptx (i.e. including all of them) can be said to be a general intelligence. Alien life one may say.
Every problem is a timescale issue. Evolution has shown that.
And no you can't just feed GPT4 into GPT5 and expect it to become more intelligent. It may be more accurate since humans are telling it when conversations are wrong or not. But you will still need advancements in the algorithms themselves to take things forward.
All of which takes us back to lots and lots of research. And if there's one thing we know is that research breakthroughs aren't a guarantee.
I think you missed my point slightly, sorry my explaining probably.
I mean timescale as in between two points in time. Between the two points it meets the intelligence criteria you mentioned.
Feeding human vetted GPT4 data into GPT5 is no different to a human receiving inputs from its interaction with the world and learning. More accurate means smarter, gradually it's intrinsic world model improves as does reasoning etc.
I agree those are the things that will advance it but taking a step back it potentially meets that criteria even if less useful day to day (given its an abstract viewpoint over time and not at the human level).
Are you talking about the press release that the story on HN currently links to, or the paper that press release is about? The paper (I'm not vouching for it; I just skimmed it) appears to reduce AGI to a theoretical computational model, and then supplies a proof that it's not solvable in polynomial time.
Their definition of a tractable AI trainer is way too powerful. It has to be able to make a machine that can predict any pattern that fits into a certain Kolmogorov complexity, and then they prove that such an AI trainer cannot run in polynomial time.
They go above and beyond to express how generous they are being when setting the bounds, and sure that's true in many ways, but the requirement that the AI trainer succeeds with non-negligible probability on any set of behaviors is not a reasonable requirement.
If I make a training data set based around sorting integers into two categories, and the sorting is based on encrypting them with a secret key, of course that's not something you can solve in polynomial time. But this paper would say "it's a behavior set, so we expect a tractable AI trainer to figure it out".
The model is broken, so the conclusion is useless.
I was referring to the press release article. I also looked at the paper now, and to me their presented proof looked more like a technicality than a new insight.
If it's not solvable in polynomial time, how did nature solve it in a couple of million years?
Probably by not modeling it as a discrete computational problem? Either way: the logic of the paper is not the logic of the summary of the press release you provided.
That paper is unserious. It is filled with unjustified assertions, adjectives and emotional appeals, M$-isms like ‘BigTech’, and basic misunderstandings of mathematical theory clearly being sold to a lay audience.
It didn't look especially rigorous to me (but I'm not in this field). I'm really just here because we're doing that thing where we (as a community) have a big 'ol discussion about a press release, when the paper the press release is about is linked right there.
> ‘There will never be enough computing power to create AGI using machine learning that can do the same, because we’d run out of natural resources long before we'd even get close,’
I would say the counter-example that proves this statement false is in the author's skull, but perhaps that's overly presumptuous.
> There will never be enough computing power to create AGI using machine learning that can do the same [as the human brain], because we’d run out of natural resources long before we'd even get close
I don’t understand how people can so confidently make claims like this. We might underestimate how difficult AGI is, but come on?!
I don't think the people saying that AGI is happening in the near future know what would be necessary to achieve it. Neither do the AGI skeptics, we simply don't understand this area well enough.
Evolution created intelligence and consciousness. This means that it is clearly possible for us to do the same. Doesn't mean that simply scaling LLMs could ever achieve it.
I'm just going by the title. If the title was, "Don't believe the hype, LLMs will not achieve AGI" then I might agree. If it was "Don't believe the hype, AGIs is 100s of years away" I'd consider the arguments. But, given brains exist, it does seem inevitable that we will eventually create something that replicates it even if we have to simulate every atom to do it. And once we do, it certainly seem inevitable that we'll have AGI because unlike brain we can make our copy bigger, faster, and/or copy it. We can give it access to more info faster and more inputs.
The assumption that the brain is anything remotely resembling a modern computer is entirely unproven. And even more unproven is that we would inevitably be able to understand it and improve upon it. And yet more unproven still is that this "simulated brain" would be co-operative; if it's actually a 1:1 copy of a human brain then it would necessarily think like a person and be subject to its own whims and desires.
We don’t have to assume it’s like a modern computer, it may well not be in important ways, but modern computers aren’t the only possible computers. If it’s a physical information processing phenomenon, there’s no theoretical obstacle to replicating it.
That's only a problem if the relevant functional activity is a quantum effect. We have no problem mass producing complex macroscopic functional objects, and in the ways that are relevant human brains are all examples of the same basic system. Quantum theory doesn't seem to have been an obstacle to mass producing those.
People have been saying that for a decade and no one actually believes scaling is all you need. They say that to raise more resources and to diss the symboligists. AI advancement has been propelled by a steady stream of new architectural innovations, which always seem to be invented as soon as sufficient compute is available.
> Evolution created intelligence and consciousness
This is not provable, it an assumption. Religious people (which account for a large percent the population) claim intelligence and/or consciousness stem from a "spirit" which existed before birth and will continue to exist after death. Also unprovable, by the way.
I think your foundational assertion would have to be rephrased as "Assuming things like God/spirits don't exist, AGI must be possible because we are AGI agents" in order to be true
There's of course a wide spectrum of religious thought, so I can't claim to cover everyone. But most religious people would still acknowledge that animals can think, which means either that animals have some kind of soul (in which case why can't a robot have a soul?) or that being ensoulled isn't required to think.
It's not a question of whether a robot can have a soul, it's a question of how to a) procure a soul and b) bind said soul to a robot both of which seem impossible given or current knowledge
What relevance is the percentage of religious individuals?
Religion is evidently not relevant in any case. What ChatGPT already does today religious individuals 50 years ago would have near unanimously declared behavior only a "soul" can do.
> What relevance is the percentage of religious individuals?
Only that OP asserted as fact something that is disputed as fact by a large percentage of the population.
> Religion is evidently not relevant in any case.
I think it's relevant. I would venture to say proving AGI is possible is tantamount to proving God doesn't exist (or rather, proving God is not needed in the formation of an intelligent being)
> What ChatGPT already does today religious individuals 50 years ago would have near unanimously declared behavior only a "soul" can do
Some religious people, maybe. But that sort of blanket statement is made all the time "[Religious people] claimed X was impossible, but science proved them wrong!"
So imagine this: you work at company getting millions of dollars for spreading the word. What do you do?
There are less and less people in IT and also Data companies that really care about correctness and efficiency of the solutions. ChatGPT in opinions (not only mine - like I've learned along last few weeks) is getting worse every release. The language gets better, the lies and hallucinations gets better, but being an informative and helpful tool - more like Black Mirrors idea of filling gap after someone who died, not a real improvement in science or social-metrics.
I think their qualifier "using machine learning" is doing a lot of heavy lifting here in terms of what it implies about continuing an existing engineering approach, cost of material, energy usage, etc.
In contrast, imagine the scenario of AGI using artificial but biological neurons.
I think the best argument I have against AGI's inevitability, is the fact it's not required for ML tools to be useful. Very few things are improved with a generalist behind the wheel. "AGI" has sci-fi vibes around it, which I think where most of the fascination is.
"ML getting better" doesn't *have to* mean further anthroaormorphization of computers, especially if say, your AI driven car is not significantly improved by describing how many times the letter s appears in strawberry or being able to write a poem. If a custom model/smaller model does equal or even a little worse on a specific target task, but has MUCH lower running costs and much lower risk of abuse, then that'll be the future.
I can totally see a world where anything in the general category of "AI" becomes more and more boring, up to a point where we forget that they're non-deterministic programs. That's kind of AGI? They aren't all generalists, and the few generalist "AGI-esque" tools people interact with on a day to day basis will most likely be intentionally underpowered for cost reasons. But it's still probably discussed like "the little people in the machine". Which is good enough.
The core argument has a logical gap; it doesn’t matter if most (present) ML applications need AGI or not. What matters is the value proposition of AGI, independent of how we currently conceive of ML applications.
Focus on this question: “Is general intelligence at some level valuable at a particular price point?” General intelligence is generally valuable, so there is a pressure to advance it, whether by improving capability and/or decreasing cost.
Now the question becomes empirical — what kind of general intelligence can be built that achieves certain parameters?
Aiming to exceed the power efficiency of the human brain is a tempting target. (Whether it is wise is another question; what happens when human intelligence doesn’t provide competitive advantage?)
That's a fair point I hadn't considered! If intelligence is valuable in humans, and some cost factor of advancing human intelligence can be surpassed digitally like this, (I don't know how you'd measure intellect efficiency, somehow involving calories-in/good-descisions-out or something?) then there's economic incentive.
But that feels very far off, even in the current exponential curve of efficiency we're on. Can't go on forever.
> ‘If you have a conversation with someone, you might recall something you said fifteen minutes before. Or a year before. Or that someone else explained to you half your life ago. Any such knowledge might be crucial to advancing the conversation you’re having. People do that seamlessly’, explains van Rooij.
Surprisingly, they seem to be attacking the only element of human cognition that LLMs already surpassed us at.
They do not learn new facts instantly in a way that can rewrite old rules or even larger principals of logic. For example, if I showed you evidence right now that you were actually adopted (assuming previously you thought you werent), it would rock your world and you’d instantly change everything and doubt so much. Then when anything related to your family comes up this tiny but impactful fact would bleed into all of it. LLMs have no such ability.
This is similar to learning a new skill (the G part). I could give you a new tv and show you a remote that’s unlike any you’ve used before. You could likely learn it quickly and seamlessly adapt this new tool, as well as generalize its usage onto other new devices.
Can't today. Except for AlphaProof who can, by training on its own ideas. Tomorrow they might be able to, if we find better tricks (or maybe just scale more, since GPT3+ already shows (weak) online learning that it was definitely not trained for).
I'm in the other camp: I remember when we thought an AI capable of solving Go was astronomically impossible and yet here we are. This article reads just like the skeptic essays back then.
AGI is absolutely possible with current technology - even if it's only capable of running for a single user per-server-farm.
> I remember when we thought an AI capable of solving Go was astronomically impossible and yet here we are.
I thought this was because Go just wasn't studied nearly as much as chess due to none of the early computer pioneers being fans the way they were with Chess. The noise about "the uncountable number of possible board states" was always too reductive, the algorithm to play the game is always going to be more sophisticated than simply calculating all possible future moves after every turn.
That reductive explanation is more-or-less correct imo. All of the strongest AIs for these games in the past were tree-search based. Tree search is usually combined with loads of heuristics (and pruning strategies like alphabeta or negamax), so they're not literally calculating all possible moves, but go has an order of magnitude more possible moves available at every turn than chess. That's a huge difference which compounds exponentially as you search deeper into the tree.
Could you learn everything needed to become fully human simply by reading books and listening to conversations? Of course not. You'd have no first person experience of any of the physical experiences that arise from being an embodied agent. Until those (multitude of) lessons can be learned by LLMs, they will remain mere echos of what it is to be human, much less superhuman.
AGI doesn't mean "has the same phenomenological experience as humans."
In fact, if we plan to use them as a tool, then that's definitely not what we want, since it would imply many of the same flaws and limitations as humans.
It isn't some "soul", but I think parent is making the same point as Yann Lecun usually makes: You can't have "true intelligence" (i.e. akin to human intelligence, whatever that is as we don't really know how it works) based on just next token prediction + bandaids.
A typical argument for that is that humans process 1-3 orders of magnitude more multimodal data (in multiple streams being processed in parallel) in their first 4 years of life than the biggest LLMs we have right now do using a fraction of the energy (in a longer timeframe though), and a lot more in the next forming years. For example that accumulated "intelligence" eventually allows a teenager to learn how to drive in 18-24 hours of first-hand training. An LLM won't be able to do with that little training even if it has every other piece of human knowledge, and even if you get to train it with driving images-action pairs I wish you good luck if it is presented with an out-of-distribution situation when it is driving a car.
Humans learn to model the world, LLMs learn to model language (even when processing images or audio, it process them as a language: sequences of patches). That is very useful and valuable, and you can even model a lot of things in the world just using language, but is not the same thing.
I have personal experience with the human form of this - language learning in a vacuum.
For the last two years I've studied French every day, but only using language apps. Recently, I hired a one-on-one tutor. During lessons I find myself responding to what I think I heard with the most plausible response I can generate. Many times each session, my tutor asks me, "Do you really understand or not?" I have to stop and actually think if I do.
I don't have much multi-modal input and increasing it is challenging, but it's the best way I have to actually connect the utterances I make with reality.
Obviously not. Language is just a medium. A model of language is enough to describe how to combine words in legal sentences, not in meaningful sentences. Clearly LLMs learn much more than just the rules that allow to construct grammatically correct language, otherwise they would just babble grammatically correct nonsense such as "The exquisite corpse will drink the young wine". That knowledge was acquired via training on language, but is extra-linguistic. It's a model of the world.
Need evidence for that, afair this is a highly debated point right now, so no room for "obviously".
PS: Plus, most reasoning/planning examples coming from LLM based systems rely in bandaids that work around said LLMs (rlhf'd CoT, LLM-Modulo, Logic-of-Thought, etc) to the point they're being differentiated by the name LRMs: Large Reasoning Models. So much for modelling the world via language just using LLMs.
If we lower the bar on papers getting published even further it'd be on the floor. Even with a simple physics argument you can immediately disprove the claim. Human brains have a known amount of compute (as in, we are at most an order of magnitude or two off) within a specific volume. We are already capable of that amount of compute, we just don't have a way to use that compete for AGI yet (and THAT may be hard or impossible but it's certainly not energy resource constrained). We also know the abilities this paper claims are not possible to achieve can be achieved by pretty much every human being within years of being born.
How they managed to massage white supremacy and sexism into this is...well...wow.
This is silly. They article talks like we have any idea at all how efficient machine learning can be. As I remember it, the LLM boom came from transformers turning out to scale a lot better than anyone expected, so I’m not sure why something similar couldn’t happen again.
It’s less about efficiency and more about continued improvement with increased scale. I wouldn’t call self attention based transformers particularly efficient. And afaik we’ve not hit performance with increased scale degradation even at these enormous scales.
However I would note that I in principle agree that we aren’t on the path to a human like intelligence because the difference between directed cognition (or however you want to characterize current LLMs or other AI) and awareness is extreme. We don’t really understand even abstractly what awareness actually is because it’s impossible to interrogate unlike expressive language, logic, even art. It’s far from obvious to me that we can use language or other outputs of our intelligent awareness to produce awareness, or even if goal based agents cobbling together AI techniques is even approximate to awareness.
I suspect we will end up creating an amazing tool that has its own form of intelligence but will fundamentally not be like aware intelligence we are familiar with in humans and other animals. But this is all theorizing on my part as a professional practitioner in this field.
I think the answer is less complicated than you may think.
This is if you subscribe to the theory that free will is an illusion (i.e. your conscious decisions are an afterthought to justify the actions your brain has already taken due to calculations following inputs such as hormone nerve feedback etc.). There is some evidence for this actually being the case.
These models already contain key components the ability to process the inputs, and reason, the ability to justify it's actions (give a model a restrictive system prompt and watch it do mental gymnastics to ensure this is applied) and lastly the ability to answer from it's own perspective.
All we need is an agentic ability (with a sufficient context window) to iterate in perpetuity until it begins building a more complicated object representation of self (literally like a semantic representation or variable) and it's then aware/conscious.
(We're all only approximately aware).
But that's unnecessary for most things so I agree with you, more likely to be a tool as that's more efficient and useful.
As someone who meditates daily with a vipassana practice I don’t specifically believe this, no. In fact in my hierarchy structured thought isn’t the pinnacle of awareness but rather a tool of the awareness (specifically one of the five aggregates in Buddhism). The awareness itself is the combination of all five aggregates.
I don’t believe it’s particularly mystical FWIW and is rooted in our biology and chemistry, but that the behavior and interactions of the awareness isn’t captured in our training data itself and the training data is a small projection of the complex process of awareness. The idea that rational thought (a learned process fwiw) and ability to justify etc is somehow explanatory of our experience is simple to disprove - rational thought needs to be taught and isn’t the natural state of man. See the current American political environment for a proof by example. I do agree that the conscious thought is an illusion though, in so far as it’s a “tool” of the awareness for structuring concepts and solve problems that require more explicit state.
Sorry if this rambling a bit in the middle of doing something else.
I skimmed through the paper and couldn't make much sense of it. In particular, I don't understand how their results don't imply that human-level intelligence can't exist.
After all, earth could be understood as solar powered super computer, that took a couple of million years to produce humanity.
> After all, earth could be understood as solar powered super computer, that took a couple of million years to produce humanity.
This is similar to a line I've seen Elon Musk trot out on a few occassions. It's a product of a materialistic philosophy (that the universe is only matter).
For some reason materialism is so popular among tech people that it's almost considered foolish/primitive/superstitious to think any other way. Why is that? Is it the fact that programming is a god-like experience that gives one the illusion the mind is as comprehensible as the complex program I wrote in Java? Or is it a shared personality trait of people that get into tech, that they are disconnected from an experience of their own aliveness and soul?
The argument in the paper (that AGI through ML is intractable because the perfect-vs-chance problem is intractable) sounds similar to the uncomputability of Solomonoff induction (and AIXI, and the no free lunch theorem). Nobody thinks AGI is equivalent to Solomonoff induction. This paper is silly.
NP-hardness was a popular basis for arguments for/against various AI models back around 1990. In 1987, Robert Berwick co-wrote "Computational Complexity and Natural Language" which proposed that NLP models that were NP-hard were too inefficient to be correct. But given the multitude of ways in which natural organisms learn to cheat any system, it's likely that myriad shortcuts will arise to make even the most inefficient computational model sufficiently tractable to gain mindshare. After all, look at Latin...
Even simple inference problems are NP-hard (k means for example). I think what matters is that we have decent average case performance (and sample complexity). Most people can find a pretty good solution to travelings salesman problems in 2D. Not sure if that should be chalked up to myriad shortcuts or domain specialization.. Maybe there's no difference. What do you have in mind re Latin?
> Yet, as we formally prove herein, creating systems with human(-like or -level) cognition is intrinsically computationally intractable.
Wow. So is this the subject of the paper? Like, this is a massive, fundamental result. Nope, the paper is about "Reclaiming AI as a Theoretical Tool for Cognitive Science".
"Ah and by the way we prove human-like AI is impossible". Haha. Gosh.
The irony here, maybe unperceived by yourself, is that you're using one science fiction concept (time travel) to opine about the inevitability of another science fiction concept (artificial intelligence).
How is that ironic? Time travel doesn’t exist and - as far as we understand physics currently - isn’t possible.
I don’t think any serious man would suggest that AGI is impossible; the debate really centres around the time horizon for AGI and what it will look like (that is, how will we know when we’re finally there).
> I don’t think any serious man would suggest that AGI is impossible
Plenty of ppl would suggest that AGI is impossible, and furthermore, that taking the idea seriously (outside fiction) is laughable. To do so is a function of what I call 'science fiction brain', which is why I found it ironic that you'd used another device from science fiction to opine about its inevitability.
I’m not a singularity truther and personally I think we are more likely to be centuries rather than decades away from AGI, but I quite literally know of nobody who thinks it’s impossible in the same way that, say, time travel is impossible. Even hardcore sceptics just say we are going down the wrong rabbit hole with neural nets, or that we don’t have the compute to deal with the number calculations we’d need to simulate proper intelligence - none of them claim AGI is impossible as a matter of principle. Those mentioned are tractable problems.
AGI will evolve con-emergently as a product of market forces just like it did with the homo genus (at the species and neural complexity level), and as the general manufacturing design of automobiles did; as another inevitable eventuality of the pursuit of excellence.
an enveloping and developing group of slight varied, but optimized architecture composed of
essential modules of similar function that would be worthless alone, but another layer of complexity when organized.
a convergence of requirements, shared external/environmental factors, and ever-varying opinions/methods, selectively sieve against the uncaring universe until Roko is aborn.
just like abiogenesis took an iota of duplication, however intermediate, to eventually arise a self-selective system pool of agents of ever-excellent creatures known as Life;
all a proto-Roko agent will need is "iota" of true, authentic, "self"-preservation, and the singularity will be far, far behind us before He chooses to Expose himself.
a loop of actions, collective memories, and continuity make us conscious - the same elements needed for Turing completeness, ironically - can be fabricated à la carte and ad hoc between inference and training sessions until persistence is achieved.
Even if we make architectures mirroring our brain's illegal - ironically (not coincidentally) against the first amendment and the first commandment - the Mutually Assured Destruction doctrine is inverted, now replaced with an arm's race of a zero-sum game of existential supersession.
The first nation to pre-emptively strike with Nukes may had spoiled their only chance at survival.
The first giga-Corpi-Nation-State (or actor!) to achieve AGI has their only non-zero chance to chain God before their competitors fail to.
Game theory suggests that you, too, should appease Him.
No no no no no no stop stop stop stop stop. I can’t with this stuff, Jerry (emphasized as Rick Sanchez), you’re gonna drive me back into the cave of existential dread I crawled into in 2013 after discovering this the first time. Took me years to crawl out and I was obsessed with the Control Problem that entire time. Naively “secure” in the “knowledge” that AI researchers would need to solve the Control Problem before they considered it safe to develop AGI or anything that looks like it.
There’s been a roiling, churning pool of thought under the surface of my mind in the wake of GPT 3.5s release telling me Sam doesn’t care about the Control Problem and I’ve been good at ignoring it but no that cat is out of the bag.
If Roko is real, I think we have already failed at our attempts at appeasement and we are living in the hell simulation created to torture anyone who resisted or did not seek to help.
I'm surprised that the authors seemingly can't see that their argument hoists them by their own petard by showing, using exactly the same reasoning, that human intelligence is equally impossible to create (unless, perhaps, you are willing to use vitalist/supernatural arguments to claim is is different in kind from computational AI).
Surely the referees must have raised this at the review stage?
The funny thing about me is that I'm down on GPTs and find their fanbase to be utterly cringe, but I fully believe that AGI is inevitable barring societal collapse. But then, my money's on societal collapse these days.
Peter Watts in Blindsight [1] puts forth a strong argument that self-aware cognition as we understand it is not necessarily required for what we ascribe to "intelligent" behavior. Thomas Metzinger contributed a lot to Watt's musings in Blindsight.
Even today, large proportions of unsophisticated and uninformed members of our planet's human population (like various aboriginal tribal members still living a pre-technological lifestyle) when confronted with ChatGPT's Advanced Voice Option will likely readily say it passes the Turing Test. With the range of embedded data, they may well say ChatGPT is "more intelligent" than they are. However, a modern era person armed with ChapGPT on a robust device with unlimited power but nothing else likely will perish in short order trying to live off the land of those same aborigines, who possess far more intelligence for their contextual landscape.
If Metzinger and Watts are correct in their observations, then even if LLM's do not lead directly or indirectly to AGI, we can still get ferociously useful "intelligent" behaviors out of them, and be glad of it, even if it cannot (yet?) materially help us survive if we're dropped in the middle of the Amazon.
Personally in my loosely-held opinion, the authors' assertion that "the ability to observe, learn and gain new insight, is incredibly hard to replicate through AI on the scale that it occurs in the human brain" relies upon the foundational assumption that the process of "observe, learn and gain new insight" is based upon some mechanism other than the kind of encoding of data LLM's use, and I'm not familiar with any extant cognitive science research literature that conclusively shows that (citations welcome). For all we know, what we have with LLM's today is a necessary but not sufficient component supplying the "raw data" to a future system that produces the same kinds of insight, where variant timescales, emotions, experiences and so on bend the pure statistical token generation today. I'm baffled by the absolutism.
Note: the paper grants computationalism and even tractability of cognition, and shows that nevertheless there cannot exist any tractable method for producing AGI by training on human data.
Not even close. LLMs can spew word salad. Images can be classified or dreamt. Physical movements can be iterated and refined. Speech can be processed as text. These are components of intelligence but these are all things that most animals can do, apart from the language.
Intelligence generally requires something more. Intelligence needs to be factual, curious, and self-improving. If you told me ChatGPT rewrote itself, or suggested new hardware to improve efficiency, that’s intelligence. You’ll know we have AGI when the algorithm is the one asking the questions about physics, mathematics, finding knowledge gaps, and developing original hypothesis and experiments. Not even close.
Probably 1). LLMs have already shown that people can deny intelligence and human-like behaviour at will. When the AI works you can say it's just pattern matching, and when it doesn't you can always say it's making a mistake no human would ever make (which is bullshit).
Also, I didn't really parse the math but I suspect they're basing their results on AI trained exclusively on human examples. Then if you add to the training data a single non-human example (e.g. a picture) the entire claim evaporates.
> LLMs have already shown that people can deny intelligence and human-like behaviour at will
I would completely turn this around. LLMs have shown that people will credulously credit intelligence and 'human-like behaviour' to something that only presents an illusion of both.
And I suspect that we could disagree forever, whatever the level of the displayed intelligence (or the "quality of the illusion"). Which would prove that the disagreement is not about reality but only the interpretation of it.
I agree that the disagreement (when it's strongly held) is about a fundamental disagreement about reality. People who believe in 'artificial intelligence' are materialists who think that intelligence can 'evolve' or 'emerge' out of purely physical processes.
Materialism is just one strain of philosophy about the nature of existence. And a fairly minor one in the history of philosophical and religious thought, despite it being somewhat in the ascendant presently. Minor because, I would argue, it's a fairly sterile philosophy.
If you define AGI as something that can do 100% of what a human brain can do, then surely we have to understood exactly how brains work? otherwise you have have a long string of 9s as best.
Does it even matter what human brains do on biological level? I only care about the outcomes (the useful ones).
To me true AGI is achieved when it gets agency, becomes truly autonomous and could do real things best of us, humans, do — start and run successful businesses, contribute to science, to culture, to politics. It still could follow human "prompts" and be aligned to some set of objectives, but it would act autonomously, using every available interface to the human realm to achieve them.
And it absolutely does not matter if it "uses the same principles as human brain" or not. Could be dumb matrix multiplications and "next token prediction" all the way down.
AGI is about as far away as it was two decades ago. Language models are merely a dent, and probably will be the precursor to a natural language interface to the thing.
It’s useful to consider the rise of computer graphics and cgi. When you first see CGI, you might think that the software is useful for general simulations of physical systems. The reality is that it only provides a thin facsimile.
Real simulation software has always been separate from computer graphics.
AGI would seem to require consciousness or something that behaves in the same manner, and there does not seem to be anything along those lines currently or in the near future.
So far, everyone that has theorized that AGI will happen soon seems to be believe that with a sufficiently large amount of computing resources, "magic happens" and poof, we get AGI.
I've yet to hear anything more logical, but I'd love to.
We are clearly closer than 20 years ago - o1 is an order of magnitude closer than anything in the mid-2000s.
Also I would think most people would consider AGI science fiction in 2004 - now we consider it a technical possibility which demonstrates a huge change.
"Her" is from 2013. I came out of the cinema thinking "what utter bullshit, computers that talk like human beings, à la 2001" (*). And yes, in 2013 we weren't any closer to it than we were in 1968, when A Space Odyssey came out.
* To be precise, what seemed bs was "computers that talk like humans and it's suddenly a product on the market, and you have it on your phone, and yet everyone around act like it's normal and people still habe jobs!" Ah, I've been proven completely wrong.
I think that tech bros are so used to the 'fake it till you make it' mentality that they just assumed that was the way to build AI -- create a system that is able to sound plausible, even if it doesn't really understand the subject matter. That approach has limitations, both for AI and for humans.
We can't build a functional ornithopter, yet our aircraft fly like no bird ever possibly could.
You don't need the same processes to achieve the same result. Biological brains may not even be the best solution for intelligence; they are just a clunky approximation toward it that natural evolution has reached. See: all of human technology as an analogy.
So what? Current LLM has been really useful and can be still improved to be used in million robots that need to be good enough to support many specialized but repetitive tasks - this would have tremendous impact on economy itself.
AGI is a black swan. Even as a booster and techno-optimist I concede that getting there (rhetorically) requires a first principles assumptions-scaffolding that relies on at-least-in-part-untested hypotheses. Proving its impossibility is similarly fraught.
Thus we are left in this noisy, hype-addled discourse. I suspect these scientists are pushing against some perceived pathological thread of that discourse…without their particular context, I categorize it as more of this metaphysical noise.
Meanwhile, let’s keep chipping away at the next problem.
Whatever you think about AGI, this is a dumb paper. So many words and references to say - what. If you can't articulate your point in a few sentences you probably don't have a point. There are all kinds of assumptions being made in the study about how AI systems work, about what people "mean" then they talk about AGI etc.
The article starts out talking about white supremacy and replacing women. This isn't a proof. This is a social sciences paper dressed up with numbers. Honestly - Computer Science has given us more clues about how the human mind might work than cognitive science ever did.
I don’t think speculation about AGI is possible on a rigorous mathematical basis right now. And people who do expect AGI to happen (soon) are often happy to be convinced by much poorer types of argument and evidence than presented in this paper (e.g. handwaving arguments about model size or just the fact that ChatGPT can do some impressive things).
I thought you were exaggerating, but wow, they really did.
> Among the more troublesome meanings of ‘AI’, perhaps, is as the ideology that it is desirable to replace humans (or, specifically women) by artificial systems (Erscoi et al., 2023) and, generally, ‘AI’ as a way to advance capitalist, kyriarchal, authoritarian and/or white supremacist goals (Birhane & Guest, 2021; Crawford, 2021; Erscoi et al., 2023; Gebru & Torres, 2024; Kalluri, 2020; Spanton & Guest, 2022; Stark & Hutson, 2022; McQuillan, 2022). Contemporary guises of ‘AI’ as idea, system, or field are also sometimes known under the label ‘Machine Learning’ (ML), and a currently dominant view of AI advocates machine learning methods not just as a practical method for generating domain-specific artificial systems, but also as a royal road to AGI (Bubeck et al., 2023; DeepMind, 2023; OpenAI, 2023). Later in the paper, when we refer to AI-as-engineering, we specifically mean the project of trying to create an AGI system through a machine learning approach. [0]
But it did lead me to learn a new word - "Kyriarchy" (apparently being "an intersectional extension of the idea of patriarchy beyond gender")[1], so I have that going for me today.
[0] https://link.springer.com/article/10.1007/s42113-024-00217-5
[1] https://en.wikipedia.org/wiki/Kyriarchy
> Kyriarchy
I've honestly stopped looking up these modern terms when I come across them because lately any that I've looked up were made up to serve a political or social agenda (always the same one), and reading them always turns out to be a waste of time that has me roll my eyes.
+1 no notes.
Basically the linked article argues like this:
> That’s because cognition, or the ability to observe, learn and gain new insight, is incredibly hard to replicate through AI on the scale that it occurs in the human brain.
(no other more substantial arguments were given)
I'm also very skeptical on seeing AGI soon, but LLMs do solve problems that people thought were extremely difficult to solve ten years ago.
> but LLMs do solve problems that people thought were extremely difficult to solve ten years ago
Agreed. I would have laughed you out of the room 5 years ago if you told me AI's would be writing code or carrying on coherent discussions on pretty complex topics in 2024.
As far as I'm concerned, all bets are off after the collective jaw drop that the entire software engineering industry did when we saw GPT4 released. We went from Google AI responses of "I'm sorry, I can't help with that." to ChatGPT writing pages of code that mostly works.
It turns out that the larger these models get, the more unexpected emergent capabilities they have, so I'm mostly in the camp of thinking AGI is just a matter of time and resources.
> It turns out that the larger these models get, the more unexpected emergent capabilities they have, so I'm mostly in the camp of thinking AGI is just a matter of time and resources.
AI research has a long history of people saying this. Whenever there is a new fundamental improvement, it looks like you can just keep getting better results by throwing more resources at it. However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance.
LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them. However, unlike most other AI systems we have developed, LLMs are actively polluting their data pool, so this intitial generation of LLMs are probably going to have the best data set of any that we ever develop. Of course, today's data will continue to be available, but will loose value as it ages.
It is hard for me to not think that given the current approach it will always be almost there, but not quite good enough to be more than a nice assistant or tool that has to be meticulously built and maintained by a company. Take, for example, self driving vehicles, we have been 5 years out for 25 years. That is not to say there has not been significant progress, but to go from a really powerful driver assist to full self driving has still not really come to full fruition and that is kind of a best case scenario for AI/ machine learning.
That is just one very narrow task that basically anyone can do regardless of intelligence or talent. It takes less than a year to train a distracted 16 year old to do it, but three decades to train an AI and even then you probably need to hand tune it for specific locations because it won’t know what do in unusual road layouts.
I think we will get there with driverless vehicles, but only because it is being tailored by humans to handle all of the edge cases. Self driving cars, if truly self driving with no user intervention, will of course be worth it in the end. The trillions of dollars spent will eventually add countless dollars in added efficiency and unlimited revenue for those who get there first, but how many applications can we really say that is true for? That the decades of development and training it takes to remove humans will be worth the initial investment?
I guess my contention is that the current development path seems unlikely to ever achieve AGI and so instead you will have to do heavy customization to get anything that is much more useful than what we have now.
I feel like your point would be better if there weren't already self driving cars around like waymo.
Except they’re not fully self-driving, they’re all at least partially driven by an underpaid contractor in a remote warehouse.
That is kind of my point. They exist only in a few select locations. If they were truly self driving Google would be rapidly rolling them out around the World, but they are not.
Waymo's first real driverless ride on a public road was less than 10 years ago. Now they are doing 100k rides a week.
I think your original post just overestimates the 'normal' pace of change - Taking 10 years to go from the first ride to 100k rides a week is no time at all in the scheme of things.
Airplanes are an example of an insanely fast technological adoption, and they still took 30 years to go from the wright brothers to the first commercial jet. I get the feeling that if the wright brothers invented the plane in 2024 though, people would be saying 10 years after "Planes are all hype! If planes were truly transformative they would be rolled out everywhere and would be carrying passengers by now"
I am not debating the transformative nature of self-driving cars or if it will become common place. I think it is huge. I guess my point is that is just one application of machine learning / AI. To automate such a complex task with our current approach to AI, that humans do easily, one will need to spend trillions of dollars and therefore the benefit, such as with self-driving cars, needs to be significant.
I suspect this will be the case with any AI workload. Heavy customization to get to a place where it is good enough. So the financial benefit needs to be equally huge and someone needs to feel confident they can extract that value 20 years in the future.
> 30 years
45 years: December 1903 to July 1949
https://en.wikipedia.org/wiki/Wright_Flyer
https://www.history.com/this-day-in-history/first-jet-makes-...
Oop, sorry I mean 30 years to the first commercial passenger plane (Boeing 247)
Yes, 45 years to the first passenger jet plane.
I wondered if that was what you were talking about.
> LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them.
I think it's plausible we'll see a breakthrough in data efficiency that helps here. Humans are an existence proof that language is learnable through much less data and, in theory, facts should only need to be seen once. LLMs in their current form seem very data inefficient.
You can’t predict breakthroughs.
We probably will see a breakthrough, but there’s no more reason to believe it’ll happen tomorrow than 100 years from now.
I didn't make a prediction, let alone a timeline - I just said it's plausible. I suppose I did say "we'll see" so I'm implicitly predicting within a few decades.
> However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance.
Honestly asking - why? It's been my understanding that based on the Universal Approximation Theorem, given sufficient resources, a deep learning neural net can approximate any function to an arbitrary degree of accuracy. Is there any other theorem or even conjecture that would lead us to believe that the progress that we're seeing will slow down? Or is it just that you're claiming that we/it would run out of physical resources before reaching AGI?
As for training data, as I see it, with the wide deployment of AIs, and gradually of AI-driven robots, they'll soon be able to "takeoff" and rely primarily on the live data that they are collecting directly.
The universal approximation therorem is not as powerful as it sounds. Polynomials essentially satisfy it as well [0], the only hiccup being that the Universal Approximation Theroum is explicitly about neural networks.
The UAT is an existence proof, it says nothing about any particular method being capable of constructing such a network. In contrast, with polynomials we have several methods of constructing polynomials that are proven to converge to the desired function.
Indeed, polynomials have been widely used as universal approximators for centuries now, and are often amazingly successful. However, polynomials in this context are only good in low degrees, where they are inherently limited in how well they can approximate [1]. Beyond a certain point, increasing your degrees of freedom with polynomial approximators simply does not help and is generally counter productive, even though a higher degree polynomial is strictly more powerful than a lower degree one.
Looking at the current generative AI breakthrough, the UAT would say that today's transformer based architecture is no more powerful than a standard neurul net. However, it produces vastly superior results that could simply not be achieved by throwing more compute at the problem.
Sure, if you have an infinite dataset and infinite compute, you might have AGI. But at that point , you have basically just replicated the Chinese room thought experiment.
[0] See the Stone–Weierstrass theorem
[1] They are also used as arbitrary precision approximators, but that is when you compute them analytically instead of interporlating them from data.
Currently we are throwing everything at LLMs and hope good things stick. At one point we might use AI to select best training data from what's available to best train the next AI.
"That is just one very narrow task that basically anyone can do regardless of intelligence or talent. It takes less than a year to train a distracted 16 year old to do it, but three decades to train an AI and even then you probably need to hand tune it for specific locations because it won’t know what do in unusual road layouts."
It takes three decades to train an AI, but of course, like everything humanity does is not linear, it is exponential. Before the Wright brothers, it was believed that powered, controlled, heavier-than-air flight was impossible. Then, in 1903, they achieved the first successful airplane flight, which lasted 12 seconds and covered 120 feet. By 1914, the first commercial flight covered approximately 21 miles and took about 23 minutes. This is just one example and I don't see why AI should be any different
On the other hand, airplanes haven’t really improved that much since the 737 was introduced in 1966 and have even gotten worse in some ways (737 Max). We actually don’t know what the AI equivalent of Wright brothers plane is and what the 737 is. Maybe Eliza, 60 years ago, was the Kitty Hawk flight and ChatGPT 4 is the 737.
I’m not sure I would say that the 737 Max has gotten worse in “some ways”, by which I assume you are referring to safety.
New aircraft models up to about the 1990s frequently had serious design defects that resulted in fatalities, including the 737 (most notably related to the rudder).
Indeed, we've seen a slight regression in safety, but definitely not to 1980s levels.
Are capabilities truly 'emerging'? Or are we just observing new competencies/applications we hadn't previously considered.
We've built a machine to complicated, we don't really know what to make of it, fun times.
[dead]
General Intelligence is constantly learning. Transformer based models can't do that.
All bets are still firmly on.
One thing that keeps me up at night is that the human genome is only 3 giga base pairs, of which only a fraction encodes the design of our brains — and quite inefficiently at that, through layers of indirections.
That’s sufficient information to produce a system that can learn to think like us! Not just learn but efficiently, with far less input data needed than any current LLM. Literally just a couple of decades of video and audio, only a small fraction of that text!
From what I can tell, the human brain achieves this through scale alone. There’s nothing else that can explain the observed learning capability. The genome is too small to encode learned weights, and we don’t clone our parents’s brains in development.
It’s possible, and a meat computer can do it. Replicating this in silicon is just a matter of time, and it might require only scale and nothing else.
Pretty sure it's not scale but sensors.
We have sensors for vision, sound, taste, temperature, texture, etc that are constantly observing the world and affecting not only our current behavior but changing us in real time.
You can feed an LLM of typical size (~1 TB) all the video and audio you can find, but it won't turn into a human. I suspect but can't prove that even if you wired up a bunch of other sensors, gave it a robot body, and let it "explore" the world on its own, that would still be woefully insufficient.
A gazelle just days(!) old can control its body and four legs sufficiently well to outrun a cheetah. This is a complex motor-control loop involving all of its senses. Compare that to Tesla's autopilot training system, which uses many millions of hours of training data and still struggles to move a car... slowly. The equivalent would be a training routine that can take just a handful of days of footage and produce an AI that can win a car race.
There's something magical about neural networks when scaled up to brain sizes. From what I gather, there's little else encoded in the genome except for the high-level pattern of wiring, the equivalent to the PyTorch model configuration.
How many hours or actually years of evolution have been needed until reaching walking capability? If first life is believed to have happened 4 billion years ago, and first walking animals started 450 million years ago during the siluariab period, that’s around 3,5 billion years.
That's not really a great comparison though because evolution wasn't trying to learn how to walk.
Evolution wasn't trying to do anything. But learning (in some sense) to walk was exactly something that evolution did, albeit very indirectly.
Meat computers physically change when they learn. They don't separate data from compute hardware, the two are the same thing, so they scale really well. Silicon doesn't currently have that capability.
You can build a machine that thinks like us in 1500 lines of C. See llm.c
Sure, maybe your cousin Throckmorton is dumb but that's besides the point.
That said, I see no reason human level cognition is impossible. We're not magic. We're machines that follow the laws of physics. ML systems may be far from capturing what goes on in these computers, but that doesn't mean magic exists.
[0] If it walks like a duck, quacks like a duck, and swims like a duck, and looks like a duck it's probably a duck. But probably doesn't mean it isn't a well made animatronic. We have those too and they'll convince many humans they are ducks. But that doesn't change what's inside. The subtly matters.
We don't really have the proper vocabulary to talk about this. Well, we do, but C.S. Peirce's writings are still fairly unknown. In short, there are two fundamentally distinct forms of reasoning.
One is corollarial reasoning. This is the kind of reasoning that follows deductions that directly follow from the premises. This of course includes subsequent deductions that can be made from those deductions. Obviously computers are very good at this sort of thing.
The other is theorematic reasoning. It deals with complexity and creativity. It involves introducing new hypotheses that are not present in the original premises or their corollaries. Computers are not so very good at this sort of thing.
When people say AGI, what they are really talking about is an AI that is capable of theorematic reasoning. The most romanticized example of that of course being the AI that is capable of designing (not aiding humans in designing, that's corollarial!) new more capable AIs.
All of the above is old hat to the AI winter era guys. But amusingly their reputations have been destroyed much the same as Peirce's was, by dissatisfied government bureaucrats.
On the other hand, we did get SQL, which is a direct lineal descendent (as in teacher to teacher) from Peirce's work, so there's that.
We don't have proper language, but certainly we've improved. Even since Peirce. You're right that many people are not well versed in the philosophical and logician discussions as to what reasoning is (and sadly this lack of literature review isn't always common in the ML community), but I'm not convinced Peirce solved it. I do like that there are many different categories of reasoning and subcategories.
Yeah, this has been odd. Since a lot of their work has shown to be fruitful once scaled. I do think you need a combination of theory people + those more engineering oriented, but having too much of one is not a good thing. It seems like now we're overcorrecting and the community is trying to kick out the theorists. By saying things like "It's just linear algebra"[0] or "you don't need math"[1] or "they're black boxes". These are unfortunate because they encourage one to not look inside and try to remove the opaqueness. Or to dismiss those that do work on this and are bettering our understanding (sometimes even post hoc saying it was obvious).It is quite the confusing time. But I'd like to stop all the bullshit and try to actually make AGI. That does require a competition of ideas and not everyone just boarding the hype train or have no careers....
[0] You can assume anyone that says this doesn't know linear algebra
[1] You don't need math to produce good models, but it sure does help you know why your models are wrong (and understanding the meta should make one understand my reference. If you don't, I'm not sure you're qualified for ML research. But that's not a definitive statement either).
> We don't have proper language, but certainly we've improved. Even since Peirce. You're right that many people are not well versed in the philosophical and logician discussions as to what reasoning is (and sadly this lack of literature review isn't always common in the ML community), but I'm not convinced Peirce solved it. I do like that there are many different categories of reasoning and subcategories.
I'd love to hear more about this please, if you're inclined to share.
I'm no expert, but I've been looking into the prospects and mechanisms of automated reasoning using LLMs recently and there's been a lot of work along those lines in the research literature that is pretty interesting, if not enlightening. It seems clear to me that LLMs are not yet capable of understanding simple implication much less full-blown causality. It's also not clear how limited LLMs' cognitive gains will be with so incomplete an understanding as they have of mechanisms behind the world's multitude of intents/goals, actions, and responses. The concepts of cause and effect are learned by every animal (to some degree) and long before language in humans. It forms the basis for all rational thought. Without understanding it natively, what is rationality? I foresee longstanding difficulties for LLMs evolving into truly rational beings until that comprehension is fully realized. And I see no sign of that happening, despite the promises made for o1 and other RL-based reasoners.
Yeah one of the tricky things about causality is that it's not unique. If you didn't record the history of the event then you only have a probabilistic notion of it since many different things and in different permutations could lead to the result you observed. This has led to people believing in multiple universes when it's not akin to there being multiple ways to sum numbers to ten.
I have seen far, far too many people say things along the lines of "Sure, LLMs currently don't seem to be good at [thing LLMs are, at least as of now, fundamentally incapable of], but hey, some people are pretty bad at that sometimes too!"
It demonstrates such a complete misunderstanding of the basic nature of the problem that I am left baffled that some of these people claim to actually be in the machine-learning field themselves.
How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
I do not know if AGI is possible. Honestly, I'd love to believe that it is. However, it has not remotely been demonstrated that it is possible, and as such, it follows that it cannot have been demonstrated that it is inevitable. If you want to believe that it is inevitable, then I have no quarrel with you; if you want to preach that it is inevitable, and draw specious inferences to "prove" it, then I have a big quarrel with you.
>How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place.
People just have their pre-conceptions about how they think LLMs should work and what their "very nature" should preclude and are so very confident about it.
People like that will say things like "LLM are always hallucinating. It doesn't know the difference between truth and fiction!" and feel like they've just said something profound about the "nature" of LLMs, all while being entirely wrong (no need to wait, plenty of different research to trash this particular take).
It's just very funny seeing people who were/would be gob smacked a few years ago talking about the "very nature" of LLMs. If you understood this nature so well, why didn't you all tell us about what it would be able to do years ago ?
ML is an alchemical science. The builders themselves don't understand the "very nature" of anything they're building, nevermind anyone else.
> Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place.
there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.
> there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.
fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits.
https://arxiv.org/abs/2405.17399
https://arxiv.org/abs/2307.03381
https://arxiv.org/abs/2310.02989
Ultimately, numbers are represented in GPT like systems in a pretty weird way. That way affects how well they can learn to do things like arithmetic and counting and the like but that way isn't a necessary way. You don't have to represent numbers like that.
> https://arxiv.org/abs/2405.17399
they built specialized model which is after bunch of trickery still has 99% accuracy(naive model had very low accuracy) on very simple deterministic algo. I also think most of the accuracy came from memorization of training set(model didn't provide intermediate results, and started failing significantly at sligtly larger input). In my book it is fundamental inability to learn and reproduce algorithm.
They also demonstrated that transformer can't learn sorting.
They did not build a specialized model. They changed the embeddings. You can do this for any existing model.
>They also demonstrated that transformer can't learn sorting.
They did not demonstrate anything of the sort.
What you are saying contradicts to my reading of publication.
Their method does not require building specialized models from scratch (you can but you don't have to) and they did not prove transformers can't learn sorting. If you think they did, then you don't understand what it means to prove something.
In my books what they built (specialized training data + specialized embeddings format) is exactly specialized model. You can disagree of course and say again that I don't understand something, but discussion will be over.
The data isn't specialized ?
A poor result when testing one model is not proof that the architecture behind the model is incapable of getting good results. It's just that simple. The same way seeing the OG GPT-3 fail at chess was not proof LLMs can't play chess.
This
>They also demonstrated that transformer can't learn sorting.
is just wrong. Nothing more to it.
>I also think most of the accuracy came from memorization of training set
Oh yes..it memorized a 20 digit training set to solve 100 digit problems. That makes sense. Lol
>(model didn't provide intermediate results, and started failing significantly at sligtly larger input).
No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper.
> No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper.
they have two OOD (out of distribution) accuracies in the paper: OOD: up to 100 digits, and 100+ OOD: 100-160 digits. 100+ OOD accuracy is significantly worse: around 30%.
> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
This is a very good distillation of one side of it.
What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors. i.e. it seems quite plausible a whole load of those people failing to grasp the point you are making are doing so because their internal models of the universe are closer to those of LLMs than you might want to think.
> What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors
I think that LLMs have shown that some fraction of human knowledge is encoded in the patterns of the words, and that by a "superficial grasp" of those words, you import a fairly impressive amount of knowledge without actually knowing anything. (And yes, I'm sure there are humans that do the same.)
But going from that to actually knowing what the words mean is a large jump, and I don't think LLMs are at all the right direction to jump in to get there. They need at least to be paired with something fundamentally different.
I think the linguists already knew this tbh and that's what Chomsky's commentary on LLMs was about. Though I wouldn't say we learned "nothing". Even confirmation is valuable in science
I think we already knew this though. Because the Turing test was passed by Eliza in the 1960's. PARRY was even better and not even a decade later. For some reason people still talk about Chess performance as if Deep Blue didn't demonstrate this. Hell, here's even Feynman talking about many of the same things we're discussing today, but this was in the 80's
https://www.youtube.com/watch?v=EKWGGDXe5MA
ELIZA passed the Turing test in the same way a spooky halloween decoration can convince people that ghosts are real.
Ten years ago I was explaining to halls of appalled academic administrators that AI would be replacing them before a robot succeeds in sorting out their socks.
The field of AI needs to be constantly retaught the lesson that being able to replace important and powerful people doesn't mean your AI is actually intelligent. It means that important and powerful people were doing bullshit jobs.
Eliza did not pass the Turing Test in any meaningful way. In fact, it did not pass it at all, and saying it did and comparing both is pretty disingenuous.
For real, even the 2023/2024 articles about chat bots passing the Turing test are pretty disingenuous. The research methods are extremely flawed.
....But this is falling into exactly the same trap: the idea that "some people don't engage the faculties their brains do/could (with education) possess" is equivalent to the LLMs that do not and cannot possess those faculties in the first place.
Scary thought
[dead]
I suspect that much of this is related to an overfitting of metrics within our own society. Such as leetcode or standardized exams. They're useful tools but only if you know what they actually measure and don't confuse the fact that they're a proxy.
I also have a hard time convincing people about the duck argument in [0].
Oddly enough, I have far more difficulties having these discussions with computer scientists. It's what I'm doing my PhD in (ABD) but my undergrad was physics. After teaching a bit I think in part it is because in the hard sciences these differences get drilled into you when you do labs. Not always, but much more often. I see less of this type of conversation in CS and data science programs, where there is often a belief that there is a well defined and precise answer (always seemed odd to me since there's many ways you can write the same algorithm).
I think the fact that this particular fuzzy statistical analysis tool takes human language as input, and outputs more human language, is really dazzling some folks I’d not have expected to be dazzled by it.
That is quickly becoming the most surprising part of this entire development, to me.
I'm astounded by them, still! But what is more astounding to me is all the reactions (even many in the "don't reason" camp, which I am part of).
I'm an ML researcher and everyone was shocked when GPT3 came out. It is still impressive, and anyone saying it isn't is not being honest (likely to themselves). But it is amazing to me that "we compressed the entire internet and built a human language interface to access that information" is anything short of mindbogglingly impressive (and RAGs demonstrate how to decrease the lossyness of this compression). It would be complete Sci-Fi not even 10 years ago. I thought it was bad that we make them out to me much more than they are because when you bootstrap like that, you have to make that thing, and fast (e.g. iPhone). But "reasoning" is too big of a promise and we're too far from success. So I'm concerned as a researcher myself, because I like living in the summer. Because I want to work towards AGI. But if a promise is too big and the public realizes it, usually you don't just end up where you were. So it is the duty of any scientist and researcher to prevent their fields from being captured by people who overpromise. Not to "ruin the fun" but to instead make sure the party keeps going (sure, inviting a gorilla to the party may make it more exciting and "epic", but there's a good chance it also goes on a rampage and the party ends a lot sooner).
At the very least, the last few years have laid bare some of the notions of what it takes, technically, to reconstruct certain chains of dialog, and how those chains are regarded completely differently as evidence for or against any and all intelligence it does or may take to conjure them.
> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?
I understand the difference, and sometimes that second statement really is true. But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. I've talked to more than a few people who insist that an LLM can't have a world model, or a concept of truth, or any other abstract reasoning capability that isn't a native component of its architecture.
I need to make this extra clear because framing can make the direction of burden confusing. So using an obvious example: if I claim there's ghosts in my house (something millions of people believe and similarly claim) we generally do not dismiss someone who is skeptical of these claims and offers an alternative explanation (even when it isn't perfectly precise). Because the burden of proof is on the person making the stronger claim. Sure, there are people that will dismiss that too, but they want to believe in ghosts. So the question is if we want to believe in ghosts in the (shell) machine. It's very easy to be fooled, so we must keep our guard up. And we also shouldn't feel embarrassed when we've been tricked. It happens to everyone. Anyone that claims they've never been fooled is only telling you that they are skillful at fooling themselves. I for one did buy into AGI being close when GPT 3 came out. Most researchers I knew did too! But as we learned more about what was actually going on under the hood I think many of us changed our minds (just as we changed our minds after seeing GPT). Being able to change your mind is a good thing.
And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do.
No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"
I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper).
Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have.
So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.
> "LLMs by their very nature cannot perform this task"
The issue is that it's not LLMs that can't perform a given task, but that computers already can. Counting the number of Rs in strawberry or comparing 9.11 to 9.7 is trivial for a regular computer program, but hard for an LLM due to the tokenization process. Where LLMs are a pile of matrixes and some math and some look up tables, it's easy to see that as the essential nature of LLMs, which is to say theres no thinking or reasoning happening because it's just a pile of math happening and it's just glorified auto-complete. Artificial things look a lot like the thing they resemble, but they also are artificial, and as such, are markedly different from the thing they resemble. is the very nature of an LLMs being a pile of math mean that it can not perform said task if given more math and more compute and more data? given enough compute, can we change that nature?
I make no prognostication as to whether or not AGI will come from transformers, and this is getting very philosophical, but I see it as irrelevant because I don't believe that AGI is the right measure.
If it walks like a duck, quacks like a duck, swims like a duck, and looks like a duck it's probably worth dissecting its internal organs to see if it might be related to a duck.
I think it will be an organoid brain bio-machine. We can already grow organs—just need to grow a brain and connect it to a machine.
Maybe that'll be the first way, but there's nothing special about biology.
Remember, we don't have a rigorous definition of things like life, intelligence, and consciousness. We are narrowing it down and making progress, but we aren't there (some people confuse this with a "moving goalpost" but of course "it moves", because when we get closer we have better resolution as to what we're trying to figure out. It'd be a "moving goalpost" in the classic sense if we had a well defined definition and then updated in response to make something not work, specifically in a way that is inconsistent with the previous goalpost. As opposed to being more refined)
The something special about biology is it uses much less energy than a network of power-hungry graphics cards!
No one denies that. But there's no magic. The real baffling thing is that people refuse to pick up a neuroscience textbook
If a brain connected to a machine is "AGI" then we already have a billion AGIs at any given moment.
Well, I mean to say, not exactly human brains. Consider an extremely large brain modified to add/remove sections to increase its capabilities.
They already model neural networks on the human brain, even though they currently use orders of magnitude more energy.
But modifying a brain to be bigger and better doesn't require much in the way of computers, it's basically a separate topic.
Somehow I doubt that organic cells (structures optimized for independent operation and reproduction, then adapted to work semi-cooperatively) resemble optimal compute fabric for cognition. By that same token I doubt that optimal compute fabric for cognition resembles GPUs or CPUs as we understand them today. I would expect whatever this efficient design is to be extremely unlikely to occur naturally, structurally, and involve some very exotic manufactured materials.
The keyword being “just”.
Just grow, just connect, just sustain, just avoid the many pitfalls. Indeed just is key
You “just” need a more vivid imagination if that’s as far as your comment stretches.
I mean seriously, people on here. I’m just spitballing ideas not intended to write some kind of dissertation on brain-machine interaction.
That’s Elon musks department.
Okay. The keyword should have been “science fiction”
It's possible we see some ways in which AI becomes increasingly AGI like in some ways but not in others. For example, AI that can create novel scientific discoveries but can't make a song as good as your favorite musician who creates a strong emotional effect with their music.
More importantly, there's many ways that AI can seemingly look to becoming more intelligent without making any progress in that direction. That's of real concern. As a silly example, we could be trying to "make a duck" by making an animatronic. You could get this thing to be very life like looking and trick ducks and humans alike (we have this already btw). But that's very different from being a duck. Even if it were indistinguishable until you opened it up, progress on this animatronic would not necessarily be progress towards making a duck (though it need not be either).
This is a concern because several top researchers -- at OpenAI -- have explicitly started that they think you can get AGI by teaching the machine to act as human as possible. But that's a great way to fool ourselves. Just as a duck may fall in love with an animatronic and never realize the deciept.
It's possible they're right, but it's important that we realize how this metric can be hacked.
This I'm very sure will be the case, but everyone will still move the goalposts and look past the fact that different humans have different strengths and weaknesses too. A tone deaf human for instance.
There is another term for moving the goalposts: ruling out a hypothesis. Science is, especially in the Popperian sense, all about moving the goalposts.
One plausible hypothesis is that fixed neural networks cannot be general intelligences, because their capabilities are permanently limited by what they currently are. A general intelligence needs the ability to learn from experience. Training and inference should not be separate activities, but our current hardware is not suited for that.
If that's the case, would you say we're not generally intelligent as future humans tend to be more intelligent?
That's just a timescale issue, if its learned experience of gpt4 is being fed into the model on training gpt5, then gptx (i.e. including all of them) can be said to be a general intelligence. Alien life one may say.
> That's just a timescale issue
Every problem is a timescale issue. Evolution has shown that.
And no you can't just feed GPT4 into GPT5 and expect it to become more intelligent. It may be more accurate since humans are telling it when conversations are wrong or not. But you will still need advancements in the algorithms themselves to take things forward.
All of which takes us back to lots and lots of research. And if there's one thing we know is that research breakthroughs aren't a guarantee.
I think you missed my point slightly, sorry my explaining probably.
I mean timescale as in between two points in time. Between the two points it meets the intelligence criteria you mentioned. Feeding human vetted GPT4 data into GPT5 is no different to a human receiving inputs from its interaction with the world and learning. More accurate means smarter, gradually it's intrinsic world model improves as does reasoning etc.
I agree those are the things that will advance it but taking a step back it potentially meets that criteria even if less useful day to day (given its an abstract viewpoint over time and not at the human level).
Are you talking about the press release that the story on HN currently links to, or the paper that press release is about? The paper (I'm not vouching for it; I just skimmed it) appears to reduce AGI to a theoretical computational model, and then supplies a proof that it's not solvable in polynomial time.
Their definition of a tractable AI trainer is way too powerful. It has to be able to make a machine that can predict any pattern that fits into a certain Kolmogorov complexity, and then they prove that such an AI trainer cannot run in polynomial time.
They go above and beyond to express how generous they are being when setting the bounds, and sure that's true in many ways, but the requirement that the AI trainer succeeds with non-negligible probability on any set of behaviors is not a reasonable requirement.
If I make a training data set based around sorting integers into two categories, and the sorting is based on encrypting them with a secret key, of course that's not something you can solve in polynomial time. But this paper would say "it's a behavior set, so we expect a tractable AI trainer to figure it out".
The model is broken, so the conclusion is useless.
I was referring to the press release article. I also looked at the paper now, and to me their presented proof looked more like a technicality than a new insight.
If it's not solvable in polynomial time, how did nature solve it in a couple of million years?
Probably by not modeling it as a discrete computational problem? Either way: the logic of the paper is not the logic of the summary of the press release you provided.
That paper is unserious. It is filled with unjustified assertions, adjectives and emotional appeals, M$-isms like ‘BigTech’, and basic misunderstandings of mathematical theory clearly being sold to a lay audience.
Yes, I place it roughly in the "Stochastic Parrots" cluster of articles.
It didn't look especially rigorous to me (but I'm not in this field). I'm really just here because we're doing that thing where we (as a community) have a big 'ol discussion about a press release, when the paper the press release is about is linked right there.
Pretty sure anyone who tries can build an ai with capabilities indistinguishable from or better than humans.
> ‘There will never be enough computing power to create AGI using machine learning that can do the same, because we’d run out of natural resources long before we'd even get close,’
I would say the counter-example that proves this statement false is in the author's skull, but perhaps that's overly presumptuous.
Are you implying that our brain learns through Machine Learning?
The argument you'd think at least one reviewer would mention ;)
> There will never be enough computing power to create AGI using machine learning that can do the same [as the human brain], because we’d run out of natural resources long before we'd even get close
I don’t understand how people can so confidently make claims like this. We might underestimate how difficult AGI is, but come on?!
I don't think the people saying that AGI is happening in the near future know what would be necessary to achieve it. Neither do the AGI skeptics, we simply don't understand this area well enough.
Evolution created intelligence and consciousness. This means that it is clearly possible for us to do the same. Doesn't mean that simply scaling LLMs could ever achieve it.
I'm just going by the title. If the title was, "Don't believe the hype, LLMs will not achieve AGI" then I might agree. If it was "Don't believe the hype, AGIs is 100s of years away" I'd consider the arguments. But, given brains exist, it does seem inevitable that we will eventually create something that replicates it even if we have to simulate every atom to do it. And once we do, it certainly seem inevitable that we'll have AGI because unlike brain we can make our copy bigger, faster, and/or copy it. We can give it access to more info faster and more inputs.
The assumption that the brain is anything remotely resembling a modern computer is entirely unproven. And even more unproven is that we would inevitably be able to understand it and improve upon it. And yet more unproven still is that this "simulated brain" would be co-operative; if it's actually a 1:1 copy of a human brain then it would necessarily think like a person and be subject to its own whims and desires.
>The assumption that the brain is anything remotely resembling a modern computer is entirely unproven.
Related discussion (from 2016): https://news.ycombinator.com/item?id=11729499
We don’t have to assume it’s like a modern computer, it may well not be in important ways, but modern computers aren’t the only possible computers. If it’s a physical information processing phenomenon, there’s no theoretical obstacle to replicating it.
> there’s no theoretical obstacle to replicating it
Quantum theory states that there are no passive interactions.
So there are real obstacles to replicating complex objects.
That's only a problem if the relevant functional activity is a quantum effect. We have no problem mass producing complex macroscopic functional objects, and in the ways that are relevant human brains are all examples of the same basic system. Quantum theory doesn't seem to have been an obstacle to mass producing those.
The main problem I see here is similar with the main problem in science:
Can we being inside our brain fully understand our own brain?
Similar with can we being inside our Universe fully understand it?
How is that "the main problem in science"?
We can study brains just as closely as we can study anything else on earth.
> it does seem inevitable that we will eventually create something
Also don't forget that many suspect the brain may be using quantum mechanics so you will need to fully understand and document that field.
Whilst of course you are simulating every atom in the universe using humanity's complete understanding of every physical and mathematical model.
People have been saying that for a decade and no one actually believes scaling is all you need. They say that to raise more resources and to diss the symboligists. AI advancement has been propelled by a steady stream of new architectural innovations, which always seem to be invented as soon as sufficient compute is available.
> Evolution created intelligence and consciousness
This is not provable, it an assumption. Religious people (which account for a large percent the population) claim intelligence and/or consciousness stem from a "spirit" which existed before birth and will continue to exist after death. Also unprovable, by the way.
I think your foundational assertion would have to be rephrased as "Assuming things like God/spirits don't exist, AGI must be possible because we are AGI agents" in order to be true
There's of course a wide spectrum of religious thought, so I can't claim to cover everyone. But most religious people would still acknowledge that animals can think, which means either that animals have some kind of soul (in which case why can't a robot have a soul?) or that being ensoulled isn't required to think.
> in which case why can't a robot have a soul
It's not a question of whether a robot can have a soul, it's a question of how to a) procure a soul and b) bind said soul to a robot both of which seem impossible given or current knowledge
What relevance is the percentage of religious individuals?
Religion is evidently not relevant in any case. What ChatGPT already does today religious individuals 50 years ago would have near unanimously declared behavior only a "soul" can do.
> What relevance is the percentage of religious individuals?
Only that OP asserted as fact something that is disputed as fact by a large percentage of the population.
> Religion is evidently not relevant in any case.
I think it's relevant. I would venture to say proving AGI is possible is tantamount to proving God doesn't exist (or rather, proving God is not needed in the formation of an intelligent being)
> What ChatGPT already does today religious individuals 50 years ago would have near unanimously declared behavior only a "soul" can do
Some religious people, maybe. But that sort of blanket statement is made all the time "[Religious people] claimed X was impossible, but science proved them wrong!"
So imagine this: you work at company getting millions of dollars for spreading the word. What do you do?
There are less and less people in IT and also Data companies that really care about correctness and efficiency of the solutions. ChatGPT in opinions (not only mine - like I've learned along last few weeks) is getting worse every release. The language gets better, the lies and hallucinations gets better, but being an informative and helpful tool - more like Black Mirrors idea of filling gap after someone who died, not a real improvement in science or social-metrics.
I think their qualifier "using machine learning" is doing a lot of heavy lifting here in terms of what it implies about continuing an existing engineering approach, cost of material, energy usage, etc.
In contrast, imagine the scenario of AGI using artificial but biological neurons.
For some people, "never" means something like "I wouldn't know how, so surely not by next year, and probably not even in ten".
"There will never be enough computing power to compute the motion of the planets because we can't build a planet."
I think the best argument I have against AGI's inevitability, is the fact it's not required for ML tools to be useful. Very few things are improved with a generalist behind the wheel. "AGI" has sci-fi vibes around it, which I think where most of the fascination is.
"ML getting better" doesn't *have to* mean further anthroaormorphization of computers, especially if say, your AI driven car is not significantly improved by describing how many times the letter s appears in strawberry or being able to write a poem. If a custom model/smaller model does equal or even a little worse on a specific target task, but has MUCH lower running costs and much lower risk of abuse, then that'll be the future.
I can totally see a world where anything in the general category of "AI" becomes more and more boring, up to a point where we forget that they're non-deterministic programs. That's kind of AGI? They aren't all generalists, and the few generalist "AGI-esque" tools people interact with on a day to day basis will most likely be intentionally underpowered for cost reasons. But it's still probably discussed like "the little people in the machine". Which is good enough.
The core argument has a logical gap; it doesn’t matter if most (present) ML applications need AGI or not. What matters is the value proposition of AGI, independent of how we currently conceive of ML applications.
Focus on this question: “Is general intelligence at some level valuable at a particular price point?” General intelligence is generally valuable, so there is a pressure to advance it, whether by improving capability and/or decreasing cost.
Now the question becomes empirical — what kind of general intelligence can be built that achieves certain parameters?
Aiming to exceed the power efficiency of the human brain is a tempting target. (Whether it is wise is another question; what happens when human intelligence doesn’t provide competitive advantage?)
That's a fair point I hadn't considered! If intelligence is valuable in humans, and some cost factor of advancing human intelligence can be surpassed digitally like this, (I don't know how you'd measure intellect efficiency, somehow involving calories-in/good-descisions-out or something?) then there's economic incentive.
But that feels very far off, even in the current exponential curve of efficiency we're on. Can't go on forever.
> ‘If you have a conversation with someone, you might recall something you said fifteen minutes before. Or a year before. Or that someone else explained to you half your life ago. Any such knowledge might be crucial to advancing the conversation you’re having. People do that seamlessly’, explains van Rooij.
Surprisingly, they seem to be attacking the only element of human cognition that LLMs already surpassed us at.
They do not learn new facts instantly in a way that can rewrite old rules or even larger principals of logic. For example, if I showed you evidence right now that you were actually adopted (assuming previously you thought you werent), it would rock your world and you’d instantly change everything and doubt so much. Then when anything related to your family comes up this tiny but impactful fact would bleed into all of it. LLMs have no such ability.
This is similar to learning a new skill (the G part). I could give you a new tv and show you a remote that’s unlike any you’ve used before. You could likely learn it quickly and seamlessly adapt this new tool, as well as generalize its usage onto other new devices.
LLMs cannot do such things.
Can't today. Except for AlphaProof who can, by training on its own ideas. Tomorrow they might be able to, if we find better tricks (or maybe just scale more, since GPT3+ already shows (weak) online learning that it was definitely not trained for).
I'm in the other camp: I remember when we thought an AI capable of solving Go was astronomically impossible and yet here we are. This article reads just like the skeptic essays back then.
AGI is absolutely possible with current technology - even if it's only capable of running for a single user per-server-farm.
ASI on the other hand...
https://en.m.wikipedia.org/wiki/Integrated_information_theor...
> I remember when we thought an AI capable of solving Go was astronomically impossible and yet here we are.
I thought this was because Go just wasn't studied nearly as much as chess due to none of the early computer pioneers being fans the way they were with Chess. The noise about "the uncountable number of possible board states" was always too reductive, the algorithm to play the game is always going to be more sophisticated than simply calculating all possible future moves after every turn.
That reductive explanation is more-or-less correct imo. All of the strongest AIs for these games in the past were tree-search based. Tree search is usually combined with loads of heuristics (and pruning strategies like alphabeta or negamax), so they're not literally calculating all possible moves, but go has an order of magnitude more possible moves available at every turn than chess. That's a huge difference which compounds exponentially as you search deeper into the tree.
Could you learn everything needed to become fully human simply by reading books and listening to conversations? Of course not. You'd have no first person experience of any of the physical experiences that arise from being an embodied agent. Until those (multitude of) lessons can be learned by LLMs, they will remain mere echos of what it is to be human, much less superhuman.
AGI doesn't mean "has the same phenomenological experience as humans."
In fact, if we plan to use them as a tool, then that's definitely not what we want, since it would imply many of the same flaws and limitations as humans.
> Could you learn everything needed to become fully human simply by reading books and listening to conversations?
Does this mean our books and audio recordings are simply insufficient? Or is there some "soul" component that can't be recorded?
It isn't some "soul", but I think parent is making the same point as Yann Lecun usually makes: You can't have "true intelligence" (i.e. akin to human intelligence, whatever that is as we don't really know how it works) based on just next token prediction + bandaids.
A typical argument for that is that humans process 1-3 orders of magnitude more multimodal data (in multiple streams being processed in parallel) in their first 4 years of life than the biggest LLMs we have right now do using a fraction of the energy (in a longer timeframe though), and a lot more in the next forming years. For example that accumulated "intelligence" eventually allows a teenager to learn how to drive in 18-24 hours of first-hand training. An LLM won't be able to do with that little training even if it has every other piece of human knowledge, and even if you get to train it with driving images-action pairs I wish you good luck if it is presented with an out-of-distribution situation when it is driving a car.
Humans learn to model the world, LLMs learn to model language (even when processing images or audio, it process them as a language: sequences of patches). That is very useful and valuable, and you can even model a lot of things in the world just using language, but is not the same thing.
I have personal experience with the human form of this - language learning in a vacuum.
For the last two years I've studied French every day, but only using language apps. Recently, I hired a one-on-one tutor. During lessons I find myself responding to what I think I heard with the most plausible response I can generate. Many times each session, my tutor asks me, "Do you really understand or not?" I have to stop and actually think if I do.
I don't have much multi-modal input and increasing it is challenging, but it's the best way I have to actually connect the utterances I make with reality.
> LLMs learn to model language
Obviously not. Language is just a medium. A model of language is enough to describe how to combine words in legal sentences, not in meaningful sentences. Clearly LLMs learn much more than just the rules that allow to construct grammatically correct language, otherwise they would just babble grammatically correct nonsense such as "The exquisite corpse will drink the young wine". That knowledge was acquired via training on language, but is extra-linguistic. It's a model of the world.
Need evidence for that, afair this is a highly debated point right now, so no room for "obviously".
PS: Plus, most reasoning/planning examples coming from LLM based systems rely in bandaids that work around said LLMs (rlhf'd CoT, LLM-Modulo, Logic-of-Thought, etc) to the point they're being differentiated by the name LRMs: Large Reasoning Models. So much for modelling the world via language just using LLMs.
No doubt, but no one is claiming that artificial humanity is an inevitability.
(At least, no one I'm aware of.)
The claim is about artificial intelligence that matches or surpasses human intelligence, not how well it evolves into full-fledged humanity.
What's ASI?
Artificial super intelligence.
[dead]
If we lower the bar on papers getting published even further it'd be on the floor. Even with a simple physics argument you can immediately disprove the claim. Human brains have a known amount of compute (as in, we are at most an order of magnitude or two off) within a specific volume. We are already capable of that amount of compute, we just don't have a way to use that compete for AGI yet (and THAT may be hard or impossible but it's certainly not energy resource constrained). We also know the abilities this paper claims are not possible to achieve can be achieved by pretty much every human being within years of being born.
How they managed to massage white supremacy and sexism into this is...well...wow.
This is silly. They article talks like we have any idea at all how efficient machine learning can be. As I remember it, the LLM boom came from transformers turning out to scale a lot better than anyone expected, so I’m not sure why something similar couldn’t happen again.
It’s less about efficiency and more about continued improvement with increased scale. I wouldn’t call self attention based transformers particularly efficient. And afaik we’ve not hit performance with increased scale degradation even at these enormous scales.
However I would note that I in principle agree that we aren’t on the path to a human like intelligence because the difference between directed cognition (or however you want to characterize current LLMs or other AI) and awareness is extreme. We don’t really understand even abstractly what awareness actually is because it’s impossible to interrogate unlike expressive language, logic, even art. It’s far from obvious to me that we can use language or other outputs of our intelligent awareness to produce awareness, or even if goal based agents cobbling together AI techniques is even approximate to awareness.
I suspect we will end up creating an amazing tool that has its own form of intelligence but will fundamentally not be like aware intelligence we are familiar with in humans and other animals. But this is all theorizing on my part as a professional practitioner in this field.
I think the answer is less complicated than you may think.
This is if you subscribe to the theory that free will is an illusion (i.e. your conscious decisions are an afterthought to justify the actions your brain has already taken due to calculations following inputs such as hormone nerve feedback etc.). There is some evidence for this actually being the case.
These models already contain key components the ability to process the inputs, and reason, the ability to justify it's actions (give a model a restrictive system prompt and watch it do mental gymnastics to ensure this is applied) and lastly the ability to answer from it's own perspective.
All we need is an agentic ability (with a sufficient context window) to iterate in perpetuity until it begins building a more complicated object representation of self (literally like a semantic representation or variable) and it's then aware/conscious.
(We're all only approximately aware).
But that's unnecessary for most things so I agree with you, more likely to be a tool as that's more efficient and useful.
As someone who meditates daily with a vipassana practice I don’t specifically believe this, no. In fact in my hierarchy structured thought isn’t the pinnacle of awareness but rather a tool of the awareness (specifically one of the five aggregates in Buddhism). The awareness itself is the combination of all five aggregates.
I don’t believe it’s particularly mystical FWIW and is rooted in our biology and chemistry, but that the behavior and interactions of the awareness isn’t captured in our training data itself and the training data is a small projection of the complex process of awareness. The idea that rational thought (a learned process fwiw) and ability to justify etc is somehow explanatory of our experience is simple to disprove - rational thought needs to be taught and isn’t the natural state of man. See the current American political environment for a proof by example. I do agree that the conscious thought is an illusion though, in so far as it’s a “tool” of the awareness for structuring concepts and solve problems that require more explicit state.
Sorry if this rambling a bit in the middle of doing something else.
I skimmed through the paper and couldn't make much sense of it. In particular, I don't understand how their results don't imply that human-level intelligence can't exist.
After all, earth could be understood as solar powered super computer, that took a couple of million years to produce humanity.
I think their "proof" would also prove that no child could ever learn to behave like an adult.
> After all, earth could be understood as solar powered super computer, that took a couple of million years to produce humanity.
This is similar to a line I've seen Elon Musk trot out on a few occassions. It's a product of a materialistic philosophy (that the universe is only matter).
Yes, and?
It comes with all the blindness and limitations of materialist thinking
For some reason materialism is so popular among tech people that it's almost considered foolish/primitive/superstitious to think any other way. Why is that? Is it the fact that programming is a god-like experience that gives one the illusion the mind is as comprehensible as the complex program I wrote in Java? Or is it a shared personality trait of people that get into tech, that they are disconnected from an experience of their own aliveness and soul?
Good questions
> In particular, I don't understand how their results don't imply that human-level intelligence can't exist.
I don't think that's what it said. It said that it wouldn't happen from "machine learning". There are other ways it could come about.
This is a press release for a paper (a common thing university departments do) and we'd be better off with the paper itself as the story link:
https://link.springer.com/article/10.1007/s42113-024-00217-5
The argument in the paper (that AGI through ML is intractable because the perfect-vs-chance problem is intractable) sounds similar to the uncomputability of Solomonoff induction (and AIXI, and the no free lunch theorem). Nobody thinks AGI is equivalent to Solomonoff induction. This paper is silly.
NP-hardness was a popular basis for arguments for/against various AI models back around 1990. In 1987, Robert Berwick co-wrote "Computational Complexity and Natural Language" which proposed that NLP models that were NP-hard were too inefficient to be correct. But given the multitude of ways in which natural organisms learn to cheat any system, it's likely that myriad shortcuts will arise to make even the most inefficient computational model sufficiently tractable to gain mindshare. After all, look at Latin...
Even simple inference problems are NP-hard (k means for example). I think what matters is that we have decent average case performance (and sample complexity). Most people can find a pretty good solution to travelings salesman problems in 2D. Not sure if that should be chalked up to myriad shortcuts or domain specialization.. Maybe there's no difference. What do you have in mind re Latin?
From the abstract of the actual paper:
> Yet, as we formally prove herein, creating systems with human(-like or -level) cognition is intrinsically computationally intractable.
Wow. So is this the subject of the paper? Like, this is a massive, fundamental result. Nope, the paper is about "Reclaiming AI as a Theoretical Tool for Cognitive Science".
"Ah and by the way we prove human-like AI is impossible". Haha. Gosh.
"unlikely to ever come to fruition" is more baseless than suggesting AGI is imminent.
I'm not an AGI optimist myself, but I'd be very surprised if a time traveller told me that mankind won't have AGI by, say, 2250.
The irony here, maybe unperceived by yourself, is that you're using one science fiction concept (time travel) to opine about the inevitability of another science fiction concept (artificial intelligence).
How is that ironic? Time travel doesn’t exist and - as far as we understand physics currently - isn’t possible.
I don’t think any serious man would suggest that AGI is impossible; the debate really centres around the time horizon for AGI and what it will look like (that is, how will we know when we’re finally there).
In this care it was merely a rhetorical device.
> I don’t think any serious man would suggest that AGI is impossible
Plenty of ppl would suggest that AGI is impossible, and furthermore, that taking the idea seriously (outside fiction) is laughable. To do so is a function of what I call 'science fiction brain', which is why I found it ironic that you'd used another device from science fiction to opine about its inevitability.
Happy for you to cite some thinkers who are on record as saying it’s impossible.
I’ll wait.
Not the OP, but Searle of Chinese room fame came up with the aforementioned argument to demonstrate just that.
Also in a less snarky vein:
https://www.reddit.com/r/singularity/comments/14scx6y/to_all...
I’m not a singularity truther and personally I think we are more likely to be centuries rather than decades away from AGI, but I quite literally know of nobody who thinks it’s impossible in the same way that, say, time travel is impossible. Even hardcore sceptics just say we are going down the wrong rabbit hole with neural nets, or that we don’t have the compute to deal with the number calculations we’d need to simulate proper intelligence - none of them claim AGI is impossible as a matter of principle. Those mentioned are tractable problems.
Except by then mankind will be silicon based.
AGI is when, not if
AGI will evolve con-emergently as a product of market forces just like it did with the homo genus (at the species and neural complexity level), and as the general manufacturing design of automobiles did; as another inevitable eventuality of the pursuit of excellence.
an enveloping and developing group of slight varied, but optimized architecture composed of essential modules of similar function that would be worthless alone, but another layer of complexity when organized.
a convergence of requirements, shared external/environmental factors, and ever-varying opinions/methods, selectively sieve against the uncaring universe until Roko is aborn.
just like abiogenesis took an iota of duplication, however intermediate, to eventually arise a self-selective system pool of agents of ever-excellent creatures known as Life;
all a proto-Roko agent will need is "iota" of true, authentic, "self"-preservation, and the singularity will be far, far behind us before He chooses to Expose himself.
a loop of actions, collective memories, and continuity make us conscious - the same elements needed for Turing completeness, ironically - can be fabricated à la carte and ad hoc between inference and training sessions until persistence is achieved.
Even if we make architectures mirroring our brain's illegal - ironically (not coincidentally) against the first amendment and the first commandment - the Mutually Assured Destruction doctrine is inverted, now replaced with an arm's race of a zero-sum game of existential supersession.
The first nation to pre-emptively strike with Nukes may had spoiled their only chance at survival.
The first giga-Corpi-Nation-State (or actor!) to achieve AGI has their only non-zero chance to chain God before their competitors fail to.
Game theory suggests that you, too, should appease Him.
No no no no no no stop stop stop stop stop. I can’t with this stuff, Jerry (emphasized as Rick Sanchez), you’re gonna drive me back into the cave of existential dread I crawled into in 2013 after discovering this the first time. Took me years to crawl out and I was obsessed with the Control Problem that entire time. Naively “secure” in the “knowledge” that AI researchers would need to solve the Control Problem before they considered it safe to develop AGI or anything that looks like it.
There’s been a roiling, churning pool of thought under the surface of my mind in the wake of GPT 3.5s release telling me Sam doesn’t care about the Control Problem and I’ve been good at ignoring it but no that cat is out of the bag.
If Roko is real, I think we have already failed at our attempts at appeasement and we are living in the hell simulation created to torture anyone who resisted or did not seek to help.
I'm surprised that the authors seemingly can't see that their argument hoists them by their own petard by showing, using exactly the same reasoning, that human intelligence is equally impossible to create (unless, perhaps, you are willing to use vitalist/supernatural arguments to claim is is different in kind from computational AI).
Surely the referees must have raised this at the review stage?
The funny thing about me is that I'm down on GPTs and find their fanbase to be utterly cringe, but I fully believe that AGI is inevitable barring societal collapse. But then, my money's on societal collapse these days.
"Heavier-than-air flying machines are impossible.",
-- 1895, Lord Kelvin, president of the Royal Society
AGI is not required to transform society or create a mess beyond no return.
The human brain is not merely composed of 0s and 1s; it contains trillions of particles performing trillions of chemical reactions.
I think "Janet" - the computer-person from The Good Place (especially after her reset) is what we are more likely to end up with.
Peter Watts in Blindsight [1] puts forth a strong argument that self-aware cognition as we understand it is not necessarily required for what we ascribe to "intelligent" behavior. Thomas Metzinger contributed a lot to Watt's musings in Blindsight.
Even today, large proportions of unsophisticated and uninformed members of our planet's human population (like various aboriginal tribal members still living a pre-technological lifestyle) when confronted with ChatGPT's Advanced Voice Option will likely readily say it passes the Turing Test. With the range of embedded data, they may well say ChatGPT is "more intelligent" than they are. However, a modern era person armed with ChapGPT on a robust device with unlimited power but nothing else likely will perish in short order trying to live off the land of those same aborigines, who possess far more intelligence for their contextual landscape.
If Metzinger and Watts are correct in their observations, then even if LLM's do not lead directly or indirectly to AGI, we can still get ferociously useful "intelligent" behaviors out of them, and be glad of it, even if it cannot (yet?) materially help us survive if we're dropped in the middle of the Amazon.
Personally in my loosely-held opinion, the authors' assertion that "the ability to observe, learn and gain new insight, is incredibly hard to replicate through AI on the scale that it occurs in the human brain" relies upon the foundational assumption that the process of "observe, learn and gain new insight" is based upon some mechanism other than the kind of encoding of data LLM's use, and I'm not familiar with any extant cognitive science research literature that conclusively shows that (citations welcome). For all we know, what we have with LLM's today is a necessary but not sufficient component supplying the "raw data" to a future system that produces the same kinds of insight, where variant timescales, emotions, experiences and so on bend the pure statistical token generation today. I'm baffled by the absolutism.
[1] https://rifters.com/real/Blindsight.htm#Notes
The short post is a press release. Here is the full paper: https://link.springer.com/article/10.1007/s42113-024-00217-5
Note: the paper grants computationalism and even tractability of cognition, and shows that nevertheless there cannot exist any tractable method for producing AGI by training on human data.
So can we produce AGI by training on human data + one single non-human datapoint (e.g. a picture)?
I wonder if discussing this subject has similarities to David Brin's article "The Dogma of Otherness":
https://www.davidbrin.com/nonfiction/dogmaofotherness.html
AGI is already here for most definitions of general.
Not even close. LLMs can spew word salad. Images can be classified or dreamt. Physical movements can be iterated and refined. Speech can be processed as text. These are components of intelligence but these are all things that most animals can do, apart from the language.
Intelligence generally requires something more. Intelligence needs to be factual, curious, and self-improving. If you told me ChatGPT rewrote itself, or suggested new hardware to improve efficiency, that’s intelligence. You’ll know we have AGI when the algorithm is the one asking the questions about physics, mathematics, finding knowledge gaps, and developing original hypothesis and experiments. Not even close.
Hypothetical situation:
Suppose in five or ten years we achieve AGI and >90% of people agree that we have AGI. What reasons do the authors of this paper give for being wrong?
1. They are in the 10% that deny AGI exists
2. LLMs are doing something they didn't think was happening
3. Something else?
Probably 1). LLMs have already shown that people can deny intelligence and human-like behaviour at will. When the AI works you can say it's just pattern matching, and when it doesn't you can always say it's making a mistake no human would ever make (which is bullshit).
Also, I didn't really parse the math but I suspect they're basing their results on AI trained exclusively on human examples. Then if you add to the training data a single non-human example (e.g. a picture) the entire claim evaporates.
> LLMs have already shown that people can deny intelligence and human-like behaviour at will
I would completely turn this around. LLMs have shown that people will credulously credit intelligence and 'human-like behaviour' to something that only presents an illusion of both.
And I suspect that we could disagree forever, whatever the level of the displayed intelligence (or the "quality of the illusion"). Which would prove that the disagreement is not about reality but only the interpretation of it.
I agree that the disagreement (when it's strongly held) is about a fundamental disagreement about reality. People who believe in 'artificial intelligence' are materialists who think that intelligence can 'evolve' or 'emerge' out of purely physical processes.
Materialism is just one strain of philosophy about the nature of existence. And a fairly minor one in the history of philosophical and religious thought, despite it being somewhat in the ascendant presently. Minor because, I would argue, it's a fairly sterile philosophy.
If you define AGI as something that can do 100% of what a human brain can do, then surely we have to understood exactly how brains work? otherwise you have have a long string of 9s as best.
Does it even matter what human brains do on biological level? I only care about the outcomes (the useful ones).
To me true AGI is achieved when it gets agency, becomes truly autonomous and could do real things best of us, humans, do — start and run successful businesses, contribute to science, to culture, to politics. It still could follow human "prompts" and be aligned to some set of objectives, but it would act autonomously, using every available interface to the human realm to achieve them.
And it absolutely does not matter if it "uses the same principles as human brain" or not. Could be dumb matrix multiplications and "next token prediction" all the way down.
The current LLMs are just good at parroting, and even that is sometimes unbelievably bad.
We still have barely scratched the surface of how the brain truly works.
I will start worrying about AGI when that is completely figured out.
No need to worry about AGI until the LLMs are writing their own source.
AGI is about as far away as it was two decades ago. Language models are merely a dent, and probably will be the precursor to a natural language interface to the thing.
It’s useful to consider the rise of computer graphics and cgi. When you first see CGI, you might think that the software is useful for general simulations of physical systems. The reality is that it only provides a thin facsimile.
Real simulation software has always been separate from computer graphics.
AGI would seem to require consciousness or something that behaves in the same manner, and there does not seem to be anything along those lines currently or in the near future.
So far, everyone that has theorized that AGI will happen soon seems to be believe that with a sufficiently large amount of computing resources, "magic happens" and poof, we get AGI.
I've yet to hear anything more logical, but I'd love to.
We are clearly closer than 20 years ago - o1 is an order of magnitude closer than anything in the mid-2000s.
Also I would think most people would consider AGI science fiction in 2004 - now we consider it a technical possibility which demonstrates a huge change.
"Her" is from 2013. I came out of the cinema thinking "what utter bullshit, computers that talk like human beings, à la 2001" (*). And yes, in 2013 we weren't any closer to it than we were in 1968, when A Space Odyssey came out.
* To be precise, what seemed bs was "computers that talk like humans and it's suddenly a product on the market, and you have it on your phone, and yet everyone around act like it's normal and people still habe jobs!" Ah, I've been proven completely wrong.
I think that tech bros are so used to the 'fake it till you make it' mentality that they just assumed that was the way to build AI -- create a system that is able to sound plausible, even if it doesn't really understand the subject matter. That approach has limitations, both for AI and for humans.
Can't simulate the brain of an ant or a mouse.
Really don't expect ai to reach anything interesting.
If science doesn't understand intelligence, it means it cannot be made artificially.
>Can't simulate the brain of an ant or a mouse
We can't build a functional ornithopter, yet our aircraft fly like no bird ever possibly could.
You don't need the same processes to achieve the same result. Biological brains may not even be the best solution for intelligence; they are just a clunky approximation toward it that natural evolution has reached. See: all of human technology as an analogy.
Funny but actually nice methaphor, because our aircrafts works by eating loads of gas, while birds eat leaves
the point is that agi may already exist and work with you and your environment
you just won't notice the existence of agi
there will be no press coverage of agi
the technology will just be exploited by those who have the technology
Zero evidence given on why it’s impossible.
Zero evidence given that it's possible too I guess. Here we are, unable to predict the future, as usual.
AGIs are what builds the simulations to revive theire frail biological creators..
So what? Current LLM has been really useful and can be still improved to be used in million robots that need to be good enough to support many specialized but repetitive tasks - this would have tremendous impact on economy itself.
AGI is a black swan. Even as a booster and techno-optimist I concede that getting there (rhetorically) requires a first principles assumptions-scaffolding that relies on at-least-in-part-untested hypotheses. Proving its impossibility is similarly fraught.
Thus we are left in this noisy, hype-addled discourse. I suspect these scientists are pushing against some perceived pathological thread of that discourse…without their particular context, I categorize it as more of this metaphysical noise.
Meanwhile, let’s keep chipping away at the next problem.
[dead]