Absolutely
A couple of posts I've done on Reddit:
_____________
I've been using ChatGPT since 2022. I still fiddle with it a bit for some non-plot bunnies just for cheap-o discussion/note analysis, except not even that anymore because of a major reason:
It fucks it up
every. Single. Time.
My favorite thing to do with Chat now is to see how its commonsense reasoning utterly
implodes at tasks even a 2 year old could figure out. It's actually incredibly easy to do this, and it makes me
severely skeptical of the whole "vibe coding"/"vibe [x]-ing" trend. Chat, and other LLMs, will
consistently output content that looks good. It seems even professional.
Until you actually read it. Which I do; I tend to listen to everything I read, so almost
immediately there are
severe logical problems. Sometimes it's just down to its context window, where it fails to parse data because there's too much. But very often, it's not that at all. You give it an instruction to do something ultra basic, it fucks it up. You ask it to figure out how it fucked up, it can't at all and winds up running in circles,
devolving in coherence over time, panicking and flailing desperately to output anything that sounds right. You track its "logic" in gasping awe as it you witness it commit to 36,360° circles around the actual commonsense answer because it lacks any sort of grounded state or adversarial agency that could act as a fact checker. You might wind up spending a full hour screaming at it to just use
basic 3-year-old-tier logic to figure out a problem that is obscenely obvious to you,
and a three year old, even without access to the correct information. Except it
does have access to the correct information and yet refuses to use it. You are all-capsing at it to stop being stupid and figure out why someone who is 28 years old is
not older than someone who is 33. And then it
fails. Then you retry the output and it succeeds. Then, curious, you reroll it again.
And it fails again.
It is
outrageous how unintelligent ChatGPT is when you really start testing it for commonsense reasoning.
This is what really hinted to me that the Attention-based Transformer was not the direct path to AGI and the mania that got kicked up for scaling them is going to end catastrophically
at best.
If not for China keeping genuine AI research going, I would not hesitate to say that the AI bubble bursting could very well lead to
nuclear AI winter, just due to the sheer negative reputation AI had developed that wound up leading to people shitting on or even denying the existence of good AI research.
I could even tell you why generative AI could have led to good stuff but we got caught up on a single architecture whose limitations were literally well known and was never intended to be the endpoint in and of itself.
It's just frustrating. So much good could be done, but now AI and robotics has become a byword for everything wrong in modern times when it shouldn't be, and eventually the Potemkin village will fall over and collapse.
_________
Post 2
Transformers are inherently stateless as a rule, and our heavy handed ways of grounding them is too ineffective to be of any use. Almost all use of LLMs doesn't require exploring the breaking points of commonsense reasoning and logical deduction, so it’s extremely easy to be fooled by their outputs into thinking they're closer to AGI than they appear.
Attempting to get an LLM to think logically always collapses into statistical probability. This can mimic intelligence but it differs extraordinarily because it's predicting the most likely next token in a sequence without actually checking to see if it's the correct one.
Typically most likely
is correct for most casual instances. The edge cases outside of distribution are where the models completely collapse.
Unlike fluid intelligence which can form abstractions to predict out-of-data distribution, transformers don't "know" that they're out of distribution and keep assuming everything they predict remains the most accurate output due to most likely prediction.
They possess extremely rudimentary understanding of concepts through statistical pattern correlations in their weights.
With a proper tree search algorithm, they could more effectively ground concepts between higher and lower dimensional states.
(i.e.
High-dimensional states = rich vectors storing many independent semantic features at once;
low-dimensional states = compressed bottlenecks for routing and final decision-making)
Scaling up increases the corpora from which it can draw to make predictions. Time test compute can sort of brute force a kind of search. Thing is, there are more effective ways to do both, and limits to both as well.
I'm not saying I know for sure how to create AGI. Just that there are clear flaws to relying solely on transformers, and these flaws have been known for a long time.
The reason why the field is so sure they're the way comes down more to a bet that you can brute force out-of-distribution generalization and capability with big enough data and enough compute to power agents.
That's my response to /u/SgathTriallair as well. It makes logical sense why the labs are betting big on scale. It's not like they don't have some operational logic.
My point is more that a lot of this momentum was started by legit hysteria. Everyone points to ChatGPT, but the actual turning point came half a year earlier, with DeepMind's Gato. That was the first model in history that displayed signs of true generalization as a result of multimodal training. So the hypothesis is that, through enough scaling and raw brute force, you can get to AGI. This isn't my guess of it; this is
literally the operational logic, they're open about this, that's why they're doing what they're doing and have never shied away from this being why we're seeing the current paradigm pushed. Because ostensibly, it doesn't matter
how you get to AGI, even if it's through the most blisteringly inefficient method. As long as you get there, the AGI will optimize itself.
It's like it doesn't matter
how you create a black hole (you can create one by compressing
any matter or energy into a dense enough point, even if that energy is light or the electromagnetic force); once you create one, it has no hair.
The problem in retrospect is that scaling this up wasn't easy and the generalization could actually be explained as the result of multimodal data prediction
mimicking generalization. The result being that the American AI labs fanatically maniacally rushed to exploit the first possible sign of AGI without really testing extensively to see if this was
actually smoke from fire or actually just vapor coming off lukewarm water.
Even
one of the co-creators of transformers has come out and said what most people in 2021 were saying, that the extreme focus on transformers has not
actually resulted in anything materially better than what we had back then, but picking the low hanging fruit of what transformers can do made us think otherwise because there
was a lot of fruit to pick.
Source
2
Because again, transformers
are important.
A tokenizer can be used to organize the data of a proper AGI system. It's arguably the most important part of it. But it's like pretending the ICE
is the car, or the mitochondria
is the cell.
This is why I said China is likely ahead. They aren't neglecting neurosymbolic and neuromorphic AI research. In fact, one of the most interesting AI news of the entire year was their "Darwin Monkey" computer. It could very well be that slow and steady wins the race after all.
To that end, machine learning is important to reach AGI, but my philosophy is there's still a bit more beyond it. Probably the biggest reason why a lot of the big names aren't budging is because some of the proponents of those other architectures (Yann LeCun, Gary Marcus) are assholes. Even if they're assholes with a point.
LeCun outright says that you need transformers to get to AGI, but it would be wrapped by a neurosymbolic system. I believe Demis Hassabis also has a point with a Tree Search tool added to it.
Edit:
This
tweet should be seen as ground zero for the Generative AI bubble. It was at this point, this
exact point that the American AI field began obsessing over transformers and scale as the path to AGI, because of the tease of generalization Gato gave us.
And to the idea that surely the AI labs wouldn't be so stupid as to invest this much time and effort into something they know can't reach AGI, you'd be surprised. Even Demis Hassabis, for a brief moment in time, claimed that AGI was less than 5 years away, and that tone shift came only after Gato and the hype it caused, that was supercombined into the ChatGPT explosion. Then
that was further compounded with the discovery of reasoning.
Ironically
, LLM reasoning dates as far back as 2020
That's the cold ugly truth of the matter: we
think there's way more AI progress happening than might be true under the surface because we're simply seeing what token prediction can do. It's literally like inventing the internal combustion engine and cranking up its power, and being surprised that it can cause things to move faster or even lift off the ground. But there's a reasonable limit to its power, and once you've created a 10,000 pound ICE, you start hitting diminishing returns before you get to the point you can attach it to your car and fly to the moon.
That's this. There was a crazy amount of stuff LLMs
could do that we've been discovering, but because it resembles the emergence of general AI, it gets obfuscated that what we're actually doing it prodding the capabilities of transformers, not actually constructing AGI systems.
Now if you
change tactics and
design, things will shift accordingly.
__________
This is the same reason why I'm completely unimpressed with Gemini 3. The higher level stuff— you know, vibe coding simulations and small games, that's almost expected. That sort of STEM-heavy stuff seems to be what LLMs are currently aimed towards, so it's become decreasingly amazing to see them do anything that isn't a high level program that requires using something like C or C++ or Assembly
Because then I try to use Gemini for those plot bunnies
And it just fucks it up
every. Single. Time. Even now.
It's reaching a level I could comfortably describe as an "uncanny valley." Somehow it's able to do somewhat impressive coding tasks, it can intelligent search a data corpus to make minor scientific discoveries, it can do all this and that, but give it a commonsense reasoning task telling it that if a person is in one place and then they're in another place an hour later, what is the logical assumption about what happened in that hour, and it's a complete crapshoot if it guesses "they traveled to the second place" or tasks like that
"This character lived in place A in X year, so what does that mean about when that character lived in place B in Y year? (read the document I gave you, it has the answer"
A:
most logically incoherent gobbledeebook you've ever heard
Me: "That's stupendously incorrect. Try again, here is the literal quote, copy pasted
A:
doubling down on the incorrect answer, gets even more ass-backwards incorrect
Me: You incompetent fucking clanker!!
Commonsense reasoning is one of those signs of general intelligence, at least to me. The fact all current LLMs suffer so catastrophically and magnificently at logical deduction tells me they all still are horrifyingly ungrounded and misaligned.
But in the short term, this is why I think there's a bubble. The ELIZA effect causes us to overestimate just how genuinely "good" these AIs are
Being that LLMs are, ahem, LANGUAGE models, and we are language-driven creatures, that's not really surprising that we're so totally fooled
When the venture capitalists realize how little LLMs and diffusion models can actually do without extensive guidance (which is the opposite of what people expected AI to be used for, the AI themselves are supposed to be the automation, not human "prompt engineers"), that's when I think the curtains will drop
The reason why I don't think it will happen all at once is due to the same effect
The ELIZA Effect isn't the same as the futureshock of being able to order pets from some shady website or visiting a dinky Web 1.0 webforum and desperately trying to convince yourself you live in a William Gibson cyberpunk novel surrounded by flying cars and androids but it's also 1998 and you're on dial-up in a flat watching VHSes. It's an actual psychological effect that can make you far more irrational and think you're conversing with an actual genuine artificial human entity.
It works on ME every time I use ChatGPT or Gemini and wind up freaking out in a rage over these statistical probabilistic language models not having humanlike commonsense reasoning
So the bubble might not "pop" as much as it could "deflate" over a longer period of time
Which gives the alternative methods just enough time to get up and running (most likely in China, maybe DeepMind has something too, maybe even Ilya?) and that could make it seem like the bubble "never" burst, like we went from AI models that stole everything to produce barely passable slop, and then suddenly we got proto-general AIs that actually legitimately learn as they go
