Future Timeline

I realise we have the proto-AGI and a couple of related threads, but I wanted to discuss this topic specifically. I'll probably lock it at some point.

So, below is a little animation I put together. It shows GPT-1, 2, and 3, with varying exponential trend lines depending on GPT-4.

For those who might be new to this forum, and/or unfamiliar with AI language models, GPT is a series of programs that uses deep learning to create human-like text. Think of them as like very, very advanced forms of smartphone autocorrect. The most recent of these, GPT-3, emerged in June 2020 and has demonstrated phenomenal capabilities, with dialogue that often seems like a real person.

This is largely thanks to its massive number of parameters, which can be thought of as like the individual synapses in a brain. GPT-3 has orders of magnitude more parameters than its predecessors. The program isn't perfect, however.

An even more advanced version, GPT-4, is strongly rumoured to be releasing this year. Estimates of the parameter count vary wildly – ranging from those who believe it won't be much larger than the 175 billion of GPT-3, to those who predict another huge leap with perhaps 100 trillion or more.

Rather like the "megahertz myth" of the early 2000s, and the qubit claims of D-wave Systems, it may be that we're reaching a point where large gains in parameter counts don't actually matter as much. Perhaps other factors will now be more important in determining how productive and efficient these language models are.

Anyway, the exact number of parameters that will feature in GPT-4 is publicly unknown, but I wanted to get the opinions of FT forumers. What do you think it will be? And does it even matter that much? Please vote in the poll I'm about to create, thanks!

If it's dense, probably 1 trillion at max. A dense 1 trillion GPT-4 sounds terrifying. But even if it's still only around 175 billion, considering how much more efficient scaling has become since 2020, it'd be ridiculously too good.

If it's a mixture of experts/MoE, than maybe, just barely maybe, it could be well over 10 trillion. Though it'd not be nearly as strong.

Something in my gut says just above 1 trillion. Maybe 1.5 or 2 trillion.

If they want to make a bold statement about 3 years after GPT3, being over 1 trillion rather than below it sounds good for PR. OpenAI is probably above that kind of logic in decision making but it is a possible influence.

Stacking on more and more parameters is only practical up to a point. Can't say what that point is because I don't know shit about AI but 300-500 billion sounds reasonable for GPT4.

As it stands I'm already very impressed with GPT3 so GPT4 will be sure to blow people's minds.

As it stands I'm already very impressed with GPT3 so GPT4 will be sure to blow people's minds. It seems crazy that just 10 years ago chatbots were utter crap (anyone remember cleverbot? )and now we have ChatGPT.

And what are parameters?

ººº wrote: ↑Mon Jan 02, 2023 8:15 pm And what are parameters?

Simply put, parameters in machine learning and deep learning are the values your learning algorithm can change independently as it learns and these values are affected by the choice of hyperparameters you provide. Some have equated parameters to our own brain synapses if it's easier to visualize the concept this way.

Learn more here: https://towardsdatascience.com/paramete ... 609601a9ac

Jakob wrote: ↑Mon Jan 02, 2023 8:12 pm As it stands I'm already very impressed with GPT3 so GPT4 will be sure to blow people's minds. It seems crazy that just 10 years ago chatbots were utter crap (anyone remember cleverbot? )and now we have ChatGPT.

I remember very clearly. I remember creating whole threads talking about how unimpressive Cleverbot was on this very forum's papa version (I'm pretty sure you could even find that thread archived somewhere). I distinctly remember the frustration of the JOHN Test, where I tried getting Cleverbot to rationalize how to spell John:
"JHN. Where do you put the O to spell John?"
And it ALWAYS failed. It failed. Another chatbot that Kurzweil made (Ramona, I think it was?) failed even harder somehow. A deep reinforcement learning-based chatbot I tried out in 2016 or 2017 failed spectacularly. It was astounding to me that these chatbots could be thwarted by an impossibly simple task.

The fact ChatGPT succeeded on its first try blew my mind. And then it went above and beyond when I asked it "WHY does the O go between the J and H?" and it responded logically that the O is a vowel and that placing it elsewhere would cause the name to be pronounced differently.

Just.... what!!!! WHAT!!!!!!!!

Also, when I tried to find answers to questions in different chatbots, I was shocked by the accuracy of GPTchat, to which question it found a reasoned answer. I wonder if the new version will have the ability to create illustrative stories, because I have dreamed of creating a comic since childhood.

Welcome to the forums Hersen! Perhaps we should rename this thread? Common consensus was that GPT-4 has around 1.7(?) trillion parameters in a mixture of experts style architecture.

How many parameters will Orion have? Or the full version of GPT 01 which will train synthetic data for this model for that matter? The fact that it's been barely over a year and a half since GPT-4's release and yet we're already discussing models many times more powerful shows the exponential curve we're on.