Alberto Romero who writes some of the most knowledgeable and thoughtful commentary on AI has just posted "GPT-4: The Bitterer Lesson".
https://thealgorithmicbridge.substack.com/p/gpt-4-the-bitterer-lesson
Originally an idea credited to Peter Sutton, the "Bitter Lesson" is that "humans have contributed little to the best AI systems we have built".
And with each iteration of GPT-X that contribution is becoming less and less. From bitter, to bitterer.
Worth a read.
I'm not sure if you can have a #state #phase #transition with just #computation. You can definitely "scale up" some existing capability by ading more computational power, but can you get (evolve to) something radically new?
Absolutely. All of biological evolution is like that. Once you start selecting for something, and you have enough parameters, you will achieve it. LLM training and biological evolution are analogous in that respect.
What's rather surprising is the richness of emergent behaviour that we get from merely predicting the next token.
Apparently, training for language creates thinking as a byproduct.
🙂
I don’t know. #Language is a fairly new “improvement” in biological evolution, and I’m not sure you can reverse engineer (artificial) #intelligence from it.
You could argue that intelligence evolved before language. After all, you have quite a few intelligent animals with no language or with a very simple vocabulary.
@boris_steipe
Reading this from The Bitter Lesson by Rich Sutton:
researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
For me, the last sentence means that the real value of #tools like #AI for humans is that they can free us from tedious, repetitive, unimaginative work such as #computation in favor of more elaborate creative thinking.
People have always used previously developed more primitive tools to develop better ones. This was true for all tools and machines we invented so far, and is also true for #AI’s ability of “upgrading itself”.
