Sabine makes a point about the limits of LLM AI -- it can only go as far as human knowledge has achieved, an LLM cannot deduce something it doesn't have underlying data for. This suggests LLMs will never be more than just a tool. And you know, Ilya tries to talk around how LLMs work (through statistical selection) to jump to the conclusion that LLMs will do something they aren't designed to achieve...unless that's how real intelligence works, that is.
@gkmizuno Well the thing is, there is a lot of signal that we haven't pulled out of the data that we have already.
Even in current methods we employ grad students to sift through old data looking for new ideas, new messages, new processing to figure things out. That's nothing groundbreaking.
So that's not much of a limit for AI. We know there is a lot of gold in the tailings that human knowledge has amassed. There's a lot of information in the data that we haven't pulled out yet. Currently we try to manually find it, and that's a lot of fertile ground for AI to make advancements.
@gkmizuno I'm saying the second.
We have so much data right now that we can't analyze it in time. In fact we're throwing away a lot of data because we don't think it's worth storing because we won't be able to analyze it anytime soon, so it's not worth the cost of storage.
We are currently awash in data that we can't analyze because we don't have enough ability to analyze it. Maybe AI can help fill that gap, finding patterns in the data that humans don't have the time or energy to tease out.
@volkris Are you saying we don't have enough data or that we don't have enough tokens/processing channels concurrently analyzing the existing data to find the optimal statistical responses?
For simplicity, let's say you have 4 bits of binary data; you only need 2^4 tokens, right?
So, there is a hard limit -- a ceiling -- to such an approach: the amount of data you have.
But you're still putting the cart before the horse; humans still have to develop the data before an LLM can read it.