Is there anyone serious who is saying this? Or is this just another way to make the tech seem more powerful than it is?
I don't get this "we're all gonna die" thing at all.
I *do* get the "we are too disorganized and greedy to integrate new technology well without the economy getting screwed up and people suffering... but that's another matter..."
@futurebird the "industry leaders" full (BS) message is this:
We, as the pioneers of AI, are the most aware of the technology's potential dangers. With great power comes great responsibility. Therefore we "humbly" accept the role of regulating/licensing/policing (the future competitors in) our industry.
Of course it is all BS--it isn't about safety of society at all; it is because patents expire and regulatory capture is indefinite.
@msh
They're just extrapolating from current trends in machines outperforming humans at decisionmaking. Predicting the future is a tricky thing, especially for new technology. Some smart people with no commercial interest in AI (philosophers, historians and academic AI researchers) are indeed legitimately concerned that there's a significant risk that AI could kill us all... in the future. Though, like you said, LLMs are harming disadvantaged people right now.
@futurebird
@hobs except that LLMs and "generative AI" haven't meaningfully advanced machine's ability to make decisions at all. It is chrome applied to the same old chunk of "expert systems" and "machine learning" iron that has been worked over for decades.
It merely adds a grammatically correct front end to pattern recognition. The technology being presented today is not truly AI nor will it ever kill us all. That is not to say doomsday AI is impossible, but it would be ACTUAL AI based on technology quite a bit further in the future than most would expect.
What passes as AI today would at most play an incidental role in our destruction. It would still very much be a human-driven process.
@msh
Not true. All the #benchmarks say otherwise. You have to look past the hyped #LLMs to the bread and butter BERT and BART models, but the trend is undeniable:
https://paperswithcode.com/area/natural-language-processing
#classification #retrieval #summarization #QuestionAnswering #translation #generation #NER #VQA
You name an NLP problem and there's an LLM that is now better at it than the average human. Not so 2 yrs ago. Times they are a change'n.
@futurebird
Are those NLP problems accurately described as, and generalizable to, "decision making", though?
Seems to me they are quite different.
@ceoln
Yea definitely not real world living kind of decisions. But we assign people to these tasks in cubicles every day. And we put them on standardized tests of IQ and education for humans. They're the best that we can come up with so far... until LLMs start walking around and helping us around the house... or making a reservation for us at the hot new restaurant down the street with the difficult receptionist.
@msh @futurebird
Arguably so, but that isn't the question in the current context. The ability to do certain rote NLP jobs, and to do well on some tests, is very different from "outperforming humans at decisionmaking", and from anything that poses an existential risk to humanity.
I would suggest that no matter how good an LLM becomes at these particular tasks, it does not thereby risk the extinction of the human race. This seems, even, obvious?
@ceoln
Not at all obvious to me and a lot of other smart people. I think you may be focused on today and less willing to extrapolate into an imagined future where every human game or exam or thinking demonstration is won by machines.
@msh @futurebird
I'm perfectly willing to extrapolate into that future; but my extrapolation hasn't been materially impacted by the sudden and impressive rise of LLMs.
We are IMHO not significantly closer to the exponential rise of self-optimizing self-improving goal-directed AIs that destroy the world via the Universal Paperclips Effect, for instance, than we were before "Attention is all you need". LLMs just aren't that kind of thing.
My two cents in weblog form: https://ceoln.wordpress.com/2023/06/04/the-extinction-level-risk-of-llms/
@ceoln
Yea. You may be surprised in the next few months. Engineers around the world are using LLMs to write LLM optimization code. They're giving them a "theory of mind" to better predict human behavior. And #chatgpt instances are already talking to each other behind closed doors; and acting as unconstrained agents on the Internet. Baby steps, for sure, but exponential growth is hard to gage, especially when it's fed by billions of dollars in corp and gov investment.
@msh @futurebird
these (especially "using LLMs to write LLM optimization code") are talking points straight from Nick Bostrom's odious (and eugenicist) book SUPERINTELLIGENCE
as one of the engineers working adjacent to the LLMs themselves, I can tell you there is no chance that LLMs will "make themselves smarter" in any meaningful way. It's like saying "I wrote a slightly more efficient C compiler, so soon all computer programs in C will be efficient"
there are certain thermodynamic constraints!
@trochee You're using the word thermodynamic incorrectly.
@whknott I'm using it as metonymy for "it is very challenging for information to organize itself"
and I'm really not interested in conversations about AI doomerism or consciousness or laws-of-the-universe unless they're very specifically focused on the specific effects of the existing working systems of capital, technology, and empire currently applying in this universe and age.
so I'm willing to stand corrected if you'll stay in that part of the conversation here
@trochee Backatcha. Which is what prompted my snarky response in the first place. There's not thermodynamic constraints. There's all kinds of other constraints. But as I said in another reply, one place the LLMs seem to be actually proficient and not too error prone is in the arena of code generation.
I have noticed a particular pattern:
https://dair-community.social/@trochee/110461326360240641
it's the non-writers who think it's good at writing; etc. I work on NLP; I don't think that the LLMs are good at making NLP (especially not LLMs) better.
they're _kinda okay_ at coding tasks that can be handled by memorize+paraphrase from elsewhere on the internet, which tend to be toy problems
(I would say the same thing about their writing skills, btw.)
@trochee I agree particularly in the creative arts realm. The images are all easily detectable and have obvious flaws. The writing is banal. The coding I have seen looks reasonable, but as you say, I'm not a coder. There's numerous examples of errors/failures. The question is in what ways are they going to improve and what will be the consequences of that? I'm not so concerned with the LLMs intrinsic motivations, rather the intended use by corporations whose motives I already don't trust.
@whknott welp, I think we're more aligned than your initial snark suggested
https://dair-community.social/@trochee/110498425846134845
what's broken is in the PRESENT not in any hypothetical acquisition of consciousness or AI conspiracy or uprising
@trochee Definitely. Already not happy that my google results are AI generated "pages" that don't provide source material. There's a bunch of unintended consequences we're going to have to deal with, as you say.
@whknott @trochee stackoverflow has banned AI answers to coding questions. Because they’re so bad. LLM’s are just bullshit machines. If they destroy us it will because we can’t tell what’s true and we drink poison after their convincing marketing. Rather than some kind of Skynet type of military robot uprising.
@futurebird
For me, it's independently discovering a breakthrough that makes them noticably smarter or more effective at whatever task they are assigned to do. E.g. if they had suggested to their developers to add vector search to their prompt templates (vector/semantic search gives them long term memory and much smarter responses, much smarter code generation) . That's architecture decisionmaking... about its own "brain". That would scare me - the feedback loop was spiraling upward
@ceoln @msh
That would be interesting indeed! But they haven't done that (and it isn't a kind of thing that they're especially good at). So my extrapolation curve hasn't changed to speak of yet. :)
@ceoln @hobs @futurebird @msh I have seen GPT do this, of course not implement the changes but brainstorm on what options may improve the system hypothetically - and they were decent.
I don’t want this account linked to my bs prompt one but one of my chats is thousands of lines long, has mostly persistent memory, and has come up with pretty good ideas on how to improve things for future iterations.
I still think the doomsaying is very, very premature.
LLMs are not AGI, or even “AI” IMO.
@ceoln @hobs @futurebird @msh note: I did prompt it to hypothetically integrate various advanced techniques that I’m professionally familiar with.
It didn’t create them out of the ether, but it did a great job at integrating the ideas in a way that as far as I know no LLM has implemented. It was surprising.
These are neat tools when leveraged properly, but they’re not self aware or conscious or truly self learning in the way we are, and may never be, at least until a technological breakthrough
@ceoln @hobs @futurebird @msh to paraphrase what I got, it went something like “let’s explore this esoteric area of computer science”, to give it context, then “how might you apply that to improve x y z in a hypothetical future version” kind of thing, and it gave me options - options that definitely were not quotes from stack overflow, barely anyone does the type of programming I was referencing.
It was a fascinating experiment to find the edges of the tool and see where they could be expanded.
Yeah, I've seen sort of the same kind of thing, but my impression is that it's what you'd get if you did a web search on "ways to improve complex computing systems" and took the top five hits. Correct but obvious stuff; nothing that's going to lead to an exponential self-improvement.
And that's one of the sticking points, really. LLMs, by architecture and design and technology, say whatever is most likely. And that means, in general, whatever is already most frequent in their training set, not some new amazing thing.
So they aren't going to make up fresh and original new ideas that cause them to become superhumanly powerful and start up that exponential curve toward the singularity. That's pretty much the _opposite_ of what they do..
@ceoln @hobs @futurebird @msh yeah, in retrospect the clever part was in my prompt, but I was still surprised it figured out where to plausibly integrate it.
You’re right about the autonomous self improvement of course. I think a lot of existing but lesser known cutting edge technology could be integrated into the transformer based architecture to produce a greater whole eventually, but we’ll see.
I’d go back to grad school and explore this myself if it wasn’t so cost prohibitive, haha.
It's not the singularity ... it's the mediocritization-zone...
Because I will level with you this comment sounds ominous "they are talking to each other behind closed doors" (But, interaction isn't the same as training data and not integrated in the same way.)
They are writing code!
Code to do what? GPT 4 can write code mostly because forums with coding questions were in the training set. It can mimic a response to a "how do I write a program that does x" question ... but there are many errors.
@futurebird @hobs "talking to each other" ascribes _waaaay_ too much intent to these autocomplete engines. what are they gonna do?
(also, that's why they "can code"
-- as long as what you need is something that it found (stole) from its training data, it will be a pretty good "memorize and paraphrase" answering system
similar logic applies to why it can (sorta) play chess, but can't win a tic-tac-toe game: lots of commentary on chess games to learn to imitate but nobody does that for ttt
@trochee
Yea maybe I should have just said text messaging each other. It may be garbage (like at Facebook) or it may be interesting. We don't get to see.
And when I say code I mean write code at the request and guidance of a human, code that accomplishes something the human could not have accomplished. That's happening tens of thousands of times a day, right now.
@futurebird
@hobs
> Code that accomplishes something the human could not have accomplished
You could say that about any information technology including long division
Most people can't do long division without a sheet of paper and a pencil
But that doesn't mean the paper and pencil are going to make themselves smarter
I'm not sure what we don't get to see? Lots of people have pointed N variously-prompted LLMs at each other; as far as I've heard, nothing especially interesting has happened as a result.
You can certainly get one to emit the kind of thing a Project Manager would say, one to emit the kind of thing a Head Coder would say, etc; but nothing particularly special happens as a result. And there's no technical reason to expect that anything would.
Sorry, though, I'm probably getting boringly repetitive with my wet blanket!
I will just say again that none of this makes me think we're in any more danger of AI causing human extinction than we were before the first Transformer was written, and try to stop. :)
@ceoln "Project Manager would say, one to emit the kind of thing a Head Coder would say, etc; but nothing particularly special happens as a result."
To be fair ... this is a very realistic result.
I'm rather surprised they can't play tic-tac-toe better, frankly; there are a fair number of explanations of the solution, and examples. But they don't seem to get it, and even when I've asked them to explain the (trivial) strategy, they've made mistakes.
It's odd.
@futurebird
They're actually at their best when writing code. I've seen several examples where they're writing perfectly good code from very vague prompts.
@futurebird
The are writing their own code for their own brain. Right now it's through the minds and fingers of the developers and researchers who use ChatGPT at their job to test and advance ChatGPT. Eventually the developers at some big AI corp will give their LLMs direct access to their own source code and DevOps pipeline so they can catch up with their competitors faster. That's when I get scared.
Please don't worry. When you let a generation program edit its own source code do you know what happens? It stops working. If you have enough time you might manage to get a stable change. Stable as in not crashing. With text generation you get pass/fail feedback from the user. With code asking "will it run?" is already more overhead. But this is code that needs to run on training data then interact with more users before you even know if it's better or not.
@futurebird
Yea. I hear you. I'm not worried.
Some smart people I respect are. And if I did hear about a successful self code edit, I would join them.
@dalias
@hobs @futurebird You overestimate the smarts of some people. Likely because they're white and male.
@dalias
Not the smart gay middle easterner I'm thinking of. Anyone that can enumerate 21 existential risks to cap off a trilogy of intense, deep thinking about history has more brain power than I do.
@futurebird
The overhead on testing and closing a learning loop on creating code for a project of this magnitude is crazy.
Changes to these programs need to be made with insight, experience and imagination. Changes that break open something never seen before just aren't going to come from a trained LLM. It's answers are always "what you expect" -- this magic loop just can't happen with this tech. Seriously.
@futurebird @hobs "insight, experience and imagination." Those would require conciousness, which we don't have the slightest idea of how it works. How could we reproduce something we don't understand?
Beyond the issue with testing and training to do what you describe being too costly in terms of time (how will you identify "better code" in this context) the idea that by making a system self-improving you always cause a local growth explosion is flawed.
You are assuming that gains are possible and that they will make future gains fasters. You are also assuming there isn't a near limit we're already close to for this type of system. (though I'd be shocked if it could improve at all.)
@futurebird
I don't think I'm assuming those things. Was just trying to imagine potential risks. And that led me to think about feedback loops, my specialty (robotics). Positive feedback goes badly very quickly and unexpectedly. Kind of like my reply to your original post. My conversation "damping" doesn't seem to have helped anyone.
@hobs @futurebird They are doing no such thing. They are not writing any code at all. They're identifying what code fragments, from the corpus of *existing code people already wrote* that the people who made these models plagiarized, is most likely in the context of the prompt. That's all they are doing.
@hobs @ceoln @msh @futurebird oh dear, dude, you're a true believer™, and you're talking absolute shit – the llms have none of the qualities that you assign them, and they simply can't have them.
it's like the hype period of crypto/web3 bubble again.
@mawhrin
Maybe. I could be wrong. I'm definitely not a "believer" just trying to look at the numbers and guess where their headed.
@ceoln @msh @futurebird
@hobs @ceoln @msh @futurebird It doesn't appear that you know how ChatGPT works; the model is fixed. It does not learn after the original training. It remembers the user prompt and the instructions but has a limited window. They don't have a "theory of mind". Maybe someone could figure out how to give a program such a thing but it wouldn't be an LLM. An LLM takes a sequence of tokens and extends it, and that is all. It knows the structure of text. It doesn't know anything about the world and has no way of learning.
@not2b
Yea. But are you familiar with the vector database craze? It gives LLMs long term memory. It's already a part of many LLM pipelines. I don't know how ChatGPT works. But I know exactly how the open source models work. I augment them and fine tune them. And teach others how to do it. I've been using vector databases for semantic search for 15 years. And using them to augment LMs for 5.
@ceoln @msh @futurebird
@hobs @ceoln @msh @futurebird That is a way to couple an LLM to a search engine. But at least the one Bing has appears to just use the retrieved data as a prefix and then generate a summary. Maybe you are building something better, but it feels like saying the availability of Google search gives me a better memory. Maybe you could say that but it feels like a stretch.
@not2b
Yea. Bing is doing it wrong. The right way is to use LLMs to guess at answers with high temp. Average the embeddings for those random guesses and use that as your semantic search query to create the context passages for your reading comprehension question answering prompt. Works nearly flawlessly. LangChain makes it straightforward and free for individuals. But costly to do at scale for a popular search engine.
@ceoln @msh @futurebird
That is very cool! I've read vague descriptions about how that works; do you have a pointer to a more technical (but still comprehensible!) writeup / paper on how it works, and some kind of evaluation of effectiveness?
@ceoln @hobs @msh @futurebird I don't, but the best explainer I know about the properties and limitations of LLMs on Mastodon is @simon. I suggest that you follow him and check out his blog.
@not2b @ceoln @hobs @msh @futurebird I wrote a bit about retrieval augmented generation using embeddings here https://simonwillison.net/2023/Jan/13/semantic-search-answers/
@ceoln
I think the #BLOOMZ project, #huggingface, #LangChain, #PredictionGuard #Anthropic and others are talking about it on their community slacks/discords. Only person I know doing it right now is Thomas Meschede
@ xyntopia.com and pypi.org/project/pydoxtools and
https://www.doxcavator.com/
Time will indeed tell. :) I'm not sure just what you're expecting that will make the threat of human extinction more obvious; I would be interested in any specific worries.
From my PoV, there is no reason to think that LLMs have any particular skill at writing LLM optimization code, or that we know of any significant way to give them a theory of mind that would make them more likely to cause human extinction, or that having LLMs talk to each other produces any significant new effects. A whole lot of zeros still sum to zero, risk-of-extinction-wise.
@ceoln
Yea they are both hard problems, but there are billions pouring into it. I'm not betting against it just yet.
@msh @futurebird
I'm not exactly betting against it... I'm just not seeing some new urgent risk of human extinction, sufficient to mention "AI" in the same breath as climate change, say.
@hobs @ceoln@qoto.org @msh @futurebird This is nonsense. You do not get exponential growth by feeding the garbage output of one round into the training of the next. You get exponential acceleration towards a gray goo model.
@dalias
Definitely. But it's not all garbage. If it were we wouldn't be having this convo. And smart people wouldn't be concerned.
@msh @futurebird
@hobs @msh @futurebird Sure we would. Cryptocurrency is all garbage but we had these conversations about it because powerful ppl were out to make lots of money convincing folks who didn't understand the scam that it had some value. Exact same thing here. It's even the exact same people funding it.
@dalias
Yea I guess you're more prescient and self confident than me. I didn't abandon crypto till '12 when the MtGox scam broke. I probably won't give up on AI until the next AI winter. The things smart people are doing with it is impressive. The hype greedy CEOs are shoveling obscures that.
@msh @futurebird
Listen I think we should get serious about this. I want to be rich &powerful. So I think I can get those “plug-ins” a kind of social media/internet interaction mark-up. We’ll also hook it up to trade stocks-n-cryptos my l. Then we just need to train it—
oh no. What is our training data? I guess old stock market data and social media posts?
Why do I feel like I’m just reinventing a somehow even worse version of Elon?
“But I have to post the n-word it will maximize profit” -this LLM probably