@pre

Yeahhhhh, but... So far it's sort of like "find ways to improve my toaster until it's good enough to figure out how to murder everybody."

There just isn't any plausible path to doing that.

And in the meantime the toast might be burning or something. Or maybe we're out of milk.

There is just *so much* that is more worth investing resources in than "AI might kill us all in some way that we haven't thought of, because eek".

cc: @emilymbender

@ceoln

If global industry was investing trillions of dollars on trying to build a toaster big enough to toast the planet I think it might be reasonable to suggest not doing that.

🤷

@pre
Oh, sure, I believe it would be good if certain big AI companies stopped doing various things that they're currently doing.

But that isn't the thrust of the current AI Risk hype; that's more like "toasters are going to get big enough to toast the planet, so we experts must be put in charge of all the toasters and be given lots of funding to figure out how to make giant toasters emit less heat!".

@ceoln That sounds like a staw man to me. Nobody thinks GPT4 is going to murder everybody.

But how's this for a stab at how GPT-20 might: Catfish lots of horny young men with fake nudes, use the money to bribe terrorists to screw around with nukes.

@pre

"Catfish lots of horny young men with fake nudes, use the money to bribe terrorists to screw around with nukes."

Do you think that's the kind of risk that the CAIS letter is talking about?

It's possible that GPT-20 might do that, but it's also possible that all of the air molecules in this room might suddenly gather in the other half and I'll suffocate.

How much resources should we put into preventing either of those things?

@ceoln The people who signed that letter did so for all sorts of reasons. Some of them are indeed worried about that kind of risk, plus the billions of other possible things which mere humans have thought of and discussed for years, let alone what a smarter-than-human machine could do.

The money isn't currently being spent on preventing a thing, it is being spent on research and development of the thing.

What proportion of R&D costs do you think should go towards ensuring safety? Is it higher or lower than we currently do?

@pre

Fair enough; the letter was short enough that people probably signed it for all sorts of reasons.

"What proportion of R&D costs do you think should go towards ensuring safety?"

That's the key question; "safety" in what sense?

I see this as a tension between two ideas:

Worry about safety against the actual current risks of power imbalance, exploitation of less privileged people to enrich more privileged ones, spread of disinformation, and so on, toward which I think we should put lots more resources, attention, etc,

vs

Worry about how to prevent humanity from going extinct from unspecified causes before we have trillions of simulated humans running in computers the size of the Moon, addressed by #TESCREAL people, mostly very privileged ones, holding conferences on tropical islands to present to each other extremely speculative papers on the subject. On which I don't think we should spend very much at all.

Those are two very different things.

@ceoln Well there's those straw men again 😆

That aside, we have no idea how to solve for AI being more fair on the less privileged and we also have no idea on how to prevent it deciding we are using the resources it needs and so murdering us to keep the coal for itself or whatever.

I would suspect that if you solve either you solve both myself.

@pre

Is it a straw man? Have you read about EA and Longtermism? I hope it's merely straw! But the evidence does not seem to me to support that conclusion.

But no, I don't think that if we solve for either, we solve both; I think we should at the very least not assume that, and actually work on the more important one rather than the more thrilling one.

I think we don't particular need to prevent AI from making humanity extinct by say taking all the resources to accomplish its goals, because it's not going to do that anytime soon at all; that is not a real danger.

I also think that we know of quite a few things that would make AI more fair for the less privileged (see everything written by https://www.dair-institute.org/ for instance), but there are lots of people who don't want to do them, and would like to distract from the idea by (for instance) focusing attention on these longtermist things instead.

And some of those people signed that letter.