The only thing less-helpful than hyperventilating that "AGI is going to kill all humans" is the (far more common) claim that AI is inherently just a "fancy autocomplete", "stochastic parrot", or a "passing fad", and thus the challenges it presents are primarily just limited to the amplification of existing problems.
@ceoln Some thoughts:
* AI tech is advancing very rapidly. Substantial improvements are happening on 6-month timeframes. Opinions we previously formed may already be out-of-date.
* AI LMs are benchmarked by how well they score on SAT, law, and graduate school exams. Today they score better than the average human test-taker, though not quite as well as the top humans.
* Right now, you can hire this "thing" that can pass law and graduate school exams, and works 24/7/365, as your personal research or writing assistant, or teacher, for about $0.01 per page. We haven't even started understand the economic effects of that.
* There are reasons to suspect that multiple 10x improvements in performance and capability are ahead in coming years, and there's no obvious limit in sight.
* This makes it easy to put in small computers too. In 5 years, your home appliances will likely be able to speak with you in every major human language plus Elvish and Klingon. Is the dishwasher whispering to the refrigerator at night? We won't even be able to know.
* We humans have a deep belief that we are special, and we have some special thing in our brains that is impossible to replicate with transistors, and this gives us a secure privileged position.
* It's common to hear "but it can't experience sensations or physical reality". Think about how much important work gets done in an office or at home, on a computer, with minimal physical interaction needed.
* Avoid the terms "conscious", "thinking", "AGI" in any discussion. They are actively misleading and always lead to a dead-end.
We may be the last generation of humans in 2M years to think that we were the smartest minds in the world. Congratulations?