If we ever develop a real AI and convince ourselves that it’s conscious according to some criteria, the very next thing we will do is figure out how to make an AI that doesn’t meet those criteria. Rinse repeat.
What actually worries me about this is that a lot of people will say “if it doesn’t have a persistent memory it can’t be conscious.” Imagine if your brain was trapped in a simulation where your memory got wiped every 5 minutes, would you stop being conscious?
Follow
A thing that I expect will most certainly not help, but will happen, is the insistence to treat consciousness as a binary classification (in which we might be uncertain of whether something is conscious, but it either is or isn't, and we can try to reason from excluded middle).
