Obligatory shout out about how this AI panic is trouble[1].
Back to the topic at hand:
This is something which we call the "Scunthorpe Problem"[2]. The name comes from (censorious and half-baked, though not that rare) algorithms which kept finding the word "cunt" inside the word "Scunthorpe" despite it being a completely valid location in the U.K.
As for this specific matter with the "child porn warnings", when I say that warnings are fuzzy[3], I really do mean "fuzzy" as in "Oh, it might appear... Maybe." (though, in that case, moderation is often the better option).
Also, bad actors might change their language to... Something weird... There are a few examples which I won't mention. In one well-documented one from 2020, they started talking about "pasta".
Also: "Part of the problem is that much of it is a demand-side problem, not a supply side problem. If people are demanding certain types of content, they will go to great lengths to get it"
For the second example that @mmasnick gives, there is actually no legitimate reason for this platform to be doing this.
A few policies, as written here, actually appear to originate from lobbying from QAnon. It's not a policy decision, it's a blunt and misguided PR move. Even when it does not involve "QAnon" (so not that entire passage), they still seem to be clustered in the years 2018, 2019, and 2020, and crucially, involve Morality in Media.
1 https://qoto.org/@olives/111866367418212996
