First case I've heard of where this kind of incident has gone this far, but it's just the start. Apple has a very similar system in place.

A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal. https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html?unlocked_article_code=AAAAAAAAAAAAAAAACEIPuonUktbfqYhkQFUZAibJUNMnqBqCgvfeh7M7gXD6IW2OVCxcwuURB4GG8EqIea1va98-xXGDRNhdN_kvWPl2hKd5DnBadjOJ8NGCiYhXZGI8s56yVWc7mJmXAr02rTK2Kze1dbw6me3vtFfWbmfpRKXU2CUjJQ1lqJBuIUq4miBdntezGe5y09h83_snF8o6EW9GPH_WyqGuXxZuO9yGbQXe4x02W4xZWTLSnt2f79EQYVkYSAKGHD4kvzFKuJ4LM8gXPa3_MxcrZMP45L4SAWZoJ4-ubIYjCqT9JtmYWV9ayGCU7zkONQ