Have you noticed the weirdnesses around "visible to followers only"?
The technical weirdness is that when you reply to that, you end up replying (by default in Mastodon) with a reply visible to _your_ followers. (And depending on various antispam setups, you might not even be able to reply in a way visible to OP's followers[1].
The nontechnical weirdness is that it's followers and not followees. The latter is a collection you always control yourself, and a reason for having someone in there you don't want to share stuff with is less likely to materialize than for followers.
[1] You can only get OP's instance to forward to them (https://www.w3.org/TR/activitypub/#inbox-forwarding), which requires signed objects (because you might have no clue who the OP's followers are, if they hide the collection, so you can't let them fetch the post from you) that people don't want to use because it makes things nonrepudiable. Even if they do, then most likely such messages would be pushed only (and not pullable), so would have less reliably delivery.
@robryk yeah, followers-only is quite broken, and i've posted about this before x3
it's actually quite frustrating