@haploc it's important to realize that every post on #fediverse, even if it's in reply to someone else, is its own standalone bit of content being shared out into the public.
Even if you set the post to have restricted visibility, that's more of a suggestion than a requirement, and the post gets transmitted out to be displayed to whomever each instance decides to display it to.
So in theory an instance COULD be smart and try to display the reply to a followers only post only to those followers, but it's up to each instance what to do with the post.
In other words, questions like these have no single answer since it's up to each instance how to handle it, even though it can be said that instances generally handle the situation in such and such a way.