I like the wording here "allegedly sexually explicit books from shelves?" so that implies the policy to remove books is not actually based on actual evidence, just someome saying these books are sexually explicit without actually providing evidence to back that up, or evidence that can be challenged with counter viewpoints again backed by evidence.
As a library you are a place where people can read, learn and develop, you can't make judgements without looking at different narratives and forming a conclusion. That conclusion doesn't have to take one side or the other, you conclusion can be depends on the situation and context for example.