Well, it's like saying fans give the umpire at a baseball game the authority to say whether it was a ball or a strike.
Yes, you can quibble over whether to trust the umpire when you're really, really insistent that it was a bad call. The fans decide whether to believe you over the umpire, so the fans give or don't give that authority. Sure.
But at the end of the day, the umpire still has the call that matters to the outcome.