I think most of the discussion around free speech on social media is a red herring. The concept itself is rather nebulous, beyond being a restriction on the state to not restrict speech of the people (and even that seems to have vaguely defined exceptions).
I think the important thing is not free speech, but effective free association. I don't care if I have 'free speech' when going to my friends house, I care about being with my friends. If I'm too offensive to them, they can kick me out. Otherwise, we're free to hang out together and talk in ways which are mutually acceptable to each of us.
But it's also important to be able to close the door to people who don't belong. If I'm at a rave, I don't want people there who are trying to grill steaks on the dancefloor, even if there's nothing wrong with dancing or grilling. The answer (in a social media setting) can't just be 'just block them'. In the example above, not every raver should have to tell the griller that they don't want them there. There should be a sign at the door and a bouncer. This is the 'effective' part of the free association. If you constantly have to worry about people you don't want to see barging in (or even harassing you), it's not free association. So we need good admins and moderators to help you with that and keep a good community going on a server.
A centralized system can not effectively handle this. It's as if I went to my friend, put on a kpop song, and then someone from the association for the advancement of classical music busts in and throws me out because they think it's trash. This is what happens at twitter, currently.
There's no way for them to handle this without splitting up in multiple fluid communities, something we already have in the fediverse. You will never make everybody happy, but with the twitter kind of centrally planned moderation, you don't make anybody happy.
So that's what I think is what makes the fediverse great: It's an effective way to stay in contact with the people you want, and to keep out the people you don't want, without removing access for anyone globally.