I think it depends on what your priorities are.
As someone who currently self-hosts a Matrix server and have tried multiple XMPP apps (admittedly a few years ago), they just aren't as user-friendly/polished as Signal (not to say that some of them don't even have E2EE as the default).
For security, IIRC the Signal protocol is considered better than, say, OMEMO. Even when it comes to privacy/metadata, sealed sender seems to mitigate a lot of the issues with a centralized server. So the main reason for decentralization is probably to eliminate the SPoF, but given that the lock-in for mobile chat apps aren't really that strong IMO, the advantages outweigh the downsides for me.
My main beef with Signal is the use of phone numbers as the only ID, but they seem to have plans to improve this.
