That'd a fair point and surprising that it's persisted for so long. Claiming that "University of London" colleges are allowed to complete on their own is not the same, not least cos many have their own degree-awarding charters, but they're completely separate organisations.
Let Oxford and Cambridge have their own separate preliminaries, maybe even allow their two finalists in.

https://www.theguardian.com/tv-and-radio/2023/mar/26/bbc-rejects-charge-of-elitist-oxbridge-bias-in-university-challenge