He is, if the contributors:
1) signed a CLA
2) were doing paid work for him.
Otherwise he cannot relicense the contributions (but he can remove them).
And if he try to, he is going to lose the right to even use such contributions (and the plugin containing them), but only a court could enforce such license termination.
In any case the copyright holder cannot "remove AGPL" from code that was released as such.