I'm not saying you're wrong, but I don't know how you would make a generalization like that given that ARM hardware is so painfully disparate. I don't think I've run X11 on my Pinebook Pro, but I have on Raspberry Pis for a number of years.
One of the biggest pain points on ARM is that hardware support is so dependent on the hardware vendor. It's not like amd64 where you basically just have three major video card vendors.