I usually install Ubuntu server edition, with a minimal set of packages. Then, I add things if and when I need them (using 'apt install <package>').
I use the XMonad tiling WM, most of the time, I only need a "desktop environment" to switch between applications. I could add panels and whatnot, but it just doesn't seem worth the screen real estate.