Oh, but the productivity gains DO translate into more leisure time for everyone.
It's just that people go on to trade that leisure for wanting more stuff.
It's easy to work for the standard of living of a century ago. It's just that now we want air conditioning and YouTube in addition to food.