You're overlooking a couple of things: just because the US spends money doesn't mean it's being spent effectively.
Too often people look at the price tag and assume it bought something good, when it didn't.
Secondly, so much infrastructure is state and local responsibility, where projects can be better managed and local leaders held more accountable by their communities.
National defense is definitely a federal matter, so it has to be in the federal budget, but so much of this other stuff is better funded through processes closer to home, where they can better serve residents.