A reason why I try to phrase things differently is that "privilege" often carries the notion of something that's very much not a right or should not be a right (compare: "driving is not a right but a privilege"). People sometimes refer to things that I think _should_ be rights as privileges (e.g. being safe when walking around at night). I feel that I'm normalizing the not-a-right status of things when I call them privileges (or part of privilege).
I just realized that I don't have the same impression when I see the word underprivileged. Thanks for making me notice that.