We, as a society, have agreed that there are things we can't use to discriminate people, and that using those things in that way is bad. Some people seem to forget that throwing around accusatory labels ("you're sexist!") is a grave issue, as it hampers preventing the actual discrimination from happening.
@jezza aha yes, these are very wise words!