Surely, being associated with a person is NOT a problem, The issue is that some people kept associating with him post him being convicted as a paedophile or when suspicions became more than that.
Seems like as with Jimmy Saville, or Russell Brand, there is the usual gossip, but no one actually takes that and looks further in to some of the things that were alleged to have happened.