"lolicon" isn't a Japanese term. It was invented by a random American observing things going on in Japan. Then, the term was borrowed, presumably out of convenience. Indirect inspiration by Lolita (even that was a work of fiction) is given too much credit.
Also, honestly, it's not a real term (particularly in the West). It means whatever the person speaking wants it to mean which is not a whole lot. I see it used as a slur against supposed "subhumans" (so being a jerk) or "Cool and feels taboo."
And, frankly, I'm concerned by how some guy from the U.S. can "make an observation" about a *completely different country*, and get paraded around as an authority, especially when their observation might have negative connotations attached to them.
Continuing on the "not a real term" point, I've seen different people, in different countries, use it in different ways. Someone might even have an opinion on what the "right way" to use it is.