So, growing up I always heard them referred to as I described above, backwards people clinging to the past over fears of the future.
You'd refer to someone as a Luddite because they refused to change to newer things, like a person refusing to get a smartphone even though they'd like it better because they were just used to an old flip phone.
That's the use of the idiom that I've so commonly heard.