The reality from what I can make out is British workers hated workers from the EU especially Polish workers because they work like trojans!
I worked with some Polish gangs way back, years before they joined the EU. Forget stopping for breakfast at 10 am - they don't, forget stopping for fag breaks - they don't, forget stopping for lunch they eat and work simultaneously. The problem with them is they turn out quality work and their output is around 30 - 50% more than the equivalent British worker (at least in the building trade where I used to work). Of course the Brits hate them!