It's not a secret that the far-right really wanted Brexit to happen. And I do count a big chunk of Tories as part of the far right (the rest of them and much of Labour being the regular kind of right wing).
A big chunk of the reason was human rights.
EU is awful in its own ways, but it's relatively less awful than much of the Western world.