Wow. No wonder we thought we were slowly becoming crazy. Turns out the spirit level we used is actually broken and it showed different information depending on which side is facing us.

So yeah, it was always 5cm too high or 5cm too low. With another one we finally managed to get it right, and yes, that was 2,5 cm difference. :P