We introduce a family of quantum spin Hamiltonians on $\mathbb{Z}^2$ that can be regarded as perturbations of Kitaev's abelian quantum double models that preserve the gauge and duality symmetries of these models. We analyze in detail the sector with one electric charge and one magnetic flux and show that the spectrum in this sector consists of both bound states and scattering states of abelian anyons. Concretely, we have defined a family of lattice models in which abelian anyons arise naturally as finite-size quasi-particles with non-trivial dynamics that consist of a charge-flux pair. In particular, the anyons exhibit a non-trivial holonomy with a quantized phase, consistent with the gauge and duality symmetries of the Hamiltonian.