Let $X$ be a double EPW sextic, and $ι$ its anti-symplectic involution. We relate the $ι$-anti-invariant part of the Chow group of zero-cycles of $X$ with Voisin's rational orbit filtration. For a general double EPW sextic $X$, we also relate the anti-invariant part of the Chow motive of $X$ with the motive of a Gushel-Mukai fourfold. As an application, we obtain a similar result for certain Fano varieties of lines in cubics with infinite-order birational automorphisms.