@_thegeoff btw I've realized that heterodynic(sp?) interferometry is a thing, but haven't managed to dig up how it works exactly (in particular, what's the nonlinear mixer that can accept light and can emit RF, or how does it work without that).
IIUC you interfere two frequencies that are closeby and thus you can read out the value by looking for a non-zero-frequency envelope in the output. I haven't read enough to know how that works: do they use nonlinear material to literally create a MHzish signal, or do they sample the output light at a MHzish frequency.
(This gives you better tolerance wrt stray light, but I don't know if this is the only reason.)