I think I figured it out in the meantime.
Assume a star is a collection of ideally monochromatic point sources. Then you can clearly see how they will interfere with each other at infinity in a particular direction. The net phase you observe there will be a function of phases of these sources, which avoids the "where does the timeshift asymmetry come from" problem. (Also, amusingly, the "you can't emit polarisationful wave in all directions" seems to still be the case, so there will be a line along which nothing gets emitted. If we allow the sources some bandwidth, this restriction disappears because it doesn't apply across different frequencies.)
Re "which photons we actually see": the question is ill-formed, because they interfere with each other.
Not exactly average phase, but something along these lines.
The reason I say along these lines instead of exactly that is quibbles around the difference between "there's literally no way to distinguish" and "our description considers these equivalent".
BTW. If the source is not monochromatic (as it never is), the notion of in phase becomes weird and complicated.
@robryk @drskyskull Yeah, "resultant" or "net" rather than "average"?
Yes, sum-up amplitudes and take argument of that. (And it might be brighter/dimmer in different directions. I haven't figured out what will happen if I e.g. take a bunch of sources with positions taken from a Gaussian and with polarisations picked uniformly from a sphere.)
The other thing I wanted to point out is that this is not the thermodynamic case, where the information on which particle went where is there, but unretrievable without investing lots of entropy. In this case the information about source of the particular photon you captured doesn't exist.
@robryk @drskyskull Ah...you're saying the *resultant* light can be considered in phase, because the "non-average-phases" cancel and we're left with the average phase?
Analogy - particles in a gas cloud have random angular momentum, but by the time they collapse to a star only the average angular momentum is left?