Can data augmentation benefit from a separation of "online" and "target" network?
BYOL's answer is yes!
Furthermore, they suggest that using negative examples might be obsolete, as they achieved new SOTA without them.
How does it work? 👇
1/4
First of all, they specify a fast-learning "online" network, and a slow-learning, "target" one.
For a given sample the online network is trying to predict target network's embedding.
The catch?
They are using different augmentations!
Follow
This way they are taught to achieve consistent embeddings of observations across different ways of introducing noise.
Finally, the online network is trained using gradient descent, while target network's weights are updated by averaging them (exponential moving average) with online's weights.