BERT got most of the early attention in the topic of foundation models, but from my experience MPNet does at least as good in most of the tasks that I've tested it on.
How does it work? Besides MLM it uses permutation learning for modeling seemingly distant relationships
🔗👇

Sign in to participate in the conversation
Qoto Mastodon

QOTO: Question Others to Teach Ourselves
An inclusive, Academic Freedom, instance
All cultures welcome.
Hate speech and harassment strictly forbidden.