Thrilled to announce the Regular Expression Inference Challenge (REIC), with Mojtaba Valizadeh, Ignacio Iacobacci, Martin Berger.

REI is a supervised machine learning (#ML) and program synthesis task, and poses the problem of finding minimal regular expressions from examples: Given two finite sets of strings P and N and a cost function cost(⋅), the task is to generate an expression r that accepts all strings in P and rejects all strings in N, while no other such expression r' exists with cost(r')<cost(r).

Turns out, this sort of inference seems to be really hard for current DL (#llms ) approaches. Prompting StarChat-beta -- a SOTA large LM for code with 15.5B parameters -- yields extremely low results.
Even a fully supervised 300M parameter model, which we call ReGPT, only achieves around 14% precise and minimal expressions.

Check out our preprint on arXiv: https://arxiv.org/abs/2308.07899
The challenge is available on CodaLab: https://codalab.lisn.upsaclay.fr/competitions/15096

We formally define the problem, and provide training and validation data, as well as starter code for all our baselines.

We invite researchers anywhere to participate in tackling our challenge.

#machinelearning #inference #challenge #AI #ML #llm #llms #huawei