Some of it is actual science, theorems about convergence criteria, or approximation ability of these kinds of functions. There are also computer security people that try to make these systems more robust to counterexamples, and that forces practitioners to understand the system better.
Most of it is not worth much. For pretty much all of the ML papers at this session, an approximated function was discovered that could perform the desired behavior, given just the right inputs, and that was it.