A #preprint (#tootprint?, whatever), on how the theories we build depend on the problems we use them to solve.
#pragmatism #philosophyofscience #neuroscience
#cognitivescience
This paper came out of a workshop called "What makes a good theory?", organized by
@Iris, @devezer, J Skewes, S Varma, and T Wareham; and an engaging discussion with @AnyDes, @NeuroStats, F Oude Maatman, S Heignen, J Rawski, and C Wright.
So, what makes a good theory?
One answer comes from Larry Loudan, who said that "science is essentially a problem-solving activity", and "the first and essential acid test for any theory is whether it provides satisfactory solutions to important problems."
I like this answer because it matches what we say we *do* in science. "Problems" are everywhere. Grants and papers are written around them (cf.
@kordinglab's 10 rules for writing). They shape our day-to-day research decisions.
But what's a problem? And what makes one scientific?
For this, we turn Steve Elliott, who summarizes the history of answers to "what's a problem?" and proposes an all-encompassing account:
A problem is a situation in which an agent's aims are unmet, with a set of constraints on what counts as a solution.
https://www.journals.uchicago.edu/doi/abs/10.1093/bjps/axz052
Elliott's account well-captures the problems of everyday and professional life. But what about the problems encountered in scientific research?
What is the situation?
Who are the agents?
What are their unmet aims?
What are the constraints on their solutions?
To answer this, we consider a research community to be a collective "agent": a group of researchers joined by a communal aim: to build a body of knowledge that can be used to solve others and future problems in a specific domain.
Scientific problems can then be seen as problems-for a research community, or "field".
The situation is the state of the field's knowledge (a paper's "background").
The constraints are designated by a research question, the answer to which would constitute a solution.
Scientific problems are "wicked" problems – research questions aren't sufficient to specify their constraints, which are often unstated and change over time.
Unless specified, they can only be provisionally solved via community-based methods of evaluation and acceptance.
In addition to the phenomena that are subject of their research (e.g. biology studies living things), a field's domain includes the set of problems related to those phenomena - what we call a field's "problem-space".
Some of these are problems-for other agents. These are "external" problems for the field. Others are about the field's body of knowledge itself. These scientific problems are “internal” for the field.
Cf. Frankel (1980) for this distinction: https://jstor.org/stable/192551
Internal problems can be e.g. a gap in the field's knowledge, a methodological challenge, or a disconnect between its theories. They range from pure to applied concerns per their relevance to external problems, and include what Laudan called empirical and conceptual problems.
To solve a problem, we use theories from a relevant body of knowledge. E.g. an engineer combines Newtonian mechanics & specialized theories to design a bridge.
These "thoery-users" judge a theory based on problem-sufficiency: can it meet the constraints of my problem-at-hand?
Scientists are theory-users too. We use to theories to explain our data, to design experiments and experimental apparatus, and even to build new theories. (e.g. we use Cajal's neuron doctrine to build a theory of visual processing)
However, scientists are predominently theory-developers. We solve scientific problems to improve our current theories in some way.
As such, we judge a theory based on its problem-coverage: the set of problems in the field’s problem-space it facilitates the solution of.
Unfortunately, we can’t actually calculate a theory's problem-coverage. Instead, we use heuristics: rules of thumb in place of an informed decision.
Our central claim is that theoretical virtues (the properties by which we judge theories) are heuristics for a theory’s problem-coverage.
Theoretical virtues were proposed by Kuhn (1977) in response to critiques that his paradigm-defining work left theory choice a matter of “mob psychology” which “cannot be based on good reasons of any kind”.
cf. Keas (2018) for a recent classification: https://link.springer.com/article/10.1007/s11229-017-1355-6
Evidential accuracy, or empirical adequacy, is usually considered the most important virtue. One of the top things a scientist wants is that their theory corresponds to what they measure/observe in the world.
But just *how much* accuracy is adequate?
For a theory-user, the answer is simple: however much is sufficient to meet to constraints of your problem.
For a theory-developer we need to consider problem-space. That is, a field judges a theory for its accuracy *relative to that needed for the problems in its domain*.
There's more to theory-observation correspondence than accuracy.
Otherwise we'd just fit a deep net to our data and call it a day.
We care about causal adequacy (for counterfactual "what-if-things-were-different" reasoning), and explanatory depth (for extrapolation to phenomena with similar principles).
These virtues can be seen as different ways to estimate the breadth of a theory's problem-coverage.
The world is causally complex, and the aims of different agents require different idealizations. As a result, no theory can solve all problems.
We consider this to be a "no free lunch” principle for scientific theories.
Because there's no free lunch, a research community has to maintain a pluralistic population of theories that collectively cover its problem-space. For example, neuroscience maintains both highly abstract and detailed theories about neuronal operations.
This is where the "coherential" virtues come in to play. Things like: theories should be internally consistent, and agree with other well-founded theories. These virtues moderate pluralism and prevent new problems from emerging where theories' coverage overlaps in problem-space.
Theories don't solve problems, people do.
We introduce a set of "agential" virtues that estimate a theory's problem-coverage by virtue of how it facilitates its use by agents, supports the health and productivity of a research community, and aligns with the needs of society.
For example, agent appropriateness reflects the degree to which a theory aligns with the capacities of its intended theory-users. In doing so, the theory can be more readily used to solve more problems. This includes (but is not limited to) the virtues of beauty and simplicity.
Communal facilitation captures e.g. the tendency of some problems to render many more problems solvable (think: methodological problems that open new techniques for the rest of the research community).
External, or societal, alignment reflects the degree to which a theory meets the needs of societally-relevant external problems. This gives a theory broad coverage outside the internal problems of a field.
We might think that scientists shouldn't worry about stuff like this -- scientists are supposed to be "value-free", its other people's job to figure out how to use our theories if they want to, and scientific truth doesn't care about us...
However, we’re limited beings in a complex world. Theories are made by people, problems are solved by people, and problem-space is defined w.r.t a research community and the needs of other people.
Acknowledging these factors means we can critically assess and prioritize them.
Further, given the power of scientific knowledge, scientists have a duty to consider the social impacts of their work and societally relevant phenomena in their domain.
Cf. Heather Douglas' "Science, Policy, and the Value-Free Ideal":
One last thing: problem-space isn't static. It changes constantly... new data or new priorities bring new problems, and problems that used to be "purely" scientific are suddenly closer to external problems when it's found they relate to societally-relevant phenomena.
Theories themselves change problem-space, and can do so in ways that achieve communal facilitation or societal alignment. Consider e.g. the importance of finding new problems, as pointed out by @fedeadolfi, @mariekewoe, and @braaklvande
in this paper from from the same workshop: https://osf.io/preprints/psyarxiv/jthxf
So what to take away? Theories are problem-laden. They are developed with an eye to external problems, through the solution of internal problems, and are judged based on their problem-solving ability. Through our decisions, these influences shape the form and content of a theory.
This is not a bad thing - it grounds science in effective action.
As scientists, we should 1) pay attention to these influences and how they shape our theories 2) put effort into specifying the problems we solve and 3) review our field's problem space (not just its results).
Thanks for reading, and especially to the organizers of the Lorentz center workshop that facilitated this work.
I promise there will be more "real" neuroscience coming soon 😜
@dlevenstein for further criticism, I don't think focusing on theory as problem solving is inclusive enough to really grapple with the role of the scientific method in the most fundamental science.
One could say that theorizing about string theory is trying to solve the problem of understanding that approach to reality, but I don't think that's the way most people think about a phrase like problem solving.
Maybe technically true but a strained description.
Yes, I was wondering how string-theory would fit in to this conceptualisation.
And while both general relativity and evolution by natural selection have proved to be useful in solving external problems, that was far from obvious at the time of their conception.
@IanSudbery
Created to explain what is now described by quantum chromodynamics and then seen as a promising way of creating quantum gravity theories is the origin story Wikipedia tells, which seems to fit the "problem-solving" paradigm fine?
Meanwhile abstract math is *right there*, waiting to be invoked as an example; the creator of Knot theory even famously saying they didn't expect anyone to find a practical use for the work when interviewd after getting a economics "Nobel prize".
More seriously, I think you might be feeling there is a gap in the above picture because it doesn't fully explain why theorists take tools from one field and apply them in another without any specific problem in mind. But then it doesn't have to because it only claims to explain why a particular theory is seen as "good", not explain how it was created.
@volkris @dlevenstein
@tobychev @volkris @dlevenstein
For example take relativity. It seems to me special relativity came out of the blue, rather than as a way to solve a specific real world problem, and Wikipedia describes general relativity thus "general relativity did not appear to be as useful, beyond making minor corrections to predictions of Newtonian gravitation theory."
@tobychev @volkris @dlevenstein
It seems to me, that some theories, rather than solving extant problems in an existing field, or using result in a field to solve external problems, almost create new fields of study or generate problems that people didn't know existed. These theories general have no apparent coverage in external problem space.
We would thus have to judge them as not good theories under this paradigm, even if they go on to change our whole conception of the world.
@tobychev @volkris @dlevenstein
Perhaps there is something here like Kuhn's different modes of science.
@IanSudbery I think you're missing how vitally important these theories were.
Special relativity did not come out of the blue. Rather it was an extremely difficult mental leap brought on by extremely critical gaps of knowledge as theorists had struggled mightily to figure out how to fit together different parts of the model of the universe around them.
Same with general relativity: you describe it as cleaning up minor details, but those minor details loomed very large! They weren't exactly minor.
It's been my experience talking with philosophers that someone is telling them these really incorrect versions of what physicists are doing today and have been doing for the last 100 years. It says if there are standard textbooks out there that are just plain factually incorrect about what the field has been doing, and this has been leaving philosophy students with really incorrect ideas about how it works in practice.
@tobychev @volkris @dlevenstein
Yeah. My point was not that these things can't be explained by the above paradigm, but rather is interesting to see how they fitted, because I think many of what we think of as the great theories wouldn't come out of this looking very good. At least not at the time they were created .