Over the last decade, a series of applied mathematics papers have explored a type of inverse problem--called by a variety of names including "inverse sensitivity", "pushforward based inference", "consistent Bayesian inference", or "data-consistent inversion"--wherein a solution is a probability density whose pushforward takes a given form. The formulation of such a stochastic inverse problem can be unexpected or confusing to those familiar with traditional Bayesian or otherwise statistical inference. To date, two classes of solutions have been proposed, and these have only been justified through applications of measure theory and its disintegration theorem. In this work we show that, under mild assumptions, the formulation of and solution to all stochastic inverse problems can be more clearly understood using basic probability theory: a stochastic inverse problem is simply a change-of-variables or approximation thereof. For the two existing classes of solutions, we derive the relationship to change(s)-of-variables and illustrate using analytic examples where none had previously existed. Our derivations use neither Bayes' theorem nor the disintegration theorem explicitly. Our final contribution is a careful comparison of changes-of-variables to more traditional statistical inference. While taking stochastic inverse problems at face value for the majority of the paper, our final comparative discussion gives a critique of the framework.