Regulation of cell growth and division is essential to achieve cell-size homeostasis. Recent advances in imaging technologies, such as ``mother machines" for bacteria or yeast, have allowed long-term tracking of cell-size dynamics across many generations, and thus have brought major insights into the mechanisms underlying cell-size control. However, understanding the governing rules of cell growth and division within a quantitative dynamical-systems framework remains a major challenge. Here, we implement and apply a framework that makes it possible to infer stochastic differential equation (SDE) models with Poisson noise directly from experimentally measured time series for cellular growth and divisions. To account for potential nonlinear memory effects, we parameterize the Poisson intensity of stochastic cell division events in terms of both the cell's current size and its ancestral history. By applying the algorithm to experimentally measured cell size trajectories, we are able to quantitatively evaluate the linear one-step memory hypothesis underlying the popular ``sizer",``adder", and ``timer" models of cell homeostasis. For Escherichia coli and Bacillus subtilis bacteria, Schizosaccharomyces pombe yeast and Dictyostelium discoideum amoebae, we find that in many cases the inferred stochastic models have a substantial nonlinear memory component. This suggests a need to reevaluate and generalize the currently prevailing linear-memory paradigm of cell homeostasis. More broadly, the underlying inference framework is directly applicable to identify quantitative models for stochastic jump processes in a wide range of scientific disciplines.