Follow
Batching substantially improved my computation time.
Now I should maybe address the recomputation of already computed values. I do fetch results from the DB when fetching entities. But I do not fetch the results when generating new ones which happen to have been previously generated.
Will it be better if I fetch the DB or worse?
I want a software engineer to setup some good performance metrics!
Developing a large project in python.
All very cool, we have type annotations that solve a lot of problems.
But interfaces, I'd really like to have interfaces. No, abstract classes don't do the job. I want compile time reminding me I fucked up.
"reasonable amount of time" = before your retirement age 
We have good news though. One million rows should take "just" about 5GB from these initial results.
That is not too bad. With 5 tera I can store a billion. Should be good enough. I probably do not have enough computational power to get to a billion in any reasonable amount of time.