I set up my Python code to make sure that (a) we're in a clean git state and (b) we haven't already run an experiment with the same ID number.
This has saved my bacon more than once, preventing either wasting days of computation or overwriting hard-won data.
