@regehr
Polish Olympiad in Informatics used to do something like this for already-compiled programs using Intel's pintools. Sadly, I can't find sources for the thing (for some reason there was a separate repository with binaries and all the scripts given to contestants, which I did find at https://github.com/olimpiada/oitimetool-bin).
Ah, it seems that the new version also has some such functionality, but seemingly using perf counters: https://github.com/sio2project/sio2jail/tree/master/src/perf
Yeah, both of these versions implemented an instrument count limit in some form. The pintool-based one had pin rewrite (valgrind-style) the program so that each basic block incremented some counter. The new one seemingly uses performance counters to count (which is probably not exact in some silly way -- e.g. I would expect there to be some slop around context switching).
(I'm somewhat surprised they still use instruction counting, given that it caused very unrealistic handling of any cache effects.)
