The robots.txt issue this guy highlights are the least of his worries. He doesn't seem to realize that what he is describing as having built is embarrassingly bad.

https://james-william-fletcher.medium.com/how-a-benign-web-crawler-found-and-exploited-a-critical-vulnerability-in-my-server-f17c30e061e7