You just gave me an idea for a fun project (like i need more projects!). I use openresty for my webserver, which has lua already built in. Might be fun to see if i can just hog up a crawler instance indefinitely by generating fake pages and links that just go back to make a new fake page and link and takes the crawler in circles that it thinks are unique