It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.

If only that script worked for every website. But, alas, it does not.