Finding Hidden Directories

Web Spidering

zaproxy | Kali Linux ToolsKali Linux

spiderfoot | Kali Linux ToolsKali Linux

Mirroring a Website

Wget

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent certifiedhacker.com

HTTrack Website Copier - Free Software Offline Browser (GNU GPL)

Other tools

Website data scrappers

ParseHub | Free web scraping - The most powerful web scraper