This crawler uses Breadth First Search (BFS) Algortihm to track and follow all links on a given website.
It can be used to download the complete html code of a website for further analysis.
Additionally you have the possibility to decide if you also want to have links from files that can be found on the website or not.