SiteCrawler is a straight-forward and intuitive tool to crawl and analyze websites.

The system finds, collects and checks all links within a website’s ecosystem. SiteCrawler will give you a quick overview of all the subpages that are reachable from a specific website. In its workings the system makes sure to always stay within the scope of the base domain you specified and thus excludes links leading to external sites or sources from its analysis. Since the collection is based on the website’s frontend source code you will only receive links that are also accessible to a normal user looking at the website with a webbrowser. The algorithm is able to build a recursive search collection and follows any chain of links until it reaches a dead end or leaves the website, so you won’t miss a single subpage. SiteCrawler will also follow all redirects and record their origin to give you an even more detailed overview of all the links that are active on any given website.

In order to get going simply specify an origin domain and let the algorithm do its work. SiteCrawler will then examine the website’s source code and collect all reachable links belonging to the same domain. If the recursive crawl is enabled, those member sites will in turn be analyzed once more to detect further links within the website’s ecosystem.

After the search algorithm has completed its work, SiteCrawler will provide you a complete collection of all the subdomains and links belonging to the origin domain and its member sites, that are valid and reachable from the frontend. The results can be viewed and copied directly in the application as well as exported to a textfile.

Escape the obscure jungle of sublinks and let SiteCrawler walk the web for you and bring clarity into your domain-maze.