webcheck is a Web site checking tool for Web
masters. It crawls a given Web site and generates
a number of reports. The whole system is
pluggable, allowing extra reports and checks to be
added easily. It supports retrieving Web sites
over HTTP, file, and FTP protocols and produces
reports on site structure, broken links, old Web
pages, overviews of external links, and more. The
links that webcheck considers external are
configurable through regular expressions, and
webcheck honors robots.txt.