How do I crawl all the pages on my internal website?

Question

I want to hit every page on my internal website to see if any throw an error just from looking at them. The website does its own error logging, so I just need something to follow links.

I am running Windows XP and IIS.

Make it public and wait for Google? :P – alex Jun 04 '10 at 04:33 — alex, Jun 04 '10 at 04:33
You can use www.crawlmysite.in . Works good – Tushar Jul 09 '14 at 06:41 — Tushar, Jul 09 '14 at 06:41

score 2 · Answer 1 · answered Jun 04 '10 at 05:00

2

wget --spider <URL>

edit- just noticed you said windows:

http://gnuwin32.sourceforge.net/packages/wget.htm

answered Jun 04 '10 at 05:00

sml

2,102
13
12

you might want to check out this though: http://stackoverflow.com/questions/762235/how-do-web-spiders-differ-from-wgets-spider – sml Jun 04 '10 at 05:03

score 2 · Accepted Answer · answered Jun 04 '10 at 05:24

2

Try the IIS SEO toolkit. Inspects your site for SEO related issues which includes broken links.

IIS SEO Toolkit

answered Jun 04 '10 at 05:24

Tommy

38,802
9
88
118

How do I crawl all the pages on my internal website?

2 Answers2