preissen wrote:I have only recently installed SEO tools and tried to generate a sitemap in XML format and it only generated a map of 21 URL's however my site has far more pages than this 100's of URL's. I can't find any settings that I need to change can anyone provide any ideas of what is causing it to only read 21 pages?
hellyson wrote:Spiders are the automated programs that are crawling the web all the time to get up to date data for search purposes. The spider visits the given URL, gets all the hyperlinks in the visited page, saves the new links to the URLs list to visit (crawl frontier), saves the page itself to a database file, traverse the URLs list, pick a new URL to visit, and do the same cycle again.
Users browsing this forum: No registered users and 1 guest