Motive [Home].Glossary.

Guides

The Motive Web Design Glossary

crawler/robot/spider

Tracking spiders

Each major search engine has one or more spider. These show up in server logs as unique user agents. ‘Polite’ spiders identify their parent search engine. For example the Google spider identifies itself as ‘Googlebot’.

A spider is a program run by a search engine to build a summary of a website’s content (content index). Spiders create a text-based summary of content and an address (URL) for each webpage.

When a person searches, the keyword(s) they enter are compared with the available website content indexes. Due to the large number of webpages indexed, direct text-only-matching is rare, rather search engines use sophisticated logics (algorithms) to rank potential matches. For example, the underlying information hierarchy of a webpage (semantic markup) may be factored into the ranking a webpage is assigned.

A spider will only index the text-based content of your website. Images (such as photos or graphical-text), video and Flash elements require supplementary HTML-text to be ‘seen’ by a spider.

Spider simulator

Enter a webpage address into the form below to simulate how that page will be indexed by a spider.


Spider simulator provided by Webconfs.com.

Related terms: alt attribute, HTML-text, hyperlink, link farm, PageRank, robots.txt, search engine, SEO.

 

Motive Web Design Glossary Trivia