Each major search engine has one or more spider. These show up in server logs as unique user agents. ‘Polite’ spiders identify their parent search engine. For example the Google spider identifies itself as ‘Googlebot’.
When a person searches, the keyword(s) they enter are compared with the available website content indexes. Due to the large number of webpages indexed, direct text-only-matching is rare, rather search engines use sophisticated logics (algorithms) to rank potential matches. For example, the underlying information hierarchy of a webpage (semantic markup) may be factored into the ranking a webpage is assigned.
Enter a webpage address into the form below to simulate how that page will be indexed by a spider.