How Do Search Engines Work?

The Three Stages of a Search Engine

How do search engines work? In broad terms, they run in three stages: they discover pages on the web, they store information about those pages in a large database called an index, and they rank the stored pages when someone runs a search.

These stages are a simplification, but they explain most of what happens between publishing a page and seeing it in search results. A page must pass through each stage before it can appear for a query.

Crawling: Finding Pages on the Web

Crawling is the process of discovering URLs and fetching them. Search engines run automated programs, often called crawlers or bots, that follow links from page to page and also read the sitemaps that site owners provide. When a crawler requests a page, it receives the same HTML and files a browser would receive, and it records what it found.

Crawling is neither instant nor complete. Engines choose which URLs to visit and how often, based on how useful the pages seem and how much load the server can handle. A new page with no links pointing to it may wait a long time to be found. For more on the programs themselves, see what a web crawler is.

Indexing: Storing and Understanding What Was Found

Indexing is the step in which a search engine processes a fetched page, works out what it is about and stores it in its index. The index is the database that results are drawn from. A page that is crawled but never stored in the index cannot appear in results.

During indexing, the engine reads the text, headings, images, links and structured data on the page. It also checks whether the page duplicates another one and which URL should represent the group. Pages can be left out at this stage, for example when they carry a noindex instruction or when they are judged to offer little value. Reasons a page may be missing are explained in why a page is not indexed.

Ranking: Ordering Results for Each Search

Ranking happens when someone searches. The engine takes the pages in its index that might answer the query and orders them by estimated usefulness. Many signals are involved, and their weight changes over time. Common signals include how well a page matches the words and intent of the query, the number and quality of links from other sites, the freshness of the content, how usable the page is on a phone and how fast the site loads.

Ranking can also vary by place. The same query may return different results in different countries, on different devices or in different languages. This is why no single position can be guaranteed for any phrase.