A search engine sends out a rover, or "bot", that looks at various web pages, and indexes those pages into a large database that the search engine keeps. When you enter a search query, the engine looks in its database of indexed pages for a match, and returns results accordingly.

But at the end of the day, not all pages on the Web will be in the search engine's database. These parts of the Web that search engines cannot reach is often called by names like the UnderWeb, the Hidden Web. etc., while what is available through standard search engine technologies is, logically enough, called the Visible Web.

Common reasons that sites are not part of the Visible Web are as follows:

• Most Web sites make money through advertising. They sell ad space based on the number of people that visit their site; and people visit the site because of the data that is there. It is not in the site's interest to have their data available through a search engine, like Google; if that was the case, then people could just go to Google to get that data. Google would get the "hit", therefore make the advertising money, instead of the site that originally had the data. (Special commands can be embedded in a Web site's page data to stop search engine bots from entering and indexing what is there.)
• When a search engine indexes the data on a Web site, that consumes a certain amount of the search engine's computing resources. If a site's data is of interest to a relatively small number of people, then the search engine company will not want to index the data there; it is not a profitable use of resources. Why clog up their database with information hardly anybody is going to want? Similarly, some data changes too fast to keep up with economically, like the millions of listings on eBay and their current prices. No search engine would want such data.
• Even if a search engine wants to index a certain Web page, sometimes the data is hidden behind an interface that requires a human hand. If the data cannot be indexed automatically, then it cannot be indexed.

Now, it is obviously not in Switchboard's interest to have all of their names and numbers stored in Google's database. If people could find Switchboard's data by just going to Google, then, again, it would be Google making money from selling advertising, not Switchboard. Likewise, Google probably doesn't want to clog up their database with every phone number in the country; it is too low a return for the resources consumed. And lastly, the interface may be impenetrable to Google's automatic search bot. If the bot cannot access the database of names and numbers directly, it usually has no idea how to use this one-at-a-time-lookup interface to get at the data.