2nd October 2019

bbc
16

How does a search engine?

A search engine makes this index using a program called a 'web crawler'. Some websites stop web crawlers from visiting them. These pages will be left out of the index, along with pages that no-one links to. The information that the web crawler puts together is then used by search engines.

Likewise, people ask, how do Google web crawlers work?

Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index. We use a huge set of computers to fetch (or "crawl") billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider).
Write Your Answer

Rate

94% people found this answer useful, click to cast your vote.

4.7 / 5 based on 3 votes.

Bookmark

Press Ctrl + D to add this site to your favorites!

Share