A web crawler, also called a spider, bot, or web robot, is an automated software program that systematically browses the web by following hyperlinks from page to page, fetching the content of each page it discovers, and processing that content for a specific purpose. Search engine crawlers, Googlebot, Bingbot, Yandexbot, crawl the web to discover, analyse, and index content for inclusion in search results. Other crawlers monitor websites for changes, check for broken links, gather research data, or perform competitive analysis.