Apple has quietly added a new page to its support site that details its first Web crawler.
A crawler is a bot, used by search engines such as Google and Bing, that automatically scans the Web for pages and information to add to an index and provides information when pages change.
Apple’s crawler, which existed previously but was never detailed by the company, identifies itself as “Applebot” and could indicate a future push into full Web search.
Right now, Apple says its crawler is used to add data such as movie information to Siri and Spotlight. You can exclude Applebot from searching your site by updating the robots.txt file on your site.
The company’s deal to use Google as the default search engine on iOS is reportedly due to expire soon and rumors have indicated in the past that Apple may be quietly building a search engine of its own.
Via: TNW
