Googlebot : Everything You Need To Know
Googlebot is web application bot that crawl webpages with purpose of index in Google's search engines then finally rank them with the help of algorithms like pagerank , penguin, hummingbird etc. Its also known as crawlers, robots and web spider. These three terms meant same thing and when we says google bot refering to two different type google bot: smartphone crawlers ( to crawl for mobile users) and a desktop crawler ( to crawl for desktop users). Its probably possible that your website can crawled by two different type of google bot. You can't opt-out either Mobile crawler or desktop crawler by robot.txt but spiders follows robot.txt directive to crawl webpage or not and its request can be recognise with user-agent strings . In the below table show different types crawler used by Google . Names : Not only these crawler but google uses variety of different crawler for each kind of products and services for example : images, video, news, mobile apps android a