How to identify all search engine bots and crawlers?

by Ian Y.   Last Updated April 16, 2018 06:04 AM

I'm recently doing keyword advertising on Bing, which also powers AOL and Yahoo searches, to promote a product. And I'm using a third-party tracker to track the clicks of my advertisements.

I found that the third-party tracker I'm using efficiently isolates all "clicks" (not sure if those are really "clicks") made by various search engine bots/crawlers into a different result group (temporary screenshot). I didn't even heard of some of those bots/crawlers before. That bot-identifying feature is very useful in the analysis of performance of my advertisments.

And I'm planning to create my own tracker so that I don't have to rely on a third-party tracker in the future. Therefore I need to know how to identify all of those search engine bots/crawlers in my to-be-built tracker.



Answers 1


You can identify (good) bots by their user agent in PHP with HTTP_USER_AGENT.

Here is an example:

if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot"))
{
    // what to do
}

Source: https://stackoverflow.com/questions/677419/how-to-detect-search-engine-bots-with-php

Here is a directory of bot names: http://www.searchenginedictionary.com/spider-names.shtml


You can also detect bots in your log files on your server. Good bots will have their user agent listed in your logs. A log viewer application can sort the IP addresses of user agents chronologically so that you can see how many visits each one made. Or, after downloading the log file, you can search the log for the bot name and based on how many results there are will be how many times that bot crawled your site.

Michael d
Michael d
April 16, 2018 05:31 AM

Related Questions



Unidentified events in google analytics

Updated April 30, 2015 21:01 PM


How To Track Outbound Clicks On Banner Ads Without GA

Updated October 19, 2017 16:04 PM