How to block RiskIQ crawlers?

by Nathan Rabinovich   Last Updated September 15, 2018 22:04 PM

RiskIQ is kind of "private Google" which crawls and index entire Internet. Unlike Google:

  • they do not respect robots.txt, so there is no opt-out nor rate-limiting.
  • they use User-Agent of normal browsers.
  • they use thousands of IPs from different providers (mainly QuadraNet and ColoCrossing, but also tens of others; some of the IPs looks residential).

On my websites, the estimated number of their "visits" is comparable to that of GoogleBot (tens of thousands per day).

I would like to block their bots as I do not like to pay for CPU and bandwidth serving those bots.

Google at least brings some traffic to the website, but serving RiskIQ bot is a pure waste.

May "Cease & Desist" letter help here ?

Related Questions

How to successfully block bingbot?

Updated May 03, 2017 09:04 AM

Pieces simulating a mechanical assembly

Updated February 18, 2017 19:04 PM