Major Crawlers not honoring robots.txt?

by Joshi   Last Updated October 06, 2018 14:04 PM

few of my websites were compromised, there were thousands of links submitted to google, bing etc which are not relevant to my site.

I cleaned the websites in question and updated the robots.txt.

the robots.txt file look like this:

User-agent: *
Disallow: /
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

which means the first line is Disallow: / yet all major crawlers are hitting(including google, bing, apple) non existing pages in the root directory and hitting 404 not found

I have updated the robots.txt file around 12 hours earlier.

Am I missing something here?



Related Questions


Help with robots meta tag and X-Robots-Tag

Updated December 21, 2016 08:01 AM


How to remove the CNAME of your site?

Updated March 03, 2016 01:01 AM

How to get rid of non-reputable Bots

Updated July 17, 2016 08:01 AM