Previously, Naiba sharedView WordPress search bot logsmethod, and from the logs, I saw some junk search bots crawling the site endlessly every day, bringing no traffic or indexing, just crawling for fun, such as YandexBot, DotBot, BLEXBot
So, since they are useless, don't occupy my server bandwidth and resources. Block these useless junk spiders.
Using the robots.txt file
People who have worked on websites should have heard of the robots.txt file, so I won't explain its purpose here.
Just insert the following content and save it.
User-agent: YandexBot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: /
Using the User Agent Blocker plugin
The User Agent Blocker plugin modifies the .htaccess file to block spiders and crawlers you don't want on your site.
Because it directly modifies the .htaccess file, it is more effective against spiders and crawlers that do not follow the robots protocol.
Extension Download Address:
nginx configuration blocking method
The above plugin modifies the .htaccess file, so it is only suitable for websites using Apache as the web server. For those using Nginx, like Naiba Web Development, you can use the following method to block them.
This method is shared on GitHub. Original URL:https://github.com/mariusv/nginx-badbot-blocker
The Installation section describes the detailed usage.
Simply put, include the following two configuration files in your virtual host's Nginx configuration file.
include nginx-badbot-blocker/blacklist.conf; include nginx-badbot-blocker/blockips.conf;
Then restart the nginx service.
Using security plugins to block
Previously, Naiba recommendedInstall a firewall for WordPress! Try Wordfence
In Wordfence's firewall settings, you can also block specific crawlers and set crawl frequency. When the crawl frequency is too high, it will automatically block them.
Related articles:How to Block the Spam Spider AhrefsBot (Including AhrefsBot Crawler IP List)