Methods to Block Specific Bots and Crawlers from Accessing WordPress Websites

Previously, Naiba sharedViewing WordPress Search Robot Logsmethods, and then from the logs, I saw some spam search robots crawling the website relentlessly every day, bringing no traffic or indexing, just crawling for fun, such as YandexBot, DotBot, BLEXBot. So, since they are useless, don't occupy my server bandwidth and resources. Block these useless spam spiders.

Using the robots.txt File

Anyone who has worked on websites should have heard of the robots.txt file, so I won't explain its purpose here. Just insert the following content and save it, and you're done.

User-agent: YandexBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: BLEXBot
Disallow: /

Using the User Agent Blocker Plugin

The User Agent Blocker plugin modifies the .htaccess file to block unwanted spiders and crawlers from accessing your website. Since it directly modifies the .htaccess file, it is more effective against spiders and crawlers that do not follow the robots protocol. Plugin download address:Download Link

nginx Configuration Blocking Method

The plugin above modifies the .htaccess file, so it is only suitable for websites using Apache as the web server environment. For example, Naibabiji uses nginx, so you can use the following method to block them. This method is shared on GitHub, original address:https://github.com/mariusv/nginx-badbot-blockerThe Installation section describes the detailed usage method. Simply put, you need to include the following two configuration files in your virtual host's nginx configuration file.

include nginx-badbot-blocker/blacklist.conf;
include nginx-badbot-blocker/blockips.conf;

Then restart the nginx service.

Using Security Plugins to Block

Previously recommended by NaibaInstall a firewall for WordPress! Try WordfenceDirectly in Wordfence's firewall settings, you can also block specific crawlers from accessing, and set crawling frequency. When a crawler's frequency is too high, it will be automatically blocked. Related article:How to Block the Spam Spider AhrefsBot with AhrefsBot Crawler IP List