How to Block Specific Bots and Crawlers from Accessing a WordPress Website

Previously, Naiba sharedView WordPress search bot logsmethod, and then from the logs, I saw some spam search bots crawling the website relentlessly every day, bringing no traffic and no indexing, just crawling for fun, such as YandexBot, DotBot, BLEXBot. So, since they are useless, don't occupy my server bandwidth and resources anymore, block these useless junk spiders.

Using the robots.txt file

Anyone who has built a website should have heard of the robots.txt file, so I won't explain its function here. Just insert the following content and save it, and you're done.

User-agent: YandexBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: BLEXBot
Disallow: /

Using the User Agent Blocker plugin

The User Agent Blocker plugin blocks the spiders and crawlers you don't want to crawl your website by modifying the .htaccess file. Because it directly modifies the .htaccess file, it is more effective against spiders and crawlers that do not follow the robots protocol. Plugin download address:Download Link

nginx configuration blocking method

The plugin above modifies the .htaccess file, so it is only suitable for websites using Apache as the web environment. For example, Naibabiji uses nginx, so you can use the following method to block. This method is shared on GitHub, original address:https://github.com/mariusv/nginx-badbot-blockerThe Installation section describes the detailed usage method. Simply put, you need to include the following two configuration files in your virtual host's nginx configuration file

include nginx-badbot-blocker/blacklist.conf;
include nginx-badbot-blocker/blockips.conf;

Then restart the nginx service.

Using security plugins to block

Previously, Naiba recommendedInstall a firewall for WordPress! Try WordfenceDirectly in Wordfence's firewall settings, you can also prohibit specific crawlers from crawling, and you can set the crawling frequency. When a crawler's frequency is too high, it will be automatically blocked. Related article:How to Block the Spam Spider AhrefsBot (Including AhrefsBot Crawler IP List)