When we are doing
SEO Optimizationfor a website, a very important task is to analyze spider crawling. If spiders don't even crawl your website, there will definitely be no indexing. How to observe spider crawling logs? Naiba recommends this Spider Analyser WordPress Plugin; it's simple and easy to use.
What is a spider?
The spiders mentioned in this article refer to the crawler robots of various search engines. Because they crawl through the links on your website one by one, we all call them search engine spiders.
A web crawler (also known as a web spider, web robot, or more frequently called a web chaser in the FOAF community) is a program or script that automatically crawls World Wide Web information according to certain rules. Other less commonly used names include ant, automatic indexer, simulator, or worm.
Why observe spider logs?
The internet is filled with various web crawlers, including search engine spiders, such as the bot robots of Baidu, Bing, Google, etc. Additionally, there are many spam bot crawlers, such as MJ12Bot, AhrefsBot, MauiBot, etc. Spam crawlers consume extra server resources. Small servers might even be brought down by spam crawlers, causing the website to be inaccessible. Therefore, we need to regularly observe whether the website is being crawled by spam crawlers and also pay attention to whether search engine crawlers are coming to fetch pages, their frequency, etc., to make targeted SEO adjustments.
Introduction to the Spider Analyser Plugin
Spider Analyser is a Plugin used to track the crawling logs of various search engine spiders on WordPress websites, providing detailed spider crawling data statistics, spider behavior analysis, spider crawling analysis, and fake spider blocking. This Plugin comes in free and Pro versions. You can directly search for and install the free version from the Plugin library in the WordPress Admin Dashboard. Some features demonstrated in this article are only available in the Pro version.
In the spider statistics section, we can see which search engines or web crawlers visit your website most frequently. From the chart above, it can be observed that Naibabiji is crawled most often by Google daily, with MauiBot ranking second. A quick search reveals that MauiBot is a spam crawler, which we can block later to prevent it from wasting server resources without bringing any traffic.
From here, you can see which URLs on the website are crawled most frequently by spiders. You can appropriately insert internal links to your newly published articles or those you want to boost in ranking on these pages to guide the spiders to crawl and index them.
Similarly, for popular articles, you can also insert internal links to other articles to guide the spiders. Additionally, you can check the indexing status and discover articles that are not indexed. We can review the quality of these articles and then submit them to search engines for crawling.
Click the icon next to the number of URLs to view detailed spider crawling activities. For example, in the chart above, we can see that a spider named coccocbot crawled the website's sitemap. After searching online, Naiba found that this search engine is from Vietnam. For Chinese websites, crawling by Vietnamese search engines is meaningless, so it can be directly blocked. This blocking is done directly on the server, not via robots.txt, making it simple, straightforward, and effective.
In the spider list, we can see the spider inventory, spider IP ranges, suspected fake spiders, and spider blocking. The SemrushBot shown in the chart above is a robot from the well-known SEO tool provider. If you don't want your website to be analyzed too thoroughly, you can directly block its crawling. As for whether this IP is genuinely a disguised robot, it's not easy to determine. Although the IP query shows it's from Shanghai, it's best to check the official IP ranges on the spider's website to confirm.
The access path feature visualizes the spider crawling activities on your website. If you run an e-commerce website, you should pay attention to the crawling proportion of product pages to perform targeted optimizations.
The article crawling feature allows you to see whether articles are indexed, the amount of spider traffic, and the number of inbound and outbound links. Focus here on pages that are crawled by spiders but not indexed, and consider whether to insert inbound or outbound links for link building.
Download the Spider Analyser Plugin
Spider Analyser is divided into a free version and a professional version. Below are the feature differences between the two versions.
The free version of the plugin can be directly searched for and installed from the Plugin Library in the WordPress Admin Dashboard. For the professional version, please purchase it from the link below, then download and upload it for installation.
Download Link
Comments are closed
The comment function for this article is closed. If you have any questions, please feel free to contact us through other channels.