When we perform
SEO Optimizationfor a website, a very important task is to analyze the crawling situation of spiders. If spiders don't even crawl your website, then there will definitely be no indexing. How to observe spider crawl logs? Naiba recommends this Spider Analyser WordPress Plugin here, it's simple and easy to use.
What is a spider?
The spiders mentioned in this article refer to the crawler robots of various search engines. Because they crawl along the links of your website one by one, we all call them search engine spiders.
A web crawler (also known as a web spider, web robot, and more frequently called a web chaser in the FOAF community) is a program or script that automatically crawls World Wide Web information according to certain rules. Other less commonly used names include ant, automatic indexer, simulator, or worm.
Why observe spider records?
The internet is filled with various web crawlers, including search engine spiders, such as bots from Baidu, Bing, Google, etc. Additionally, there are many spam bot crawlers, such as MJ12Bot, AhrefsBot, MauiBot, etc. Spam crawlers will consume extra server resources; some small servers might even be brought down by spam crawlers, causing the website to be inaccessible. Therefore, we need to regularly observe whether the website is being crawled by spam crawlers, and also pay attention to whether search engine crawlers are coming to fetch pages, their frequency, in order to make targeted SEO adjustments.
Introduction to Spider Analyser Plugin
Spider Analyser is a Plugin used to track various search engine spider crawl logs on WordPress websites, and perform detailed spider crawl data statistics, spider behavior analysis, spider crawling analysis, and fake spider blocking, etc. This Plugin is divided into free and Pro versions. You can directly search and install the free version from the Plugin library in the WordPress Admin Dashboard. Some features demonstrated in this article are only available in the Pro version.
In the spider statistics here, we can see which search engines or web crawlers have crawled your website the most times. From the image above, you can see that Naibabiji is crawled the most times daily by Google, and the second is MauiBot. Searching reveals it's a spam crawler, so we'll block it later directly, as it wastes server resources and brings no traffic.
From here, you can see which URLs on the website are crawled the most by spiders. You can appropriately insert an internal link to your newly published articles or articles you want to boost the weight of on these pages, to guide spiders to crawl them.
The same applies to popular articles; you can also insert internal links to your other articles for spider guidance. Then you can also check the indexing status, and for articles found not indexed, we can check the article quality and then submit them to search engines for crawling.
Clicking the icon next to the number of URLs allows you to view the specific spider crawling situation. For example, in the image above, we can see that a spider named coccocbot crawled the website's sitemap. Then Naiba searched online and found that this search engine is from Vietnam. For our Chinese website, crawling by a Vietnamese search engine is meaningless, so it can be blocked directly. This blocking is done directly on the server, not via robots.txt, simple, crude, and effective.
In the spider list, we can see the spider inventory, spider IP ranges, suspected fake spiders, and spider blocking. The SemrushBot in the image above is the robot from the famous SEO tool. If you don't want your website to be analyzed too thoroughly, you can directly block its crawling. As for whether this IP is really a disguised robot, it's not easy to judge. Although the IP query shows it's a Shanghai IP, it's best to search for the official IP ranges on the spider's official website to determine.
The access path can visualize the spider access situation on your website. If you have an e-commerce website, you should pay attention to the crawling proportion of product pages for targeted optimization.
The article crawling feature allows you to see whether it's indexed, spider visits, and the number of inbound/outbound links. Here, focus on pages that have been crawled by spiders but are not indexed, and consider whether to insert inbound/outbound links for link building.
Download Spider Analyser Plugin
Spider Analyser is available in Free and Pro versions. Below are the feature differences between the two versions.
The free version of the plugin can be installed directly by searching in the WordPress Admin Dashboard plugin repository. For the Pro version, please purchase from the link below, then download and upload it for installation.
Download Link
Comments are closed
The comment function for this article is closed. If you have any questions, please feel free to contact us through other channels.