In our last article, we went over some robust ways to sort, filter, and block WordPress traffic as needed to keep your site running at top speed. In this article, we will go further looking at the trade off between speed and security using the performance impact of specific security measures related to WordPress. This is an important step in security as set it and forget it can mean severe speed compromises as we will see.
- Speed impact of robots.txt and ai.txt
In the last article, we covered using robots.txt and ai.txt to control legitimate robotic traffic like search engine robots and AI training scrapers/crawlers. Some may wonder, how much of an impact does this have on non-robotic traffic such as customers? The answer may surprise you.
None.
The only visitors that reference robots.txt or ai.txt are spiders run by providers like Google. Your browser on your workstation or phone will never request or reference these documents. They only improve the speed of your website without any speed compromises involved. You can also use more than basic directives in robots.txt to reduce robotic traffic resource usage even more.
- robots.txt extras: crawl delay
robots.txt supports a variety of extra directives to help make search engine spiders and other robotic visitors as friendly as possible. The one we’ll cover today is Crawl-delay. This requests legitimate spiders visiting your site wait a set number of seconds before requesting and crawling another page. For dynamic sites with many visitors, this can help even out traffic so you don’t get a flood of robotic requests slowing down things for human customers and visitors.
To implement this, you just need to add the following for a 5 second delay between page crawls:
Crawl-delay: 5
or even 10:
Crawl-delay: 10
Beware setting this too high as your pages won’t appear in the search engine as fresh new content for visitors to find. A setting of 1 – 10 will usually be best depending on your traffic levels and website requirements.
The final block might look like this:
User-agent: *
Crawl-delay: 5
- Performance impact of .htaccess directives and filtering
Did you know that you can severely slow down your website by using the wrong .htaccess directives?
Some web guru articles will recommend IP filtering in .htaccess such as using https://www.ip2location.com/free/visitor-blocker to filter out abusive spiders via country filtering. Unfortunately, this method can reduce your site speed by 10x or more. The following tests show more than 5000 requests through reduce to a around 500 just through .htaccess:
As we can see, folder structure and .htaccess can play a key role in website performance and throughput. Using .htaccess to block IP addresses can be an extremely performance draining technique. It is better to use other methods to secure your website against robotic visitors. For example, support staff can block these bad IPs at the firewall without any performance impact. You can also look to services like Cloudflare which block abusive IPs by default reducing the need to keep long IP lists in .htaccess.
In our next article, we will cover more quality of life improvements in WordPress optimizing, security, and hardening. Stay tuned to our blog for the latest articles and updates for WordPress and website.