Eleven2 has begun the process of upgrading our servers to Apache 2.4. In this post you will find very detailed information on why we will be switching to Apache 2.4.
* Apache 2.4 – A big step forward.
While other web servers such as NGINX have captured social media’s attention, Apache still powers most of the internet as the rock solid platform we have depended on for almost 20 years. The Apache platform continues to modernize itself without sacrificing what makes it into the platform that has built so much of the internet. You can see clear examples of this in each system’s design as NGINX drops support for .htaccess configuration while Apache builds in architecture changes to support MPM event taking on NGINX where it gets most of its performance.
Usability is key.
As we looked at each piece of the shared web hosting platform, our focus was performance, compatibility, and usability in equal measure. If the system was slower for compatibility, we needed to do better. If the system performs extremely well, but it requires a support ticket to change website rewrite, we need to do better. If the performance wasn’t showing in benchmarks, we definitely needed to do better.
We started with static file performance, MPM event, and Apache 2.4.
Much like NGINX’s event based architecture, Apache’s MPM event takes a few more steps in that direction than other MPMs recycling more threads in different ways including secure threads for TLS/HTTPS connections improving https performance. This reduces overhead as more sites can securely use the same threads ensuring requests meet the hardware faster than before. If a visitor requests resources from website1.com and another user requests different resources from website3.com, there is no pause or startup/shutdown as the server handles each seamlessly. This massively increases the maximum number of requests the server can handle while reducing the amount of time it takes to handle each of the requests. Win/Win.
In addition to improvements introduced by MPM event, we have also optimized every aspect of the system including mod_expires/mod_headers intelligent caching, mod_deflate optimized compression, and a lot more. These changes optimize caching for CDN/proxy, mobile usage such as iOS/Android, and a lot more without interfering with website settings. These transparently instruct CDNs such as Cloudflare, mobile devices, and even browsers to:
Keep certain items locally cached instead of constantly rechecking/downloading,
Accept compressed content such as CSS, javascript, HTML coding, and more for faster loading on mobile. The slower connections on many mobile devices make this a key performance improvement.
Deny access to restricted resources to reduce robot intrusions reducing resource usage. With more of your resources available for visitors, more of your traffic is legitimate instead of just more robots lining up to attack. This is in addition to other measures such as mod_security and denies access to files such as .sql which are usually database backups left in place by mistake.
Optimizes TLS/HTTPS connections introducing Forward Perfect Secrecy, HSTS, SNI, faster TLS KeepAlive responses, and a lot more. With Google’s recent changes to search engine rankings making HTTPS sites list higher, these changes will be key going forward.
* mod_fcgid FastCGI – The replacement for suPHP. Good riddens.
When suPHP was first introduced years ago, it was a significant improvement over the older phpsuexec patch set for Apache that had been in use. It supported user level php.ini, significantly improved security, and a full CGI implementation with several “fixes” to ensure compatibility among many other things. With mod_fastcgi (not mod_fcgid which would be introduced much later) a buggy mess, it was a stable, production, and fast option.
Fast? Yes, suPHP was fast during a time when a single page request was a single PHP request.
With AJAX/Web 2.0, automatically refreshing WordPress themes, and a lot more, one PHP request per page request became 15 – 20 PHP requests per page request increasing the stress put onto the system by suPHP. As the system starts an interpreter process, runs the code provided by Apache, flushes the result back to Apache, and shuts down for each request, the overhead became significant. There was also no way to introduce opcode caches to cache compiled PHP code as suPHP shuts down the interpreter after each request closing the cache along with it. To resolve this, we started to upgrade to mod_fcgid.
What is the difference?
The most important aspect of the difference is how much they are the same. suPHP and mod_fcgid are both CGI based which improves compatibility between the two allowing us to make only minor alterations to the variable environment as it changes. The result has been tens of thousands of websites upgraded with less than one support ticket per server upgraded.
To improve on the performance of suPHP, mod_fcgid avoids the startup/shutdown overhead by keeping the workers resident in memory for a short time after the request has been completed. If a website loads the index.php, it is very reasonable to assume the visitor or search engine robot will also request more resources in the next few seconds/minutes. If no requests have been received during that time, the worker unloads itself to reduce resource usage until more requests are received. This intelligent worker management combined with opcode caching has resulted in less server instability/crashing as well as improved speeds as websites load in a fraction of the previous time thanks to opcode caching and the reduced overhead.
mod_security – Rethinking filters.
When mod_security was first released by the developers, the filters it included were basic and designed to catch a very wide variety of attack type traffic as it ran on the incoming POST/GET requests through regular expressions to validate them. This is a lot like code you would find in PHP scripts to validate credit card numbers, email addresses, or other data. It is just applied as a filter at the HTTP level as a Web Application Firewall or WAF. This was problematic for a variety of reasons including numerous false positives for WYSIWYG editors, forums, blogs, and many others.
Another problem with this approach was performance. The regular expressions were CPU heavy leading to a severe drop in performance. The amount of CPU drain was nearly as bad as PHP itself for awhile as the system struggled to process everything. Despite various attempts at making it lighter such as filtering only dynamic traffic, the software was still too heavy. That is when the authors converted to the Perl Compatible Regular Expression C library which is a much faster regular expression processing engine than the one it previously used. When this software was compared to the old regex (regular expression) engine, it was 2 – 4x faster making the rules much faster. It also encouraged more and better rule contributions as the regex syntax was from a known standard.
This same approach would continue to be used for years until Apache 2.4, advances in compilers provided by Chrome/Apple, and other changes allowed us to rethink filters.
PCRE-JIT – Borrowing from Chrome.
When Microsoft Internet Explorer 6 roamed the earth, websites did not rely on AJAX Gmail/Google Docs interfaces. Due to the speed of the javascript interpreters, those browsers would not be capable of running modern web applications. It isn’t just language compatibility, but raw speed as advances have allowed browsers to run javascript hundreds into thousands of times faster. The how is a very neat trick.
If you look at programs running on your computer, they run native machine code as binary programs. They “speak” the same language as the hardware allowing them to run without an interpreter to slow the process down. Unfortunately, the extra step of involving an interpreter can slow the process down considerably compared to native code. That is where the idea of JIT or Just In Time compilers comes in.
It is faster to compile javascript then run it as native code than it is to run the code through an interpreter. The net gain is sometimes up to 500 – 1000% faster for javascript in complex operations.
For us, we can use the same trick with PCRE to speed up execution quite a lot by compiling it before executing using a Just In Time compiler built into PCRE.
http://sljit.sourceforge.net/pcre.html
For various types of complex regular expressions, the net result is 70 – 90% faster than before. For mod_security which is a very advanced regular expression processing engine itself, this is a significant gain. This allows us to run more data through the engine much faster than before while also taking advantage of features built into Apache 2.4 to filter more data in different ways. The increased speed has allowed us to deploy different types of filters to block exploits without the same false positives we have seen before.
Whitelisting – Whitelist by the ID
With the new JIT compiler also comes new whitelisting as we work on building a system that is more compatible. The new whitelisting features will allow us to selectively disable mod_security per folder, per domain, per rule ID, in .htaccess, and so on for a very flexible system. This will allow us to avoid many of the whitelisting problems so we can disable it for a single directory instead of an entire site providing more protection from exploiting robots.
Comodo WAF – New mod_security rules from Comodo.
Since mod_security was originally released, the focus has been on blocking bad robots based on GET/POST contents directed at specific scripts. The result of this has been rules that cause a considerable amount of false positives for WYSIWYG editors, theme editors built into popular scripts, and similar POST/GET requests. These have lead to a lot of whitelisting which disables mod_security and the filters leading to scripts being exploited.
As we looked for a new approach, the new Comodo WAF rule set took an interesting strategy with validation allowing us to avoid some of the POST/GET filters in favor of Apache 2.4 enabled validation. Unfortunately, this would filter a lot more via mod_security, but with other recent enhancements such as PCRE-JIT the system runs even faster than before as we block more bad robots using less system resources.
This system has also been in testing for awhile now with very few false positives while feeding attackers into the firewall.
Linux Malware Detect – Enhancements to scanning. – Eleven2 patches.
Finally, we have also modified the Linux Malware Detect engine to start to scan more potential malware before it is able to exploit vulnerable scripts. Without giving too much away, this reduces overlap, enables more comprehensive scanning, and has resulted in a massive drop in the amount of malware separate from mod_security changes.