Benchmarking Drupal 8 on PHP 7-dev

Benchmarking Drupal 8 on PHP 7-dev

Difficulty: 
Piece of Cake

It's been a while since I quick benchmarked Drupal 7 on PHP7. But at the time of that writing it was still not possible to benchmark D8 in PHP7, there were too many compatibility issues that simply would not let D8 boot on PHP7.

There is D8 on PHP7 initiative in this queue issue at Drupal.org that has helped get D8 working on PHP7.

Today I was finally able to get D8 up and running on PHP7 and the results were astonishing. We only found 1 PHP bug that was reported (and solved in a matter of hours by the PHP team), indication that PHP7 is still under heavy development and that any test ride should expect issues.

It is not only the switch to PHP7 what makes a difference, but the fact that D8 is faster than D7 out of the box (in the sense that it leverages caching by default and in a more sofisticated way). 

I am though not very happy that the D8 performance efforts are so focused on caching. Relying on caching as the main performance driver is not good for a CMS that aims at being a CMF (sort of a development framework). Caching adds complexity to applications and makes development and troubleshooting slower and more time consuming, and in it's heart it is very dangerous because it helps cover serious performance flaws in the overal system's design. It's sort of a rat's trap because it helps you hide bad implementations or designs in the sort term, but it simply bursts once your application has grown leaving you with no other alternative than starting from scratch because - um - you have been hiding you trash under the carpet for a very long time and there is nothing to be saved. Premature optimization is after all not the root of all evil - it just makes sense to design you system from the ground to be fast and scalable. Designing at your will thinking that later you will make it scalable and fast by caching is a double edged sword.

Premature Optimization can be defined as optimizing before we know that we need to.

Now the results. I had to shift from concurrent throughput testing to sequential single requests because D8 on PHP7 D8 was hiting hardware limits leaving us with numbers such as 1200 Requests Per Second with 8 concurrent requests. Anyways, sequential non concurrent requests give us a better idea of the real performance difference.

This is a very simple 1,000 sequential requests Apache Benchmark test against the default Home Page with Page caching turned off and opcache enabled.

With PHP 7.0.0-dev (Build date: Mar 25 2015 22:22:57):  210 requests per second

With PHP 5.6.6: 22 requests per second.

[UPDATE] Take a look at the comments to see what these numbers are really supposed to be. These results seem to be all screwed up.

We are not comparing D7 to D8, but PHP 5.6 to PHP 7 on the same exact setup. Our D7 comparison threw a +16% for cached pages and +58% for uncached, but this time we are seeing a x10 fold in performance and turnning page caching on and off made no difference to the results (possible indication that we are either setting up something wrong or there's something broken in D8 page caching).

Comments

If turning off the caching made no difference you might was to rebuild the cache (which in D8 might be the new, did you clear the cache?)

If you have a drush 7, the command is drush cr

I khave to respond to this ;)

1) Caching and performance with warm caches is a focus but not the only focus. There are multiple criticals and many non-critical issues to profile scenarios with many fields for example, profiling bootstrap, saving views and so on.

2) Your argument about caching can be turned around. It is very hard to add caching to an application when the architecture doesn't support it. We've been working *very* hard to make the caching as transparent as possible and solving cache invalidation problems.

3) There's no way that you got numbers like that, and differences like that with page caching disabled. My guess would be that your 5.6 run had cold caches or so and you have page caching enabled. Note that this is now a module that you have to disable/enable. Make sure you look at the request time distribution to see if you have some slow requests. For example, when the first request is after enabling a module, then I saw a first slow request at 500ms, and the others at ~50 with page caching disabled.

Here are my numbers with ab -n 1000 (page cache enabled) and -n 200 when it is off:

PHP7, page cache on:
Requests per second: 409.04 [#/sec] (mean)
Time per request: 2.445 [ms] (mean)

PHP7, page cache off:
Requests per second: 17.52 [#/sec] (mean)
Time per request: 57.076 [ms] (mean)

PHP 5.6, page cache on:
Requests per second: 280.09 [#/sec] (mean)
Time per request: 3.570 [ms] (mean)

PHP 5.6, page cache off
Requests per second: 13.11 [#/sec] (mean)
Time per request: 76.264 [ms] (mean)

This is an empty installation with no content.

I always try to be a little bit politically incorrect when writing these articles, after all I do it for fun (but always trying to be respectful and provide useful information!) I was about not to publish this particular one because I didn't like the final numbers and had the feeling that there was something wrong. It all got messed up when I came accross the bug in PHP and spent more than expected on the benchmark because I took my time to workaround it in D8 for the tests and report it to the PHP bug queue. When the time to do the benchmarks came, I was on a rush.

1) You are right, but I have seen that the issues that get the most "commercial" visibility are those related to high level caching.

Issues like this one "Make D8 2x as fast [...]" got a lot of attention, but the title itself is missleading. D8 twice as fast in..... what scenarios?

Take a look the issue Figure out why Drupal 8 is slow. Having such an issue when the final piece of software is close to release is not a bad thing after all. But compare these two different interpretations of that same issue:

a) We designed all this to be blazing fast, we followed best practices and have been meticulous in the development, but it is still too slow, something must be wrong and we need to spot it.

b) We are close to release, and this thing is still full of performance issues, what's up?

What is that issue closer to, a or b? They are even paying attention (which is good) to 1ms/request performance gains patches, but that starts to sound like the north is lost.

2) You are right again. I must confess that Drupal 8 tag based caching system and its fine integration with the overal architecture is extremely good. It will sure come in handy and make a big difference. But that does not invalidate any of the statements I made. Many people rely on high level caching on a regular basis to cover for an application's poor perfomance. That happens too often in D7. I've heard people say like yes this "site" is fast - with page caching turned on - but it's barely usable for anything that is not read-only anonymous pages. 

3) Let's run it again with more time! I am unable to disable internal page caching. This is supposed to be a module now, but it is required in the standard install profile, so I cannot disable it? The numbers have gotten worse compared to the last benchmark. This is about 1/5 slower than Berdir's benchmarks.

PHP7, page cache on:
Requests per second: 81.23 [#/sec] (mean)
Time per request: 12.311 [ms] (mean)

PHP7, page cache off:
Requests per second: --- [#/sec] (mean)
Time per request: --- [ms] (mean)

PHP 5.6, page cache on:
Requests per second: 24.47 [#/sec] (mean)
Time per request: 40.863 [ms] (mean)

PHP 5.6, page cache off
Requests per second: --- [#/sec] (mean)
Time per request: --- [ms] (mean)

 

 

You are welcome to help!

https://www.drupal.org/node/1744302 was open since almost day 1 of Drupal 8 or at least as soon as the router was merged.

We (as the twig team) early on worked hard to micro-improve the performance of l() while improving its DX. Later all that work was thrown away as the Symfony CMF generator took over.

Are there things that are too complex in Drupal 8? - Yes, there unfortunately are. We have put layers on top of layers, which had been put on top of other layers already. I imagine in Drupal 9 we will start cleaning up, but its way too late to do anything about it right now - except convert likely some of the most slowest subsystems (TypedData, Yaml parsing, Twig's getVariable()) to native C extensions for PHP < 7 (and maybe even PHP 7) [for yaml and twig such extensions already exist].

So our best chance is caching and micro-caching. Or in other words the battle plan is:

  • Can we reduce the work we do in the "critical path"?
  • If not, can we generate it before we get to the "critical path"?
  • If not, can we cache it?
  • Can we cache it with a good granularity?
  • Can we cache it with a long max-age?
  • If not, can we defer rendering / executing it?

If you have any ideas, benchmarks, etc. how to improve the "critical path" of Drupal 8's performance, please chime in now! Benchmarking (XHProf), proposing solutions, good new ideas are very very very welcome. Looking forward to see you in the issue queues and in #drupal-performance! (if you want) Thanks,

Add new comment

By: root Sunday, April 26, 2015 - 10:00