APC vs Zend Optimizer+ Benchmarks with Symfony2

If you are following some of the biggest names in the PHP scene you may have seen that Zend Optimizer+ is now open-source on Github at https://github.com/zend-dev/ZendOptimizerPlus and it seems that will be included in PHP 5.5 distribution.

This is awesome news as we will finally get a PHP Accelerator bundled with the main language distribution and there should not be (let’s hope) weird issues like the ones happening in early versions of PHP 5.4 with APC that has prevented massive adoption of the new stable version of our favorite language.

There are some really promising performance tests available here but at the time of writing this post there are neither Symfony2 nor Zend Framework 2 benchmarks so I decided to grab the extension and do my own performance tests.

So… let’s see the results

Hardware and software used

What is tested? Basically the simplest application you can develop with Symfony2 (Version 2.2.0).
Composer was run with -o flag which, as you should know, optimizes autoload creating a classmap (except the test which is said that it war ran without it)

Machine used is my laptop, basically a Macbook Air with 1.8 Ghz i5 processor, 8Gb RAM and SSD HD

PHP version is 5.4.12, APC is 3.1.13 and OS is MacOSx Mountain Lion

And the tests are run with Apache Benchmark against a Macports default Apache installation with these settings:

1ab -n 5000 -c 10 <symfony2_url>

A Symfony2 Hello world

I’ve seen some “Hello world” performance tests around the Internet using Twig but I guess the fairest test for a Symfony2 minimal footprint is some controller like this one:

 1<?php
 2 
 3namespace RicardClau\Bundle\BenchmarkBundle\Controller;
 4 
 5use Symfony\Bundle\FrameworkBundle\Controller\Controller;
 6use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
 7use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;
 8use Symfony\Component\HttpFoundation\Response;
 9 
10class DefaultController extends Controller
11{
12    /**
13     * @Route("/benchmark", name="benchmark")
14     * @return \Symfony\Component\HttpFoundation\Response
15     */
16    public function benchmarkAction()
17    {
18        return new Response('APC vs ZO+');
19    }
20}
21

Results with Xdebug

PHP 5.4.12 without APC, but with composer optimization

~ 37 req / s (really poor performance)

PHP 5.4.12 with APC 3.1.13 no custom, but without composer autoload optimization

~ 155 req / s

PHP 5.4.12 with APC 3.1.13 no custom

~ 180 req / s (nice boost with composer autoload optimization!)

PHP 5.4.12 with APC 3.1.13 and apc.stat = Off

~ 195 req / s

PHP 5.4.12 with APC 3.1.13, APC Universal ClassLoader and apc.stat = Off

~ 195 req / s (almost same performance if you use composer autoload optimizer and stat is set to off)

PHP 5.4.12 with Zend Optimizer+ no custom

~ 211 req / s

PHP 5.4.12 with Zend Optimizer+ recommended settings

1  zend_optimizerplus.memory_consumption=128
2  zend_optimizerplus.interned_strings_buffer=8
3  zend_optimizerplus.max_accelerated_files=4000
4  zend_optimizerplus.revalidate_freq=60
5  zend_optimizerplus.fast_shutdown=1
6  zend_optimizerplus.enable_cli=1

~ 220 req / s

PHP 5.4.12 with Zend Optimizer+ extreme settings

All of the above plus

1  zend_optimizerplus.save_comments=0<br />
2  zend_optimizerplus.enable_file_override=1

~ 202 req / s (which is really weird to be lower actually but I’ve tested it many times and seems to happen in my setup)

Please note that to make this last test work properly, you should have your Symfony2 cache warmed before the test if you are using annotations. Otherwise these settings will make your annotations get ignored!

Results WITHOUT Xdebug

(thx to @pborreli suggestion)

PHP 5.4.12 without APC, but with composer optimization

~ 40 req / s (really poor performance)

PHP 5.4.12 with APC 3.1.13 no custom, but without composer autoload optimization

~ 249 req / s

PHP 5.4.12 with APC 3.1.13 no custom

~ 260 req / s (nice boost with composer autoload optimization but less impressive than before!)

PHP 5.4.12 with APC 3.1.13 and apc.stat = Off

~ 280 req / s (still nice boost)

PHP 5.4.12 with APC 3.1.13, APC Universal ClassLoader and apc.stat = Off

~ 280 req / s (almost same performance again if you use composer autoload optimizer and stat is set to off)

PHP 5.4.12 with Zend Optimizer+ no custom

~ 307 req / s

PHP 5.4.12 with Zend Optimizer+ recommended settings

1  zend_optimizerplus.memory_consumption=128
2  zend_optimizerplus.interned_strings_buffer=8
3  zend_optimizerplus.max_accelerated_files=4000
4  zend_optimizerplus.revalidate_freq=60
5  zend_optimizerplus.fast_shutdown=1
6  zend_optimizerplus.enable_cli=1

~ 337 req / s

PHP 5.4.12 with Zend Optimizer+ extreme settings

All of the above plus

1  zend_optimizerplus.save_comments=0
2  zend_optimizerplus.enable_file_override=1

~ 312 req / s (again slower)

Some Excel graphics

Conclusions

First of all, I would like to say that these results must be taken with a pinch of salt as they were made in my personal laptop which is NOT a production server. Operating System is MacOSx so they might differ a little bit on a Linux machine prepared for production. Having said that, there are some interesting facts:

With APC, Symfony2 performance is dramatically boosted!. What would we do without PHP Accelerators? Thus, the importance of having an accelerator bundled with the standard distribution
With composer autoload optimization we might get a benefit of ~20-25% even with a small test like this. Don’t forget it in your production environments!. However, disabling xDebug made this benefit get reduced to ~10%.
With apc.stat set to Off, we get an extra boost of ~8-10% even in such a small test. This gets bigger when the amount of files opened increases. Don’t forget to consider it. However, remember that whenever a change is uploaded you should reset APC so that your changes are noticed.
Xdebug make things SLOW. However, most of the % are similar, except maybe the performance boost with composer autoloader optimizer. Please check that you have not it enabled in your production servers!
Zend Optimizer+ is faster than APC, even with the default installation. With recommended settings and in the best scenario we get our requests to be 10 times faster than PHP with no Accelerator!
With recommended settings we get a performance boost of ~11% with xDebug and of ~15% without it, if we compare with APC best performance, but there seems to be something wrong with the extreme settings which were getting worse performance. This was consistent in 5 test I did in different moments, so it seems we can discard that there might have been some background process consuming CPU.
Is it worth to switch? Well, it depends, at least at the moment of writing these lines. If you are not using APC as a local storage cache, it seems that you will get an extra boost for free, which is always good. Also, if it is bundled with the standard distribution, there is one less PECL package to get into account for automated machine deploys. However, if you are using APC as local storage in your application, I don’t think the performance boost is worth if we’re losing the easy caching that APC provides.
As Lukas Smith states in his comment, there is some work being done with ApcU (which can be seen at https://github.com/krakjoe/apcu) which will be APC minus the byte choice cache. If they make it, with the launch of PHP 5.5 we will have an optimizer bundled with PHP and we will still be able to install ApcU. This will allow us to benefit of the performance boost in ZO+ while not losing APC storage and upload files hook. It definitely seems PHP 5.5 will be another amazing version of our favorite language with Generators and Zend Optimizer+ being the biggest improvements!

Please, feel free to add any comments to these tests!