Hello, everyone.
That’s gonna be our first post in English here. The reason is that I personally believe that it might help someone in case they meet with those same strange issues we ran into recently.
In November of 2011 we started a new project for our client and chose Symfony2 as a base framework. First of all, we really wanted to try it in real life as lately there are a lot of things happening in the PHP community around Symfony2 and related projects. Also, we had a very limited amount of time to bring this project to life. And gathering information about different frameworks we decided that Symfony2 would be the fastest in means of speed of development.
As for the latter, it happened to be absolutely true. Working with Symfony2 is great pleasure and fun. We developed the base functionality in about 1.5 moths and then started to clear out bugs and add new features. I think we’ll describe some of the moments about development with Symfony2 in nearest future.
A couple of weeks ago the site replaced the old one on its main domain and strange things started to happen. Our logs started to fill with occasional “non-object” errors with stack traces in framework code and the site just returned blank pages on them. We started to experiment and researching. Our first mistake was that we placed this project on Amazon EC2 Micro intstance and haven’t read about CPU resources on it. Symfony2 appeared to eat CPU quite extensively. We switched fast to Small Intstance, CPU graphics was fine there but errors continue to appear repeatedly.
After some time I posted a question on stackoverflow hoping that anybody met this kind of bugs. Thanks for everyone who tried to help with ideas.
We made all kinds of possible logs, tried the project with different APC settings and without APC at all. Also we tried both apache2+mod_php and nginx+php-fpm. Ryan’s suggestions about memory issues were a most possible variant, playing with APC didn’t help, but it gave me the idea and we started to scan bug trackers and changelogs of all pieces of our server software stack for memory and non-object issues.
Finally, we just upgraded PHP on the latest server that we tried to the latest version 5.3.10 replacing the latest version available from aptitude. And error flow just stopped. Actually, I don’t know why it wasn’t the first thing that came to our minds.
Here I want to say special thanks to our big friend and good specialist Ilya, who assisted us with his help during all the time we were fighting with the problem. Actually, it was him who gave us one more server to test site and it was he who just upgraded PHP and everything became just good.
This situation gave us a good lesson: both in business and technology. During those days I learned a lot of things about server configurations and interactions that I couldn’t imagine I would ever learn. From a business point of view, we learned, how to explain bad situations to the client. But the main conclusions are of course about all sort of possible improvements that we’ll introduce to our development, deployment and testing processes to avoid the same situations in the future.
Thanks for your attention. Hope you’ll never meet any kind of Heisenbugs!
