Server optimization and bottleneck troubleshooting.

We are not looking for someone to optimize the server on our behalf, instead we are looking for suggestions to what may be causing the unstability described below and possible fixes. Background information: We have two Dual-CPU 32GB RAM SSD servers. One is used as a web server and one is mysql database server, the servers are also privately connected through LAN. The server serves mostly PHP pages with only around 5% of requests requiring mysql lookups. Currently, the web server hits a bottleneck at around around 150/requests per second. Once it hits the bottleneck the server becomes very slow and unresponsive (check the attachment graph). We have checked the following: * MySQL process list shows little or no current running query, meaning that SQL isnt the bottleneck * When server becomes unstable, you can see lots of requests getting stuck in apache queue. Apache status would show thousands of current requests waiting in queue but are not even processing any of them, it looks as if one process is blocking all the other processes. When these requests are stuck the analytics would show a downfall in realtime traffic, after around 5~班seconds Apache starts processing the requests again, the thousands of processes gets completed by Apache causing a huge spike in analytics. * We believe the hardware are not being fully utilized since when it hits the bottleneck, the CPU load on both server are very low. We already tried the following: * The server is running with fcgi+xcache with logging disabled to reduce disk IO. * We have increased apache server numbers and limits, but didnt help at all. * We have moved some caching into tmpfs but didnt help Now we need suggestions on what else we could do to increase the system bottleneck.