Narayan Newton

I've recently been setting up a new Apache Solr search cluster for drupal.org. Hopefully this is going to be the "final" solution to our search issues. During this project, I noticed that Solr can now set headers for Reverse Caching Proxies. This is really cool! I went off to test this and setup Varnish to proxy my Solr connections. First, I setup Solr to send Cache-Control headers with a 60 second timeout for all requests. I may tweak this later, but this is fine for testing. (Note: Solr can also send If-Modified-Since headers) I then setup varnish with a very simple configuration. We don't need to deal with cookies, so I really just entered my two servers into varnish as backends and set it up to listen on 8080.

To get some data, I took a day of request logs from our current solr server and ran it through sed/awk to get a URL list for siege. I then set siege to a concurrency of 100 and unleashed it on the server.

Without Varnish:
Transaction rate: 178.62 trans/sec

With Varnish:
Transaction rate: 250.96 trans/sec

Not bad and more importantly, vastly less load on the server with varnish. During testing without varnish, the CPU was constantly pegged at 99% usage. With varnish, it hovers between 15% and 40%. The hit-rate against the cache after testing was around 87%. I'll be interested in the hit-rate with real usage.