Narayan Newton

I'm going to be honest, I'm kind of obsessed with caching. Since becoming involved in web performance, it has pretty much taken over my life. Yes there is the mysql tuning, query profiling, solr deployments....but all in all 90% of my day is looking at hit rates of one form or another. Due to this, HTTP caching for drupal.org has been a mild obsession for me. This week my wish came very close to true.

I've recently been setting up a new Apache Solr search cluster for drupal.org. Hopefully this is going to be the "final" solution to our search issues. During this project, I noticed that Solr can now set headers for Reverse Caching Proxies. This is really cool! I went off to test this and setup Varnish to proxy my Solr connections. First, I setup Solr to send Cache-Control headers with a 60 second timeout for all requests. I may tweak this later, but this is fine for testing. (Note: Solr can also send If-Modified-Since headers) I then setup varnish with a very simple configuration.

I'm a pretty big fan of Google's 20% time. To those who don't know, this is a policy where employees are allowed to work on "their own" project for 20% of the time. For some managers, this would seem terrifying or just stupid. However, in reality it allows your employees to work on what they are passionate about and, shocking I know, sometimes they have good ideas and their ideas make you money. At the very least they gain experience. I'd much rather have an employee learning new things on my dime than stagnating on "their own time."

I've been following MySQL 5.1 for awhile now. It has many interesting/exciting features and improvements, but I have had a long-standing view that it was released too early and is not production ready. With the partial release/sneak-beta of MySQL 5.4, a performance-centric release based on 5.1, I've decided to take a look at the bugs I've been following in 5.1 and re-evaluate its production ready-ness.

So, long ago in a galaxy far far away there was a table engine called InnoDB. It was transactional, it was fast...it was briefly not owned by Oracle. Life was pretty awesome. However....backup sucked. Big time.... To do a consistent point in time backup, we needed to do a single transactional dump of the entire database. This means that we need to hold a short-lived lock at the beginning of the dump (which can be problematic) and that it takes _forever_ to dump and restore.

At Drupalcon DC this year, Kieran and I gave a talk on the current state of drupal.org infrastructure. David Strauss and Derek Wright ended up joining us on stage. It was an interesting chance to get a large part of the drupal.org infra team all on stage together to answer questions.

The video is available here: http://www.archive.org/details/DrupalconDc2009-Drupal.orgInfrastructureS...
The slides are available here: http://nnewton.org/sites/default/files/Infrastructure_team_presentation.ppt