Jeff Foster is a Software Engineer working on Red Gate’s SQL Source Control product. Jeff and his team have recently begun investigating the use of Amazon’s EC2 to run engine tests and he was kind enough to sit down and talk about the process.

What made you decide to look into using the cloud for your testing?

Currently, our test runs take about 24 hours to go through in total, during which they consume around 70 hours’ worth of CPU time. We already have quite a few virtual machines running where one machine takes about 8 hours on one sort of test while another machine runs 8 hours on a different test, eventually it adds up to about 70 hours.  This means is takes about 24 hours to get feedback which isn’t as quick as we’d like it to be.

What we want to do is move testing to the cloud and run the tests on a million machines getting feedback in a second! In reality we’re using about 80 virtual machines and get feedback in under an hour.

What sort of testing are you doing?

We’re running engine tests. SQL Source Control is licensed per user so something like load testing doesn’t really apply. We have a series of tests that work the application without using the graphical user interface so they cover all the features we’ve done, all the regressions we have, and so on. The tests make sure everything works together as it’s supposed to.

One of the main problems that has frustrated us is when we add a new feature, we commit that, we go home, the tests finish a few days later, THEN we notice we’ve broken something else. By shortening the feedback cycle it’s going to make a massive difference to our response time.

Sounds a little like continuous integration?

Kind of, but continuous integration is a bit different. With continuous integration you run a commit, you run your unit tests, then you get your result so that’s kind of okay, but it still takes too long for us. The engine tests are closer to integration tests than unit tests so they test the app as a whole. There’s a limited amount of value we can get from unit testing in my opinion as unit tests look at the individual pieces of the puzzle without putting it together, so for us engine tests are a lot more valuable.

How far into the process are you?

So we had a week-long time period booked out a little while ago to look into improving testing which is where the project started. By the end of that week we demonstrated how one lot of tests used to take 6 hours, with EC2 we could get them dramatically faster to about half an hour.

After that week, we didn’t have a whole lot of time to look at it again for a while until recently. We poked around making changes to the algorithms we used to take into account what we did wrong. Now we have something that works, just not quite as efficiently as we’d like, which is the final bit of polishing.

How easy was it to get started with EC2?

The biggest thing was that the EC2 machines aren’t in the office, it sounds like a little thing, but, a lot of our tests were trying to use things in the office network. We had to change quite a bit of our testing code to make them more independent of the machine it’s run on.

Is that ultimately a positive thing?

Absolutely, it was brilliant because it forced us to go through the exercise. On the EC2 side, it was a cakewalk. EC2 is really easy to use so we didn’t have too many problems with that.

Can I get an overview of how it works?

If you imagine there’s one machine at the front, we run the tests to that machine, then that machine talks to all the other machines and tells them to run these tests.  Those machines then report back and say, ‘I found 1 million and 1 tests to run in that package’.  We need to make sure they all report the same number otherwise weird things will happen, really, it can get very creepy. The master machine in the front then allocates tests from the package, so one machine does every tenth test starting at number 1, a different machine does every tenth test starting at number 2, and so on until each machine has a slice of the testing.

Do the machines cater for difference scenarios, such as different OSs?

No, they’re all exactly the same OS.  With Amazon you get to provide your own image to them with what the machines should have on them, for instance SQL Server. Alternatively you can take one of Amazon’s images as a base then install whatever else you need.   Then you start up the VMs with whichever image as the base. EC2 only supports the server operating system.

So, what’s next?

We want this type of testing to become part of our daily lives on the SQL Source Control team, so we want to see that benefit of being able to break something and finding out very quickly rather than waiting another day to find out.

What do you see the time-saving becoming?

You have to pay by hour so if it takes 30 minutes we’ve wasted the second part of that hour. We’re looking for results in under an hour, that’s the goal. There isn’t an unlimited budget so we want to make sure we’re making the most out of our investment. Our testing isn’t really tied to Amazon, so we can change providers if we find something cheaper.

How long did it take from the initial idea to when you completed your first test?

We were able to get our first test running within the first week we looked at it – it was really only 2 days of actual work to get it going. The biggest thing was our code depended on being in the Red Gate building, so that wasn’t any part of EC2, it was just something we had to do.

When do you expect it to be ready?

As soon as a gap opens up really, it should be about 2 weeks of work to get it to where we need it to be.