The dangers of performance testing …

October 19, 2011

I spend a lot of my time talking to people about RAID performance numbers. Almost everyone as their own bent on this subject, and many organisations have specific tests they run against hardware to “check” performance.

The real problem is … what exactly do the numbers mean and do they relate to your real data. An example comes to mind when I sat with a customer in a foreign land using a Linux performance testing tool that I was not particularly familiar with. The customer has an absolute bent on IOPs, and was very keen to see the performance of our card under the single particular performance metric test they commonly use.

The answer: 1.9 million IOPs

I smiled and should have left there and then. That’s possibly the highest number of IOPs any machine has ever produced, let alone a single RAID card. I could sell a million of these things!

However … commonsense took over and I asked the customer how much RAM was in their system. Due to language barriers this took a bit of time, but finally they cottoned on to what I was asking and the penny dropped as to why I was asking it.

Answer – 96gb (in a machine with 2 8-core processors).

So where do you think all those IOPs came from – simple answer is that the system RAM had cached the entire performance test file and all IOPs were being read from system RAM – the test was not going anywhere near the hard disks because the test was only 60Gb in size.

While this might seem like a pretty funny example (I certainly thought it was), it was a simple example of the fact that performance testing is fraught with dangers and variables. Add to that the fact that you really need to be testing a data pattern that is as close as possible to your real-world application and it all gets very messy from here on in.

I have had plenty of customers quoting Mb/sec speeds to me – when I know that customer is using a SQL database and Mb/sec is meaningless for that particular application. IOPs are the go here. I also come across plenty of customers who play with the variables of their testing software – doing such thing as a 128 queue depth when their application is never going to get anywhere near that number (very few applications are that well written, or parallel in nature that they even get to a queue depth of 16) … again, a meaningless test that will produce all sorts of numbers that will never be matched in the real world.

Therefore my recommendation to most customers is to study their applications first, and try to work out what the software will be doing under heavy load (and this one is totally out of my field – I’m not going to study every customer’s individual application) – then they can set relevant metrics in their testing software to determine what sort of performance they “should” get from any given hardware configuration. Be careful of what you ask for – you might just get numbers that look good but don’t mean anything. The trick is knowing “what” to ask for.

There are “lies, damned lies and statistics”, which I believe is a quote that can be attributed to Benjamin Disraeli, the famous English Politician.

I tend to agree.



Leave a Reply

Your email address will not be published. Required fields are marked *

4 − = one

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>