Here is an observation which I’ve been been playing back to clients for a year or so now:
- There are two types of company in the world, those who talk about automated testing and those who do automated testing.
- Those who talk about automated testing buy tools like Quick Test Pro, WinRunner and Quality Centre and Rational Test Workbench, Rational Quality Manager, i.e. expensive products from the likes of HP and IBM.
- Those who do automated testing use tools like Selenium, Fit & Fitnesse, Cucumber (inc. Gerkin, RSpec, etc.) and other Open Source products they don’t need to buy
For completeness I should add:
- There are a few, very few, companies which don’t talk about automated testing but on the whole everyone thinks it is a good idea and those who don’t do it would like to
- There are a some companies who talk about automated testing and have no tools; this is a better position to be in than talking and spending money
This is my observation, and if I’m being honest I did see a company 12 years ago which used WinRunner in a load testing environment. But, on the whole the software test suites from IBM and HP tend to be shelfware. In fact, this blog is motivated by a trip to a company which found my observation very funny because QTP is sitting on a shelf unused.
What I would really like to know is: has anyone else seen this? Or, can anyone give a counter example?
I think there is even a logic here. The IBM and HP products are expensive, so expensive you have to ask the price (actually IBM does give the price of Test Workbench at $5,500 for a single user license). Consequently they need to be sold, it also means that they need to be bought at a high level in the corporation. The people who are sold these products are very disconnected from the day-to-day work of developers and testers. This means the products might not be suitable and even if they are then developers and testers need to adopt them: they aren’t part of the change decision.
In addition: I just don’t think these tools are any good. To be fair, I’m no expert on testing tools but I’ve seen Quality Centre in action. Quality Centre (QC) isn’t an automated testing tool, its a very expensive test tracker. Testers write their test scripts in natural language, e.g. English, into the tool, the tool runs on the side and as they execute the tests they click Success or Fail. At the end QC gives a report.
To my mind QC is a fancy version of Microsoft Word. English language test scripts, if they are any good, are so detailed that they take almost as long to create as automatic scripts but they are expensive to execute because they are manual.
QTP and similar products are often linked to the OS or browser. They result in fragile tests which break easily when a box moves a few pixels or the browser changes.
HP also follow a razor blades model, I’m told: once you’ve bought QTP you need to buy a plugin for every browser to OS you want to use it with. Every time a new version of the browser comes out you need to buy a new plugin – or so I am told, tell me if I’m wrong.
The net result is: these tools are high maintenance and easily fall into disuse if they are ever used.
On the contrary: Open Source tools are adopted by people who do the work because they see the benefit they bring and they don’t need budget or signatures to get the tools. People get them because they want to use them not because a Salesman comes around and convinces the boss that you need a tool.
And because many of the Open Source tools require developers to create glue code to interface the tool to the application they lead to a more collaborative style of working.
(Perhaps some companies who just talk about automated testing have downloaded some of the Open Source tools but never use them tool. Since they are free nobody notices.)
So there is my theory. Anyone else seen this? Anyone else explain it? Anyone got a counter example?