April 14, 2009

Test automation candidates

Filed under: automation — SiKing @ 4:59 pm

This post is about a topic near and dear to me: test automation. Surprisingly, the first one. 😛

I am working on a new project which requires UI (front-end) automation. I am one of the few that actually makes a distinction between front-end and back-end automation – completely different set of tools for the task IMHO. Anyway, new project … UI automation. My boss asked me to do a little analysis of what is out there, and come up with some recommendations. I immediately grabbed for what I know, but was told to make things fair and that I should also look at some alternatives, especially the big boys.

HP: A lot of people in the industry keep gushing about “QTP”. Check out this totally obvious and easy to remember URL:^1322_4000_100__ – this is not the download URL, this is the main page! 🙄 So I get on there. They are obviously going to try and get a vacuum hooked into my wallet, so I go through the 5 minute registration process, didn’t even lie about my contact info, then they mail me a URL to the 1.3GB download; servers kept timing out, but eventually I got through it somehow. 2 hours later I’m ready for the tutorial, in the hopes that I will pick up the gist of the thing. The only thing I found out is that the trial license does not include enough components to even get through the tutorial. I e-mail sales@hp, nobody got back to me.

Borland: A good friend of mine is working with Silk, and can’t say enough good things about it. Way back, I used to work with some of their stuff; these guys invented the idea of an IDE AFAIK. Did not even realize the company is still around today. Basically same situation with the download as above. However, sales were a little more proactive, very little. Basically, the sales drone called me up with what is the problem. I told him that servers keep timing out, and if there is any chance they could ship me a CD. He, very slowly, responded with: “Well, you know, our software is rather complex and not easy to use.” I felt like he is telling me that I am probably too dumb to use their software. So I thanked him and hung up.

IBM: I was, by far, the most impressed with them … almost! First, I got here via Rational Robot. Apparently IBM gobbled up the whole thing (Rational that is). Their site, out of the three (maybe four), is the worst to navigate. Somehow I downloaded Robot, and started going through the tutorial. After a little while I simply gave up in frustration. This has to be the most unintuitive piece of software out there. Their sales got in touch with me, without any prompt from my end, and immediately invited me to a one-day seminar. Sure, why not. I told them about my download troubles, and asked for a CD. “Absolutely not a problem, we’ll have one waiting for you at the seminar!” At the seminar, they confirmed that Robot indeed sucks, and they are only keeping it around for the name and legacy projects! But they got this new thing called Rational Functional Tester: installs as a plug-in into Eclipse or Visual Studio (whichever is your poison), and the test recorder dumps out either Java or VB.NET tests. We are a C# house. Then they have this entire management platform built around it, which they spent the next 6 hours of the 7 hour seminar showing off. My reaction to it: we would need to at least double our management staff to have enough people to click through everything. I mean I can see a lot of potential in this platform for a very large company … which is exactly the reason why I don’t like working for very large companies: because they use software exactly like this to manage stuff, and it ends up being just a time sink. However, during breaks I kept hassling one of their consultants about the Functional Tester. He showed me this thing they call Data-Driven Testing: you record one test, parametrize it, and then you feed into it data from something like an Excel spreadsheet, and it goes through everything. “Neat! So how do you set this up in RFT?” “Unfortunately, RFT does not do any of the work for you, you have to code everything up yourself: the data, the data parser, and the driver.” “Hmmm, OK, so how difficult is to code up tests from scratch.” “Well, first you need to point RFT at your application, so that it learns all the controls.” “Can’t I code those up myself?” “No, that part is proprietary. But it will become part of the IDE as methods and you can then use code-completion when you are coding up your tests.” “What if the tool cannot recognize some obscure control of my app that I need?” “That’s what we have consultants for!” “And how do you hook it all up to a continuous build system?” “Well you have something monitor say a file on a server, and then start up the IDE and kick off the tests.” “No, I mean how do you run it unattended?” “Yea … monitor a file … start the IDE …” “You mean I need to install the IDE (in my case Visual Studio, a licensed IDE) on the server where I am going to run tests unattended?” “Yes” 😀 “So in my case, the license that I pay you guys would basically be only for the UI control recognition mechanism, as everything else I have to code up myself, and if that happens to fail I have to pay you more money?” “Yes” 😀

We’re going with Selenium. More info, less fluff, coming soon…



  1. There was a day not so long ago when Selenium was evil incarnate to you.

    Comment by Paul H. — September 1, 2009 @ 9:41 am | Reply

    • Still is. Only people who were really bad in previous life are forced to do UI development/testing. Interacting with Users, on any level, is just wrong for a computer guy.

      Comment by siking — September 1, 2009 @ 1:07 pm | Reply

  2. Avoid most of the HP software partners like Utopia, Cortechs, SPAN, Orasi, Genilogix, Pepperweed, RTTS, Powertest, Loadtester, etc. They suck.

    Brad Purcell
    Sky IT Group – New York City

    Comment by Brad Purcell — June 17, 2009 @ 7:24 pm | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: