When buying a tool...

...or pretty much anything to bring into your group there are some basic things to consider.  Now most of this can be found in many of the automation books out there, or in the QA Forums if you search for it, but I wanted to put down my own personal checklist.

1. Consider who is going to use the tool - if its just you, and only ever you, then whatever fits your skillset is fine.  If other people are going to use this, not everyone will have the same experience, and some people like using detail-driven tools, others do not.  Make sure it fits the audience.
2. Don't go for whizz-bang cool stuff - there is always a hot tool out there but it may not be a fit.  Picking something just because it looks cool, or is going to be resume fodder, is not the way to make a wise decision.  Go for a tool that works, not a tool that sounds neat.
3. Make sure it fits the project - not every tool is simple and easy to use, in the midst of a project bringing in a tool that requires a lot of focus and start-up time, such as training and installation, is not wise.
4. Plan for future needs - again, focusing on the here and now is fine to get you through the immediate needs, but at some point all you end up with is a bunch of tools that get you through a defined set of problems.  Thinking ahead, what will be other platforms that will be supported, other technologies coming in the next project, all of this should be taken under consideration.
5. Do a demo - take a small subset of what will work and run a prototype, or a demo of the tool.  Open Source projects can be used at any time without issue, but if its a commercial product see if they will allow you to demo it against your tool yourself, many will give you 30 days or so to check things out.  If you build your own, do small prototypes that you can expand on or gain the knowledge you need to be able to do full builds.
6. Check more than one tool - don't decide on the first one that works, do some due diligence.  This is the hard part, because it often when you find one that looks like its the best thing ever, and will do everything you want its good to take off.  Do your research and check every item on your list, sometimes you will come up with more than one, then its a matter of what will fit the team, environment and needs.  You can often end up with a better candidate this way.
7. Talk to people - while we all have our own ideas of what the needs are, others who may need to use the tool will also have requirements.  Sometimes something at the back of their minds that is hard to use, or do or something that would make work easier, get all these items into a requirements list and document them.

A lot of these steps are important in one critical step, the rationale behind why the tool is needed and why it will work, you never want to be stopped by someone who says "Well why do we need this new thing if the old one works?".  You should have the information at your fingertips to say, "oh, that's because we can do A and not have to worry about B and C..."

If you can't talk about why the change will help, then you haven't done your work.

Rethinking the Test Harness

Recently I went through the exercise of updating the test harness I was using, and I was glad to see that I did it very similarly to Elfriede Dustin's recent book, I did get the book after I was done and read it as I was going through my coding but the important part that set me up for everything was the planning.  I already knew the test harness we had was inadequate for the tasks we were adding onto it, this was one of those organic harnesses that grew over time in the company, a lot of people used it and added what they needed to it.  Starting from seeds (scripts) it blossomed over time into a very complex organism with lost of scripts, branches (directories of new tests) that were linked into others and eventually its own library of functions to pass among the various scripts.  At one point I helped in added some object orientation to the scripts, which at this point were 90% in Perl, this helped us have more reuse and there was a lot of reuse, that is calling, of scripts that were previously written.

Then there was an adjustment in the projects and the test harness needed some adjustment, there were some options that were needed, such as more remote access into scripts being run, especially if they locked up a machine's resources.  Also logging was something I wanted for a long time, not just for the scripts that generated output, but also for any script run I was looking to having a database maintain some results so we can have a more permanent record, and review times when the scripts would on occasion fail without any real understanding of why.  So I slimmed down the directories, removed scripts that were no longer called and decided it was no longer a good idea to check in compiled binaries, I never liked that anyway.  Of course to do this required keeping the current harness working, while redoing everything we had, and we had a project lull so I stepped back and took a look at what we had.

What we had was a large organism that had grown through acquisition of tests by anyone who wanted to contribute, some tests added just to meet a specific condition a customer encountered long ago and was now tested by something else.  Also, there were scripts that were no longer called and basically wanted to run in directories of people who had not worked at the company for years.  So I stepped back further and said, if I was going to do this over again, what would I do?

So I started looking at other open source options, I like toolsmithing and the exploration process so this was a good fit for my needs, plus it fit the budget and there was already plenty of information on the internet from other people who had tried the tools.  So rather than redo everything myself I was going to use a tool to help us organize everything, with open source we had an option to meet the company budget for tools at the time ($0) and there was support in forums and on websites.  Next I had to look at what we needed to run it on, at the time we were supporting Windows and Unix (Linux, AIX, HPUX and Solaris) with multiple versions of each platform, so any tool needed to run on all those - a tall order, and eventually in a reorganization the platforms got pulled down to Windows and Linux and finally Windows.  Always a good idea to know what platforms you are supporting, and since it was not going to be just Windows forever, I kept all the platforms under consideration.  Once that was done I looked at the pros and cons of each tool I came across, made myself a matrix on each tool then wrote out what I wanted to have the test harness do, I could have written the harness document first but I wanted to get some influence first on what the capabilities of other tools can do.  That way I was not making a hard to reach design, knowing I wanted logging, a database, a UI for the database and reuse existing scripts was enough to do my research.

At the end I had a document specifying what I wanted the harness to do, a matrix of possible tools and having meetings with other people in the group, and in Development, I got together everyone who might possibly use the tool and solicited opinions.  In particular I asked the Architect of the product for opinions, having a high level thinker add even more to the document helped me generate a good tool, I did need to explain what it was that I designed, but it was good practice to be able to talk out what the harness was and should do.  In the end I think these steps helped me create a very solid foundation to create the harness on, and before the company folded I had the structure in place with a database, a UI to the database, some agents and clients running to do tests and check results and store them.  Very nice, sadly I won't see the design come to fruition but I think the experience was time well served so when I have to do this again I know all the steps I need to do.

Ruby CLI Example

This took me awhile to go through and write but this is the main class I am using for my test Harness command line argument parser.

class TestCLI
    # This is the main CLI for the Test Harness, all
    # test options should be listed here
    OPTIONS ={
        :agent  => ['-ag', '--agent'],
        :driver => ['-dri', '--driver'],
        :help   => ['-h', '--help'],
    }
    USAGE =<    This program understands the following options:
      -ag, --agent      : Run agent based tests
      -dri, --driver    : Run driver based tests
      -h, --help        : Displays a message with usage instructions
    With no command line flags, this message is displayed.
END_OF_USAGE
    def parse_opts(args)
        return option_by_args(args[0]) if understand_args?(args)
        # options are not understandable, so display Usage
        puts "Options are not understandable."
        display(USAGE)
    end
    def display(content)
        puts content
    end
    def do_default()
        puts "This is the default section."
    end
    def option_by_args(arg)
        return agent_tests      if OPTIONS[:agent].include?(arg)
        return driver_tests     if OPTIONS[:driver].include?(arg)
        return display(USAGE)   if OPTIONS[:help].include?(arg)
        do_default()
    end
    def understand_args?(args)
        OPTIONS.keys.any? { |key| OPTIONS[key].include?(args[0]) }
    end
    def agent_tests
        agt = system("rutemax -c conf\\agent.rutema all >> logs\\agent_tst.log")
        puts "Ran agent tests.\n#{agt}"
    end
    def driver_tests
        # Probably need something like this
        #   rutemax -c conf\driver.rutema all
        drt = system("rutemax -c conf\\driver.rutema all >> logs\\drv_tst.log")
        puts "Ran driver tests.\n#{drt}"
    end
end

I am running two major test types right now, for the Agent that we produce and the filter driver that is mainly what I test, I utilize Rutema, an open source test execution and management tool to run my scripted tests.  I am using HEREDOC to get the options text out because its multi-line, something I need to convert in another script due to its use of multiple puts lines.  Basically this class parses through each option and then goes to the appropriate function when it finds one it uses, the de_default is a placeholder until I figure out what I want to have there.

All in all I think its pretty good, considering I have only been doing stuff in Ruby for a few months and basically jumped into doing this project to rewrite our Test Harness into something with more reporting and stability.  I'm also not a trained programmer, so if I can do it, so can you.

Ruby Backtick Capabilities

Ruby can run some programs as if on the command line, the mechanisms are pretty much the same as Perl would run them, backticks are a blessing in many respects.  It took some finding but the main three mechanisms are:

• Kernel.system
• Kernel.exec
• %x[]

For more on these I went through the Ruby book but also this site to find out more on how these options operate.  This took me hours to find so I wanted a space to hold this info so I could find it again.

The First 24

Another subject that has come up in company meetings is the concept of the First 24.  Something we have all encountered whenever using a new product, piece of software or even a new car.  Those first 24 hours help to make or break a User's perspective of a product, if you can't win the User over in that time you may not over time, as the saying goes you only have one chance to make a first impression.

Consider the time taken to install a product, if the install process is onerous, complex or unintelligable in parts Users won't proceed, the install process is really the first contact a User has with a product.  Forget all that fancy packaging, its all to get eyes on a rack in the store, and with a lot of software being sold online these days it doesn't matter as much as reviews, and if Users can't install your product then bad reviews will follow.  If you follow the Amazon model and have a rating system, then most people are not going to buy software that has bad reviews, future Users who do research online will look for comments from other people.  If there are numerous comments about installation issues or problems, or people looking for advice and help on forums, that information will be there and can be a first negative to potential Customers.

Once past install is the Out of the Box experience adequate?  Is the product usable from the get-go or is there need for Customization?  When I have used software I often like to just get in and check it out, basically my tester's instincts to do a post-install review - is everything in place and working.  Can I start the product, access the help screens, am I going to get a license or registration  notification right away, these are things I want to know.  I may even start clicking around on screens and check out menus to see what is where, and if its understandable to me before I've gone into the manual or tutorials in-depth.  Problems at this stage will probably engender a support call, or a search on forums or through an internet search engine to see if someone else has the same issue.  If it can be fixed easily or needs an update I often forgive updates, after all I work with Windows and software, there are always updates to be had and its normal - so I may forgive that.  What if the software has a nasty crash?  That tends to be irritating, and if it can't be fixed or has a large number of steps to fix then my view of the product goes down, I shouldn't need to go through a lot of configuration to get something working, unless it is a complex product in which case I know this up front, or I am engaged with a consultant to help me out.  If I have gotten past installation and the first use then I'm going to be happy and have a positive view of the product, this is a good test to run in addition to any others, if you have a User Acceptance Test it will incorporate these kinds of things to make sure it will meet your Users expections and make their first 24 a good experience.

I mentioned using a car earlier, and let me close with the best example I can come up with, while comparing software to a car is not really the best example one can make the steps one goes through to purchase a car and software is the same.  When you pick up your new car you want it clean and ready for you, maybe with a new tank of gas, but there should still be that new car smell inside and the mileage should be similar to what it was in the test drive or what was needed to get the car off the delivery truck.  Still, when you get that car and sit in it when its yours for the first time, drive it off the lot and home you expect it to be smooth, smell nice and run well until you get it home.  The first 24 in your car should be a good smelling, smooth and happy experience to make you feel good about putting down all that money on the car, just as that experience should be nice so should software, but without the new car smell.

(Sorry to dissappoint you Jack Bauer fans but yes, it was intentional that I not mention him at all, well until now to mention that I did not mention him.)

Monthly Topics

It's June and it means its another montly topic, and even though the group likes having the QA Talks every month I find it hard to keep people coming up with something they want to talk about, or even do a presentation.  So most months it falls to me to make something, last month I did Regular Expressions since there is more Perl being used, and a few others on the team are learning how to write scripts for the automation framework.  So either I am better than I think, I doubt it, or no one wants to do the work.

I usually need to be inspired to write a topic, its not easy to just up and present for 20 minutes, and I try to maintain interest to keep people's attention so its tough to make good technical topics that are enjoyable.  Not sure yet what this month will be about, but having read some more lectures by Yifa, a Buddhist nun associated with my wife's temple, I may have something on how to break down tests.  Funny how we find inspiration in unusual places.

I'll probably come back and add but topics I have done or have thought about include:

• Test Heuristics - DONE!
• Agile & Scrum - done but always needs more
• Basic Debugging - DONE!
• Driver Debugging - DONE!
• What is Quality? - DONE!
• Regular Expressions - DONE!
• Find your Buddha (seeking Test Nirvana)  - DONE!
• Test Frameworks
• Database Test Techniques
• Ruby
• QTP Testing - DONE!
• Accessing Data Query Methods in Web Services API - DONE!
• Blink - Thinking Without Thinking (Overview of the Book by Malcolm Gladwell) - DONE!
• DNS and product usage - DONE!
• Product Installer - DONE!
• Virtualization - DONE!

That's all for now, I may update the list later on.

Automation is not easy, nor should it be

If you've ever been in an organization that starts to get some process around itself you will discover in testing that someone, at some point, wrote a script.  That script was the beginning of other scripts, which eventually became under the control of maybe a master script or were all bundled into a suite.  When that bundle, or suite, was run time and time again and the results were known and used, you suddenly have the good beginnings of a Test Harness.

It can take time to develop, but that's what those scripts are, development.  Script Development, with all the process and care that goes into the code that comes under test, so to should those scripts being written be vetted and reviewed by someone else.  It's not "just for QA" when the scripts are doing the work and validating someone elses code, this is something that has become a tool which is being used for a purpose, that purpose is validation.  When you validate something there needs to be a story to back it up, someone has to be able to stand up in front of the crowd and proudly say "this is my script and here is what it does!".  This invites all kinds of questions and discussion and debate, it does happen, but you know this is not a bad thing.  Discussion and debate is good for not just the Requirements and the Product, but for those little things that help the product get out the door, because that script is code and code needs reviewing, and commenting, and that is work that needs to be done.

Automating work takes time, dedication, patience and practice.  It's not easy to generate a framework that will allow you to run tests, especially a framework that you can run and say that when a test case fails its because the product has a problem and not the automation.  Automation is hard work because you need to develop something that will aid you in your goal of running tests and validating the product, but its also a project on its own and needs its own schedule, set of tasks, care and review.  Just going out and writing a script is fine, if all you want or need is a script that does X and never Y, but sometimes there is a point where there is a need to do X, Y, Z and all the other letters before them.  It's not easy, and it should not be because a badly written and implemented Automation system does no one any good, but it doesn't all need to be out at once either, it can be done in stages, just like a real project because after all that is what it is.

I've done this already, and this is the background for what I have here.  When I came to my current place of employment there were a lot of scripts written, there was even a beginning stab at automation, done by someone else who was here.  We worked together and extended the automation scripts and built a nice little test harness, it didn't test everything, but it tested a good piece of functionality consistently and routinely and gave results.  So we knew where things were and we had discussions about where we wanted the automation to go, saw what was lacking and talked about how to fill in the gaps and brought Developers into the discussion to get their feedback.  The other person left and I took over, I kept up with what we had discussed but now that I was driving I was talking about the automation but fewer and fewer people were talking back, eventually it was just me.  Then the product took a new direction and I said to myself "hey me!  Let's get back on board and do this right and fill in those gaps!"  Because I had been patching the system we had for awhile, working around it when the scripts we had were breaking because they were starting to be used in ways we never intended, in between testing and writing the automation the automation became a part time job.  During the break while the new direction was being created, I came up with a full blown plan on what we needed, how we were going to get there, brought people into the conversation who were excited about it and we made a staged release plan.

Now we are building some cool stuff, people are talking about it, other groups in the company find it cool and think its a nice idea, and they want to talk about it, and I am feeling reenergized in what I am doing because we will have some really nice stuff when this is all done.

About a year from now.