Fitnesse and Selenium as Services

Because I am working with Fitnesse and Selenium, I wanted a way to be able to remotely access tests on a central server, right now I am setting up the services on my own machine. These are the instructions I used to get things in place.

Set-up Fitnesse as a Service:

• Install the Windows Resource Kit if its not already in place
• Run the following command:

instsrv Fitnesse "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"

• If the resource kit is not in that location update the path, but you MUST use the full path to srvany.exe or it will not work
• Modify the Registry key in HKLM\SYSTEM\CurrentControlSet\Services\Fitnesse
• Change the Image Path to point to Srvany.exe's full path (if not already done so)
• Add a new key - Parameters HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Fitnesse\Parameters
• and three string values,
  AppDirectory - AppDirectory is the root directory of our service where the Fitnesse bat file resides
  Application - is the java.exe executable needed to run our service
  AppParameters - is the parameters used by the java.exe to start our Fitnesse service.
• In the Services panel verify that the Account being used to start the service has the right permissions
• Start the service
• To access the Fitnesse Web UI use the URL - http://local_ip_address:port/

Items to note:

• If the Service starts, but does not open the port make sure Windows Firewall is set to allow access to the port.
• If the Port is open but the open port is not appearing in a netstat -a then use the command line arguments to verify that the command line options work.

Example settings for Fitnesse Service:
AppDirectory - c:\fitnium
Application - C:\Program Files\Java\jre6\bin\java.exe
AppParameters - -cp fitnesse.jar;selenium-java-client-driver fitnesseMain.FitNesseMain -p 8888

Selenium Server:

• In order to assure that Fitnesse can run its tests Selenium needs to be set up the same way
• Modify the Registry key in HKLM\SYSTEM\CurrentControlSet\Services\Selenium
• Change the Image Path to point to Srvany.exe's full path (if not already done so)
• Add a new key - Parameters (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Service\Selenium\Parameters)
• and three string values,
  AppDirectory - AppDirectory is the root directory of our service where the Selenium bat file resides
  Application - is the java.exe executable needed to run our service
  AppParameters - is the parameters used by the java.exe to start our Selenium service.
• In the Services panel verify that the Account being used to start the service has the right permissions
• Start the service

Example settings for Selenium Service:
AppDirectory - c:\fitnium
Application - C:\Program Files\Java\jre6\bin\java.exe
AppParameters - -jar .\selenium-server-1.0\selenium-server.jar   

Fitnesse and Selenium

I had decided to go with Fitnesse and Selenium as part of our Acceptance Test Framework, it was a challenge to say the least, and with the options that were out there it was bit of work and testing to find the best combination to get things to work in our environment. Basically, I am testing an update to a HealthCare Process Improvement site, nothing like being a QA person on a project like that I must say! So my tests boil down to being able to login and do certain functions, as well as just roam the site and check for pages that are not working, in the midst of this I wanted some structure so that I could do things like have login and logout before the tests so I just set up a session and do everything. Sadly, it did not work out that way.

WebTest Fixture
I went through Gojko Adzic's Web Test Fixture for Selenium and Fitnesse, this I struggled with for a little while but I liked the ability to use .Net or Java and the language to call Selenium was pretty simple. Though you have to remember to use those !'s on certain commands, something I found through trial and error with Start Browser, and it worked out pretty well. Getting Suites to work is something that I started with here, and had a lot of problems with, I kept getting fixture errors in trying to set up my suite that really became a stumbling block, seems as if I am destined never to get the fixture library to carry through the suite and putting it on every page became a problem after awhile. Biggest problem though was the version that is complete on the site is an older version of Selenium that does not work with FireFox 3, so if you have that on your machine you can either do your own buildup, or try to update the Selenium libraries in the installation. You could probably compile your own code and get things working that way as well, but its not the method I wanted to go with so I eventually gave up on it.

Fitnium
Something I came across and this worked out pretty well, moreso once I resolved my whole Login and Logout issue, and the fixture on every page problem that seems to plague me is something that carried here as well. Still, I was able to work out my issues before getting responses to my posts on the Fitnesse Yahoo Group, not sure if that is good or not but I will say its a positive that I could do it. Again a big win for me was being able to work out the fixture and suite problem here, although some of the Fitnium commands could be a little easier to remember but that I can work through.

Fitnesse
I do like the Acceptance Tests you can make, and the Suites, they will definitely help with some of the work we need to do for our product, I think my biggest issue is like what most people say - you need to write the tests. Structuring them in a logical way, or what seemed logical tome, did not work as well and gave me trouble until I used SetUp and TearDown for my tests and put the Login and Logout steps in those pages rather than the SuiteSetUp and SuiteTearDown. I don't know why that happens, but it'd be interested to know. Writing the tests can get time consuming, and I still want an easy way to get Selenium scripts to output into Wiki pages, Cory Foy's script did not work for me and I have no idea why (I don't do JavaScript much). The documentation can be confusing and incomplete, I mean where is that !r command defined? I even bought Gojko Adzic's book to help me get going, and I was able to get some nice stuff out of it, but being a doer rather than a reader I didn't get as much out of it as I thought I would, but I'm keeping it handy, who knows, I may get more into this.

Overall, it worked out well, but I would say if you are going to set these up:

• Plan out your tests, and how you want them to work
• Consider a few alternatives, and your Users and see what works
• Read up on the tutorials and see if you can go through the code there, if not getting knee deep in this will only be frustrating in the future
• Be sure these are the tools for you, Fitnesse can be daunting to a techie, and it will be moreso to a business user

Other than that, I'm glad I went through this and now have Selenium and Fitnesse working in my environment.

Automation in its many forms

So in order to get things working smoother I have spent a couple weeks researching some automation tools so that once we really start getting into coding we have something in place to make testing easier, not only for me but anyone else who wants to check out their code.  I've gone through a number of demos on my machine, yes there has been much installing and uninstalling the past couple weeks, in order to assure that I have chosen the right tools.  Not just GUI test tools, since the project is an update to an existing web site, but test case tools and frameworks that will help with automation, also being able to do load and performance testing at some point would be nice.  This is what I have come up with:

Seleniun is the GUI tester of choice, why was because it was easy to generate tests and although the Ajax and Rad Editors on the site don't work as well with Selenium as I'd like I am ok with the manual checking of a couple parts of the site.  Better than all of it, plus being able to export the scripts into C# has allowed me to take the tests we have, import them into a test library and then run those through the PowerShell build scripts with Gallio to give us test reports.

I wanted a Test Case Management Tool so went through some research and the only ones I came up with as possibilities were:

• Test Link was my first choice, since I had heard good things about it from people in my last company, though I didn't like having to install Apache, PHP and MySQL on my machine I figured it's worth a shot.  Basically I wanted something simple, and I got around the MySQL option by utilizing my local MSSQL install, with some work, and then I got to work.  While it was interesting I thought the interface was a little unwieldy, not that intuitive, and even with the help files and forums I had a hard time getting my fields to show up on Test Cases at the times I wanted them.  My simple case was to show Bugs added to a Test Case, so they could be seen on early reviews, and modified when they were worked on, I couldn't get it to work right even after a struggle.  Some of the optioins were right but with some other comments about large numbers of test cases causing issues, I decided it was not worth it, considering half my Users were going to be non-Engineering folks it was not going to work for me.
• Fitnesse was my next choice, as a wiki with some human readable language would be a good way to generate some Acceptance Tests, and Test Cases, for the project.  I figured at least some of the Business Users could get used to a wiki, after all most are writers for the sites content, although I was later told that no it would not happen I knew the wiki idea would be more acceptable in some form.  Besides, with Selenium, if I could get the scripts exported into a wiki format then I could easily script tests then put them on the wiki, so we'd have them as smoke tests for the build and as automated tests on the wiki.  All I needed was something simpler for the Business folks.
• Web Test is an add-on developed by Gojko Adzic that works with Fitnesse, he does a lot with Fit and Fitnesse with training, seminars and books (one of which is being delivered to my home today [Sept 11, 2009] as I write this).  This add-on makes writing the tests in the wiki simpler, and works in Java and .Net forms, but I had a lot of trouble getting this to work properly with the tests when I was trying to write my Suites, and really get some power out of Fitnesse.  Not sure this is being actively developed since the last comments on the site were from 2008, so I had to let this go and see if there was another way.  Note: Gojko has noted in the Fitnesse Yahoo Group that he is no longer maintaining this, so I'd skip it as an option. 
• Fitnium looked to be that other way, right now I am implementing my tests in it, the install is easy and it did not take much for me to get the libraries in place, WebTest was tougher for me to use since the libraries I could get did not work with the 1.2 version of Selenium I wanted to use and I was not sure how to compile new ones, yet Fitnium worked right away.  It's language is even simpler, and I had all I needed right out of the box, so its the one I am going with because of that ease of use, and hopefully understandable code for Users who can then write their own tests in the future.  It's very nice, and I like it alot, once I get my Fitnesse and Fit books I can expand my use of it.

Now that I have Selenium and Fitnium going I am starting to look at some Load and Performance tools, so far I have the following to check out:

• JMeter, which I think I used in the past but it was almost 10 years ago.
• Pylot, a python based tool developed by Corey Goldberg, a fellow forum member
• DieselTest, never heard of this but its Windows based and matches out environment
• OpenSTA, which I used many years ago and had a hard time with, maybe this time will be different.

It's nice starting up a group from scratch, even when you are really the only official QA person on the project and in the group, but its also nice when you can share tools with people who also take a look and they all agree, "that one is cool!".

Defining Agile

I was running through some blog links this morning and came across this gem from Elizabeth Hendrickson, someone whose books I love but hadn't put her blog in my aggregator for some strange reason.

From this blog post Defining Agile Results, Characteristics and Practices
comes this gem:
"I define Agile in terms of results. Specifically, Agile teams:

• Deliver a continuous stream of potentially shippable product increments
• At a sustainable pace
• While adapting to the changing needs and priorities of their organization"

I like the way its spelled out simply, concise and is the goal of many of the Agile groups I have worked with, or have been trying to get that way.  I am definitely going to be using it, and thinking of it, when I work on my upcoming projects.