Running Fitnesse with PowerShell - RESTful Tests

Previously I noted how I had written my original tests with Fitnesse to use the TestRunner and access Fitnesse in a way that worked, but as I recently discovered is no longer supported. Oh well, so I update my tests. Which was necessary anyway, since on the Build Machines where these tests would be run Fitnesse is not going to be running, and if it was I shouldn't need to usurp it for my tests. So I moved over to using the RESTful method, which I had a hard time with earlier, but now is so insanely simple I wonder why I had a hard time with it before.

Without futher ado, here is the RESTful method to run Fitnesse tests.

First I set up my variables, I've also added in an archive file now, so I can keep track of my results, the RESTful results are nice so I actually output each test into its own results file, and I make an aggregate to be an archive. Eventually I will get rid of the individual results and keep the archive around - this also let's me go onto any machine that ran a test and see how things have been going. The Arrays I have are suites that I set up in Fitnesse to contain my tests, I have one for QA that I would run, and a Smoke suite that is a selected set to verify the build worked.

# Get the current directory Push-Location .
# Variables for the script
# This location should not change, since its the standard checkout location [string]$fitnesseLocale = "C:\MAIN\trunk\AutomatedTests\Fitnesse\Fitnesse"
[string]$fitnesseJar = "fitnesse.jar"
# Need to get the local machine here, it will be how Fitnesse runs [string]$serverName = $env:COMPUTERNAME
[string]$portName = "8080"
# These are needed for the CheckFitnessProcess function,
# placing them here $fitRunning = $false
# Set Java executable to use within PowerShell $Java = "java.exe"
# Archive file $archiveFile = "fitnesseArchive.txt"
# Use a Test Array to be able to bundle all the tests
# Tests that are commented out are having issues being run
# Smoke Testing is Development Based
# although its now geared towards checking Login $smokeArray = ( "SpringsLoginLiteTests" )
# Tests for the QA environment
# although its now geared towards checking page links $qaArray = ( "SpringsLinkChecks" )
# Tests to check how this script runs, this is a hidden command
# This just validates the Registration Verify Fields $checkArray = ( "SpringsTest.UserLoginFfTests" )
# Location of the Test Results, this is hardcoded within the SVN structure
# so it should always be available, I am sure there are better ways of doing this $testResultsDir = "C:\IHI_MAIN\trunk\AutomatedTests\Fitnesse\Fitnesse\testResults"
if (!(Test-Path -path $testResultsDir))
{
New-Item $testResultsDir -type directory > $null
}

Then I still run my check on whether Fitnesse is running, it doesn't really need to be running anyway; so I decided that since REST only runs as needed then I was going to shut down Fitnesse if it was going.

# CheckFitnesseProcess determines if Fitnesse is running, if Fitnesse is not running
# then the script continues. The current test configuration does not require Fitnesse
# to run and does a better job when it's not.
# Adapted from: http://odetocode.com/Blogs/scott/archive/2006/07/17/5330.aspx function CheckFitnesseProcess {
param ([string]$AppPoolName="")
$ProcessId = $null
if ($AppPoolName.ToUpper() -ne $NOT_PASSED) {
[string]$filter = $null
if ((Get-Process java -ErrorAction SilentlyContinue) -ne $null) {
Get-Process java | ForEach {
# use WMI API to get process details
$filter = "Handle='" + $_.Id + "'"
$wmip = Get-WmiObject Win32_Process -filter $filter
# CommandLine property has AppPool name; grab name using contains
if($wmip.CommandLine.contains("fitnesse")) {
# Uncomment the next line for debugging
# Write-Host "Found this java process: $wmip.CommandLine `n"
$fitRunning = $true
# Saving the Process ID to stop the process later,
# using a Global as I could not get this work if I declared
# the variable at the beginning.
$Global:fitProcess = $_.Id
}
}
}
if ($fitRunning -eq $false) {
# All is good
}
if ($fitRunning -eq $true) {
Write-Host "Fitnesse is running, it's best if its not - shutting it down...`n" -ForegroundColor Red
# Stop the Fitnesse process
Stop-Process -Id $fitProcess
}
}
}

I'm going to skip the Help function, if you need that it should be in my previous post, the important one is the Main function.

function main()
{

# Need to be in the Fitnesse directory to access TestRunner, and

# the Fitnesse batch file in case Fitnesse needs to be started

cd $fitnesseLocale

#Check and see if Fitnesse is running

CheckFitnesseProcess

foreach ($test in $testArray)

{

# Start the test

$myDate = Get-Date

Write-Host "`nRunning $test`n"

# Build RESTful URL

$testLink = $test +"?" + $testType +"&format=text"

# Get the output from TestRunner so we can scrape it for test passes and fails

$testResults = & $Java "-jar" "$fitnesseJar" "-c" "$testLink"

Write-Host "Completed $test`n"

# Write-Host "Test Results..."

# Scan through the output, then display the results

# Dump $testResults out to a File

$testFile = $test + $myDate.Day + $myDate.Month + $myDate.Year + $myDate.Hour + $myDate.Minute + ".txt"

$testResults | Out-File "$testResultsDir\$testFile"

# Print out date to an archive file

$testDate = "Last run time: " + $myDate.Day + "/" + $myDate.Month + "/" + $myDate.Year + " at " + $myDate.Hour + ":" + $myDate.Minute

$testDate >> "$testResultsDir\$archiveFile"

foreach ($line in $testResults)

{

if ($line -match "\A\.\s"){

# This is a Pass

Write-Host $line -ForegroundColor Green

# Output to archive file

$line >> "$testResultsDir\$archiveFile"

}

if ($line -match "\AX\s") {

# This is an Exception, a warning but not a Failure

Write-Host $line -ForegroundColor Yellow

# Output to archive file

$line >>  

Fitnesse TestRunner Scripting in PowerShell

I'm running Fitnesse with SharePoint and in trying to automate my tests I ended up wrapping everything up in PowerShell, this is my first full script that handles all the tests I wanted to do.

First set up all the variables needed.
[string]$fitnesseJar = "fitnesse.jar"
[string]$fitnesseClass = "fitnesse.runner.TestRunner"
# Need to get the local machine here, it will be where Fitnesse runs
[string]$serverName = $env:COMPUTERNAME
[string]$portName = "8080"
# These are needed for the CheckFitnessProcess function, placing them here
$fitRunning = $false
# Since we need to manipulate this variable in a few different functions, its Global
[int]$Global:startTries = 0
# Set Java executbale to use within PowerShell
$Java = "java.exe"
# Use a Test Array to be able to bundle all the tests
$smokeArray = ( "FitNesseTestPageName1" )
$qaArray = ( "FitNesseTestPageName2" )

I use a function to check if Fitnesse is running, I wanted to only run it when I needed to and since this script can be automated I cannot be sure that Fitnesse will always be running
# CheckFitnesseProcess determines if Fitnesse is running; in which case
# it jumps down to running the tests, as long as Fitness is running the Check does
# not need to do anything else.
# If Fitnesse is not running then the script attempts to start it, does a
# check to make sure that Fitnesse started then runs the test. If this script
# starts Fitnesse then it will attempt to clean up after itself and stop
# Fitnesse by doing a check on $startTries, if this is > 0 then this script
# started Fitnesse and can shut it down, otherwise it leaves the existing
# process running
# Adapted from: http://odetocode.com/Blogs/scott/archive/2006/07/17/5330.aspx
function CheckFitnesseProcess {
param ([string]$AppPoolName="")
$ProcessId = $null
if ($AppPoolName.ToUpper() -ne $NOT_PASSED) {
[string]$filter = $null
if ((Get-Process java -ErrorAction SilentlyContinue) -ne $null) {
Get-Process java | ForEach {
# use WMI API to get process details
$filter = "Handle='" + $_.Id + "'"
$wmip = Get-WmiObject Win32_Process -filter $filter
# CommandLine property has AppPool name; grab name using contains
if($wmip.CommandLine.contains("fitnesse")) {
# Uncomment the next line for debugging
# Write-Host "Found this java process: $wmip.CommandLine `n"
$fitRunning = $true
# Saving the Process ID to stop the process later,
# using a Global as I could not get this work if I declared
# the variable at the beginning.
$Global:fitProcess = $_.Id
}
# The next statement was put in to check for the processes, there was a problem
# when updating Java that the process find command did not work, if this happens
# again just uncomment the Write-Host line to find out what is going on, otherwise
# this just gets bypassed but caught later on.
if ($fitRunning -eq $false)
{
# Uncomment the next line for debugging
# Write-Host "Did not find a Java/Fitnesse Process. Found $wmip.CommandLine `n"
}
}
}
if (($fitRunning -eq $false) -and ($Global:startTries -lt 3)) {
# Don't need to note that we have had 0 attempts
if ($Global:startTries -gt 0) {
# Need to give Fitnesse some attempts at starting, 3 is a good max
Write-Host "Fitnesse is not running, have tried to start $Global:startTries time(s),`n"
}
$Global:startTries++
startFitnesse
}
if (($fitRunning -eq $false) -and ($Global:startTries -eq 3)) {
# At some point we need to give up starting it
Write-Host "Fitnesse is not running, tried to start $Global:startTries time(s).`n"
Write-Host "Strike Out!" -ForegroundColor Red
Write-Host "Looks like Fitnesse cannot be started or found,`n"
Write-Host "check the server and try again.`n"
# Go back to where we started
Pop-Location
exit;
}
if ($fitRunning -eq $true) {
# Success!!
Write-Host "Fitnesse is good to go.`n" -ForegroundColor Green
}
}
}

Then I needed something to start Fitnesse, this uses the Fitnesse Batch script that comes with many of the installs
function startFitnesse()
{
Write-Host "Attempting to start up Fitnesse...`n"
# Starts the Fitnesse process using the batch configuration file
# Not sure we need the runAs Admin but that can be adjusted
Start-Process -FilePath fitnesse.bat
# Giving it time to wake up
Start-Sleep -Seconds 2
# Need to check that the process actually started, do the check again
CheckFitnesseProcess
}

Every script needs a help option
function myHelp()
{
Write-Host "The Fitnesse Tester understands the following command line arguments:`n"
Write-Host " qa - will run the QA Test Suites, a long set of tests`n"
Write-Host " smoke - which will run the Build Smoke Tests, a short set`n"
Write-Host "`n Running this script with no arguments will display this message`n"
}

Then there is the main and rest of the script that runs everything
function main()
{
# Need to be in the Fitnesse directory to access TestRunner, and
# the Fitnesse batch file in case Fitnesse needs to be started
cd $fitnesseLocale
#Check and see if Fitnesse is running
CheckFitnesseProcess
foreach ($test in $testArray)
{
# Start the test
Write-Host "`nRunning $test`n"
# Get the output from TestRunner so we can scrape it for test passes and fails
$testResults = & $Java "-cp" "$fitnesseJar" "$fitnesseClass" "-v" "$serverName" "$portName" "$test"
Write-Host "Completed $test`n"
Write-Host "Test Results..."
# Scan through the output, then display the results
# Dump $testResults out to a File
$testResults | Out-File "$test.txt"
foreach ($line in $testResults)
{
if ($line -match "Assertions:") {
# Write-Host "$line"
# Do a regular expression to get the Assertion info
# Example: Assertions: 8 right, 0 wrong, 0 ignored, 0 exceptions
$line -match ":\s(?\d+)\sr.*(?\d+)\sw.*(?\d+)\si.*(?\d+)\se.*"
# Send a summary of the results to the console
Write-Host "Passed: "$matches.right -ForegroundColor Green
Write-Host "Failed: "$matches.wrong -ForegroundColor Red
Write-Host "Ignored: "$matches.ignores
Write-Host "Exceptions: "$matches.exception -ForegroundColor Yellow
}
}
}
# Go back to where we started
Pop-Location
# Stop the Fitnesse process so we don't leave things running
# But only if this script started it, which means $startTries is > 0
if ($Global:startTries -ge 1){
Write-Host "`nSince this script started Fitnesse, it'll be shut down now.`n"
Stop-Process -Id $fitProcess
}
}
# Check command line values to see what test is running
if ($args -eq "smoke")
{
$testArray = $smokeArray
Write-Host "Running Smoke Test.`n"
}
elseif ($args -eq "qa")
{
$testArray = $qaArray
Write-Host "Running QA Test Suite.`n"
}
elseif ($args -eq "check")
{
$testArray = $checkArray
Write-Host "Running Check Suite.`n"
}
else
{
myHelp
exit;
}
# Run main
main

I've moved onto REST but in case anyone else is trying to work with the TestRunner and PowerShell you have something you can start with, and not have to work it all from the beginning like I did.

Testing with SharePoint

I've been learning a lot about SharePoint over the past 9 months, the project I am working on is to bring a non-profit health community onto SharePoint and increase the offerings the non-profit has and to expand the social networking. My work is revolving around testing the conversion of the site, making sure that what we end up with is going to be equivalent with what is currently in place. It's interesting work and I've had some interesting tasks over the past few months. Some of the more difficult work has been dealing with SharePoint and its eccentricities in order to assure that what we end up with is what was asked for.

Web Parts are like little applications that SharePoint uses to do all kinds of work, this could be anything from displaying a document library to a search result. While they are like little applications Web Parts are quite different and aren't very testable on their own, I haven't seen much on them regarding Unit Testing yet and with their reliance on SharePoint data its probably not that easy. Mostly the plan is to test the Web Parts in an exploratory fashion, when they are ready and added to pages they will be used and abused to put them through their paces. Not the way I'd like to do it, but I am also pulling some information from the Microsoft page on how to test Web Parts.

Automation with SharePoint is kind of tough, not that its difficult to work with, but the page layouts and Web Parts don't always make things run smooth. Web Parts can update like Ajax driven pages will, some of the ID's given by SharePoint for page elements can also be cryptic or dynamic. It's not impossible, I have done tests with Selenium, Fitnium and Fitnesse after some time to work out the issues in calling page elements. It just seems to take more time than it should.

The Social Networking aspects make things a little hard to automate everything, it would be nice to free myself up for other work but that's more a dream than a reality. Testing Discussion Groups, Comments and Ratings means eyes need to be on the page as things are done in order to verify that SharePoint responds properly.

When trying to find errors there are two important places to look:

1. ULS Logs - Get the Microsoft ULS Log Viewer, if there is something happening real time, this is the place to see it happen. Access to the SharePoint server under test is required to get this to work though.
2. Event Log - the Event Log is where alot of the captured errors go, if the Developers have not set this up in the code they should otherwise you lose and important resource.

I'll be putting in more links to some of the material I have captured over the past few months.

Cost of Bug Fixing

I got an IM from a developer I used to work with and he had that good old familiar question "do you have a copy of that graph that shows the cost of fixing a bug?"

Well I didn't at the time, mostly I dealt with this with the book I keep on Quality Assurance, it has the graph and a little blub about it. Since he needed one online, and pretty much everything is on the internet I did a check and lo and behold, I found a developer centric version.
Rate of Diminishing Returns Of Fixing Software Bugs
He needed an online version because he had a VP who wouldn't drink the water on catching defects early, I guess there are still plenty of them out there. Either way, I found the link and he went away happy, another QA-centric developer satisfied.

It feels good to help on a Monday.

Regular Expressions with PowerShell

I came into Regular Expressions using Perl, and I must say I liked it and enjoyed it immensely once I was able to actually figure out how to do the syntax, never mind doing captures in order to use specific values from strings later. Although, like most things, once you do it then its simple and you think, "that was not so bad" when you look at something that looks like you kid typed when you left the computer on.

Perl was one thing, PowerShell is another. Now that I have this figured out its simpler, but finding the information on it was not so simple, although I will say one thing that made it easier was having a Designer tool that went over some of it. The Rad Software Designer is a very good tool for being able to help figure out the syntax as well as give you a place to assure that the results will be what you want, and since PowerShell is based off the .Net Regular Expression implementation the result will work fine there.

My goal was to be able to match a line like this:
Assertions: 8 right, 0 wrong, 0 ignored, 0 exceptions
So I could get the numerical values for display, turns out I only needed this:
:\s(?\d+)\sr.*(?\d+)\sw.*(?\d+)\si.*(?\d+)\se.*
...and then I could use some named capture values to then display the information I need. Now if I could only figure out how to turn off the True statement being displayed, that would be great.

More SharePoint News!

I am doing some of the Microsoft Hand On Labs (HOL's) for SharePoint 2010, normally I would post a link to these but they are so amazingly buggy with typos and code that doesn't compile (or write without errors appearing) that I would rather let anyone really wanting them to seek them out. I downloaded them from Microsoft somewhere, they are from early 2009 and it looks like someone wrote them up quick and then somehow got them on the Microsoft site; I can't get past two pages without a typo and I hate that. The code has errors, I know enough to fix some of the warnings that show up in Visual Studio 2010 when I write out the C# examples, so I would not push this on anyone.

Still, because of this I have had to go out and research errors now and again, what I have found are a couple of interesting posts on sites:

• Sharepoint 2010 beta release known issues
• (I encountered this in the lab I was running [HOL 8]) - Interesting little bug in VS 2010 beta-2

So far I am having middling luck in scripting many of the Web Parts I have been able to compile with Fitnesse, I have also found that I right now need to deploy and add Web Parts on the same environment. I cannot deploy through Visual Studio 2010 on one machine then add that Web Part on another machine, I kept getting errors regarding that the object did not exist even though it appeared in the Web Parts list. So, I am running things locally until I know they are on the site and working, then I go to another machine and script with Fitnesse.

Fitnesse also doesn't seem to like the naming conventions that SharePoint gives for certain objects, I am trying to find a solution for that as select lists may have their own ID's and Name's with the SharePoint strings in the future - so I am working that out now.

Enjoy!


Note: I've edited this page to now note the Labs, they are for 2010 and I figure why not share the fun.

SharePoint and Fitnesse - a match made somewhere...

Now that my current web redesign project is going to be based on SharePoint 2010, yes its in Beta and we know this but we are still going forward with this anyway...not my decision but it should be interesting, I have migrated some of the old Fitnesse scripts I had to the new platform. In addition, I have upgraded from using the Fitnium add-on to FitLibrary and the SpiderFixture since it gives better Internet Explorer integration than Fitnium did, while FireFox will still be tested it won't be the focus that it was in the past.

So far I have encountered some issues with the way Fitnesse sees the pages, since SharePoint's web parts can act like Ajax some adjustments need to be made, also SharePoint masks some of the standard ID's of the base pages with its own ID scheme. Some of those ID's are not only odd, but sometimes they are very long, in addition SharePoint's ability to be made of many dynamic pieces makes validating some pages very tedious.

In the past I did make Fitnesse a service on my machine, that way I always had it up and running, however, in recent releases I have had trouble with the connections from the browser to the site under test. Today I noticed something odd, when I was running a new install on the command line everything worked, running the same script on the Fitnesse server that ran as a service there would be failures. On the command line the browser would open a new instance and run its tests, on the service version nothing would show up. So I did a check and sure enough I did a nice default, when I set up the service I set it to run as Local System, but when I run command line I run as myself. Changing the Log On to myself, and granting myself access rights, I can now get the same scripts to pass on the new install.

I'm now running FitLibrary 20091021 release with Fitnesse 20100103, so far basic stuff looks fine and good, I just hope I can keep this running.