Setting up Pylot

I was in need of a Load Test Tool, now that I have Fitnesse up and running in my environment, I wanted to be able to check and see how our environment was going to behave under a significant amount of Users. Significant at this point is more than 5, so in looking around I came back to Pylot, not that I needed another language installed on my machine but I liked the simplicity and the ease by which within a day I was up and running with tests that were already confirming to me that the pages I knew were slow, are indeed slow.

Check the Getting Started Guide on how to get it up and running. But there is one thing missing, if you want to run the Script Agent you will need the Python Win 32 Extensions in addition to the normal set up.

When running the Script Agent and running back the XML script I did get a couple of errors about:
ERROR: can not parse testcase file: not well-formed (invalid token): line #, column #
It took a bit to work through this but basically what happened, in my case, was that the recording took in a NULL value somewhere in the CDATA line on the login and logout of my site, removing those allowed my script to run without a problem.

Once you get used to the XML format its easy to parameterize the testcases, and adding in new test steps for text validation and new pages, while the URL's are not the best mechanism in my site to use they do allow me to do a check against bookmarks on the site rather than clicking on the page elements.

I'm liking this tool a lot.

Fitnesse and Fitnium Fun

Some things I have learned when using Fitnium and Fitnesse.

SuiteSetUp and SuiteTearDown are not for actions, those need to go into regular tables, the SetUp and TearDown are not for Action Fixtures.

fit.exception.FitParseException: Can't find tag: table - typically means malformed HTML in a page but MOSTLY means you don't have a SetUp or TearDown in your Suite, and within those pages you need something, ANYTHING, even if its simple text.
Another possibility is if the wiki page containing the test has both the Suite and Test properties showing in the left property bar, both properties can also generate the error as its trying to run a Test but has nothing to do.

The jar file usually needs to be on any page its called on, don't trust the inheritance with the Parent pages or the Root page, you can try it but if you start getting function errors on the page - well now you know what needs to be added.


When adding URL's make sure they are not in camel case, especially URL's that have been parameterized otherwise the URL may not be sent to Selenium the way you think...and oh yes, PARAMETIZE everything.

Fitnesse Parameterization

In order to adjust for the multiple environments that the Fitnesse tests will be running in, within my environment later on, I have spent the time to parameterize the framework now and just added in some Symbolic Links. The Symbolic Links is a wonderful feature for Fitnesse, as it allows a wiki page to be called from another, so I can set up one page with a test, or a set of tests, and have it called from another. The true power in this comes from being able to set up multiple high level pages with specific environment information, so one set of tests can be run on Server A if its called from the Server A Test Page which has a Symbolic Link to Server Tests, I can also call the same tests from the Server B Test Page, which has different test settings, and reuse Server Tests within it. Simple, yet powerful.

It did take a little bit of playing around to get the set up working, so here is what I found out.

• Create the page that will contain the Symbolic Link
• Go to Properties and set the Suite property on the page and save it, as this page is calling another Suite is needed so the Page will go down the Hierarchy
• Find the path that will be in the Symbolic Link
• Go to Properties and set the Name for the Link then the path to the Page, click on Create/Replace
• Save the settings and edit the page with whatever is needed, the Symbolic Link if used once is automatically called

I don't know what will happen with the need for multiple calls, I have seen examples where instead of looping a page was set up as follows:
[Username1]
[Password1]
[Symbolic Link]
[Username2]
[Password2]
[Symbolic Link]
This should allow testing each Username and Password on that Linked Test Page, I have not tried this yet, have not had a need to, but it should be workable as I believe Fitnesse will take the settings it comes to last.

There is probably more to get out of Fitnesse that I haven't discovered yet, I can't wait to see what I find next.

Books On My Shelf

Over the years I have collected quite a few books, not always on testing as I used to be able to get enough online and through discussions with people to keep up. Occasionally I still bought the book that to me, was useful, or was a necessary part of the books I wanted to always take with me from Job to Job. So here is the list of what I have (your mileage may vary):

Testing Books (some of these come into the office with me, especially Lessons Learned In Software Testing, which I have pulled out from time to time to get a point across to Dev Managers), this is not in any special order:

• Quality Software Management - Volume 1: Systems Thinking, Weinberg
• The Complete Guide To Software Testing, Hetzel
• Effective Software Testing, Dustin
• How To Break Software, Whittaker
• Lessons Learned In Software Testing, Kaner & Bach & Petticord
• Managing The Testing Process, Black
• Customer Orientated Software Quality Assurance, Ginac
• Software Testing In The Real World, Kit
• Change-Based Test Management, Sistowicz & Arell
• How We Test Software At Microsoft, Page, Johnston, Rollison
• Writing Solid Code, Maguire
• Debugging The Development Process, Maguire
• How To Break Software Security, Whittaker, Thompson
• Implementing Automated Software Testing, Dustin, Garrett, Gauf

Technology Books (I use Perl ALOT as you can see), since I am mostly self taught these are all the things I have needed for my job in past...most of this is O'Reilly, a geeks best friend.

• Learning Perl On Win32 Systems, Programming Perl, Advanced Perl Programming, Perl Best Practices, Perl CD Bookshelf, Mastering Regular Expressions, Perl Cookbook
• Debugging Perl
• Learning Bash Shell
• Sed & Awk
• JavaScript
• CGI Programming
• SQL In A Nutshell
  
• Unix Shell Programming
• Unix In A Nutshell
• Python Visual Quickstart Guide
• Python Programming For Beginners
• Python Essential Reference
• Unit Test Frameworks
• Teach Yourself Java
• Java Development With Ant
• Open Source Programming with CVS
• Unit Test Framework, O'Reilly
• Pragmatic Unit Testing, Hunt, Thomas
  
• Practical Performance Analyst, Gunther
• Aptana RadRails, Ramirez
• Test Driven .NET Development with Fitnesse, Adzic
• Fit For Developing Software, Mugridge, Cunningham
• Head First C#
• Windows PowerShell In Action

I've been asked if I really had to choose my top 3 I'd say:

1. Hetzels book The Complete Guide (as it was my first)
2. Maguire's Debugging The Development Process
3. Lessons Learned (which I actually bring into my office a lot)

Others I received about the same time I originally made this post are:

Web testing handbook - Splaine and Jaskiel
Software testing - Ron Patton
Systematic software testing - Craig and Jaskiel
Software testing techniques - Boris Beizer
Black box testing - Boris Beizer
Test process improvement - Koomen & Pohl
The testing practitioner - Erik van Veenandaal
The Art of Software Testing – Glenford Myers (yes, I know it is old, but still good) 

PowerShell fun with Fitnesse

Fitnesse is the newest automation tool I have introduced here, we also use PowerShell to drive a lot of the builds, because there is some integration between the two (think Smoke Test) I worked on getting the TestRunner working with PowerShell. I couldn't get the .Net version to run properly, it kept dying when it would render the page, but the Java runner worked great once I was able to work through the java command line with the BuildMaster. If not for his help I wouldn't have been able to finish.

This script launches the Java TestRunner, points it to the Fitnesse server, and runs the three tests (one Test and two Suites) I currently have set up, then scans through the XML results to display some results in the command line window.

# Variables for the script
[string]$fitniumLocale = "C: itnium"
[string]$fitnesseJar = "fitnesse.jar"
[string]$fitnesseClass = "fitnesse.runner.TestRunner"
[string]$serverName = "my_server"
[string]$portName = "8888"
# Use a Test Array to be able to bundle all the tests
$testArray = ( "Test1" , "Test2", "Test3" )

function main()
{
Write-Host "Running Fitnesse Tester.`n"
cd $fitniumLocale
# Run through each test in the bundle
$Java = "java.exe"
foreach ($test in $testArray)
{
# Make a date string to get the right file date/format
$dateStr = [string] (get-date)
$Month,$Day,$Year,$Hour,$Minute,$Second = $dateStr.split(" :/")
$testResults = "$test" + "$Month" + "$Day" + "$Year" + "$Hour" + "$Minute" + ".xml"
[string]$opts = "-xml $testResults"
# Start the test
Write-Host "`nRunning $test`n"
# Get the output from Java so we can scrape it for the test passes and fails
& $Java "-cp" "$fitnesseJar" "$fitnesseClass" "-xml" "$testResults" "$serverName" "$portName" "$test"
Write-Host "Completed $test`n"
Write-Host "Test Results"
# Scan through the output file, we want that for archiving
[System.Xml.XmlDocument] $xd = new-object System.Xml.XmlDocument
$file = resolve-path($testResults)
$xd.load($file)
$nodelist = $xd.selectnodes("/testResults/finalCounts") # XPath is case sensitive
foreach ($testCaseNode in $nodelist)
{
$right = $testCaseNode.selectSingleNode("right").get_InnerXml()
$wrong = $testCaseNode.selectSingleNode("wrong").get_InnerXml()
$ignores = $testCaseNode.selectSingleNode("ignores").get_innerXml()
$exception = $testCaseNode.selectSingleNode("exceptions").get_InnerXml()
}
# Send a summary of the results to the console
Write-Host "Passed: $right" -ForegroundColor Green
Write-Host "Failed: $wrong" -ForegroundColor Red
Write-Host "Ignored: $ignores"
Write-Host "Exceptions: $exception" -ForegroundColor Yellow
}
}
main


It needs some cleaning up, but its a good first step that runs well, I especially like the color coding for Pass, Fail and Exceptions so I can visually be drawn to the values I want to see.

I'm in QA and I'm a Doubter

Yes, I doubt. I doubt a lot, especially on projects that tell me that things are working good and we're on target, I sit back and think - am I missing something? I think its partly endemic to people who work in Test, because let's face it we are in the position of being the person next in line from when something is done and there needs to be a step to make sure that the something is done right. If you've been in this position this may be familiar to you:

Developer: Hey, the code is ready I'm going to hand it over to you later, I'm done with it and it looks good!
Tester: Thanks for letting me know, I'll get back to you once I check it out.

Then you do, and your getting back to the Developer involves defect reports. Was it good? Probably, it met what the Developer was trying to do with it, there were Unit Tests which ran and passed, or what was the minimun to pass did, so it was handed to Test. Now, Test has its own tests to run, its plan and its criteria to be able to say that things are stable enough to put it into Production and let Customers use it. Testers usually have Tests that Developers never run, or want to get into, so they are looking at things from a higher level, sort of like the Ranger in the Watchtower, we have our scope up and we are looking all around. Meanwhile, the Developer was just doing something over in one place and getting that to look nice, while nearby another Developer was doing the same thing but somehow they will collide. Testers are there to raise the warning, hey there is some cross-purposes here or to see that there may be, if its been communicated that Developers are working together on something and maybe no one involved realizes there is some dependency.

So where does the doubting come in? Remember those Customers? Well, if a Customer sees something wrong and calls up with a problem, or is angry that there is a problem, many companies will look to find out why that problem got out there. Someone has to take responsibility for it, and because we have that person in Test between the Developer and Production its that person's fault that anything got past, not that the Developer put it there - by mistake or accident - but that the Customer saw it. To avoid that I became a doubter.

Now, being a doubter can be a strength, it makes you question everything and you look for issues that may need resolution, but to be KNOWN as a doubter is pretty bad. Then you get a reputation, and sometimes people won't want to work with you because they know you'll come back with issues, complaints or will be negative right away. What is needed is something I like to call Constructive Doubt.

Constructive Doubt is what I do when I get code and need to look at it, I won't immediately snipe back and say "heh, more bugs for me to find?" when a Developer comes and says something is ready. Diplomacy is part of Constructive Doubt, because you want to build with it, anyone can tear down something, but it takes real skill to build something. Use that doubt to figure out some new way to test, use a new tool, a framework or something that will allow you to doubt in an automated fashion. Sounds like I am talking about Testing, but in reality part of a Test is a doubt, you doubt something is going to follow the requirement, sure you could call it a Verification or a Validation but in the middle of it all is a seed of doubt; you want to KNOW that something is doing the right thing in the right way. Sometimes when I have tested something enough I know it inside and out I sit back and think, how else could this fail? How else could I figure out that something failed? Where else could I doubt the process is working? Then you look, you learn and sometimes you find something new.

I'm in QA, I doubt and I build up on that.

Updating Subversion

Yesterday I went through the process of upgrading our soon to be used Subversion server, we are moving from VSS to SVN and the initial install of Subversion was done by our BuildMaster. I got asked to do the upgrade and bring things up to current versions and document the steps, so for posterity here they are.

Subversion installation used 1.6:

http://subversion.tigris.org/svn_1.6_releasenotes.html

From Compatibility Concerns

• Older clients and servers interoperate transparently with 1.6 servers and clients. However, some of the new 1.6 features may not be available unless both client and server are the latest version. There are also cases where a new feature will work but will run less efficiently if the client is new and the server is old.
• There is no need to dump and reload your repositories. Subversion 1.6 can read repositories created by earlier versions. To upgrade an existing installation, just install the newest libraries and binaries on top of the older ones.
• Subversion 1.6 maintains API/ABI compatibility with earlier releases, by only adding new functions, never removing old ones. A program written to the 1.0, 1.1, 1.2, 1.3, 1.4 or 1.5 API can both compile and run using 1.6 libraries. However, a program written for 1.6 cannot necessarily compile or run against older libraries.

Subversion upgrade process

o Turn off the Apache Service

o Prepare Subversion for upgrade (back-ups)

1. Zip up directories to keep a rollback in place

1. Back up the configuration files under Apache (can be done within the Zip file)

1. svnaccessfile

2. conf/httpd.conf

3. other stuff!

2. Move ZIP files to a new location as storage

o Upgrade Subversion

1. Unzip the upgrade

2. Move the binaries into place

1. Copy /bin directory to install location

2. Copy D:Program FilesSubversion inmod_dav_svn.so and mod_authz_svn.so to the Apache modules directory

3. Copy the /iconv directory to install location (this helps to update the version numbers for Subversion within the Apache footer - at least I think it did, my cache was acting up in FireFox)

o Upgrade Tortoise

1. Install Updated software package

o Reboot

o Create a new Repository:

1. Svnadmin create d:SourceControlSVNMAIN

o Test/Verify Upgrade

1. Connect using the Tortoise Repo-Browser from a remote machine

1. Connection should allow viewing of repository

Backup Process

Using FSFS as the database backend.

Criteria for the backup:

The process should run [nightly?? (this can be adjusted depending on how long it takes)] using Windows Scheduled Tasks.

Use the following command from a Windows Scheduled Task:

svnadmin hotcopy (repository path) (backup path)

The backup process will copy server configuration and hook scripts.

Keep the backups on an area that is backed up (get location from IT)

Is it possible to "share files" in SVN?

If you use the svn:externals –r option you can, but it seems SVN can only do directories not files

http://blog.kowalczyk.info/article/Short-tutorial-on-svn-propset-for-svnexternals-p.html

Security/branches:

· Authentication via AD

· Groups and user access defined in svnaccessfile

· We want to have two groups with users defined to each

· By default all Company group users will have all access; Consultant group users have access to only a branch (and possibly build stuff)

· Out of the gate just the trunk, all users edit there. More branches down the road.

Sites with additional information:

http://www.sublimesvn.com/blog/2009/02/configuring-subversion-with-active-directory-authentication/

http://concise-software.blogspot.com/2009/02/instant-windows-svn-server-with-ssl-and.html