In my current position, part of my work involves utilizing a Test Harness to perform some distributed testing. This is a Harness written in Perl, which is one of my favorite languages, it covers multiple operating systems (windows, Linux, Solaris, HPUX and AIX) and while parts of it are the usual Perl hack most of it is in Object-Perl. The developer of this and I are working to get it even more distributed, objectified (if thats a word) and also just spreading out how to use this tool so we can also allow others to use it for Development testing.
We ran it on one of the latest builds, and sure enough there were lots of failures, so we talked and the initial response was that this happens and we should run it again. So we did, and of course there were less failures. Over the weekend some network upgrades happened, never a good time in a test cycle, so I started running them again after the lab was back up. I got different errors, and in come cases fewer. So I started thinking, what was it we were trying to prove here? Did we simply want to be running the harness over and over again until the errors were gone? Did we want to investigate the initial errors and see where the problems were from? Did we have enough confidence in the Harness to know the results we received were accurate?
After some discussion, it came down to what I considered the usual response. Until we know its not the Test Harness creating the issues, there is no reason to report the issues as bugs, unless its for the Harness. We could run some of the tests manually, but the reason for having the Harness is to take the manual time and cut it down...so we are in that state where we debug the Test Harness until we are sure it works, in parallel we will use it on the product and try to get some confidence with repeatability on the Test Results. I don't think that will happen for awhile, but at least we will spend time getting to see how the tool reacts, and be able to judge better how it will work out in the future.
No comments:
Post a Comment