Let the crowd in...

I was reading James Whittaker's posts on  Crowd Sourcing at http://blogs.msdn.com/james_whittaker/archive/2008/08/20/the-future-of-software-testing-part-1.aspx, where he sees the next big leap to be one where everyone who wants to can be a part of the Testing Solution.  While I'm not quite sure I agree with it, and some of the quotes reflect my own opinion, I can see where it can be of benefit as an additional avenue of allowing people time and a chance to test software and submit their issues to be fixed.  This is a good thing, most people, as he says, do find problems not always found in a test cycle either because of contraints, or some quirk of the User Environment that is not mimiced in the lab; getting those issues in early is a good thing.  However, I am a realist, and while its nice to get many of these bugs in I personally don't see where the value is going to be if those bugs are not being resolved but end up in the bug bucket waiting to be looked at.  Though I am getting ahead of myself, there were a couple of points in this I thought was interesting.

The Cloud.  I'm still unclear on what this is, or whether its just an anticipation to what Cloud Computing is going to be, but I'm not expecting people to be sharing configurations and environments across the Net.  In a QA lab you can have all kinds of virtual environments, but they are not personalized, they tend to be representable of what Customers have, either by being proactive and knowing what Customers use, or by being reactive and adding in software or configurations that became known trouble spots.  I'll share a specific browser configuration and plug-in configuration with someone, but I'll be damned if I am going to sit there and share someone's Hello Kitty theme with kitty cat icons made up of the heads of someone's little babies.  I'm sure that works fine for some people, including some of my co-workers (not the Hello Kitty theme, but the cat picture backgrounds) but its not something I'd expect to see in my lab nor would I expect it to have issues with commercial software.

Bug Reporting.  As I said, just because Jim from South Carolina found a particular GUI issue with the tool bar, and it was confirmed with Vijay from Pune as well as Klaus from Dresden, doesn't mean that bug is going to be fixed.  I'm all for finding as many bugs as possible before I release, but honestly, do I need 100+ minor defects or 250 Enhancement Requests sitting in my bug queue because the Crowd found them, and the serious issues they had were already reported and entered?  I don't see what this gains me, other than more defects for triage that may or may not ever get fixed, but from a business standpoint I can say that we found them with the Crowd!  Serious defects should hopefully be caught prior to the release to the Crowd, or maybe specific configurations can be found, and I am all for getting as many of those as possible, but when I look at the numbers of Critical and high priority defects being added in, there are very few found later on; while I'd like those fixed is there any guarantee they will be?  I'm still waiting to see the results on that.

Isn't this really Beta?  I can see this point, and I partially agree with it.  If you are working on an open source project you expect updates that may or may not be very well tested, with a Beta you are taking this knowing that the software may crash (and badly) in your environment because that is what it is.  So what is this?  Beta?  Well, I'd say not really since it seems like the releases are still very early in the cycle, unless the company has a long timeframe between a release build and eventually getting it out.  So where does the testing stop?  Does the release mean everything found is put on hold and we now wait for the next release, especially since Customers now have a copy we can be getting reports from them.  I'm waiting to see more on this, but I am trying to keep an open mind.

Talent Pool.  Who are the people who are joining?  Are these people new to the field and they want more experience?  Are they bored?  What is their experience?  As someone who has trained people off and on I can tell you that a bug report from someone who is just learning compared to someone who has been doing this for a long time is very different.  I'm not sure who is signing up here, the money doesn't seem like much, and maybe I am not their talent pool, but when I get out of work the last thing I want to be doing is testing some other kind of software, heck I don't like to practice programming much outside of work.  I'm not the target audience for the community, but I'm curious as to who is.

Unlike Mr. Whittaker I don't know if this is the next logical step, but I like to take a long view and a higher view of testing as a whole, some things get tried, some change and some just end when its seen they don't lead anywhere.  I don't know where uTest and the Crowdsourcing is going to go, but I am curious to watch and see.

IO Stress, Part 2!

After the Development team came back from the latest microsoft PlugFest there was a new version of the IOStress tool, which has a better structure than the last one without all the useless command scripts in the root directory.  It seems to run better, and has picked up on configuration I had set for the previous one, as I was able to just put the directory down and run it with fewer errors than I had seen in the last version.  Running with the Verifier and our Driver still causes a crash, but that is expected for the way the Assertions were being used in 2008, supposedly in windows 2003 this was more forgiving, but 2008 doesn't like it.  I've crashed the 2008 box 3 times since installing the new version, go me!

I need to get a listing of the tests, and what they do, but for now I have a nice basic structure that runs on our driver and doesn't crash the box, always a good thing.  The one thing I don't like is the reboots the IOStress program does when setting up certain tests and when completing them - since I run this in a remote desktop window and do other things - the test goes away when I don't notice that the remote desktop is gone.  Part of the problem with the reboot is that the test results window disappears quickly after the reboot, I may have to look at getting the mail piece working so I can get the results in email, they may be formatted better.

Undocumented and unofficial tools are so much fun to work with....right.

Setting up IO Server Stress Tests for Windows

In my current position I test a filter driver, basically a driver placed into the stack on multiple platforms that gathers events on the system and passes them up to a Java agent to send to a central server for review.  Some events are filtered out, and some are not, but default and this is done to keep the messaging down to a level where the driver is not stressed out, but sometimes its good to know how the driver operates under stress.  There are a few scripts we use to load the system with events of all kinds, but windows provides a suite of tests in something it calls IOStress, and is given out at Plugfest every year.  I have spent some time setting it up for testing in my environment and this is what I have had to do so far to get it to work:

• The IO Stress program needs a net share set up for ntiosrv that points to the iostress folder and supporting files
• Copy the iostress folder from \Software\Microsoft\iostress to a local drive (say D:\) and rename the io.stress60 (plug-iosrv) folder to iostress
• Create the share for ntiosrv

net share ntiosrv=d:\iostress /GRANT:Administrator,FULL /REMARK:"IOStress Share"

• This will set up a share to the D:\iostress directory, change to that location and run - iostress.cmd /ignoredebugger
• The I/O Stress program main window may be minimized, select it
• In the Registration and Verifier screen enter the driver to be tested
  • Select Low Resources Simulation
  • Select I/O Verification Level 1
• In the Registration and Verifier screen, if run with No Debug, there will be a note regarding No Debugger Selected, this is due to the Email and Contact information being empty. Enter values, any will do, into the three fields.
• Select the drives needed to be tested, all available drives on the machine will appear in the Run Information tab, on this page as well the Test Time can be entered in the Specify Number of hours to run field.
• The Stress Tests tab allows selection of specific tests, a slight overview of the tests is included with the zip file that contains the tests, it gives some information on the basic tests.
• If the system reboots you must log in to the machine, the IO Stress kit likes the Administrator account to have a blank password, but that is insecure

Still, even with all this I have encountered a few problems when running the tests:

• There is a complaint about a registry value not discovered, I have yet to find documentation on what that value is, but it seems to still run
• Do not run with the Verifier, this seems to cause crashes, possibly because the IOStress program runs with the driver in the Verifier and having them in twice causes conflicts
• When setting up the environment it claims some network drives are not found, I am not sure if its the ntiosrv share or not, I can never seem to find it in the net view list, but without the share being available the entire test suite will not even start.

I don't mind Windows tests, and undocumented Microsoft tools are such a joy, but it does some checking and has allowed me to verify some issues and find new ones especially on Windows 2008 which is a new platform for my group and we don't have a complete set of tools for that platform yet.  If I get these issues resolved, I'll note the fixes.

Command Windows Where You Want Them

For most Windows platforms I have used a registry hack to add a Command Window option to the right click command window within Windows Explorer  This is on the Microsoft Site, as well as available though the PowerToys add-on.

I just discovered that Windows 2008 has this embedded.  To get a Command Window in any directory right click the folder name while holding down the shift key, this will create an option to Open a Command Window within the directory.  This should work for Vista and Windows 2008, so if you need DOS windows like I do, this is how to do it within W2K8.