I was reading James Whittaker's posts on Crowd Sourcing at http://blogs.msdn.com/james_whittaker/archive/2008/08/20/the-future-of-software-testing-part-1.aspx, where he sees the next big leap to be one where everyone who wants to can be a part of the Testing Solution. While I'm not quite sure I agree with it, and some of the quotes reflect my own opinion, I can see where it can be of benefit as an additional avenue of allowing people time and a chance to test software and submit their issues to be fixed. This is a good thing, most people, as he says, do find problems not always found in a test cycle either because of contraints, or some quirk of the User Environment that is not mimiced in the lab; getting those issues in early is a good thing. However, I am a realist, and while its nice to get many of these bugs in I personally don't see where the value is going to be if those bugs are not being resolved but end up in the bug bucket waiting to be looked at. Though I am getting ahead of myself, there were a couple of points in this I thought was interesting.
The Cloud. I'm still unclear on what this is, or whether its just an anticipation to what Cloud Computing is going to be, but I'm not expecting people to be sharing configurations and environments across the Net. In a QA lab you can have all kinds of virtual environments, but they are not personalized, they tend to be representable of what Customers have, either by being proactive and knowing what Customers use, or by being reactive and adding in software or configurations that became known trouble spots. I'll share a specific browser configuration and plug-in configuration with someone, but I'll be damned if I am going to sit there and share someone's Hello Kitty theme with kitty cat icons made up of the heads of someone's little babies. I'm sure that works fine for some people, including some of my co-workers (not the Hello Kitty theme, but the cat picture backgrounds) but its not something I'd expect to see in my lab nor would I expect it to have issues with commercial software.
Bug Reporting. As I said, just because Jim from South Carolina found a particular GUI issue with the tool bar, and it was confirmed with Vijay from Pune as well as Klaus from Dresden, doesn't mean that bug is going to be fixed. I'm all for finding as many bugs as possible before I release, but honestly, do I need 100+ minor defects or 250 Enhancement Requests sitting in my bug queue because the Crowd found them, and the serious issues they had were already reported and entered? I don't see what this gains me, other than more defects for triage that may or may not ever get fixed, but from a business standpoint I can say that we found them with the Crowd! Serious defects should hopefully be caught prior to the release to the Crowd, or maybe specific configurations can be found, and I am all for getting as many of those as possible, but when I look at the numbers of Critical and high priority defects being added in, there are very few found later on; while I'd like those fixed is there any guarantee they will be? I'm still waiting to see the results on that.
Isn't this really Beta? I can see this point, and I partially agree with it. If you are working on an open source project you expect updates that may or may not be very well tested, with a Beta you are taking this knowing that the software may crash (and badly) in your environment because that is what it is. So what is this? Beta? Well, I'd say not really since it seems like the releases are still very early in the cycle, unless the company has a long timeframe between a release build and eventually getting it out. So where does the testing stop? Does the release mean everything found is put on hold and we now wait for the next release, especially since Customers now have a copy we can be getting reports from them. I'm waiting to see more on this, but I am trying to keep an open mind.
Talent Pool. Who are the people who are joining? Are these people new to the field and they want more experience? Are they bored? What is their experience? As someone who has trained people off and on I can tell you that a bug report from someone who is just learning compared to someone who has been doing this for a long time is very different. I'm not sure who is signing up here, the money doesn't seem like much, and maybe I am not their talent pool, but when I get out of work the last thing I want to be doing is testing some other kind of software, heck I don't like to practice programming much outside of work. I'm not the target audience for the community, but I'm curious as to who is.
Unlike Mr. Whittaker I don't know if this is the next logical step, but I like to take a long view and a higher view of testing as a whole, some things get tried, some change and some just end when its seen they don't lead anywhere. I don't know where uTest and the Crowdsourcing is going to go, but I am curious to watch and see.
No comments:
Post a Comment