Months ago we had a discussion between the Dev and QA teams as to what was going on with our project and make some plans going forward, strangely even though we talk and documentation is given for review a comment came up in the discussion that said "I don't know what QA is testing?"
To me and the Manager it was a WTH (what the hell??) moment. We had never heard this before, and even when we discuss bugs and defect fixes with the Developers this never came up, neither did this when we would go over reproducing bugs. To me, I was testing and they were seeing results, but maybe someone was just not seeing things the same way I was. We got over that, went into a project and did a lot of work over the past few months, adding some tests, discussing a few and verifying many fixes. Yet, from those earlier discussions a document was made that still had that comment in it, so now that we are between versions I am going back to this and starting to think out, what is it that someone was missing and we never picked up on.
One of the projects for the next release is to update our test harness, adding some wrappers to it so that the scripts are more integrated and a database to store results, yes I know we still record some results in Excel and if not for that - not much historical tracking. Well, that and the output documents I have from my tests, that I save for each release and archive away in case we ever need to go through old data. I am also taking time to go to the Developers and show them what we are doing, get some input and putting the question to them "what should we be testing?".
Getting people involved is a good way to eliminate any confusion, since a follow-up to that is going to be "what would you like to test with this?" and offer our Test Harness to those who really do not have one. Some parts of our product don't have many Unit Tests and its a shame, or they have them and they are kept secret, that's a future task I'm not going to be giving them a fish, I intend to give them a rod, boat, ocean of fish and pilot and let them see what they can catch.
Wednesday, December 3, 2008
Monday, October 20, 2008
STOMPing around with the kids
I started a program today with a 5th Grade Elementary class that basically uses Lego Mindstorms kits to introduce kids to Engineering concepts, this is through the STOMP Network. We had a 4 hour training class over the summer, and received some kits to build robots with, I took it home but my 3 year old was not as patient with the kit as I had hoped; yeah I know, not realistic. Still it was fun to watch him see the robot go around, and follow it some where it went and did a course through the house.
First day with this group was basically build a durable structure, in this case a chair, that can be dropped with a stuffed animal in it, I had talked about the class with someone else involved who said it was good to come in with a good and bad example. I made a good chair, that holds up when dropped from knee height, and a chair that would break when it hit the floor, as expected most kids were pointing to the one that would break because they were familiar with those Legos. Still it didn't stop their design process when the kids were told to make their own chair, in teams of 2's and 3's. Funny how many different designs there were, and how many of them wanted to make wheel chairs, whether it was because there were wheels in the kits or they wanted to see something move as well I have no idea.
This continues for the next 8-10 weeks and looks like it will be fun. Next week the kids should finish their chairs, then we move onto something else, eventually making a robotics car that they can program. I have to get used to the programming interface again, thankfully I have time for that.
First day with this group was basically build a durable structure, in this case a chair, that can be dropped with a stuffed animal in it, I had talked about the class with someone else involved who said it was good to come in with a good and bad example. I made a good chair, that holds up when dropped from knee height, and a chair that would break when it hit the floor, as expected most kids were pointing to the one that would break because they were familiar with those Legos. Still it didn't stop their design process when the kids were told to make their own chair, in teams of 2's and 3's. Funny how many different designs there were, and how many of them wanted to make wheel chairs, whether it was because there were wheels in the kits or they wanted to see something move as well I have no idea.
This continues for the next 8-10 weeks and looks like it will be fun. Next week the kids should finish their chairs, then we move onto something else, eventually making a robotics car that they can program. I have to get used to the programming interface again, thankfully I have time for that.
Friday, October 10, 2008
Who needs the Customer? We do.
I was recently discussing an upcoming project with the VP of our group, part of the discussion went to process (as I am the process wonk in my group) and inevitably it drifted to Scrum and Agile methods as they are something I have been talking about over the past year. We do have a good methodology that works well for the development and testing we need to do, its not Agile and potentially could be, but would take a lot of restructuring that upper management is not yet convinced we need since we still deliver very stable software. The next project has a PRD that was coming out, and the teams were reviewing the document I asked who was going to play the part of the Customer Advocate? Who were our domain experts for the product?
The answer - QA.
I was a little dumbfounded, while I will admit we know the product and have a good idea of what a Customer is going to do with our software, were we really in tune with the day to day usage of the product? Were we too close to see some things? Was Development equally as capable since they do some design, which even our lab manager questions and he is representative of the Customer, but the response is the Customer wants it that way. Which makes no sense to some of us, especially the lab manager as he is as close to a Customer as we get, but the response is sometimes that if we needed improvements or changes like we have mentioned they would come through support. To date few have. Still I kept coming back to the real question - is QA really the Customer? Do we really have a good grasp of the Customer's needs and wants and know how they will use the product day to day?
I was not so sure.
Individually the QA Team has an understanding of how the product works, and what it should do, one or two of us have a wide grasp of the product while the rest know our areas really well and have some pockets of knowledge in others. Its a complex product, but would a Customer really use all its features or just those that help them resolve their problems? What were the problems we were trying to solve? Those problems were stated and we knew them, and knew the areas of the product that solved those problems, the new PRD was more on topic with not only stating the problems but gave more detailed User Scenarios than we've had before. Did that close the knowledge gap? Not really, because we have a good grasp of the high level issues and stated problems, we know the solutions for them and how to test them - but something still nagged at me and it was historical.
Last year we had some discussions on how to improve things, we split into teams that discussed how to improve our process and product more - one item showed up on multiple lists - more meetings and discussions with Customers. We don't often know who our Customers are, and in the two years I have been asking I have yet to get anything that shows what platforms our Customers run software on, the only time I know is when I get an issue escalated from a Customer. Since "more meetings with Customers" kept showing up, were we really meeting the Customer's stated needs, or were we meeting the needs as filtered through the Product Managers who talked to the Customers about their needs? Sort of like playing Operator, need A gets said to Product Manager A who then tells Need A to Product Manager B who is also hearing Need B from Product Manager C and its all filtered up into a document that states Need C, an amalgamation of Need A and Need B. But does that make it a real example of the Customer Need or is there still a disconnect? I'm of the opinion that regardless of how much a Product Manager discusses needs with the Customer there is a gap in either experience, knowledge or use that is not quite bridged.
When the same question came up later with a different group of people and there were discussions about the PRD and what we need, some said it was a good document while I and others said its nice but we need Customers to look at it. Regardless of how much we know about the product and its use we often can't bridge the How, this is why people use the mantra of "eat your own dog food", if you cannot get the product to work in your environment for its intended use then no one else will. Customer Advocates are good, and this is what I am trying to bring in for Agile because I think it will improve our product, we can have Customer Advocates at all levels and test or develop with that in mind, but sometimes you need someone outside all of that who will use the product and give you honest feedback.
In otherwords, a Customer.
The answer - QA.
I was a little dumbfounded, while I will admit we know the product and have a good idea of what a Customer is going to do with our software, were we really in tune with the day to day usage of the product? Were we too close to see some things? Was Development equally as capable since they do some design, which even our lab manager questions and he is representative of the Customer, but the response is the Customer wants it that way. Which makes no sense to some of us, especially the lab manager as he is as close to a Customer as we get, but the response is sometimes that if we needed improvements or changes like we have mentioned they would come through support. To date few have. Still I kept coming back to the real question - is QA really the Customer? Do we really have a good grasp of the Customer's needs and wants and know how they will use the product day to day?
I was not so sure.
Individually the QA Team has an understanding of how the product works, and what it should do, one or two of us have a wide grasp of the product while the rest know our areas really well and have some pockets of knowledge in others. Its a complex product, but would a Customer really use all its features or just those that help them resolve their problems? What were the problems we were trying to solve? Those problems were stated and we knew them, and knew the areas of the product that solved those problems, the new PRD was more on topic with not only stating the problems but gave more detailed User Scenarios than we've had before. Did that close the knowledge gap? Not really, because we have a good grasp of the high level issues and stated problems, we know the solutions for them and how to test them - but something still nagged at me and it was historical.
Last year we had some discussions on how to improve things, we split into teams that discussed how to improve our process and product more - one item showed up on multiple lists - more meetings and discussions with Customers. We don't often know who our Customers are, and in the two years I have been asking I have yet to get anything that shows what platforms our Customers run software on, the only time I know is when I get an issue escalated from a Customer. Since "more meetings with Customers" kept showing up, were we really meeting the Customer's stated needs, or were we meeting the needs as filtered through the Product Managers who talked to the Customers about their needs? Sort of like playing Operator, need A gets said to Product Manager A who then tells Need A to Product Manager B who is also hearing Need B from Product Manager C and its all filtered up into a document that states Need C, an amalgamation of Need A and Need B. But does that make it a real example of the Customer Need or is there still a disconnect? I'm of the opinion that regardless of how much a Product Manager discusses needs with the Customer there is a gap in either experience, knowledge or use that is not quite bridged.
When the same question came up later with a different group of people and there were discussions about the PRD and what we need, some said it was a good document while I and others said its nice but we need Customers to look at it. Regardless of how much we know about the product and its use we often can't bridge the How, this is why people use the mantra of "eat your own dog food", if you cannot get the product to work in your environment for its intended use then no one else will. Customer Advocates are good, and this is what I am trying to bring in for Agile because I think it will improve our product, we can have Customer Advocates at all levels and test or develop with that in mind, but sometimes you need someone outside all of that who will use the product and give you honest feedback.
In otherwords, a Customer.
Tuesday, October 7, 2008
Those Three Little Words (or is it four?)
It's something that I used to dread, after all the hype and the information on this that is out there I feared one day I will come into work and hear the words "we're going Agile". If you wish to dispute my counting, feel free. It's my blog and I count "we're" as one word, so get over it.
While expecting this to come out at some point, my fear was recently fired up again when I got a notice about a company reorganization (I think we have one planned yearly), and in it were the words - "we're going to be using Agile processes". Yup, we're going AGILE! There was no date on it, or really any reason why, I have my suspicions but I will keep it to myself. Last summer I gave a short presentation to my group, I do one a month and let others do it when they want or if they have something good, and in that I went over my experiences with Agile and Scrum. Agile was done badly, Scrum was done well, though after a recent test conference I can see it being improved upon.
We discussed the upcoming change at our group meeting soon after the notice came and generated two important questions we thought would be good to answer before proceeding:
While expecting this to come out at some point, my fear was recently fired up again when I got a notice about a company reorganization (I think we have one planned yearly), and in it were the words - "we're going to be using Agile processes". Yup, we're going AGILE! There was no date on it, or really any reason why, I have my suspicions but I will keep it to myself. Last summer I gave a short presentation to my group, I do one a month and let others do it when they want or if they have something good, and in that I went over my experiences with Agile and Scrum. Agile was done badly, Scrum was done well, though after a recent test conference I can see it being improved upon.
We discussed the upcoming change at our group meeting soon after the notice came and generated two important questions we thought would be good to answer before proceeding:
- What is the problem we are trying to solve?
- How will Agile processes solve the problem?
- They have no process and saying you are going Agile allows you to say you are just in a phase of "adaptation"
- The process that is in place has so many exceptions that its not followed, this is the second path to "adaptation"
Friday, September 19, 2008
Ahoy Matey!
Since Talk Like A Pirate Day (TLAPD) seems to be recognized by a lot of tech companies I figured I would add some of my own thoughts on the matter.
Pirates have that certain nautical attitude that basically says "don't mess with me". Or it may even be that certain aptitude to make things happen, or just do what they want, damn the consequences, Piratitude if you go by the original creators of TLAPD. That is an attitude that can help in testing as well, as others have noted specifically Phil Kirkham there is such a thing as Pirate Testing, take what you want and steal the rest, that is a common trait that I have seen in testers and I think its a good one.
First the attitude, when your job revolves around testing and the basic premise of your job is to find faults in others work (mind you this it not my total thinking on the matter just a realistic view on part of it) then you need to have a sense of humor about it. Who has a better sense of humor about life than pirates? When your time is spent wenching, drinking and fighting you better be able to find enjoyment in anything cause when your time comes that's it, if you haven't enjoyed the ride on the way there then you are missing out. Being able to come into work with a smile on your face, and a skip in your walk, every day or even when facing down a Developer for a specific bug, having a devil-may-care attitude is sometimes your best friend. Giving difficult news is tough enough, but when you can come in with an "Arrrrr Matey" and a cutlass at your side, its amazing how the levity and sharp blade can turn things around to your advantage. Pirates have it all, and they never let you forget it.
Testers are like pirates in another way as well, as the Beastie Boys once said, "What do you call Pirate's Trease? BOOTY!!". What's Test Booty? Well its not that hot chick/guy who just got hired out of college. No. It's the treasure and self satisfaction you can get from a job well done, cause after all treasure is just a pirate's way of saying "we took that ship unawares with its cannon locked up and its sails limp". Test Booty can be anything from writing a script that catches bugs, to that program you need to write which once you get it working and see it doing its job are able to sit back and say "my job just got 10% easier". You can shoot for more, but lets keep those goals realistic, we are pirates after all, men in the real world, not philosophers who look towards some Eternal Truth. Unless you can find it at the bottom of a cup of grog.
Yes, Testers are like Pirates, because we go our own way, earn our booty and still at the end of the day can enjoy a nice cup of grog.
Pirates have that certain nautical attitude that basically says "don't mess with me". Or it may even be that certain aptitude to make things happen, or just do what they want, damn the consequences, Piratitude if you go by the original creators of TLAPD. That is an attitude that can help in testing as well, as others have noted specifically Phil Kirkham there is such a thing as Pirate Testing, take what you want and steal the rest, that is a common trait that I have seen in testers and I think its a good one.
First the attitude, when your job revolves around testing and the basic premise of your job is to find faults in others work (mind you this it not my total thinking on the matter just a realistic view on part of it) then you need to have a sense of humor about it. Who has a better sense of humor about life than pirates? When your time is spent wenching, drinking and fighting you better be able to find enjoyment in anything cause when your time comes that's it, if you haven't enjoyed the ride on the way there then you are missing out. Being able to come into work with a smile on your face, and a skip in your walk, every day or even when facing down a Developer for a specific bug, having a devil-may-care attitude is sometimes your best friend. Giving difficult news is tough enough, but when you can come in with an "Arrrrr Matey" and a cutlass at your side, its amazing how the levity and sharp blade can turn things around to your advantage. Pirates have it all, and they never let you forget it.
Testers are like pirates in another way as well, as the Beastie Boys once said, "What do you call Pirate's Trease? BOOTY!!". What's Test Booty? Well its not that hot chick/guy who just got hired out of college. No. It's the treasure and self satisfaction you can get from a job well done, cause after all treasure is just a pirate's way of saying "we took that ship unawares with its cannon locked up and its sails limp". Test Booty can be anything from writing a script that catches bugs, to that program you need to write which once you get it working and see it doing its job are able to sit back and say "my job just got 10% easier". You can shoot for more, but lets keep those goals realistic, we are pirates after all, men in the real world, not philosophers who look towards some Eternal Truth. Unless you can find it at the bottom of a cup of grog.
Yes, Testers are like Pirates, because we go our own way, earn our booty and still at the end of the day can enjoy a nice cup of grog.
Wednesday, September 3, 2008
Windows IPv6 Socket Clients
Just got through building my very first windows IPv6 socket client and server, I am using them for validating that the application I am testing, can detect the IPv6 sockets across machines. Lots of fun stuff.
In trying to get the code to build on Windows 2003 Server, using Visual Studio, I had to add the following into the include directives to get them fixed. Since it took me some searching to get this done let me go over some of this. My code is as follows (and yes its probably not good form, but like I said, this is my first Windows app, I've taken 1 C++ course and read a couple of books and looked at some example code...I'm a total newbie on this).
First the ifdef, I needed to define WIN32 in the project properties under C/C++/PreProcessor Directives
#ifndef WIN32
#else
This pragma was to get rid of a Warning due to a conversion from size_t to int problem where I was holding the result of the bind operation in an int in order to be able to tell what the status was later on.
#pragma warning(disable:4267)
There were problems due to linker errors which are due to ws2_32.lib and iphlpapi.lib not being found on the path, I also needed to add in the libraries to the Visual Studio Platform SDK, and I even placed in the latest Microsoft SDK on my local box, then in the project options under Linker/General/Additional Libraries added the paths to the two SDK's. I'm still not sure I need both of these or not, but they are there just in case.
#pragma comment(lib, "IPHlpApi")
#pragma comment(lib,"ws2_32")
Because I am trying to use IPv6 operations in these apps, I need the updated winsock library, not the older one that only supports IPv4.
#include
End the if directive.
#endif
A couple of other items I had to add:
In trying to get the code to build on Windows 2003 Server, using Visual Studio, I had to add the following into the include directives to get them fixed. Since it took me some searching to get this done let me go over some of this. My code is as follows (and yes its probably not good form, but like I said, this is my first Windows app, I've taken 1 C++ course and read a couple of books and looked at some example code...I'm a total newbie on this).
First the ifdef, I needed to define WIN32 in the project properties under C/C++/PreProcessor Directives
#ifndef WIN32
#else
This pragma was to get rid of a Warning due to a conversion from size_t to int problem where I was holding the result of the bind operation in an int in order to be able to tell what the status was later on.
#pragma warning(disable:4267)
There were problems due to linker errors which are due to ws2_32.lib and iphlpapi.lib not being found on the path, I also needed to add in the libraries to the Visual Studio Platform SDK, and I even placed in the latest Microsoft SDK on my local box, then in the project options under Linker/General/Additional Libraries added the paths to the two SDK's. I'm still not sure I need both of these or not, but they are there just in case.
#pragma comment(lib, "IPHlpApi")
#pragma comment(lib,"ws2_32")
Because I am trying to use IPv6 operations in these apps, I need the updated winsock library, not the older one that only supports IPv4.
#include
End the if directive.
#endif
A couple of other items I had to add:
- When trying to build this for IPv6 I kept getting warnings about sprintf being insecure, and deprecated, this was removed by adding _CRT_SECURE_NO_WARNINGS in the PreProcessor directives.
- I needed the pragma lines because the Linker was having problems with the socket operations and would give me errors regarding LNK2019: unresolved external symbol
Wednesday, August 27, 2008
Let the crowd in...
I was reading James Whittaker's posts on Crowd Sourcing at http://blogs.msdn.com/james_whittaker/archive/2008/08/20/the-future-of-software-testing-part-1.aspx, where he sees the next big leap to be one where everyone who wants to can be a part of the Testing Solution. While I'm not quite sure I agree with it, and some of the quotes reflect my own opinion, I can see where it can be of benefit as an additional avenue of allowing people time and a chance to test software and submit their issues to be fixed. This is a good thing, most people, as he says, do find problems not always found in a test cycle either because of contraints, or some quirk of the User Environment that is not mimiced in the lab; getting those issues in early is a good thing. However, I am a realist, and while its nice to get many of these bugs in I personally don't see where the value is going to be if those bugs are not being resolved but end up in the bug bucket waiting to be looked at. Though I am getting ahead of myself, there were a couple of points in this I thought was interesting.
The Cloud. I'm still unclear on what this is, or whether its just an anticipation to what Cloud Computing is going to be, but I'm not expecting people to be sharing configurations and environments across the Net. In a QA lab you can have all kinds of virtual environments, but they are not personalized, they tend to be representable of what Customers have, either by being proactive and knowing what Customers use, or by being reactive and adding in software or configurations that became known trouble spots. I'll share a specific browser configuration and plug-in configuration with someone, but I'll be damned if I am going to sit there and share someone's Hello Kitty theme with kitty cat icons made up of the heads of someone's little babies. I'm sure that works fine for some people, including some of my co-workers (not the Hello Kitty theme, but the cat picture backgrounds) but its not something I'd expect to see in my lab nor would I expect it to have issues with commercial software.
Bug Reporting. As I said, just because Jim from South Carolina found a particular GUI issue with the tool bar, and it was confirmed with Vijay from Pune as well as Klaus from Dresden, doesn't mean that bug is going to be fixed. I'm all for finding as many bugs as possible before I release, but honestly, do I need 100+ minor defects or 250 Enhancement Requests sitting in my bug queue because the Crowd found them, and the serious issues they had were already reported and entered? I don't see what this gains me, other than more defects for triage that may or may not ever get fixed, but from a business standpoint I can say that we found them with the Crowd! Serious defects should hopefully be caught prior to the release to the Crowd, or maybe specific configurations can be found, and I am all for getting as many of those as possible, but when I look at the numbers of Critical and high priority defects being added in, there are very few found later on; while I'd like those fixed is there any guarantee they will be? I'm still waiting to see the results on that.
Isn't this really Beta? I can see this point, and I partially agree with it. If you are working on an open source project you expect updates that may or may not be very well tested, with a Beta you are taking this knowing that the software may crash (and badly) in your environment because that is what it is. So what is this? Beta? Well, I'd say not really since it seems like the releases are still very early in the cycle, unless the company has a long timeframe between a release build and eventually getting it out. So where does the testing stop? Does the release mean everything found is put on hold and we now wait for the next release, especially since Customers now have a copy we can be getting reports from them. I'm waiting to see more on this, but I am trying to keep an open mind.
Talent Pool. Who are the people who are joining? Are these people new to the field and they want more experience? Are they bored? What is their experience? As someone who has trained people off and on I can tell you that a bug report from someone who is just learning compared to someone who has been doing this for a long time is very different. I'm not sure who is signing up here, the money doesn't seem like much, and maybe I am not their talent pool, but when I get out of work the last thing I want to be doing is testing some other kind of software, heck I don't like to practice programming much outside of work. I'm not the target audience for the community, but I'm curious as to who is.
Unlike Mr. Whittaker I don't know if this is the next logical step, but I like to take a long view and a higher view of testing as a whole, some things get tried, some change and some just end when its seen they don't lead anywhere. I don't know where uTest and the Crowdsourcing is going to go, but I am curious to watch and see.
The Cloud. I'm still unclear on what this is, or whether its just an anticipation to what Cloud Computing is going to be, but I'm not expecting people to be sharing configurations and environments across the Net. In a QA lab you can have all kinds of virtual environments, but they are not personalized, they tend to be representable of what Customers have, either by being proactive and knowing what Customers use, or by being reactive and adding in software or configurations that became known trouble spots. I'll share a specific browser configuration and plug-in configuration with someone, but I'll be damned if I am going to sit there and share someone's Hello Kitty theme with kitty cat icons made up of the heads of someone's little babies. I'm sure that works fine for some people, including some of my co-workers (not the Hello Kitty theme, but the cat picture backgrounds) but its not something I'd expect to see in my lab nor would I expect it to have issues with commercial software.
Bug Reporting. As I said, just because Jim from South Carolina found a particular GUI issue with the tool bar, and it was confirmed with Vijay from Pune as well as Klaus from Dresden, doesn't mean that bug is going to be fixed. I'm all for finding as many bugs as possible before I release, but honestly, do I need 100+ minor defects or 250 Enhancement Requests sitting in my bug queue because the Crowd found them, and the serious issues they had were already reported and entered? I don't see what this gains me, other than more defects for triage that may or may not ever get fixed, but from a business standpoint I can say that we found them with the Crowd! Serious defects should hopefully be caught prior to the release to the Crowd, or maybe specific configurations can be found, and I am all for getting as many of those as possible, but when I look at the numbers of Critical and high priority defects being added in, there are very few found later on; while I'd like those fixed is there any guarantee they will be? I'm still waiting to see the results on that.
Isn't this really Beta? I can see this point, and I partially agree with it. If you are working on an open source project you expect updates that may or may not be very well tested, with a Beta you are taking this knowing that the software may crash (and badly) in your environment because that is what it is. So what is this? Beta? Well, I'd say not really since it seems like the releases are still very early in the cycle, unless the company has a long timeframe between a release build and eventually getting it out. So where does the testing stop? Does the release mean everything found is put on hold and we now wait for the next release, especially since Customers now have a copy we can be getting reports from them. I'm waiting to see more on this, but I am trying to keep an open mind.
Talent Pool. Who are the people who are joining? Are these people new to the field and they want more experience? Are they bored? What is their experience? As someone who has trained people off and on I can tell you that a bug report from someone who is just learning compared to someone who has been doing this for a long time is very different. I'm not sure who is signing up here, the money doesn't seem like much, and maybe I am not their talent pool, but when I get out of work the last thing I want to be doing is testing some other kind of software, heck I don't like to practice programming much outside of work. I'm not the target audience for the community, but I'm curious as to who is.
Unlike Mr. Whittaker I don't know if this is the next logical step, but I like to take a long view and a higher view of testing as a whole, some things get tried, some change and some just end when its seen they don't lead anywhere. I don't know where uTest and the Crowdsourcing is going to go, but I am curious to watch and see.
Subscribe to:
Posts (Atom)