Multi-Lingualism - One Testers View

I've been asked, and often seen asked, "I am a Tester (QA Engineer or whatever the title is) and I want to learn a language, which one should I learn?".  Well...

Answer: Whatever will help you in your job.  There is a lot out there, its like deciding what to learn for a second language, you learn one you enjoy or one that will help you progress, its the same with software and computer languages.  If you work testing Java apps, then learn Java it will only help you out, Scheme or Lisp no, they may allow you to understand jokes from some old programmers but thats about it; unless you work in a place that uses Scheme or Lisp.

For more keep going.

This is going to be a HOP (Highly Opinionated Post), and will cover my own career in this space, so bear with me.

I started in QA with no education in this field except what I had learned myself, some BASIC, some more Visual Basic, a little C and at the time I was hacking PPP and SLIP, plus some HTML.  I decided I needed something to help do things in a more automated fashion, and at the time the language of choice for that was Perl.  Over the years I have become a good Perl hacker, and can pretty much get to happen what I want when I need to, only now though I am doing it in an object orientated fashion.  I ended up using a few test tools over the years, WinRunner and LoadRunner, so I became familiar with their scripting languages because if you want to use them effectively you need to.  I used a Bug Tracking tool years ago called Clarity, for that I needed to be trained in their proprietary language, so I took their courses and got our system set up to run.  It worked well, but over the years I have also come to like Bugzilla - and what do you know its written in Perl!

Product-wise I spent many years testing web sites and multi-tier systems.  I have had to test ASP, Visual Script, use a tool written in Visual Basic, and databases.  Well, learning ASP was not all that hard, its just getting used to its syntax (could I code in it?  No.  But I can read and follow it...I've even hacked it to perform date testing).  SQL, that was a little tricky and I got one of those SQL in 21 Days books, that followed me around for years.  Windows Installer was another item we tested, as we were installing some Applications, so I got used to the syntax for running it on the command line, and then check out how to read its logs.  Java came into much more use as I worked, I started learning it twice over time, never got good at it, but hey I can follow it!  So testing Java Apps became easy.  Especially Java Apps that used databases since I already knew SQL.  In one job we started using JUnit, and that meant working with Java, it was some doing but I got some JUnit tests created and running.  Shell scripts were useful in a couple of jobs, as we needed to be able to work some packages properly, and there is just some stuff you SHOULD NOT do in Perl.

Taking a look back I started with very little, but what can I do now?

• SQL, can query, create, delete, destroy, manipulate and do what I want with data
• Perl, anything Perl can do I can almost do, not that great with Objects yet but getting there
• ASP, I can read, follow, hack and troubleshoot it
• Shell Scripting, can set up something to run what I want
• Batch, can set up Windows Batch scripts to do what I want
• Java, can read, follow and troubleshoot it.  Hacking is a bit different
• JUnit, can make a JUnit test since I can follow Java
• Perl, most useful tool I came across and its multiplatform!
• C/C++, can't code much in it, but I can follow it - great for Code Reviews!

Recently I started tackling Python, because at home I decided I needed some way to track my downloaded music videos, a bad hobby I know.  But what else was there to do with the tools I had but install MySQL and then add in a front end to query data, I used Perl before but wanted something graphical and Python seemed the next logical step.  So now I can add Python in my little bag of tricks.

If a project comes up and I need to work with something, learn about it, that is what is great about this industry, you can read all you want and there is always something new.

Note: I know Perl is listed twice, it's on purpose and you can make your own jokes about it.

Tools I've Used - AutoIt

Over the years I have used many tools, and tried many more, for testing.  One of them that I have grown with, and grown to like, is simple and free with a pretty good fan base.  AutoIt. 

I originally found it when I needed to automate a Customer Test, this was for a simple app that just found documents on a Customer's harddrive and uploaded them to a server; which the Customer could then distribute.  But since the issue dealt with doing this repeatedly I started looking at this tool to handle it, you can make your script based on screen positioning or by the use of tabs in an application.  I opted for tabs since I don't like screen positioning.  It worked well and did a fairly good job, though I ended up using Perl to generate the file upload list for the application, and used AutoIt to run the Perl Script, then start the application which made things easier!  Then with our actual Web Application I used it to make small scripts that would do Customer functions on the web site, again using tabs since I liked that - which as usual needed maintenance when the Developers changed the pages.  Still it worked well, and as the team grew others used it as part of their testing, one member of my staff was using it to generate Customers by having it run on the Customer Creation pages over and over - which made it easier to generate needed accounts the same each time; and check for changes in the Customer Admin pages.  It really fit a small niche.

For a free tool I have gotten some good use from, its been great.

As was noted when this was originally posted - the right tool is the smallest tool.

Pros:

• Scripting Language is Basic-like, so its fairly simple and easy to read
• Can do Tab (keystroke) movement or Mouse positioning
• Has a lot of add-ons with an active development community
• A good community of support, both on the Web and on Yahoo
• Scripts can be converted to EXE's

Cons:

• Windows based, if you want multiplatform its not that useful
• Syntax was sometimes not that clear, but this seems to be solved in the latest version
• Web Scripts can work differently on Multiple Browser types, I never was able to make a generic script (but that may just be me)

Finding the Right Help

Since there seems to be much being made about getting employees, on this site and in conversations I have had with people recently, I'll add my two cents about this.  I'll try to put it in some kind of order, but its going to be difficult, so bear with me if I go out of order.  Since I usually started as the first person in QA it was easy for me to grow the group as I understood completely the space and the work, so I had a very good idea about what skills mattered.

• Getting the people in.  Referrals, Referrals, Referrals.

I've never found an easy way to get people in, but the best first option is to use your local resources, co-workers who might have worked with some brilliant QA Engineer in the past or knows someone that may fit into the company really well.  If you have been lucky enough to hire some QA staff already ask them if they know people who want to come work with your company, its also a way to check enthusiasm since people who WANT to be there will want to bring in their friends.  Most companies offer referral bonuses, or some sort of compensation, for referring people.  Also make sure you know where people have worked, if you get an applicant who worked with someone you have on staff, or is a colleague, see what they remember - it may be worth pursuing or not.  But take in mind that its one persons opinion.

Although the only other option to this is posting the job on the company site, internet job site or to a recruiter and they all have their drawbacks.  When you do it this way, clear time on your schedule to go over resumes, while its been nice to have HR go over resumes, unless you have an HR person who REALLY understands your needs you will either get resumes that don't fit or never see ones that might.  In all these situations there are some options.

1. HR - meet with the HR person or Rep who will be handling the job, make sure they understand the needs and what the job description is, sometimes I've had Reps who knew Development but not Testing.  You get some odd resumes that way.
2. Recruiters - they can be annoying, and we've all heard the negative stories, but if they are a good company and you get a good relationship with the company or its Rep they can be good.  I always send back my feedback to the Rep after a meeting with a candidate they send, telling them the good and bad and why I reject or accept them.  It's helped to give me better candidates, and if a company doesn't seem to get it I drop them, its not worth my time to get bad candidates in.  I also go over the experience on the resume with the person, that way I know if the resume has been doctored.
3. Company Site Posting - usually a small deluge when the posting goes up, that eventually fades away.  This I will usually go through with the HR Rep, because some resumes give you an idea about what the persons communication skills are like and some may have the right skills but not written in a way someone with experience in QA might recognize.
4. Internet Job Site - this will be a floodgate opening when it goes up, you may find responses from recruiters on behalf of applicants in this flood as well, keep it all in mind.  Again, go over the resumes with the HR rep, but take time for this, as it will not be quick.
5. Oh yeah, SAVE those resumes with comments too.  I've had cases where I interviewed one person, then in another year got another resume and the name seemed familiar so I checked my files and we had rejected the person as a bad fit and not the right skills.  In a year I sincerely doubted the person would change, so we did not bring them in for another set of interviews.

• The initial interview - phone or otherwise

I would always check the past few jobs and ask them to describe in more detail what the job entailed, trying to get an idea of what the person really did on the projects in the past and what their feeling was on the work.  Wording on the resume may or may not match what the person really did, and I like to find that out in the beginning, but I also want to make sure the person can communicate effectively.  Since QA is a lot of face to face discussions with Developers and others the person you hire must be able to communicate with your colleagues, and without much difficulty.  I'm not picking on people with second languages at all, but if in speaking and writing you cannot understand what the person is talking about, especially in regards to a defect report, then no one else will either and it will make work difficult for others.  During this interview I asked most of the questions as well, as I am trying to decide whether this person can come in and do the job, if the potential is not there I will not bother, there will be time for candidate questions in the face-to-face.

• The face-to-face

This is when I go into more detail on the job and really try to grill the candidate on experience, skills and test methodology.  Depending on the level of the position I may ask more technical questions or just try to find out why the person wants to get into testing for a Junior Position.  I also ask for examples, if someone knows how to write SQL I ask them to tell me some commands, how to select from a table, two tables, multiple tables, do a sort - just to make sure they have a grasp of things.  There is also The Example.  Taking an example of something that is under test, I give some vague description as just ask what they would do.  See if it matches what we do, this gives an idea of mind fit.  Then there is the Technical Example, something that really challenges them to use their skills, this can be anything from how to test a product to my latest favorite, how do you test a bank of elevators in a building?

• Group Melding.  How is the fit?

During the interview process I have some members of the QA staff interview the candidate, this is to determine how everyone will get along, sure we may spend time with the candidate during the interview process and a couple of days after start date, but the day to day interaction will be with the QA staff.  If there are going to be issues, find them out now, before you have a serious issue on your hands.  Since there is only so much time in the day we broke out the interviews to focus on each persons special skill areas, one person may ask the more technical questions while another may ask the more process related items, use the strengths of your staff to your advantage.
Also I like to ask the same question multiple times, with different people, there should be different answers unless the person has been coached, in which case a warning should go up.  It's happened to me in the past.  Also figure out a way to end things early, if the process is going then a large warning flag goes up, you may decide to cut the candidate loose, plan for this.

• Contractors

This is always tricky, you want someone in for a short time, with only specific skills but do you really need to do the whole interview process?  In a way I say yes, because this person will be there day to day, and if its long term you will get to know them like any other co-worker, in some cases the contract can become permanent so plan for permanence now; it may save you headaches later.  But as with regular applicants I send my feedback immediately to the company during the interview process, while it is still fresh in my mind, in many cases the company has responded by sending Contractors better meeting my skills.  There have been a few I have hired in the past, some I took on full time, but I had that option in mind on the interview and made sure the fit was good; some have been the best employees I ever had.

---

There ya have it, and while this has worked for me over the years, use parts on your own and in your own environment, my expectations and needs were probably different than yours.  A lot of this is planning and managing the risk, items we should be well familiar with in QA, these are skills you can use in any endeavor, not just the latest project.

Sharing the Environment

Years ago,  my first QA job actually, we were sparse on resources and had one lab for everyone.  Yup.  A server for all, and a database to share.  This was for a start-up that had limited resources and money, QA was brought in after Development really got under way so it was a lot of catch-up.  Originally, until we got the middle-ware stable the Development Leads did the actual releases as well, with input from me, so if there were issues or something missed they could discern the issues easier.  Still it was all we could do, so we needed to work out the risks.

Here is how we ended up doing it.

• Web Site, seperated out the main home directories, easy enough to do.
• Web Server, this was the tricky one, because it was a compiled executable and when it was updated mostly they were library files.  Really we just promoted the new files, even if they were the same as the existing ones, not something I would recommend but it worked at the time.
• Database, sort of tricky but we just used different Oracle instances and that allowed us to test in isolation.
• Other middleware, which was the bulk of the code, we handled this by reinstalling.

Was it full of risk?  Sure.  But we also had weekly or bi-weekly releases, once a package was out we had another coming in.  This did work for us for a year and a half, it was all for one Customer so in that we even had an easier time.  That Customer went on to be a fairly large ISP in the UK so we were doing something right on the back end, there were few problems that arose after deployment.  So sharing an environment can be done,  but you need to be aware of EVERY potential risk, especially when you are testing in a dev environment that already has the tweaks made for fixes - and may not be included in the package.

To me the whole idea of Risk Management is what needs to be done for every release or build coming into Test, if you don't know your environment enough to see the Risks then you can set yourself up for trouble.  Its worth time to ask "What if..?" questions all the time, some Risks you won't know until you have experience to ask the questions, and some will come from doing the releases and seeing the code work in the field.  The goal is to minimize those issues that arise from the field, or to eliminate them entirely (a good pipe dream).

Would I do testing like this again?  No.  I've learned that having your own environment that is yours and yours alone to test with is the best way to mitigate everything.  I know it can be done to be in the same environment as Development, I just would never recommend it.

Software I hated...

This is a personal story and the names are changed to protect somebody, but honestly, there are no innocent parties here.

The beginning of the year I was looking for some DVD Authoring software, because my son was 6 months old and we had lots of pictures and small digital movies I wanted to do something with them.  Saving them onto a DVD of my own devising seemed like a good plan, and I figured it would be good blackmail material when he gets older.  So I looked around for some different types and settled on a couple after some research to features and price, it all reminded me of looking for testing tools.  A couple had some Try Before You Buy options, I settled on the one that seemed most promising.  Downloaded the software and got to play with it.

Initially I was happy with it, slow in converting some movies but it suited my needs and I decided to buy it, I ordered it from the company web site and downloaded an update which basically gave me the full version.  I used it for quite a few months, though I did get the occasional crash when trying to convert a movie, and larger ones took a long time to do, I put it down at the time to my machine and Windows.  There was a feature to make Audio CD's, and this is when my delight turned.  Wanting to put some of my music on CD's I pulled together some songs off CD's I own and they all came out distorted in places, after going through the ripping process and finding nothing wrong I decided to look on the company site for this issue.  Originally I did not find it but through a thorough search of the User BBS for the company did find that this was a known problem, as were the long conversion times and crashes.  But there was an update!

So I downloaded the update, it said it was for my version and was linked into my Company Support and Info page so I downloaded it and started to run it, Install Shield started then came up with a message box "This version is not installed".  But there it was on my computer, and I was using it for months!  Back to the BBS, yes this was a known issue, you need to reinstall from the CD.  I didn't want the CD since I could download the software and I can make my own; besides why have more CD's sit around.  So I couldn't update because the way I had purchased and downloaded the software was an unsupported path to patches, though I was given the option by the manufacturer.  Well Tech Support should handle this I though.  Wrong.  My initial email to support provided me at first an auto response with 5% off the next version.  Why buy another when all I wanted was an update?  In my email I noted all the steps I took, pretty much like I would document a bug report, and their answer was to send the initial email of my purchase with its links to the company headquarters in another country.

I didn't have the email, I have no idea where it went actually so that was my fault, but told them they could see I was a registered user, gave them my build number and offered to send them a copy of my credit card statement showing the purchase.  After a week with still no response I emailed again asking a status, it still occurred to me that if they offered my particular update path then it should be tested, so I reinstalled and found my update files for the full version would only work with the Try Before You Buy installed; so I know what I had and told them this again in a follow up email.  After 4 days of no response I opened another ticket with Support, referencing my first one, got another 5% off coupon which I deleted.  After another 5 days of no response to either one I have decided to move on, I will never buy this company's products again and I will always be SURE to get a CD for the software if I ever buy anything, or just not get Downloaded software anymore.

Lots of QA and Support lessons here, its too bad in this day and age that companies cannot be more responsive, but after seeing and hearing many horror stories about testing and support its always infuriating to be a vicitim of one.  Especially when you work hard to remove these kinds of situations from your own little corner of the world.

Harnessing the Harness

In my current position, part of my work involves utilizing a Test Harness to perform some distributed testing.  This is a Harness written in Perl, which is one of my favorite languages, it covers multiple operating systems (windows, Linux, Solaris, HPUX and AIX) and while parts of it are the usual Perl hack most of it is in Object-Perl.  The developer of this and I are working to get it even more distributed, objectified (if thats a word) and also just spreading out how to use this tool so we can also allow others to use it for Development testing.

We ran it on one of the latest builds, and sure enough there were lots of failures, so we talked and the initial response was that this happens and we should run it again.  So we did, and of course there were less failures.  Over the weekend some network upgrades happened, never a good time in a test cycle, so I started running them again after the lab was back up.  I got different errors, and in come cases fewer.  So I started thinking, what was it we were trying to prove here?  Did we simply want to be running the harness over and over again until the errors were gone?  Did we want to investigate the initial errors and see where the problems were from?  Did we have enough confidence in the Harness to know the results we received were accurate?

After some discussion, it came down to what I considered the usual response.  Until we know its not the Test Harness creating the issues, there is no reason to report the issues as bugs, unless its for the Harness.  We could run some of the tests manually, but the reason for having the Harness is to take the manual time and cut it down...so we are in that state where we debug the Test Harness until we are sure it works, in parallel we will use it on the product and try to get some confidence with repeatability on the Test Results.  I don't think that will happen for awhile, but at least we will spend time getting to see how the tool reacts, and be able to judge better how it will work out in the future.

Those pesky nightly tests...

It's come up a few times over the years, how do I make sure my coverage is good and make sure we are testing all we should all the time?  Or in time enough?  Is there enough time to test all we want to?  The questions just keep coming over and over.

I've actually ended up doing this a couple of different ways over the years, and I don't guarantee any one of them will work for anyone else, and I am not exactly sure they worked for me all the time either.  There was always more analysis we needed to do.

1. Jump in there and do it.  Oh yes, we jumped headfirst into making some nightly tests, anything was good so long as it tested SOMETHING.  This was the worst way, and it was only done because the Engineering VP wanted something in place, mostly I think it was a way to tell the Board that we have automation or nightly testing in place.  Has this topic ever been covered in an in-flight magazine?  While it was a good exercise in getting QA involved with the Nightly Tests (we previously had none) and allowing us some time to do some scripting we actually got more out of the scripts used for our regular Test Phase.  This is the worst way to do it, since you are basically just doing anything for the sake of doing something, I think we ended up calling this the Exploratory Nightlies; no plan, no review of technology, just seat of your pants Nightly Testing.
2. Use what Dev had.  As there were some tests that Developers had written already, in both Perl and a few other Unit Test frameworks we had a good setup.  Tests were written, established, covered needed areas of code as these were always written to cover bug fixes; they had Unit Tests as well but they were shorter to run and the Nightly Tests ran longer so they became the place to put all longer running tests.  This worked fine, as the reporting mechanism was also automated and plugged into many of the Dev and IT infrastructure in place.  Problem was if new Unit Tests were added that did not quite fit the model in Perl, which the Nightly Framework was written in, then reporting would cover an entire suite of tests as one success or failure.  When we added JUnit, it also caused some issues with both the reporting, and some issues with environments which was not always translated to the individual test; which was sometimes running Ant and JUnit started by Perl.  It was a framework that worked for a time, but was not essentially scalable with new technology as we added new applications and languages to the product.
3. Where was that Matrix again?  Ah yes, the Test Matrix.  Each item of functionality had a test point and a test case, so we wrote up something to cover it.  This was a way to get the coverage we wanted, and we could say that if a test failed we knew what work was being done, and if a whole section of tests failed we knew a checkin was bad.  So we would take note of it and spend more time later on testing that area.  Technology here was not so important, though we tried to make sure all of the Matrix we wanted to put in was capable of being done so with the technology we chose, the fit was more important.  Still we ended up with a blend of technologies, but that seems to be the norm.  It became easier to manage though, in the point of view, where we could match tests with cases, though how this will go in the long run I can't perceive yet.

I actually have never considered buying anything, though its been tantalizing, the issue I have with buying a product is that I tend to work on products that are a mix of technologies or have hooks already in them for Unit Tests.  QA takes those hooks and expands on them, it also allows us to program our tests and know what works, and we can also keep an eye on the framework and make sure its working and the failure is actually from the product and not the framework itself; often a cause of issues.  Homegrown systems are also more managable as there is experience building them, so you have more intimate detail on what it is doing, rather than just learn a scripting language.  I'm not knocking testing tools, I like some of them, but I've had better success, even with some of the massive failures I have seen, in having the diy framework in house.  But its not part-time, this means dedicated time and resources, and the ability and will to take time to learn new things, concepts and more; its not for the faint of heart but its worth the effort.

Flavor Of The Month - Part 2 (Agility with Agile)

At the same company that never got off the ground with Six Sigma, though we had a pretty efficient cappacino process, at some point someone got tuned in to Agile.  This time we had a meeting between the VP of Engineering, the President of the company, the Dev Director, the Architect (who became the other Dev Director...long story there) and me (the QA Manager).  We sat and discussed it, the benefits of it, then with the Cockburn Agile book in hand went off to read about it and see what we could do about implementing the process.  The system was nice, QA and Dev working together, with PM to get the projects completed on time.  This was all done because we were consistently late with the projects, requirements changed during the development and test cycles, often times at the last minute, while also trying to do these ambitious projects in short cycles because "we were the best!".  You can probably guess what happened.

After a couple of days of discussion we broke it to the teams, they would get broken up into 3 groups, each one focused on a different product the company provided, and they would all sit in the open areas designated for the groups.  QA moved out of the QA lab (which I will admit was pretty noisy with the 20 or so computers and video hardware) and into the Agile Groups; which I will call Groups 1, 2 and 3.  Furniture was arranged, the spaces were broken up with help from everyone and the stuff was set up over the weekend so that on Monday morning everything was ready to go.  Here's how it all broke down...goes to show, even with good intentions if the entire business process doesn't change nothing will really get fixed.

Group 1:  This one I called the "Puppy Pound", mostly due to the half walls ringing the space outside my office, there were about 16 desks around the edge where everyone sat; reminded me of a puppy pen in a pet store.  This space was also on the way to the kitchen from the rest of the company, so the opening that let people walk though was closed off at one point to keep the noise down; didn't do anything about the noise in the kitchen though.  The group worked together fine, QA had two members with about 12 developers, one doc person (who was never hired) and a PM delegate (who surprisingly worked on maybe half the projects until she left).  The Team Lead was constantly called into fixing legacy issues on the core product, items that were always pushed off to be fixed in the next release because our DB was overtaxed and constantly being updated without enough testing to all the components, plus we had excessive amounts of data; we tried to manage the changes in testing but sometimes failed - though we did know all the changes happening.  Another member on the team worked with the Architect on special projects, that were supporting the core product, but he was distracted.  So the team while being together did what they could, but did not do a lot of new stuff, they ended up being a sort of triage and special projects team; my personal view.

Group 2: This was a group that was a bit smaller but had more PM and Doc people, including a UI person!  They had really high walls around them on most of the sides, they did the next generation project and were overseen by the Architect who sat in the next room.  They did ok, but they also had a lot of turnover, the project they were on was behind for months, and had extremely tight schedules, which I disagreed with but attempted to work out anyway.  This group was the unfriendliest group, they never really talked as a team (you'll soon see why), and even by sitting next to each other, the QA members sometimes had no idea what was going on.  The Architect never managed them effectively, and the dates slipped because what oversight there ended up being was minimal, turned out when one of the DB guys left he hadn't done work for quite awhile and no one seemed to have noticed.  Apparently the Architect had also told the Developers on the team to not really mix with anyone else, they didn't, but a few of them talked to the other members in the group but it was nothing constant.  I kept saying the dates were in danger, but the Architect claimed it was all under control, as he was a coffee buddy with the VP I was ignored (the two of them beat anyone into submission who disagreed with them), eventually I left because it was not worth my time to deal with someone who was not going to listen.  The product went out a little late, and much scaled back, and took months to work out the kinks.  From everyone I talked to about the project afterwards, it was a shambles.  Did I mention the schedule for this was decided upon by the company officers at a meeting in the Spring, for a Winter release, without knowing the technology or talking to anyone who might work on it?  Did I also mention that the main database and software we were using to base this work on, was from a release that the company which created it was no longer developing, or supporting, after our initial release date?  Oh yeah, those are kind of important.

Group 3: This one personifies my personal belief in geography killing team work.  This one had an area that snaked along the wall, with one of the main doors in the middle of it, the QA and Dev members gravitated to one side, PM and Doc on the other.  Never the twain really met.  They pushed hard and got a lot of things out on tight deadlines, the Dev Director was totally involved in this and often was seen coding with the rest of them.  Dev and PM rarely seemed to talk.  There was some change as things went on, and a reliance on technology from a company in another country, that constantly claimed to do things that it did not.  When they were on site they would tweak and fix things for us, but never trained us on everything they were doing, though we tried and did get some training by watching them over time; even getting friendly with one or two who helped us out by showing us some tricks.  The weakness here was a reliance on technology we had no control over, which was being pushed to the edges of what it was claimed it could do, and be on a schedule that was totally unrealistic.  Items Agile was not going to solve.

The company is in the process of being liquidated, right after I left there was a mass exodus, which was not my intent I actually did leave for personal reasons.  Still much here is my view and from discussions I had with people who were still here, if you worked at this place (and I am sure you will know it) then take this all with a grain of salt.  We were doomed from the beginning.

Documentation

When I have had projects come in previously there is usually not much in the way of documentation, with minor updates or when I was dealing with a service provider on the web we had the occasional page update.  Occasionally we had a Release Note, or a help file, that was sent with the software or was linked in to a web page somewhere, but then the question would come up....how much do we test the documentation?

Granted, the Documentation Groups know how to spell and use grammar check in whatever software they use, sometimes using the software to check the steps out; that has to be the best group I ever worked with!  Still, when we get a package of code into QA we like to look at everything, and I always add a little time somewhere for documentation.  Even though there is someone reviewing the pages, I have encountered the odd missing comma, double period, mistyped word or homonym.  So I at least try to look over the page before it goes out, and my keen editorial third eye often finds something wrong on the page before I read a word.

This was something that I was reminded of recently when helping out a friend.  She is writing documentation in Chinese that needs to be translated to German.  To handle the intermediate step, though i have never seen Chinese to German translations I bet its interesting, they were going to do Chinese to English to German, but the English was awkward.  I was asked to review it, and sure enough there were some edits needed.  I made a few, tweaked a word or two, then sent it back to her; I have not heard back but I hope it made for a better document to be made into German.  Not that I will ever understand it.

In major projects I push to have an entry on the project schedule to get the docs in, and in with enough time for revision as there is usually something that may need a tweak or two.  Never a lot of time, but enough to make sure we cover it completely.  Its not really Testing the Documentation, more like a review, but in some ways it is important to make sure what the Customer has will also convey in a language they can understand what the software should do.  To me its an added dimension to Testing, more of a tangent, but still part of the whole.  As a generalist I find that QA is often more suited to this than say Developers, as we are technical enough to get down into the guts of the code and test appropriately, but can still stand back and as a Customer think and use it like they do.  This adds a good portion of completeness to me, and I think helps increase the Customer Experience a bit better while also making sure what they are delivered is good, understandable and won't make them think that something was rushed.

Transition Plans

Being as I am changing jobs this week, from a pretty large startup to a multi-national, its been on my mind that the "knowledge transfer" needs to be done.  I've done it a few times before, and I am well known to be a document hound; if something has to be done its documented.  No questions asked.  We'll probably end up doing it again in 6 months or more, so I like to review how we did it previously.

Ever since I worked at a company I will call NaviPath, which was its name and since it no longer exists I figure its safe, we had an initiative to get ISO certified.  My job at NaviPath was in Release Engineering, but I also did Smoke testing of the builds to make sure everything worked properly, I've done Release and QA in my many years.  I have no idea why ISO, but we needed to do it.  There was the documentation reviewer who made sure everything going into source control, in the appropriate document form, was correct and numbered properly.  The pyramid went around to test people on the procedures (you can be sure the pyramid got lots of movement by others as well) and to make sure they understood what we were doing.  True the company folded before the initiative finished but from it all I took the document format with me, because it seemed a good QA fit.

So what does this have to do with Knowledge Transfer?  All will become clear.

When I started my next company, I again wrote down procedures and processes.  Using the ISO format, and using my own numbering, I kept track of tasks I was doing.  Generating a Test Plan, short and simple document on creating Test Cases.  Installing parts of the software, a document with small screen shots that showed all the steps and things to watch for, gotchas that happen and everything else we could think of; these increased over time.  All in all, though not all contained excessive detail, we had about 30 documents by the end of my employment as the QA and Release Manager; as I brought new people in and they wanted to know how to do something like rebuild the QA servers in the lab - I pointed them to the documents.  Dev wanted to know how something was done in QA, there was a doc that explained it.  Or I talked it out with them, then wrote something to be used the next time.  These documents became our official departmental records, and even after I left I heard from people who stayed behind that they appreciated them because they had forgotten to ask about one of the tasks we only did once or twice a year, but the document was there with all the steps.

So in looking at transitioning now, I can think of all the work I do and have done in time, but there might be something I miss.  Or maybe I am not thinking of it in the last few days, because I am transitioning all the work I am doing now, and anything that is on my mind for planning in the next month.  Not the two hour task I did 6 months ago, which won't need to be done until next year; if its written down and documented properly then I don't need to.  Or I can review all the documented items and remind myself of anything that is needed.  I've also found that going over all the documents with someone brings up other tasks that were slight, or may need to be added somewhere.  Sure there is a maintenance cost, but after 3 years I found that 10 minutes a week to review the documents was enough to keep them updated, and ones that needed drastic changes either were deprecated for new ones or we added a new section or document for the new way; if we had legacy items that still required the old way.  Even though I spent 4 days going over everything I knew about the systems, how we tested and the best ways to test the next generation, I also spent time reviewing all the departmental documents that I had written over time.

The Knowledge you leave behind as an artifact, is just as important as handing off the project you are on now, and often that work you leave documented somewhere will help everyone out more in the long run.  That is how I would rather be remembered.

dding QA to Scrum (one Company's view)

My current company has been running Scrum for a Development process but as this tends to be a wrapper for Development, at least in my view, its been tough to fit QA into this process.  We do have release iterations where there is a focus on making sure that the code is stable, and we focus specifically on testing and bug fixing, though we do tend to freeze late in the iteration so the final testing gets cut.  What's new about that?

So in trying to decide how we wanted to set up QA the thinking came that QA needs to be more in the Analysis camp and not so much in the Process camp.  This is kind of a switch, at least to those of us who have been squarely in the Process camp, as we have less documentation now to develop test cases and plan out mini-release cycles every Iteration; but is this really much of a change?  The way we are changing to is most of the Release Testing will get backloaded into the Release Iterations, with the preparation of Test Cases and Unit Testing being done ahead of time and increased during the iterations.

With each Iteration QA can analyze the new functionality being developed, discover the Test Cases needed as code is written, generate the Unit Tests necessary to fulfill the Test Cases; QA can pair up with the Developers to check their Unit Tests as well.  We tend to have two levels of Unit Tests, one which we do call Unit Tests that the Developers create and use to test their code and Black Box tests which QA creates and is one step up from the Unit Tests which can start to test the classes as a whole and start Path testing.  Did I mention we are doing a lot of Java and JUnit?  This allows both the Unit and Black Box tests to be run in our Nightly Test Suite, so we have continual coverage once we add a new test.  So we cover method and function testing at the low level, and the Functionality and Use/Test Cases at the high level, providing more coverage as the work progresses.

So if this is followed, we generate new cases each Iteration, adding to our Test Plans, which may or may not be run during pre-Release Iteration, and we increase our Automation.  Not much different really than being in the Process camp, except we are sticking with the Developers to find out what we need to do and test, rather than looking at documentation and understanding the Use and Business Cases.  If that Case information is needed we can get it from the Product Owner, or Product Director depending on your terminology, so the only change here is the information source, rather than written materials its going to be dynamic where we gather the data we need from the flows between the Developers and the rest of the Scrum Team.

Originally we thought this might be difficult, and a tough change, but when we spelled it out then its actually not much different than we do it now, its not the What that has changed but the How.  Of course none of this has been in practice for very long, so it will depend on how its handled over the coming Iterations, it would be interesting to see if it were not for the fact that I am leaving the company this month.

Flavor of the Month - Part 1 (Six Sigma)

The advantage of working at many start ups is the occasional exposure to new business processes, especially those that are considered and decided upon at the high level.  While there may be the need for education, and some additional hires, the process (whatever it is) will help the company streamline itself, or even make the company more competitive.  Or at least that is the thinking.

This is how Six Sigma was presented to us at one company, whose name will remain anonymous at this point since it is still around.  This company was a service provider using ASP and MSSQL along with apps written in both Java and C++ that utilized COM and COM+, depending on the various item.  So in the middle of the yearly managers meeting we learned that Six Sigma was the way to go.  A business process person was hired, books were bought and sent to all the managers, and we were encouraged to help bring this process into the company and help expand our productivity and effectiveness.  Were we really ready for any of this?

In a word - no.

The company (especially the Development Heads) was more concerned with getting new software and features out to customers, even with a huge backlog of issues that needed to be fixed in our production code the push was on to get the newest and sexiest feature out.  While the issues kept rising, and the bug reports noted the increase, the Development Managers ended up ignoring the Six Sigma and kept moving on the way they were.  We could have improved our code some, we actually did two bug fests when we had some massive system hemorrhages, and then moved on to improving our process; or done it in parallel.  But we didn't.

Instead we ended up reading the books, doing what we normally do then sit in the kitchen and look at each other saying things like "Did you make that coffee by Six Sigma?  It doesn't look very efficient to me."  Which was when we knew the process was gone...oh that and the fact the Business Process Person got promoted to the Operations group, and spent her time getting new offices and sites up and running.

Myopia or Familiarity

The other day while changing my son's diaper I noticed that I am following a process, every time I go to change the diaper I get my materials together and set them up in a certain way to make the process go quick.  He hates being placed still for too long, this also eliminates the time it takes when he is still asleep and I need to change the diaper without waking him up; I have this down to 5 minutes.  But then it happened, part of my process had changed to where I no longer was using the cloth to cover him in case of a liquid projection, and when it happened I was unprepared.  Luckily none on me, though I have been close, but the area around him and his clothes required changing.  Yet another bullet dodged.

When it comes to testing something on a consistent basis eventually I discover a sort of too familiar attitude crops up to the point where I am going over some areas quickly, and others not so quick.  In the same way as my process just becomes second nature, to where I don't think about it at all.  It's then I consider that I am either too familiar with the product, which does allow me to test it faster as I know the Test Cases by heart, or its that I have become myopic in my evolution of the Test Cases and pretty much just do the ones I know and maybe do a tinge of Exploratory Testing.  I could say that familiary breeds contempt, though I don't exactly hate the application or feel any sort of disdain for the testing I am doing, its just that I have spent so much time with it that I have come too close.  I no longer see the whole picture.  Being so focused that I see only the design, and the paths that the Test Cases have generated over time.

In a growing group this sort of work can be shared, fresh eyes can come in and look at the Test Plan and find a new way to do things, or enhance the Test Cases.  Perhaps adding more for that new piece of functionality that I considered covered by another Test Case; duplicate coverage is another thing best described in the future.  Or I can find myself in front of the monitor again, looking to test the application for the umpteenth time and try to pull myself back, in order to get back that freshness and see the whole picture once again.  Sometimes I find that I circle around and get back to the same close up view, and its hard to pull myself back to review all the Test Cases again and see if perhaps something needs to be added.

Reading the Cases backwards is one way I've found to add in some novelty to get myself thinking again about how I should go about testing once more; almost like starting over.  That's really what I try to do, even if its the thousandth time I have looked at the same Test Plan, I at least try to fool myself into thinking its the first time.

Just a review of where I started

Originally published in May 2006, I am moving this to my new home.  Enjoy it again or for the first time!

Well as a way to start things here, let me sort of recap my employment, and my foray's into Quality of all kinds over the years.

I pretty much ended up into this after Managing a couple of Support groups, Customer and Technical, and as the person who was the liason to the Engineering department I got exposed to Quality techniques in opening up ways to troubleshoot and report issues.  The more I got involved with the QA department the more I realized how interesting it was, the deconstruction of problems, and the generalized knowledge you gain by having to be familiar with all aspects of the software.  So I buckled down and started studying some programming and process techniques.

Landing a job as the first QA employee at a startup I worked close with the Engineering Team, who were mostly an offshoot of the group I worked with when I was in Support, ah yes the heady days of working with the evil Empire of NewsCorp.  This was the mid-90's when they were in their first move into the internet, and then realized (well the first time anyway) that they didn't get it and laid us all off.  By this time the QA group was 3 people including myself this only took 2 years.

So I took my experience of a few years as a tester, and as management experience pretty much translates across jobs, started at another place as a one man team who grew the team as I began formulating what I knew a group needed at a new place.  Again, as the lovely days of the DotBomb went on, we had fun and lots of beer lunches, then all got laid off.  So I went on to another place, this time we grew the QA group to about 6 people in a year, thinking it was not me I took some time to find a new place and worked as a contributor.  That work was more like a summer job, it lasted from June to September, but at least by the end of it I had been introduced to my future wife.  Moving on took another year as a Contributor and when that didn't work out went back to the start up phase again.  That job lasted 3 years, we built up a solid organization where QA was also the Release Team, as I had done some release work in some of the previous jobs I was given the task, and we at the highest had 8 people total in the group.

Now I am back to being a Contributor, tired of Management, but still enjoying the deconstruction and the learning as I am moving into more Automation and Unit Testing Frameworks.  Never went to school for programming, I am all self taught, but I've been lucky to work with talented people to bounce ideas off of, and its good to have some online communities to share experiences with as well.

Since I am caught up, with just a little background, I'll go over many of the lessons I have learned and am learning...if its of any use to you then good.  If not, I will at least try to make it an interesting read.  Judge for yourself if its worth it.