Tools I've Used - AutoIt

Over the years I have used many tools, and tried many more, for testing.  One of them that I have grown with, and grown to like, is simple and free with a pretty good fan base.  AutoIt. 

I originally found it when I needed to automate a Customer Test, this was for a simple app that just found documents on a Customer's harddrive and uploaded them to a server; which the Customer could then distribute.  But since the issue dealt with doing this repeatedly I started looking at this tool to handle it, you can make your script based on screen positioning or by the use of tabs in an application.  I opted for tabs since I don't like screen positioning.  It worked well and did a fairly good job, though I ended up using Perl to generate the file upload list for the application, and used AutoIt to run the Perl Script, then start the application which made things easier!  Then with our actual Web Application I used it to make small scripts that would do Customer functions on the web site, again using tabs since I liked that - which as usual needed maintenance when the Developers changed the pages.  Still it worked well, and as the team grew others used it as part of their testing, one member of my staff was using it to generate Customers by having it run on the Customer Creation pages over and over - which made it easier to generate needed accounts the same each time; and check for changes in the Customer Admin pages.  It really fit a small niche.

For a free tool I have gotten some good use from, its been great.

As was noted when this was originally posted - the right tool is the smallest tool.

Pros:

• Scripting Language is Basic-like, so its fairly simple and easy to read
• Can do Tab (keystroke) movement or Mouse positioning
• Has a lot of add-ons with an active development community
• A good community of support, both on the Web and on Yahoo
• Scripts can be converted to EXE's

Cons:

• Windows based, if you want multiplatform its not that useful
• Syntax was sometimes not that clear, but this seems to be solved in the latest version
• Web Scripts can work differently on Multiple Browser types, I never was able to make a generic script (but that may just be me)

Finding the Right Help

Since there seems to be much being made about getting employees, on this site and in conversations I have had with people recently, I'll add my two cents about this.  I'll try to put it in some kind of order, but its going to be difficult, so bear with me if I go out of order.  Since I usually started as the first person in QA it was easy for me to grow the group as I understood completely the space and the work, so I had a very good idea about what skills mattered.

• Getting the people in.  Referrals, Referrals, Referrals.

I've never found an easy way to get people in, but the best first option is to use your local resources, co-workers who might have worked with some brilliant QA Engineer in the past or knows someone that may fit into the company really well.  If you have been lucky enough to hire some QA staff already ask them if they know people who want to come work with your company, its also a way to check enthusiasm since people who WANT to be there will want to bring in their friends.  Most companies offer referral bonuses, or some sort of compensation, for referring people.  Also make sure you know where people have worked, if you get an applicant who worked with someone you have on staff, or is a colleague, see what they remember - it may be worth pursuing or not.  But take in mind that its one persons opinion.

Although the only other option to this is posting the job on the company site, internet job site or to a recruiter and they all have their drawbacks.  When you do it this way, clear time on your schedule to go over resumes, while its been nice to have HR go over resumes, unless you have an HR person who REALLY understands your needs you will either get resumes that don't fit or never see ones that might.  In all these situations there are some options.

1. HR - meet with the HR person or Rep who will be handling the job, make sure they understand the needs and what the job description is, sometimes I've had Reps who knew Development but not Testing.  You get some odd resumes that way.
2. Recruiters - they can be annoying, and we've all heard the negative stories, but if they are a good company and you get a good relationship with the company or its Rep they can be good.  I always send back my feedback to the Rep after a meeting with a candidate they send, telling them the good and bad and why I reject or accept them.  It's helped to give me better candidates, and if a company doesn't seem to get it I drop them, its not worth my time to get bad candidates in.  I also go over the experience on the resume with the person, that way I know if the resume has been doctored.
3. Company Site Posting - usually a small deluge when the posting goes up, that eventually fades away.  This I will usually go through with the HR Rep, because some resumes give you an idea about what the persons communication skills are like and some may have the right skills but not written in a way someone with experience in QA might recognize.
4. Internet Job Site - this will be a floodgate opening when it goes up, you may find responses from recruiters on behalf of applicants in this flood as well, keep it all in mind.  Again, go over the resumes with the HR rep, but take time for this, as it will not be quick.
5. Oh yeah, SAVE those resumes with comments too.  I've had cases where I interviewed one person, then in another year got another resume and the name seemed familiar so I checked my files and we had rejected the person as a bad fit and not the right skills.  In a year I sincerely doubted the person would change, so we did not bring them in for another set of interviews.

• The initial interview - phone or otherwise

I would always check the past few jobs and ask them to describe in more detail what the job entailed, trying to get an idea of what the person really did on the projects in the past and what their feeling was on the work.  Wording on the resume may or may not match what the person really did, and I like to find that out in the beginning, but I also want to make sure the person can communicate effectively.  Since QA is a lot of face to face discussions with Developers and others the person you hire must be able to communicate with your colleagues, and without much difficulty.  I'm not picking on people with second languages at all, but if in speaking and writing you cannot understand what the person is talking about, especially in regards to a defect report, then no one else will either and it will make work difficult for others.  During this interview I asked most of the questions as well, as I am trying to decide whether this person can come in and do the job, if the potential is not there I will not bother, there will be time for candidate questions in the face-to-face.

• The face-to-face

This is when I go into more detail on the job and really try to grill the candidate on experience, skills and test methodology.  Depending on the level of the position I may ask more technical questions or just try to find out why the person wants to get into testing for a Junior Position.  I also ask for examples, if someone knows how to write SQL I ask them to tell me some commands, how to select from a table, two tables, multiple tables, do a sort - just to make sure they have a grasp of things.  There is also The Example.  Taking an example of something that is under test, I give some vague description as just ask what they would do.  See if it matches what we do, this gives an idea of mind fit.  Then there is the Technical Example, something that really challenges them to use their skills, this can be anything from how to test a product to my latest favorite, how do you test a bank of elevators in a building?

• Group Melding.  How is the fit?

During the interview process I have some members of the QA staff interview the candidate, this is to determine how everyone will get along, sure we may spend time with the candidate during the interview process and a couple of days after start date, but the day to day interaction will be with the QA staff.  If there are going to be issues, find them out now, before you have a serious issue on your hands.  Since there is only so much time in the day we broke out the interviews to focus on each persons special skill areas, one person may ask the more technical questions while another may ask the more process related items, use the strengths of your staff to your advantage.
Also I like to ask the same question multiple times, with different people, there should be different answers unless the person has been coached, in which case a warning should go up.  It's happened to me in the past.  Also figure out a way to end things early, if the process is going then a large warning flag goes up, you may decide to cut the candidate loose, plan for this.

• Contractors

This is always tricky, you want someone in for a short time, with only specific skills but do you really need to do the whole interview process?  In a way I say yes, because this person will be there day to day, and if its long term you will get to know them like any other co-worker, in some cases the contract can become permanent so plan for permanence now; it may save you headaches later.  But as with regular applicants I send my feedback immediately to the company during the interview process, while it is still fresh in my mind, in many cases the company has responded by sending Contractors better meeting my skills.  There have been a few I have hired in the past, some I took on full time, but I had that option in mind on the interview and made sure the fit was good; some have been the best employees I ever had.

---

There ya have it, and while this has worked for me over the years, use parts on your own and in your own environment, my expectations and needs were probably different than yours.  A lot of this is planning and managing the risk, items we should be well familiar with in QA, these are skills you can use in any endeavor, not just the latest project.

Sharing the Environment

Years ago,  my first QA job actually, we were sparse on resources and had one lab for everyone.  Yup.  A server for all, and a database to share.  This was for a start-up that had limited resources and money, QA was brought in after Development really got under way so it was a lot of catch-up.  Originally, until we got the middle-ware stable the Development Leads did the actual releases as well, with input from me, so if there were issues or something missed they could discern the issues easier.  Still it was all we could do, so we needed to work out the risks.

Here is how we ended up doing it.

• Web Site, seperated out the main home directories, easy enough to do.
• Web Server, this was the tricky one, because it was a compiled executable and when it was updated mostly they were library files.  Really we just promoted the new files, even if they were the same as the existing ones, not something I would recommend but it worked at the time.
• Database, sort of tricky but we just used different Oracle instances and that allowed us to test in isolation.
• Other middleware, which was the bulk of the code, we handled this by reinstalling.

Was it full of risk?  Sure.  But we also had weekly or bi-weekly releases, once a package was out we had another coming in.  This did work for us for a year and a half, it was all for one Customer so in that we even had an easier time.  That Customer went on to be a fairly large ISP in the UK so we were doing something right on the back end, there were few problems that arose after deployment.  So sharing an environment can be done,  but you need to be aware of EVERY potential risk, especially when you are testing in a dev environment that already has the tweaks made for fixes - and may not be included in the package.

To me the whole idea of Risk Management is what needs to be done for every release or build coming into Test, if you don't know your environment enough to see the Risks then you can set yourself up for trouble.  Its worth time to ask "What if..?" questions all the time, some Risks you won't know until you have experience to ask the questions, and some will come from doing the releases and seeing the code work in the field.  The goal is to minimize those issues that arise from the field, or to eliminate them entirely (a good pipe dream).

Would I do testing like this again?  No.  I've learned that having your own environment that is yours and yours alone to test with is the best way to mitigate everything.  I know it can be done to be in the same environment as Development, I just would never recommend it.