Monday, September 5, 2016

Adding an external file to JIRA

As a part of my companies personal project time, a yearly occurrance they call Off The Grid, I was part of a project to add external files to a JIRA ticket.  JIRA is used to track EVERYTHING in the company.

This is the python script I hobbled together.

#!/usr/bin/env python

from stat import S_ISREG, ST_CTIME, ST_MODE
import os,sys,re,requests
from requests.auth import HTTPBasicAuth
import urllib3
def openFile(location,key,filetype,fileKey):
    Tests that the file location exists and searches through each file for a specific keyword
    :param location:
    :param key:
    :param filetype:
    :param fileKey
    :return: outputFile:
    outputFile = filetype + "output.txt"
    except OSError:
    f = open(outputFile,'w')
    print >> f, "Looking in %s" % location
    for dirpath, dirnames, files in os.walk(location):
        if not files:
            print dirpath, 'is empty'
    entriesf = (os.path.join(location, fn) for fn in os.listdir(location))
    entries = ((os.stat(path), path) for path in entriesf)
    entries = ((stat[ST_CTIME], path)
               for stat, path in entries if S_ISREG(stat[ST_MODE]))
    # for cdate, path in sorted(entries):
    #    print time.ctime(cdate), os.path.basename(path)
    check_file = re.compile(fileKey)
    for file in entriesf:
        if os.path.isfile(file): # and
            for i,line in enumerate(open(file)):
                #if re.finditer(key,line):
                match =,line)
                if match:
                    print >> f, "Found in %s \n %s" % (file, line)
                    print >> f, "No entries that matched keyword %s" % fileKey
            print >> f, "No files that matched %s" % fileKey
    return outputFile

def addSqlTraceToJira(location,ticket):
    Add a SQL Trace file to the Jira Ticket
    :param location:
    :param ticket:
    entriesf = (os.path.join(location, fn) for fn in os.listdir(location))
    check_file = re.compile('trc$')
    check_ticket = re.compile(ticket)
    for file in entriesf:
        if os.path.isfile(file) and and
            print "No files found with %s" % ticket
def attachFileToJira(logfile,ticket):
    Attach the output file to the Jira ticket
    :param logfile:
    :param ticket:
    url = "https://jira-test.fakedomain.comnet/rest/api/2/issue/%s/attachments" % ticket
    headers = {'X-Atlassian-Token':'no-check'}
    files = {'file': open(logfile, 'r')}
    auth = HTTPBasicAuth('username', 'password')
    # print "Sending %s to ticket %s" % (logfile,url)
    r =,headers=headers,files=files,auth=auth,verify=False)
    print 'Jira file attach sent me a %s with \n %s' % (r.status_code,r.text)

def addCommentToJira(keyword,ticket):
    This attaches a comment about the search parameter that was added to the ticket
    :param keyword:
    :param tickiet:
    url = "https://jira-test.fakedomain.comnet/rest/api/2/issue/%s/comment" % ticket
    headers = {'X-Atlassian-Token': 'no-check'}
    auth = HTTPBasicAuth('username', 'password')
    comment = "Just added an attached log file about the results for the %s search." % keyword
    json = {"body": comment}
    r =,headers=headers,json=json,auth=auth,verify=False)
    print 'Jira adding a comment sent me a %s with \n %s' % (r.status_code, r.text)

def main():
    if len(sys.argv) < 5:
        from os.path import basename
        print ('Usage: %s location key type ticket' % basename(__file__))
        # fileLocation
        LOCATION = sys.argv[1]
        # searchKey
        SEARCHKEY = sys.argv[2]
        # type
        FILETYPE = sys.argv[3]
        # Jira Ticket
        TICKET = sys.argv[4]
        # File Keyword
        FILEKEY = sys.argv[5]
        #TIMESTART = sys.argv[3]
        #TIMEEND = sys.argv[4]
    # attachFileToJira(logfile,TICKET)
    # addCommentToJira(SEARCHKEY,TICKET)
if __name__ == '__main__':

I claim some responsibility for it, mostly by hacking together a lot of python and doing a lot of TDD as I built out the functions.

Friday, July 15, 2016

Robot Framework and RESTful APIs

Been awhile.  Yup, it has.  Just a short one this time, more to get something down that vexed me for a little bit.

Dealing with a RESTful API that returned JSON in the body to my Robot Framework test caused me no end of grief.  Since the response always came back with the following:
        "account_organizations": [
Originally I thought, it's just a JSON response, I can deal with it.

Oh no.

Not at all.

Turns out what I was getting ended up being byte order markers that were messing up the JSON compare I was trying to do.  Knowing what I was to get back was one thing, comparing that with the body of the response was another thing since the byte order markers did not behave like a regular list as I thought it was.  This turned out to be a dict, since Robot Framework basically is Python we have to be Pythonic here.

So, treating that as a dict we get the 3rd index from the list:
THAT now pulls the right values for comparison.

I also run this against a make unicode Python library, just in case.  So in Robot I use the following to cover it all:

${l_response_body} Get Response Body
${l_decoded_body} = make_unicode        ${l_response_body[3:]}

So all is good with the world, for now.

Monday, October 6, 2014

Testing Beliefs

This comes off of Pete Whalen's wonderful post on Testing and Wars of Religion.

Oftentimes we cling to our faiths in processes and procedures, too much in some cases.  It's hard to let go of something that you have worked hard to understand and struggled with for something new.  Change is hard.  Very hard.  But that is the point.  At some point beliefs and processes no longer work the same way and a new way to look at things is required.  Why?  Because just as work and processes change, new tools, languages and methods of achieving goals come and enter the environments we work in so to do these new things often challenge beliefs and require adjustment in how we look at things, or a new vision of the world.

Change is hard, but if it was easy then in many cases it would not be worth it.  I always look at them as educational opportunities, though even I challenge some of the changes that come up, its normal and once I get over my initial reluctance I can take a wide-view of things and move on.

Short, sweet and to the point.  Happy Monday!

Friday, September 19, 2014

The Power of PowerShell pipelining

Every now and again I come back to PowerShell, it's useful as a tool, simple, clean and while I can sometimes say its easy to use, many times I smack my head on the desk.  Working in a Windows environment PowerShell scripts make doing many things easier, though it usually takes me some time to work them out.

For example, I am trying to output some results from failures on running SSIS Tasks, but I only really need to know certain statuses, or in most cases just the failures.  Focusing on the failures there can be a lot of them, if something fails at the beginning EVERYTHING runs and can easily fill up the log.  So while a command like:

    $Results = &'C:\Program Files\Microsoft SQL Server\110\DTS\Binn\DTExec.exe' /Project $Path /Package "$File.dtsx"

Can give me a huge result set, do I really need to display a couple of hundred lines of errors to the console, or to the Test Driver?  No.

I wanted to only pull out the error details, I am skipping over the step where I know there are errors here (just to make it easy) and what I want are the details to notify someone that "Hey, something went wrong here!  Here are some examples, want to know more, log in to the box and do some research!!"

So at first I step through everything (that's how I understand where things are working) and first came up with this:

foreach ($_ in $Results)
if ($_ -match "Description")
$exec_errors += $_

Now that is way too long.  A lot of white space, plus do I really need to step through that much?  No I don't, thankfully PowerShell makes this easy, since $Results is an array, I can pipeline it and step through it to match only what I want:

    $Results|?{ $_ -match "Description" }

This gets me only the data I want to display, the rest of what you get for the error messages is meaningless to convey the details of what went wrong.  I pipeline the $Results array into ? which is shorthand for the Where-Object and within the curly braces put what it is I was looking before in my if statement, under my foreach.  So now instead of a 3 or so line statement (not quite counting the curly braces there) I have 1 line!  Yes, there can be only 1.

Still I need someplace to put those results:

    $exec_errors = $Results|?{ $_ -match "Description" }

Now, I have an array of just the details I want and using $exec_errors I can output only what I need to communicate:

Write-Output "Errors in execution: "
if ($exec_errors.length -ge 2) {
Write-Output "$exec_errors[0] `n $exec_errors.length - 1"
} else {
Write-Output $exec_errors

I will have to see about making a one-liner of the rest later on.

Tuesday, September 9, 2014

SSIS Task Variables

In some current Testing I am doing work with SSIS.  One task imports data into a database that we baseline for future Test Cases, but to get the data in I need to modify some of the dates.  With SSIS there is an option to adjust data using Derived Columns, with that it's possible to adjust data using Task Scoped Variables; I'll have to find a reference but using project scoped variables did not work initially.

Basically the variables were created this way:

  1. Gave a descriptive name (need to have those so you know what to look for later!)
  2. Scope, was selected to be the Task being worked on, which was easy since the project only had one task for data loading.
  3. Data Type, since it was for Dates I made this DateTime
  4. Value, just hit enter/return here and it took the current date and time
  5. Expression, this is the real work here
    1. For Now this was just left as the default - GetDate()
    2. For a Year I made an expression - DATEADD("Year",1,(GetDate()))
    3. Same adjustments for Adding Minute or Month, just adjust Year
When using these, in the Derived Columns select the Expression column and in the upper right panel there is a tree for Variables and Parameters.  Select the User: and this will update the value for that column when the Derived Columns task runs.

Learning about DateAdd came from the Microsoft Site

Monday, August 11, 2014

Why History really relates to Testing

I am a History buff.  Of course I also went back to school and did a major in it, getting my Bachelor's from Boston University while still working in Software.  I was going to do a CS degree but honestly, much as I like it, programming during the day and for homework didn't really suit me.  After some discussion with my wife I ended up changing to History, since it was something I loved, I think it was a great decision since I got exposed to subjects and topics I never would have thought to take on my own.  From those I learned a lot about how to analyze situations and topics, do research and apply that to the question or topic at hand; something I do a lot of while Testing.

Testing IS History.  Regression Tests are a way of compounding a Historical Record of Fact that says, these things happened and if we want to prevent them we can do THIS or THAT.  Something that in History we often table about, the What If format is not only a plot line from Fiction novels but sometimes is a way of viewing an event from a certain perspective.  Of course as our understanding of events increases so does the Historical Record and our understanding of it, I find that Regression Tests are the same way.  The more I understand a certain issue, or feature, the more I can improve my test case and get the confidence level of the product higher.

When scheduling I rely on my knowledge of prior events, but being the person I am I document everything as I go along.  This gives me a record to use and base my future estimates on rather than giving my "gut feel" for a specific project or set of features.  When I did releases we often put times for specific steps to take, this way in long releases we were able to say when certain people were needed to help, or to even frame the maintenance window needed.

These are just two examples, but very deep and broad ones.

How do you use and document your prior knowledge on events and tests?  Do you document your test code, like a historical record, so others who come later can figure out what you were doing?

Sunday, April 27, 2014

The Person Month Revisited

I was walking down the street and noticed a few people were picking up trash in the road, thought to myself "hey, I bet with one or two more they get done real quick!"

Of course, then my thoughts turned to Software and then in some weird connection I jumped into the Man Month Myth, renamed the Person Month to be more gender neutral, and it occurred to me that yes it still happens and yes in some circumstance it may work.  Though those are slight, and slim.  It of course depends.

Let's look at some examples:

Street Cleaners: the more the merrier!  It doesn't take much domain knowledge to pick up trash on a street, or rake leaves from a yard.  All you need is some coordination and a slight understanding of tools.  Of course, coordination with those tools will make the job faster, a group of 4 or 5 teens could do well, if those are the only resources we have available.  It's like after a party at a house, or a bbq, the more people taking little tasks and doing them, the quicker the whole task get's done.  I don't want to be demeaning but at it's core there is not a lot of skill involved in cleaning up, using a broom is fairly simple as is carrying a bag to put trash in - at this point the main skill is knowing what is trash.  So at this point the skill level is not high, and the tool usage ability is low.

Too many cooks spoil the soup: yes, that adage does have some relevance.  The reason you want one cook is because he understands how this dish should taste, everyone is slightly different and the more you have the more additions you get until the soup is a mess.  Also, hardly tasty.  Not everyone has the same taste or background to know what a particular dish should be, the best restaurants do well because every dish they have has consistency but this is due to long hours of training and knowledge transfer.  So we have a medium skill level and a tool usage ability is getting higher, but now we have an added knowledge transfer quotient to add in as there is a lot to know about making good soup.

Software: it depends on the domain knowledge and skill set.  The high up the domain tree you go the more it requires a specific fit.  Sure you could add one or two more Engineers to your project BUT those Engineers had better be really familiar with the project, specs, and domain or you do more harm than good.  The specs have been discussed, meetings have been held to discuss code and flow and design, there may be a lot of documents, depending on your environment but there is also a lot of knowledge transfer.  There is a high skill level, a high usage requirement for tools and a lot of knowledge transfer.

So, can you add more people and get more done?  Yes, but only in specific situations.  If people have the skills, tool and domain knowledge then adding them in is not an impediment, but this is rarely the case.  Although if you have an environment with lots of switching between teams then some of this may be in place but you have to know.  If you don't analyze the situation first you will just make a mess of it.  Like the soup.