Friday, February 25, 2022

Rubber Duck Debugging

 



Talk through the problem

Have you ever been stuck on something?

As you are talking about it or telling someone about it, suddenly the solution arrives or you get that eureka moment where you stop and realize you have the answer.  Even when writing out the issue you are working on to someone, then the answer comes as you are writing it out.  This happens to me a lot.  Sometimes just looking for something at work, or I am writing an email and send it, then I realize I needed to try some step. BOOM! The solution presents itself.

Talking it out

I used to just call it talking things out and leave it at that, then I find out that there is a name for this.


Rubber Duck Debugging!


I've never spent my time talking to inanimate objects, maybe myself but while I may be an object I am not inanimate, so while talking to myself or others I do get in this situation. At times its kind of embarrassing to send an email or message, then immediately follow it up with a never mind.


That sort of makes me feel like the old SNL character Roseanne Roseannadanna.




Never get stuck alone thinking

To me the whole point of it is to not get stuck on a problem and just sit there thinking and thinking and thinking on it.  Getting stuck on something small that distracts me from moving onto the solution.


Talk it out, step away, or speak to a duck if you have one. You never know, the answer may just suddenly appear.

Sunday, December 19, 2021

Recipes for Success

There are multiple recipes for success in technology, and like chefs, they vary from place to place and person to person.

What works for one person won't for another, without some adjustments.

In many ways the Local Sourcing movement changed the ways in which cuisine was made, gone in some places were the Special of the Day (to go on a tangent that was often a way for kitchen's to get rid of stock that was getting old) to be replaced with a full seasonal menu.  Just like companies that use different technologies in different ways so the environment is different in what a Developer, Tester, or Manager needs to know in order to stay current or be ready to make the next step.

Recipes handed down in a family are based on a point in time where ingredients were local and available, some at only certain times of year so they were "in Fall we had" or "on this holiday my grandmother made...".  When shipping became more immediate and being able to send produce and meats long distances with a relative amount of freshness so more ingredients became available in wider times.  That removed the specialness of certain things that could only be found at certain times of the year, and how foodies looked only for certain ingredients at certain times because only THEN were they tasty.

Technology evolves but in companies with established practices there are certain things that remain in a specific way with people having to learn how to use that version of a software with another.  Change comes slow, and with new versions of Java, Python, and dare I say Perl (though who even uses that anymore...) at times you need to adjust how software is built and when bringing build recipes current there are adjustments that need to be made.  Some easy, some not.  Some complex, some not.  In the end its the result you want, and like a tasty dish, you want the build to be complete, whole, and workable.

Correlations are fun, and there is more to be done with cooking (a fave of mine) and software, but more on that later.  I have a special dish I need to get ready for....

Thursday, December 16, 2021

Keep Moving Forward

 It's hard to keep current, especially with all the options out there, and stay on a single trajectory to both improve oneself and keep progress going on a career path.  It's no longer a single line that moves one up a traditional line of promotions to some sort of mythical Manager or VP or CEO position but often a zig zag of moving between hard and soft skills or between languages and tools that often don't lead to one place.

It's like a continually evolving buffet where tools, languages, processes, and management styles continually update or are reviewed, and reviewed again.  Deciding on what trajectory to take is difficult, especially if as usually happens, one tends to change jobs every couple of years.  Anecdotally people change jobs every 3 to 5 years, the time of the company man has been long gone.  Very few stay on at a company for decades working for one place, many have worked for company after company where they only thing they take with them are skills, knowledge, and experiences.

So what to do?  I take it in three steps on what I want to learn:

  1. What am I doing now that I need to improve on?
  2. What am I doing next that I need to know more about?
  3. What is coming up that I need to understand?
This makes the choices fairly simple for me, as languages evolve and there are those I am using constantly then I want to keep up and improve my coding ability.  Being able to make code more concise, readable, or find new ways to be DRY (Don't Repeat Yourself) are what I look to be able to improve at the moment.  Keep current skills fresh and improved, move up to the next level, and that keeps my toolbox fresh and up to date.

Anything that may be coming up, some new tool, idea, or something the company is using is always a good option.  It's hard to implement things, or use things, you don't understand.  Becoming familiar with the next thing coming is a good way to be prepared, don't wait until it comes.  Take the time to figure out what it is, learn it, then plan for it.  Knowing about the next project makes planning much easier.

Along the same lines there are new projects that may be coming, in companies change doesn't happen in a vacuum.  People know, they are aware and plan for change.  As the Scout motto says - Be Prepared.  Which also works well for personal advancement, being ready for the next change or the next project is the best way to succeed and keep the time working on it down and being ready to jump in and start not only makes one look better, but also starts you on the path to success.

Keep moving forward, but also keep preparing yourself.

Thursday, December 9, 2021

Accessibility Testing in Automation

 One of the new items I have been adding into the Automation Frameworks that I support, is the Axe Tool.  This does a check against page source to determine if the HTML structure is compatible with various assistive technologies.  This is something that can be added into a browser, and allows manual testing, but why do things manually if you can automate?

Automate or die!

That's been the mantra, well not die, but it's got to be serious.  After all why do things manually if you don't have it?  So to start we create a UI test to Login and go to a home page like the following (forgoing most of the basic Class setup, which I am assuming you, dear reader, know how to do - if not, just read on and learn what you can!)

@Test(description = "Verify Login Accessibility")
public void testLoginPageAccessibility() {
logger.info("This is to test the a11y functionality with the AXE library.");
LoginPage loginPage = new LoginPage(driver);
try {
loginPage.goToPage(new URL(getRunTimeConfig().getUrl()));
User user = getUser(userId);
logger.info("Checking page for " + user.getUsername());
loginPage.login(user.getUsername(), password);
logger.info("Getting the Accessibility Report for the Landing page.");
getAxeReport(getCurrentMethodName());

The Current Method Name is the loginPage method, that is built upon the WebDriver GET so what is being sent to getAxeReport() is the current page being checked.

So let's look at how it generates the Report:

public void getAxeReport(String testMethodName) {

logger.info("Checking for accessibility on page: " + driver.getCurrentUrl() + " for " + driver.getTitle());
org.json.JSONObject responseJson = new AXE.Builder(driver, scriptUrl).setTimeout(180).analyze();
org.json.JSONArray violations = responseJson.getJSONArray("violations");
String pageName = driver.getTitle().trim().replaceAll(" +", "");
String reportName = OUT_DIR + testMethodName + "/" + pageName + ".json";
String rawViolations = OUT_DIR + testMethodName + "/" + "raw" + pageName + ".json";
try {
if (new File(OUT_DIR + testMethodName).mkdirs()) {
try (Writer file = new OutputStreamWriter(new FileOutputStream(new File(reportName)),
StandardCharsets.UTF_8)) {
file.write(cleanUpAxeJson(violations.toString()));
logger.info("Successfully wrote Axe Report to File...");
}
} else {
logger.error("failed to create directory");
}
if (violations.length() == 0) {
Assert.assertTrue(true, "No violations found");
} else {
logger.debug("Found violations, generating raw report.");
AXE.writeResults(rawViolations, responseJson);
}
} catch (FileNotFoundException e) {
logger.debug("File Not Found exception: " + e);
} catch (IOException e) {
logger.debug("IO Exception: " + e);
}

}

What's happening is the page URL is being obtained, and using the AXE JavaScript library against the source of the page, collect the results into a file.  From the results most of what we care about are the violations, and as this is all JSON, we just need to parse the JSON tree and save the parts that tell us what is wrong with the page.

Results we save locally so the Report can be stylized, or saved and reviewed later by people who know what violations are really a problem.

Easy peasy.

Wednesday, December 1, 2021

Looking for the next BIG THING!!

There is always the cool item on the horizon.  The next tool that YOU MUST HAVE!!  Something that your Team cannot do without, or that cool presentation at the conference that took you by the shirt and shook you until you could do nothing else but think about IT!!

It's something all too familiar to many of us.  We want to improve, we want to be current, no one wants to be left behind or fail at being competitive.  A lot of this is psychological, we're human and in some ways the tools, with savvy marketing behind them, know how to get inside your head and get you thinking about them.  It's not a bad thing, but falling for it without knowing is.  Don't just jump into something without checking it out first, spend time looking at those tools and seeing if they will work.  Talk to co-workers, do a POC, and make sure the fit is right.

A warm glove is just as good when made out of different materials, but if you need to use your fingers, a mitten won't work. Check the fit, ensure that what you innovate or enhance meets the needs of the Teams that will use it, and that the Teams have the background to readily accept the change.  Nothing gets a nose out of joint than a tool you are forced to use that doesn't meet all the needs of a Team.

When looking to make improvements I look at the following questions to start.

  • Is this a programming language the Team already knows?
  • Does the interface merge with the existing without a lot of retraining?
  • Can you slip this into the builds and use it without people noticing?  The best updates are those invisible to the Users, they suddenly find their existing functionality works, and they have new options.  WIN!
  • Does it fit the culture?  The worst changes require significant retraining.

When working with people, they have emotions, feelings, and wants same as you.  Ensure you meet them and people will accept what you offer, when you give them junk, expect complaints.

Monday, September 3, 2018

Building out Tomcat and MySQL with Docker

We have a Test Queuing and Archive server that runs on Tomcat, with the data stored in MySQL.  This currently runs on vm's in our lab, but as part of the process to move this to AWS I moved the manual setup for these to Docker images.  This gives us a few advantages:

  1.  A long, multi-day wiki page install is distilled down into one docker image that builds in 5 minutes
  2. We get a repeatable environment usable whenever we need to deploy, without worrying about underlying changes of configuration files
  3. Being able to build a local Development version of the environment that can be brought up and used in minutes, rather than move to the Development vm and run some install scripts for an environment we barely use.  Updates to this system are rare.
Because I love automation I added some local shell scripts to check for running Docker images, containers, and added cleanup for previous builds.  In testing I was building these over and over and found I had at one point 5 GB of previous build images that I don't need.  The scripts also handle stopping the Containers so I don't have collisions in case I forget to check if the Containers are running, sometimes I don't check my laptop after the weekend to see what I might have left running.  I know, that's a bad habit, but this way I don't need to worry.

For my Local Development environment I know what IP I am going to start my Containers on, so I added an entry in my HOST file to be able to use a named instance to the Container when it comes up.  I like that better than worrying about trying to connect to an IP address.

As part of adding this into Jenkins I added parameters to the shell scripts to build for the environments I need, a Local Development environment needs MySQL the Production environment has an RDS DB running, so all I need for that is Tomcat.  This way all the configuration is set in the Dockerfile and I just build for the environment I want.

#!/bin/bash 
# Build script for Local Tomcat usable on local environments
BUILD='LOCAL'

show_help()
{
cat << EOF
    usage:
        $0 -b local     for local Thunder and MySQL Docker  (DEFAULT)
        $0 -b prod      for remote Thunder Docker image to connect to AWS RDS
EOF
}

while getopts "hb:" opt; do
    case $opt in
        h)
            show_help
            exit 0
        ;;
        b)
            BUILD=$OPTARG
        ;;
    esac
done

# WAR file and web server files
echo "We should always want to do a clean then a new build"
wget -O www/scripts/jquery.js code.jquery.com/jquery-1.11.1.js
cp www/index.html.template.templ www/index.html
./gradlew clean build
if [ -e build/libs/my.war ]then
    echo "my.war was built correctly"
else
    echo "my.war was not built, some kind of problem?"
    exit 1
fi

# Stop the existing images, if they are running
LOCAL=`docker ps -q --filter=ancestor=local`
PROD=`docker ps -q --filter=ancestor=prod`
MYSQL=`docker ps -q --filter=ancestor=mysql`
if [ ! -z "$LOCAL" ]; then
    echo "Stopping running Local instance"
    docker stop local
fi
if [ ! -z "$PROD" ]; then
    echo "Stopping running Prod instance"
    docker stop prod
fi
if [ ! -z "$MYSQL" ]; then
    echo "Stopping MySQL instance"
    docker stop mysql
fi
# Let's just be cleanly and remove all those old dangly images
docker system prune -f

# Create a Docker image
if [ "${BUILD}" == 'LOCAL' ]then
    echo -e "Running a $BUILD build"
    # Build all for local but only need Tomcat for Prod
    docker build -t mysql -f deployment/docker/local/mysql/Dockerfile .
    docker build -t local -f deployment/docker/local/tomcat/Dockerfile .
else
    echo -e "Running a $BUILD build"
    # Prod version should be prod name, don't need a database
    docker build -t prod -f deployment/docker/prod/tomcat/Dockerfile .
fi
# Cleanup the files we only need for the war file
rm www/index.html
rm www/scripts/jquery.js
For Tomcat I have some configuration to do with the context and server files, that will vary for environment, but what I especially needed was a way in Docker to build up a self-signed certificate without manual intervention.  After some searching around and trial/error I was able to come up with the commands that I needed to build that.

FROM tomcat:7.0-jre8

# Install useful toolsRUN apt-get update && \
    apt-get -y upgrade && \
    apt-get install -y wget vim curl procps tar libssl-dev --fix-missing --no-install-recommends


# Configure Tomcat Container
COPY build/libs/my.war /usr/local/tomcat/webapps/
COPY deployment/docker/local/resources/local-server.xml /usr/local/tomcat/conf/server.xml
COPY deployment/docker/local/resources/local-context.xml /usr/local/tomcat/conf/context.xml
COPY deployment/docker/local/resources/local-environment.xml /usr/local/tomcat/conf/environment.xml
COPY deployment/docker/local/resources/local-testserver.xml /usr/local/tomcat/conf/testserver.xml

# MySQL Connector
WORKDIR /usr/local/tomcat/lib/
RUN curl -L -o mysql-connector-java-5.1.46.tar.gz https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
# Add in untar of the mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar
RUN tar -xf mysql-connector-java-5.1.46.tar.gz mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar
RUN cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar .

# Configure Tomcat User and GroupENV RUN_USER    tomcat
ENV RUN_GROUP   tomcat
RUN groupadd -r ${RUN_GROUP} && useradd -g ${RUN_GROUP} -d ${CATALINA_HOME} -s /bin/bash ${RUN_USER} && \
    chown -R ${RUN_USER}:${RUN_USER} $CATALINA_HOMERUN chown -R ${RUN_USER}.${RUN_USER} /data/RUN chown -R ${RUN_USER}.${RUN_USER} /usr/local/resource-manager/
# Setup SSL for Tomcat
WORKDIR /usr/local/tomcat/conf/RUN $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA -storepass changeit -keypass changeit -noprompt -dname "CN=local.test.server, OU=Central, O=MYCOMP, L=BOSTON, S=Massachusetts, C=US" -keystore "/usr/local/tomcat/conf/.keystore"
RUN openssl req -newkey rsa:2048 -x509 -subj "/C=US/ST=Massachusetts/L=BOSTON/O=MYCOMP/CN=local.test.server" -keyout cakey.pem -out cacert.pem -passout pass:mypass
RUN openssl pkcs12 -export -in cacert.pem -inkey cakey.pem -out identity.p12 -name "local" -passin pass:mypass -password pass:mypass
RUN keytool -importkeystore -destkeystore identity.jks -deststorepass mypass -srckeystore identity.p12 -srcstoretype PKCS12 -srcstorepass mypass
RUN keytool -import -file cacert.pem -keystore trust.jks -storepass mypass -noprompt

WORKDIR /usr/local/tomcat/
USER tomcat

For the Production build I need an actual certificate, but since that environment is not fully ready I don't need to import that yet.

There is a shell script that gets the Local Development environment running, using Docker-Compose I am able to get the environment I need up and running.  Again the shell script has a path for Local and Production environments setting them up appropriately.

#!/bin/bash
BUILD='LOCAL'
show_help()
{
cat << EOF
    usage:
        $0 -b LOCAL     for local Thunder and MySQL Docker  (DEFAULT)
        $0 -b PROD      for remote Thunder Docker image to connect to AWS RDS
EOF
}

while getopts "hb:" opt
do
    case "$opt" in
        h)
            show_help
            exit 0
        ;;
        b)
            BUILD=$OPTARG
        ;;
    esac
done
# Stop the existing images, if they are running
LOCAL=`docker ps -q --filter=ancestor=local`
PROD=`docker ps -q --filter=ancestor=prod`
MYSQL=`docker ps -q --filter=ancestor=mysql`

if [ ! -z "$LOCAL" ]; then
    echo "Stopping running Local instance"
    docker stop local
fi
if [ ! -z "$PROD" ]; then
    echo "Stopping running Prod instance"
    docker stop prod
fi
if [ ! -z "$MYSQL" ]; then
    echo "Stopping MySQL instance"
    docker stop mysql
fi
# Check for Local or Prod build then use the right directories
if [ "${BUILD}" == 'LOCAL' ]then
    echo -e "Starting a $BUILD environment"
    docker-compose -f deployment/docker/local/tomcat.yaml --verbose up -d
else
    echo -e "Starting a $BUILD environment"
    docker-compose -f deployment/docker/prod/tomcat.yaml --verbose up -d
fi

The YAML file I set for the environment gets the Containers up and running with the settings I needed.

version: '2.2'
services:
    tomcat:
        container_name: tomcat
        image: local:latest
        environment:
              CATALINA_OPTS: "-server -Xms1024m -Xmx2048m -XX:PermSize=1024m -XX:MaxPermSize=1536m -Dhost_default_domain=tlocal.test.server -Djava.net.preferIPv4Stack=true -agentlib:jdwp=transport=dt_socket,address=1043,server=y,suspend=n"
        ports:
            - "127.0.0.1:80:8080"
            - "443:8443"
        restart: on-failure
        links:
            - mysql
    mysql:
        container_name: mysql
        image: mysql:latest
        ports:
          - 3306:3306
        environment:
          MYSQL_DATABASE: testdb
          MYSQL_USER: testadmin
          MYSQL_PASSWORD: testpass
          MYSQL_ROOT_PASSWORD: testpass
        mem_limit: 1000000000

With that I have my Tomcat and MySQL Containers up and running, I can point some local tests to the Test Server URL and have my tests run against and archive their runs locally.  This gives me flexibility in testing changes as I can do my code locally build my Docker environment and test without worrying about checking into the repository and checking out in another environment that may or may not be in use.

Friday, August 24, 2018

Pull Requests: How the Process Can Get Stuck

At my current workplace we have the usual Pull Request process:


  • Open a branch
  • Do work
  • Commit all my work
  • Push to the branch
  • Previous steps...ad nauseum
  • When ready open a PR!
A Pull Request can be either a Review or a Merge.  A Pull Request I will heretofore refer to as the PR.

I do Reviews at times, to assure that the work fits everything we are doing and usually its when trying something new, novel, or dangerous.  Once there is some sign off on the Review, and we try to turn these around in < 48 hours, then it becomes a Merge or we generate a new one as a Merge to get all the updates and changes (if any) in the repo.

Merges are fairly straightforward, get some comments or review on the Code.  Does it fit the style guides, are core files updated properly and cross-tested with other peoples Code so we don't have regression failures in other areas.  Once all that is done, again we turn these around quick, or try to, then the merge happens and we go on our way.

Occasionally we have things that throw lots of wrenches in the process.

Big PRs.  I mean HUGE!  Something that touches codes and libraries all over the place, these tend to be rare or due to some major functional change.  A PR like this can take days to review and go over, sometimes with a lot of back and forge, merge conflicts due to others that have been merged along the way.  They become a huge mess and while rare can rear up like a nightmare that sits in the queue for a long period of time until it is finally ready and gets put in, with a little hesitation on that merge button because you just KNOW that within a day something will break and there will be follow-up.  Though usually that's natural, you just hope the follow-up with be slight.

PRs that get opened due to people doing work that will stretch over time, are waiting for core functionality to be finished in another area, or its a huge group effort.  These are the ones that irk me the most because they should never be opened that early in the first place, at least to me.  When I see them I can feel my breath stop and I just know I need a little walk before I continue, they just irk me to no end.
With a global company where people live and work in different areas, and different schedules, it feels like this is the best way to communicate things.  With all of the other tools at our disposal (Teams, Slack, HipChat, Email [remember that one?]) sometimes people think this is the best communication mechanism.  I may be in the minority and disagree, that you can work on a branch with different people if needed, or that branch may need to wait, but there is no need to open it so early, its not ready there is nothing here more than a sign that pops up saying "review me, there was yet another checkin!".  Diplomatically I mention to people there are other mechanisms, but so far its been like tilting at windmills.  Though I dream some day people will listen.

Vacation Days!  Yes, there are even the PRs where the owner opens in the final hours of a Friday afternoon and submits everything, then goes on a well deserved break for the next week and won't be checking emails.  These sit around, with people wondering if the feedback will be looked at, or if that one Unit Test that failed is going to be resolved.  Not much you can do, but wait for them to get back and after catching up, get back to the PR.

Just like a Vacation Day the author can get swamped with something new, or a higher priority task, so the PR sits calmly in queue with comments and questions, which the author decides to get back to but doesn't really notify anyone.  So it sits waiting.

Most of this can be solved with good communication, but at times its just "the way things are".

Personally I think we can always do better, I have a few areas I need to work on constantly so I am a work in progress myself, but I like the see PRs be done quick, easy, and get out of my queue.

Especially when my manager is monitoring that queue and keeps wondering why there is one there.

Shh...it might be sleeping.

Monday, September 5, 2016

Adding an external file to JIRA

As a part of my companies personal project time, a yearly occurrance they call Off The Grid, I was part of a project to add external files to a JIRA ticket.  JIRA is used to track EVERYTHING in the company.

This is the python script I hobbled together.

#!/usr/bin/env python

from stat import S_ISREG, ST_CTIME, ST_MODE
import os,sys,re,requests
from requests.auth import HTTPBasicAuth
import urllib3
urllib3.disable_warnings()
def openFile(location,key,filetype,fileKey):
    """
    Tests that the file location exists and searches through each file for a specific keyword
    :param location:
    :param key:
    :param filetype:
    :param fileKey
    :return: outputFile:
    """
    outputFile = filetype + "output.txt"
    try:
        os.remove(outputFile)
    except OSError:
        pass
    f = open(outputFile,'w')
    print >> f, "Looking in %s" % location
    for dirpath, dirnames, files in os.walk(location):
        if not files:
            print dirpath, 'is empty'
            exit()
    entriesf = (os.path.join(location, fn) for fn in os.listdir(location))
    entries = ((os.stat(path), path) for path in entriesf)
    entries = ((stat[ST_CTIME], path)
               for stat, path in entries if S_ISREG(stat[ST_MODE]))
    # for cdate, path in sorted(entries):
    #    print time.ctime(cdate), os.path.basename(path)
    check_file = re.compile(fileKey)
    for file in entriesf:
        if os.path.isfile(file): # and check_file.search(file):
            for i,line in enumerate(open(file)):
                #if re.finditer(key,line):
                match = re.search(key,line)
                if match:
                    print >> f, "Found in %s \n %s" % (file, line)
                else:
                    print >> f, "No entries that matched keyword %s" % fileKey
        else:
            print >> f, "No files that matched %s" % fileKey
    f.close()
    return outputFile

def addSqlTraceToJira(location,ticket):
    """
    Add a SQL Trace file to the Jira Ticket
    :param location:
    :param ticket:
    :return:
    """
    entriesf = (os.path.join(location, fn) for fn in os.listdir(location))
    check_file = re.compile('trc$')
    check_ticket = re.compile(ticket)
    for file in entriesf:
        if os.path.isfile(file) and check_file.search(file) and check_ticket.search(file):
            attachFileToJira(file,ticket)
        else:
            print "No files found with %s" % ticket
def attachFileToJira(logfile,ticket):
    """
    Attach the output file to the Jira ticket
    :param logfile:
    :param ticket:
    :return:
    """
    url = "https://jira-test.fakedomain.comnet/rest/api/2/issue/%s/attachments" % ticket
    headers = {'X-Atlassian-Token':'no-check'}
    files = {'file': open(logfile, 'r')}
    auth = HTTPBasicAuth('username', 'password')
    # print "Sending %s to ticket %s" % (logfile,url)
    r = requests.post(url,headers=headers,files=files,auth=auth,verify=False)
    print 'Jira file attach sent me a %s with \n %s' % (r.status_code,r.text)

def addCommentToJira(keyword,ticket):
    """
    This attaches a comment about the search parameter that was added to the ticket
    :param keyword:
    :param tickiet:
    :return:
    """
    url = "https://jira-test.fakedomain.comnet/rest/api/2/issue/%s/comment" % ticket
    headers = {'X-Atlassian-Token': 'no-check'}
    auth = HTTPBasicAuth('username', 'password')
    comment = "Just added an attached log file about the results for the %s search." % keyword
    json = {"body": comment}
    r = requests.post(url,headers=headers,json=json,auth=auth,verify=False)
    print 'Jira adding a comment sent me a %s with \n %s' % (r.status_code, r.text)

def main():
    if len(sys.argv) < 5:
        from os.path import basename
        print ('Usage: %s location key type ticket' % basename(__file__))
    else:
        # fileLocation
        LOCATION = sys.argv[1]
        # searchKey
        SEARCHKEY = sys.argv[2]
        # type
        FILETYPE = sys.argv[3]
        # Jira Ticket
        TICKET = sys.argv[4]
        # File Keyword
        FILEKEY = sys.argv[5]
        #timeStart
        #TIMESTART = sys.argv[3]
        #timeend
        #TIMEEND = sys.argv[4]
    # logfile = openFile(LOCATION,SEARCHKEY,FILETYPE,FILEKEY)
    # attachFileToJira(logfile,TICKET)
    # addCommentToJira(SEARCHKEY,TICKET)
    addSqlTraceToJira(LOCATION,TICKET)
if __name__ == '__main__':
    main()

I claim some responsibility for it, mostly by hacking together a lot of python and doing a lot of TDD as I built out the functions.


Friday, July 15, 2016

Robot Framework and RESTful APIs

Been awhile.  Yup, it has.  Just a short one this time, more to get something down that vexed me for a little bit.

Dealing with a RESTful API that returned JSON in the body to my Robot Framework test caused me no end of grief.  Since the response always came back with the following:
\xef\xbb\xbf{
        "account_organizations": [
Originally I thought, it's just a JSON response, I can deal with it.

Oh no.

Not at all.

Turns out what I was getting ended up being byte order markers that were messing up the JSON compare I was trying to do.  Knowing what I was to get back was one thing, comparing that with the body of the response was another thing since the byte order markers did not behave like a regular list as I thought it was.  This turned out to be a dict, since Robot Framework basically is Python we have to be Pythonic here.

So, treating that as a dict we get the 3rd index from the list:
${l_response_body[3:]}
THAT now pulls the right values for comparison.

I also run this against a make unicode Python library, just in case.  So in Robot I use the following to cover it all:

${l_response_body} Get Response Body
${l_decoded_body} = make_unicode        ${l_response_body[3:]}

So all is good with the world, for now.

Monday, October 6, 2014

Testing Beliefs

This comes off of Pete Whalen's wonderful post on Testing and Wars of Religion.

Oftentimes we cling to our faiths in processes and procedures, too much in some cases.  It's hard to let go of something that you have worked hard to understand and struggled with for something new.  Change is hard.  Very hard.  But that is the point.  At some point beliefs and processes no longer work the same way and a new way to look at things is required.  Why?  Because just as work and processes change, new tools, languages and methods of achieving goals come and enter the environments we work in so to do these new things often challenge beliefs and require adjustment in how we look at things, or a new vision of the world.

Change is hard, but if it was easy then in many cases it would not be worth it.  I always look at them as educational opportunities, though even I challenge some of the changes that come up, its normal and once I get over my initial reluctance I can take a wide-view of things and move on.

Short, sweet and to the point.  Happy Monday!

Friday, September 19, 2014

The Power of PowerShell pipelining

Every now and again I come back to PowerShell, it's useful as a tool, simple, clean and while I can sometimes say its easy to use, many times I smack my head on the desk.  Working in a Windows environment PowerShell scripts make doing many things easier, though it usually takes me some time to work them out.

For example, I am trying to output some results from failures on running SSIS Tasks, but I only really need to know certain statuses, or in most cases just the failures.  Focusing on the failures there can be a lot of them, if something fails at the beginning EVERYTHING runs and can easily fill up the log.  So while a command like:

    $Results = &'C:\Program Files\Microsoft SQL Server\110\DTS\Binn\DTExec.exe' /Project $Path /Package "$File.dtsx"

Can give me a huge result set, do I really need to display a couple of hundred lines of errors to the console, or to the Test Driver?  No.

I wanted to only pull out the error details, I am skipping over the step where I know there are errors here (just to make it easy) and what I want are the details to notify someone that "Hey, something went wrong here!  Here are some examples, want to know more, log in to the box and do some research!!"

So at first I step through everything (that's how I understand where things are working) and first came up with this:

foreach ($_ in $Results)
{
if ($_ -match "Description")
{
$exec_errors += $_
}
}

Now that is way too long.  A lot of white space, plus do I really need to step through that much?  No I don't, thankfully PowerShell makes this easy, since $Results is an array, I can pipeline it and step through it to match only what I want:

    $Results|?{ $_ -match "Description" }

This gets me only the data I want to display, the rest of what you get for the error messages is meaningless to convey the details of what went wrong.  I pipeline the $Results array into ? which is shorthand for the Where-Object and within the curly braces put what it is I was looking before in my if statement, under my foreach.  So now instead of a 3 or so line statement (not quite counting the curly braces there) I have 1 line!  Yes, there can be only 1.

Still I need someplace to put those results:

    $exec_errors = $Results|?{ $_ -match "Description" }

Now, I have an array of just the details I want and using $exec_errors I can output only what I need to communicate:

Write-Output "Errors in execution: "
if ($exec_errors.length -ge 2) {
Write-Output "$exec_errors[0] `n $exec_errors.length - 1"
} else {
Write-Output $exec_errors
}

I will have to see about making a one-liner of the rest later on.

Tuesday, September 9, 2014

SSIS Task Variables

In some current Testing I am doing work with SSIS.  One task imports data into a database that we baseline for future Test Cases, but to get the data in I need to modify some of the dates.  With SSIS there is an option to adjust data using Derived Columns, with that it's possible to adjust data using Task Scoped Variables; I'll have to find a reference but using project scoped variables did not work initially.

Basically the variables were created this way:

  1. Gave a descriptive name (need to have those so you know what to look for later!)
  2. Scope, was selected to be the Task being worked on, which was easy since the project only had one task for data loading.
  3. Data Type, since it was for Dates I made this DateTime
  4. Value, just hit enter/return here and it took the current date and time
  5. Expression, this is the real work here
    1. For Now this was just left as the default - GetDate()
    2. For a Year I made an expression - DATEADD("Year",1,(GetDate()))
    3. Same adjustments for Adding Minute or Month, just adjust Year
When using these, in the Derived Columns select the Expression column and in the upper right panel there is a tree for Variables and Parameters.  Select the User: and this will update the value for that column when the Derived Columns task runs.

Learning about DateAdd came from the Microsoft Site

Monday, August 11, 2014

Why History really relates to Testing

I am a History buff.  Of course I also went back to school and did a major in it, getting my Bachelor's from Boston University while still working in Software.  I was going to do a CS degree but honestly, much as I like it, programming during the day and for homework didn't really suit me.  After some discussion with my wife I ended up changing to History, since it was something I loved, I think it was a great decision since I got exposed to subjects and topics I never would have thought to take on my own.  From those I learned a lot about how to analyze situations and topics, do research and apply that to the question or topic at hand; something I do a lot of while Testing.

Testing IS History.  Regression Tests are a way of compounding a Historical Record of Fact that says, these things happened and if we want to prevent them we can do THIS or THAT.  Something that in History we often table about, the What If format is not only a plot line from Fiction novels but sometimes is a way of viewing an event from a certain perspective.  Of course as our understanding of events increases so does the Historical Record and our understanding of it, I find that Regression Tests are the same way.  The more I understand a certain issue, or feature, the more I can improve my test case and get the confidence level of the product higher.

When scheduling I rely on my knowledge of prior events, but being the person I am I document everything as I go along.  This gives me a record to use and base my future estimates on rather than giving my "gut feel" for a specific project or set of features.  When I did releases we often put times for specific steps to take, this way in long releases we were able to say when certain people were needed to help, or to even frame the maintenance window needed.

These are just two examples, but very deep and broad ones.

How do you use and document your prior knowledge on events and tests?  Do you document your test code, like a historical record, so others who come later can figure out what you were doing?


Sunday, April 27, 2014

The Person Month Revisited

I was walking down the street and noticed a few people were picking up trash in the road, thought to myself "hey, I bet with one or two more they get done real quick!"

Of course, then my thoughts turned to Software and then in some weird connection I jumped into the Man Month Myth, renamed the Person Month to be more gender neutral, and it occurred to me that yes it still happens and yes in some circumstance it may work.  Though those are slight, and slim.  It of course depends.

Let's look at some examples:

Street Cleaners: the more the merrier!  It doesn't take much domain knowledge to pick up trash on a street, or rake leaves from a yard.  All you need is some coordination and a slight understanding of tools.  Of course, coordination with those tools will make the job faster, a group of 4 or 5 teens could do well, if those are the only resources we have available.  It's like after a party at a house, or a bbq, the more people taking little tasks and doing them, the quicker the whole task get's done.  I don't want to be demeaning but at it's core there is not a lot of skill involved in cleaning up, using a broom is fairly simple as is carrying a bag to put trash in - at this point the main skill is knowing what is trash.  So at this point the skill level is not high, and the tool usage ability is low.

Too many cooks spoil the soup: yes, that adage does have some relevance.  The reason you want one cook is because he understands how this dish should taste, everyone is slightly different and the more you have the more additions you get until the soup is a mess.  Also, hardly tasty.  Not everyone has the same taste or background to know what a particular dish should be, the best restaurants do well because every dish they have has consistency but this is due to long hours of training and knowledge transfer.  So we have a medium skill level and a tool usage ability is getting higher, but now we have an added knowledge transfer quotient to add in as there is a lot to know about making good soup.

Software: it depends on the domain knowledge and skill set.  The high up the domain tree you go the more it requires a specific fit.  Sure you could add one or two more Engineers to your project BUT those Engineers had better be really familiar with the project, specs, and domain or you do more harm than good.  The specs have been discussed, meetings have been held to discuss code and flow and design, there may be a lot of documents, depending on your environment but there is also a lot of knowledge transfer.  There is a high skill level, a high usage requirement for tools and a lot of knowledge transfer.

So, can you add more people and get more done?  Yes, but only in specific situations.  If people have the skills, tool and domain knowledge then adding them in is not an impediment, but this is rarely the case.  Although if you have an environment with lots of switching between teams then some of this may be in place but you have to know.  If you don't analyze the situation first you will just make a mess of it.  Like the soup.

Friday, April 18, 2014

Sauce Connect Script

I am currently working with Sauce Labs and using their Cloud Based testing to work with some automation, what I wanted first was an easy way to handle starting up the tunnel to Sauce Labs.  This was a jar file so I tried a few other methods to set the environment up the way I want, but considering that most environments I am working with are Windows I figured may as well use the old stand by of PowerShell.

Here is my current version of the script, this just checks to see that the environment is set up so I can output the log files in the places I want, and checks if a job is running then just continues on and uses the existing one - for now.  I heavily comment code when I first write it, so it should be enough to answer most questions, if not comment here and let me know!

----

# Sauce Labs Connect script
# Description:
# This script checks and sees if there is an existing Job that has an open tunnel
# to Sauce Labs in the current session; because of the way PowerShell works you
# cannot easily detect what is running in another session.  Basically this script
# was intended to generate the necessary Tunnel and run the test scripts with
# fewer keystrokes
# Author: Michael Furmaniuk
# Last Updated: April 16, 2014

# Command line values, so there is no reliance on the Username and Access Key
# being set in the environment or in the configuration files (although if they already are...)
param([string]$u, [string]$k)
Set-StrictMode -v 2

#######################################
# Check and make sure some of the specific needs are met
# Sauce Connect
[string]$jobName = "sauceConn"
[string]$rootPath = ""
[string]$sauceLog = "sauceLog.log"
[string]$sauceReady = "sauceReady.log"
[string]$sauceJar = "C:\Sauce-Connect-jar\Sauce-Connect.jar"
# If necessary the following could be completed to make this easier to run, otherwise these are
# command line arguments given as $u $k
# [string]$u = ""
# [string] $k = ""

#######################################
# Start the Sauce Jar
function start-SauceConnectJar()
{
<#
.Synopsis
This function starts the Sauce Connect Jar utilizing some known and command line parameters
.Description
This function starts the Sauce Connect Jar utilizing some known and command line parameters
#>
[CmdLetBinding()]
Param(
[Parameter(Mandatory=$true)][string]$sauceLog,
[Parameter(Mandatory=$true)][string]$sauceReady,
[Parameter(Mandatory=$true)][string]$sauceJar,
[Parameter(Mandatory=$true)][string]$u,
[Parameter(Mandatory=$true)][string]$k
) #end param
Process
{
# Let's check and see what drives and temp files we have
if (Test-Path "C:\TEMP") {
$rootPath = "C:\TEMP"
} elseif (Test-Path "D:\TEMP") {
$rootPath = "D:\TEMP"
} else {
Write-Host "I can't seem to detect any disk drives here with a TEMP directory, that is a problem.`n"
exit
}
# Build up the file paths
$sauceLog =  $rootPath + "\" + $sauceLog
$sauceReady = $rootPath + "\" + $sauceReady
# If these already exist we want to remove them
if (Test-Path $sauceLog) {
Remove-Item $sauceLog
}
# Note: if this call fails then there is an existing job in another session
# need to somehow handle this and fail gracefully: this is a TODO
if (Test-Path $sauceReady) {
Remove-Item $sauceReady
}
# Start Sauce Connect and see what is returned for the process
# Build up the latter part of the script block
$commandLineOption = "-l $sauceLog -f $sauceReady -d"
# Now for the full command line
$scriptblock = [scriptblock]::Create("java -jar $sauceJar $u $k $commandLineOption")

# Now let's start this as a Job, next line is for debugging
# Write-Host "Starting Sauce Connect with: $($scriptblock).`n"
# Should now be suppressing the start text
$output = Invoke-Expression "start-job -name $jobName -ScriptBlock { $scriptblock } 2>&1"

# Check for the Ready File
while (!(Test-Path $sauceReady)) {
Write-Host "Sauce Connect is still starting...`n"
sleep(10)
}
# Next statement is for debugging
# Since you only get a Tunnel ID when the connection is complete using this as a defining
# point to know the tunne is up, next statement was for debugging
# Write-Host "Sauce Connect should be up...getting the Tunnel ID.`n"
# Now acually getting the Endpoint/Tunnel ID as a verification step
get-SauceEndPoint $sauceLog
}
}

#######################################
# Get the endpoint ID from the log
function get-SauceEndPoint()
{
<#
.Synopsis
This function scans the initial Sauce Connect log file for the EndPoint ID to pass back
.Description
This function scans the initial Sauce Connect log file for the EndPoint ID to pass back
to the User through the console, so it can be used by the User to access the active Tunnel.
At some point this may be used for further automation
#>
[CmdLetBinding()]
Param(
[Parameter(Mandatory=$true)][string]$sauceLog
) #end param
Process {
# Now that the Connection is up, and the Ready File is in place
# read the log file to get the Tunnel ID or endpoint in the case of Java
$runningLog = Get-Content $sauceLog
# Run a regex to get the endpoint line, only one in the file at a time
# when this script runs, then print this out so it can be used (or saved for something else later)
$myMatch = $runningLog -match "endpoint ID:\s.*$" | %{$_ -split "ID:\s"}
# Next statement was a way to message that the Tunnel is up, if not needed comment out
Write-Host "Endpoint ID this time is: $($myMatch[1])"
}
}

#######################################
# The Sauce branch depending on what is running,
# I can either start a new Job or shut it down as/if necessary
$jobs = Get-Job 2>&1
if ($jobs) {
if ($jobs.State -eq "Running") {
# There is an existing Job, either we want to stop it and restart or drop out
# Right now going with using an existing Tunnel
Write-Host "An existing job is running, quitting for now."
Write-Host "...but getting the EndPoint ID to be useful."
get-SauceEndPoint $sauceLog
exit
}
elseif ($jobs.Name -eq $jobName -and $jobs.State -eq "Completed") {
Write-Host "Found $($jobs.Name) that was $($jobs.State) getting rid of it...`n"
Remove-Job $jobName
# Now that the previous one is gone, let's move on
Write-Host "Removed the old job, so starting a new Tunnel.`n"
start-SauceConnectJar $sauceLog $sauceReady $sauceJar $u $k
} else {
Write-Host "Weird, found a job but its not running, it's $($jobs.State).`n"
}
} else {
# Shouldn't be any other states to worry about at this point
# so just start the Jar file and move along
Write-Host "Nothing running, so starting the Tunnel.`n"
start-SauceConnectJar $sauceLog $sauceReady $sauceJar $u $k
}

Tuesday, January 21, 2014

My 2013 Year in Review

Since January seems to be the time for reflection, and looking forward, I'll oblige and review the major milestones I had for 2013 and then see if I can learn anything from it all.

The year began with a compressed timeline to set up a new hosting site, the SharePoint environment we had in one hosting center needed to be recreated in another as we had to move data centers to a new provider.  Basically I spent two months prior to the end of the year, and most of January 2013 getting the environment ready, this provided some much needed updates to the existing scripts and documentation we used to be able to create these environments.  There were some mistakes in setting up the environment, I know now the best way NOT to update IIS redirects, and have much more experience with those under my belt.  In February we brought the site up and live, with some delay as the method I was using to copy over files that were needed for various apps was not completing properly and with some work with the IT guys we got that worked out.

Of course that hosting site lasted about two months until we started getting issues with up time, even having a 36 hour site downtime due to the center trying to update their disc software for the virtual machines.  I am not a hardware guy but apparently the disk update did not go well and brought down everything on that disk for a day and a half, we got reports for the first 6 hours or so then nothing until the site came up.  We knew it was up before they called us.

So another site move happened!  This one was much smoother and with the documentation updated it went much better.  Still two in one year,quite a feat!

All through the year I am also updating the Automation Framework that runs the Regression Tests, Feature Tests, and Acceptance Tests I have in place.  This is a Selenium setup with Web Driver, SpecFlow to utilize some BDD, driven with PowerShell scripts and written within Visual Studio so I get some experience coding in C#.  It's a nice framework and has had some awesome improvements and abstraction added, I'd like to do some more but have not had a chance yet.

I completed my O'Reilly online course in JavaScript programming, I learned a lot in that and am still amazed that I passed considering that I think my own programming skills are subpar.  I have learned how better to structure my code, and have improved my own Test Framework with abstraction and reading from external data files so its easier to update information when our site has changes.

With the PowerShell scripts that we have for our Build and Deploy network I have added some improvements and also am planning a PowerShell 3 update for 2014.  I've already worked out some serious issues with the changes and how we will deal with them, I just need to test for them and then begin implementation.

There is a site redesign coming, so I need to adjust my Test Framework, determine how to deal with changing existing redirects to new locations and also add new ones for people looking for information from the outside.  Should be fun!

All in all a good year and I have some interesting work coming up in 2014 with log scanning for people trying to access some information on our site that is ONLY tracked in our IIS files, since the data is on our secondary site that is not under Google Analytics tracking.  Can't wait!

Wednesday, October 23, 2013

No Software is Perfect, but the ACA was not the best

"If you believe the doctors, nothing is wholesome; if you believe the
theologians, nothing is innocent; if you believe the military, nothing is safe."
Lord Salisbury

To paraphrase, "if you think a software deployment never has problems, nothing is shipped".

Originally I started this post thinking about deployments but with the release of the ACA web site, portal, whatever it really is I saw a very public forum airing something on which I deal with on a consistent basis.  Anyone, and I mean ANYONE, who has worked in software has been involved with or seen a botched deployment on code either being shipped to customers, or as is more the case these days, on code released to a live site.  Things generally go well, and the scale of problems varies, but it is always there.  To mitigate we test, check, use the code in an environment that will simulate the production environment.  There are too many test types to mention that can be used, but in any of them actually USING the code or application will show up issues of some kind, at the very least.  I will say something that anyone working in tech knows, No Software is Perfect.

Listening to On Point on NPR, I support Public Radio and typically I am agreement with most of what Tom Ashbrook says but this time I sadly think he was out of his depth.  Sure it looks bad, but this is a political hot potato and anything that could have gone bad would be a candidate for hyperbole.  Was the rollout of the government web site bad?  Oh yes, just from the comments I wonder what sort of process they ran it through, but doing this for a living I know how hard it is and what it takes.  My older relatives who barely use computers would probably get a different perspective.  Yet I also know people who have worked on government projects, even looking back at many of the governments weapon's procurement and development programs, none of this should be a surprise.  The US Government still does a large amount of work on a waterfall process where they just start down a road and keep driving, filling up the tank as it gets empty so the work continues and many times no one is navigating, so the project just keeps driving along.

Still, after hashing through many of the reports over the past two weeks here is what I see as the major issues, and yet I see this on many projects, some even today, and no one has died yet.

  • No Testing, or whatever kind there was seems to be minimal.  If the site was crashing as people said I would have to ask what sort of Load Testing there was.  I don't even think I want to bring up Security testing.
  • Coding done by Government Contract, seems like a big waterfall of a project with a process to match, seems like there was little Acceptance testing done until the end when it was deployed.
  • Outsourcing, not a bad thing, but was the company that really doing this the best choice?  From some of what I had seen few traditional software companies wanted in on this, and the company that did do it seems to have a spotty track record at best.
  • No Communication or Transparency, this was a big project (I am using that sarcastically since more Web Applications and sites don't have millions of lines of code, but it's big in the scale of visibility) and with the size it seemed much of the status was held close to the leaders.  Sure its a political liability, but when you aren't transparent people make up their own rationales.
  • Tech Surge, oh yeah we all know throwing more people and money at something gets it done faster!  Right?
  • Were the people really the "best and brightest"?  Or if they are only coming in now why would they want to refactor code?  Seems a waste of a skill set to me.  And if they are only coming in now then who was working on this?
  • Bureaucracy.  'nuff said.

Rocky roll outs are a norm, but maybe this time people who actually run projects, or those on them that they feel are going wrong can point out to the ACA web site rollout and say, "let's not be like them."

Maybe, just maybe, then, someone will listen.

Monday, September 16, 2013

Reading about Quality

On occasion I read books on Quality that are basically throwbacks, or what I like to think of as historical documents.  Recently I was going through one of Juran's books on Quality to see what I might be able to learn and apply from what Quality Control and Process Design was all about before software.  This was sort of inspired by a recent book I read on Toyota and its evolution of Lean Manufacturing leading up to Kanban and its current practices, I also found out some interesting stuff on Toyota cars and how they are designed.  Considering I drive a Highlander and my wife a Camry it was cool stuff!

So while reading through Juran's book I noticed the following formula:

Quality = (Frequency of deficiencies) / ( Opportunity for deficiencies)

I wondered how I might be able to use this in my own arena.  Dealing with web sites that have hundreds or thousands of visitors a day there is no direct linkage here.  So just toying around with ideas I thought I might be able to use something along the lines of Visitors from our web analytics and determine then some number through contacts to our Support Organization to see if I could get an idea of Opportunity and/or Frequency through these numbers.  I'm no statician at all, so not sure if this will work at all but I would like to try out, as a mental exercise, whether I can get something useful from this and see if I can get an idea of what our Quality is.

Typically I am against just making up metrics, but in some cases I think its fairly good to be able to give a representable number to Business Users if that number CAN be representative and provide a USEFUL idea of what is going on.

Friday, September 6, 2013

Who am I reading?

Like many others out on the interweb, I tend to read certain blogs or newsfeeds on a consistent basis.  I do have a focus on Testing and Quality but I have a few other divergent interests, some align and some don't.

Without further ado, my list of favorites includes:
Enjoy, if you are not already doing so!

Tuesday, September 3, 2013

Working with CSV and PowerShell for Link Checkers

One of the current tools in my toolbox is Xenu Link Sleuth which does a great job of scanning through our site and finding links that are broken or need updating.  Although while the report it gives is good to review afterwards it is difficult to send out to others, so to make the data useful to my business users I needed to make something that was more readable.  Since PowerShell is something that we use often it seemed an easy choice to write something that reads through the exported data and generates a useful report.

# Crawl Report
$crawlReport = Import-Csv "" -Delimiter "`t"
$crawlList = @()
# Page Map
$siteMap = Import-Csv "" -Delimiter "`t"
$siteList = @()
# Links List
[string] $brokenLinksList = "BrokenLinksList.txt"
[string] $brokenLinksOverview = "BrokenLinksList.csv"
# Clean up from last time
if (Test-Path $brokenLinksList) {
  Remove-Item $brokenLinksList -Force
}
if (Test-Path $brokenLinksOverview) {
  Remove-Item $brokenLinksOverview -Force
}
 
"Just some initial information on what we are dealing with."
"Crawl Report is " + $crawlReport.Length + " lines."
"Site Map is " + $siteMap.Length + " lines."
# Check each Address that has a Status-Code of 404, add it to an array we will
# use to get its origin from the Site Map
foreach ($row in $crawlReport) {
  if ($row.StatusCode -eq "404") {
    $crawlList += $row.Address
  }
}
"We now have " + $crawlList.Count + " broken links to deal with."
# Check in the Site Map for the Address as a Link to Page
foreach ($link in $crawlList) {
  foreach ($line in $siteMap) {
    if ($link -eq $line.LinkToPage) {
      # Pull these together to make it simpler to review later on
      $data = $link + "," + $line.LinkToPage + "," + $line.OriginPage
      $siteList += $data
    }
  }
}
"Overall there are " + $siteList.Count + " broken links to fix."
# Output the Address, Link To Page and Origin Page to a separate file
# since these were put together as CSV's earlier let's just make an
# exported CSV file for ease of use later on
"BrokenLink,LinkToPage,OriginPage" >> $brokenLinksList
foreach ($entry in $siteList) {
  $entry >> $brokenLinksList
}
# Now that we have everything in one file let's make it a CSV
$brokenLinksList | Export-Csv $brokenLinksOverview

This puts the data into a CSV file that can be filtered and sorted in Excel, which puts the data in a format that's more useful for my business users.