Building out Tomcat and MySQL with Docker

We have a Test Queuing and Archive server that runs on Tomcat, with the data stored in MySQL.  This currently runs on vm's in our lab, but as part of the process to move this to AWS I moved the manual setup for these to Docker images.  This gives us a few advantages:

1.  A long, multi-day wiki page install is distilled down into one docker image that builds in 5 minutes
2. We get a repeatable environment usable whenever we need to deploy, without worrying about underlying changes of configuration files
3. Being able to build a local Development version of the environment that can be brought up and used in minutes, rather than move to the Development vm and run some install scripts for an environment we barely use.  Updates to this system are rare.

Because I love automation I added some local shell scripts to check for running Docker images, containers, and added cleanup for previous builds.  In testing I was building these over and over and found I had at one point 5 GB of previous build images that I don't need.  The scripts also handle stopping the Containers so I don't have collisions in case I forget to check if the Containers are running, sometimes I don't check my laptop after the weekend to see what I might have left running.  I know, that's a bad habit, but this way I don't need to worry.

For my Local Development environment I know what IP I am going to start my Containers on, so I added an entry in my HOST file to be able to use a named instance to the Container when it comes up.  I like that better than worrying about trying to connect to an IP address.

As part of adding this into Jenkins I added parameters to the shell scripts to build for the environments I need, a Local Development environment needs MySQL the Production environment has an RDS DB running, so all I need for that is Tomcat.  This way all the configuration is set in the Dockerfile and I just build for the environment I want.

#!/bin/bash 

# Build script for Local Tomcat usable on local environments

BUILD='LOCAL'

show_help()

{

cat << EOF

    usage:

        $0 -b local     for local Thunder and MySQL Docker  (DEFAULT)

        $0 -b prod      for remote Thunder Docker image to connect to AWS RDS

EOF

}

while getopts "hb:" opt; do

    case $opt in

        h)

            show_help

            exit 0

        ;;

        b)

            BUILD=$OPTARG

        ;;

    esac

done

# WAR file and web server files

echo "We should always want to do a clean then a new build"

wget -O www/scripts/jquery.js code.jquery.com/jquery-1.11.1.js

cp www/index.html.template.templ www/index.html

./gradlew clean build

if [ -e build/libs/my.war ]then

    echo "my.war was built correctly"

else

    echo "my.war was not built, some kind of problem?"

    exit 1

fi

# Stop the existing images, if they are running

LOCAL=`docker ps -q --filter=ancestor=local`

PROD=`docker ps -q --filter=ancestor=prod`

MYSQL=`docker ps -q --filter=ancestor=mysql`

if [ ! -z "$LOCAL" ]; then

    echo "Stopping running Local instance"

    docker stop local

fi

if [ ! -z "$PROD" ]; then

    echo "Stopping running Prod instance"

    docker stop prod

fi

if [ ! -z "$MYSQL" ]; then

    echo "Stopping MySQL instance"

    docker stop mysql

fi

# Let's just be cleanly and remove all those old dangly images

docker system prune -f

# Create a Docker image

if [ "${BUILD}" == 'LOCAL' ]then

    echo -e "Running a $BUILD build"

    # Build all for local but only need Tomcat for Prod

    docker build -t mysql -f deployment/docker/local/mysql/Dockerfile .

    docker build -t local -f deployment/docker/local/tomcat/Dockerfile .

else

    echo -e "Running a $BUILD build"

    # Prod version should be prod name, don't need a database

    docker build -t prod -f deployment/docker/prod/tomcat/Dockerfile .

fi

# Cleanup the files we only need for the war file

rm www/index.html

rm www/scripts/jquery.js

For Tomcat I have some configuration to do with the context and server files, that will vary for environment, but what I especially needed was a way in Docker to build up a self-signed certificate without manual intervention.  After some searching around and trial/error I was able to come up with the commands that I needed to build that.

FROM tomcat:7.0-jre8

# Install useful toolsRUN apt-get update && \

    apt-get -y upgrade && \

    apt-get install -y wget vim curl procps tar libssl-dev --fix-missing --no-install-recommends

# Configure Tomcat Container

COPY build/libs/my.war /usr/local/tomcat/webapps/

COPY deployment/docker/local/resources/local-server.xml /usr/local/tomcat/conf/server.xml

COPY deployment/docker/local/resources/local-context.xml /usr/local/tomcat/conf/context.xml

COPY deployment/docker/local/resources/local-environment.xml /usr/local/tomcat/conf/environment.xml

COPY deployment/docker/local/resources/local-testserver.xml /usr/local/tomcat/conf/testserver.xml

# MySQL Connector

WORKDIR /usr/local/tomcat/lib/

RUN curl -L -o mysql-connector-java-5.1.46.tar.gz https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

# Add in untar of the mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar

RUN tar -xf mysql-connector-java-5.1.46.tar.gz mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar

RUN cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar .

# Configure Tomcat User and GroupENV RUN_USER    tomcat

ENV RUN_GROUP   tomcat

RUN groupadd -r ${RUN_GROUP} && useradd -g ${RUN_GROUP} -d ${CATALINA_HOME} -s /bin/bash ${RUN_USER} && \

    chown -R ${RUN_USER}:${RUN_USER} $CATALINA_HOMERUN chown -R ${RUN_USER}.${RUN_USER} /data/RUN chown -R ${RUN_USER}.${RUN_USER} /usr/local/resource-manager/

# Setup SSL for Tomcat

WORKDIR /usr/local/tomcat/conf/RUN $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA -storepass changeit -keypass changeit -noprompt -dname "CN=local.test.server, OU=Central, O=MYCOMP, L=BOSTON, S=Massachusetts, C=US" -keystore "/usr/local/tomcat/conf/.keystore"

RUN openssl req -newkey rsa:2048 -x509 -subj "/C=US/ST=Massachusetts/L=BOSTON/O=MYCOMP/CN=local.test.server" -keyout cakey.pem -out cacert.pem -passout pass:mypass

RUN openssl pkcs12 -export -in cacert.pem -inkey cakey.pem -out identity.p12 -name "local" -passin pass:mypass -password pass:mypass

RUN keytool -importkeystore -destkeystore identity.jks -deststorepass mypass -srckeystore identity.p12 -srcstoretype PKCS12 -srcstorepass mypass

RUN keytool -import -file cacert.pem -keystore trust.jks -storepass mypass -noprompt

WORKDIR /usr/local/tomcat/

USER tomcat

For the Production build I need an actual certificate, but since that environment is not fully ready I don't need to import that yet.

There is a shell script that gets the Local Development environment running, using Docker-Compose I am able to get the environment I need up and running.  Again the shell script has a path for Local and Production environments setting them up appropriately.

#!/bin/bash

BUILD='LOCAL'

show_help()

{

cat << EOF

    usage:

        $0 -b LOCAL     for local Thunder and MySQL Docker  (DEFAULT)

        $0 -b PROD      for remote Thunder Docker image to connect to AWS RDS

EOF

}

while getopts "hb:" opt

do

    case "$opt" in

        h)

            show_help

            exit 0

        ;;

        b)

            BUILD=$OPTARG

        ;;

    esac

done

# Stop the existing images, if they are running

LOCAL=`docker ps -q --filter=ancestor=local`

PROD=`docker ps -q --filter=ancestor=prod`

MYSQL=`docker ps -q --filter=ancestor=mysql`

if [ ! -z "$LOCAL" ]; then

    echo "Stopping running Local instance"

    docker stop local

fi

if [ ! -z "$PROD" ]; then

    echo "Stopping running Prod instance"

    docker stop prod

fi

if [ ! -z "$MYSQL" ]; then

    echo "Stopping MySQL instance"

    docker stop mysql

fi

# Check for Local or Prod build then use the right directories

if [ "${BUILD}" == 'LOCAL' ]then

    echo -e "Starting a $BUILD environment"

    docker-compose -f deployment/docker/local/tomcat.yaml --verbose up -d

else

    echo -e "Starting a $BUILD environment"

    docker-compose -f deployment/docker/prod/tomcat.yaml --verbose up -d

fi

The YAML file I set for the environment gets the Containers up and running with the settings I needed.

version: '2.2'

services:

    tomcat:

        container_name: tomcat

        image: local:latest

        environment:

              CATALINA_OPTS: "-server -Xms1024m -Xmx2048m -XX:PermSize=1024m -XX:MaxPermSize=1536m -Dhost_default_domain=tlocal.test.server -Djava.net.preferIPv4Stack=true -agentlib:jdwp=transport=dt_socket,address=1043,server=y,suspend=n"

        ports:

            - "127.0.0.1:80:8080"

            - "443:8443"

        restart: on-failure

        links:

            - mysql

    mysql:

        container_name: mysql

        image: mysql:latest

        ports:

          - 3306:3306

        environment:

          MYSQL_DATABASE: testdb

          MYSQL_USER: testadmin

          MYSQL_PASSWORD: testpass

          MYSQL_ROOT_PASSWORD: testpass

        mem_limit: 1000000000

With that I have my Tomcat and MySQL Containers up and running, I can point some local tests to the Test Server URL and have my tests run against and archive their runs locally.  This gives me flexibility in testing changes as I can do my code locally build my Docker environment and test without worrying about checking into the repository and checking out in another environment that may or may not be in use.

Pull Requests: How the Process Can Get Stuck

At my current workplace we have the usual Pull Request process:

• Open a branch
• Do work
• Commit all my work
• Push to the branch
• Previous steps...ad nauseum
• When ready open a PR!

A Pull Request can be either a Review or a Merge.  A Pull Request I will heretofore refer to as the PR.

I do Reviews at times, to assure that the work fits everything we are doing and usually its when trying something new, novel, or dangerous.  Once there is some sign off on the Review, and we try to turn these around in < 48 hours, then it becomes a Merge or we generate a new one as a Merge to get all the updates and changes (if any) in the repo.

Merges are fairly straightforward, get some comments or review on the Code.  Does it fit the style guides, are core files updated properly and cross-tested with other peoples Code so we don't have regression failures in other areas.  Once all that is done, again we turn these around quick, or try to, then the merge happens and we go on our way.

Occasionally we have things that throw lots of wrenches in the process.

Big PRs.  I mean HUGE!  Something that touches codes and libraries all over the place, these tend to be rare or due to some major functional change.  A PR like this can take days to review and go over, sometimes with a lot of back and forge, merge conflicts due to others that have been merged along the way.  They become a huge mess and while rare can rear up like a nightmare that sits in the queue for a long period of time until it is finally ready and gets put in, with a little hesitation on that merge button because you just KNOW that within a day something will break and there will be follow-up.  Though usually that's natural, you just hope the follow-up with be slight.

PRs that get opened due to people doing work that will stretch over time, are waiting for core functionality to be finished in another area, or its a huge group effort.  These are the ones that irk me the most because they should never be opened that early in the first place, at least to me.  When I see them I can feel my breath stop and I just know I need a little walk before I continue, they just irk me to no end.

With a global company where people live and work in different areas, and different schedules, it feels like this is the best way to communicate things.  With all of the other tools at our disposal (Teams, Slack, HipChat, Email [remember that one?]) sometimes people think this is the best communication mechanism.  I may be in the minority and disagree, that you can work on a branch with different people if needed, or that branch may need to wait, but there is no need to open it so early, its not ready there is nothing here more than a sign that pops up saying "review me, there was yet another checkin!".  Diplomatically I mention to people there are other mechanisms, but so far its been like tilting at windmills.  Though I dream some day people will listen.

Vacation Days!  Yes, there are even the PRs where the owner opens in the final hours of a Friday afternoon and submits everything, then goes on a well deserved break for the next week and won't be checking emails.  These sit around, with people wondering if the feedback will be looked at, or if that one Unit Test that failed is going to be resolved.  Not much you can do, but wait for them to get back and after catching up, get back to the PR.

Just like a Vacation Day the author can get swamped with something new, or a higher priority task, so the PR sits calmly in queue with comments and questions, which the author decides to get back to but doesn't really notify anyone.  So it sits waiting.

Most of this can be solved with good communication, but at times its just "the way things are".

Personally I think we can always do better, I have a few areas I need to work on constantly so I am a work in progress myself, but I like the see PRs be done quick, easy, and get out of my queue.

Especially when my manager is monitoring that queue and keeps wondering why there is one there.

Shh...it might be sleeping.

Adding an external file to JIRA

As a part of my companies personal project time, a yearly occurrance they call Off The Grid, I was part of a project to add external files to a JIRA ticket.  JIRA is used to track EVERYTHING in the company.

This is the python script I hobbled together.

#!/usr/bin/env python

from stat import S_ISREG, ST_CTIME, ST_MODE
import os,sys,re,requests
from requests.auth import HTTPBasicAuth
import urllib3
urllib3.disable_warnings()
def openFile(location,key,filetype,fileKey):
    """
    Tests that the file location exists and searches through each file for a specific keyword
    :param location:
    :param key:
    :param filetype:
    :param fileKey
    :return: outputFile:
    """
    outputFile = filetype + "output.txt"
    try:
        os.remove(outputFile)
    except OSError:
        pass
    f = open(outputFile,'w')
    print >> f, "Looking in %s" % location
    for dirpath, dirnames, files in os.walk(location):
        if not files:
            print dirpath, 'is empty'
            exit()
    entriesf = (os.path.join(location, fn) for fn in os.listdir(location))
    entries = ((os.stat(path), path) for path in entriesf)
    entries = ((stat[ST_CTIME], path)
               for stat, path in entries if S_ISREG(stat[ST_MODE]))
    # for cdate, path in sorted(entries):
    #    print time.ctime(cdate), os.path.basename(path)
    check_file = re.compile(fileKey)
    for file in entriesf:
        if os.path.isfile(file): # and check_file.search(file):
            for i,line in enumerate(open(file)):
                #if re.finditer(key,line):
                match = re.search(key,line)
                if match:
                    print >> f, "Found in %s \n %s" % (file, line)
                else:
                    print >> f, "No entries that matched keyword %s" % fileKey
        else:
            print >> f, "No files that matched %s" % fileKey
    f.close()
    return outputFile

def addSqlTraceToJira(location,ticket):
    """
    Add a SQL Trace file to the Jira Ticket
    :param location:
    :param ticket:
    :return:
    """
    entriesf = (os.path.join(location, fn) for fn in os.listdir(location))
    check_file = re.compile('trc$')
    check_ticket = re.compile(ticket)
    for file in entriesf:
        if os.path.isfile(file) and check_file.search(file) and check_ticket.search(file):
            attachFileToJira(file,ticket)
        else:
            print "No files found with %s" % ticket
def attachFileToJira(logfile,ticket):
    """
    Attach the output file to the Jira ticket
    :param logfile:
    :param ticket:
    :return:
    """
    url = "https://jira-test.fakedomain.comnet/rest/api/2/issue/%s/attachments" % ticket
    headers = {'X-Atlassian-Token':'no-check'}
    files = {'file': open(logfile, 'r')}
    auth = HTTPBasicAuth('username', 'password')
    # print "Sending %s to ticket %s" % (logfile,url)
    r = requests.post(url,headers=headers,files=files,auth=auth,verify=False)
    print 'Jira file attach sent me a %s with \n %s' % (r.status_code,r.text)

def addCommentToJira(keyword,ticket):
    """
    This attaches a comment about the search parameter that was added to the ticket
    :param keyword:
    :param tickiet:
    :return:
    """
    url = "https://jira-test.fakedomain.comnet/rest/api/2/issue/%s/comment" % ticket
    headers = {'X-Atlassian-Token': 'no-check'}
    auth = HTTPBasicAuth('username', 'password')
    comment = "Just added an attached log file about the results for the %s search." % keyword
    json = {"body": comment}
    r = requests.post(url,headers=headers,json=json,auth=auth,verify=False)
    print 'Jira adding a comment sent me a %s with \n %s' % (r.status_code, r.text)

def main():
    if len(sys.argv) < 5:
        from os.path import basename
        print ('Usage: %s location key type ticket' % basename(__file__))
    else:
        # fileLocation
        LOCATION = sys.argv[1]
        # searchKey
        SEARCHKEY = sys.argv[2]
        # type
        FILETYPE = sys.argv[3]
        # Jira Ticket
        TICKET = sys.argv[4]
        # File Keyword
        FILEKEY = sys.argv[5]
        #timeStart
        #TIMESTART = sys.argv[3]
        #timeend
        #TIMEEND = sys.argv[4]
    # logfile = openFile(LOCATION,SEARCHKEY,FILETYPE,FILEKEY)
    # attachFileToJira(logfile,TICKET)
    # addCommentToJira(SEARCHKEY,TICKET)
    addSqlTraceToJira(LOCATION,TICKET)
if __name__ == '__main__':
    main()

I claim some responsibility for it, mostly by hacking together a lot of python and doing a lot of TDD as I built out the functions.

Robot Framework and RESTful APIs

Been awhile.  Yup, it has.  Just a short one this time, more to get something down that vexed me for a little bit.

Dealing with a RESTful API that returned JSON in the body to my Robot Framework test caused me no end of grief.  Since the response always came back with the following:

\xef\xbb\xbf{

        "account_organizations": [

Originally I thought, it's just a JSON response, I can deal with it.

Oh no.

Not at all.

Turns out what I was getting ended up being byte order markers that were messing up the JSON compare I was trying to do.  Knowing what I was to get back was one thing, comparing that with the body of the response was another thing since the byte order markers did not behave like a regular list as I thought it was.  This turned out to be a dict, since Robot Framework basically is Python we have to be Pythonic here.

So, treating that as a dict we get the 3rd index from the list:

${l_response_body[3:]}

THAT now pulls the right values for comparison.

I also run this against a make unicode Python library, just in case.  So in Robot I use the following to cover it all:

${l_response_body} Get Response Body

${l_decoded_body} = make_unicode        ${l_response_body[3:]}

So all is good with the world, for now.

Testing Beliefs

This comes off of Pete Whalen's wonderful post on Testing and Wars of Religion.

Oftentimes we cling to our faiths in processes and procedures, too much in some cases.  It's hard to let go of something that you have worked hard to understand and struggled with for something new.  Change is hard.  Very hard.  But that is the point.  At some point beliefs and processes no longer work the same way and a new way to look at things is required.  Why?  Because just as work and processes change, new tools, languages and methods of achieving goals come and enter the environments we work in so to do these new things often challenge beliefs and require adjustment in how we look at things, or a new vision of the world.

Change is hard, but if it was easy then in many cases it would not be worth it.  I always look at them as educational opportunities, though even I challenge some of the changes that come up, its normal and once I get over my initial reluctance I can take a wide-view of things and move on.

Short, sweet and to the point.  Happy Monday!

The Power of PowerShell pipelining

Every now and again I come back to PowerShell, it's useful as a tool, simple, clean and while I can sometimes say its easy to use, many times I smack my head on the desk.  Working in a Windows environment PowerShell scripts make doing many things easier, though it usually takes me some time to work them out.

For example, I am trying to output some results from failures on running SSIS Tasks, but I only really need to know certain statuses, or in most cases just the failures.  Focusing on the failures there can be a lot of them, if something fails at the beginning EVERYTHING runs and can easily fill up the log.  So while a command like:

    $Results = &'C:\Program Files\Microsoft SQL Server\110\DTS\Binn\DTExec.exe' /Project $Path /Package "$File.dtsx"

Can give me a huge result set, do I really need to display a couple of hundred lines of errors to the console, or to the Test Driver?  No.

I wanted to only pull out the error details, I am skipping over the step where I know there are errors here (just to make it easy) and what I want are the details to notify someone that "Hey, something went wrong here!  Here are some examples, want to know more, log in to the box and do some research!!"

So at first I step through everything (that's how I understand where things are working) and first came up with this:

foreach ($_ in $Results)
{
if ($_ -match "Description")
{
$exec_errors += $_
}
}

Now that is way too long.  A lot of white space, plus do I really need to step through that much?  No I don't, thankfully PowerShell makes this easy, since $Results is an array, I can pipeline it and step through it to match only what I want:

    $Results|?{ $_ -match "Description" }

This gets me only the data I want to display, the rest of what you get for the error messages is meaningless to convey the details of what went wrong.  I pipeline the $Results array into ? which is shorthand for the Where-Object and within the curly braces put what it is I was looking before in my if statement, under my foreach.  So now instead of a 3 or so line statement (not quite counting the curly braces there) I have 1 line!  Yes, there can be only 1.

Still I need someplace to put those results:

    $exec_errors = $Results|?{ $_ -match "Description" }

Now, I have an array of just the details I want and using $exec_errors I can output only what I need to communicate:

Write-Output "Errors in execution: "
if ($exec_errors.length -ge 2) {
Write-Output "$exec_errors[0] `n $exec_errors.length - 1"
} else {
Write-Output $exec_errors
}

I will have to see about making a one-liner of the rest later on.

SSIS Task Variables

In some current Testing I am doing work with SSIS.  One task imports data into a database that we baseline for future Test Cases, but to get the data in I need to modify some of the dates.  With SSIS there is an option to adjust data using Derived Columns, with that it's possible to adjust data using Task Scoped Variables; I'll have to find a reference but using project scoped variables did not work initially.

Basically the variables were created this way:

1. Gave a descriptive name (need to have those so you know what to look for later!)
2. Scope, was selected to be the Task being worked on, which was easy since the project only had one task for data loading.
3. Data Type, since it was for Dates I made this DateTime
4. Value, just hit enter/return here and it took the current date and time
5. Expression, this is the real work here
1. For Now this was just left as the default - GetDate()
2. For a Year I made an expression - DATEADD("Year",1,(GetDate()))
3. Same adjustments for Adding Minute or Month, just adjust Year

When using these, in the Derived Columns select the Expression column and in the upper right panel there is a tree for Variables and Parameters.  Select the User: and this will update the value for that column when the Derived Columns task runs.

Learning about DateAdd came from the Microsoft Site