Rubber Duck Debugging

 

Talk through the problem

Have you ever been stuck on something?

As you are talking about it or telling someone about it, suddenly the solution arrives or you get that eureka moment where you stop and realize you have the answer.  Even when writing out the issue you are working on to someone, then the answer comes as you are writing it out.  This happens to me a lot.  Sometimes just looking for something at work, or I am writing an email and send it, then I realize I needed to try some step. BOOM! The solution presents itself.

Talking it out

I used to just call it talking things out and leave it at that, then I find out that there is a name for this.

Rubber Duck Debugging!

I've never spent my time talking to inanimate objects, maybe myself but while I may be an object I am not inanimate, so while talking to myself or others I do get in this situation. At times its kind of embarrassing to send an email or message, then immediately follow it up with a never mind.

That sort of makes me feel like the old SNL character Roseanne Roseannadanna.

Never get stuck alone thinking

To me the whole point of it is to not get stuck on a problem and just sit there thinking and thinking and thinking on it.  Getting stuck on something small that distracts me from moving onto the solution.

Talk it out, step away, or speak to a duck if you have one. You never know, the answer may just suddenly appear.

Recipes for Success

There are multiple recipes for success in technology, and like chefs, they vary from place to place and person to person.

What works for one person won't for another, without some adjustments.

In many ways the Local Sourcing movement changed the ways in which cuisine was made, gone in some places were the Special of the Day (to go on a tangent that was often a way for kitchen's to get rid of stock that was getting old) to be replaced with a full seasonal menu.  Just like companies that use different technologies in different ways so the environment is different in what a Developer, Tester, or Manager needs to know in order to stay current or be ready to make the next step.

Recipes handed down in a family are based on a point in time where ingredients were local and available, some at only certain times of year so they were "in Fall we had" or "on this holiday my grandmother made...".  When shipping became more immediate and being able to send produce and meats long distances with a relative amount of freshness so more ingredients became available in wider times.  That removed the specialness of certain things that could only be found at certain times of the year, and how foodies looked only for certain ingredients at certain times because only THEN were they tasty.

Technology evolves but in companies with established practices there are certain things that remain in a specific way with people having to learn how to use that version of a software with another.  Change comes slow, and with new versions of Java, Python, and dare I say Perl (though who even uses that anymore...) at times you need to adjust how software is built and when bringing build recipes current there are adjustments that need to be made.  Some easy, some not.  Some complex, some not.  In the end its the result you want, and like a tasty dish, you want the build to be complete, whole, and workable.

Correlations are fun, and there is more to be done with cooking (a fave of mine) and software, but more on that later.  I have a special dish I need to get ready for....

Keep Moving Forward

 It's hard to keep current, especially with all the options out there, and stay on a single trajectory to both improve oneself and keep progress going on a career path.  It's no longer a single line that moves one up a traditional line of promotions to some sort of mythical Manager or VP or CEO position but often a zig zag of moving between hard and soft skills or between languages and tools that often don't lead to one place.

It's like a continually evolving buffet where tools, languages, processes, and management styles continually update or are reviewed, and reviewed again.  Deciding on what trajectory to take is difficult, especially if as usually happens, one tends to change jobs every couple of years.  Anecdotally people change jobs every 3 to 5 years, the time of the company man has been long gone.  Very few stay on at a company for decades working for one place, many have worked for company after company where they only thing they take with them are skills, knowledge, and experiences.

So what to do?  I take it in three steps on what I want to learn:

1. What am I doing now that I need to improve on?
2. What am I doing next that I need to know more about?
3. What is coming up that I need to understand?

This makes the choices fairly simple for me, as languages evolve and there are those I am using constantly then I want to keep up and improve my coding ability.  Being able to make code more concise, readable, or find new ways to be DRY (Don't Repeat Yourself) are what I look to be able to improve at the moment.  Keep current skills fresh and improved, move up to the next level, and that keeps my toolbox fresh and up to date.

Anything that may be coming up, some new tool, idea, or something the company is using is always a good option.  It's hard to implement things, or use things, you don't understand.  Becoming familiar with the next thing coming is a good way to be prepared, don't wait until it comes.  Take the time to figure out what it is, learn it, then plan for it.  Knowing about the next project makes planning much easier.

Along the same lines there are new projects that may be coming, in companies change doesn't happen in a vacuum.  People know, they are aware and plan for change.  As the Scout motto says - Be Prepared.  Which also works well for personal advancement, being ready for the next change or the next project is the best way to succeed and keep the time working on it down and being ready to jump in and start not only makes one look better, but also starts you on the path to success.

Keep moving forward, but also keep preparing yourself.

Accessibility Testing in Automation

 One of the new items I have been adding into the Automation Frameworks that I support, is the Axe Tool.  This does a check against page source to determine if the HTML structure is compatible with various assistive technologies.  This is something that can be added into a browser, and allows manual testing, but why do things manually if you can automate?

Automate or die!

That's been the mantra, well not die, but it's got to be serious.  After all why do things manually if you don't have it?  So to start we create a UI test to Login and go to a home page like the following (forgoing most of the basic Class setup, which I am assuming you, dear reader, know how to do - if not, just read on and learn what you can!)

@Test(description = "Verify Login Accessibility")
public void testLoginPageAccessibility() {
    logger.info("This is to test the a11y functionality with the AXE library.");
    LoginPage loginPage = new LoginPage(driver);
    try {
        loginPage.goToPage(new URL(getRunTimeConfig().getUrl()));
        User user = getUser(userId);
        logger.info("Checking page for " + user.getUsername());
        loginPage.login(user.getUsername(), password);
        logger.info("Getting the Accessibility Report for the Landing page.");
        getAxeReport(getCurrentMethodName());

The Current Method Name is the loginPage method, that is built upon the WebDriver GET so what is being sent to getAxeReport() is the current page being checked.

So let's look at how it generates the Report:

public void getAxeReport(String testMethodName) {

    logger.info("Checking for accessibility on page: " + driver.getCurrentUrl() + " for " + driver.getTitle());
    org.json.JSONObject responseJson = new AXE.Builder(driver, scriptUrl).setTimeout(180).analyze();
    org.json.JSONArray violations = responseJson.getJSONArray("violations");
    String pageName = driver.getTitle().trim().replaceAll(" +", "");
    String reportName = OUT_DIR + testMethodName + "/" + pageName + ".json";
    String rawViolations = OUT_DIR + testMethodName + "/" + "raw" + pageName + ".json";
    try {
        if (new File(OUT_DIR + testMethodName).mkdirs()) {
            try (Writer file = new OutputStreamWriter(new FileOutputStream(new File(reportName)),
                    StandardCharsets.UTF_8)) {
                file.write(cleanUpAxeJson(violations.toString()));
                logger.info("Successfully wrote Axe Report to File...");
            }
        } else {
            logger.error("failed to create directory");
        }
        if (violations.length() == 0) {
            Assert.assertTrue(true, "No violations found");
        } else {
            logger.debug("Found violations, generating raw report.");
            AXE.writeResults(rawViolations, responseJson);
        }
    } catch (FileNotFoundException e) {
        logger.debug("File Not Found exception: " + e);
    } catch (IOException e) {
        logger.debug("IO Exception: " + e);
    }

}

What's happening is the page URL is being obtained, and using the AXE JavaScript library against the source of the page, collect the results into a file.  From the results most of what we care about are the violations, and as this is all JSON, we just need to parse the JSON tree and save the parts that tell us what is wrong with the page.

Results we save locally so the Report can be stylized, or saved and reviewed later by people who know what violations are really a problem.

Easy peasy.

Looking for the next BIG THING!!

There is always the cool item on the horizon.  The next tool that YOU MUST HAVE!!  Something that your Team cannot do without, or that cool presentation at the conference that took you by the shirt and shook you until you could do nothing else but think about IT!!

It's something all too familiar to many of us.  We want to improve, we want to be current, no one wants to be left behind or fail at being competitive.  A lot of this is psychological, we're human and in some ways the tools, with savvy marketing behind them, know how to get inside your head and get you thinking about them.  It's not a bad thing, but falling for it without knowing is.  Don't just jump into something without checking it out first, spend time looking at those tools and seeing if they will work.  Talk to co-workers, do a POC, and make sure the fit is right.

A warm glove is just as good when made out of different materials, but if you need to use your fingers, a mitten won't work. Check the fit, ensure that what you innovate or enhance meets the needs of the Teams that will use it, and that the Teams have the background to readily accept the change.  Nothing gets a nose out of joint than a tool you are forced to use that doesn't meet all the needs of a Team.

When looking to make improvements I look at the following questions to start.

• Is this a programming language the Team already knows?
• Does the interface merge with the existing without a lot of retraining?
• Can you slip this into the builds and use it without people noticing?  The best updates are those invisible to the Users, they suddenly find their existing functionality works, and they have new options.  WIN!
• Does it fit the culture?  The worst changes require significant retraining.

When working with people, they have emotions, feelings, and wants same as you.  Ensure you meet them and people will accept what you offer, when you give them junk, expect complaints.

Building out Tomcat and MySQL with Docker

We have a Test Queuing and Archive server that runs on Tomcat, with the data stored in MySQL.  This currently runs on vm's in our lab, but as part of the process to move this to AWS I moved the manual setup for these to Docker images.  This gives us a few advantages:

1.  A long, multi-day wiki page install is distilled down into one docker image that builds in 5 minutes
2. We get a repeatable environment usable whenever we need to deploy, without worrying about underlying changes of configuration files
3. Being able to build a local Development version of the environment that can be brought up and used in minutes, rather than move to the Development vm and run some install scripts for an environment we barely use.  Updates to this system are rare.

Because I love automation I added some local shell scripts to check for running Docker images, containers, and added cleanup for previous builds.  In testing I was building these over and over and found I had at one point 5 GB of previous build images that I don't need.  The scripts also handle stopping the Containers so I don't have collisions in case I forget to check if the Containers are running, sometimes I don't check my laptop after the weekend to see what I might have left running.  I know, that's a bad habit, but this way I don't need to worry.

For my Local Development environment I know what IP I am going to start my Containers on, so I added an entry in my HOST file to be able to use a named instance to the Container when it comes up.  I like that better than worrying about trying to connect to an IP address.

As part of adding this into Jenkins I added parameters to the shell scripts to build for the environments I need, a Local Development environment needs MySQL the Production environment has an RDS DB running, so all I need for that is Tomcat.  This way all the configuration is set in the Dockerfile and I just build for the environment I want.

#!/bin/bash 

# Build script for Local Tomcat usable on local environments

BUILD='LOCAL'

show_help()

{

cat << EOF

    usage:

        $0 -b local     for local Thunder and MySQL Docker  (DEFAULT)

        $0 -b prod      for remote Thunder Docker image to connect to AWS RDS

EOF

}

while getopts "hb:" opt; do

    case $opt in

        h)

            show_help

            exit 0

        ;;

        b)

            BUILD=$OPTARG

        ;;

    esac

done

# WAR file and web server files

echo "We should always want to do a clean then a new build"

wget -O www/scripts/jquery.js code.jquery.com/jquery-1.11.1.js

cp www/index.html.template.templ www/index.html

./gradlew clean build

if [ -e build/libs/my.war ]then

    echo "my.war was built correctly"

else

    echo "my.war was not built, some kind of problem?"

    exit 1

fi

# Stop the existing images, if they are running

LOCAL=`docker ps -q --filter=ancestor=local`

PROD=`docker ps -q --filter=ancestor=prod`

MYSQL=`docker ps -q --filter=ancestor=mysql`

if [ ! -z "$LOCAL" ]; then

    echo "Stopping running Local instance"

    docker stop local

fi

if [ ! -z "$PROD" ]; then

    echo "Stopping running Prod instance"

    docker stop prod

fi

if [ ! -z "$MYSQL" ]; then

    echo "Stopping MySQL instance"

    docker stop mysql

fi

# Let's just be cleanly and remove all those old dangly images

docker system prune -f

# Create a Docker image

if [ "${BUILD}" == 'LOCAL' ]then

    echo -e "Running a $BUILD build"

    # Build all for local but only need Tomcat for Prod

    docker build -t mysql -f deployment/docker/local/mysql/Dockerfile .

    docker build -t local -f deployment/docker/local/tomcat/Dockerfile .

else

    echo -e "Running a $BUILD build"

    # Prod version should be prod name, don't need a database

    docker build -t prod -f deployment/docker/prod/tomcat/Dockerfile .

fi

# Cleanup the files we only need for the war file

rm www/index.html

rm www/scripts/jquery.js

For Tomcat I have some configuration to do with the context and server files, that will vary for environment, but what I especially needed was a way in Docker to build up a self-signed certificate without manual intervention.  After some searching around and trial/error I was able to come up with the commands that I needed to build that.

FROM tomcat:7.0-jre8

# Install useful toolsRUN apt-get update && \

    apt-get -y upgrade && \

    apt-get install -y wget vim curl procps tar libssl-dev --fix-missing --no-install-recommends

# Configure Tomcat Container

COPY build/libs/my.war /usr/local/tomcat/webapps/

COPY deployment/docker/local/resources/local-server.xml /usr/local/tomcat/conf/server.xml

COPY deployment/docker/local/resources/local-context.xml /usr/local/tomcat/conf/context.xml

COPY deployment/docker/local/resources/local-environment.xml /usr/local/tomcat/conf/environment.xml

COPY deployment/docker/local/resources/local-testserver.xml /usr/local/tomcat/conf/testserver.xml

# MySQL Connector

WORKDIR /usr/local/tomcat/lib/

RUN curl -L -o mysql-connector-java-5.1.46.tar.gz https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

# Add in untar of the mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar

RUN tar -xf mysql-connector-java-5.1.46.tar.gz mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar

RUN cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar .

# Configure Tomcat User and GroupENV RUN_USER    tomcat

ENV RUN_GROUP   tomcat

RUN groupadd -r ${RUN_GROUP} && useradd -g ${RUN_GROUP} -d ${CATALINA_HOME} -s /bin/bash ${RUN_USER} && \

    chown -R ${RUN_USER}:${RUN_USER} $CATALINA_HOMERUN chown -R ${RUN_USER}.${RUN_USER} /data/RUN chown -R ${RUN_USER}.${RUN_USER} /usr/local/resource-manager/

# Setup SSL for Tomcat

WORKDIR /usr/local/tomcat/conf/RUN $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA -storepass changeit -keypass changeit -noprompt -dname "CN=local.test.server, OU=Central, O=MYCOMP, L=BOSTON, S=Massachusetts, C=US" -keystore "/usr/local/tomcat/conf/.keystore"

RUN openssl req -newkey rsa:2048 -x509 -subj "/C=US/ST=Massachusetts/L=BOSTON/O=MYCOMP/CN=local.test.server" -keyout cakey.pem -out cacert.pem -passout pass:mypass

RUN openssl pkcs12 -export -in cacert.pem -inkey cakey.pem -out identity.p12 -name "local" -passin pass:mypass -password pass:mypass

RUN keytool -importkeystore -destkeystore identity.jks -deststorepass mypass -srckeystore identity.p12 -srcstoretype PKCS12 -srcstorepass mypass

RUN keytool -import -file cacert.pem -keystore trust.jks -storepass mypass -noprompt

WORKDIR /usr/local/tomcat/

USER tomcat

For the Production build I need an actual certificate, but since that environment is not fully ready I don't need to import that yet.

There is a shell script that gets the Local Development environment running, using Docker-Compose I am able to get the environment I need up and running.  Again the shell script has a path for Local and Production environments setting them up appropriately.

#!/bin/bash

BUILD='LOCAL'

show_help()

{

cat << EOF

    usage:

        $0 -b LOCAL     for local Thunder and MySQL Docker  (DEFAULT)

        $0 -b PROD      for remote Thunder Docker image to connect to AWS RDS

EOF

}

while getopts "hb:" opt

do

    case "$opt" in

        h)

            show_help

            exit 0

        ;;

        b)

            BUILD=$OPTARG

        ;;

    esac

done

# Stop the existing images, if they are running

LOCAL=`docker ps -q --filter=ancestor=local`

PROD=`docker ps -q --filter=ancestor=prod`

MYSQL=`docker ps -q --filter=ancestor=mysql`

if [ ! -z "$LOCAL" ]; then

    echo "Stopping running Local instance"

    docker stop local

fi

if [ ! -z "$PROD" ]; then

    echo "Stopping running Prod instance"

    docker stop prod

fi

if [ ! -z "$MYSQL" ]; then

    echo "Stopping MySQL instance"

    docker stop mysql

fi

# Check for Local or Prod build then use the right directories

if [ "${BUILD}" == 'LOCAL' ]then

    echo -e "Starting a $BUILD environment"

    docker-compose -f deployment/docker/local/tomcat.yaml --verbose up -d

else

    echo -e "Starting a $BUILD environment"

    docker-compose -f deployment/docker/prod/tomcat.yaml --verbose up -d

fi

The YAML file I set for the environment gets the Containers up and running with the settings I needed.

version: '2.2'

services:

    tomcat:

        container_name: tomcat

        image: local:latest

        environment:

              CATALINA_OPTS: "-server -Xms1024m -Xmx2048m -XX:PermSize=1024m -XX:MaxPermSize=1536m -Dhost_default_domain=tlocal.test.server -Djava.net.preferIPv4Stack=true -agentlib:jdwp=transport=dt_socket,address=1043,server=y,suspend=n"

        ports:

            - "127.0.0.1:80:8080"

            - "443:8443"

        restart: on-failure

        links:

            - mysql

    mysql:

        container_name: mysql

        image: mysql:latest

        ports:

          - 3306:3306

        environment:

          MYSQL_DATABASE: testdb

          MYSQL_USER: testadmin

          MYSQL_PASSWORD: testpass

          MYSQL_ROOT_PASSWORD: testpass

        mem_limit: 1000000000

With that I have my Tomcat and MySQL Containers up and running, I can point some local tests to the Test Server URL and have my tests run against and archive their runs locally.  This gives me flexibility in testing changes as I can do my code locally build my Docker environment and test without worrying about checking into the repository and checking out in another environment that may or may not be in use.

Pull Requests: How the Process Can Get Stuck

At my current workplace we have the usual Pull Request process:

• Open a branch
• Do work
• Commit all my work
• Push to the branch
• Previous steps...ad nauseum
• When ready open a PR!

A Pull Request can be either a Review or a Merge.  A Pull Request I will heretofore refer to as the PR.

I do Reviews at times, to assure that the work fits everything we are doing and usually its when trying something new, novel, or dangerous.  Once there is some sign off on the Review, and we try to turn these around in < 48 hours, then it becomes a Merge or we generate a new one as a Merge to get all the updates and changes (if any) in the repo.

Merges are fairly straightforward, get some comments or review on the Code.  Does it fit the style guides, are core files updated properly and cross-tested with other peoples Code so we don't have regression failures in other areas.  Once all that is done, again we turn these around quick, or try to, then the merge happens and we go on our way.

Occasionally we have things that throw lots of wrenches in the process.

Big PRs.  I mean HUGE!  Something that touches codes and libraries all over the place, these tend to be rare or due to some major functional change.  A PR like this can take days to review and go over, sometimes with a lot of back and forge, merge conflicts due to others that have been merged along the way.  They become a huge mess and while rare can rear up like a nightmare that sits in the queue for a long period of time until it is finally ready and gets put in, with a little hesitation on that merge button because you just KNOW that within a day something will break and there will be follow-up.  Though usually that's natural, you just hope the follow-up with be slight.

PRs that get opened due to people doing work that will stretch over time, are waiting for core functionality to be finished in another area, or its a huge group effort.  These are the ones that irk me the most because they should never be opened that early in the first place, at least to me.  When I see them I can feel my breath stop and I just know I need a little walk before I continue, they just irk me to no end.

With a global company where people live and work in different areas, and different schedules, it feels like this is the best way to communicate things.  With all of the other tools at our disposal (Teams, Slack, HipChat, Email [remember that one?]) sometimes people think this is the best communication mechanism.  I may be in the minority and disagree, that you can work on a branch with different people if needed, or that branch may need to wait, but there is no need to open it so early, its not ready there is nothing here more than a sign that pops up saying "review me, there was yet another checkin!".  Diplomatically I mention to people there are other mechanisms, but so far its been like tilting at windmills.  Though I dream some day people will listen.

Vacation Days!  Yes, there are even the PRs where the owner opens in the final hours of a Friday afternoon and submits everything, then goes on a well deserved break for the next week and won't be checking emails.  These sit around, with people wondering if the feedback will be looked at, or if that one Unit Test that failed is going to be resolved.  Not much you can do, but wait for them to get back and after catching up, get back to the PR.

Just like a Vacation Day the author can get swamped with something new, or a higher priority task, so the PR sits calmly in queue with comments and questions, which the author decides to get back to but doesn't really notify anyone.  So it sits waiting.

Most of this can be solved with good communication, but at times its just "the way things are".

Personally I think we can always do better, I have a few areas I need to work on constantly so I am a work in progress myself, but I like the see PRs be done quick, easy, and get out of my queue.

Especially when my manager is monitoring that queue and keeps wondering why there is one there.

Shh...it might be sleeping.