Accessibility Testing in Automation

 One of the new items I have been adding into the Automation Frameworks that I support, is the Axe Tool.  This does a check against page source to determine if the HTML structure is compatible with various assistive technologies.  This is something that can be added into a browser, and allows manual testing, but why do things manually if you can automate?

Automate or die!

That's been the mantra, well not die, but it's got to be serious.  After all why do things manually if you don't have it?  So to start we create a UI test to Login and go to a home page like the following (forgoing most of the basic Class setup, which I am assuming you, dear reader, know how to do - if not, just read on and learn what you can!)

@Test(description = "Verify Login Accessibility")
public void testLoginPageAccessibility() {
    logger.info("This is to test the a11y functionality with the AXE library.");
    LoginPage loginPage = new LoginPage(driver);
    try {
        loginPage.goToPage(new URL(getRunTimeConfig().getUrl()));
        User user = getUser(userId);
        logger.info("Checking page for " + user.getUsername());
        loginPage.login(user.getUsername(), password);
        logger.info("Getting the Accessibility Report for the Landing page.");
        getAxeReport(getCurrentMethodName());

The Current Method Name is the loginPage method, that is built upon the WebDriver GET so what is being sent to getAxeReport() is the current page being checked.

So let's look at how it generates the Report:

public void getAxeReport(String testMethodName) {

    logger.info("Checking for accessibility on page: " + driver.getCurrentUrl() + " for " + driver.getTitle());
    org.json.JSONObject responseJson = new AXE.Builder(driver, scriptUrl).setTimeout(180).analyze();
    org.json.JSONArray violations = responseJson.getJSONArray("violations");
    String pageName = driver.getTitle().trim().replaceAll(" +", "");
    String reportName = OUT_DIR + testMethodName + "/" + pageName + ".json";
    String rawViolations = OUT_DIR + testMethodName + "/" + "raw" + pageName + ".json";
    try {
        if (new File(OUT_DIR + testMethodName).mkdirs()) {
            try (Writer file = new OutputStreamWriter(new FileOutputStream(new File(reportName)),
                    StandardCharsets.UTF_8)) {
                file.write(cleanUpAxeJson(violations.toString()));
                logger.info("Successfully wrote Axe Report to File...");
            }
        } else {
            logger.error("failed to create directory");
        }
        if (violations.length() == 0) {
            Assert.assertTrue(true, "No violations found");
        } else {
            logger.debug("Found violations, generating raw report.");
            AXE.writeResults(rawViolations, responseJson);
        }
    } catch (FileNotFoundException e) {
        logger.debug("File Not Found exception: " + e);
    } catch (IOException e) {
        logger.debug("IO Exception: " + e);
    }

}

What's happening is the page URL is being obtained, and using the AXE JavaScript library against the source of the page, collect the results into a file.  From the results most of what we care about are the violations, and as this is all JSON, we just need to parse the JSON tree and save the parts that tell us what is wrong with the page.

Results we save locally so the Report can be stylized, or saved and reviewed later by people who know what violations are really a problem.

Easy peasy.