Operational Quality

In order to make sure that our production site is running properly, twice a week I do a check on the live site.  Mostly what I want to see is that the site is responding to search crawls, since our content needs to be available for queries by Users who come looking for specific topics.  I do a check to make sure current content is being crawled and indexed.  We don't want a stale index, and since the crawls run incrementally each night I check that the index is up to date.

Coming up with a name for this was hard at first since its not really maintenance but its not really an Acceptance Test in the strict definition of the word, if there is one in Software Testing, since the site is live and there are no changes.  I asked a bit in the Software Quality Assurance and Testing Stack Exchange and received the great suggestion of Operational Quality from a fellow poster.

So what does this comprise?

• I reuse existing SpecFlow and Selenium Web Driver tests in my Test Framework to query against the index and make sure I get results, checking the pages for terms I know need to be in the result set
• Query the SharePoint a backup of the content database, I don't want to interfere with production, for what has been added recently and look for those pages in the index
• Log in to the SharePoint farm Administration server and check the Search Service Application for Crawl Log errors.  There typically are some false negatives for access denied, but I have become accustomed to what to ignore over time (isn't that always the case?)
• Run some search queries on topics I know are in the site content or just want to be sure still come up

Twice a week I get a calendar reminder, to make sure I have time for this, to do the work.  I have made it easier for myself over time, which is something I try to do in a way on continual improvement, so it takes me maybe more time to log in to all the servers I need to check than for me to do the actual work.