Test Automation: "Hero" for Agile projects

In times where customers and competitors demand a shorter "time to market" every day, it is more important than ever to make sure that all the processes and activities in the software development cycle evolve and adopt the tools and alternatives existing in the market to optimize the effort/resources that they demand.

In the agile cycle, the testing process is one of critical activities due to its timing (ending the sprint) within the project. When issues are found at this time, it directly affects the project schedule and there is usually not enough time to react or find an appropriate alternative to address the issue. If you want to save/recover your time, test automation is the "Hero" you have to ask for. It can "save" the project.

There are many solutions that can be easily incorporated to the testing process, according to the project features (budget, schedule, resources).

However, the reality is that is not uncommon to find projects that fail when try to incorporate the Test Automation to their testing processes. There are many enemies to defeat in the Test Automation road:

HOW (Tool)

The critical factors to consider when selecting a test automation tool are its learnability and support. Whether you go with a commercial or an open source tool, you have to research other successful implementations and analyze its pros and cons. You have to look for a solution that can be easily adapted and implemented, so create and maintain your test automation suite will not become your kryptonite.

Do's:

• Self-checking*: Your choice (tool) must auto-generate a report for every test run.
• Research for others' experiences.
• Clear*: The test script code must be readable and understandable.
• Maintainable*: Needs to be easy maintained or reused for other team members.

Don'ts:

• Commercial does not always mean it will be properly supported.
• If "X" tool works for project "A", that does not necessarily mean it will for "B".
• Think that the same test automation tool will be compatible with all the technologies (development languages, browsers, operating systems).

WHO (Skill)

The big difference between a "Test Analyst" and a "Test Automation Analyst" is SKILL. Many times "the functional testing team" is forced to become "the functional + test automation team", resulting in an automated test case that does not completely (or even partially) cover the functionality to test (traceable*).

It is important to look inside the team for the appropriate resources for automation tasks. A strong background in Functional testing + Development/scripting is a must.

Do's:

• Fluent communication between "testing team" and "test automation team" to avoid duplicated test cases (necessary*).

Don'ts:

• Assume that a "good test analyst" will become a “good test automation analyst”.
• Assume that a "good developer" will become a “good test automation analyst”.

WHAT (Effort - Cost/Benefit)

Let's face it, trying to automate all the project test cases is as complex as automating all your system users' behaviors. There is a thin line between where the automation cost is less expensive than its advantages and sometimes, that line can be difficult to find.

The selection of the test cases to automate must focus on high effort test cases, not quantity. For example, some simple test cases could be frequently executed, so its related effort is higher.

Do's:

• Concise*: Start working on the test cases that are quick and easy to record/run (Note: Only if they are going to be frequently executed).
• Test Cases that have to be evaluated with different data combinations are good candidates.
• Regression test cases (going to give back your investment).

Don'ts:

• Projects with constant UI changes are definitively a non-candidate for test cases.
• On your latest sprints, try to add new complex (record) test cases.
• Independent*: Features dependent on modules/components that will be changed on next sprints must be carefully selected.
• Include test cases that required human intervention pre-requisites.
• Consider test cases dependent on others.

WHEN (Time)

How to know when to start? … How to know when to stop? ... These are two major questions to answer if you want to be successful in your test automation process. Even the test automation record activities does not start early in the project, there are some critical tasks as test automation planning and test cases selection that need to be performed at the project start.

Do's:

• The answer to the first question is as soon as it is possible (parallel to the manual testing).
• Start planning and selecting the test cases to automate during the manual test cases writing (usually first sprint).
• Include a task for test cases maintenance, remember that your test cases (automated) will be constantly executed (repeatable*). If this (maintenance) becomes a high consumption activity, it is a clear indicator that you are in the wrong way.
• The final task is to validate that your test scripts are still working properly and store them in your test cases library.

Don'ts:

• Start test recording activities if the UI is not stable enough or it is expected big changes in the future sprints.
• Try to add/automate high complex test cases on finals sprints.

References:

Test Automation Manifesto*.
http://xunitpatterns.com/~gerard/xpau2003-test-automation-manifesto-paper.pdf

Effective Test Automation in an Agile Environment
http://agile.techwell.com/articles/weekly/effective-test-automation-agile-environment