Selecting a test automation approach:
Following assessment is designed to help you evaluate where you stand in terms of your application, test team and test process. Based on the results, you can choose the automation approach that is right for your needs. Start by answering these questions:
1) What phase of development application is in?
- Planning
- Analysis
- Design
- Code
- Test
- Maintenance
2) What is the skillset of the test team?
- Primary technical
- Some technical, some non-technical
- Primarily non technical
3) How stable is the application?
- Stable
- Somewhat stable
- Unstable
Approach:
- Capture/Playback:
Profile:
Application in test or maintenance phase
Primarily non technical test team
Somewhat or not documented test process
Stable application
- Data driven:
Profile:
Application in code or early test phase
Some technical, some non technical test team
Well or somewhat documented test process
Stable or somewhat stable application
- Table driven:
Profile:
Application in planning, analysis or design
Some technical, most non technical test team
Well documented test process
Unstable or stable application
- With little prior planning, it is usually possible to migrate from one method to another as time and expertise permits.
Capture/Playback:
- Tests are performed manually while inputs and outputs are captured in the background.
- During subsequent automated playback, script repeats same sequence of actions to apply inputs and compare the actual responses to captured results, differences are reported as errors.
- Capture/playback is available for almost all automated test tools, although it may be implemented differently.
Notice that the inputs - selection from menus, radio buttons, list boxes, check boxes, and push buttons as well as text and keystrokes- are stored in the script.
Structure:
In order to allow capture/playback script recording to be distributed among multiple testers, common structure should be adopted.
- One script, one requirement:
Ideal structure is to have one script per requirement.
This also allows requirements to be distributed among multiple testers.
Name of the script can be used as cross reference to requirement and clearly indicate the content and
purpose of each script.
- Associate scripts by application areas:
Scripts can be packaged together into test suites that are related by common characteristics.
This makes it easier for single tester to focus on certain areas of the system, and simplifies later maintenance when changes are needed.
- Callable scripts:
If your tool does not have built in mechanism, you should consider making each script callable from another, so that when it completes it returns processing to the next instruction in the calling script.
A master or driver script can then be created which contains series of calls to the individual scripts for execution.
Advantages of Capture/Playback:
- One of the most earliest and and most common automated test approaches.
- Little training and setup time: It requires least training and setup time.
- Develop tests on the fly: can be defined on the fly by the test operator.
- Audit trail: Provides excellent audit trails for ad hoc or usability testing, in the event an error occurs, precise steps that created it are captured for later diagnosis.
Disadvantages of Capture/Playback:
- Requires manual capture
- Application must be stable
- Redundancy and omission
- Tests must be combined
- Lack of maintainability
- No logic means more tests fail
- Short useful script life
Data Considerations:
As capture/playback expects the same inputs to produce the same outputs, state of the data is critical.
- Static data:
Beginning state of the application database is essential to predictable results in any automated test method.
Assure that your test cycle contains steps to prepare the database to known state, either by refreshing it or populating it with known records.
- Dynamic data:
Application will generate data value dynamically that cannot be known in advance but must be used later during the test.
For ex, Unique ID is assigned to each new record as it is entered, which must be input later in order to access the record.
- Using variables:
It may be necessary to implement variable capability for dynamic field, so that the value can be retreived during playback and saved for later reference.
Data Driven:
Difference between capture/playback and Data driven is that in former case input and outputs are fixed, while in the latter the inputs and outputs are variable.
Following page contains example of the previous capture/playback script, modified to add an external file and replace the fixed values with variables.
Comments can be added for documentation:
Structure:
- One script, one process:
Data driven script is tied to single processing sequence but will support multiple test cases.
A sequence of steps that enters or processes data may include many test cases relating to individual
elements of the data or steps in the sequence
Select sequence of steps that require consistent set of data for each iteration, and name the script for process or application window it addresses.
- One record, one test case:
Each record in test data should relate to the single test case, and test case ID should be stored in data record.
This allows single script to process multiple test cases while logging results of each.
Test results are logged at each record, instead of at the end of the script.
- Data intensive:
Application is fairly data intensive, same steps tend to be repeated over and over with different data.
- Consistent behavior:
Same steps are repeated without significant variance, so different inputs do not have major impact on sequence of actions.
If value of one field causes a completely different processing path to take effect, then amount of logic required to process each test case increases exponentially.
Advantages of data driven approach:
- Create test cases earlier: Data driven allows test cases- inputs and expected outputs to be created in advance of the application.
- Flexible test case creation: As they are stored as data, set of inputs and outputs can be entered through spreadsheet, word processor, database or other familiar utility, then stored for later use by test script. Familiarity with test tool is not required for test cases.
- Reduced maintenance: This approach reduces required maintenance by not repeating the sequence of actions and logic to apply each test case.
Disadvantages of data driven approach:
It requires additional expertise in test tool and in data file management.
- Technical tool skills required: To convert script to process variable data, at least one of the testers must know test tool and understand the concept of variable values
How to implement external data files
Programming logic such as if/then/else expressions and processing loops
- Data file management needed: Depending on utility used, this may require expertise in creating and manipulating spreadsheet macros, database forms or word processor templates, then exporting data in the file compatible to the test tool.
Data consideration:
- One script, one file
- Using multiple files
- Dynamic data
Table driven:
- Sequence of actions to apply and evaluate inputs and outputs are stored external to the script.
- Test does not have to be performed manually at all.
- Inputs and expected outputs, as well as sequence of actions are created as data records.
Below is excerpt of the test script routines and test data files that would process same entry of chart of accounts:
Structure:
- Like Data-Driven, this method requires that the names of the application data elements be known, however it also requires that the type of object and the valid methods it supports also be defined and named, as well as windows and menus.
One file, multiple scripts:
- Single test data file in Table-Driven script is processed by multiple scripts.
Multiple records, one test case:
- Single test case is comprised of multiple records, each containing a single step.
- Test case ID should be stored in each data record to which it relates.
- This allows single set of scripts to process multiple test cases.
- Test results are logged at each step instead of at the end of the test case.
Advantages of Table Driven approach:
- Develop test cases, scripts earlier:
- Minimized maintenance:
- Portable architecture between application: Test library can be easily ported from one application to another.
- Portable architecture between tools: This approach is also portable between test tools.
- No tool expertise needed to create test cases: As logic is defined and stored only once per method, and there is substantial implied logic for verifying the context and state of the application to assure proper playback, individual testers may create test cases without understanding logic or programming. All needs is understanding of the application components, their names and valid methods and values which apply to them.
Disadvantages:
- Extensive technical and tool skills needed: Extensive programming and test tool expertise are needed.
- Data conventions and management critical:
Following assessment is designed to help you evaluate where you stand in terms of your application, test team and test process. Based on the results, you can choose the automation approach that is right for your needs. Start by answering these questions:
1) What phase of development application is in?
- Planning
- Analysis
- Design
- Code
- Test
- Maintenance
2) What is the skillset of the test team?
- Primary technical
- Some technical, some non-technical
- Primarily non technical
3) How stable is the application?
- Stable
- Somewhat stable
- Unstable
Approach:
- Capture/Playback:
Profile:
Application in test or maintenance phase
Primarily non technical test team
Somewhat or not documented test process
Stable application
- Data driven:
Profile:
Application in code or early test phase
Some technical, some non technical test team
Well or somewhat documented test process
Stable or somewhat stable application
- Table driven:
Profile:
Application in planning, analysis or design
Some technical, most non technical test team
Well documented test process
Unstable or stable application
- With little prior planning, it is usually possible to migrate from one method to another as time and expertise permits.
Capture/Playback:
- Tests are performed manually while inputs and outputs are captured in the background.
- During subsequent automated playback, script repeats same sequence of actions to apply inputs and compare the actual responses to captured results, differences are reported as errors.
- Capture/playback is available for almost all automated test tools, although it may be implemented differently.
Notice that the inputs - selection from menus, radio buttons, list boxes, check boxes, and push buttons as well as text and keystrokes- are stored in the script.
Structure:
In order to allow capture/playback script recording to be distributed among multiple testers, common structure should be adopted.
- One script, one requirement:
Ideal structure is to have one script per requirement.
This also allows requirements to be distributed among multiple testers.
Name of the script can be used as cross reference to requirement and clearly indicate the content and
purpose of each script.
- Associate scripts by application areas:
Scripts can be packaged together into test suites that are related by common characteristics.
This makes it easier for single tester to focus on certain areas of the system, and simplifies later maintenance when changes are needed.
- Callable scripts:
If your tool does not have built in mechanism, you should consider making each script callable from another, so that when it completes it returns processing to the next instruction in the calling script.
A master or driver script can then be created which contains series of calls to the individual scripts for execution.
Advantages of Capture/Playback:
- One of the most earliest and and most common automated test approaches.
- Little training and setup time: It requires least training and setup time.
- Develop tests on the fly: can be defined on the fly by the test operator.
- Audit trail: Provides excellent audit trails for ad hoc or usability testing, in the event an error occurs, precise steps that created it are captured for later diagnosis.
Disadvantages of Capture/Playback:
- Requires manual capture
- Application must be stable
- Redundancy and omission
- Tests must be combined
- Lack of maintainability
- No logic means more tests fail
- Short useful script life
Data Considerations:
As capture/playback expects the same inputs to produce the same outputs, state of the data is critical.
- Static data:
Beginning state of the application database is essential to predictable results in any automated test method.
Assure that your test cycle contains steps to prepare the database to known state, either by refreshing it or populating it with known records.
- Dynamic data:
Application will generate data value dynamically that cannot be known in advance but must be used later during the test.
For ex, Unique ID is assigned to each new record as it is entered, which must be input later in order to access the record.
- Using variables:
It may be necessary to implement variable capability for dynamic field, so that the value can be retreived during playback and saved for later reference.
Data Driven:
Difference between capture/playback and Data driven is that in former case input and outputs are fixed, while in the latter the inputs and outputs are variable.
Following page contains example of the previous capture/playback script, modified to add an external file and replace the fixed values with variables.
Comments can be added for documentation:
Structure:
- One script, one process:
Data driven script is tied to single processing sequence but will support multiple test cases.
A sequence of steps that enters or processes data may include many test cases relating to individual
elements of the data or steps in the sequence
Select sequence of steps that require consistent set of data for each iteration, and name the script for process or application window it addresses.
- One record, one test case:
Each record in test data should relate to the single test case, and test case ID should be stored in data record.
This allows single script to process multiple test cases while logging results of each.
Test results are logged at each record, instead of at the end of the script.
- Data intensive:
Application is fairly data intensive, same steps tend to be repeated over and over with different data.
- Consistent behavior:
Same steps are repeated without significant variance, so different inputs do not have major impact on sequence of actions.
If value of one field causes a completely different processing path to take effect, then amount of logic required to process each test case increases exponentially.
Advantages of data driven approach:
- Create test cases earlier: Data driven allows test cases- inputs and expected outputs to be created in advance of the application.
- Flexible test case creation: As they are stored as data, set of inputs and outputs can be entered through spreadsheet, word processor, database or other familiar utility, then stored for later use by test script. Familiarity with test tool is not required for test cases.
- Reduced maintenance: This approach reduces required maintenance by not repeating the sequence of actions and logic to apply each test case.
Disadvantages of data driven approach:
It requires additional expertise in test tool and in data file management.
- Technical tool skills required: To convert script to process variable data, at least one of the testers must know test tool and understand the concept of variable values
How to implement external data files
Programming logic such as if/then/else expressions and processing loops
- Data file management needed: Depending on utility used, this may require expertise in creating and manipulating spreadsheet macros, database forms or word processor templates, then exporting data in the file compatible to the test tool.
Data consideration:
- One script, one file
- Using multiple files
- Dynamic data
Table driven:
- Sequence of actions to apply and evaluate inputs and outputs are stored external to the script.
- Test does not have to be performed manually at all.
- Inputs and expected outputs, as well as sequence of actions are created as data records.
Below is excerpt of the test script routines and test data files that would process same entry of chart of accounts:
Structure:
- Like Data-Driven, this method requires that the names of the application data elements be known, however it also requires that the type of object and the valid methods it supports also be defined and named, as well as windows and menus.
One file, multiple scripts:
- Single test data file in Table-Driven script is processed by multiple scripts.
Multiple records, one test case:
- Single test case is comprised of multiple records, each containing a single step.
- Test case ID should be stored in each data record to which it relates.
- This allows single set of scripts to process multiple test cases.
- Test results are logged at each step instead of at the end of the test case.
Advantages of Table Driven approach:
- Develop test cases, scripts earlier:
- Minimized maintenance:
- Portable architecture between application: Test library can be easily ported from one application to another.
- Portable architecture between tools: This approach is also portable between test tools.
- No tool expertise needed to create test cases: As logic is defined and stored only once per method, and there is substantial implied logic for verifying the context and state of the application to assure proper playback, individual testers may create test cases without understanding logic or programming. All needs is understanding of the application components, their names and valid methods and values which apply to them.
Disadvantages:
- Extensive technical and tool skills needed: Extensive programming and test tool expertise are needed.
- Data conventions and management critical:
