Testing would parallel systems development life cycle for the application. This cycle is generally depicted as:
Test team:
- To automate your testing you need to assemble the dedicated test team and obtain the assistance of other areas in the company.
- Developing test scripts is essentially form of programming.
- For this role, more technical background is needed.
- Team leader must have the control over the workflow of the team members.
Team leader:
- Responsible for developing the test plan and managing team members according to it.
- Coordinating with other areas to accomplish the team effort.
- Assign duties and control the workflow of those who are dedicated to the test team.
Test Developers:
- Responsible for developing the test cases, executing them, analyzing and reporting the results
- Develop tests, whether as data records or as scripts, and use the test framework.
Script developers:
- Experts in the testing tool, ideally with technical programming experience
- Developing and maintaining test framework.
Test Librarian:
- Responsible for managing configuration, change and version control for all elements of the test library.
Customer Liaison:
- Represents the user community of application under test and is responsible for final approval of the test plan or any changes to it.
- Working with test developers to identify test cases and gather sample documents and data.
Development Liaison:
- Responsibility to Test team to assure the software is properly unit tested and delivered in known state to the Test Team.
Systems Liaison:
- Represent the system or network support group and database administrator.
- Responsible for supporting test environment to assure that the test team have access to the proper platform configuration and database for test execution.
Test Automation Plan:
- Describes the steps needed to automate testing.
- Outlines necessary components of test library, resources which will be required, schedule, and entry/exit criteria for moving from one step to next.
This section is used to control additions and changes to the plan.
Keeping track of changes is important.
- Describe the application under test in this section.
- Be sure to specify the version number.
- Statement of the scope is as important as what will be tested and what will not be, as well as who will be responsible.
- List the names and roles of test team members and cross reference each of the steps to the responsible parties.
Schedule:
Planning test Cycle:
- Test cycle must be carefully planned to minimize the amount of supervision or interaction required.
- Execution cycle should be capable of automatically preparing and verifying test environment, executing test suites or individual tests in sequence.
Test Suite Design:
- Test suite is the set of tests that are related, either by their function or by area of application they impact and which are executed as a group.
- Not only set of tests but their sequence within the suite should be considered.
Related tests:
- Test suite usually contains tests that are related by the area of the application they exercise.
Context:
- All tests in the suite should share the same beginning and ending context, as well as expected state of the database.
Documentation:
- Include the set and sequence of the individual tests, beginning and ending context, as well as any data or sequence dependencies with other test suites.
Test Cycle Design :
Different types of test cycles needed:
- regression cycle that exercises entire test library or
- fix cycle that tests only targeted areas of the application where changes have been made.
Setup:
- Verifying configuration and all other variables that affect test execution.
- Test cycle executed against the wrong platform configuration may be worthless.
Context:
- Beginning and ending context of the cycle should be same point, usually command prompt or program manager.
- Initialize state of the database or other data elements.
Schedule sequence:
- Test schedule is often comprised of set of test cycles.
- Sequence should reflect any dependencies of either context or data, and standard tests.
- Test schedule template may be useful for assuring that all standard tests and tasks are included for each run.
Cleanup:
- Cycle should end with the cleanup of the test environment.
- Deleting work files, making file backups, assembling historical results, any other housekeeping tasks.
Test Execution:
- Must thoroughly document results.
- Documentation should be sufficient to determine which test passed or failed, what performance was, diagnosis of failure.
Test Log:
- reports results of test execution for each test case.
Pass/Fail:
- Each individual test case, whether data records or scripts, should be logged as to whether it executed successfully or not.
- Keep track of high priority requirements.
Performance:
- Many service level agreement maximum allowable response time, and/or expected average, for given areas of the system.
- Performance measurements may also include the overall time required to execute certain functions, such as file update or other batch process.
Configuration:
- Each test log should clearly indicate the configuration against which it was executed.
Totals:
- Total test cases executed, passed and failed, as well as elapsed time overall, should be provided at the end of the execution log to simplify the updating of historical trends.
Error log:
- For every test that fails, there should be corresponding entry in the error log.
- Provides more information about the test failure to support diagnosis of the problem and determination about its cause.
Test case and script:
- Test case which failed and script being executed should be documented in the error log to enable later review of error condition.
State of the application:
- When error occurs, document actual state of the application for comparison against what was expected.
- This may include snapshot of screen at time of failure, date and time, test case being executed, expected result, and whether recovery was attempted.
Diagnostics:
- Error log should include as much diagnostic as possible about state of appliation as is available: stack or memory dumps, file listings, and similar documentation.
Analyzing results:
- At the conclusion of each test case, test results in form of execution, performance and error logs must be analyzed.
- Automated testing may yield results which are not necessarily accurate or meaningful.
- For ex: execution log may report hundreds of errors, but close examination may reveal that an early, critical test failed which in turn jeopardized integrity of the database for all subsequent tasks.
Inaccurate results:
- Test results do not accurately reflect state of the application.
- Three types:
1) False failures
2) Duplicate failures
3) False successes
- False failure from test environment:
Fails for reason other than error or defect in application
Reasons:
- Test environment is not properly configured.
- Different error has cause the test to loose context
- False failure from application changes:
New field or control is added , causing script to get out of context and report failures for other fields.
- False failures from test errors:
Failure may be result of an error in the test itself.
There may be missing test case record or error in the script itself.
- Duplicate failure:
This failure is attributable to same cause as another failure.
For ex: if heavily used transaction window has an error, this error may be reported for every transaction that is entered into it.
If there are five hundred transactions, there will be five hundred errors reported.
- False success from test defects:
False success occurs when test fails to verify one or more aspects of the behavior, thus reporting the test was successful when in fact it was not.
If the test completes too quickly, it might indicate that it did not execute properly.
- False success from missed errors:
Another false success might occur if the test is looking only specific response, thus missing an incorrect response that indicates an error.
Defect tracking:
- Each reported defect should be given a unique identifier and tracked as to the test case that revealed it, date it was logged as defect, developer it was assigned to, and when it was actually fixed.
Test team:
- To automate your testing you need to assemble the dedicated test team and obtain the assistance of other areas in the company.
- Developing test scripts is essentially form of programming.
- For this role, more technical background is needed.
- Team leader must have the control over the workflow of the team members.
Team leader:
- Responsible for developing the test plan and managing team members according to it.
- Coordinating with other areas to accomplish the team effort.
- Assign duties and control the workflow of those who are dedicated to the test team.
Test Developers:
- Responsible for developing the test cases, executing them, analyzing and reporting the results
- Develop tests, whether as data records or as scripts, and use the test framework.
Script developers:
- Experts in the testing tool, ideally with technical programming experience
- Developing and maintaining test framework.
Test Librarian:
- Responsible for managing configuration, change and version control for all elements of the test library.
Customer Liaison:
- Represents the user community of application under test and is responsible for final approval of the test plan or any changes to it.
- Working with test developers to identify test cases and gather sample documents and data.
Development Liaison:
- Responsibility to Test team to assure the software is properly unit tested and delivered in known state to the Test Team.
Systems Liaison:
- Represent the system or network support group and database administrator.
- Responsible for supporting test environment to assure that the test team have access to the proper platform configuration and database for test execution.
Test Automation Plan:
- Describes the steps needed to automate testing.
- Outlines necessary components of test library, resources which will be required, schedule, and entry/exit criteria for moving from one step to next.
This section is used to control additions and changes to the plan.
Keeping track of changes is important.
- Describe the application under test in this section.- Be sure to specify the version number.
- Statement of the scope is as important as what will be tested and what will not be, as well as who will be responsible.
- List the names and roles of test team members and cross reference each of the steps to the responsible parties.
Schedule:
Planning test Cycle:
- Test cycle must be carefully planned to minimize the amount of supervision or interaction required.
- Execution cycle should be capable of automatically preparing and verifying test environment, executing test suites or individual tests in sequence.
Test Suite Design:
- Test suite is the set of tests that are related, either by their function or by area of application they impact and which are executed as a group.
- Not only set of tests but their sequence within the suite should be considered.
Related tests:
- Test suite usually contains tests that are related by the area of the application they exercise.
Context:
- All tests in the suite should share the same beginning and ending context, as well as expected state of the database.
Documentation:
- Include the set and sequence of the individual tests, beginning and ending context, as well as any data or sequence dependencies with other test suites.
Test Cycle Design :
Different types of test cycles needed:
- regression cycle that exercises entire test library or
- fix cycle that tests only targeted areas of the application where changes have been made.
Setup:
- Verifying configuration and all other variables that affect test execution.
- Test cycle executed against the wrong platform configuration may be worthless.
Context:
- Beginning and ending context of the cycle should be same point, usually command prompt or program manager.
- Initialize state of the database or other data elements.
Schedule sequence:
- Test schedule is often comprised of set of test cycles.
- Sequence should reflect any dependencies of either context or data, and standard tests.
- Test schedule template may be useful for assuring that all standard tests and tasks are included for each run.
Cleanup:
- Cycle should end with the cleanup of the test environment.
- Deleting work files, making file backups, assembling historical results, any other housekeeping tasks.
Test Execution:
- Must thoroughly document results.
- Documentation should be sufficient to determine which test passed or failed, what performance was, diagnosis of failure.
Test Log:
- reports results of test execution for each test case.
Pass/Fail:
- Each individual test case, whether data records or scripts, should be logged as to whether it executed successfully or not.
- Keep track of high priority requirements.
Performance:
- Many service level agreement maximum allowable response time, and/or expected average, for given areas of the system.
- Performance measurements may also include the overall time required to execute certain functions, such as file update or other batch process.
Configuration:
- Each test log should clearly indicate the configuration against which it was executed.
Totals:
- Total test cases executed, passed and failed, as well as elapsed time overall, should be provided at the end of the execution log to simplify the updating of historical trends.
Error log:
- For every test that fails, there should be corresponding entry in the error log.
- Provides more information about the test failure to support diagnosis of the problem and determination about its cause.
Test case and script:
- Test case which failed and script being executed should be documented in the error log to enable later review of error condition.
State of the application:
- When error occurs, document actual state of the application for comparison against what was expected.
- This may include snapshot of screen at time of failure, date and time, test case being executed, expected result, and whether recovery was attempted.
Diagnostics:
- Error log should include as much diagnostic as possible about state of appliation as is available: stack or memory dumps, file listings, and similar documentation.
Analyzing results:
- At the conclusion of each test case, test results in form of execution, performance and error logs must be analyzed.
- Automated testing may yield results which are not necessarily accurate or meaningful.
- For ex: execution log may report hundreds of errors, but close examination may reveal that an early, critical test failed which in turn jeopardized integrity of the database for all subsequent tasks.
Inaccurate results:
- Test results do not accurately reflect state of the application.
- Three types:
1) False failures
2) Duplicate failures
3) False successes
- False failure from test environment:
Fails for reason other than error or defect in application
Reasons:
- Test environment is not properly configured.
- Different error has cause the test to loose context
- False failure from application changes:
New field or control is added , causing script to get out of context and report failures for other fields.
- False failures from test errors:
Failure may be result of an error in the test itself.
There may be missing test case record or error in the script itself.
- Duplicate failure:
This failure is attributable to same cause as another failure.
For ex: if heavily used transaction window has an error, this error may be reported for every transaction that is entered into it.
If there are five hundred transactions, there will be five hundred errors reported.
- False success from test defects:
False success occurs when test fails to verify one or more aspects of the behavior, thus reporting the test was successful when in fact it was not.
If the test completes too quickly, it might indicate that it did not execute properly.
- False success from missed errors:
Another false success might occur if the test is looking only specific response, thus missing an incorrect response that indicates an error.
Defect tracking:
- Each reported defect should be given a unique identifier and tracked as to the test case that revealed it, date it was logged as defect, developer it was assigned to, and when it was actually fixed.
No comments:
Post a Comment