Why to automate?
- Three major benefits:
1) Cumulative coverage to detect errors and reduce cost of failure.
2) Repeatability to save time and reduce cost to market.
3) Leverage to improve resource productivity.
1) Cumulative coverage:
- Applications change and gain complexity over their useful life.
- Feature set of the application grows steadily over time.
- Thus, number of tests needed for adequate coverage is also constantly increasing.
- Just a 10% code change still requires that 100% of the features be tested.
- Test coverage constantly decline over time unless you increase the test resources and cycle time.
- Automation helps to accumulate test cases over life of an application so that both existing and new
features can always be tested.
2) Repeatability to save time and reduce cost to market.
Faster time to market:
- Automation can help reduce time to market by allowing test execution to happen 24*7.
- Once the test library is automated, execution is faster and run longer than manual testing.
- This benefit is only available when your tests are automated.
Reduce cost:
- Automation can reduce cost of failure by allowing increased coverage so that errors are uncovered
before they have chance to do real damage in production.
3) Leverage to improve resource productivity.
- By generating tests cases programmatically, we can yield thousands or more- when only hundreds might be possible with manual resources.
When not to automate:
- Unstable design: (Ex: weather mapping system or one that needs real time data does not give predictable results for automation).
- Inexperienced testers: If person automating the test are not sufficiently experienced with the application to know the expected behavior, automating their test is also of doubtful value. Save automation for the experts.
- Temporary testers: Let temporary resource(consultant or users) handle the manual test support while permanent staff handles automation.
- Insufficient time, resources: In case of lack of time, dont expect tool to help you. Look at automation for the longer term. Keep in mind that automation is strategic solution, not short term fix.
How not to automate:
- Test automation tools are just specialized programming languages.
- Developing an automated test library is development project requiring commensurate skills.
Automation is more than capture/replay:
- Although it is most commonly recognized technique, capture/replay is not the most successful approach.
Dont write a program to test a program:
- Developing an automated test library that provides comprehensive coverage would require more
code than exists in the application itself.
- This is because test must account for positive, negative and invalid cases for each feature or function.
Automation is more than test execution:
- You are going to build application that automates your testing, more than just running the tests.
- We need complete process and environment of creating and documenting tests, managing and maintaining them, executing them and reporting the results, as well as managing test environment.
Duplication of efforts:
- If you just hand out automation tool out to individual testers and command that they automate their
tests, each one of them will address all of these issues in their own unique way.
- This leads to the tremendous duplication of effort and can cause conflict when tests are combined.
Need for the framework:
- Approach automation of testing just as you would automation of any application.
- This framework should make the test environment efficient to develop, manage and maintain.
- We will discuss how to develop a framework and select best automation approach.
Setting realistic expectations:
- Test automation is strategic:
If test process is in crises and management wants to throw money at a tool to fix it, dont fall for it.
Test automation is long term, strategic solution, not short term band aid.
Not everything can be automated:
- Allow time for tasks that cannot be automated.
- You still need to:
1) Gather test requirements
2) Define test cases
3) Maintain your test library
4) Administer the test environment
5) Review and analyze test results.
On ongoing basis, there is need to add new test cases based on enhancements or defects, so that your
coverage can constantly be improving.
Accept gradual progress:
- If you cannot afford time in short term, then do your automation gradually.
- Target those areas where you get biggest paycheck first, then reinvest time savings in additional areas, till you get it all automated.
Primary goal of automation should be to increase test coverage, not to cut testing costs.
Plan for long term:
- Test automation project will last as long as application under test is being maintained.
- Achieving automation is not sprint, it is a long distance run.
Terminology:
1) Requirement:
- "What" of a system.
- Performance requirement spells out the volume and speed of the application, such as maximum
acceptable response or processing time and highest number of simultaneous users.
2) Test:
- Combination of test case and test script.
3) Test case:
- Set of inputs and expected application response that will confirm that requirement has been met.
- Test case may be stored as one or more data records, or may be stored within test script.
4) Test Script:
- Series of commands or events stored in a script language file that execute test case and report results.
- Create multiple possible pathways.
- Contain logical decisions that affect the execution of the script.
5) Test cycle:
- It is set of individual tests that are executed as a package in particular sequence.
- Cycles are usually related to application operating cycles, or by their priority or content.
- We have build verification cycle to establish acceptance of new software build.
- Regression cycle to assure that previous functionalities has not been disrupted by changes or new features.
6) Test Schedule:
- series of test cycles and comprises complete execution set, from initial setup of test environment through reporting and cleanup.
- Three major benefits:
1) Cumulative coverage to detect errors and reduce cost of failure.
2) Repeatability to save time and reduce cost to market.
3) Leverage to improve resource productivity.
1) Cumulative coverage:
- Applications change and gain complexity over their useful life.
- Feature set of the application grows steadily over time.
- Thus, number of tests needed for adequate coverage is also constantly increasing.
- Just a 10% code change still requires that 100% of the features be tested.
- Test coverage constantly decline over time unless you increase the test resources and cycle time.
- Automation helps to accumulate test cases over life of an application so that both existing and new
features can always be tested.
2) Repeatability to save time and reduce cost to market.
Faster time to market:
- Automation can help reduce time to market by allowing test execution to happen 24*7.
- Once the test library is automated, execution is faster and run longer than manual testing.
- This benefit is only available when your tests are automated.
Reduce cost:
- Automation can reduce cost of failure by allowing increased coverage so that errors are uncovered
before they have chance to do real damage in production.
3) Leverage to improve resource productivity.
- By generating tests cases programmatically, we can yield thousands or more- when only hundreds might be possible with manual resources.
When not to automate:
- Unstable design: (Ex: weather mapping system or one that needs real time data does not give predictable results for automation).
- Inexperienced testers: If person automating the test are not sufficiently experienced with the application to know the expected behavior, automating their test is also of doubtful value. Save automation for the experts.
- Temporary testers: Let temporary resource(consultant or users) handle the manual test support while permanent staff handles automation.
- Insufficient time, resources: In case of lack of time, dont expect tool to help you. Look at automation for the longer term. Keep in mind that automation is strategic solution, not short term fix.
How not to automate:
- Test automation tools are just specialized programming languages.
- Developing an automated test library is development project requiring commensurate skills.
Automation is more than capture/replay:
- Although it is most commonly recognized technique, capture/replay is not the most successful approach.
Dont write a program to test a program:
- Developing an automated test library that provides comprehensive coverage would require more
code than exists in the application itself.
- This is because test must account for positive, negative and invalid cases for each feature or function.
Automation is more than test execution:
- You are going to build application that automates your testing, more than just running the tests.
- We need complete process and environment of creating and documenting tests, managing and maintaining them, executing them and reporting the results, as well as managing test environment.
Duplication of efforts:
- If you just hand out automation tool out to individual testers and command that they automate their
tests, each one of them will address all of these issues in their own unique way.
- This leads to the tremendous duplication of effort and can cause conflict when tests are combined.
Need for the framework:
- Approach automation of testing just as you would automation of any application.
- This framework should make the test environment efficient to develop, manage and maintain.
- We will discuss how to develop a framework and select best automation approach.
Setting realistic expectations:
- Test automation is strategic:
If test process is in crises and management wants to throw money at a tool to fix it, dont fall for it.
Test automation is long term, strategic solution, not short term band aid.
Not everything can be automated:
- Allow time for tasks that cannot be automated.
- You still need to:
1) Gather test requirements
2) Define test cases
3) Maintain your test library
4) Administer the test environment
5) Review and analyze test results.
On ongoing basis, there is need to add new test cases based on enhancements or defects, so that your
coverage can constantly be improving.
Accept gradual progress:
- If you cannot afford time in short term, then do your automation gradually.
- Target those areas where you get biggest paycheck first, then reinvest time savings in additional areas, till you get it all automated.
Primary goal of automation should be to increase test coverage, not to cut testing costs.
Plan for long term:
- Test automation project will last as long as application under test is being maintained.
- Achieving automation is not sprint, it is a long distance run.
Terminology:
1) Requirement:
- "What" of a system.
- Performance requirement spells out the volume and speed of the application, such as maximum
acceptable response or processing time and highest number of simultaneous users.
2) Test:
- Combination of test case and test script.
3) Test case:
- Set of inputs and expected application response that will confirm that requirement has been met.
- Test case may be stored as one or more data records, or may be stored within test script.
4) Test Script:
- Series of commands or events stored in a script language file that execute test case and report results.
- Create multiple possible pathways.
- Contain logical decisions that affect the execution of the script.
5) Test cycle:
- It is set of individual tests that are executed as a package in particular sequence.
- Cycles are usually related to application operating cycles, or by their priority or content.
- We have build verification cycle to establish acceptance of new software build.
- Regression cycle to assure that previous functionalities has not been disrupted by changes or new features.
6) Test Schedule:
- series of test cycles and comprises complete execution set, from initial setup of test environment through reporting and cleanup.
No comments:
Post a Comment