- Use automation to verify what you expect, and use manual testing for what you dont.
1) Test process must be well defined:
- You cannot automate process that is not already well defined.
- You need to know what needs to be tested.
2) Testware is software:
- Bridge gap between what should be tested and how it should be automated.
3) Test automation is two different disciplines:
- Test automation is really two different things:
- There is testing which is one discipline and automation, which is another.
- In automation software testing, computer is being instructed to perform task previously performed manually.
- Whether these instructions are stored in something called script or program they both have all characteristics of source code.
- Test: Application expertise, What to test, Test cases.
- Automation: Development expertise, How to automate, Test scripts
Maintainability:
Applications are maintained continuously:
- Instead of gradually improving your test coverage over time by accumulating more and more test cases, you will be discarding and recreating tests instead.
- Maintainability is so important as without it you cannot accumulate tests.
Changes must be known in advance:
- It is important to know where and how to make changes to test library in advance.
- If person must watch test to determine the results, then the test is not truly automated.
Optimization:
When designing tests, remember that more is not always better.
The more tests you have, more time it will take do develop, execute and maintain them.
You need to have enough tests to do the job without having too many to manage.
One test, one requirement:
- Each test must be designed to map to specific business or design requirement or to previously reported defect.
- It allows tests to be executed selectively when needed: for ex, to confirm whether the defect has been corrected.
Having no requirements is no excuse:
- Examine your tests and decide what feature or function this test verifies, then state this as a requirement.
- You must know what test cases are affected if an application requirement changes, it is simply not practical to review every test case to see whether it remains valid.
Understanding test results:
- Specify as precisely as possible what each test case covers is that, if test case fails, it reduces the level of diagnostics required to understand the error.
- Lengthy involved test case that covers multiple features or functions may fail for any number of reasons.
- Time it takes to analyze a failure is directly related to complexity of the test case itself.
- A crisp tie in between requirements and test cases will quickly indicate the type and severity of the failure.
Requirements measure readiness:
- Requirements can be assigned priorities and used to measure readiness for release.
- Having requirements tied to the tests, also reduces confusion about which requirements have been satisfied or failed based on results of the test, thus simplifying the test and error log reports.
Requirement matrix:
- Handy way of keeping track of which requirements have an associated test.
- Requirement that has too many tests may be too broadly defined, and should be broken down into
separate instances. It may also have more tests than are needed to get job done.
- Test that is associated with too many requirements may be too complex and should be broken down
into smaller, separate tests more targeted to specific requirements.
Independence:
- Refers to degree to which each test case stands alone.
- Does success or failure of one test case depend on another, what is the impact of sequence of execution?
Independent data:
- Most easily accomplished if each test case verifies atleast one feature or function by itself and without reference to other tests.
- For ex, test case that exercises delete capability of record in the file should not depend on previous test case that creates record:
Independent context:
- Independence is needed where application context is concerned.
- For ex, one test is expected to commence at particular location, but it relies on previous test to navigate through application to that point.
- If first test does not execute, second test could fail for wrong reason.
- Your framework should give consideration for selecting common entry and exit points to areas of application and assuring that related tests begin and end at one of them.
Result independence:
- It is also risky for one test case to depend on successful result of another.
- For ex, test case that does not expect an error message should provide assurance that, no message was issued.
- If one is found, steps should be added to clear the message.
If proper attention is paid to independence, test execution cycle will be greatly simplified.
In cases where independence is not desirable or possible, be certain that dependencies are well documented.
Ex: ADD RECORD 01, ADD RECORD 02..
Modularity:
- refers to test scripts, whereas test independence refers to test cases.
Tie Script design to the application design:
- Ideally, test script should be comprised of modules that correspond to the structure of application itself, so that when change is made to the application , script changes are as localized as possible.
- Modularity should not be taken to an extreme: scripts should not be broken down so minutely that they lose all individual meaning.
Identify common scripts:
- Common functions needed by all tests should not be duplicated within each individual script,
- Instead they should be shared as part of the overall test environment.
Context:
- Refers to state of the application during the test playback.
- As automated test is executing same time as application is, it is critical that they remain synchronized.
- Synchronization takes two forms:
1) Assuring that the application is infact located at the point where test expects to be.
2) Assuring that the test does not run ahead of application while it is waiting or processing.
Context control results:
- Tests are performing inputs and verifying outputs.
- It is imperative that inputs be applied at proper location in the application, and that the output appears where expected.
- Also, when multiple test run one after the another, result from one test can affect the next.
- If one test begins at main menu and ends at sub- menu, following test must either expect to begin at sub menu or risk failure.
Main menu approach:
- Design all test to begin and end at same point in application.
- In most cases, there will be main menu or SIGNON area.
- By designing every test so that it commences at this point and ends there, tests can be executed in any order without considering context.
Enabling error recovery:
- A test which fails can, after logging its error, call a common recovery function to return context to proper location so that next tests can be executed.
- Designing test suites, or combinations of tests, will also be more complex as consideration must be given to grouping tests which share common contexts.
Documentation:
Documentation of testware means that, in crunch, test library could be executed manually.
It can be:
- extensive comments sprinkled throughout test case or scripts.
- narrative descriptions stored either within the tests or in seperate documentation files.
Document for transferability:
- Only person who created the script will know what was expected.
- Anyone else attempting to execute the script may not understand what went wrong if window does
not appear and subsequent actions are out of contexts.
- So, without adequate documentation, transferability from one tester to another is limited.
Mystery tests accumulate:
- Always provide enough documentation to tell what test is expected to do.
More is better:
- More documentation is better.
- Assume as little knowledge as possible, and provide as much information as you can think of.
1) Test process must be well defined:
- You cannot automate process that is not already well defined.
- You need to know what needs to be tested.
2) Testware is software:
- Bridge gap between what should be tested and how it should be automated.
3) Test automation is two different disciplines:
- Test automation is really two different things:
- There is testing which is one discipline and automation, which is another.
- In automation software testing, computer is being instructed to perform task previously performed manually.
- Whether these instructions are stored in something called script or program they both have all characteristics of source code.
- Test: Application expertise, What to test, Test cases.
- Automation: Development expertise, How to automate, Test scripts
Maintainability:
Applications are maintained continuously:
- Instead of gradually improving your test coverage over time by accumulating more and more test cases, you will be discarding and recreating tests instead.
- Maintainability is so important as without it you cannot accumulate tests.
Changes must be known in advance:
- It is important to know where and how to make changes to test library in advance.
- If person must watch test to determine the results, then the test is not truly automated.
Optimization:
When designing tests, remember that more is not always better.
The more tests you have, more time it will take do develop, execute and maintain them.
You need to have enough tests to do the job without having too many to manage.
One test, one requirement:
- Each test must be designed to map to specific business or design requirement or to previously reported defect.
- It allows tests to be executed selectively when needed: for ex, to confirm whether the defect has been corrected.
Having no requirements is no excuse:
- Examine your tests and decide what feature or function this test verifies, then state this as a requirement.
- You must know what test cases are affected if an application requirement changes, it is simply not practical to review every test case to see whether it remains valid.
Understanding test results:
- Specify as precisely as possible what each test case covers is that, if test case fails, it reduces the level of diagnostics required to understand the error.
- Lengthy involved test case that covers multiple features or functions may fail for any number of reasons.
- Time it takes to analyze a failure is directly related to complexity of the test case itself.
- A crisp tie in between requirements and test cases will quickly indicate the type and severity of the failure.
Requirements measure readiness:
- Requirements can be assigned priorities and used to measure readiness for release.
- Having requirements tied to the tests, also reduces confusion about which requirements have been satisfied or failed based on results of the test, thus simplifying the test and error log reports.
Requirement matrix:
- Handy way of keeping track of which requirements have an associated test.
- Requirement that has too many tests may be too broadly defined, and should be broken down into
separate instances. It may also have more tests than are needed to get job done.
- Test that is associated with too many requirements may be too complex and should be broken down
into smaller, separate tests more targeted to specific requirements.
Independence:
- Refers to degree to which each test case stands alone.
- Does success or failure of one test case depend on another, what is the impact of sequence of execution?
Independent data:
- Most easily accomplished if each test case verifies atleast one feature or function by itself and without reference to other tests.
- For ex, test case that exercises delete capability of record in the file should not depend on previous test case that creates record:
Independent context:
- Independence is needed where application context is concerned.
- For ex, one test is expected to commence at particular location, but it relies on previous test to navigate through application to that point.
- If first test does not execute, second test could fail for wrong reason.
- Your framework should give consideration for selecting common entry and exit points to areas of application and assuring that related tests begin and end at one of them.
Result independence:
- It is also risky for one test case to depend on successful result of another.
- For ex, test case that does not expect an error message should provide assurance that, no message was issued.
- If one is found, steps should be added to clear the message.
If proper attention is paid to independence, test execution cycle will be greatly simplified.
In cases where independence is not desirable or possible, be certain that dependencies are well documented.
Ex: ADD RECORD 01, ADD RECORD 02..
Modularity:
- refers to test scripts, whereas test independence refers to test cases.
Tie Script design to the application design:
- Ideally, test script should be comprised of modules that correspond to the structure of application itself, so that when change is made to the application , script changes are as localized as possible.
- Modularity should not be taken to an extreme: scripts should not be broken down so minutely that they lose all individual meaning.
Identify common scripts:
- Common functions needed by all tests should not be duplicated within each individual script,
- Instead they should be shared as part of the overall test environment.
Context:
- Refers to state of the application during the test playback.
- As automated test is executing same time as application is, it is critical that they remain synchronized.
- Synchronization takes two forms:
1) Assuring that the application is infact located at the point where test expects to be.
2) Assuring that the test does not run ahead of application while it is waiting or processing.
Context control results:
- Tests are performing inputs and verifying outputs.
- It is imperative that inputs be applied at proper location in the application, and that the output appears where expected.
- Also, when multiple test run one after the another, result from one test can affect the next.
- If one test begins at main menu and ends at sub- menu, following test must either expect to begin at sub menu or risk failure.
Main menu approach:
- Design all test to begin and end at same point in application.
- In most cases, there will be main menu or SIGNON area.
- By designing every test so that it commences at this point and ends there, tests can be executed in any order without considering context.
Enabling error recovery:
- A test which fails can, after logging its error, call a common recovery function to return context to proper location so that next tests can be executed.
- Designing test suites, or combinations of tests, will also be more complex as consideration must be given to grouping tests which share common contexts.
Documentation:
Documentation of testware means that, in crunch, test library could be executed manually.
It can be:
- extensive comments sprinkled throughout test case or scripts.
- narrative descriptions stored either within the tests or in seperate documentation files.
Document for transferability:
- Only person who created the script will know what was expected.
- Anyone else attempting to execute the script may not understand what went wrong if window does
not appear and subsequent actions are out of contexts.
- So, without adequate documentation, transferability from one tester to another is limited.
Mystery tests accumulate:
- Always provide enough documentation to tell what test is expected to do.
More is better:
- More documentation is better.
- Assume as little knowledge as possible, and provide as much information as you can think of.
No comments:
Post a Comment