Training Day #4- Test Execution, Bug Tracking, Test Metrics, and Test Sign off
Test Execution is also the “Test” part of the SDLC.
Once the test cases are written, shared with the BAs and Dev team, reviewed by them, changes are notified to the QA team (if any), the QA team makes necessary amends- Test design phase is complete. Now getting the Test cases ready does not mean we can initiate the test run. We need to have the application ready as well among other things.
Test Execution Guidelines
Let us now make a list of all things that are important to understanding the Test Execution phase:
#1) The build (the code that is written by the dev team is packaged into what is referred to a build- this is nothing but an installable piece of software (AUT), ready to be deployed to QA environment.) being deployed (in other words, installed and made available) to the QA environment is one of the most important aspects that needs to happen for the Test Execution to start.
#2) Test Execution happens in the QA environment. To make sure that the dev team’s work on the code is not in the same place, where the QA team is testing, the general practice is to have a dedicated Dev and QA environment. (There is also a production environment to host the live application).
#3) Test team size is not constant from the beginning of the project. When the Test Plan is initiated the team might just have a Team lead. During the test design phase, a few testers come on board. Test Execution is the phase when the team is at its maximum size.
#4) Test Execution also happens in at least 2 cycles (3 in some projects). Typically in each cycle, all the test cases (the entire test suite) will be executed. The objective of the first cycle is to identify any blocking, critical defects, and most of the high defects.
The objective of the second cycle is to identify remaining high and medium defects, correct gaps in the scripts and obtain results.
#5) Test Execution phase consists of- Executing the Test scripts + Test script maintenance (correct gaps in the scripts) + Reporting (defects, status, metrics, etc.) Therefore, when planning this phase schedules and efforts should be estimated taking into consideration all these aspects and not just the script execution.
#6) After the Test script being done and the AUT is deployed – and before the Test execution begins, there is an intermediary step. This is called the “Test Readiness Review (TRR)”. This is a sort of transitional step that will end the test designing phase and ease us into the test execution.
#7) In addition to the TRR, there are few more additional checks before we ensure that we can go ahead with accepting the current build that is deployed in the QA environment for test execution.
Those are the Smoke and Sanity tests. Detailed information on what these are is at: What is Smoke and Sanity Test?
#8) Upon successful completion of TRR, Smoke and Sanity tests, the test cycle officially begins.
#9) Exploratory Testing would be carried out once the build is ready for testing. The purpose of this test is to make sure critical defects are removed before the next levels of testing can start. This exploratory testing is carried out in the application without any test scripts and documentation. It also helps in getting familiar with the AUT.
#10) Just like the other phases of the STLC, the work is divided among team members in the Test Execution phase also. The division might be based on module wise or test case count wise or anything else that might make sense.
#11) The primary outcome of the test execution phase is in the form of reports primarily i.e. Defect Report and Test Execution Status report. The detailed process for reporting can be found at Test Executions reports.
Bug Tracking
While Test Execution was going on, we encountered a situation where the expected result of the test case was not met. Also, we identified some unexpected behavior during Exploratory Testing.
What happens when we encounter these deviations?
We obviously have to record them and track them to make sure that these deviations get handled and eventually fixed on the AUT.
#1) These deviations are referred to as Defects/bugs/issues/incidents/errors/faults.
#2) All the following cases can be logged as defects
- Missing requirements
- Incorrectly working requirements
- Extra requirements
- Reference document inconsistencies
- Environment-related issues
- Enhancement suggestions
#3) Defect recording is mostly done in excel sheets or via the use of a Defect Management software/tool.
For information on how to handle defects via tools, try using the following links:
Typically, the following columns are a part of the Defect Report:
- Defect ID: For unique identification.
- Defect Description: This is like a title to describe the issue briefly.
- Module/section of the AUT: This is optional, just to add more clarity as to indicate the area of the AUT where the problem was encountered.
- Steps to Reproduce: What is the exact sequence of operations to be performed on the AUT to recreate the bug are to be listed here. Also, if any input data is specific to the problem that information is to be entered as well.
- Severity: To indicate the intensity of the issue and eventually the impact this might have on the functioning of the AUT. The guidelines on how to assign and what values to assign in this field can be found in the test plan document. So, please refer to the Test Plan document.
- Status: What is the status of the defect.
- Screenshot: A snapshot of the application to show the error when it happened.
- Sprint: Select the sprint in which is defect is found.
These are some of the ‘must-have’ fields. This template can be expanded (E.g. to include the name of the tester who reported the issue) or contracted (For Example, the module name removed) as needed.
Test Metrics
We have established that during the Test Execution phase, reports are sent out to all the other project team members to give a clear idea about what is happening in the QA Execution phase. This information is important to everyone in order to get validation about the overall quality of the final product.
Imagine I report that 10 test cases passed or 100 test cases were executed- these numbers are merely raw data and do not give a very good perspective about how things are going on.
Metrics play a vital role in filling this gap. Metrics are in simple words, intelligent numbers that the testing team collects and maintains. For Example, if I said 90% of the test cases passed, it makes more sense than saying 150 test cases passed. Isn’t it?
There are different kinds of Metrics collected during the test execution phase. What metrics exactly are to be collected and maintained for what periods of time- this information can be found in the test plan document.
The following are the most commonly collected test metrics for most projects:
- Pass Percentage of the Test cases
- Defects density
- Critical Defects percentage
- Defects, Severity wise number
Test Sign Off /Completion Report
As we have to notify all the stakeholders that testing has begun, it is also the QA team’s duty to let everyone know that testing has been completed and share the results. So, typically an email is sent from the QA team (usually the Team Lead/QA Manager) giving an indication that the QA team has signed off on the product attaching the test results and the list of open/known issues.
Sample Test Sign off Email:
To: Client, PM, Dev team, DB team, BA, QA team, Environment Team (and anyone else that needs to be included)
Email: Hello Team,
QA team signs off on the Axonator version 3.0 software after the successful completion of the 2 cycles of functional testing the website.
The test cases and their execution results are attached to the email. (Or mention the location where they are present. Or if using test management software, provide details regarding the same.)
The list of known issues is attached to the email too. (Again, any other references that make sense can be added.)
Thanks,
QA team lead.
Attachments: Final Execution Report, Final issue /defect report, Known issues list
0 Comments