Test process improvement with the TMMi model
I feel i should warn everybody at the beginning of this post that this is gonna be a very long one. Because this is a subject that i am passionate about.
The size and complexity of systems and applications are constantly increasing. This leads to increased requirements and demands from the customer receiving the application or system. Users are not just interested in the functions that the systems provide, it’s also necessary that they are of high quality. And how do we meet the users expectations? By testing, but you can’t just test the software and claim that you have a good test process.
A test process is complex, it involves a lot of sub processes, actors, goals and subgoals etc. More and more organisations make efforts to improve their software development processes. The reference model that is most often used is the Capability Maturity Model Integration (CMMI). In this model practices that are related to verification and validation activities are described, but the level of detail is too limited from the viewpoint of the test professional.
The Test Maturity Model Integration (TMMi) has been developed by the TMMi Foundation, using the TMM framework as developed by the Illinois Institute of Technology as one of its major sources. The TMMi provides a structured approach for test process improvement. Testing as defined in the TMMi is applied in its broadest sense to encompass all software quality-related activities. My experience of the TMMi model comes from performing a test process improvement project based on the TMMi model. The goal of the project was to identify improvement areas for the organization. The result defined the maturity level of the test process and a range of improvement proposals were presented.
Overview of TMMi
Just like the CMMI staged representation, the TMMi has a staged architecture for process improvement. It contains levels that an organisation passes through as its testing process evolves from one with an ad-hoc and unmanaged nature to a mature and controlled process. Each maturity level in the model except Level 1 includes a number of process areas. In order for the test process to achieve a certain level of maturity the specific goals for the process areas have to be achieved. specific goals are achieved with practices, activities, tasks and responsibilities.
Bellow follows a quick explanation of the different levels in the TMMi-model.
At level 1 testing is a chaotic, unclearly defined process, and is usually considered as a part of troubleshooting. The goal at this level is to show that the software works without any major problems. This level is defined by a lack of resources, tools and trained testers.
At Level 2 testing is a manageable process distinct from debugging/troubleshooting. Despite this, many of the stakeholders still see testing as a project phase that follows after the “coding” phase. At this level, the testing is multilayered, we have component, integration, system and acceptance level testing. The main objective of Level 2 is basically to verify that the product satisfies the requirements.
At level 3, testing is no longer just a phase that comes after coding of the product but is fully integrated in the software development life cycle and its associated milestones. Test planning is done at an early stage in the project and documented most often in a test plan for the project. Test plans are based on experience and knowledge from Level 2. Testing is seen at this level of as a profession and within the organization there exists some form of internal test training.
Level 4: Measured
Organizations that have achieved fully Level 2 and 3 have already staff and even the technical infrastructure to measure test process at level 4. This is important because it encourages process improvement and leads to a well defined process that fulfills functionality and efficiency requirements. At this level measures are implemented and used within the organization to evaluate the quality of the testing process, and evaluate and monitor productivity improvements. Data collected through evaluations of the testing process is used to optimize the approach and make testing more efficient and productive.
Level 5: Optimization
Levels 1-4 create an infrastructure that supports a completely defined and measurable process. Activities at level 5 are capable of continuously improving the testing process. This is done through a series of process and technology improvements. A flawless and optimized test process according to level 5 is:
- Manageable, defined, measured, efficient and productive.
- Statistically controlled and predictable.
- Focused on prevention of defects.
- Supported with automation.
- Can support technological changes.
- Can reuse test resources.
- Focused on process changes to achieve continuous improvement.
For example in company’s at this level there is an process improvement group focused on the test process. The group is formal, and its members have training in the area, such groups are called Test Process Group (TPG). Support for such a group within the business starts at level 3 by the introduction of an testing organization.
TMMi improvement project
In the book “The litle TMMi” by Erik Van Veenendaal he breaks down test process improvement with the help of the TMMi model into 4 different phases:
- Initiation (determining the scope and objective of the process improvements)
- Planning (define the project structure, resources and ownership)
- Implementation (development of procedures, templates and strategies)
- Delivery (publishing templates, strategies and initialization)
Define the current status
Before an improvement project begins, it is essential to know the maturity level of the current test process. Therefore, the current situation is evaluated. There are formal and informal evaluations. Usually an informal evaluation is sufficient to determine the current maturity level of the testing process. The results of the informal evaluation is compiled in a report that has the maturity profile and recommendations for improvement. The report is the base that the improvement project is built on.
It is desirable to define improvement project objectives clearly, it’s very important that these objectives are directly linked to organizational business goals to ensure management support. Examples of such cases is the increase of productivity and an more efficient process. But ultimately it is up to the organization to establish these goals.
Define the project organization
A common problem that exists in all types of improvement projects is that the initiative is not receiving adequate attention and resources. Therefore it is recommended to organize the project according to the informal evaluation. Each recommendation is assigned to a workgroup that implements the improvements. These teams consist of representatives from all test projects within the business.
The most important part of the improvement is communication. It is very important to continuously inform the individuals within the company. This is to ensure that they understand why the changes are necessary to ensure acceptance of the changes.
Finding a balance between process and improvement project is very important, optimally, it can be achieved by an external consultant who works with an internal test manager. It is important that the external consultant is not the driving factor for changes in the improvement project. A common pitfall with external consultants as a driving factor is that the external consultant doesn’t adapt to the client’s level of maturity, but bases the project on their own perception of what level the client should be on. It is therefore recommended that ownership of the improvement project goes to an internal grouping.
Depending on the activity, it may be unclear who the stakeholders in the test process are, therefore it is sometimes necessary to do an analysis of stakeholder. This step is important both to ensure that the improvement project is getting enough resources and also to ensure that the changes will be accepted.
Software process maturity level
According the TMMi model, it is equally important to look at the development processes maturity level. This is because in order to achieve a mature testing process it is required that the supporting processes that the process depends on are also mature processes. If the support processes are not mature, it is very difficult, even impossible to improve the test process. If the support processes are not mature, there are two options either start one or several process improvement projects to increase the maturity level of these or not to start any test process improvement at all because the business is not ready for that step in the current situation.
Identification of test process improvement activities
Within businesses, several improvement initiatives run simultaneously. To avoid conflicts, or that the same activity is performed several times, it is important to identify how the test process improvement activities affect and are affected by other improvement activities. This helps the project manager work with the process improvement.
The implementation phase
During this phase, all changes are implemented based on recommendations from the informal evaluation. It is important to reuse existing processes and “best practices” as much as possible. Sometimes the changes are already implemented within the business but just not documented. It is important to focus on all areas in each level.
The organization has to define a test policy that defines the business approach to testing. It answers questions such as why we are testing, how important is the quality, if it is a time-driven or quality-driven process and so on. Usually, the answers to these questions are related to business objectives or alternatively company’s business strategy. It is important that the test policy is complete, clear, and should preferably use test terminology instead of business language that is typically used in the activity statements.
Test Policy is implemented through a test strategy. The TMMi model requires definition of a test strategy for each project within the organization. This strategy provides clear insight into how products are tested all the way through their lifecycle. It must fit into the organization’s development process. The strategy contains a definition of test levels and the organizational groups that perform testing. It also includes an approach such as regresionstest, input and output criteria, etc.. It is important that the strategy created is easy to explain to stakeholders and testers.
Measuring the success and quality of testing it is important to monitor the status and show the benefits of the improvements. This refers to such areas as efficiency, performance and utilization. Goal-question-metric is a good method to define ways to measure these areas.
An important part of the process is documenting the changes of the process, with a clear documentation structure it is easier to create a “red thread”, linking the various documents produced during the process.
The delivery phase
It is important that all documents, templates, and the like are readily available to testers and all other stakeholders.
Delivery is the hardest and most time consuming part of the improvement project. This is because in order to achieve a given level of maturity in TMMi model is is required that at least 80% of the people in the organization operate according to the process. Therefore it is very important to involve the stakeholders in the project to increase the chances that the new process and the new way of working is accepted. If the existing procedures are not followed, it is important to address the root cause first. Only when you determined why the current processes are not followed, is it worth changing the process and enforcing changes. The time required for this depends on several factors including the structure of the business, its culture and the number of ongoing development projects. It can sometimes be useful to test the new changed process in a “pilot project” to see if the changes have the desired effects. It is always best to break down the change in small steps and communicate changes to stakeholders in all possible ways. This to ensure that the changes are understood, accepted and enforced.
Hopefully you now have a basic understanding of the TMMi model and test process improvement. If you are interested and want to read more about the TMMi model i can recommend the following books:
- “Test Maturity Model integration (TMMi) – Guidelines for Test Process Improvement” by Erik van Veenendaal and Brian Wells
- “The Little TMMi – Objective-Driven Test Process Improvement” by Erik van Veenendaal and Jan Jaap Cannegieter
And also the following pdf links: