Build trust in every step of your workflow.
Tools Qualification in Compliance with Safety Standards
The use of software development tools can be greatly beneficial in the development of safety-related electric and/or electronic (E/E) systems. In safety-critical domains such as automotive, aerospace, and medical devices, adherence to safety standards like ISO 26262, DO-178C, and IEC 62304 cannot be neglected. The ability to automate repetitive and time-consuming activities can increase confidence in safety-critical systems by eliminating the possibility of human error during such predictable activities. One key aspect of compliance is ensuring that the tools used in software development are qualified for their intended purpose.
In this article, we will be looking through the process of integrating some of these tools, namely the static analysis automation tools, into our work environment. Static analysis tools such as Cppcheck and Clang are increasingly favored for their ability to enhance code quality, detect potential bugs, and enforce coding standards.
The Significance of Tool Qualification
To minimize the risks associated with tool usage, and to ensure their reliability, functional safety standards emphasize the importance of gaining confidence in the tools used during the development of electronic and electrical systems. In the development of safety-critical automotive software, meeting the tool classification and qualification requirements outlined in ISO 26262 is crucial for ensuring compliance with these safety standards.
Tool Qualification Process
Part 8 of the ISO 26262 standard calls for a 2 step qualification process:
- Tool classification
To determine the tool confidence level (TCL) of the tool under qualification, we first need to analyze and document the actual/intended usage of the tool. Use cases are used for this purpose, and potential tool errors that could occur in the context of the considered use case need to be identified and documented. Then the tool is classified to have a tool impact of 1 (TI1) if there is no possibility that the tool could introduce such errors into the system (or fail to detect such error within the system), or to have a tool impact of 2 (TI2) if that is not the case (meaning the tool could possibly introduce/fail to detect said error).
Then, an evaluation is done of the measures taken to either prevent or at least detect such errors, and the efficacy is rated based on the confidence level of these measures, having either high (TD1), medium (TD2), or low (TD3) confidence levels. Note that TD1 means there is high confidence that the measures applied to the tool will prevent/detect the tool error under analysis.
Eventually, we assign a tool confidence level (TCL) based on the following matrix, with TCL1 indicating the highest level of confidence and TCL3 the lowest:
| Tool Error Detection | ||||
| TD1 | TD2 | TD3 | ||
| Tool Impact | TI1 | TCL1 | TCL1 | TCL1 |
| TI2 | TCL1 | TCL2 | TCL3 | |
- Tool qualification
After classifying the tool to one of the described TCLs, we assess the need for a tool qualification process. Generally, a tool having the confidence level of TCL1 will not require additional actions.
For TCL2 and TCL3, the ISO 26262 standard calls for a tool qualification process using any or a combination of the following methods:
(1a) Increased confidence from use
(1b) Evaluation of the tool development process
(1c) Validation of the software tool
(1d) Development in compliance with a safety standard
Note that ISO26262 highly recommends methods 1a and 1b for projects with all ASIL levels except for ASIL D, with which it is recommended to use methods 1c and 1d for the tool qualification process.
Project Overview
The project focused on developing a car Body Control Module (BCM), a critical component in modern automotive systems responsible for managing vehicle functions such as lighting, door locking, and windshield wipers. As the BCM is a safety-related system, the development process needed to adhere to stringent functional safety and coding standards, including ISO 26262 for functional safety, MISRA-C for coding guidelines, and ASPICE for software development process assessment. The development team needed to qualify static analysis tools to meet such compliance requirements. The team selected Cppcheck and Clang tools due to their robust analysis capabilities and support for coding standards such as MISRA-C.
Implementing Static Analysis Tools
Why Cppcheck and Clang?
Cppcheck and Clang were chosen for their complementary capabilities:
- Cppcheck: Lightweight, open-source static analysis tool specializing in MISRA-C compliance and customizable checks.
- Clang: Advanced analysis capabilities for runtime defect detection and enforcing CERT-C guidelines.
Both tools offer seamless integration into CI/CD pipelines and support for custom rule definitions.
Integration Process
The integration of these tools involved:
- Configuration:
We customized the tool configurations to meet MISRA-C and project-specific coding rules. We opted to choose configurations that would not alter already existing tools so as to avoid introducing tool-born errors into the system, choosing instead to make any modifications manually after reviewing the result of the static analysis. We also analyzed the pre-existing checks for both tools for homogeneity and consistency, to avoid conflicting results and unnecessary checks.
- Pipeline Automation:
We integrated the tools into the CI/CD pipeline to automate static analysis checks by creating automated batch scripts to invoke the static analysis activities and analyze the results for confirmation before each build process. This provided control over the level of restrictions enforced by static analysis activities over the different stages of development.
- Baseline Establishment:
Initial scans identified and documented existing issues, which helped us to create a baseline of the state of each tool, which assisted in further stages of tool analysis and qualification process.
Validation and Qualification of Tools
Validation Activities
Cppcheck and Clang were validated using an open-source python test suite that assessed the tools’ ability to detect a range of known defects. Additionally, we used real-world scenarios from the BCM project as project benchmarks to evaluate tool performance.
Qualification Activities
Qualification followed ISO 26262 guidelines:
- Increased Confidence from Use: Historical data from similar projects were found to demonstrate tool reliability
- Validation of Software Tools: Verification tests using a variety of testing techniques confirmed the tools’ ability to detect specific categories of injected errors.
Challenges Faced and their Solutions
- False Positives:
One of the primary challenges was the high volume of false positives generated by the static analysis tools during the initial scans. This led to unnecessary time spent reviewing non-issues, which slowed down the development process and diverted attention from genuine defects. The team carefully fine-tuned the tools’ configurations and rulesets. Custom scripts were developed to filter out warnings unrelated to the project’s coding guidelines.
- Frequent Updates:
Static analysis tools like Clang and Cppcheck undergo frequent updates to enhance functionality, fix bugs, and add new rules. While beneficial in the long term, new versions introduced unexpected behavior, altered outputs, and flagged additional issues, requiring revalidation and requalification efforts. To mitigate this, a staging environment was set up where tool updates could be rigorously tested before introducing into the development environment. Versioning the static analysis solution while ensuring backward compatibility provided confidence for use in the project.
- Performance Bottlenecks:
When scanning large codebases, the analysis tools exhibited performance bottlenecks, leading to prolonged execution times. This slowed down the CI/CD pipeline. The team optimized the analysis scope by excluding non-critical files from scans and parallelizing the analysis tasks where possible. Additionally, incremental analysis was employed, focusing only on modified or newly added code. Lastly, tiering the solution to allow for partial runs of the analysis tools proved to be time saving while still maintaining the benefits of using the tools.
- Documentation Overhead:
ISO 26262 mandates detailed documentation of the tool qualification process. Generating this documentation in a manner that satisfied both internal and external audits was challenging. Without proper documentation, demonstrating compliance during certification audits could be problematic. The team developed templates and checklists to streamline the documentation process. Automated reporting scripts were also created to capture tool outputs and validation results, reducing manual effort and improving traceability.
Results and Observations
Impact on Code Quality
- Improved Compliance: Adherence to MISRA-C and CERT-C standards.
- Defect Reduction: Early detection of potential runtime and logical errors.
Process Efficiency
- Automated checks reduced manual review effort.
- CI/CD integration accelerated defect resolution timelines.
Compliance Benefits
Tool qualification ensured certification audits readiness, demonstrating compliance with ISO 26262 and ASPICE.
Conclusion
Tool qualification is a critical component in safety-critical projects, enabling the reliable development of compliant systems. By integrating and qualifying Cppcheck and Clang, the BCM project achieved enhanced code quality, streamlined workflows, and adherence to stringent safety standards. This case study underscores the importance of a structured approach to tool qualification for delivering reliable and certifiable software.
