Software Reliability & Quality Assurance

Jelinski and Moranda Model (1972):

• Basic Assumptions: The Jelinski and Moranda model, proposed in 1972, relies on two key assumptions:
  • The reliability increases by a constant increment with each error detection and repair, implicitly assuming perfect error fixing in every instance.
  • All errors contribute equally to reliability growth, assuming a uniform impact across different types of errors.
• Realism Considerations: Acknowledging the limitations of the model, it's important to note that these assumptions are unrealistic. In reality, different errors contribute differently to reliability growth, and error fixes may not always be perfect, potentially introducing new types of errors.
• Reliability Growth Prediction: Despite its limitations, the Jelinski and Moranda model predicts reliability growth, as illustrated in the accompanying Figure. The graph represents a step function model of reliability growth over time.
• Instantaneous Failure Rate: The model defines the instantaneous failure rate (also called the hazard rate) as Z(t) = ∆ × (N – i), where ∆ is a constant, N is the total number of errors in the program, and t is any time between the ith and (i + 1)th failure. This means the failure rate remains constant between two failures and improves by ∆ after every failure.
• Graphical Representation: Refer to the accompanying Figure for a visual representation of the step function model of reliability growth, providing insights into how the reliability of a software product evolves over time according to the Jelinski and Moranda model.

• On the graph diagram, with time (on the x-axis) and ROCOF (on the y-axis), the plot visualizes how the rate of failure occurrence changes over time as errors are detected and repaired. The ROCOF curve reflects the model's assumption that reliability increases by a constant increment each time an error is identified and fixed. This visualization helps in understanding the dynamic nature of reliability improvement predicted by the Jelinski and Moranda model.

Littlewood and Verall’s Model:

• Negative Reliability Growth: Littlewood and Verall’s model allows for negative reliability growth, acknowledging that fixing a bug may introduce additional errors, potentially decreasing the overall reliability of the software.
• Imperfect Bug Fixes: In contrast to the Jelinski and Moranda model, it recognizes the imperfections in bug fixes, understanding that they may introduce new errors, affecting the software's reliability.
• Diminishing Returns: The model accounts for diminishing returns as errors are repaired over time. Initially, the average improvement to product reliability per repair may be substantial, but as testing and fixing progress, this improvement decreases.
• Independent Error Contribution: An error's contribution to reliability improvement is treated as an independent random variable following a Gamma distribution. This captures the idea that error corrections with larger impacts on reliability are addressed first.

Goel-Okutomo Model:

• Exponential Distribution of Execution Times: The Goel-Okutomo model assumes that execution times between failures follow an exponentially distributed pattern.
• Expected Number of Failures: The expected number of failures at any time is represented as µ(t) and is calculated as the expected value of failures between time t and time t + ∆t.
• Non-Homogeneous Poisson Process (NHPP): The reliability growth in this model is assumed to follow a Non-Homogeneous Poisson Process. This means that the expected number of error occurrences is proportional to the expected number of undetected errors existing at time t.
• Immediate and Perfect Error Correction: Once a failure is detected, the model assumes immediate and perfect error correction, enhancing the reliability of the software.
• Formula for Number of Failures: The number of failures at time t is given by the formula µ(t) = N(1 – e–bt), where N is the expected number of defects, and b is the rate at which the failure rate decreases.
• Graphical Representation: The change in the number of failures over time has been graphically plotted, providing a visual representation of how failures evolve as the software undergoes testing and error correction.

What is ISO 9000 Certification?

• Reference for Contract: ISO 9000 certification acts as a reference for contracts between independent parties. The standard provides guidelines for maintaining a quality system, addressing both operational and organizational aspects.
• Operational and Organisational Aspects: ISO 9000 addresses operational aspects, including processes, and organizational aspects such as responsibilities and reporting. It offers recommendations for high-quality product development without being directly concerned about the product itself.
• Types of ISO 9000 Standards: ISO 9000 consists of three standards—ISO 9001, ISO 9002, and ISO 9003. Each applies to specific types of organizations in different industries.
  • ISO 9001: Applicable to organizations engaged in design, development, production, and servicing of goods, including most software development organizations.
  • ISO 9002: Applies to organizations not involved in product design but focused on production, excluding software development organizations.
  • ISO 9003: Applies to organizations exclusively involved in the installation and testing of products.

ISO 9000 for Software Industry

• Challenges in Interpretation: Many clauses of ISO 9000 use generic terminologies, posing challenges for interpretation in the context of software development. Software's intangibility and reliance on data as the only raw material contribute to these challenges.
• Software Development Differences: Software's intangibility makes it challenging to control and manage until development is complete. Unlike traditional manufacturing, software development doesn't involve physical raw materials like iron-ore or coal, rendering certain ISO 9000 clauses irrelevant.
• ISO 9000 Part-3: Due to these differences, ISO released ISO 9000 Part-3 in 1991, offering guidance specifically tailored for the software industry. However, official guidance remains limited, requiring continuous cross-referencing with ISO 9000-3.

Usage of SEI CMM

SEI CMM can be utilized in two ways: Capability Evaluation and Software Process Assessment. Capability Evaluation assesses an organization's software process capability and is conducted by the contract awarding authority. Software Process Assessment is for internal use, enabling organizations to enhance their own process capability.

Maturity Levels

• Level 1: Initial

  The initial level has no specific requirements. Few or no defined processes lead to chaotic development efforts, with engineers following individual processes.

• Level 2: Repeatable

  Basic project management activities are prepared, including cost and schedule tracking. Configuration management tools are used, but processes are not documented. Repeatability is achieved when producing similar products.

• Level 3: Defined

  Processes for management and development activities are defined and documented. There is a common organization-wide understanding of activities, roles, and responsibilities. Process and product qualities are not yet measured.

• Level 4: Managed

  Focus shifts to software metrics, collecting both process and product metrics. Quantitative quality goals are set, and tools like Pareto charts and fishbone diagrams are used for measurement.

• Level 5: Optimizing

  Process and product metrics are collected and analyzed for continuous improvement. The organization adapts to innovative ideas and technologies based on quantitative feedback from process measurements.

CMM Shortcomings

• Lack of Guidance: Organizations often express a need for more guidance on how to improve, despite understanding the areas requiring improvement.
• Documentation Overload: CMM's emphasis on thicker documents and detailed information contrasts with agile practices, which prioritize reducing complexity and minimizing documentation without sacrificing relevant details.
• Maturity Level Measurement: Getting an accurate measure of an organization's current maturity level is challenging, as CMM takes an activity-based approach without assessing the effectiveness of these activities in delivering intended results.