Realtime Systems treats Verification and Validation as a separate part of a project from development. This part is driven by its own quality and user goals that are different from development. However, like the development activity, the V&V of a product must be planned and scheduled to be executed efficiently. To be most effective, the V&V team members should have intimate, unbiased participation in requirements definition, design reviews, code inspections, and configuration change board decisions.

Verification and Validation Planning:

1. V&V Plan and Effort Estimations - Just like the project Development Plan and Quality Plan, Verification and Validation planning should be generated early, initially as part of the Development Plan, and later as a standalone document. During the development activities, system requirements & architecture will likely change and impact the test strategy, or conversely the V&V strategy for regulatory compliance may change which in turn can impact system requirements and architecture tradeoffs. For regulated industries, V&V effort is a significant portion of the projects resources (sometimes up to half of the total cost), and therefore deserves to be estimated and tracked like any other development activity.
2. Regulatory Compliance Strategies - This activity requires both research and expert knowledge. It typically includes UL testing, EMI/EMC testing, CE Mark strategy, and compliance with established development SOPs and guidelines for ISO. For wireless RF products and for many medical devices involving disposables, special certifications and biocompatibility testing are needed. These marketing and business requirements can be so onerous that a incremental strategy is sometimes appropriate. Regardless of the approach, the project must plan for the needed financial and personnel resources.
3. Development of Test Objectives - Test objectives should be developed early to help remove ambiguity from the system requirements. A common fate of a project is to release too early, or become mired at the other extreme, stuck in testing because there are no clear objects established to determine completion criteria. The project Development Plan should state whether testing is a debugging exercise or does the business use testing to access project and release risks. In addition to strictly functional testing, the project V&V team needs to consider usage goals for the project: performance, reliability, or usability.
4. Use of Simulation Environments - Simulations assist in both development and verification activities. During development, simulation can be used to take the place of physical components that are not yet available. The same is true for verification activities in that Test Objectives and procedures may be evaluated before the product is available using simulated components. Simulation can also be used to control or establish a known test environment for regression testing purposes and automated testing. Early and informed selection of simulation environments can result in reuse of these in product development, debugging, testing, clinical trials, and manufacturing.
5. Software Maintenance and Upgrades - Modifications to existing products these are typically underestimated because they tend to appear smaller and more limited than a full development activity. However, all of the processes followed for the initial development project need to be executed in some form for maintenance and upgrades. Otherwise, all of the good work can be undone very quickly. The project Development Plan should be revised before initial product release to include special needs of the Maintenance life-cycle.

Back to V&V Overview:

Design Assurance Activities:

1. Standard Operating Procedures - SOPs and documented guidelines establish the roadmap and framework for the project. Team members need to be trained on all of the processes that they will utilize. Realtime maintains a full Quality Systems program that includes a library of Procedures, Policies, and Guidelines to draw from. Any inadequate or incomplete procedures are identified and communicated by the team to program management. The project Development Plan and project Quality Plan will state any tailoring necessary to the established Realtime practices.
2. Team Training - Participants typically come from different backgrounds and environments which can be a big advantage during activities that require diversity of perspective. However, if not placed on the same foundation, team members could be at odds with each other. Team Training is very important to effectively make progress and implement the established quality-based processes; it is often overlooked or too limited in scope.
3. Requirements and Design Reviews - Historically, 65 % of the project defects originate in the requirements phase; it is very important that the verification and validation efforts mature these elements quickly before development resources are wasted in the wrong direction. A diverse group of participants, modeling, and formal inspections can be a valuable part of these reviews. Requirements Management tools (like DOORS, SLATE, or RequisitePro) are extremely useful for tracking attributes and change history of requirements, design, and test coverage. The project Development Plan will indicate which review milestones will be scheduled and tracked.
4. Development Infrastructure Control and Certification - These are the tools that the team members will use to perform the needed development tasks. These tools include change control, issue tracking, quality analysis (e.g., PC-LINT and Promet), checklists, compilers, schematic capture and layout, CAD, authoring, testing, monitoring, and any project specific tools which support the development activities. All of these tools should be validated (or alternatively certified for use within the Manufacturer’s intended use claims) and configuration controlled. The list of tools should also include templates, and forms.

Back to V&V Overview:

Quality Assurance:

1. Dedicated specialists in Software Quality Assurance - These are people who are not directly responsible for the development of the code, but will ensure that the defined processes are followed and quality expectations are met. Designated personnel are appointed by the Quality Manager, as identified in the Quality Plan, to perform the Quality Assurance roles on each of the design disciplines.
2. SEI Capability Maturity Model Processes - These elements are used by business to establish the maturity and reliability of their current process and also recommend the next items for improvement. A goal is for projects to become less risky, more predictable, and more reliable without substantially increasing overhead costs. The Quality Systems that Realtime has in place are designed for any project to operate at SEI Level 3, which can be tailored down as appropriate for the size and purpose of the development effort.
3. QSR, GPSD, and 21 CFR Part 11 (Electronic Records) for FDA - Guidelines have been developed by the FDA that establish minimum expectations for product developers, manufacturers, auditors, and reviewers. It is important that these are understood at the start of the project and integrated into the plan. Many projects will provide the needed elements at the end of the project when there is no way for the project to benefit from the deliverable in a natural sequence. In fact late delivery of these items sometimes forces project rework and redirection.
4. DO-178B for Avionics - The avionics industry has special practices that must be followed based on the “Level” classification from ‘A’ through ‘E’. Because our staff has experience in medical, military, and other regulated industries, we are able to offer development and verification services at each of the five levels. We also have an association with a DER if that is able to benefit a program.
5. Internal Audits - Each project typically establishes a Development Plan and a Quality Plan at a minimum. Typically these will incorporate the needs of ISO 13485 for processes and ISO 14971 for Risk Management. Internal Audits are  self checks performed by a resource independent from the project in order to assess compliance to the established project plans and the overall corporate wide Quality System. These audits can be valuable during the development activities or for the purposes of showing compliance to ISO 13485 and ISO 14971. For instance, an audit of deliverables required to exit a project phase is very timely and important to preventing projects from skipping ahead before it is ready. Risk Assessments and audits of computer systems/software/processes are important for both certification and business operation efficiency.
6. Metrics and Static Analysis Reports - Metrics are not just used for long term trend analysis. There are a number of special purpose static analysis tools that are typically used during software and hardware development to assess the qualities of those deliverables. Reports that are generated by such tools should be available to everyone on the project to encourage uniformity and immediate improvement opportunities. For example, Software developers should be able to generate qualitative reports on their code before code check in and inspections. Minimum acceptance levels are established as phase exit and entry criteria.

Back to V&V Overview:

Test Execution and Reports:

1. Test Case and Protocols Development - Test cases validate each of the requirements and verify each of the design elements. Protocols (or test procedures) establish the testing processes, steps, methods, and pass/fail evaluation criteria before the test is executed.
2. Test Execution and Report Generation - Test cases are executed following practices that establish evidence of the results of the execution. A report of the results is made to document the testing effort, and provide a baseline for future regression testing. This wrap up is unbiased and separate from any release decisions, which are made by the project and business leaders.
3. Problem Tracking and Change Requests - All Problems are not created equal. All Change Requests have a cost. Each must be evaluated on its own merits to the project in both the current phase and future commitments. Many of these must be implemented immediately, and others may not be important enough for immediate consideration or may even be counterproductive. It is the responsibility of the Configuration Control Board, a qualified group of team members with diverse backgrounds and perspectives, to manage these decisions effectively. Many projects that do not manage change or problem resolution well get into requirements creep, which could result in indefinite delays in product launch or feature upgrades.

Back to V&V Overview: