Objective
Our objective was to assess the methods and technical rationale in developing the Advanced Combat Helmet (ACH) testing protocols issued by the Office of the Director, Operational Test and Evaluation (DOT&E) to determine whether the test protocols are appropriate for the ACH. Specifically, our assessment focused on the First Article Testing (FAT) Resistance to Penetration (RTP) requirement of at least 90 percent Probability of no Penetration (P(nP)) with 90 percent Confidence Level, commonly termed the “90/90 standard,” and the Lot Acceptance Testing (LAT) RTP requirement of 4 percent Acceptable Quality Level (AQL). Lastly, we assessed the participation of various stakeholders and industry experts such as active ACH manufacturers and test facilities.
Findings
After reviewing the methods and technical rationale in developing the helmet RTP requirements, we found the following.
A. The DOT&E test protocol for the ACH adopts a statistically principled approach and represents an improvement from the legacy test protocol with regard to increased sample size. However, future protocol revisions necessitate further refinement by anchoring the RTP requirements to helmet specific empirical data such as manufacturing capabilities and test performance.
B. The DOT&E LAT protocol is an improvement from the legacy LAT and adopts a widely established and industrially accepted American National Standards Institute (ANSI Z1.4-2008, Sampling Procedures and Tables for Inspection by Attributes). In selecting the LAT RTP requirement of 4 percent AQL, DOT&E considered the government risk of accepting underperforming helmets, manufacturer risk of failing LAT with acceptable helmets, and historical LAT data. However, DOT&E did not consider selecting an AQL that was based on the safety criticality of the helmet.
C. In accordance with authorizing statutes, DOT&E has the authority to establish test standards for personnel protective equipment such as the ACH. However, despite the significance and broad impact of these protocols, DOT&E did not explicitly consult with heads of the Military Departments to provide them an opportunity to comment on new or changed test protocols and did not adequately document the adjudication of inputs provided by program offices and subject matter experts in the staffing process. The program office also did not solicit comments on the helmet test protocols with the helmet vendors and Defense Contract Management Agency (DCMA).
D. The FAT RTP acceptance was based on an aggregate of all test outcomes under varying conditions to achieve 90/90, or 17 penetrations out of 240 shots. This could result in passing FAT, despite test results showing clusters of failures for a unique helmet size or in a particular test environment.
Recommendations
During the course of this assessment, DOT&E and the Army’s Program Executive Office (PEO) Soldier were very responsive and have already committed to address most of the findings identified. Our recommendations pertaining to the findings are detailed on the next page.
Recommendation for Finding A. Origin of Resistance to Penetration Requirements
We recommend that DOT&E and PEO Soldier fully characterize the performance of all helmet designs included in the combat helmet test protocols. Performance characterization should consider threat, historical test data, prototype test data, and manufacturing capabilities. Based on helmet performance characterizations, DOT&E and PEO Soldier should determine if modification to the FAT and LAT protocols are appropriate.
Recommendation for Finding B. Acceptable Quality Level Based on Safety Criticality
We recommend that in addition to considerations made to Government risk, manufacturer risk, and historical LAT data, DOT&E should also consider the safety criticality of the helmet in establishing the AQL requirement for LAT RTP in future protocol revisions.
Recommendation for Finding C. Coordination of Helmet Test Protocols
Due to the significance and broad impact of these protocols, we recommend that DOT&E ensure the affected organizations, including heads of the Military Departments, are consulted in developing the protocols and have an opportunity to provide input on new or changed test protocols. DOT&E should also ensure that inputs received are documented and adjudicated.
We recommend that PEO Soldier solicit input from helmet vendors and DCMA on new or changed test protocols. PEO Soldier should also ensure that inputs received are documented and adjudicated.
We recommend that DOT&E include an explicit statement in future protocol revisions that allows program managers to modify the test protocols, provided that program managers submit a well-justified request for approval.
Recommendation for Finding D. Evaluation of Aggregate Test Results
We recommend that DOT&E and PEO Soldier describe the method of identifying and addressing statistically significant differences in performance due to environmental conditions, helmet sizes, shot locations, and different vendor designs for all FAT results under the DOT&E helmet test protocol.
We recommend that PEO Soldier and Defense Logistics Agency (DLA) consider the contractual implications of not fielding an ACH design that passes FAT yet shows significant clusters of inadequate performance.
Overall DOT&E and PEO Soldier Comment
DOT&E and Program Executive Office (PEO) Soldier have reviewed the referenced report and agree with the findings and recommendations.
Overall DoD OIG Response
DOT&E and PEO Soldier’s comments were fully responsive. We reviewed their response and have additional points for consideration. However, no additional comments are required.
This report is a result of Project No. D2012-DT0TAD-0007.000.