Consistent with  the U.S. Department of Defense’s (DoD) Artificial Intelligence (AI) Strategy, as we previously reported on here, on April 13, 2020, DOD published a Request for Information (RFI) requesting assistance from academia and industry with the development and planning of a potential new requirement for DOD’s Joint Artificial Intelligence Center’s (JAIC) Testing & Evaluation (T&E) office.  DOD seeks industry assistance with and provision of various AI-technology and tools, testing and evaluation services, and any other technologies not specifically identified in the RFI.  Interested parties are encouraged to submit white papers by 10:00 AM CST on May 12, 2020.

Purpose of RFI

To help facilitate the adoption of AI technologies, the JAIC is creating a Joint Common Foundation (JCF).  The JCF will provide a “secure development ecosystem” for all DoD components to discover, develop, and test solutions for their AI problems and host AI technologies developed by the JAIC’s Mission Initiatives (MIs).  These include National MIs led by a functional team in JAIC to address broader, cross-cutting AI challenges across DoD, as well as Component MIs led by DoD components, with support from JAIC, to use AI for component-specific goals and to solve discrete problems.  Where JCF provides the centralized support network, T&E will be the centralized point of contact for AI-subject matter expertise.

To ensure that MIs are delivering safe, effective products, the T&E office will provide testing services – e.g., algorithm, system, and operational testing – on the JCF network to evaluate these products and independently analyze AI performance of the products.  In addition, T&E will develop AI processes, tools, and standards to be used across DoD on the JCF network.

RFI Focuses on Three Distinct Areas with Identified Needs

For these reasons, T&E office is requesting information “on the development of T&E software for AI, creation of AI T&E processes and best practices, and the ability of vendors to provide AI test and evaluation services.”  As provided from the RFI, T&E is focused on these three areas with the following identified needs:

• Various AI-technology and tools to develop T&E’s capabilities and support DoD’s emerging AI technologies – e.g., Machine Learning (ML), Deep Learning (DL), and Neural Networks / Neural Nets (NN) – with a focus on these capabilities:
  • Conversational interface applications using voice to text.
  • Speech enabled products and services for DoD applications and systems.
  • Image analysis, testing deep learning based visual search and image classifier.
  • Natural Language Processing (NLP) enabled products and services.
  • Humans augmented by machines, to include human-machine interfaces and improved methods to measure warfighter cognitive and physical workloads, to include augmented reality (AR) and virtual reality (VR) test services.
  • Autonomous systems.

• Test and evaluation services from across the government, academia, and industry to assist in the test/evaluation of technologies provided by MIs or other DoD contributors. As provided from the RFI,  T&E is looking for support in:
  • Data Set Development/Curation: Each MI or DoD AI project has unique data that must be acquired and curated such that it can be used by modern AI software systems. Efforts include, but are not limited to, data quality assessment, image annotation, and data coverage analysis.
  • Test Harness Development: Creation of automated, containerized, cloud-enables test harnesses that facilitate automatic ingestion of data into a containerized AI model and aggregates the results.
  • Model Output Analysis: Analysis expertise/software is needed to assess the quality of results from the AI models developed by the MIs. This effort involves both development of containerized, cloud-based, analysis software and creation of context specific and operationally relevant performance metrics.
  • Test Planning, Documentation, and Reporting: Testing expertise to create the required documentation for an AI algorithm undergoing the testing lifecycle (Algorithm test, System test, and Operational test). This effort includes, but is not limited to, creation of Test and Evaluation Master Plans (TEMPS), Test Plans, creation of Test Report documenting findings, and management of a test schedule.
  • Testing Services: This effort involves the services and expertise to conduct test and evaluation of AI products.  This includes, but is not limited to, performing testing on AI algorithms and/or AI enabled systems across all T&E areas (Algorithm test, System test, Developmental test, and Operational test).

• The identification of other technologies that are not included in the RFI and would be beneficial to T&E’s AI efforts.

Conclusion

As called for in DoD’s AI Strategy, this RFI is looking for assistance from industry and academic partners to provide AI tools and standardized processes for testing and evaluation across DoD.  These partnerships will help fulfill DoD’s vision in its AI Strategy for the JAIC by creating a DoD-wide mechanism offering tools, frameworks and standards, and unique services tailored to and in support of DoD’s deployment of AI.  Notwithstanding, any opportunities that result from this RFI and the information gathered by DoD may present challenges in navigating non-traditional procurement approaches (such as OTAs and public-private partnerships).