Photo of Kate GrowleyPhoto of Michelle ColemanPhoto of Laura J. Mitchell Baker

Consistent with  the U.S. Department of Defense’s (DoD) Artificial Intelligence (AI) Strategy, as we previously reported on here, on April 13, 2020, DOD published a Request for Information (RFI) requesting assistance from academia and industry with the development and planning of a potential new requirement for DOD’s Joint Artificial Intelligence Center’s (JAIC) Testing & Evaluation (T&E) office.  DOD seeks industry assistance with and provision of various AI-technology and tools, testing and evaluation services, and any other technologies not specifically identified in the RFI.  Interested parties are encouraged to submit white papers by 10:00 AM CST on May 12, 2020.

Purpose of RFI

To help facilitate the adoption of AI technologies, the JAIC is creating a Joint Common Foundation (JCF).  The JCF will provide a “secure development ecosystem” for all DoD components to discover, develop, and test solutions for their AI problems and host AI technologies developed by the JAIC’s Mission Initiatives (MIs).  These include National MIs led by a functional team in JAIC to address broader, cross-cutting AI challenges across DoD, as well as Component MIs led by DoD components, with support from JAIC, to use AI for component-specific goals and to solve discrete problems.  Where JCF provides the centralized support network, T&E will be the centralized point of contact for AI-subject matter expertise.

To ensure that MIs are delivering safe, effective products, the T&E office will provide testing services – e.g., algorithm, system, and operational testing – on the JCF network to evaluate these products and independently analyze AI performance of the products.  In addition, T&E will develop AI processes, tools, and standards to be used across DoD on the JCF network.

RFI Focuses on Three Distinct Areas with Identified Needs

For these reasons, T&E office is requesting information “on the development of T&E software for AI, creation of AI T&E processes and best practices, and the ability of vendors to provide AI test and evaluation services.”  As provided from the RFI, T&E is focused on these three areas with the following identified needs:

  • Various AI-technology and tools to develop T&E’s capabilities and support DoD’s emerging AI technologies – e.g., Machine Learning (ML), Deep Learning (DL), and Neural Networks / Neural Nets (NN) – with a focus on these capabilities:
    • Conversational interface applications using voice to text.
    • Speech enabled products and services for DoD applications and systems.
    • Image analysis, testing deep learning based visual search and image classifier.
    • Natural Language Processing (NLP) enabled products and services.
    • Humans augmented by machines, to include human-machine interfaces and improved methods to measure warfighter cognitive and physical workloads, to include augmented reality (AR) and virtual reality (VR) test services.
    • Autonomous systems.
  • Test and evaluation services from across the government, academia, and industry to assist in the test/evaluation of technologies provided by MIs or other DoD contributors. As provided from the RFI,  T&E is looking for support in:
    • Data Set Development/Curation: Each MI or DoD AI project has unique data that must be acquired and curated such that it can be used by modern AI software systems. Efforts include, but are not limited to, data quality assessment, image annotation, and data coverage analysis.
    • Test Harness Development: Creation of automated, containerized, cloud-enables test harnesses that facilitate automatic ingestion of data into a containerized AI model and aggregates the results.
    • Model Output Analysis: Analysis expertise/software is needed to assess the quality of results from the AI models developed by the MIs. This effort involves both development of containerized, cloud-based, analysis software and creation of context specific and operationally relevant performance metrics.
    • Test Planning, Documentation, and Reporting: Testing expertise to create the required documentation for an AI algorithm undergoing the testing lifecycle (Algorithm test, System test, and Operational test). This effort includes, but is not limited to, creation of Test and Evaluation Master Plans (TEMPS), Test Plans, creation of Test Report documenting findings, and management of a test schedule.
    • Testing Services: This effort involves the services and expertise to conduct test and evaluation of AI products.  This includes, but is not limited to, performing testing on AI algorithms and/or AI enabled systems across all T&E areas (Algorithm test, System test, Developmental test, and Operational test).
  • The identification of other technologies that are not included in the RFI and would be beneficial to T&E’s AI efforts.

Conclusion

As called for in DoD’s AI Strategy, this RFI is looking for assistance from industry and academic partners to provide AI tools and standardized processes for testing and evaluation across DoD.  These partnerships will help fulfill DoD’s vision in its AI Strategy for the JAIC by creating a DoD-wide mechanism offering tools, frameworks and standards, and unique services tailored to and in support of DoD’s deployment of AI.  Notwithstanding, any opportunities that result from this RFI and the information gathered by DoD may present challenges in navigating non-traditional procurement approaches (such as OTAs and public-private partnerships).

Tags: AI, DoD, JAIC, JCF, MIs
Print:
Email this postTweet this postLike this postShare this post on LinkedIn
Photo of Kate Growley Kate Growley

Businesses around the globe rely on Kate M. Growley to navigate their most challenging digital issues, particularly those involving cybersecurity, artificial intelligence, digital infrastructure, and their intersection with national security. Clients seek her guidance on proactive compliance, incident response, internal and government-facing investigations…

Businesses around the globe rely on Kate M. Growley to navigate their most challenging digital issues, particularly those involving cybersecurity, artificial intelligence, digital infrastructure, and their intersection with national security. Clients seek her guidance on proactive compliance, incident response, internal and government-facing investigations, and policy engagement. With a unique combination of legal, policy, and consulting experience, Kate excels in translating complex technical topics into advice that is practical and informed by risk and business needs.

Kate has extensive experience working with members of the U.S. government contracting community, especially those within the Defense Industrial Base. She has partnered with contractors from every major sector, including technology, manufacturing, health care, and professional services. Kate is an IAPP AI Governance Professional (AIGP) and a Certified Information Privacy Professional for both the U.S. private and government sectors (CIPP/G and CIPP/US). She is also a Registered Practitioner with the U.S. Cybersecurity Maturity Model Certification (CMMC) Cyber Accreditation Body (AB).

Having lived in Greater China for several years, Kate also brings an uncommon understanding of digital and national security requirements from across the Asia Pacific region. She has notable experience with the regulatory environments of Australia, Singapore, Japan, and Greater China—including the growing regulation of data flows between the latter and the United States.

Kate is a partner in the firm’s Washington, D.C., office, as well as a senior director in the firm’s consultancy Crowell Global Advisors, to which she was seconded for several years. She is a founding member of the firm’s Privacy & Cybersecurity Group and part of the firm’s AI Steering Committee. She has been internationally recognized by Chambers and named a “Rising Star” by both Law360 and the American Bar Association (ABA). She has held numerous leadership positions in the ABA’s Public Contract Law and Science & Technology Sections and has been inducted as a lifetime fellow in the American Bar Foundation.

Read more about Kate Growley
Show more Show less
Photo of Michelle Coleman Michelle Coleman

Michelle D. Coleman is a partner in Crowell & Moring’s renowned Government Contracts Group in the firm’s Washington, D.C. office. Michelle advises clients from diverse industries in connection with contract disputes and other government contract matters, including Contract Disputes Act (CDA) claims and…

Michelle D. Coleman is a partner in Crowell & Moring’s renowned Government Contracts Group in the firm’s Washington, D.C. office. Michelle advises clients from diverse industries in connection with contract disputes and other government contract matters, including Contract Disputes Act (CDA) claims and requests for equitable adjustments, terminations, prime-sub disputes, other transaction authority, and AI.

Michelle also has an active pro bono practice, representing clients as an attorney volunteer with the Washington Legal Clinic for the Homeless. Michelle has helped multiple clients receive long term housing through the Rapid Rehousing Program and other permanent voucher programs. In addition to being a volunteer, Michelle serves as an ambassador and as co-chaired the firm’s fundraising campaign for the Clinic for the last two years.

Prior to working at Crowell & Moring, Michelle served as an attorney in the Air Force’s Acquisition Law and Litigation Directorate, where she provided acquisition and litigation risk advice on procurements valued over $14 billion on major Air Force procurements. She also served as a trial attorney in the Air Force Legal Operations Agency, Commercial Law and Litigation Directorate. As a trial attorney, Michelle litigated complex contract disputes before the Armed Services Board of Contract Appeals (ASBCA) and bid protests before the Government Accountability Office (GAO).

As an Air Force litigator, Michelle litigated a broad range of issues before the ASBCA, including organizational conflicts of interest; small business issues; price realism analysis; past performance; NAICS code issues; technical acceptability; nonmanufacturing rule, brand name, or equal issues; construction claims; commercial items; terminations; assignment of claims; reprocurement; limitation of funds; release; differing site conditions; setoffs/withholding; and evidentiary issues. Among the construction cases, Michelle litigated a $28 million Air Force design-build construction claim involving complex differing site conditions and delay issues, and she also litigated and won a claim for alleged defective specifications, undisclosed information, constructive interpretation, and technical impossibility for a contract for the design and construction of an Air Force dynamic break test stand.

Before her Air Force career, Michelle was employed by a defense contractor, where she gained valuable government contract experience in her roles as a business analyst and a subcontracts administrator. Michelle’s government and contractor experience gives her the unique ability to take both parties’ perspectives into consideration when providing advice on government contract issues.

Read more about Michelle Coleman
Show more Show less
Photo of Laura J. Mitchell Baker Laura J. Mitchell Baker

Laura J. Mitchell Baker is a counsel with Crowell & Moring’s Government Contracts Group in the firm’s Washington, D.C. office.

Laura represents government contractors in litigation and administrative matters, including contract disputes with state and federal entities, suspension and debarment proceedings, mandatory disclosures…

Laura J. Mitchell Baker is a counsel with Crowell & Moring’s Government Contracts Group in the firm’s Washington, D.C. office.

Laura represents government contractors in litigation and administrative matters, including contract disputes with state and federal entities, suspension and debarment proceedings, mandatory disclosures to the government, prime-sub disputes, and False Claims Act investigations. Her practice also includes counseling on federal, state, and local government contracts, government contracts due diligence, and regulatory and compliance matters, as well as conducting internal investigations.

Read more about Laura J. Mitchell Baker
Show more Show less
Related Posts
DOJ Announces Safe Harbor for Acquirers Who Disclose Pre-Acquisition Misconduct
October 13, 2023
Biden Administration Moves Closer to Establishing Framework for Giving Preference to Bids and Contractors with Lower GHG Emissions
October 3, 2023
A Brief Primer on the Impact of a Federal Government Shutdown
September 29, 2023