A Systematic Approach to Evaluation of Delivered AI Solutions Torch Technologies, Huntsville, AL

Format:

Book

Conference/Event

Author/Creator:

Wells, David, author.

Contributor:

Elrod, Ivy

Foley, Jacob

Fowler, Stuart

Walters, Josh

Conference Name:

2024 NDIA Michigan Chapter Ground Vehicle Systems Engineering and Technology Symposium (2024-08-13 : Novi, Michigan, United States)

Language:

English

Physical Description:

1 online resource cm

Place of Publication:

Warrendale, PA SAE International 2024

Summary:

An approach to performance assessment and evaluation of robustness' of delivered AI (artificial intelligence) solutions (e.g., Aided Target Detection and Recognition (AiTDR) or ground vehicle autonomy behaviors) is presented. The initial development assumes the AI solution is delivered as a blackbox solution taking specified inputs and delivering output or outputs per stated requirements. The methods developed seek to not only confirm that requirements are met but to confirm performance in a manner that provides supporting evidence of which input information contributed to the AI output. Methods are developed for both AI solutions that take a single sample input as well as AIs that take a sequence of inputs to produce an output. Examples are included for convolutional neural networks (CNN). Planned extensions to the blackbox methods are discussed. Extensions include adaptations to support Residual Network (ResNet) solutions, evaluation of solution robustness when hidden layer activations are exposed, and development of metrics and methodologies to support AI solution training by ensuring the AI solution is properly trained, not over trained, or undertrained. A process is proposed for assessment of the AI solution with metrics to identify primary contributing inputs to the solution and an approximate indication of where the solution falls along the training scale

Notes:

Vendor supplied data

Publisher Number:

2024-01-4068

Access Restriction:

Restricted for use by site license