Virtual Paper Review – The TEVV-Athlon Framework

Join us virtually this Wednesday at 6 pm to continue our monthly Paper Review series! This month we read a new NIST draft, “The TEVV-Athlon Framework for Evaluating AI Systems” (NIST AI 200-2). TEVV stands for test, evaluation, verification, and validation.

The question behind it is simple to ask and hard to answer: what evidence shows that an AI system does what we need it to do, and how much can we trust that evidence? The framework borrows its name from the triathlon. No single event measures athleticism, and no single benchmark measures an AI system.

A TEVV-Athlon starts from an organization’s goals and works through four stages to a custom assessment. It keeps the concepts being measured separate from the activities and instruments that produce the evidence. Every result then traces back to the goal it was meant to answer.

We will walk through the four stages and the paper’s guidance on benchmarks, red teaming, user testing, and measurement validity. Then we will walk through an implementation of the framework in the context of a tabletop gaming platform.

Links:

• NIST AI 200-2 ipd (Initial Public Draft) – https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.200-2.ipd.pdf