CIRCLE: A Framework for Evaluating AI from a Real-World Lens
Authors: Westling, C., Schwartz, R., Briggs, M., Carlyle, M., Holmes, M., Fadaee, M., Waters, G., Taik, A. and Lacerda, T.
Journal: Intellisys
Abstract:Most AI evaluations rely on static benchmarks that measure model outputs in isolation, offering little evidence about how systems behave once embedded in real-world workflows. As a result, decision- makers lack systematic evidence about downstream effects, operational risks, and long-term impacts that matter for deployment, governance, and procurement. We introduce CIRCLE, a six-stage lifecycle-based framework that links stakeholder concerns to context-sensitive evaluation meth- ods, longitudinal measurement, and ongoing monitoring of deployed AI systems. The framework integrates evaluation methods such as A/B testing, field testing, red teaming, and longitudinal studies into a coordi- nated evaluation pipeline rather than treating them as isolated activities.
Together, these methods support more contextualized, iterative, and decision-relevant assessments of AI systems. By aligning constructs, meth- ods, and metrics with real deployment contexts, CIRCLE supports more actionable, iterative, and governance-relevant evaluation of AI systems and their secondary and tertiary effects.
Source: Manual