CIRCLE: A Framework for Evaluating AI from a Real-World Lens
Authors: Schwartz, R., Westling, C., Briggs, M., Fadaee, M., Nejadgholi, I., Holmes, M., Rashid, F., Carlyle, M., Taïk, A., Wilson, K., Douglas, P., Skeadas, T., Waters, G., Chowdhury, R., Lacerda, T.
Journal: Lecture Notes in Networks and Systems
Publication Date: 01/01/2027
Volume: 2075 LNNS
Pages: 737-756
eISSN: 2367-3389
ISSN: 2367-3370
DOI: 10.1007/978-3-032-32732-1_41
Abstract:This study proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the “reality gap” between model-centric performance metrics and AI system outcomes in deployment. Current approaches such as MLOps frameworks and AI model benchmarks offer detailed insights into system stability and model capabilities, but they do not provide decision-makers outside the AI stack with systematic evidence of how these systems actually behave in real-world contexts or affect their organizations over time. CIRCLE operationalizes the “Validation” phase of TEVV (Test, Evaluation, Verification, and Validation) by translating priorities of stakeholders outside the stack into measurable signals. Unlike participatory design which often remains localized, or algorithmic audits which are often retrospective, CIRCLE provides a structured, prospective protocol for linking context-sensitive qualitative insights to scalable quantitative metrics. By integrating methods such as field testing, red teaming, and longitudinal studies into a coordinated pipeline, CIRCLE produces “systematic knowledge”—evidence that is comparable across sites yet sensitive to local context. This can enable governance based on materialized downstream effects rather than theoretical capabilities.
https://eprints.bournemouth.ac.uk/41721/
Source: Scopus
CIRCLE: A Framework for Evaluating AI from a Real-World Lens
Authors: Westling, C., Schwartz, R., Briggs, M., Carlyle, M., Holmes, M., Fadaee, M., Waters, G., Taik, A., Lacerda, T.
Journal: Intellisys
Publication Date: 30/09/2026
Abstract:Most AI evaluations rely on static benchmarks that measure model outputs in isolation, offering little evidence about how systems behave once embedded in real-world workflows. As a result, decision- makers lack systematic evidence about downstream effects, operational risks, and long-term impacts that matter for deployment, governance, and procurement. We introduce CIRCLE, a six-stage lifecycle-based framework that links stakeholder concerns to context-sensitive evaluation meth- ods, longitudinal measurement, and ongoing monitoring of deployed AI systems. The framework integrates evaluation methods such as A/B testing, field testing, red teaming, and longitudinal studies into a coordi- nated evaluation pipeline rather than treating them as isolated activities.
Together, these methods support more contextualized, iterative, and decision-relevant assessments of AI systems. By aligning constructs, meth- ods, and metrics with real deployment contexts, CIRCLE supports more actionable, iterative, and governance-relevant evaluation of AI systems and their secondary and tertiary effects.
https://eprints.bournemouth.ac.uk/41721/
Source: Manual