Operationalizing AI Auditability Measures: An AI Audit Case Study of College Admissions System

Abstract

The pervasive integration of artificial intelligence (AI) technologies promises to improve efficiency across various facets of society but to add unforeseen risks to society. Auditing AI systems is critical to mitigating these risks and to ensure lawfulness, ethicality, morality, and technical robustness in AI systems. To enable audits for AI systems, auditability measures must be implemented during various stages of the lifecycle of an AI system. Previous studies have identified measures of AI audit success; these measures are however conjectures without concrete evidence of their efficacy. In this study, we survey AI audit practitioners to identify most important auditability measures in practice and conduct an experiment to examine how auditability measures implemented in AI systems affect AI audit feasibility, effectiveness, and efficiency. Our findings will provide valuable insights to policymakers, technology providers, audit practitioners, and business users on auditability measures that contribute to AI audit success.