Differential Assessment of Black-Box AI Agents

Rashmeet Kaur Nayyar, Pulkit Verma, Siddharth Srivastava

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations


Much of the research on learning symbolic models of AI agents focuses on agents with stationary models. This assumption fails to hold in settings where the agent's capabilities may change as a result of learning, adaptation, or other post-deployment modifications. Efficient assessment of agents in such settings is critical for learning the true capabilities of an AI system and for ensuring its safe usage. In this work, we propose a novel approach to differentially assess black-box AI agents that have drifted from their previously known models. As a starting point, we consider the fully observable and deterministic setting. We leverage sparse observations of the drifted agent's current behavior and knowledge of its initial model to generate an active querying policy that selectively queries the agent and computes an updated model of its functionality. Empirical evaluation shows that our approach is much more efficient than re-learning the agent model from scratch. We also show that the cost of differential assessment using our method is proportional to the amount of drift in the agent's functionality.

Original languageEnglish (US)
Title of host publicationAAAI-22 Technical Tracks 9
PublisherAssociation for the Advancement of Artificial Intelligence
Number of pages9
ISBN (Electronic)1577358767, 9781577358763
StatePublished - Jun 30 2022
Event36th AAAI Conference on Artificial Intelligence, AAAI 2022 - Virtual, Online
Duration: Feb 22 2022Mar 1 2022

Publication series

NameProceedings of the 36th AAAI Conference on Artificial Intelligence, AAAI 2022


Conference36th AAAI Conference on Artificial Intelligence, AAAI 2022
CityVirtual, Online

ASJC Scopus subject areas

  • Artificial Intelligence


Dive into the research topics of 'Differential Assessment of Black-Box AI Agents'. Together they form a unique fingerprint.

Cite this