Reasoning About Interventions in Data and Machine Learning Systems

Sep 09, 2026

Episode #920

Download: MP4 Video Size: 268.7MB  
Watch on YouTube

Abstract

Modern machine learning systems are only as reliable as the data and pipelines that support them, yet improving their behavior often requires navigating enormous spaces of possible data and system configurations. These systems are complex enough that improving them by changing everything at once is both inefficient and unreliable. This talk explores a simple principle for building smarter data and machine learning systems: identify which intervention is most likely to change the outcome, and act there.  I will first introduce DataSift that applies this principle to model behavior, identifying the most influential data to expand the training data when the goal is to improve fairness without sacrificing predictive performance. By combining data valuation, influence functions, and multi-armed bandits, DataSift identifies small, high-impact subsets of candidate data rather than indiscriminately adding data to the training set. Next, I will present PipeLens that applies the same principle to data science pipelines, identifying the components and parameters whose intervention is most likely to repair a malfunctioning pipeline. By learning from successful and failed pipeline executions, PipeLens identifies causally relevant root causes and efficiently searches for interventions that restore pipeline utility. Despite addressing different problems, both systems replace brute-force search with targeted intervention using principled reasoning about influence and causality to determine what to change, why it matters, and how to change it efficiently.