Visibility and monitoring in deployed ML systems

Modern machine learning has ushered in an era of unparalleled system capabilities, exemplified by self-driving cars and synthetic speech indistinguishable from human. However, these techniques bring with them the challenge of monitoring and understanding the behaviour of live ML systems.

In this talk, we share insights learned from building tools and workflows for monitoring ML systems. We cover a variety of topics including the need to reevaluate software monitoring practices in light of ML systems, the specifics of what to monitor and how to monitor it (ranging from population drift or domain shift to historical backtests), and the significance of involving machine learning engineers in the monitoring process.