Model Drift

Model drift is the phenomenon where a model becomes less accurate, less precise, and overall less predictive over time due to changes in the relationship between the target and input variables. We can think of a model, in its most simple form, as an identified 'relationship' between two or more variables, such that a change in one causes a systematic change in another. For example, you could have a model of the relationship between temperature and ice cream sales, where when the temperature goes up so too do sales in ice cream.

However, the world is always changing, and the underlying relationship between two variables is not guaranteed to always be consistent. If ice cream suddenly becomes unpopular - maybe everyone in the group you’re modeling decides they prefer frozen yogurt - the predictive relationship of temperature to ice cream will weaken. The model that once accurately reflected reality has drifted away from it.

Model drift can have different causes, but broadly speaking it's due to either 'concept drift', where the variable definitions have changed, or 'data drift', where the underlying real world data has changed.

The key to fighting model drift is continuous monitoring and optimization. Models can and should be refitted frequently against new, updated datasets, to ensure that the models still reflect ground truth, and inspected to ensure that the concepts are still accurate.