Machine learning models vulnerable to distribution shift, not overfitting.
The article explores how machine learning models can be reliable and robust. The researchers developed new ways to measure overfitting and generalization in these models. They found that different optimization algorithms can affect how well a model generalizes to new data. They also discovered that commonly used datasets may not always accurately represent real-world scenarios. Surprisingly, despite efforts to make models perform well on specific benchmarks, there was little evidence of overfitting in machine learning competitions. The study suggests that the main concern for machine learning models is adapting to changes in data distribution, rather than overfitting.