Neural networks can perform well even when fitting noisy data, leading to two types of overfitting: "tempered" and "benign". Researchers studied ReLU neural networks and found that in one-dimensional data, overfitting is tempered, but in high dimensions, it becomes benign. This means that the input dimension plays a crucial role in determining the type of overfitting in neural networks.