There were two groups of scientists working on the same task in two separate labs. One set of scientists had vague credentials. A couple physicists, chemists, and biologists grouped up together. The other set were scientists that were experts in their field in E. Coli. And both were experimenting with proteins from E. Coli.
Psychologist Dunbar was studying how scientific research was being conducted. And he noticed an interesting case between these two groups. Both of them had found a similar problem where the proteins would get stuck to a filter. But the group of general scientists quickly drew upon knowledge from their separate fields and solved the problem in the same day. However, the E. Coli specialists took weeks before they could finally solve the problem.
What allowed a group of scientists to solve a challenge that specialized individuals couldn’t, even with their disadvantage in relative experience? Well it turns out this disadvantage was the very thing that allowed them to solve the problem. And it allows a group of generalists to actually have an advantage over experts not just in E. Coli, but nearly any field.
The Overfitting Issue
Most machine learning applications try to learn some type of data. Computer Scientists can build a model so that, based on input data, the model can make some prediction. We train this model so it learns to build the general function that makes these predictions. A good example of this would be to determine the price of a house based on the square feet inside of it. You can see the general problem more clearly in the image below.
Once the model learns the function, we can use it to make predictions on data it hasn’t seen before. This can be useful, for example, to predict the price of a house based on the number of rooms it has and the total space. But there is something we have to watch out below
This is what we call overfitting. While the model learns how to make predictions based on the data it has seen, it doesn’t generalize outside of that data. Even though we are told that its accuracy is 100% for this task, which should be better than the previous model, we can clearly see it will not perform well when given new data.
Is this the mistake we often make in our learning? In our pursuit of getting test scores, we learn only the test material without understanding fundamental concepts. In our pursuit of perfection, we memorize a specific technique or algorithm that works for the time being. In our pursuit of deep knowledge, we trust those who have achieved the highest degree of specialization in the field.
But it doesn’t matter how well I understand SAT problems if I didn’t understand the math. It doesn’t matter how well I’ve memorized Beethoven’s 5th Symphony on the piano if I can’t learn how to play his 4th. It doesn’t matter how capable I am of recalling an E. Coli enzyme if there’s a problem with the protein its interacting with.
Cognitive psychologist Nate Kronell noted that “some people argue that part of the reason U.S. students don’t do as well on international measures of high school knowledge is that they’re doing too well in class.” We’re overfitting to the metrics we’ve defined in our learning. And it results in a hefty price when we branch outside of that scope. We no longer have the generalization we need to face new problems. Because it’s a lot simpler to say we’re already at 100% and just use our old methods. So how do we solve this?
Regularization
During the 17th century, many scientists imagined all planetary movement to be caused by individual spirits. And these spirits were guided by pure crystalline spheres which were invisible from Earth. This was the wisdom that Johannes Kepler himself believed as common fact. But only until he observed a comet across the sky, which made him think that such an object should have broken these crystalline spheres in the heavens.
He started wondering why planets moved more slowly than others and what is actually causing these movements. But he was so far out of his reach, with no scientific evidence, that he had to draw upon things outside of his experience to find this invisible guiding motion.
He started by analyzing the nature of light and wondered if this was the source of the Sun’s pull. But he found the planetary motions unaffected by an eclipse. So he had to move on to something else. He then read a new paper on magnetism and suddenly was stricken by the idea of invisible but interacting magnets at the planetary poles. But that didn’t explain why the planets would move in their orbits.
Like this, Kepler kept experimenting with analogy after analogy to find what was happening, concluding that there was some notion of bodies pulling one another, and that the larger a body was, the more pull it had. He was even able to predict based off of this that tides were caused by the moon. And keep in mind, this was before there was any conception of physical forces governing the universe. He essentially found the idea of gravity by letting go of his previous model of the universe to keep experimenting.
David Epstein recounts this tale in his book Range as an example of someone willing to think outside their domain and outside their experience. Something that’s only possible when one isn’t focused on optimizing towards a certain field. And in fact, Epstein is able to showcase numerous studies showing the top athletes, musicians, artists, and scientists were not those who focused deeply on their subject, but went through a sampling period where they experimented with a variety of sports, instruments, etc.
If you don’t want to overfit your knowledge, you have to be willing to go outside your own expertise. Most people gravitate towards using familiar techniques and intuitions to solve unexpected problems. But it’s those that are willing to completely jump outside their box that generate the larger solution.
Find a new hobby. Develop a new interest. Never get fixed upon a certain mode of thinking. And learn to think beyond your experience. Then you’ve allowed yourself to generalize well to the challenges ahead of you.
