“Microsoft’s Ronny Kohavi, a pioneer in online experimentation since his days as director of data mining and personalization at Amazon.com, likes to challenge his Internet-savvy audiences with a little experiment. Kohavi … shows screenshots of simple but real A/B experiments …. Some visitors saw the A version, some the B version … Kohavi then challenges his audience to predict the real-world outcomes of the test: Did Option A win? Did Option B win? Or were the results a statistical dead heat?
Almost without exception, his audiences fragment; not even a rough consensus emerges. ‘Stop debating,’ Kohavi advises. ‘It’s easier to get the data … Data trumps intuition.’
Data from experiments such as Kohavi’s should ideally encourage executive curiosity and humility. Unfortunately, that rarely happens. Too many executives worldwide would rather use their expertise to ‘think’ or ‘intuit’ what’s best for their customers than run even the simplest experiment. They take pride in their’ gut.’ That’s a huge managerial mistake.
… Leading researchers, such as Daniel Kahneman, who received the 2002 Nobel Prize in Economic Sciences, and psychologist Philip E. Tetlock, have persuasively demonstrated that ‘experts’ are particularly prone to predictive overconfidence.
For example, in a study where experts were asked to make predictions related to world events, Tetlock reported in a 1999 article that’ across all seven scenarios, experts were only slightly more accurate than one would expect from chance.’ What’s more, Tetlock found that the ‘experts who assigned confidence estimates of 80 percent or higher were correct only 45 percent of the time’. The overconfidence — arrogance? — of presumed expertise is counterproductive. Pride goeth before a fall. Data trumps intuition. Serious innovators take data seriously.”