Leaders too often misinterpret empirical patterns and miss opportunities to engage in data-driven thinking. To better leverage data, leaders need to understand the types of problems data can help solve as well as the difference between those problems that can be solved with improved prediction and those that can be solved with a better understanding of causation.
Too many leaders take an incomplete approach to understanding empirical patterns, leading to costly mistakes and misinterpretations. As we have discussed before, one extremely common mistake is interpreting a misleading correlation as causal. We’ve advised countless organizations on the topic. We’ve written research papers, managerial articles, and even a book dedicated to the power of experiments and causal inference tools — a toolkit that economists have adopted and adapted over the past few decades. Yet, while we are deep believers in the causal inference toolkit, we’ve also seen the reverse problem — leaders who overlook useful patterns because they are not causal. The truth is, there are also times when a correlation is not only sufficient, but is exactly what is needed. The mistake leaders make here is failing to understand the distinction between prediction and causation. Or, more specifically, the distinction between predicting an outcome and predicting how a decision will affect an outcome.
Consider a manager that is struggling with the following question: Should I subsidize college degrees for my employees? She might start by examining the relationship between college degrees and productivity. Yet, even if she sees a positive association between college degrees and productivity, it is hard to know — without further analysis — whether this relationship is causal. After all, there are likely to be other underlying differences between people with and without degrees. And offering education subsidies to the degree-less employees won’t make them identical to the other employees who already have a degree. She would need an experiment, or natural experiment, to better understand whether this relationship is causal.
Now, suppose the same manager was grappling with a slightly different question: Should I hire more college graduates? She might again look at the correlation between college degrees and productivity to consider whether she’d hire more productive workers by tweaking hiring to place more weight on a degree. In this case, the correlation is useful — since it is helping to predict who will be productive, even if it says nothing about whether the degree is causing productivity.
A subtle but critical difference exists between these two questions. “Should I hire more college graduates?” is a prediction problem. “Should I subsidize college degrees for my employees?” is a causal inference problem. In the former, she is trying to assess whether college degrees are predictive of productivity. In other words, are the kinds of people who get college degrees good employees? In the latter, she is trying to determine whether college degrees cause higher productivity.
This distinction is critical for decision makers: When considering hiring employees with a college degree, the manager needs predictive tools, which can range from basic correlations to more advanced machine learning algorithms. She might not need to know whether degrees are having a causal effect (or if, instead, the kind of people who get college degrees also happen to be productive employees). When considering subsidizing college degrees for her employees, however, understanding whether it’s the actual college education that causes higher productivity should be her core question. To successfully determine whether degrees will help improve current employee performance, she needs the tools of causal inference, such as experiments or natural experiments, which are focused on understanding the causal impact of making a change.
Here we provide examples of common causal inference and prediction problems. We draw out key distinctions between the two types of problems and point to different tools leaders need when confronting each.
Common causal inference problems
Managers regularly face decisions that involve thinking through the causal impact of different options. Will hiring consultants improve our company’s productivity? Will higher wages reduce turnover? Will advertising on social media draw in new customers?
These questions have all been answered using the causal inference methods from social science. For instance, economists Emma Harrington and Natalia Emanuel, in conjunction with a large tech company, examined wages within the company’s call centers and warehouses. In 2019, the company increased pay for warehouse workers from $16 an hour to $18 an hour. Looking at the timing of the pay raise, the researchers were able to see the effect of higher wages on productivity using a difference-in-differences approach. They found that the raises not only increased productivity, but also that a $1 increase reduced the chances an employee would quit by 19%. As it turns out, it was profitable to increase wages, as the pay hikes more than paid for themselves through the productivity boost and decline in turnover.
As a second example, consider a recent analysis by Brett Gordon, Florian Zettelmeyer, Neha Bhargava, and Dan Chapsky which looks at advertising campaigns run on Facebook. Looking at 15 U.S.-based advertising campaigns consisting of roughly 1.6 billion advertising impressions, the researchers compare the estimates of the impact of advertisements on Facebook from experiments to the estimates from non-experimental correlations. The team found that the non-experimental correlations between advertisements and purchase intentions were misleading, as advertisements are targeted and tend to be shown to users who are already inclined to purchase a product. For instance, laundry detergent ads are going to be shown to people who are already inclined to buy laundry detergent even in the absences of the ad. The authors then investigated different non-experimental approaches to controlling for characteristics of users, and found that the correlation remained misleading despite the controls. Even more advanced statistical controls didn’t eliminate this ‘selection bias’ problem. This is because selection bias is especially severe in the context of online advertisements, where advertisements are heavily targeted and where effects tend to be small on a per impression basis, which means that even small amounts of bias can lead to very misleading estimates overall. In that context, experiments can be a powerful way to overcome selection bias and to identify the causal impact of advertisements.
A third example comes from the world of financial products, where one of us (Dean), with colleagues Jeremy Burke, Julian Jamison, Kata Mihaly, and Jonathan Zinman, ran a study with a credit union in St Louis. It looked at a popular “credit builder” loan product designed to help those who wanted to establish a credit history do so. Indeed, if you just looked a correlation, you’d find that people who availed themselves of the product designed to build credit scores did go on to build credit scores — success! But because the credit union had randomized the offers, they found plenty of people similar to those successful clients who hadn’t been offered that product also went on to build good credit scores on their own. Again, we have problem of the college degree correlation — the people who are the type of people who want it, tend to be the type to be successful. It wasn’t the product that did it, but the correlation might make you think it was.
These are just three of many examples of how the causal inference toolkit can answer critical questions in areas ranging from operations to strategy to marketing.
Common prediction problems
If your employees or customers are a self-selecting group, does that mean you’re out of luck? No, finding out a credit improvement product seemed to lead to no increase in scores, might be interpreted as a failure of the product, but it’s not a failure of information. Recall that a user’s decision to use the product turned out to be quite predictive of whether their score would improve. If you are the bank, that is information you can use. For example, you may want to use similar information to assess credit risks. Banks might be more willing to give credit to individuals with low credit scores who elect to use a credit improvement product compared to individuals who don’t use the product. The reason is simple: Using the product is predictive of future behavior, even though it is not causing the behavior.
Managers in all industries regularly face decisions that involve making predictions.
Machine learning and artificial intelligence are extremely valuable in these contexts. Our own research has documented the potential for algorithms to lead to more efficient hiring and promotion processes in areas ranging from teachers to police officers. Recent work has further explored these ideas, and found that algorithms have the potential to increase both efficiency and equity of hiring. For instance, consider a recent paper by economists Danielle Li, Lindsey Raymond, and Peter Bergman, which examines the value of using an algorithm to screen resumes — with data on roughly 90,000 job applications to a Fortune 500 firm between 2016 and 2019. Comparing multiple algorithms to human decision makers, the researchers found the algorithms helped to identify better candidates in the screening than the people did, leading to a higher likelihood that the candidates were hired. Moreover, when carefully designed, the algorithms led to both higher quality candidates and more demographically diverse candidates. But, to get there, the organization needed to realize that there is an element of prediction in hiring and needed to be clear about what its hiring goals were are.
As a third example, suppose that you were to see a correlation between a given year’s most popular cuisines in Boston and the prior year’s most popular cuisines in New York. Even if the link is not causal, the correlation is valuable. For instance, it can be insightful for restaurants that are looking to innovate in their menus. One of us (Mike) has seen this type of question come up in his work with Yelp, where it is possible to look at large scale data sets to answer this type of question. This work has helped to ways in which data from tech companies can shed light on the evolution of economic activity. For instance, Yelp data can help to provide insight into the ways in which gentrification affects different types of businesses. It can also help to predict changes in economic activity. More broadly, data from tech companies has been one important new source of information — and has now been widely used for both causal inference and prediction problems.
Choosing the right machinery
“We are drowning in information but starving for wisdom.” This quote, from biologist E.O. Wilson, captures the essence of the modern business ecosystem. The world is awash in data. And advances in data analytics over recent decades have the potential to improve managerial decisions in virtually all sectors and for a wide range of problems. A large body of economics and statistics literature has explored the ways in which artificial intelligence has reduced the cost of making predictions, in settings ranging from hiring to investing to driverless cars. In parallel, the development of causal inference tools has been recognized in the 2019 and 2021 Nobel Prizes in Economics. Both are important for business decisions.
Yet, leaders too often misinterpret empirical patterns and miss opportunities to engage in data-driven thinking. To better leverage data, leaders need to understand the types of problems data can help solve as well as the difference between those problems that can be solved with improved prediction and those that can be solved with a better understanding of causation.