New research method reveals how businesses can run better experiments

Many companies today use experiments to gather the data that drives their strategies. That’s how Facebook knows whether a newsfeed change increased user engagement, or how a bank knows whether its new promotional strategy increased deposits.

But a phenomenon called interference has always made getting to that clear cause-and-effect difficult. Now researchers at The University of Texas (UT) at Austin have designed a new method that illuminates how interference affects the results of these randomized control trials, enabling social and behavioral researchers to account for interference across a broad range of applications.

Whenever an experimental treatment can change the behavior of the units outside the treatment group, there’s interference, and it muddies causal inference—the important cause-and-effect conclusions researchers draw.

“This phenomenon occurs any time I run a randomized experiment and my experimental units are connected with each other in some way—in social networks, friend groups, households, trade networks, and city streets,” said lead researcher David Puelz, UT McCombs School of Business clinical assistant professor of finance.

In social media research, people in the treatment and control groups may be friends. In e-commerce research, people in each study group may live in the same household.

“Without accounting for interference, estimates of causal effects will be very wrong,” Puelz said. “But if incorporated into a statistical analysis in the right way, interference can be a blessing. So we designed a method that has network interference directly baked in.”

The findings are published online in advance in the Journal of the Royal Statistical Society. The research team included Guillaume Basse of Stanford University, Avi Feller of the University of California-Berkeley, and Panos Toulis of the University of Chicago’s Booth School of Business.

The new method can test directly for interference effects, such as spillovers and peer effects in social networks. For example, Facebook can test whether a newsfeed change affects not only the user, but also friends of that user. Policymakers can test whether mask wearing alters viral spread within and across households, and human resources divisions in companies can investigate whether wellness programs change health outcomes among groups of employees.

The research focused on spatial interference, drawing on data from a large-scale policing experiment conducted by a separate team in Medellín, Colombia, in 2015. That team measured crime on all the city’s streets, identified hotspots, and then asked whether extra police patrols on hotspot streets affected crime on adjacent streets. Did the intervention reduce crime or just shift it to another street?

To find out, the researchers constructed a graph that encoded information about the interference structure of Medellín’s street segments and every possible combination of policing assignments. Their method then uses this novel graph construct with a new algorithm to break it up into two relevant subsets: Spillover units that weren’t treated but may be linked to a treated unit, and pure control units that weren’t treated or linked.

On this smaller graph, they can then perform what’s called a Fisher randomization test to determine whether there is a spillover effect—whether crime on streets near policed streets differed from those far away from policed streets.

The research confirmed previous findings that extra patrolling on one street reduces crime on adjacent streets. But beyond crime in Medellín, the computational approach Puelz and team created has application potential for any research that is confounded by network interference.

“It can also be used at Facebook and Google in all of the thousands of experiments that they’re conducting every single day, or at any other company conducting A/B tests and internal experiments to improve operations,” Puelz said.