NSORS Modeling Surprise WHO Eric Horvitz, Microsoft Research Combining massive quantities of data, insights into human psychology, and machine learning can help humans manage DEFINITION surprising events, says Eric Horvitz. Surprise modeling combines data mining and machine learning to By M. Mitchell Waldrop help people do a better job of anticipating and coping with Much of modern life depends on forecasts: where the next unusual events. hurricane will make landfall, how the stock market will react IMPACT to falling home prices, who will win the next primary. While Although research in the field is existing computer models predict many things fairly preliminary, surprise modeling accurately, surprises still crop up, and we probably can’t could aid decision makers in a wide eliminate them. But Eric Horvitz, head of the Adaptive range of domains, such as traffic management, preventive medicine, Systems and Interaction group at Microsoft Research, thinks military planning, politics, business, we can at least minimize them, using a technique he calls and finance. “surprise modeling.” CONTEXT A prototype that alerts users to Horvitz stresses that surprise modeling is not about building a surprises in Seattle traffic patterns technological crystal ball to predict what the stock market will has proved effective in field tests do tomorrow, or what al-Qaeda might do next month. But, he involving thousands of Microsoft says, “We think we can apply these methodologies to look at employees. Studies investigating broader applications are now under the kinds of things that have surprised us in the past and then way. model the kinds of things that may surprise us in the future.” The result could be enormously useful for decision makers in fields that range from health care to military strategy, politics to financial markets. Granted, says Horvitz, it’s a far-out vision. But it’s given rise to a real-world application: SmartPhlow, a traffic-forecasting service that Horvitz’s group has been developing and testing at Microsoft since 2003. SmartPhlow works on both desktop computers and Microsoft PocketPC devices. It depicts traffic conditions in Seattle, using a city map on which backed-up highways appear red and those with smoothly flowing traffic appear green. But that’s just the beginning. After all, Horvitz says, “most people in Seattle already know that such-and-such a highway is a bad idea in rush hour.” And a machine that constantly tells you what you already know is just irritating. So Horvitz and his team added software that alerts users only to surprises—the times when the traffic develops a bottleneck that most people wouldn’t expect, say, or when a chronic choke point becomes magically unclogged. TR10: Modeling Surprise TechnologyReview.com But how? To monitor surprises effectively, says Horvitz, the machine has to have both knowledge—a good cognitive model of what humans find surprising—and foresight: some way to predict a surprising event in time for the user to do something about it. Horvitz’s group began with several years of data on the dynamics and status of traffic all through Seattle and added information about anything that could affect such patterns: accidents, weather, holidays, sporting events, even visits by high-profile officials. Then, he says, for dozens of sections of a given road, “we divided the day into 15-minute segments and used the data to compute a probability distribution for the traffic in each situation.” That distribution provided a pretty good model of what knowledgeable drivers expect from the region’s traffic, he says. “So then we went back through the data looking for things that people wouldn’t expect—the places where the data shows a significant deviation from the averaged model.” The result was a large database of surprising traffic fluctuations. Once the researchers spotted a statistical anomaly, they backtracked 30 minutes, to where the traffic seemed to be moving as expected, and ran machine-learning algorithms to find subtleties in the pattern that would allow them to predict the surprise. The algorithms are based on Bayesian modeling techniques, which calculate the probability, based on prior experience, that something will happen and allow researchers to subjectively weight the relevance of contributing events (see TR10: “Bayesian Machine Learning,” February 2004). The resulting model works remarkably well, Horvitz says. When its parameters are set so that its false-positive rate shrinks to 5 percent, it still predicts about half of the surprises in Seattle’s traffic system. If that doesn’t sound impressive, consider that it tips drivers off to 50 percent more surprises than they would otherwise know about. Today, more than 5,000 Microsoft employees have this “surprise machine” loaded on their smart phones, and many have customized it to reflect their own preferences. Horvitz’s group is working with Microsoft’s traffic and routing team on the possibility of commercializing aspects of SmartPhlow. And in 2005 Microsoft announced that it had licensed the core technology to Inrix of Kirkland, WA, which launched the Inrix Traffic application for Windows Mobile devices last March. The service offers traffic predictions, several minutes to five days in advance, for markets across the United States and England. Although none of the technologies involved in SmartPhlow is entirely new, notes Daphne Koller, a probabilistic-modeling and machine-learning expert at Stanford University, their combination and application are unusual. “There has been a fair amount of work on anomaly detection in large data sets to detect things like credit card fraud or bioterrorism,” she says. But that work emphasizes the detection of present anomalies, she says, not the prediction of events that may occur in the near future. Additionally, most predictive models disregard statistical outliers; Horvitz’s specifically tracks them. The thing that makes his approach unique, though, is his focus on the human factor, Koller says: “He’s explicitly trying to model the human cognitive process.” The question is how wide a range of human activities can be modeled this way. While the algorithms used in SmartPhlow are, of necessity, domain specific, Horvitz is convinced that the overall approach could be generalized to many other areas. He has already talked with political TR10: Modeling Surprise TechnologyReview.com scientists about using surprise modeling to predict, say, unexpected conflicts. He is also optimistic that it could predict, for example, when an expert would be surprised by changes in housing prices in certain markets, in the Dow Jones Industrial Average, or in the exchange rate of a currency. It could even predict business trends. “Over the past few decades, companies have died because they didn’t foresee the rise of technologies that would lead to a major shift in the competitive landscape,” he says. Most such applications are a long way off, Horvitz concedes. “This is a longer-term vision. But it’s very important, because it’s at the foundation of what we call wisdom: understanding what we don’t know.” Copyright © 2008, Technology Review Inc. All Rights Reserved. http://www.technologyreview.com TR10: Modeling Surprise TechnologyReview.com .