V. 16 no. 5 • May/Jun 2013

Does Yelp Affect Restaurant Demand? Michael L. Anderson and Jeremy Magruder

eliefs on product quality play an of the display system at Yelp. com—a important role in shaping con­ popular site that allows users to leave Internet review forums increasingly sumer demand. For many goods, reviews of local businesses—to esti­ supplement expert opinion and social B consumers face ex ante uncertainty mate the effect of positive Yelp rat­ networks in informing consumers regarding the quality of the good and ings on restaurant customer flows. about product quality. We estimate the effect of positive Yelp.com ratings rely on imperfect signals to infer qual­ Yelp reviewers assign businesses on restaurant reservation availability ity. Traditionally, expert opinion and ratings from one to five stars in whole- and find that an extra half-star social learning have helped consumers star increments. When a user searches rating causes restaurants to sell out resolve these information asymmetries. Yelp.com, Yelp presents a list of busi­ 19 percentage points (49%) more For an expert’s take, consumers may nesses that meet the search criteria frequently, with larger impacts when consult Consumer Reports when buying or fall within the category of interest. alternate information is more scarce. an automobile or household appliance Figure 1 on the next page reproduces or they may read reviews by professional an excerpt from a sample search on critics when selecting a movie or choos­ Yelp.com. Businesses are sorted accord­ Also in this issue ing among dining options. Alternatively, ing to relevance and rating, and for consumers may confer with peers who each business the average rating is Immigration Reform 2013: own the automobile or who have eaten prominently displayed, rounded to the Implications for California at the restaurant. In recent years, how­ nearest half star. The number of stars in the average rating is easily visible, Agriculture ever, online sites that cheaply aggre­ gate consumer reviews have recently particularly because the color of the Philip L. Martin...... 5 expanded and have begun supplement­ stars changes at whole star thresholds. ing both of the traditional mechanisms. We downloaded the entire history Some Impacts of Recession on But are these sites playing an important of reviews from Yelp.com for each California’s Nursery and Floral role in determining consumer demand? restaurant in San Francisco, CA and Industry Despite the theoretical potential recorded the date of the review, the of digital word-of-mouth to influ­ rating assigned (1–5), and the reviewer’s Hoy Carman...... 9 ence consumer choices, it is difficult unique user identifier. We then recon­ to estimate its impact on purchas­ structed the average rating and total In the Next Issue... ing decisions. Products that receive number of reviews for each restaurant at positive reviews are ones that appeal to every point in time and matched these China’s Growing Role in consumers (i.e., they have high unob­ data with reservation availability data Agricultural Trade served quality), and these products from a large online reservation website. would likely experience high sales even As Figure 1 demonstrates, Yelp aggre­ by Colin A. Carter in the absence of positive reviews. In gates all reviews for a given business and Sandro Steinbach a recent paper published in the Eco­ and displays the average rating promi­ nomic Journal, we leverage a feature nently. However, when Yelp computes the average rating, they round off to Figure 1. Results of a Sample Search on Yelp.com the nearest half-star. Two restaurants that have similar average ratings can thus appear very differently on Yelp. For example, a restaurant with an aver­ bean bag coffee house IPA San Francisco age rating of 3.24 displays a 3-star Show Filters average rating, while a restaurant with an average rating of 3.26 displays a 1. Bean Bag Coffee House 462 reviews 3.5-star average rating. In actuality, Category: Coffee & Tea 601 Divisadero St Neighborhood: Western Addition/NOPA San Francisco, CA 94117 the true underlying quality of these (415) 563-3634 two restaurants is similar on average, stopping at the bean bag every morning on my way to work. The bean bag coffee is NOT like that. allowing us to identify the effect of the They sell coffee that tastes like roasted, fiery, burning charred blackness, the way coffee is supposed Yelp rating on customer demand while 2. Mojo Bicycle Café 295 reviews controlling for unobserved quality. If Categories: Coffee & Tea, Bikes 639 Divisadero St Neighborhood: Western Addition/NOPA San Francisco, CA 94117 Yelp reviews have significant impacts (415) 440-2338 on consumer demand, then we should Would it be too much to ask for the baristas here to know a thing or two about coffee? I have had the same experience twice when trying to buy beans. It goes something like this. I pick up a bag observe a sharp increase in reserva­ tions at each major rounding thresh­ 3. 21st Amendment Brewery 1081 reviews Categories: Breweries, Pubs, American (Traditional) 563 2nd St old (e.g., 3.25 stars and 3.75 stars). Neighborhood: SOMA San Francisco, CA 94107 In the paper we use a technique (415) 369-0900 known as regression discontinuity to Been coming here regularly for a couple of years. Not too much to say except the beers are fantastic. My fave is the 21st Amendment IPA which is their house beer. The drawback is that they estimate the effect of Yelp. Here, we present several figures that graphically 4. Salt House 1085 reviews Category: American (New) 545 Mission St summarize the results from the regres­ Neighborhood: SOMA San Francisco, CA 94104 (415) 543-8900 sion discontinuity estimator. Figure 2

Salt House is the kind of restaurant you're only going to find in Manhattan, SF or maybe Chicago. plots mean 7:00 p.m. reservation avail­ The focus is on the cuisine where it should be. Even though the decor and staff are West Coast laid ability by Yelp rating. Panel A focuses

5. NOPA 2218 reviews on the window where restaurants Category: American (New) 560 Divisadero St have either 3 or 3.5 stars, and Panel B Neighborhood: Western Addition/NOPA San Francisco, CA 94117 (415) 864-8643 focuses on the window where restau­ the right amount of meat/bread/condiments 3) Baked white bean appetizer - perfectly melded tomato rants have either 3.5 or 4 stars. There and feta topped with crunchy breadcrumbs that are perfectly juxtaposed against the beans I'm a fan. are clear jumps in the mean availability 6. Acme Burgerhaus 173 reviews at both the 3.5 and 4 star thresholds. Category: Burgers 559 Divisadero St Moving from 3 to 3.5 stars—which Neighborhood: Western Addition/NOPA San Francisco, CA 94117 (415) 346-3212 occurs when a restaurant’s rating crosses The fries were crisp and had plenty of garlic on them. * 1.95 draft beers. not quite as cheap as bean 3.25 stars—reduces the likelihood of bag but I can't get ostrich burgers at bean bag cafe. did I mention you can eat an ostrich here? Not availability from about 85% to about 7. Brickhouse Cafe 585 reviews 60%. Moving from 3.5 to 4 stars—which Categories: American (Traditional), Breakfast & Brunch, Bars 426 Brannan St Neighborhood: SOMA San Francisco, CA 94107 occurs when a restaurant’s rating crosses (415) 369-0222 3.75 stars—reduces the likelihood of breakfast or brunch. You can't go wrong with the Vanilla Bean French Toast. Oh, oh! There's also a question of the day, and if you answer it correctly you get 25 cents off your coffee. I'm not a coffee availability further to below 40%. Interestingly, for the most part, it 8. Radius 197 reviews appears that a step function is a good Category: American (New) 1123 Folsom St Neighborhood: SOMA San Francisco, CA 94103 approximation to the overall relation­ (415) 525-3676 Special Offer ship between Yelp ratings and restaurant because they source everything from within a 100 miles. Obviously, exceptions are made for the availability. That is, restaurant avail­ coffee beans, appliances, etc. Hopefully I'll have a chance to meet the restaurant personality of this ability appears to respond primarily 9. Ironside 280 reviews to the displayed rating, and not the Categories: American (New), Caterers 680 2nd St Neighborhood: SOMA San Francisco, CA 94107 underlying average review score (which (415) 896-1127 presumably measures the restaurant’s feel like they're missing a big opportunity to have smaller portions at lower prices. 3. The coffee! Ironside gets their beans from Four Barrel (delivered by bicycle messenger) so you'd expect their true quality from the perspective of

10. AT&T Park 1125 reviews 2 Category:Giannini Stadiums &Foundation Arenas of Agricultural Economics • University24 of W Californiaillie Mays Plz Neighborhood: SOMA San Francisco, CA 94107 Special Offer (415) 972-2000

and out for food, lol. Every Friday, two hours before a game starts [when doors open], they offer mystery grab bags. Though it was a Saturday game.. they offered mystery grab bags but it pretty much Figures 2a and 2b. Average Reservation Availability at 7 p.m. by Restaurant Yelp Rating

Panel A 1.0 Panel B 1.0 0.9 0.9

0.8 0.8

0.7 0.7

0.6 0.6

0.5 0.5

0.4 0.4

0.3 0.3

0.2 0.2 Average Reservation Availibility at 7 p.m. at 7 p.m. ReservationAverage Availibility at 7 p.m. ReservationAverage Availibility

3.15 2.75 2.85 2.95 3.05 3.25 3.35 3.45 3.55 3.65 3.75 3.25 3.35 3.45 3.55 3.65 3.75 3.85 3.95 4.05 4.15 4.25 Average Yelp Stars Average Yelp Stars consumers). Overall, we find that a half- important for restaurants excluded above the rounding threshold would star increase in Yelp ratings decreases from these prestigious rankings. We not be directly comparable to restau­ reservation availability by 19 percent­ again find that an extra half-star on rants below the rounding threshold. age points during peak dining hours. Yelp reduces reservation availability However, if specific restaurants If Yelp is providing information by 20 to 30 percentage points at all manipulate their reviews to fall right about new restaurants, that information three times for restaurants without above the thresholds, then some of should be most valuable among res­ external recognition but that the Yelp restaurants above the thresholds taurants that are unfamiliar to patrons. ranking does not similarly advantage have “true” Yelp ratings (i.e., the We divide restaurants into familiar/ restaurants which have been externally ratings they would receive absent unfamiliar groupings along two dimen­ accredited. These results support the manipulation) that are lower than sions. First, restaurants with fewer hypothesis that Yelp is most valuable their observed Yelp ratings. than 500 reviews are likely to be less when there is less external informa­ To generate a significant drop in frequented and less well known than tion about restaurants, though other reservation availability at the threshold, those with more than 500 reviews. differences between the two groups of these restaurants must sell out virtu­ For restaurants with fewer than 500 restaurants may also play some role. ally all the time, despite the fact that reviews, an extra half-star on Yelp The high return to positive Yelp they receive low ratings from true Yelp reduces reservation availability by 20 ratings naturally creates an incentive reviewers. It seems ex ante surprising to 30 percentage points depending on for restaurants to manipulate their that a restaurant that receives poor the reservation time. In contrast, for own ratings by leaving false reviews. reviews would be extremely crowded, restaurants with more than 500 reviews, Manipulation is feasible in this context though it is theoretically possible. for whom there is likely less hidden because Yelp is crowd-sourced—any Using a short theoretical model, we information about quality, there is no restaurateur can, in principle, leave show that although restaurants face discontinuous change at any threshold himself a 5-star review. Furthermore, incentives to manipulate Yelp ratings, associated with additional Yelp stars. the significant increases in business at it does not make sense for them to try A second test for whether the Yelp Yelp thresholds create a strong incentive to stay right above the Yelp round­ effect is due to solving information for restaurants to attempt to manipulate ing threshold. The intuition is simple: problems groups restaurants according their ratings to fall above a threshold. given that a random stream of reviews to whether there are external sources of Is it possible that the increases in will change each restaurant’s average quality information. Here, we note that demand that we observe in Figure 2 at rating over any time period, a restaurant quality information is easily available Yelp rounding thresholds are the result which is just above a threshold has a for restaurants which have a Michelin of specific restaurants strategically very similar likelihood of just missing star or those which appear in the San manipulating their ratings so that they that threshold after new reviews come Francisco Chronicle’s annual Top 100 fall right above the rounding thresh­ in as a restaurant which is just below Restaurants listing. In contrast, crowd- olds? If so, this would invalidate our the threshold. Both restaurants therefore sourced information may be more research design, because restaurants

Giannini Foundation of Agricultural Economics • University of California 3 face similar incentives to try and push of each restaurant in a subsample of of consumed meals via two mecha­ their Yelp scores into safer territory. 73 restaurants. Next, we assumed that nisms. First, it can redirect consumers We also present a variety of empiri­ a restaurant has no reservation avail­ to higher quality restaurants. Second, cal tests that consistently show no ability if the number of seats reserved it can induce lower quality restaurants evidence of any manipulation behavior for a given evening reaches its capacity. to shut down or improve their qual­ that would cause restaurants to cluster Finally, we examined the average cus­ ity in response to changes in customer just above the thresholds. For example, tomer flows that would be consistent demand. We provide direct evidence restaurants leaving fake reviews for with reservation availability rates of of the first mechanism, but we cannot themselves should have more 5-star 58% (the average rate above the Yelp speak to the second mechanism. With reviews and fewer reviews per reviewer. thresholds) and 39% (the average below the rapid spread of Yelp and other simi­ We find no evidence that restaurants the Yelp thresholds) under different lar crowd-sourcing websites, this sug­ with these types of characteristics assumptions about the distribution of gests that market evolution may be an cluster just above the thresholds. arriving customers. Our calibrations important avenue of future research. Two questions emerge when con­ suggest that the median restaurant sidering the effects of Yelp ratings on might experience an increase in cus­ Suggested Citation: restaurant demand. First, do the effects tomer flows of 6% or more if its reserva­ Anderson, M.L., and J. Magruder. 2013. represent the transmission of informa­ tion availability drops from 58% to 39%. "Does Yelp Affect Restaurant Demand?" tion on restaurant quality or do they Modest changes in customer flows, ARE Update 16(5):1-4. University of California Giannini Foundation of represent a marketing effect generated however, can have a significant impact Agricultural Economics. by Yelp’s ranking system? Second, on profits in an industry with high what changes in customer flows and fixed costs and high margins. A typi­ profits are consistent with the observed cal mid-to-high-end restaurant with Michael Anderson and Jeremy Magruder are both assistant professors in the Department changes in reservation availability? $20,000 per week in sales and a margin of Agricultural and Resource Economics at Our estimates may not represent a of 68% on food and beverage sales, UC Berkeley. They can be contacted by e-mail pure effect of information regarding earns approximately $2,000 per week at [email protected] and jmagruder@ restaurant quality if the order in which in pre-tax profit. In comparison, a berkeley.edu, respectively. Yelp lists restaurants on its website 6% increase in customer flow trans­ (e.g., in Figure 1) is a function of a lates into an additional revenue gain For additional information, the restaurant’s displayed average rating of $816 per week in pre-tax profit. authors recommend: rather than its true average rating. In Of course, the increase in profit will Anderson, M.L. and Magruder, J. that case, restaurants just above a Yelp be lower if the restaurant is capacity- 2012. “Learning from the Crowd: threshold would be significantly more constrained or if it has to expand Regression Discontinuity Estimates likely to be seen by consumers brows­ staffing levels to maintain service. of the Effects of an Online ing Yelp than restaurants just below a Nevertheless, the calibrations sug­ Review Database,” The Economic Yelp threshold. However, we find that gest that a typical restaurant could Journal, 122(563): 957–989. the order in which Yelp lists restau­ experience substantial gains in profit rants is not affected by the displayed when crossing a Yelp threshold. average rating (after controlling for In summary, the effects we estimate the restaurants’ true underlying aver­ are large, and they indicate a valuable age ratings). We thus conclude that use of crowd-sourced information: increased information about restaurant because Yelp collects and aggregates quality causes higher-rated restaurants the experiences of a large number of to have lower availability, rather than patrons, Yelp provides a convenient any effect of increased visibility. forum to solve asymmetric informa­ To gauge what changes in customer tion problems about the quality of flows could be consistent with our unfamiliar restaurants. Tightening result that an extra half-star on Yelp the link between restaurant quality causes a 19 percentage point decrease in and restaurant patronage may well reservation availability, we performed have positive benefits for society. a series of simple statistical calibra­ Crowd-sourced quality informa­ tions. First, we recorded the capacity tion may improve the average quality

4 Giannini Foundation of Agricultural Economics • University of California