Quentin Hardy, “How Big Data Gets Real,” Bits Blog, June 4, 2012. Available at: http://bits.blogs.nytimes.com/2012/06/04/how-big-data-gets-real/

How Big Data Gets Real

The business of Big Data, which involves collecting large amounts of data and then searching it for patterns and new revelations, is the result of cheap storage, abundant sensors and new software. It has become a multibillion-dollar industry in less than a decade. Growing at speed like that, it is easy to miss how much remains to do before the industry has proven standards. Until then, lots of customers are probably wasting much of their money.

There is essential work to be done training a core of people in very hard problems, like advanced statistics and software that ensures data quality and operational efficiency. Broad-based literacy in the uses of data should probably happen too, along with new kinds of management, better tools for reading the information, and privacy safeguards for corporate and personal information.

That such a huge number of tasks are taking place is a good indicator that, even with the hype, Big Data is a big deal. Last Friday, a number of technologists gathered at a forum hosted by the University of California, Berkeley, iSchool and talked about ways many of these jobs are being done. (Disclosure: I lecture at the iSchool, which is the school of information science, and moderated several panels there.) They talked about the progress so far, and identified a number of good ideas and businesses left to pursue.

In some ways, Big Data is about managing all kinds of weird new data, like social media updates from a mobile phone. It is hard to categorize in the first place, and may be used in lots of different ways, from advertising to traffic management. The so-called unstructured database of choice is by now pretty clearly Hadoop.

Cloudera, a leading software producer, is now training 1,500 people a month, mostly online, in how to use both the database and associated applications. According to Amr A. Awadallah,

Cloudera's chief technical officer, over 10,000 people have been trained on its system. Data quality from new diverse sources is still a big problem, as is persuading companies and organizations to let others see data that might be more valuable in a commonly shared algorithm.

"I've tried paying money for it, but it's easier for companies to decide not to share," said Gil Elbaz, the founder of Factual, a company that seeks to hold lots of online data. "The only way that works is to get them to take risks in exchange for data that is valuable to them."

Much of the fear about exposing data, he said, has to do with competitors learning secrets. Mr. Elbaz thinks there is a good business in developing "de-identifiers" that can make data anonymous, and privacy insurers specializing in covering the costs of exposure.

On a personal level, others think the government or a trusted private institution will hold the personal identifiers of things like medical data, releasing it to trusted parties. "It's a little scary that right now a cab driver using Uber knows more about you than a doctor, who has to take all of your information for the first time," said Peter Skomoroch, principal data scientist at LinkedIn. Another data-improving business consists of moving the world's older data online. A company called Captricity aims to couple image-capture from things like cellphone cameras with cheap workers in Amazon.com's Mechanical Turk service, in order to put older handwritten documents into digital databases. The company's early business is from government and charity sites in Africa and India, but there is no reason why it shouldn't be valuable for most medical records. If someone took the trouble to write it down, the company figures, that is a good way to assume it is valuable data.

There are other businesses trying to take the arcane side of Big Data into the mainstream, with easy-to-use statistical tools and new ways of visualizing data that make it easier to understand. Companies like ClearStory and Platfora "want to make it possible for businesses, for history majors, to use," said Ben Werther, chief executive of Platfora. "We're in the pre-industrial age of Big Data." Martin Wattenberg, creator of a well-known wind map and who is now at Google, talked about a necessary revolution in design of data outcomes that have yet to become widespread.

Away from big online companies like Google, some of the earliest data-driven businesses include hedge funds and insurance companies, which have the money and tradition of using lots of math. Another, Mr. Skomoroch said, is the Mormon Church. "It is the focus on genealogy data," he says. "If you move, they'll always find you."

What Big Data is seeing now looks like the classic industrial curve. There is the first discovery of something big, leading to establishing principles like scientific rules. Science moves toward engineering as a means to manufacturing, resulting in mass deployment. Then things really change.

“The Personal Data Landscape,” in Rethinking Personal Data: Strengthening Trust, World Economic Forum, May 2012, 7-12. Available online at http://www3.weforum.org/docs/WEF_IT_RethinkingPersonalData_Report_2012.pdf ______Chapter 1: The Personal Data Landscape

Introduction Historically, the strength of a major economy is tightly linked to its ability to move physical goods. The Silk Route, the Roman The digital world is awash in personal data.1 Every day, people roads and the English fleet all served as the economic backbones send 10 billion text messages, make 1 billion posts to a blog or connecting vast geographies. Even though it is a virtual good, data social network and generate millions of entries into their growing is no different. Data needs to move to create value. Data sitting electronic health records. alone on a server is like money hidden under a mattress. It is safe and secure, but largely stagnant and underutilized. In addition, with approximately 6 billion mobile telephone subscriptions in the world, it is now increasingly possible to track As an emerging asset class, personal data currently lacks the rules, the location of nearly every person on the planet, as well as their norms and frameworks that exist for other assets. The lack of social connections and transactions.2 And mobile phones are not trading rules and policy frameworks for its movement have resulted the only devices recording data – Web applications, cars, retail in a deficit of trust among all stakeholders and could undermine the point-of-sale machines, medical devices and more are generating long-term potential value of personal data. Different jurisdictions are unprecedented volumes of data as they embed themselves into looking to tackle this deficit of trust through different approaches, our daily lives. Estimates are that by 2015, 1 trillion devices will be ranging from fundamental rights-based approaches to harm- connected to the Internet.3 minimization approaches.

Companies and governments are using this ocean of Big Data to Several characteristics of personal data make establishing rules unleash powerful analytic capabilities. They are connecting data and frameworks uniquely challenging: from different sources, finding patterns and generating new insights – all of which adds to the ever deepening pool of data. - The digital nature of personal data means it can be copied infinitely and distributed globally, thereby eliminating many of In short, the growing quantity and quality of personal data creates the trade barriers which exist for physical goods. enormous value for the global economy. It can help transform the lives of individuals, fuel innovation and growth, and help solve many - Data, unlike most tangible assets, is not consumed when used; of society’s challenges. As the first-phase World Economic Forum it can be reused to generate value. report on personal data elaborated, personal data represents an emerging asset class, potentially every bit as valuable as other - Data grows ever more connected and valuable with use. assets such as traded goods, gold or oil. Connecting two pieces of data creates another piece of data and with it new potential opportunities (as well as new potential harms).

- The role of the individual is changing. Individuals are no longer primarily passive data subjects. They are also increasingly

1 There are many definitions of personal data across different jurisdictions and even different sectors the creators of data. In addition, personal data is intimately within the same jurisdiction. For the purposes of this report, personal data is used to refer to data and linked with an individual’s background and identity, unlike metadata relating to a specific, identified or identifiable person. This is similar to definitions used by both the proposed US Privacy Bill of Rights and the EU’s Data Protection Regulation. interchangeable commodity goods. 2 “Key Global Telecom Indicators for the World Telecommunication Service Sector”. International Telecommunications Union, http://www.itu.int/ITU-D/ict/statistics/at_glance/KeyTelecom.html. 3 King, Rachel. “IBM panel discusses tackling big data storage as problem escalates”. smartplanet, September 2011, http://www.smartplanet.com/blog/smart-takes/ibm-panel-discusses-tackling-big- data-storage-as-problem-escalates/19010.

Rethinking Personal Data: Strengthening Trust 7 Chapter 1: The Personal Data Landscape

All stakeholders in the personal data ecosystem face a challenge flow of personal data on which e-commerce and other economic of unprecedented size, speed and complexity. Rules and norms activity depends becomes overly restricted. change faster in a hyperconnected world and outstrip the ability of traditional rule-setting approaches to keep pace. Solutions Consider the many ways personal data can create economic and that focus on isolated examples or one-size-fits-all approaches social value for governments, organizations and individuals: quickly grow outdated as social, commercial and regulatory contexts change. Enterprises, governments and individuals need - Responding to global challenges. Real-time personal data and to creatively collaborate to develop new rules and frameworks that social media can help to better understand and respond to are both robust enough to be enforceable, yet flexible enough to global crises like disaster response, unemployment and food accommodate the world’s accelerating and constant change. security. It represents an unprecedented opportunity to track the human impacts of crises as they unfold and to get real- time feedback on policy responses.5 Or consider the case of The Opportunity Google Flu Trends, which uses individuals’ ostensibly private flu-related search words and location data6 to detect potential Personal data plays a vital role in countless facets of our everyday flu outbreaks in real time, as opposed to the weeks-old lives. Medical practitioners use health data to better diagnose government data that currently exists.7 Researchers have found illnesses, develop new cures for diseases and address public health that the data has a high correlation with upswings in emergency issues. Individuals are using data about themselves and others to room activity, and it could provide the basis for early-warning find more relevant information and services, coordinate actions systems to detect pandemics and save millions of lives.8 and connect with people who share similar interests. Governments are using personal data to protect public safety, improve law - Generating efficiencies. For centuries, increased access to enforcement and strengthen national security. And businesses are information has created more efficient ways of doing business.9 using a wide range of personal data to innovate, create efficiencies These days, organizations in every industry are using vast and design new products that stimulate economic growth. amounts of digital data to streamline their operations and boost overall productivity. For example, US$ 700 billion in health cost Estimates are that the Internet economy amounted to US$ 2.3 trillion in value in 2010, or 4.1% of total GDP, within the G20 5 UN Global Pulse, http://www.unglobalpulse.org/about-new. group of nations. Larger than the economies of Brazil or Italy, the 6 It is important to note that Google Flu Trends data can never be used to identify individual users Internet’s economic value is expected to nearly double by 2016 to because it relies on anonymized, aggregated counts of how often certain search queries occur each week. US$ 4.2 trillion.4 But this growth could be severely constrained if the 7 Google Flu Trends, http://www.google.org/flutrends/about/how.html. 8 Dugas, A. F., Y. H. Hsieh, S. R. Levin, et al. “Google Flu Trends: Correlation with Emergency Department Influenza Rates and Crowding Metrics.” Clinical Infectious Diseases, 2012. Described in 4 Dean, David, Sebastian DiGrande, Dominic Field and Paul Zwillenberg. “The Digital Manifesto: How Science Daily, http://www.sciencedaily.com/releases/2012/01/120109155511.htm. Companies and Countries Can Win in the Digital Economy.” The Boston Consulting Group. January 2012. 9 Gleick, James. The Information: A History, a Theory, a Flood. London: Fourth Estate, 2011.

8 Rethinking Personal Data: Strengthening Trust Chapter 1: The Personal Data Landscape

savings in the US, or about 30% of total healthcare spending The above list is far from comprehensive. Some of the most today, could result in large part through the increased flow important value-creation opportunities from personal data remain of personal data. Improved information flow could reduce as yet unknown. Most personal data still resides in silos separated duplicative lab testing and imaging, fraud and inefficiencies, by different technology standards and legal contracts. And a lack as well as lead to better care coordination and treatment.10 of an effective system of permissions prevents data from moving in In financial services, personal data is already being used to a trusted and secure way to create value. Creating such systems generate significant efficiencies through facilitating online will allow ever greater “data leverage”. But such opportunities are commerce and payments as well as saving billions of dollars not without potential harms and risks, which manifest themselves through fraud prevention. in a loss of trust among all stakeholders in how personal data is protected and used. - Making better predictions. Personal data is stimulating innovative new products tailored to and personalized for Given the variety of applications in which personal data can be the specific needs of individuals. For example, Amazon’s leveraged, estimating the impact of the “economics of trust” is “Customers Who Bought This Also Bought” collaborative difficult to measure. Existing research by BCG on the Internet filtering tool suggests related items to buy that customers might economy in the G20 has forecast that online retail will grow to not have discovered otherwise.11 In financial services, tailored US$ 2 trillion by 2016.17 However, this estimate is influenced by insurance products are being developed based on devices consumer perception of trust in how personal data is used. Online that track driving behaviour rather than just age, gender and retail could grow even faster to US$ 2.5 trillion by 2016 with neighbourhood. News and content websites can customize enhanced trust or to only US$ 1.5 trillion if trust were to be eroded. the articles each individual views based on their interests Given that this US$ 1 trillion range is from just one small part of the and preferences. In addition, organizations can use business broader personal data ecosystem, it provides an indication of the intelligence derived from the aggregation of millions of individual magnitude of the potential economic impact when other sectors consumer transactions to prepare for likely events. For (health, financial services, etc.) are considered – potentially in the example, before a hurricane strikes, Wal-Mart knows to stock tens of trillions of dollars. its shelves with not only flashlights and batteries, but also with Pop-Tarts.12 The Evidence of a Decline in Trust - Democratizing access to information. Consumers benefit from “free” services like search engines, e-mail, news sites and Ample evidence suggests that there is a decline in trust in the social networks that previously either did not exist or have a personal data ecosystem. Many of the existing regulatory, significant monetary cost in other forms in the offline world. commercial and technical mechanisms for strengthening trust However, individuals are beginning to realize that targeted are no longer fit to do the job. The widespread loss of trust is advertising, based on data about them and their online unmistakable: security breaches, identity theft and fraud; concern behaviour, 13 fuels most of these ostensibly free services either from individuals and organizations about the accuracy and use of directly or indirectly. As is widely quoted online: “If you’re not personal data; confusion from companies about what they can and paying for something, you’re not the customer; you’re the cannot do; and increasing attention and sanctions from regulators. product being sold.”14 Some recent events serve as leading indicators of the potential - Empowering individuals. Empowered consumers are taking for future instability on a much more massive scale. In 2011, Sony greater control over the use of data created by and about revealed breaches in its online video game network that led to the them. Rather than individuals being passive agents, they are theft of names, addresses and possibly credit card data belonging engaging with organizations in a collective dialogue.15 As a to more than 100 million accounts, in what was one of the largest- result, people are starting to volunteer information that only they ever Internet security break-ins.18 Experts said the breaches could know. Such volunteered personal information (VPI) includes cost Sony and credit card issuers US$ 1-2 billion.19 their updated contact details as they happen, reasons why they took an action or made a purchase, and future plans and Data breaches are also growing more anarchic and motivated preferences. Some observers have estimated that VPI could by state-sponsored political and anti-business sentiments. The reach approximately US$ 32 billion in value by 2020 in the UK loose-knit hacking movement known as Anonymous claimed to alone.16 In addition, individuals are using the information they have stolen thousands of credit card numbers and other personal share about themselves, their beliefs and their preferences to information belonging to clients of US-based security think tank connect like never before. In the past few years, social media Stratfor.20 tools built on the foundation of widely shared personal data have played a key role in bringing down governments.

10 “Where Can $700 Billion in Waste Be Cut Annually from the U.S. Healthcare System?” Thomson 17 For country-by-country forecasts see Dean, David, Sebastian DiGrande, Dominic Field, Andreas Reuters, http://www.factsforhealthcare.com/whitepaper/HealthcareWaste.pdf, 2009. Lundmark, James O’Day, John Pineda and Paul Zwillenberg. “The Internet Economy in the G-20: The 11 Tene, Omer and Jules Polonetsky. “Privacy in the Age of Big Data”. Stanford Law Review, 2 $4.2 Trillion Growth Opportunity.” The Boston Consulting Group. March 2012. February 2012. http://www.stanfordlawreview.org/online/privacy-paradox/big-data. 18 Baker, Liana B. and Jim Fincle. “Sony PlayStation Suffers Massive Data Breach”. Reuters. 26 April 12 “A Different Game”. The Economist. 25 February 2010. http://www.economist.com/ 2011. http://www.reuters.com/article/2011/04/26/us-sony-stoldendata-idUSTRE73P6WB20110426; node/15557465. “Sony suffers second data breach with theft of 25m more user details”. The Guardian, Technology Blog. http://www.guardian.co.uk/technology/blog/2011/may/03/sony-data-breach-online- 13 Most online advertising today relies on anonymous cookies and is linkable to a device rather than entertainment. an individual. 19 Miller, Mary Helen. “Sony data breach could be most expensive ever”. Christian Science Monitor. 3 14 Lewis, Andrew. MetaFilter Weblog. 26 August 2010. http://www.metafilter.com/95152/ May 2011. http://www.csmonitor.com/Business/2011/0503/Sony-data-breach-could-be-most- Userdriven-discontent#3256046. expensive-ever. 15 Searles, Doc. Cluetrain Manifesto. http://www.cluetrain.com. 20 “Anonymous targets US security think tank Stratfor”. Associated Press. 25 December 2011. 16 Ctrl-Shift, http://www.ctrl-shift.co.uk/themes/volunteered_personal_information. http://www.guardian.co.uk/technology/2011/dec/25/anonymous-security-thinktank-stratfor

Rethinking Personal Data: Strengthening Trust 9 Chapter 1: The Personal Data Landscape

In addition, some data breaches happen accidentally or different names and e-mail addresses for different contexts, to use unintentionally; think of the employee who leaves an unencrypted pseudonyms, or to prevent their data being captured or linked to laptop on a train containing thousands of data records. And them in the first place. breaches are not limited to the companies that originally collected the personal data, which adds to the loss of trust. In 2011, hackers Government Policy-makers compromised the database of Epsilon, a marketing company Policy-makers and regulators around the world share similar objectives – to stimulate innovation and economic growth while at responsible for sending 40 billion marketing e-mails on behalf of the same time protecting individuals from harmful uses of personal 2,500 customers, including such companies as Best Buy, Disney 21 data. Striking this balance lies at the heart of the tension facing the and Chase. Very few customers had ever heard of Epsilon, nor regulatory community. However, regulators are growing concerned did they know it held data about them. about the loss of individuals’ trust in how organizations are using and protecting data about them. Meanwhile, different regulators are taking different approaches to balancing these objectives, potentially Stakeholders Have Different Perspectives and adding to the instability as they work out how to protect individuals Concerns through new laws, policies and regulations. This ranges from privacy bills of rights and “do-not-track” options for consumers, to Tension is rising. Individuals are growing concerned that companies requirements that consumers be granted access to their personal and governments are not protecting data about them and that they data. A more comprehensive approach is needed that does not limit are instead using it in ways not necessarily in their best interests. the solution to one sector or jurisdiction to allow for a global flow of Many organizations are struggling to protect and secure the data. explosion of data they have access to, and they are unsure what Following the release of the National Strategy for Trusted Identities they can and cannot do with it. in Cyberspace in 2011, which focused on identity assurance to support online interactions, the US government has outlined Governments are trying to strike a balance between protecting a comprehensive online Privacy Bill of Rights that aims to give individuals and encouraging innovation and growth. Such individuals more control over how personal information about them uncertainty creates instability, which in turn manifests itself is used on the Internet.23 The Privacy Bill of Rights includes the goals differently across three main groups of actors in the personal of individual control, transparency, respect for context, security, data ecosystem – individuals, government policy-makers and access and accuracy, focused collection and accountability. The organizations. government intends to work with stakeholders to translate the document into specific practices or codes of conduct, to develop Individuals legislation based on these rights and to provide the US Federal Trade Surveys show that individuals are losing trust in how data about Commission (FTC) with enforcement authority. The approach builds them is being collected, used, shared and combined by both on existing US self-regulation practices in which the FTC steps in to organizations and governments. For example, according to enforce unfair or deceptive practices. European Justice Commissioner Viviane Reding, 72% of European citizens are concerned that their personal data may be misused, In Europe, on the other hand, the European Commission and they are particularly worried that companies may be passing on has approached the issue from the perspective of protecting their data to other companies without their permission.22 However, a fundamental rights, although it shares with the US the same disconnect exists between what people say and what they do; the common goal of providing individuals with greater control to restore world of personal data is no exception. While many people say they trust. The proposed Data Protection Regulation issued in January care about privacy, they also share information quite widely on social 2012 includes a requirement that Internet companies obtain explicit networks and elsewhere online. consent from consumers about the use of their personal data and delete the data forever at the users’ request or face the prospect of A large part of concern stems from the fact that individuals often fines for failing to comply.24 The rules would also extend the reach sign up to services not knowing how their data will be protected or of European law to companies outside the EU that handle data if it will be shared. While legally they have given organizations their relating to EU residents. This raises, however, jurisdictional questions consent to the stated rules for usage, few individuals actually read given the global nature of data flows. While many observers have these privacy policies or terms of services. Individuals therefore praised these efforts as an important step in giving individuals more have little visibility into the practices of the organizations they are control over data about them, some have raised questions about putting their trust in – until their data is breached or misused. As the possible unintended consequences on innovation and economic government’s use of personal data grows, concern likewise grows growth and the use of data for purposes such as health research.25 over government protections of individual privacy. Many of these regulatory requirements have reasonable motives, but Another concern of individuals is how to manage their online identity could also have unintended consequences that might undermine and the different aspects of their digital lives. Health data about economic and social value. For example, the Indian government an individual has a different impact when shared in a healthcare, introduced rules in April 2011 it said would protect individuals, work, family or social context. The lack of contextual control and permissions represents another cause for concern among individuals. At the moment, one of the few ways for individuals 23 “We Can’t Wait: Obama Administration Unveils Blueprint for a ‘Privacy Bill of Rights’ to Protect Consumers Online”. Office of the Press Secretary, US White House, 23 February 2012, http://www. to keep different parts of their digital lives separate is to use whitehouse.gov/the-press-office/2012/02/23/we-can-t-wait-obama-administration-unveils- blueprint-privacy-bill-rights. 24 Sengupta, Somini. “Europe Weighs Tough Law on Online Privacy”. The New York Times. 23 21 Lennon, Mike. “Massive Breach at Epsilon Compromises Customer Lists of Major Brands”. January 2012. http://www.nytimes.com/2012/01/24/technology/europe-weighs-a-tough-law-on- Security Week, 2 April 2011. http://www.securityweek.com/massive-breach-epsilon-compromises- online-privacy-and-user-data.html?_r=1&pagewanted=al.l. customer-lists-major-brands. 25 22 For example, see the US government’s position on the draft EC regulation available at http://www. Reding, Viviene. “The EU Data Protection Reform 2012: Making Europe the Standard Setter for edri.org/files/US_lobbying16012012_0000.pdf; http://blogs.law.harvard.edu/infolaw/2012/01/25/ Modern Data Protection Rules in the Digital Age.” Speech at the Innovation Conference Digital, Life, more-crap-from-the-e-u; http://www.pcworld.com/businesscenter/article/251573/proposed_eu_ Design, Munich, Germany, 22 January 2012. http://ec.europa.eu/commission_2010-2014/reding/ data_laws_under_fire_from_both_sides.html; http://www.marketingweek.co.uk/news/facebook- pdf/speeches/s1226_en.pdf. data-laws-could-stifle-innovation/4000557.article.

10 Rethinking Personal Data: Strengthening Trust Chapter 1: The Personal Data Landscape

including requiring companies to take consent in writing from Organizations individuals about the use of the sensitive personal information they Commercial enterprises, not-for-profits and governments that have collect.26 The requirement could have significantly constrained the access to personal data are responding in unique ways to the US$ 75 billion Indian outsourcing industry, which employs 2.5 million explosion of information created by and about individuals. Some people.27 In August 2011, the Indian Ministry of Communications are unsure of the rules of the game and are concerned about legal and IT issued a clarification effectively exempting outsourcers from liabilities and the negative brand impact of being seen as unfairly the new law and calling into question the law itself.28 exploiting personal data given the heightened media attention on privacy and data breaches. As a result, some organizations are The role of regulators is likely to be very different when it comes currently restricting the use of the personal data they hold and are to ensuring accountability for organizational stewardship of data underinvesting in the systems that can help to generate value from as opposed to the setting of rules for what can and cannot be this data. done with personal data. In the latter areas, the personal data ecosystem is increasingly too complex, fast-moving and global for A number of private organizations, particularly those in the telecom traditional regulatory mechanisms to be effective. Some observers sector, face significant legal and long-standing regulatory constraints say approaches that treat governance as an afterthought and on how they can use personally identifiable information. This lies in economic externality (through regulatory oversight and mechanisms contrast with many of their competitors, which are using personal such as notification and consent, hearings, rulings, legal challenges, data much more freely to generate value. Others are innovating and injunctions and warrants) create huge costs, uncertain liabilities and developing new business models that offer individuals the tools to problematic social acceptance.29 receive some form of control or even payment for the use of data about them.

The relationship individuals hold with Internet giants such as Google and Facebook was tested in 2011 when the FTC announced that it 26 Wugmeister, Miriam and Cynthia Rich. “India’s New Privacy Regulations”. Morrison & Foerster planned to heavily monitor the companies during the next 20 years. Client Alert. http://www.mofo.com/files/Uploads/Images/110504-Indias-New-Privacy-Regulations. The settlements stemmed from charges related to the unauthorized pdf. sharing with third parties of personal data, failure to delete personal 27 “India’s share in global outsourcing market rises to 55 pct in 2010”. International Business Times, 3 February 2011. http://www.ibtimes.com/articles/108475/20110203/nasscom-global-outsourcing- data from deactivated accounts and engaging in deceptive tactics share-india-it-bpo-sector-revenue-growth-of-it-bpo-sector.htm. that violated established privacy policies.30 28 Ribeiro, John. “India Exempts Outsourcers From New Privacy Rules”. IDG News, 24 August 2011. http://www.pcworld.com/businesscenter/article/238706/india_exempts_outsourcers_from_new_ 30 privacy_rules.html. “Facebook and privacy: Walking the tightrope”. The Economist. 29 November 2011. http://www. economist.com/blogs/babbage/2011/11/facebook-and-privacy; “FTC Charges Deceptive Privacy 29 Clippinger, John Henry. “Design for a Trusted Data Platform and a Data-driven Bank: Overview Practices in Google’s Rollout of Its Buzz Social Network”. Federal Trade Commission, http://www.ftc. and Next Steps”. ID Cubed, January 2012. gov/opa/2011/03/google.shtm.

Exhibit 1: A lack of trust has the potential to pull the ecosystem apart

Enhance Control Increase Transparency Obtain Fair Value Distribution

– Lack of Transparency – Significant Liabilities – No Clear Rules – Rapid Pace of Change Drive Growth Create Value Stimulate Innovation Generate Efficiencies Protect Individuals Predict Behaviour

Rethinking Personal Data: Strengthening Trust 11 Chapter 1: The Personal Data Landscape

In the meantime, some industry sectors are coming together to Re-establishing Trust in a Complex Ecosystem help establish best practices and norms to guide behaviour. In the marketing world, the Digital Advertising Alliance, which represents The existing dialogue about personal data is currently anchored more than 90% of online advertisers, has established self-regulatory in fear, uncertainty and doubt. This fear is potentially made worse principles for online behavioural advertising.31 The guidelines aim to by the increasingly shortened cycle between the discovery of an give individuals a better understanding of and greater control over event or vulnerability and widespread media coverage. A researcher ads that are customized based on their online behaviour. They also discovers that a website or mobile app has used data improperly, establish specific limitations for the use of this data in cases when the mainstream press picks up the story, setting off a popular the potential harm is considered highest, such as employment, firestorm, which creates pressure on politicians to react. credit and healthcare eligibility information.32 In addition, the Network Advertising Initiative has established monitoring and accountability While exposing company missteps is clearly important, the 33 mechanisms to encourage compliance with these rules. focus on fear and concern could result in a reactive and one- sided response. Unintended consequences could reduce the Similar efforts can be found in the mobile space with the recently announced GSM Association privacy principles.34 In the online opportunities for value creation. It seems to be an intractable identity arena, the Open Identity Exchange was formed as a not- problem: how to create the rules and tools so that all stakeholders for-profit to support the creation of private sector-led legal rules and can capture the value from data in a trusted way. policy development and the creation of an open market for related identity and privacy products and services relating to online data and identity challenges.35

Governments are also increasingly using personal data for law enforcement and national security, including the monitoring of SMS messages, blogs, social network posts or geolocation data to fight criminal and terrorist activity, which is raising significant surveillance and privacy concerns. A 2011 report by the Brookings Institute noted that rapidly declining storage costs make it technologically and financially feasible for authoritarian governments “to record nearly everything that is said or done within their borders – every phone conversation, electronic message, social media interaction, the movements of nearly every person and vehicle, and video from every street corner”.36

Governments are trying to improve service delivery and achieve significant cost savings by leveraging personal data. For example, a recent World Economic Forum report estimated the cost savings to emerging market governments could range up to US$ 100 billion per year with increased use of mobile financial services and the ability to utilize personal data more efficiently.37

31 http://www.aboutads.info/obaprinciples 32 http://www.aboutads.info/msdprinciples 33 http://www.networkadvertising.org/pdfs/NAI_2011_Compliance_Release.pdf 34 http://www.gsma.com/mobile-privacy-principles 35 http://openidentityexchange.org 36 Villasenor, John, “Recording Everything: Digital Storage as an Enabler of Authoritarian Governments”, Brookings Institute, 2011. 37 Galvanizing Support: The Role of Government in Advancing Mobile Financial Services. World Economic Forum, March 2012.

12 Rethinking Personal Data: Strengthening Trust David Talbot, “A Social Network that Pays You.” MIT Technology Review, November 8, 2011. Available at http://www.technologyreview.com/news/426064/a-social-network-that-pays-you/

A Social Network that Pays You

Chime.in lets users create pages about their own interests—and plans to give them a cut of the resulting ad revenue.

For all the differences among them, the juggernauts of social media rely on a common business model: create free services, then sell ads against users' information. In a dramatic departure, a new social network plans to give its users a 50 percent commission—or even let them sell their own ads and keep all the revenue.

Chime.in is built around users' interests—think photography, politics, or travel—as opposed to friends, professional contacts, or news. The site's founders hope that by creating pages around those interests, the users will attract people with similar affinities, an attractive combination for targeted advertising.

"Because social is going to be so powerful, I feel that the people who are creating the engaging social content should have some stake," says Bill Gross, the serial entrepreneur who is the CEO of both Idealab, a startup incubator, and Ubermedia, a social media developer that launched Chime.in. "Right now that's sort of a heresy—but I almost like it that people think it's heresy. It gives me more of a lead."

Gross is no stranger to creating disruptive business models. The pay-per-click concept for advertising in search listings was born in 1998 at his startup Goto.com, a search engine that was later renamed Overture and sold to Yahoo in 2003 for $1.6 billion. "It took five years [to go from calling Overture] heresy to 'We want to own it,' " Gross recalls.

Chime.in launched last month in beta; the site will officially launch at the end of this year, with the advertising model kicking into gear in 2012.

In terms of technology, Chime.in is a highly derivative platform. It most resembles Facebook, with a string of posts and comments beneath them. Like Twitter, the content is public by default, and users can follow anyone (no friending required), but posts can be longer: 2,000 characters. Users can vote on content, much as they do on Digg.

But the emphasis is on users' interests; after joining already existing groups or creating their own, users can sort content by those interests. Chime.in says the site is now home to 5,000 interest-based groups that have so far shared more than 25 million "chimes."

In a sense, Chime.in is offering a social-networking version of Web-publishing platforms like Wordpress, with full social features like reposting and comment threads. And each post or "chime"—often with a photo or video—fits nicely on a smart-phone screen. The overall idea is that this technology—as well as a promise of at least 50 percent of all ad revenue—will prod people to add and develop state-of-the-art content that other people find trustworthy. With time spent on social networks rising and search engines falling, "more and more people will make decisions based on social cues from people they trust, than from something they found on a search engine," Gross says.

The ads would appear on a person's personal profile page, or on a community page created by an individual, brand, or celebrity. Whoever created the page would get 50 percent of the revenue from any advertising Chime.in placed there. Other Chime.in users, most likely companies, could also place ads themselves on their own pages, and collect 100 percent. Gross estimates that some successful pages eventually could bring in thousands of dollars in ad revenue. The site already has gotten several entertainment companies to set up their own Chime.in pages, including E! Entertainment, Universal Pictures, Walt Disney Studios, and Bravo.

The site is not without glitches. I created two accounts: one through Facebook (which let Chime.in search my Facebook profile information) and another directly through Chime.in. Each time it offered me a rather strange, seemingly random collection of 11 interests from which to initially choose: Apple, autos, blogworld, blogging, celebrity chefs & restaurants, comic books & superheroes, Google, marketing & advertising, macro photography, and music discovery.

I chose "music discovery" and "autos," but I landed in the "macro photo" group. Still, this meant I got to meet my first follower: Kayla Connelly, of Moosic, Pennsylvania, a prodigious Chimer and macro photographer. I followed her, too, and was soon enjoying her intimate portraits of vodka labels, grilled-cheese sandwiches, and snow-dusted angel statues. I also learned she likes Coldplay. Her post of a white Christmas-tree light was captioned with this purloined lyric: "Lights will guide you home and ignite your bones, and I will try to fix you."

I wondered why Connelly was Chiming. Finding no way to e-mail her directly within Chime.in, I posted a comment under one of her photos (of paint pots) and disclosed my journalistic purpose. I asked her why she would bother with Chime.in; we already have Facebook, Twitter, Google+, Digg, and many others. "Immediately addicted!" she replied. "So much easier to connect with knowledgeable users with similar interests and get feedback."

Poking around the site, I saw groups working on crowdsourced efforts. One such group is writing a work of fiction. It's called "The Great Story." Here's one of the latest passages: "Chapter 32: Suddenly, I feel a sharp pain in both the right side of my head and in my left triceps. Everything is spinning and my vision is blurry. The pain in my head greatly intensifies before ..."

Within a few minutes of my chat with Connelly, I heard from Chime.in's PR team. It turns out that Chime.in has community managers who do "human curation" of the content, to bring out the higher-quality material. One such manager—who I later learned was Joy Hepp, an "expert on Mexican travel with five Frommer's titles under her belt"—had alerted the authorities to my inquiry.

That curated setup has some advantages, but the long-term success of Chime.in will likely depend on users eventually being able to create and manage high-quality, spam-free content without such assistance. Albert-László Barabási, Linked: The New science of Networks (Cambridge: Perseus Publishing, 2002), 67-72

The Sixth Link: The 80/20 Rule

***

If you are not a physicist or mathematician, most likely you have never heard of power laws. That is because most quantities in nature follow a bell curve, a distribution rather similar to the peaked distribution characterizing random networks. For example, if you measure the height of all your adult male acquaintances and prepare a histogram counting how many of them are four, five, six, or seven feet tall, you will find that most people in your sample are between five and six feet tall. Your histogram will have a peak around these values. Indeed, unless you hang out a lot with basketball players, you will have very few seven-or eight-foot people in your sample. The same is true for shorter people: Three-or four-feet-tall individuals will be rather rare. As most quantities in nature follow such a peaked distribution, ranging from our IQs to the velocity of molecules in a gas, many people are familiar with these ubiquitous bell curves.

In the past few decades scientists have recognized that on occasion nature generates quantities that follow a power law distribution instead of a bell curve. Power laws are very different from the bell curves describing our heights. First, a power law distribution does not have a peak. Rather, a histogram following a power law is a continuously decreasing curve, implying that many small events coexist with a few large events. If the heights of an imaginary planet's inhabitants followed a power law distribution, most creatures would be really short. But nobody would be surprised to see occasionally a hundred-feet-tall monster walking down the street. In fact, among six billion inhabitants there would be at least one over 8,000 feet tall. So the distinguishing feature of a power law is not only that there are many small events but that the numerous tiny events coexist with a few very large ones. These extraordinarily large events are simply forbidden in a bell curve.1

Each power law is characterized by a unique exponent, telling us, for example, how many very popular Webpages are out there relative to the less popular ones. As in networks the power law describes the degree distribution; the exponent is often called the degree exponent. Our measurements indicated that the distribution of incoming links on Web pages followed a power law with a unique and well-defined degree exponent close to two. A similar power law was present when we looked at outgoing links, the degree exponent this time being slightly larger.2 Our tiny robot offered compelling evidence that millions of Web-page creators work together in some magic way to generate a complex Web that defies the random universe. Their collective action forces the degree distribution to evade the bell curve-a signature of random networks-and

1 Note that there is an important qualitative difference between a power law and a bell curve when it comes to the tail of the distribution. Bell curves have an exponentially decaying tail, which is a much faster decrease than that displayed by a power law. This exponential tail is responsible for the absence of the hubs. In comparison, power laws decay far more slowly, allowing for "rare events" such as the hubs.

2 This implies that the number of Webpages with exactly k incoming links, denoted by N(k), follows N(k) ~ k-r, where the parameter r is the degree exponent. The slope of the straight line on the log-log plot indicates that the degree exponent had a value close to 2.1. When we counted how many outgoing I inks were on a given World Wide Web document, we observed the same pattern: The log-log plot revealed that the number of pages with exactly k outgoing links follow N(k) ~ k-r, with r = 2.5. to tum the Web into a very peculiar network described by a power law. The robot failed to answer our most pressing question, however. What was it about the Web that prompted it to defy the strict predictions of random networks?

Then we realized that there was another way to approach this problem. Could it be that equally simple laws characterize most complex networks and we had not seen them because we had not looked for them before? This second line of questioning turned out to be much more fruitful. Indeed, a few months later, while analyzing the actor network behind Hollywood, we found that it too followed the same mathematical relationship: The number of actors that had links to exactly k other actors decays following a power law. Later we learned that Erdos and his mathematician colleagues obeyed this law, too. The web within the cell joined the list as we learned that the number of molecules interacting with exactly k other molecules decays following a power law. We also discovered a paper by Sid Redner, a professor of physics at Boston University, who found that the distribution of citations in physics journals follows a power law. Viewing citations as links of a network whose nodes are publications, Redner's finding implied that the citation network is also described by a power-law degree distribution. Subsequently, in numerous large networks that we and many other scientists have had a chance to investigate, an amazingly simple and consistent pattern has emerged: The number of nodes with exactly k links follows a power law, each with a unique degree exponent that for most systems varies between two and three.

The striking visual and structural differences between a random network and one described by a power-law degree distribution are best seen by comparing a U.S. roadmap with an airline routing map. On the roadmap cities are the nodes and the highways connecting them the links. This is a fairly uniform network: Each major city has at least one link to the highway system, and there are no cities served by hundreds of highways. Thus most nodes are fairly similar, with roughly the same number of links. As we saw in Chapter 2, such uniformity is an inherent property of random networks with a peaked degree distribution. The airline routing map differs drastically from the roadmap. The nodes of this network are airports connected by direct flights between them. Inspecting the maps displayed in the glossy flight magazines placed on the back of each airplane seat, we cannot fail to notice a few hubs, such as Chicago, Dallas, Denver, Atlanta, and New York, from which flights depart to almost all other U.S. airports. The vast majority of airports are tiny, appearing as nodes with at most a few links connecting them to one or several hubs. Thus, in contrast to the highway map, where most nodes are equivalent, on the airline map a few hubs connect hundreds of small airports (Figure 6.1).

A similar unevenness characterizes networks with power-law degree distribution. Power laws mathematically formulate the fact that in most real networks the majority of nodes have only a few links and that these numerous tiny nodes coexist with a few big hubs, nodes with ananomalously high number of links. The few links connecting the smaller nodes to each other are not sufficient to ensure that the network is fully connected. This function is secured by the relatively rare hubs that keep real networks from falling apart.

In a random network the peak of the distribution implies that the vast majority of nodes have the same number of links and that nodes deviating from the average are extremely rare. Therefore, a random network has a characteristic scale in its node connectivity, embodied by the average node and fixed by the peak of the degree distribution. In contrast, the absence of a peak in a power-law degree distribution implies that in a real network there is no such thing as a characteristic node. We see a continuous hierarchy of nodes, spanning from rare hubs to the numerous tiny nodes. The largest hub is closely followed by two or three somewhat smaller hubs, followed by dozens that are even smaller, and so on, eventually arriving at the numerous small nodes.

The power law distribution thus forces us to abandon the idea of a scale, or a characteristic node. In a continuous hierarchy there is no single node which we could pick out and claim to be characteristic of all the nodes. There is no intrinsic scale in these networks. This is the reason my research group started to describe networks with power-law degree distribution as scale-free. With the realization that most complex networks in nature have a power~ law degree distribution, the term scale-free networks rapidly infiltrated most disciplines faced with complex webs.

Neither the hierarchy of omnipresent hubs nor the accompanying power laws were accounted for in either of the network theories available at the time that we discovered them in 1999. If anything, they were considered merely accidental. The random network theory of Erdos and Renyi and its cluster-friendly extension by Watts and Strogatz both insisted that the number of nodes with k links should decrease exponentially—a much faster decay than that predicted by a power law. They both told us, in rigorous mathematical terms, that hubs do not exist.

The surprising discovery of power laws in the Web forced us to acknowledge the hubs. The slowly decaying power law distribution accommodates such highly linked anomalies in a natural way. It predicts that each scale-free network will have several large hubs that will fundamentally define the network's topology. The finding that most networks of conceptual importance, ranging from the World Wide Web to the network within the cell, are scale-free gave legitimacy to hubs. We would come to see that they determine the structural stability, dynamic behavior, robustness, and error and attack tolerance of real networks. They stand as proof of the highly important organizing principles that govern network evolution.

*** Don Peck, “Can the Middle Class Be Saved?” The Atlantic, September 2011, 1-2. Available online: http://www.theatlantic.com/magazine/archive/2011/09/can-the-middle-class-be- saved/8600/

Can the Middle Class Be Saved?

In October 2005, three Citigroup analysts released a report describing the pattern of growth in the U.S. economy. To really understand the future of the economy and the stock market, they wrote, you first needed to recognize that there was “no such animal as the U.S. consumer,” and that concepts such as “average” consumer debt and “average” consumer spending were highly misleading.

In fact, they said, America was composed of two distinct groups: the rich and the rest. And for the purposes of investment decisions, the second group didn’t matter; tracking its spending habits or worrying over its savings rate was a waste of time. All the action in the American economy was at the top: the richest 1 percent of households earned as much each year as the bottom 60 percent put together; they possessed as much wealth as the bottom 90 percent; and with each passing year, a greater share of the nation’s treasure was flowing through their hands and into their pockets. It was this segment of the population, almost exclusively, that held the key to future growth and future returns. The analysts, Ajay Kapur, Niall Macleod, and Narendra Singh, had coined a term for this state of affairs: plutonomy.

In a plutonomy, Kapur and his co-authors wrote, “economic growth is powered by and largely consumed by the wealthy few.” America had been in this state twice before, they noted—during the Gilded Age and the Roaring Twenties. In each case, the concentration of wealth was the result of rapid technological change, global integration, laissez-faire government policy, and “creative financial innovation.” In 2005, the rich were nearing the heights they’d reached in those previous eras, and Citigroup saw no good reason to think that, this time around, they wouldn’t keep on climbing. “The earth is being held up by the muscular arms of its entrepreneur- plutocrats,” the report said. The “great complexity” of a global economy in rapid transformation would be “exploited best by the rich and educated” of our time.

Kapur and his co-authors were wrong in some of their specific predictions about the plutonomy’s ramifications—they argued, for instance, that since spending was dominated by the rich, and since the rich had very healthy balance sheets, the odds of a stock-market downturn were slight, despite the rising indebtedness of the “average” U.S. consumer. And their division of America into only two classes is ultimately too simple. Nonetheless, their overall characterization of the economy remains resonant. According to Gallup, from May 2009 to May 2011, daily consumer spending rose by 16 percent among Americans earning more than $90,000 a year; among all other Americans, spending was completely flat. The consumer recovery, such as it is, appears to be driven by the affluent, not by the masses. Three years after the crash of 2008, the rich and well educated are putting the recession behind them. The rest of America is stuck in neutral or reverse.

Income inequality usually shrinks during a recession, but in the Great Recession, it didn’t. From 2007 to 2009, the most-recent years for which data are available, it widened a little. The top 1 percent of earners did see their incomes drop more than those of other Americans in 2008. But that fall was due almost entirely to the stock-market crash, and with it a 50 percent reduction in realized capital gains. Excluding capital gains, top earners saw their share of national income rise even in 2008. And in any case, the stock market has since rallied. Corporate profits have marched smartly upward, quarter after quarter, since the beginning of 2009.

Even in the financial sector, high earners have come back strong. In 2009, the country’s top 25 hedge-fund managers earned $25 billion among them—more than they had made in 2007, before the crash. And while the crisis may have begun with mass layoffs on Wall Street, the financial industry has remained well shielded compared with other sectors; from the first quarter of 2007 to the first quarter of 2010, finance shed 8 percent of its jobs, compared with 27 percent in construction and 17 percent in manufacturing. Throughout the recession, the unemployment rate in finance and insurance has been substantially below that of the nation overall.

It’s hard to miss just how unevenly the Great Recession has affected different classes of people in different places. From 2009 to 2010, wages were essentially flat nationwide—but they grew by 11.9 percent in Manhattan and 8.7 percent in Silicon Valley. In the Washington, D.C., and San Jose (Silicon Valley) metro areas—both primary habitats for America’s meritocratic winners—job postings in February of this year were almost as numerous as job candidates. In Miami and Detroit, by contrast, for every job posting, six people were unemployed. In March, the national unemployment rate was 12 percent for people with only a high-school diploma, 4.5 percent for college grads, and 2 percent for those with a professional degree.

Housing crashed hardest in the exurbs and in more-affordable, once fast-growing areas like Phoenix, Las Vegas, and much of Florida—all meccas for aspiring middle-class families with limited savings and education. The professional class, clustered most densely in the closer suburbs of expensive but resilient cities like San Francisco, Seattle, Boston, and Chicago, has lost little in comparison. And indeed, because the stock market has rebounded while housing values have not, the middle class as a whole has seen more of its wealth erased than the rich, who hold more-diverse portfolios. A 2010 Pew study showed that the typical middle-class family had lost 23 percent of its wealth since the recession began, versus just 12 percent in the upper class.

The ease with which the rich and well educated have shrugged off the recession shouldn’t be surprising; strong winds have been at their backs for many years. The recession, meanwhile, has restrained wage growth and enabled faster restructuring and offshoring, leaving many corporations with lower production costs and higher profits—and their executives with higher pay.

Anthony Atkinson, an economist at Oxford University, has studied how several recent financial crises affected income distribution—and found that in their wake, the rich have usually strengthened their economic position. Atkinson examined the financial crises that swept Asia in the 1990s as well as those that afflicted several Nordic countries in the same decade. In most cases, he says, the middle class suffered depressed income for a long time after the crisis, while the top 1 percent were able to protect themselves—using their cash reserves to buy up assets very cheaply once the market crashed, and emerging from crisis with a significantly higher share of assets and income than they’d had before. “I think we’ve seen the same thing, to some extent, in the United States” since the 2008 crash, he told me. “Mr. Buffett has been investing.”

“The rich seem to be on the road to recovery,” says Emmanuel Saez, an economist at Berkeley, while those in the middle, especially those who’ve lost their jobs, “might be permanently hit.” Coming out of the deep recession of the early 1980s, Saez notes, “you saw an increase in inequality … as the rich bounced back, and unionized labor never again found jobs that paid as well as the ones they’d had. And now I fear we’re going to see the same phenomenon, but more dramatic.” Middle-paying jobs in the U.S., in which some workers have been overpaid relative to the cost of labor overseas or technological substitution, “are being wiped out. And what will be left is a hard and a pure market,” with the many paid less than before, and the few paid even better—a plutonomy strengthened in the crucible of the post-crash years.

The Culling of the Middle Class

One of the most salient features of severe downturns is that they tend to accelerate deep economic shifts that are already under way. Declining industries and companies fail, spurring workers and capital toward rising sectors; declining cities shrink faster, leaving blight; workers whose roles have been partly usurped by technology are pushed out en masse and never asked to return. Some economists have argued that in one sense, periods like these do nations a service by clearing the way for new innovation, more-efficient production, and faster growth. Whether or not that’s true, they typically allow us to see, with rare and brutal clarity, where society is heading—and what sorts of people and places it is leaving behind.

Arguably, the most important economic trend in the United States over the past couple of generations has been the ever more distinct sorting of Americans into winners and losers, and the slow hollowing-out of the middle class. Median incomes declined outright from 1999 to 2009. For most of the aughts, that trend was masked by the housing bubble, which allowed working- class and middle-class families to raise their standard of living despite income stagnation or downward job mobility. But that fig leaf has since blown away. And the recession has pressed hard on the broad center of American society.

“The Great Recession has quantitatively but not qualitatively changed the trend toward employment polarization” in the United States, wrote the MIT economist David Autor in a 2010 white paper. Job losses have been “far more severe in middle-skilled white- and blue-collar jobs than in either high-skill, white-collar jobs or in low-skill service occupations.” Indeed, from 2007 through 2009, total employment in professional, managerial, and highly skilled technical positions was essentially unchanged. Jobs in low-skill service occupations such as food preparation, personal care, and house cleaning were also fairly stable. Overwhelmingly, the recession has destroyed the jobs in between. Almost one of every 12 white-collar jobs in sales, administrative support, and nonmanagerial office work vanished in the first two years of the recession; one of every six blue-collar jobs in production, craft, repair, and machine operation did the same.

Autor isolates the winnowing of middle-skill, middle-class jobs as one of several labor-market developments that are profoundly reshaping U.S. society. The others are rising pay at the top, falling wages for the less educated, and “lagging labor market gains for males.” “All,” he writes, “predate the Great Recession. But the available data suggest that the Great Recession has reinforced these trends.”

For more than 30 years, the American economy has been in the midst of a sea change, shifting from industry to services and information, and integrating itself far more tightly into a single, global market for goods, labor, and capital. To some degree, this transformation has felt disruptive all along. But the pace of the change has quickened since the turn of the millennium, and even more so since the crash. Companies have figured out how to harness exponential increases in computing power better and faster. Global supply chains, meanwhile, have grown both tighter and more supple since the late 1990s—the result of improving information technology and of freer trade—making routine work easier to relocate. And of course China, India, and other developing countries have fully emerged as economic powerhouses, capable of producing large volumes of high-value goods and services. Some parts of America’s transformation may now be nearing completion. For decades, manufacturing has become continually less important to the economy, as other business sectors have grown. But the popular narrative—rapid decline in the 1970s and ’80s, followed by slow erosion thereafter—isn’t quite right, at least as far as employment goes. In fact, the total number of people employed in industry remained quite stable from the late 1960s through about 2000, at roughly 17 million to 19 million. To be sure, manufacturing wasn’t providing many new jobs for a growing population, but for decades, rising output essentially offset the impact of labor-saving technology and offshoring.

But since 2000, U.S. manufacturing has shed about a third of its jobs. Some of that decline reflects losses to China. Still, industry isn’t about to vanish from America, any more than agriculture did as the number of farm workers plummeted during the 20th century. As of 2010, the United States was the second-largest manufacturer in the world, and the No. 3 agricultural nation. But agriculture is now so mechanized that only about 2 percent of American workers make a living as farmers. American manufacturing looks to be heading down the same path.

Meanwhile, another phase of the economy’s transformation—one more squarely involving the white-collar workforce—is really just beginning. “The thing about information technology,” Autor told me, “is that it’s extremely broadly applicable, it’s getting cheaper all the time, and we’re getting better and better at it.” Computer software can now do boilerplate legal work, for instance, and make a first pass at reading X-rays and other medical scans. Likewise, thanks to technology, we can now easily have those scans read and interpreted by professionals half a world away.

In 2007, the economist Alan Blinder, a former vice chairman of the Federal Reserve, estimated that between 22 and 29 percent of all jobs in the United States had the potential to be moved overseas within the next couple of decades. With the recession, the offshoring of jobs only seems to have gained steam. The financial crisis of 2008 was global, but job losses hit America especially hard. According to the International Monetary Fund, one of every four jobs lost worldwide was lost in the United States. And while unemployment remains high in America, it has come back down to (or below) pre-recession levels in countries like China and Brazil.

Anxiety Creeps Upward

Over time, both trade and technology have increased the number of low-cost substitutes for American workers with only moderate cognitive or manual skills—people who perform routine tasks such as product assembly, process monitoring, record keeping, basic information brokering, simple software coding, and so on. As machines and low-paid foreign workers have taken on these functions, the skills associated with them have become less valuable, and workers lacking higher education have suffered.

For the most part, these same forces have been a boon, so far, to Americans who have a good education and exceptional creative talents or analytic skills. Information technology has complemented the work of people who do complex research, sophisticated analysis, high-end deal-making, and many forms of design and artistic creation, rather than replacing that work. And global integration has meant wider markets for new American products and high-value services—and higher incomes for the people who create or provide them.

The return on education has risen in recent decades, producing more-severe income stratification. But even among the meritocratic elite, the economy’s evolution has produced a startling divergence. Since 1993, more than half of the nation’s income growth has been captured by the top 1 percent of earners, and the gains have grown larger over time: from 2002 to 2007, out of every three dollars of national income growth, the top 1 percent of earners captured two. Nearly 2 million people started college in 2002—1,630 of them at Harvard—but among them only Mark Zuckerberg is worth more than $10 billion today; the rise of the super-elite is not a product of educational differences. In part, it is a natural outcome of widening markets and technological revolution, which are creating much bigger winners much faster than ever before—a result that’s not even close to being fully played out, and one reinforced strongly by the political influence that great wealth brings.

Recently, as technology has improved and emerging-market countries have sent more people to college, economic pressures have been moving up the educational ladder in the United States. “It’s useful to make a distinction between college and post-college,” Autor told me. “Among people with professional and even doctoral [degrees], in general the job market has been very good for a very long time, including recently. The group of highly educated individuals who have not done so well recently would be people who have a four-year college degree but nothing beyond that. Opportunities have been less good, wage growth has been less good, the recession has been more damaging. They’ve been displaced from mid-managerial or organizational positions where they don’t have extremely specialized, hard-to-find skills.”

College graduates may be losing some of their luster for reasons beyond technology and trade. As more Americans have gone to college, Autor notes, the quality of college education has become arguably more inconsistent, and the signaling value of a degree from a nonselective school has perhaps diminished. Whatever the causes, “a college degree is not the kind of protection against job loss or wage loss that it used to be.”

Without doubt, it is vastly better to have a college degree than to lack one. Indeed, on a relative basis, the return on a four-year degree is near its historic high. But that’s largely because the prospects facing people without a college degree have been flat or falling. Throughout the aughts, incomes for college graduates barely budged. In a decade defined by setbacks, perhaps that should occasion a sort of wan celebration. “College graduates aren’t doing badly,” says Timothy Smeeding, an economist at the University of Wisconsin and an expert on inequality. But “all the action in earnings is above the B.A. level.”

America’s classes are separating and changing. A tiny elite continues to float up and away from everyone else. Below it, suspended, sits what might be thought of as the professional middle class—unexceptional college graduates for whom the arrow of fortune points mostly sideways, and an upper tier of college graduates and postgraduates for whom it points progressively upward, but not spectacularly so. The professional middle class has grown anxious since the crash, and not without reason. Yet these anxieties should not distract us from a second, more important, cleavage in American society—the one between college graduates and everyone else.

If you live and work in the professional communities of Boston or Seattle or Washington, D.C., it is easy to forget that nationwide, even among people ages 25 to 34, college graduates make up only about 30 percent of the population. And it is easy to forget that a family income of $113,000 in 2009 would have put you in the 80th income percentile nationally. The true center of American society has always been its nonprofessionals—high-school graduates who didn’t go on to get a bachelor’s degree make up 58 percent of the adult population. And as manufacturing jobs and semiskilled office positions disappear, much of this vast, nonprofessional middle class is drifting downward.

*** Eduardo Porter, “How Superstars’ Pay Stifles Everyone Else,” New York Times, December 25, 2010. Available online: www.nytimes.com/2010/12/26/business/26excerpt.html

How Superstars’ Pay Stifles Everyone Else

In 1990, the Kansas City Royals had the heftiest payroll in Major League Baseball: almost $24 million. A typical player for the New York Yankees, which had some of the most expensive players in the game at the time, earned less than $450,000.

Last season, the Yankees spent $206 million on players, more than five times the payroll of the Royals 20 years ago, even after accounting for inflation. The Yankees’ median salary was $5.5 million, seven times the 1990 figure, inflation-adjusted.

What is most striking is how the Yankees have outstripped the rest of the league. Two decades ago, the Royals’ payroll was about three times as big as that of the Chicago White Sox, the cheapest major-league team at the time. Last season, the Yankees spent about six times as much as the Pittsburgh Pirates, who had the most inexpensive roster.

Baseball aficionados might conclude that all of this points to some pernicious new trend in the market for top players. But this is not specific to baseball, or even to sport. Consider the market for pop music. In 1982, the top 1 percent of pop stars, in terms of pay, raked in 26 percent of concert ticket revenue. In 2003, that top percentage of stars — names like Justin Timberlake, Christina Aguilera or 50 Cent — was taking 56 percent of the concert pie.

The phenomenon is not even specific to the United States. Pelé, from Brazil, the greatest soccer player of all time, made his World Cup debut in Sweden in 1958, when he was only 17. He became an instant star, coveted by every team on the planet. By 1960, his team, Santos, reportedly paid him $150,000 a year — about $1.1 million in today’s money. But these days, that would amount to middling pay. The top-paid player of the 2009-10 season, the Portuguese forward Cristiano Ronaldo, made $17 million playing for the Spanish team Real Madrid.

Of course, the inflated rewards of performers at the very top have to do with specific changes in the underlying economics of entertainment. People have more disposable income to spend on entertainment. Corporate sponsorships, virtually non-existent in the age of Pelé, account today for a large share of performers’ income. In 2009, the highest-earning soccer player was the English midfielder David Beckham, who made $33 million from endorsements on top of a $7 million salary from the Los Angeles Galaxy and AC Milan.

But broader forces are also at play. Nearly 30 years ago, Sherwin Rosen, an economist from the University of Chicago, proposed an elegant theory to explain the general pattern. In an article entitled “The Economics of Superstars,” he argued that technological changes would allow the best performers in a given field to serve a bigger market and thus reap a greater share of its revenue. But this would also reduce the spoils available to the less gifted in the business.

The reasoning fits smoothly into the income dynamics of the music industry, which has been shaken by many technological disruptions since the 1980s. First, MTV put music on television. Then Napster took it to the Internet. Apple allowed fans to buy single songs and take them with them. Each of these breakthroughs allowed the very top acts to reach a larger fan base, and thus command a larger audience and a bigger share of concert revenue. Superstar effects apply, too, to European soccer, which is beamed around the world on cable and satellite TV. In 2009, the top 20 soccer teams reaped revenue of 3.9 billion euros, more than 25 percent of the combined revenue of all the teams in European leagues.

Pelé was not held back by the quality of his game, but by his relatively small revenue base. He might be the greatest of all time, but few people could pay to experience his greatness. In 1958, there were about 350,000 television sets in Brazil. The first television satellite, Telstar I, wasn’t launched until July 1962, too late for his World Cup debut.

By contrast, the 2010 FIFA World Cup in South Africa, in which Ronaldo played for Portugal, was broadcast in more than 200 countries, to an aggregate audience of over 25 billion. Some 700 million people watched the final alone. Ronaldo is not better then Pelé. He makes more money because his talent is broadcast to more people.

IF one loosens slightly the role played by technological progress, Dr. Rosen’s framework also does a pretty good job explaining the evolution of executive pay. In 1977, an elite chief executive working at one of America’s top 100 companies earned about 50 times the wage of its average worker. Three decades later, the nation’s best-paid C.E.O.’s made about 1,100 times the pay of a worker on the production line.

This has separated the megarich from the merely very rich. A study of pay in the 1970s found that executives in the top 10 percent made about twice as much as those in the middle of the pack. By the early 2000s, the top suits made more than four times the pay of the executives in the middle.

Top C.E.O.’s are not pop stars. But the pay for the most sought-after executives has risen for similar reasons. As corporations have increased in size, management decisions at the top have become that much more important, measured in terms of profits or losses. Top American companies have much higher sales and profits than they did 20 years ago. Banks and funds have more assets.

With so much more at stake, it has become that much more important for companies to put at the helm the “best” executive or banker or fund manager they can find. This has set off furious competition for top managerial talent, pushing the prices of top-rated managers way above the pay of those in the tier just below them. Two economists at New York University, Xavier Gabaix and Augustin Landier, published a study in 2006 estimating that the sixfold rise in the pay of chief executives in the United States over the last quarter century or so was attributable entirely to the sixfold rise in the market size of large American companies.

And therein lies a big problem for American capitalism.

Capitalism relies on inequality. Like differences in other prices, pay disparities steer resources — in this case, people — to where they would be most productively employed.

Despite the great danger and cost of crossing the border illegally into the United States, hundreds of thousands of the hardest-working Mexicans are drawn by the relative prosperity they can achieve north of the border — where the average income of a Mexican-American household is more than $33,000, almost five times that of a family in Mexico.

In poor economies, fast economic growth increases inequality as some workers profit from new opportunities and others do not. The share of national income accruing to the top 1 percent of the Chinese population more than doubled from 1986 to 2003. Inequality spurs economic growth by providing incentives for people to accumulate human capital and become more productive. It pulls the best and brightest into the most lucrative lines of work, where the most profitable companies hire them.

Yet the increasingly outsize rewards accruing to the nation’s elite clutch of superstars threaten to gum up this incentive mechanism. If only a very lucky few can aspire to a big reward, most workers are likely to conclude that it is not worth the effort to try. The odds aren’t on their side.

Inequality has been found to turn people off. A recent experiment conducted with workers at the University of California found that those who earned less than the typical wage for their pay unit and occupation became measurably less satisfied with their jobs, and more likely to look for another one if they found out the pay of their peers. Other experiments have found that winner- take-all games tend to elicit much less player effort — and more cheating — than those in which rewards are distributed more smoothly according to performance.

Ultimately, the question is this: How much inequality is necessary? It is true that the nation grew quite fast as inequality soared over the last three decades. Since 1980, the country’s gross domestic product per person has increased about 69 percent, even as the share of income accruing to the richest 1 percent of the population jumped to 36 percent from 22 percent. But the economy grew even faster — 83 percent per capita — from 1951 to 1980, when inequality declined when measured as the share of national income going to the very top of the population.

One study concluded that each percentage-point increase in the share of national income channeled to the top 10 percent of Americans since 1960 led to an increase of 0.12 percentage points in the annual rate of economic growth — hardly an enormous boost. The cost for this tonic seems to be a drastic decline in Americans’ economic mobility. Since 1980, the weekly wage of the average worker on the factory floor has increased little more than 3 percent, after inflation.

The United States is the rich country with the most skewed income distribution. According to the Organization for Economic Cooperation and Development, the average earnings of the richest 10 percent of Americans are 16 times those for the 10 percent at the bottom of the pile. That compares with a multiple of 8 in Britain and 5 in Sweden.

Not coincidentally, Americans are less economically mobile than people in other developed countries. There is a 42 percent chance that the son of an American man in the bottom fifth of the income distribution will be stuck in the same economic slot. The equivalent odds for a British man are 30 percent, and 25 percent for a Swede.

NONE of this even begins to account for the damage caused by the superstar dynamics that shape the pay of American bankers.

Remember the ’80s? Gordon Gekko first sashayed across the silver screen. Ivan Boesky was jailed for insider trading. Michael Milken peddled junk bonds. In 1987, financial firms amassed a little less than a fifth of the profits of all American corporations. Wall Street bonuses totaled $2.6 billion — about $15,600 for each man and woman working there.

Yet by current standards, this era of legendary greed appears like a moment of uncommon restraint. In 2007, as the financial bubble built upon the American housing market reached its peak, financial companies accounted for a full third of the profits of the nation’s private sector. Wall Street bonuses hit a record $32.9 billion, or $177,000 a worker.

Just as technology gave pop stars a bigger fan base that could buy their CDs, download their singles and snap up their concert tickets, the combination of information technology and deregulation gave bankers an unprecedented opportunity to reap huge rewards. Investors piled into the top-rated funds that generated the highest returns. Rewards flowed in abundance to the most “productive” financiers, those that took the bigger risks and generated the biggest profits.

Finance wasn’t always so richly paid. Financiers had a great time in the early decades of the 20th century: from 1909 to the mid-1930s, they typically made about 50 percent to 60 percent more than workers in other industries. But the stock market collapse of 1929 and the Great Depression changed all that. In 1934, corporate profits in the financial sector shrank to $236 million, one- eighth what they were five years earlier. Wages followed. From 1950 through about 1980, bankers and insurers made only 10 percent more than workers outside of finance, on average.

This ebb and flow of compensation mimics the waxing and waning of restrictions governing finance. A century ago, there were virtually no regulations to restrain banks’ creativity and speculative urges. They could invest where they wanted, deploy depositors’ money as they saw fit. But after the Great Depression, President Franklin D. Roosevelt set up a plethora of restrictions to avoid a repeat of the financial bubble that burst in 1929.

Interstate banking had been limited since 1927. In 1933, the Glass-Steagall Act forbade commercial banks and investment banks from getting into each other’s business — separating deposit taking and lending from playing the markets. Interest-rate ceilings were also imposed that year. The move to regulate bankers continued in 1959 under President Dwight D. Eisenhower, who forbade mixing banks with insurance companies.

Barred from applying the full extent of their wits toward maximizing their incomes, many of the nation’s best and brightest who had flocked to make money in banking left for other industries.

Then, in the 1980s, the Reagan administration unleashed a surge of deregulation. By 1999, the Glass-Steagall Act lay repealed. Banks could commingle with insurance companies at will. Ceilings on interest rates vanished. Banks could open branches anywhere. Unsurprisingly, the most highly educated returned to banking and finance. By 2005, the share of workers in the finance industry with a college education exceeded that of other industries by nearly 20 percentage points. By 2006, pay in the financial sector was again 70 percent higher than wages elsewhere in the private sector. A third of the 2009 Princeton graduates who got jobs after graduation went into finance; 6.3 percent took jobs in government.

Then the financial industry blew up, taking out a good chunk of the world economy.

Finance will not be tamed by tweaking the way bankers are paid. But bankers’ pay could be structured to discourage wanton risk taking. Similarly, superstar effects are not the sole cause of the stagnant incomes of regular Joes. But the piling of rewards on our superstars is encouraging a race to the top that, if left unabated, could leave very little to strive for in its wake.

This article was adapted from “The Price of Everything: Solving the Mystery of Why We Pay What We Do,” by Eduardo Porter, an editorial writer for The New York Times.