ON A CHINESE SCREEN

‘Face, Face, Face.’ PC: Terhide Tomori, Flickr. The Global Age of com. Algorithm Social Credit and the Financialisation of Governance in China

Nicholas LOUBERE Stefan BREHM

Much has been made of the Orwellian social Beautiful credit! The foundation of modern control aspects of the emerging ‘social credit society. Who shall say that this is not the system’ in China. However, social credit golden age of mutual trust, of unlimited is more than simply a Chinese version of reliance upon human promises? big brother: it is an unprecedented climax of the global financialisation project Mark Twain and Charles Dudley Warner, and a signal of a potential dark digital The Gilded Age: A Tale of Today (1873) future dominated by algorithmic rule. n recent years, few news items out of China have resulted in as much anxiety and fear Iin western media and public discourse than the Chinese government’s ongoing attempts to create a ‘social credit system’ (shehui xinyong

142 MADE IN CHINA YEARBOOK 2018 ON A CHINESE SCREEN

tixi) aimed at rating the trustworthiness big data generated from economic, social, of individuals and companies. Most major and commercial behaviour. The stated aim Western media outlets have spent significant is to ‘provide the trustworthy with benefits energy warning about China’s efforts to create and discipline the untrustworthy … [so that] an Orwellian dystopia. The most hyperbolic integrity becomes a widespread social value’ of these—The Economist—has even run with (State Council 2016). While official policy menacing headlines like ‘China Invents documents are light on detail with regard to the Digital Totalitarian State’ and ‘China’s how the social credit system will ultimately Digital Dictatorship’ (The Economist 2016b; operate, they have suggested various ways The Economist 2016c). These articles both to punish untrustworthy members of society implicitly and explicitly depict social credit as (i.e. those with low ratings), such as through something unique to China—a nefarious and restrictions on employment, consumption, perverse digital innovation that could only be travel, and access to credit. In recent months, conceived of and carried out by a regime like there have already been reports of blacklisting the Chinese Communist Party (Daum 2017). resulting in restrictions for individuals, but Social credit is thus seen as signalling the as of yet this only applies to those who have onset of a dystopian future that could only broken specific laws or ‘failed to perform exist in the Chinese context. But how unique to certain legal obligations’ (Daum 2018a). China is this attempt to ‘build an environment The Chinese social credit system is emerging of trust’—to quote the State Council (2016)— rapidly, and the aforementioned blacklists are using new digital forms of data collection and connecting data from dozens of governmental analysis? Is this Orwellian social credit system departments. However, it is still far from indicative of an inherently Chinese form of being a unified or centralised system. Like digital life, or is it a dark manifestation of our most new policies in China, social credit is collective impulses to increase transparency being subjected to the country’s distinctive and accountability (at the expense of privacy), policy modelling process (Heilmann 2008), and to integrate everyone into a single where local governments produce their own ‘inclusive’ system to more easily categorise, interpretations of policies, which then vie monitor, and standardise social activity? In this to become national models. Over 30 local essay, we propose that Chinese social credit governments have already started piloting should not be exoticised or viewed in isolation. social credit systems, which utilise different Rather, it must be understood as merely one approaches to arrive at their social credit manifestation of the global age of the algorithm. scores, and which use the scores to achieve different outcomes. In contrast to other policies, however, large Internet companies Engineering a have also been given licenses to run their own pilots (Loubere 2017a). The most widely used Trustworthy Society private social credit system is Alibaba’s Sesame Credit, which utilises opaque algorithms While there have long been discussions to arrive at social credit scores for their about creating an economic and social rating customers. Those with high scores have been system in China, they took a much more able to access a range of benefits from other concrete form in 2014, with the publication of Alibaba businesses and their partners (Bislev a high-level policy document outlining plans 2017). Sesame Credit is significant due to to create a nationwide social credit system the huge amounts of economic data held by by 2020 (State Council 2014). The proposed Alibaba through Alipay and Ant Financial, but system will assign ratings to individuals, it should not be conflated with governmental organisations, and businesses that draw on

MADE IN CHINA YEARBOOK 2018 143 ON A CHINESE SCREEN

social credit system pilots. It is not clear how inclusion project, which seeks to integrate or if the government and private systems will marginal and impoverished populations into be integrated in the future, which seems to be the global capitalist system—primarily through causing a degree of tension between regulators expanded access to credit—as a means of and the Internet giants (Hornby, Ju, and promoting economic development and social Lucas 2018). empowerment. In the same way that Chinese social credit appears poised to extract huge amounts of Financial(ised) Inclusion personal data from individuals in its quest to create a trustworthy society, proponents of financial inclusion justify intrusive methods While social credit can be seen as an of assessing creditworthiness in order to outgrowth of our collective impulse to achieve reduce lender risk from untrustworthy a more trustworthy society, a unified fully- borrowers. Indeed, just months before their functioning social credit system will ultimately hyperbolic headlines about China’s digital turn the quest for trust through transparency authoritarianism, The Economist praised the and accountability upside down, because it use of psychometrics and other personal digital would hold citizens responsible vis-à-vis their data by lenders in developing contexts as being a rulers. At the core of the emerging system, beneficial financial innovation (The Economist the state and financial actors define, quantify, 2016a). In this way, the financial inclusion and calculate trustworthiness and honesty—it project depicts the application of financialised is a technocratic fix based on the logic that, logics as the means of producing a more fair with the correct set of algorithms, the good and accountable inclusive system, where the citizen can be engineered into society. Social trustworthy reap rewards that were denied credit therefore seeks to transform individuals them in the past. However, underpinning this into a new ‘civilised’ (and ‘credit-conscious’) neoliberal fantasy is a glaring contradiction population through the imposition of an that shatters the illusion of inclusion as being incentive and disincentive system that can unbiased and fair—those with capital are able mould logical profit-maximising citizens into to set the terms of their engagement with the civilised subjects. capitalist system much more easily than those In the case of China’s proposed social credit without. system—as with any credit rating system— This points to the fact that the rich will these rewards and punishments are meted out largely be able to extract more of the rewards through engagement with, and incorporation from their participation in financialised rating into, the market. The calculation of credit scores systems—such as the social credit system— requires market activity, which in turn requires while avoiding the sanctions. Moreover, a credit score. Moreover, if social credit is to live punishments are much more dramatic for up to its technocratic promise of systematically those without accumulated capital, as their eliminating untrustworthiness, everyone must very existence depends on their continued be assessed equally—i.e. everyone must be participation in the capitalist system for daily included in the system. In the absence of a survival. From this perspective, the spectre social credit score, the worst must be assumed, of China’s financialised social credit system meaning that the burden of proving ones portents a society comprising individual micro- trustworthiness falls to the individual. Thus, in entrepreneurs operating in a shared economy a society dominated by social credit, integration mode where livelihoods are determined by into the socioeconomic system is a necessity credit scores. Indeed, Sesame Credit already rather than a choice. In this way, China’s social works with sharing economy apps, such as credit resonates with the global financial Daowei, which provides a platform for a literal

144 MADE IN CHINA YEARBOOK 2018 ON A CHINESE SCREEN

gig economy comprised of individuals (with maximisation into the realm of interpersonal their credit scores listed) advertising the sale relationships. Through managing networks, of their services or products (Loubere 2017b). digital activity, and private action, an individual Those looking for a plumber in the area can or organisation can impact social value and, by select one with the highest score, just as people extension, financial capital. Thus, social credit in the west decide hotels and restaurants based creates new incentives that can be used to on Yelp or TripAdvisor reviews. align the interests of citizens and organisations with those of the government. The state, as a shareholder in ‘the people’, enjoys the Financialisation Gone dividends of good behaviour and loyalty, which are rewarded through economic privileges. In Wild this all-encompassing financialised system, social action becomes increasingly entrenched In this sense, the emergence of social credit within the economic realm, and individual represents an unprecedented climax of the behaviour is more and more shaped by financial global financialisation project. Financialisation motives. In a nutshell, social credit represents can be broadly defined as ‘the increasing the ultimate marketisation of political control role of financial motives, financial markets, because it provides incentives for maximising financial actors, and financial institutions in citizen value through politically and the operation of domestic and international commercially aligned social behaviour. economy’ (Epstein 2005, 3). Social credit opens Using algorithms to render citizens and the door for financialising social behaviour. organisations compliant with the visions To elaborate this claim, consider the relation and rationales of the ruling regime reduces between social and financial capital. The the state’s information and monitoring costs OECD, for example, defines social capital as dramatically. In the context of China, this ‘networks together with shared norms, values has the potential to reshape the ‘fragmented and understandings that facilitate cooperation authoritarian’ model, which is characterised within or among groups’ (Keeley 2007, 103). by decentralised decision-making and policy In the digital age, these networks become the implementation (Mertha 2009). One could linchpins between the social and economic envision a future in which the many local spheres. On the one hand, networks are officials and bureaucrats that enjoy privileges more concrete and easier to observe than the due to the central leadership’s reliance on their norms and values shaping the perceptions and support to govern the masses will be subject to behaviour of network members. On the other the rule of algorithms themselves. Only a small hand, social networks represent a crucial means elite would be needed to manage algorithmic for both gaining access to material resources rule, entailing a dramatic reconcentration of and shaping the rules for resource distribution. power. If Chinese experiments are successful, Thus, analysing and contextualising social they will certainly serve as a model for many networks in a big-data driven world allow for other countries: authoritarian regimes, inferences to be made about both the social and democratic systems with authoritarian economic attributes of an individual. tendencies, and eventually democracies The invention of social credit establishes that struggle to maintain legitimacy in an an explicit and tangible link between social increasingly polarised and fragmented political behaviour and economic benefits. In this landscape. context, the State is able to assume a new role not dissimilar to that of a corporate shareholder. Social credit creates a market for social capital and transplants the rationale of profit

MADE IN CHINA YEARBOOK 2018 145 ON A CHINESE SCREEN

The Repressive Logics infrastructure primarily targeting the Uyghur ethnic minority. This includes police of Financialised checkpoints, iris scans, mandatory spyware Governance installed on mobile devises, and pervasive CCTV with facial recognition software. These surveillance technologies feed into, and draw As noted above, despite the discourse of on, a database that includes information about inclusion resulting in transparency and fairer personal identity, family and friends, movement distribution or resources, social credit and and shopping behaviour, and even DNA that the financialisation of social behaviour are is collected at medical check-ups organised inherently biased and paradoxically result by the government. Ultimately, these data are in socioeconomic exclusion within an all- run through algorithms that assign residents encompassing inclusive system. In addition to with public safety scores deeming them ‘safe’, being partial to those with capital, social credit ‘unsafe’, or somewhere in between (Millward will likely also widen other socioeconomic 2018). Those who are deemed to be a threat cleavages. Tests and experiments again and are often detained and sent to reeducation again confirm that data and algorithms are just centres (Foreign Policy 2018). While this is as biased as society is and inevitably reproduce not the government’s proposed social credit real life segmentation and inequality (Bodkin system per se—as these types of data are not 2017). Cathy O’Neil, the author of Weapons legally allowed to be collected for public or of Math Destruction, for instance, warns that market information (Daum 2018b)—the logics we need algorithmic audits (O’Neil 2016). underpinning this type of coercive surveillance After all, algorithms are not some naturally infrastructure and the dreams a nationwide occurring phenomena, but are the reflections citizen rating system are the same. of the people (and societies) that create them. These developments represent a new For this reason, the rule of algorithms must reality that, while shocking initially, has not be mistaken as an upgraded, more rational, become a banal part of everyday existence in and hyper-scientific rule of law 2.0. This is a few short years. It is becoming increasingly particularly true in China, where the concept of clear that Xinjiang is a testing ground for the rule of law has been increasingly developed technologies and techniques that will be rolled and theorised by the Party-state to justify its out nationwide—and even beyond—in the near attempts to consolidate control over society future. For instance, over the spring festival (Rosenzweig, Smith, and Treveskes 2017). period railway police in Henan used glasses In recent years, China has already been augmented with facial recognition software providing glimpses of the repressive connected to a centralised database to identify possibilities of algorithmic rule. In particular, suspected criminals (Wade 2018). China’s recent reports about the construction of a massive surveillance market is also a global sophisticated high-tech surveillance state in affair, with companies from around the world the Xinjiang Autonomous Region anticipate lining up to develop products for both the a near future where a digital social credit Chinese state and private businesses operating system sits at the core of a coercive security in the country (Strumpf and Fan 2017). This apparatus that is inherently biased against points to the fact that China is not developing its certain segments of society—producing surveillance capabilities in isolation but is at the dramatically inequitable and ultimately violent forefront of a global push towards increasingly results (Human Rights Watch 2018). In an op- centralised and interconnected surveillance ed for the New York Times, James A. Millward apparatuses. Rating systems like the proposed describes the extent of the surveillance social credit system will inevitably sit at the

146 MADE IN CHINA YEARBOOK 2018 ON A CHINESE SCREEN

centre of surveillance regimes, providing the of a digital future where we are all locked in basis for how individuals and organisations are a continuous and banal system of monitoring, monitored and assessed, and what they are able accounting, categorising, and tracking—which to do (and not do) within society. has potential far reaching consequences for those who challenge the hegemony in any way, or even just for those who do not have Our Dark Digital Futures the resources or capacity to participate in the socioeconomic system on the terms mandated. Big-data driven social benchmarking sparks China’s proposed social credit system and entrepreneurs’ and politicians’ imaginations the ongoing construction of a surveillance about the opportunities lying ahead. And even state in Xinjiang represent the vanguard of though not all visions will be economically more efficient means of socioeconomic control or politically viable at any place in the world, that are being taken up around the globe. They the general trend appears to be global and are dark outgrowths of the digital revolution’s irreversible. Social credit and the dreams of supposed ‘liberation technologies’— financialised governance are not Chinese or underpinned by our very human compulsions authoritarian particularities, but are, perhaps, for transparency, security, and fairness. Credit our ‘shared destiny’ (gongtong mingyun)— systems are, of course, not new, nor are they to use a term employed by Xi Jinping when Chinese in origin. Most industrialised nations talking about the Chinese vision for the future have been relying on credit ratings for a long of humanity (Barmé, Jaivin, and Goldkorn time in order to quantify the financial risk of 2015). This, however, does not make it less, but countries, firms, and individuals (Yu et al. rather more worrisome. The logical conclusion 2015). Indeed, some of the most disturbing of society-wide financialisation is the blurring aspects of Chinese social credit, such as of the border between political and commercial its integration into social media, are not realms, and the sharpening of the repressive uniquely or originally Chinese. In the United tools wielded by the rich and powerful. In States, Affirm, a San Francisco-based lender China this scenario appears to be inevitable. headed by PayPal cofounder Max Levchin, To quote Lucy Peng, the chief executive of Ant has been experimenting with social media financial, ‘[Sesame Credit] will ensure that the data to evaluate the credit risk of car buyers bad people in society don’t have a place to go, since 2013. And Lenddo, a Hong Kong-based while good people can move freely and without company, took an even bolder approach and obstruction’ (Hvistendahl 2017). ■ informed debtors’ friends on Facebook when they did not pay instalments in time. Even the Orwellian nightmare unfolding in Xinjiang has its parallels elsewhere, such as with the recent revelations that in the United States, the New Orleans Police Department and Immigration and Customs Enforcement have been working with Peter Thiel’s company Palantir Technologies (which also has connections with the CIA and the Pentagon) to experiment with ‘predictive policing’ based on data collected from police databases, social media, and elsewhere (Fang 2018; Winston 2018). Taken together, these developments reveal a vision

MADE IN CHINA YEARBOOK 2018 147 This text is taken from Dog Days: A Year of Chinese Labour, Civil Society, and Rights, Made in China Yearbook 2018, edited by Ivan Franceschini and Nicholas Loubere, published 2019 by ANU Press, The Australian National University, Canberra, Australia.

doi.org/10.22459/MIC.04.2019.22