Neural Networks, Biometrics and the New Artificial Intelligence Mathias Fuchs & Ramón Reichert 5

ISBN: 2838

Digital

Vol. 4, Issue 1/2018

Ramón Reichert, Mathias Fuchs, Pablo Abend, Annika Richterich, Karin Wenz (eds.) Rethinking AI: Neural Networks, Biometrics and the New Artifical Intelligence Ramón Reichert, Annika Richterich (eds.) Digital Material/ism

The journal is edited by Annika Richterich, Karin Wenz, Pablo Abend, Mathias Fuchs, Ramón Reichert

Editorial Board Maria Bakardjieva, Brian Beaton, David Berry, Jean Burgess, Mark Coté, Colin Cremin, Sean Cubitt, Mark Deuze, José van Dijck, Delia Dumitrica, Astrid Ensslin, Sonia Fizek, Federica Frabetti, Richard A. Grusin, Orit Halpern, Irina Kaldrack, Wendy Hui Kyong Chun, Denisa Kera, Lev Manovich, Janet H. Murray, Jussi Parikka, Lisa Parks, Christiane Paul, Dominic Pettman, Rita Raley, Richard Rogers, Julian Rohrhuber, Marie-Laure Ryan, Mirko Tobias Schäfer, Jens Schröter, Trebor Scholz, Tamar Sharon, Roberto Simanowski, Nathaniel Tkacz, Nanna Verhoeff, Geoffrey Winthrop-Young, Sally Wyatt

Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available on the Internet at http://dnb.d-nb.de

All rights reserved. No part of this book may be reprinted or reproduced or utilized in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publisher.

Cover layout: Kordula Röckenhaus, Bielefeld Typeset: Michael Rauscher, Bielefeld

ISSN 2364-2114 eISSN 2364-2122 Print-ISBN 978-3-8376-4266-7 PDF-ISBN 978-3-8394-4266-1 Content

Introduction Rethinking AI. Neural Networks, Biometrics and the New Artificial Intelligence Mathias Fuchs & Ramón Reichert 5

I Methodological and Conceptual Reflections

Can We Think Without Categories? Lev Manovich 17

Secret Agents A Psychoanalytic Critique of Artificial Intelligence and Machine Learning Clemens Apprich 29

Voices from the Uncanny Valley How Robots and Artificial Intelligences Talk Back to Us Tiina Männistö-Funk & Tanja Sihvonen 45

II Epistemologies and Media Genealogy

Educational AI A Critical Exploration of Layers of Production and Productivity Franz Krämer 67

Competing Visions for AI Turing, Licklider and Generative Literature Oscar Schwartz 87

Pervasive Intelligence The Tempo-Spatiality of Drone Swarms Sebastian Vehlken 107

Where the Sun never Shines Emerging Paradigms of Post-enlightened Cognition Johannes Bruder 133 III Politics and Media Research

The Coming Political Challenges of Artificial Intelligence Benjamin Gregg 157

On the Media-political Dimension of Artificial Intelligence Deep Learning as a Black Box and OpenAI Andreas Sudmann 181

Automated State of Play Rethinking Anthropocentric Rules of the Game Sonia Fizek 201

IV Entering the Field

Visual Tactics Toward an Ethical Debugging Catherine Griffiths 217

Unconventional Classifiers and Anti-social Machine Intelligences Artists Creating Spaces of Contestation and Sensibilities of Difference Across Human-Machine Networks Monica Monin 227

Biographical Notes 239 Introduction Rethinking AI. Neural Networks, Biometrics and the New Artificial Intelligence

Mathias Fuchs & Ramón Reichert

Recently, the long-standing research tradition of Artificial Intelligence has undergone a far-reaching re-evaluation. When Herbert Simon in 1956 announced in one of his classes that “[…] over Christmas Allen Newell and I invented a thinking machine” (Gardner 1985: 146) the pioneers of Artificial Intelligence overstated the possibilities of algorithmic problem solving and they underestimated the pragmatic potential of it. They overrated the power of their program by proposing that human thinking can be performed algorithmically and they underestimated it by not being able to foresee what machine learning algorithms would be able to accomplish some 60 years later. Simon and Newell’s “thinking machine” was the Logic Theorist, a programme that could create logical statements by combining any out of five logical axioms. This was a scientific sensation in the 1950s and was celebrated as evidence-by-machine of Alfred North Whitehead and Bertrand Russel’s theoretical exposition in the Principia Mathematica. (Russel and Whitehead 1910) Russel and Whitehead demonstrated in an intelligent way that logical theorems could be deduced in an entirely formal manner, i. e. without creative intelligence. Raymond Fancher reports that Russel later admitted, that one of the machine deductions was “more elegant and efficient than his own”. (Fancher 1979) Today we have arrived at a state of computational power, that makes automated problem solving of many tasks more efficient than the ones under human conduct. “Elegance” however, seems not to be an issue any longer. The “Winter of Artificial Intelligence” that started in the late 1970s (Crevier 1993: 203) pointed out that machines can perform various tasks that look like human thinking, but that an artificial intelligence that thinks in the way humans think is an impossible thing to accomplish. “Human-level artificial Intelligence? Be serious!” was Nils Nilsson’s title for an important paper from 2003. Google Assistant, Deep Face, Alexa and Siri all work, and they definitely work well, but they do not tell us how humans think. The current interest in modelling information processing, closely related to the automation of quasi-intelligent behaviour and machine-based learning, is now permeating numerous research and development areas. This affects a wide range of scientific applications, which are highly affected by the regulation and analysis of complex processes. Thus, the interest for artificial neural networks is especially pronounced in fields of practice in which there is little explicit knowledge about the object area to be investigated. This concerns,

DOI 10.14361/dcs-2018-0102 DCS | Digital Culture and Society | Vol. 4, Issue 1 | © transcript 2018 6 Mathias Fuchs & Ramón Reichert

for example, the systematic recognition and processing of patterns (texts, images) which are to be controlled by means of deep learning. The methods of neuroinformatics change large areas of scientific knowledge and have great influence on the knowledge fields of prognosis and diagnostics, classification, simulation and modelling, time series analysis, language development, image processing and pattern recognition. The continuing upswing in life sciences, especially neurobiology and brain research, has led to neuroinformatics being instrumental in the development of automated infrastructures (Internet of Things), intelligent sensor networks (Sensor Information Technology) and learning environments (Deep Learning) and thus exerts a major influence on the digital society, its culture and its social practice. (Shaviro 2014) This has an enormous effect on intelligent sensor networks (robotics and sensor information technology), on learning environments (deep learning) and on digital cultures as a whole but even on the scale of special- ised methods and practices like technical computing, decision theory, computer vision, semantic networks, linguistics, multi-sensor technology, knowledge-based systems, automated deduction and reasoning, machine learning, robotics and planning. Theoretical approaches that have been suggested to accomplish this redefine such basic and fundamentally vital operations asdecision making, sensing, network control and agency. An important question for the purpose of this issue is: In which ways are the recent trends of AI deconstructing the limits of the human? (cf. Hayles 2014: 199–210) In a specific sense, neuroinformatics is of great importance for technical information processing and artificial intelligence and influences large areas of sensory and cognitive data modelling and processing, among others. (Halpern 2014) Within the fields of sensor technology (seeing), semantics and linguistics (language), robotics (manipulation of movement and behaviour) and cognitive science (learning) the (self-)learning algorithms and predictive models of neural information processing not only create epistemic frameworks for the design of multi-agent environments for communication, knowledge transfer, and education, but also create forms of knowledge and power of machine-based intelligence that enable new spaces of action for political and human resources economic processes and decisions. (Keedwell/Narayanan 2005) The New Artificial Intelligence movement has abandoned the cognitivist perspective and now instead relies on the premise that intelligent behaviour should be analysed using synthetically produced equipment and control architectures. (cf. Munakata 2008) In order to explore artificial intelligence, today’s researchers build robots that are supposed to show self-regulatory, learning behaviour within complex, and also learning, systems. (cf. Wenger 2014) Using (complete) auton- omous agents, New Artifical Intelligence examines certain issues and basic concepts such as “self-sufficiency”, “autonomy and situatedness”, “embodiment”, “adaptivity” and “ecological niches and universality” (cf. Pfeifer/Scheier 1999) involving vast areas of human and social sciences. On the other hand, New AI Introduction 7 methods such as machine learning are employed in producing prognostic information on prospective behavioural patterns of web users: “A study publishing this month used machine learning to predict with 80 to 90 percent accuracy whether or not someone will attempt suicide, as far off as two years in the future. Using anonymized electronic health records from two million patients in Tennessee, researchers at Florida State University trained algorithms to learn which combination of factors, from pain medication prescriptions to number of ER visits each year, best predicted an attempt on one’s own life.” (Molten 2017) Learning algorithms and predictive models create new epistemic conditions for digital biometrics while at the same time also opening spaces of agency for political and economic processes and decisions, which we intend to subject to close scrutiny as part of our special issue. (cf. Stewart 2014: 67–99; Neidich 2014: 264–286) This issue will also apply a historical perspective through analysing the past occurrences of AI discourses. The meaning of AI has undergone drastic changes during the last 60 years of international cooperation on research and development activities in the domain of intelligent technologies and infrastructures. What we talk about when saying AI is not what it meant in 1958, when John McCarthy, Marvin Minsky and their colleagues started using the term. Biological information processing is now firmly embedded in commercial applications like the intelligent personal Google Assistant, Facebook’s facial recognition algorithm, Deep Face, Amazon’s device Alexa or Apple’s software feature Siri to mention just a few. (Manning 2015: 701–707) In this context, the research design of Artificial Intelligence has changed signif- icantly. Today, for example, Natural Language Processing (NLP) researchers operate on large portfolios of sample data and collaborate with the research department of the online search engine provider Google. The researchers of the Artificial Neural Networks (ANN) use the huge digitized textbooks available online (known as corpora) to statistically analyse linguistic conventions using big data. The “New AI” is no longer concerned with the needs to observe the congruencies or limita- tions of being compatible with the biological nature of human intelligence: “Old AI crucially depended on the functionalist assumption that intelligent systems, brains or computers, carry out some Turing-equivalent serial symbol processing, and that the symbols processed are a representation of the field of action of that system.” (Pickering 1993, 126) Artificial intelligence research has been commonly conceptualised as an attempt to reduce the complexity of human thinking. (cf. Varela 1988: 359–75) The idea was to map the human brain onto a machine for symbol manipulation – the computer. (Minsky 1952; Simon 1996; Hayles 1999) Already in the early days of what we now call “AI research” McCulloch and Pitts commented on human intelligence and proposed in 1943 that the networking of neurons could be used for pattern recognition purposes (McCulloch/Pitts 1943). Trying to implement cerebral processes on digital computers was the method of choice for the pioneers of artificial intelligence research. 8 Mathias Fuchs & Ramón Reichert

The ecological approach of the New AI has its greatest impact by showing how it is possible “to learn to recognize objects and events without having any formal representation of them stored within the system.” (Pickering 1993, 127) The New Artificial Intelligence movement has abandoned the cognitivist perspective and now instead relies on the premise that intelligent behaviour should be analysed using synthetically produced equipment and control architectures (cf. Munakata 2008). Kate Crawford (Microsoft Research) has recently warned against the impact that current AI research might have, in a noteworthy lecture titled: AI and the Rise of Fascism. Crawford analysed the risks and potential of AI research and asked for a critical approach in regard to new forms of data-driven governmentality: “Just as we are reaching a crucial inflection point in the deployment of AI into everyday life, we are seeing the rise of white nationalism and right-wing authori- tarianism in Europe, the US and beyond. How do we protect our communities – and particularly already vulnerable and marginalized groups – from the potential uses of these systems for surveillance, harassment, detainment or deportation?” (Crawford 2017) Following Crawford’s critical assessment, this issue of the Digital Culture & Society journal deals with the impact of AI in knowledge areas such as computational technology, social sciences, philosophy, game studies and the humanities in general. It can be assumed that the large-data research in the fields of advanced technology, humanities and social sciences is responsible for the reorganization of power relations in the digital society. Subdisciplines of traditional computer sciences, in particular Artificial Intel- ligence, Neuroinformatics, Evolutionary Computation, Robotics and Computer Vision once more gain attention. Biological information processing is firmly embedded in commercial applications like the intelligent personal Google Assistant, Face- book’s facial recognition algorithm, Deep Face, Amazon’s device Alexa or Apple’s software feature Siri (a speech interpretation and recognition interface) to mention just a few. In 2016 Google, Facebook, Amazon, IBM and Microsoft founded what they call a Partnership on AI. (Hern 2016) This indicates a move from academic research institutions to company research clusters. In this context we were interested in receiving contributions on the aspects of the history of institutional and private research in AI. We invited articles that observe the history of the notion of “artificial intelligence” and articles that are able to point out how specific academic and commercial fields (e. g. game design, aviation industry, transport industry etc.) interpret and use the notion of AI. The work on Artificial Intelligence is expected to yield better results than competing models of computer science and computational neuroscience in the challenging applications of predictive modelling of knowledge, classifying pattern recognition, and fault tolerant learning. Artificial neural networks are used in applied computer science and mathematics because they allow alternative formal- izations of computability. Today, neural networks are used for optical (Image Clas- Introduction 9 sification) and acoustic (Speech Recognition) pattern recognition and in robotics. In the military field, neural networks are used in automatic image analysis for target recognition. Against the background of these trends and upheavals, we not only question the cultural, social and political significance of today’s dominant biological information processing, but also the historical dimension of the mutual influence of biology, cybernetics and early computer science since the mid-20th century training of research centres and networks. (Hauptmann/Neidich 2010) Against this background, the special issue Rethinking AI explores and critically reflects the hype of neuroinformatics in AI discourses and the potential and limits of critique in the age of computational intelligence. (Johnston 2008) Digital societies increasingly depend on smart learning environments that are technologi- cally inscribed. Our special issue is asking for the role and value of open processes in learning environments. Therefore, we invited contributions that are historical and comparative or critically reflective about the biological impact on social processes, individual behaviour and technical infrastructure in a post-digital and post-human environment (e. g. artificial neural networks, fuzzy systems, genetic algorithms, evolutionary computation, deep learning, prognostics and predictive modelling, computer vision). We had put together texts that discuss empirical findings from studies that approach the relationships between neurobiology, brain research, computational intelligence, biopolitics, psychological research and the new AI movement. Against this background, we (1) want to examine the scientific-historical, epistemological, and media-theoretical dimensions of New AI in order (2) to focus on their historical and performative dimensions, which provide a ground breaking foundation for understanding the basic requirements of the data society. In his contribution, Lev Manovich discusses important challenges for Cultural Analytics research. In his survey on key texts and propositions from 1830 on until the present he examines the fundamental paradigms of data visualization, unsupervised machine learning, and supervised machine learning. He wants to clarify that the observation and analysis of culture means to be able to map and measure three fundamental characteristics: diversity, structures, and dynamics. Clemens Apprich’s contribution presents the idea of a “psychoanalysis of things” and applies it to artificial intelligence and machine learning. His approach reveals some of the hidden layers within the current AI debate and hints towards a central mechanism in the psycho-economy of our socio-technological world. At a time, when algorithms, in the form of artificial neural networks, operate more and more as secret agents, the question of “Who speaks?” achieves high relevance. This question, situated at the centre of a psychoanalysis of paranoia, becomes central for the analysis of AI in Digital Cultures. Tiina Männistö-Funk & Tanja Sihvonen analyse the attempts at making speaking machines commercially successful on various occasions. They investigate how speech producing devices such as the actual digital assistants that operate our current technological systems fit into their historical context. Franz Krämer’s article is about Educational AI. It is not only governments and the traditional educational institutions, but increasingly 10 Mathias Fuchs & Ramón Reichert

also companies, especially big technology firms like Facebook, Google, Amazon or Microsoft, that appear to see unlimited potential in educational AI, and steadily elevate their sponsorship and investments. His text approaches the phenomenon from a view that sees the educational relevance of AI as rooted in AI’s characteristics as software. Software can be described – as another author of this Journal’s issue has famously stated - as “our interface to the world, to others, to our memory and our imagination – a universal engine on which the world runs” (Manovich 2013) AI systems and the notion of educationally applied AI can be viewed as a result of social, economic, political and cultural production processes, including practices, structures and discourses interlinking education, educational technology and the notion of AI. Oscar Schwartz investigates how two competing visions of machine intelligence put forward by Alan Turing and J. C. R. Licklider have informed experiments in computational creativity, from early attempts at computer-generated art and poetry in the 1960s, up to recent experiments that utilise Machine Learning to generate paintings and music. Sebastian Vehlken‘s leading hypothesis argues that Swarm Robotics create a multifold “spatial intelligence”, ranging from the dynamic morphologies of such collectives via their robust self-organization in changing environments to representations of these environments as distributed 4D-sensor systems. Johannes Bruder elaborates on delibera- tions of “post-enlightened cognition”. His article explores links between cognitive neuroscience, psychology and artificial intelligence research. Bruder demon- strates how the design of machine learning algorithms is entangled with research on creativity and pathology in cognitive neuroscience and psychology through an interest in “episodic memory” and various forms of “spontaneous thought”. Spon- taneous thought, long time stigmatised as a sign of distraction or even potentially pathological, achieves a new valuation. Recent research in cognitive neuroscience conceptualises spontaneous thought as serving the purpose of creative problem solving and therefore builds upon earlier discussions around the links between creativity and pathology. Benjamin Gregg starts from the worrying observation that there is no consensually held scientific understanding of intelligence. He states that the term “intelligence” is no less indeterminate in the sphere of artificial intelligence than in general theoretical thought. This is obviously no hindrance to using the term “AI”, because technical applications and biotechnical developments do not wait for scientific clarity and definitional precision. The near future, he proposes, will bring significant advances in technical and biotechnical areas, including the genetic enhancement of human intelligence (HI) as well as artificial intelligence (AI). Gregg shows how developments in both areas will challenge human communities in various ways and why the danger of AI is distinctly political. Andreas Sudmann investigates the media-political dimension of modern AI technology. The main focus of his paper is centred around the political implica- tions of AI’s technological infrastructure, especially with regard to the machine learning approach that since around 2006 has been called Deep Learning (also known as the simulation of Artificial Neural Networks). Sonia Fizek’s contribution Introduction 11 is about the automated state of play. She attempts to review and critically rethink anthropocentric rules of games. Two aspects guide her investigation: Firstly, she is interested in self-playing game worlds, self-acting characters and non-human agents traversing multiplayer spaces. These automated actors and environments are more than a merely technological enhancement of gaming. Secondly, based on a decentralised post-humanist reading, we might have to rethink digital games and play. This serves also for a critical reflection of games AI, which due to the fictional character of video games, often plays by very different rules than the so-called “true” AI. Catherine Griffiths writes about the visual tactics toward an “ethical debugging”. Her critical study of artificially intelligent algorithms, strate- gies from the fields of critical code studies and data visualisation are combined to propose a methodology for computational visualisation. She argues that computational visualisation seeks to elucidate the complexity and obfuscation at the heart of artificial intelligence systems. This is the source for ethical dilemmas, that are a consequence of the use of machine learning algorithms in socially sensitive spaces, such as in determining criminal sentencing, job performance, or access to welfare. The paper of Monica Monin is concerned with the ways in which present day artists are engaging with artificial intelligence, specifically material practices that endeavour to use these technologies and their potential non-human agencies as collaborators with differential objectives to commercial fields.

References

Al-Jobouri, Laith (2017): CEEC 2017, 2017 9th Computer Science and Electronic Engineering Conference, https://www.aconf.org/conf_104832.html. Barad, Karen (2007): Meeting the Universe Halfway. Quantum Physics and the Entanglement of Matter and Meaning. Durham/London: Duke University Press. Crevier, Daniel (1993): AI: The Tumultuous Search for Artificial Intelligence, New York, NY: BasicBooks. McCulloch, Warren and Walter Pitts (1943): A Logical Calculus of the Ideas Imma- nent in Nervous Activity. In: Bulletin of Mathematical Biophysics, Vol. 5, pp. 115–133. Fancher, Raymond E. (1979): Pionieers of Psychology. New York/London: WW Norton & Co. Gardner, Howard (1985): The Mind’s New Science: A History of the Cognitive Rev- olution. New York: Basic Books. Halpern, Orit (2014): Beautiful Data: A History of Vision and Reason since 1945. (Experimental Futures.) Durham, N. C.: Duke University Press. Hauptmann, Deborah and Warren Neidich (2010): Cognitive Architecture. From Bio-politics to Noo-politics. Rotterdam: 010 Publishers. 12 Mathias Fuchs & Ramón Reichert

Hayles, Katherine N. (2014): Cognition Everywhere: The Rise of the Cognitive Nonconscious and the Costs of Consciousness. In: New Literary History 45 (2): pp. 199–220. Hayles, Katherine N. (1999): How We Became Posthuman. Virtual Bodies in Cybernetics, Literature, and Informatics, Chicago/London. Hern, Alex (2016): “Partnership on AI’ formed by Google, Facebook, Amazon, IBM and Microsoft | Technology” In: The Guardian, online, https://www.theguard ian.com/technology/2016/sep/28/google-facebook-amazon-ibm-microsoft- partnership-on-ai-tech-firms. Johnston, John (2008): The Allure of Machinic Life: Cybernetics, Artificial Life, and the New AI. Cambridge, MA and London, UK: The MIT Press. Keedwell, Edward and Ajit Narayanan (2005): Intelligent Bioinformatics: The Application of Artificial Intelligence Techniques to Bioinformatic Problems. Chichester, UK: John Wiley and Sons, Ltd. Manning, Christopher D. (2015): Computational Linguistics and Deep Learning Computational Linguistics, 41 (4), pp. 701–707. Manovich, Lev (2013): Software Takes Command. New York, London: Bloomsbury Academic, https://issuu.com/bloomsburypublishing/docs/9781623566722_web. Minsky, Marvin (1952): A neural-analogue calculator based upon a probability model of reinforcement. Harvard University Psychological Laboratories inter- nal report. Molten, Megan (2017): Artificial Intelligence Is Learning to Predict and Prevent Suicide. In: Wired, online, https://www.wired.com/2017/03/artificial-intelli gence-learning-predict-prevent-suicide/. Munakata, Toshinori (2008): Fundamentals of the New Artificial Intelligence. Neural, Evolutionary, Fuzzy and More, New York: Springer. Neidich, Warren (2014): “The Architectonics of the Mind’s Eye in the Age of Cog- nitive Capitalism.” Brain Theory. Palgrave Macmillan UK, pp. 264–286. Nilsson, Nils (2005): “Human-level artificial intelligence? Be serious!” in AIMag 26 (4). Pfeifer, Rolf and Scheier, Christian (1999): Understanding Intelligence. Cam- bridge, MA: The MIT Press. Pickering, John (1993): “The New Artificial Intelligence and Biological Plausibil- ity”, in: Valenti, Stavros and John Pittenger (eds.), Studies in Perception and Action II, London/New York: Psychology Press, pp. 126–129. Russell, Bertrand and Alfred North Whitehead (1962 [1910–1913]): Principia Math- ematica. Cambridge University Press. Russell, Stuart J. and Peter Norvig (2003): Artificial Intelligence: A Modern Approach (2nd ed.), Upper Saddle River, New Jersey: Prentice Hall. Shaviro, Steven (2014): The Universe of Things: On Speculative Realism. Univer- sity of Minnesota Press. Simon, Herbert (1996): The Sciences of the Artificial. Cambridge, MA: The MIT Press. Introduction 13

Stewart, Patrick (2014): “Introduction to Methodological Issues in Biopolitics”, in: Politics and the Life Sciences: The State of the Discipline. Emerald Group Pub- lishing Limited, pp. 67–99. Varela, Francisco J. et al. (1988): Cognitive networks: Immune, neural, and other- wise, in: Alan Perelson: Theoretical Immunology, Redwood City, pp. 359–75. Wenger, Etienne (2014): Artificial intelligence and tutoring systems: computational and cognitive approaches to the communication of knowledge. Morgan Kaufmann.

Methodological and I Conceptual Reflections

Can We Think Without Categories?

Lev Manovich

Abstract In this article methods developed for the purpose of what I call “Media Analytics” are contextualized, put into a historical framework and discussed in regard to their relevance for “Cultural Analytics”. Large- scale analysis of media and interactions enable NGOs, small and big businesses, scientific research and civic media to create insight and information on various cultural phenomena. They provide quantitative analytical data about aspects of digital culture and are instrumental in designing procedural components for digital applications such as search, recommendations, and contextual advertising. A survey on key texts and propositions from 1830 on until the present sketches the development of “Data Society’s Mind”. I propose that even though Cultural Analytics research uses dozens of algorithms, behind them there is a small number of fundamental paradigms. We can think them as types of data society’s and AI society’s cognition. The three most general paradigmatic approaches are data visualization, unsupervised machine learning, and supervised machine learning. I will discuss important challenges for Cultural Analytics research. Now that we have very large cultural data available, and our computers can do complex analysis quite quickly, how shall we look at culture? Do we only use computational methods to provide better answers to questions already established in the 19th and 20th century humanities paradigms, or do these methods allow fundamentally different new concepts?

Media Analytics and Cultural Analytics

Since the middle of the 2000s, global digital culture has entered a new stage that I call “media analytics.” (Manovich 2018: 473–488) Computational analysis of massive numbers of cultural artefacts, their online “lives,” and people’s interactions with these artefacts and each other has redefined dynamics and mechanisms of culture. Such analysis is now used by numerous players – the companies who run social networks, NGOs planning their outreach, millions of small businesses around the world advertising online, or millions of people who are using social media and web analytics tools and dashboards to refine their online posts and self-presentation and understand their social and professional value. For example, I am using Google Analytics

to understand how visitors interact with all pages on my websites. I can also look up my score on academia.edu, computed by comparing popularity of my publications on this network with publications of millions of other academics and students. This large-scale analysis of media and interactions also enables other key components of digital culture such as search, recommendations, and contextual advertising. For example, to make its search service possible, Google continuously analyses full content and mark-up of billions of Web pages. It looks at every page on the Web that its spiders can reach – its text, layout, fonts used, images, and so on, using over 200 signals in total (Google, 2016a). E-mail spam detection relies on analysis of texts of numerous e-mails. Amazon analyses purchases of millions of its customers to recommend books. Netflix analyses choices of millions of subscribers to recommend films and TV shows. It also analyses information on all its offerings to create more than 70,000 genre categories (Madrigal, 2014). Contextual advertising systems such as AdSense analyse content of Web pages and automatically select the relevant ads to show. Video game companies capture gaming actions of millions of players and use this to optimize game design. Other examples include automatic translation and recommendations for people to follow or add to your friends list on social networks). The same core algorithms used in the industry also make possible new research about cultures and societies in fields that include computer science, data science, computational social science, digital humanities, urban studies, media studies, data visualization, and data design. Since the research that uses large cultural datasets and data science to create, manage, and analyse them is spread between all these institutional and design disciplines, I have been using the umbrella term “Cultural Analytics” to refer to it. (Manovich 2009) Here are a few examples of Cultural Analytics research: “Toward Automated Discovery of Artistic Influence,” (Saleh 2015) “Measuring the Evolution of Contem- porary Western Popular Music,” (Serrà 2012) and “Shot durations, shot classes, and the increased pace of popular movies.” (Cutting/Candan 2015: 40–62) The first paper presents a mathematical model for automatic discovery of influence between artists. It then tests it using images of 1710 well-known paintings created by 66 well-known artists over a number of centuries. While some of the discov- ered influences are the same ones often described by art historians, the model also suggested other visual influences between artists not discussed previously. The second paper investigates changes in popular music using a dataset of 464,411 songs from 1955 to 2010. The dataset included “a variety of popular genres, including rock, pop, hip hop, metal, or electronic.” The authors concluded that over time, there was “the restriction of pitch transitions” and “the homogeniza- tion of the timbral palette” – in other words, some of the musical variability has decreased. The third paper analyses gradual changes in average shot duration across 9400 English-language narrative films created during 1912–2013. In this article I will discuss a few general challenges for Cultural Analytics research. Now that we have very large cultural data available, and our computers Can We Think Without Categories? 19 can do complex analysis quite quickly, how shall we look at culture? Do we only use computational methods to provide better answers to questions already established in the 19th and 20th century humanities paradigms, or do these methods allow fundamentally different new concepts? I think that such perspectives are necessary because contemporary culture itself is now driven by the same or similar methods. And this is the key difference between using computational methods and concepts to analyse cultural data today vs. twenty years ago. Now these methods and concepts are driving everyday digital culture lived by billions of people. When small numbers of humanists and social scientists were analysing cultural data with computers in the second part of the 20th century, their contemporary culture was mostly analogue, physical, and non-quantifiable. But today we as academic researchers live in the “shadow” of a world of social networks, recommendations, apps, and interfaces that all use media analytics. As I already explained, I see media analytics as the new stage in the development of modern technological media. This stage is characterized by algorithmic large-scale analysis of media and user interactions and the use of the results in algorithmic decision making such as contextual advertising, recommendations, search, and other kinds of information retrieval, filtering of search results and user posts, document classification, plagiarism detection, video finger- printing, content categorization of user photos, automatic news production etc. And we are still only at the beginning of this stage. Given the trajectory of gradual automation of more and more functions in modern society using algorithms, I expect that production and customization of many forms of at least “commercial culture” (characterized by conventions, genre expectations, and templates) will also be gradually automated. So, in the future already developed digital distribution platforms and media analytics will be joined by the third part: algorithmic media generation. (Of course, experimental artists, designers, composers, and filmmakers have been using algorithms to generate work since the 1960s, but in the future, this is likely to become the new norm across culture industry.) We can see this at work already today in automatically generated news stories, online content written about topics suggested by algorithms, production of some television shows, and TV broadcasts during sport events where multiple robotic cameras automatically follow and zoom into dynamic human perfor- mances. So, for instance, if we want to analyse intentions, ideology and psychology of an author of certain cultural artefact or experience, this author maybe not a human but some form of AI that uses a combination of data analysis, machine learning and algorithmic generation. Until ten years ago, key cultural techniques we used to represent and reason about the world and other humans included natural languages, lens-based photo and video imaging, various other media for preserving and accessing information, calculus, digital computers, and computer networks. The core concepts of data/AI society are now as important. They form data society’s “mind” – the particular ways of encountering, understanding, and acting on the world and the humans. And this is 20 Lev Manovich

why even if you have no intention of doing practical Cultural Analytics research yourself, you need anyway to become familiar with these new data-centred cultural techniques. (The concept of “cultural techniques” has been mostly used in recent German media theory. See Winthrop-Young/Irascu/Jussi Parikka 2013). While both media analytics in industry and Cultural Analytics research use dozens of algorithms, behind them there is a small number of fundamental paradigms. We can think them as types of data/AI society’s cognition. The three most general ones are data visualization, unsupervised machine learning, and supervised machine learning. Others are feature extraction, clustering, dimension reduction, classification, regression, network science, time series analysis, and information retrieval. (Others may have a different list depending on their field).

The Development of Data Society’s Mind: Short Timeline

Given that the terms “AI,” “machine learning” and “data science” have entered public discourse in 2010s, many people may think that these fields are very new. In fact, almost all ideas and methods used today for data analysis were developed in the 19th and 20th century. The following is my timeline of selected concepts and methods in statistics and computational data analysis. (What is included and excluded is based on my own experience with data science and is consciously biased towards Cultural Analytics research rather than all industry applications.)

The nineteenth century: 1. Normal distribution observed in physical characteristics of groups of people: Adolphe Quetelet in the early 1830s. 2. Social physics, 1835: Quetelet’s book Sur l’homme et le développement de ses fa- cultés, ou Essai de physique sociale (translated in English as Treatise on Man) outlines the idea of “social physics.” Quetelet appropriates this term from Compte who was already using it earlier. 3. Sociology, 1838: Learning that Quetelet uses his concept, Compte introduces the new concept of “sociologie.” 4. Standard deviation: Francis Galton in late 1860. 5. Correlation: Aguste Bravais in 1846; rediscovered by Galton in 1888. 6. Regression analysis: Galton in the 1880s. 7. Histogram: Pearson in 1895. 8. Regression analysis, extended and formalized – Udny Yule and Karl Pearson, 1897–1903. 9. Statistical Hypothesis Test theory foundations: Pearson in 1900. 10. Power Law, 1896: Economist Vilfredo Pareto publishes his observations of “80–20 rule” that is latter named “Pareto Law”; it exemplifies more general “power law” that describes many phenomena in network culture and social media today such as “long tail.” Can We Think Without Categories? 21

The twentieth century: 1. Principal Component Analysis: Pearson in 1901. 2. Factor analysis: Charles Spearman in 1904. 3. Multi-variable regression – first used the 1900s. 4. Markov’s Chains, 1906: Andrew Markov starts working on theory and methods for statistical analysis of time series which are later names “Markov’s chains.” 5. Network analysis, the 1930s: Jacob Moreno develops methods for representing social networks; in the 1950s measurement methods and concepts for network analysis are formalized. 6. Neural networks, 1957: Frank Rosenblatt develops neural networks for classification (“perceptrons”). This research was based on the earliest model of neural networks developed by Ukrainian-born American scientist Nicolas Ra- shevsky. His students Walter Pitts and Warren McCulloch further developed his model and published their famous 1943 article in Rashevsky’s journal. (McCulloch/Pitts 1943: 115–133) 7. “Software,” 1958: John Tukey is the first to use this term. (Tukey 1958: 1–9) 8. Multi-dimensional Scaling (MDS), the 1950s. (Torgerson 1958; Green 1975: 24–31) 9. Support Vector Machines (SVM), 1963: Vladimir Vapnik and Alexey Chervo nenkis (Soviet Union) develop the original algorithm. A version of the algorithm published by Vapnik in 1995 becomes one of the most popular algorithms for classification. 10. Deep Learning, 1965: Alexey Ivakhnenko and V. G. Lapa publish the first working algorithm for deep learning. In the late 2000s deep learning becomes most popular approach to classification. Ivakhnenko is often referred as the “father of deep learning.” 11. Vector space concept, late 1960s: developed for text comparisons by Gary Salton. 12. Exploratory data analysis, 1971: Tukey starts developing ideas of exploratory data analysis. 13. Statistical programming languages, 1976: The language called “S” is developed in Bell Labs by John Chambers and others. It makes Exploratory Data Analy- sis possible since it allows researchers to perform every kind of statistical and data analysis from a command line and make graphs. (Chambers 2016) “R” language developed later from “S.” 14. Backpropagation, 1986: This new method for teaching deep networking starts to become popular after the publication of the article by David Rumelhart, Geoffrey Hinton, and Ronald Williams inNature . (Rumelhart et al. 1986: 533) 15. PageRank, 1996: Larry Page and Sergey Brin developed the PageRank algorithm. The idea was proposed earlier and can be traced back to the work of Eu- gene Garfield, the founder of bibliometrics and scientometrics in the 1950s. (Although PageRank is not the key algorithm in data analysis, it had fundamental effect on how people interact with information and cultural content online.) 22 Lev Manovich

16. Item-item collaborative filtering, 1998: Amazon invents and starts using Item- item collaborative filtering, a recommendation algorithm that calculates simi- larity between objects (such as books) based on people’s ratings of those objects. (Sarwar 2001: 285–295)

The twenty-first century: 1. Latent Dirichlet allocation (LDA), 2003: The model for “topic modelling” in text documents called Latent Dirichlet allocation is published by David Blei, Andrew Ng, and Michael I. Jordan. J. K. Pritchard, M. Stephens, and P. Don- nelly published the original topic modelling paper in 1999. 2. Deep learning takes off, 2006: The current wave of deep learning research starts with publication “A Fast Learning Algorithm for Deep Belief Nets” by Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh. (2006: 1527–54) Com- bination of bigger data available for training, the use of computer clusters and GPUs, and sharing of datasets and code online, and competition such as ImageNet lead to very fast progress. 3. Use of deep learning for image classification, 2012: A paper “ImageNet Classi- fication with Deep Convolutional Neural Networks” by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton shows that “deep learning outperforms other methods for image classification.”

Summarizing this timeline, I can say that practically all concepts and methods relevant for computational analysis of culture on large scale were invented by already the middle of the 1970s, although their massive applications in cultural industry and in Cultural Analytics research is only about ten years old.

Do We Want to “Explain” Culture?

Approaching cultural processes and artefacts as “data” can lead us to ask the kinds of questions about culture that people who professionally write about it, curate and manage it do not normally ask today – because such questions would go against the accepted understanding of culture, creativity, aesthetics, and taste in humanities, popular media or art world. For example, would collectors and museums pay millions for the works of contemporary artists if data analysis shows that they are completely unoriginal despite their high financial valuations? Or, if data analysis reveals that trends in art world can be predicted as accurately as the ones in fashion? The most well-known and influential quantitative analysis of cultural data within social sciences remains Pierre Bourdieu’s Distinction (1979). The data used in this analysis comes from the surveys of French public. For analysis and visualization of this data, Bourdieu used recently developed method of correspon- dence analysis. It is similar to PCA but works for discrete categories, showing their Can We Think Without Categories? 23 relations in graphical form. For Bourdieu, this form of data analysis visualization went along with his theoretical concepts about society and culture, and that’s why it is plays a central role in this book. Distinction is Bourdieu most well known book, and in 2012 Bourdieu was the second most quoted author in the world in academic publications, just behind Michel Foucault. (Truong/Weill 2012) Bourdieu did not use the most common method of quantitative social science – “explaining” some observed phenomena using mathematic models such as linear regression. However, given the variety and the scale of cultural data available today, maybe today such method can produce interesting results? What would happen if we also take other standard methods of quantitative social science and use them to “explain” the seemingly elusive, subjective, and irrational world of culture? For example, we can use factor analysis to analyse choices and preferences of local audiences around the world for music videos from many countries to understand the dimensions people use to compare musicians and songs. Or we can use regression analysis and combination of demographic, social, and economic variables to model choices made by “cultural omnivores” – people who like cultural offerings associated with both elite and popular taste. (Peterson 1992: 243–258) In quantitative marketing and advertising research, investigators ask similar questions all the time in relation to consumer goods and cultural artefacts. And computer scientists also do this when they analyse social media and web data. But this does not happen in humanities. In fact, if you are in the arts or humanities, such ideas may make you feel really uncomfortable. And this is precisely why we should explore them. The point of any application of quantitative or computational methods to analysis of culture is not whether it ends up being successful or not (unless you are in media analytics business). It can force us to look at the subject matter in new ways, to become explicit about our assumptions, and to precisely define our concepts and the dimensions we want to study. So at least as a thought experiment, let’s apply quantitative social science paradigm to culture. Quantitative social science aims to provide “explanations” of social phenomena expressed as mathematical relations between small numbers of variables (what influences what and by how much). Once such models are created, they are often used for prediction. The common statistical methods for such “explanations” are regression models, versions of factor analysis or fitting a probability distribution to the data. The latter means determining if observed data can be described using a simple mathematic model, e. g., Gaussian distribution, log-normal distribution, the Paretto distribution etc. (In quantitative film studies, a number of researchers found that shot frequencies in the twentieth-century Hollywood films follow a log-normal distribution. See, for example, DeLong 2015: 129–36) Are we interested in trying to explain what influences what in culture or predicting its future with mathematic models? Do we need to explain culture