Is a picture worth a thousand words?
“On written text and the (moving) image, their relationship within Facebook’s interface and the possible humanistic consequences”
Nena Snoeren
University of Amsterdam
MA Media Studies: New Media and Digital Culture
28th of June 2019
2 Index Page
Abstract 4
1. Introduction 5
2. Text/Image 8
2.1 On text 9
2.2 On image 11
2.3 The relationship and the divide 13
3. Text/Image within new media 17
3.1 Affordances 21
3.2 Methodology 24
3.3 Text/Image within Facebook’s interface 26
4. Case studies 27
4.1 Case study #1: The status update 28
4.2 Case study #2: Emoji, symbols, stickers and GIFs 39
4.3 Case study #3: The person 47
5. Discussion 53
6. Conclusion 58
7. References 61
3 Abstract
Our media world is made up out of two visual components: images and written text. Together they form a big part of our understanding and online communication. In the recent past images have become significantly easier to produce and share. This has had its effect on the position of written text in media and communication, that now operates within the ‘logic of the image’. In this work, the relationship of written text and images will be explored through a historical and philosophical overview of these two concepts. Furthermore, Facebook’s interface will be analysed through the ‘discursive interface analysis’, focussing on three case studies: ‘the status update’, ‘emoji, symbols, stickers and GIFs’ and ‘the person’. These analyses will be conducted from a contemporary historical perspective, mapping the changes in Facebook’s interface updates in relation to written text and images over time. Considering the concept of productive power, the features are researched through the concept of affordances, seeing how certain changes in features produce a specific normativity towards images and text. To conclude, the image-shift that becomes apparent is analysed from a humanistic perspective, discussing the possible humanistic consequences of the change in the text/image relationship.
4 1. Introduction
Does the title page of this work feel a bit empty to you? Would it look more appealing with an image on it, or a bigger title? You are reading this work now, but if it were laying on a table amongst a few other papers with images on them, would you feel drawn to reading this one? The answer is probably: no. From my personal and academic experience, it has been my observation that the world around us, and new and social media in particular, have become more (moving) image orientated. Our technologies have increasingly become compatible with (moving) images in the last -and even more so in this- century. Moving images have migrated from living in cinemas to living in our homes on the television, then on the computer and now in our hands via our smartphones. Cinema and television were made to show the moving image, initially the computer and the telephone were not. Computer to computer communications were written-text based and phone communications verbal-text based. As computer technologies advanced, the image and later the moving image (first in the GIF format) made its way onto the internet, onto our home computers and not much later onto our internet connected phones. Now our timelines, news feeds and digital communications have shown a shift from predominantly textual information towards images, symbols, videos, GIFs and emoji. Not only has the image found its way on the text-based medium of the internet, it is now not merely being supported, but it has been thoroughly integrated into its structures. Modern social media platforms are specifically built to support and stimulate the use and viewing of (moving) images. User posts that contain images or moving images are far more visible than those only containing text. Newer popular social media platforms like Instagram and Snapchat have even kept the use of written text to a minimum, making images the main form of content and communication from the start. Academic research into written text and images has been conducted by many fields, including psychology, philosophy, linguistics, communication, television studies, film studies and new media studies. Research into internet platforms, social media and web interfaces has been done extensively within new media studies, focussing on the content and structure of certain online platforms and their interfaces. A combination of research into written text and images as ontological research subjects, within an online platform, however, has not been conducted. I aim to make my work here a contribution to platform studies within new
5 media studies, in this new but necessary way. Research into digital platforms and their interfaces are highly relevant at this time, seeing their immense user base and thus their far-reaching influence and power. Research into the interfaces of the most popular social media platforms discussing the state of text and image is essential as they are the cornerstones of modern human culture, understanding and communication. This work will therefore be focussing on the changing relationship between written text and the (moving) image on the most popular social media platform until this day: Facebook. Neil Postman talks about not seeing the point of studying media unless doing so in a moral or ethical context in “The Humanism of Media Ecology”, the keynote address at the Inaugural Media Ecology Association Convention in 2000. Other well-known media scholars like Marshall McLuhan did not agree with this idea and thought one ought to stay ‘neutral’ (Postman 11). I agree with Postman in the sense that one can and should reflect on media within a moral context. We can reflect, analyse and dissect what certain media are, how they function and how they work, and then? What deeper meaning and possible consequences does their influence have? These, I believe, are the important things to consider. Humanism concerns itself with the ethical consequences of human actions and decisions. A humanistic approach thus evaluates the morality behind certain matters. I want to touch upon this humanistic point of view in relation to the use of written text and images within new media in this work. If we are looking at a platform that currently has 2.3 billion active users, facilitating a practice at the core of our human experience: communicating with one another through text and images, the humanistic consequences of this practise must be reflected upon. This is not to say that my research here will hold all the answers to what is ‘good’ or ‘bad’ for us humans about the changing text/image relationship on Facebook, rather it is to be seen as a (limited) exploration into the humanistic issues this changing relationship could be bringing about. Therefore, the research question of this work is as follows: “What has the relationship been between written text and the (moving) image within Facebook’s interface structure and what could this mean in terms of humanistic consequences?” Firstly, to lay a good foundation for a discussion on changing text/image relationships within a new media platform, I will be exploring what written text, images and their relationship are in the first section of this work. I will use a multi-disciplinary approach
6 for this as to explore the text/image question from multiple perspectives in relation to new media. I believe such a multi-disciplinary approach is necessary in order to rightfully address the diverse nature of these research subjects. In order to research what the ontology of written text and (moving) images are, academic works from the disciplines of psychology, communication, linguistics, anthropology, film studies and new media studies are used. Following, I will use certain main features of Facebook’s interface as case studies to analyse the specific text/image relationships over time. The case studies will be: ‘The status update’, ‘Emoji, symbols, stickers and GIFs’ and ‘The person’. These contemporary historical analyses will be conducted through the ‘discursive interface analysis’ to thoroughly research what changes occurred to the interface structure at what time and what this signifies in terms of normativity. The concept of affordances and productive power will provide helpful here. Conducting these analyses on Facebook’s interface will serve as an example for the greater changes that are happening within the (new) media landscape. A discussion of the above sections will follow, exploring the deeper implications of the changes analysed in the case studies in connection with the first section on written text and images. This is the place where the humanistic context will come forward and possible moral consequences will be explored.
Image 0. Picture dictionary point it
7 2. Text/Image
Traveling without the internet, some people used -and maybe still use- a little picture dictionary called point it, for basic communication with people that do not speak their language. Obviously, more complex communication was not possible with this little book, but with the internet not being around and/or accessible yet, it made travelling a lot easier. From a text-image relationship perspective, this is the idea of a picture dictionary (as point it calls itself on its cover). It is using a set of images to function as a type of language, organised by its ‘universal’ uses and associations independent of any written language. A dictionary, both in definition and etymology, has a strong connection to words and written language. In the case of point it one could thus argue a picture dictionary is a contradictio in terminis; as a picture or an image is not a word or language but something different, yet connected. We all know that written text and images are different from one another, at least, they are to us humans. We read text and we see or read images in a different way. Written text is language and to be able to ‘make sense’ to us it works following reasonably strict rules. We must learn to understand the system of language as it is not innate to us. Almost all of us learn to speak, most to read and write too. We are highly verbal beings. Not only are we very verbal beings, we are visual beings too. We have a lively imagination and produce and share images like never before. Technological development has made it easier to communicate with images. It has never been as easy for so many people to participate in visual communication. The creation of images (especially photography and video) is done with hardly any effort in contrast to a hundred years ago. One could image the number of images (let alone moving images) an average person was exposed to daily in 1919 has increased significantly over a hundred years. So, we like communicating with images, a lot. It seems the more it is technologically possible to communicate with images, the more we like to do it, especially within new media. What does this mean for written text within new media? Is it being replaced in some form or another? Or are images only a welcome addition to textual communication? To make any kind of analysis into this direction, we first need to lay out what written text and images are. How they are different and how they might be the same. In the next section, I will lay out a discussion of the ontology of written text and the image.
8 2.1 On text
The first stories we will be exposed to in our lifetime will be either visual (picture books) or verbal, as we cannot properly read until we reach the age of about six or seven years old. You probably do not remember anymore what it felt like to slowly be able to make more sense of this highly textual world. One could also take a trip to China to experience this feeling of lostness (with the obvious exception of the ones able to read Chinese). But what exactly is text? A text can be written or spoken. You can hear a text, or read one like you are doing now, unless someone is reading this to you. When written text is spoken about in this work, it is referring to language that is written down. To the writings of spoken languages specifically. So the written down versions of other types of ‘languages’ that are not spoken by humans (today) such as Morse code, hieroglyphs or hypertext will not be part of this discussion. Although language has a lot of ways of expression other than written text (verbal, sign language, Braille), the focus will be on written text as this form of language is often used in combination with images and this relationship is the subject of my study. I will not be researching the spoken word or spoken text in this work, as it would fall outside of the possible scope of this particular (limited) research into image-text relations. I would also like to point out that the ‘image-as-text strategy’ will not be used. This idea was developed during the ‘linguistic turn’ by (language) philosopher Ludwig Wittgenstein, and later on by others. In this trend, language was seen as constructing reality and everything could be considered a text of some sort (Bateman 13). Following this idea, one could read a painting, a film or even a building as being a form of text. In the field of semiotics, for example, this idea of text went quite far, making a text anything that can be studied with the techniques of the semiotic system (Bateman 13). Making images a part of what is considered a text, would make my research into image-text relations rather problematic. Written text knows many forms or systems of writing, this work is written in the Latin script. Unfortunately, I cannot read or write other forms of writing than the Latin script, therefore this research will be into the ontology of written text in the Latin script only. Undoubtedly, reading from right to left provides a (small) difference in the readers/users experience in relation to nearby images, but this concept will thus not be considered in this work. As written text is language, it now makes sense to look into the field of linguistics to further our understanding of the written word. Language works following a system that
9 ensures what is spoken or written down makes sense to us. Gibberish is still language, but to even be classified as being gibberish it means the words that are used fall outside of this systems’ logic. There are many concepts that work within this system or grammar, some of which I will briefly lay out with the help of the work “Language Descriptions” by Anthony Liddicoat and Timothy Curnow. Syntax, for example, deals with the order of words in a sentence so that it means what we want to say: “the dog sees the cat” versus “the cat sees the dog” (Liddicoat and Curnow 39). Language also works with units, for example within: “the tall plumber died”, ‘the tall plumber’ goes together as a unit and ‘plumber died’ does not (Liddicoat and Curnow 41). There are many more examples of how grammar works but this will not be discussed further as the point of this section is to demonstrate that the rules of language and written text are quite strict. If you put a question mark behind a sentence it creates a whole different context of meaning: “You saw your wife in the garden” versus “You saw your wife in the garden?”. The point here is: language seems to work following a relatively unambiguous system to create meaning. If we ask what written text is, the answer thus far would be that it is written down language and that it works following a specific set of rules that determine its interpretation. A big part of what written text is however, should contain why it is used by us and how. Written text has one purpose that is most prominent: communication. Why we use written text in certain situations compared to other means of communication is due to its relatively unambiguous nature. Written text can communicate quite specific and complex information efficiently. In contrast to the spoken word or spoken text, written text is also less ambiguous. You can ask someone to put something ‘into writing’; this is a more binding way of communicating. Written text is associated with seriousness, and being permanent: it is there in black and white. Written text, and language in general, is also very useful to describe things that happen over time. We could express these kinds of narratives through the image, or possibly through smell, touch or even taste but one can image this being highly inefficient. We use written text in a way that places itself within discourses that go beyond the language in use to “language use [that is] relative to social, political and cultural formations […], language reflecting social order but also language shaping social order, and shaping individuals’ interaction with society” (Jaworski and Coupland 3). We use language to reflect upon society and it is a reflection of society in itself. Written texts, in a broader concept than just being texts on paper or a screen, can be part of specific discourses.
10 Discourse is defined by the Cambridge Dictionary as “communication in speech or writing” (Discourse, Cambridge Dictionary), which comes down to simply being spoken or written text. What Jaworski and Coupland are referring to however, is a type of discourse in the Foucauldian sense. This refers to the specific way a group of people speak about a subject and with this also construct a specific reality of it; containing certain morals and power relations. In more modern academics, it is not only the written word that is seen as constructing a discourse. Images, art works and interfaces do too.
2.2 On image
When we discuss what an image is, it is needed to make clear what we mean by ‘an image’. The Cambridge Dictionary explains image as: “any picture, especially one formed by a mirror or a lens” (Image, Cambridge Dictionary). ‘Picture’ is explained by the dictionary as: “a drawing, painting, photograph, an image seen on a television or cinema screen, a cinematic film” (Picture, Cambridge Dictionary). When I will be referring to image in this work, this can include any type of picture as explained above. Images are all around us, at least, for most of us. They are so integrated in our daily lives, our surroundings and our technologies that we experience them as being ‘normal’. Symbols and logos are to be found almost everywhere humans are. Taking a photograph of ourselves - a selfie - and sending this to another person in a matter of seconds is not prodigious (anymore) for most of us in the Western world and beyond. Centuries ago the reproducing of images was done by woodblock printing or other kinds of stamping. Easily producing, using and reproducing ‘realistic’ images became possible through the invention of photography. First forms of photography were invented around 1830-1840 by Joseph Nicéphore Niépce and Louis Daguerre. The invention developed from being very time- consuming to the type of everyday analogue photography that became increasingly possible and popular in the 20th century. Taking photographs soon became easy with cameras becoming portable and available for consumers. Still, most people did not take photographs every day, but reserved them for special occasions such a holidays, birthdays and other significant events. As a photo that is taken by an analogue camera cannot be deleted within the device, one can imagine that the process of taking one was more special than the practice of digital photography is today. From being special, creating and sharing images has
11 now become the norm for most in the Western world. What lays at the core of our need to create these types of images? Of course, there are many types of images besides the photographic one, still and moving, symbols, drawings and graphs. Discussing the photographic image here however is helpful seeing that a large percentage of content on social media sites - including Facebook - is made up of photographic images in particular. André Bazin, influential film critic and theorist, claims that it is our obsession with realism in his work “The Ontology of the Photographic Image” (7). With the invention of photography: “For the first time, between the originating object and its reproduction there intervenes only the instrumentality of a nonliving agent” (Bazin 7). Also, Bazin explains that our obsession with recreating the world realistically (through the photograph) lies in our deeper human need to overcome time and death (6). By creating a ‘realistic’ replication of the world at a certain point in time, the subject stands still in this time. With the social media age now being here, it becomes clear that this ‘preservative nature’ of the photograph is no longer our main concern. With platforms like Snapchat being popular, where photos that are taken and send are deleted a few seconds after consumption, the photograph has clearly become something else. This is also the case for the popular features of Instagram and Facebook: ‘stories’, where photos stay online for 24 hours and are then automatically deleted. What then has the photograph become? In his work “Photography in the Age of Snapchat”, Daniel Miller concludes that photography has expanded to become a ubiquitous presence and with this it is almost analogous to language itself (14). The photograph has, in the new media landscape, become mundane by nature (Miller 15). The idea that the image now acts as a kind of language itself, calls for the question as to how the understanding of this language works exactly. Structural research has been done in this department by Gunther Kress and Theo van Leeuwen over several years, where they, together as well as independently, extensively discuss the different semiotic modes of images, visual design, multimodality and modality in the digital age. It becomes clear that those different modes of the visual do not work as language works, as the ‘signs’ do not refer to set meanings like the signs of language (words) do. With discussing what images are, it is important to discuss how they carry information. In the case of visual objects, knowing what information content is carried is
12 more complex than it is for text (Santini, Gupta and Jain 37). “In text, every word has a finite number of meanings, and the correct semantic value of a word, if not immediately clear, needs to be disambiguated […] through sentence level or paragraph-level analysis” (Santini, Gupta and Jain 37). With images, this ambiguity is clearly more of an issue. Although people who do not speak each other’s language can see the same image and understand what it is, what it means is more open to interpretation and personal context than the written word. Overall, what is to be concluded from this section is that how images ‘work’, how they make meaning, is different and not as straight forward as how language works. In the next section the relationship between written text and images will be discussed.
2.3 The relationship and the divide
Going back as far as Plato a specific distinction was made by people between words and images. Plato is credited with first systematising this distinction in the Cratylus. Words or language were seen as something created and artificial whereas images were seen as natural and effortlessly understood (Mitchell 75). Thomas Mitchell also makes this observation in relation to forms of art in his work Iconology: Image, Text, Ideology from 1986. Mitchell says: “[…] poetry, or verbal expression in general, sees its signs as arbitrary and conventional—that is, ‘unnatural’ in contrast to the natural signs of imagery” (47). Mitchell believes this distinction between text and image is so normal for humans to make that there is no reason to believe Plato first systematised it (75). In this logic, there is a “commonplace distinction between words and images as conventional and natural signs” (Mitchell 77). In other words: language and written text is something we have to learn to understand whereas understanding images is more innate to us. Looking at images and words this way, the visual/verbal divide seems strong. There are cases where written text can manifest itself in the form of an image (see Image 1). We can read the words individually and together they make up an image. So written text can form an image and strictly speaking, written language is visual. Does this mean that maybe they are more similar than we initially think? John Bateman states in his work Text and image: a critical introduction to the visual/verbal divide (2014) that in researching text and image one needs to avoid assuming that the two are more similar than they actually are (7). Bateman states that if text and image were more or less the same,
13 combining them would not result in anything substantially new (7). Going in the opposite direction: arguing that images and text are completely different, is also not productive because this would mean combining them does not produce anything sensible (Bateman 7). And clearly they do. In case of Image 2 for example, the fact that the words are shaped in the image of a face, changes the ‘reading’ (the interpretation) of the words in question. One could imagine the different context of the words if they were shaped like, say, an elephant. Now we do not very often encounter text and image like Image 1 in daily life. Most of the time we see images and text next to each other or on top of each other (text on image especially). Like for example with the online consumption of text-based news.
Image 1. Words form an abstract image of a face
To see how images and text are typically used in our daily news consumption, one can take a look at some popular news websites. Popular news websites like The Times, the NOS, The New York Times and many others all contain a lot of images, especially on their homepages. Surfing around the internet will result in noticing that the online written news media rely heavily on the use of images. The structure of the NOS.nl homepage even makes it necessary for a new article to be accompanied by an image. We can conclude from this that people seem to really like photographic images to go with text-based news consumption. I have noticed especially photographic images with humans on them. Another conclusion to come to here is that the images are an important part of drawing attention to the article, since they most often appear next to the headline and preview. Most of the time the images are not an essential part of the article, or even not necessary at all (see image 2).
14
Image 2. The preview of a Dutch article on the NOS.nl home page about “Kijkwijzer” accompanied by the logo of “Kijkwijzer”
In this text-based news media example, images are used to make the news more appealing even when the image is not necessary to tell the news story. Why is it that the image, and especially the moving image, can have such a powerful and seemingly immediate attention-demanding effect on people that written text just not seems to possess? Obviously, when the moving image contains audio there is an extra component to draw a persons’ attention, but even without audio the movement in images draws our attention more than something that is still. A study done by Aman Yadav et al. into cognitive and affective processing and the differences here between text, video, and video plus text in 2011 is called “If a picture is worth a thousand words is video worth a million?” This study concludes that “video is more powerful than text in the affective/engagement realm” (34). Participants in the study were more engaged when experiencing video or video plus text content compared to the text-only content. Earlier research by James Clark and Allan Paivio from 1991 shows that visual imagery, such as pictures and moving images trigger a more affective response than written words do (160). A more recent study done by Marko Horvat, Davor Kukolja, and Dragutin Ivanec in 2015 states that “emotional reactions can be induced by virtually any multimedia format: films, pictures, sounds, voice and even text.” (1394, emphasis added). “And even” suggests here that it is the least obvious or apparent option. They concluded that video material was more powerful in stimulating emotions than pictures (1396). A hierarchy is noticeable: text evokes the least affect/emotion, images more so and moving images even more. The same goes for
15 the attention hierarchy. Text and images thus are different in how or if we see them and what our reactions are towards them. A conclusion to draw here is that in cases where texts and images are placed in the same space, the image and especially a moving image, would ‘win’ in terms of grabbing our attention, holding it, and transferring a stronger feeling. One could imagine this effect being even stronger if the moving images have accompanying sound. What becomes clear by simply looking at homepages of text-based news websites is that images are also used with the purpose of drawing attention to text. Images here make a written text news website, and especially the homepage, more appealing. As if a written news article does not possess the power of being appealing enough by itself. Written text is not as sexy, you could say. Is the rule of the written word broken? Has the image won out over the word? These questions are asked by Bateman in his book Text and Image (11). His answer in short: no. “[…] despite the increase in ‘visuality’, it is rarely the case that the written word disappears. What we instead find all around us is a far richer range of combinations of different ways of making meanings” (Bateman 11, emphasis in original). Over time, their relationship has seemed to become closer and more intertwined. Bateman continues with explaining that when combinations of written text and the image are done well, it results in something more than either could achieve alone (11). French literary theorist, philosopher, linguist and semiotician Roland Barthes has written quite a bit on written text and images and their relationship. In his work “Rhetoric of the Image” (Rhétorique de l'image) from 1964, Barthes explains image-text relationships as being equal or unequal. Barthes lays out a three-way classification of text-image relations: anchorage, relay and illustration (Barthes 38) (Bateman 35). Anchorage and illustration are unequal relationships between text and the image. With ‘anchorage’, the text describes how to ‘read’ an image, for example with an image caption. The freedom of the image is being dictated by the text (Bateman 35). The other way around would be ‘illustration’, in which case the text needs the image. Barthes calls more equal text-image relations ‘relay’. According to him these types of relationships are less common. “Here text […] and image stand in a complementary relationship; the words, in the same way as the images, are fragments of a more general syntagma […]” (Barthes 41). He goes on to explain that the modes can co-exist in one iconic whole (41). With a social media site instead of more traditional media that appear on the page, this co-existence of the different modes of text-image relationships seems the default. In
16 the Discussion section, these different modes will be discussed in relation to the case studies. Both written text and images have unique characteristics, that go together well in human communication. Together, they create meanings that they would not be able to create on their own.
3. Text/Image within new media
New media, as their name suggests, have indeed not been around for long. Especially in contrast to written text and images, and the forms of media that they were native to for centuries. Who thinks of new media probably thinks of the internet first. Seeing it changed the way we live our lives so drastically, it might be the most important development in media thus far. In “What Is New Media” (2001), Lev Manovich lays out that we are living in the middle of a new media revolution, the revolution of ‘new media’. He describes this as: “the shift of all culture to computer-mediated forms of production, distribution, and communication” (2001, 19). This shift of all culture might seem radical, and maybe it is. Never before were different forms of media, text, audio and/or images together in a way in one space that the internet now offers. Manovich describes how new media became new in detailing the history of the old media and computing. Their intertwinement changed the identity of both media and the computer (Manovich 2001, 25). Manovich then goes on to explain that computerisation will affect deeper and deeper layers of culture (2001, 27). I would argue that nowadays, the identity of media and ‘the computer’ are not only changed and intertwined, but inseparable. The old media have found their ways onto the internet where they now live as if they always have. Though, as Manovich claims, we have to be wary of defining new media as old/analogue media that has solely been digitised (49, 50). Indeed, if we look at the state of new media today, one does not simply identify the ‘old media’ within it as being digitised. In the case of cinema, television or radio viewing/listening online, there is a clear feeling of experiencing the old media in a new media environment, but this is not at the core of what new media is. So, what are new media? What makes new media ‘new’ and so different from the older media forms? I will touch upon these questions briefly with the works of Lev Manovich, Mark Deuze and Terry Flew. Manovich describes in his work “New Media from Borges to HTML” (2003) that a
17 window of a web browser will come to supplement a cinema screen, a book and a CD player with which all culture will be filtered through a computer (8). The human-computer interface plays a big part in this as it “[…] comes to act as a new form through which all older forms of cultural production are being mediated” (Manovich 2003, 8). Manovich has since written a lot about these human-computer interfaces and their software in particular. In 2019, new media and the human-computer interface are still ‘filtering’ culture but most of all, they are creating it on an immense scale. When talking of new media today, we are talking about massive quantities of digital cultural production, distribution and consumption. We can identify a shift from ‘filtering’ or ‘translating’ the old media to new media forms towards mass consumption and now to mass new media production. This mass new media production mostly consists of user-generated media content. This shift is touched upon by Manovich in his work “The Practise of Everyday (Media) Life” (2009), where he discusses user-generated content in particular. Manovich shares some numbers of media creation and consumption from 2006-2008, a few years before the first photo was shared on Instagram. “Facebook: 14,00,000 photo uploads daily. The number of new videos uploaded to YouTube every 24 hours (as of July 2006): 65,000” (Manovich 2009, 4). In 2013, an average of 350 million new photos are uploaded to Facebook every day (Smith n. pag.). These numbers have been rising and if we are talking about what new media are, user- generated content in the form of text and images seems to be a big part of it. This type and scale of bottom-up production structure was not present in the older media. This omnipresence of media is another aspect of what makes these media ‘new’. Digitalisation, embeddedness, connection and fluidity are all concepts that work within this omnipresence of new media. What is happening now and what will continue happening in the (near) future is that new media will become more and more invisible (Deuze 40). This next shift will be from device orientated media consumption to a more ‘natural’ media experience such as the further implementation of the Internet of Things (Deuze 40). A part of experiencing new media today, is experiencing it with less consciousness about it. The new media we use and speak about today are in itself different than the new media from twenty or even ten years ago. Answering what new media are exactly proves to be quite difficult, as the changes in technology and new media use change so rapidly. But the concepts that I have discussed above, combine to be helpful in our understanding. A combination of the internet, digitalisation, bottom up content creation and their
18 omnipresence are at heart. On the next page, Image 3 shows the three “C’s” in a definition of new media given by Terry Flew in his book New Media an Introduction (2008). Flew explains that three C’s: Computing and information technology, Communications networks and digitised media and information Content, converge in the case of new media. In its centre is the internet, at the heart of new media. Now new media is discussed in general briefly, let is investigate how written text and images occur and have occurred within them.
Image 3. From New Media an Introduction by Terry Flew (page 3)
When we look at new media, we can identify that it is a space for the visual to thrive. In contrast with the older media, image production, image sharing and image viewing is common practice here. Over time, a shift has become visible from text orientated new media into a more visual experience. If we look at the popular social media platforms from the early 2000s such as MySpace and LinkedIn, written text was playing a prominent role on the platforms. Images were present, but not in abundance. Twitter became popular a few years later and this social media site was and is mainly text orientated. Around the same time YouTube became very popular, and here an early shift towards a focus on images can be identified. The video-based platform has since in part taken over the role of the television, with many young people consuming the majority of the moving images this way (Businesswire.com). Tumblr was launched in 2007, gaining popularity quickly. This site focusses around micro-blogging and has also seen a shift towards more image sharing.
19 Facebook, the most popular social media site in the world, became popular in 2006. In 2009, Facebook was ranked the most used social networking service by monthly active users. In 2006, Facebooks interface was mainly text orientated, with a single profile picture in the top left corner of someone’s profile and the odd tagged photo here and there. Now a person’s Facebook profile page is cluttered with images. After Facebook’s rise, other social media platforms have since become very popular. It is noticeable that these were very image orientated from the start, like Instagram and Snapchat, where text plays a significantly smaller role. Snapchat, mainly popular amongst young people, has seen the most dramatic shift towards an image-based structure. The app has no central user profile with textual information, no textual descriptions or bio’s and mostly focusses around sending photographs and other images. The image-shift that is clearly noticeable within new media raises the question what the state is of text in this landscape. Do we envision a future of (new) media where the consumption of text is kept to a bare minimum? As the technological possibilities for creating and sharing images has become more advanced, more and more images are used. It becomes clear that the more images we can produce and consume, the less we need and want written text. As the book and the page were replaced by the screen, the ‘logic of writing’ has been replaced by the ‘logic of the image’ (Kress 9). The power of text has been giving way to the power of the image after centuries of dominance, with the new media as a perfect example. New media of today, and especially the new social media platform, centres around the image. Images play a key role in presenting online identities, through profile pictures, everyday snapshots and created media (Highfield and Leaver 49). Next to this, as Gunther Kress puts it in his work Literacy in the New Media Age: “When writing now appears on the screen, it does so subject to the logic of the image” (10). Kress explains ‘the logic of the image’ as being spatial/simultaneous, in contrast to ‘the logic of writing’ as being temporal/sequential (20). According to Kress, the logic of the image now dominates the sites and the conditions of appearance of all ‘displayed’ communication (9). Following this logic, written text is now a visitor within this world of the image, and the world of new media. Existing within this other logic, it has to follow new rules. “After a long period of the dominance of the book as the central medium of communication, the screen has now taken that place. This is leading to more than a mere displacement of writing. It is leading to an inversion in semiotic power” (Kress 9). Kress’ work was published in 2003, the same year
20 MySpace was founded, before smartphones and iPads existed. One can image the magnitude of the shift in semiotic power now screens are everywhere, in our pockets, on our faces, multiple ones in our homes and in classrooms. Now the image has taken over our new media spaces and written text is visiting within this logic of the image, it seems fair to assume that our affective responses are changing too. As discussed above, images cause and hold a different affective response and quality than written text does by its nature alone. Living within media that follows the logic of the image must have a great impact on the affective quality and potential that written text now possesses. The new media networks that images and text travel through to reach our bodies for their consumption, are often just considered as passive sets of objects. James Ash argues in his work “Sensation, Networks, and the GIF: Toward an Allotropic Account of Affect” that rather than doing this, the networks can be considered as “transmitting and translating sense itself, which in turn generates affects as these sensations encounter bodies” (123). If we consider these networks to be operating within the logic of the image, this means that the generating of affect within these networks is structured to benefit the image. Computational objects and networks structure the affects that a particular object can potentially generate (Ash 123). The potentials for generating affect within these computational objects and networks are thus noticeably not equal towards images versus texts.
3.1 Affordances
In the introduction of their work “The Affordances of Social Media Platforms” Taina Bucher and Anne Helmond describe the switch of the ‘favourite button’ on Twitter to the ‘heart button’. This switch was answered with noticeable dislike from users. Bucher and Helmond write that “a feature is clearly not just a feature”, they are objects of intense feelings (2). Social media platforms are structured in a way so that users can interact with them in specific ways through these features. This interaction needs to be easy to understand, intuitive and fun. The way a social media interface is structured, affords a user to use it in a certain way. I will be discussing the concept of affordances in relation to social media interfaces and images and text in the next section. The concept of affordance was first laid out by James Gibson in his work The
21 Ecological Approach to Visual Perception in 1979. Gibson discusses the concept of affordances as being specific relationships between the animal and their environment. The surroundings of the animal afford the animal to do certain things, it provides things. These affordances are not the same for all animals, as one can imagine, they are relative to the animal (Gibson 120). Affordances thus say something about the relationship between the animal (or human) and the environment. The concept of affordances has since been used in many academic fields including design, Human-Computer Interaction, sociology and new media studies. When using the concept of affordances within a new media interface analysis, the interface is the environment the human is using. The features and specificities of this environment afford the user specific things to do (and to not do). This understanding of affordances is described by Bucher and Helmond as ‘low-level affordances’ and is feature orientated. Next to low-level affordances, Bucher and Helmond describe ‘high-level affordances’: a more abstract conceptualisation that is much broader than specific features such as buttons and screens. This conceptualisation of affordances as high-level affordances is closer to the original concept Gibson described: “In a Gibsonian sense, technical features—understood as the furniture of the digital landscape—afford certain actions such as clicking, sharing, or liking” (Bucher and Helmond 13, emphasis added). The old Twitter ‘favourite button’ simply affords a user to click on it, and with it the user communicates a post is one of his/her favourites. But many users also used this button to say “I agree”, to save the post to look at later or to end the conversation (Bucher and Helmond 2). Bucher and Helmond call for being more sensitive to platform specificities in empirical analyses of affordances in social media. This entails considering how affordances of platforms are relational and multi-layered. If we are speaking about interface affordances, a concept to consider is productive power. With affording certain actions, a platform constructs a certain power structure: creating regulatory power and normalisation. The concept of productive power was introduced by philosopher Michel Foucault in his work on social control and its relation to power and knowledge. A productive power framework, as Mel Stanfill describes it, “operates from the premise that making something more possible, normative, or ‘common sense’ is a form of constraint encouraging that outcome” (1060). Stanfill introduces the ‘discursive interface analysis’ in their work “The interface as discourse: The production of norms through web design”. With this form of analysis, a websites’ affordances are analysed
22 to reveal what users should do, thus producing the normative. Stanfill wants us to ask which beliefs drive design and which are built in, and also what the consequences are of these design choices. This type of analysis uses the concept of low- and high-level affordances, seeing it looks at specific features and also looks beyond these functions, how they shape the norm. Stanfill uses three types of affordances (out of four) from the work of Rex Hartson, to conduct the discursive interface analysis: cognitive, sensory and functional. Functional affordances show what a site can actually do, cognitive affordances how users know what a site can do and sensory affordances enable the user in sensing something (design choices) (Stanfill 1063). With these three components, the discursive interface analysis produces knowledge into what a platform affords the user and how these affordances, through productive power, create certain norms. Now there is a better understanding of affordances and how this concept relates to social media, we will come back to the two core concepts of this work: written text and images. How do they relate to affordances in the world of social media? Written text and images are essential parts of the affordances of a social media interface. In fact, they are the only two components of the visual affordances a social media platform has to offer. They often occur in combination. A word (text) in a box (image) is often a button, which will afford you to click on, often taking a specific action. Beyond the fact that visual affordances are made up out of written text and images, these affordances offer the user to interact with written text and images too. A button may afford you to upload or take a photo or write a text. Where those buttons or options are and what a user can do with them, has certain implications. These two ‘levels’ of affordances of images and text are comparable with the high- and low-level affordances Bucher and Helmond describe. We know that written text and images have different affordances themselves; you interact with one differently than with the other, one mode affords you different things and actions than the other. Their differences have been discussed earlier in this work, though I will sum up some core differences in the light of affordances. Written text affords a person to imagine an accompanying image, and affords ‘understanding’ following a narrative logic. Whereas images do not evoke this same imagination and have a more spatial/simultaneous logic. Images afford us – in the case of the photograph – to look at a ‘real’ representation of the world. Written text affords us a type of clear and precise communication, that images (without speech) do not afford us in certain situations. In their turn images also afford us
23 another type of clear and precise communication, that written text does not. A good example of this is the use of traffic signs, instead of long textual information next to the roads.
3.2 Methodology
To obtain a better understanding of how written text and images relate to each other within a new media environment, I will conduct three case studies of Facebook’s user interface. These case studies will be done through a historical overview of updates in certain interface features. The first case study is an analysis of the (changes in the) Facebook status update feature and how these changes afford different options in terms of the use of written text and images. The second case study is an analysis into how emoji, stickers, symbols and GIFs were and are supported by the social media site, in relation to written text. The third case study is an analysis into how ‘the person’ was and is represented on Facebook through images and text, specifically through ‘the profile’. The analyses are conducted in the ‘discursive interface analysis’ style laid out by Mel Stanfill, introduced upon above. Within my analyses I will make use of the concept of affordances to describe what certain functions afford the end-user to do. The three types of affordances – functional, cognitive and sensory – laid out by Stanfill in their discursive interface analysis will provide useful. Following the advice of Bucher and Helmond to be more sensitive to platform specificities, I take these Facebook specificities in account whilst conducting the analyses. The contemporary historical period I will be considering is from September 2006 until current day (June 2019). I choose September 2006 because that is the time the Facebook platform became open to anyone. From this starting point, the most noticeable changes in Facebook’s platform structure in relation to the three subjects of the case studies are discussed. A study worth mentioning here is “Becoming a semiotic technology – a historical study of Instagram's tools for making and sharing photos and videos” by Søren Vigild Poulsen. Poulsen researches Instagram’s interface tools for making images and describes these tools in semiotic terms while considering their meaning making potentials. Poulsen also conducts a contemporary historical analysis and discusses two kinds of changes in the semiotic function that emerges from the analysis. “One is the addition of new tools with semiotic functions that were not previously available, and another is the elaboration of
24 the meaning potential of existing tools” (Poulsen 133). My work here will also consider the changes in platform tools of an influential social media site, both existing and new, and their ‘meaning potentials’. Though our approaches differ as I will be researching the features through the concept of affordances and further, how this produces the normative in relation to productive power. Nonetheless, Poulsens’ method will provide a useful context in the way it concerns platform features that are supporting images. By using the three types of affordances as part of the analysis, a variety of functions can be discussed. I analyse the low-level affordances of the specific features in relation to written text and images as well as the high-level affordances they provide. This means that besides discussing (the changes in) buttons, placements, options, menus and lay-outs, I am discussing what actions they afford the end-user and what this produces. In this way, combining a historical approach with the discursive interface analysis allows me to map the changes in certain Facebook’s interface features and study their normalising effects on a higher level. With doing this, the relationship between written text and the (moving) image will be thoroughly investigated. Knowing that new and social media are environments where images thrive, and are operating within the ‘logic of the image’, special attention is paid to how text is being represented. Being that (moving) images clearly are playing a bigger role on social media sites today than they were ten years ago, the starting point of the analyses is to research how this is specifically represented by Facebook’s interface. In researching these specific features, a clearer idea emerges as to what ‘the shift’ to the logic of the image actually looks like. Keeping the relationship between text and images central is key to understanding what the limitations of images are in these environments and vice versa. The discursive interface analysis is described by Stanfill as examining “norms produced by ‘affordances’ of websites – defined by H. Rex Hartson (2003: 316) as what a site ‘offers the user, what it provides or furnishes’ (emphasis in original)” (1062). Stanfill explains that this type of analysis is different from the social-scientific Human–Computer Interaction framework as it is not orientated towards designers goals, but takes into account that there is an unequal power balance between industry and the end-user (1062). Something that is especially true for a platform like Facebook. The discursive interface analysis shows how through productive constraint technically possible uses become more or less normative. Of course, the concept of the discourse takes a central place in the
25 discursive analysis, as it provides a fruitful base for researching productive power. In the analyses the interface is analysed as discourse, it is telling us something. This Foucauldian concept is used widely within the critical discourse analysis in Media and Cultural Studies, and proves useful as discourses “structure how we think about things and accordingly how it makes sense to us to act” (Stanfill 1061). Foucault introduced the idea of the discourse in relation to social power structures over fifty years ago, and it still provides an applicable base for analysis. Foucault was mainly talking about governance from the national establishment like the government and other powerful institutions like prisons, schools, hospitals, the military and religion. Which in his time were considered the main producers of power. However, I would like to add that in this day and age, the massive new media companies that we spend so much of our time within are to be considered serious producers of productive power too. Their interfaces are part of our everyday lives and communications, structuring the normative through their affordances. Analysing the interface of the social media site with the most users worldwide in relation to the two main pillars of our understanding of the world and human communication: written text and (moving) images, will help provide knowledge into the formation of modern-day productive power. In the Discussions section after the case studies, a discussion of the possible implications of the normalising effects of the changes within the interface will be provided.
3.3 Text/Image within Facebook’s interface
This section will be a relatively short overview of how written text and images have existed within Facebook’s interface. No specific features, functions and tools will be discussed in detail at this point. The purpose of this overview is to establish a base on which to further built the case studies, and to demonstrate the more general ‘image-shift’. Facebook became available to anyone with an internet connection in September 2006. At the start of its existence, Facebook’s interface supported the profile picture on the top left of someone’s profile, photos that are uploaded in ‘Albums’ and photos a person is tagged in. These Albums however are not centrally linked to on someone’s profile. Most of the links and options are consisting of text. The next year, a few symbols are added in the top left menu. The ‘Mini-Feed’ still mainly contains text updates. It was not until 2010 that photos really began to play a bigger role, with more images appearing on the profile, bigger
26 symbols and more sponsored content containing images. In 2011 the “Timeline update” was introduced, bringing the most significant redesign of the interface thus far. With Timeline the interface became a lot more image centred, with the profile and the home page supporting significantly more images than before. At the Facebook ‘f8 developers conference’ in 2011, Mark Zuckerberg reveals this new Timeline redesign for the first time. Zuckerberg says: “The first thing you’re going to notice is that, it’s just, a lot more visual” (F8 2011 Keynote, 14:19). After the 2011 shift to a more visual platform, changes in favour of images kept steadily being implemented. The Facebook Messenger App was also released in 2011, making instant messaging via Facebook on mobile a lot easier. Facebook Messenger has become more and more supportive of image and photo sharing over the years. April 2012 is when Facebook buys the very visual social media platform Instagram for one billion US dollars. Hashtags, video stories and more options to personalise photos soon became available on Facebook too. Though, until this day, we see that Facebook still makes use of a considerable amount of written text in its interface structure. Especially compared to the other popular social media websites. In contrast to, for example, Instagram and Snapchat, written text is found in every feature. To explore in what ways text is used, and images are implemented the case studies section will follow.
4. Case studies
In the next few sections I will conduct three analyses as case studies of Facebook’s user interface: ‘The status update’, ‘Emoji, symbols, stickers and GIFs’ and ‘The person’. These particular features are chosen as case studies within this work as they represent the nature of social media and Facebook in particular well. Sharing (updating status), the profile (how a person is represented) and the use of emoji, symbols, stickers and GIFs are at the heart of what makes a social media platform social. Within ‘The status update’, the Facebook feature that allows users to update their status is looked at, from its earlier forms until the way it takes shape today. Attention will be paid to how textual and visual elements within this feature evolve. With ‘Emoji, symbols, stickers and GIFs’ these visual elements of Facebook’s interface will be analysed throughout the platform throughout its history, paying attention to their relationship with text. The third case study, ‘The person’, will look at how a person is represented through text and images within Facebook’s interface. This includes the personal
27 profile and a persons’ representation across the platform. To collect data of older Facebook interface layouts, click through YouTube tutorials, screenshots, news articles on design updates and tech blog posts are used. YouTube tutorials in particular provide a useful source of data in this case, as the videos show how the older versions of Facebook were actively used in movement. Unfortunately, these do not exist from before 2009. With this data collection, the functions that are subject to the case studies will be discussed through the three types of affordances. First the functional affordances, second the cognitive affordances and last the sensory affordances will be discussed per change in feature in the case studies. First, the low-level affordances will be discussed per case study (what changes and how) followed by the high-level affordances (what do the changes mean). These findings will be discussed in the Discussion section, together with the findings of the first section on written text, (moving) images and their relationship. The discursive interface analysis method will provide a structural base for analysing the different features across the interface, in relation to the affordances of written text and images, and their productive power.
4.1 Case study #1: The status update
When Facebook became open to the public on the 26th of September 2006, the interface of Facebook looked a lot different than it does today. The status update did exist at this point, however not in a central position of one’s profile. Analysing the status update through the concept of functional affordances, I will discuss here what the feature was able to do. Looking at Image 4, we can see what the status update function looked like in 2006. A person was able to upload a status, only made up out of written text. Like a lot of other Facebook features at the time, the functions of the status update were limited. Looking at the cognitive affordances of the status update here, the question to answer is: how does a user know what this function can do? The word ‘edit’ at the top right of the feature provides the user with the cue that by clicking it the status can be edited. Two other clickable items are the words ‘See All’ and ‘[2] updates’. This early version of the status update thus provides the user with three options to click on, all indicated with written text. As an attribute to cognitive affordances, the concept of sensory affordances consists of certain design features to help the user with sensing. There are two basic symbols used in the
28 status update function at this point, an arrow at the top left and an image of an envelope next to the status.
Image 4. Mark Zuckerberg’s Facebook profile page in 2006, viewed from the user
In 2007, Facebook’s interface sees some changes, but the status update feature still only supports written text (see Image 5). The two options to see past status updates together ‘See All’ and ‘[2] updates’ have been removed. What is possible to do with the status update function, thus has been limited. In terms of the functional affordances, not much has changed with the status update feature. The cognitive affordances did change, however. From the word ‘edit’ has now been made ‘Update your status…’. The status update in 2007 now only has one clickable item at a time, both written text only. A sensory affordance that has changed from 2006 to 2007 is that the placing of the status update feature has changed. The function now has a more visible and prominent place on the profile page, since it moved from below the profile picture and links to personal information to the top centre of the page. As most of the text and the two symbols have been removed, the feature now has a more minimalistic and sleek appearance.
29
Image 5. A Facebook profile page in 2007, viewed from the user
In 2008 the status update feature sees a lot of changes. The functional affordances increase significantly with a new profile layout (see Image 6). Beyond written text, it is now possible to include written notes, images, videos and links in a status update. I would argue that this is the biggest change in the status update that Facebook has ever made. Next to being able to upload anything other than written text as one’s status update or wall post, people can now comment on status updates.
30
Image 6. A Facebook profile page in 2008, viewed from the user
The cognitive affordances of this new status update and all its new functions have obviously also changed significantly, with new symbols, text and buttons to go with the all the new options. The written text option as a status update is still the default option. Sharing a written note is the next option, followed by (moving) images. A clear ‘Post’ button on the right side of the function now first appears within the status update. The sensory affordances now include several symbols and better legibility. The addition of the text entry box makes the function more noticeable and prominent. The status update became a lot more prominent on the News Feed (Live Feed), where it was moved from a position on the side of the screen to the top centre in 2009 (see Image 7). The “Add photo”, etc, texts are now gone, only leaving their symbols with the clear indication “Attach” in front of them. The ‘Write note’ option has downgraded into the drop-down menu. Furthermore, in uploading a status update now a person is no longer ‘posting’ but ‘sharing’ as the publish button to update one’s own status has changed.
31
Image 7. Mark Zuckerberg’s profile page in 2009, viewed from the user
In terms of the sensory affordances, this version of the status update has gotten back its more simplistic and clean look, with a more minimalistic design without the written text. The symbols are now meant to be sufficient on their own for the user to understand what to do with them. But mousing over them will reveal a written explanation like: ‘photo’, ‘video’ and ‘event’. These words came back in 2010, with the text entry box missing completely (see Image 8). The text entry box or photo upload options would appear upon clicking on the text/symbol buttons. This is the most minimalistic the feature has looked in Facebook’s history, taking up the least amount of space.
Image 8. A Facebook profile page in 2010, viewed from the user
32 The next change to the status update feature was with the big ‘Timeline redesign’ in 2011 and 2012 (see Image 9). With the Timeline redesign, the original Wall became more like an overview of one’s life, where one can add ‘life events’ and easily scroll back and forward through time.
Image 9. A Facebook profile page in 2011, viewed from the user
Considering the status update, the text entry box is back, and the written text next to the ‘photo’ and ‘status’ buttons has stayed. With the addition of being able to share the particular life events, the functional affordances have changed significantly. These life events are highly visual and visible important posts on a person’s timeline, categorised in different areas of life. With a status update, upon clicking on the text entry box, one can now tag locations and friends, that will appear as written text links in the update. A part of the cognitive affordances to support this new function are the five new symbols at the top right. The ‘video’ and ‘link’ option buttons have been removed. The ‘share’ or ‘post’ buttons have also been removed from the outlook of the feature at this point, and is only to be found after clicking in the text entry box (it is back to ‘post’). The sensory affordances have thus also changed, although the font and original symbols have stayed the same. All Facebook users were forced to use the new Timeline redesign in the spring of 2012. In the following year, the life event update symbols have disappeared and made place for a more general ‘life event’ button (see image 10). This does not change the functional affordances, but the introduction of ‘feelings and actions’ to the status update
33 feature does (see Image 11). After clicking on the text entry box in ‘status’ mode, a user was able to select a particular feeling or action to go with their status update. This part of the status update is made up out of words and an accompanying symbol. As can be seen in Image 11, categories to share within are either what you are feeling or consuming.
Image 10. A Facebook profile page in 2013, viewed from the user
Image 11. Feelings and Actions added to the status update feature
Upon clicking on the text entry box now, next to being able to add a person and a location, two extra buttons have appeared next to those: an ‘add photo’ button and the ‘add
34 feelings/actions’ button (see Image 12). In terms of cognitive affordances, the order of the top row buttons has not changed. The fact that the five symbols of the ‘life event’ function are now bundled under one general button gives the status update a less cluttered look. The addition of the photo upload button within the written text status update is particularly important to the text/image relationship within the status update. The sensory affordances at this phase of the status update feature highly depend on the variety of the different symbols, that gives the function a more visual but more cluttered look.
Image 12. Options when uploading a textual status update in 2013
The next changes to the status update feature are around the year 2016 when the cognitive and sensory affordances change (see Image 13). The ‘photo’ button at the top bar changes into ‘photo/video’, the word video had been gone from the top bar since 2010. The button ‘place’ has been removed. Most clearly, the font and the symbols have changed significantly, giving the status update feature a less dated look.
Image 13. Facebook profile page in 2016, viewed from the user
35 In 2017, the option was added to stream a live video from the status update. This button is placed before the ‘life event’ button (see Image 14). Another big addition to the status update is the function of the ‘image background’: making a text-only post to have a specific visual background (see Image 15). A user can choose between a variety of designs as a background image for their text-only status update, resulting in a text-image combination. Also part of the update is the addition of a wide range of emoji, GIFs and stickers that can be added to the written text. The cognitive affordances have changed significantly too, with the buttons for the written text options now being in a different place and having different symbols. They no longer take up a relatively small part of the total status update, but are larger and more visible and visual. Obviously, the image background function brings about its own new set of cognitive affordances, mainly consisting of the text entry box being a lot bigger and busier upon clicking within it. The new button to add emoji is placed within the text entry box, to give it a more prominent position. Different image backgrounds can be selected easily with multiple buttons at the bottom of the text entry box. The buttons and symbols are in a different style and all have colours now. An important change is that a miniature of a person’s profile picture is now placed within the text entry box. The feature overall has a more visual, cluttered and busy feel.
Image 14. Facebook profile page in 2017, viewed from the user
36
Image 15. The image background function introduced in 2017
At present day (spring 2019), the same options are possible while some options are added like ‘play with friends’ and ‘support charity’. Posting something can now directly be added to a person’s ‘story’. In terms of cognitive affordances, buttons for adding extra media to the written status update have returned under the text entry box (see Image 16). The option to add photos and or videos is now represented by two buttons. The button to ‘share’ has been removed and will only appear upon clicking within the box (see image 17). After some years, updating a status has now become ‘sharing’ again. The sensory affordances have yet a more visual, playful, rounded feel again, with more colourful symbols being present.
Image 16. Facebook status update feature in 2019, viewed from the user
37
Image 17. Facebook status update feature upon clicking on it in 2019, viewed from the user
In the previous section, the changes in low-level affordances of the status update function have been discussed extensively. In the next section, these findings will be analysed from a high-level affordances perspective, laying out the normalising effects the changes in the status update feature bring about in relation to written text and the image. In the earliest days of Facebook, the status update feature was text-based only. This stayed this way until an important update in 2008. Having this origin, Facebook used to consist mostly of written text content. Moving the status update to a more prominent position within the profile and removing the symbols in 2007, makes the function more important and written-text focussed. Until this time, images such as photos, symbols, GIFs and emoji were not yet prioritised on the platform. With the rise in popularity of the smartphone in the mid-2000s, taking photographs became a more normal practise. This reflects in the changes the status update sees from 2008 onwards. With making it possible and easy to upload images with the 2008 update, this practise instantly becomes more normal and common for over a hundred million active users (Clement n. pag). Soon, the content on Facebook sees a shift to the visual, and the choices in status update layout clearly reflect this. The buttons to share visual content get a more prominent position with every update and the cognitive and sensory affordances themselves become more visual as well. This entails the changing in order of the buttons to prioritise the sharing of image- based content; the buttons themselves becoming more visual; the addition of symbols and emoji and being able to share GIFs, stickers and video material. The position of written text
38 within the status update features’ cognitive and sensory affordances, however, does not seem to have suffered much. This means that the end-user knows how to use the feature of the interface partially based on textual cues. The placement of the buttons and options have clearly shifted in favour of the sharing of the visual though. Interestingly, the change in word choice from the ‘post’ to ‘share’ button, indicates a move away from the literary, as posting something is strongly associated with sending letters. ‘Sharing’ however, does not hold this association per se. In terms of the overall functional affordances however, written text has seen a strong loss compared to the use of the visual via the status update feature. This is made the clearest by the introduction of the ‘image background function’, where written text must become image-like in order to stay visible on the platform. Overall, the conclusion can be drawn that especially in terms of the functional affordances, the status update has become a lot more (supportive of the) visual over time.
4.2 Case study #2: Emoji, symbols, stickers and GIFs
As can be observed from Image 4 in the previous case study, the Facebook profile did not contain a lot of visual elements in 2006. Symbols were kept to a minimum and although GIFs have been around the internet since its very beginning, they are not supported on Facebook until fairly recent years. In this case study, the four visual elements “emoji, symbols, stickers and GIFs” will be analysed (or a lack thereof) throughout the platform’s history. The analysis will give more attention to the later years as most of the visual elements are not supported until later in time. In terms of functional affordances, the Facebook interface did not support the use of emoji, symbols, stickers or GIFs in any of its features for people to interact with in 2006. All actions that could be taken was communicating through written text or uploading images. This means in terms of what a user can do, none of the actions involved the use of the above-mentioned elements. Cognitive affordances surrounding the use of emoji, symbols, stickers or GIFs did not exist either. The only affordances in relation to those visual elements are of a sensory nature and consist of the symbols that are used accompanying updates on the Mini-Feed. Also, Facebook’s logo at the time (the graphic image of a man’s face) can be seen at the top left corner (see image 4). The platform, at this stage, looks like a text-based medium where visual elements are secondary. Written text is used as the main form of
39 content and design in most functions. In 2007, we see an increase in the use of symbols as they appear next to the textual buttons in the menu on the top left (see image 5). These additions are cognitive affordances as they aid the user in knowing what they can do with the buttons. At this time adding certain ‘applications’ was popular on Facebook, which would appear on someone’s Mini- Feed. Some applications did not come with a logo, however, leaving the space where the logo should go awkwardly empty. This shows that, at this point in time, Facebook was not too concerned about their visual elements. The Facebook logo of the man’s face has disappeared. Although the interface appears more visual due to the addition of showing six friends (and their profile pictures) instead of three on one’s profile, Facebook still does not support the use of emoji, symbols, stickers or GIFs. In terms of the affordances types in 2007 in relation to the use of the visual elements, some cognitive affordances appear but most are still of a sensory nature. Overall, the platform still looks like a mainly text-based medium. This continues for over a couple of years, with more cognitive and sensory affordances being added, only in terms of symbols. In 2008, for example, symbols are added to the cognitive affordances of the status update, as discussed in the previous case study. The ‘Mini-Feed’ has disappeared and made place for the ‘Wall’ in the new redesign. This redesign appears more visual through the addition of peoples’ profile pictures now appearing next to their name, but in terms of the use of the other visual elements, no functional affordances are added. The top left menu and its symbols are removed and replaced by tabs under a persons’ name. In 2009, an important change is made to Facebook as a functional affordance, when the ‘like button’ is introduced (see image 7). At that time this button is represented by the word ‘like’, although next to the notification that something has been liked, the now famous symbol of the ‘thumbs up’ first appears. In the following years, the amount of likes a status or comment has, is represented next to the “thumbs up” symbol. Until this day, the word ‘like’ has not been removed. The status update feature introduces some new symbols as discussed in the previous case study, but overall in terms of the three types of affordances the interface looks quite ‘empty’ in the department of emoji, symbols, stickers and GIFs. This does not change until in 2012 the use of emoji becomes supported in the Messenger function and in the comments sections. This is a big change in terms of functional affordances in the visual department of the interface.
40 Before this, the use of emoji was possible, but a user would have to know -or copy and paste- specific textual codes that would turn into emoji. Since this function was added, the use of emoji increased significantly and are now a standard part of any comment section across the platform. In terms of cognitive affordances, if one wants to add an emoji to a comment in 2012, there is a button shown as a smiley emoji at the right side of the text entry field (see Image 18). The choice of emoji was very limited at this time via this application, but more emoji could be used using their textual codes.
Image 18. Adding emoticons to Facebook Messenger in 2012
A year later, ‘stickers’ are introduced: the bigger and in some cases moving cousin of the emoji. At this time Facebook users have a variety of options to express themselves with visual elements other than photographic images. Stickers are found under the same button as emoji (see image 19). Different themes of stickers were popular and are still available today like ‘Meep’ the big expressive smiley face and ‘Pusheen’ the grey cat that moves. The sensory affordances of the new sticker and emoji features are still quite simple and ‘rugged’. With the Timeline redesign, Facebook becomes more visual in terms of (the placing of) photographic images and in the form of other visual cognitive affordances (see Image 20). The ‘About’ section on someone’s profile now includes a symbol in front of every line, introduced just before the Timeline redesign in 2010. The buttons on the left side of the top bar that used to read: ‘home’, ‘profile’, ‘friends’ and ‘inbox’ are now replaced by three symbols and ‘home’ is moved to the right side of the bar.
41
Image 19. A variety of stickers became available in Messenger and in the comment section
Image 20. A Facebook profile after the Timeline redesign in 2011, viewed from the user
In 2013, the ‘feelings’ option is added to the status update feature, making use of some emoji and extra symbols like discussed in the previous case study. Bit by bit the three types
42 of affordances around the use of emoji, symbols, stickers and GIFs are now taking shape to make the platform less text focussed and offer the user more ways to interact with, and express themselves through, images. An important thing to notice is that the Facebook logo changes from the word ‘facebook’ to a symbol of the letter F with a new redesign in 2014 (see image 21). Another important update in terms of functional affordances is when in 2015 the use of GIFs is finally supported. It is now possible to post GIFs to Facebook via a third-party, using a link to the GIF in the status update feature. An extension via Chrome can be used to post GIFs in comment sections at this time as well (see Image 22). With this extension, the cognitive affordance is to show a ‘GIF button’, with the written word representing the option. It now appears as the third option next to sharing an image and a smiley or sticker. It is not until 2017 however that posting a GIF can be done directly through the status update feature and in the comment section without a third-party.
Image 21. The new Facebook logo as shown on the website
Image 22. The GIF button in the comment section through a Chrome extension
43 In 2016, another big change in the functional affordances around emoji, symbols, stickers and GIFs is when Facebook introduces ‘reactions’ on posts, next to the like-function. It is now possible for users to express a range of ‘emotion’ reactions on a post. The five extra reactions that are added next to a ‘like’ can be seen in Image 23, these emoji are animated when appearing on Facebook however (their movements loop). What cognitive affordances concerned, the new function works by a mouseover over the original like button, which will make the rest of the reactions appear and ready to select. Upon hoovering over the options, the words of their meaning appear (see Image 24). This design is keeping the like-button the default setting though.
Image 23. New Facebook reactions in 2016
Image 24. Facebook reactions in mouseover in 2019
As can be seen in comparing Image 23 with Image 24, the sensory affordances changed from 2016 to 2019. Also, the animations change slightly, making the faces even more expressive and moving more. From this update in 2016 onwards, the like, comment and share buttons are now bigger and all include a symbol. Interestingly, the words accompanying the reactions differ in meaning in different language settings. For example, in Dutch, the heart emoji says ‘geweldig’ meaning ‘amazing’, which is significantly different than expressing ‘love’. Thus, the cognitive affordances of the same functional affordance differ user to user
44 here, because written text is not universal. From 2016 to 2019 Facebook has not seen mayor changes in terms of the visual elements’ three types of affordances. More and more emoji become available on Facebook over time, they now have a smooth not as ‘rugged’ look with a lot more detail. Facebook has its own design of emoji, different from the design of the Unicode Standard Emoji set found on most smartphones, although they are fully compatible and supported. The GIF function is now a standard part of the status update feature, although only visible through a drop-down menu at the moment. Having discussed the low-level affordances, the next section will touch upon the high-level affordances and their normalising effects in relation to text and images on the platform. Facebook’s interface started out in 2006 with a strong position for the written word and was not a very visual place. This was not strange at that time, however other social media platforms that existed before and at that time were a more visual-orientated like Hyves (Dutch 2004-2013) and MySpace. As can be seen in Image 25, Hyves’ logo was image (emoji) orientated from the very beginning. In contrast, from the start Facebook looked ‘cleaner’ and less ‘flashy’ in terms of colours, logo, symbols, emoji and possible background settings. The platform clearly prioritised written text; their affordances were less orientated on those visual elements then. Rapidly becoming more and more popular as a social media platform, this was the norm in online communications for millions of people. As shown in the above written analysis, the addition of emoji, symbols, stickers and GIFs happened increasingly over time, with a spike in new visual affordances between 2012 and 2016. Between then, as the analysis shows, the implementation and use of emoji, stickers and symbols strongly increases. Interestingly, Hyves for example, supported the sharing of looping animations and other types of (moving) images as early as 2006 (see Image 26). Due to the increasing worldly popularity of Facebook and Twitter however, Hyves was shut down in 2013 (see Image 27). This seems to indicate that people at that time went where other people went to communicate, regardless of the ‘visual attractiveness or possibilities’ of the environment. Around 2011-2012 with the Timeline redesign, more use of symbols and the support of emoji, Facebook has collected around one billion active users word wide. The image-shift is taking place, and with it the use of images in all aspects becomes increasingly normalised. Especially interesting, considering that the use of these emoji, stickers, symbols and GIFs is occurring in close interaction with written text.
45
Image 25. The Hyves logo
Image 26. The dancing banana looping animation that was popular on Hyves
Image 27. An image used by many ex-Hyves users as their profile picture reading: “Sorry, I have also moved to Facebook” (own translation)
As the analysis shows, most of these visual elements are introduced to be an addition to written text. Symbols appearing next to written text (buttons), emoji and stickers to go in the comment section as addition to text. ‘Reactions’ appear with their written meaning description. Although some symbols are used as a complete replacement of text buttons,
46 like the ones on the top bar and the Facebook logo that is used. Interacting with written text surrounded by, and in direct relation with, (moving) images becomes the new norm.
4.3 Case study #3: The person
If one wants to be active on Facebook, they need a Facebook profile. Expressing who you are through its features is what the profile is all about. Elements of this profile such as the profile picture and your name, is what represents a person across the platform. This next case study will be an analysis into how a person is being represented on Facebook through written text and images, over time. When Facebook went public in 2006, a person was represented through their profile picture, other pictures, their name and personal information, and amongst other elements, they still are. The functional affordances were simple. The options for a profile picture are plain: a person can upload one picture and that is all. The cognitive and sensory affordances are also still quite plain at this point, much resembling what a CV would look like. The name a person gives up through the profile settings appears on the right next to their picture, as well as on an extra bar under the top bar accompanied by “[…]’s profile” (see Image 4). Under this name personal details can be found, with quite official categories like ‘sex’, ‘birthday’ and ‘hometown’. Other pictures could be added to albums and viewed through a link under a persons’ profile picture saying “View more pictures of […] (# of pictures)”. A person is referred to on other parts of the platform solely by their name, that forms a link to their profile. This is with the exception of appearing amongst someone’s featured friends on their profile, with this the profile picture is also shown. The next year, the functional affordances do not change significantly. The options for the profile picture and name are the same. More information can be added to personal information however, like ‘interested in’ and ‘political views’. The number of friends shown in featured friends went up from three to six. In terms of cognitive affordances, the awkward bar under the top bar with someone’s name on it is removed (see Image 28). Except for the featured friends, a person is still represented only by their name appearing on other parts of the platform. A change in sensory affordances is that the name now appears in black instead of blue. When ‘The Wall’ is introduced in 2008, an option is added to write something about one’s self above the personal information box (see Image 6). The
47 functional affordances around the profile picture and the name are still the same. The cognitive affordances that changed are the about section that moves to under the profile picture. More importantly, a person is now represented across the platform with both their name and profile picture; this makes the platform instantly appear more visual-orientated. Other photos of a person can now also be viewed through a tab next to the ‘Wall-tab’ named ‘Photos’. In terms of sensory affordances, the profile picture is now horizontal instead of vertical. Overall, the profile is starting to look and feel less CV-like and less official.
Image 28. A Facebook profile page in 2007, not viewed from the user
In the following year 2009, not a lot of affordances change around the features of the profile. The personal information box is now named ‘Information’ at the top (see Image 7), which keeps the official tone strong at that time. In 2010 however, one year before the big Timeline redesign, the profile features already go through some changes. As can be seen in Image 29, a change in functional affordances is that there is an option now to add an extra name within round brackets at the top of a persons’ profile. This extra name will not appear when a person is represented on other places on the platform. All the affordances around the ‘Information’ section have changed too, with a lot of extra options to add, and a place back at the top of the profile. In terms of cognitive affordances, all the lines of the personal
48 information section now have a symbol in front of them, and the whole section looks less official and less structured. A big change that hints at what the Timeline redesign will look like a year later, are the new photos below the information section that are now clearly visible. These new visual elements appear right at the top of the profile, giving the whole profile a more visual look instantly in terms or sensory affordances.
Image 29. The Facebook profile in 2010, viewed from the user
Now with the Timeline redesign in 2011 (mandatory for every user in 2012), a very important new functional affordance pops up that will make the platform significantly more visual. The personal profile now offers the option to upload a cover photo that spreads all across the top of the profile page (see Image 30). Uploading one is recommended by the platform by the ‘add a cover button’ that is placed in the blank space. In terms of cognitive affordances, the placing of important items of the profile have shifted. The five featured photos on the top of the profile have now made place for different categories, such as Friends, Photos, Locations and more, all accompanied by images and their label in text. Someone’s photos can now easily be accessed from here. Also, in some cases the featured friends function has also moved to the top of the Timeline, now displaying eight friends (see Image 30). The information section is now called ‘About’. With this update, the profile
49 picture gets its square shape that it will keep for several years. With this new shape, a user can now edit their own thumbnail. The sensory affordances change in line with the functional affordances in that the functions that represent a person now feel a lot more visual too.
Image 30. A Facebook profile after the Timeline redesign in 2011, viewed from the user
This design of the Timeline changed in 2013 when the bar of visual tiles next to the about section is removed and replaced by tabs again, much like in 2009. The functional affordances around the representation of the person do not change much, but it is now possible to add textual descriptions and place tags to the profile picture. Both the cognitive and sensory affordances in many aspects move back to being less visual. The tabs with the features of the profile in it (Timeline, About, Friends, Photos) are written words only with no use of symbols. Until this day they still are. In 2015, it became possible to upload a short video clip as one’s profile picture. This function is especially important as a person can now be represented by moving images as a main part of their profile. This video will not play when a person is represented on other parts of the platform, only on their profile. The sensory affordance of this new function, seeing moving profile pictures, lets users know that video is now a more normal part of interacting with people on Facebook.
50 Another update on the profile picture in 2015 is that setting temporary profile pictures with a certain filter or theme becomes possible. This functional affordance started when same sex marriage became legal in the United States, and users could put a rainbow flag filter over their profile page. In under an hour, one million people participated. Another example is the French flag filter that became available in the wake of the Paris attack in November 2015. In 2016, multiple frames became available to choose from as a standard part of the feature. This has as a consequence that profile pictures across the platform become more graphic looking. This function was expanded when in 2017, the option to add a frame to their profile picture was explicitly offered to the user. In terms of sensory affordances, the user once again is exposed to an increase of visual elements in the representation of the person. Another change to the functional affordances around this time is that the five featured photos are back. They now appear right under the personal information section.
Image 31. The Facebook profile in 2018 with the new round profile picture
The classic square profile picture is being replaced by a circle in 2018 (see Image 31). Users can edit their thumbnail to ensure this new frame fits their profile photo well. This new roundness is also found in the new personal information section, that is now called ‘Intro’, where the globe symbol is placed next to. Adding a ‘bio’ is now possible, giving the user more space to express themselves using words. More featured photos are made possible to
51 pick in 2018 when the user can pick up to nine images instead of five, once again filling the profile page up with extra images. Overall, the biggest change in affordances of the profile in terms of text and images was the addition of the cover photo, filling a good portion of the screen with image content ‘above the fold’. Having discussed the low-level affordances of how a person is represented through Facebook features over time, the high-level affordances will be discussed in the next section. From the start of Facebook in 2006, what this analysis shows is a move away from the ‘official’ surrounding a person. With the profile page looking much like what a traditional CV would look like in the early years, year by year the features make use of more (moving) images and symbols, leaving the formal look behind. Profile picture filters and moving profile videos are a good example of that. The cover photo that came with the Timeline redesign takes away the attention from the non-visual even more. Categories of the personal information section like ‘sex’ and ‘political views’ are dropped whilst symbols are added. The number of photos that can be added as featured photos and the number of featured friends including their profile pictures that are shown keeps increasing over the years. With this the use of images to represent oneself is being normalised increasingly. Even the textual expression of the personal information section now appears with symbols and appears with increasing images all around it. It is clear that the image-shift is also taking place what the expression of the person is concerned. Although the number of images expressing the person varies from year to year and even goes down at one point, almost all of the updates concern expanding the use of (moving) images. The name and the personal information section do not see a very significant increase in affordances or amount of space they get, with the exception of it now being possible to add a ‘bio’. Another observation to highlight is the move to a more ‘rounded’ looking profile. This signifies another step away from formalness. In conclusion, all three types of affordances play a big role in the move to the visual of the representation of the person on Facebook. Especially functional and sensory affordances contribute however with more options to use images and more design choices away from the textual. The normalisation around the use of images in relation the person has clearly taken place on Facebook.
52 5. Discussion
Having had a good look at Facebook’s interface structure, it is now time to come back to the first part of this work and continue answering its central question to completion. The investigation into possible humanistic consequences of the relation between written text and images within a social media interface will start by looking into what humanism is. The term humanism dates back to pre-Socratic philosophers, whose writings of such topics were lost in Europe for centuries and later rediscovered by Renaissance scholars, translated from Arabic (Fordham University, “Islam”). In the Renaissance period the term was re-used within The Enlightenment movement to indicate that humans can be ethical and moral beings without taking part in any religion or believing in a god. I would argue this believe is now the norm for a large part of the world population, and certainly the believe on which most democratic systems are built. Humanism thus concerns itself with the ethical consequences of human actions and decisions. In relation to the research subject, the concept of humanism will be used as a framework here to investigate what possible ethical consequences the changes in Facebook’s interface can have. To come back to Neil Postman’s “The Humanism of Media Ecology”, it is clear what Postman thought of new media (in the year 2000):
As far as I can tell, the new media have made us into a nation of information junkies; that is to say, our 170-year effort have turned information into a form of garbage. My own answer to the question concerning access to information is that, at least for now, the speed, volume, and variety of available information serve as a distraction and a moral deficit; we are deluded into thinking that the serious social problems of our time would be solved if only we had more information, and still more information. (14)
This does not leave any doubts about what Postman would say about the social media content practices of today. Postman poses a final question: “To what extent do new media enhance or diminish our moral sense, our capacity for goodness?” (15). This question is clearly not to be answered in this work. But the answers to the question as to what extend the changing relationship between written text and the (moving) image within Facebook’s interface structure enhances or diminishes our moral sense, are to be explored.
53 Knowing what written text and images are, how they carry information and how humans respond to these two different modes of communication in a different way, we can begin to assess what effect their changing relationship could have within the new media environment. This next section will shortly summarise what has been discussed and concluded according to the theories laid out in the first part of this work on written text, (moving) images and their relationship. In part 2.1 it is discussed that written text works following a relatively unambiguous system to create meaning. It works following a temporal logic as language is very well suited to express things that happen over time. In section 2.3 written text is discussed as being created by man and therefore artificial; language is not innate to us. I have argued that written text does not draw our attention as much in a multimedia environment and that it does not possess a strong affective or engaging power. Written text often needs images to draw attention to its content. In part 3. the idea of ‘the logic of the screen’ is discussed in which is argued that written texts’ place on the screen is under the logic of the image instead of the logic of the page, where text used to belong. This indicates a loss of power and a lostness in general in terms of the position of text within the digital landscape. Now looking at what is argued about (moving) images in this work, in section 2.2 the ‘naturalness’ of images is discussed. In the way that they are all around us in this day and age and producing them is easier than ever before. It is also argued that the photographic image is a product of our obsession with realness and overcoming time and death. The photograph, and images in general, have become so omnipresent that they have become a type of language themselves. Understanding how their system of meaning-making works though, is a lot more ambiguous than how language works. In part 2.3 images are explained to be seen as ‘being natural’ as early as the third century BCE. It is also shown that images are used to draw attention to text in news media. It is proven that images are better at grabbing and holding our attention, and that images evoke a stronger affective/ emotional response than text does. In section 3. is discussed that images play a big part in shaping online identities. Images also operate within their own ‘logic’: ‘the logic of the screen/image’ within new media and this gives them a clear advantage in terms of awareness. In the next section the results of the three case studies will be discussed from a humanistic perspective in relation to the conclusions about written text and images above. The first case study on the status update feature on Facebook shows that the feature
54 has become increasingly more visual especially in terms of functional affordances. Looking at the results of the second and the third case study, the same rings true: in terms of functional affordances all features become increasingly more visual. So, what users can actually do with the feature has changed considerably in favour of the image over time. The status update feature lets users express their thoughts and feelings and share content, which now has been normalised to do through images. Emoji, stickers, symbols and GIFs are introduced to the interface, more and more, over time. Noticeably at a much slower pace than its early competitors like the Dutch Hyves, which indicates that people go simply where other people go. The affordances in that case do not matter significantly for the user, they use what they get. The third case study shows that the way the person is represented on the platform increases with functional affordances in favour of the image. This process continues over time until a large portion of the profile page is filled with (moving) images. All case studies show the same shift to the visual in terms of functional affordances over time. This is not to say that the options that already existed for written text got less, but rather that developing functional affordances in their favour did not occur significantly. As is discussed in the methodology section, Foucault’s notion of productive power plays an important part in this whole. Foucault’s work focusses on certain ‘institutions’ that carry out this form of power, in his day and age consisting of for example the government, prisons and universities. Nowadays, influential companies that work on a global scale are to be seen as such influential institutions. By making the sharing and interacting with images more available each year, Facebook is exercising grand productive power. And, to quote Uncle Ben from Spiderman on this suitable occasion: “With great power comes great responsibility”. This responsibility is not discussed and reflected upon enough within media studies, maybe precisely because this is a case of power being productive rather than restrictive. Facebook is, slowly but surely, changing the way we ought to communicate, promoting the visual, exercising its power onto billions of people. The question to be answered remains: what could be the consequences of this? Seeing that is concluded that the meaning-making of images is more ambiguous to decode than is with written text, the move to the visual in our online communication could have the consequence of making our communications quite literally mean less. This is with the exception of signs and emoji for example who have a pre-set meaning (like words do), that are used with the opposite goal of making the meaning of a text or context clearer. The
55 temporal logic of the written word is replaced by the spatial logic of the image, and spatial relations are not particularly oriented to causality (Kress 57). The written word even lives within this spatial logic of the image now. This could have as a consequence that we are missing crucial causal relationships and that truly understanding and putting down more complex ideas through language is no longer actively supported by the context language is now occurring in. Written text must become more image-like within the logic of the screen, as we have seen in the case studies. Before the logic of the screen and image existed, humans were used to reading within the logic of the page. From left to right (in our case) and from top to bottom. Every sentence more or less has the same form -the same design so to speak- so in order to extract the information you need or want, you must read or scan the entire text in this way. ‘Ctrl + F’ did not exist. There was an index and some sub headers and images here and there; this was the logic of the page that existed like this for centuries. Indeed, it does not sound that appealing. The logic of the screen and image ‘works’ differently. Within it, sentences are becoming shorter and less complex. More bullet points and symbols are used. As Kress puts it: “Seen in that light, the move from complex sentence syntax to the simpler sentence syntax […] gives away the means for constructing conceptually complex things” (58). On the question of what consequences this image-shift could have on the transmission of knowledge, the answer is that knowledge changes shape in different modes. The changing of shape within the multimodal design could thus have deep epistemological effects (Kress 51). The way we know things, and how we know them, is changing. Adding to this that written text does not hold the same attention-grabbing and affective power that images do, it seems the effects of these consequences will only be amplified. If a picture would be really worth a thousand words, we are going in the right direction with the image-shift on social media. Putting out a thousand-word long message is a lot. But this proverb stems from a time where pictures were consumed on more special and rare occasions. Now we take, send and delete pictures like they are nothing special. We throw them away like garbage. Facebook’s interface does not promote the putting out of long textual messages at all anymore. It promotes uploading pictures in a landscape where this action does not mean anything special anymore. We send a selfie over Snapchat that is deleted after ten seconds, is that action worth a thousand words? Or are we living in a time where sending someone a thousand-word text message would mean more? A profile video
56 of you, smiling somewhere on a beach, or a thousand-word long bio about yourself, what would mean more? I think the point here is clear. But is there not still a lot of text used within the Facebook interface? As the analyses show written text mainly occurs where cognitive and sensory affordances is concerned. They do not make the same heavy image-shift as the functional affordances do. The cognitive and sensory affordances; how users know what they can do and the design choices of these signs, have not become considerably more visual than textual in this case. It seems the cognitive and sensory affordances need the written word as a form of ‘anchorage’ (Barthes) to make sure they are being interpreted in the right way. A good example of this are the button labels that keep their words although symbols are added and the words that clarify the meaning of the ‘reactions’ when mousing-over them. Using written text has become a way to make certain images less ambiguous (personal tags, location tags, descriptions). In turn, certain images (symbols, signs, emoji) are made to give written text the context or ‘attractiveness’ it needs. Actually expressing one’s self with written words is becoming more rare. Maybe the ‘naturalness’ of images versus the written word that is experienced by us as being ‘conventional’, as Mitchell argues, has caught up with us. If we indeed experience images as far more ‘natural’ than text, it seems logical they fit the trend of media becoming more and more invisible, as Deuze shows. This computerisation, according to Manovich, will affect deeper and deeper layers of culture. In combination with what Bazin calls our obsession with the photograph and its ‘realness’, this can now come to a fuller expression. Adding to this, as discussed in part 2.3, that research from the field of psychology shows that we have a more affective response to images than text, it is to be concluded that our world is becoming more emotion-triggering. Written text, although it would probably never disappear within the media landscape, is clearly in a state of disempowerment. Facebook’s interface started out as a place where text was welcomed, even encouraged. Its competitors at that time were making use of much more images and visual possibilities. Slowly but surely the interface lost its ‘official’ look and made the move to being a more playful, image rich environment. It is trying to survive next to Instagram and Snapchat, which were visually orientated from the start. In this way our media world is becoming more shallow, more fast paced, less causal, more emotion-triggering and more ambiguous. This is not a good development in terms of humanism. The faster you go, the more you need to rely on (traffic) signs to quickly get a
57 message across. Essays next to motorways will not work. And vice versa, the more signs there are in your environment, the faster you think you can go. Either way, (traffic) signs are made to be unambiguous images, their meaning is made clear following a set system that you ought to know. For other images, photographic ones for example, this pre-set meaning making system does not function. The logic of the screen is the ideal home for the traffic sign and the photographic image. And here we are, lost between speeds and signs.
6. Conclusion
Living within something probably makes it harder to reflect on. As Mark Deuze says: “We no longer live with media, we live within them” (12, own translation). I see the irony of writing a work on text and images within the logic of the image. But the logic of the image is now virtually everywhere. Nevertheless, my work here shows a clear trend. The three case studies display this trend over time in orderly manner, resulting in a straightforward conclusion: the image-shift is real. And as argued, the consequences of this shift are too. In this work, the foundations of our understanding of the world have been thoroughly discussed: written text and the (moving) image. The first sections builds on the theory from multiple academic disciplines to lay a foundation for analysing Facebook’s infrastructure. Through text and images, we communicate, learn, consume, understand, love, and share. It is through them we are who we are. They serve as cornerstones for our culture and civilisation. Written text and images are two different modes through which we experience the world differently. We know that they complement each other well, and that their combinations have been used for centuries. This was under the logic of the page, meaning that the images that were used were subjected to its logic. That they have been freed from this structure, is probably a good thing. As observed, they thrive like never before. The shift to the logic of the image is therefore not necessarily a bad thing in and of itself. But a lack of understanding into the position of written text within this logic is. As I have argued, the changing relationship between text and images within Facebook’s interface structure has resulted in an unequal power balance in favour of the image. Foucault’s concept of productive power, together with the specific methodology making use of affordances introduced by Stanfill, served as a useful frame for the analyses. This unequal power balance is particularly noticeable when functional affordances are concerned. When
58 and where the written word is used in the interface structure, it is there mostly merely to serve as a type of clarification to the image. The analysis into the cognitive and sensory affordances show this clearly. We can see this shift happening on a much larger scale, within the entire new media landscape and beyond. And it is changing, slowly but surely, the way we think. Interesting connections can and should be made between this change and other changing aspects of society relating to for example the global mental health epidemic and global politics. Large corporations within the new media landscape through which we live our lives exercise grand power, one button at a time. Without it sounding too sinister, this power is shaping you and me, structurally. It is reshaping our textual and visual context. Questions of responsibility are shied away from, probably because the task at hand seems overwhelming. Facebook, Google, Amazon, Netflix, eBay, Uber, are they not our new world rulers? Have their platforms and their algorithms not more productive power over our lives than the restrictive power of the law? Academic research into these conglomerates in relation to the shift to the visual and its precise effects is scarce. Academic research into this shift within new media studies has not been conducted -outside of this work- within this humanistic framework. The possible humanistic consequences of this shift more and more into the logic of the screen lie in a certain lostness. A lostness of interpretation, causality and critical, deep thinking. A focus lies instead on what possibilities are added; what technological developments are gained. Of course, these developments have also caused good in the world. Technically, we are more connected than ever, but does our society show it? With each new function into the support of the image lies a real loss. A serious loss. A loss of the logic of the page, a loss that is to be acknowledged, to be seen and to be understood. This understanding will hopefully result in the realisation that written text is begging for attention, and that it is deserving of this. The written word will not disappear, instead it is doomed to shrivel within a context it has no home. It seems clear that more academic research needs to be done to explore the depths of this loss and its very real consequences for humankind. Within new media studies this calls for stepping off the beaten track and having a more multidisciplinary approach. As there are many sides to the text/image relationship, I would recommend that other academic fields such as communication, psychology, linguistics, information studies and philosophy should also be looking into this
59 issue with their specialised knowledge. After all, if we do not understand how we understand things, what is the point of research?
60 7. References
Ash, James. “Sensation, Networks, and the GIF: Toward an Allotropic Account of Affect.” Networked Affect. Eds. K. Hillis, S. Paasonen and M. Petit. Cambridge Massachusetts: MIT Press, 2015. 119-134.
Barthes, Roland. “Rhetoric of the Image”. Image, Music, Text. Ed. and trans. Stephen Heath. New York: Hill and Wang, 1977. 152-163.
Bateman, John. Text and Image: A critical introduction to the visual/verbal divide. New York: Routledge, 2014.
Bazin, André. “The Ontology of the Photographic Image.” Trans. Hugh Gray. Film Quarterly 13.4 (Summer, 1960): 4-9.
Bucher, Taina and Anne Helmond. “The Affordances of Social Media Platforms.” The SAGE Handbook of Social Media. Eds. J. Burgess, A. Marwick and T. Poell. Sage Publications, 2018. 233–253.
Businesswire.com. “More Denim, Netflix and YouTube; Less Handbags and Broadcast Media, According to Survey of 9,400 Teens.” 2015. 15 March 2019.
Clark, James and Allan Paivio. “Dual Coding theory and Education.” Educational Psychology Review 3.3 (1991): 149-210.
Clement, J. Statista.com. “Number of Facebook users worldwide 2008-2019.” 2019. 17 April 2019.
Deuze, Mark. Leven in media. Amsterdam: Amsterdam University Press, 2018.
“Discourse”. Cambridge Dictionary.
“F8 2011 Keynote.” YouTube. 24 September 2011. 17 March 2019.
61 Fordham University. “Islam.” 26 January 1996. 3 April 2019.
Flew, Terry. New media: an introduction. Melbourne: Oxford University Press, 2008.
Gibson, James. The Ecological Approach to Visual Perception. 1979. Classic Editions. New York: Psychology Press, 2015.
Hartson, H. Rex. “Cognitive, physical, sensory, and functional affordances in interaction design.” Behaviour & Information Technology 22.5 (2003): 315–318.
Highfield, Tim and Tama Leaver. “Instagrammatics and digital methods:
studying visual social media, from selfies and GIFs to memes and emoji.” Communication Research and Practice 2:1 (2016): 47-62
Horvat, Marko, Davor Kukolja and Dragutin Ivanec. “Comparing affective responses to standardized pictures and videos: A study report.” MIPRO Proceedings of the 38th International Convention IEEE (May 2015): 1394 –1398
“Image”. Cambridge Dictionary.
Jaworski, Adam and Nikolas Coupland. “Introduction.” The Discourse Reader. London: Routledge, 1999. 1-36.
Kress, Gunther. Literacy in the New Media Age. London and New York: Routledge, 2003.
Liddicoat, Anthony and Timothy Curnow. “Language Descriptions.” The Handbook of Applied Linguistics. Eds. lan Davies and Catherine Elder. Hoboken: Blackwell Publishing, 2004. 25-53.
Manovich, Lev. “New Media from Borges to HTML.” The New Media Reader. Eds. Noah Wardrip-Fruin and Nick Montfort. Cambridge: The MIT Press, 2003. 13-25.
Manovich, Lev. “The Practise of Everyday (Media) Life.” Critical Inquiry 35.2 (Winter 2009): 319-331.
Manovich, Lev. “What is new media.” The language of new media. Cambridge: The MIT Press, 2001. 18-55.
62 Miller, Daniel. “Photography in the Age of Snapchat.” Anthropology & Photography No. 1 (December 2015): 1-17.
Mitchell, Thomas. Iconology: Image, Text, Ideology. Chicago: University of Chicago Press, 1986.
“Picture”. Cambridge Dictionary.
Postman, Neil. “The Humanism of Media Ecology.” Proceedings of the Media Ecology Association Volume 1 (June 16–17 2000): 10-16.
Poulsen, Søren Vigild. “Becoming a semiotic technology – a historical study of Instagram's tools for making and sharing photos and videos.” Internet Histories 2.1-2 (2018): 121- 139.
Santini, Simone, Gupta Amarnath, and Ramesh Jain. “In Search of Information in Visual Media.” Communications of the ACM 20.12 (December 1997): 34-42.
Smith, Cooper. “Facebook Users Are Uploading 350 Million New Photos Each Day.” Businessinsider.com. 2013. 5 March 2019.
Stanfill, Mel. “The interface as discourse: The production of norms through web design.” New media & society 17.7 (2015): 1059–1074.
Yadav, Aman, Michael M. Phillips, Mary A. Lundeberg, Matthew J. Koehler, Katherine Hilden and Kathryn H. Dirkin. “If a picture is worth a thousand words is video worth a million? Differences in affective and cognitive processing of video and text cases.” Journal of Computing in Higher Education 23.1 (2011): 15–37
63