What we need is a grammar of understanding, a way of describing how meaning making occurs in and around videogames. We’ve already made some progress across various disciplines and discourses, from design and art theory to cultural studies and educational applications. Often these approach games by examining what they mean. I care less about what videogames mean than how they mean it. I’ll leave meanings to critics; I care about the ways they mean, the perceptual features, the structures that support the creation of meaning. These are questions which inform both the making and playing of games. Indeed, I feel strongly that questions of meaning serve as a unifying space for both ends of videogames, design and play. Often it’s easy to treat these as wholly separate dimensions, where play is simply an emergent property of the design of the game, or where players interact with a “text” that comes pre-packaged and is simply decoded. I think this stems not from ignorance of the interdependent nature of design and play but in issues of access: critics from either dimension often lack effective ways to interact with and interrogate what the “other side” is doing, from the actual process of design and development to the multitudes of ways to play. Instead, it is far more productive to conceive of games as both designed and enacted.
This is why I feel how meaning happens (or doesn’t) is so important: it provides a more robust vocabulary across the practices of making games and enacting them. For designers it’s important to understand the conventions of design but also the processes of meaning making in order to more effectively create specific ways of not just performing but understanding. For players, it’s critical to know how they make meaning themselves but also how designers choose the forms and signs of the game to more fully reflect on their play and to also begin to position themselves as “designers” of their own experiences in games and the everyday. This last point is perhaps the most important, if not beyond the scope of this essay: we face tremendous challenges in the coming century, and engaged global citizens will have great agency—and responsibility—to design their experiences and their own solutions. The process of promoting a change in their worldview beyond passive reception to active participation is certainly happening in many domains; videogames provide another form of conceiving of the world in this way, and one which appears to be an exceptionally effective one (see Barab et. al., 2010 for example). None of this is possible without knowledge of how to make the world meaningful and how to use this meaning for some purpose. Meaning is the precursor of action.
But meaning is an incredibly broad concept. Even in a videogame, with constraints on the scope and scale of the experience, meaning happens across a broad range of domains, from kinesthetic feedback to textual interfaces to aural cues to the mechanics of the game itself to the social spaces in which the games is played and shared, and many more. As a dominantly visual medium, however, videogames offer a particularly rich space for discussing the various methods for creating and negotiating meaning. This is a relatively unexplored space, yet it is one which both informs design and influences play. Developing a way of talking about meaning making across a range of visual features is both a first step towards a comprehensive vocabulary and an invitation to open this space for analysis and refinement.
The scope of this essay dictates that this first step should be a small one. I’ve outlined three different perspectives when talking about visual features. There are many more. This is not an attempt to capture all the various visual features; rather, it is an outline for how to talk about how to talk about them. That is, I intend that this model for interrogating the semiotic features of a broad range of videogames’ visual elements is a chance to develop a practice of looking at games as vehicles for meaning making. The perspectives outlined here, and the tools I’ve provided for performing a visual analysis of games both in general and specifically, serve as a building blocks for a fuller dialogue.
Videogames and the Work of Meaning
What do I mean by “meaning”? I mean that players have to make sense of the various conceptual domains they inhabit in order to understand the actions they take and their purposes in taking them. Already it should be clear from my use of the term meaning that things don’t simply mean something in and of themselves; they mean in order to be used for some purpose. Existential crises aside, this is how humans view the world, and make it meaningful to them. This does not discount what Searle (1995) terms “brute facts” about the universe; instead, there is much about the universe that we have no knowledge of and no way to meaningfully interact with it. X-rays serve as a good example. X-rays have been bombarding us for billions of years; they have produced genetic mutations and other phenomena. Until the very recent past, though, humans had no way of knowing about them; they were “meaningless” in that we had no sense of them and no idea of how to use them. Since their discovery, we’ve adapted them to several uses, from looking at the human body in new ways to understanding how energy moves in space and more. Perhaps someday we’ll use them to trigger genetic mutations that we control. The point is that X-rays have always existed; they only became meaningful when we could adapt them to some purpose. Meaning is an ascribed feature, not an inherent state.
Since games are designed spaces, it’s important to think about how designers can promote certain meanings towards some preferred action. I say “promote” here since this analysis follows a social semiotics approach (Kress, 2010; see also Halliday, 1978). Social semiotics suggests that signs (and videogames use many of them, as we shall see) are created with intentionality, but are only ever interpreted, and that there is no set or fixed meaning in a given sign. A designer chooses the most “apt” sign that fits their intended purpose; the interpreter brings with them a lifetime of ideological frames and experiences and assigns some meaning to the sign; indeed, Kress suggests that these are actually two different examples of sign-making, one by the designer of the sign and one by the interpreter of it.
Importantly, sign making is a “motivated” act, where the “sign-maker…brings meaning into an apt conjunction with a form, a selection/choice shaped by the sign-maker’s interest” (Kress 2012, p. 62; emphasis in original). But this “apt” fit must be recognizable to the user of the sign, and there is inherently some chance that the interpreter will make a completely unintended meaning for themselves. A fork provides another example. It is a designed object, a “motivated” construction. The culturally conventional “meaning” of a fork is as a tool used to eat food. But I could just as easily use it to stab someone, to pry open a stuck window, or as part of a piece of art. The fork is a physical thing, yes; it even has some motivated, intended design. But what it “means” depends on how it is actually used. Some signs become so conventionalized that they become a storehouse for common cultural values (a rose for love in most Western cultures); others become quite contested and take on very different meanings to different people (the Confederate flag, for example).
What are these “interests” of the designers of the signs (and the game)? Certainly, games are meant to be played, so one interest is providing a meaningful space for the experience of the game. But games are (generally) about something in varying degrees. Designers must make meaning happen across several spaces, then: they need to make playing the game meaningful (using the tools of the game) as well as the context of the game meaningful (what is being done and how). I’m treading carefully here; there are definitely games that do a poor job of this. The famous case of Atari’s 1982 E.T. comes to mind. While much of the problem was the game’s broken mechanics, I’d argue that those mechanics are tied to an equally faulty representational system. There are plenty of other examples; outliers are interesting and worth looking at in a more robust analysis. For now, I’ll restrict myself to mostly “good” examples of aptness and fit when talking about the design of sign systems.
The motivation of a designer, then, is two-fold: to help players understand what they are doing and why they are doing it. They create signs which help players actually play the game. More directly these signs work to both orient the player to the game world as well as to teach them when to call on specific information (Holmes, 2012). Here I’d like to extend this argument to include orientation to the tools available and the narrative context for their actions in addition to visual elements like the avatar, the gameworld and data/feedback. Orientation takes several forms: features like framing, point-of-view, salience, composition, and others provide important clues on how the to make the images meaningful. Orientation is to help the player both make informed choices about how to act and contextualize their actions within the game. Orientation is a way of structuring the experience so that players learn to call on the appropriate information to achieve their intended actions. So a designer wants to create signs that push players towards understanding how to play the game and then giving them the space to do so.
What of the “interests” of the interpreter (here, the player)? Playing is also a “motivated” act in which the player wants to work towards some goal. They need to be able to see the game, make some sense of what they are seeing, and then take actions which should re-trigger the chain of events again. At a deeper level, though, players should realize not just what their affordances for action are (Gee, 2003) but why they are taking them and have some access to understand what effects their actions had on the game. They need to be able to contextualize their actions effectively and, just as importantly, efficiently to keep the game moving (Holmes, 2012). That is, what they see is not an isolated event but a sequence of iterative interactions with the orientational structures outlined above. They play the game, and as they play, they refine their understanding to keep progressing in their play. Importantly, players are not blank slates that come to the game and make meaning from it whole cloth; rather, they have experiences with other games, other media, and other semiotic resources that inform their choices. Further, they have a particular way of seeing the world that informs any meaning that they create from the semiotic resources they encounter. This worldview may or may not be compatible with that of the designer; creating meaning from the semiotic resources may be difficult.
This is an example of a gap in both convention as well as in ideologies. It further reinforces the idea that these visual features are not only interrelated but interessential. You can’t isolate one feature from the others clearly, nor can you discount the effects they have on each other. The perspectives below highlight this interdependence more clearly. For now, it’s important to recognize that these perspectives neither exist by themselves nor occur without the influence of other visual structures. The usefulness of the tools I’ve provided is in trying to foreground a particular perspective, but always with this conceit in mind.
Tools and Stories
One way to think about games is that they are both operating systems (or tools to accomplish some work) as well as stories (the authored narrative, the fictional world, the contextual space, as well as the players directly enacted story) (see Bogost, 2006). Following this model, then, a game contains two primary visual domains: the interface and the story space. By interface, I mean things like icons, mini-maps, action buttons, pop-up text, chat windows, menus, avatars, even the world itself – essentially, all the things a player can use to do the game. These are the operative features but also things like status markers and orienting information; they are the OS of the game in many ways. By story, I mean both the narrative context in which the game occurs (the characters, the world, the objects that populate it and so on) as well as the space for the player’s story (where they went left, where they died repeatedly, etc.). It should be clear that there is significant overlap between these domains: an NPC in World of Warcraft is both a story element (it’s a night elf merchant) and a tool (it sells items to the player). How a designer balances these elements provides insight into how to make their inclusion meaningful, and how to use them.
One way to analyze the visual semiotics of videogames, then, is to look at how these two domains are represented. This can take many forms (as we shall see), but perhaps the most immediately useful way is to consider how apparent (or not) the tools/interface features are, and how apparent (or not) the story-space is. We could graph the features (see Figure 1).
A game like Angry Birds has very little tool space visible during the main play sequences (a distinction I’ll return to shortly). Most of what a player sees is the space in which the action takes place, with some indications of what the tools might be. Again, Angry Birds demonstrates the interconnectedness of many interface elements with story-space: the birds are the objects players use to play the game; they are both characters in the story and tools of the game. Nevertheless, Angry Birds might be categorized as Low Tool/High Story and appear in Quadrant 1 of our working graph. A game like StarCraft II, on the other hand, has a very rich and elaborate interface element present during the main play sequences as well as detailed spaces for the stories of the game to unfold. StarCraft II might be categorized as High Tool/High Story and appear in Quadrant 2. Other games can be mapped accordingly.
Mapping games based on these criteria might demonstrate patterns on how different genres align; many RTS games likely are High Tool/High Story, while FPS games might be Medium Tool/High Story, and puzzle games may be High Tool/Low Story. Genre groupings play a large role in the Conventions perspective described below by helping players call on previous experiences within the genre to predict and act on expectations, and for designers to use forms that follow these genre conventions as a kind of cognitive shortcut. For this perspective, however, it helps formalize those distinctions that make the genre itself. That is, the conventions of the tool/story perspective help form what can be considered a particular genre. More directly, how the game balances the tool/story elements is a critical part in understanding what the game is and how it means.
A High Tool space indicates several important clues for making the screen meaningful for players. The inclusion of the tools suggests that they are important, since players see them at all times during play (or, at the least, in a just-in-time/on-demand manner [Gee, 2003]). Their presence indicates that they might be important to the player. How and where they appear similarly influence how players can attach meaning to them; a small button tucked away in the corner of the screen might be easily missed or appear unimportant, while a large, flashing icon in the middle of the screen is more noticeable. While this again encroaches on notions of convention (both within games and in other visual media), it nevertheless demonstrates that the use of these features helps indicate significance and an opportunity for a player to do something (react to incoming fire, talk to another character, open a map, access system menus and so on). Designers can use the elements to “point to” important or meaningful elements for the player to notice and use in their play.
This kind of mapping can also be applied to different parts of the same game. Angry Birds is a Low Tool/High Story game, at least during the main play sequences. Interface elements appear in greater importance and quantity at other times (the completion of level where a score is assigned, for example). Games are not static, and most modern games have a number of different types of visual resources they employ during different segments. Deus Ex: Human Revolution provides a good example here. During the main sequences of play, the player navigates a three-dimensional world. Players see this world, the people in it, the buildings and spaces, the objects of the world; they are “in” the world when navigating it. There is also a persistent interface overlay with information about health, ammo, action icons, and a mini map, among others elements. Players have direct access to some status information and some action capacities on top of the story-space. However, Deus Ex also includes a number of pre-rendered cinematic cut scenes where there is no interface to speak of (save some conversation choice trees which I’ll ignore for a moment). Deus Ex also has a stylized menu system for assigning talent and ability points, manipulating the inventory and other procedural elements. These moments are primarily interface driven in that players care more about the tools they are manipulating (sorting their weapons, assigning points); the story is still present in that these icons fit within the fictional narrative world but only to keep the metaphors of the game cohesive enough that players can use them effectively. Players don’t care much about the story when they are in the menus, only enough to keep the fiction of their actions cohesive. Similarly, players don’t care much about the tools of the game while watching a conversation cutscene play out, only enough to direct the conversation at certain points.
So in one game, different moments can map to different quadrants of the graph; during the main play sequences, players encounter Medium Tool/High Story (Quadrant 2) moments, while cutscenes have Low Tool/High Story elements (Quadrant 1), and menus and inventory screens have High Tool/Low Story elements (Quadrant 3). This suggests several important things. First, the balance of Tool/Story changes depending on what players are doing. Second, that understanding how these elements appear can help designers to create an “apt” fit in what they are representing and what they want the player to be doing. Third, in a similar manner, players can learn what is important and what they should focus their attention on by considering how and when each of these elements appears and with what emphasis. Learning to play the game requires knowing when and how to act; the presence and emphasis on tool and story elements can help players know what information is meaningful and when they should be using it. Of course, this implies that players progress through the game by repeated interactions with the visual resources and learn over time what is important (Holmes, 2012). Indeed, as I’ve argued with World of Warcraft, players have access to lots of tool information and story information simultaneously, but they may only call on a portion of it at any given time. After many hours of interacting with the space, they learn what to call on and what to ignore situationally. This is partly a feature of the just-in-time/on-demand nature of some interface elements like damage text or inventory. But it’s also a matter of learning when and why they should call on a given semiotic resource for whatever purpose they want to accomplish.
Tool/Story Perspective: For any game, ask how the elements of the interface appear on the screen, and how the world in which the stories (player’s and game’s) appear. What is happening at a given moment in the game, and how are these elements balanced? Is there a correlation between the types of actions you take and how the game presents its visual resources? What expectations do you have on how to act based on the alignment of these features?
Conventions Old and New
Videogames have a 40-or-so-year history; there have been many thousands of videogames made. Videogames are also part of a longer heritage of visually representational media like TV, pictures, and paintings among others. As such, there are some elements which cross these various media and become conventionalized. Some elements appear in many games and so too have become conventionalized. What do conventions do? One way of viewing conventions is as a way of taking a shortcut to meaning by normalizing the sign so that I can act quickly. I must constantly balance the detail of the information I process with the speed in which I need to act; if I focus too closely on the number of stripes on the tiger chasing me, I might not be able to actually get out of the way in time. Conventions are a way to help me quickly notice, assign a particular meaning, and act on that understanding. Importantly, conventions are socially constructed, which will come into play more in the next perspective, but which again reinforces the notion that these perspectives all influence the others.
In videogames, a common convention is the health bar. Many games include a meter for understanding the player’s status, which in turn helps drive how the player acts; a wounded player may move more cautiously or be unable to progress to another part of the game at all. Since many games use a health meter, when I play a new game that includes one I have a quicker time knowing where to access this information and when to call on it, and thus have a greater likelihood of effectively playing the game. I can quickly make the sign (a green and red horizontal bar in the corner of the screen that changes the proportions of the colors) into a meaningful representation of my status. The convention gives me a shortcut to guide my actions more efficiently since I don’t have to spend much time recognizing the sign, attaching a meaning to it and then using it. Players without this prior knowledge of the convention may not know what the sign is and may need to spend some time learning what it might mean. This might hold them back or slow their progress or, worse still, never become a meaningful sign to the player and they may never be able to play the game at all.
Health meters also are often accompanied by a cross icon, either next to the bar or as the bar itself. Other meters might use a certain number of heart icons to represent health. These games are calling on multiple conventions from both other games as well as other signs in the everyday world to help players take a shortcut to the meaning of the sign. Crosses have a longstanding connection to health through things like the Red Cross and the doctor’s sign. Similarly, health can often be represented by a heart (on the front of a “healthy lifestyle” cereal or on many medical products). Designers can call on these conventions of the everyday to provide another framing tool to help orient the player to the intended meaning of the sign of the health meter. Of course, crosses can also refer to Christianity, and hearts can mean love. Players may look at a meter and assume it is a “love meter” of some kind. Here, their meaning mismatches the intended one. Game conventions can help minimize this risk; not only is it a heart icon pulled from the “real” world, but it is also a feature of many games that indicates status. Knowing both conventions (the everyday, the game) provides a reinforcing structure to ensure that the designer’s intended meaning is made by the player to help them through the game.
Games use many other features that straddle this line between game-specific conventions and conventions from other media and the everyday. For example, many games use light to help point players to important or useable features of the world. Portal has many instances of a light shining specifically on a certain part of the room to indicate where a portal should be placed; Fallout New Vegas uses light to emphasize useable computer terminals or the entrances to rooms and buildings. There are many other examples. These are elements that occur in many videogames, but which are also tools used in photography and painting as ways of emphasizing important parts of the subject matter, as ways of suggesting the importance (or lack thereof), as methods for composing the frames of the image (see Kress and van Leeuwen, 1996). These are ways of orienting players to what is important both by using them across many different games but also calling on a history of these framing devices. Designers can promote meaning by calling on these conventions and designing signs which match the expectations inherent in them. They can also use convention to undermine the player’s expectations (a game like Perspective provides an interesting case, which uses elements of 2D and 3D perspective to change how the player sees the world). Similarly, players can bring the conventions not just of the everyday, and not just of games in general, but in the genre of the game (or of a particular designer like Blizzard of BioWare) to form quick shortcuts on how to assign meaning to the various signs they encounter and how they play the game.
Conventions Perspective: For any game, ask what conventions are called on to make the various elements of the interface and story-space. Are there correlates in other games? In other genres? What conventions are used from other media? How do these conventions relate to the actions performed in the game, and in the story of the game?
As I mentioned earlier, players don’t come to a game as blank slates but as actors with lived experiences and ways of seeing the world. When they play the game, they play through their experience. A feminist player sees the game differently than a post-structuralist. They bring different views of the world to the game, different assumptions about the way the world operates. We don’t have to limit this to academic worldviews, certainly. A teenage girl sees a game differently than a father; a teacher sees it differently than a critic. The point is that any gameplay rests at the intersection of what the game is (the code that displays) and what the player already believes. Further, the game is not ideologically free either. Games are designed experiences (Squire 2006); they are made by people who themselves have world views and embed potential interpretations within the game. The player enacts it, only insofar as they align their own worldview with the “text” of the game.
One way to think about visual analysis is that both making and “reading” images happens at three levels: the perceptual, structural, and ideological (Serafini, 2010). Most importantly for this view, the reader looks inward through these levels; that is, their first frame is always an ideological one which informs the structures they notice and even the things they perceive. So I, as a middle-class, white male academic may never have noticed that all the restaurants in Grand Theft Auto: San Andreas are unhealthy fast food chains (except for the “nicer” restaurant available on dates). The stimuli were there, and I knew I was in a pizza or burger shop. But I never attached any more meaning to the fact that a game that occurs in a very low-income city has only fast food to eat. Someone with a different world view may attach that meaning. I never “noticed” the stimuli in this way, even if I saw it.
Just as importantly, since games are made, it’s important to consider the ideological frames that the designers bring. As Sturken and Cartwright argue, images “are produced within dynamics of social power and ideology” (2001, p. 21). They are indivisibly created through these ideologies. They are often the product of many, many voices (particularly in AAA games but even in small design teams much negotiation undoubtedly occurs), but still come together in the form of the game proper as a cohesive expression of that ideology. Even more importantly, the signs chosen by the designers work to reinforce this ideology; not only is a game usually about something, but the way it is represented is an expression of those beliefs. So Grand Theft Auto: San Andreas’ designers deliberately chose the representations they did for the restaurants, which can belie their ideological motivations for creating those signs. Journey specifically does not include another player’s avatar name when playing co-op. The designers here are both playing with the convention of identifying other players through their names well as indicating an ideologically motivated choice about representation; here, one which suggests that your companion’s identity is not important but that what you do along your journey is. By deliberately not including this information, the designers signal what they intend to mean and provide a clue to where players should work to assign meaning.
Ideological Perspective: For any game, ask what ideological frames do you as a player bring. What do you perceive, and how did you know to look at it? What choices did the designers make in picking the “aptness” of their signs? What ideological frame might this indicate? How do these ideological frames influence the ways you act in the game?
It should be very clear that this analysis leaves out many important ways of talking about games and the structures they use to convey meaning, even when restricted to a visual analysis. More work needs to be done in both identifying other useful perspectives and in applying them to specific cases. The perspectives included here are not exhaustive, but are meant to serve as a model for a more robust analysis and the beginning of a larger conversation. Creating a framework for analyzing the semiotic resources used to make sense of the game and the actions of play is an essential next-step in game studies, one which opens new avenues of analysis and critique as well as new discussions on how to best create and experience games more broadly.
Angry Birds [Computer software]. (2009). Espoo, Finland: Rovio Entertainment.
Barab, S., Gresalfi, M., and Ingram-Goble, A. (2010). Transformational play: Using games to position person, content, and context. Educational Researcher. 39.7: 525-536.
Bogost, I. (2006). Unit operations: An approach to videogame criticism. Boston, MA: MIT Press.
Deus Ex: Human Revolution [Computer software]. (2011). Tokyo: Square Enix.
Fallout: New Vegas [Computer software]. (2010). Rockville, MD: Bethesda Softworks.
Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York, NY: Peter Lang Publishing.
Gee, J. P. (2007). Good video games and good learning: Collected essays on video games, learning and literacy. New York, NY: Peter Lang Publishing.
Grand Theft Auto: San Andreas [Computer software]. (2004). New York: Rockstar Games.
Halliday, M.A.K. (1978). Language as a social semiotic. Baltimore: University Park Press.
Holmes, J. (2012). Seeing action: A visual analysis of World of Warcraft. GLS 8.0 Conference Proceedings. Pittsburgh, PA: ETC Press.
Journey [Computer software]. (2012). Los Angeles: Thatgamecompany.
Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. New York, NY: Routledge Press.
Kress, G. and van Leeuwen, T. (1996). Reading images: The visual grammar of design. New York, NY: Routledge.
Perspective [Computer software]. (2012). Redmond, WA: DigiPen Institute of Technology. (unreleased).
Portal [Computer software]. (2007). Bellevue, WA: Valve Corporation.
Searle, J. (1995). The construction of social reality. New York, NY: Free Press.
Serafini, F. (2010): Reading multimodal texts: Perceptual, structural, and ideological perspectives. Children’s Literature in Education (41): 85-104.
Squire, K. (2006). From content to context: Videogames as designed experiences. Educational Researcher (35.8): pp. 19-29.
StarCraft II [Computer software]. (2010). Irvine, CA: Blizzard Entertainment, Inc.
Sturken, M and Cartwright, L, (2001). Practices of looking: An introduction to visual culture. Oxford: Oxford UP.
World of Warcraft [Computer software]. (2004). Irvine, CA: Blizzard Entertainment, Inc.