A Framework for Cloud Aesthetics in Mixed-Realities


Dr. Troy Innocent
Senior Lecturer, Games and Interactivity, Swinburne University of Technology, Melbourne, Australia
Email: tinnocent@swin.edu.au
Web: http://troyinnocent.net/
Reference this essay: Innocent, Troy. “A Framework for Cloud Aesthetics in Mixed Realities.” In Leonardo Electronic Almanac 22, no. 1, edited by Lanfranco Aceti, Paul Thomas, and Edward Colless. Cambridge, MA: LEA / MIT Press, 2017.
Published Online: May 15, 2017
Published in Print: To Be Announced
ISSN: 1071-4391
ISBN: 978-1-906897-62-8
https://contemporaryarts.mit.edu/pub/aframeworkforcloudaesthetics


Abstract

Augmentation of urban space generates mixed realities, producing hybrid spaces that emerge from ecologies of virtual and actual signs and entities. This essay presents a framework for different aspects of fiducial markers that have been used within a series of street games. This augmentation of cities reconciles repositories of data with the city that generated them by ‘actualizing’ the data in situ, often making the experience of the city more game-like. A fiducial marker, often an object located in space, visible to both machines and humans, tethers a cloud of data to its location, connecting flows of information and players. The street games explore multisensory feedback generated by the markers in a series of public artworks in Ogaki, Istanbul, and Adelaide that connect street art, formal abstraction, augmented reality, and game design. Seven different aspects of fiducial markers are identified in this discussion. These aspects are analyzed by applying four different models: Lévy’s flattening of the virtual/actual hierarchy, Galloway’s “gamic actions,” Bogost’s “unit operations,” and Harman’s “quadruple object.” This analysis is centered upon the ways in which machine and human factors play equal roles in the construction of meaning, and makes it possible to explore a framework for analyzing the ontologies that emerge in mixed realities, blending fiducial markers, digital entities, human players, and public space in clouds of data.

Keywords

Mixed realities, cloud aesthetics, pervasive game, fiducial markers, public art, Ludea, transmedia, electronic art


Introduction

How can fiducial markers be experienced in new ways via an exploration of cloud aesthetics? The cloud is a kind of media ecology—a complex, adaptive system. [1] Arguably, the modern city is an expression of these systems in action. Fiducial markers are visual patterns used in augmented reality (AR) machine vision systems to recognize objects and places. My recent public art practice has developed an approach to fiducial markers as aesthetic objects, exploring what happens to these markers in a mixed reality, a type of media ecology that blends AR, games, code, and urban space. As will be explored in detail, this practice is focused on decoding cities, revealing the multiplicity of fiducial markers. Situated in the Micronation of Ludea, [2] this mixed reality invited players to recode and reinvent their urban environment through real-world street games in Ogaki, Istanbul, and Adelaide.

Figure 1. Troy Innocent and Indae Hwang, noemaflux 2011. Urban art environment at ISEA2011 Istanbul, Turkey. © Troy Innocent, 2011. Used with permission.
Figure 1. Troy Innocent and Indae Hwang, noemaflux 2011. Urban art environment at ISEA2011 Istanbul, Turkey. © Troy Innocent, 2011. Used with permission.


In 2011, one of these games was played on the streets of Istanbul as part of the ISEA exhibition program. The project is described as follows:

noemaflux describes an act of shifting perception. In this work it is centered on an augmented reality that enables different ways of seeing the city. This experience is constructed via a network of relationships that connect fiducial markers, urban space, generative writing systems and abstract virtual spaces.”

To make this kind of mixed reality experience possible, physical tags, called “fiducial markers,” were placed in the city. A fiducial marker, often a material object located in space, equally visible to both machines and humans, tethers a cloud of data to its location, connecting flows of information and players. The noemaflux project focused on the characteristic multiplicity of each marker—the different sets of relations that each had with its environment, its habitat, players, and digital entities.

In the context of a street game such as noemaflux, the marker becomes a particular type of glyph, ideogram, symbol, or ‘IdeoTag’ [3] used to construct places. The meaning of each glyph emerges through play and experience. It is a unit of language; a complex signifier that finds its ideal habitat in mixed realities. Each of these fiducial markers may operate as a machine-readable code, a game token, or as an element of a wayfinding system.

Multisensory Mixed Realities

In noemaflux, fiducial markers map out space on the streets. Rather than being hidden or embedded, the markers appear as an overt part of the urban space. They are distinct enough to be noticed by players who are looking for them, but to others, they appear as fragments of street art. In this way, the markers operate as a kind of reverse psychogeography: once found, they tether their location to the mind of the player. The sites are identified by whoever stages the game (often the artist) and are usually selected for their appearance or a feeling of being a ‘ready-made’ game location; they look like sites of action, transaction, or interaction.

Figure 2.  noemaflux:ia, Troy Innocent, 2013. Multisensory experience of fiducial markers. © Troy Innocent, 2013. Used with permission.
Figure 2.  noemaflux:ia, Troy Innocent, 2013. Multisensory experience of fiducial markers. © Troy Innocent, 2013. Used with permission.


In the most recent iteration of this project, noemaflux:ia, the markers, the markers are discovered by two players who activate and shape a musical score that is both heard (via headphones) and felt (via triggered vibrations on a wearable device). Building on the relationships between street art, mixed reality, virtual art, and gaming established in earlier versions of noemaflux, [4] the latest iteration connects sight, sound, and touch in a synesthetic relationship. This is in contrast to the primary mode of operation for mixed and augmented realities, which are primarily concerned with visual data. Although embedded in the environment, this exclusively visual mode still places the viewer at a distance from information—one looks at the cloud, rather than being in the cloud.

In addition to being visual and navigational tags, the markers and their locations are also elements of a musical score, a kind of graphical notation distributed throughout the city, waiting to be played. There is a long history of graphical notation that originated in the avant-garde of the early twentieth century and was connected with an interest in synesthesia [5] and the phenomenological experience of art. Embedded in the origins of formal abstraction were early systems, such as Point and Line to Plane[6] and the Light-Space Modulator. [7] Later, the electronic music pioneers Stockhausen and Xenakis employed notation [8] in their scores to express sonic elements not present in traditional music. noemaflux:ia transcodes these formal systems into a mixed reality using overt fiducial markers as elements of a graphical score embedded in an urban environment.

noemaflux:ia establishes a multisensory engagement with mixed reality that combines sight, sound, and touch. Primarily, visual feedback takes the form of fiducial markers, as they are aesthetic objects in themselves; however, feedback is also experienced via the color-coded patterns that appear on the iPad’s screen, generated by scanning each marker. Audio feedback manifests via the sonification of the code embedded within each marker. Touch is expressed as a rhythmic vibration, in time with the beat of the score being generated by the markers.

The fiducial marker systems devised for these experiences offer opportunities to make the cloud visible by blending the city around us with invisible, digital realms in a mixed reality. To make this possible, the machines need to be taught to see. Machines do not see the way we do, and have their own view of the city. Tracking; decoding; matching; searching; mapping. These constitute a set of operations for processing visual complexity and searching for patterns. A machine may detect a range of patterns in order to navigate – the edges of buildings, road markings, and structures. Traditionally, AR has employed fiducial markers to calibrate machines with their environment, employing them to provide points of reference in order to make spatial calculations and identify locations and objects. Fiducial markers first appeared as clusters of giant pixels, signifying code embedded in the city; the markers have evolved to include just about anything with a unique visual pattern that may be recognized by a machine.

Fiducial markers allow places, things, and people to be tagged and made visible to machines, and position the machine in a new relationship with the city. Although the goal of most commercial AR systems is to render these markers invisible, a more interesting—and transparent—mixed reality can be constructed by making these codes visible; by making them vulnerable, tangible, and overt.

This raises questions about how mixed realities are experienced; for example, their phenomenology, especially when framed by a play experience. To make a game, we often need to define an ontology; how do we explore that ontology, particularly the how it shifts across a mixed reality? Noemaflux:ia blends three different aspects in its design in order to establish a multimodal, multisensory, mixed reality experience. First: visual notation, drawing, and other symbolic systems for the notation of sound and music. Second: asemic writing, a form of writing with no specific semantic content. Third: synesthesia, a neurological condition in which sensory inputs are blended (for example, image and sound), and which has become a popular approach to constructing digital media experiences.

Seven Aspects of Fiducial Markers in noemaflux:ia

Using a multisensory approach in noemaflux:ia, several new aspects of the fiducial markers became apparent when compared with the earlier game, noemaflux, situated in Ogaki and Istanbul. [9] In noemaflux:ia, nine overt fiducial markers designed as elements of a graphical score were embedded into the Adelaide CBD. The iPad app that hosted the experience was configured as a sixteen-step sequencer, set to loop. As the player found and activated each marker, each step of the sequence was filled with the sound of the corresponding marker, accompanied by a visual effect and label. The label identifies the Ludean guild connected to the marker: Crossmedia Ecologist, Master Codemaker, Locative Urbanist, etc. Over time, an audiovisual sequence plays through the iPads carried by the players as they search for further markers, generating sound as they go.

Figure 3.  noemaflux:ia, Troy Innocent, 2013. Seven aspects of a fiducial marker. © Troy Innocent, 2013. Used with permission.
Figure 3.  noemaflux:ia, Troy Innocent, 2013. Seven aspects of a fiducial marker. © Troy Innocent, 2013. Used with permission.

Noemaflux:ia was created during a residency at the University of South Australia, leading up to an exhibition at ISMAR2013 in the central business district of Adelaide. Below are seven aspects of the role that the fiducial markers come to play and the ways in which they are experienced:

  1. Wayfinding markers: As they are stylistically consistent and typically located within short distances from one another, the makers signify the presence of an abstract signage system.

  2. Machine code: The device sees the marker as a visual pattern by identifying points of difference in the image.

  3. Graphical notation: Each marker has a collection of sounds and micro-compositions mapped to it that play—quite loudly—as the markers are activated on the street.

  4. Nonobjective art: Each marker is equally at home in an art gallery; they are aesthetic objects in themselves.

  5. Material object: Many fiducial markers are flat images that look like machine codes. These are tangible objects that blend into their environment as street art.

  6. Game token: Activating markers on the street locates you in space, increments scores and other data, changes game states, and reveals narrative information; in this case, each marker is a guild sign from the Micronation of Ludea, the fictional world associated with the street game.

  7. Signifier of a mixed reality: Once the role of the tag is decoded, it then becomes more than a wayfinding marker; it becomes a portal into a mixed reality. A player encounters the marker and recognizes its potential to reveal a cloud of data that surrounds it.

So, how might we examine each of these aspects of the media ecology that comprises this particular mixed reality? The multiplicity of fiducial markers may be seen as an expression of cloud aesthetics. Not all of these aspects are equal; they are activated hierarchically, and they constitute layers of the mixed reality. The mixed realities explored in this game are based on media ecologies, which is to say that the systems and relationships present in the game are constructed through networks and interactions across the main elements of the experience: the city, the markers, the players, and digital entities. These elements are arranged in relation to one another like a constructed language; each language is an artificial system constructed from small bits (letters) arranged to form larger constructs (words) connected with one another to form constructs larger still (sentences), and so on. The interconnection of these elements is a unique aesthetic experience; each element is a small bit—or molecule—in a larger, more complex whole. As a media ecology, this larger whole operates as a complex, adaptive system and explores the connections between artificial systems and natural processes abundant in life. The synesthetic codes of the fiducial markers operate as an abstract language, with a hierarchy structuring the layers of expression in the system, analogous to the processes occurring in living systems.

The highest level of this hierarchy is composed of the wayfinding markers and signifiers of a mixed reality, orientating players into a new relationship with the city. At the next level, the markers operate as graphical notation, material object, and game token when players encounter and decode the markers using the iPad. Finally, a third level pertains to the markers’ role as machine code (invisible to the player) and nonobjective art (highly visible to the player but not performing a function in the mixed reality). The role of these different layers in manifesting the mixed reality can be articulated further by decoding the ontology of this game.

Mixed Reality Ontologies

My earlier analysis of the aesthetics of fiducial markers [10] explored a linguistic context, such that the markers were framed as a kind of code switching. In this model, three aspects of fiducial markers are considered: human-readable, machine-readable, and mixed reality signifier. As outlined above, this model has been expanded into seven aspects in the more recent project. When constructing a mixed reality, ontology is key in the sense that such constructions allow one to define the possibilities of what can be and ways “to be.” The focus on the fiducial markers’ overt nature (emphasizing rather than obscuring their artificiality) remains, but this focus is now supplemented through considerations of a number of other factors: context, materiality, function, and relationship. The marker now has a number of simultaneous ontological positions. While the earlier model saw the marker as a language unit, this approach sees it as a means of binding multiple aspects of a single ontology within a mixed reality; that is, to provide a solid construct in a cloud of data. Four different approaches are applied to decode this ontology and the relationships within. First, Lévy’s model of the actual-virtual; second, Galloway’s gamic actions; third, Bogost’s unit operations; and fourth, Harman’s quadruple object.

Figure 4. Lévy model mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.
Figure 4. Lévy model mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.


Figure 4 considers relations between players, markers, entities, and locations in a mixed reality within the context of Lévy’s idea of the virtual. [11] He considers four processes: virtualization, realization, actualization, and potentialization. A significant aspect of this model is its balance between the virtual and actual, which represent a shift away from the virtual-physical divide and toward a better understanding of what occurs cognitively and experientially. A fiducial marker presents a potential that is realized when activated by digital data. As Lévy describes it, the “virtual” mindset means seeing everything as the product of processes and systems, regardless of whether the thing is solid, physical, and material. This is an important point, marking an significant shift in our understanding of the aesthetics of mixed realities and fiducial markers. In this light, markers may be considered as an alphabet of sorts, a kind of ecosystem of codes or media ecology—a system that generates meaning through play. How, then, can we articulate this system?

Levy’s model recognizes the nature of fiducial markers in urban spaces. Until activated, they do not tell the full story, even though their presence indicates that there is something to be decoded. They are virtual objects with a material presence. When a player finds a marker, they actualize it into image and sound on the iPad, a process that then sets up new possible relations between sounds and images interacting with one another, or that may be manipulated to open up new potential within the system. A perceptual shift in the urban space also occurs as the role of each code is realized, changing the players’ behavior as they search for additional markers. The altered behavior of the player also impacts spectators by bringing further attention to the codes.

Figure 5. Galloway gamin actions mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.
Figure 5. Galloway gamin actions mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.


Of great importance here is the equivalent role that humans and machines play in the system. This can be understood in terms of a distinction Galloway draws between operator and machine acts. [12] He breaks down the system in a way that equalizes the contributions of human and machine agents, and it is this balancing of the media ecology that is most relevant to mixed realities. Operator acts are those of the player, while machine acts are those of the software system. The latter is made up of machine vision that has decoded the markers, rules for connecting these to images and sounds, and the overall operational context of the iPad. Diegetic acts are those embedded into this system and that cannot be changed; non-diegetic acts are those that happen outside the system; in this case, mostly through interaction between players. Players are important but not central; there is a need for a decentralized approach to understanding the relationships within the system.

Possible models for mapping the various marker roles are suggested in figure 5. The non-diegetic acts are interesting in this context, as players interpret and discuss the system amongst themselves. As noted above, their behavior also modifies the public space, as searching and playing are not common practices at the Adelaide CBD. There is a symbiotic relationship between operator and machine. The player-operator needs the machine to decode the markers to hear them, while the game-machine can’t move on its own, and needs to be carried around and shown the markers in order to decode them. This relationship has been explored in the earlier version of noemaflux, in which players carried a digital seed that pollinated data at each marker site.

Figure 6. Bogost mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.
Figure 6. Bogost mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.


However, the fiducial markers also undergo transformations in noemaflux:ia as the player—in dialogue with the machine—activates various states that transform their meaning in the system and release more data. Signifier to object, machine code to music, then onto game token, and so on. Bogost’s unit operations [13] explore the construction of different forms of language via computational processes, particularly those that occur in computer games and often combine different languages in fluid forms of expression. Each unit (image, sound, code, and word) operates on others through complex processes using consistent rules and systems. This approach may be employed to understand how the various aspects of the fiducial markers interact with one another and with the player, and to understand how these transformations unfold to build a mixed reality—even a story.

In noemaflux:ia, every aspect of the fiducial marker undergoes a transformation via the unit operations embedded in the hybrid system of players, public space, iPads, markers, and digital game. The marker is recognized by a player as a signifier of a mixed reality, encountered as a material object, decoded by the iPad as machine code, transcoded into a musical note, and finally read as a game token—“there are more to be found!” The mixed reality, while a complex system with many layers, is interconnected by these repeated operations. Connections and associations are quickly established in the system, making it tangible for the players as a mixed reality with a particular ontology that shares its space with the Adelaide CBD. In other words, the Adelaide CBD becomes its habitat. The act of tagging locations with markers to construct the space prior to play is also a unit operation at another level that initiates the entire process.

Figure 7. Harman mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.
Figure 7. Harman mapped to aspects of fiducial marker. © Troy Innocent, 2013. Used with permission.


This idea of urban space as providing a habitat for the fiducial markers suggests consideration of the markers’ role as objects and the multiplicity of interconnected roles and functions they have in the ontology of a mixed reality—an ontology that balances the player, the machine, the cloud, and the street, all equally important parts of this ecology. Harman’s fourfold object [14] provides a possible framework for analyzing the dynamics of mixed realities that blend fiducial markers, digital entities, human players, and public space in data-clouds (see fig. 7). Through this approach, he identifies real and sensual objects, which have real and sensual qualities. Most importantly, the real object always lies underneath the surface, its real qualities unchanged and defining the object itself, while the sensual object serves as its interface with other objects, each of which will encounter different sensual qualities depending on their perspective.

In a fiducial marker, the real object is the code. The material object read by the iPad contains some of its sensual qualities, but some of these appear on the screen of the iPad or via its speakers. These images and sounds have a special relationship with the marker. A musical note is a sensual quality associated with the marker, and perhaps, an object in itself. The game tokens triggered by the decoding of the marker change states in the system, again creating different relationships—perhaps these are real qualities? It is tempting to place the machine code operations within this domain as well, as they are invisible to the player; however, the code-machine-space-player interactions and the diverse range of objects they identify provides a richer source of meaning. The marker has a relationship to the urban space, to the player, to the machine (the iPad), and to the game, and presents different sensual qualities to each. Each of these objects relates to each other in this system, demonstrating the complexity of this media ecology.

Conclusion

Mixed realities are emblematic of cloud and molecular aesthetics. They represent the antithesis of discrete, homeostatic virtual reality, in that they are clusters of data, contingent on player interaction, that are embedded within many layers of the city. Not that this is a dialectic relationship; one aspect that makes mixed realities worthy of investigation is the broad spectrum of possible relationships they represent. They are models of perception that blend multiple modes of existence into a cohesive, collective whole, bound not by territorial borders, but by sets of relationships and operations. Everything is overlapping and interstitial, and there is no clear distinction between the game and the city around it, except in the minds of the players, who collectively define the mixed reality through their shared experience.

The mixed reality games discussed above build in layers as the player decodes and experiences their various aspects, allowing the many different perceptions and meanings of the fiducial markers to coexist. Wayfinding markers enable traversal. Computer vision facilitates the decoding of these markers as real objects. Sound is activated as one of many sensual qualities. The objects are also seen as nonobjective art, and their materiality renders them tangible as sensual objects. Many of their real qualities emerge through play, as the player decodes the invented psycho-geography and becomes part of the ontology of the mixed reality. Operating somewhat like a portal, a fiducial marker is a material object located in space, equally visible to machines and humans, that tethers a cloud of data to its location and connects flows of information and players. The players are left with their heads in the cloud, embedded in a multisensory mixed reality.

Figure 8. noemaflux:ia, Troy Innocent, 2013. Cloud of ideas with fiducial markers. © Troy Innocent, 2013. Used with permission.
Figure 8. noemaflux:ia, Troy Innocent, 2013. Cloud of ideas with fiducial markers. © Troy Innocent, 2013. Used with permission.


These seven aspects demonstrate the complexity of mixed reality ontologies. Lévy provides one approach toward how we might consider players, markers, entities, and locations as interconnected in a cohesive mixed reality. Bogost’s unit operations represent another approach, identifying the processes that occur within the game. Harman provides further understanding of the phenomenology of this experience by grounding this concept in a clear ontology—useful in constructing these works, and in understanding the equal roles that humans and machines play in the system, a concept also reflected in Galloway’s distinctions between machine and operator acts. He breaks down the system in a way that equalizes the contributions of human and machine agents, and it is this aspect of these ontologies that is most relevant to mixed realities. Players are important, but not central; there is a need for a decentralized way of understanding the relationships within the system.

The works analyzed above are a series of prototypes that explore these themes through games played in public spaces. Although this represents preliminary research, it suggests a useful framework for understanding mixed realities already being used to direct further development of prototypes in the field. Further work will focus on creating a hierarchy for the various aspects of fiducial markers, allowing some to recede while bringing others to the fore of the mixed reality, tuning the experience in order to make the player aware of these relations. This practice is informed and made possible by the findings articulated in this essay.

References and Notes

[1] Jussi Parrika, Insect Media (Minneapolis, MN: University of Minnesota Press, 2010).

[2] Troy Innocent, Scenes from Ludea (Melbourne: Tolarno Galleries, 2005).

[3] Troy Innocent, “Code Switching in Mixed Realities,” in Proceedings of the 19th International Symposium of Electronic Art (ISEA2013), Sydney, 11-13 June 2013 / Kathy Cleland, Laura Fisher and Ross Harley (eds.), 2013, pp. 1-5.

[4] Ibid.

[5] Richard E. CytowicDavid M. Eagleman, and Dmitri NabokovWednesday Is Indigo Blue: Discovering the Brain of Synesthesia (Cambridge, MA: The MIT Press, 2011).

[6] Wassily Kandinksy, Point and Line to Plane (New York: Dover Editions, 1979).

[7] László Moholy-Nagy, Light-Space Modulator (1922-1930, replica 1970 Bauhaus-Archiv, Berlin).

[8] Theresa Sauer, Notations 21 (New York, NY: Mark Batty Publisher, 2009).

[9] Ibid.

[10] Ibid.

[11] Pierre Lévy, Becoming Virtual (New York, NY: Plenum Trade, 1998).

[12] Alex Galloway, Gaming: Essays On Algorithmic Culture (Electronic Mediations)(Minneapolis, MN: University of Minnesota Press, 2006).

[13] Ian Bogost, Unit Operations: An Approach to Videogame Criticism (Cambridge, MA: The MIT Press, 2008).

[14] Graham Harman, The Quadruple Object (Alresford, Hants: Zero Books, 2011).

Author Biography

Troy Innocent explores the multiplicity of codes in the contemporary mediascape, particularly connections between language and reality. His work invites people to play in worlds that emerge from transmedia ecologies—complex systems of virtual and actual signs and entities. He has developed a unique aesthetic vocabulary that spans a hybrid practice and traverses interaction, design, code, sculpture, animation, sound, and installation. Innocent’s recent public art practice manifests in mixed realities, such as an interactive sculpture garden entitled Colony (2008), Urban Codemakers(2010), and noemaflux, an ongoing work that has appeared on the streets of Ogaki, Instanbul, and Adelaide. Innocent is represented by Anna Pappas Gallery.