Skip to main content
SearchLogin or Signup

Capture the Rapture

This essay explores the manifestation of an individual’s experience of art captured through a stream of emotive data.

Published onNov 05, 2017
Capture the Rapture

Capture the Rapture

Birgitte Aga
Creative Director, i-DAT, Plymouth University
Reference this essay: Aga, Birgitte. “Capture the Rapture.” In Leonardo Electronic Almanac 22, no. 1, edited by Lanfranco Aceti, Paul Thomas, and Edward Colless. Cambridge, MA: LEA / MIT Press, 2017.
Published Online: May 15, 2017
Published in Print: To Be Announced
ISSN: 1071-4391
ISBN: 978-1-906897-62-8


This essay explores the manifestation of an individual’s experience of art, in real-time, as a visual and evolving dialogue captured through a stream of emotive data. Its starting point is the preoccupation contemporary cultural organizations and funding bodies have with evaluating the experience of art through disconnected techniques, represented as static imagery. As such, it challenges the hegemony of the eye and the illusion of certainty that the image produces, and instead requires a temporal, polysensory engagement with fleeting moments of ‘insubstantial uncertainty.’

The essay will also reflect on the notion of capturing and manifesting an emotive experience by exploring a recent technological intervention, the Qualia Project, developed by i-DAT and partners. It attempts to propose the ways in which playful techniques for capturing the emotional experience of art, and the rapture itself, could create dynamic visual dialogues that juxtapose and enrich contemporary evaluation methods. Furthermore, it asks, through the meaning generated from such emotive data, can one affect arts programming, precipitate new ways of curating, and change how we think about the aesthetic experience of the arts?

Keywords: Qualia, emotive data, audience evaluation, cultural impact measurement, visual data dialogues


The issue of value and impact measurement is one of the issues which characterizes current concerns in the cultural sector. This is demonstrated by the research commissioned and published by such organizations as the Arts Council England, NESTA, and The Arts and Humanities Research Council [1-11], as well as through initiatives such as the Manchester Metric Pilot. [12] The research has one thing in common: it explores how people 'experience’ art and culture—“the encounter between people and cultural works or their engagement in cultural activities.” [13] These organizations not only debate the notion of ‘value’ itself, employing the related intrinsic verses instrumental value discourse, [14,15] but also grapple with a range of different methodologies of measurement. Some social and economic impact measurements are borrowed from other sectors, while others are piloted as new approaches, as is the case with the Manchester Metrics Pilot [16] and Public Value Measurement Framework, a project of the Government of Western Australia Department of Culture and the Arts. [17]

However, some of these organizations and funding bodies appear to be preoccupied with demonstrating their value to the public and justifying their investment of public funds rather than exploring ways by which they can truly capture the experience of art itself, such as engaging in deeper dialogues with their audiences. While it is apparent that current methods of measuring impact and outcomes are increasingly being challenged, somewhere in the midst of this pursuit of the perfect metric of art’s worth, the capture of the experience of art itself is forgotten. This means that dialogue with the experiencing individual and the expression of a distributed empathy with and between audiences are muted by static data residing in data silos preserved by the public sector.

The Nature of Emotions

This enquiry into the nature of emotion has been underway since Aristotle, [18] with a more scientific, methodological study of emotions emerging in the late 19th century in the research of such luminaries as Darwin, Dewey, Irons, and James. [19] Many different and often competing models of emotion and emotional expression have since been proposed, but the ‘mechanics’ of emotion and its constructs have remained intangible, its parameters subjective. Only recently, through the introduction of neuroimaging techniques, are we able to identify and visualize which brain areas are consistently and specifically associated with particular emotional states. Central to the wider discourse of understanding emotional states, however, is the measurement of behavior and experiences.

At the heart of emotion and emotionally charged events is this notion of experience. Through experience, different states are perceived as feelings, which emerge when people make meaning out of sensory input from the body. [20] These feelings—internal emotions, affective states—are influenced by knowledge of prior experiences. They manifest through a broad response disposition that may include observable and/or measurable language; behavior, facial, and postural expressions; avoidance or performance deficit, [21] and physiological changes such as increased heart rate and changes in galvanic skin response. [22]

These manifestations are not only projected, voluntarily or involuntarily, by the individuals in physical spaces, but also across virtual spaces, recorded as personal data on such platforms as social media posts, mobile GPS, and, more recently, wearable bio-sensing devices. Such data have the potential to weave rich tapestries of experience, forming the basis for deeper dialogues between people experiencing the world and the other actors (individuals, cultural producers, and funders) around them.

Capturing the Emotive Experience of Art

The value for the arts in measuring subjective and personal data harvested directly through human experience does not lie in the act of harvesting the data itself, but rather in the opportunity—thanks to the emergence of the cloud—to process such dynamic data through an evolving, real-time dialogue between individuals and art producers, curators, and funders.

By transcending the physical and virtual, one may start to better understand the individual experience of the arts and begin to share emotions, which involve communicating the circumstances, thoughts, and feelings surrounding an emotional event. [23] Such sharing offers intrapersonal and interpersonal benefits, providing individuals with an inner satisfaction and relief after sharing, which again strengthens social bonds. [24]

If the value of art is, as outlined by Carnwath and Brown in their Literature Review for the Arts Council England, “created in the encounter between a person (or multiple people) and an object (which may be tangible or intangible, as in an idea or activity),” [25] then the notion of measuring the individual and collective experience of these encounters must be central to our understanding of the value of art. John Holden [26] emphasizes that measuring impact should be about improvement of the cultural provision itself, not about auditioning for more funding, and that the public, first and foremost, cares about their experience of the arts. In order to collect a broader response disposition from such encounters, emerging technological interventions can be implemented. By collecting, processing, and sharing such observable and/or measurable data sets (physiological, verbal, behavioral, and environmental) and juxtaposing these with contemporary evaluation techniques and related data sets, it becomes possible to see new realities and meanings as they are revealed.

Technological Interventions Capturing and Manifesting Emotive Experiences

There are several recent initiatives influencing the harvesting and manifestation of emotive data, encapsulating a playful dialogue in their approach to representing such data sets. There is also a growing body of research into the biological functions that underlie the aesthetic experience. [27,28] This research attempts to unravel what happens when people engage in cultural experiences. [29] These initiatives span physiological, verbal, and behavioral aspects of experiences and transcend the environments of the body and physical and virtual space. They span social commentary, data visualizations, scientific experiments, commercial products, and artworks. Some examples include We Feel Fine by Jonathan Harris and Sepandar Kamvar (2005); [30] Emoto by Moritz StefanerDrew Hemment, and Studio NAND(2012); [31] CrowdEmotion by CrowdEmotion and BBC Worldwide (2014); [32] Lightwave by Lightwave (2014); [33] Supernova Suite  by Sensum (2014); [34] Project "Pulse Room" by Rafael Lozano-Hemmer (2006); [35] and Emotions in Space by Maurice Benayoun (2014). [36]

Qualia: Prototyping the Capture of the Experience of Art

The initiatives mentioned above represent an important frontier in the exploration of experience measurement through the capture of emotive data. Another project that attempts to play at this frontier while integrating more traditional evaluation systems is the ongoing research project Qualia, [37] developed by i-DAT in partnership with Cheltenham Festivals, University of Warwick, Elixel, and Intercity Studio (2014) through seed-funding from NESTA’s Digital R&D Fund for the Arts. The project has developed several technological interventions to capture audience experience by harvesting, processing, manifesting, and sharing audience mood and facial gestures. This data is cross-referenced with personal geospatial and sentiment data to create a more dynamic dialogue amongst audiences, as well as between audiences and contemporary arts organizations.

Figure 1. Qualia Logo, i-DAT, 2013. © i-DAT, 2016. Used with permission.

The Qualia project is a real-time system designed to capture and display how audiences at cultural events "feel." It thus extends contemporary audience evaluation methods by incorporating techniques to harvest, analyze, and visualize more deeply emotive, qualitative data. Qualia employs a range of digital processes coupled with ethnographic practices, which can be described as a “techno-ethnography.’ These processes challenge traditional methods of evaluating cultural impact that are based on quantitative measurements of parameters only indirectly related to the ‘quality’ of the event itself; for example, demographic studies that measure economic impact. By taking this approach, Qualia aims to enhance engagement and to enable arts organizations to reflexively program cultural events, thereby enhancing their economic, cultural, and social impact, and, most importantly, to maximize the experience itself.

The first manifestation of the Qualia project as a complete set of prototypes was tested during Cheltenham Literature Festival in 2013. i-DAT has since delivered new projects that build on Qualia's systems, such as the FLUX Festival app launched in Liverpool in July 2014, and is in the process of launching a new progression through the incentivized and personalized cultural listing app in Plymouth, Artory.

The Qualia system is broken down into three prototypes, described below.

Prototype #1: Qualia Smile

Figure 2. Qualia Smile, i-DAT, 2013. Smile recognition software—The capture of real-time information from the Qualia Smile probes add another layer of mood sensing to the Qualia landscape. © i-DAT, 2016. Used with permission.

"Qualia Smile" (figure 2) is a playful tool for counting smiles that encourages audiences to participate in measuring the collective happiness of festival-goers. Qualia Smile gamifies the collection of qualitative data, generating a positive feedback loop: smile, and the festival smiles with you. The tabulated smiles generate an aggregate visualization of the mood of the festival. Qualia Smile incorporates image analysis techniques found in most common cameras. These open-software libraries are used to create a playful and participatory record of the happiness level at the festival.

Prototype #2: Qualia App:

Figure 3. Qualia App, i-DAT, 2013. The Qualia App for Android and iOS smart phones allows audiences to act as human sensors, feeding back real-time responses to an event, and allows the organization to respond and to map the audience's movement through the venue in real time. © i-DAT, 2016. Used with permission.

The "Qualia Mobile App" (figure 3) provides visitors with an intimate window into the Qualia System. It fulfills the basic functions of a "what’s on" app in terms of listing event information, schedules, maps, etc., but goes a step further by integrating personalization and incentivizing based on an individual’s preferences. Central to the system is the app’s ability to enable users to express their feelings about their cultural experiences in real time through user-generated images, mood snaps, social media sharing, and other feedback.

The app also provides individuals with a unique user ID/ticket and user account used to register the individual’s presence at specific locations and track their interactions. This enables the cultural organization to monitor hotspots and visitor flow, enabling responsive targeting to clusters of users. The purpose of this tracking feature was to allow event organizers to alert visitors to specific locations and provide a responsive management of resources in order to support more popular sub-events. These functionalities were intended to challenge the organizers to develop a much more dynamic and responsive relationship with the individuals experiencing their program.

Prototype #3: Qualia Probe:

Figure 4. Qualia Probe, i-DAT, 2013. The Qualia Probes (figure 4 & 5) are sensing ‘pods’ which were placed strategically around the event site. The Probes provided a focus for public interaction, and a platform that mediated feedback to and from visitors through dynamic visualizations and general information. By capturing real-time information, the probes add another layer of mood-sensing to the Qualia landscape. © i-DAT, 2016. Used with permission.

Figure 5. Qualia Probe Interface, i-DAT, 2013. The Qualia Probes provide a focus for public interaction, providing feedback to visitors through dynamic visualizations and information. © i-DAT, 2016. Used with permission.

"Qualia Probes" are sensing ‘pods’ placed strategically around an event site. Each probe provides its user with access to the Qualia web engine, where the user can view general information but also contribute annotations and blog posts and record and annotate images of their experiences at the festival. These probes measure audio levels in order to understand the ambient hum of the event, and integrate a camera that acts as a visitor-tabulator, measuring both the flow and mood of people using smile recognition technology and image capture. All these data sets, together with the social media feed generated by the festival, are processed and displayed back to the user, in real time, as dynamic data visualizations. These visualizations use a visual language, illustrated in figure 6, for the psychometric scale employed throughout the system, both in the front-end Qualia app and probes and in the web-engine analytics dashboard system.

Figure 6. Qualia Psychometric Scale, i-DAT, 2013. Derived from the Likert Scale. © i-DAT, 2016. Used with permission.

Prototype #4: Qualia Web-Engine:

Figure 7. Qualia Web-Engine: Dashboard, i-DAT, 2013. The open-source web engine, shared under a GNU license, is what drives all Qualia protoypes. It integrates APIs, data scraping, and sentiment analysis, in addition to taking data feeds from Qualia Mobile App users and onsite feedback probes. © i-DAT, 2016. Used with permission.

The open-source Qualia web engine (figures 7, 8, and 9), shared under a GNU license, is what drives all the Qualia prototypes. It integrates APIs, data scraping, and sentiment analysis in addition to taking data feeds from Qualia Mobile App users and onsite feedback probes. It not only allows cultural organizations to see how their audiences are feeling and how they are engaging in real time, but also enables an evolving, real-time dialogue with participants by visualizing the cumulative experience of the crowd.

Although it exists as a separate prototype, the "Qualia Web-Engine" provides a substrate for the Qualia Project. The Qualia Web-Engine manages and synchronizes interactions between all of the processes described above. The server-side system provides an administrative backend to the website, apps, and probes, while also serving as an analytics tool that looks for patterns and behaviors within the collected data. The system takes advantage of a variety of APIs including Facebook, Twitter, and Google Maps in order to track image posting and messaging. In addition, it accesses these APIs to scrape data from social networking activity surrounding the event. Central to the engine is its "sentiment analysis," which is used to evaluate the mood of the event by analyzing the language used by visitors to describe their experience. These messages are extracted from social networking feeds and onsite feedback through the probes and the Qualia mobile apps.

Figure 8. Qualia Web-Engine: Dashboard Analytics, i-DAT, 2013. Visualizing Audience feedback data sets over time through using the Qualia psychometric scale. Measuring the following (in successive order):  Number of Visitors Using the App, Attendance, Feedback Ratings, Social Media Posts, CPS Points Logged, Hotpoints Detected, Mood and Sentiment. © i-DAT, 2016. Used with permission.

Figure 9. Qualia Web-Engine: Dashboard Analytics 2, i-DAT, 2013. Visualizing mood (shape) and sentiment (colour) analysis over time. © i-DAT, 2016. Used with permission.

By integrating these functions while processing the other data captured by the probes and apps, the system is able to analyze this auxiliary data and to look for patterns and behaviors before the data are visualized through the website, pods, and mobile apps. The backend also allows the event organizers to administer the system, respond to real-time information, and post to visitors through the mobile apps, website, and probes. It thus laid the foundation for a technology-driven platform that encourages a real-time dialogue centered on the expression of feelings generated from real-time experiences.

The Qualia system described above was intended to explore new mechanisms for measuring and evaluating cultural impact by gauging the audience emotions. While these mechanisms may merely provide another set of indices for funders to evaluate their expenditure in these austere times, the intention is that the system will also provide a more open and inclusive set of tools for measuring how people feel, and for entering into a dialogue with them regarding their experiences. Although this may suggest a form of cultural surveillance, it is critical to recognize that the system enables the sharing of data and stimulates a level of participation and interaction beyond an audience's normally passive reception. Qualia’s linking of incentives, its personal scheduler and social media feeds, and its provision of a shared overview of the rest of the audience's interactions attempt to entice users into a dialogue with event organizers. In order to do this, there must be a shared understanding of benefit: arts organizations must understand that audiences are not just an open data-mine, and audiences need to understand that the events they attend are actually defined by their own experiences. This vantage point is grounded in, but also transcends, the notion of data privacy and ownership by empowering users to make choices about their personal data shadows and the potential impact these shadows can have on their own (and collective) experiences of art. It thus provides an opportunity to challenge the status quo of the producer/consumer power dynamic. The wider implications for curation and authorship are not yet fully grasped, but traditional modes are being challenged by the Qualia project and by emergent initiatives in the field. Collectively, these initiatives visualize dynamic dialogues that are products of a real-time negotiation with participants—captured through a stream of data—and that paint new realities.


Emotive and dynamic data harvested directly through human experience—shared through cloud-based systems wherein humans become part of the sensory apparatus and the experience of the individual becomes central to an evolving dialogue about the experience of the arts—open up new meanings and challenge the status quo. While some of the methods for mapping experiences outlined above are experimental and operate outside the currently recognized, rigid statistical models for audience evaluation, they do represent an important frontier in measurement. As Carnwath and Brown argue, "any conceptual framework for impact that ignores the immediate physiological effects of cultural experiences—regardless of the difficulties in measurement—would be incomplete." [38]

Essentially, such systems should not replace but should contribute toward contemporary audience evaluation techniques, thereby enriching the tapestry of experiences and building upon the representation of knowing the world through frozen moments of time. They should be used as tools to encourage an evolving, real-time dialogue that transcends the physical and virtual. Can we, producers and curators of the arts, deal with the level of reality this exposes and embrace the emotions of the crowds that lie somewhere in the cloud? Can we engage with such polysensory data streams of experiences, represented as fleeting moments of ‘insubstantial uncertainty’ to better create and capture the rapture?

References and Notes

[1] Arts Council England, The Value of Arts and Culture to People and Society—an Evidence Review (London: Arts Council England, 2010).

[2] Jackie Bailey, Meaningful Measurement: A Review of the Literature about Measuring Artistic Vibrancy (NSW: Australia Council for the Arts, 2009).

[3] Jackie Bailey and Lance Richardson, “Meaningful Measurement: A Literature Review and Australian and British Case Studies of Arts Organizations Conducting “Artistic Self-assessment”,” Cultural Trends 19 no. 4 (2010), 291-306.

[4] Hasan Bakhshi, “The Impact of the “Valuing Culture” Debates in the UK, 2003-2011,” Cultural Trends 21 no. 3 (2012): 213-214.

[5] Hasan Bakhshi and David Throsby, Culture of Innovation: An Economic Analysis of Innovation in Arts and Cultural Organizations (London: Nesta, 2010).

[6] Hasan Bakhshi, Juan Mateos-Garcia and David Throsby, Beyond Live: Digital Innovation in the Performing Arts (London: Nesta, 2010).

[7] Hasan Bakhshi, Alan Freeman and Graham Hitchen, “Measuring Intrinsic Value: How to Stop Worrying and Love Economics,” in Mission Money Models (London: Mission Money Models, 2009), (accessed September 15, 2014).

[8] Hasan Bakhshi, I. Hargreaves and J. Mateos-Garcia, A Manifesto for the Creative Economy (London: Nesta, April, 2013).

[9] J. Bennett and M. Phillips, Measuring Cultural Value and Cultural Impact Using Technology-enhanced Methods (London: AHRC, 2014).

[10] BOP Consulting, Measuring the Economic Benefits of Arts and Culture: Practical Guidance on Research Methodologies for Arts and Cultural Organisations (London: Arts Council England, 2012).

[11] Catherine Bunting et al., Achieving Great Art for Everyone: A Review of Research and Literature to Inform the Arts Council’s Ten-year Strategic Framework (London: Arts Council England, 2010).

[12] Catherine Bunting and John Knell, “Measuring Quality in the Cultural Sector - The Manchester Metrics Pilot: Findings and Lessons Learned,” 2014, (accessed 15 June 2014).

[13] John D. Carnwath and Alan S. Brown, Understanding The Value And Impacts Of Cultural Experiences a Literature Review (London, commissioned by Arts Council England: WolfBrown, 2014).

[14] J. Bennett and M. Phillips, Measuring Cultural Value and Cultural Impact Using Technology-enhanced Methods (London: AHRC, 2014).

[15] J. Holden, Capturing Cultural Value: How Culture Has Become a Tool of Government Policy (London: Demos, 2004).

[16] Catherine Bunting and John Knell, “Measuring Quality in the Cultural Sector - The Manchester Metrics Pilot: Findings and Lessons Learned,” 2014, (accessed 15 June 2014).

[17] Michael Chappell and John Knell, Public Value Measurement Framework: Valuing and Investing in the Arts—Towards a New Approach (Perth: Government of Western Australia, Department of Culture and the Arts, 2012).

[18] David Konstan, The Emotions of the Ancient Greeks: Studies in Aristotle and Classical Literature (Toronto: University of Toronto Press, 2006).

[19] T. Wager, L. F. Barrett, E. Bliss-Moreau, K. A Lindquist, S. Duncan, H. Kober,  J. Joseph, M. Davidson and J. Mize, “The Neuroimaging of Emotion,” in Handbook of Emotions, 3rd edition (New York: Guilford Press, 2008).

[20] K. A. Lindquist, T. D. Wager, H. Kober, E. Bliss-Moreau and L.F Barrett, “The Brain Basis of Emotion: A Meta-analytic Review,” Behavioral and Brain Sciences 35, (2012): 121-143.

[21] Carroll E. Izard, Jerome Kagan and Robert B. Zajonc, eds. Emotions, Cognition, and Behavior (Cambridge, New York: Cambridge University Press,1984).

[22] B. Aga and M. Phillips, Human Geography v1.0 (Plymouth, UK: i-DAT, 2013) (accessed September 10, 2014).

[23] Bernard Rime, “Interpersonal Emotion Regulation,” in Handbook of Emotion Regulation, ed. James J. Gross (New York: Guilford Press, 2007), 466–68.

[24] Ibid.

[25] John D. Carnwath and Alan S. Brown, Understanding The Value And Impacts Of Cultural Experiences:  A Literature Review (London, commissioned by Arts Council England: WolfBrown, 2014).

[26] J. Holden, Cultural Value and the Crisis of Legitimacy: Why Culture Needs a Democratic Mandate (London: Demos, 2006).

[27] Catherine Stevens, Renee Glass, Emery Schubert, Johnson Chen and Heather Winskel, “Methods for Measuring Audience Reactions,” in Proceedings of the Inaugural International Conference on Music Communication Science, ed. E. Schubert, K. Buckley, R. Eliott, B. Koboroff, J. Chen and C. Stevens (Sydney, Australia: ARC Research Network in Human Communication Science (HCSNet), University of Western Sydney, 2007) 155–158.

[28] Wolfgang Tschacher, Steven Greenwood, Volker Kirchberg, Stéphanie Wintzerith, Karen van den Berg and Martin Tröndle, “Physiological correlates of aesthetic perception of artworks in a museum,” Psychology of Aesthetics, Creativity and the Arts 6, no.1, (2012): 96-103.

[29] John D. Carnwath and Alan S. Brown, Understanding The Value And Impacts Of Cultural Experiences  a Literature Review (London, commissioned by Arts Council England: WolfBrown, 2014).

[30] Jonathan  Harris and Sepandar Kamvar“We Feel Fine,” 2005, (accessed June 15, 2014).

[31] Moritz Stefaner, Drew Hemmet and Studio NAND, “Emoto,” 2012, (accessed June 15, 2014).

[32] CrowdEmotion and BBC Worldwide, “CrowdEmotion,” 2014, (accessed June 15, 2014).

[33] Lightwave, “Lightwave,” 2014, (accessed June 15, 2014).

[34] Sensum official website, “Supernova Suite,” (accessed June 15, 2014).

[35] Rafael Lozano-Hemmer’s official website, “Project "Pulse Room," 2006, (accessed June 15, 2014).

[36] Maurice Benayoun’s official website, “Emotions in Space,” 2014, (accessed June 15, 2014).

[37] i-DAT, “Qualia,” 2014,, (accessed June 15, 2014).

[38] John D. Carnwath and Alan S. Brown, Understanding The Value And Impacts Of Cultural Experiences a Literature Review (London, commissioned by Arts Council England: WolfBrown, 2014).


Qualia, was originally developed and piloted by i-DAT, University of Warwick and Cheltenham Festivals, for the Cheltenham Festival 2013, and is currently ‘powering’ a range of national cultural projects such as Flux Liverpool, Artory Plymouth. The App was produced by Elixel with branding and design by Intercity Studio.

For further information on Qualia, please see and

Author Biography

Birgitte Aga is part of the i-DAT Research and Design Collective who are playfully experimenting with data ( She is an interdisciplinary designer, researcher and technologist and has produced a range of playful and experimental cultural probes/prototypes, interventions, festivals, and exhibitions. Aga’s focus is on the evolution and convergence of personal, wearable and interconnected technologies, data and predictive algorithms and the influence of such on human experience. Through prototyping intimate data-driven and narrative based experiences she questions the use of such predictive algorithms and personal data for human enhancement and commercial gain.


No comments here