Observing with Machines: An investigation into human–machine collaboration during ethnographic field work
Thesis Proposal Fall 2019
Introduction
Smart homes, smart cars, smart watches, smartphones, in smart cities, are everywhere. We are surrounded with invisible agents, algorithms, sensors, and actuators that not only have an effect on culture, but are concurrently observing culture (Redstroem, J. & Wiltse, H. 2019). These agents are woven into the fabric of our everyday lives in ways we do not notice. They interact with us through our phones, track our movements, and facilitate our mobility through public space. These agents influence our behavior and establish complex relations with us (Iaconesi & Persico, 2016). They have varying levels of agency—they record information and generate data—lots and lots of data that influence decisions in the social sphere. Furthermore, these agents and the information they generate are fractured and dispersed across different spatio-temporal environments that range from the cyber, physical, and social (De.S. et al 2017).
The “agents” I speak about are non-human computational agents which make up the Internet of Things in smart cyber-physical-social environments, or Smart Cities. The Internet of Things describes the interconnectedness of everyday objects embedded with sensors and computing devices that are able to send and receive data. A smart environment is a space that is mediated by these sensors or other computational agents such as artificial intelligence (AI) and machine learning. A smart home might have a digital assistant like Alexa to help pay the bills, and a google nest to help with climate control, with little use for human activity. A smart city might have cars equipped with sensors that engage with their networked environments (sensors in traffic lights and crosswalks, all of which work with each other to facilitate mobility in public space for the driver and streetwalker to safely cross the street.
More than half of the world’s population live in cities, many of which are “smart” (Barth et al 2019). Even Raleigh, North Carolina has its own smart initiatives. Because of this trend, it is important for designers to consider the implications of smart infrastructures. By understanding the affordances of new technologies and its working schematics, including the networked capabilities of the Internet of Things, designers can begin to think critically who they are designing for and in what context (Forlano 2016).
Designers and researchers are calling for new frameworks to design in smart environments. Carl DiSalvo, designer and author of AdVersarial Design, structures workshops around perceiving a city from a robotic perspective. Thing ethnography by the Think Tank Project in Amsterdam studies human-object relations through the perspective of a thing (Giaccardi 2016). More than Human Participatory Design leverages non-human agencies, perspectives, and voices, leading to interesting and provocative questions about our environment, our future and ourselves.
This investigation is a plunge into what it is like to design and understand culture within a smart, dynamic,and urban environment. As pervasive, invisible and ubiquitous computing (as AI, sensing artifacts, robotics) occupies corners of public and private space, there is a design opportunity to speculate on future interfaces that bridge the cyber-physical-social landscape. The data generated by these actors (human, non-human, natural or technological) are qualitative and quantitative. This investigation focuses a lens on how these subjective (and objective) worlds can be blended, reframed and restructured in order to not simply see more, but to see differently.
Problem Statement and Justification
Smart Cities
A Smart City is a multidimensional and multifaceted conceptualization. It is made of networked sensors and algorithms embedded in objects—wearables, traffic lights and smart cars that are able to “unobtrusively and seamlessly connect and exchange information” (Mahmood 2019). Despite promises of these technologies to make cities smarter and better, their networked character has forced collisions between the city, its infrastructure, and its citizens (Forlano 2016. These agents make up the Internet of Things and can tell a story about human culture through the data they generate. This data is referred to as “big data” and can be difficult to read without expertise. As urban trends point towards adopting smart infrastructures, design researchers should develop methods to attend to their complexity, one of which, is the consideration of non-human agents the data they generate.
Ethnography and Data
Ethnographic observation is the study of human culture. Traditionally, its practice is conducted through teams who gather qualitative information in the form of observations, field notes, and interviews. These qualitative data are not numerically measurable, but are inherently rich, enabling ethnographers paint a multi-dimensional, context-driven, and “thick” understanding of humans and their environment (Colson, E., & Geertz, C. (1975). Traditional ethnographic methodology becomes complicated, however, when smart agents in the Internet of Things generate a different type of data that aggregates at a multitude much higher than traditional ethnographic data. These data are “big and thin” and provide a macroscopic but myopic view of culture. Big data lacks the inherent richness of thick data and without context, can only reveal the “what” as opposed to the “why. (Bornakke et al. 2018). Conversely, without big data, ethnographic thick data could fall short in asking the right questions (Wang 2016).
According to technology ethnographer Tricia Wang, integrating big and thick data forms a more complete picture—big data offers insights at scale at best of machine intelligence where thick data rescues the context loss and integrates the best of human intelligence (2016). According to Curran, the justification lies in their common characteristics and in their situatedness within the same epistemological field—within human behavior and cultural interpretation (2013). He coins the term “big ethnographic data” and lists synergistic ties between qualitative and quantitative data:
- Both are interested in the everyday culture
- Both explore patterns, movement and networks
- Both are interested in the physical (how the body interacts with products and space)
- Both can offer holistic and synchronic approaches to analysis (2013)
Blok et al refer to this new form of data as “big social data” which is computational, transactional, and digital and in need of re-alignment across the social sciences (2017). Others call for “the mixing of big decontextualized data with highly contextualized thick data can help uncover the meaning behind Big Data and analysis and entirely new interfaces and polyphonies can arise” (Bornakke et al. 2018).
By merging the two types of data, researchers could have a more holistic understanding of culture in complex smart environments, which could result in a number of things for design research including understanding users in socio-technical smart environments and understanding the affordances and materiality of smart agents and their heterogeneous data types. All of which would have an effect on the services we provide, the infrastructure we design, and roads we create, the interfaces we design.
Blending Data
While the incentive to blend data worlds evident, there is no systematic method for integrating quali-quanti methods in ethnographic research, much less an interface that allows for generating its blending in collaboration in situ (Bornakke, R and Due, B 2018). There are, however, some explorations:
Aipperspach et al uses a methodology of ethno-mining which draws on techniques from ethnography and data mining that is characterized by close iterative loops to incorporate both quantitative and qualitative in a “mutual dance of space observation” (2006). Bornakke et al proposes a big-thick blending methodology that is rapid, iterative, and collaborative with respect for individual expertise (Bornakke et al 2018). Blok et al explores a multi-methodology of “combining heterogeneous data: through a “stiching together” of digital and ethnographic data worlds (2017). These methods use visualizations to guide the ethnographic process and make visible the blended and stitched data worlds as well as a call to consider the heterogeneous nature of each data world, in its orientation towards time, space, and content (Aipperspach et al 2006, Blok et al 2017.
Given the affordances of new technologies in smart environments, there is a design opportunity to help blend the disparate data worlds together through explorations of a tool to mediate collaboration and co-production of heterogeneous data as well as visualizations to blend and analyze their results.
This investigation merges two frameworks: ethno-mining and big-thick blending during sequential stages of ethnographic research: in field work and analysis. I intend to explore how the affordances of networked objects—sensors and actuators that exist in smart environments can contribute to the co-production of data during a study—in part, to speculate about the future of human-machine co-production of knowledge in the future. Through exploring visualizations strategies, I will investigate how heterogeneous data types can be blended, mapped, and visualized to enable novel insights.
Research Question
How can the design of a multi-agent data (system) used by ethnographers during a study of a smart urban environment gather and visualize data to aid comprehension and analysis?
- How can geo-tagging connect ethnographers during field work to facilitate the co-production of data?
- How can multi-modal features promote heterogeneity among data types?
- How can dynamic visualizations generate novel insights through blending data?
- How can interactive mapping provide context to aid spatial analysis?
Frameworks
More than Human Participatory Design
A framework that challenges traditional binaries of Western thought such as City/Nature and Human/Non human to consider the entanglements between human and nonhuman worlds which range through plants, technologies, animals, and materials within space-time in both topological and topographical formations in order to overcome problematic narratives of human privilege and exceptionalism. It acknowledges the co-production of knowledge within (and between) communities, taking into account the voices, needs, and agencies of non-humans (Bastian, M 2017).
Speculative Design
Speculative design is a discursive practice, based on critical thinking and dialogue, which questions the practice of design. However, the speculative design approach takes the critical practice one step further, towards imagination and visions of possible scenarios. By speculating, designers re-think alternative products, systems and worlds. Dunne, A., & Raby, F. (2013).
Big and Thick Blended Framework
Blending bridges big and thick heterogeneous data into shared analytic spaces. Analytical insights are determined by data materiality with different affordances. The blending is interpretive, distributed cognitive and embodied process performed by the researchers. It must happen iteratively and in rapid pace to consider how analytical insights tend to stabilize over time. The blending takes place before the analysis in each input space is finished to secure the full potential of the blending process. Blended spaces are entirely new analytical results created through complementary, extension, and calibration (Bornakke, T. & Due, B.L. 2018).
Ethnomining Framework
A mixed methods analysis that draws on techniques from ethnography and data mining. It is characterized by close, iterative loops that integrate the results and the processes of ethnographic and data mining techniques to interpret data. It makes use of both qualitative and quantitative data to study phenomena that are inaccessible to either data type alone. It then provides a means for interpreting the data which produces novel insights by exposing the biases inherent in each data (Aipperspach, et al 2006).
Presentation Slides
