Month: March 2007

The work in mapping the connection between physical geography and virtual representations of that geography is a component of the work I have done that is relevant to the question as to what the basis of my tenure case is. In retrospect, I realize that I picked up the story in the middle, and that there is a preceding “chapter” to my intellectual work since my work as a graduate student. I would like to recast this personal narrative with a brief supplement that explains the continuity of my work since graduate school.

The vector of the scholarly and art-technology work I have done investigates ways to connect the physical world with the virtual world. The online virtual world brought about manifest changes for how we live within and socialize as human beings. It is my speculation that a “connected world” in which the virtual and the physical are linked, not only amongst human beings, but also network connected non-sentient “things” — will be the next evolution in the new networked age.

As explained in my Personal Statement, many of those “connecting things” projects took the form of creating linkages between the physical world as represented as geography, and the virtual world as in visualizations and representations of that geography. This was some early projects in exploring digital cartography, the most literal kind of link between the virtual and the physical. There was work, though, that preceded these projects that I feel is significant and crucial to a more complete description of my scholarly trajectory.

There are three “chapters” to my research work, and each has consumed between 3-5 years. Virtual reality was the main character in the first chapter, and was the focus of my master’s of engineering studies and my doctoral dissertation. Mobile devices was the focus of the second chapter and consisted of a series of art-technology commissions that lasted 4 years during the completion of my doctoral dissertation and into the first year of teaching at USC. The third chapter, which I began at the early part of my second year at USC, is focused on how electronic games that link physical world data sensing with virtual world games can address two real-world challenges — the state of the environment and adolescent physical fitness.

The first “chapter” of this research trajectory began with my master’s studies on virtual reality and the cultural and technological ways that virtual reality set itself off as distinct from physical reality. For my master’s research, I studied computer-human interaction and learned how the human body can become an interface — a connecting bridge — between the virtual and the physical. My chief interest was to study how virtual reality could possibly lead to richer, more evocative, more natural connections between our physical bodies and virtual bodies as a form of human-computer interface. This was the core of my studies as a master’s student and a doctoral student — studying virtual world to physical world connections — the human-computer interface. One chapter of my dissertation became well-regarded as one of the earliest analyses of video games, in this case SimCity, a game that links representations of physical worlds (cities) to virtual worlds (the computer simulation.) In the work that lead to this chapter — a series of studies of people playing this game — I began to recognize that the computer-human connection can be understood both at the physical level as well as the cognitively level. This realization would become a valuable consideration in the “third chapter” as I work to find ways to create connections between physical and virtual worlds that persuade people to modify their habits and behaviors.

The second chapter is about digitally connected mobile devices as connectors between things happening “on the ground” in physical environments beyond the heads-down computer desktop experience, and the virtual representation of those activities online, on the Internet. In this chapter, I created mobile devices and device-based experiences that allows users to create and share their own personal maps of the world around them. This work presaged the popularity of online, user-created maps of popular or significant locales before the introduction of the popular commercial online mapping services.

The third chapter, just now beginning, is focused on linking the virtual and the physical, where the specific parameter of the physical world is represented by two different things, for two separate but related projects: physical fitness and the physical environment. I will explain each briefly in turn. (As this work is in progress, a complete description and summary results are premature. One project is sufficiently advanced at this stage that it is being submitted for consideration at user-interface design professional society conferences.)

The first project in this third chapter is body-based fitness activities. The second project in this chapter is sensor-based readings of the “micro-local” state of the environment.

For the first project, my course of research, which is approximately 30% complete, is to create small evocative games that connect physical fitness — real-world physical activities such as might be compelling for young adults, for example, skateboarding, basketball, snowboarding, walking — with online digital games. I am constructing a “theory object” that creates a bridge between physical activity and virtual activity. This particular “theory object” is an inexpensive device that can record arbitrary physical activity, and do so in a way that addresses many of the important characteristics of a successful physical fitness regimen. The objective is to allow physical play to also appear as compelling as virtual play. The project is meant to evoke in the user the sense that their physical play will also contribute to virtual play. By connecting physical activity with virtual activity I am hoping that the propensity many youth have for online experiences will encourage them to invest their time with a mix of physical and virtual gaming.

The second project for this chapter is to create a series of games about the environment by connecting physical world sensing of the “micro-local” environment to a virtual world representation of the sensor data. I am designing and constructing a hand-held, portable device capable of sensing environmental pollutants that will produce data that can feed into a small game. The objective is to evoke concern for the environment and turn attention more directly toward environmental issues.

With both of these “third chapter” projects, my hope and expectation is that USC’s preeminence as a center for research and development of electronic, online games will further buoy the significance and importance of this third chapter of research. This kind of virtual/physical hybrid electronic game is a conceptually rich area of research and is just at its early stages. A search of the canonical idiom for this new arena — “exergaming” — on the ACM digital library yields four citations, three of which are informal interviews with leaders in the field of human-computer interfaces. Only one citation makes an extended study of pedometers as a way of encouraging physical fitness, and the paper itself yields no remarks on the construction of more complete game scenarios. In other words, the ground is fertile for defining the terms and debate of this area of gaming that connects the virtual and the physical, an area of research practice that I have called “offline gaming.”

Results and Publication
The first chapter as described above yielded a long (100+) page essay that became my master’s thesis and further investigation and study of that first chapter yielded my doctoral dissertation, completed in 2004. One chapter from the dissertation was published as a journal article, and reprinted in a book on the popular computer game SimCity. The third and current chapter serves as material for a book project with the working title “Connecting Things”, about how connections between the virtual world and the physical world can mitigate the deleterious effects of relying too much on one rather than finding a careful, connected balance between the two. The underlying theme of this book has been presented and refined in a series of invited public and conference lectures as well as small research workshops I have facilitated over the last two years. (I have included slides from several of these presentations, as well as reports and essays on the book’s main theme in the dossier.)

The book will consists of scholarly theoretical essays, research results, and designed devices of my own making (what I refer to as “theory objects” and others refer to as evocative knowledge objects) constructed as open-designs (open-source/open-hardware/open-process) so as to help explain the book’s thesis through their actions.

More Detail of My “Connecting Things” Research Vector

From VR to Mobile Devices to Connecting Sensor Things That Link the Physical to the Virtual

1. “Chapter One” — “Virtual World” as in Virtual Reality
My master’s thesis, completed in 1993, investigated how the physical and virtual became conceptually distinct. I looked closely at popular news media, science-fiction and film to understand how “virtual reality” was represented and how its representation changed what people understood to be the possibility of unique connections between the physical world and the virtual world. This was a watershed moment in the study of human-computer interfaces, and helped pave the way for expanding the study of connections between humans and machines.

For my doctoral studies and my dissertation, completed in 2004, I performed a close study of the technology behind virtual reality and computer simulations.
I did this so I could understand how technology can shape the way we think about and understand the world. My goal was to learn how to do such a scholarly analysis — and do it convincingly — so as to learn how I might myself create technology devices and objects that evoke within the mind of the technology’s user that the physical world matters.

As I completed my dissertation I had embarked on a series of “art-technology” theory objects. These were technology objects commissioned as artistic projects that continued my theme of finding ways to connect the physical world with the virtual world, and do so as to deepen a sense of commitment to the physical world. These projects were driven by a deep personal sense of anxiety regarding the state of the physical world’s ecology. My concern was driven by the enormous attention that “virtual worlds” were receiving as the Internet continued to grow in importance. I felt an obligation to research ways to make the Internet more “physical” or to make it more directly associated with activities and places in the normal, geographic world.

2. “Chapter Two” — Virtual Worlds in Mobile Devices
The PDPal series of projects were a collaborative effort to turn to mobile devices that were connected to the Internet. The reasoning behind choosing the PDA (personal digital assistant) was that they were a digital device associated with digital networking, only they were chiefly designed for use while one was engaged in a physical world activity — walking or at least away from the canonical digital device, the desktop personal computer. These projects force the user to consider their surroundings by asking them to record the activities going on around them. By looking up and engaging the physical world, the user’s appreciation and attention to the physical world increases. This application runs counter to many of the early PDA softwares, which were simply smaller versions of the same desktop applications one was used to, with no evocative link to the physical world. To actually create the connection between physical and virtual, PDPal had an online community where people could upload their recordings to be shared in a PDPal virtual world.

Two art-technology projects (theory objects) I developed on my own used the developing technology known as WiFi, used for wireless digital networking. I chose to use WiFi because it was most often associated with mobile computing, hence offered a new (at the time) style of networked computing that is happening in the physical world — at a cafe, in a park, airport, etc.

My research agenda was to find ways in which WiFi could create connections between the physical world and the virtual world of the Internet. WiFi was suitable because it is safe to assume that one is going to be “closer” to and more attentive to activities going on in public, physical places.

I created two devices to allow me to learn about how WiFi in quotidian physical places could allow a user to simultaneously be online in the virtual world of the Internet, while still attentive to their physical world surroundings. WiFi.Bedouin was an apparatus that created a fully mobile wireless access point that could be carried in a backpack. It was one of the first instances of making WiFi mobile. The second device called WiFi.ArtCache, performed a similar function, but provided a unique networked service by reversing the notion that digital media is infinitely reproducible. All of the digital resources on the device — files, media, etc. — were limited in quantity and were unique in the sense that one had to be physically within range in order to download the media. Thus I was able to study the affects of an enforced scarcity of digital media and begin to question how scarcity of resources in virtual worlds can teach us about scarcity of real-world resources. (This project won the “People’s Choice Award” in 2006 at the International Society of Electronic Arts Biennial Festival and Symposium.)

The capstone to my work in connecting the physical and the virtual was a project I conducted during my first summer as a professor at USC. In collaboration with Peter Brinson, another faculty member, I created a project called “Vis-a-Vis” (Face-to-Face) that was a kind of digital binocular. The device was a viewing portal into which one looked at a scene rendered as a video game world. The scene was fairly conventional, except that it was rendered on a small, flat-screen, portable Tablet-PC. One held the Tablet-PC in front of oneself, as if reading a newspaper, and as one pivoted the screen to the left or right or tilted it up and down, the point of view seen through the screen trained appropriately, as if the Vis-a-Vis device were a magic portal into the game world.

This project was an early attempt at extending the connection between the virtual and the physical to a very direct, physical movement interface. This project lead directly to project designs and experiments in connecting physical movement to virtual worlds activities, such as effects in electronic games.

3. “Chapter Three” — Virtual Worlds as in Connected Devices
(Preliminary work, in-progress.)
The projects I am working on presently are very much in-progress. They are experiments in connections between virtual and physical worlds that operate “asynchronously.” This means that the data gathered in the physical world is not immediately sent to the virtual world. The purpose of this particular design strategy is to explore the possibility that some connections between the virtual and physical need not be “always on.” The notion that the world will be blanketed by continuous network coverage is, I believe, a conceit born out of the hubris of engineers and network service providers. With every possibility, there are very many alternative possibilities. In this case, I would like to explore how connections can be partial and “seamful.”

For this stage of my current research, I am designing devices and software for a series of electronic games specifically designed for non-gamers, such as myself. The first translates “ambient” physical activity into game play events. The second uses CO2 sensors to capture the level of pollution in the micro-local ecosystem, turning pollution levels sensed into a game about saving the world. The third is an extension of Eric Paulos’ and Liz Goodman’s Bluetooth sensing “Jabberwocky” project, turning the concept of urban “Familiar Strangers” hypothesized by Stanley Milgram into a game that forces the player to recognize the density of invisible connections between people in the world.

The first project uses a device I designed to record ambient physical activity by the device’s owner. This movement is richer and more subtle than that captured by a pedometer, as one of my design requirements was that it be able to capture nearly any kind of physical, body-based activity. This kinetic activity is translated to “energy” or “power-up” in an electronic game context, which is still to be determined. This project is the framework for what I imagine to be a style of gaming I’m calling “offline gaming” in which a player can play the game without having to be sitting in front of a screen or distract themselves from the immediate, physical world activities immediately in front of them.

The second project uses another device I am designing to capture the levels of CO2 in the immediate environment. These readings are accumulated and become “negative energy” in an electronic game. This negative energy will result in a decline of some sort in the game world, which can only be mitigated by the player engaging in some physical world activity that is generally perceived to be eco-friendly, such as riding their bike, recycling or driving an eco-friendly vehicle.

Both of these projects continuing my research in the area of connecting the physical and virtual, or “1st life” and “2nd life” with the purpose of encouraging behaviors that will hopefully lead to more habitable, life-affirming, sustainable, healthful worlds.

For the last 10 years, I have been working on ways to represent and visualize physical, geographic space using portable, mobile, networked digital technologies. (Cf. an Extended Research Statement.) This work makes use of the many mechanisms available to describe geographic space — including mobile devices, wireless networks, maps, positioning equipment as well as theoretical scholarship on the topic of how humans conceptualize, imagine and occupy space.
Space and the ways it is occupied and represented is a significant pivot upon which humanity’s place in history balances. Arguably the most tenacious and intractable conflicts of any age rest upon the stakes and claims individuals and civilizations place upon space and territory. Personal and family lives are referred to through the places and locations of significant events. Quotidian experiences are often linked directly to specific locations. Capturing a mood for most any human experience requires specific consideration of space and time.
This work has taken the form of essays, commissioned and exhibited art and technology projects, public lectures, structured professional society workshops and keynote lectures, public workshops, conferences, interviews, book essays and several years of blog essays covering topics related to the social, technological and political aspects of space, place and geography.
My inspiration comes from an interest in the changing ways in which space has been imagined, written about, fought over and visualized over the course of human history. I have approached this work through commissioned art projects and exhibition in art contexts, professional consulting to the mobile technology industry, university teaching, writing, the construction of experimental devices, lectures and hands-on workshops.
The digital networked age has the chance to create a significant inflection point in the evolution of the ways space is represented. In a similar way, the history of film can be seen as an inflection point in the evolution of story telling. This continuing research vector works towards the goal of empowering the imagination to experience the spatial world differently by developing creative projects, instruments, idioms and metaphors, vocabularies and approaches to teaching — a “practical language” of space for digital interactive media.
Learning how to represent space through the lens of digital media is a crucial ingredient for understanding how to become better behaved occupants of the planet. And my research instincts say that there is something particularly important about learning how to do so with the new mechanisms for sociability made available in the digitally networked age.
The questions I address through my work are inspired by the fluidity with which space can be represented. There are well-known examples that describe the cultural and historical shifts in the way space is represented. For example, Australian aborigines use what anthropologists refer to as “Songlines”, which are cycles of songs whose rhythms and ordering represent navigable paths that can cover vast distances. Micronesian sailors have been thought to use star paths for their long, open-canoe sea journeys, sharing none of the tools of navigation and landscape representation we take for granted, such as the flat map, compass or sextant. By recognizing the multivalent ways in which cultures have shifted the ways they live within space, it becomes possible to reorient our current tenancy to encourage more habitable, less deleteriously impactful occupancy.
The goal of this continuing work is to find ways to empower the imagination to see the world differently — if only for a short time — so as to reflect upon our condition and perhaps have the courage to reshape one’s world towards a more habitable, playful, life-affirming one.
My approach to achieving this goal is to mix pedagogy, research and project execution. Through courses and public and invited workshops, I am trying to emphasize learner-centered education, with an emphasis on hands-on exploration of meaning, creativity and experimentation. Through my own research and projects, I attempt to continuously evolve my own knowledge and share that process publicly through my own very frequently published research bulletins. I use digital networks and various forms of social software to disseminate my activities, and share both high-level research insights, as well as practical aspects of projects. I do this in the spirit of open design and learner-centered education. It is my goal that sharing the details of my research — what I am doing, how it is being done — will encourage peers and students to feel comfortable exploring and experimenting themselves. These bulletins often become courseware for classroom and workshops very soon after they are published online. In such a way, the topics I cover tend to be fresh, immediate and representative of current trends and areas of interest within the relevant research arenas.
I am committed to professional service within the various disciplines wherein I feel my research has a voice. I have served as a session co-chair for a number of professional society workshops, as well as a papers reviewer for the Ubiquitous Computing society’s annual conference, amongst several other peer-reviewing assignments for various other conferences. I have juried several international art-technology festivals and competitions, and served as a peer reviewer for a special issue on Urban Computing for the IEEE’s flagship journal “Computer.” In 2007, I was elected as an international reviewer for grants for the Israel Science Foundation.
As I am multidisciplinary in my education and background, having degrees in both engineering as well as the humanities, I layer my research approach by combining physical construction with written scholarship. Because my research extends traditional engineering disciplinary boundaries, I combine engineering with creative practices and exhibit these works as art-technology projects.
All of my work has been referred to as exemplars of interdisciplinary research, and has been exhibited internationally, specifically in the United States, Western Europe (Spain, England, Scotland, Holland, Germany) and Asia (Japan, South Korea). Many of the projects were supported by competitive commissions or invited for inclusion in juried exhibitions, festivals and conferences.
Several of my art-technology projects were included in the Rhizome Art Base, an invitation-only, curated database of internationally significant and influential art-technology projects. In 2006, my project “WiFi ArtCache” won the Audience Choice Award at the International Society of Electronic Arts Symposium and Festival in San Jose, California. In 2005 I was a finalist for an international Creative Capital Grant, selected from an initial pool of more than 2,200 applicants to reach the final round of 120. In 2004, “Pussy Weevil”, a project I developed to explore physical interaction in 3D space, which anticipates the trend of physical interaction with gaming as seen by the Nintendo Wii, was selected for exhibition at the prestigious 26th Ars Electronica Festival in Linz Austria. In 2003, my collaborative project “PDPal” was competitively selected for exhibition on the Times Square (NYC) Panavision screen, where for three months a video we produced was presented at the 59th minute of every hour. Also in 2003, the University of Minnesota Design Institute commissioned myself and the two other PDPal collaborators to create a PDPal-inspired map of Minneapolis-St.Paul, which they published in their limited edition Twin Cities Knowledge Maps artists maps set. In 1994, I wrote one of the first scholarly essays on video games and culture on the topic of urbanism reflected through the popular computer game SimCity. The essay was published in the journal “Socialist Review” and then solicited for re-publication in the book “SimCity”, part of the popular “Ludologica” series of books on game scholarship. I was a member of the creative team that produced the Peabody Award-winning “Sonic Memorial” radio and web documentary on the events of September 11. An essay I wrote in 2006 on the rise of digital networked sensors and the social-political ramifications therein developed into a series of invited public lectures, conference keynotes, invitations to facilitate workshops and press interviews. The essay was designed to lay the groundwork for a series of projects on the topic, which I intend to define the work I pursue over the next 3-4 years.

Great questions from Sascha Pohflepp that helped me figure out what the heck I do and why the heck I do it.