Extended Research Statement

The work in mapping the connection between physical geography and virtual representations of that geography is a component of the work I have done that is relevant to the question as to what the basis of my tenure case is. In retrospect, I realize that I picked up the story in the middle, and that there is a preceding “chapter” to my intellectual work since my work as a graduate student. I would like to recast this personal narrative with a brief supplement that explains the continuity of my work since graduate school.

The vector of the scholarly and art-technology work I have done investigates ways to connect the physical world with the virtual world. The online virtual world brought about manifest changes for how we live within and socialize as human beings. It is my speculation that a “connected world” in which the virtual and the physical are linked, not only amongst human beings, but also network connected non-sentient “things” — will be the next evolution in the new networked age.

As explained in my Personal Statement, many of those “connecting things” projects took the form of creating linkages between the physical world as represented as geography, and the virtual world as in visualizations and representations of that geography. This was some early projects in exploring digital cartography, the most literal kind of link between the virtual and the physical. There was work, though, that preceded these projects that I feel is significant and crucial to a more complete description of my scholarly trajectory.

There are three “chapters” to my research work, and each has consumed between 3-5 years. Virtual reality was the main character in the first chapter, and was the focus of my master’s of engineering studies and my doctoral dissertation. Mobile devices was the focus of the second chapter and consisted of a series of art-technology commissions that lasted 4 years during the completion of my doctoral dissertation and into the first year of teaching at USC. The third chapter, which I began at the early part of my second year at USC, is focused on how electronic games that link physical world data sensing with virtual world games can address two real-world challenges — the state of the environment and adolescent physical fitness.

The first “chapter” of this research trajectory began with my master’s studies on virtual reality and the cultural and technological ways that virtual reality set itself off as distinct from physical reality. For my master’s research, I studied computer-human interaction and learned how the human body can become an interface — a connecting bridge — between the virtual and the physical. My chief interest was to study how virtual reality could possibly lead to richer, more evocative, more natural connections between our physical bodies and virtual bodies as a form of human-computer interface. This was the core of my studies as a master’s student and a doctoral student — studying virtual world to physical world connections — the human-computer interface. One chapter of my dissertation became well-regarded as one of the earliest analyses of video games, in this case SimCity, a game that links representations of physical worlds (cities) to virtual worlds (the computer simulation.) In the work that lead to this chapter — a series of studies of people playing this game — I began to recognize that the computer-human connection can be understood both at the physical level as well as the cognitively level. This realization would become a valuable consideration in the “third chapter” as I work to find ways to create connections between physical and virtual worlds that persuade people to modify their habits and behaviors.

The second chapter is about digitally connected mobile devices as connectors between things happening “on the ground” in physical environments beyond the heads-down computer desktop experience, and the virtual representation of those activities online, on the Internet. In this chapter, I created mobile devices and device-based experiences that allows users to create and share their own personal maps of the world around them. This work presaged the popularity of online, user-created maps of popular or significant locales before the introduction of the popular commercial online mapping services.

The third chapter, just now beginning, is focused on linking the virtual and the physical, where the specific parameter of the physical world is represented by two different things, for two separate but related projects: physical fitness and the physical environment. I will explain each briefly in turn. (As this work is in progress, a complete description and summary results are premature. One project is sufficiently advanced at this stage that it is being submitted for consideration at user-interface design professional society conferences.)

The first project in this third chapter is body-based fitness activities. The second project in this chapter is sensor-based readings of the “micro-local” state of the environment.

For the first project, my course of research, which is approximately 30% complete, is to create small evocative games that connect physical fitness — real-world physical activities such as might be compelling for young adults, for example, skateboarding, basketball, snowboarding, walking — with online digital games. I am constructing a “theory object” that creates a bridge between physical activity and virtual activity. This particular “theory object” is an inexpensive device that can record arbitrary physical activity, and do so in a way that addresses many of the important characteristics of a successful physical fitness regimen. The objective is to allow physical play to also appear as compelling as virtual play. The project is meant to evoke in the user the sense that their physical play will also contribute to virtual play. By connecting physical activity with virtual activity I am hoping that the propensity many youth have for online experiences will encourage them to invest their time with a mix of physical and virtual gaming.

The second project for this chapter is to create a series of games about the environment by connecting physical world sensing of the “micro-local” environment to a virtual world representation of the sensor data. I am designing and constructing a hand-held, portable device capable of sensing environmental pollutants that will produce data that can feed into a small game. The objective is to evoke concern for the environment and turn attention more directly toward environmental issues.

With both of these “third chapter” projects, my hope and expectation is that USC’s preeminence as a center for research and development of electronic, online games will further buoy the significance and importance of this third chapter of research. This kind of virtual/physical hybrid electronic game is a conceptually rich area of research and is just at its early stages. A search of the canonical idiom for this new arena — “exergaming” — on the ACM digital library yields four citations, three of which are informal interviews with leaders in the field of human-computer interfaces. Only one citation makes an extended study of pedometers as a way of encouraging physical fitness, and the paper itself yields no remarks on the construction of more complete game scenarios. In other words, the ground is fertile for defining the terms and debate of this area of gaming that connects the virtual and the physical, an area of research practice that I have called “offline gaming.”

Results and Publication
The first chapter as described above yielded a long (100+) page essay that became my master’s thesis and further investigation and study of that first chapter yielded my doctoral dissertation, completed in 2004. One chapter from the dissertation was published as a journal article, and reprinted in a book on the popular computer game SimCity. The third and current chapter serves as material for a book project with the working title “Connecting Things”, about how connections between the virtual world and the physical world can mitigate the deleterious effects of relying too much on one rather than finding a careful, connected balance between the two. The underlying theme of this book has been presented and refined in a series of invited public and conference lectures as well as small research workshops I have facilitated over the last two years. (I have included slides from several of these presentations, as well as reports and essays on the book’s main theme in the dossier.)

The book will consists of scholarly theoretical essays, research results, and designed devices of my own making (what I refer to as “theory objects” and others refer to as evocative knowledge objects) constructed as open-designs (open-source/open-hardware/open-process) so as to help explain the book’s thesis through their actions.

More Detail of My “Connecting Things” Research Vector

From VR to Mobile Devices to Connecting Sensor Things That Link the Physical to the Virtual

1. “Chapter One” — “Virtual World” as in Virtual Reality
My master’s thesis, completed in 1993, investigated how the physical and virtual became conceptually distinct. I looked closely at popular news media, science-fiction and film to understand how “virtual reality” was represented and how its representation changed what people understood to be the possibility of unique connections between the physical world and the virtual world. This was a watershed moment in the study of human-computer interfaces, and helped pave the way for expanding the study of connections between humans and machines.

For my doctoral studies and my dissertation, completed in 2004, I performed a close study of the technology behind virtual reality and computer simulations.
I did this so I could understand how technology can shape the way we think about and understand the world. My goal was to learn how to do such a scholarly analysis — and do it convincingly — so as to learn how I might myself create technology devices and objects that evoke within the mind of the technology’s user that the physical world matters.

As I completed my dissertation I had embarked on a series of “art-technology” theory objects. These were technology objects commissioned as artistic projects that continued my theme of finding ways to connect the physical world with the virtual world, and do so as to deepen a sense of commitment to the physical world. These projects were driven by a deep personal sense of anxiety regarding the state of the physical world’s ecology. My concern was driven by the enormous attention that “virtual worlds” were receiving as the Internet continued to grow in importance. I felt an obligation to research ways to make the Internet more “physical” or to make it more directly associated with activities and places in the normal, geographic world.

2. “Chapter Two” — Virtual Worlds in Mobile Devices
The PDPal series of projects were a collaborative effort to turn to mobile devices that were connected to the Internet. The reasoning behind choosing the PDA (personal digital assistant) was that they were a digital device associated with digital networking, only they were chiefly designed for use while one was engaged in a physical world activity — walking or at least away from the canonical digital device, the desktop personal computer. These projects force the user to consider their surroundings by asking them to record the activities going on around them. By looking up and engaging the physical world, the user’s appreciation and attention to the physical world increases. This application runs counter to many of the early PDA softwares, which were simply smaller versions of the same desktop applications one was used to, with no evocative link to the physical world. To actually create the connection between physical and virtual, PDPal had an online community where people could upload their recordings to be shared in a PDPal virtual world.

Two art-technology projects (theory objects) I developed on my own used the developing technology known as WiFi, used for wireless digital networking. I chose to use WiFi because it was most often associated with mobile computing, hence offered a new (at the time) style of networked computing that is happening in the physical world — at a cafe, in a park, airport, etc.

My research agenda was to find ways in which WiFi could create connections between the physical world and the virtual world of the Internet. WiFi was suitable because it is safe to assume that one is going to be “closer” to and more attentive to activities going on in public, physical places.

I created two devices to allow me to learn about how WiFi in quotidian physical places could allow a user to simultaneously be online in the virtual world of the Internet, while still attentive to their physical world surroundings. WiFi.Bedouin was an apparatus that created a fully mobile wireless access point that could be carried in a backpack. It was one of the first instances of making WiFi mobile. The second device called WiFi.ArtCache, performed a similar function, but provided a unique networked service by reversing the notion that digital media is infinitely reproducible. All of the digital resources on the device — files, media, etc. — were limited in quantity and were unique in the sense that one had to be physically within range in order to download the media. Thus I was able to study the affects of an enforced scarcity of digital media and begin to question how scarcity of resources in virtual worlds can teach us about scarcity of real-world resources. (This project won the “People’s Choice Award” in 2006 at the International Society of Electronic Arts Biennial Festival and Symposium.)

The capstone to my work in connecting the physical and the virtual was a project I conducted during my first summer as a professor at USC. In collaboration with Peter Brinson, another faculty member, I created a project called “Vis-a-Vis” (Face-to-Face) that was a kind of digital binocular. The device was a viewing portal into which one looked at a scene rendered as a video game world. The scene was fairly conventional, except that it was rendered on a small, flat-screen, portable Tablet-PC. One held the Tablet-PC in front of oneself, as if reading a newspaper, and as one pivoted the screen to the left or right or tilted it up and down, the point of view seen through the screen trained appropriately, as if the Vis-a-Vis device were a magic portal into the game world.

This project was an early attempt at extending the connection between the virtual and the physical to a very direct, physical movement interface. This project lead directly to project designs and experiments in connecting physical movement to virtual worlds activities, such as effects in electronic games.

3. “Chapter Three” — Virtual Worlds as in Connected Devices
(Preliminary work, in-progress.)
The projects I am working on presently are very much in-progress. They are experiments in connections between virtual and physical worlds that operate “asynchronously.” This means that the data gathered in the physical world is not immediately sent to the virtual world. The purpose of this particular design strategy is to explore the possibility that some connections between the virtual and physical need not be “always on.” The notion that the world will be blanketed by continuous network coverage is, I believe, a conceit born out of the hubris of engineers and network service providers. With every possibility, there are very many alternative possibilities. In this case, I would like to explore how connections can be partial and “seamful.”

For this stage of my current research, I am designing devices and software for a series of electronic games specifically designed for non-gamers, such as myself. The first translates “ambient” physical activity into game play events. The second uses CO2 sensors to capture the level of pollution in the micro-local ecosystem, turning pollution levels sensed into a game about saving the world. The third is an extension of Eric Paulos’ and Liz Goodman’s Bluetooth sensing “Jabberwocky” project, turning the concept of urban “Familiar Strangers” hypothesized by Stanley Milgram into a game that forces the player to recognize the density of invisible connections between people in the world.

The first project uses a device I designed to record ambient physical activity by the device’s owner. This movement is richer and more subtle than that captured by a pedometer, as one of my design requirements was that it be able to capture nearly any kind of physical, body-based activity. This kinetic activity is translated to “energy” or “power-up” in an electronic game context, which is still to be determined. This project is the framework for what I imagine to be a style of gaming I’m calling “offline gaming” in which a player can play the game without having to be sitting in front of a screen or distract themselves from the immediate, physical world activities immediately in front of them.

The second project uses another device I am designing to capture the levels of CO2 in the immediate environment. These readings are accumulated and become “negative energy” in an electronic game. This negative energy will result in a decline of some sort in the game world, which can only be mitigated by the player engaging in some physical world activity that is generally perceived to be eco-friendly, such as riding their bike, recycling or driving an eco-friendly vehicle.

Both of these projects continuing my research in the area of connecting the physical and virtual, or “1st life” and “2nd life” with the purpose of encouraging behaviors that will hopefully lead to more habitable, life-affirming, sustainable, healthful worlds.

Statement of Research Motivations

For the last 10 years, I have been working on ways to represent and visualize physical, geographic space using portable, mobile, networked digital technologies. (Cf. an Extended Research Statement.) This work makes use of the many mechanisms available to describe geographic space — including mobile devices, wireless networks, maps, positioning equipment as well as theoretical scholarship on the topic of how humans conceptualize, imagine and occupy space.
Space and the ways it is occupied and represented is a significant pivot upon which humanity’s place in history balances. Arguably the most tenacious and intractable conflicts of any age rest upon the stakes and claims individuals and civilizations place upon space and territory. Personal and family lives are referred to through the places and locations of significant events. Quotidian experiences are often linked directly to specific locations. Capturing a mood for most any human experience requires specific consideration of space and time.
This work has taken the form of essays, commissioned and exhibited art and technology projects, public lectures, structured professional society workshops and keynote lectures, public workshops, conferences, interviews, book essays and several years of blog essays covering topics related to the social, technological and political aspects of space, place and geography.
My inspiration comes from an interest in the changing ways in which space has been imagined, written about, fought over and visualized over the course of human history. I have approached this work through commissioned art projects and exhibition in art contexts, professional consulting to the mobile technology industry, university teaching, writing, the construction of experimental devices, lectures and hands-on workshops.
The digital networked age has the chance to create a significant inflection point in the evolution of the ways space is represented. In a similar way, the history of film can be seen as an inflection point in the evolution of story telling. This continuing research vector works towards the goal of empowering the imagination to experience the spatial world differently by developing creative projects, instruments, idioms and metaphors, vocabularies and approaches to teaching — a “practical language” of space for digital interactive media.
Learning how to represent space through the lens of digital media is a crucial ingredient for understanding how to become better behaved occupants of the planet. And my research instincts say that there is something particularly important about learning how to do so with the new mechanisms for sociability made available in the digitally networked age.
The questions I address through my work are inspired by the fluidity with which space can be represented. There are well-known examples that describe the cultural and historical shifts in the way space is represented. For example, Australian aborigines use what anthropologists refer to as “Songlines”, which are cycles of songs whose rhythms and ordering represent navigable paths that can cover vast distances. Micronesian sailors have been thought to use star paths for their long, open-canoe sea journeys, sharing none of the tools of navigation and landscape representation we take for granted, such as the flat map, compass or sextant. By recognizing the multivalent ways in which cultures have shifted the ways they live within space, it becomes possible to reorient our current tenancy to encourage more habitable, less deleteriously impactful occupancy.
The goal of this continuing work is to find ways to empower the imagination to see the world differently — if only for a short time — so as to reflect upon our condition and perhaps have the courage to reshape one’s world towards a more habitable, playful, life-affirming one.
My approach to achieving this goal is to mix pedagogy, research and project execution. Through courses and public and invited workshops, I am trying to emphasize learner-centered education, with an emphasis on hands-on exploration of meaning, creativity and experimentation. Through my own research and projects, I attempt to continuously evolve my own knowledge and share that process publicly through my own very frequently published research bulletins. I use digital networks and various forms of social software to disseminate my activities, and share both high-level research insights, as well as practical aspects of projects. I do this in the spirit of open design and learner-centered education. It is my goal that sharing the details of my research — what I am doing, how it is being done — will encourage peers and students to feel comfortable exploring and experimenting themselves. These bulletins often become courseware for classroom and workshops very soon after they are published online. In such a way, the topics I cover tend to be fresh, immediate and representative of current trends and areas of interest within the relevant research arenas.
I am committed to professional service within the various disciplines wherein I feel my research has a voice. I have served as a session co-chair for a number of professional society workshops, as well as a papers reviewer for the Ubiquitous Computing society’s annual conference, amongst several other peer-reviewing assignments for various other conferences. I have juried several international art-technology festivals and competitions, and served as a peer reviewer for a special issue on Urban Computing for the IEEE’s flagship journal “Computer.” In 2007, I was elected as an international reviewer for grants for the Israel Science Foundation.
As I am multidisciplinary in my education and background, having degrees in both engineering as well as the humanities, I layer my research approach by combining physical construction with written scholarship. Because my research extends traditional engineering disciplinary boundaries, I combine engineering with creative practices and exhibit these works as art-technology projects.
All of my work has been referred to as exemplars of interdisciplinary research, and has been exhibited internationally, specifically in the United States, Western Europe (Spain, England, Scotland, Holland, Germany) and Asia (Japan, South Korea). Many of the projects were supported by competitive commissions or invited for inclusion in juried exhibitions, festivals and conferences.
Several of my art-technology projects were included in the Rhizome Art Base, an invitation-only, curated database of internationally significant and influential art-technology projects. In 2006, my project “WiFi ArtCache” won the Audience Choice Award at the International Society of Electronic Arts Symposium and Festival in San Jose, California. In 2005 I was a finalist for an international Creative Capital Grant, selected from an initial pool of more than 2,200 applicants to reach the final round of 120. In 2004, “Pussy Weevil”, a project I developed to explore physical interaction in 3D space, which anticipates the trend of physical interaction with gaming as seen by the Nintendo Wii, was selected for exhibition at the prestigious 26th Ars Electronica Festival in Linz Austria. In 2003, my collaborative project “PDPal” was competitively selected for exhibition on the Times Square (NYC) Panavision screen, where for three months a video we produced was presented at the 59th minute of every hour. Also in 2003, the University of Minnesota Design Institute commissioned myself and the two other PDPal collaborators to create a PDPal-inspired map of Minneapolis-St.Paul, which they published in their limited edition Twin Cities Knowledge Maps artists maps set. In 1994, I wrote one of the first scholarly essays on video games and culture on the topic of urbanism reflected through the popular computer game SimCity. The essay was published in the journal “Socialist Review” and then solicited for re-publication in the book “SimCity”, part of the popular “Ludologica” series of books on game scholarship. I was a member of the creative team that produced the Peabody Award-winning “Sonic Memorial” radio and web documentary on the events of September 11. An essay I wrote in 2006 on the rise of digital networked sensors and the social-political ramifications therein developed into a series of invited public lectures, conference keynotes, invitations to facilitate workshops and press interviews. The essay was designed to lay the groundwork for a series of projects on the topic, which I intend to define the work I pursue over the next 3-4 years.

Exergaming and Instrumented Measures of Fitness

The Economist had a short article titled Let’s Get Physical on “exergaming” — gaming that combines play with exercise. At the same time, Fabien wrote a thought-provoking blog post on the Nokia Sports Tracker and Tracing Personal Mobility. This stuff got me thinking about these weak-signals around exercise, play, gaming, fitness. I wrote a little comment on Fabien’s blog about the different ways to turn sport into electronic gaming and play in some fashion, or the challenges around that. Much work is associated with GPS or pedometers and that sort of thing, or stats-based stuff.

I wonder how far beyond the more or less obvious solutions like GPS and pedometers we can go? And then what beyond the usual stats cards and graphs?

I think that the instrumented approach to measuring activity like this is quite compelling to many fitness enthusiasts. Knowing the numbers and tracking progress through spreadsheets and graphs has its appeal for those who want to measure very detailed incremental increases in their fitness. I think that there is the possibility for re-calibrating what gets to count as “fitness” so that it has a less instrumental meaning. So, rather than fitness measured as how far you can run in what amount of time, fitness could be shaped around less self-centered characteristics, such as how much CO2 your super hero avatar prevented from escaping into the atmosphere, or how many dinosaurs you saved by avoiding turning them into the fuel from their fossils. I only say this because, plainly, the GPS thing is wonky at best as you describe. In many cases it works perfectly fine — but in the off case that it does not work well, there’s a real issue in terms of user adoption or satisfaction. If you measure physical activity more “ambiently” or with less instrumentalized rigor (sum-of-squares acceleration versus meters moved), you can tap into much less expensive techniques. Also, there’s an exciting challenge there — can we redefine the culture of fitness, tie it into the booming electronic games business — all in the service of elevating ecoawareness? It’d be like crossing the streams in Ghost Busters!

Technorati Tags: , ,

Flavonoid v.03 Notes: Testing the QT113H proximity sensor

Here’s the v03 Flavonoid prototype pretty much all assembled, except for attaching a battery on back and the DS1340 real-time clock and its crystal — you can see the landing pad there near the center.. I’d be surprised if this all works. I mean..something must be wrong, right?

Okay, everythings on there..now what? I basically try to test each component one at a time to see if it works. Here I’m checking the QT113H, which is a touch/proximity sensor. The sensor can detect whether the Flavonoid is being held, or is near your body. Touch/proximity interface semantics are something I’m interested in experimenting with — more than just physical contact, but nearness as well.

What I’m doing is checking to see if the sensoris working. My finger is over a contact to which will be attached a flexible probe, like a super thin wire that can be run along the inside of whatever enclosure this ends up in. So, it triggered — I can tell because my multimeter went to about 3.3v.

But, then something annoying happened..it basically locked up, the sensor. And then I remembered that when I was first experimenting with the QT113H, it had to be recalibrated, basically by cycling the power. I never really fully understood when that had to be done, but regardless — this is a problem because, well..Flavonoid has no power switch so I can’t cycle the power on the thing, even for debugging. Drat..


I threw a tri-color LED on there for debugging and diagnostics. I think I’ll probably put a surface-mount one on there in the next version.


I put an in-system programmer jack on the board so I could program it with just about any in-system programing jig. That ribbon cable there is running to my STK500 on my Windows machine, although one of these days I guess I’ll figure out how to use the rig I got for my Mac. Anyway, I use AVRStudio’s little AVR programming interface to flash the program memory from the STK500. I comple using the WinAVR (avr-gcc) tools, which works like a charm..no problems.


Okay, back to this snafu with the QT113H..basically it turns out that you can “power” the thing directly from an I/O pin on the microcontroller. It uses so little current to run, that one of the pins on the ATmega32 can supply it. So, that means I can connect the power pin of the QT113H directly to one of the port pins, which I did here with a little light gauge white wire wrap. I had to cut a trace on the board that had been the VCC supply for the QT113H. Easy peasy. Now i can power the thing up and down programmatically. So, every reset or whenever. Convenient.


Here I attached a little bit of white wire as the sensor probe for the QT113H. Works like a charm. If I hold my hand near it, it triggers the sensor.

That’s it — proximity sensor works. Did a little redesign in the original schematic.

Next, I’ll check to see if the real-time clock is working..

Game Week Musings — Offline Gaming? A Near Future for Electronic Play?


Trying to come up with some “log line” style idioms to describe the whole vector of near future research I’m doing around game gestures that elongate the scales of motion, time and contact (proximity, touch, etc.) that electronic games have completely shrunk down to nil. There’s definitely a trend towards considering stretching these out a bit — the Wii, Warioware, Teku Teku Angel, Nike+, MobZombies. How far can you go? Can there be “offline gaming” where the screen disappears to the point of it not even being necessary? Where you sort of ambiently know that you’re gaming in the sense that your actions and activities “offline” will register in the game world once you get back to your normal human computer later? Can you still be gaming while you’re doing a run to the market, without being consciously and actively “in” the game while doing the grocery shop? But still, knowing in the back of your mind that, hey, cool! I’ll get my shopping done and probably get a +2 power up!

What’s the language and name for something like that?

* Hands-Free Gaming
* Offline Gaming
* Off-Screen Gaming

Evolution In Electronic Game Gestures Near Future Electronic Game Interfaces?

Evolution In Game Gestures?

How far can you go with a game gesture? Can shopping at the Farmer’s Market become an interface to an offline game, such that my action and activity while shopping powers-up my electronic game avatar while I’m away from it? Is our imagination open to the idea of gaming..even while not in front of a screen? Can you keep it in the back of your head that “stuff” in the online game world will be affected by your activities in the offline world? How will that change our perception of activities offline? How will it affect our attention to what’s actually happening around us? What will it feel like to know you’re gaming even though you’re not twiddling keys on your mobile phone, nor looking at its little postage stamp screen? How about this — you’re not even networked! Everything is stored up and then uploaded to the mothership later, when you’re back home ready to be an online agent again. You get to participate in what’s going on in 1st life, all the while knowing in the back of your head that your actions get to become something valuable in 2nd life!


The last couple of months has been an investigation of what this could be, through a mix of mostly technology craftwork with the Flavonoid project. Some notes, scribblings and sketches, too, but mostly construction of a theory object that I hope will help me answer these questions perhaps more expediently than head-scratching alone. We’ll see how it goes. Flavonoid came to life yesterday, so that’s promising!


Technorati Tags: , , , ,