Human Joystick Interface

Aaron Meyers deployed this turn of phrase “human joystick” during his final presentation for the course I taught this semester — “Design and Technology for Mobile Experiences.” He’s been working hard mostly on his thesis project, Torrent Raiders, but for my class he worked on programming a J2ME version of the MobZombies game that’s been percolating around the Interactive Media Division since 2002.

I’ve been interested in expanding the kinds of interfaces we have to digital worlds, and doing so to explore what computing can mean, besides the kind of computing we assume computing means. My speculation is that, to a significant degree, the “point of entry” defines and shapes what we imagine computing is, and it will not become much more than what it is so long as our point-of-entry are a flat visual display, a small squares of plastic that we push about 2 millimeters, a ball of plastic we swish around a flat surface. It boggles me when I think that this basic setup has been around, little changed, for 15 years, and much longer if you factor the computer mouse out of the framework. Boggles me.

And it’s not for lack of effort. The Wii-style wand concept has been bandied about at any number of professional/academic research contexts. Ten years ago a music conductor’s baton was the concept behind a project that, initially, was designed to control electronic music using conducting gestures. The researcher, Teresa Marrin, became so enamored with the possibility of gesture as a computer-human interface that she saw it not only as a device that could be used as a “new instrument on which to perform computer music” but also as “a model for the design of new interfaces and digital objects.” The interesting thing is that it’s more than a 3D mouse in many regards — it’s usage context is explicit in the object. The hardware is remarkably prescient:

The sensors on the baton include an infrared LED for positional tracking, five piezo-resistive strips for finger and palm pressure, and three orthogonal accelerometers for beat-tracking. Both the infrared sensor and the baton send separate data streams (including values for absolute 2D position, 3-axis accelerometer, 3-axis orientation, and surface pressure) via cable to the tracking unit, which converts the signals to the computer.

Over the last month or so, I’ve been constructing a sensor prototype that would turn the human into a human joystick for the MobZombies game. It combines a 2-axis gyroscope and 3-axis accelerometer, a microcontroller and a Bluetooth radio to transmit the data to the game display device, a mobile phone. The motivation here is two fold. First, investigate what a mobile game can be, that evokes both traditional playground style pre-digital action. Second, set up a baseline experiment for how computing can move away from the fixed office desk and make use of human body movement as an interface.

If you haven’t seen the “trailer” for MobZombies, I recommend checking it out.


(I’ve always wanted to make one of these specimen style graphics, with the ruler and callouts? You know?)

DIY Wearable Sensor

When I first cobbled the sensor together, I was hoping to use a magnetic compass like the previous version of the game controller used. It was a bit too sensitive to the RF energy created by the Bluetooth radio and I couldn’t easily find a way to separate the two without making a large design or diminishing the capabilities of one or the other. So, I turned to a MEMS gyroscope by InvenSense — the IDG300, which is a fast little gyro. I ran some quick tests which pretty much showed me that it was plenty fast. In fact, I could probably even go slower and maybe use a lower-end unit and save a few scheckles. I quickly cobbled together a bit of code bolted onto Aaron’s J2ME prototype. Turning motion is spot-on, which was a welcome surprise.


The MobZombies human joystick style interface is fun and suggestive more than it is a tectonic shift in how we interface with our devices. But, this style of interface is coming — it’s already here in some contexts. Nokia has introduced the 5500 Sports Phone with integral tri-axis accelerometer. I’ve heard tell of mobile phones that use the camera to detect and interpret motion by computing changes to the visual field. Etc.

MobZombies is a baseline experiment because I’m actually somewhat more interested in how very broad movement can become an interface to what I imagine would be a very different kind of computing than what we have today. Broad as in extended gestures beyond semaphore antics. Why? I would like to re-interpret human activities as fodder for computational expression. But this requires shifting the general notion of what computation is, which will require more than words. It will require some designed objects that express this shift through perhaps what some will see as peculiar usage scenarios.

Why do this? Why shift what computing means? Part of that answer comes from a sense that there must be much more to what “computing can become” than smaller or faster or cheaper. But, specifically in this case, my reason for thinking and doing this is because of this thing that boggles me — that the interface for the instrument that dramatically refashioned the ways in which humans make culture — whether entertainment, leisure, maintaining and acquiring social relationships, waging war, circulating knowledge, knitting together the fabrics of societies — the whole smash..that interface is set up to make you sit down and punch little plastic best. At worst, just sit down and look at a screen. I feels incredibly protozoic. There must be something beyond that computing can become. Why hasn’t that next bit come to pass? Why is “computing” so instrumentalized and so sedentary in this way?

I like to think about an entirely revitalized notion of “mobile computing” that isn’t about a small phone with a relatively powerful computer on which you’re able to run spreadsheets while you’re out and about. I’m wondering about a kind of mobile computing that puts more emphasis on the “mobile” part of that framework, where motion, in the broadest sense, is the computational activity.

1. Teresa, M. Possibilities for the digital baton as a general-purpose gestural interface CHI ’97 extended abstracts on Human factors in computing systems: looking to the future, ACM Press, Atlanta, Georgia, 1997.

A Gyroscope Game Input Control

I’ve been playing with a gyroscope lately as a game input control element for MobZombies. The challenge is to come up with a less cumbersome sensor rig, and less expensive. The sensor will need to communicate with a mobile phone because no one’s going to buy a $2000 tablet pc to play a game. The sensor should also be able to capture forward and backward motion.

The IDG300 gyroscope seemed decent, albeit “expensive” in the context of trying to make an under $100 sensor rig. But, you know..modern times and all.

I ran a few tests to see how well it captured tight and wide turns. I think it will require lots of moving average calculations. The graphs below are 20 sample averages which have also had a 16 point moving average calculation applied to them to smooth out the rough bits.

Tight left turn, basically a slow Whirling Dervish

Forward several steps, turn left, forward several steps, turn right

So, with lots of smoothing, the data looks clean enough to be able to translate turning motion into the equivalent of pressing the “A” or “D” keys to turn your guy left or right. An accelerometer will take care of forward or backward translational movement, I’m pretty sure.

The goal here really is to get to a point of capturing enough embodied movement sufficiently to use physical action as direct input into a game. This is different from the “offline gaming” idea, and closer to the “whoosh” style input.

Technorati Tags: ,

MobZombies on and New Kinds of Physical Electronic Play


No, that isn’t an historical photo of children playing MobZombies..but it may as well be. MobZombies is being featured on the web site..that’s pretty cool. I’ve started helping out on the project’s evolution, mostly in the area of lowering the cost of the device. This is important. I think it’s a very simple and pretty brilliant kind of user interface that really strongly suggests one way in which electronic games can get out of the Protozoic and into something much more like traditional playground-style play. I also think it could be a great test-bed for research in that area of physical electronic play or whatever you want to call it. Play that mixes the idioms of the playground and the world of digital screens. This isn’t quite Offline Gaming, but something in between the Wii and Offline Gaming.

Why? Not because I hate regular normal video games. I like to grumble about them, and about 90% of them aren’t particularly interesting to me. But so is about 90% of most media I’d say, so it’s not like I’m hating on games in particular.

No, it’s because there are a kind of electronic game I would really enjoy playing and, like..I figure I may as well think about and build those, if only to sate my ambition to create something that I find fun and that I think would actually produce some real impactful change. Physical play is not only fun, it’s generally agreed by everyone to be important for reasons of social development and to have a healthy, fit body.

So, related — I heard David Elkind on one of LA’s local NPR shows yesterday. He had some things to say that weren’t surprising based on his thorough-going arguments about childhood development. But, still — I found it interesting to hear his emphasis on a balance of activities that are about play in the traditional, skinned-knee variety, as well as those of the new 2nd life world of networked social interaction. His new book is “Power of Play”, which I’m looking forward to reading.

For a recent paper submission, I came across this from the President’s Council on Physical Fitness and Sports, which I thought was long defunct after I stopped hearing about that fitness test we did in 5th grade where you could win a cool patch if you got up in the whatever..95th percentile. (I never made it, although I tried as hard as I could.) It’s called Taking Steps Toward Increased Physical Activity: Using Pedometers To Measure and Motivate (President’s Council on Physical Fitness), which resonates with all of the step-counting programs out there. It’s cool. Step-counting and pedometers still feel a bit like..low res. What are kids going to want, that’s cool and vibes with their sensibilities around high-res electronic games?

By the way, the Robert Wood Johnson Foundation is committing $500 million over the next five years to fight childhood obesity. Their initial grant calls in the first two years are not quite appropriate to the kinds of projects described here — more instrumented research that gives something to people to try and does measurement of results and such all. But, the topics for which they’re soliciting proposals is still pretty interesting for anyone doing research in this area, such as perception of facilities for fitness and exercise, e.g. playground and recreational facilities.

Technorati Tags: ,

World's Slowest Instant Messenger

I’ve been fixated on a story my wandering mind told me many months ago about a little theory object that forced me to think about how the connected things in the era of IP networks always do their thing as fast as possible, approaching speeds that are imperceptible to normal humans. Almost without question, this is seen as a good thing. But I wondered what it would take to disrupt that assumption. How hard would the apparatus of connected things fight back? Would it be hard to write slow networked communications software? What is “slow” in the era of connected things? Can their be a slow instant messenger device?

I decided the best way to figure these questions all out was to sketch out what a super slow communications device might be, how it would operate, what it would create in terms of affect for those participating in the messaging, and what it could be “good” for.

Naturally enough, I ran up against all kinds of brick walls. Most people thought I had definitely gone completely misheggeneur. Why in the world would I want something that communicated really, really slow? Everything is supposed to be faster, quicker, more instant that last year’s instant. I mean, processor speeds keep flying through the roof. Broadband gets thicker and quicker. Rates go up when it comes to speed, not down. And I had no clear way to explain why I was drawn to this idea, other than trying to do the opposite of the dominant trend for the sake of seeing what other possibilities for connected digital networking there might be.


This may not be as weird an experiment as it sounds, particularly in an age where the Internet is splitting up into all kinds of tiers of service, with for-pay super high speed networks and bottom-tier, low-rent slow networks. Is it really safe to assume that we’ll always have fast networks available to us? Suppose you had to make a choice for economic reasons – you can send this E-FedEx for $43 and it’ll get there in 1200 milliseconds, or send it E-Postal for $3.19 and it’ll arrive sometime early next week, probably. I mean, it wouldn’t surprise me if this were a likely near-future world. It really wouldn’t.

So, how can slow be good? In my wandering mind I imagined a little device that slowly, very slowly, spilled a message out one letter at a time. Like a slow-scan signal from an interplanetary probe, feeding back a nice galactic photo over the course of 32 hours. Some of those messages might have a certain enjoyable anticipation to them — that’s a good thing..affect in messages where we’ve perhaps re-oriented our sense of affect for communication because we’ve been learning how to expect our communications faster or we expect less from our communication because most of the electronic kind gets all gummed up with crap and spam?

I’m also sort of speculating that this experiment might teach me more about how the “weight” of pre-digital interaction rituals can be re-invested with their pre-digital semantic heft even in the age of electronic mail. That is, can the momentum and weight (of time, of material things moving so as to make connections between people, of haptic/touch/proximity connections based on material coming in contact with things) imbue digital communications with something other than the transfer of information?

Boy, that’s out there. What I’m wondering is — what happens when I have to invest some material energy to get a message between (or from) someone and myself? That’s all this is — it takes three things to get the message going and finally delivered in its entirety.

1. Time, lots of it.

2. Commitment — the thing only works if I keep it close. If it’s off on its own, it slows down its delivery to glacial proportions.

3. Movement — I basically need to carry it with me wherever I go. And if I don’t go anywhere..if I sit at my computer all day, kind of like I did this entire afternoon and evening? That message just isn’t going to move anywhere.

At the end, perhaps a week or so, the message will start revealing itself, one character at a time.

Obviously, this is for the dedicated communicator, who enjoys the anticipation of a message from someone extra special.

Part II Is Here

Technorati Tags: , ,