Has anyone discussed the submissions process for professional conferences — the important, if infamously flawed, social mixer for the knowledge production class? No? I think it would be worth taking the time to do start a short introduction to the topic. Let’s give it a shot.
In the world of academic, scholarly knowledge production you produce knowledge. What gets to count as “production” includes circulating your ideas as they are framed by experiments or studies, the results of those experiments or studies.
Amongst the other knowledge production practices (talks, seminars, colloquia, workshops, keynotes — various means of “holding forth” on your special insights within a miniscule, boggish corner of “the field”) the experiment or study is the Big Kahuna. You get appropriately “long” venues for dissemination (10 pages or so for a paper) presented at an professional society conference. There are statistics somewhere about this whole business and some indications that most of this stuff is kruft that gets lost in the hinterlands of digital document depositories. A few get to stand out, but most — let’s face it — are penned and published to add a line to the CV. Nonetheless, we make our incremental contributions to “the field.” No news there.
The process goes like this. You identify your professional affinity group — I’m thinking some of my work is good for the Computer-Human Interaction set, best represented in most of the western world by the ACM SIGCHI — then begin parsing the various calls-for-papers/projects/submissions/proposals and thinking about what your current harvest of ideas, projects, experiments, study results, etc. are appropriate for responding to any number of those calls.
So, over the last two years, I’ve attempted to become a lively member of that society — participating in a workshop in 2004, involving myself in the student design competition as a jurist, and, this year, submitting some of my junior-tenure-track ideas to a workshop, and some early laboratory research (partially funded out of my own pocket) for the SIGCHI’s new Work-in-Progress call for papers.
This year for me continues 2006’s theme as My Years of Accumulating Rejection. It’s okay — as several of you out there know, I have deliberately adoped this life style of accumulating rejection. It’s not the philosophy of The Downward Spiral, but one that encourages me to attempt many things, even if I feel that achievement of those things may be far beyond my range. Shakespeare wrote about that — something like, it’s better to eat something that looks weird than not to eat at all, or something like that.
Okay, what better way to describe the magic of turning your brilliant ideas into scholarly and intellectual coin than a real life example of such an attempt failing miserably? I’ll go ahead and not bother to remove my name, as my innocence is well known to be suspect anyway, and the hideous process I describe in what follows is protected through a process that should be familiar to anyone who’s watched an episode of CSI — the anonymous grand jury, or in this case, the anonymous reviewer.
A miserable failure
I submitted a short paper within a new category called “Works-in-Progress” for the annual SIGCHI conference held every year around the winter/spring change. “Works-in-Progress” (“WiP”) call is described thusly:
Work-in-Progress submissions provide an opportunity for both practitioners and researchers to present a concise report of new findings or other types of innovative or thought-provoking work relevant to the HCI community.
Like interactive posters of the past, Work-in-Progress focuses more on visual presentation and discussion between the author and attendees around the poster. Accepted work will be published as 6-page papers in the CHI 2006 Extended Abstracts.
This submission category aims to attract attention from a broad range of disciplines covering a spectrum of topics and methodologies. The following topics are especially encouraged: multidisciplinary work, social impact of technology, design research, human-centered design and innovation, and cross-cultural design. We encourage submissions from all of CHI’s communities: design, education, engineering, management, research, and usability.
It goes on in other details, but you get the idea. WiPs are not completed studies with crunched numbers, ANOVA analyses, lots of tables and charts based on the results of experimental data processed through algorithms or stuff like that. WiPs don’t have definitive conclusions. They don’t have final answers to a research question. They’re probes — they’re testing the atmosphere of a potentially new area of research. They are work that’s, you know..like..work that isn’t done and is “in progress” and, presumably, progressing nicely enough to be thought-provoking and a decent topic of discussion, or something that can generate some excitement about a new take on an existing research area (those miniscule, boggy corners) or even suggest whole new research vectors for “the field.”
I’m a new guy. I like the idea of WiPs because at this point in my career — all my work is “in progress.” I have yet to do a full-fledged study or generate such as could go into a full paper with any hope of being accepted.
So, I sent up a balloon to check the conditions in the stratosphere of scholarship that my trusted mentors and their peers regualarly jet about in. I had been working on a project that some of you may know about — using a nifty, albeit expensive (for my self-fundedness), three degree of freedom accelerometer I came across during a research expedition to Japan. This thing is quick. It’s got negligible drift. I can program computers with the almost-the-best of them, so I can DIY hack it to talk to a whole bunch of things, including an open-source game engine called Torque, and a Max/MSP patch I wrote to control a QuicktimeVR. Now, with this sensor attached in various ways to lightweight, portable things like small TabletPCs (8.4″ display) and even smaller TabletPCs (5″ display), I now have my very own DIY, non-NSF funded platform for research into the use of fast, negligible drift, three-degree of freedom accelerometers as an interface element for first-person perspective electronic play and cinematic experiences..
What does that mean? It means I am working on (and making some decent progress) on using this kind of sensor as a game controller. I’m also working on (and making really decent progress) on using this platform to create a kind of cinematic surround experience using a portable visual display. Imagine holding up your PSP and having the point-of-view of a game change as you turn around or look up. Imagine a location-based visual story where you follow a character (cf. Janet Cardiff’s visual story projects) and can pan left or right or tilt up or down and have the camera’s perceived POV change precisely in synchronization.
Okay. You get the idea. I think this is great material for a forum that is looking for submissions that are:
Evaluation of systems, techniques and other phenomenon relevant to HCI – this can include either experimental or other types of evaluation.
Reflections from Practice – lessons learned, broad conclusions, or principles derived from practice, backed by thought-provoking and well-substantiated analyses.
Design Briefings – accounts of the design (rationale, process, outcomes and evaluation) of an innovative application, user interface, or system.
Methodologies and Tools – new methods, processes, techniques, and tools for use in interactive system design and development.
Here I’ve got a DIY platform that I think is thought-provoking, innovation, uses some new tools (open-source game engines, Max/MSP, a new sensor), it’s certainly brief (you only get 6 pages, with references and images and everything.
Well, I was wrong. But, maybe I’m taking this too professionally. I think it would be a solid and useful pedagogical exercise to show you what a real, live review looks like. For someone who maybe doesn’t have the juice of a seasoned researcher in “the field.”
Let’s call the jury in. What did the reviewers say?
Title: Vis-Ã -Vis: An Orientation Sensing Interface for Electronic Games and Digital Cinema
Reviewer: external
Overall rating: 2 (scale is 1..5; 5 is best)
Reviewer expertise: 4 (scale is 1..4; 4 = “Expert”)
Overall Rating
2 (Probably reject: I would argue for rejecting this paper.)
Expertise
4 (Expert)
The Review
This paper proposed a technique named Vis-a-vis, which uses
spatial/acceleration sensing devices as an additional input modality in
games and media navigation applications. Unfortunately, using
spatial/motion aware sensors to augment interactions had been studied
many times by HCI researchers and two relative recent work include [1]
and [2]. With the drastic price drop of accelerometers, motion sensors
had already become a commodity and had been widely used in many
commerical products (laptops, digital cameras, handheld gaming devices
etc) and research projects (text input, elder care, gaming, monitoring,
authentication etc).
In this paper, the authors didnÂ’t provide any new paradigms of using
accelerometers. all their prototypes simply take the input from the
accelerometers and map them into directional changes, which is the most
basic use of such sensors. I didnÂ’t find any interesting qualitative
results from their informal experiments either.
Suggestions: If the authors would like to explore further in the area of
accelerometer-based gaming, I highly recommend them to get a copy of
“WarioWare : Twisted” for Gameboy Advance, which includes nearly 200
accelerometer enabled mini-games. The usage of accelerometers in many
mini-games is pretty innovative.
[1] Tsang, M., et al, Boom Chameleon: Simultaneous capture of 3D
viewpoint, voice and gesture annotations on a spatially-aware display.
UIST 2002
[2] Yee, K., Peephole Displays: Pen Interaction on Spatially Aware
Handheld Computers, CHI 2003
Work-in-Progress?
This is a “work-in-progress” but the authors still haven’t successfully
defined their intended innovations at this stage.
Formatting and Style Problems
———————— Paper 318, Review 2 ————————
Title: Vis-Ã -Vis: An Orientation Sensing Interface for Electronic Games and Digital Cinema
Reviewer: external
Overall rating: 3 (scale is 1..5; 5 is best)
Reviewer expertise: 4 (scale is 1..4; 4 = “Expert”)
Overall Rating
3 (Borderline: Overall I would not argue for accepting this paper.)
Expertise
4 (Expert)
The Review
The paper describes ongoing work that investigates the use of orientation
sensors as another modality for mobile computing. This is an important
topic that should be interesting to the wide audience in CHI. The authors
clarified their research goal and scope. Two examples of electronic games
and mobile cinematic experience are described.
However, exploring more modalitiies for mobile computing is a big topic.
The authors should concretize further the problem they try to solve. The
findings of the work should also be described in a more focused way.
An important reference is missing.
Hinckley, K., Pierce, J., Sinclair, M., Horvitz, E., Sensing Techniques
for Mobile Interaction, ACM UIST 2000 Symposium on User Interface
Software & Technology, CHI Letters 2 (2), pp. 91-100.
Work-in-Progress?
Yes
Formatting and Style Problems
———————— Paper 318, Review 3 ————————
Title: Vis-Ã -Vis: An Orientation Sensing Interface for Electronic Games and Digital Cinema
Reviewer: external
Overall rating: 3 (scale is 1..5; 5 is best)
Reviewer expertise: 3 (scale is 1..4; 3 = “Knowledgeable”)
Overall Rating
3 (Borderline: Overall I would not argue for accepting this paper.)
Expertise
3 (Knowledgeable)
The Review
This paper presents an interaction technique based on orientation for two
applications running on a tablet pc. The description of the prototype
device is well done and shows that the device could be very compelling
for a certain type of application. There were also experiments
presented of each prototype application, however the results of the two
experiments were very sparse. Since this is such a short paper venue, 2
columns of references seems like too much. Some of this space could be
reclaimed for more information about the experimental results. The
experiments obviously took some time to run, so it would be worth writing
a bit more about the experiences of the users.
Work-in-Progress?
Yes, I think that future work and experiments on this device will be very
interesting.
Formatting and Style Problems
Turning Test –> Turing Test
———————— Paper 318, Review 4 ————————
Title: Vis-Ã -Vis: An Orientation Sensing Interface for Electronic Games and Digital Cinema
Reviewer: external
Overall rating: 3 (scale is 1..5; 5 is best)
Reviewer expertise: 2 (scale is 1..4; 2 = “Passing Knowledge”)
Overall Rating
3 (Borderline: Overall I would not argue for accepting this paper.)
Expertise
2 (Passing Knowledge)
The Review
This paper presents a funny and interesting scenario of using orientation
sensors for games. The scenario, technology and prototype games are
convincing, but there is little qualitative and quantitative data to
evaluate the benefit of using the orientation sensors. In addition, the
references and some parts of the text need polishing. What is a ‘Turning
test’?, for instance.
Work-in-Progress?
Perhaps too much a work-in-progress, as the authors does not give a
roadmap to further improvement of their work.
Formatting and Style Problems
I find it implausible that a new work would be dismissed based on the fact that previous work in the area has been done. Our approach and the content of our experience are, based on my review of the literature, distinct. And, certainly it is the case that continuing a vector of research, seeking additional innovations is part of the process of advancing knowledge. For instance, the “DJ Boombox” mentioned further below is another gesture-based interaction scenario, yet it contributes additional material to the field. I’ll eat my words if this reviewer can find prior art demonstrating work that uses 360 degree azimuth, +/-89 degree tilt, with negligible delay and drift as an interface to an electronic game and QuicktimeVR experience.
Drops in prices of components such as accelerometers likely means that more and more researchers are likely to contribute innovative ideas to their use. Cost and innovation are inversely related in an open community of research. It is the rare researcher who would simply do the same thing because she can get her hands on some cheap parts. As financial accessibility yields increasing experimentation we can expect increasingly innovative usage scenarios, designs, projects which is an argument suggesting that more papers, works-in-progress, posters, workshops, etc., should appear, not fewer. Arguing against a vector of research on the basis of a proliferation of components just doesn’t make sense. The same sort of argument could be made against bounding papers because they’re components have been commercialized. DIY cultures have embarrassed formal R&D communities such as ours with their innovations through low-cost, clever, insightful and entirely legitimate re-mix of existing commercial designs.
Reviewer 2
Reviewer 2 suggests including Hinckley’s early paper (from UIST ’00 — or from CHI Letters), while Reviewer 3 suggests that we have too many references. At some point it is necessary to either highlight new, previously under represented literature, or continue to cite the same literature in every paper on a particular topic. It is this author’s opinion that a paper should contribute to the field not only in its substance but in its indexing other than the usual relevant prior art, not always the same prior art.
Work-in-progress suggests that we’re clarifying our approach, precise research questions, techniques and methods. It seems that this (as well as the other) reviewer is calling for more finality in a work-in-progress, which is counter to the objectives of the call, particularly so as to receive early, substantive feedback from colleagues in the field.
Reviewer 3
It is my understanding that a work-in-progress is distinguished from a completed research project in that it has not yet or has not completely formed experimental results. Therefore, I would claim that emphasizing the ground work within the very short amount of space allocated would include the reference material and prior-art that frames the work-in-progress, not experimental results for work-in-progress that has not yet conducted formal experimentation.
Review 4
The expectation for qualitative or quantitative data in a work-in-progress does not seem to keep within the work-in-progress idiom. I’m not sure how such data would be presented when the work is not complete and experimental design is “in-progress.” And, I’m not sure it is helpful to refer to a work-in-progress as “too much” a work-in-progress. Does that mean it’s been worked on too much, or it’s progress has continued too long? In either case, a roadmap to further improvements could really only be properly assessed at the completion of a study, rather during the process of conducting the work.
Considering the review criteria — contribution and benefit, originality and innovation, clarity — I would ask that this submission be re-evaluated, with an explicit focus on responding to the paper within the context of the three criteria (Contribution and Benefit; Originality and Innovation; Clarity).
Alongside of CHI2005, for instance, our work is arguably innovative and original, and contributes to what is in fact an underrepresented, and important area of CHI research. At CHI 2005 there were a limited number of projects that focused their investigation on the use of accelerometers for manipulating visual displays, and all were late-breaking results — no full papers. It would seem that there is interest, but no furtherance of this important research vector within the field. I would challenge the reviewers to point to a completed research project using the kind of interface modality we are researching, with the technical characteristics (fast-acting, low-drift, absolute measured, 3 degree of freedom sensors) and the semantic characteristics (game point-of-view control, control of point-of-view in a cinematic digital visual experience), which are innovations described in our paper. Having presented this project at several talks, and informal presentations to visiting colleagues, I have yet to come across an individual who was aware of this sensor device, which has the unique characteristic of an unprecedented ease-of-use and off-the-shelf availability. I’m sure a large community of researchers would benefit from being made aware of this technology and our work. Similarly, the growing and important emphasis on play and games within larger cultures suggests that CHI should participate in the innovation of new styles of game interaction and interface. Arguably, this work-in-progress makes just such an contribution.
At CHI2005 I found three candidate instances of prior art. One project that used accelerometers as an entertainment innovation, and focused on movement as an interface control (The DJammer: “Air Scratching and Freeing the DJ to Join the Party.) Another used accelerometers but with significantly reduced sensor range, acuity, and with fewer degrees of freedom than the three degree of freedom accelerometer we use in our project (“The Virtual Raft Project: A Mobile Interface for Interacting with Communities of Autonomous Characters.”) A third that used accelerometers as a gesture interface for interpreting hand movements, thus not directly within the idiom of one-to-one use of absolute movement as a way to control point-of-view within a visual world. I find it hard to support the overall conclusion of the reviewers that this project does not warrant inclusion in the WiP sessions if for its contributions and innovation alone.
As to the clarity of the writing or technical issues if there is a typo or a misaligned image, I think that should be subject to reconsideration as a professional courtesy — everyone makes this mechanical errors, and CHI professionals in particular should understand the challenges spelling checkers have when a misspelled word is another dictionary word spelled correctly.
Overall, I found the WiP reviews, despite the expertise of the reviewers, to be without foresight. Although I would expect SIGCHI to perhaps be conservative in the area of “long papers”, I am quite surprised that this new WiP call would reject work such as this.
Another miserable failure
I submitted what I thought was a decent POV for the SIGCHI Workshop on Mobile Social Software. The POV was called What’s Your Social Doing In My Mobile?, a category treatment breaking down three conceptual idioms within the mobile social software arena — spatial annotation, proximity-based interaction, and presence awareness. (In the longer essay, which’ll become a book chapter someday, it’s four — add on media sharing.)
Politely rejected.
Why? I don’t know. They didn’t send reviews and I’m still waiting to hear back from the organizers after I sent a note asking for comments. Might not hear back. So much for transparency.
I suspect they were looking for less fluffy such things and wanted some stats on privacy technology and such all.
Onward
So, there you have it. It’s a tough game. I enjoy playing it immensely, despite the pain. I don’t have expectations that I will obtain more than an average and low acceptance record, particularly because I play at the boundaries of acceptable theory, objects, practice and approach to knowledge production. That’s okay — I enjoy being at the edge, much more than in the center. I am also much more satisfied than this post might sound — not perfectly satisfied, but okay. And I am really happy to have a mechanism for circulating what I think to a wider audience. Sharing such things as these reviews makes it a bit more tolerable.
Accumulate Rejection. It means you’re out there taking risks, probing for your audience, and continuing to circulate ideas, insights and culture. It’s kind of like being an actor, going on auditions. Or worse, a comedian, playing the circuit.