Positive And Negative Aspects That The Reviewers Education Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

This paper follows up and reports some of the same techniques and results that were published in a recent ACM ITS 10 paper and that they do not cite or discuss. The authors need to explicitly address this in the rebuttal. Apart from that, the reviewers gave generally high scores to this paper, although relatively minor weaknesses were also highlighted.

Double publication concern:

Before I detail the positive and negative aspects that the reviewers

commented on, I need to address several issues related to publication


This paper describes a system that has been published before by a

superset of the current authors (ACM ITS'10).

Towards a teacher-centric approach for multi-touch surfaces in

classrooms. In ACM International Conference on Interactive Tabletops and

Surfaces (ITS '10). ACM, New York, NY, USA, 187-196.

DOI=10.1145/1936652.1936688 http://doi.acm.org/10.1145/1936652.1936688

This previous publication also reported on a study. R3 argues that the

work presented here provides significant extension on the previous paper.

However, I have been a reviewer for the previous paper and I have several


-The previous paper described already the interaction techniques

(although without the solid design justification from this paper). How is

then possible to present these as the second contribution of this paper


-The previous paper might report results on an version of the experiment

that was already partially reported in the previous paper (they might

have added new groups).

-The authors do NOT cite, mention, or compare the current work with the

work already published. This is particularly surprising because the

authors did not maintain anonymity in their current submission, making it

easy for both reviewers and chairs to find the other paper.


Almost all reviewers commented that this is a very well-written paper. R2

and R3 comment on how they learned something and how there are few

studies that look at the application of multi-touch tabletop technology

to large class scenarios as these. R4 also thinks that the combination of

a formative and a summative evaluation is strong, and the evaluation is


R1 is probably the most critical, citing that a lot of what is presented

in the paper is not really novel, a concern somewhat shared by R3.

R1 and R4 express concerns about the value and origin of the design

requirements. More specifically, R1 does not believe that most of the

design requirements are not obvious or can directly derived from a

general educational perspective. R4 thinks that Design Requirement 4 is

artificial and put there just by convenience. R1 thinks that DR5 is not

really a requirement.

R1 and R2 commented on a possible longitudinal study. I think that a

longitudinal study is not necessary to make a paper like this a

publication-worthy; however, these comments from the reviewers are more

relevant in the light that at least a part of the results from one of the

studies might have been reported before.

Finally there are some concerns by R2 on the lack of discussion of how

the data was collected and analyzed, of how the results would generalize

to other tasks (there are many different tasks that can take place in

classrooms), and how some conclusions are not traceable to the reported

data (Visiting students is a burden?).

It should also be noted that the paper is almost half a page over the

page limit.

Points to address in the rebuttal:

-the issue of double publication/the novelty of the contributions if

considered against the ITS paper.

-the relationship between the data reported in both papers

-the lack of a discussion on the data collection and analysis methods

-the lack of a discussion on the generalizability of the results

------------------------ Submission 346, Review 4 ------------------------

Reviewer: secondary

Overall Rating

4 (Probably accept: I would argue for accepting this paper.)


3 (Knowledgeable)

The Review

This paper describes two studies, one formative and one summative, of

user interface techniques geared toward supporting the use of multi-touch

tabletops in school classrooms. Through analysis of video data of

teacher-student interaction and application use, the strengths and

weaknesses of the different features designed into the classroom

management system are presented.

This is a well-written paper with good quality prose. I appreciate the

presentation of material, and the authors clear use of both formative and

summative evaluation to assess system design. Together these studies

provide insight into how best to design tabletop interfaces.

I am also aware of no work that fully replicates the work presented in

this paper, but I am not an expert in this field. I do know that many

researchers have expressed an interest in applying tabletop technology to

the classroom environment. I hope that some domain experts are among the

reviewers for this paper to ensure that the contributions outlined here

are novel. Because I am not aware of significant prior art in this

field, I assume that the work is novel and evaluate it as such.

While the paper is well-written and significant analysis was clearly done

by the authors, there are components of this paper that seemed less than

impressive to me. It seems to me that the use of tools observed in the

final design could easily have been predicted based on an understanding

of the teaching process. Teachers' goal in assessing student learning

during a classroom activity, the _reason_ they wander around the room, is

to ensure that groups are staying on task, making progress toward a

solution, and that all group members are participating. Dealing with

issues like occlusion of material seem immaterial and childish in a

design that adapts in an informed way to teachers' goal in classroom

management. It almost seems some features, particularly DR4, was

inserted because it was assumed it should be important, rather than

arising naturally from an understanding of users. Even in the formative

study, it seemed that this feature was promoted by the researchers, as

opposed to being an issue that arose naturally from teachers' goal. We

saw occlusion, asked teachers about it (Prompting them, essentially) and

they said, "Oh, yeah, that'd be nice." It seems almost that some of the

design features were promoted by multi-touch researchers pushing their

view of technology, and this researcher bias persists in classroom

layout. Consider Figure 1. Is it really a good idea for the teacher

table to be positioned in such a way that teachers need to turn their

backs on a class? One thing any elementary of junior high school teacher

will tell you is that classroom control is all about constant monitoring

of the physical classroom environment. Hunched over a desk facing away

from students is not a natural orientation for teachers.

While I do have minor concerns with the work, as outlined in the previous

paragraph, I think the evaluation of a deployed solution using teachers

and students in realistic settings was largely effective in evaluating

the use of tabletop technology for classroom learning. I still don't

believe that there exists much benefit to this technology over

paper-based interaction in classrooms. I believe that better solutions

can be designed that are less expensive and more versatile in terms of

monitoring and sharing information than the electronic tabletop interface

described, and that these solutions need not have the inherent drawbacks

of forcing students to work with virtual artifacts when tangible

artifacts are more effective. However, the goal of this paper wasn't to

argue that tabletops were a panacea for the education system. Given that

we want to create this artificial market for tabletops in classroom

(analogous to the artificial market that has been created for

smartboards) I can conclude that this design study is a useful step in

this direction.

------------------------ Submission 346, Review 1 ------------------------

Reviewer: Program Review Committee Member

Overall Rating

3 (Borderline: Overall I would not argue for accepting this paper.)


2 (Passing Knowledge)

The Review

This paper has a lot in it that is really not overly novel. For

instance, the following design requirements that were elicited from

instructor feedback:

DR1: Lots of computer assisted learning software has this as a principle

feature. Here's a naive version that shows up when I first google for

related examples http://www.neuber.com/usermonitor/index.html, but there

are plenty of examples of similar systems that allow for monitoring and

control. I don't think this design requirement is novel.

DR2: Same thing as DR1. Perhaps if the design requirement touched on

novel aspects of the table or novel methods of interaction that were

needed for instructors to control views that would be good, but as is

it's not a very significant revelation.

DR4: Occlusion and orientation of artifacts on tables is a well known


DR5: Interesting, but I would like to know more. Would teachers actually

take the time to review and trace back through student activities to

build individual evaluations? The evaluation suggested that this wasn't

very popular, and that fits in line with my understanding of the time

pressures teachers are in during the classroom.

The evaluation is ok, but I would have liked to see (of course), a

longitudinal study. Table top computing is really new, and there is a

huge novely effect. That teachers would rate it highly is not a

surprise, it's pretty cool stuff. But would they rate it so highly after

it is deployed in their classroom for a year? And compared to other

IT-based classroom management tools, is it really much better? What's

novel about this system that makes it better? This is only alluded to in

section 4.3 ("Two teachers, who had prior...") but the authors don't dig

into why their system is better. How many other teachers used classroom

management tools and didn't find this one better?

------------------------ Submission 346, Review 2 ------------------------

Reviewer: Program Review Committee Member

Overall Rating

4 (Probably accept: I would argue for accepting this paper.)


4 (Expert)

The Review

This paper investigates interaction techniques that will effectively

support teachers within a tabletop classroom. It reports on a pre-design

study that identified potential issues from the teachers' part and

provided the design requirements; exhibits the design solutions that

addressed these requirements; and provides the reader with the findings

from an evaluation study.

Overall this paper is well written; it is clear, easy to follow and well

argumented. The design process described is robust and the findings

reported are quite interesting. It is true that there are very few

studies reporting on an ecology of multi-touch devices within a classroom

environment and for this, the contribution of this work is valuable for

HCI. On a personal note, I would like to see similar studies carried out

in real classrooms as I would suspect there is a whole range of other

challenges and difficulties for the teachers involved. Still, this work

identifies some real issues of management and control that can occur

within a real classroom.

However, there are a few issues to considerate. Firstly,it would be

useful if the authors added some more details about their data collection

and analysis. While the method is briefly stated, there is no mention of

any detail on how the data was collected and/or analysed - that is both

for the observational and the questionnaire data.

Secondly, the 'Related Work' section is somewhat confusing. Usually

related work is preceding and is used to frame the findings reported in

the paper, while a discussion section is more appropriate after the

findings. What the authors provide here is something between a discussion

and a related work section and some disambiguation is necessary.

Thirdly -as the authors briefly mention in the conclusion, it would be

interesting to see if the same issues of design requirements and/or

implementation emerge in the context of a different task/activity. Also,

a long-term study would be advisable to better understand the effects of

the tabletops in the children's engagement and determine possible false

indications due to the novelty effect.

Some other issues to address: a)why the authors state on page 6 (Design

Solutions) that visiting student tables was a burden? The teachers do not

seem to mention this in their reports.

b) on the teachers' feedback section, it would be nice to know why this

one teacher did not rate the system positively.

c)references were not alphabetised,

d) paper exceeded the 18-page limit

------------------------ Submission 346, Review 3 ------------------------

Reviewer: Program Review Committee Member

Overall Rating

5 (Definite accept: I would argue strongly for accepting this paper.)


3 (Knowledgeable)

The Review

I thoroughly enjoyed reading this submission. It was well-written, and I

feel like I really learned something!

The authors address the problem of how to design a meaningful

multi-display environment for classroom teaching. Their particular focus

in this context is multi-touch table-based interfaces where students

collaborate with one another. The goal of the work appears to be to

explore how to design an instructor-centered counterpart to this system

in the classroom. The authors present a formative study where the

evaluate the potential for the instructor's system, actually go ahead and

design the system (providing considerable detail on this), and then

finally evaluating the system as a whole with several classrooms. The

contribution of this work is a set of design principles for a

teacher-centric system for monitoring and managing tabletop collaboration

classroom systems.

This piece of work extends some work that was published at the ITS 2010

conference, and adds substantial new content. In particular, the ideas

coming out of the study from ITS (and presented here as the "preliminary

study") have been (appropriately) reframed here as design

recommendations, and used to iterate on the existing design. This

discussion of the iterations and the design recommendations are

interesting. While the ideas are not groundbreaking, they provide a

logical "next step" in this particular application space that are

appropriate. This submission also provides an additional study that is

used to evaluate the subsequent design, providing additional observations

on how such a system might be employed in the real world.

This is an interesting, comprehensive piece of work. It raised some

interesting questions about the pedagogical uses of this kind of tool.

What are the learning objectives that can be achieved using such a tool

in a classroom? In some sense, this is merely a platform (so in theory,

facilitates "anything"); however, its design implies a certain way of

interacting with information and classmates (along with the teacher):

what implications does this have for learning? I imagine also that

teachers enjoy walking around a classroom while group activity is going

on -- was this considered in the design? Would an "iPad"-like version of

the teacher tool be useful? And, how would it be used/used differently?