A Wearable Gestural Interface Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

This paper gives an over view of a wearable gestural interface ,"The Sixth Sense", developed by an young Indian scientist Pranav Mistry of MIT. This device is designed on gestural recognition technology .We give input to our system by gestures of our hands. This wearable device is mobile and can be carried any where a user wants to. This device converts the whole world to its screen .It projects the screen onto any opaque object that's opposite to it. Later in this paper, the features provided by this device and how the motion of the fingers is mapped to the input command are explained. At the end some future proposals like changing size and usability audience is discussed.


Our grandsons laugh at us when they know that we distinguished digital and physical world.. unknown

We grow up interacting with the physical objects in the world. When we talk about objects we talk about gestures. We use gestures not only to interact with objects but also with each other like saying Namaste by folding hands and waving hand horizontally means 4 runs in cricket .it comes from part of everyday learning. Thus made Pranav interesting and he wanted to interact with computer rather than devices but with gestures. Pranav wanted to connect physical world with digital world. He want to use the same physical language to interact with digital pixels. He developed "The Sixth Sense". This is mobile and can carry with you where ever you go. This device projects the screen onto any wall near you. So by this you can carry your digital world with you. The backbone of this device is Gestural Interface Technology.

Gestural Interfaces

Gesture recognition is a way of interpreting human gestures via computer algorithms. Gestures are any bodily motion that comes from the face or hand. Near future is mainly focused on face and hand gesture recognition. Computer vision algorithms play a main role in interpreting the sign language. In this there's a camera that senses the movement of hands or fingers or face and analyses the motion and converts the motion into the input of the system.

Gesture recognition helps humans to interact with machines directly without any external mechanical devices .Using this we can map our finger to a mouse pointer or any pointing device on the screen and control the system as we do with an input device .Gestural interfaces limit the input methods to our hands. This is the way where we can connect our physical world to the real world. Some of the input commands are pointing, selecting, moving, tracking, tracing, entering, dragging, pushing, pulling, displacing, directing, grabbing, releasing, hitting, rubbing.

According to Cadoz, hand gestures serve three functional roles, semiotic, ergotic , and epistemic. The semiotic function is to communicate information, the ergotic function corresponds to the capacity to manipulate objects in the real world, and the epistemic function allows us to learn from the environment through tactile experience.[4 ]

Perceptive and Multimodal User Interfaces

Two main scenarios for gestural interfaces can be distinguished. One aims at developing Perceptive User Interfaces (PUI), as described by Turk [5], striving for automatic recognition of natural, human gestures integrated with other human expressions, such as body movements, gaze, facial expression, and speech. However, in this paper the focus is on using hand gestures given purposefully as instructions, this falls within the second approach to gestural interfaces, Multimodal User Interfaces, where hand poses and specific gestures are used as commands in a command language. The gestures need not be natural gestures but could be developed for the situation, or based on a standard sign language. [ 7]


Wearable gestural interface consists of a camera to track the gesture, a projector to display the screen and to map the fingers and a smart phone or a laptop to connect the device to the internet cloud. The GUI projector projects the display screen or input area onto a wall or a paper or any solid opaque object that is right before the device. The camera may be mounted on our cap or it may be attached to a pendent in the neck joined with the projector hanging with it.

Our fingers are the only input devices to this sixth sense device. Two thumbs and two forefingers act like the pour pointers to the device or screen. The motion of the fingers can be tracked using the color coding technique. Each finger should have a colored knob or a sticker. The camera records the motion of the fingers and maps it with the input command gestures and determines the required task and performs it.

Gestural interaction with The Sixth Sense device.

This device primarily identifies 3 types of gestures:

1. Gestures supported by multi-touch systems

2. Freehand gestures

3. Iconic gestures (in-the-air drawings)

The figure on left shows the examples of gestures of each type listed above. Gestures supported by multi-touch are used in map browsing or increasing and decreasing of the size of images etc.

Freehand gestures are used to recognize a specific task to perform such as to zoom in or zoom out, take a snap by framing or Namaste.

Iconic gestures are similar to the work of a stylus. If u draw a shape in the air, the camera recognizes the gesture and do appropriate work for it.

From the figure, movement of hands back and forth as shown zooms the camera lens. Forming a rectangle by your 4 fingers means that camera is to take a snap shot .

Lifting thumb up and down commands the device to on a pen such that user can write something o the screen. The iconic gestures are single input gestures such that camera tracks the motion of the finger and traces out the shape user had drawn and map it to default gestures and performs tasks. Here are some default gestures.

A "star" indicates that user want to see weather information

" @" symbol denotes that user want to check his emails

"X" means to undo the operation.

And a magnifying glass means to go to the map

What is Sixth Sense?

Humans pursue the world with their five senses, Eyes, nose, tongue, ears and skin. We tend to do everything in our day to day life with these senses that means we interact with the world by these senses. This sixth sense helps us to interact with the digital world, the world which we have been interacting with many mechanical devices these days. Sixth sense interprets physical gestures into computer input, majorly hand gestures plays a major role in this sixth sense technology.

What our "The Sixth Sense" gives us:

The Sixth Sense affords any things that we depend on a Computer on our day to day life. Almost all things that we can do with a PDA or a laptop can be done with this if we have one.


All the world's a screen:

This sixth sense device has no screen. So it converts the whole world around us into a screen. The projector we use contains a mirror which projects the display on to any object before it. You can project on a wall, a card board, a news paper or even on a person before you. Many gesture interfaces developed past date or not mobile. They stick to a machine or a digitally interfaced screen like "Tom Cruise super computer in the movie Minority Report". But as this Sixth Sense device converts the whole world into its screen, its mobile, as you can wear it and use it anywhere.

Take a photo snap:

We can take photo graph of any substance in this world if we stand before it. The camera which is mounted on your cap or on the pendent captures the image between a rectangular gesture that you make with your fingers. Later you can project these pictures on to a wall or a screen or any opaque object to view, modify, send, and zoom or whatever you want to do with the picture.

Sixthsense uses the multi touch screen technology with zoom in or zoom out by resizing the picture with your fingers comfortably. The most sophisticated and interesting thing here is, you can even drag the image from the wall or a screen and drop it on any computer or a storage device.

Check e-mail :

The sixth sense device displays your inbox on to a wall or anything of the world when you make a gesture "@" when the camera mounted tracks this moments of your figure it maps the symbol to the default input gesture and displays the mailbox of the your on to the wall. The user can check mails, compose mails or delete mails by using his finger as a point on the screen. The user also gets a virtual onscreen keyboard on the wall so that he may compose mails by touching the key on the keys in the keyboard.


If you are travelling and got stuck, if you want to use a navigating map service to know where you are and where to go, or simply if you want to view a map of a particular location no matter by longitude or latitude or name of a location you can draw a symbol of a magnifying glass in the air, the device a map on the screen. Using your four fingers you can navigate, zoom in, zoom out as you wish.

At the mart:

If you are shopping and if want to but a product such as toilet paper and if you want to know which brand is good and what is the range of prices of a particular in different stores. Just hold the product in your hand the camera upon you scans the product identifies it and gives you all the details about the product.

Buy a book:

If you are in a store and want to see the rating of a book the camera identifies the book if you hold it before it and displays the Amazon ratings, price of the book, reviews on the book. If you show the page of the author to the camera it will show you details about the author to including the pages in the book. If you show the paragraph in the book to the camera then it will show you citations and references of the paragraph.

Make a call:

If you want to make a call, show your palm to the camera. It will display the dialing numbers and options on you palm . You can just touch the numbers and make a call.

A person standing and if you want to know about the person before you give him a shake hand you don't need to open you PDA or log on to the facebook to search for him profile the Sixth Sense displays every detail including the name the age fields of interest , work position and everything on the shirt of that person.

Are you late for a flight:

If you hold a boarding pass is your hand and you are on the way to the airport and you the know the flight is on time or not, the Sixth Sense scans the boarding pass and displays the status of the flight, delayed time, and change of the gate on the boarding itself. Not only this Sixth Senses allows you to do many things browsing internet, weather forecast, write notes on a wall, play games on a cardboard sheet and so on.

How we map physical world to digital world:

Every input we give is by the hand gestures. The movement of your hand is everything to the camera, so different gesture of different movements of our hand are inputs to the input device.

Several mappings in the device are:

If you form a frame with your fingers it maps to a capture button of the camera.

If you show your thumb and your fore finger perpendicularly it toggles the pen.

If you move your hands back and forth it maps to the taller and wide of the camera.

If you write "A" in the air , it maps to the e-mail feature

This sophisticated wearable gestural interface do has some constraints. As this device projects the screen on to a wall or to any open world object display may be dull and little bit clear as the outer light may dominate the display. The physical constraint of this device is that its size, its 10 inch tall pendent, it can be hanged in your neck. All the information in possible only if the cloud is present and available (the cloud of Internet).

Future of the Sixth Sense:

Pranav Mistry planning to open source the code of the Sixth Sense such that anyone can develop their applications and gestures for the device so that they can use the Sixth Sense in the field they want. Pranav's next step is to convert the Sixth Sense as a fifth sense as the last fifth sense to the physically challenging people. In the near future this long pendent may be converted into a shape of a silver dollar which you can hang up any were and bring the world at your fingers.