Overview On Virtual Air Writing Technology Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

This chapter describes a brief introduction about the project Virtual Air Writing (VAW) , the literature review and the main objectives of the project. Furthermore the methods and techniques that are used in the project are also described in this chapter.

1.1 Brief Introduction

VAW is all about human gesture recognizing using Nintendo Wii remote as a gesture device .When a user holding a gesture device in one of their hands performs a gesture then our system recognizes those arm motions made by the user and on success invokes a virtual key board event. To recognize the human arm gestures two different techniques are used i.e. Dynamic Time Warping (DTW) and 2D trajectories. These techniques are described in detail in the chapter No.6.

1.2 Literature Review

The literature review helps us to know what the gestures are and how to perform them. To study pattern of gesture recognition we have gone through many research papers and articles published in the IEEE Consortium [1...4]. Before going into the implementation the research was made that is related to our project like (Wii Remote Libraries, Visual Studio 2008, Gesture recognition pattern matching and some of the already build Wii Remote software's) which is discussed in the chapter No.2.

1.3 Project Background

Hand gestures are the natural form of non verbal communication used by human beings. People interact by using different gestures every day, even without knowing it. In recent years a wide range of study has been made, in the field of Human Computer Interaction (HCI). The VAW interactive application recognizes the human arm motion made by the user. Gesture based recognition is a highly useful technique for computer interaction because recognizing gestures as input allows computers to be more accessible for the physically-impaired and makes interaction more natural in a 3-D virtual world environment [21].

1.4 Research Question

'Virtual Air Writing'' facilitates the user to interact with the system using Nintendo Wii remote as an input device.

* Determining a gesture recognition technique that recognizes 3d gestures in VAW application?

1.5 Objectives

VAW facilitates the user to interact with the system through Wii remote as a gesture device. The goal of our project is to recognize the gestures/arm movements made by the user using Wii remote. Our system currently recognizes numeric characters from (0'9) and mathematical operators (+, - , /, and *).

1.6 Methodology

User sitting in front of the TV/Computer shall make simple arm movements by holding the gesture device in one of his/her hand. Our application recognizes the arm motion made by the user and on the success invokes a virtual keyboard event. Wii remote has 3 built in accelerometers that provide 3d motion sensing capability.Wii remote is connected with the system through blue tooth device. We get the accelerometer events i.e. x, y and z axis the time when Wii remote connects with the system. To analyze these accelerometer events and to recognize the arm motions made by the user two different techniques are used i.e. Dynamic Time Warping (DTW) and 2D trajectories. These techniques are described in detailed in Chapter No.6 .

1.7 Thesis Outline

Chapter 2 describes the related work in the field of Human Computer interaction(HCI) .We have described some of the Wii remote projects and how these projects are related with our project. The Arm gestures are defined in chapter 3. The detailed description of Hardware Specification of Wii remote is present in Chapter 4; this chapter also focuses on 'How to Communicate with the Wii remote'. Chapter 5 provides the Detailed Design and software requirement specification (SRS) including functional requirements and the UML diagrams for our project. Gesture Recognition techniques have been described in Chapter 6. Chapter 7 provides the interfaces of the application, User test cases and results are present in chapter 8 and Chapter 9, and the report is summed up in the chapter 10 having conclusion and future work.




This chapter describes several technologies that have associations with our study. The Nintendo' Wii Remote has been assembled as a revolutionary technology. Development of the Wii Remote began in or around 2001, coinciding with development of the Wii console. Wii remote was first time introduced in the field of computer in 2006, by Johnny Chung lee from Carnegie Melon University, he had introduced Wii remote into the field of HCI as an Infrared device for hands free Interaction, the first project under his compendium was Wii remote White board followed by Head Mounted Displays and Virtual Reality. In the following decade, many students and researchers have worked and developed many interactive applications using Nintendo Wii remote, this field is still under research and development after the launch of Wii Console, in our case it is Accelerometer based approach.

2.1 Wii remote projects:

Following are some of the projects related to Wii remote:-

* Wii remote Smooth Board (Multi Touch White Board) [5]:

Developed by Goh Boon Jin easily transforms your flat screen display (projected screen or flat panel) into a interactive whiteboard with just a Wii Remote and IR Pen. This software provides an intuitive interface that will enable users to interact with their computer directly at the screen. Smooth board also allows you to control the computer when you are away the screen with the use of the Wii Remote. With Smooth board, you will have impressive presentations that can be free from the mouse and keyboard.

* Wii Earth ( Wii Remote Interface for Virtual Earth) [6]:

Developed by Paul Wegener allows you to use your Wii Remote to navigate (and zoom-in on) the Earth, using Google Maps and similar services. You can switch between viewing with Open Street Map, Live (Maps, Satellite) and Google (Maps, Satellite, Terrain).

Figure 2.2 Wii Earth [6]

* Head Tracking for Desktop VR Displays using the Wii Remote [7]:

Developed by Johnny Chung Lee in 2008, using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space.

Figure2.3: Virtual Display for head tracking[7]

* Nintendo WII ( as a gaming Device) [8]:

Revolution Nintendo breaks with more than 20 years of video game history by abandoning the traditional controller held with two hands, and introduces an innovative freehand-style unit held with one hand. The intuitive, pioneering interface allows players to run, jump, spin, slide, shoot, steer, accelerate, bank, dive, kick, throw and score in a way never experienced in the history of gaming. When picked up and pointed at the screen, the controller gives a lightning-quick element of interaction, sensing motion, depth, positioning and targeting dictated by movement of the controller itself. The controller also allows for a variety of expansions, including a Nunchuk style analog unit offering the enhanced game-play control hard-core gamers demand. Several games are already revealed at the E3 gaming conference, Las Vegas 2006. Users were able to play real-life golf and baseball, but could also catch a fish by raising the controller fast enough.

Figure2.4 : Nintendo Gaming Devices[8]


The accelerometer based approach enables three axis (x, y and z) motion sensing. The acceleration approach is used in VAW Gesture recognition application, as used in the arm motion recognition for XWand and other applications for Nintendo Wii remote. The Dynamic time warping technique used is simple and efficient for examining the real time acceleration information. The technique utilizes less mathematical calculations that enable it to execute on the comparably low specification of the system.



Hand gestures are natural form of non-verbal communication. Everyday people interact with each other using different gestures without knowing it. This chapter describes different human arm gestures and how there gestures are classified into different groups. Furthermore this chapter describes, how different arm gestures are used in our project .

3.1 What is a gesture?

* Gesture can be defined as

i. 'The notion to embrace all kinds of instances where an individual engages in movements, whose communicative intent is paramount, manifests, and openly acknowledged'. [10]

ii. 'A motion of the body as a means of communication or expression'.[11]

3.2 Handwritten gesture techniques:

Handwritten gestures can be classified into two types [3]: uni-stroke and multi-stroke gestures. Uni-stroke gestures are those that are performed with a single stroke of an input from the device while holding a button ('ZERO' in figure), on the other hand multi-stroke gestures are those which can be performed with the multiple inputs from the device ('P' in the figure), the example of multiple strokes can be an input from the mouse or a pen that are recognized on a Tablet PC. In our project we have worked on unistroke gestures because they are simple to understand , easy to perform and easy to remember.

Figure 3.1: Difference between a Uni-stroke and Multi-Stroke gesture.

3.3 Classifications of Gestures

Gestures that are associated with speech are referred to as gesticulation, and gestures which function independently of speech are referred to as autonomous. Autonomous gestures can be organized into their own communicative language, such as American Sign Language (ASL)[], but can also represent commands with movement. Recapitulated, gestures are defined as a physical movement of the hand, arm, face, and body with the intention to convey information or meaning. Gesture recognition in this study means not only tracking arm movement, but also interpreting that movement as being a semantically meaningful command. Classifications of gestures are done by function, linguistic, and role in communication.

Cadoz (1994) [13] divided the class of functional gestures into three groups:

1. Semiotic (used to communicate meaningful information).

2. Ergotic (used to manipulate the physical world and create artifacts).

3. Epistemic (used to learn from the environment through tactile or haptic exploration).

3.4 Characteristics of Human Arm Gestures:

Communicating gestures by means of motion (arm or hand movement) has some benefits over static arm or hand postures. Firstly, arm gestures are natural form of interaction. Gestures help people to convey the exact meaning of their sentences. Secondly gesture is a powerful form to interact. There are also some negative aspects of arm gestures input, e.g. user can experience possible fatigues while performing a gesture as there is a lot muscle involvement in it. So arm commands therefore must be concise and fast to minimize efforts. Our project currently recognizes 3D unistroke gestures like numeric characters (0'9) and mathematical operators (+,-, / and *) as shown in the figure 3.2.

To perform a gesture user holds the wiiremote in one of his/her hand and presses the button and holds the button until he/she completes the gesture and then releases the button as shown in the figure 3.3.

Figure 3.3: keeping a button pressed while gesturing a number '2'




This chapter is intended for the readers who are interested in getting the knowledge of what actually the Wii remote is and how does it work. This chapter describes the basic structure of the Wii remote hardware, its accelerometer capabilities and communication methods.

4.1 Internal Structure of Wii Remote

This section explains the internal structure of Nintendo Wii remote controller and influence of gravity on Wiimote.

* Accelerometer capability

Wii remote has 3 built-in accelerometers that provide 3d motion sensing capability. Accelerometer can be defines as:

'An accelerometer is a device that measures the vibration, or acceleration of motion of a structure. The force caused by vibration or a change in motion (acceleration) causes the mass to "squeeze" the piezoelectric material which produces an electrical charge that is proportional to the force exerted upon it. Since the charge is proportional to the force, and the mass is a constant, then the charge is also proportional to the acceleration.' [13]

Accelerometer Provides the Wii remote's motion- sensing capability [9]. It has a +/?3G sensitivity range, 8 bits per axis, and a 100 Hz update rate.

Figure 4.1: Acceleration on the X, Y and Z plane while making a gesture of '3'

Where starting points are X=0, Y=0 and Z=1

* Batteries of the Wii remote

The Wii remote uses two AA batteries and has an operating time between 20 and 40 hours, depending on the number of active components. Approximately 8 bits of battery-level resolution are available.

* Wii Remote Buttons

There are 12 buttons on Wii remote. Four are arranged in a standard directional pad layout. One button is on the bottom providing a trigger-like affordance for the index finger. The remaining seven buttons are intended to be used by the thumb. The remote design is symmetric, allowing use in either the left or right hand. We have used buttons 'A' and 'B' for recording and performing a gesture.

Figure 4.2: Nintendo Wii Remote


4.2 Communication with the Wii Remote

To communicate with Computer over a wireless channel, Wii remote has a built-in HID that gets connected with the PC through the Blue tooth medium, although, the Wii remote is not fully compliant with the HID standards but it can connects to many Bluetooth capable computers .

4.3 Technical Information on the Wii remote system

The Wii remote device uses two basic sensors, i.e. the accelerometer and IR camera. The generic (absolute x,y and z coordinates) data is captured with the accelerometer with the help of Bluetooth medium, the accelerometer data does not require line?of?sight and provide robust motion and position sensing. The device includes three?axis motion sensors, IR camera, keypad, and Bluetooth HID antenna. The block diagram of Wii remote is described in figure 4.3.

Figure 4.3: Block Diagram of a Wii remote controller




The system specification of gesture recognition provides comprehensive features of all the software design, and is intended for the reader who belongs to software engineering field, the decisions for the project and comprehensive overview of designing the system is explained. Gesture recognition with the top level components and modular design are defined.

5.1 Software Development Life Cycle (SDLC)

The life cycle of the project follows water fall model. The water fall consists of Analysis, Requirement Specification, Design, Implementation, Testing and Integration, and Deploy phases. In Analysis phase we analyze the existing products related to Gesture Recognition; which includes the study of different recognition techniques and algorithm used for the system. After the analysis phase we gathered the requirements for the desired product. We also identify additional requirements that support our product. In the Design phase we develop the system design consists of UML diagrams which are discussed in system specifications. In the development phase we build user interface and recognition algorithms that are the main objectives of the project. Then we work on the required functionality of the product. After implementation of the product we design test case for the product to test the correctness of the product; that it could generate the desired results or not.

5.2 Development Environment

The development environment for gesture recognition application involves the description of necessary features that the project needed to design and implement in the system.

5.2 System Features

* Description and Priority

Highest priority of the system is to recognize the gestures correctly that needs to be stored in the queue.

* Stimulus/Response Sequences

User will perform gestures using a Wii remote. System will store the gestures performed by the user and will perform mathematical calculations and finally the system will display the results on the screen.

* Functional Requirements

FR1: VAW must contain data and gesture recognition analyzer.

FR2: Must have some built in object libraries for alphabets and geometric shapes recognition.

FR3: Motion sensing ability.

* Hardware

The primary hardware that is used in the development of the application is a computer system with 2.2 GHz processor, 2 GB main memory, and 320 GB hard disk. The gesture device aims to connect with Windows Os via Bluetooth HID. The Wii remote is used for motion sensing and gestures.

* Software

The input/output device needs software solutions to make them available for the VAW application. The software used in the development environment consists of the following.

' Microsoft Windows Operating System

' Visual Studio '08 .Net

* Wiimote Managed Libraries and Gesture Recognition.

5.3 Application Design

This section describes gesture recognition applications design for VAW system. The phases involved in the development and the basic interaction techniques are discussed.

1. Use Case Diagram

The users' holds the Wii remote in the hand, and uses the 'A and B' button on the device for recording and performing different gestures. Figure 5.1 illustrates the use case diagram of the system.

Figure 5.1: Use Case Diagram

A use case scenario represents the basic functionality of the interaction with the system, the Table 5.1 describes how to perform a gesture while holding a Wii remote device in the hand, the system gets the input signals from the device and stores the data into the XML file, the Table 5.2 describes the recognition process of the system, the system gets the data from the XML file and checks with the inertial information coming from the Wii remote device, if the match is successful the gesture is displayed on the screen.

Use case Name

2. Move the Wii Remote for making gestures

3. Release the button

5. Stores the gestures in the XML

6. Match with the pre-defined gestures.

7. Display on the screen

Table 5.1: Use case for performing a gesture

Use case Name

2. Get the inertial information from the Wii remote via Bluetooth.

4. Display the best matched gesture on the screen.

Table 5.2: The recognition process for gestures.

2. System Diagram

The system is created into two parts, first is for recognition mode and second is the interfacing. The figure 5.2 describes the system diagram for gesture recognition system. The software module captures the gestures using motion sensors (raw acceleration data) and after recognizing the motion, the system displays the recognized numeric or symbol on the screen.

Figure 5.2: System Diagram

3. Block Diagram

Block diagram has been divided in to two phases.

1. Data Analyzer

2. Gesture Recognition

1) Data Analyzer

This phase involves analyzing the data (in our case accelerometer signals) that we are getting from the Wii remote.

* Bluetooth HID Driver

First of all we have to make sure that Wii remote is connected with Blue Tooth. For this purpose Blue Tooth HID driver is used that supports HID devices For example mouse devices, keyboards and joysticks over a Blue Tooth wireless network.

* Detecting start/stop of gesture

User presses the button 'A' of the Wii remote while performing a gesture and after completion of a gesture user releases the same button.

* Low Pass Filter

The accelerometer values are passed through LPF to reduce the noise and produce a noise free signal.

Figure 5.3: Acceleration signal passed through LPF

(a) Signal with noise (b) signal after filtering

The figure 5.3 explains the basic schematic diagram, on passing through the LPF the signals gets smoothened. The figure 5.4 explains the function of the LPF, where S is the sampling rate with respect to 10ms; K is the present moment of time. Sk is the input signal and Sk?1 is the filtered output signal. The variable Z is the time factor. The Sk?1 always relies on its past and current signal values. The process is known as infinite input response filter. The output signal can be calculated by using the following equation.

'. Eq1

In the above equation, FC is the 'filtration constant' and its value is between 0 and 1. The low pass filter is called as pass band, if the value of FC is equal to 1; it allows all the frequency to pass through filter without any change in accelerometer signals. The Filter is stop band, if the value of FC is equal to 0; it stops all the frequency signals.

Figure 5.4: Low Pass Filter

By getting the real time analysis of the graph, the value of FC is set to 0.1. This value gives improved results for noise free acceleration signals.

* Store in a XML

For performing a gesture user is asked to press the button 'B' of the Wii remote. Similarly to record a gesture user is asked to press the button 'A' of the Wii remote. For the accelerometer values we have maintained an XML file. This file will store the name of Gesture and its x, y and z coordinates.

2) Gesture Recognition

This phase involves recognition of gestures. For this purpose we have used two different techniques i.e. 2D trajectories and Dynamic Time Warping. Both of these techniques are used in gesture recognition and these two techniques have been described in more detail in chapter #6 Gesture Recognition. The system match the real time results of a particular gesture with predefined gestures stored in XML file for that gesture. For example if user performs a gesture for '2' then the x, y and z coordinates of this gestures will be compared with the predefined gestures library (XML file) of 2. If the results are matched then the system will display the result on the output screen and will finally invoke a Virtual keyboard event for that gesture.

Figure 5.5: VAW Block Diagram

4. Activity Diagram

The gesture recognizer software module supports the physical interactive feature for VAW application. The software module utilizes accelerometer signals for pattern recognition. The activity diagram for gesture recognition system is described in figure 5.4. The device sense wrist activity by hardware derived accelerometer [20]. The user press the button on Wii remote for starting the gesture and releases it at the ending the gesture. The acceleration (x, y, and z) data is passed through the Low?Pass?Filter and is sent to the data?analyzer. The gesture recognizer gets the stored date from the data?analyzer to determine arm motion. On successful recognition of a gesture, the system displays the best scoring gesture on the screen.

Figure 5.6: Activity Diagram

5. Class Diagram

Figure 5.7: Class Diagram

6. Sequence Diagram




In this chapter techniques for gesture recognitions are described. The techniques that we have used in our project have been described in detail .We look at how acceleration data is converted to velocity and distance measures, described by Cartesian coordinates, and furthermore how the gesture data is visualized. We describe how features are calculated and used for recognition. Additionally, the two different types of classifiers are discussed.

6.1 Visualization of Acceleration

Data from three different accelerometers represent an arm gesture. If the Wii-Remote is stable and at rest, the sensors always measure the same values:

X = 0G, Y = 0G, and Z = 1G.

When the Wii-Remote is moved around, the 3 sensors measure acceleration and produce variable sine waveforms. Moving a hand upward generates a sine curve on the Z-axis, where the first peak is high (highest acceleration) and the second peak is low (highest deceleration). When the movement would be directed to the downward instead, the peaks would be opposite.

Figure 6.1: Straight line when Wii remote is stable and at rest.

6.2 Gesture Recognizing Techniques

This section focuses on the techniques and approaches for recognizing a gesture. In our project we have used two different techniques for recognition of a gesture which are as follows.

1. 2D Trajectories

Provided the fact that the Wii remote delivers acceleration data to the computer, a technique is developed to visualize the trajectory of the arm for some period of time. This technique is called 2D trajectory. The trajectory is considered as a 3 dimensional digital representation of the arm gesture. Many features can be calculated from it e.g. the number of horizontal lines, the number of curves, the period of time, etc, which can be used for comparison with the existing data set.

* Visualization of trajectory

It is rather straightforward to calculate the distance between two data points when the acceleration for a given time unit is provided. When the interval is constant (10 ms) and the acceleration is input from the Wii remote, the velocity between two points can be calculated by using equation 1:

'. Eq1

Where V is velocity, a acceleration (can be a negative value), t time (10 ms) and g gravitation (9.8 m/s2). Once the velocity is calculated for a particular data point, the distance to the origin of the axis is calculated by using equation 2:


Where d is distance. Now after plotting the values from equation (2) in to an Excel graph, we obtained the following Results.

Figure6.3: Gestures Distance Values Plotted using Microsoft Excel Sheet

2. Dynamic Time Warping (DTW)

The second technique for gesture recognition, Dynamic Time Warping [16], it is a Dynamic Programming technique to find the distance between two accelerometer signals [17]. Basically, it is an algorithm used for speech recognition, and can be used for measuring similarities between two acceleration and deceleration of making a gesture in air during the course of an observation, a data having linear representation can be examined through DTW. In general, it allows finding an optimal match between two given sequences for example time series. The given sequences are deformed or warped non-linearly in the time dimensions to determine the similarities independent of the variations in time sequence.

* Matching paths and Calculating Distance

A matching path technique is used to calculate the distance between two curves. Following are the steps involved in matching technique.

1. A matching path is a list of combinations of the points of the first curve and the points of the second curve that needs to be created. This creation let the distinguishing of different matching-path-using methods, of which DTW is one.

2. For all combinations of points i, j in the matching path, the distance D(i,j) between them is calculated. This DTW-implementation uses Euclidean distance. The resulting value is the distance between the curves.

The Euclidean distance [18] between two points P = (x1, y1, z1) and Q = (x2, y2, z2) is calculated as:

'.. Eq3

6.3 Orientation Filter

* Influence of Gravitation

The earth's gravity plays an important role for accelerometers; a definition:

'In physics, gravitation or gravity is the tendency of objects with mass to accelerate toward each other. Gravitation is one of the four fundamental interactions in nature, the other three being the electromagnetic force, the weak nuclear force, and the strong nuclear force. Gravitation is the weakest of these interactions, but acts over great distances and is always attractive. In classical mechanics, gravitation arises out of the force of gravity (which is often used as a synonym for gravitation). In general relativity, gravitation arises out of space time being curved by the presence of mass, and is not a force. In quantum gravity, the graviton is the postulated carrier of the gravitational force. The gravitational attraction of the Earth endows objects with weight and causes them to fall to the ground when dropped. Moreover, gravitation is the reason for the very existence of the earth, the sun and other celestial bodies; without it matter would not have coalesced into these bodies and life as we know it would not exist. Gravitation is also responsible for keeping the earth and the other planets in their orbits around the sun, the moon in its orbit around the earth, for the formation of tides, and for various other natural phenomena that we observe.' [14]

* Influence of gravity on X-Y-Z Coordinates

The x-axis measures the acceleration from left to right, the y-axis measures acceleration from front to back while the z-axis measures acceleration from bottom to top. As a result of the gravitational force of the earth, the z-axis always measures 1G acceleration upwards (if the Wii Remote is at rest and has a steady position). The x-axis and y-axis do not feel this force under these conditions.

Figure 6.4: Gravity affects on all 3 dimensions if the Wii Remote is not in a steady position [15]

Wii remote includes 3 motion sensors to get the directional motion in 3D space, for calculation Euler angle mapping is applied on the accelerometer values to find the coordinates of the orientation. It doesn't matter how the accelerometer device is kept, as long as the orientation remain the same. In contrast of orientation changing, the accelerometer sensors cannot identify the change in values by the arm gestures or gravity effect as it is influenced by Wii remote orientation. Solution can be defined as; when the user presses the button for performing a gesture the system stores the starting orientation of device.

Figure 6.5: Yaw, pitch and roll for the hand [19]

Similarly, when the button is released at the end of gesture, the ending orientation is stored. An average between the starting and ending point is calculated and subtracted with all the accelerometer data with respect to the given time interval. The orientation of a device is characterized by three angular values, yaw, pitch and roll which are commonly used to measure the direction of ships and airplanes. Hand orientation is also calculated in the same manner as of airplanes in respect to its zenith position.

Yaw is the direction (around Z-axis), roll the horizontal angle (around Y-axis), and pitch the vertical angle (around X-axis). We can only calculate X and Y axis to determine the orientation of the Wii remote. The Zenith position can be calculated as:

==> Rm is a rotational matrix.

For Rotation along X,Y and Z the matrices are:



The 'reverse location' of zenith is provided by a sensor that calculates the gravitation, so that the angular values of pitch and roll are able to be calculated. It is impossible to calculate yaw values with 3?axis accelerometer, as in perspective of downward force directly opposite to z?axis (1G), x axis and y axis motion sensors (x=0G and y=0G) are not influenced by any additional force.

The reverse (nadir) location is provided as:


So that :

Gravity is described as:

where unit is measured in 1G = 9.8m/s2

To calculate the polar coordinates (?, ?) from a rectangle (XYZ) we apply a 'standard' Rect2Polar formula. As a result we can calculate pitch and roll for any orientation of the Wii remote, and subtract the average orientation from all dependant accelerometer data points.



This chapter explains the interface of our project. How the interface is designed, what tools and techniques are used and finally snap shots are also included at the end of this chapter.

7.1 Importance of a good interface:

To design an interface that will attract the users is very important and there are several reasons for designing a good user interface:-

1. If the user interface is better, it is easier to train people to use it.

2. If the user interface is easier to use , it reduces the computational and accessing cost.

3. If the user interface is graphically easy to understand, the users are more satisfied to use the system.

The user interface of an application will often make or break it.' Although the functionality that an application provides to users is important, the way in which it provides that functionality is just as important.

So we have focused a lot while designing the interface of our project .The aim was to keep it simple and user friendly. We have designed our interface using .NET technologies. For the accelerometer events we have implemented a real time live graph for x, y and z axis. The variation in the graph ensures that the Wii remote is alive and generating accelerometer values.

7.2 Explanation:

First a user needs to connect a Wii remote using Bluetooth HID, then in application, a user performs a gesture while pressing 'A' button, and release at the end, the application recognizes the motion and match with the predefined libraries (XML), and prompts the best matched number.

Components of the recognition phase interface:

1. Connect button and Exit button for connecting/disconnecting the Wii remote from the application.

2. Graph showing live generation of the accelerometer values in X, Y and Z Axis when a gesture of 2 is made.

3. Loaded gesture in the predefined library.

4. Showing idle status and best match when Wii remote is at rest and recognizing status when gesture is performed.

5. Displaying best matched gesture on the box.

Figure 7.1: Recognition of Gesture' 2' in the Application

Component of the recording phase Interface:

In recording phase (figure 7.2) a user presses 'B' button on the Wii remote to record the gesture and save it to the xml file, those xml files are loaded every time in the left panel of the interface when a new gesture is recorded and for viewing.

Figure 7.2: Recording of a gesture.

Components of the calculator Interface:

1. A user is given with the facility to enter 3 digits (0-999) in the Values text box.

2. Calculated answer after getting input and making a gesture of a equal operator '=' as 'e'.

3. Operator symbol displayed after getting a gesture of a symbol.

4. Clear button for clearing the values and Equal operator for getting answer into Answer panel.

Figure 7.3: calculator interface.




An overview of the experimental environment is provided in this chapter. It describes the setting where participants are currently in, how they are informed about the tasks, and what is expected from them during testing. Furthermore, this chapter portrays the pre and post questionnaires participants were requested to fill out. A required trainings phase precedes the main testing. This instructional phase is designed to familiarize the user with the futuristic, but mysterious concept of gestural interaction. The main experiment itself is profoundly discussed next, together with the experiment variables and their hypotheses. The results are to be read in the next chapter.

8.1 Procedure

Participants are unfamiliar with the concept of gestural communication. As a result, users are needed to be trained how to use arm movements for proper communication; both to adapt to the concept as well as to be able to handle the Wii Remote appropriately. Before any interaction takes place, participants are asked to fill out a questionnaire about their overall experience with the common Wii remote, and their opinions, beliefs, and expectations concerning gesture based interaction. Subsequently, a textual and oral explanation gives the aspects on the purpose of the user tests, along with a description of the exact tasks that need to be accomplished. The user manual is provided in Appendix B. After the testing is finished, a second questionnaire is presented with additional inquiries relating to the types of gestures used during the testing. The questionnaires can be found in Appendix C.

8.2 Participants

During the experiment we selects 15 participants (with ages ranging from 18 to 25 years old) to perform gestures with the available interface; each of them are from different educational background and varying ages.

8.3 Experiment

Prior to the user tests, a short questionnaire needed to be filled out by all participants, exploring their beliefs and opinions with respect to this communication method. Immediately after that, participants were informed what to expect during tests. This is done by instructions on paper, with a verbal verification afterwards. The training phase, being the next step in procedure, introduced the Wii Remote Controller and the gestures in practice: participants learned how to make (complex) commands gradually. Once this phase was finished, the main test cases are started.

* Environment

Users' are given with the good environment, seating plan and comfortable postures to make gestures in an open room with a projector placed on the side wall that displays the main interactive application, a laptop was attached to the system under the supervision of one of the experts that controls the follow of information.

* Trainings phase

Participants are confronted with the interface for the first time, unaware of the available commands and their corresponding gestures. For this reason an introduction to the environment seems necessary. Participants were instructed to complete a trainings phase that would simplify the available gestures to them. A description and images are shown for each numeric gesture, explaining the exact goal, meaning and trajectory. Participants were instructed to memorize all gestures. It can be seen in Figure 8.1.

8.4 Main testing

The gestures that were taught to the participants during training phase, needed to be applied throughout the main user tests: participants were requested to complete all 0 to 9 numbers as accurate as possible. Hitting the 'A' button on the Wii Remote let them to record the gestures and memorize them for future presentation (see figure 7.2 in chapter 7). After completing the (first) assignment, the same instruction window automatically appears again, initiating the next task. When a task cannot be completed (the participant cannot remember the appropriate gesture, or the participants is unable to perform the gesture properly), two consequences are concerned. One, the corresponding command needs to be retrained instantaneously, and two; the recording tasks start all over again. The second consequence seems harsh, but is expected to stimulate participants to really remember the gestures. This trial setup furthermore has the productive consequence that performance can be compared between singular participants for each solitary task, while in the meantime it is getting clear which gestures are tricky to remember and complicated to perform (since difficult gestures need retraining), per task. Consequently, retraining a certain command stimulates analysis of its corresponding gesture, which means that retraining enhances gesture examination, while on the other hand it seems that, without any retraining, the usability of the system is rather satisfactory (because the user finishes the tasks without a flaw). For each assignment the elapsed time and the number of actions are considered. The program additionally records and stores all acceleration data, enabling the pre-load of all gestures. A trainer observes the participants carefully during testing, reporting considerable performance. When all the gestures are completed satisfactory, the test is finished.

8.5 Hypothesis

This section describes the expectations concerning the variables used in this study. Several variables are manipulated and their influence is to be determined by above described testing. The following hypotheses are proposed.

* Instructions about the Wii Remote

To get the complete information of handling the Wii Remote, the participants were instructed how to hold the Wii remote while gesturing. For this reason, participants can perform better while testing the application. (The user manual is given in Appendix B).

* Different gesture types

Numeric gestures are easy to remember, but since different people have varying ways to write them, trouble is expected for numeric gestures '0', '4' and '9'. See Figure 8.1 for their trajectories. Although, every participants need to learn all the gesture that they record on the first phase.

Figure 8.1: Trajectories for Numbers.



The Gesture Recognition system was experienced by participants while observing and interviewing them. To further evaluate the performances of gesture recognition techniques, a qualitative user test on the participants was performed. The design of experiments was to measure users' performances and find out their feelings towards handling and facilitation techniques. The important features of the system such as the speed, accuracy, and the recognition rate with user satisfaction were analyzed. After performing all the tasks, the participants had to fill out the questionnaire, which helped to understand the users better. This chapter describes an overview of the experiment results. The questionnaires are evaluated, and experimental variables are analyzed. After collecting the data from the questionnaires, a simple analysis of the data was created. The average mean values were calculated for each variable. The questionnaires presented to participants are provided in Appendix C.

9.1 Performance

The performance of gesture recognition application is analyzed by experimental variables. Figure 9.1 shows average recognition rate for each numeric gestures and figure 9.2 represents recognition rate of each symbols used for the calculator.

Figure 9.1: Analysis graph for each numeric gestures.

Figure 9.2: Analysis graph for each mathematics operator.

the average recognition rate for system defined gestures was above 80% because, the users are not fully trained to make the gesture on the basis of predefined training set, in comparison the recognition rate gets 90% and above on average for user dependent training sets i.e. each user records their own gestures before testing on the recognition phase, problem arises where user hold device and the orientation of the Wii remote changes.

According to the figure 9.1, the best numeric gestures that were recognized 100% were '2,4,5,6,8,9' while the other lacks the orientation or either they get mixed with other gestures while performing them, for example '0' and '6', '1' and '7', '3' and '8' therefore their recognition rate falls a little while taking tests. In the same manner the best recognition rate for mathematics operators '+,-,*, /' was 82% for predefined gestures and above 90% for user dependent. The problem arises for the gesture of minus operator, where we chose 'S' as a representation for the gesture, and it get mixed up with the numeric gesture '5' while performing predefined training sets.

9.2 User satisfaction

The participants used the application and give feedback based on their user experience. The average mean of the user satisfaction for the system was determined. The average mean was calculated, for items with a scale (from 1 to 7), where 1 was 'Strongly Disagree' and 7 was 'Strongly Agree'. The user satisfaction for the application was satisfactory and lies in the range of 5:6. Observing user interaction with the system gave an understanding of pros and cons for the features of gesture recognition application.

Figure 9.3: analysis graph for user satisfaction

9.3 Discussion:

By observing user interaction with the system gave an understanding of using gesture recognition application. Overall analyses of the techniques are better regarding performance, completion time, and user fulfillment. The participants used gesture recognition system and gave feedback based on their experience. All the gestures performed were with the recognition rate on average of 82% with default based gestures and above 90% with custom based gesture training sets. The overall user satisfaction for using application was 5.7/7. From the comments of the users, we learned that 'ease of access' was their primary concern to interact with the application.




This study has focused on the aspects of gesture based Human Computer Interaction (HCI) where user perform gestures without restraints in the air. For further improvements we will be focusing on shape recognition, character recognition. Moreover we will try to develop a Virtual Air Calculator which will help the user to perform mathematical calculations in the air using a Wii Remote.

10.1 Techniques Used

Two different classifiers, DTW and 2D Trajectories, are merged into one. Both of these techniques have been discussed in Chapter 6 Gesture Recognition. Initial phase of our project was to recognize numeric gestures from 0-9 for which we are succeeded. Now for further improvements we will be using the same classifiers to recognize different shapes and characters from A to Z or a to z.

10.2 Arm gestures

Different Arm gestures in our study have been divided into three groups. Ergotic gestures, 'acting' semiotic gesture, and 'symbolic' semiotic gesture, where ergotic mean manipulative and semiotic means communicative. Symbolic semiotic gestures (e.g. Trajectories) appear to be most appropriate for direct interaction; most of them are easy to remember and also easy to perform. Furthermore, their overall recognition rate is the highest of all. Numerical gestures are fairly well recognized, and extremely well remembered. The best recognized gesture is the number '2' with a recognition rate of 100%, Number '5' and number '8' are recognized properly 80% to 95% of the times being second best recognized. Gesture '6' appeared to be more difficult to remember than '9'.

10.3 User Help

For the user help we will be providing a training manual for the users who are not familiar with Wii Remote usage. This training manual will help them in the following areas

-->How to connect Wii Remote with the system?

-->How to record and perform gestures?

10.4 Future work

There are number of things that can be created using the same techniques used for creating this application, for the future work we will be focusing on the following steps.

1. Intelligent application which can recognize the gestures without the press of a button. For example a user only needs to hold a device in one of the hands and perform a gesture without the button pressing.

2. We will be focusing on the multiple recordings of gestures as a multi-stroke input.

3. Handling the home device such as TV, DVD and other electronic devices with the accelerometer motion sensor.