Usability Testing Of Mobile Applications Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

With the rapid improvement of mobile technology and its application, efficient usability testing becomes progressively more significant for the design, development and deployment of successful applications. Despite the great potential of mobile devices and applications, there are some limitations due to the features of limited bandwidth, unreliability of wireless networks and changing context. The traditional methodologies of testing desktop applications are not appropriate for mobile applications [1]. A suitable testing method can be created by proposing a framework which includes an appropriate selection of research methodologies, usability attributes, mobile tools and data collection methods. The usability testing of the consumer application can be done in two environments: field and laboratory. The overall performance of both the methods is in the same level but it differs in the type of mobile and the application to be tested. The testing of applications in current mobile trends like android, RIM blackberry, windows and iOS will help in the evolution of new technologies.

Key words: Effectiveness, efficiency and environment.


For the past few years, the usage of mobile phones, Personal Digital assistants, palms and pocket PCs has had a drastic change and has become a necessity of people's day-today life. Among these, mobile phones play a vital role in communication and have stepped up to help people in accessing the internet in their hand held devices. With this rapid improvement, many companies started deploying innovative mobile applications like daily news alert services, classified mobile advertising, restaurant and entertainment listings, wireless Web portals, and mobile commerce (m-commerce) applications [1]. This development attracted extensive research areas that focus on how to conduct an appropriate and efficient usability testing for mobile applications.

Usability testing is "examining the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and contentment in a specified context of use" [2].

The testing methods used for the desktop application is not applicable for mobile application [1] due to distinct features like mobility, wireless network and different operating system.. So, usability testing method should be more specific to mobile application.

The main objective of this article is to discuss a generic frame work and different methodologies in usability testing of mobile application. It also deals with the major issues in usability studies and how to conduct the testing on the basis of nature of the application and mobile device.


Several interesting studies were performed [1], [2], [3] in usability testing of mobile application and attempted to find solution for the following questions.

Can the presentation methods help users in browsing or perceiving information of their own interest or work easily on mobile application?

The researchers [1], [4] focused on exploring and testing of different style to effective content and presentation on the small screen of mobile devices. They also concentrate on usage of font size, color and style that help makes the users more comfortable in the web pages in mobile phones.

What are the relevant design of menus, link structure and cues that assist the user in navigation?

This category of research [5] focussed on evaluating the design of menus and links that helps the user to reach the destination page and several guidelines have been suggested. Firstly, the menus should be clear, easily interpretable, predictable and consistent. Secondly, the designers should avoid displaying long list of menus in small screen so that users' cognition can be reduced. Finally, the menus should help the users to finish the task with minimum interaction.

Can users perform specific activities like query search, filling form and making notes in the application?

Researchers [4], [6], [7] aimed to check how the users like to perform variety of activities in mobile device. The users feel unlikely, in using the same degree of comfort in mobile application as they do on desktops, due to constraints of mobile devices.

What kind of data entry method helps the users to enter the data easily and correctly?

This line of research [8], [9] has investigated how the application should from different modes like external keyboard, stylus and soft keyboard. Even though the mode of input is different, the application should perform its operation consistently and efficiently.

How well the application works in different context, mobility and slow network connections?

The researchers [1], [10] investigated the usability of application in different mobile context and in different mobile phone signal strength.

In a controlled laboratory experiment, several contextual issues were ignored but usability testing in real time experiment helped in ensuring that a mobile application can work properly and help the users in achieving their goal in real-time environment [1].


The main motivation behind this work is lack of guidance for testing a mobile application in the current trend. On the other hand, there is a debate between laboratory and field method in usability testing. The testing should not be performed in a same procedure for all kind of applications. The type of testing method should depend on mobile device or operating system such as RIM, android and OS. It should also depend on application type such as banking, geo application, gaming and m-commerce.

The usability testing of mobile application has been a hot topic in recent researches because in current technology, the deployment of different mobile applications are increasing very high rate. Generally, the present testing method depends only on 'where' and with 'whom' the testing going to be performed but it should be extended by considering the nature of application and mobile device.


There are several disputes that need to be tackled during the usability process. The following are some of the common challenges [1] that the user faces in using the mobile application.

Mobile context: It can be defined as "any information that represents the situation related to interaction between user, application and environment" [11]. It is very difficult to represent all possibilities of mobile context such as location identity of nearby people and environmental objects that may distract users' attention

Connection to network: The slow and unreliable wireless network connection is major obstacle for using mobile applications [12]. It affects the data transfer speed, downloading time, quality of video and audio streams. So, connectivity is the vital thing to be considered in testing a mobile application.

Screen size and display resolutions: The major constraint in mobile device is smaller screen size which reduces the usability of the application. On connecting to the World Wide Web, it possesses some navigation problem, illegible and difficult for the users to perceive information [13].

Even though the mobile features are developed a lot in current trend, it possesses very less screen resolution on comparing with desktop [7]. On testing, it should be checked that whether the application is adaptable to kind of mobile phones.

Processing capability and power: On comparing with desktop, mobile phones have very low memory and processing power which reduces the working efficiency of application [1]. In latest technology, companies have started deploying mobile phones with higher memory. The designers should create the application which can work in all kind of mobile models. So, testing of compatibility of application to a specific mobile phone model is necessary.

Input entry methodology: Providing a input to o small device like mobile phone is difficult and ir requires some expertise to give an accurate input [12]. The application should accept the input modes like multimodal, voice and touch which help the novice to use it easily.

On creating a methodology for usability testing of mobile application, the above challenges should be considered and must overcome.


Usability testing is an evaluation of degree of effectiveness and efficiency of a product that it can provide to the user [14]. The main reason behind the testing of a mobile application is to measure its working ability and detect the problems in it and to find an appropriate solution for the detected problems. The attributes and variables are necessary to measure the characteristics of the product that encompass all qualities expecting by the users.

The three major variables used are time, events and navigation. The variable time usually denotes the time period taken by user to perform an action or time take by application to respond to user (e.g. process time). The event describes the user interaction with the application that represents button click, moving back and forth between the pages, zoom a picture or dragging an object.

The following table represents the essential attributes and variables [14] to be used in usability testing of mobile applications.





Time taken by the user to accomplish the task at the first attempt [15]; Time for the novice users to perform the task; Number of trials ; Amount of training required to use the application.



Task completion time, Time taken by application to receive and process the input from the user; Time spent by the user in searching for icons or cues ; How far the application navigate the user to attain their goal.



Time, number of clicks, pages and steps to finish the task after not using the application for a long period of time like 3-4 weeks.



Number of errors in performing a task like diversion from the right path, entering the wrong input by the user and misinterpret the navigation shown by the application.



Comparing the user performance with a predefined level in terms of speed (e.g. user completes the job in 3 mins with only 4 click in the application); errors and number of steps.



Effort to attain the solution; Number clicks, pages and menus that user should go through, to reach the required result or page; Time to search for button, options or cues to perform a specific function.



Reading speed, accuracy of answers in a predefined test. How correctly the user can perceive the things like picture, tables and key things

Table 1. Attributes and measuring variables for usability testing of mobile application.


In this section, the general framework for testing procedure to facilitate the researchers who performs the usability studies for mobile applications [1] is discussed. This framework also includes some issues and suggestions that evaluator should consider during the testing process. The framework embraces the selection of research methodology, tools factors to be measured and data collection technologies.


There are two types of methodologies for performing usability testing in mobile application: Laboratory and field testing. In laboratory method, the evaluator analyses the usage of product in a testing room and collect the data about the performance. Conversely, in the field testing, the examiner conducts the research in the real time environment and gathers the information about functioning of the product. Both methods have pros and cons on testing different kind of products. This paper deals with selecting appropriate method for mobile applications.

6.1.1 Laboratory method:

The main advantage in laboratory method is that the evaluator holds the control over the experiment [1]. The examiner knows the objective of the testing and they explain the participants about complete procedure. Secondly, the examiner can collect all the data about the users' performance. Thirdly, video recording is also possible which is helpful for future analysis. For example: if the research is to check the efficiency of the mobile application, then the examiner can record and analyse how the application interact the user easily and correctly.

The major disadvantage in the laboratory method of testing is, the user will not get the real time environment. So, the users' behaviour may be different from their usual interaction with mobile application. It also ignores the mobile context and reliability of wireless connection during the testing process.

6.1.2 Field method:

The main advantages of field method over laboratory method are consideration of dynamic mobile context and unreliable wireless network during the testing of mobile application [1]. Secondly, the users are examined at the real time environment which leads the evaluator to collect some realistic data.

On the other hand, the field method possesses three major disadvantages: Firstly, it is difficult for the examiner to apply an evaluation technique. Secondly, most of the field testing takes place at the dynamic environment which leads the evaluator to face difficulties in data collection and condition control. Thirdly, it is not easy all the time to establish the realistic environment which captures the richness of mobile context.

To solve the above the problems, the evaluators need to have more concentration on data collection method and defining a mobile context.

6.1.3 Comparative study between Laboratory and Field method:

To show the influence of environmental effects on the usability testing, we'll discuss the empirical study of comparing both laboratory and field method [15] in this section. In this experimental study, the evaluation of mobile application called "Mobile wire" is being tested. Mobile wire is an application which performs the file transfer operation between computer and mobile phone. As this application is not familiar to all the uses, it shows all the problems facing by the initial users.

6.1.3 (A) Test scenarios:

On laboratory and field testing, 20 subjects of each whose age group of 22-35 were participated. On laboratory, the environment setting was quiet similar to real time environment because closer the setting to actual environment, smaller the variation between the result in laboratory and field method. The field testing was conducted a rush hour with the subjects in the office in Ruoholahti. In each method, 4 evaluators were joined who were having 5-13 years experience in usability testing. On both the conditions, the users were given nokia 60 series mobile with Symbian platform. During the test process, the subjects were asked wear special equipment which contains three cameras for video recording. The cameras were focused on different places: user's face, surroundings, mobile screen and keyboard. The figure .1 shows the usage of cameras.

In this test session, two successful usability testing methods were used: think aloud protocol and performing a predefined task. In think aloud method, the users were asked to explain what they expected and what happened in the experiment session.

6.1.3 (B) Test method:

On Laboratory and field environment, the subjects were asked to perform the following 10 tasks [15].

Getting familiar with features and usage of mobile.

Receive a SMS from the evaluator which contains the WAP page.

Download the mobile wire application to the mobile.

Opening the downloaded application.

Using the application, receive an image from a friend and view it.

Download the image to the specified folder.

Capture an image with mobile phone camera and save the image in a computer.

Giving rights to a friend to view the saved image.

Close the application.

Delete the application.

After finishing all these process, the subjects were allowed to have questionnaire session with the evaluator. The users were asked to explain their experience with the usage of the application and problems faced during the process. Figure .2 shows the questionnaire session for the users after finishing the tasks [18].

Figure 1: Usability testing in Laboratory method Figure 2: Usability testing in Field method

6.1.3 (c) Test results:

The empirical study between the laboratory and field study brought some unexpected results. The evaluator identified identical set of 46 distinct problems in both the methods [15]. There is no significant difference in number of problems detected and they detected same kind of problems. Even though there are some difficulties in data collection method of field method, it does not affect the evaluators in analysing users' behaviour. As the surrounding in the laboratory maintained like an actual environment, the user does not have any distinct feel of working in laboratory.

But the major thing to consider is time consumption, which is very high in field testing. Field testing method possesses more interruptions than the laboratory method which leads greater task execution time. Graph 1. shows the execution time for each task in both the methods [19]. The evaluators suggested found that field testing is suitable only when the mobile application with real context is to be tested. From this study, it is concluded that laboratory method is the most appropriate method for testing mobile application, if the mobile context is does not play a major part in that application .

Graph 1: Comparing the execution tome for Usability testing in Laboratory and Field method


There are two different kinds of tools which can be used in testing of mobile application: Real mobile devices loaded with the required applications and simulation software in desktop computer. Both the methods have merits and demerits on their usage [12].

On performing testing in simulation software, the emulator stores some of the data like number of clicks done by the user, mouse pointer motion which reveals the user's knowledge and ability to use the mobile phone. The captured data can be used for analysing the performance and errors in the design which impede the user to reach their goal. But on the other hand, it could not perceive users' real time experience with the mobile phones. For example: for touch screen mobile, we could not check how the application receives the input from the user by virtual keyboard or stylus.

On testing with mobile devices, it allows the examiners to get the realistic values due to the direct interaction of user or examiner with the device in a real time environment. It possesses some difficulties in gathering the data about users' behaviour in accessing mobile application. To overcome these problems, it is possible to attach a small camera to record the activities going on in the mobile for the future experimental purpose [16]. These allow the investigator to note down some of the important data or users' behaviour on using the mobile application.

On comparing to emulator, the mobile device possesses more influential effect on usability testing of mobile application. The emulator can be use for device related issues like user interface design.It is strongly recommended to use mobile device rather than simulation software in desktop, for finding usability problems involving in mobile context

The figure .3 illustrates the testing of mobile application with simulation software. The figure .4 shows the video recording for testing the iPhone mobile application in the real time environment.

Figure 3- Usability testing of iPhone application with emulator Figure 4-Usability testing of iphone application in real time environment


The selection of appropriate usability testing methodology for mobile applications relies mainly on specific research questions and objective. Several researches [1], [2] were performed and they came to two different conclusions.

Firstly, in case of examining the applications in a stand-alone application without considering the network connectivity, then laboratory method is more suitable. In this method, evaluation of components like interface layout, information presentation schemes, design of menu and link structures, and data entry methods could be done efficiently. But testing of network connectivity, mobility and other contextual factors could not bring an accurate result.

Secondly, the field testing method is more appropriate for testing the application performance related issues which relies mainly on mobile context. But the main disadvantage in this method is that the data collection is difficult and time consuming.

Figure .5 shows the complete framework of selecting a methodology for usability of mobile application [1].

Figure from: Zhang, Dongsong and Adipat, Boonlit(2005) 'Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications', International Journal of Human-Computer Interaction, 18: 3, p. 293 - 308

Figure 5-Framework for Usability testing of mobile application Picture from:

Initially, the evaluator should have identification research question and the purpose of the testing which plays a key role in selecting the methodology.

If the testing process completely relies on real context, then the evaluator should go for the field method or else laboratory method is highly sufficient. In case of laboratory testing, the examiners have to choose the type of tool to be used in testing the application. As we discussed earlier (in section), there are two types of tools: emulator and actual mobile device. But this kind of categorization is not necessary for testing the application in the fields. The attributes selection is the next stage in usability testing process and it is common to both the laboratory and field method. But the attributes used in testing is not same all the time, it may vary depending on the type of application to be tested.


The main purpose of data collection is to keep record, make decision and for future references. In case of data collection method, laboratory testing is more convenient and efficient than the field testing. There are two different methods is data collection: traditional approach and new data collection method [1].

In the traditional method of data collection in usability testing of mobile application, observation, interview, survey questionnaire, and verbal protocol, have been employed in usability testing of mobile applications are the major techniques employed. In observation method, video cameras, microphones and several examiners can be used to record the users' interaction with the application. The two techniques in this method are ethnography and user observation. Both the method provides the significant information about the interaction problems, users' reaction and context where the activities takes place. In the questionnaire and interview method, the users must be asked to answer a series of questions about the context, their experience on using the mobile application and their suggestion about the application. In this type, two different techniques [3] can be used: probing and Experience Sampling Method (ESM). It is better to entangle both the techniques and propel innovations by presenting a provoking questionnaire session.

If direct observation is not possible at the time of testing, then the following techniques [3] should be followed:

The ESM or contextual questionnaire session for users to give back the information about pros and cons on using the mobile application, problems at the time of testing and efficient features of the application.

Probing techniques, provoking the users' imagination by asking them to provide opinion and suggestions about the application.

Data gathering can be extended by several contexts, monitoring transition and users' reaction to the changes.

Initially, the evaluators faced many problems in data collection process in field testing. So, they put forth some new technologies voice mail diaries and pocket and web diaries [1]. In the voice mail method, the users have to call the specified voice mail line report some problems and suggestions about the usage of mobile application. With that collected information, the evaluator can overcome the problems and update the application. In the pocket and web diaries method, the users have to write about the mobile application, then connect to predetermined web server and upload their notes in it. In addition to these methods, other techniques are regular meetings, e-mail reports, daily online questionnaires, and audio or video recorders can also be employed for collecting data.


This study [17] explains the perfect example for using the framework for usability testing of mobile application (as we discussed section)

7.1 Objective and hypotheses:

The goal of the study was to test the usability of mobile map application and how the frame work for usability testing works in the real time.

The hypotheses we need to study in this experiment [17]:

What dataset does the user prefer to use in map application and under what condition?

Does the user able to complete the task with the given material?

What are the disadvantages of the application?

Does the user able to perceive the cues and symbols used?

What are the features needed to complete the task efficiently and quickly?

7.2 Selection of testing methodology:

The usability testing of map application mainly concerned about the real context. It is better to choose field method rather than laboratory testing because it is difficult for the evaluator to check the efficiency of the map application inside the laboratory. To test such application, the users must be given a real time environment and it is difficult to bring such settings in the laboratory. The main aim of this experiment is to bring some qualitative results rather than a quantitative one. In case of field testing, it is easy for both user and evaluator to check the efficiency of the application and bring out the accurate qualitative results about it.

7.3 Test scenarios:

To test the map application in the tougher situation, the experiment was performed in the Nuuksio national park, is an area noted for hiking and camping possibilities. Totally, six users from different professions were involved to perform the usability testing. The table .2 shows the complete information of the subjects who participated in the usability testing [17].

The task given to the subjects was to reach to reach the nearest camp from the current position in the Nuuskio national park. The users have to note down their positions periodically on the way to their destination. The users can choose any path to reach the camping site but they have to give reason why they choose this path and they have to mention what are the cues and options used in the application.

Table from: Nivala. A, Sarjakoski. L.T, Jakobsson, A and Kaasinen. E, 'Usability evaluation of topographic maps in Mobile devices', 21st International Cartographic Conference, 2003.

Table 2- Details about the participants of usability testing map application

7.3 Test method:

At the study area, the users were clearly explained about the aim of the experiment that it is to test the usability of application and not the users. In this field study, three different methods were used for the data collection: think aloud, observation and interviews. Two evaluators with very good experience in the field of usability testing were participated for observation, interview and evaluation process.

7.4 Test tool:

In case of field testing, the real mobile devices would bring the better results and it does not need any emulator to perform experiment. The users were given different types of mobile phone like iPhone, Blackberry and Nokia in which the application can be installed. The reason behind the usage of different mobile phones was to check ability and efficiency of the application in different operating system like iOS, RIM and symbian. The users were advised to use the topographical maps for easier exploration of the study area.

The figure .6 shows the maps generated by the application in the mobile phone [17].

Figure 6-Maps generated by the mobile phone application

After reaching the destination, the users were asked to questionnaire session in which they have to answer set of predefined questions. The following questions were used in the question session [17]:

What do you think about the application? Do you think it is useful? Did you trust the application?

Did you like to have the route suggested by the application?

Were there enough or too many suggestions or target given by the application?

Does the mobile phone helps in accessing the application?

What are the features to be added and to be reduced?

What topographical information do you think is relevant?

How was the quality of visualization on the application?

What about the colors used in the maps? Did they help in differentiating the regions?

What other improvements can be made to the application?

After finishing all the testing process, the evaluators asked the subjects to send their suggestions about the map application and the test procedure in the voice mail method.

7.5 Results:

The results of the field testing were categorized into three parts: map-related results, technical issues about the software and finally users' experience in participating the field testing [17].

In map related issues, the users had a difficult experience in differentiating the symbols used in the map application. The colours used in representing the symbols are in same range and so it is difficult for them to perceive some information. For example: both contour lines and path marking were indicated in the brown colour and both hiking trails and roads were drawn in red colour. The fact is that the experiment was conducted in outdoor in bright sunlight which made the users to differentiate the colours. The figure .9 shows the results about the symbols used in the map application [17].

Table 3. Map symbols that the users commented on or had problems with during the Nuuksio tests.

Table from: Nivala. A, Sarjakoski. L.T, Jakobsson, A and Kaasinen. E, 'Usability evaluation of topographic maps in Mobile devices', 21st International Cartographic Conference, 2003.

The main advantage over the map that users felt was zooming option between different map scales which shows a clear path for the users to the destination. Stepping between the layers of map is smooth and properly implemented which gives the proper guidance to the users.

Many problems arose when the application is connected to network, to retrieve the data from the database. Among these the severe one is the slowdown of application speed when the users were trying to perform multiple operations continuously without any break.

Another problem is that the users were not feeling comfortable with using keyboards for giving input to map application. Rather, users were sophisticated in using touch screen mobiles which is helpful in zooming and dragging operation.

The users participated with more interest in field testing by using map application to reach their destination place and gave valuable feedback on usage of application. The evaluators had full control over the experiment by tracking subjects' position on the study area and recording feedback about the testing. The users showed 100% involvement in giving opinion about the testing process and map application, using the latest method voice mail process.


In the future, the researches should be done to overcome the difficulty of data collection process in field method. The evaluators should use some sensors, small cameras, microphones and large number of observers to collect the accurate data from the subjects. The evaluator should search some peaceful environment for field testing, to have a better data collection and very less time consumption.

In further testing, the evaluators should have different testing procedures for different categories of mobile application like gaming, m-commerce, education, etc. The testing environment for gaming and educational application should not be same environment because users' mindset relies mainly on it which brings major difference in the result. On testing the above two applications, it is not necessary to check the security factor but it is the key feature to be tested in the m-commerce application.

In the laboratory study, the evaluator should have a casual communication with the subjects and must try to bring the real time environment. It leads the examiner to collect some real time data about the users' performance.


With the rapid improvement of technology in mobile devices and its application, it allows the users to access the communication media such as text, audio, video, graphics and audio (available on internet) from their hand held mobile device. Simultaneously, the problems on the mobile applications are also increasing at a very high rate. So, the proper testing methodology should be implemented for mobile application. On testing mobile application, the selection of examining environment must depend on the type of application and aim of the testing. Both the laboratory and field testing have pros and cons. The evaluators should choose one of those testing method, depending on the features of mobile application. The further researchers could find some new attributes for measuring mobile application but they must inherit some previously used mandatory features like discipline, interactivity, quality, usefulness and aesthetics [1]. The implementation of such testing process should help the designers of mobile application, for their future work or updating their current work.