Model-based software testing

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

If we look to today's technology world and increasing number of hand held portables devices. The biggest and easy to find example could be the smart phones, e.g. Nokia devices, black berry or iphone. All these smart phones are equipped with smart interactive user interface (UI), and all these devices are becoming more and more complex devices containing tens of different applications almost for any thing. There are also numerous online stores available who sell these APS. However, not only APS are getting complicated but also hardware add-ons , although these have little to do with the actual telephone part of the phone, such as cameras and GPS-devices.

This increase in the hardware and other complexity of smart phones also leads to increased number of errors. In order to get applications which are error free, all the APS and operating systems need to be  thoroughly tested before deployment or use, and the best way to achieve this level of accuracy the testing process need some sort of automation. But to design automated test cases for today's smart phones is not a simple matter. As old practice there exist three different views on how an application should work or designed:

First a UI designer writes a broad specification of how the user interface should look, these specification are like architectural drawings of the buildings where each and every functions of different keys in different states of the application are explained and it is also clarified that how the application responds to user actions. Second, a developer reads the specification of the designer or UI architect and based on this writes the application according. Third and last is the job of the tester, where the tester takes the application and the respective specification and tries to find out how the program should work and does it work that way. In simple this could be considered as the process of validation.

These three views do simplify the tasks but also do lead into troubles, as they are seldom consistent. All the applications get mature as the time passes as with time many new features are added. In order to add a new feature to application or updating an existing a feature means that all three views of the application must be updated accordingly. It is evident that most time in testing is spending on adjusting the test cases into always changing applications.

The process of software engineering cycle is best described with the help of the popular V-model [1] (Figure 1). The V-model gives the picture of the software engineering process. If we look to the left side of the model which represent specification and design activities, going through  the coding stage, and ultimately ending up in the right tails testing activities. The model shows that the design and testing are connected through the planning phase, but in general they are clearly two separate entities. The planning phase normally consists of testers studying the different sections of the software system and then ultimately writing and executing individual test scenarios that could be applied [2].

This approach has at least two of the major challenges [2]: First is that since, all traditional tests suffer from the “pesticide paradox” [3]. In paradox approach the bug fixing leads to less and less efficient test cases where the bugs they reveal have already been fixed. Second issues is since with time software tends to change over time as new features are added, the tests cases must be modified to support those changes. But modification of the tests cases takes much time. It said that almost half of the development time is spend on testing. Since, each project has time as will as financial implications therefore maintenance costs can raise high if modifications are required often. One reason behind this as with time initial designed have changed and may be the relevant documentation which exists are for the initial version, where as the initial version have evolved and not always the design documents are updated accordingly.

One approach to those problems is to take a model of the software that is currently into use. If we use model based approach there is no need for separate test planning or even coding. The model could be used to generate all the test cases along with respective code.

Model based testing [16, 17, and 18] is a new and inventive approach in software testing because it completely automates the validation testing process. There exists testing tools that automatically generate test cases from a model. The output or the generated tests are executable and include which an oracle component which assigns a pass/fail verdict to each test.

Model-based testing makes sure that repeatable and scientific basis for product testing, gives sufficient coverage of all the behaviours of the product and allows tests to be linked directly to requirements. The detailed research on model-based testing approach in the last 5-10 years has confirmed the practicability of this approach. Hence, it showed that it can be cost-effective and has been developed in a variety of test generation strategies and model coverage criteria. Some of the popular commercial tools have started to emerge, which include from the USA (T-Vec, Reactive Systems, I-logix), in Europe (Conformiq, Leirios Technologies, Telelogic), as well as a wide variety of academic and research tools [19]

In order to understand the model based approach fully we need to define and understand the general description of a software model. In [4], model is defined as following:

“Simply put, a model of software is a depiction of its behaviour. Behaviour can be described in terms of the input sequences accepted by the system, the actions, conditions, and output logic, or the flow of data through the application's modules and routines. In order for a model to be useful for group of testers and for multiple testing tasks, it needs to be taken out of the mind of those who understand what the software is supposed to accomplish and written down in an easily understandable form. It is also generally preferable that a model be as formal as it is practical. With these properties, the model becomes shareable, reusable, precise description of the system under test.”

Models can be used to predict behaviour of software. As models are abstract representation therefore they are e generally simpler than the actual software [5]. The model must accurately predict behaviour as it is in general essential for the automated test case creation and particularly for model based testing. As stated earlier the due requirement of software models is to be more abstract than the actual system for which it is used to model [6]. This requirement is essential for the purpose of validation. In general simple models could be easily validated where as detailed or low level of modelling may make the system much more complex and validation and test cases generation will also become travail. If the model is at the same abstraction level as the software system then the actual efforts required for validating that the model, by validation we mean to find out that whether that model is correct or not, same as validating that the software is correct.

Software can be modelled in a number of ways and several approaches exist.  Each of this approach has it own pros and cons. In order to select a modelling approach mostly it is depended upon suit the software that is to be modelled. In [4] the most common approaches to software modelling are as under:

Finite State Machines:

[20] Finite state machine (FSM) is mostly used to mode the behaviour. FSM consists of states and transitions between those states. An example FSM is shown in Figure 2.

The simple FSM is a binary machine, where in this FSM q0 is the start state, and q3 the final or terminal state. Transitions are based on the occurrence of binary 1 or with binary 0. In this FSM the sequence 00 would go through states q0, q1 and end to q3, and where as the sequence 10101 use the states q0, q2, q0, q2, q0, and end to q3. In order to define FSM it is formally considered as quintuple (I, S, T, F, L), where

  • I is the set of inputs.
  • S is the set of all states in the model
  • T is a function which finds out whether a transition can occurs when an input is

Applied to the system in a particular state

  • F is the set of final states where the model can end up
  • L is the state where the software is launched

Markov Chains

Markov chains are stochastic models which make use of the probability distribution. Markov Chains can be used to software modelling. We can say that they are also state machines but these make use of some probability distribution for the transition to occur from one state to another. These are especially useful when are concentrating mostly on the reliability and mean time of failure.

State charts

State charts are considered as an extension of finite state machines. They are normally used to model complex or real-time systems. These systems can not be easily modelled using the finite state machines. In state charts, it is also possible to specify state machines in a hierarchy, whereas the upper level states contain a complete lower level state machine, or several machines. We can also add certain external conditions into the model that could determine whether certain transitions could occur or not. State charts are an equivalent to the Turing machine. They are easier to read compared to the finite state machines, but nontrivial to work with and require training beforehand.


Grammars are mostly used to describe the syntax of programming and other input languages. The grammars are alike to different forms of finite state machines. The grammars could be used for example to represent a model of a system in a more compact form than finite state machines do. They are also easy to write and maintain. On the other hand lack some property such as visualization of the language or model they represent.

The Unified Modelling Language

The unified modelling language (UML) is considered and equivalent to the finite state machines. But UML is commonly used to describe complicated behaviours of software, thus we can say that the simple graphical representation of a state machine is replaced by a structured language. UML Model contains two parts which is a model itself, and a set of diagrams. The model could be best depicted as a formal description of the diagrams, which are mostly used for visualization, and it has three essential parts: first is a functional model, this shows the system functionality from the user's perspective. Second is object model, entails system structure (objects corresponding methods and attributes and respective relationships between them). Third is dynamic model, this shows the internal behaviour of the system using for example a sequence, activity and state machine diagrams. The UML diagrams are further divided into three parts:

  • Structure diagrams reveal all the components of the modelled system which contains; e.g. classes, packages, objects and the overall structure of the modelled system.
  • Behaviour diagrams are used to describe how the system actually Works/behave. These could the respective. workflows and use cases.
  • Interaction diagrams illustrate the data and control flows of the system.

If the diagrams are studied in much more deeper than what may be discovered that the UML may contain more or less every model type described earlier in essay at least in some form. This actually leads to increased complexity. In fact UML can be used, and is designed to be used for modelling complicated software systems at every level possible low level.

Graph Transformation

Graphs represent the most basic model for entities and relations. Graphs are popular to be used for abstract representation of models. In UML model, a collection of object graphs is represented by means of a metamodel as abstract notation. Formally, the basic version of a graph consists of a set of vertices V and a set of edges E such that each edge e in E has a source and a target vertex s(e) and t(e) in V , respectively. More, advanced version of graph models use attributed graphs [21] in which the nodes and edges can be associated with attributes carrying textual, boolean and numeric information. Graphs can also use the concept of inheritance in their node types. Graphs occur at two level: type level and instancelevel. The type level is used to represent the metamodel and instance level as object graphs. In simple words, graphs provide a modeling language, where graph model is states and rules provide state changing operation. The concept can be more generally described by a type graph TG where the fixed type graph TG represents all the possible states in the model and the instance graph represents individual snap shots, whereas the transformation of instance graph results from application of transformation rules.

Use Case Model

Use case can be considered as the description simple task that a user performs. For example taking a picture is an activity which a user performs with help of the camera, from the user's point of view. Whereas in other words, use cases are used to collect the user requirements without excessive technical terminology, which is reflected in software design documents.

Use case model actually consists of several user events and their corresponding responses and respective system events. When events are combined together, they actually define the whole use case. This combination is called the flow of events. One way of defining use cases is UML is to use case diagram.

Application Model

The application model is opposite of the use case model which describes the behaviour of a whole application in general for example consider a music player or a clock. If we speak in terms of UML, the application model is considered as the same as the behaviour model. The model actually contains every entry point as well as exit point of an application and every s possible state between them. Hence, every user action is also modelled.

In fact software models are not self created but they must be created somehow. This fact is often ignored in research papers on model based testing approach. In reference at [4], the following three steps are considered for software modelling:

First step is to understand the system from functional and non functional aspects about the software and its environment. This could include for example to determine the components that need to be tested, reviewing available documentation, communicating with other teams involved in the software engineering, potential user identification and  documenting every input, response and the sequences of inputs that need to be modelled. The second step is to select the right model type for the software being modelled. There are also other important factors for choosing the model than that the model is adequate for the software  such as the people who are going to actually work with the model have to

be taken into consideration. The third step is to develop the model by making a list of all possible inputs to the system, document the situations where user can and cannot actually apply the input and document the situations where input causes different responses from the system.

Once the model is complete, it has to be transformed into state machine. There are several algorithms for automating this task.  Some of them are presented in reference [10].

Another approach to model creation is by Harry Robinson which is called operational modes in reference [5]. Operational mode states that what user actions are possible in a state and what will be the system responses, when the actions are executed. However, the use of operational modes is suggested because of “A finite state model used to represent the behaviour of an application is likely to have many, by many states we mean that so many that it would be tedious and unrealistic to create and maintain the model by hand”, however, no solution is presented how the actual operational modes should be created and maintained so that the task would not be tedious and unrealistic.

Both of the approaches seem to be most suitable when the model is to be created manually. It is worth mentioning that there exist several algorithms which could automate the process of this conversion to the state machines but still it require a lot of manual work.

One solution to this sort problem in order to reduce manual creation work, is probably to use the model throughout the software engineering process, starting right from the design phase. If the software is depicted as a model from the start then we don't need to create a separate creation process. This approach is commonly referred as model driven development.

The aim of model driven development is that the primary focal point and products of software development are models rather than computer programs [11]. The prime importance of this approach is that models are expressed using concepts such as that are less bound to the underlying implementation technology and are much closer to the actual problem domain. In fact that modelling can be done in any language or with any existing method; for example a UML can be used as well as UI specifications. The choice of the modelling language or method mostly depends on the level of modelling required.

The basic idea behind the model driven development is to automate process of the transformation of models from one form to another [12]. This means that models are prime imported from some form which could be, such as the methods described before and exported to another form. For example a C source code. In order for the transformation to work we must define precisely the import and export languages. The choice language definition is open. This means that the models can be transformed practically from one form to other no matter whatever the form could be. The key issue in transformation is the mapping of source language to the target language. For example, UML class diagrams can be easily mapped to Java class skeletons, and UI specifications to test automation scripts.

Models are not considered useful if the ultimate product of the modelling is the documentation which could be used for the software development. The maximum benefits of modelling could only be achieved if models are used as seeds for automatic code generation and verification activities. In this process of the verification part is known as model based testing.

The prime vision of automatic code generation is that the model would take the role of implementation languages similar to the way the third-generation programming languages (such as C++) displaced assembly languages [11]. It is important to note that code generation would not produce only code skeletons but also complete and working programs. After this automation we would not need to examine or modify the generated programs. There is e is no need to examine or modify the assembly language produced by third-generation programming language compilers. Hence, this is a goal that is hard to achieve, but as it is referred in [11], there do exists tools and techniques that could generate better code than handcrafted would ever be, even when the models describe large-scale industrial applications.

In short Model based testing can be defined as a technique for automatic creation of test cases from specified software model. The prime advantage of this technique is that the test generation can systematically derive all combination of tests associated with the requirements represented in the model to automate both the test design and test execution process [13].

In order to generate a test case from a finite state machine is usually straightforward and easy work. However, the states in the machine contain information about what the state are and in the case of actions, they reveal the action that is to be executed, and in the case of responses, they describe formally, what the response is actually. If we execute the test case manually manually, this is all the information that is needed. If the motivation is to generate automated test cases then the states need to contain all the necessary test automation parameters that are needed by the test automation system used. The test automation parameters can be of course be derived also from some other information, such as by using a text parser to generate the parameters from the verbal descriptions automatically.

Since all states contain all the required information so the actual test case generation is

Fairly simple task. The simplest approach to this problem is to traverse randomly through the state transition diagram, from start to the end. Whereas, one test case is constituted of the states visited during traversing. Normally it is wanted that the transition coverage criterion is hundred percent and then random traversing is out of the question and a more complicated algorithm is needed. In reference [2], graph traversal algorithms are presented the goal is to meet the criterion, and the core challenge is to choose the most effective one.

In several advantages and disadvantages of model based testing are presented, and

those are explored next.

In [4] many research studies have shown that model based testing is efficient, particularly when testing is used for small applications, embedded systems, user interfaces and state-rich systems with reasonably complex data. Rosaria and Robinson (2000) study concentrated on testing graphical user interfaces (GUI), Agrawal and Whittaker (1993) studied the embedded control software and whereas Avritzer and Larson (1993) studied phone systems.

It is important to note that the most attractive attribute of model based testing is considered to be the automatic generation of test cases, but that is not the end. Model of software can help in specifying the unclear and poorly defined requirements [13]. The Model could be used to remove defects before even the coding begins and automating the test case creation the result is significant cost savings and higher quality code. Figure 6 [14] shows the differences between

Current defect discovery and removing process (marked “Old”) and early defect discovery (marked “New”). This graph shows that the quicker the defects are detected and fixed, the less the costs of fixing them will be. Some other benefits of the testing are as described in [13]. If we have complete model then it is possible to generate all the possible test case. In this test case could ultimately cover every possible transition of the model by using graph algorithms. Another important advantage is that model based test automation has the capability to discovers defects more effectively than manual methods.

It would be wrong to say that almost all research on model based testing agrees on one one problem agenda which is the deployment of the model based testing into an organization requires significant efforts and as well as financial investments.

In [4] the reason behind this is amount of skills is required from the testers. The tester needs to be familiar with the model, hence, knowledge of different forms of state machines, respective formal languages, and automata theory would be essential required. Along with this the testers would also need familiarity with the tools and scripts are required when test automation is going to be used. This will also requite sufficient amount of time mean working houres. Another point is that the model must be carefully selected. The divide and conquer rule be applied. This mean that large problem shall be divided into small parts and each addressed by modelling separately. Models do have some draw backs. The most popular the explosion of state-space needed. It is important to note that even a simple application may contain so many states that the maintenance of the model becomes difficult and tedious job.

Here we now discuss some of the existing tools. The tools below are mostly take from the paper at the reverence [15], and some can be easily seen on the internet as each tool has there own dedicated we link. Some of these tools listed here are either commercially or otherwise publicly available free of cost.

AETGâ„¢ by Telecordiaâ„¢ Technologies is a tool which is web based and this tool could be used for pair wise test case generation, which uses the input data domain as the model. This means that this tool could be used cases such as databases or user input data validation.

Case Maker®by Diaz & Hilterscheid Unternehmensberatung GmbH this too is also used to generate pair

wise test cases. This is also using the input data domain as the model but it has the additional capability to use constraints specified by business rules, i.e. the business domain.

Conformiq Test Generatorâ„¢ by Conformiq this tool takes UML state chart and generate respective test cases. This represent a high-level graphical test script. It has to be noted that the state charts do not represent the actual system under test but only the respective test script. This means that the tool is more a test script editor rather than a real model based testing tool.

CTesK, JTesK by UniTESK makes used of the formal representations of software requirements as the input for test case generation. All these requirements are written directly to the source code with the help of extensions to the programming language, and the process oftransforming the requirements into test cases is automated.

MaTeLo by All4Tec generates test cases based on the model of the system under test. The respective model is created manually through the editor which is provided by the software. This tool uses the theory of the Markov chain theories.

Qtronicâ„¢ by this tool automates the process of e test case generation from software design models. The specification of the tool on the web says that UML model can be imported from other tools as well.

Raveâ„¢ by T-VEC: this tool is used to model and analyze system requirements before even the coding begins. These requirements models are subsequently used to generate test cases automatically. The modelling is done with so-called tabular notation editor that is also available from T-VEC.

Reacts® by Reactive System is is mostly used for the testing of the embedded Systems. This software uses the Simulink® and Stateflow® models for test case generations.

SmartTest by Smartware Technologies make use of the same pair wise technologies as used by the AETG and Case Maker. The model they use for test case generations is the input data domain.

Statemate™ Automatic Test Generator™ / Rhapsody® Automatic Test Generator™(ATG) by I-Logix use state charts of the system and it can also use UML

state machine for test case generations. It is worth mentioning that it also provide a tool for the stat char creation.

TAU Tester by Telelogic makes use of the TTCN-3 as the modelling notation and further use this for respective test case generation automatically. The only drawback of this tool is that models have to be created by hand.

Test Cover by is a web-based testing service. This tool utilize the same concepts of the  pair wise techniques and input data domain..

TEMA Tool by Research Group on Software Engineering in the Institute of Software

Systems in Tampere University of Technology is currently used such as for S60 UI

Testing. This tool is based on test models. The model is used to specify the behaviour of the system under test from the perspective of testing.


[1] Fewster, M. & Graham, D. Software Test Automation. Harlow, United Kingdom: Pearson Education Ltd, 1999. 574 p. ISBN 0-201-33140-3.

[2] Robinson, H. Graph Theory Techniques in Model-Based Testing. 16th International Conference and Exposition on Testing Computer Software, Los Angeles, California, USA, 1999.

[3] Beizer, B. Software Testing Techniques. 2nd Ed. New York, USA: Van Nostrand Reinhold Co, 1990. 550 p. ISBN 0-442-20672-0.

[4] El-Far, I. K. & Whittaker, J. A. Model-based Software Testing. In: Marciniak, J. (ed.), Encyclopedia on Software Engineering, Volume 1. New York, USA: John Wiley & Sons Inc, 2001. pp. 825-837. ISBN 0- 471-21008-0.

[5] Robinson, H. Finite state model-based testing on a shoestring. International Conference on Software Testing Analysis and Review, San Jose, California, USA, 1999.

[6] Pretschner, A., Prenninger, W., Wagner, S., Kühnel, C., Baumgartner, M., Sostawa, B., Zölch, R. & Stauner, T. One Evaluation of Model- Based Testing and its Automation. In: Proceedings of the 27th International Conference on Software Engineering. St. Louis, MO, USA. 15.-21.5.2005. New York, NY, USA: The Association for Computing Machinery. pp. 392-402. ISBN 1-59593-963-2.

[7] Apfelbaum, L. & Doyle, J. Model Based Testing. 10th International Software Quality Week, San Francisco, California, USA, 1997.

[8] Nikkanen, M. Use Case Based Automatic User Interface Testing in Mobile Devices. Master's thesis. Helsinki University of Technology, Department of Computer Science and Engineering. Espoo. 2005. 79 p.

[9] Collard, R. Test Design: Developing test cases from use cases. Software Testing & Quality Engineering Magazine, 1999. Vol. 1:4. pp. 30-37.

[10] Prowell, S. J. TML: a description language for Markov chain usage models. Information and Software Technology, 2000. Vol. 42:12. p.  825-833.

[11] Selic, B. The Pragmatics of Model-Driven Development. IEEE Software, 2003. Vol. 20:5. pp. 19-25.

[12] Mellor, S. J., Clark, A. N. & Futagami, T. Model-Driven Development. IEEE Software, 2003. Vol. 20:5. pp. 14-18.

[13] Blackburn, M., Busser, R. & Nauman, A. Why Model-Based Test Automation is Different and What You Should Know to Get Started. International Conference on Practical Software Quality and Testing, Washington, USA, 2004.

[14] Safford, E. Test Automation Framework, State-based and Signal Flow Examples. Twelfth Annual Software Technology Conference, Salt Lake City, USA, 2000.

[15] Utting, M. & Legeard, B. Practical Model-Based Testing: A Tools Approach. 1st ed. San Francisco, CA, USA: Morgan Kauffmann, 2007. 456 p. ISBN 0-12-372501-1.

[16] I.K. El-Far and J.A. Whittaker. Model-based software testing. In John J. Marciniak, editor, Encyclopedia of Software Engineering, volume 1, pages 825-837. Wiley- nterScience, 2002.

[17] [BBN04] M. Blackburn, R. Busser, and A. Nauman. Why model-based test automation is different and what you should know to get started. In International

Conference on Practical Software Quality and Testing, 2004. See

[18] B. Legeard, F. Peureux, and M. Utting. Controlling Test Case Explosion in Test Generation from B Formal Models. The Journal of Software Testing, Verification and Reliability, 14(2):81-103, 2004.

[19]A. Belinfante, L. Frantzen, and C. Schallhart. Tools for Test Case Generation. In M. Broy, B. Jonsson, J.-P. Katoen, M. Leucker, and A. Pretschner, editors, Model-Based Testing of Reactive Systems [BJK+05], Springer LNCS 3472, pages 391-438. Springer-Verlag, 2005.

[20] Wagner, F., "Modeling Software with Finite State Machines: A Practical Approach", Auerbach Publications, 2006, ISBN 0-8493-8086-3

[21] Conrad, M.;Dörr, H.; Stuermer, I.; Schuerr, A.: Graph Transformations for Model-based Testing. Proc. of Modellierung 2002, Tutzing (D), März 2002