Chapter 1

1.1 Introduction

Software engineering [PRE01] is the process associated with industrial quality software development, the methods used to analyze, design & test computer Software, the management techniques associated with the control & monitoring of Software projects & the tools used to support process, methods, & techniques. In Software Development Life Cycle, the focus is on the activities like feasibility study, requirement analysis, design, coding, testing, & maintenance.

Feasibility study involves the issues like technical/economical/ behavioral feasibility of project. Requirement analysis [DAV93] emphasizes on identifying the needs of the system & producing the Software Requirements Specification document (SRS), [JAL04] that describes all data, functional & behavioral requirements, constraints, & validation requirements for Software.

Software Design is to plan a solution of the problem specified by the SRS document, a step in moving from the problem domain to the solution domain. The output of this phase is the design document. Coding is to translate the design of the system into code in a programming language. Testing is the process to detect defects & minimize the risk associated with the residual defects. The activities carried out after the delivery of the software comprises the maintenance phase.

1.2 Evolution of Software Testing Discipline

The effective functioning of modern systems depends on our ability to produce software in a cost-effective way. The term software engineering was first used at a 1968 NATO workshop in West Germany. It focused on the growing software crisis. Thus we see that the software crisis on quality, reliability, high costs etc. started way back when most of today's software testers were not even born.

The attitude towards Software Testing [BEI90] underwent a major positive change in the recent years. In the 1950's when Machine languages were used, testing was nothing but debugging. When in the 1960's, compilers were developed, testing started to be considered a separate activity from debugging.

In the 1970's when the software engineering concepts were introduced, software testing began to evolve as a technical discipline. Over the last two decades there has been an increased focus on better, faster and cost-effective software. Also there has been a growing interest in software safety, protection and security and hence an increased acceptance of testing as a technical discipline and also a career choice.

Now to answer, What is Testing? we can go by the famous definition of Myers [MYE79], which says, Testing is the process of executing a program with the intent of finding errors. According to Humphrey, software testing is defined as, the execution of a program to find its faults. Testing is the process to prove that the software works correctly [PRA06].

Software testing is a crucial aspect of the software life cycle. In some form or the other it is present at each phase of (any) software development or maintenance model. The importance of software testing and its impact on software cannot be underestimated. Software testing is a fundamental component of software quality assurance and represents a review of specification, design and coding. The greater visibility of software systems and the cost associated with software failure are motivating factors for planning, through testing. It is not uncommon for a software organization to spend 40-50% of its effort on testing.

During testing, the software engineering produces a series of test cases that are used to rip apart the software they have produced. Testing is the one step in the software process that can be seen by the developer as destructive instead of constructive. Software engineers are typically constructive people and testing requires them to overcome preconceived concepts of correctness and deal with conflicts when errors are identified.

A successful test is one that finds a defect. This sounds simple enough, but there is much to consider when we want to do software testing. Besides finding faults, we may also be interested in testing performance, safety, fault-tolerance or security. Testing often becomes a question of economics. For projects of a large size, more testing will usually reveal more bugs. The question then becomes when to stop testing, and what is an acceptable level of bugs. This is the question of good enough software.

Testing is the process of verifying that a product meets all requirements. A test is never complete. When testing software the goal should never be a product completely free from defects, because it's impossible. According to Peter Nielsen, The average is 16 faults per 1000 lines of code when the programmer has tested his code and it is believed to be correct. When looking at a larger project, there are millions of lines of code, which makes it impossible to find all present faults. Far too often products are released on the market with poor quality. Errors are often uncovered by users, and in that stage the cost of removing errors is large in amount.

1.3 Objectives of Testing

Glen Myers [MYE79] states a number of rules that can serve well as testing objectives:

  • Testing is a process of executing a program with the intent of finding an error.
  • A good test is one that has a high probability of finding an as yet undiscovered error.
  • A successful test is one that uncovers an as yet undiscovered error.
  • The objective is to design tests that systematically uncover different classes of errors & do so with a minimum amount of time & effort.

Secondary benefits include

  • Demonstrate that Software functions appear to be working according to specification.
  • That performance requirements appear to have been met.
  • Data collected during testing provides a good indication of Software reliability & some indication of Software quality.
  • Testing cannot show the absence of defects, it can only show that Software defects are present.

1.4 Software Testing & Its Relation with Software Life Cycle

Software testing should be thought of as an integral part of the Software process & an activity that must be carried out throughout the life cycle.

Each phase in the Software lifecycle has a clearly different end product such as the Software requirements specification (SRS) documentation, program unit design & program unit code. Each end product can be checked for conformance with a previous phase & against the original requirements. Thus, errors can be detected at each phase of development.

  • Validation & Verification should occur throughout the Software lifecycle.
  • Verification is the process of evaluating each phase end product to ensure consistency with the end product of the previous phase.
  • Validation is the process of testing Software, or a specification, to ensure that it matches user requirements.

Software testing is that part of validation & verification associated with evaluating & analysing program code. It is one of the two most expensive stages within the Software lifecycle, the other being maintenance. Software testing of a product begins after the development of the program units & continues until the product is obsolete.

Testing & fixing can be done at any stage in the life cycle. However, the cost of finding & fixing errors increases dramatically as development progresses.

Changing a Requirements document during the first review is inexpensive. It costs more when requirements change after the code has been written: the code must be rewritten. Bug fixes are much cheaper when programmers find their own errors. Fixing an error before releasing a program is much cheaper than sending new disks, or even a technician to each customer's site to fix it later. It is illustrated in Figure 1.1.

The types of testing required during several phases of Software lifecycle are described below:


Requirements must be reviewed with the client; rapid prototyping can refine requirements & accommodate changing requirements.


The specifications document must be checked for feasibility, traceability, completeness, & absence of contradictions & ambiguities.

Specification reviews (walkthroughs or inspections) are especially effective.


Design reviews are similar to specification reviews, but more technical.

The design must be checked for logic faults, interface faults, lack of exception handling, & non-conformance to specifications.


Code modules are informally tested by the programmer while they are being implemented (desk checking).

Thereafter, formal testing of modules is done methodically by a testing team. This formal testing can include non-execution-based methods (code inspections & walkthroughs) & execution-based methods (black-box testing, white-box testing).


Integration testing is performed to ensure that the modules combine together correctly to achieve a product that meets its specifications. Particular care must be given to the interfaces between modules.

The appropriate order of combination must be determined as top-down, bottom-up, or a combination thereof.

Product Testing

The functionality of the product as a whole is checked against its specifications. Test cases are derived directly from the specifications document. The product is also tested for robustness (error-handling capabilities & stress tests).

All source code & documentation are checked for completeness & consistency.

Acceptance Testing

The Software is delivered to the client, who tests the Software on the actual h/w, using actual data instead of test data. A product cannot be considered to satisfy its specifications until it has passed an acceptance test.

Commercial off-the-shelf (or shrink-wrapped) Software usually undergoes alpha & beta testing as a form of acceptance test.


Modified versions of the original product must be tested to ensure that changes have been correctly implemented.

Also, the product must be tested against previous test cases to ensure that no inadvertent changes have been introduced. This latter consideration is termed regression testing.

Software Process Management

The Software process management plan must undergo scrutiny. It is especially important that cost & duration estimates be checked thoroughly.

If left unchecked, errors can propagate through the development lifecycle & amplify in number & cost. The cost of detecting & fixing an error is well documented & is known to be more costly as the system develops. An error found during the operation phase is the most costly to fix.

1.5 Principles of Software Testing

Software testing is an extremely creative & intellectually challenging task. The following are some important principles [DAV95] that should be kept in mind while carrying Software testing [PRE01] [SUM02]:

Testing should be based on user requirements: This is in order to uncover any defects that might cause the program or system to fail to meet the client's requirements.

Testing time & resources are limited: Avoid redundant tests.

It is impossible to test everything: Exhaustive tests of all possible scenarios are impossible, because of the many different variables affecting the system & the number of paths a program flow might take.

Use effective resources to test: This represents use of the most suitable tools, procedures & individuals to conduct the tests. Only those tools should be used by the test team that they are confident & familiar with. Testing procedures should be clearly defined. Testing personnel may be a technical group of people independent of the developers.

Test planning should be done early: This is because test planning can begin independently of coding & as soon as the client requirements are set.

Test for invalid & unexpected input conditions as well as valid conditions: The program should generate correct messages when an invalid test is encountered & should generate correct results when the test is valid.

The probability of the existence of more errors in a module or group of modules is directly proportional to the number of errors already found.

Testing should begin at the module: The focus of testing should be concentrated on the smallest programming units first & then expand to other parts of the system.

Testing must be done by an independent party: Testing should not be performed by the person or team that developed the Software since they tend to defend the correctness of the program.

Assign best personnel to the task: Because testing requires high creativity & responsibility only the best personnel must be assigned to design, implement, & analyze test cases, test data & test results.

Testing should not be planned under the implicit assumption that no errors will be found.

Testing is the process of executing Software with the intention of finding errors.

Keep Software static during test: The program must not be modified during the implementation of the set of designed test cases.

Document test cases & test results.

Provide expected test results if possible: A necessary part of test documentation is the specification of expected results, even though it is impractical.

1.6 Software Testability & Its Characteristics

Testability is the ability of Software (or program) with which it can easily be tested [PRE01] [SUM02]. The following are some key characteristics of testability:

  • The better it works, the more efficient is testing process.
  • What you see is what you test (WYSIWYT).
  • The better it is controlled, the more we can automate or optimize the testing process.
  • By controlling the scope of testing we can isolate problems & perform smarter retesting.
  • The less there is to test, the more quickly we can test it.
  • The fewer the changes, the fewer the disruptions to testing.
  • The more information we have, the smarter we will test.

1.7 Stages in Software Testing Process

Except for small programs, systems should not be tested as a single unit. Large systems are built out of sub-systems, which are built out of modules that are composed of procedures & functions. The testing process should therefore proceed in stages where testing is carried out incrementally in conjunction with system implementation.

The most widely used testing process consists of five stages that are illustrated in Table 1.1.

Errors in program components, say may come to light at a later stage of the testing process. The process is therefore an iterative one with information being fed back from later stages to earlier parts of the process. The iterative testing process is illustrated in Figure 1.2 and described below:

Unit Testing: Unit testing is code-oriented testing. Individual components are tested to ensure that they operate correctly. Each component is tested independently, without other system components.

Module Testing: A module is a collection of dependent components such as an object class, an abstract data type or some looser collection of procedures & functions. A module encapsulates related components so it can be tested without other system modules.

Sub-system (Integration) Testing: This phase involves testing collections of modules, which have been integrated into sub-systems. It is a design-oriented testing & is also known as integration testing.

Sub-systems may be independently designed & implemented. The most common problems, which arise in large Software systems, are sub-systems interface mismatches. The sub-system test process should therefore concentrate on the detection of interface errors by rigorously exercising these interfaces.

System Testing: The sub-systems are integrated to make up the entire system. The testing process is concerned with finding errors that result from unanticipated interactions between sub-systems & system components. It is also concerned with validating that the system meets its functional & non-functional requirements.

Acceptance Testing: This is the final stage in the testing process before the system is accepted for operational use. The system is tested with data supplied by the system client rather than simulated test data. Acceptance testing may reveal errors & omissions in the systems requirements definition (user-oriented) because real data exercises the system in different ways from the test data.

Acceptance testing may also reveal requirement problems where the system facilities do not really meet the user's needs (functional) or the system performance (non-functional) is unacceptable.

1.8 The V-model of Testing

To test an entire software system, tests on different levels are performed. The V model [FEW99], shown in figure 1.3, illustrates the hierarchy of tests usually performed in software development projects. The left part of the V represents the documentation of an application, which are the Requirement specification, the Functional specification, System design, the Unit design.

Code is written to fulfill the requirements in these specifications, as illustrated in the bottom of the V. The right part of the V represents the test activities that are performed during development to ensure that an application corresponding to its requirements.

Unit tests are used to test that all functions and methods in a module are working as intended. When the modules have been tested, they are combined and integration tests are used to test that they work together as a group. The unit- and integration test complement the system test. System testing is done on a complete system to validate that it corresponds to the system specification. A system test includes checking if all functional and all non-functional requirements have been met.

Unit, integration and system tests are developer focused, while acceptance tests are customer focused. Acceptance testing checks that the system contains the functionality requested by the customer, in the Requirement specification. Customers are usually responsible for the acceptance tests since they are the only persons qualified to make the judgment of approval. The purpose of the acceptance tests is that after they are preformed, the customer knows which parts of the Requirement specification the system satisfies.

1.9 The Testing Techniques

To perform these types of testing, there are three widely used testing techniques. The above said testing types are performed based on the following testing techniques:

Black-Box testing technique

Black box testing (Figure 1.4) is concerned only with testing the specification. It cannot guarantee that the complete specification has been implemented. Thus black box testing is testing against the specification and will discover faultsofomission, indicating that part of the specification has not been fulfilled. It is used for testing based solely on analysis of requirements (specification, user documentation).

In Black box testing, test cases are designed using only the functional specification of the software i.e without any knowledge of the internal structure of the software. For this reason, black-box testing is also known as functional testing. Black box tests are performed to assess how well a program meets its requirements, looking for missing or incorrect functionality. Functional testing typically exercise code with valid or nearly valid input for which the expected output is known. This includes concepts such as 'boundary values'.

Performance tests evaluate response time, memory usage, throughput, device utilization, and execution time. Stress tests push the system to or beyond its specified limits to evaluate its robustness and error handling capabilities. Reliability tests monitor system response to represent user input, counting failures over time to measure or certify reliability.

Black box Testing refers to analyzing a running program by probing it with various inputs. This kind of testing requires only a running program and does not make use of source code testing of any kind. In the security paradigm, malicious input can be supplied to the program in an effort to cause it to break. If the program breaks during a particular test, then a security problem may have been discovered.

Black box testing is possible even without access to binary code. That is, a program can be tested remotely over a network. All that is required is a program running somewhere that is accepting input. If the tester can supply input that the program consumes (and can observe the effect of the test), then black box testing is possible. This is one reason that real attackers often resort to black box techniques. Black box testing is not an alternative to white box techniques. It is a complementary approach that is likely to uncover a different type of errors that the white box approaches.

Black box testing tries to find errors in the following categories:

  • Incorrect or missing functions
  • Interface errors
  • Errors in data structures or external database access
  • Performance errors, and
  • Initialization and termination errors.

By applying black box approaches we produce a set of test cases that fulfill requirements:

  • Test cases that reduce the number of test cases to achieve reasonable testing
  • Test cases that tell us something about the presence or absence of classes of errors.

The methodologies used for black box testing have been discussed below: Equivalent Partitioning

Equivalence partitioning is a black box testing approach that splits the input domain of a program into classes of data from which test cases can be produced. An ideal test case uncovers a class of errors that may otherwise before the error is detected. Equivalence partitioning tries to outline a test case that identifies classes of errors.

Test case design for equivalent partitioning is founded on an evaluation of equivalence classes for an input condition [BEI95]. An equivalence class depicts a set of valid or invalid states for the input condition. Equivalence classes can be defined based on the following [PRE01]:

  • If an input condition specifies a range, one valid and two invalid equivalence classes are defined.
  • If an input condition needs a specific value, one valid and two invalid equivalence classes are defined.
  • If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined.
  • If an input condition is Boolean, one valid and invalid class is outlined. Boundary Value Analysis

A great many errors happen at the boundaries of the input domain and for this reason boundary value analysis was developed. Boundary value analysis is test case design approach that complements equivalence partitioning. BVA produces test cases from the output domain also [MYE79].

Guidelines for BVA are close to those for equivalence partitioning [PRE01]:

  • If an input condition specifies a range bounded by values a and b, test cases should be produced with values a and b, just above and just below a and b, respectively.
  • If an input condition specifies various values, test cases should be produced to exercise the minimum and maximum numbers.
  • Apply guidelines above to output conditions.
  • If internal program data structures have prescribed boundaries, produce test cases to exercise that data structure at its boundary.

White-Box testing technique

White box testing (Figure 1.5) is testing against the implementation as it is based on analysis of internal logic (design, code etc.) and will discover faultsofcommission, indicating that part of the implementation is faulty. Designing white-box test cases requires thorough knowledge of the internal structure of software, and therefore the white-box testing is also called the structural testing. White box testing is performed to reveal problems with the internal structure of a program.

A common goal of white-box testing is to ensure a test case exercises every path through a program. A fundamental strength that all white box testing strategies share is that the entire software implementation is taken into account during testing, which facilitates error detection even when the software specification is vague or incomplete. The effectiveness or thoroughness of white-box testing is commonly expressed in terms of test or code coverage metrics, which measure the fraction of code exercised by test cases.

White box Testing involves analyzing and understanding source code. Sometimes only binary code is available, but if you decompile a binary to get source code and then study the code, this can be considered a kind of white box testing as well. White box testing is typically very effective in finding programming errors and implementation errors in software. In some cases this activity amounts to pattern matching and can even be automated with a static analyzer.

White box testing is a test case design approach that employs the control architecture of the procedural design to produce test cases. Using white box testing approaches, the software engineering can produce test cases that:

  • Guarantee that all independent paths in a module have been exercised at least once
  • Exercise all logical decisions
  • Execute all loops at their boundaries and in their operational bounds
  • Exercise internal data structures to maintain their validity.

There are several methodologies used for white box testing. We discuss some important ones below. Statement Coverage

The statement coverage methodology aims to design test cases so as to force the executions of every statement in a program at least once. The principal idea governing the statement coverage methodology is that unless a statement is executed, we have way of determining if an error existed in that statement. In other words, the statement coverage criterion [RAP85] is based on the observation that an error existing in one part of a program cannot be discovered if the part of the program containing the error and generating the failure is not executed. However, executed a statement once and that too for just one input value and observing that it behaves properly for that input value is no guarantee that it will behave correctly for all inputs. Branch Coverage

In branch coverage testing, test cases are designed such that the different branch conditions are given true and false values in turn. It is obvious that branch testing guarantees statement coverage and thus is a stronger testing criterion than the statement coverage testing [RAP85]. Path Coverage

The path coverage based testing strategy requires designing test cases such that all linearly independents paths in the program are executed at least once. A linearly independent path is defined in terms of the control flow graph (CFG) of the program. Loop testing

Loops are very important constructs for generally all the algorithms. Loop testing is a white box testing technique. It focuses exclusively on the validity of loop constructs. Simple loop, concatenated loop, nested loop, and unstructured loop are four different types of loops [BEI90] as shown in figure 1.6.

Simple Loop: The following set of tests should be applied to simple loop where n is the maximum number of allowable passes thru the loop:

  • Skip the loop entirely.
  • Only one pass thru the loop.
  • Two passes thru the loop.
  • M passes thru the loop where m < n.
  • N-1, n, n+1 passes thru the loop.
  • Nested Loop: Beizer [BEI90] approach to the nested loop
  • Start at the innermost loop. Set all other loops to minimum value.
  • Conduct the simple loop test for the innermost loop while holding the outer loops at their minimum iteration parameter value.
  • Work outward, conducting tests for next loop, but keeping all other outer loops at minimum values and other nested loops to typical values.
  • Continue until all loops have been tested.

Concatenated loops: These can be tested using the approach of simple loops if each loop is independent of other. However, if the loop counter of loop 1 is used as the initial value for loop 2 then approach of nested loop is to be used.

Unstructured loop: This class of loops should be redesigned to reflect the use of the structured programming constructs. McCabe's Cyclomatic Complexity

The McCabe's Cyclomatic Complexity [MCC76] of a program defines the number of independent paths in a program. Given a control flow Graph G of a program, the McCabe's Cyclomatic Complexity V(G) can be computed as:


Where E is the number of edges in the control flow graph and N is the number of nodes of the control flow graph.

The cyclomatic complexity value of a program defines the number of independent paths in the basis set of the program and provides a lower bound for the number of test cases that must be conducted to ensure that all statements have been executed at least once. Knowing the number of test cases required does not make it easy to derive the test cases, it only gives an indication of the minimum number of test cases required.

The following is the sequences of steps that need to be undertaken for deriving the path coverage based test case of a program.

  • Draw the CFG.
  • Calculate Cyclomatic Complexity V(G).
  • Calculate the basis set of linearly independent paths.
  • Prepare a test case that will force execution of each path in the basis set. Data Flow based Testing

The data flow testing method chooses test paths of a program based on the locations of definitions and uses of variables in the program. Various data flow testing approaches have been examined [FRA88] [NTA88] [FRA93]. For data flow testing each statement in program is allocated a unique statement number and that each function does not alter its parameters or global variables. For a statement with S as its statement number,

DEF(S) = {X| statement S contains a definition of X}

USE(S) = {X| statement S contains a use of X}

If statement S is if or loop statement, its DEF set is left empty and its USE set is founded on the condition of statement S. The definition of a variable X at statement S is live at statement S, if there exists a path from statement S to S' which does not contain any condition of X.

A definition-use chain (or DU chain) of variable X is of the type [X,S,S'] where S and S' are statement numbers, X is in DEF(S), USE(S'), and the definition of X in statement S is live at statement S'.

One basic data flow testing strategy is that each DU chain be covered at least once. Data flow testing strategies are helpful for choosing test paths of a program including nested if and loop statements

1.9.3 Grey-Box testing technique

Grey box testing [BIN99] designs test cases using both responsibility-based (black box) and implementation-based (white box) approaches. To completely test a web application one needs to combine the two approaches, White-box and Black-box testing. It is used for testing of Web based applications. The Gray-box testing approach takes into account all components making up a complete web application, including the environment on witch it resides.

Nguyen Hung Q [NGU01] states that Gray-box testing is integral to web application testing because of the numerous components, both software and hardware, that make up the complete application. The factors considered in Gray-box testing are high-level design, environment and interoperability conditions.

1.10 Some more testing types

From the V-model (Figure 1.3) [FEW99], we see that there are various levels or phases of testing, namely, Unit testing, Integration testing, System testing, User Acceptance testing etc. A brief definition on some more widely employed types of testing is given below:

Regression Testing: Selective re-testing of a system to verify the modification (bug fixes) have not caused unintended effects and that system still complies with its specified requirements

Performance Testing: To evaluate the time taken or response time of the system to perform it's required functions in comparison.

Stress Testing: To evaluate a system beyond the limits of the specified requirements or system resources (such as disk space, memory, processor utilization) to ensure the system do not break unexpectedly.

Load Testing: Load Testing, a subset of stress testing, verifies that a web site can handle a particular number of concurrent users while maintaining acceptable response times.

Alpha Testing: Testing of a software product or system conducted at the developer's site by the customer.

Beta Testing: Testing conducted at one or more customer sites by the end user of a delivered software product system.

Chapter 2 Literature Survey & Objectives of Research

2.1 Literature Survey

In order to carry out the research in the domain of software testing, an exhaustive survey of the literature was carried out resulting in the elicitation and acquisition of the knowledge. The crux of various research papers related to the field of software testing and web testing in chronological order is discussed in the following section:

Kung David C. et al [KUN00] in the paper An Object-Oriented Web Test Model for Testing Web Applications presented a methodology that used an Object-Oriented Web Test Model (WTM) to support Web application testing. The test model captured both structural and behavioral test artifacts of Web applications and represented the artifacts form the object, behavior, and structure perspectives. Based on the test model, both structural and behavioral test cases were derived automatically to ensure the quality of Web applications.

From the object perspective, entities of a Web application were represented using Object Relation Diagram (ORD) in terms of objects and inter-dependent relationships. From the behaviour perspective, a Page Navigation Diagram (PND) was used to depict the navigation behaviour of a Web application. At the same time, a set of Object State Diagrams (OSDs) were used to describe the state behaviour of interacting objects.

From the structure perspective, the Block Branch Diagram (BBD) and Function Cluster Diagram (FCD) were employed to capture the control flow and data flow information of scripts and functions in a Web application. Moreover, special testing concerns were expressed in terms of textural test constraints that could be imposed on the corresponding objects. The model could also be used as a road map to identify change ripple effects and to find cost-effective testing strategies for reducing test efforts required in the regression testing.

In the study Thinking Aloud as a Method for Testing the Usability of Websites: The Influence of Task Variation on the Evaluation of Hypertext, Waes Luuk Van [WAE00] focused on three variants of task types for the thinking-aloud method. In the study, the participants were asked to think aloud while looking for information on a website from different perspectives.

The author analyzed the results of three usability studies in which different thinking-aloud tasks were used: a simple searching task, an application task, and a prediction task. In the task setting the profile of the Web-surfer, the communication goal of the website and other quality aspects were taken into account. The qualitative analysis of these studies showed that the task variation had some influence on the results of usability testing and that, consequently, tasks should be matched with the evaluation goals put forward.

The problem of analyzing frequently traversed spots in a Web site has been explored by Lin Yuh-Chi and Paul Hadingham [LIN00] in the paper Tracking Frequent Traversal Areas in a Web Site via Log Analysis. Firstly, a support knowledge base from Web logs was constructed and the problem of tracking frequent traversal areas as a problem of finding strongly connected components in a digraph was formulated. The tracking features into their test Web site were successfully incorporated. This approach could be useful in the innovative applications such as Web promotion, banner ads, and Web content design in the electronic commerce.

Data flow testing techniques to Web applications have been extended by Liu Chien-Hung et al [LIU00a] in the paper Structural Testing of Web Applications. Several dataflow issues for analyzing HTML and extensible Markup Language (XML) documents in Web applications were discussed. An approach to supporting dataflow testing for Web applications was presented by the authors. In this approach, the structural test artifacts of a Web application were captured in a Web Application Test Model (WATM) where each component of the Web application was treated as an object.

An Object Relation Diagram was employed to depict objects and their relationships. The elements of an HTML or XML document were considered as structured variables within an object. For the variables of interest, data flow test cases were derived in five levels from corresponding flow graphs based on the scope of data flow interactions within an object, across objects, or across clients. An example was used to demonstrate each level of the testing approach based on the types of def-use chain for the variables of interest.

Statistical testing and reliability analysis can be used effectively to assure quality for Web applications. This was stated by Kallepalli Chaitanya and Tian Jeff [KAL01] in their paper Measuring and Modeling Usage and Reliability for Statistical Web Testing. To support this strategy, Web usage and failure information from existing Web logs was extracted. The usage information was used to build models for statistical Web testing.

The related failure information was used to measure the reliability of Web applications and the potential effectiveness of statistical Web testing. This approach was applied to analyze some actual Web logs. The results demonstrated the viability and effectiveness of this approach. Various tools and utility programs previously developed by the authors for general test management and reliability analysis were also used to support the reliability analysis for Web applications. This combination provided a comprehensive yet inexpensive alternative to the automated support for this approach.

A proposal in the field of structural testing of Web applications was suggested by Ricca and Tonella [RIC01], who proposed a first approach for white box testing of primarily static Web applications. This approach was based on a test model named the navigational model that focuses on HTML pages and navigational links of the application. Later, the same authors presented an additional lower layer model, the control flow model, representing the internal structure of Web pages in terms of the execution flow followed [RIC04]. This latter model has also been used to carry out structural testing.

In the navigational model two types of HTML pages are represented: static pages, whose content is immutable, and dynamic pages, whose content is established at run time by server computation, on the basis of user input and server status. Server programs (such as scripts or other executable objects) running on the server side of the application, and other page components that are relevant for navigational purposes, such as forms and frames, are also part of the model. Hyperlinks between HTML pages and various types of link between pages and other model components are included in this code representation.

As for the control flow model, it takes into account the heterogeneous nature of statements written in different coding languages, and the different mechanisms used to transfer control between statements in a Web application. It is represented by a directed graph whose nodes correspond to statements that are executed either by the Web server or by the Internet browser on the client side, and whose edges represent control transfer.

Different types of nodes are shown in this model, according to the programming language of the respective statements. A test case for a Web application is defined as a sequence of pages to be visited, plus the input values to be provided to pages containing forms. Various coverage criteria applicable to both models have been proposed to design test cases: they include path coverage (requiring that all paths in the Web application model are traversed in some test case), branch coverage (requiring that all branches in the model are traversed in some test case), and node coverage (requiring that all nodes in the model are traversed in some test case).

McGough Jeff et al [MCG01] described one solution to the problem of presenting students with dynamically generated browser based exams with significant engineering mathematics content in the paper A Web-Based Testing System with Dynamic Question Generation. This proof of concept was called Web Testing Markup Language (WTML), an extension of HTML.

The web server was extended to process the WTML and support features necessary for mathematics exams. The result was then supplied to the client (student's browser). The student completed an exam and returned it for grading and database recording. The system was used in a calculus class and then assessed. This was a complete success in terms of proof of concept. A web based testing system was constructed, which supported the needs of mathematics intensive courses.

Scholtz Jean [SCH01] in his paper Adaptation of Traditional Usability Testing Methods for Remote Testing discussed the approach to design the usability testing tools. Several tools available for public use such as WebCAT, WebVIP and VISVIP were also discussed in the paper. Remote testing could be used to assess a current site or it could be used when a new site or a new version of a site was being developed.

A link from the current site to a remote evaluation test might be setup to solicit input from current users. This was valuable in two ways. First, it provided information about the usability of the new site. Second, it helped prepare users for a change. It was difficult to drastically change a heavily used web site as users were often frustrated when navigation paths and shortcuts they finally figured out were changed. For web sites that had a global user population, remote assessment and analysis methods were necessary to continually evaluate global usability.

Some open issues in the field of web application testing were discussed by Lucca Giuseppe Antonio Di et al [LUC02] in the paper Testing Web Applications. These issues included the problem of defining a test model, for representing the components of the application to be tested and their inter-connections; a test strategy to create test cases from the test model; various testing levels, specifying the different scopes of the tests to be run; and a testing tool supporting the test engineer during test case design and execution.

An object-oriented model of a web application was exploited as a test model, and a definition of the unit level for testing the web application was proposed. Based on this model, a method to test the single units of a web application and for the integration testing was proposed. In order to experiment the proposed technique and strategy, an integrated platform of tools comprising a web application analyzer, a repository, a test case generator and a test case executor, had been developed. The proposed testing approach and the supporting tool had been submitted to a preliminary experimentation, carried out for validating them. The obtained results showed the feasibility of the testing approach, and the effectiveness of the tool in automating many activities required by the testing process.

Kirchner Michele [KIR02] dealt with the concept of accessibility of Web sites to people with disabilities. Major problems arose when there was the need of special browsers, which could understand the standard HTML, but not the extensions introduced by the most used browsers. The WAI (Web Accessibility Initiative) had pursued some guidelines in order to build accessible sites and the new W3C standards were converging towards those guidelines. It was shown that several tools existed that were able to check the Web pages for the accessibility for impaired people, and some tools were able to repair non accessible pages.

A strategy was proposed by integrating existing testing techniques and reliability analyses in a hierarchical framework by Tian Jeff et al [TIA03]. This strategy combined various usage models for statistical testing to perform high level testing and to guide selective testing of critical and frequently used subparts or components using traditional coverage-based structural testing. Reliability analysis and risk identification were an integral part of this strategy to help assure and improve the overall reliability for web-based applications. Some preliminary results were included to demonstrate the general viability and effectiveness of the approach.

This three-tiered hierarchical approach offered an integrated framework that provided both a potentially effective testing strategy and a vehicle for reliability assessment and improvement. The user focus of web-based applications were supported in this strategy by testing functions, usage scenarios, and navigation patterns important to and frequently used by end users under our top-tier usage model based on the list-like operational profiles as well as our middle-tier usage models based on Markov chains. Internal components and structures for web based applications could also be thoroughly exercised by using these bottom-tier models based on traditional coverage based testing, under the guidance of the upper-level usage models.

An application of agent technology in the testing of web-based applications was presented by Huo Qingning et al [HUO03] in A Multi-Agent Software Environment for Testing Web-based Applications. A prototype was described and discussed whose multi-agent architecture and the infrastructure of the system satisfied the requirements of testing web-based applications. The dynamic nature of web information systems demanded constant monitoring the changes of the system and its environment. Sometimes, the change in the system and its environment might require changes in testing strategy and method accordingly.

Agents were adaptive, and they could adjust their behaviours based on environment changes. These changes could be integrated to the system lively. The diversity of platforms and the formats of media and data of web information systems demand using a wide variety of test methods and tools. Multi-agent systems could provide a promising solution to this problem. Different agents were built to handle different types of information, to implement different testing methods and to integrate different tools. Thus, each individual agent could be relatively simple while the whole system was powerful and extendible.

The distribution of large volume of information and functionality over a large geographic area required testing tasks carried out at different geographic locations and to transfer information between computer systems. The agents could be distributed among many computers to reduce the communication traffic. Although a single agent could only perform as a normal program, the strength of agents came from the intelligent dynamic task assignment, the dynamic control of agent pool, the dynamic interactions between agents, the live update of agent systems.

Xu Lei et al [XU03] put the slicing method into practice to simplify the problems in regression testing of web applications, i.e., only analysed the related contents and did not consider the irrelated ones, in order to carry through the regression testing quickly and effectively. So focus was on the simplified contents and to improve the working efficiency. In order to make sure of what to test, the possible changes and their influences in Web applications were analysed, and the dependent relationships were divided into direct and indirect.

On the base of the dependent relationships, it was emphasized on how to test the indirect-dependent among data, and the regression testing method for Web applications based on slicing was introduced, which obtained the dependant sets of Web changes based on the Forward and Backward Search Method and generated the testing suits. The quality and efficiency of the regression testing can be greatly improved since the testing suits were tightly related to the changes and there was no lack or redundancies in the testing suits.

Some useful statistic data in the log file on server side depending on the users' responses and attitudes was used by Lei Xu et al [XU03a] in Website Evolution Based on Statistic Data. Then, based on the user-visiting information, maintenance and evolution was carried out more efficiently. The authors firstly proposed a simplified Website model, which was a weighted structure model; then gained the critical pages, predominant pages and user-visiting patterns based on this model.

These concepts were then discussed for using them in the evolution for Websites in detail. A usability improving method which aims at critical pages was proposed, then interaction function testing method which focused on predominant pages was introduced; and finally the group testing method which takes place after dividing the module based on user-visiting pattern was produced. The authors focused on how to obtain a valid and evolutive Website in the maintenance phase, and gave some practicable testing methods. Since this work was performed in the maintenance phase, it considered recursive testing also, which was different from the testing occurred in before phases.

Arnold Tom [ARN03] in the paper Testing Web Services (.NET and Otherwise) presented an overview about a web service, gave an example of how web services were being put to use, and determined how to approach testing a web service manually through a web page and with automation using different programming languages.

In cases where the source code wasn't available to the test engineer, working with a web service was very much a black box approach to testing, even though using a programming language to test a web service's exposed interfaces into its functionality seems to indicate otherwise. It all boiled down to sending and receiving HTTP traffic, and the data that was sent depended on what format the web service expected. The required format could be as simple as a query string sent from a browser's Address line, or, an XML document wrapped in a SOAP envelope, all embedded within an HTTP header. The return values also depended on how the web service was written, anything could be returned: XML, HTML, or a single byte of information indicating the response.

Building a web page to provide a user interface for testing a web service was a good way to verify the service is behaving as expected. However, getting into the guts of testing a web service through the use of a programming language like Perl, or an automation tool like Application Center Test, that provided specific support for sending and receiving HTTP data, these programming languages allowed a test engineer to take more of a data-driven approach to testing. Perl, Java, Python, C, and even VB Script can all be used for testing. What the test engineer needed to focus on was what the web service expected as input and what it defined as its output.

The quality issue of the web systems was investigated and a systematic framework for web testing was proposed by Chang Wen-Kui et al [CHA03], in which a web application model was established to denote all components of a website by the Markov usage chain. To demonstrate its usability, a practical example of link validity testing was presented. The suggested validation mechanism not only generated automatically a complete set of the test scripts for link validation, but also provided quantitative analysis of the invalid components in a web application.

In comparison with other existing tools, the proposed systematic framework was shown to be effective for quickly certifying link validity in websites applications. It had many benefits such as: helping in forming a testing plan and allocating testing resources, generating test scripts systematically, and reaching maximal testing coverage. Hence, the proposed testing mechanism was more practical and efficient than those web-testing technologies.

An approach to complement Web Application (WA) testing strategies, such as black box, white box or even strategies exploiting session data, with a state-based testing to verify the correct interaction of the WA with a web browser was proposed by Lucca Giuseppe A. Di et al [LUC03] in the paper Considering Browser Interaction in Web Application Testing. The aim was to discover failures due to particular sequences of navigations performed using the web browser buttons.

A testing model, considering the role of the browser while navigating a web application, and some coverage criteria, were presented in the paper. The test case generation could be automated once baseline test cases, produced with other traditional approaches, were available. The test execution could also be automatically performed developing a tool that simulated the actions of a web browser, with the ability of going forward, backward and reloading a page. Finally, heuristics could be adopted to automatically reveal some inconsistencies caused by following particular sequences of actions to navigate among pages.

Elbaum Sebastian et al [ELB03] in the paper Improving Web Application Testing with User Session Data explored the notion that user session data gathered as users operate web applications can be successfully employed in the testing of those applications, particularly as those applications evolved and experienced different usage profiles.

Results of an experiment comparing new and existing test generation techniques for web applications were reported, assessing both the adequacy of the generated tests and their ability to detect faults on a point-of-sale web application. Results showed that user session data could produce test suites as effective overall as those produced by existing white-box techniques, but at less expense. Moreover, the classes of faults detected differ somewhat across approaches, suggesting that the techniques might be complimentary.

Sampath et al [SAM04] explored the possibility of using concept analysis to achieve scalability in user-session based testing of Web applications. Concept analysis was a technique for clustering objects that have common discrete attributes. It was used in to reduce a set of user sessions to a minimal test suite, which still represented actual executed user behaviour. In particular, a user session was considered as a sequence of URLs requested by the user, and represented a separate use case offered by the application. Starting from an original test suite including a number of user sessions, this test suite was reduced by finding the smallest set of user sessions that covers all the URLs of the original test suite.

At the same time, it represented the common URL of the different use cases represented by the original test suite. This technique enabled an incremental approach that updating the test suite on-the-fly, by incrementally analysing additional user sessions. The experiments carried out showed the actual test suite reduction is achievable by the approach, while preserving the coverage obtained by the original user sessions' suite, and with a minimal loss of fault detection. A framework was thus developed that automated the entire testing process, from gathering user sessions through the identification of a reduced test suite to the reuse of that test suite for coverage analysis and fault detection.

Zhu Hong [ZHU04] discussed why testing web applications and quality assurance was difficult and proposed a cooperative agent approach to overcome the problems. The application of Lehman's theory of software evolution to web-based applications showed that they are by nature evolutionary and, hence, satisfied Lehman's laws of evolution. The essence of web applications implied that supporting their sustainable long term evolution should play the central role in developing quality assurance and testing techniques and tools. Therefore, two basic requirements of such a software environment were identified.

First, the environment should facilitate flexible integrations of tools for developing, maintaining and testing various kinds of software arte-facts in a variety of formats over a long period of evolution. Second, it should enable effective communications between human beings and the environment so that the knowledge about the system and its evolution process could be recorded, retrieved and effectively used for future modification of the system.

The solution to meet these requirements was a cooperative multi-agent software growth environment. In the environment, various tools were implemented as cooperative agents interacting with each other and with human users at a high level of abstraction using ontology. A prototype system for testing web applications designed and implemented whose preliminary experiments with the prototype showed some promising results.

A framework for Web application testing was established by Xu Lei et al [XU04]. Based on the analysis for Web application characters and traditional software testing process, the process for Web application testing was modeled, which described a series of testing flows such as the testing requirement analysis, test cases generation and selection, testing execution, and testing results analysis and measurement. The realization techniques were also investigated so as to integrate each testing step and implemented the whole testing process harmoniously and effectively. Thus the framework considered the specialties of the Web testing fully and it guided each task during the testing process, which was focused on the unique characters of Web applications.

An agent-based framework based on the Belief-Desire-Intention (BDI) model of rational agents and the Unified Modeling Language (UML) for testing of web applications was proposed by David Kung [KUN04]. It was described how Web applications testing could be modeled and reasoned using the framework. An agent-oriented framework for testing Web applications was sketched. The framework was an application of a general agent-oriented software engineering methodology.

The general agent-oriented software engineering framework had been applied to modeling and design of an intelligent elevator system and a garden care system that consisted of several garden care agents and the easiest action plan using the classical AI approach (theorem-proving approach) was successfully generated.

An experimental framework for testing a huge variety of web architecture related aspects in order to provide flexibility and accuracy was proposed by Domenech Josep et al [DOM04]. This framework was a hybrid implementation that combined both real and simulated parts in order to provide flexibility and accuracy. It reproduced in detail the behavior of web users, proxy servers and original servers. The simulator also included a module to provide performance results such as precision (prefetching accuracy), recall & response time and byte transference. It was quantified how the prefetch requests could interfere in the performance of the user requests.

Self-interference and the cross-interference could be quantified since the environment included the entire system. The evaluation of prefetching techniques for comparison has been standardized. The authors also created novel prefetching architectures that interrelate the three parts of the global system. Therefore, prefetching algorithms in any point of the system as well as to check their impact on the overall system performance can be implemented.

The paper Testing a Web Application by Sneed Harry M. [SNE04] described the requirements and constraints of testing a web application system for the Austrian Chamber of Commerce. It separated the testing task into two parts - one for testing the web architecture and one for testing the internet application itself. The high costs of testing an internet application were driven not only by the comlexity of the architecture, but also by the complex usage profiles of the application.

Non-functional quality characteristics such as usability, performance, security and interoperability took on particular importance in web-based systems. The evolution of such systems required that the evolved application be tested against the old one. A comparison tool was developed to validate both the backend data base contents and the frontend web page contents.

Sneed Harry M. [SNE05] in the paper Testing an E-Government Web Site dealt with the problem of testing web sites where the code was not available. The website in question was an E-Government website of the state of Saxony in eastern Germany. The experience of the author in extracting test cases from natural language specifications, generating test data for relational databases from the database schemas and generating XML/WSDL requests from the interface descriptions was described.

The test included testing the web pages in dialog mode as well as testing the web services in batch mode. In the end more than 450 errors were reported of which some 52 were critical. Most of the errors were related to misinterpretations of the specification. There were also some errors in the accessing of the database and many in the processing of the web service requests. The test was ended in May 2005 and the system turned over.

Xu Wuzhi et al [XU05] in their paper Testing Web Services by XML Perturbation had introduced a novel test method for XML based communication. A formal model for XML schema was defined and a method to create virtual schema when XML schemas did not exist was presented. Based on this formal model, perturbation operators were designed. The operators were used to modify virtual or real schema to satisfy defined test coverage criteria. Test cases, as XML messages, were generated to satisfy the perturbed XML schema. The method was applied as a case study to two web services. As case studies, there was no expectation of external validity of the results.

Mei Hong et al [MEI05] focused on facilitating the testing of Web services and proposed a framework for testing Web services, which can help a tester of Web services in two ways: Firstly, it could help the tester to acquire effective test data. Secondly, it could help the tester to execute the test data for the testing. In the framework, Hong mainly focused on test data acquisition, test data selection, and test data execution. For test acquisition, approaches suggested were:

1) generating test data using contract information of the service, and 2) obtaining test data via recording runtime information in real execution. For test data selection, the use of contract-based mutation was exploited that was effective in killing mutants. For test data execution, a general-purpose test driver to execute test data was used. Experimental study to evaluate the test data generation and selection strategy, showed the effectiveness of the strategy.

The black box approach for Web application testing considered by Andrews et al [AND05] in the paper Testing Web Applications by Modeling with FSMs exploited Finite State Machines (FSMs) for modelling software behaviour and deriving test cases from them. The state-dependent behaviour of Web applications was taken into account, and specific solutions for addressing the problem of state explosion were proposed. The process for test generation comprised two phases. In the first phase, the Web application was modelled by a hierarchical collection of FSMs, where the bottom-level FSMs were formed by Web pages and parts of Web pages, while a top-level FSM represented the whole application.

In the second phase, test cases were generated from this representation. The model of the Web application was obtained as follows. First, the application was partitioned into clusters that were collections of Web pages and software modules that implement a logical function. Second, Web pages that included more than one HTML form, each of which was connected to a different back-end software module, were modelled as multiple Logical Web Pages (LWP), in order to facilitate testing of these modules.

Third, an FSM was derived for each cluster, starting from bottom-level clusters containing only modules and Web pages (no clusters), and therefore aggregating lower-level FSMs into a higher level FSM. Ultimately, an Application FSM (AFSM) defined an FSM of the entire Web application. In each FSM, nodes would represent clusters and edges would represent valid navigation among clusters. Annotated FSMs and aggregate FSMs were thus used to generate tests.

Tests were considered as sequences of transitions in FSM and the associated constraints. Test sequences for lower-level FSMs were combined to form the test sequences for the aggregate FSMs. Standard graph coverage criteria, such as all nodes and all edges, were used to generate sequences of transitions for clusters and to aggregate FSMs. The approach mainly addresses integration and system testing.

A preliminary fault model for the Web applications was described by Ricca Filippo and Tonella Paolo [RIC05a] in the paper Web Testing: A Roadmap for the Empirical Research. This model was obtained by conducting a series of observations of faults reported for existing Web applications. The available Web testing techniques had been classified into three main groups: Functional, Code coverage and Model-based techniques.

Each of them was assessed in terms of the faults directly addressed by it. Preliminary results indicated that most faults observed on real Web applications are addressed only indirectly by the available testing techniques. This clearly demanded for novel techniques which more aggressively attack those fault types. When compared with the alternative techniques, their complementary nature emerged suggesting that one single technique was usually not enough and a combination of techniques was expected to reveal faults which belong to different fault categories.

Siblini Reda and Mansour Nashat [SIB05] in the paper Testing Web Services proposed a technique for testing Web Services using mutation analysis. The technique was based on mutation operators to the Web Service Description Language (WSDL) document in order to generate mutated Web Service interfaces that would be used to test the Web Service. For the purpose, mutant operators that are specific to WSDL documents were defined. Nine mutant operators that could be applied to the main WSDL elements were identified. These operators were useful for revealing WSDL interface errors as well as some logical errors in the Web Service itself.

In an attempt to understand the characteristics of document data and searchers' information needs, and their relationship to search methods to help improve the effectiveness of navigational retrieval systems, the test collection in relation to the evaluation results of the search systems was investigated, in particular, the relationships among the search methods, topic types, and the styles of relevant Web pages by Oyama K. et al [OYA05] in the paper Analysis of Topics and Relevant Documents for Navigational Retrieval on the Web.

The run results were evaluated using Discounted Cumulative Gain (DCG) and Mean Reciprocal Rank (MRR) as metrics. Of the three groups of systems, the one that utilized anchor text information and link information performed best, the group utilizing link information but anchor text information performed second, and the third group using only content information performed worst.

Relationships between difficulty of topics and several attributes of topics and relevant documents were then discussed. It was suggested that topics with search terms that were likely to specify the representative Web pages and with a relevant document of the actual Web pages with many in-links and without frame structure were the easiest to search.

The quality is more important in Web applications. Web applications are crucial vehicles for commerce, information exchange, and a host of social and educational activities. Since a bug in a Web application could interrupt an entire business and cost millions of dollars, there was a need of better methodologies, techniques and tools for developing, maintaining and testing Web applications.

So, Ricca F and Tonella P [RIC05] in their paper Anomaly detection in Web applications: A review of already conducted case studies aimed to investigate, define and apply a variety of analysis and testing techniques able to support the quality of Web applications. Validity of the solutions was assessed by extensive empirical work which showed that only 40% of the randomly selected Web applications exhibit no anomalies/failures.

This datum showed that the common idea of researchers and practitioners was correct, i.e., process, tools and in general methodologies for Web development and testing should be improved. This paper reported six interesting case studies, conducted during this long period of time, to give an idea of the anomalies and failures found in real-world applications. Considering the relation between type of anomalies and analyses applied, the most important of them seemed to be the structural view. It is the analysis more frequently used and more useful in practice.

A technical approach based on existing WWW browsers which offered a clear solution to most of the technical needs for assuring Web Site quality was presented by Miller Edward [MIL] in the paper Web Site Testing. All of these needs and requirements imposed constraints on the test automation tools used to confirm the quality and reliability of a Web Site. A real opportunity to amplify human tester/analyst capabilities had also been presented, which resulted in more reliable Web Sites.

Looker Nik et al [LOO] in the paper Testing Web Services described the current state of the art in testing techniques that could be applied to web services. The challenges encountered, when testing web services in a distributed environment, were reviewed. Test methods and tools that could be applied to the problem were detailed and finally, a promising line of research that can be applied to this problem was outlined.

Bogardi MA et al [BOG06] in the paper Performance Factors in ASP.NET Web Applications with Limited Queue Models discussed the effects of two configurable settings of the ASP.NET application server i.e. the limit of the global queue and the limit of the application queue. The response time of a test web application were measured, while changing those settings. The results were analyzed in a qualitative manner which was followed by using statistical methods: independence tests to investigate which factors influence principally the performance.

The experiments have shown that the global queue limit and the application queue limit were performance factors. Finally, optimal settings according to the performance-related requirements were determined as a function of client workload and the settings of the thread pool attributes. Results showed that the requestQueueLimit and appRequestQueueLimit parameters had a considerable effect on the performance and they were performance factors.

A tool called WSDLTest for supporting Web service testing was reported on in the paper WSDLTest - A Tool for Testing Web Services by Sneed Harry M and Huang Shihong [SNE06]. The tool generated Web service requests from the WSDL schemas and adjusted them in accordance with the pre-condition assertions written by the tester. It dispatched the requests and captured the responses. After testing it then verifying the response contents against the post-condition assertions composed by the tester. The tool had been deployed in an eGovernment project to expedite the testing of Web services offered by a German State Government.

The problems in testing Web Services (WS) applications were analysed by Zhu Hong [ZHU06] in A Framework for Service-Oriented Testing of Web Services and a service oriented framework for testing WS applications was proposed. It enabled collaborations between various parties involved in the development of WS applications via service request and service providing. It also enabled special testing services to be provided as WS to perform testing tasks on behalf of their customers.

The utilization of software testing ontology STOWS was discussed to register testing services with semantics about the capability of the services, to request testing service with semantic specification of the testing task, and to query and retrieve testing services to match the capability of services to the requested test tasks. This approach to testing WS applications not only imposed technical challenges, but also social challenges for the approach to be adopted by IT industry and software testing tool vendors.

A serial of models for of Model-Driven Testing (MDT) for Web application was defined and a framework supporting them was presented in the paper A Framework of Model-Driven Web Application Testing by Nuo Li et al [NUO06]. The framework supported building tested WA model and test case model by a semi-automatic procedure, enabled testers to design test deployment model and test control model, and implemented distributed test execution. It was designed as Eclipse plug-in, and separated into a tester and a modeler. The tester and the modeler were independent. The modeler not only supported to build the test models defined in this paper, but also specified any other models extended from UML2.

Miao Huaikou et al [MIA07] in their paper A Formal Open Framework Based on Agent for Testing Web Applications employed and extended the Gaia methodology to design a formal open framework based on agent for automatically testing Web applications. In the framework, each test task was corresponding to a role and the agent took the role to achieve its test task or cooperated via interaction with other agent to finish the test tasks. The agent could not only join or leave agent society at will, but also took or released roles at run time dynamically. The internal design of multi-agent system (MAS) was independent of the Web applications. The framework helped to implement automatically testing of Web applications.

Torkey F A et al [TOR07] in the paper A New Methodology for Web Testing contributed a practical and cost-effective testing methodology for web applications. Because web site could not test as a whole in one attempt it was relied in this implementation to decompose the behavior of the web site into testable components then mapping these components onto testable objects. The web application components decomposed into many components like images, links, text, etc.

Then the process of testing became first testing each component of the applications and record the result of testing. Then by using Jmeter, it was proved that performance testing tool that broken components on a web site has a bad effect in its performance. This methodology was cost-effective, maintainable, and user friendly because it improved the total quality for performance testing of web applications.

An approach to build a practical abstract-based structural framework to support testing for composed web services was proposed in the paper An Abstract Workflow-Based Framework for Testing Composed Web Services by Karam Marcel et al [KAR07]. The construction purpose and merits of the framework have been showed.

The framework supported the representation of composed web services and their complex relationships; in particular those that were related to the semantic, control, and data that interact in a composed web service. The framework captured these relationships at a granularity level that allowed many structural based testing techniques to be applied. A promising line of testing criteria that could be applied to the framework was also outlined.

Software developers need automated techniques to maintain the correctness of complex, evolving Web applications. While there has been success in automating some of the testing process for this domain, there exists little automated support for verifying that the executed test cases produce expected results. Sprenkle Sara et al [SPR07] through their paper Automated Oracle Comparators for Testing Web Applications assisted in this tedious task by presenting a suite of automated oracle comparators for testing Web applications.

To effectively identify failures, each comparator was specialized to particular characteristics of the possibly non deterministic Web applications' output in the form of HTML responses. The authors also described combinations of comparators designed to achieve both high precision and recall in failure detection and a tool for helping testers to analyze the output of multiple oracles in detail. Results from an evaluation of the effectiveness and costs of the oracle comparators were presented.

It was found that the best comparator depends on the application's behavior and the fault. In general, unioning Content-CollapsedWS-Dates and TagNames+ImptAttrs was the most effective comparator. The authors also provided recommendations to testers on applying effective oracle comparators based on their application's characteristics.

The testing of Web application has been one of the most effective methods to ensure the security and reliability of Web application. Chen Shengbo et al [CHE07] in their paper Automatic Generating Test Cases for Testing Web Applications combined Logic Components (LCs) with component interaction and an agent to assist to automatically generate test cases for testing Web applications.

From Pages-Flow-Diagram (PFD) of Web application under test, this research avoided the state space explosion problem by successively partitioning Web applications into LCs, each of which was composed of Web pages and other LCs. The authors used an automaton to model each LC, and used compositions of automata to model interaction of LCs. This approach supported concurrent access and interaction between LCs. It also enhanced the reuse of component interactions and overcomes differences in actions between LCs by using an agent as a coordinator.

Graphical User Interface (GUI) APplications (GAPs) are ubiquitous and provide various services. Since many GAPs are not designed to exchange information (i.e., interoperate), companies replace legacy GAPs with web services, that are designed to interoperate over the Internet. However, it is laborious and inefficient to create unit test cases to test the web services. Conroy Kevin M et al [CON07] in the paper Automatic Test Generation from GUI Applications for Testing Web Services proposed a novel approach for generating tests for web services from legacy GAPs.

This approach combined accessibility technologies for accessing and controlling GAPs in a uniform way with a visualization mechanism that enables nonprogrammers to generate unit test cases for web services by performing drag-and-drop operations on GUI elements of legacy GAPs. A tool based on the approach was designed, and this tool was used to extract test cases from reference GAPs. Experiments suggested that the approach is efficient and effective. Choosing Smart to extract data for generating tests from reference GAPs saved a lot of time that would otherwise be spent to write programs that perform data extractions.

Murlewski Jan et al [MUR07] in the paper Comprehensive Approach to Web Applications Testing and Performance Analysis presented a comprehensive overview of testing software and analyzing performance of applications with JUnit framework. By introducing test into software development cycle, the quality and reliability of the software could be enhanced. Moreover such an approach might have an influence on reducing the cost of system development. Well-written set of test allowed the programmers to stress less modifications of existing code. It was because after modifications programmer was able to run all tests and check if last changes didn't cause any errors in the application.

A practical and cost-effective testing methodology for web applications has been contributed in the paper Ensuring the Quality Testing of Web Using a New Methodology by Keshk Arabi and Ibrahim Amal [KES07]. The web application components were decomposed into many components like images, links, text, etc. Then the processes of testing became first testing each component of the applications and record the result of testing. This methodology was cost effective, maintainable, and user friendly methods because it improved the total quality for performance testing of web applications.

Tauhida Parveen et al [TAU07] in the paper On the Need for Teaching Web Application Testing argued that there was a need for an additional course focused exclusively on testing Web applications. Such applications represented a sufficiently unique and increasingly important area that new theories and practical experience would be beneficial in maintaining such an application. A case study based on personal experience in an industrial context was used to illustrate the need for teaching Web application testing in a software engineering degree program.

A model-driven ontology-based approach for automatic testing on Web Services (WS) with the purpose to enhance test formalism and intelligence was proposed by Wang Yongbo et al [WAN07] in Ontology-Based Test Case Generation for Testing Web Services. Petri-Net was introduced to model the operation semantics of the composite WS specified by OWL-S. Ontologies were built to enable intelligent test generation by reasoning. The method, techniques and prototype systems of the proposed approach was presented in the paper.

Web Service is widely used nowadays, and whether implementation of Web Service-based System composed of individual services, especially interaction among individual services, meets requirements is an important research area in testing Web Service-based System. Yu Ying et al [YU07] in the paper OWL-S Based Interaction Testing of Web Service-based System studied interaction testing of Web Service-based System and tested implementation based on interaction requirements.

OWL-S model was extended to characterize requirements and the method of requirements description based on ontology gave concepts in requirements explicit semantics. The mapping mechanism from requirements property specification to implementation property specification was established, code instrumentation could be automatically accomplished to collect and record runtime data and testing analysis code could be automatically generated to analyze recorded runtime data according to implementation property specification. The aim of testing whether implementation satisfied requirement property was obtained by testing whether implementation satisfies implementation property.

2.2 Objectives of Research

After going through the research papers available of the subject, it was observed that though a lot of work has been done in the field of web applications testing, yet there seems to be some problems that have not been thrown more light upon. The testers are not fully aware as what quality attributes should be there in case of web based applications that should be tested in order to get a quality web application. Once it is decided what quality attributes to test then the testers can improve the quality of the web applications.

It was also observed that usability and maintainability are more important quality attributes then others. There is no reason to assume that users of web applications will be computer literate, so web applications should be easy to use. Maintenance is difficult in web applications due to the limited time given to designers to modify it & upload it. The researcher understands that development of web applications also requires a disciplined approach & to implement discipline, measurement of these attributes should be available.

So, further research is carried out to identify the metrics for these two quality attributes. As the dependency on web sites is increasing at an exponential rate, the numbers of hits to a particular site at any instance are unimaginable, so load testing is a force to reckon with for mission-critical web sites. Therefore the factors affecting the load levels and the issues related in testing the load level of the web application should also be identified. In view of the observations cited, the following objectives were set:

  • Identification of the attributes affecting the quality of web-based applications.
  • Designing of metric for quantitative evaluation of the attributes identified.
  • Web applications are characterized by two quite important attributes, usability and maintainability, so a specific focus is required on these two issues.
  • Identification of various issues related in designing realistic and accurate web site load tests.

Chapter 3 Quality Attributes of Web Based Applications

3.1 Introduction

Programming languages were invented to make machines easier to use. They thrive because they make problems easier to solve. The primary purpose of programming languages is to make human-machine communication easy instead of just possible. Programming languages are designed to be both high-level independent of the underlying machines & general level purpose that can be applied to a wide range of problems.

The earliest programming language was machine language, which was an ideal formalism from the computer's point of view, but a very awkward one for humans. Assembly language, which came next, was a big improvement since programmers could at least understand what they were saying; but they still had to describe their algorithms by specifying each internal step the computer would carry out. However, programming in assembly language was time consuming, expensive & machine dependent.

Early seventies observed the advent of structured programming paradigm that necessitate the similarity of dynamic & static structure of the program implemented by single entry & single exit constructs to facilitate the engineering approach to software development & maintainability. Procedure oriented software development using structured approach is monitored & controlled with the identification of a number of quality attributes. But very soon it was realized that structured approach is not able to cope with the problems of software crisis & world observed the commencement of object-oriented paradigm stressing & striking on reusability & extendability attributes of software.

The Object Oriented programming languages support interacting objects. An object is a group of procedures that share a state since data is also a part of a state, data & all procedures or functions that apply to it can be captured in a single object. Other features of Object Oriented programming such as encapsulation, inheritance, polymorphism, dynamic binding & state independent behavior provides visible benefits in software design & implementations.

To realize the development of intelligent software & Expert systems, logic based paradigm was introduced that consists of series of facts, rules of inference & a theorem or query to be proved. The output is true if facts support the query & false otherwise.

To cater the need of different types of applications, a number of other programming languages paradigms were inundated such as problem oriented paradigm for data base applications, event oriented for window programming etc. The 21st century characterized by the age of information is based on Internet technology & the popularity of Internet is growing on accord of WWW technology.

The last decade observed the tremendous growth of web technology. But the dynamic nature of the web has forced the designers of web applications to think seriously to find an engineering approach to the development of web applications. This is the motivating factor to identify the quality attributes of web software.

3.2 Software Quality & Measurement

Quality is something for which no organization and no customer / end user would like to make compromise with. Poor quality software results into number of problems e.g. inconsistent product, unreliability, poor performance, difficulties in debugging and maintaining the product. Developing high quality software is an issue of great importance. The question, which therefore immediately comes to our mind is -what exactly do we mean by quality. A number of definitions exist in literature.

Quality is very hard to define, difficult to measure & at the same time easy to recognize. According to International Standards Organization [ISO86], quality is defined as: "The totality of features & characteristics of a product or service that bear on its ability to satisfy specified or implied needs". Kitchenhan [KIT89] defines the software quality as "fitness for needs". Alan [ALA97] provides the insights about quality as: Quality is not absolute, multidimensional, subject to constraints, about acceptable compromises & quality criteria are not independent but interact with each other causing conflicts.

Software quality also has some inherent problems as compared to other areas of manufacturing such as (1) users are not clear about requirements at the start of the project, (2) Requirements are changing quite frequently, (3) Technology are changing very fast, (4) High expectations of clients as far as adaptability to new environment is concerned, (5) Lack of good quality processes, and (6) No physical existence of the software.

As quality is quite subjective & multi-dimensional, a number of views & models of quality are proposed in the literature. The ISO 9001 has twenty quality clauses, for development of quality assurance systems, such as Quality policy, Management review, Quality system procedure, Contract review, Design control, Document control, Purchasing, Software identification & tracing, Purchase supplied product, Process control, Inspection & testing, Inspection, measuring & testing equipment, Test status, Control of Non-conforming products, Corrective action, Handling, storage & delivery, Quality records, Internal quality audits, Training, Software maintenance service & Statistical techniques. These guidelines define the essential features of the software quality management system & suggest controls & methods that allow the software to meet customer needs.

SEI-CMM model (1987) prescribes a systematic path from an adhoc process to a mature process by moving through five levels i.e. Initial, Repeatable, Defined, Managed & Optimizing.

Garvin's [GAR84] has five views of quality: (1) transcendent view, (2) product based view, (3) user based view, (4) the value based view & (5) manufacturing view. McCall's [MCC77] attempted to synchronize & relate the user's view of quality to developer's view. McCall explains that Software quality has three perspectives (product revision, product transition & product operation) each having specific characteristics such as usability, integrity, efficiency, correctness, reliability, maintainability, testability, flexibility, reusability, portability, & interoperability.

Boehm's [BOE78] quality model has the quality criteria such as portability, reliability, efficiency, human engineering, testability, understandability, modifiability & corresponding metrics are also suggested. ISO9126 (1991) (Figure 3.1) quality model describes quality in terms of functionality, reliability, efficiency, usability, maintainability, & portability.

Figure 3.1 Overview of ISO 9126

Since, quality is very subjective, these must be ways to measure the quality of software. Metric helps the developer to predict & control the software product quality. Goodman [GOO93] defines software metrics as The continuous application of measurement based techniques to the software development processes & products to supply meaningful information together with use of those techniques to improve software development process & its products.

Fenton [FEN87] defines software metrics as quantitative measures, which can be derived from any attribute of software life cycle in a mechanical & algorithmic fashion. Metrics can be used to (1) Predict success and failure quantitatively of a product, person or process, (2) Control quality of the software, (3) Make important and meaningful decisions, (4) improve code quality and maintainability, (5) Predict quantified estimates, and (6) Improve scheduling and productivity.

Measurement is the process of assigning numbers to attributes of entities according to some defined rules. For software, the entities are products, processes & resources. An attribute is the property of an entity e.g. its length. Measurement can be direct or indirect. Direct measures are numbers that can be derived from the entity without using any other information, while indirect measures are computed by combining two or more direct measures.

Software quality metrics can be classified into three categories viz. Product, Process & Project. In procedure-oriented software, we have a number of metrics available being an old paradigm such as: LOC - a Size-Oriented metric measuring the length of program, Halstead's software science metric [HAL77] to measure the complexity of the software, McCabe's Cyclomatic Complexity Metric [MCC76] to measure complexity on the basis of control flow, Function Point Analysis [ALB79] to measure the cost & efforts required to develop the software, & Information Flow metrics [HEN81] to measure the level of information flow among different modules.

Some of the popular object oriented software metrics [CHI94] are Depth of Inheritance Tree (DIT), Number of children (NOC), Number of methods per Class (NOM), Coupling between Object Classes (CBO), Response for a class (RFC) & Lack of cohesion in Methods (LCOM). DIT & NOM primarily evaluates Efficiency & Reuse but also relates to Understandability & Testability [CHI94] [HUD94] [LOW99] [MCC94] [SHA93], NOC primarily evaluates Efficiency, Reusability & Testability [CHI94] [LOW99] [MCC94] [SHA93], CBO Improves modularity & promotes encapsulation & evaluates Efficiency & Reusability [CHI94] [HUD94] [LEE93] [LOW99] [MCC94] [SHA93], RFC evaluates Understandability, Maintainability & Testability [CHI94] [LOR94] [MCC94] [SHA93], LCOM evaluates Efficiency & Reusability [CHI94] [HUD94] [LOR94] [MCC94] [SHA93], Weighted Methods per Class (WMC) metric measures Understandability, Maintainability & Reusability [CHI94] [LOR94] [MCC94] [SHA93].

3.3 Web Based Software & Quality Attributes

Designers are always striving for identifying the quality attributes of software so that their efforts may yield into good quality product. A lot of research work has been carried out in the field of procedure & object oriented software. In this chapter, the applicability & limitations of those attributes have been identified in web-based applications & some attributes that are specific to web based softwares are suggested.

Portability: Portability is the effort required to transfer a program from one hardware/software system environment to another. No. of sub-factors support Portability such as:

  1. Adaptability-opportunity for adapting the application to different environments without additional efforts
  2. Installability-effort for installing the software
  3. Conformance-conformance of an application to standards relating to portability
  4. Replaceability -opportunity & effort to use an application as a replacement for another application.

In procedure-oriented software, portability can be achieved by modularizing the software so that the dependencies on the environment are isolated in only a few modules that must be modified to port the software to another environment. It is bigger issue in web-based applications, since the execution environment consists of different kinds of computers & operating systems.

A web application must be able to work correctly on a wide variety of system configurations including browsers, operating systems, & hardware systems. All of these are out of control of the developer of the application.


Integrity is that quality factor of software which can be explained as the extent to which access to software or data by unauthorized persons can be controlled. The web applications run on distributed platforms & are more vulnerable to this type of threats. The data entered into the web application must be validated to ensure its correctness. In addition, measures must be taken to insure the data stays correct after it is entered into the application.


Efficiency is defined in terms of resources consumed by the software to perform its functions. Resources are time, storage, transmission channels & peripherals etc. It is the most important quality attribute of a Web application. A web application is termed efficient if the response time to any action is very quick. The next is the page-generation & graphics-generation speed.

Correctness & Reliability

Correctness & Reliability are interrelated quality attributes of software. Correctness is a mathematical property establishing the equivalence between the software & its specification. It is essential that the functionality & information obtained from web application is correct. This can include not only the functionality of buttons & behind the scene instructions, but also calculations & general navigation through a web site. Reliability refers to the factors such as Maturity, which is the frequency of software faults, Fault or error tolerance which is the ability of software to deal with software faults or infringement of its specified interface, & Recoverability which is the capability to recover data affected in case of a failure & is measured by the time & effort needed for it.

Reliability of web sites is a factor that is to be dealt with seriously. The familiar error 'The Page cannot be displayed' is an example of the failure. Correctness is an absolute quality & any deviation from its requirements makes a system incorrect, regardless of how minor or serious the consequences of the deviation are, while the notion of Reliability is relative. If the consequence of software is not serious, the incorrect software may still be reliable. Reliability of web applications means that all triggers execute correctly & completely, & records are kept of completed actions.

Correct link processing; downloading, error recovery, input validation & server & system availability increase the reliability of the Web application. Web sites have vastly increased the customer base for software. Knowledgeable users will tolerate many more mistakes in the software than less knowledgeable users will. If web software is unreliable, web sites that depend on the software will lose customers & the businesses may also lose large sums of money. These factors combine to make reliability of web software crucial, & most importantly, companies can afford to spend resources to ensure high reliability.

Maintainability: Maintainability is a set of attributes that bear on the effort needed to make specified modifications [ISO86]. The factor consists of the sub-factors such as (1) Analysability - the effort needed for the diagnosis of deficiencies or failures & for the identification of parts to be modified, (2) Changeability - effort needed for modification, fault removal or for environmental change, (3) Stability - the tolerance of the application towards unexpected effects of modifications, & (4) Testability - the effort for validating modification. Web-based applications have a higher frequency of new releases. Maintenance updates are installed & immediately made available to customers through the web site. Thus, even small individual changes can be installed immediately. Web sites have maintenance cycles of days or even hours. Since the users do not always upgrade their software, thus software vendors must ensure compatibility of new versions with old versions.


Reusability is analogous to evolvability. It is difficult to achieve a posteriori, one should strive for reusability when software components are developed. One of the promising techniques is the use of object-oriented design, which can unify the qualities of evolvability & reusability. A web application can be easily reused by making the required changes at a short period of time.


Testability of software is the suitability for allowing the developer to follow program execution & for debugging. It depends on modularity & structured ness, since they prove more suitable for systematic testing. A web application must be testable for functionality, usability, navigation, forms, page content, configuration, compatibility, reliability, availability, performance & security.


Usability is a set of attributes that bear on the effort needed for use, & on the individual assessment of such use, by a stated or implied set of users [ISO86]. The quality factor consists of three sub-factors such as (1) Understandability - the users' effort for recognizing the logical concept of an application & the applicability of that logical concept, (2) Learnability - the users' effort for learning the application as for example, operation control, input & output, & finally (3) Operability is the users' effort for operation & operation control.

Usability means ease of use of the application by one & all. Ease of use decides how quickly the user understands the application & its scope at the global level. Ease of use gets considerable boost if the Web application has interfaces to various other applications & links & hyperlinks to other sites to fetch information, process & execute the transaction without logging out from the current site. Usability describes the user's ability to manipulate the web-based application's features in order to accomplish a particular goal. Traditional software applications have users, but web sites have customers.

Most web applications have a broad customer base. These customers have grown to expect software to be as simple to learn as shopping at a store. Usable web-based applications will be efficient, easy to learn & help users accomplish their goals in a satisfactory & error-free manner. A high degree of understandability of the website where the application resides gives a tremendous boost to web application usage. The web site design features should be such that it gives quick understanding of the application through appropriate entry point, provides on-line feed back for every action of the user, simplifies the use by provision of help features, drop-down boxes, automatic insertion of choice of an entity such as country, position etc.

Functionality: Functionality is a set of attributes that bear on the existence of a set of functions & their specified properties [ISO91]. This factor consists of five sub-factors: (1) Suitability - the functionality of the application fits the needs of a user to fulfill a certain task without overwhelming the user, (2) Accuracy - the results or application's behavior is correct, (3) Interoperability - the application is able to interact with the specified systems, & (4) Compliance - the application is compliant with standards, conventions or regulations in laws & similar prescriptions.

The functionalities in Web application refer to Searching, Retrieving, Navigating, Browsing, & Domain-related features. All these functions expedite application processing by way of searching the appropriate site for information, data & so on. Retrieving is a process of fetching the data in the current application for use. Navigating is the process feature that helps the user to traverse at ease through a maze of links & hyperlinks & allows the user to come back to local application that is being executed. Browsing is a capability that makes the application versatile.

The capabilities are speed & responsiveness while in use. These four functionalities become more productive when domain-related features are also provided. The features could be domain-specific formula processing, information access, downloading of forms & references to domain-specific rules & regulations & procedures & practices. These types of features make the application rich in scope & fast in processing & execution.

In addition to the above quality attributes, the web applications have some more attributes, which are; Content-driven, Network intensive, Security, Aesthetics, Recoverability, Availability, & Scalability.

Content Driven

The web applications are Content-driven since they run on the Internet/ intranet/ extranet to meet the needs of diverse groups of users. The characteristics of content are text, graphics, audio & video clippings mixed in such a way as to make it interesting & provoking to the user of the application. Web applications should present only the correct & updated in real-time mode information. The contents are under continuous modification in web applications. They are used for showing snap-shots, information communication & transaction processing. This requires continuous evolution of the content as an attribute.


An application runs on client-server architecture. Application of engineering makes it web-enabled & it runs on the Internet across the globe. Hence, they are, network-intensive & have to follow Web/Internet protocols & standards in addition to the normal standards of application system design & development.


One of the major risks of web applications is security. It is very important to validate the application & data that are protected form outside intrusion or unauthorized access, though, it is not always possible to restrict users & make web application secure. External intrusion from people trying to gain access to sensitive information, intentionally sabotage the information or application is a major security risk. The transactions over the Internet should also be made secure. Security from the virus attacks on web-based applications should also be taken care into. The access controls for the authorized users should be defined & authorization levels should be set.


The Web applications characteristic, important from the user's point of view is aesthetics. A good, attractive design of the application generates appropriate responses from users. The use of colours, presentation, layout, & creation of guiding buttons & domain-specific icons are the main requirements of Web applications. Web application is appealing & is in a great demand if the look, feel & ease of use is high. The continuous evolution of Web Applications makes the work performed on them analogous to gardening [LOW99].


Availability means more than just being up & running; the Web software must also be accessible to diverse browsers. In the seemingly never-ending browser wars of the past few years, some Web software developers actively sought to make sure their software would not work under competitors' browsers.

By using features only available for one browser or on one platform, they become foot soldiers in the browser wars, sometimes unwittingly. Web sites must adapt their presentations to work with all browsers, which require significantly more knowledge & effort on developers' part.


Scalability concerns the web applications' ability to handle the volumes & types of activities that can occur after launch. The organization needs to ensure the delivery of acceptable performance under many situations & that it can support transaction levels after the launch of web application. Scenarios affecting scalability are, (1) how closely the these environment matches the production environment, (2) millions of users accessing the site during launch, (3) plan for capacity by modeling the users expected to access the site, (4) identify the levels & types of activity that can degrade performance, & (5) product performance measures such as requests per second & transactions per second.

The need for scalability has driven many technology innovations of the past few years. The industry has developed new software languages, new design strategies, & new communication & data transfer protocols in large part to allow Web sites to grow as needed. Designing Web applications that scale well represents one of today's most interesting & important software development challenges. So, the web-based applications should be developed that can scale & still remain reliable, secure & available.

From the discussion, it is perceived that two dimensions of quality attributes of web-based applications can be visualized, one from the viewpoint of the user & another from the viewpoint of the designer as illustrated in figure 3.2. The user is concerned with the attributes Portability, Correctness, Functionality, Usability, Recoverability, Availability, Aesthetics & Efficiency, & is not supposed to go in the implementation intricacies of Reliability, Maintainability, Reusability, Testability, Content-driven, Network Intensive, Security & Scalability. Rather these attributes are in the domain of designer.

3.4 Summary

From the above discussion, it can be inferred that the issues, which are relevant in procedure oriented & object oriented Software, are also relevant in web-based applications, although the priorities may be different. It can be concluded that efficiency, usability, maintainability, scalability & reliability are more important then others.

There is no reason to assume that users of web applications will be computer literate, so web applications should be easy to use. Maintenance is difficult in web applications due to the limited time given to designers to modify it & upload it. The need of scalability can never be undermined but reliability & availability should not be compromised.

The researcher understands that development of web applications also requires a disciplined approach & to implement discipline, measurement of these attributes should be available. So, further research is carried out to identify the metrics for these attributes.

Chapter 4 Web Usability: Factors and Measurement

4.1 Introduction

The World Wide Web has had a significant impact on access to the large quantity of information available through the Internet. Web-based applications have influenced several domains, by providing access to information and services to a variety of users with different characteristics and backgrounds.

Users visit Web applications, and return to previously accessed applications, if they can easily find useful information, organized in a way that facilitates access and navigation, and presented according to a well-structured layout. In other words, the acceptability of Web applications by users relies strictly on the applications' usability.

Usability is one relevant factor of a Web application's quality. Recently, it has received great attention, and been recognized as a fundamental property for the success of Web applications. Defining methods for ensuring usability is therefore one of the current goals of web testing research. Also, much attention is currently paid to usability by industry, which is recognizing the importance of adopting usability methods during the development process, to verify the usability of Web applications before and after their deployment. Some studies have demonstrated how the use of such methods reduces costs, with a high cost benefit ratio, as they reduce the need for changes after the application is delivered [MAD99] [NIE93].

Usability is generally taken as a software quality factor that aims to provide the answer to many frustrating problems caused by the interaction between people and technology. It describes the quality of products and systems from the point of view of its users.

Different definitions of usability have been proposed, which vary according to the models they are based on. International standard ISO 9241 (Ergonomic Requirements for Office Work with Visual Display Terminals) provides guidance on usability, introducing requirements and recommendations to be used during application design and evaluation [ISO97].

The standard defines usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. In this definition, effectiveness means the accuracy and completeness with which users achieve specified goals, efficiency is the resources expended in relation to the accuracy and completeness with which users achieve goals, and satisfaction is described as the comfort and acceptability of use. Usability problems therefore refer to aspects that make the application ineffective, inefficient, and difficult to learn and use.

Although the ISO 9241-11 recommendations have become the standard for the usability specialists' community, the usability definition most widely adopted is the one introduced by Nielsen [NIE95]. It provides a detailed model in terms of usability constituents that are suitable to be objectively and empirically verified through different evaluation methods. According to Nielsen's definition, usability refers to:


The ease of learning the functionality and behavior of the system.


The level of attainable productivity, once the user has learned the system.


The ease of remembering the system functionality, so that the casual user can return to the system after a period of non-use, without needing to learn again how to use it.

Few errors

The capability of the system to feature a low error rate, to support users making few errors during the use of the system, and, in case they make errors, to help them recover easily.

Users' satisfaction: the measure in which the user finds the system pleasant to use.

The previous principles can be further specialised and decomposed into finer-grained criteria that can be verified through different evaluation methods. The resulting advantage is that more precise and measurable criteria contribute towards setting an engineering discipline, where usability is not just argued, but systematically approached, evaluated, and improved [NIE92] [NIE95].

When applying usability to Web applications, refinements need to be applied to the general definitions, to capture the specificity of this application class. Main tasks for the Web include finding desired information and services by direct search, or the discovery of others by browsing; comprehending the information presented; invoking and executing services specific to certain Web applications, such as the ordering and downloading of products. Paraphrasing the ISO definition, Web usability can therefore be considered as the ability of Web applications to support such tasks with effectiveness, efficiency, and satisfaction. Also, Nielsen's usability principles mentioned above can be interpreted as follows [NIE95a]:

Web application learnability must be interpreted as the ease for Web users to understand the contents and services made available through the application, and how to look for specific information using the available links for hypertext browsing. Learnability also means that each page in the hypertext front-end should be composed in a way such that its contents are easy to understand and navigational mechanisms are easy to identify.

Web applications efficiency means that any content can be easily reached by users through available links. Also, when users get to a page, they must be able to orient themselves and understand the meaning of this page with respect to the starting point of their navigation.

Memorability implies that, after a period of non-use, users are still able to orient themselves within the hypertext; for example, by means of navigation bars pointing to landmark pages.

Few errors mean that when users erroneously follow a link, they are able to return to their previous location.

Users' satisfaction refers to the situation in which users feel they are in control with respect to the hypertext, since they comprehend the available content and navigational commands.

4.2 Quality Attributes and Web Applications

The World Wide Web is expanding rapidly. As more and more information becomes available, individuals will use the web for many more purposes. Students and scholars use the web for research. Individuals use the web to order books and clothes, make travel arrangements, send flowers, read newspapers, track investments, etc.

Companies use the intranet to track orders and to disseminate information to employees. However, many times web users are frustrated because they are unable to locate the sites or the information they want on a particular site. User-centered design and usability testing of user interfaces is being applied in some traditional software development processes, but is not yet commonplace.

Issues of time, resources, and perceived value are obstacles to be overcome in moving usability into the software development cycle. Moving usability-engineering procedures to the development of web sites and web applications is even more difficult. The normal software development time has been shortened considerably for web site development. Most companies want a web presence within three months [VOR98].

Usability is suited to an iterative design environment. Cycles of design, usability issues, and reiterating the design are difficult to carry out in reduced time scales. Sites and applications developed for the web become "unusable" for external reasons. That is, traditional software is tested when changes are made to the software. Web sites may become unusable because of outside forces.

For example, a new capability becomes available and other sites that use that capability become more desirable than sites lacking the capability. A site that was "state of the art" yesterday may no longer be appealing. There are too many variations of types of users, tasks that can be carried out on a site, and different configurations of machines and browsers to attempt to bring a representative sample to a usability lab for testing.

Although researchers and practitioners have spent years developing processes and technologies to improve and measure software quality attributes, many software companies have had little motivation to improve the quality of their software. Software contractors are usually paid no matter how poor the quality of their delivered software is.

Sometimes they can even get additional resources to correct problems of their own making. Commercial software vendors are driven almost entirely by time-to-market, which means they usually make more money by delivering poor quality products sooner than high quality products later. Bug fixes are often delivered as new releases and can be sold to generate more revenue for the company. For most types of applications, there has traditionally been very little motivation for producing high quality software [OFF02].

However, the situation is quite different for web-based software. A recent survey of web software development managers and practitioners shows that web-based companies depend on customers using their sites, and most importantly, returning to their sites. Users will only return to sites that satisfy their needs. Thus, unlike many other applications web software applications only make money if their web sites satisfy customers' needs. Moreover, the time-to-market becomes much less important.

Unlike many software vendors, if a new company releases competitive site that is of higher quality, customers will almost immediately start using the new site. Thus, instead of sooner but worse, it is often advantageous to be later-than and better. Although the use of sticky web sites has been discussed and ways to encourage users to come back have been developed [DAN00], thus far the primary mechanism that has brought repeat users to web sites has been high quality. It seems unlikely that this will change anytime soon.

4.3 Factors Affecting Usability of Web Applications

In web applications, it is analyzed and observed that usability of the web site is more important quality attributes than others. Great web-based applications are truly useful to their users. Usefulness can be thought of as a combination of utility and usability. Utility describes the Web-based application's functionality and hopefully meets a user's requirements.

Usability describes the user's ability to manipulate the web-based application's features in order to accomplish a particular goal. If a web-based application does not function properly, either because of designer mistakes or poor design choices that makes it difficult to use, it is not a good application. Usable web-based applications will be efficient, easy to learn and help users accomplish their goals in a satisfactory and error-free manner.

These days the websites come in many forms, ranging from simple brochure ware sites to complex web-based applications and the resulting variety and complexity have put a premium on usability as a key element of web-based applications. Like procedure-oriented software, the web-based application should only be considered excellent if it is useful, usable, correct and pleasing. For a well-designed web-based application, the execution must be excellent meaning that the application must not break in any way. The navigation of site must work at all times. There should not be any broken links.

In web-based applications, several usability aspects should be taken into account when setting up a design such as the user's aim and related surfing behavior, the quality objectives of usability testing, the possibility of measuring (quantitative) efficiency e.g. time needed, the possibility of measuring (quantitative) effectiveness e.g. number of correct answers, the focus on certain characteristics of the web-application itself (text and graphics) and its goals (informative, persuasive, divertive or instructive) and the focus on the different components of the web-based applications e.g. homepage, frames and navigation tools, forms, specific content branches, java scripts.

The more a web-based application helps people find the information they are looking for, the more usable it is. Information retrieval is an important goal in website usability, but it is certainly not the only objective in usability testing. A very useful classification of quality objectives is given by Jong & Schellens [JON97], and Jong & Heuvelman [JON99]. They break down the overall concept of effectiveness into a series of conditions for effectiveness and identify the quality objectives, (1) Contact: how is the (information on the) website brought to the potential readers' attention? (2) Selection: how does the target group select the information within the site? (3) Comprehension: does the reader understand the information on the site correctly? (4) Application: is the reader able to apply the information in a realistic setting (especially for instructive texts)? (5) Acceptance: is the presentation of the information acceptable to the readers? (6) Appreciation: are the style, the tone, and the register, the design appreciated by the readers? (7) Relevance and completeness: are the most important topics in the site relevant for the intended reader?

It is impossible to address the full spectrum of quality objectives with a single task or a one-dimensional task setting. Therefore, when developing ideas for thinking-aloud tasks, it is important to strike the right balance between the type of texts and themes that are focused on in the evaluation and the other aspects of web-based application's usability.

So, the factors to be considered for improving and measuring the usability of web sites are:

  • Lightness of text, graphics, and backgrounds (f1)
  • The color of the items to be distinguished (f2)
  • Color of the visited and unvisited links (f3)
  • Availability of load time progress bar if load time is more (f4)
  • Response time (f5)
  • Location of navigation buttons (f6)
  • Appearance and size of navigation buttons (f7)
  • Caption on navigation buttons (f8)
  • Tool tips with buttons (f9)
  • Dissimilarity between the dissimilar pages (f10)
  • Time consumed in mouse movement and keyboard access (f11)

Identifying the quality attributes and factors affecting quality is not sufficient to assure the quality. One must have a quantitative measure available. Usability can be measured as per following metric:

where fi is the ithfactor to be considered for improving and measuring the usability of web site and n is the number of factors.

4.4 Summary

To improve the quality of any product, one must be able to measure the un-measurable. Whenever one is concerned with measurements, immediately three questions are posed - (1) Why to measure? (2) What to measure? and (3) How to measure? The World Wide Web is the biggest library of the world, but unstructured and unorganized due to a common trend of migrating each and every information to the web without considering the quality. And the sufferer is the poor user. The most important quality attribute of a web application is its usability - affected by a number of factors. The metric like Usability Index can be used to measure the usability and help in assuring the quality.

Table 4.1 Weight Assignment to the Usability Factors


Very low




Very high



































































Chapter 5 Web Maintainability: Factors and Metrics

5.1 Introduction

Software maintenance can be defined as set of activities undertaken on a software system following its release for operational use [BEN91]. There is evidence that maintenance costs exceed 60 percent of the total costs of software. There can be more than one reason for maintaining the software and there can be different kind of changes in the software such as simple changes to repair coding problems, more extensive changes to repair design errors, changes to accommodate the new requirements and changes to the architecture of the software.

Software maintenance can be categorized into four types (figure 5.1) viz. Corrective, Adaptive, Perfective and Preventive maintenance. Corrective maintenance deals with the reporting errors such as coding errors, design errors and requirement errors. Adaptive maintenance concerns external changes such as introduction of new versions of operating systems, new hardware or removing support for existing facilities. Perfective maintenance concerns improving the delivered software as a result of change in user requirements or efficiency improvements. Preventive maintenance is done to anticipate future problems and to improve the maintainability.

Extended ISO Model (Figure 5.2) defines Maintainability as set of attributes that bear on the effort needed to make specified modifications. Maintainability has certain characteristics such as Analysability, Changeability, Stability, Testability, Manageability, and Reusability [ZEI96].

Analysability can be defined as attributes of software that bear on the effort needed for diagnosis of deficiencies or causes of failures, or for identification of parts to be modified [ZEI96]. It can be measured by fault position recognition ratio and mean failure analysis time. A higher value is preferred for fault position recognition ratio, while in case of mean failure analysis time, a lower value is preferred.

Changeability is the set of attributes of software that bear on the effort needed for modification, fault removal or for environmental change [ZEI96]. It can be measured by modification effort per unit volume, which is the average amount of effort needed to modify the software product, per unit volume of the modification. Here, we have to measure the (1) effort needed to analyse the modification, (2) the effort needed to modify the design, (3) effort needed to modify the software product, (4) effort needed to test the modification, (5) effort needed to implement the modified software product, (6) extent of the modification and to calculate the modification effort per unit volume, which is the sum of the efforts/extent of modification.

Considerable efforts are required when fictional changes are proposed to test changeability. Another measuring way is correction effort per defect, which is mean effort needed to repair a defect in the software product. For this, we measure (1) the effort needed to find the cause of the defect, (2) the effort needed to correct the design, (3) the effort needed to correct the software product, (4) the effort needed to test the correction, (5) the effort needed to implement the corrected product, (6) the correction effort for all defects during a set period, then (7) calculate the correction effort, which is the sum of 1, 2, 3, 4 and 5, and then (8) calculate the correction effort per defect, which is the sum of the correction efforts divided by the number of defects.

We also measure the Mean fault correction time, which is the mean time from receiving the failure report to sending the corrected software in fault correction. A lower value if preferred for this factor. Mean failure treatment time, which is the mean time from the failure occurrence to the restoration for end users, is another measuring way. Other indicators are mean fault correction work time, which is mean work time for correcting the discovered software fault to be corrected in fault correction, in consequence of analysing the failure and mean revise work time per changed line of source code, which is mean work time for revision per updated and added source code line. The last factor is fulfillment degree of maintenance document, which means fulfillment degree of product documents that are usable in the maintenance stage.

Stability can be defined as attributes of software that bear on the risk of unexpected effect of modifications. It is measured by the ratio of new faults made at revision [ZEI96].

Testability is the attribute of software that bears on the effect needed for validating the modified software [ZEI96]. This characteristic has great influence on reliability and the expected maintenance effort. It can be measured by different ways. One of them is test effort per unit volume, which is the effort needed to test one unit volume of the software product with a certain testing coverage degree.

We firstly specify the required test coverage degree, then measure the effort needed to test certain parts of the software product and size of these parts; and then calculate the test effort per unit volume. This can only be measured after completion of the software product. However, an estimate can be made after finishing a representative part of the product. Another measure is number of test cases per unit volume, which is the number of test cases that have to be made to test a unit volume of the software product with a certain testing coverage degree.

The required test coverage degree is specified, and the number of test cases needed to test certain parts of the software product is estimated. Then the size of the tested parts of the software product is measured and finally the number of test cases per unit volume is calculated. Other factors are Mean user's work time to verify the fault correction, Mean maintainer work time to test the fault correction after correcting the fault and Work time to test updated and added source code per unit line in case of correcting faults and/or case of re-modelling.

Manageability is described as attributes of software that bear on the effort needed to (re) establish its running status [ZEI96]. It is measured in terms of control effort ratio, which is the ratio of effort put in controlling the software product (including maintenance) in man-hours to the number of hours the product is available to the users.

We first determine the duration of the measurement period in order to achieve a sufficiently reliable measurement, and the amount of time needed to control the software product (including maintenance) during this period is registered. Then we register the availability of the software product in hours during this period and the ratio is determined.

Reusability is described as attributes of software that bear on its potential for complete or partial reuse in another software product [ZEI96]. It is measured by ratio of reusable parts and ratio reused parts, which is the ratio of reusable/reused parts of the software product to the total number of parts of the software product.

5.2 Problems with Web Maintenance

Software reengineering and maintenance are two classical activities in software engineering. Software maintenance is a time consuming task because requirements change and evolve frequently. Moreover, these changes happen very often in case of web based applications. The growth of the World Wide Web and the accelerated development of web sites and associated web technologies have resulted in a variety of maintenance problems.

We have focused on the maintenance problems in the web domain because it is a rapidly changing area and the operations involved in the development and maintenance processes usually must be completed in a very short time. Therefore, the factors that must be kept in mind in case of maintenance of web based applications are a) Cost reduction, b) Time to market improvement, and c) Faster development of new products.

The following maintainability requirements have been specified for web based applications:

  • Not more than 1 hour spent correcting any identified fault (corrective maintenance).
  • Each new function point (system enhancement) shall be implementable within 2 working days (perfective maintenance).
  • Each new change required to adapt to changes in environment shall be implementable within 2 working days (adaptive maintenance).
  • Each change in the system designed to reduce future maintenance costs should be undertaken within 1.5 working days (preventive maintenance).

The World Wide Web has grown to be a huge information and storage medium. The speed and breadth of its growth and its inherent complexity have meant that the web has not seen many of the rigorous development and maintenance principles applied to its contents and applications that traditional Software Engineering applies to more conventional large scale software applications [WAR99] [BRE98]. Large sites often contain many thousands of lines of code split into many modules stored in many places with large amounts of data often important to the owner of the site.

In most traditional software systems, the code is segregated from the data by data abstraction, and many systems are data processors in which the program only interacts with the data when that data is provided to it. However, due to the design of HTML, in web documents, the data is marked up by tags to format both its structure and style of presentation. Thus, the data is an integral part of the code, and this complicates web maintenance. Others have noted that the blurred distinction between data and software in the HTML model presents developers with some interesting and problematic consequences [WAR99] [ANT99].

Due to the ease with which web pages can be generated using a variety of tools (MS Frontpage, Netscape Composer, Macromedia Dreamweaver to name but a few), web sites can be created by people with little or no formal knowledge of software engineering and in very little time. For small scale personal web sites, it is not important that the site is error free and up-to-date; however, for larger corporate sites, this is often key to the businesses' on-line success.

It is argued that currently web sites and the WWW lack both data abstractions and structures that could facilitate maintenance. Classical web maintenance operations can be divided in several categories, such as a) Modification and updating of web pages, b) Populate databases with updated information, c) Graphical design changes, and d) Other technical operations not related to the web development and information production, such as server maintenance tasks.

In addition, the main problem in web development and maintenance operations is that time to market is frequently too short, because the customers demand a rapid solution for their needs. In this way, the staff employed in web development and maintenance tasks may vary depending on the time-to-market requirements. Hence, a solution to satisfy time requirements is to increase the size of the development group but this leads to an increment of the staff costs.

Another problem occurs when development and technical staff roles are intermixed. The technology under the web that has been developed includes markup languages (e.g.: HTML, XHTML, XML), script languages (e.g.: JavaScript, PERL) and other programming languages that support database functions such as ASP and PHP pages. This diversity of solutions means that all the staff needs to know about several programming languages in order to develop suitable web pages.

Finally, another problem occurs when the HTML code is completely embedded with JavaScript, ASP or PHP code. This is particularly important in maintenance operations due to the reasons that, a) The web files are larger and more difficult to read and maintain than if they were divided into different source code files, b) Programmers and web developers need to know several languages and techniques, c) Maintenance operations are time consuming because all the code is usually mixed, and d) the modularity of the web files is low.

Due to the mentioned problems, if we want to reduce the time spent in the modifications or even to develop new web products quickly, then we will need to change part of the development and maintenance processes to achieve a reduction in the complexity of the web files. Therefore, a reengineering process can be a good choice to analyze the existing code and try to develop web components reliable and faster from which new web systems can be built or already existing systems can be maintained with less costs.

In addition to the problems mentioned above, there are more causes of web site maintenance problems such as Duplication of files or data, the file structure of web sites, the HTML format of combined code & data storage and the problems of using database storage.

If the same information is required in two or more pages, it is common for that information is to be duplicated and placed on all the pages. HTML lacks an "include" directive common in many other languages. This problem is caused by the method of storage within HTML and forces the designer to choose between linking pages and copying the data. Likewise if a file, typically an image file is needed in several places and it is awkward to reference it in a single location, then copies may be made and placed throughout the site. This causes the problems associated with the failure to successfully update all copies.

Web sites consist of files containing the site contents. The predominant file format is HTML, with some embedded content such as images and increasingly the inclusion of scripts both as part of the HTML document or as a means of generating the HTML page. The files, which can be likened to modules in a conventional software system, are usually stored in directories to further modularise the site into comparmentalised blocks of similar data type or related content.

When this approach is followed uniformly and sensibly across the site, it is very useful and effective in creating a site whose content is easy to manage [WAR99]. However, the huge scale of some web sites and the inherent complexity of so many interlinked files make the organized de-centralisation of the pages a complex task which, if not carried out carefully and with planning, can result in illogical arrangements of files which quickly become too complicated to manage.

HTML files are a combination of data and tags. The tags control both the style and layout of pages. As these contents are intermixed, a maintainer editing a page to alter content can inadvertently alter tags. It is important to carefully design the file structure associated with a web site and to consider the implications of file contents on the maintainability of a site.

One of the main uses for databases on the WWW is for the storage of company data such as customer or product details. This information may be stored in specialized databases designed and used solely for the company web site, or it may utilize a connection to existing company databases to produce the same result [ANT99]. Obviously it is wasteful to maintain the same information in more than one database although for security reasons it may be sensible to make one database a partial or complete replica of another.

Many companies and organizations holding information in existing databases have sought to integrate these data stores with their web sites, rather than incur the penalties of having to maintain separate data sources. Dynamic page generating systems and associated scripting languages make such integration possible although considerable redevelopment of existing web pages may be necessary. In addition, co-ordination of the database maintenance with the web site maintenance is required. Database systems can also be used to store more complex components of web sites or references to these.

Examples of these are databases to store links [ARN99] [HAR97]. These enable maintainers to easily verify and update links throughout a site. However, in the case of older web sites, some reverse engineering of the site may be required to ensure that currently replicated elements within the web site are identified, abstracted, and, where appropriate, moved to a database or subsumed into a scripting language program.

5.3 Factors Affecting Web Maintenance

There are certain factors that affect the maintainability of web applications. These are hyper-links, information content and style [BOL01]. Hyper-links are major cause for the inconsistency and inaccuracy of data in case of web based applications. This is due to the links explicitly hard-wired into the HTML of a web page no longer pointing to the target page and its contents.

A broken link may be due to the facts that

  • a) the linked site has moved or closed down
  • b) The linked page has been removed or renamed
  • c) The link has been incorrectly specified (either mis-spelt or simply nonexistent)
  • d) The page exists, but its access permissions have been wrongly set; and
  • e) The page exists, but the relevant information has changed or been removed making the linked content no longer relevant to the originating page with a link to it.

There are two categories of the ways developed to minimize the first three situations described above i.e. dynamic link storage and link testing tools. Dynamic storage usually consists of a database that stores the links centrally [ARN99]. The HTML page containing the link is dynamically generated with the data for the database when requested. This enables the maintainer to easily update all occurrences of a link in a site when the link changes.

It also simplifies the process of re-structuring a web site. Link testing is carried out to find links that are no longer valid [ARN99]. Typical systems parse the HTML document and systematically test every link to ensure that a valid page exists at the end of the link. These tools can be used in conjunction with dynamic storage to provide more comprehensive aid to maintainers. The final two situations are almost impossible to prevent unless there is co-operation between the maintainers of the pages at both ends of the link. Typically this will only be possible for links within a site.

Another factor is Information Content. Here, there can be problem of up-to-date information, redundancy and incompleteness. There is large amount of data that changes frequently in case of many web sites. One must have one central information point and it should be kept up-to-date and accurate. Even then the problems of out-of-date information are still common and many of the sites give no indication of the currency of their information, nor do they indicate the last time of update.

Redundancy is separate from the previous category because, although it will invariably include incorrect information, in the case of redundancy (inconsistent information), some of it may be correct. In a typical website, the same information may be stored in one or more different places. If this information changes and the maintainer doesn't up-date all occurrences of the information, the inconsistencies will be introduced into the site [WAR94] and there will be problem of redundancy. Consequently some of the information will be incorrect and misleading [WHI97]. Incomplete information is of no use for the users of the web site. This problem can also be due to broken links, as discussed earlier.

Style is a separate consideration from the design of the user interface in so much as it is not concerned with the actual choice of style used, but rather the consistent application of the styles used. Research into Human Computer Interaction has shown that systems should use a consistent and standard style throughout [WHI97].

However, it is not uncommon to find different fonts, font sizes, colours, backgrounds and other style variations within the same web site across several pages or even in single pages of the site. Obviously this gives the site an unprofessional image as well as causing problems for users with navigation and readability. The Internet communities, and the standards bodies in particular, have started to make great efforts to counter this problem.

5.4 Measuring Maintainability

Lord Kelvin once said that when you can measure what you are speaking about and express it in numbers, you know something about it. Measurement is fundamental to any engineering discipline. If the maintainability of a system can be accessed, the relative amount of effort required for maintenance can be predicted, so will not come as an unexpected organizational expense. Maintainability metric can help management make an informed decision on whether a component should be maintained or completely rewritten to reduce future maintenance cost.

A metric should possess the following characteristics:

Simple and computable: It should be easy to learn how to derive the metric and its computation should not be effort and time consuming.

Empirically and intuitively persuasive. The metric should satisfy the engineer's intuitive notion about the product under consideration. The metric should behave in certain ways, rising and falling appropriately under various conditions.

Consistent and Objective: The metric should always yield results that are unambiguous. The third party would be able to derive the same metric value using the same information.

Consistent in its use of units and dimensions: It uses only those measures that do not lead to bizarre combinations of units.

Programming language independent.

An effective mechanism for quality feedback: Metric should provide search information that can be used to evaluate the effectiveness of the products.

In addition to the above-mentioned characteristics, metric should be defined in an unambiguous manner, should be tailored to best accommodate specific products and processes. Keeping in view the above-mentioned characteristics, Maintainability Index for Web (miw) is proposed that ranges from 0 to 1 in order of most desirable to least desirable:

miw=(a*x+b*y+g*z)/3 (i)

where a,b,g is the maintainability effort multiplier (constant) depending upon the type of web-page (text, multimedia etc.) with respect to hyperlinks, information contents, and style attributes respectively : 0<= a,b,g <=1.

x=f(lp, bp, ip, irp) (ii)

where 0<=x<=1 and

f(lp, bp, ip, irp) =[lp-(bl+ip+irp)] * ValidLinkEffort + bp * BrokenLinkEffort

+ ip * IncorrectLinkEffort + irp * IrrelevantLinkEffort.



bp=broken links/page

ip=incorrectly specified link/page

irp=irrelevant page's link/page

y=f(re, up, in) (iii)

where 0<=y<=1 and f(re, up, in) =(re*Ere + up*Eup + in*Ein)


Ere , Eup , Ein are the effort multipliers with respect to re, up & in, and

re=redundant information content

up=not up-to-date information content

in=inconsistent information contents

z=f(style attributes) (iv)

where 0<=z<=1 and

f(style attributes) = (TotalPagesOfInconsistentStyle) * E

where E is the average effort to convert the page in uniform style.

5.5 Summary

Maintenance is the most time and effort-consuming activity for any type of software and web sites are also not an exception. If the efforts required in maintaining web software can be predicted quantitatively then quality of the web software can be assured, and necessary resources can be allocated in advance.

In this research, it has been identified that factors affecting the maintainability can be categorized into three classes i.e. hyperlinks, information content, and style. Although to decide the values of a, b, and g is not an easy task, yet these can be decided by empirical studies. Further researches can be carried out to identify other factors in the above-mentioned three classes to further refine miw.

Chapter 6 Issues In Designing Of Web Load Test

6.1 Introduction

Test Engineers' and web developers' perception of a web site: A distributed software consisting of components like servlets, applets, cookies etc., of which some executes on servers awhile others get downloaded to the client machines and execute there itself. All the testing strategies and techniques applicable to traditional software are relevant in web testing also.

Additionally, web systems pose new challenges for the test engineers, owing mainly to their unique nature [SIN03]. Usage and access patterns of web sites are highly random owing to stochastic components like human involvement, communications channels etc. Moreover, the user community of web site form a highly heterogeneous group, therefore these software could be tested for stress / load in right earnest [SIN04].

Of lately, load testing has become one of the top quality assurance priorities for organizing with mission-critical Web sites. How many users will the web site be able to serve while still maintaining acceptable response times? Results of the web load test are aimed at answering this question. And the results (if otherwise correct) prove indispensable piece of information for planning market campaigns, estimating technology budgets etc. [SIN04]. Despite being central to very important decisions, practically all web site load tests are not designed properly. The load simulated by load tests should be realistic one [LIN04].

This chapter describes some important issues in the design of realistic and accurate web site load tests namely, a) understanding the nature of the load, b) estimating target load levels, and c) documenting the design.

6.2 Understanding the Nature of the Load

Before a web site is tested for performance under estimated load levels, the very nature of the load itself need to be understood rightly. Various factors affecting the load levels need to be understood, and lastly, the load need to be generated using random variates from suitable distribution [JOH01].

To understand the nature of the load accurately and objectively is different since web load testing is a new discipline and is poorly understood and documented. Considering the amount of effort required for planning and executing a web site load test, and that most Internet-based projects have strict delivery schedules, most test experts generate any (read unplanned and unrealistic) load and get back any (read possibly incorrect) results. But, any laxity in load testing may cost the organization very dearly in that it may lead to misleading performance projections and an eventual abandonment of disgruntled users [JOH01].

The first step in designing a Web Site load test id to understand as accurately and as objectively as possible, the nature pf load that must be generated. Unfortunately, very often, even the experienced, well-meaning and hard-working test engineers may also design and develop load tests that don't simulate the real loads.

Perhaps, the primary reason for this is that we load testing is a new area of testing. Even if the recently added internet components (like audio, graphics, animations and video etc.) are ignored, load testing has always been one of the more mysterious and less covered types of test, with relatively few test experts available [SIN04].

However, if significant time and expense has already been put in for load testing, the additional effort required making it realistic is minor compared to the significant gain in confidence in the results. The first step toward realism is to understand and characterize the anticipated load as objectively and systematically as possible. Cederholm [CED02] suggests using the web site load files as a resource and the log file analyzer as a tool understanding the true nature of web site load.

Log Files and Log Analyzers: Every time a visitor visits the web site, the details of that visit are recorded (provided the logging option is enabled on the server). Following details are recorded for each user session:

  • The IP address of the visitor
  • The Data and Time of access of files / pages
  • The size of the accessed files / pages
  • A code indicating the success of a session
  • Information about the visitor's browser, operating system, search engine etc.

These log files are stored in the machine-readable format that must be processed to extract the required information. Several public domain and commercially available log file analyzers are currently available online [CED02].

User Sessions versus Concurrent Users: Number of concurrent users is a popular metric when it comes to describing a Web load. But this approach is riddled with inherent problems as described by Cederholm [CED03].

The number of concurrent users should not be seen as input for a load test run, but as the result of a number of factors. But, still it is very common to see something like this: The Web Site will be tested with a load of 1,000 concurrent users.

A better input variable to a web site load test is the number of user sessions per unit of time [CED03]. The concept of the user session, which is defined as a sequence of related page views by a unique visitor, is vital to Web site load testing. A wide range of users with a wide range of objectives visits a typical Web site. Of all the visitors to a university site, for example, some may be there to browse, some others to download an admission form, and others to know about their results.

Some will move from one page to another in rapid succession, barely taking any time to read the content; others will take their time with each page and move more slowly. Understanding this wide range of actions and behaviors is critical for load test design, since a well-designed load test should replicate these actions and behaviors as accurately as possible. The best way to capture the nature of Web site load is to identify and track some key variables of user sessions that are applicable and relevant to Web site traffic. Some of the variables that should always be tracked are:

  • The length of the session (measured in pages)
  • The duration of the session (measured in minutes and seconds)
  • The type of pages that were visited during the session (e.g., home page, faculty pages, result card preparation page)

Other Factors

In addition to these basic metrics, however, other variables that are likely to make a nontrivial difference in the load, may also be tracked. One can easily observe that, for example, a seven-page session that include a form downloading or registration with university or preparation and display of result card is going to create more load on the Web site than a seven-page session that involves only browsing.

A browsing session might only involve the serving of static pages, while a download/registration session will involve transfer of a file or updating of registration database. A single registration session might put as much load on some of the system's resources as twenty browsing sessions. Similar reasoning may apply to transactions carried out by new versus frequent users.

After having determined the user session variables that need to be observed, the log analyzer may be used to find out the range and distribution of values for these variables. Most log analyzers automatically generate information on basic metrics such as session length or session duration [CED03].

Statistical Data

For each of the session variables being tracked, at the very least, its average value must be known; however, it is worthwhile and often necessary to go well beyond that. The average number of pages per session, for example, may turn out to be 4.0, but since that average may be the result of 50% of the users just visiting the home page and 50% of the users visiting seven pages, a load test in which all sessions are four pages long would match the average - but it would not be realistic.

Use of statistical tools such as standard deviation could be better, but the still better way to characterize most Web load testing variables is to use a discrete distribution. In the case of pages per session, the distribution might look as follows:

Number of pages/session Percentage of sessions




4 or more 06%

The amount of detail and precision with which these variables are tracked depends on the complexity of the Web site, as well as the time required to design, develop, execute, and analyze the results.

For pre-launch sites, where log files are not available, data can be used from the logs created during beta tests to get a first approximation of load test variables. Designing a load test also requires understanding volume patterns and load levels. It is necessary to estimate key variables - target load levels based on expected growth of traffic, the peak load level that might occur, how fast the number of users can ramp up to the peak level and how long the peak level may last. Use of the metric user sessions per unit of time is suggested by Shea [SHE03].

User Abandonment: Another usually overlooked load testing factor that has a huge impact on load testing results is user abandonment. I have left a Web site because its pages were loading too slowly. You too must have. I am sure you have. Abandoned Internet sessions are extremely common.

Considering that the magnitude of user abandonment is going to be quite high at the levels that are likely to be used in a load test, and that this user abandonment will have a significant impact on the resulting load, it is very surprising that most Web load tests don't even attempt to simulate this very common behavior with any degree of realism. User abandonment must be simulated as realistically as possible; otherwise the web site is tested for a load that will never occur in real life [DOU03].

Page Request Distribution: One user session variable that deserves special consideration is the statistical distribution of page requests. This variable tells what pages are being requested, and in what proportions. Developing a page request distribution is a two-step process: firstly, pages are categorized, and then the percentage of page requests in each category is calculated. A set of discrete page categories are defined, because for most Web sites it is not possible to list all the pages, and moreover in terms of load many pages are equivalent.

If all faculty description pages of a university website, for example, contain similar information, i.e., a couple of photographs and several lines of text, the load they will present to the Web site will be very similar. By the same token, a home page is going to present very different load from every other page. The home page may be a purely static page (and will probably be cached in memory), so it will take the server very little effort to respond to the request [DOU03].

6.3 Estimating Target Load Levels

The next step in designing the load test is to understand the volume patterns, and to determine what load levels the Web site under test might be subjected to (and must therefore be tested for). There are four key variables that must be understood in order to estimate target load levels:

  • How the overall amount of traffic to your Web site is expected to grow
  • The peak load level that might occur within the overall traffic
  • How quickly the number of users might rise up to that peak load level
  • How long that peak load level is expected to last

Load levels might be measured in terms of concurrent users, user sessions per unit of time, or number of page views, etc. The concept of user sessions is easily understood by technical and non-technical people at all levels of the organization and is tracked easily by most log file analyzers [SIN04a].

Estimating Overall Traffic Growth: Estimating the growth of the overall amount of anticipated traffic is important, because the magnitude of the peak levels is going to be proportional to the magnitude of the overall traffic. Overall traffic growth is typically estimated from two sources, namely, historical growth data and future projections [CED03].

In many cases, both of these components are used. To estimate this number, the current weekly traffic is estimated and the appropriate people in the organization are asked about how they anticipate the traffic to vary from month to month. Also, they are asked if they anticipate any unusually high volumes due to special event like festival season.

From this conversation it may be inferred that the company anticipates a specific month-to-month variation, and that a new product launch is expected to draw a specific number of additional visitors to the Web site during the week of the launch. Combining this information, the number of visitors per week in the week of launch may be estimated.

Estimating Peak Levels: Once an estimate of overall traffic growth is in place, the peak level expected within that overall volume need to be estimated. This is necessary because Web traffic is not uniformly distributed and most Web sites exhibit very noticeable peaks in their volume patterns. Fortunately, the ratios between overall traffic and peak number of sessions can be used to make educated guesses about target volume levels.

Typically, there are a few points in time (one or two days out of the week, or a couple of hours each day) when the traffic to the Web site is highest. A weather information Web site, for example, may experience its highest traffic on Fridays and Saturdays, as people make plans for their weekend. If traffic log data for the current year are available, then it is straightforward to estimate when the peak volumes will occur in the next year.

The monthly levels will show if there is a clear seasonality in volume pattern and the weekly levels, on the other hand, will show if there is a pattern related to days of the week. At the end of this analysis, these data are compiled in form of a volume patterns table [DOU03].

The ratio of peak to average volume can be used to set target peak levels for the web load test. Information about lowest volume periods would be useful for scheduling a load test on a site. Running your load tests during hour of minimum load will allow the load test runs while causing the minimum amount of disruption to users.

6.4 Design Representation

Having a good understanding of the nature and traffic volumes of the anticipated load means having lots of information to justify the decisions regarding load testing. Thereafter, the web load tests can be designed with ease and confidence. The key elements of a load test design are:

  • Test objective
  • Pass/fail criteria
  • Script description
  • Scenario description

Of lately, time has become a very scanty thing for most people. Reading anything longer than a couple of pages are not very common nowadays, so the design document should be as simple and direct as possible.

Load Test Objective

The test objectives section must explicitly mention the objectives of load test, and what actions are going to be taken based on its results [CED02]. The objective of this load test is to determine if the Web site, as currently configured, will be able to handle the peak load level anticipated for the coming festival season measured in terms of user sessions per unit of time. If the system fails to scale as anticipated, the results will be analyzed to identify the bottlenecks, and the test will be run again after the suspected bottlenecks have been addressed [CED02].

Pass/Fail Criteria

Furthermore, a test should have some predefined pass or fail criteria associated with it. Unfortunately, most load tests are executed without a clear understanding of what constitutes success or failure. Make sure that there is an understanding in the organization about what results will be acceptable.

The load test will be considered a success, if the Web site will handle the target load of a specified number of user sessions per unit time while maintaining predefined average page response times. The page response time will be measured as the elapsed time between a page request and the time the last byte is received.

Script Description

Next, the types of scripts that are planned to be used in the load tests are documented. Complexity of the Web site determines the number of scripts required. Since in most cases the user sessions follow just a few navigation patterns, hundreds of individual scripts are not required to achieve realism. If chosen carefully, a dozen scripts will take care of most Web sites. But choosing them carefully requires some work, since this combination of scripts must be able to closely replicate the page distribution and other characteristics of web site [DOU03].

Scenario description

Finally, scripts should be combined to describe a load testing scenario. A basic scenario includes the scripts that will be executed, the percentages in which those scripts will be executed, and a description of how the load will rise up. If the time and other resources are in plenty, multiple scenarios are executed to simulate a range of different peak volumes. But then each scenario must have its own clearly stated objectives and pass/fail criteria.

6.5 Summary

Designing scripts for testing the performance of web site under projected load levels in important, for the test results are going to become the basis of marketing and budgeting etc. For a web load test to be meaningful, the tests should reproduce the anticipated loads as accurately and realistically as possible. Doing this necessitates studying previous load patterns, if available, and design test scenarios that closely reproduce them.

For maiden launch of web sites logs created during beta testing may be used to approximate load test variable. This is a task that will require a serious amount of hard work, intelligence, intuition, and communication skills. But the additional effort will pay off in terms of enhanced confidence in web site performance projections, learning new technologies, tools, and ways of thinking, and expanding the skill set of load test engineers. This in turn will significantly increase testers' marketability and career options.

Chapter 7 Conclusion and Future Scope

6.1 Conclusion

Developing successful Web applications with economic and quality issues in mind requires broad perspectives and the incorporation of a number of principles, models, methods, and techniques from diverse disciplines such as information systems, computer science, hypertext, graphic design, information structuring, knowledge management, and ultimately software engineering as well. Web engineering is therefore an amalgamation of many disciplines, but with its own challenges.

It has a very diminutive history compared with other engineering disciplines, but is rapidly sprouting. Like any other engineering science, Web engineering is concerned with the establishment and use of sound scientific, engineering, and management principles, and disciplined and systematic approaches to the triumphant development, deployment, maintenance, and evolution of Web sites and applications within budgetary, calendar, and quality constraints.

Software engineering for the World Wide Web is indeed different, but one can acclimatize much of what is already acknowledged to understand these differences. The progress in this direction is astonishing, and is allowing much of the research of the last 20 years to be prolifically used. Development of web applications necessitates a disciplined approach & to realize discipline, measurement of quality attributes should be on hand.

Defining quality attributes is an often underestimated part of the software design and development process. Most commonly used quality attributes can be easily redefined for greater utility, simply by placing more prominence on what is most concrete, measurable, and actionable about them. The quality of an entity is easy to be acquainted with but hard to define and evaluate, and sometimes expensive to incorporate in the end product. Because quality is not achieved at the closing stages of a development without a vigilantly designed quality assurance strategy in the premature stages, the three perspectives of quality internal quality, external quality, and quality in use have their own relative importance.

It is scrutinized in the literature that the issues, which are pertinent in procedure-oriented & object-oriented Software, are also relevant in web-based applications, although the priorities may be poles apart. It can be concluded that efficiency, usability, maintainability, scalability & reliability are more imperative attributes then others. Maintenance is intricate in web applications due to the limited time given to designers to modify it & upload it. The need of scalability can on no account be undermined but reliability & availability should not be compromised.

Maintenance is the most time and effort-consuming commotion for any type of software and web sites are also not an exception. Maintainability can be defined in terms of efforts required to maintain the software. If the efforts required maintaining web software can be predicted quantitatively then quality of the web software can be pledged, and necessary resources can be apportioned in advance. So, to test the maintainability of the web-based software in a quantitative way is a very significant concern.

In this research, it has been identified that factors affecting the maintainability can be categorized into three classes i.e. hyperlinks, information content, and style and subsequently a metric, maintainability index (miw), was devised ranging from 0 to 1 in order of most desirable to least desirable.

The web applications having the Maintainability Index at a lower side will be easy to maintain, while those having it at a higher side will be very time and effort consuming. Although to settle on the values of a, b, and g in the said metric is not an undemanding mission, yet these can be decided by pragmatic studies. Further researches can be carted out to discover other factors in the above-mentioned three classes to further distill miw.

Web applications are swiftly nurturing in number and convolution, and becoming the de facto standard for distributed applications that require human interaction. The escalating number of Web users, the multiplicity of application domains, content, the complexity of hypertext structures and interfaces, all hearten the use and measurement of usability as a determinant factor for the triumph of such applications.

Applications that incorporate usability engineering into their development process are anticipated to proceed in accordance with with quality obligations. In particular:

  1. Evaluation is the key for assuring quality: the effort devoted to an evaluation can unswervingly establish the quality of the final application.
  2. To be effectual, an evaluation must rely upon appropriate and validated usability criteria.

The methods currently adopted in assessing the usability of Web applications, and criteria that can be applied to the evaluation of Web applications have been premeditated. In addition, the advantages and snags of different usability methods have been identified so as to facilitate practitioners choose the most suitable method with respect to the evaluation goals.

Independent of the method chosen, practitioners and researchers advocate that a sound usability evaluation plan should include the use of different, complementary methods, to ensure the entirety of the evaluation results. The characteristics of each method determine their effectiveness in discovering a specific class of usability problems.

To improve the quality of any product, one must be proficient to measure the un-measurable. The World Wide Web is the biggest library of the world, but unstructured and unorganized due to a common trend of migrating each and every information to the web without considering the quality. And the sufferer is the pitiable user. The most important quality attribute of a web application is its usability - affected by a number of factors.

So the testing of usability of web-based applications in a quantitative way is another vital issue. The factors considered for improving and measuring the usability of web sites were: Lightness of text, graphics, & backgrounds, color of the items to be distinguished, color of the visited & unvisited links, availability of load time progress bar if load time is more, response time, location of navigation buttons, appearance & size of navigation buttons, caption on navigation buttons, tool tips with buttons, dissimilarity between the dissimilar pages and time consumed in mouse movement and keyboard access.

The metric, Usability Index, based on these factors was identified and it can be exploited to measure the usability and help in assuring the quality. All the Usability Factors were assigned a weight and the usability index of the web application was measured, which varies from 0 to 1 in the order of least desired to most desired. The web application having usability index below 0.5 are having very low usability, while the applications whose usability index is more than 0.5 are considered usable.

Web sites are integration of different components varying in performance profile such as hardware, software and networking components made available by different vendors / manufactures yielding a heterogeneous system the behavior of which is difficult to predict under real time conditions. So there is always an urge to perform a load test to gain an insight of the performance. Concerns pertinent to designing a realistic and accurate web site are familiarizing with the nature of load, identifying target load levels and documenting the design.

A good load test is characterized by the crafting of anticipated loads as accurately and realistically as possible, demanding the study of prior load blueprints, if available, and devise test scenarios that closely reproduce them. For maiden launch of web sites, logs created during beta testing may be used to approximate load test variable. It is a Herculean task asking serious amount of hard work, intelligence, intuition, and communication skills. But the additional effort will pay off in terms of enhanced confidence in web site performance projections and increased testers' marketability and career options.

6.2 Future Scope

Modern web site applications entail:

  • Software that is more complex than before
  • Software that is of high quality
  • Software that can be updated quickly and reliably.

These issues present attention-grabbing research problems. Current knowledge to create software for the Web that is both of sufficient complexity and of sufficient quality is scare. On the other hand, the field is making extraordinary progress. The technological innovations that have been developed in just the past three years are stunning both in breadth and in depth, and new research results are appearing every day.

Identifying common styles and other contents requires extensive analysis of the existing web pages. However, the rationalization of the site achieved should in theory permit maintainers to understand the contents of the web sites more straightforwardly and to separate concerns of style from other types of information content during subsequent maintenance of the site. Empirical evidence of such benefits can only be obtained in the longer term. So an extensive research is required to implement a component based approach for web-based applications development.

The quality attributes can be classified with respect to types of composition, in which each type is characterized by the required input for obtaining predictability on the system level. Some types demonstrate obvious composable characteristics, while others are not directly related to compositions. In spite of diversity of attributes, technologies, and theories, it should be possible to craft reference frameworks that by identifying type of composability of attributes can help in estimation of accuracy and efforts required for constructing component-based systems in a predictable line of attack.

Techniques from conventional Reserve Engineering can be applied to alleviate some of the maintenance problems found with existing web sites and the current storage systems employed for web sites. Certainly there is scope for further research on the reverse engineering of web sites.


[ACM94] Object-Oriented Software Testing, Communications of the ACM (special issue), ACM Press, Volume 37, Sept. 1994.

[AGR90] Agrawal H and Horgan J, Dynamic program slicing, Proceedings of the ACM SIGPLAN '90 Conference on Programming Language Design and Implementation, SIGPLAN Notices, pages 246-56, June 1990.

[AGR93] Agrawal H, Horgan J, Krauser E and London S, Incremental regression testing, Proceedings of the Conference on Software Maintenance - 1993, pages 348-357, September 1993.

[AGR94] Agrawal H, On slicing programs with jump statements, Proceedings of the ACM SIGPLAN'94 Conference on Programming Language Design and Implementation, SIGPLAN Notices, pages 302-12, June 1994.

[ALA97] Alan C G, Software Quality Theory & Management, International Thomson Computer Press, 1997.

[ALB79] Albrecht A, Measuring Application Development Productivity, Proc. of IBM Application Development Symposium, Monterey, California, Pages: 83-92, 1979.

[AND05] Andrews A A, Offutt J and Alexander RT, Testing Web Applications by Modeling with FSMs, Software Systems and Modeling, 4(2), 2005.

[ANT99] Antonio G, Canfora G, et al, Web Sites: Files, Programs or Databases?, Proceedings of WSE99, 1st Annual Workshop on Web Site Evolution, Pages: 6-8, 1999.

[ARN03] Tom Arnold, Testing Web Services (.NET and Otherwise), Software Test Automation Conference & Expo, Spring 2003.

[ARN99] Arnold SC, An Architecture for Maintaining Link Structure of a Website, Proceedings of WSE99, 1st Annual Workshop on Web Site Evolution, Pages: 9-11, 1999.

[BAN00] Bangio A, Ceri S and Fraternali P, Web Modeling Language (WebML): A Modelling Language for Designing Web Sites, Proceedings of the 9th International Conference on the WWW (WWW9), Elsevier: Amsterdam, Holland, Pages: 137-157, 2000.

[BAS84] Basili V R and Perricone B T, Software errors and complexity: An empirical investigation, Communications of the ACM, 27(1), Pages: 45-52, January 1984.

[BAS96] Basili V R, The role of experimentation in software engineering: past, current, and future, Proceedings of the 18th International Conference on Software Engineering, 25-30 March, Pages: 442-449, 1996.

[BAS99] Basili V R, Shull F and Lanubile F, Building knowledge through families of experiments, IEEE Transactions on Software Engineering, 25(4), Pages: 456-473, July-Aug 1999.

[BAT93] Bates S and Horowitz W, Incremental program testing using program dependence graphs, Proceedings of the 20th ACM Symposium on Principles of Programming Languages, January 1993.

[BAU00] Baudry B, Jezequel J M, and Traon Y L, Testing-for-Trust: The Genetic Selection Model Applied to Component Qualification, Proceedings in the Technology of OO Languages and Systems Conference (TOOLS-Europe 2000), Mont St Michel and St Malo, France, June 2000.

[BEI90] Beizer B, Software Testing Techniques. Van Nostrong Reinhold, New York, 1990.

[BEI95] Bezier B, Black-Box Testing, Wiley, 1995.

[BEN88] Benedusi A, Cimitile A, and Carlini U De, Post-maintenance testing based on path change analysis, Proceedings of the Conference on Software Maintenance, pages 352-361, October 1988.

[BEN91] Bennett KH, Automated Support for Software Maintenance, Information and Software Technology, Vol. 33, No. 1, Pages: 74-85, Jan/Feb 1991.

[BIN92] Binkley D, Using semantic differencing to reduce the cost of regression testing. Proceedings of the Conference on Software Maintenance, pages 41-50, November 1992.

[BIN95] Binkley D, Reducing the cost of regression testing by semantics guided test case selection. Proceedings of the Conference on Software Maintenance , October 1995.

[BIN99] Binder RV, Testing Object-Oriented Systems, Models, Patterns, and Tools, Addison-Wesley: Reading, MA, 1999.

[BOE78] Boehm B W, Brose J R, Kaspar H, Lipw M, Macleon G J and Merritt M J, Characteristics of Software Quality, Amsterdan, North Holland, 1978.

[BOG06] Bogardi-Meszoly A, Levendovszky T, Charaf H, Performance Factors in ASP.NET Web Applications with Limited Queue Models, Proceedings of International Conference on Intelligent Engineering Systems (INES '06), Page(s):253 - 257, 2006.

[BOL01] Boldyreff C, Kewish R, Reverse engineering to achieve maintainable WWW sites, Proceedings of Eighth Working Conference on Reverse Engineering, Pages(s):249-257, 2-5 Oct. 2001.

[BRE98] Brereton P, Budgen D, Hamiliton G, Hypertext: The next Maintenance Mountain, IEEE Computer, Vol. 31(12), pp. 49-55, December 1998.

[BRO90] Brown P A and Hoffman D, The application of module regression testing at TRIUMF, Nuclear Instruments and Methods in Physics Research, Section A, A293(1-2), Pages: 377-381, August 1990.

[BUE03] Buehler O and Wegener J, Evolutionary functional testing of an automated parking system. In International Conference on Computer, Communication and Control Technologies and The 9th. International Conference on Information Systems Analysis and Synthesis, Orlando, Florida, USA, 2003.

[BYB03] Bybro M, A Mutation Testing Tool for Java Programs, Thesis, Department of Numerical Analysis and Computer Science, Nada at the Royal Institute of Technology, KTH, June 2003.

[CED02] Cederholm D, Bulletproof Web Design: Improving flexibility and protecting against worst-case scenarios with XHTML and CSS, Voices That Matter, 2002.

[CED03] Cederholm D, Web Standards Solutions: The Markup and Style Handbook, Pioneering Series, 2003.

[CHA03] Chang Wen-Kui, Hon Shing-Kai and Chu Cheng-Chung William, A Systematic Framework for Evaluating Hyperlink Validity in Web Environments, Proceedings of the Third International Conference On Quality Software (QSIC'03), Page(s):178 - 185 , 6-7 Nov., 2003.

[CHE07] Chen Shengbo, Miao Huaikou, Qian Zhongsheng, Automatic Generating Test Cases for Testing Web Applications, International Conference on Computational Intelligence and Security Workshops (CISW 2007), Page(s):881 - 885, 15-19 Dec. 2007.

[CHI94] Chidamber Shyam, C. Kemerer, A Metrics Suite for Object Oriented Design, IEEE Transactions on Software Engineering, No. 20(6), Pages: 476-493, June, 1994.

[CHR00] Christodoulou SP, Zafiris PA and Papatheodorou TS, WWW2000: The developer's view and a practitioner's approach to Web engineering. In: Proceedings of the 2nd ICSE Workshop on Web Engineering, Pages: 75-92, 2000.

[CON07] Conroy Kevin M., Grechanik Mark, Hellige Matthew, Liongosari Edy S., and Xie Qing, Automatic Test Generation from GUI Applications for Testing Web Services, IEEE International Conference on Software Maintenance (ICSM 2007), Page(s): 345 - 354, 2-5 Oct. 2007.

[CON99] Conallen J, Building Web Applications with UML, Addison-Wesley: Reading, MA, 1999.

[DAN00] Daniel A. Menasce, Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning, Prentice Hall, 2000.

[DAV93] Davis A M, Software Requirements: Objects, Functions, & States, Prentice Hall, Englewood Cliffs, NJ, 1993.

[DAV95] Davis A, 201 Principles of Software Development, McGraw Hill, 1995.

[DEM78] DeMillo R A, Lipton R J and Sayward F G, Hints on test data selection: Help for the practicing programmer, IEEE Computer, 11(4), Pages: 34-41, April 1978.

[DES01] Deshpande Y and Hansen S, Web engineering: creating a discipline among disciplines, IEEE Multimedia, 8(2), Pages: 8-87, April-June 2001.

[DES02] Deshpande Y, Murugesan S, Ginige A, Hansen S, Schwabe D, Gaedke M and White B, Web engineering. Journal of Web Engineering, 1(1), Pages: 3-17, October 2002.

[DOM04] Domenech Josep, Pont Ana, Sahuquillo Julio and Gil Jose A, An experimental framework for testing web prefetching techniques, Proceedings of the 30th EUROMICRO Conference (EUROMICRO'04), Pages: 214-221, Aug. 31 - Sept. 3, 2004.

[DOU03] Douglas K and Duyne V, The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience, STQE, Nov.-Dec. 2003.

[ELB03] Elbaum Sebastian, Karre Srikanth, Rothermel Gregg, Improving Web Application Testing with User Session Data, Proceedings of 25th International Conference on Software Engineering, Pages: 49-59, 3-10 May 2003.

[ELB05] Elbaum S, Rothermel G, Karre S and Fisher M, Leveraging User-Session Data to support Web Application Testing, IEEE Transactions on Software Engineering, 31(3), Pages:187-202, 2005.

[EMI06] Emila M et al, Web Engineering, Springer, 2006.

[FEN87] Fenton N E, Kaposi A A, Metrics & Software Structure, Journal of Inf. Software Tech., Pages: 301-20, 1987.

[FEN97] Fenton NE and Pfleeger SL, Software metrics: a rigorous and practical approach, 2nd ed., PWS Publishing Company, 1997.

[FEW91] Fewster M, The manager wants 100 Journal of Software Testing, Verification and Reliability, 1(2), Pages: 43-45, July.Sept. 1991.

[FEW99] Fewster Mark and Graham Dorothy, Software Test Automation, Effective use of test execution tools, Adison-Wesley, Harlow, 1999.

[FIS77] Fischer K F, A test case selection method for the validation of software maintenance modifications. Proceedings of COMPSAC '11, Pages: 421-426, November 1977.

[FIS81] Fischer K F, Raji F and Chruscicki A, A methodology for retesting modified software, Proceedings of the National Telecommunications Conference B-6-3, Pages: 1-6, November 1981.

[FRA00] Fraternali P and Paolini P, Model-driven development of Web applications: the AutoWeb system. ACM Transactions on Information Systems (TOIS), 18(4), Pages: 1-35, October 2000.

[FRA88] Frankl P G and Weyuker E J, An applicable family of data flow testing criteria, IEEE Transactions on Software Engineering, 14(10), Pages: 1483-1498, October 1988.

[FRA93] Frankl P G and Weiss Stuart, An experimental comparison of the effectiveness of branch testing & data flow testing, IEEE Transactions on Software Engineering, 19(8):774-787, August 1993.

[GAR84] Garvin D., What does product quality mean?, Sloan Management Review, 4, 1984.

[GEL97] Gellersen H, Wicke R and Gaedke M, Web Composition: An object-oriented support system for the Web engineering lifecycle. Journal of Computer Networks and ISDN Systems, 29(8-13), Pages: 865-1553, September 1997.

[GEL99] Gellersen H-W, Gaedke M, Object-oriented Web application development. IEEE Internet Computing, 3(1), Pages: 60-68, January/February, 1999.

[GER00] Gerrard P, Risk-Based E-Business Testing: Part 1 - Risks and Test Strategy,, 2000.

[GIL02] Gill Nasib S, Software Reliability, Testing and Quality Assurance, Khanna Book Publishing Company Pvt Ltd, 2002.

[GIN01] Ginige A, Murugesan S, Web engineering: an introduction. IEEE Multimedia, 8(1), Pages: 14-18, January/March, 2001.

[GIN02] Ginige A, Workshop on web engineering: Web engineering: managing the complexity of Web systems development. Proceedings of the 14th International Conference on Software Engineering and Knowledge Engineering, Pages: 721-729, July 2002.

[GOO93] Goodman Paul, Practical Implementation of Software Metrics, McGraw Hill Book Co., UK, 1993.

[HAL77] Halstead M H, Elements of Software Science, New York, Elsevier, North Holland, 1977.

[HAR93] Harrold MJ and Gupta R, Soffa ML, A Methodology for Controlling the Size of a Test Suite, ACM Transactions on Software Engineering and Methodology, 2(3), Pages: 270-285, 1993.

[HAR97] Hartman JH, et al, Index-based Hyperlinks, Computer Networks and ISDN Systems, Vol. 29, Pages: 1129-1135, 1997.

[HEN81] Henry S and Kafura D, Software Structure Metrics based on Information Flow, IEEE Transactions on Software Engineering, SE-7(5), September, 1981, Pages: 510-518.

[HIE02] Hieatt E and Mee R (2002) Going Faster: Testing the Web Application. IEEE Software, 19(2), Pages: 60-65, 2002.

[HOR98] Horowitz E, Migrating software to the World Wide Web. IEEE Software, May/June, 15(3), Pages: 18-21, 1998.

[HOU94] Houghton Mifflin Company, The American Heritage Concise Dictionary, 3rd Ed, 1994.

[HOW05] Hower R, Web Site Test Tools & Site Management Tools. Software QA and Testing Resource Center, (Accessed 2005).

[HOW82] Howden W E, Weak Mutation Testing and the Completeness of Test Cases, IEEE Transactions of Software Engineering, Vol SE-8, No. 4, Pages: 371-379, July 1982.

[HUD94] Hudli, R., Hoskins C., Hudli, A., Software Metrics for Object-oriented Designs, IEEE, 1994.

[HUO03] Huo Qingning, Zhu Hong and Greenwood Sue, A Multi-Agent Software Environment for Testing Web-based Applications, Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC'03), Pages: 537 - 542, 25-27 Oct., 2000.

[IEE03] IEEE Std. 2001-2002, Recommended Practice for the Internet Web Site Engineering, Web Site Management and Web Site Life Cycle, IEEE, 2003.

[IEE90] IEEE Std. 610.12-1990, Glossary of Software Engineering Terminology, in Software Engineering Standard Collection, IEEE Computer Society Press, Los. Alamitos, CA, 1990.

[ISO86] ISO1986 Quality Vocabulary, ISO - 8042.

[ISO91] ISO1991, Quality Vocabulary ISO - 9126.

[ISO97] ISO 9241: Ergonomics Requirements for Office Work with Visual Display Terminal (VDT), Parts 1-17, 1997.

[JAL04] Jalote Pankaj, An Integrated approach to SOFTWARE engineering, Second edition, Narosa publication house, 2004.

[JOH01] Johnson K, Mining Gold from Server Logs, STQE, January - February 2001.

[JON97] Jong De M. and Schellens P. J., Reader-focused text evaluation. An Overview of Goals and Method, J. Business Tech. Communication, vol.11, no. 4, Pages: 402-432, 1997.

[JON99] Jong De M. and Heuvelman A. et al., Formative evaluation of public information on the WWW: A survey of approaches, in Jaarboek Onderzoek Communicatie management, A.A. Van Ruler, P. J. M. C. Schellens, and O. Scholten, Eds. Alphen a/d Rijn: Samsom, 1999.

[KAL01] Kallepalli Chaitanya and Tian Jeff, Measuring and Modeling Usage and Reliability for Statistical Web Testing, IEEE Transactions On Software Engineering, Vol. 27, No. 11, Page(S):1023 - 1036, November 2001.

[KAR07] Karam Marcel, Safa Haidar, Artail Hassan, An Abstract Workflow-Based Framework for Testing Composed Web Services, IEEE/ACS International Conference on Computer Systems and Applications (AICCSA '07), Page(s):901 - 908, 13-16 May 2007.

[KES07] Keshk Arabi and Ibrahim Amal Ensuring the Quality Testing of Web Using a New Methodology, IEEE International Symposium on Signal Processing and Information Technology, Page(s):1071 - 1076, 15-18 Dec. 2007.

[KIR02] Kirchner Michele, Evaluation, Repair, and Transformation of Web Pages for Web Content Accessibility: Review of Some Available Tools, Proceedings of the Fourth International Workshop on Web Site Evolution (WSE'02), Pages: 65-72, 2 Oct., 2002.

[KIT02] Kitchenham BA, Pfleeger SL, Pickard LM, Jones PW, Hoaglin DC, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, August, 28(8), Pages: 721-734, 2002.

[KIT89] Kitchenham B, Software Quality Assurance, Microprocessors and Microcomputers, 13(6), Pages: 373-381, 1989.

[KIT95] Kitchenham B, Pickard L, Pfleeger SL, Case studies for method and tool evaluation. IEEE Software, 12(4), Pages: 52-62, 1995.

[KUN00] Kung David C, Liu Chien-Hung, Hsia Pei, An Object-Oriented Web Test Model for Testing Web Applications, Proceedings of the First Asia-Pacific Conference on Quality Software,Pages: 111 - 120, 30-31 Oct., 2000.

[KUN04] Kung David, An Agent-Based Framework for Testing Web Applications, Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04), Volume: 2,Pages: 174 - 177, 28-30 Sept., 2004.

[LEE04] Lee SC, Shirani AI, A component based methodology for Web application development. Journal of Systems and Software, 71(1-2), Pages: 177-187, 2004.

[LEE93] Lee Y., Liang B., Wang, F., Some Complexity Metrics for Object-Oriented Programs Based on Information Flow, Proceedings: CompEuro, March, Pages: 302-310, 1993.

[LIN00] Lin Yuh-Chi and Hadingham Paul, Tracking Frequent Traversal Areas in a Web Site via Log Analysis, Seventh International Conference on Parallel and Distributed Systems,Pages: 321 - 325, 4-7 July 2000.

[LIN04] Linderman M & Fried J, Defensive Design for the Web, New Riders Press, March 2, 2004.

[LIU00] Liu C, Kung D C, Hsia P and Hsu C, Object-based Data Flow Testing of Web Applications, Proceedings of First Asia-Pacific Conference on Quality Software, IEEE Computer Society Press, Los Alamitos, CA, Pages: 7-16, 2000.

[LIU00a] Liu Chien-Hung, Kung David C., Hsia Pei and Hsu Chih-Tung, Structural Testing of Web Applications, Proceedings of 11th International Symposium on Software Reliability Engineering (ISSRE 2000), Page(s):84 - 96, 8-11 Oct. 2000.

[LOO] Looker Nik, Munro Malcolm, Xu Jie, Testing Web Services,

[LOR94] Lorenz, Mark & Kidd, Jeff, Object-Oriented Software Metrics, Prentice Hall Publishing, 1994.

[LOW99] Lowe D., Web Engineering or Web Gardening?, WebNet Journal, vol. I (1999), no. 2.

[LUC02] Lucca Giuseppe Antonio Di, Fasolino Anna Rita, Faralli Francesco and Carlini Ugo De, Testing Web Applications, Proceedings of the International Conference on Software Maintenance (ICSM 02), Page(s):310 - 319, 3-6 Oct., 2002.

[LUC03] Lucca Giuseppe Antonio Di and Penta Massimiliano Di, Considering Browser Interaction in Web Application Testing, Proceedings of the Fifth IEEE International Workshop on Web Site Evolution (WSE'03), Page(s):74 - 81, 22 Sept., 2003.

[MAD99] Madsen KH, Special Issue on the Diversity of Usability Practices, Journal of Communications of the ACM, 42(5), 1999.

[MCC76] McCabe T J, A Complexity Metric, IEEE Transactions on Software Engineering, SE-2(4), Pages: 308-320, December, 1976.

[MCC77] McCall J A. et al., Concepts & Definition of Software Quality, Factors in Software Quality, NTIS 1, 1977.

[MCC94] McCabe & Associates, McCabe Object-Oriented Tool User's Instructions, 1994.

[MCG01] McGough Jeff, Mortensen Jeff, Johnson Jerry and Fadali Sami, A web-based testing system with dynamic question generation, 31st ASEE/IEEE Frontiers in Education Conference, Page(s): S3C - 23-8 vol.3, October 10-13, 2001.

[MEI05] Mei Hong, Zhang Lu, A Framework for Testing Web Services and Its Supporting Tool, Proceedings of the 2005 IEEE International Workshop on Service-Oriented System Engineering (SOSE'05), 2005.

[MIA07] Miao Huaikou, Chen Shengbo, Qian Zhongsheng , A Formal Open Framework Based on Agent for Testing Web Applications, International Conference on Computational Intelligence and Security, Page(s):281 - 285, 15-19 Dec. 2007.

[MIL] Miller Edward, Web Site Testing.

[MUR01] Murugesan S and Deshpande Y, Web Engineering, Managing Diversity and Complexity of Web Application Development, Lecture Notes in Computer Science 2016, Springer Verlag, Heidelberg, 2001.

[MUR02] Murugesan S and Deshpande Y, Meeting the challenges of web application development: The Web Engineering Approach. Proceedings of the 24th International Conference on Software Engineering, Pages: 687-688, May 2002.

[MUR07] Murlewski Jan, Wojciechowski Jarosaw, Sakowicz Bartosz, Napieralski Andrzej, Comprehensive Approach to Web Applications Testing and Performance Analysis, 9th International Conference on The Experience of Designing and Applications of CAD Systems in Microelectronics (CADSM '07), Page(s):429 - 431, 19-24 Feb. 2007.

[NGU00] Nguyen H Q, Testing Applications on the Web: Test Planning for Internet-Based Systems, John Wiley & Sons, NY, 2000.

[NGU01] Nguyen Hung Q, Testing Applications on the Web: Test Planning for Internet-Based Systems, Wiley, New York, 2001.

[NIE92] Nielsen J, The Usability Engineering Lifecycle, IEEE Computer, 25(3), Pages: 12-22, 1992.

[NIE93] Nielsen J and Landauer TK, A Mathematical Model of the Finding of Usability Problems, Proceedings of the ACM International Conference on Human Factors in Computing Systems, Amsterdam, Netherlands, April, Pages: 296-213, 1993.

[NIE95] Nielsen J, Usability Engineering, Academic Press, Cambridge, MA, 1995.

[NIE95a] Nielsen J, Multimedia and Hypertext Internet and Beyond, Academic Press, London, 1995.

[NTA88] Ntafos S C, A comparison of some structural testing strategies, IEEE Transactions on Software Engineering, 14(6), Pages: 868-874, June 1988.

[NUO06] Nuo Li, Qin-qin Ma, Ji Wu, Mao-zhong Jin, Chao Liu, A Framework of Model-Driven Web Application Testing, Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06), Volume 2, Page(s): 157 - 162, Sept. 2006.

[OFF02] Offutt Jeff, Quality Attributes of Web Software Applications, IEEE Software: Special Issue on Software Engineering of Internet Software, Vol. 19(2), Pages: 25-32, March/April 2002.

[OYA05] Oyama K., Ishikawa H., Eguchi K., Aizawa A., Analysis of Topics and Relevant Documents for Navigational Retrieval on the Web, Proceedings of the 2005 International Workshop on Challenges in Web Information Retrieval and Integration (WIRI'05), Page(s):157 - 163, 8-9 April 2005.

[PFL97] Pfleeger S L, Jeffery R, Curtis B and Kitchenham B, Status report on software measurement. IEEE Software, 14(2), Pages: 33-43, March/April 1997.

[PRA06] Prasad K V K K, Software Testing Tools with case studies, Dreamtech Press, Reprint Edition, 2006.

[PRE01] Pressman Roger S, Software Engineering: A practitioner's approach - 5th edition, McGraw-Hill International Edition, 2001.

[PRE98] Pressman R S, Can Internet-based applications be engineered?, IEEE Software, 15(5), Pages: 104-110, September/October 1998.

[RAP85] Rapps S & Weyuker E J, Selecting Software test data using data flow information, IEEE Transactions on Software Engineering, 11(4), Pages: 367-375, April 1985.

[REI00] Reifer DJ, Web development: estimating quick-to-market software. IEEE Software, Pages: 57-64, November/December 2000.

[RIC01] Ricca F, Tonella P, Analysis and testing of Web applications. In: Proceedings of the 23rd International Conference on Software Engineering, IEEE Computer Society Press, Los Alamitos CA, Pages: 25-34, 2001.

[RIC04] Ricca F and Tonella P, A 2-Layer Model for the White-Box Testing of Web Applications, Proceedings of Sixth IEEE Workshop on Web Site Evolution, IEEE Computer Society Press, Los Alamitos, CA, Pages: 11-19, 2004.

[RIC05] Ricca Filippo and Tonella Paolo, Anomaly detection in Web applications: A review of already conducted case studies, Proceedings of the Ninth European Conference on Software Maintenance and Reengineering (CSMR'05), Page(s): 385 - 394, 21-23 March 2005.

[RIC05a] Ricca Filippo and Tonella Paolo, Web Testing: a Roadmap for the Empirical Research, Proceedings of the 2005 Seventh IEEE International Symposium on Web Site Evolution (WSE'05). Page(s): 63 - 70, 26 Sept., 2005

[SAM04] Sampath S, Mihaylov V, Souter A and Pollock L, A Scalable approach to user-session based testing of Web Applications through Concept Analysis, Proceedings of 19th International Conference on Automated Software Engineering, IEEE Computer Society Press: Los Alamitos, CA, Pages: 132-141, 2004.

[SAM04a] Sampath S, Mihaylov V, Souter A and Pollock L, Composing a framework to automate testing of operational Web-based software, Proceedings of 20th International Conference on Software Maintenance IEEE Computer Society Press, Pages: 104-113, 2004.

[SAM99] Samaroo A, Allott S and Hambling B, Effective Testing for E-Commerce,, 1999.

[SCH01] Scholtz Jean, Adaptation of Traditional Usability Testing Methods for Remote Testing, Proceedings of the 34th Annual Hawaii International Conference on System Sciences, Pages: 8, 3-6 Jan., 2001.

[SCH02] Schwabe D, Guimaraes RM and Rossi G, Cohesive Design of Personalized Web Applications, IEEE Internet Computing. 6(2), Pages: 34-43, 2002.

[SHA93] Sharble, Robert, & Cohen, Samuel, The Object-Oriented Brewery: A Comparison of Two Object-Oriented Development Methods, Software Engineering Notes, Vol 18, No 2., Pages: 60-73, 1993.

[SHE03] Shea D, The Zen of CSS Design: Visual Enlightenment for the Web, Voices That Matter, 2003.

[SIB05] Siblini Reda, Mansour Nashat, Testing Web Services, ACS / IEEE International Conference on Computer Systems & Applications, 2005, Pages: 135-139.

[SIN03] Singh V et al, A simulator for planning web site testing, The Journal of The Computer Society of India, vol. 33, No. 2, Apr-Jun 2003.

[SIN04] Singh V, Issues in Website Testing, Proceedings of TCTE '04, SHSLCIET, Longowal, October 15-16, 2004.

[SIN04a] Singh V etc al, A Simulator for Planning Website Development, Proceedings of SEPP'04, TIET, Patiala, March 5-6, 2004.

[SNE04] Sneed Harry M., Testing a Web Application, Proceedings of the Sixth IEEE International Workshop on Web Site Evolution (WSE'04), 11 Sept. 2004.

[SNE05] Sneed Harry M., Testing an E-Government Web Site, Proceedings of the 2005 Seventh IEEE International Symposium on Web Site Evolution (WSE'05), Page(s):3, 26 Sept. 2005.

[SNE06] Sneed Harry M, Huang Shihong, WSDLTest - A Tool for Testing Web Services, Eighth IEEE International Symposium on Web Site Evolution (WSE'06), Page(s):14 - 21, Sept. 2006.

[SPR07] Sprenkle Sara, Pollock Lori, Esquivel Holly, Hazelwood Barbara, Ecott Stacey, Automated Oracle Comparators for Testing Web Applications, 18th IEEE International Symposium on Software Reliability Engineering (ISSRE '07), Page(s):117 - 126, 5-9 Nov. 2007.

[STA02] Standing C, Methodologies for developing Web applications. Information and Software Technology, 44(3):151-160, 2002.

[SUM02] Summerville Ian, Software Engineering, 7th edition, Pearson Education, 2002.

[TAU07] Tauhida Parveen, Scott Tilley, George Gonzalez, On the Need for Teaching Web Application Testing, 9th IEEE International Workshop on Web Site Evolution (WSE 2007), Page(s):51 - 55, 5-6 Oct. 2007.

[TAY02] Taylor MJ, McWilliam J, Forsyth H, Wade S, Methodologies and web-site development: A survey of Practice, Information and Software Technology, 44(6), Pages: 381-391, 2002.

[TIA03] Tian Jeff, Ma Li, Li Zhao and Koru A. Gunes, A Hierarchical Strategy for Testing Web-Based Applications and Ensuring Their Reliability, Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC'03), Pages: 702 - 707, 3-6 Nov., 2003.

[TOR07] Torkey F A, Keshk Arabi, Hamza Taher, Ibrahim Amal, A New Methodology for Web Testing, ITI 5th International Conference on Information and Communications Technology ICICT 2007, Page(s):77 - 83, 16-18 Dec. 2007.

[VOR98] Vora P., Designing for the Web: A Survey, Interactions, Vol. 5(3), Pages: 13-30, 1998.

[WAE00] Waes Luuk Van, Thinking Aloud as a Method for Testing the Usability of Websites: The Influence of Task Variation on the Evaluation of Hypertext, IEEE Transactions On Professional Communication, Vol. 43, No. 3, September, 2000.

[WAN07] Wang Yongbo, Bai Xiaoying, Li Juanzi, Huang Ruobo, Ontology-Based Test Case Generation for Testing Web Services, Eighth International Symposium on Autonomous Decentralized Systems (ISADS'07), Page(s):43 - 50, 21-23 March 2007.

[WAR94] Warren PJ, Boldyreff C, Munro M, Characterising Evolution in Web Sites, Proceedings of WSE99, 1st Annual Workshop on Web Site Evolution, Pages: 46-48, 1994.

[WAR99] Warren, PJ, Boldyreff C, Munro M, The Evolution of Websites, International Workshop on Program Comprehension, IEEE Computer Society Press, 1999.

[WCA05] Web Content Accessibility Guidelines 2.0, WCAG20 (accessed 2005).

[WHI97] White MD, Abels EG, Hahn K, Identifying user-based criteria for Web pages, Internet Research, Vol. 7, No. 4, Pages: 252-262, 1997.

[WIL00] Wild C and Seber G, Chance Encounters: A First Course in Data Analysis and Inference, John Wiley & Sons, New York, 2000.

[WOO] Woodward, M.R. and Hennell, M.A, "On the Relationship between Two Control-Flow Coverage Criteria: All JJ-Paths and MCDC", Information and Software Technology , ISSN: 0950-5849.

[XU03] Xu Lei, Xu Baowen, Chen Zhenqiang, Jiang Jixiang and Chen Huowang, Regression testing for Web applications based on slicing, Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC'03), Pages: 652 - 656, 3-6 Nov., 2003.

[XU03a] Xu Lei, Xu Baowen, Chen Zhenqiang, Chen Huowang, Website Evolution Based on Statistic Data, Page(s):301 - 306, 28-30 May 2003.

[XU04] Xu Lei, Xu Baowen, A Framework for Web Applications Testing, Proceedings of the 2004 International Conference on Cyber worlds (CW'04), Pages : 300-305, 18-20 Nov 2004.

[XU05] Xu Wuzhi, Offutt Jeff and Luo Juan, Testing Web Services by XML Perturbation, Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05), Page(s):10, 8-11 Nov. 2005.

[YAU87] Yau S S and Kishimoto Z, A method for revalidating modified programs in the maintenance phase, Proceedings of the Eleventh Annual International Computer Software and Applications Conference, pages 272-277, October 1987.

[YU07] Yu Ying, Huang Ning, Luo Qiuzhong , OWL-S Based Interaction Testing of Web Service-based System, Third International Conference on Next Generation Web Services Practices (NWeSP 2007), Page(s):31 - 34, 29-31 Oct. 2007.

[ZEI96] Zeist VanRHJand Hendriks P R H, Specifying Software Quality with the Extended ISO model, Software Quality Journal, Volume 5, No 4, Page(s): 273-284, Dec 1996.

[ZHU04] Zhu Hong, Cooperative Agent Approach to Quality Assurance and Testing Web Software, Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04), Volume: 2,Pages: 110 - 113, 28-30 Sept. 2004.

[ZHU06] Zhu Hong, A Framework for Service-Oriented Testing of Web Services, Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC'06), Volume 2,Page(s):145 - 150, Sept. 2006.