Benefits and Limitations of Automated Software Testing: Literature Review and Survey

3367 words (13 pages) Essay in Information Technology

18/05/20 Information Technology Reference this

Disclaimer: This work has been submitted by a student. This is not an example of the work produced by our Essay Writing Service. You can view samples of our professional work here.

Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of UK Essays.





Automatic testing enables quicker iterations, more reliable test outcomes and finally better products. Automated testing tools become increasingly frequent and easily accessible. Two contributions are made to this study. First, conduct a systemic evaluation of advantages and constraints in scholarly literature in software test automation. Secondly the practitioner conducts an investigation into the advantages and constraints of software testing automation. This research tries to close the gap by examining both the benefits of test automation and its constraints. A systematic literature review examines scholarly opinions and evaluates the opinions of professionals through an enquiry. Academic opinions are explored by a systemic literary examination, while the opinions of practitioners are evaluated by a study in which 115 software experts have obtained replies. In addition, advantages were often identified from stronger evidence sources, while constraints often stem from reports of experience.

  Table of Contents


Table of Contents

List of Tables

List of Figures

Benefits and limitations of automated software testing: Systematic literature review and practitioner survey








List of Tables

Table 1: Steps

Table 2: Source of Evidence

Table 3: System Type

Table 4: Survey Results for Benefits

Table 5: Survey Results For Limitations

Table 6: Overall Satisfaction with AST

List of Figures


Figure 1: Benefits of Automated Testing

Figure 2: Limitations of Automated Testing

Benefits and limitations of automated software testing: Systematic literature review and practitioner survey

Kai Petersen, Mika Mäntylä


Automated tests are a sophisticated field of research from a research perspective. Automated tests are aimed at 100% automation (Bertolino 2007). This vision has not, however, been achieved in practice (Berner, Weber and Keller, 2005). Today, top businesses use automated testing to boost product service life, decrease expensive and recurring development and enhance the quality of iteration. Developers and technicians use automated tests to overcome this deficiency, decrease the time required for test creation and eventually save money. The advantages of automatic testing while manual testing is continuous at an overall reduced price are simpler to leverage than before.

Get Help With Your Essay

If you need assistance with writing your essay, our professional essay writing service is here to help!

Find out more

This report provides a concise overview to automated software tests. The research consists of a number of parts. Section 2 contains the study objective; Section 3 provides an explanation for the literature review; Section 4 details the advantages and constraints of automated testing; Section 5 provides study findings and Section 6 includes results of the study. The references are provided in the last Section.


The study aims at finding the question: Are the advantages and constraints of empirical studies and reports of experience found throughout the industry?


The systemic literature evaluation is used to identify the benefits and limitations of AST (Kitchenham and Charters, 2007). Over the past decade, the study concentrated on the era 1999-2011 when testing instruments were developed and strengthened, thereby influencing assessment of constraints and benefits. As a search database, we have used IEEE Explore, Engineering Village (including Inspec and Compendex), Scopus, ACM and Google Scholar. A total of 24,706 papers led from the search. The following steps were taken for the choice of papers as illustrated in Table 1. To determine if reviewers are aware of the inclusion and exclusion criteria, the research choice based on titles and abstracts was carried out using a test set of 50 papers. Kappa k was calculated (a measure for the contract between the examiners (Berry and Mielke, 1988) and k is 0.605 which show an important agreement has been reached. This value was calculated. This therefore means that the criteria for inclusion / exclusion have been obviously defined.

The review was analyzed with thematic assessment and narrative summaries. Table 2 shows which proof sources have been used. Most trials are of an empirical nature, it is obvious.

Table 1: Steps


No. of articles (in → out)

Duplicate removal

24706 → 19920

Title check

19920 → 9456

Abstract check

9456 → 1470

Introduction and conclusion check

1470 → 227

Full-text Reading

227 → 25

Table 2: Source of Evidence



No. of articles


(Saglietti and Pinte 2010), (Al Dallal,  2009), (Tan and Edwards, 2008), ( Alshraideh, 2008), (du Bousquet and Zuanon, 1999), (Malekzadeh and Ainon, 2010), ((Leitner et al., 2007), (Kansomkeat and Rivepiboon, 2003), (Burnim and Sen, 2008) and (Choi and Lee, 2006).


Industrial Case

Haugset and Hanssen, 2008), ( Liu, 2000), ( Hao et al., 2009), (Bashir and Banuri,  2008), (Shan and Zhu, 2009), (Karhu et al., 2009) and (Wissink and Amaro, 2006).



(Perry and Rice,  2013), (Pocatilu, 2002), (Pettichord, 1999), (Fecko and Lott, 2002), (Persson and Yilmazturk, 2004) and (Fewster  and Graham, 1999).


The literature review demonstrates certain advantages and restrictions of automated testing. The advantages and constraints recognized in the literature review are shown in Figure 1 and Figure 2.

Following are the benefits of automated testing:

  • Enhanced service quality: quality for fewer defects in a software product
  • Coverage testing: high code coverage (i.e. declaration, branch, route) is accomplished via automation
  • Decreased test time: time to test, i.e. capacity to carry out more trials in a period of time
  • Reliability: AST is more accurate if studies are repeated because the manual testing tester does not take advantage of the tester’s expertise.
  • Increased trust: increase trust in the system quality (e.g. as designers perceive)
  • Reusability of tests: when tests are intended with primary tenure in mind, they can be repeated often, there are advantages to a large level of repeat test instances, not to a single test case.
  • Less human effort: Automation decreases human effort to be used in other (in specific, deficiency avoidance) operations.
  • Cost reduction: saves with a high degree of automation costs
  • Enhanced fault detection: efficiency, because of a strong detection capability, to detect a big part of the failures of a system.

Following are the limitation of automated testing:

  • Automation can’t fully replace manual tests. Not all test tasks can be readily automated, particularly those requiring comprehensive domain understanding.
  • The failure to attain the anticipated objectives. Organizations are tented in a fraction of a moment by running tests, but they fail to attain sustainable or true advantages.
  • Changes in software product technology and developments lead to difficulty in keeping automated testing.
  • Test automation needs time to ripen. Creation of facilities and automation testing takes time and therefore automation maturity (and associated advantages) is time-consuming.
  • In order to save the maximum possible costs (e.g., expense on improductive testing operations), organizations have impracticable expectations with regard to AST.
  • An suitable approach (e.g. what level of testing is difficult to decide for what purpose) leads inappropriate strategies that fail to make use of AST’s advantages
  • Failure of qualified individuals. Many abilities are required to automate trials (for example, experimental instruments, general software, domain and system skills).

Figure 1: Benefits of Automated Testing

Figure 2: Limitations of Automated Testing


The research was conducted to determine whether the benefits and restrictions are applicable for the software industry in particular. The internet research was spread by AST, Yahoo Groups, Google Groups and LinkedIn. The survey has also been e-mailed to corporate contacts. There are a total of 115 valid responses. The results show that all sectors, most of which cover internet, finance and healthcare, are covered. The system type demonstrates in Table 3.

Table 3: System Type

System type

Total Answers























Embedded system









5.      RESULTS

In Table 4, the advantages of AST are shown as they were in the survey and the advantages identified in the systemic literature review are referred to to illustrate the benefits of the survey.

Table 4: Survey Results for Benefits

5=completely agree, 4=agree, 3=neutral, 2=disagree, 1=completely disagree

  1. B.R1: Overall, 86% of participants agree entirely that the test instances are highly reusable.
  2. B.R2: A total of 84% agree on this advantage. However, this should not be tested. It was stressed.
  3. B.R3: The beneficial relationship between automation, which results in improved test coverage, is 75% agreed.
  4. B.R4: This issue concerns re-run tests and saving time and expense, which 72 percent of participants agree with when comparing the manual test.
  5. B.R5: 66% of the participants agreed that AST should be associated with greater confidence in product quality and the capacity to meet schedules.
  6. B.R6: Total 66% agreed, but the practitioners gave no particular clarifications.
  7. B.R7: The total automation lowers the test cost and facilitates continuous testing at the same time, 64 percent of respondents agree.
  8. B.R8: Total 60 percent agree to the declaration. At the same moment, the participants disagree with a significant proportion (24 percent).
  9. B.R9: Just a few (33 percent) agree that automated testing makes elevated error detection possible.

 Table 5 show The restrictions are described in the same way as the advantages.

Table 5: Survey Results For Limitations

5=completely agree, 4=agree, 3=neutral, 2=disagree, 1=completely disagree

  1. L.R1: 89 percent of participants recognize the limit..
  2. L.R2: 88 percent of participants are very supportive of this declaration.
  3. L.R3: 81 percent agree on these challenges when it comes to effective automation and the necessary abilities.
  4. L.R4: 77 percent agree that AST involves elevated investment in the purchase of instruments and in training employees for instruments compared with manual testing.
  5. L.R5: This statement is also given here because it raises the disadvantage that complex defects cannot be found.
  6. L.R6: 45% agree that market-based testing instruments are inconsistent.
  7. L.R7: The response to this issue demonstrates obviously that the test testers (only 6 percent of participants) do not think in complete automation and that therefore remains an obstacle.

The general satisfaction is heavily related to unreasonable expectations as if the testers are not happy with using automation; they are mainly disappointed with expectations. Table 6 indicates a clear and extremely satisfied majority of participants.

Table 6: Overall Satisfaction with AST


Automated testing enables quicker iteration, better test outcomes and, eventually, greater quality products. It can be time consuming and expensive to invest in the original configuration for automated testing. Automated instruments are increasingly popular and easily accessible. Automated testing includes more time for planning and growth than manual testing.

Find out how can help you!

Our academic experts are ready and waiting to assist with any writing project you may have. From simple essay plans, through to full dissertations, you can guarantee we have a service perfectly matched to your needs.

View our services

Two contributions are made to this study. Firstly, a systemic examination of the advantages and constraints in scholarly literature in software tests automation. Secondly, perform a practitioner study of the advantages and constraints of software test automation.

Cite This Work

To export a reference to this article please select a referencing style below:

Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.
Reference Copied to Clipboard.

Related Services

View all

DMCA / Removal Request

If you are the original writer of this essay and no longer wish to have the essay published on the UK Essays website then please:

Related Lectures

Study for free with our range of university lectures!