1128:
590:
1095:
1284:
1498:
helps to prevent false or misleading information being posted. It still however allows others to post new and relevant information to existing posts, and so increases the usefulness and relevance of the site. This said, no system is perfect, and it may not perform to this standard on all subjects all the time but will improve with increasing input and increasing use.
1051:. A good candidate for test automation is a test case for common flow of an application, as it is required to be executed (regression testing) every time an enhancement is made in the application. Test automation reduces the effort associated with manual testing. Manual effort is needed to develop and maintain automated checks, as well as reviewing test results.
1473: – including both the data and functionality to use – from the prescription of the way the test cases are executed. As a result, it separates the test creation process into two distinct stages: a design and development stage, and an execution stage. The design substage covers the
1403:
to describe testing done using a table of conditions directly as test inputs and verifiable outputs as well as the process where test environment settings and control are not hard-coded. In the simplest form the tester supplies the inputs from a row in the table and expects the outputs which occur in
910:
What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team must make. A multi-vocal literature review of 52 practitioner and 26 academic sources found that five main factors to consider in test automation decision are:
744:
Once automated tests have been developed, they can be run quickly and repeatedly many times. This can be a cost-effective method for regression testing of software products that have a long maintenance life. Even minor patches over the lifetime of the application can cause existing features to break
1334:
Tools are specifically designed to target some particular test environment, such as
Windows and web automation tools, etc. Tools serve as a driving agent for an automation process. However, an automation framework is not a tool to perform a specific task, but rather infrastructure that provides the
866:
the code is written. However, these unit tests evolve and are extended as coding progresses, issues are discovered and the code is subjected to refactoring. Only when all the tests for all the demanded features pass is the code considered complete. Proponents argue that it produces software that is
832:
Another variation of this type of test automation tool is for testing mobile applications. This is very useful given the number of different sizes, resolutions, and operating systems used on mobile phones. For this variation, a framework is used in order to instantiate actions on the mobile device
724:
through use of a model of the system for test case generation, but research continues into a variety of alternative methodologies for doing so. In some cases, the model-based approach enables non-technical users to create automated business test cases in plain
English so that no programming of any
1497:
is what most frameworks evolve/develop into over time and multiple projects. The most successful automation frameworks generally accommodate both grammar and spelling, as well as information input. This allows information given to be cross-checked against existing and confirmed information. This
1158:
A test automation framework is an integrated system that sets the rules of automation of a specific product. This system integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks which need to be assembled to
658:
separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be
896:
is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. For
Continuous Testing, the scope of testing extends from validating bottom-up requirements or user stories to
810:. However, reliance on these features poses major reliability and maintainability problems. Relabelling a button or moving it to another part of the window may require the test to be re-recorded. Record and playback also often adds irrelevant activities or incorrectly records some activities.
1279:
of application under test. The goal of Test
Automation Interface is to simplify the process of mapping tests to business criteria without coding coming in the way of the process. Test automation interface are expected to improve the efficiency and flexibility of maintaining test scripts.
911:
1) System Under Test (SUT), 2) the types and numbers of tests, 3) test-tool, 4) human and organizational topics, and 5) cross-cutting factors. The most frequent individual factors identified in the study were: need for regression testing, economic factors, and maturity of SUT.
836:
Another variation is script-less test automation that does not use record and playback, but instead builds a model of the application and then enables the tester to create test cases by simply inserting test parameters and conditions, which requires no scripting skills.
801:
Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. The advantage of this approach is that it requires little or no
762:
is also being widely used by software testers as it enables them to verify requirements independent of their GUI implementation, commonly to test them earlier in development, and to make sure the test itself adheres to clean code principles, especially the
1185:
Choosing the right framework/scripting technique helps in maintaining lower costs. The costs associated with test scripting are due to development and maintenance efforts. The approach of scripting used during test automation has effect on costs.
1436:
requires the creation of small, independent scripts that represent modules, sections, and functions of the application-under-test. These small scripts are then used in a hierarchical fashion to construct larger tests, realizing a particular test
741:) can be laborious and time-consuming to do manually. In addition, a manual approach might not always be effective in finding certain classes of defects. Test automation offers a possibility to perform these types of testing effectively.
867:
both more reliable and less costly than code that is tested by manual exploration. It is considered more reliable because the code coverage is better, and because it is run constantly during development rather than once at the end of a
1085:
A strategy to decide the amount of tests to automate is the test automation pyramid. This strategy suggests to write three types of tests with different granularity. The higher the level, less is the amount of tests to write.
1117:
which has fewer tests due to the different attributes that make it more complex to run, for example the fragility of the tests, where a small change in the user interface can break a lot of tests and adds maintenance
696:. A testing framework that uses a programming interface to the application to validate the behaviour under test. Typically API driven testing bypasses application user interface altogether. It can also be testing
1146:
Integration tests: These tests check how different units of code work together. Although individual units may function properly on their own, integration tests ensure that they operate together coherently.
1107:
provides robustness to the software products. Testing individual parts of the code makes it easy to write and run the tests. Developers write unit tests as a part of each story and integrate them with CI.
1110:
The service layer refers to testing the services of an application separately from its user interface, these services are anything that the application does in response to some input or set of inputs.
927:
may be a better investment. This ambiguity once again leads to the conclusion that the decision on test automation should be made individually, keeping in mind project requirements and peculiarities.
1047:
Test automation tools can be expensive and are usually employed in combination with manual testing. Test automation can be made cost-effective in the long term, especially when used repeatedly in
1314:
and a test runner. The parser is present to parse the object files coming from the object repository into the test specific scripting language. The test runner executes the test scripts using a
813:
A variation on this type of tool is for testing of web sites. Here, the "interface" is the web page. However, such a framework utilizes entirely different techniques because it is rendering
1691:
Proceedings from the 5th
International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. "Test Design: Lessons Learned and Practical Implications
571:
1664:
1404:
the same row. The table typically contains values which correspond to boundary or partition input spaces. In the control methodology, test configuration is "read" from a database.
935:
Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with
688:
events such as keystrokes and mouse clicks, and observes the changes that result in the user interface, to validate that the observable behavior of the program is correct.
525:
919:
While the reusability of automated tests is valued by software development companies, this property can also be viewed as a disadvantage. It leads to the so-called
1143:
Unit tests: These are tests that test individual components or units of code in isolation. They are fast, reliable, and isolate failures to small units of code.
492:
1139:'s testing blog, unit tests should make up the majority of your testing strategy, with fewer integration tests and only a small amount of end-to-end tests.
2306:
564:
1666:
Systems, Software and
Services Process Improvement: 22nd European Conference, EuroSPI 2015, Ankara, Turkey, September 30 -- October 2, 2015. Proceedings
1077:
that are a part of it. Some test automation tools allow for test authoring to be done by keywords instead of coding, which do not require programming.
1073:
person must have software coding ability since the test cases are written in the form of source code which when run produce output according to the
334:
1844:
482:
2235:
2210:
2192:
2173:
2154:
2121:
1932:
1706:
1674:
1647:
557:
457:
198:
1326:
Object repositories are a collection of UI/Application object data recorded by the testing tool while exploring the application under test.
477:
2111:
700:
to classes, modules or libraries are tested with a variety of input arguments to validate that the results that are returned are correct.
515:
1149:
End-to-end tests: These test the system as a whole, simulating real-world usage scenarios. They are the slowest and most complex tests.
2254:
999:
424:
188:
1721:
1517:
1832:
633:
611:
291:
2043:
1335:
solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer.
2556:
1891:
Garousi, Vahid; Mäntylä, Mika V. (2016-08-01). "When and what to automate in software testing? A multi-vocal literature review".
871:
development cycle. The developer discovers defects immediately upon making a change, when it is least expensive to fix. Finally,
796:
680:
414:
409:
165:
1870:
1584:
1338:
There are various types of frameworks. They are categorized on the basis of the automation component they leverage. These are:
764:
543:
2299:
434:
147:
127:
879:, but equivalent behavior, is much less likely to introduce new defects when the refactored code is covered by unit tests.
1429:
344:
233:
223:
173:
1663:
O’Connor, Rory V.; Akkaya, Mariye Umay; Kemaneci, Kerem; Yilmaz, Murat; Poth, Alexander; Messnarz, Richard (2015-10-15).
2617:
2581:
2520:
1159:
represent a business process. The framework provides the basis of test automation and simplifies the automation effort.
520:
248:
213:
64:
1733:
1542:(SUT), or to represent testing strategies and a test environment. The picture on the right depicts the former approach.
2092:
1559:
1553:
1370:
775:, to determine if they meet expectations for functionality, reliability, performance, and security. Since APIs lack a
339:
311:
2270:
1987:
1070:
855:
462:
306:
193:
183:
122:
2612:
2525:
2292:
1589:
1421:
1415:
1347:
228:
208:
2566:
2515:
2485:
2427:
807:
776:
685:
604:
598:
369:
238:
218:
1182:
will remain the same. Ideally, there is no need to update the scripts in case of changes to the application.
859:
780:
497:
379:
258:
132:
1196:
Structured (uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements)
1166:
of assumptions, concepts and tools that provide support for automated software testing is the low cost for
1454:
1448:
1352:
1205:
1135:
One conception of the testing pyramid contains unit, integration, and end-to-end unit tests. According to
1010:
923:, where repeatedly executed scripts stop detecting errors that go beyond their frameworks. In such cases,
615:
439:
349:
301:
243:
2445:
2440:
1074:
296:
263:
79:
69:
1948:
1786:
725:
kind is needed in order to configure them for multiple operating systems, browsers, and smart devices.
2490:
2480:
1474:
1167:
803:
359:
203:
137:
104:
84:
45:
675:
There are many approaches to test automation, however below are the general approaches used widely:
2475:
2470:
2460:
2404:
2349:
2339:
1523:
1509:
1389:
1383:
1362:
1342:
1199:
1175:
851:
826:
772:
721:
715:
660:
354:
273:
99:
2263:
Hayes, Linda G., "Automated
Testing Handbook", Software Testing Institute, 2nd Edition, March 2004
2530:
2505:
2465:
2455:
2399:
1815:
1594:
1527:
1433:
1163:
1048:
946:
893:
888:
738:
664:
472:
1971:
1457:, also known as action word based testing (not to be confused with action driven testing), is a
1263:
describe tests that need to be run on the program to verify that the program runs as expected.
2540:
2366:
2334:
2329:
2250:
2231:
2206:
2188:
2169:
2150:
2117:
1928:
1856:
1702:
1670:
1643:
1539:
1466:
1094:
1030:
784:
1762:
1193:
Linear (procedural code, possibly generated by tools like those that use record and playback)
2510:
2495:
2450:
2376:
2361:
2354:
2315:
2223:
1900:
1807:
1694:
1609:
1604:
1563:
1531:
1458:
1393:
1310:
Interface engines are built on top of
Interface Environment. Interface engine consists of a
950:
876:
872:
822:
734:
697:
647:
429:
392:
374:
364:
89:
1283:
2274:
977:
968:
868:
316:
268:
152:
74:
1798:
Vodde, Bas; Koskela, Lasse (2007). "Learning Test-Driven
Development by Counting Lines".
939:), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
875:
is safer when unit testing is used; transforming the code into a simpler form with less
2586:
2561:
2500:
2409:
2381:
2050:
1614:
1535:
1494:
1488:
1462:
1357:
1276:
1179:
1017:
924:
920:
862:(TDD) or test-first development. Unit tests can be written to define the functionality
1255:) that allow the execution of unit tests to determine whether various sections of the
2606:
2219:
AST '08: Proceedings of the 3rd
International Workshop on Automation of Software Test
2143:
1066:
1060:
59:
1845:
The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola
1819:
783:. API testing is considered critical when an API serves as the primary interface to
2576:
2571:
2414:
2344:
1567:
1315:
1240:
1202:(data is persisted outside of tests in a database, spreadsheet, or other mechanism)
1104:
995:
142:
1009:
Support unattended test runs for integration with build processes and batch runs.
2217:
1904:
1698:
2591:
2435:
2371:
2074:
1571:
942:
One must keep satisfying popular requirements when thinking of test automation:
936:
759:
754:
692:
31:
1745:
2011:
1127:
1114:
991:
897:
assessing the system requirements associated with overarching business goals.
818:
17:
1774:
1619:
1470:
1272:
1260:
1171:
1003:
964:
94:
2012:"Selenium Meet-Up 4/20/2010 Elisabeth Hendrickson on Robot Framework 1of2"
1516:
990:
Common Driver (For example, in the Java development ecosystem, that means
2267:
1425:
1400:
1397:
1392:(DDT), also known as table-driven testing or parameterized testing, is a
957:
655:
467:
419:
404:
399:
2016:
1811:
1749:
1599:
1023:
1311:
1224:
creating a mechanism to hook into or drive the application under test
1136:
178:
2227:
1787:
https://www.devbridge.com/articles/automated-user-interface-testing/
2284:
1330:
Defining boundaries between automation framework and a testing tool
1640:
Automated Defect Prevention: Best Practices in Software Management
1291:
Test Automation Interface consists of the following core modules:
1282:
1252:
1248:
1244:
1126:
1093:
253:
1530:
for designing and optionally also executing artifacts to perform
1477:
and assessment and the data analysis, definition, and population.
906:
Factors to consider for the decision to implement test automation
1256:
850:
Test automation, mostly using unit testing, is a key feature of
814:
487:
2288:
1271:
Test automation interfaces are platforms that provide a single
984:
768:
583:
27:
Use of special software to control test execution and analysis
806:. This approach can be applied to any application that has a
1538:. Models can be used to represent the desired behavior of a
1275:
for incorporating multiple testing tools and frameworks for
1746:
Produce Better Software by Using a Layered Testing Strategy
1189:
Various framework/scripting techniques are generally used:
1859:, by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015
1763:
https://docs.travis-ci.com/user/gui-and-headless-browsers/
659:
difficult to do manually. Test automation is critical for
1833:
Part of the Pipeline: Why Continuous Testing Is Essential
1736:, by Cameron Philipp-Edmonds, Stickyminds August 19, 2014
1174:
then only the test case file needs to be updated and the
1734:
All About API Testing: An Interview with Jonathan Cooper
1002:). This enables tests to integrate with the developers'
1847:, by Cameron Philipp-Edmonds, Stickyminds December 2015
1022:
Support distributed execution environment (distributed
1239:
A growing trend in software development is the use of
1259:
are acting as expected under various circumstances.
1221:
defining the format in which to express expectations
30:"Automated QA" redirects here. For the company, see
2549:
2423:
2392:
2322:
2110:DESAI, SANDEEP; SRIVASTAVA, ABHISHEK (2016-01-30).
1724:, by Amy Reichert, SearchSoftwareQuality March 2015
1210:
Hybrid (two or more of the patterns above are used)
2142:
1835:, by Adam Auerbach, TechWell Insights August 2015
1722:Testing APIs protects applications and reputations
1642:. Wiley-IEEE Computer Society Press. p. 74.
1098:The test automation pyramid proposed by Mike Cohn
956:Data driven capability (Input Data, Output Data,
2044:"Conquest: Interface for Test Automation Design"
745:which were working at an earlier point in time.
720:One way to generate test cases automatically is
1857:DevOps: Are You Pushing Bugs to Clients Faster
2300:
1469:. This method separates the documentation of
1029:Distributed application support (distributed
737:tasks (such as extensive low-level interface
565:
8:
2113:SOFTWARE TESTING : A Practical Approach
2038:
2036:
2034:
1396:methodology that is used in the testing of
2307:
2293:
2285:
2183:Mark Fewster & Dorothy Graham (1999).
1775:http://phantomjs.org/headless-testing.html
1217:The Testing framework is responsible for:
572:
558:
36:
987:to be able to integrate with other tools)
634:Learn how and when to remove this message
2245:Mosley, Daniel J.; Posey, Bruce (2002).
1515:
1123:Unit, integration, and end-to-end levels
1090:Unit, service, and user interface levels
597:This article includes a list of general
2166:Implementing Automated Software Testing
1638:Kolawa, Adam; Huizinga, Dorota (2007).
1630:
44:
1988:"Just Say No to More End-to-End Tests"
1949:"Full Stack Testing by Gayathri Mohan"
833:and to gather results of the actions.
791:Graphical user interface (GUI) testing
526:Electrical and electronics engineering
2164:Elfriede Dustin; et al. (2009).
2141:Elfriede Dustin; et al. (1999).
1982:
1980:
1918:
1916:
1914:
1487:These paragraphs are an excerpt from
1447:These paragraphs are an excerpt from
983:Extensible & Customization (Open
684:. A testing framework that generates
7:
2247:Just Enough Software Test Automation
2116:(in Arabic). PHI Learning Pvt. Ltd.
829:are normally used for this purpose.
821:instead of operating system events.
1893:Information and Software Technology
1520:General model-based testing setting
1871:"When Should a Test Be Automated?"
1785:Automated User Interface Testing;
1552:This paragraph is an excerpt from
1414:This paragraph is an excerpt from
1382:This paragraph is an excerpt from
779:, API testing is performed at the
603:it lacks sufficient corresponding
25:
2205:Roman Savenkov Consulting, 2008,
1424:is a term used in the testing of
452:Standards and bodies of knowledge
2557:Graphical user interface testing
2268:Architectures of Test Automation
2203:How to Become a Software Tester.
1773:Headless Testing with PhantomJS;
1761:Headless Testing with Browsers;
1570:to describe the behavior of the
1508:This section is an excerpt from
1154:Framework approach in automation
797:Graphical user interface testing
681:Graphical user interface testing
588:
1585:Comparison of GUI testing tools
1287:Test Automation Interface Model
980:friendly – minimal binary files
767:. It involves directly testing
765:single responsibility principle
544:Outline of software development
2249:. Prentice Hall Professional.
2216:Hong Zhu; et al. (2008).
1461:methodology suitable for both
1:
2582:Software reliability testing
2521:Software performance testing
2187:. ACM Press/Addison-Wesley.
2075:"golang/go TableDrivenTests"
1905:10.1016/j.infsof.2016.04.015
1699:10.1109/IEEESTD.2008.4578383
1170:. If there is change to any
963:Customization Reporting (DB
1560:Behavior-driven development
1554:Behavior-driven development
1547:Behavior driven development
1371:Behavior driven development
1081:Testing at different levels
698:public (usually) interfaces
2634:
2424:Testing types, techniques,
2145:Automated Software Testing
1972:The Practical Test Pyramid
1551:
1507:
1486:
1446:
1413:
1381:
1277:System/Integration testing
1213:Agile automation framework
1071:software quality assurance
1065:In automated testing, the
1058:
974:Easy debugging and logging
886:
856:agile software development
794:
752:
713:
307:Software quality assurance
29:
1590:List of web testing tools
1422:Modularity-driven testing
1416:Modularity-driven testing
1409:Modularity-driven testing
1348:Modularity-driven testing
1267:Test automation interface
1247:frameworks (for example,
1113:At the top level we have
1016:Email Notifications like
2567:Orthogonal array testing
2516:Smoke testing (software)
2486:Dynamic program analysis
2185:Software Test Automation
1162:The main advantage of a
1131:Google's testing pyramid
808:graphical user interface
292:Configuration management
1243:frameworks such as the
1235:Unit testing frameworks
1103:As a solid foundation,
860:test-driven development
858:, where it is known as
846:Test-driven development
618:more precise citations.
516:Artificial intelligence
1562:(BDD) involves naming
1521:
1455:Keyword-driven testing
1449:Keyword-driven testing
1442:Keyword-driven testing
1353:Keyword-driven testing
1288:
1132:
1099:
1011:Continuous integration
825:or solutions based on
440:Infrastructure as code
286:Supporting disciplines
2446:Compatibility testing
1925:Succeeding with Agile
1748:, by Sean Kenefick,
1526:is an application of
1519:
1298:Interface Environment
1286:
1130:
1097:
1043:Test automation tools
1013:servers require this.
297:Deployment management
2491:Installation testing
2481:Differential testing
2093:"JUnit 5 User Guide"
1953:www.thoughtworks.com
1475:requirement analysis
804:software development
117:Paradigms and models
46:Software development
2618:Automation software
2476:Development testing
2471:Destructive testing
2461:Conformance testing
2405:Integration testing
2350:Model-based testing
2340:Exploratory testing
1992:Google Testing Blog
1524:Model-based testing
1510:Model-based testing
1503:Model-based testing
1390:Data-driven testing
1384:Data-driven testing
1377:Data-driven testing
1367:Code-driven testing
1363:Model-based testing
1343:Data-driven testing
1227:executing the tests
921:"Pesticide Paradox"
852:extreme programming
827:Selenium Web Driver
773:integration testing
722:model-based testing
716:Model-based testing
710:Model-based testing
661:continuous delivery
40:Part of a series on
2531:Symbolic execution
2506:Regression testing
2466:Continuous testing
2456:Concurrent testing
2400:Acceptance testing
2323:The "box" approach
2273:2021-01-26 at the
2168:. Addison Wesley.
2149:. Addison Wesley.
2134:General references
1923:Mike Cohn (2010).
1812:10.1109/ms.2007.80
1595:Continuous testing
1528:model-based design
1522:
1289:
1133:
1100:
1049:regression testing
894:Continuous testing
889:Continuous testing
883:Continuous testing
739:regression testing
729:Regression testing
693:API driven testing
671:General approaches
665:continuous testing
435:Release automation
312:Project management
2600:
2599:
2541:Usability testing
2367:White-box testing
2335:All-pairs testing
2330:Black-box testing
2237:978-1-60558-030-2
2211:978-0-615-23372-7
2194:978-0-201-33140-0
2175:978-0-321-58051-1
2156:978-0-201-43287-9
2123:978-81-203-5226-1
1934:978-0-321-57936-2
1927:. Raina Chrobak.
1873:. StickyMinds.com
1708:978-0-7381-5746-7
1676:978-3-319-24647-5
1649:978-0-470-04212-0
1540:system under test
1467:automated testing
1322:Object repository
1301:Object Repository
1230:reporting results
823:Headless browsers
817:and listening to
785:application logic
644:
643:
636:
582:
581:
473:ISO/IEC standards
16:(Redirected from
2625:
2613:Software testing
2511:Security testing
2496:Negative testing
2451:Concolic testing
2377:Mutation testing
2362:Grey-box testing
2355:Scenario testing
2316:Software testing
2309:
2302:
2295:
2286:
2260:
2241:
2201:Roman Savenkov:
2198:
2179:
2160:
2148:
2128:
2127:
2107:
2101:
2100:
2089:
2083:
2082:
2071:
2065:
2064:
2062:
2061:
2055:
2049:. Archived from
2048:
2040:
2029:
2028:
2026:
2025:
2008:
2002:
2001:
1999:
1998:
1984:
1975:
1969:
1963:
1962:
1960:
1959:
1945:
1939:
1938:
1920:
1909:
1908:
1888:
1882:
1881:
1879:
1878:
1866:
1860:
1854:
1848:
1842:
1836:
1830:
1824:
1823:
1795:
1789:
1783:
1777:
1771:
1765:
1759:
1753:
1743:
1737:
1731:
1725:
1719:
1713:
1712:
1687:
1681:
1680:
1660:
1654:
1653:
1635:
1610:Software testing
1605:Headless browser
1532:software testing
1459:software testing
1394:software testing
1306:Interface engine
1295:Interface Engine
998:and the popular
877:code duplication
873:code refactoring
735:software testing
705:Other approaches
648:software testing
639:
632:
628:
625:
619:
614:this article by
605:inline citations
592:
591:
584:
574:
567:
560:
521:Computer science
430:Build automation
37:
21:
2633:
2632:
2628:
2627:
2626:
2624:
2623:
2622:
2603:
2602:
2601:
2596:
2545:
2536:Test automation
2425:
2419:
2388:
2318:
2313:
2282:
2280:
2275:Wayback Machine
2257:
2244:
2238:
2228:10.1145/1370042
2215:
2195:
2182:
2176:
2163:
2157:
2140:
2136:
2131:
2124:
2109:
2108:
2104:
2091:
2090:
2086:
2073:
2072:
2068:
2059:
2057:
2053:
2046:
2042:
2041:
2032:
2023:
2021:
2020:. 28 April 2010
2010:
2009:
2005:
1996:
1994:
1986:
1985:
1978:
1970:
1966:
1957:
1955:
1947:
1946:
1942:
1935:
1922:
1921:
1912:
1890:
1889:
1885:
1876:
1874:
1868:
1867:
1863:
1855:
1851:
1843:
1839:
1831:
1827:
1797:
1796:
1792:
1784:
1780:
1772:
1768:
1760:
1756:
1752:January 7, 2014
1744:
1740:
1732:
1728:
1720:
1716:
1709:
1689:
1688:
1684:
1677:
1662:
1661:
1657:
1650:
1637:
1636:
1632:
1628:
1581:
1576:
1575:
1568:domain language
1557:
1549:
1544:
1543:
1513:
1505:
1500:
1499:
1492:
1484:
1479:
1478:
1452:
1444:
1439:
1438:
1419:
1411:
1406:
1405:
1387:
1379:
1332:
1324:
1308:
1269:
1237:
1156:
1125:
1092:
1083:
1063:
1057:
1045:
1040:
1018:bounce messages
978:Version control
969:Crystal Reports
933:
917:
908:
903:
891:
885:
848:
843:
799:
793:
757:
751:
731:
718:
712:
707:
673:
654:is the use of
652:test automation
640:
629:
623:
620:
610:Please help to
609:
593:
589:
578:
549:
548:
539:
531:
530:
511:
503:
502:
453:
445:
444:
395:
385:
384:
330:
322:
321:
317:User experience
287:
279:
278:
169:
158:
157:
118:
110:
109:
55:
54:Core activities
35:
28:
23:
22:
15:
12:
11:
5:
2631:
2629:
2621:
2620:
2615:
2605:
2604:
2598:
2597:
2595:
2594:
2589:
2587:Stress testing
2584:
2579:
2574:
2569:
2564:
2562:Manual testing
2559:
2553:
2551:
2547:
2546:
2544:
2543:
2538:
2533:
2528:
2526:Stress testing
2523:
2518:
2513:
2508:
2503:
2501:Random testing
2498:
2493:
2488:
2483:
2478:
2473:
2468:
2463:
2458:
2453:
2448:
2443:
2438:
2432:
2430:
2421:
2420:
2418:
2417:
2412:
2410:System testing
2407:
2402:
2396:
2394:
2393:Testing levels
2390:
2389:
2387:
2386:
2385:
2384:
2382:Static testing
2379:
2374:
2364:
2359:
2358:
2357:
2352:
2347:
2342:
2337:
2326:
2324:
2320:
2319:
2314:
2312:
2311:
2304:
2297:
2289:
2279:
2278:
2277:", August 2000
2264:
2261:
2256:978-0130084682
2255:
2242:
2236:
2213:
2199:
2193:
2180:
2174:
2161:
2155:
2137:
2135:
2132:
2130:
2129:
2122:
2102:
2084:
2066:
2030:
2003:
1976:
1974:, by Ham Vocke
1964:
1940:
1933:
1910:
1883:
1869:Brian Marick.
1861:
1849:
1837:
1825:
1790:
1778:
1766:
1754:
1738:
1726:
1714:
1707:
1682:
1675:
1655:
1648:
1629:
1627:
1624:
1623:
1622:
1617:
1615:System testing
1612:
1607:
1602:
1597:
1592:
1587:
1580:
1577:
1564:software tests
1558:
1550:
1548:
1545:
1536:system testing
1514:
1506:
1504:
1501:
1495:Hybrid testing
1493:
1489:Hybrid testing
1485:
1483:
1482:Hybrid testing
1480:
1453:
1445:
1443:
1440:
1420:
1412:
1410:
1407:
1388:
1380:
1378:
1375:
1374:
1373:
1368:
1365:
1360:
1358:Hybrid testing
1355:
1350:
1345:
1331:
1328:
1323:
1320:
1307:
1304:
1303:
1302:
1299:
1296:
1268:
1265:
1236:
1233:
1232:
1231:
1228:
1225:
1222:
1215:
1214:
1211:
1208:
1206:Keyword-driven
1203:
1197:
1194:
1180:startup script
1155:
1152:
1151:
1150:
1147:
1144:
1124:
1121:
1120:
1119:
1111:
1108:
1091:
1088:
1082:
1079:
1059:Main article:
1056:
1053:
1044:
1041:
1039:
1036:
1035:
1034:
1027:
1020:
1014:
1007:
988:
981:
975:
972:
961:
954:
932:
929:
925:manual testing
916:
915:Plateau effect
913:
907:
904:
902:
901:Considerations
899:
887:Main article:
884:
881:
847:
844:
842:
839:
795:Main article:
792:
789:
753:Main article:
750:
747:
730:
727:
714:Main article:
711:
708:
706:
703:
702:
701:
689:
686:user interface
672:
669:
642:
641:
596:
594:
587:
580:
579:
577:
576:
569:
562:
554:
551:
550:
547:
546:
540:
537:
536:
533:
532:
529:
528:
523:
518:
512:
509:
508:
505:
504:
501:
500:
495:
490:
485:
480:
475:
470:
465:
463:IEEE standards
460:
454:
451:
450:
447:
446:
443:
442:
437:
432:
427:
422:
417:
412:
407:
402:
396:
391:
390:
387:
386:
383:
382:
377:
372:
367:
362:
357:
352:
347:
342:
337:
331:
328:
327:
324:
323:
320:
319:
314:
309:
304:
299:
294:
288:
285:
284:
281:
280:
277:
276:
271:
266:
261:
256:
251:
246:
241:
236:
231:
226:
221:
216:
211:
206:
201:
196:
191:
186:
181:
176:
170:
168:and frameworks
164:
163:
160:
159:
156:
155:
150:
145:
140:
135:
130:
125:
119:
116:
115:
112:
111:
108:
107:
102:
97:
92:
87:
82:
77:
72:
67:
62:
56:
53:
52:
49:
48:
42:
41:
26:
24:
18:Automated test
14:
13:
10:
9:
6:
4:
3:
2:
2630:
2619:
2616:
2614:
2611:
2610:
2608:
2593:
2590:
2588:
2585:
2583:
2580:
2578:
2575:
2573:
2570:
2568:
2565:
2563:
2560:
2558:
2555:
2554:
2552:
2548:
2542:
2539:
2537:
2534:
2532:
2529:
2527:
2524:
2522:
2519:
2517:
2514:
2512:
2509:
2507:
2504:
2502:
2499:
2497:
2494:
2492:
2489:
2487:
2484:
2482:
2479:
2477:
2474:
2472:
2469:
2467:
2464:
2462:
2459:
2457:
2454:
2452:
2449:
2447:
2444:
2442:
2439:
2437:
2434:
2433:
2431:
2429:
2422:
2416:
2413:
2411:
2408:
2406:
2403:
2401:
2398:
2397:
2395:
2391:
2383:
2380:
2378:
2375:
2373:
2370:
2369:
2368:
2365:
2363:
2360:
2356:
2353:
2351:
2348:
2346:
2343:
2341:
2338:
2336:
2333:
2332:
2331:
2328:
2327:
2325:
2321:
2317:
2310:
2305:
2303:
2298:
2296:
2291:
2290:
2287:
2283:
2276:
2272:
2269:
2266:Kaner, Cem, "
2265:
2262:
2258:
2252:
2248:
2243:
2239:
2233:
2229:
2225:
2222:. ACM Press.
2221:
2220:
2214:
2212:
2208:
2204:
2200:
2196:
2190:
2186:
2181:
2177:
2171:
2167:
2162:
2158:
2152:
2147:
2146:
2139:
2138:
2133:
2125:
2119:
2115:
2114:
2106:
2103:
2098:
2094:
2088:
2085:
2080:
2076:
2070:
2067:
2056:on 2012-04-26
2052:
2045:
2039:
2037:
2035:
2031:
2019:
2018:
2013:
2007:
2004:
1993:
1989:
1983:
1981:
1977:
1973:
1968:
1965:
1954:
1950:
1944:
1941:
1936:
1930:
1926:
1919:
1917:
1915:
1911:
1906:
1902:
1898:
1894:
1887:
1884:
1872:
1865:
1862:
1858:
1853:
1850:
1846:
1841:
1838:
1834:
1829:
1826:
1821:
1817:
1813:
1809:
1805:
1801:
1800:IEEE Software
1794:
1791:
1788:
1782:
1779:
1776:
1770:
1767:
1764:
1758:
1755:
1751:
1747:
1742:
1739:
1735:
1730:
1727:
1723:
1718:
1715:
1710:
1704:
1700:
1696:
1692:
1686:
1683:
1678:
1672:
1668:
1667:
1659:
1656:
1651:
1645:
1641:
1634:
1631:
1625:
1621:
1618:
1616:
1613:
1611:
1608:
1606:
1603:
1601:
1598:
1596:
1593:
1591:
1588:
1586:
1583:
1582:
1578:
1573:
1569:
1565:
1561:
1555:
1546:
1541:
1537:
1533:
1529:
1525:
1518:
1511:
1502:
1496:
1490:
1481:
1476:
1472:
1468:
1464:
1460:
1456:
1450:
1441:
1435:
1431:
1427:
1423:
1417:
1408:
1402:
1399:
1395:
1391:
1385:
1376:
1372:
1369:
1366:
1364:
1361:
1359:
1356:
1354:
1351:
1349:
1346:
1344:
1341:
1340:
1339:
1336:
1329:
1327:
1321:
1319:
1317:
1313:
1305:
1300:
1297:
1294:
1293:
1292:
1285:
1281:
1278:
1274:
1266:
1264:
1262:
1258:
1254:
1250:
1246:
1242:
1234:
1229:
1226:
1223:
1220:
1219:
1218:
1212:
1209:
1207:
1204:
1201:
1198:
1195:
1192:
1191:
1190:
1187:
1183:
1181:
1177:
1176:driver Script
1173:
1169:
1165:
1160:
1153:
1148:
1145:
1142:
1141:
1140:
1138:
1129:
1122:
1116:
1112:
1109:
1106:
1102:
1101:
1096:
1089:
1087:
1080:
1078:
1076:
1072:
1068:
1067:test engineer
1062:
1061:Test engineer
1055:Test engineer
1054:
1052:
1050:
1042:
1037:
1032:
1028:
1025:
1021:
1019:
1015:
1012:
1008:
1005:
1001:
997:
993:
989:
986:
982:
979:
976:
973:
970:
966:
962:
959:
955:
952:
948:
945:
944:
943:
940:
938:
930:
928:
926:
922:
914:
912:
905:
900:
898:
895:
890:
882:
880:
878:
874:
870:
865:
861:
857:
853:
845:
841:Methodologies
840:
838:
834:
830:
828:
824:
820:
816:
811:
809:
805:
798:
790:
788:
786:
782:
781:message layer
778:
774:
770:
766:
761:
756:
748:
746:
742:
740:
736:
728:
726:
723:
717:
709:
704:
699:
695:
694:
690:
687:
683:
682:
678:
677:
676:
670:
668:
666:
662:
657:
653:
649:
638:
635:
627:
624:February 2009
617:
613:
607:
606:
600:
595:
586:
585:
575:
570:
568:
563:
561:
556:
555:
553:
552:
545:
542:
541:
535:
534:
527:
524:
522:
519:
517:
514:
513:
507:
506:
499:
496:
494:
491:
489:
486:
484:
481:
479:
476:
474:
471:
469:
466:
464:
461:
459:
456:
455:
449:
448:
441:
438:
436:
433:
431:
428:
426:
423:
421:
418:
416:
413:
411:
408:
406:
403:
401:
398:
397:
394:
389:
388:
381:
378:
376:
373:
371:
368:
366:
363:
361:
358:
356:
353:
351:
348:
346:
343:
341:
338:
336:
333:
332:
326:
325:
318:
315:
313:
310:
308:
305:
303:
302:Documentation
300:
298:
295:
293:
290:
289:
283:
282:
275:
272:
270:
267:
265:
262:
260:
257:
255:
252:
250:
247:
245:
242:
240:
237:
235:
232:
230:
227:
225:
222:
220:
217:
215:
212:
210:
207:
205:
202:
200:
197:
195:
192:
190:
187:
185:
182:
180:
177:
175:
172:
171:
167:
166:Methodologies
162:
161:
154:
151:
149:
146:
144:
141:
139:
136:
134:
131:
129:
126:
124:
121:
120:
114:
113:
106:
103:
101:
98:
96:
93:
91:
88:
86:
83:
81:
78:
76:
73:
71:
68:
66:
63:
61:
60:Data modeling
58:
57:
51:
50:
47:
43:
39:
38:
33:
19:
2577:Soak testing
2572:Pair testing
2535:
2415:Unit testing
2345:Fuzz testing
2281:
2246:
2218:
2202:
2184:
2165:
2144:
2112:
2105:
2096:
2087:
2078:
2069:
2058:. Retrieved
2051:the original
2022:. Retrieved
2015:
2006:
1995:. Retrieved
1991:
1967:
1956:. Retrieved
1952:
1943:
1924:
1896:
1892:
1886:
1875:. Retrieved
1864:
1852:
1840:
1828:
1806:(3): 74–79.
1803:
1799:
1793:
1781:
1769:
1757:
1741:
1729:
1717:
1690:
1685:
1669:. Springer.
1665:
1658:
1639:
1633:
1337:
1333:
1325:
1316:test harness
1309:
1290:
1270:
1241:unit testing
1238:
1216:
1188:
1184:
1161:
1157:
1134:
1105:unit testing
1084:
1064:
1046:
953:independence
941:
937:test oracles
934:
931:What to test
918:
909:
892:
863:
849:
835:
831:
812:
800:
758:
743:
732:
719:
691:
679:
674:
651:
645:
630:
621:
602:
420:UML Modeling
415:GUI designer
80:Construction
70:Requirements
2592:Web testing
2436:A/B testing
2372:API testing
1432:modularity
1428:. The test
1200:Data-driven
1168:maintenance
771:as part of
760:API testing
755:API testing
749:API testing
616:introducing
138:Prototyping
133:Incremental
105:Maintenance
85:Engineering
32:AutomatedQA
2607:Categories
2060:2011-12-11
2024:2010-09-26
1997:2023-02-11
1958:2022-09-13
1899:: 92–117.
1877:2009-08-20
1626:References
1471:test cases
1261:Test cases
1115:UI testing
1075:assertions
819:DOM Events
599:references
510:Glossaries
100:Deployment
2441:Benchmark
2097:junit.org
1620:Unit test
1434:framework
1273:workspace
1172:test case
1164:framework
1004:workflows
965:Data Base
869:waterfall
329:Practices
153:Waterfall
128:Cleanroom
95:Debugging
65:Processes
2550:See also
2271:Archived
1820:30671391
1579:See also
1426:software
1401:software
1398:computer
1024:test bed
967:Access,
958:Metadata
947:Platform
656:software
538:Outlines
468:ISO 9001
410:Profiler
405:Debugger
400:Compiler
375:Stand-up
2428:tactics
2017:YouTube
1750:Gartner
1600:Fuzzing
1118:effort.
612:improve
209:Lean SD
148:V model
90:Testing
2253:
2234:
2209:
2191:
2172:
2153:
2120:
2079:GitHub
1931:
1818:
1705:
1673:
1646:
1566:using
1463:manual
1430:script
1312:parser
1137:Google
864:before
601:, but
483:SWEBOK
204:Kanban
179:DevOps
143:Spiral
75:Design
2054:(PDF)
2047:(PDF)
1816:S2CID
1437:case.
1253:NUnit
1249:JUnit
1245:xUnit
1038:Roles
996:Maven
733:Some
478:PMBOK
393:Tools
254:SEMAT
249:Scrum
123:Agile
2426:and
2251:ISBN
2232:ISBN
2207:ISBN
2189:ISBN
2170:ISBN
2151:ISBN
2118:ISBN
1929:ISBN
1703:ISBN
1671:ISBN
1644:ISBN
1572:code
1465:and
1257:code
1251:and
1178:and
1000:IDEs
985:APIs
949:and
854:and
815:HTML
769:APIs
663:and
493:IREB
488:ITIL
458:CMMI
335:ATDD
244:SAFe
214:LeSS
189:DSDM
2224:doi
1901:doi
1808:doi
1695:doi
1534:or
1069:or
1031:SUT
994:or
992:Ant
777:GUI
646:In
498:OMG
425:IDE
380:TDD
370:SBE
360:DDD
345:CCO
340:BDD
264:TSP
259:TDD
239:RUP
234:RAD
229:PSP
224:MSF
219:MDD
199:IID
194:FDD
184:DAD
174:ASD
2609::
2230:.
2095:.
2077:.
2033:^
2014:.
1990:.
1979:^
1951:.
1913:^
1897:76
1895:.
1814:.
1804:24
1802:.
1701:.
1693:.
1318:.
951:OS
787:.
667:.
650:,
365:PP
355:CD
350:CI
274:XP
269:UP
2308:e
2301:t
2294:v
2259:.
2240:.
2226::
2197:.
2178:.
2159:.
2126:.
2099:.
2081:.
2063:.
2027:.
2000:.
1961:.
1937:.
1907:.
1903::
1880:.
1822:.
1810::
1711:.
1697::
1679:.
1652:.
1574:.
1556:.
1512:.
1491:.
1451:.
1418:.
1386:.
1033:)
1026:)
1006:.
971:)
960:)
637:)
631:(
626:)
622:(
608:.
573:e
566:t
559:v
34:.
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.