- Testing levels
- Classification of testing types
Software testing is an examination of a software product in order to detect software bugs and to verify that the program does what it is supposed to do. This examination includes test suites chosen according to certain parameters and specifics of software. In a broader sense, software testing is software quality assurance technique comprising test design, test execution, and analysis of test results.
Software testing is a considerable aspect of the software development process, since it helps provide security, reliability and usability of the product. Nowadays, there are many approaches to software testing. However, efficient testing of complex software systems is rather a creative process than a formal adherence to strict, definite rules.
Let's describe the main stages of software testing according to STLC (software testing life cycle):
- Requirement handling.
- Test planning.
- Test case development.
- Test environment setup.
- Test execution.
- Test cycle closure.
Unit testing is a software testing level at which the smallest components of software are tested, for example, individual classes or functions. Unit testing is often performed by software developers.
- Wikipedia. Unit testing.
- Unit testing native code with Test Explorer.
- Unit Testing in BlueJ.
- Unit testing Visual C# code in a Windows Store app.
Integration testing is a software testing level at which interfaces between components or subsystems of software are tested.
System testing is a software testing level at which a complete integrated system is tested to evaluate the system's compliance with its specified requirements. System testing includes alpha testing and beta testing subcategories.
Classification of testing types
The testing types are classified into several categories according to a number of criteria. The following are the most common ones.
By the object under test
Functional testing is a software testing type intended to validate whether a program meets functional requirements. Functional testing checks if a software product can correctly solve certain tasks as required by users.
- Wikipedia. Functional testing.
- Stack Overflow. Unit tests vs Functional Testing.
- Unit, Integration, and Functional Testing.
Performance testing is a software testing type performed to determine how a system performs in terms of performance and stability under a particular workload. Performance tests are executed before and after program optimization in order to discover any changes of performance. If an optimization leads to a performance drop and therefore is unsuccessful, the developer may refuse using it. In case of a performance gain, its degree is compared to the anticipated one to make sure that the optimization has been done successfully. The purpose of performance testing is to find out any cases of performance increase and decrease in order to avoid faulty optimizations.
- Wikipedia. Performance testing.
- Wikipedia. Load testing.
- Stack Overflow. Performance testing utilities for C++.
Load testing is a software testing type performed to determine a system's performance under normal, increased, and anticipated peak load conditions. If load testing is performed before the commercial exploitation of the system, it helps assure that no sudden performance losses will occur later (in about a year), when the system is loaded with data.
Stress testing is a software testing type that determines robustness of software beyond the limits of normal operation. It checks programs under stress conditions like large input data amounts, disk space shortage or execution on a low-performance processor.
Stress testing is also performed on a customized solution and server group to evaluate its ability to serve many users at once. Stress tests check not only the server group but also the impact of software customization on the system performance in general and its fault tolerance. This testing type requires a network of several computers emulating activity of a group of users.
Stability/endurance/soak testing is a software testing type that checks stability of software during a long testing time under medium workload.
- Wikipedia. Software reliability testing.
Compatibility testing is a software testing type that evaluates the application's compatibility with a certain computing environment.
- Wikipedia. Compatibility testing.
Security testing is a software testing type that checks the ability of software security mechanisms to react to system intrusions.
By security testing type
Penetration testing is a software testing type that simulates a real attack on the system. This simulation helps evaluate the ability of available security tools to deal with the real threat from an attacker. Moreover, penetration testing is able to detect zero-day vulnerabilities.
Web application security testing is a software testing type aimed at detecting system vulnerabilities and searching for opportunities to use them. It also includes a risk assessment of web application vulnerabilities.
The development of various web application security standards depends largely on the community and open-source projects. For example, the Open Web Application Security Project (OWASP) has developed the OWASP ASVS standard.
- GitHub. The Web Security Testing Guide.
API security testing is a software testing type that helps detect vulnerabilities in the application programming interface and web services. API security testing is used to prevent unauthorized access and programming interface overuse. APIs are highly vulnerable to threats such as Man-in-the-middle (MITM) attacks, API injections, and denial-of-service attacks (DoS).
Application security testing is a group of testing methods that are used to detect and eliminate vulnerabilities in software applications. These methods include testing, analyzing, and reporting on the security status of a software application during the software development life cycle (SSDLC).
By technologies used for application security testing
Static Application Security Testing (SAST) is a process of software source code testing by means of a static analyzer in order to detect code fragments containing potential vulnerabilities.
Dynamic Application Security Testing (DAST) is a software testing type by means of a dynamic analyzer in order to detect potential runtime vulnerabilities. It may include searching for problems with scripting, memory leaks, cookie processing, authentication, and the execution of third-party components.
Interactive Application Security Testing (IAST) is a software testing type that detects security defects in real time by analyzing the source code, data flow, configuration, and third-party libraries. IAST also applies to API testing.
Mobile Application Security Testing (MAST) is a testing methodology that combines static analysis, dynamic analysis, and the study of data generated by mobile apps. In this way, comprehensive testing of a security system is provided. Moreover, security problems related to the specifics of mobile platforms are solved. For example, jailbreaking, malicious Wi-Fi networks and data leaks from mobile devices are detected.
- Introduction to the Mobile Security Testing Guide.
- GitHub. Mobile Application Security Testing Guide.
Software Composition Analysis is an application composition analysis that identifies dependencies. The purpose of the analysis is to evaluate the security of the dependencies used and determine their compliance with licenses.
- SCA (software composition analysis).
- The risks of using vulnerable dependencies in your project, and how SCA helps manage them
By the access to source code and architecture
Black-box testing is a software testing type performed on a software system which can only be accessed through user interfaces or external interfaces. Black-box testing allows another computer or process connect to the system to run tests on it. This approach is still the most popular in everyday testing practice, but it has a lot of drawbacks. For example, some errors can only occur in specific conditions and therefore are difficult to find and reproduce.
- Wikipedia. Black-box testing.
- Black Box Testing: An In-Depth Tutorial With Examples And Techniques.
- Elliotte Rusty Harold. Fuzz testing.
White-box testing is a software testing type that allows a tester to have a free access to the source code of an application. The tester writes test cases covering the libraries that the software uses. White-box testing includes methods of code review and software inspection. This approach allows peeping into the "black box" to focus on the application's internal implementation which determines its behavior. The main problem with this approach is that runtime computations are difficult to monitor. White-box tests evaluate the program's logic, so testing is said to be complete when it tracks all the possible code paths. Even a medium complexity program may have dozens of thousands of code paths.
By the product maturity phase
Alpha testing is a software testing type implying simulated or actual operational testing (by developers or users correspondingly).
Beta testing is a software testing type when limited versions of software are released to a group of users. Beta testing allows to make sure that the product contains few bugs.
Regression testing is a software testing type performed to ensure that no previously found bugs have come back after any code modifications and bug fixes, as well as to determine the main functionality of the software product. This type of testing is usually performed on every new product version. Regression testing is the most important testing phase performed immediately before the end of the development process; it is extremely crucial before the release. Regression testing helps make sure that not only the main functions work properly but also that no previously detected bugs have been brought into the final version. Regression testing, as an integral part of functional testing, guarantees that no modifications brought into code by bug fixes have caused negative effect on all the other functions of the application.
Smoke testing is a software testing type performed as a set of tests to ensure that the product can be launched successfully. If no errors occur during the launch, the smoke test is said to be passed. If the smoke test is failed, software is sent back to the developer team for revision. The point is that sometimes different components of a program are written by different developer teams and may often appear non-compatible when being assembled into a whole. This fact makes it pointless to test the product in general.
By the automation criterion
Manual testing is a software testing type when no software tools are used to run tests and analyze test results. Instead, a tester does it manually.
Automated testing is a software testing type when software tools are used to perform tests and analyze test results. Certainly, automated testing is highly useful, as it allows saving time and resources.
In the development process, it often happens that new versions with bug fixes are released every day or even several times a day. In this case, smoke tests should be automated in the first place. This will help to make sure that the program can be successfully launched after every build. Automated tests can be completed in just a few seconds, so you can quickly verify the success of the build. The test would take much longer if done manually. Thus, smoke testing automation is an opportunity for the testing department to save incredibly much time.
There is a large number of applications providing test automation features. The most popular are HP LoadRunner, HP QuickTest Professional, HP Quality Center, and TestComplete.
Automation in general does not only save development time but also improves security and reliability of software products. Testers will also see significant advantages of this approach: reliability of the testing process and results increases, testing time decreases, and the tester's work gets less stressful. Sure, automated testing will never fully replace manual testing, but it can simplify the software test engineer's work indeed.
Dynamic and static code analysis
As a software product is passing later development phases, the cost of bug elimination may grow exponentially. Static and dynamic analysis tools can help prevent these expenses due to earlier detection of software bugs.
Dynamic analysis (runtime analysis) is a software testing type performed when executing the program. Dynamic analysis reveals defects in the source code as they occur. The analysis process can be divided into a number of steps: preparing input data, running a test program launch, gathering the necessary parameters, and analyzing the output data.
- Dynamic code analysis.
- Wikipedia. Dynamic analysis.
- Why we need dynamic code analysis: the example of the PVS-Studio project.
Static analysis is a software testing type performed without actual execution of the program. Static analysis enables quick detection of bugs in the source code before the code is ready to be executed.
In practice, developers may use both analysis techniques to speed up the development and testing processes, as well as to enhance the quality of the software product.