1
ANALYZED SOFTWARE TESTING TECHNIQUES ~ BLACK-BOX TESTING, WHITE-BOX
TESTING, GRAYBOX-TESTING
COSC 5370 - ADVANCED SOFTWARE ENGINEERING
TEXAS A&M UNIVERSITY – CORPUS CHRISTI
SPRING 2014
BY
MERT EVREN 2
Abstract
~ Testing is an important phase in software engineering Software testing methods are used for examination, verification and validation of the source code of a program. These methods are applied to software applications and their components to uncover hidden errors. Software testing includes experimentally and scientifically inspecting the correctness of the application. Software testing determines the applications quality by evaluating the capability of the application. The entire testing procedure should be well-structured. In this paper, three testing methodologies; black-box testing, white- box testing, and gray-box testing, are analyzed in detail. The remainder of this paper is structured as follows; section 2 describes black-box testing, white-box testing, and gray-box testing. Different testing techniques of each method has shown. These testing methods are analyzed by providing pros and cons of each. Section 3 presents possible improvement on testing methods. Lastly, section 4 concludes the paper.
3
1. INTRODUCTION
Testing is an important phase in software engineering. “To develop high-quality software, it is
essential to use software testing methods.” [1] Software testing is defined “as a process of accessing
the functionality and correctness of a software through analysis.” [4] Software testing methods are
used for examination, verification and validation of the source code of a program. These methods are
applied to software applications and their components to uncover hidden errors. Software testing
includes experimentally and scientifically inspecting the correctness of the application. Software
testing determines the applications quality by evaluating the capability of the application. The entire
testing procedure should be well-structured. “Testing process must balance the requirements,
technical limitation and user expectation.” [7] Testing cost changes depends on the application and
the type of test applied, however not testing the application can be even cost more. Testing is done in
every stage of software life cycle. “Testing ensures that what you get in the end is what you wanted to
build.” [7]
In this paper, three testing methodologies; black-box testing, white-box testing, and gray-box
testing, are analyzed in detail. The remainder of this paper is structured as follows; section 2 describes
black-box testing, white-box testing, and gray-box testing. Different testing techniques of each
method has shown. These testing methods are analyzed by providing pros and cons of each. Section 3
presents possible improvement on testing methods. Lastly, section 4 concludes the paper.
2. TESTING METHODS
2.1. BLACK-BOX TESTING
Black box testing, sometimes named functional testing or behavioral testing, defined as “is a
software testing techniques in which functionality of the software under test (SUT) is tested
without looking at the internal code structure, implementation details and knowledge of internal
paths of the software.” [2], mainly concentrates on testing whether or not the code does what it is 4
supposed to do developed from its functional requirements. Black box testing develops test cases
by using the specifications of the program and it is done by using the output of the user interface
as would an end-user. Black-box testing is not concerned with the process software takes to
provide particular output, it is concerned with the output that results from the input.
2.1.1. BLACK-BOX TESTING TECHNIQUES
Figure-1 - Black-Box Testing Techniques [5]
There are many black-box testing techniques exists, as shown in Figure-1. Following are some black-box testing techniques.
2.1.1 EQUIVALENCE CLASS PARTITIONING
This testing technique divides input values of a program unit into valid and
invalid partitions where test cases derived from.
5
2.1.2 BOUNDARY VALUE ANALYSIS
Boundary value analysis determines the boundaries of input values which
includes error values and valid values.
2.1.3 FUZZY TESTING
Fuzzy testing is used to discover implementation bugs by using
“malformed / semi- malformed data injection in an automated or semi-automated
fashion. Fuzzing is also used to test for security problems in software.” [2]
2.1.2. PROS
Very efficient and suited for testing large code segments. Simply focuses on if valid
and invalid inputs provide correct outputs.
Black-box testing is quite easy, advantaging tester’s usage with no coding skills or
the knowledge of software implementation.
Once the specifications are done, test cases can be designed. Tester are only
concerned with GUI, internal paths of the program are not required to be analyzed.
“The test cases can show presence or absence of classes of errors.” [7]
2.1.3. CONS
Since black-box testing is based on GUI, it makes it hard for maintaining script
when the user interface constantly changes.
Because black-box testing doesn’t look into the internal code, the program can
never be fully tested.
“Blind Coverage: cannot control targeting code segments or paths which may be
more error prone than others.” [2] 6
“Only small numbers of possible input can actually be tested.”[7]
The test may be already executed on the program by its programmer which causes
redundant testing.
2.2. WHITE-BOX TESTING
White-box testing digs deep into the internal code of software. [2] Describes why this
testing method named white-box “This method is named so because the software program, in the
eyes of the tester, is like a white/transparent box; inside which one clearly sees.” White box
testing investigates the implementation of the components in the source code, in another word
white-box testing allows testers to see what is happening inside of the software. “White box
testing is considered as a security testing method that can be used to validate whether code
implementation follows intended design, to validate implemented security functionality, and to
uncover exploitable vulnerabilities.” [7] White box testing provides full understanding of the
internal mechanism of the application. “In white box testing it is necessary for a tester to have
full knowledge of source code” [4] White-box testing is very efficient on revealing errors.
“Exhaustive correctness checking is performed, all code paths are executed to ensure quality
software outcome with reduced defect rate.” [5]
2.2.1. WHITE-BOX TESTING TECHNIQUES
There are many existing test techniques for white-box testing as show in Figure-2. Here
are the some of the white-box testing techniques analyzed.
2.2.1.1. CONTROL FLOW TESTING
Control-flow testing is an efficient techniques for most of the software. “It is a
structural testing strategy that uses the program control flow as a model control flow
and favors more but simpler paths over fewer but complicated path.”[4] 7
2.2.1.2. BASIC PATH TESTING
Basic path testing confirms that each independent path in the application source
code is taken in a predetermined order. “It allows the test case designer to produce a
logical complexity measure of procedural design and use this measure as an approach
for outlining a basic set of execution path” [2]
2.2.1.3. DATA FLOW TESTING
Data flow testing focuses on how data communicates within the system. It is used
on modules with nested if and loop statements to detect variable that are unused or not
initialized.
2.2.1.4. CONDITION TESTING
Condition testing concentrates on the logical conditions of the program methods.
Figure-2 – White-Box Testing Techniques
8
2.2.2. PROS
Allows internal code to be fully tested where objects are able to be identified
programmatically. “All independent paths in a module will be exercised at least
once.” [7]
Looking deep into implementation in detail allows programming errors to be easily
detected.
Behavior of software modules can be uncovered since testing involves accessing to
source code.
2.2.3. CONS
“It is nearly impossible to look into every bit of code to find out hidden errors,
which may create problems, resulting in failure of the application.”[7]
Testers have to be familiar with testing tools. Testers needs to be skilled in coding
languages. Because of these requirements, it becomes difficult to scale testers.
White-box testing uses test scripts. Implementation of these test scripts are tied to
the source code of the application. Maintenance becomes very hard when any
change made to the code.
2.3. GRAY-BOX TESTING
Gray-box testing, also called as translucent testing, is combination of white-box testing and
black-box testing. Gray-box testing is defined as, “a technique of testing the application with
limited knowledge of the internal working of an application and also has the knowledge of
fundamental aspects of the system” [4] Documentation of the program needs to be provided to
testers in order to execute the tests. “The aim of this testing is to search for the defects if any due
to improper structure or improper usage of applications.” [2] “Grey box testing technique will 9 increase the testing coverage by allowing us to focus on all the layers of any complex system through the combination of all existing white box and black box testing.” [4]
Figure 3 – Representation of Gray-box Testing [4]
2.3.1. GRAY-BOX TESTING TECHNIQUES
2.3.1.1. MATRIX TESTING
Status report of the application is generated which includes the relationship
between the test requirements and the test cases. This report allows testers to analyze
which requirements are included in which test cases.
2.3.1.2. REGRESSION TESTING
This test is applied after new changes are made, such as functional improvement,
to the source code of the application “Its purpose is to determine if the change has
regressed other aspects of the program.” [2] Regression testing is a regular step of the
software implementation procedure.
2.3.1.3. PATTERN TESTING
Pattern testing check if the architecture, development patterns and design of the
application are correctly implemented. 10
2.3.1.4. ORTHOGONAL ARRAY TESTING
Orthogonal array testing is a statistical testing technique. “This type of testing
use as subset of all possible combinations.”[4]
2.3.2. PROS
Testing is not depends on the applications source code. “In grey box testing, the
tester relies on interface definition and functional specification”[4]
Gray-box testing offers strengths and benefits of both white-box testing and black-
box testing. “This increases the number of test cases generated, increases the
coverage and deals with chances of blunder in user requirements and system
functional behavior.” [5]
“Based on the limited information available, a Gray Box tester can author
intelligent test scenarios, especially around data type handling, communication
protocols, and exception handling.” [2]
Gray-box testing provides unbiased testing which separates the teams of developers
and testers.
Gray-box testing is efficient on web applications. Testing “supports the Web
applications as substantial material is available in WSDL—Web service definition
language.”[2]
2.3.3. CONS
Access to source code and binaries is not available which limits the coverage of
testing and causes untested code paths.
“If the software designer has already run a test case, the tests can be redundant.”[4] 11
“Inherent to distributed applications is the difficulty associated in defect
identification. Gray box testing is still at the mercy of how well systems throw
exceptions, and how well are these exceptions propagated with a distributed Web
Services environment.” [2]
3. POSSIBLE IMPROVEMENTS
As the technology grows, “The existing new technologies like Service Oriented Architecture
(SOA), wireless technologies, mobile services etc. has opened new path to testing.” [2] In the paper,
[6] new implications are proposed for white-box testing, in order to maximize error detection rate.
“Testers will provide light weight models that developers can run against their codes.” [4] The
implication recommends testing to be mainly aimed at “revealing faults that fall into the most
common classifications.” [6] “Testers could potentially benefit by choosing testing methods which
are effective at detecting the most commonly-occurring faults, which the results indicate are faults
relating to if conditions and method call parameter changes.”[6]
4. CONCLUSION
Software testing offers an independent vision of the software. Black-box testing, white-box
testing and gray-box testing are analyzed along with the testing techniques of each method. “To
perform testing effectively and efficiently, everyone involved with testing should be familiar with
basic software testing goals, principles, limitations and concepts.”[7] The research provides
advantage and disadvantages of each testing methods. Possible improvements of these testing
methods are discussed.
12
Bibliography
[1] Lee, J.; Kang, S.; Lee, D. (2012). A Survey of Software Testing Practices. IET Software. Vol. 6 Issue
3, p275-282. 8p.
[2] Shivani Acharya; Vidhi Pandya. (2013) Bridge between Black Box and White Box – Gray Box
Testing Technique. International Journal of Electronics and Computer Science Engineering, Vol.
2, Issue 1, p.175
[3] Augusto Lazzarini Lemos, Fabiano Cutigi Ferrari, Marcelo Medeiros Eler, José Carlos Maldonado,
Paulo Cesar Masiero, (2013). Evaluation studies of software testing research in Brazil and in the
world: A survey of two premier software engineering conferences. Journal of Systems and
Software, Volume 84, Issue 4, Pages 951–969
[4] Mohd. Ehmer Khan , Farmeena Khan. (2012). A Comparative Study of White Box, Black Box and
Grey Box Testing Techniques. International Journal of Advanced Computer Science and
Applications, Vol. 3, No.6,
[5] MUNAZZA JANNISAR, RUQIA BIBI & MUHAMMAD FAHAD KHAN, (2014). Testing
Techniques Selection Based on SWOT Analysis. International Journal of Knowledge, Innovation
and Entrepreneurship. Volume 2 No. 1, pp. 56—68
[6] Shimul Kumar Nath, Robert Merkel and Man Fai Lau, (2012). On the improvement of a fault
classification scheme with implications for white-box testing. Proceedings of the 27th Annual
ACM Symposium on Applied Computing, Pages 1123-1130
[7] Abhijit A. Sawant1,Pranit H. Bari2and P. M. Chawan, (2012). Software Testing Techniques and
Strategies, International Journal of Engineering Research and Applications, Vol. 2, Issue 3,
pp.980 - 986 13
[8] Ghiduk, Ahmed S.(2014). Automatic generation of basis test paths using variable length genetic
algorithm. Information Processing Letters. Vol. 114 Issue 6, p304-316. 13p.
[9] Anand, Saswat; Burke, Edmund K.; Chen, Tsong Yueh; Clark, John; Cohen, Myra B.; Grieskamp,
Wolfgang; Harman, Mark; Harrold, Mary Jean; McMinn, Phil; Bertolino, Antonia; Jenny Li, J.;
Zhu, Hong. (2013) An orchestrated survey of methodologies for automated software test case
generation In The Journal of Systems & Software. 86(8)
[10] Xiaoqi Xing; Bin Liu; Dongyi Ling. (2014). Research on Gray-box Testing Methods for Software
Fault Injection. Applied Mechanics & Materials, Issue 543-547, p3360-3363, 4p.