1

ANALYZED TECHNIQUES ~ BLACK-BOX TESTING, WHITE-BOX

TESTING, GRAYBOX-TESTING

COSC 5370 - ADVANCED SOFTWARE ENGINEERING

TEXAS A&M UNIVERSITY – CORPUS CHRISTI

SPRING 2014

BY

MERT EVREN 2

Abstract

~ Testing is an important phase in software engineering Software testing methods are used for examination, verification and validation of the source code of a program. These methods are applied to software applications and their components to uncover hidden errors. Software testing includes experimentally and scientifically inspecting the correctness of the application. Software testing determines the applications quality by evaluating the capability of the application. The entire testing procedure should be well-structured. In this paper, three testing methodologies; black-box testing, white- box testing, and gray-box testing, are analyzed in detail. The remainder of this paper is structured as follows; section 2 describes black-box testing, white-box testing, and gray-box testing. Different testing techniques of each method has shown. These testing methods are analyzed by providing pros and cons of each. Section 3 presents possible improvement on testing methods. Lastly, section 4 concludes the paper.

3

1. INTRODUCTION

Testing is an important phase in software engineering. “To develop high-quality software, it is

essential to use software testing methods.” [1] Software testing is defined “as a process of accessing

the functionality and correctness of a software through analysis.” [4] Software testing methods are

used for examination, verification and validation of the source code of a program. These methods are

applied to software applications and their components to uncover hidden errors. Software testing

includes experimentally and scientifically inspecting the correctness of the application. Software

testing determines the applications quality by evaluating the capability of the application. The entire

testing procedure should be well-structured. “Testing process must balance the requirements,

technical limitation and user expectation.” [7] Testing cost changes depends on the application and

the type of test applied, however not testing the application can be even cost more. Testing is done in

every stage of software life cycle. “Testing ensures that what you get in the end is what you wanted to

build.” [7]

In this paper, three testing methodologies; black-box testing, white-box testing, and gray-box

testing, are analyzed in detail. The remainder of this paper is structured as follows; section 2 describes

black-box testing, white-box testing, and gray-box testing. Different testing techniques of each

method has shown. These testing methods are analyzed by providing pros and cons of each. Section 3

presents possible improvement on testing methods. Lastly, section 4 concludes the paper.

2. TESTING METHODS

2.1. BLACK-BOX TESTING

Black box testing, sometimes named functional testing or behavioral testing, defined as “is a

software testing techniques in which functionality of the software under test (SUT) is tested

without looking at the internal code structure, implementation details and knowledge of internal

paths of the software.” [2], mainly concentrates on testing whether or not the code does what it is 4

supposed to do developed from its functional requirements. Black box testing develops test cases

by using the specifications of the program and it is done by using the output of the user interface

as would an end-user. Black-box testing is not concerned with the process software takes to

provide particular output, it is concerned with the output that results from the input.

2.1.1. BLACK-BOX TESTING TECHNIQUES

Figure-1 - Black-Box Testing Techniques [5]

There are many black-box testing techniques exists, as shown in Figure-1. Following are some black-box testing techniques.

2.1.1 EQUIVALENCE CLASS PARTITIONING

This testing technique divides input values of a program unit into valid and

invalid partitions where test cases derived from.

5

2.1.2 BOUNDARY VALUE ANALYSIS

Boundary value analysis determines the boundaries of input values which

includes error values and valid values.

2.1.3 FUZZY TESTING

Fuzzy testing is used to discover implementation bugs by using

“malformed / semi- malformed data injection in an automated or semi-automated

fashion. is also used to test for security problems in software.” [2]

2.1.2. PROS

 Very efficient and suited for testing large code segments. Simply focuses on if valid

and invalid inputs provide correct outputs.

 Black-box testing is quite easy, advantaging tester’s usage with no coding skills or

the knowledge of software implementation.

 Once the specifications are done, test cases can be designed. Tester are only

concerned with GUI, internal paths of the program are not required to be analyzed.

 “The test cases can show presence or absence of classes of errors.” [7]

2.1.3. CONS

 Since black-box testing is based on GUI, it makes it hard for maintaining script

when the user interface constantly changes.

 Because black-box testing doesn’t look into the internal code, the program can

never be fully tested.

 “Blind Coverage: cannot control targeting code segments or paths which may be

more error prone than others.” [2] 6

 “Only small numbers of possible input can actually be tested.”[7]

 The test may be already executed on the program by its programmer which causes

redundant testing.

2.2. WHITE-BOX TESTING

White-box testing digs deep into the internal code of software. [2] Describes why this

testing method named white-box “This method is named so because the software program, in the

eyes of the tester, is like a white/transparent box; inside which one clearly sees.” White box

testing investigates the implementation of the components in the source code, in another word

white-box testing allows testers to see what is happening inside of the software. “White box

testing is considered as a method that can be used to validate whether code

implementation follows intended design, to validate implemented security functionality, and to

uncover exploitable vulnerabilities.” [7] White box testing provides full understanding of the

internal mechanism of the application. “In white box testing it is necessary for a tester to have

full knowledge of source code” [4] White-box testing is very efficient on revealing errors.

“Exhaustive correctness checking is performed, all code paths are executed to ensure quality

software outcome with reduced defect rate.” [5]

2.2.1. WHITE-BOX TESTING TECHNIQUES

There are many existing test techniques for white-box testing as show in Figure-2. Here

are the some of the white-box testing techniques analyzed.

2.2.1.1. CONTROL FLOW TESTING

Control-flow testing is an efficient techniques for most of the software. “It is a

structural testing strategy that uses the program control flow as a model control flow

and favors more but simpler paths over fewer but complicated path.”[4] 7

2.2.1.2. BASIC PATH TESTING

Basic path testing confirms that each independent path in the application source

code is taken in a predetermined order. “It allows the test case designer to produce a

logical complexity measure of procedural design and use this measure as an approach

for outlining a basic set of execution path” [2]

2.2.1.3. DATA FLOW TESTING

Data flow testing focuses on how data communicates within the system. It is used

on modules with nested if and loop statements to detect variable that are unused or not

initialized.

2.2.1.4. CONDITION TESTING

Condition testing concentrates on the logical conditions of the program methods.

Figure-2 – White-Box Testing Techniques

8

2.2.2. PROS

 Allows internal code to be fully tested where objects are able to be identified

programmatically. “All independent paths in a module will be exercised at least

once.” [7]

 Looking deep into implementation in detail allows programming errors to be easily

detected.

 Behavior of software modules can be uncovered since testing involves accessing to

source code.

2.2.3. CONS

 “It is nearly impossible to look into every bit of code to find out hidden errors,

which may create problems, resulting in failure of the application.”[7]

 Testers have to be familiar with testing tools. Testers needs to be skilled in coding

languages. Because of these requirements, it becomes difficult to scale testers.

 White-box testing uses test scripts. Implementation of these test scripts are tied to

the source code of the application. Maintenance becomes very hard when any

change made to the code.

2.3. GRAY-BOX TESTING

Gray-box testing, also called as translucent testing, is combination of white-box testing and

black-box testing. Gray-box testing is defined as, “a technique of testing the application with

limited knowledge of the internal working of an application and also has the knowledge of

fundamental aspects of the system” [4] Documentation of the program needs to be provided to

testers in order to execute the tests. “The aim of this testing is to search for the defects if any due

to improper structure or improper usage of applications.” [2] “Grey box testing technique will 9 increase the testing coverage by allowing us to focus on all the layers of any complex system through the combination of all existing white box and black box testing.” [4]

Figure 3 – Representation of Gray-box Testing [4]

2.3.1. GRAY-BOX TESTING TECHNIQUES

2.3.1.1. MATRIX TESTING

Status report of the application is generated which includes the relationship

between the test requirements and the test cases. This report allows testers to analyze

which requirements are included in which test cases.

2.3.1.2.

This test is applied after new changes are made, such as functional improvement,

to the source code of the application “Its purpose is to determine if the change has

regressed other aspects of the program.” [2] Regression testing is a regular step of the

software implementation procedure.

2.3.1.3. PATTERN TESTING

Pattern testing check if the architecture, development patterns and design of the

application are correctly implemented. 10

2.3.1.4. TESTING

Orthogonal array testing is a statistical testing technique. “This type of testing

use as subset of all possible combinations.”[4]

2.3.2. PROS

 Testing is not depends on the applications source code. “In grey box testing, the

tester relies on interface definition and functional specification”[4]

 Gray-box testing offers strengths and benefits of both white-box testing and black-

box testing. “This increases the number of test cases generated, increases the

coverage and deals with chances of blunder in user requirements and system

functional behavior.” [5]

 “Based on the limited information available, a Gray Box tester can author

intelligent test scenarios, especially around data type handling, communication

protocols, and exception handling.” [2]

 Gray-box testing provides unbiased testing which separates the teams of developers

and testers.

 Gray-box testing is efficient on web applications. Testing “supports the Web

applications as substantial material is available in WSDL—Web service definition

language.”[2]

2.3.3. CONS

 Access to source code and binaries is not available which limits the coverage of

testing and causes untested code paths.

 “If the software designer has already run a test case, the tests can be redundant.”[4] 11

 “Inherent to distributed applications is the difficulty associated in defect

identification. is still at the mercy of how well systems throw

exceptions, and how well are these exceptions propagated with a distributed Web

Services environment.” [2]

3. POSSIBLE IMPROVEMENTS

As the technology grows, “The existing new technologies like Service Oriented Architecture

(SOA), wireless technologies, mobile services etc. has opened new path to testing.” [2] In the paper,

[6] new implications are proposed for white-box testing, in order to maximize error detection rate.

“Testers will provide light weight models that developers can run against their codes.” [4] The

implication recommends testing to be mainly aimed at “revealing faults that fall into the most

common classifications.” [6] “Testers could potentially benefit by choosing testing methods which

are effective at detecting the most commonly-occurring faults, which the results indicate are faults

relating to if conditions and method call parameter changes.”[6]

4. CONCLUSION

Software testing offers an independent vision of the software. Black-box testing, white-box

testing and gray-box testing are analyzed along with the testing techniques of each method. “To

perform testing effectively and efficiently, everyone involved with testing should be familiar with

basic software testing goals, principles, limitations and concepts.”[7] The research provides

advantage and disadvantages of each testing methods. Possible improvements of these testing

methods are discussed.

12

Bibliography

[1] Lee, J.; Kang, S.; Lee, D. (2012). A Survey of Software Testing Practices. IET Software. Vol. 6 Issue

3, p275-282. 8p.

[2] Shivani Acharya; Vidhi Pandya. (2013) Bridge between Black Box and White Box – Gray Box

Testing Technique. International Journal of Electronics and Computer Science Engineering, Vol.

2, Issue 1, p.175

[3] Augusto Lazzarini Lemos, Fabiano Cutigi Ferrari, Marcelo Medeiros Eler, José Carlos Maldonado,

Paulo Cesar Masiero, (2013). Evaluation studies of software testing research in Brazil and in the

world: A survey of two premier software engineering conferences. Journal of Systems and

Software, Volume 84, Issue 4, Pages 951–969

[4] Mohd. Ehmer Khan , Farmeena Khan. (2012). A Comparative Study of White Box, Black Box and

Grey Box Testing Techniques. International Journal of Advanced Computer Science and

Applications, Vol. 3, No.6,

[5] MUNAZZA JANNISAR, RUQIA BIBI & MUHAMMAD FAHAD KHAN, (2014). Testing

Techniques Selection Based on SWOT Analysis. International Journal of Knowledge, Innovation

and Entrepreneurship. Volume 2 No. 1, pp. 56—68

[6] Shimul Kumar Nath, Robert Merkel and Man Fai Lau, (2012). On the improvement of a fault

classification scheme with implications for white-box testing. Proceedings of the 27th Annual

ACM Symposium on Applied Computing, Pages 1123-1130

[7] Abhijit A. Sawant1,Pranit H. Bari2and P. M. Chawan, (2012). Software Testing Techniques and

Strategies, International Journal of Engineering Research and Applications, Vol. 2, Issue 3,

pp.980 - 986 13

[8] Ghiduk, Ahmed S.(2014). Automatic generation of basis test paths using variable length genetic

algorithm. Information Processing Letters. Vol. 114 Issue 6, p304-316. 13p.

[9] Anand, Saswat; Burke, Edmund K.; Chen, Tsong Yueh; Clark, John; Cohen, Myra B.; Grieskamp,

Wolfgang; Harman, Mark; Harrold, Mary Jean; McMinn, Phil; Bertolino, Antonia; Jenny Li, J.;

Zhu, Hong. (2013) An orchestrated survey of methodologies for automated software test case

generation In The Journal of Systems & Software. 86(8)

[10] Xiaoqi Xing; Bin Liu; Dongyi Ling. (2014). Research on Gray-box Testing Methods for Software

Fault Injection. Applied Mechanics & Materials, Issue 543-547, p3360-3363, 4p.