THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Journal of Information Technology Management ISSN #1042-1319 A Publication of the Association of Management

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

HADAS CHASIDIM SCE — SHAMOON COLLEGE OF ENGINEERING [email protected] DANI ALMOG SCE — SHAMOON COLLEGE OF ENGINEERING [email protected] DOV BENYOMIN SOHACHESKI SCE — SHAMOON COLLEGE OF ENGINEERING [email protected] MARK L. GILLENSON UNIVERSITY OF MEMPHIS [email protected] ROBIN POSTON UNIVERSITY OF MEMPHIS [email protected] SHLOMO MARK SCE — SHAMOON COLLEGE OF ENGINEERING [email protected]

ABSTRACT The widely used term “unit test,” which is a quality procedure for testing modules of code, has been used for decades. To enable , specialized infrastructures were designed to enable the isolation of unit tested code from its production implementation. Currently, hundreds of tools and add-ons exist for unit testing in almost every environment and language. , on the other hand, which is a quality procedure for testing multiple modules of software code working together, demands a vastly different kind of infrastructure to enable the interaction of the code with other production implementation components. With the growing popularity of agile methods, the boundaries of unit and integration testing have become blurred with (CI) and Continuous Deployment (CD) derived from market demand for continuous reply and reaction of the software to a rapidly changing world. CICD enhances the importance of unit testing. Following that, it is important to identify the effect of new agile software development life cycles (SDLC) on unit test activities – assuming it may affect software quality in general. In this paper, we analyze the evolving different definitions and usages of the term “unit test” and attempt to understand the implication of these definitions to the actual use of the term.

Keywords: unit test, component test, unit under test

Journal of Information Technology Management Volume XXIX, Number 2, 2018 40

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

On the other hand, many people use the term “unit test” INTRODUCTION with different connotations and meanings referring to the test of a specific component or functionality. Motivation It is necessary to differentiate between the “unit Continuous Integration (CI) and Continuous under testing” and “unit test.” While unit testing is the Deployment (CD) [10] are new trends driven by market act of testing at the unit of software code, e.g., one demand. This trend is affecting the ability to provide an module, level, the term unit under test refers to the earlier and conscientious delivery of adaptation and various portions of software code that are being tested. changes to the software product. These new working The first formal definitions of “unit testing” supported by paradigms signify a different approach towards software the ANSI/IEEE Std 1008-1987, IEEE Standard for development life cycles (SDLC) and the need to Software Unit Testing reveal some flexibility concerning understand the impact of this change on software quality the meaning of the word “unit” in “unit testing,” see insert and testing procedures. The effect of the change may be below. This definition is not precise enough for today’s very significant at all dimensions of the organization and various approaches to SDLC, especially within the agile software development processes, with an examination of SDLC environments. For example, is a unit a single item its effect on unit testing being the first step. It is essential or a set of items? Moreover, the definition does not to ensure a common language among SDLC team provide us with a definite idea of the actual intention of members using the term "unit test" consistently. To model the unit definition. the CICD practices of different software industries [16], A set of one or more computer program modules the authors consider unit testing to be the basic procedure together with associated control data (for example, within the CICD approach. Following a review of the tables), usage procedures, and operating procedures that literature on CICD, research on CICD reports the satisfy the following conditions: consistent and meaningful use of unit test during the 1. All modules are from a single computer CICD process. Building on this work, this research program. explores the different definitions and usage of the term 2. At least one of the new or changed modules in "unit test" during the last ten years as reflected in the the set has not completed the unit test. academic literature. 3. The set of modules together with its associated The goal of this research study is to arrive at a data and procedures is the sole object of a better understanding of the term “unit test” and its testing process. applications. We explore the primary usages of the term Over the years, academics, open-source to see whether there are differences of opinion among participants, and the software industry have generated an leading academic researchers. To our knowledge, this is abundance of different definitions for unit tests [11]. one of the first papers that aims to better understand the While some definitions are products of necessity, others influence of current changes in software development reflect a given principle. The majority of definitions cite approaches and philosophies on testing efforts. This is principle buzzwords and concepts, attempting to remain thus an initial step in trying to arrive at a clearer accurate and authentic concerning the absolute definition understanding of the intention behind the value of unit of “unit.” Such as the case where unit tests cover only testing. functional requirements or when a unit is defined as First, we report the outcome of literature review writing many tests for one unit without actually formally research done on academic papers. The next section defining a unit. contains our definitions and suggestions to refine the The adoption of the use of xUnit tools in practice definition and use unit testing in today’s more agile is a rather straightforward process, and the administration SDLC processes. The last section concludes our findings of tests involving the use of these tools is also relatively and gives recommendations for our future research. easy. The main drawback is the misconception that such tests are categorized as unit tests. With the help of familiar testing tools, unit testing has become a Unit Testing in General chameleon that can almost imperceptibly camouflage Today, almost every programming language has itself and transform itself into other types of testing. As its individual unit testing framework (e.g., JUnit for Java, Andrew Hunter has aptly noted, “Unit tests have quickly NUnit for C#), which enables the use of small, become the proverbial hammer that makes everything automatically executable unit tests. Unit testing has look like a nail” [7]. become an accepted practice, often mandated by When codebases were still relatively small, and development processes (e.g., test-driven development). implementations were more transparent, the notion of a

Journal of Information Technology Management Volume XXIX, Number 2, 2018 41

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

unit was straightforward. Original software systems Instead of single regulatory outcomes from one unit, we delivered anywhere from a single service to a series of find ourselves imitating input from various units to services and behavior patterns. The behavior patterns conclude about a given unit. Developers find themselves were modest, and they focused on a specific goal. The uncovering the encapsulation of units and their original (still true to this the day) philosophy behind the dependencies to perform unit tests, a task that is Unix (OS) is DOTADIW: “Do One tantamount to unit integration testing, a topic that is Thing and Do It Well” [13]. The entire OS was a scaffold, beyond the purview of this paper. using “very narrow, very tightly specified interfaces” [7]. Unix and Interlisp quickly gained popularity due to their Previous Surveys on Unit Tests separation of responsibilities through the use of pipes, In 2006, a unit testing practices survey [14] was filters, and interfaces [7]. performed based on focus group discussions in a software The unit can be complex in its capabilities, but it process improvement network (SPIN) and a questionnaire is intended to be solely focused on a unique output. This was employed to validate the results (shown in Table 1). characteristic transforms the unit into a black box; when it The purpose of the survey was to investigate what is provided with input, the unit will not deviate from its practitioners refer to when they talk about unit testing. established, predetermined goal. Since this survey has been conducted, more than Unit testing is the sort of testing that is usually 11 years ago, it seems logical to reexamine the use of its close to the heart of developers [18]. The primary reason terminology and practices. Programming languages, as is that, by definition, unit testing tests distinct, well- well as design implementations, have changed due to defined parts of the system in isolation from other parts. more complex software capabilities. A more recent survey Thus, they are comparatively easy to write and use. Many 2014 [5] focused on the possible benefits build systems that have built-in support for unit tests, derived from the unit test: which can be leveraged without undue difficulty. With 1. What motivates developers to write unit tests? Maven [9], for example, there is a convention that The driving force behind unit testing are the describes how to write tests such that the build system can developers’ conviction and management find them, execute them, and finally prepare a report of requirements the outcome. Writing tests boils down to writing test 2. What are the dominating activities in unit methods, which are tagged with annotations testing? Writing new tests are perceived as less to mark the methods as being tests. If the test code starts dominant than writing, refactoring, and fixing to require complicated setup and runtime dependencies, the code. Often, a failing test s treated with an we are no longer dealing with unit tests. Here, the amendment of the test (rather than the code) or difference between unit testing and , a deletion of the test. which requires integration testing, can be a source of 3. How do developers write unit tests? Developers confusion. Often, the same underlying technologies and claim to write unit tests systematically and to libraries are reused between unit and functional testing. measure code coverage, but do not have a clear This is a good thing, as reuse is good in general and lets priority as to what makes an individual test you benefit from your expertise in one area as you work good. on another. Still, it can be confusing at times, and it pays 4. How do developers use automated unit test to raise your eyes now and then to see that you are doing generation? The main uses of automated test the right thing. generation are those that do not require any type The testability of such units is optimal and easily of specification. implemented. Units are exclusively responsible and are 5. How could unit testing be improved? independent of other units. Planning and designing Developers do not seem to enjoy writing tests; systems composed solely of such units would be ideal but they want more tool support in order to identify the possibility of the development of such systems what to test and how to produce robust tests. amounts to wishful thinking. Today’s systems are so large Our survey, on the other hand, is focused on the and intricate that the dismantling of a functional group definition of the term “unit test” and on the ability to into single units would be an exhausting process at best. differentiate between the various units participating in The majority of systems today consist of compound or unit testing. Naturally, we have chosen to explore the interdependent units. The interdependency of units option of academic publication. degenerates the testing scheme and its capabilities:

Journal of Information Technology Management Volume XXIX, Number 2, 2018 42

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Table 1: Results of Unit Testing Practices Survey 2006 [14]

Definition Strength Problem GUI test Unit identification Unit identification Test of What? Test of smallest unit or units Test scripts and harness surrounding modules maintenance Data structures Framework tailoring Structure-based Test framework How? Test selection Preferably automated Documentation Test metrics

Where? Solution domain Not found Not found

competency Independent test competence Who? By developer independence network introduction strategy When? Quick feedback Continuous regression test Stopping criteria

External requirement (safety) Why? Ensure functionality Cost versus value agile methods

UNIT TEST DEFINITION SURVEY Survey Questions To make sure our collection was representative, Our survey intends to look for most of the most of the papers we chose were published in the last publications that have appeared in major journals and at eight years. The distribution of the reviewed papers is major academic conferences from 2002 and later (see shown in Figure 1. The data gathered from each paper Appendix 1). We omitted repetitive papers and excluded included answers to the following questions: instructional books. Overall 112 papers were selected for 1. What categories and affiliations in unit test this survey, all of which are listed in Appendix 1. definitions can we identify? The following two selection criteria were 2. In addition to the formal definition within the employed: paper, what attributes of usage can be found in The term “unit test” appears in the paper’s  the employment of the term? title and abstract. 3. Is there a connection between the main topic of  The term “unit test” is used more than ten the paper and its usage of the term “unit test”? times in the paper, the assumption being that A textual and formal semantic analysis of the anyone who uses a term that frequently must papers convinced us that we needed to spend more time have a specific definition in mind. reviewing each paper to try to understand the intention of the precise usage of terms. To answer these questions, we had to understand and evaluate each of these papers.

Journal of Information Technology Management Volume XXIX, Number 2, 2018 43

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Figure 1: Articles Referred to by Publication Year

Categories  Methodology – The definition focuses on In our analysis of the content of the definitions, the methodology and technique used for the we identified the following categories of motives for unit test. defining unit test:  Automation – There is a direct relationship with test automation.  Atomic – The definition states that the unit focuses on the smallest indivisible  Domain – A specific domain is included in (atomic) fraction of code in the codebase. the definition.  Isolation – The definition states that the unit  Contribution – The definition contains a test is being administered for code that has declaration of the value of the unit test. been isolated from the rest of the program. A one-sample t-test was conducted to examine the differences between the definitions. Results showed  Code-related – The definition relates the unit to a specific code. that there were significant differences between all the definitions except for domain, environment, external  External dependency – The definition states dependency, and automation. See results in Table 2. that there is a need to externalize all We concluded from our analysis that many of the dependencies. definitions might belong to more than one category. Who is doing the test? – Who is performing  Figure 2 presents the most common category affiliations the unit test (e.g., )? found in our survey.  Environment – The definition includes the characteristics of the required test environment.

Journal of Information Technology Management Volume XXIX, Number 2, 2018 44

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Table 2: One Sample T-test report

Category (M, SD) t(79) p < 0.05

Code M=1.5, SD=6.68 2.019 Yes

Atomic M=1.3, SD=5.78 2.02 Yes

Isolation M=0.83, SD=3.7 2.0 Yes

Who is doing test M=0.53, SD=2.4 1.96 Yes

Contribution M=0.45, SD=2.2 1.96 Yes

Methodology M=0.45, SD=2.2 1.97 Yes

Environment M=0.23, SD=1.04 1.94 Yes

External dependency M=0.13, SD=0.64 1.75 Not significant

Automation M=0.13, SD=0.6 1.9 Not significant

Domain M=0.03, SD=1.6 1.43 Not significant

Figure 2: Categories of Definitions

Journal of Information Technology Management Volume XXIX, Number 2, 2018 45

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

At this stage, it was evident that there was no “classic” definitions in which the usage of xUnit test code agreed-upon definition. Thus, we first attempted to extract tools was assumed. When the definition was more affiliations and categories, relying on semantic general, we called it a “component” definition. Figure 3 relationships (semantic net) [4]. Definitions such as describes the distribution of the definitions using three atomic, isolation, and code-related were classified as categories.

Figure 3: Definitions of “Unit Test”

While most (i.e., 66%) of the definitions were between papers that centered on the use of a specific tool classical, we could not ignore the component definitions and those that were concerned with general use or with (24%); the remaining 10% represented cases where we theory. Almost half (47%) of the papers reported about could not extract a precise definition from the manner in new tool or solution in order to answer a need related to which the term was being used. the unit test as displayed in Figure 5.

"Unit Test" Usage Attributes Topic of the Paper and Usage Attributes As reflected in published academic papers (see A logistic regression was applied to examine the Appendix 1), emphasizing that the definition in itself is effect of the papers’ topic on the use of unit test (Cox & insufficient for an explanation of the usage of the term, Snell’s R2 < 0.1). We failed to notice significant relations Figure 4 maps how different publications use the term between the topic of the paper and the usage affiliation of “unit test” as they address their specific readerships. the term in this case (Figure 6). The theme of a given paper is sometimes correlated to the use of the term. We distinguished

Journal of Information Technology Management Volume XXIX, Number 2, 2018 46

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Figure 4: Attributes of Usage of the Term “Unit Test”

Figure 5: Topic Affiliation of the Research

Journal of Information Technology Management Volume XXIX, Number 2, 2018 47

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Figure 6: Topics of Papers and Affiliation of the Use of the Term “Unit Test”

Survey Summary Results in the use of the term? Emphasizing that the definition is insufficient for an explanation of 1. What categories and affiliations in the unit test the usage of the term, we distinguished between definition can we identify? There are basically papers that centered on the use of a specific tool two diverse ways to define the term “unit test”: and those that were concerned with general The classic way – About two-thirds of  usage or with theory. We failed to relate a the papers related to the unit test as the precise definition to specific usage smallest, isolated, atomic and code- terminologies. related test that is mainly performed by 3. Is there a connection between the main topic (or the developers. theme) of the paper and its use of the term “unit  The component way (24%) – The focus test”? Almost half (47%) of the papers were is on a unit of functionality, not published reports of a new tool or a solution necessarily on the perception of the unit provided to facilitate a need related to the unit test as the smallest, indivisible portion test. There was no significant connection of the program; here the unit test is between the topic of the paper and use of the administered mainly by testers. term in this case. The rest of the papers (10%) did not define the term at all and require further examination in order to reveal how they understand the OUR SUGGESTED DEFINITIONS meaning of the term “unit test.” This research illustrates that at the most general 2. In addition to a formal definition within the level unit testing refers to the practice of testing specific paper, what attributes of usage could be found functions, modules, or areas – or units – of software code.

Journal of Information Technology Management Volume XXIX, Number 2, 2018 48

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

This testing enables us to verify that the isolated piece of and building all of the affiliated infrastructures. Only then code functions as expected. In other words, for any could testers perform the specific test. Apart from running function and a given set of inputs, one can determine the program in a debug mode, when they needed to test whether the function is returning the proper values and the precise functionality they had to develop an isolation will smoothly handle failures during execution should mechanism to ensure the testing of specific code invalid input be provided. Developing software on a behavior. Unit tests are written as test classes with test module level basis allows for more straightforward unit methods. In the past, to display a similar behavior one had testing because the code under test has been isolated to develop a new code to mask the tested unit and had to and/or is independent of other procedures in the code inject artificial information into the tested object so that base. To enable well-defined unit testing, the code should the program could be executed (this was sometimes called be built with tight cohesion and loose coupling, with a a mock mechanism). In general, mocking refers to the use more significant number of smaller, more focused of replacement classes that are easily configurable to react functions that provide a single operation for a unique set to input and provide the output during testing rather than of data rather than large functions performing several the use of real classes. There are different related terms: different procedures. for example, a stub usually has a fixed default behavior, Unit tests are short code fragments created by whereas a mock-up typically must be set up as part of a or occasionally by a white box (structural) test in order to verify expected interactions. Usually, the or grey box (functional and structural) [2] – that is, by tested program and the actual final code were very testers during the development process. The unit test is different – because of the need to add instrumented most often considered a lower test level. Unit testing is, observational code to the application code. It was only roughly speaking, the testing of a small portion of the when the world moved into interpreter mode program code in isolation from the test code. This is considered the execution that isolation was enabled more naturally, and first testing step during development and the most unit test infrastructure appeared. granular aggregate of the testing scheme. By its very An important aspect of unit testing is the nature, the unit test is attached to the code from which it environment where the class within the unit under test is is created. At the beginning of software development being operated [1]. Figure 7 demonstrates the internal history, when attempting to test a particular functionality, environment needed for the execution of the test. Unit test testers were faced with the challenge of needing to have infrastructure was designed as a pivotal element to enable the program ready and operational before attempting to isolation of the tested code before the full implementation execute the test; the would not let the code be of each object. executed before completing all the necessary declarations

Figure 7: The Environment of a Test Class

Journal of Information Technology Management Volume XXIX, Number 2, 2018 49

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

In the modern development environment, a unit Differentiation of the Unit Test from an test method is a method without input parameters. It Integration Test represents a test case and typically executes a method where the class has fixed arguments (use of assert class) For clarification, this paper sees an essential and verifies that it is returning the expected answer [17]. distinction between the unit test, which refers to the use of Unit tests include a test oracle that verifies the observed xUnit testing [18], and component testing, which is more behavior with the expected result. By convention, the test general and means the testing of only a portion of the oracle of a unit test is encoded using assertions. The test program. Another distinction might derive from the fails if any assertion fails or if an exception occurs [17]. abstraction level, in which case the unit test will usually An assert class failure will usually stop all execution, and be affiliated with the code itself, and the component test no exceptional treatment mechanism will be able to may be expressed in functional or business terminology. salvage the program execution during integration testing. The unit test may form the basis for component Such a mechanism prevents us from refining the testing testing that can be considered a higher level of testing. process, and all failures are equally critical to the testing Component testing is sometimes known as module and process. The isolation and the predefined static value of program testing. Component testing is mostly done by the the tested parameters lead by their very nature to the tester. Component testing may be done in isolation from negation of any kind of integration between the different the rest of the system depending on the development life elements of the software under test. cycle model chosen for that application. In such cases, the missing software is replaced by stubs and drivers that simulate the interface between the software components in a simple manner.

Figure 8: Architectural Buildup of a Testing Level

The greatest pitfall might be encountered when projects, such as the agency’s “Flight Software Product developers test too large a unit or when they consider a Line” [6], one could see the possibility of the creation of method to be a unit. This is particularly true if you do not unit tests without the use of traditional xUnit understand Inversion of Control, in which case your unit infrastructure (and without the use of the assert class). tests will always turn into end-to-end integration testing. It is therefore vital to explicitly define the Unit testing should test individual behaviors – and most terminology when talking about the unit test and unit test methods have many behaviors [6]. In some NASA integration. For example, Unit Test Virtualization [3] is a

Journal of Information Technology Management Volume XXIX, Number 2, 2018 50

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

new approach for reducing the execution time of long test a mock is implemented in place of the actual network suites. VmVm (pronounced “vroom-vroom”) is an easy- connection, the tests will continue to pass. Thus, mocking to-use device for the implementation of Unit Test has shown itself to be a proven and effective technique Virtualization wherever Java is used. It allows us to and is a widely adopted practice. Having the right tool perform a pre-initiation of the test environment and to absolutely makes the difference. avoid the need for restarting the execution process Another important aspect worth considering is following each failure. The idea behind the approach is the identity and research specialization of those who are obvious but fails to distinguish between the unit test and administering the different testing levels. We argue here unit test integration nor does it address issues raised by that another set of skills and knowledge should be the use of an assert class. This approach is perhaps more considered when you must choose the right person to plan suitable for component tests. and perform these testing activities. Another example is work with Open Worm, as It is our experience that when developers focus reported by Sarma et al. [15], where a unit test applies to on the unit level, it becomes more difficult for them to the smallest functional unit of code and has no external comprehend the state of their code in the larger scheme of dependencies. On the other hand, tests intended to verify the code base. Consequently, their units cannot directly that different components are working together are interact with other sections of the overall codebase, as the classified as integration tests. They assess whether developed units are essentially too isolated. multiple components have been integrated correctly. On the other hand, the refactoring process of a Some of the tests discussed below focus on another tested unit of code [12] can complicate our understanding distinction that the authors make, rather than of a unit and a unit test. Let us assume, for example, that distinguishing between ordinary verification tests we have successfully tested and implemented an isolated (designed to verify that the code is working as intended) portion of code with no external dependencies. Once all and model validation tests (designed to validate a model the tests pass, the developers must refactor their code. against experimental data). During the refactoring process, the previously identified Usually, the isolation issue is presented through unit of code can be extracted to multiple methods or the use of technology [8]. While mock classes. If the extracted classes are themselves considered objects help us remove unnecessary dependencies in tests units, then we have fundamentally undermined our and make the tests fast and reliable, the use of mocks original test. Even more complicated than method manually written in C++ is problematic: extraction, code that is migrated to an abstract parent class  Someone must implement the mocks. The cripples our definition of a unit even further. Abstract job is usually tedious and error-prone, and it classes cannot be tested because their code cannot be is no wonder that researchers go great instantiated. Do we relate to the inheriting classes of the distances to avoid it. abstract class in order to test their shared parent?  The quality of these manually written mocks Under ideal code conditions for a unit (isolation, is somewhat unpredictable. You might see atomic, etc.), we can address the aptitude of the some polished mocks, but you will also see programmers or testers for adequately understanding a some that have been hacked up in a hurry given program’s requirements. A correctly written test and which have a large number of ad hoc can be executed on an isolated section of code and can restrictions. pass, if, unfortunately, the developer did not accurately  The knowledge you gain from using one understand the necessary requirements. As a result, all the mock cannot be applied to the next one. tests will pass although many of them did not actually validate the intended functionality of the code. Discussions and Considerations Regarding the degradation of code, all matter is subject to natural deterioration. The robustness of the In contrast, Java and Python programmers have matter largely influences the pace at which the some excellent mock frameworks that automate the deterioration occurs. Software, albeit abstract and creation of mocks. Tests that rely on external API intangible, is equally vulnerable to the deterioration and (application protocol interface), network connections, degeneration phenomena found in the physical world user input, threading, and other external dependencies around us. More research is imperative if we are to better must be mocked. A passing test must continue to be understand the underlying source and effects of software administered as long as the codebase remains in the same degradation (code rot) and the significance of unit tests in state. If the network connection suddenly becomes the creation of a more durable, more robust software disconnected, the code will subsequently fail. However, if product.

Journal of Information Technology Management Volume XXIX, Number 2, 2018 51

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

CONCLUSIONS test as part of the software quality assurance during a CI/CD project. Since unit testing is considered an It is apparent that the definition of the term “unit important link in a chain of quality activities aimed to test” is neither clear nor precise. Most of the literature we improve organization outcome, understanding that will have reviewed tend to consider the structural aspect of the assist organizations in focusing their quality goals and term – atomic, isolation, etc. – and relates the action to an recruitment needs. X-unit testing infrastructure. About 24% of the sources Additionally, we can identify other future define “unit test” more loosely and display a higher level research directions: of abstraction that does not restrict the definition, and  Checking whether the current tools enable which allows an integrative portion of the program to be the two activities – We propose a included in the unit being tested. continuation of research on the existing  Unit testing – This is the act of testing an testing tool with the aim of determining isolated, atomic, and code-related portion of whether there are tools that enable both unit the software (a unit). It is evident that the test definitions noted above. right candidates to perform this activity are  The present research study did not explore the developers themselves. the conditions under which unit testing is  Component testing – This is the testing of a enabled. It may be interesting to identify the functional and more substantial portion of terms of feasibility for the proper use of unit the program (a component). We claim that tests, or, in other words, to determine when another set of skills and another kind of unit testing cannot be performed. knowledge are needed to perform this  Another possible direction is to try to bridge portion of the work. the two definitions of unit testing by It is vital in our opinion to distinguish between proposing a metamorphosis of unit test the two aspects and to allocate the best resources for each artifacts into component testing. assignment, or, alternatively, to train the developers and  The recognition of the open-source provide them with new skills and knowledge so that they community as a mainstream constituent in can perform these two categories of testing. global software development has produced As stated in the Introduction, the move to CICD new research motivations. GitHub, Stack shows shifting the center of testing activities into unit Overflow, and Twitter, to mention but a few testing. Therefore, we can identify growing importance of social profiles, are the main vehicles for the role of the unit testing level, which emphasizes the individuals today to promote and publish importance of the distinction between classical unit their opinions, styles, and code. Testing and testing level and the integration (component) level. We quality constitute core values in most open- recommend that the two aspects of testing be separated in source projects. The comparison and the early stages of software development. contrast of the definitions, perceptions, and implementations of quality in open-source Threat to Validity community projects versus the competitive One of the possible threats to the validity of a software industry could provide us with an survey’s findings is related to the appropriateness of the even more profound understanding of data collection and sampling approaches. To attain testing and, more specifically, unit testing. optimal objectivity in our study, it was important to collect a representative, statistically sound sample (our REFERENCES full list of papers is included in Appendix 1). Another possible threat may stem from the fact that we had to [1] Arcuri, A., G. Fraser, and J.P. Galeotti. Automated subjectively interpret each definition and each use of the unit test generation for classes with environment terms in question. To prevent discrepancies, we had two dependencies. in Proceedings of the 29th independent teams repeat the paper reviewing process. ACM/IEEE international conference on Automated . 2014. ACM. [2] Beizer, B. and J. Wiley, Black Box Testing: The Next Step Techniques for Functional Testing of Software and To fully understand the impact to the quality of a Systems. Software, IEEE, 1996. 13(5): p. 98. software product, it is important to follow this research [3] Bell, J. and G. Kaiser. VMVM: unit test with a field study of the actual implementations of unit virtualization for Java. in Companion Proceedings

Journal of Information Technology Management Volume XXIX, Number 2, 2018 52

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

of the 36th International Conference on Software AUTHOR BIOGRAPHIES Engineering. 2014. ACM. [4] Clark, P., et al. Exploiting a thesaurus-based Hadas Chasidim is a faculty member at semantic net for knowledge-based search. in Shamoon College of Engineering's Department of AAAI/IAAI. 2000. Software Engineering. She holds B.Sc., M.Sc. and Ph. [5] Daka, E. and G. Fraser. A survey on unit testing degrees in Industrial Engineering from Ben-Gurion practices and problems. in 2014 IEEE 25th University (2013). Her research deals with software International Symposium on Software Reliability quality and testing, human-computer interaction, usable Engineering. 2014. IEEE. privacy and security. Hadas is the head of software [6] Ganesan, D., et al., An analysis of unit tests of a quality development track at Software Engineering flight software product line. Science of Computer department in Beer Sheva campus. Programming, 2013. 78(12): p. 2360-2380. [7] Hunter, A., Are unit test overused in Simple talk Dani Almog is a senior researcher and lecturer 2012: https://www.simple-talk.com/dotnet/net- on Software Quality and test automation. Contributing framework/are-unit-tests-overused/ researcher and lecturer at Shamoon College of [8] Lewis, W.E., and continuous Engineering (SCE) and Ben Gurion University (BGU). quality improvement. 2016: CRC press. Currently, the main topic of his studies are Software [9] Marschall, Philippe. "Detecting the methods under quality testing and fundamentals, addressing software test in java." Bachelor thesis (2005). engineering needs. Dani has very vast experience in the [10] Michael, H. 2016. Understanding and improving industry – former test automation managing director for continuous integration. In Proceedings of the 2016 Amdocs product development division. Dani is serving at 24th ACM SIGSOFT International Symposium on the Advisory Board of ITCB (Israeli Testing Certification Foundations of Software Engineering (FSE 2016). Board) and Leading the ISTQB academia research group ACM, New York, NY, USA, 1066-1067. DOI: stream. https://doi.org/10.1145/2950290.2983952 [11] Naik, K. and P. Tripathy, Software testing and Dov Benyomin Sohacheski is currently a quality assurance: theory and practice. 2011: John graduate student and holds a B.Sc. in Software Wiley & Sons. Engineering from SCE Shamoon College of Engineering. [12] Passier, H., L. Bijlsma, and C. Bockisch. He is a researcher and an assistant lecturer at SCE. Dov Maintaining Unit Tests During Refactoring. in has over 8 years of experience in the software Proceedings of the 13th International Conference development industry as a PHP and Python developer, on Principles and Practices of Programming on the working on projects from web-based software services to Java Platform: Virtual Machines, Languages, and data-analytics and statistical platforms. Tools. 2016. ACM. [13] Raymond, E.S., The art of Unix programming. Mark L. Gillenson is Professor of Business 2003: Addison-Wesley Professional. Information and Technology in the Fogelman College of [14] Runeson, P., A survey of unit testing practices. Business and Economics of the University of Memphis, IEEE software, 2006. 23(4): p. 22-29. USA. He received his B.S. degree in Mathematics from [15] Sarma, G.P., et al., Unit testing, model validation, Rensselaer Polytechnic Institute and his M.S. and Ph.D. and biological simulation. arXiv preprint degrees in Computer and Information Science from The arXiv:1508.04635, 2015. Ohio State University. He is an associate editor of the [16] Ståhl, D., and Bosch, J. "Modeling continuous Journal of Management and has published in integration practice differences in industry software such leading journals as MIS Quarterly, European Journal development." Journal of Systems and Software 87 of Information Systems, and Information & Management. (2014): 48-59. His latest book is Fundamentals of Database Management [17] Tillmann, N. and W. Schulte. Parameterized unit Systems 2nd edition, 2012, John Wiley & Sons. tests. in ACM SIGSOFT Software Engineering Notes. 2005. ACM. Dr. Robin Poston is the Director of the System [18] Verona, Joakim, Michael Duffy, and Paul Testing Excellence Program for the FedEx Institute of Swartout. Learning DevOps: Continuously Deliver Technology at The University of Memphis, and she is a Better Software. Packt Publishing Ltd, 2016. page Professor and Dept Chair of Business Information and 88 Technology at the Fogelman College of Business & Economics at The University of Memphis. She is holds the

Journal of Information Technology Management Volume XXIX, Number 2, 2018 53

THE UNIT TEST: FACING CICD – ARE THEY ELUSIVE DEFINITIONS?

Papasan Family Professorship for Exemplary Leadership, serves as the Interim Associate Dean of the Graduate School. Dr. Poston is a recipient of the Memphis Alumni Association Distinguished Teaching Award and she leads the annual International Research Workshop on Advances and Innovations in Software Testing attended by over a hundred academic and industry professionals.

Prof. Shlomo Mark currently the dean of the faculty of engineering at SCE College of Engineering - Ashdod campus is a professor of software engineering, and the Head of the NMCRC – the Negev Monte Carlo Research Center at SCE. His main research interests are Quality in software engineering, Agility and Agile life cycle, scientific computing and software Life Cycle for Scientific software product, Computational Modeling for Physical, Environmental and Medical Application.

Journal of Information Technology Management Volume XXIX, Number 2, 2018 54