Interactive Execution of Non-Interactive Test Framework Features - in Collaboration with Ericsson AB

Bachelor Degree Project

Interactive Execution of Non-interactive Test Framework Features - In Collaboration with Ericsson AB

Author: N iklas Eriksson Supervisor: F rancis Palma Semester. V T 2020 Subject: Computer Science

Abstract

Some software libraries and frameworks get built to assist engineers in other domains than software engineering. However, these tools can be too complicated for these engineers to utilize due to their inexperience in software engineering. The engineers must then request assistance from more experienced software engineers. And when experienced software engineers must assist, they can spend less time on their original work. This problem got identified at Ericsson AB for one of their test automation frameworks for radio development, where test cases had to be written in pure Java code by test engineers without the test engineers always having the necessary skills to do so. In collaboration with Ericsson, we address this problem by applying design science to develop a novel technique that enables interactive execution of parts of this framework, called MJE Actions. We then present a prototype that embodies this novel technique. It enables test engineers at Ericsson to interactively execute MJE Actions via command-line without having to write a single line of Java code.

Keywords: command-line interface, fluent interface, software design, software architecture, test automation framework, method chaining.

Preface

The basis for this research initially stemmed from my passion for software design and development. Due to my inexperience in developing command-line interfaces, I saw collaborating with Ericsson AB to design and develop one as an excellent opportunity to learn and gain more experience. In truth, I could not have achieved my current level of success without all the support from supervisors and family. First of all, I thank my parents, who have provided me with love and understanding. Secondly, I thank my university supervisor and my supervisor at Ericsson that has guided me through the process of succeeding with my research. Thank you all for your unswerving support.

Contents 1 Introduction 5 1.1 Background 5 1.2 Problem formulation 6 1.3 Motivation 7 1.4 Related work 7 1.5 Objectives 8 1.6 Target group 9 1.7 Scope/Limitation 9 1.8 Outline 9 2 Method 10 2.1 Design Science 10 2.2 Instantiation of Design Science 11 2.3 Reliability and Validity 11 2.4 Ethical Considerations 11 3 Objectives for a solution 12

4 Implementation 13 4.1 Abstract Design 13 4.2 Concrete Design 15 4.3 Development 17 5 Results 23

6 Analysis 29 6.1 Fulfillment of requirements 29 6.2 Evaluation of the prototype 30 6.3 Concluding discussion 30

7 Conclusion & Future work 32

References 34

Appendix A - The requirements of the prototype 36

Appendix B - The test cases for the prototype 37

1 Introduction This report presents the execution of a bachelor's degree project at Linnaeus University of 15 credits. The project is a collaboration with the telecom company Ericsson AB targeting a problem about their test engineers having a hard time using a Java Test Framework practically and efficiently to fulfill their needs and goals.

1.1 Background Most software libraries and frameworks are designed and implemented for trained software engineers and software developers. Some software packages are built for domain experts in other domains than computer science or software engineering. These domain experts might not have enough programming skills to efficiently use various software tools [1], resulting in heavily requested assistance by more experienced software developers. In Development Services and Infrastructure (DSI), Ericsson develops Continuous Integration framework, test tools, test automation framework, test orchestration, and test environment. The mission for DSI is to enable cost-efficient radio development worldwide with their development environment as a competitive advantage, both for present and future business needs. In the test department of DSI, they develop the test automation framework, called MJE (MSRAN 1 JCAT [Java Common Automatic Test] Extension). It is developed in Java and uses, except for functionality developed in the organization, open-source software like TestNg [2], and Spring beans [3]. MJE provides functionality that provides nodes (e.g., base stations), user equipment (e.g., mobiles, simulators), logging features, and much of other common functionality to the test programs written in Java, i.e., test cases. In other words, an MJE test program is a Java program implemented on top of the MJE framework. For using MJE as described above, test engineers and developers need to write Java programs that implement the test cases. Finally, they can execute these programs in the test environment. In case of failure, the test engineers can study the logs to determine the root cause. MJE has three abstraction levels where the top-level, i.e., Action, could be

1 MSRAN is an organization name in Ericsson AB.

used to write short test programs by using the built-in functionality provided by the framework. The test engineers would benefit considerably from improved test support that enables interactive execution of MJE actions via a command-line interface.

1.2 Problem formulation Having to request more experienced software developers for assistance when utilizing a given software library demands more time and other resources. This approach is not practical since the domain experts must spend much time on things that are not in their area of expertise, and the software developers assisting the domain experts must spend much time doing tasks that are very much routine for them.

This degree project investigates how to enable interactive MJE Action execution via a command-line-interface(CLI).

Currently, MJE Actions can only execute via invocations declared in the source code of an MJE-based test program. The command-line based solution makes it possible to execute MJE Actions as if they were commands, interactively, and with faster turnaround time; from writing a test case to the execution of the test case. As a proof of concept, we would like to find out if there is a way to enable interactive execution of MJE Actions via the command-line and develop a prototype as a proof of concept.

The core problem addressed is How can we create a novel technique for a CLI that supports the interactive execution of MJE Actions?

A new technique will assist test engineers to explore and find new ways to test the System Under Test [10] that was never thought of before. The prototype will have generated commands based on the existing MJE Actions, that the user can run. After the completion of this degree project, Ericsson will further investigate the prototype by letting test engineers try it, to find out if the test engineers would gain any value of the prototype and if further development

should take place or not.

1.3 Motivation By nature, an MJE-based test program is non-interactive. The life cycle of writing a test case and executing it takes in the best case some hours, up to a few days. Also, for writing the test case, Java competence is needed, which is not the case for all test engineers. An interactive and command-line based abstraction on top of MJE is supposed to help the above obstacles. The interactivity of the system increases the possibility of doing Exploratory Testing [11] and finding the weak parts of the System Under Test. The test engineers who lack Java experience would also be able to use the system quickly to realize their test ideas. The time for performing spontaneous and quick testing gets shortened. Ericsson also has a similar system, MoShell, which operates on a single node (e.g., base station). The simplicity and popularity of an interactive and command-based test system are already proven. Ericsson is interested in evaluating the same idea (a command-based interactive system) with MJE, which, unlike MoShell, operates on multiple nodes, tools, and protocols. Ericsson, with its mobile networks, among other product offerings, has a significant impact on society. Therefore, we believe that the findings of this study will positively affect society by prototyping easier test writing for Ericsson mobile networks.

1.4 Related work There exists a considerable body of work regarding CLI based development. Wesley Bland et al. developed a command-line interface (CLI) to interact with the software tool called OSCAR (open-source cluster application resources) [4]. Users could only interact with OSCAR via a graphical user interface (GUI). The CLI is an alternative user interface to the GUI and is a close mirror of the GUI, providing the same functionality [4]. In our case, there is no user interface to the MJE framework already, as test programs are written manually in Java. Similar to OSCAR CLI, our CLI does not change any functionality of the underlying system. Accelerating Test Automation Platform (ATAP) was developed to make test automation accessible to non-programmers by Anurag Dwarakanath et al. It works by automatically converting English-like test scripts to machine-executable code using Selenium WebDriver [1]. Comparing the

ATAP project with ours, we find the similarity in that both want to enable non-programmers to utilize an underlying test framework. Martin Fowler created a framework for thinking about domain-specific languages (DSLs) [5]. It divides DSLs into internal DSLs, external DSLs, and language workbenches. An external DSL is an entirely separate language having a full written parser. In contrast, an Internal DSL is an idiomatic way of using a general-purpose language. A language workbench is a tool that, other than parsing and code generation, provides a better editing experience for DSL users. In all these three cases, it is important to have an explicit semantic model so that they form a "thin skin" over an underlying library [5]. Considering this, we see that our CLI can probably be regarded as a kind of DSL in that it is a "thin skin" over the MJE framework and that we should, therefore, use his ideas and framework to create a useful CLI. Picocli [6] is a framework for creating Java command-line applications. It aims to be the easiest way to create rich command-line applications to be run off the JVM. To implement a command in Picocli, one has to annotate a Java class using Picocli's command-annotation. Then, by having the annotated Java class implementing the java.lang.Runnable or java.lang.Callable interface the command can be executed in one line [6]. We use Picocli as our command-line argument parser. We chose Picocli since it was easy to use and because of its popularity among many software projects. Among many, projects like Maven, JUniT4, Debian, and Groovy have adopted Picocli. For example, in Groovy, all command-line tools are based on Picocli [7]. JLine is a Java library for the handling of console input. Except for its additional features, JLine is very similar in functionality to the GNU Readline [8]. We use JLine as our console input handler since Jline and Picocli complement each other very well and have minimal functional overlapping [9]. As we can see, there are several techniques and tools available. However, none of these solves the problem of enabling interactive execution of MJE Actions via command-line.

1.5 Objectives The first objective for a solution to the addressed core problem is to find a novel technique that enables the interactive execution of MJE Actions via command-line. The second objective for a solution is to implement a prototype-tool that

embodies the found novel technique that enables interactive execution of MJE Actions. To fulfill the objectives above, we must first investigate the MJE Framework to understand how MJE Actions work. From this project, we first expect to find a way to execute MJE Actions via commands interactively. We then hope to be able to execute these commands from an isolated shell-like environment.

1.6 Target group The target group is, except for the Ericsson test engineers, developers looking for a solution to implement a command-line-based application on top of a programming language framework to increase the availability and usability of the framework.

1.7 Scope/Limitation To limit the scope of the thesis project, only a few MJE Actions are to be executable from the system prototype. We set this limitation because we consider the process of implementing so that all MJE Actions can be executed from the system prototype as a very repetitive and time-consuming process since there exist over 240 of them, not bringing much value to the project. Due to the project's coupling to an Ericsson project, the findings and implementations are possibly not fully applicable to all external users and organizations.

1.8 Outline The following chapter describes how we apply design science to solve the core problem addressed in this study. We then present the objectives for a solution and how they became defined in chapter 3, before presenting how we design and develop the novel technique and prototype that embodies this technique in chapter 4. Chapter 5 presents our results with a demonstration of the prototype, including a traceability matrix showing that the prototype fulfills its requirements specification. To analyze our results, we evaluate the prototype and novel technique through a comparison in chapter 6, ending with a concluding discussion of our study. We end the report with chapter 7 by concluding our findings and by presenting future work.

2 Method In this chapter, we explain the method we apply to target the addressed problem, followed by a description of the method's instantiation in this study. We then present any threats concerning the validity and reliability of our study and ethical considerations.

2.1 Design Science As for the method, we use Design Science [12] (DS) with a process model consisting of 5 activities. Figure 2.1 shows a visual image of the process model.

Figure 2.1: Shows the activities in the design science process model used.

The process starts with the activity of identifying the actual problem to be solved (see subchapter 1.3 for our problem formulation). To then justify the research study regarding solving the identified problem, motivation is written (see subchapter 1.4 for our motivation). The following activity is to specify how to solve the actual problem by defining the objectives for a solution. The identified problem, together with knowledge of what is possible and feasible, infers the objectives for a solution . When the objectives for a solution are defined, the following activity is to design and develop the actual artifact to solve the identified problem. In our case, it is the prototype.

After the creation of the artifact, it gets demonstrated through examples that show how it solves the identified problem. In the last and final activity, the artifact becomes evaluated through a comparison. The comparison compares the observed results from the demonstration with the defined objectives for a solution .

2.2 Instantiation of Design Science We instantiate the use of DS in this study by doing the first activity in the DS process, which is to identify the problem and justify the research. We first identify the problem and then define the research question, which is, “How can we create a novel technique for a CLI that supports the interactive execution of MJE Actions?”. We then justify our research by motivating why answering this research question is crucial. In the following activity of the DS process, we frame the prototype requirements, together with stakeholders, at Ericsson, which we then apply to define the objectives for a solution. For this, we conduct a series of interviews and stakeholder workshops and use the results to derive a requirements specification. The third activity focuses on designing and developing a solution based on the objectives for a solution . We present the solution in the fourth activity via a demonstration together with a requirements traceability matrix. In the fifth and last activity of the DS process, we evaluate the solution by comparing the demonstration with it to show that it fulfills the objectives for a solution and thereby the core problem.

2.3 Reliability and Validity The reliability of a study indicates how reproducible the results are when repeating the study under the same conditions. In this current study, we cannot find any significant reliability threats. External validity in the context of a scientific study is determined based on to which extent the results can be generalized outside the context of the specific study. We acknowledge an external validity threat of that the results are built upon a framework of Ericsson AB and can, therefore, be hard to generalize. In contrast to external validity, the internal validity of a study is determined based on how well it can rule out alternative explanations for its findings. We feel that for this current study, the internal validity is very high due to our chosen method.

2.4 Ethical Considerations Ethical considerations are not relevant to this project since it is a pure software development project.

3 Objectives for a solution To understand and define feasible and possible objectives for a solution, we conducted stakeholder interviews (or similar workshops) with the Ericsson supervisor and MJE framework developers. We conducted interviews with the Ericsson supervisor to understand his expectations, and with the MJE framework developers to know what is feasible and possible to accomplish as a solution. The interviews took place iteratively, and in each iteration, we started by interviewing the Ericsson supervisor to understand some of his expectations of our solution. We then analyzed those expectations to understand if those were possible and feasible to meet, from an MJE framework technical perspective by interviewing the MJE developers. Before each interview, we prepared ad hoc questions on a piece of paper or document that we discussed during the interview. Not all interviews were very much prepared, depending on what the goal of the corresponding interview was. We prepared less for the interviews with the Ericsson supervisor but more for the interviews with the MJE developers because then we could base our questions on the results from the recent interview with the supervisor. After the interviewing phase, we analyzed the results with the Ericsson supervisor and translated them into the requirements of a solution. We then selected the most critical requirements and filtered away those not crucial for this proof-of-concept study. From the interviews and the requirements specification, we defined the objectives for a solution . The first objective for a solution is to find a novel technique that enables the interactive execution of MJE Actions via command-line. The second is to implement a prototype-tool that embodies the found novel technique that enables interactive execution of MJE Actions. The final requirements specification locates in Appendix A and their corresponding test cases in Appendix B.

4 Implementation In this chapter, we describe the MjeShell prototype. We first describe the design and then the prototype. See Appendix A for a complete requirements specification for the prototype.

4.1 Abstract Design An initial investigation took place to analyze the MJE framework and its abstraction layer MJE Actions. It resulted in the conclusion that all implemented MJE Actions followed a specific design pattern when being implemented. Each MJE Action implemented uses method chaining [13] and is categorized as a retrieve or perform MJE Action. Each retrieves MJE Action is executing a read operation and is therefore not changing the state of the System Under Test (SUT). Each performs MJE Action is executing a write operation which might change the state of the SUT. Because the MJE Actions are executed using method-chains, we decided that the commands that are executed from the prototype (i.e., CLI) to execute MJE Actions should be parent-subcommand (PS) built commands. Each part of the PS command shall direct-map to the corresponding method in the corresponding method-chain. Figure 4.1 shows this idea by showing a straightforward example of how one (random) Java method-chain is direct-mapped to by a PS built command.

Fi gure 4.1: Mapping of (random) PS command “method1 method2 method3” to a Java method chain.

During the execution, each part of the PS command knows its corresponding method of the corresponding method-chain, enabling the

generation of the actual Java method-chain to execute an MJE Action. With runtime compilation, the method-chain then gets executed. Figu re 4.2 shows a high-level dynamical view of h ow these PS built commands get executed. Figure 4.3 then shows a high-level static view of the prototype (i.e., the CLI in the figure) of how the end-user can execute commands that, in turn, execute underlying MJE Actions Java statements.

Figure 4.2: Shows a sequence diagram of how the end-user can execute a command via the CLI (i.e., prototype) to execute MJE Actions.

Fi gure 4.3: Shows a high-level static view of the prototype’s (i.e., CLI’s) relation to the end-user and the MJE Framework.

4.2 Concrete Design Each MJE Action from the end-users perspective is composed of the components described in table 4.4. For this project, all these components are relevant except for the ActionFilter component. Needed for executing an MJE Action, is access to the class Actions, which provides access to all the available MJE Actions. After the selection of an MJE Action, the ActionRequest is method-chained to it. The ActionRequest itself consists of one or more methods that are chained together in a particular order. When executing an MJE Action chained with an ActionRequest, an ActionResult is the output, which can be referenced by a Java object for later use. Code 4.5 shows a use case scenario where two MJE Actions implement a test case to verify that there are no new alarms on the SUT after executing a particular write operation.

Name of the component Description of the component

ActionRequest Is the container of all parameters needed for the action. It contains zero or more mandatory parameters, and zero or more optional attributes.

ActionResult Is the output of an Action execution.

Action Is the Action itself which takes an ActionRequest and produces an ActionResult.

Actions Is the class providing access to all available Actions.

ActionFilter Provides Action filters, selecting a subset of all available Actions. Table 4.4: The components of an MJE Action.

// Get access to all available MJE Actions // This is only necessary to do once in each test case. Actions actions = Actions.create(n ew TestBase());

// Retrieve and store all active alarms on all available nodes FmGetActiveAlarmResult alarmsBefore = actions.fmGetActiveAlarm().retrieve();

// Execute the Moshell command 'cvls' on all available nodes actions.cliSendMoshellCommand().command(' cvls') .perform();

// Retrieve and store all active alarms on all available nodes FmGetActiveAlarmResult alarmsAfter = actions.fmGetActiveAlarm().retrieve();

// Verify that there are no new alarms on the available nodes actions.fmGetActiveAlarm().verifyNoNewAlarms(alarmsBefore, alarmsAfter); Code 4.5: Shows a code snippet of the MJE specific code necessary to test that there are no new alarms after a particular write operation.

Due to the requirement R4, that all the prototype’s commands that are to execute underlying MJE Actions shall follow a specific design pattern, a design pattern was created. Because MJE Actions are composed using method-chains, and that those method-chains follow a specific tree-like structure, the decision was that the commands to execute underlying MJE Actions should follow that same tree-like structure. We chose to design these tree-like structures using a PS hierarchy, as mentioned earlier. In this hierarchy, each part (i.e., top-level parent command, subcommand, nested subcommand, command parameter, or option) presents one method of the corresponding MJE Action method-chain. By doing so, these PS built commands can generate the actual Java method-chains that the MJE Actions are composed of and compile these during runtime, which will be further explained later in this chapter. In these PS hierarchies that we have designed, each top-level parent command presents an Action component. The optional attributes of the corresponding ActionRequest are presented as command parameters or options to the top-level parent command. The main reason for that decision is that the methods presenting optional attributes are not order-dependent of each other and must always appear before the methods presenting mandatory parameters in the corresponding method-chain. However, the methods presenting mandatory parameters are order-dependent of each other. The first mandatory parameter, in the valid order, is therefore presented as the subcommand of the top-level parent command, and the rest of the mandatory parameters as nested subcommands in the valid order. Code 4.6 shows a composed MJE Action statement in Java

code where the Action component is presented first in the method-chain, then the optional attributes, and lastly, the mandatory parameters. Figure 4.7 then shows how the MJE Action statement of code 4.6 looks like when followin g the PS hierarchy design described above.

// Get access to all available MJE Actions // This is only necessary to do once in each test case. Actions actions = Actions.create(n ew TestBase());

// Select one Action actions .fmGetActiveAlarm() / / The selected Action.

// Create the ActionRequest

// Set two optional attributes .testStepComment(" get all the active alarms for all available nodes of the type Rbs.") .byNodeType(NodeType.Rbs)

// Set one mandatory parameter. .retrieve(); Code 4.6: Shows a code snippet explaining how an MJE Action is composed.

4.3 Development The Java command-line interface framework called Picocli supports the idea of PS hierarchies. Picocli works in the way that each command gets implemented by creating a corresponding Java class that implements the Java interface java.lang.Runnable and has a Picocli specific annotation that sets the name of the command, i.e., what the user must enter to execute the corresponding command, and the subcommands of the command itself. Each command can also have options and parameters [6]. The java.lang.Runnable interface forces each command Java class to implement a method named run. This method shall contain the business logic of a command Java class as the Picocli framework will invoke this method during the execution of a command. By representing each method in an MJE Actions method-chain with one corresponding command Java class, a direct mapping establishes between the

MJE Action method-chains (i.e., MJE Action compositions), and the PS built commands. Figure 4.7 shows how this direct mapping looks like to the MJE Action composed in code 4.6. Note from the figure that a decision was that all commands, command parameters, and options of each PS command should have the same name as the method mapped to has. This decision was to ensure that the end-user, used to compose MJE Actions as Java statements, shall know what to write when using the prototype, to get the intended results.

fmGetActiveAlarm --testStepCommand " get all the active alarms for all available nodes of the type Rbs" - -byNodeType Rbs retrieve Figure 4.7: Shows the (one line) command that is a direct mapping to the MJE Action composed in Code 4.6.

Picocli enables an execution strategy of the PS commands so that each subcommand is executed separately in the occurring order, starting from the top-level parent command. We use that execution strategy. We then let each command Java class of each PS command know the exact syntax of the direct-mapped part (i.e., method) of the corresponding MJE Action method-chain. That makes it possible to concatenate each command Java class’s known syntax information together during execution. That, in turn, creates the actual string literal that the actual Java statement (i.e., method-chain) equals. Through runtime compilation, the prototype executes these string literals to execute actual test cases built of MJE Actions. Each command Java class knows its corresponding syntax information by implementing a toString method. Each toString method returns the syntax that the corresponding part of the corresponding MJE Action method-chain equals. Some methods in the method-chains use function parameters. Therefore, their corresponding direct mapping command Java classes’ toString methods must dynamically generate their result based on set command parameters and options. Code 4.8 shows the implementation of the Java command class direct mapping to the Action component fmGetActiveAlarm. Note, at code 4.8, how the toString method is being invoked by the method named run, to add the result to a buffer. The buffer contains an ordered list of all the methods of the corresponding MJE Action statement (i.e., method-chain) that the corresponding PS command has

generated during execution. Code 4.9 Shows how this buffer becomes executed; as a test case, after that, each command Java class of a PS command has added its known syntax to it.

@CommandLine. Command(name = " fmGetActiveAlarm", description = "returns all active alarms", subcommands = { Retrieve.class, VerifyNoNewAlarms.class }) public c lass F mGetActiveAlarm e xtends R eusableOptions i mplements Runnable {

@Override public String t oString( ) { String result = " actions.fmGetActiveAlarm()"; result += s uper. toString(); / / get optional parameters. r eturn result; }

@Override public v oid r un( ) {

/ * This method gets invoked by the Picocli framework.

It will, based on its direct mapping to fmGetActiveAlarm, generate a string literal that will be concatenated with the other subcommands of the parent-subcommand command used to generate MJE Action statements with the Action component being fmGetActiveAlarm. */

MjeShellLauncher.getUnderlyingMjeActionStatementBuffer() .addToUnderlyingMjeActionStatement(t his. toString()); } } Code 4.8: Shows a code snippet of the implementation of the Java command class direct mapping to the Action component fmGetActiveAlarm, excluding imports and package names.

public s tatic v oid e xecuteMjeShellSpecificCommand( String[] arguments, String line) { / * Note that code has been removed from this code snippet for the reader in mind. Also some lines have been formated a little to become more readable. */

/ / Get Mje Action Statement generated by the command executed. String mjeActionStatement = getUnderlyingMjeActionStatementBuffer() .getUnderlyingMjeActionStatement(); i f (mjeActionStatement.length() > 1 / * greater than one since the string always contains a semicolon */) { Reflect.compile( " com.ericsson.msran.mjeshell.MjeActionCommandExecutor", " package com.ericsson.msran.mjeshell;\n" + / / Note that imports has been removed from this code snippet " class MjeActionCommandExecutor {\n" + " public void execute() {\n" + " com.ericsson.msran.actions.Actions actions =\n" + " com.ericsson.msran.actions.Actions.create(new TestBase()" + " );\n" + / / Here we put our generated MJE Action statement. mjeActionStatement + " }\n" + " }\n" ).create().call(" execute") ; System.out.println(" Executed action statement(s):\n" + mjeActionStatement); / / Prepare the buffer for new statements to be executed. getUnderlyingMjeActionStatementBuffer() .clearUnderlyingMjeActionStatement(); Code 4.9: Shows a code snippet on how the system executes the generated MJE Action statements.

For the system to execute commands, it must be able to read user input. It uses Jline for this, which reads console input and lets the system decide what to do with it afterward [8]. Code 4.10 shows how the system does this.

public s tatic v oid m ain( String[] args) { / /Note: boiler code has been removed from this code snippet.

/ / start the shell and process input until the user quits // with Ctl-D. String line; w hile (t rue) { / / Read user input line. line = reader.readLine(prompt, rightPrompt, (MaskingCallback)

null, n ull) ; i f (line.matches(" ^\\s*#.*") ) { c ontinue; } ParsedLine pl = reader.getParser().parse(line, 0); String[] arguments = pl.words().toArray(n ew String[0]); String command = Parser.getCommand(pl.word()); i f (builtins.hasCommand(command)) { / / it's a builtin jline command, e.g., 'history'. builtins.execute(command, Arrays.copyOfRange(arguments, 1, arguments.length), System.in, System.out, System.err); } e lse { / / The line matches a system specific command. executeMjeShellSpecificCommand(arguments, line); } } Code 4.10: Shows a code snippet on how the system reads console input that gets executed as commands.

The system supports the execution of multiple MJE Actions combined by not immediately executing MJE Action statements generated by PS commands that have a trailing semicolon. Instead, it executes them together with the MJE Action statement generated by the next PS command executed that did not have a trailing semicolon. That, combined with the fact that the system supports storing of ActionResult, implies that the user can combine and execute as many MJE Actions statements in a test case as she wants to. Figure 4.11 shows how to execute the test case in code 4.5 via the system.

fmGetActiveAlarm retrieve --toVariable alarmsBefore; cliSendMoshellCommand command --name cvls perform; fmGetActiveAlarm retrieve --toVariable alarmsAfter; fmGetActiveAlarm verifyNoNewAlarms --before alarmsBefore --after alarmsAfter Figure 4.11: Shows how to execute the test case in code 4.5 via the system.

The system uses Picocli features to implement common suggestions and autocomplete functionality that gets triggered when the user presses tab. It provides the command record that tells the system to record every

successfully executed command. The user must then run the command stop to stop the recording and to be able to save the recorded commands into an executable file. Executable files can then be executed from the system via the command execute .

5 Results Table 5.1 shows a Requirement Traceability Matrix overall of all the gathered requirements for the prototype, where the first column is the requirement id. The second column is the corresponding requirement description. The third column contains all the corresponding test case IDs and the fourth the status of those. The fifth column shows if the Ericsson supervisor has got his expectations met regarding that specific requirement. Note that meeting the Ericsson supervisor’s expectations for each requirement means that the prototype becomes validated with a positive result.

Req Req Test Status Ericsson ID Description ID Supervisor’s expectation is met

R1 The prototype shall be executable T1 T1 - pass check from a terminal on an Ericsson server with the permissions to execute MJE Actions.

R2 The prototype shall interpret each T2, T2 - pass, check read line as one single command. T3, T3 - pass, T4, T4 - pass, T5 T5 - pass

R3 The prototype shall, by default, T2, T2 - pass, check execute a command when the T3, T3 - pass, user presses enter. T4, T4 - pass, T5 T5 - pass

R4 The commands that are based on T2, T2 - pass, check underlying MJE Actions in the T3, T3 - pass, prototype shall follow a specific T4, T4 - pass, design pattern. T5, T5 - pass, T6, T6 - pass, T7, T7 - pass

R5 The prototype shall support the T5 T5 - pass check execution of multiple commands

together.

R6 The prototype shall provide a T2, T2 - pass, check command to retrieve all the T3, T3 - pass, active alarms of the specified T5 T5 - pass nodes and optionally assign them to a variable.

R7 The prototype shall provide a T4, T4 - pass, check command to execute MoShell T5 T5 - pass commands over the MJE CLI towards MoShell.

R8 The prototype shall provide the T5, T5 - pass, check necessary commands to verify T10 T10 - that there are no new alarms after pass the state has been changed on the specified nodes.

R9 When the user presses tab, the T6 T6 - pass check prototype shall suggest matching commands based on what the user has already entered as an input.

R10 When the user presses tab and T7 T7 - pass check there is only one command matching what the user has already entered as input, the prototype shall autocomplete that one command.

R11 The prototype shall provide a T8, T8 - pass, check command that, after it has been T9 T9 - pass run, records all commands that were successfully executed.

R12 The prototype shall provide a T9 T9 - pass check command that stops the recording of all commands that were executed successfully and lets the user save what was recorded into a file.

R13 The prototype shall provide a T10 T10 - check command that executes an pass executable file, consisting of executable commands. Table 5.1: Requirement Traceability Matrix with the requirements for the prototype.

Figure 5.2 shows a UML use case diagram of how the end-user, i.e., Ericsson test engineer, can interact with the prototype to fulfill a goal, and a number identifies each use case. By presenting these use cases, we show how our results satisfy several real-world needs and goals of the test engineers. Three corresponding PS commands realize use cases 1-3, enabling test engineers to execute two MJE Actions interactively. The fourth use case is the execution of multiple commands together, enabling the execution of multiple MJE Actions combined. That helps in creating more advanced and useful test cases. The last two use cases, 5 and 6 make the prototype more user friendly, creating a better user experience for the test engineers. Figure 5.3 shows a UML activity diagram combining all the use cases of figure 5.2. It illustrates an actual real-world example of how an Ericsson test engineer can use the prototype. Figure 5.4 then shows a screenshot of executing the first six steps of the UML activity diagram in figure 5.3, and figure 5.5 shows the last two steps.

Figure 5.2: Shows a UML use Figure 5.3: Shows a UML case diagram representing how activity diagram combining the end-user, i.e., Ericsson test all the use cases of figure engineer, can interact with the 5.2. resulting prototype.

Figure 5.4: Shows a screenshot of executing steps 1-6 of the UML activity diagram in figure 5.3.

Figure 5.5: Shows a screenshot of executing steps 7 and 8 of the UML activity diagram in figure 5.3.

6 Analysis The core problem addressed in this study is, “How can we create a novel technique for a CLI that supports the interactive execution of MJE Actions?”. Through stakeholder interviews and workshops, we have defined the objectives of a solution to solve the core problem. The objectives for a solution are to find a novel technique that enables the interactive execution of MJE Actions via command-line and to implement a prototype-tool that embodies the found novel technique that enables interactive execution of MJE Actions. In defining the objectives, we have also defined a complete requirements specification for the prototype. To evaluate the result, we compare our prototype demonstration in the result chapter with the objectives for a solution . Before that, we argue the fulfillment of the prototype requirements that concerns the interactive execution of MJE Actions via the command-line.

6.1 Fulfillment of requirements The fourth requirement specifies that “The commands that are based on underlying MJE Actions in the prototype shall follow a specific design pattern”. The commands of the prototype follow the design pattern presented in subchapter 4.1 and 4.2. Hence we have fulfilled the requirement. “The prototype shall support the execution of multiple commands together”. Our prototype meets this fifth requirement since it allows multiple commands to execute together if they end with a semicolon except for the last one. Then the prototype executes all the MJE Actions statements that the commands have generated together as one test case (i.e., one Java program). We argue that the sixth requirement - “The prototype shall provide a command to retrieve all the active alarms of the specified nodes and optionally assign them to a variable” - is satisfied. The prototype provides the command fmGetActiveAlarm. When executing the command, the user can specify which nodes to retrieve active alarms from and to then optionally assign the result to a variable for later use. The seventh requirement specifies that “The prototype shall provide a command to execute MoShell commands over the MJE CLI towards MoShell”. The prototype provides the command cliSendMoshellCommand, allowing the user to execute any MoShell command. Therefore this requirement is arguably met.

We finally declare the fulfillment of the eighth requirement - “The prototype shall provide the necessary commands to verify that there are no new alarms after the state has been changed on the specified nodes”. The command fmGetActiveAlarm enables the user both to retrieve the active alarms at a particular time point as well as to compare the active alarms of different time points. Figure 5.4 in the result chapter shows a scenario where the active alarms before and after a write operation are retrieved and then compared to verify that there are no new alarms.

6.2 Evaluation of the prototype To compare our prototype (i.e., the artifact of the DS process) with the defined objectives for a solution, we, via the result chapter, demonstrate the prototype use cases in figure 5.2 and how to fulfill these in a corresponding activity diagram, in figure 5.3. With figure 5.4 and 5.5, we present screenshots of executing the steps of the activity diagram to prove that the prototype works. We can arguably show the fulfillment of the objectives for a solution based on the four first use cases of the prototype presented in figure 5.2, and the fact that the corresponding commands enabling the use cases are executing MJE Actions. The last two use cases are use case 5, which enables recording and saving of successfully executed MJE Action commands into an executable file and use case 6, which enables execution of these executable files. These two use cases do not solve the core problem but make it much easier to use the prototype since they allow the user to store successfully executed commands to then analyze the execution history, as well as re-execute the commands. They, therefore, add much value for the test engineers and Ericsson. From the above, we can conclude that the prototype arguably fulfills the objectives that we defined for a solution and that the objectives have helped solving the core problem of this study.

6.3 Concluding discussion We begin the DS process by identifying the problem and then justifying our research through motivation. We continue the DS process by trying to understand how we can feasibly and possibly solve the addressed problem, through stakeholder interviews and workshops. From that, we define and specify the objectives for a

solution . After specifying the objectives for a solution , we design a novel technique that enables interactive execution of MJE Actions via command-line, to fulfill the first objective. We then develop a prototype that embodies this technique to fulfill the second objective. We show the results from the development of the prototype by first showing a requirements traceability matrix, and then by demonstrating how the prototype fulfills real-world goals and needs of the Ericsson test engineers. In the final activity of DS, we compare the observed results from the demonstration with the objectives for a solution . From this, we can see that our technique and prototype fulfill the objectives for a solution and, therefore, arguably solves the core problem that this study addresses.

7 Conclusion & Future work We can conclude from our results that it is possible to enable the interactive execution of MJE Actions via the command-line. Although our results are very Ericsson specific, we believe that our solution can help and inspire other developers to do the same but for other frameworks and features, primarily built of method-chains. Nowadays, it is common to create fluent interfaces, i.e., object-oriented APIs whose design relies extensively on method chaining [14]. Ericsson does not explicitly call their MJE Actions a fluent interface, even though it probably is one. That means that our findings can probably be used by other projects where there is a need for a CLI to a fluent interface, or similar. As mentioned, our results are primarily relevant for Ericsson. However, we hope that they can also be relevant to others in the industry with similar problems to be solved. Because our prototype only enables interactive execution of 2 out of approximately 240 MJE Actions, further development is required to enable interactive execution of all MJE Actions to fulfill all the needs of the Ericsson test engineers (regarding MJE Actions). Since it takes some time to implement these PS bu ilt commands, we have investigated if it would be possible and feasible to generate these commands automatically. We have found a promising solution via a thorough investigation, where these commands can be implemented using the Java Reflection application programming interface (API) [15]. The Reflection API makes it possible to inspect the MJE Actions to understand how to implement the corresponding PS commands. Since our solution and design to implement these PS commands ought to be generic, we believe it is feasible and possible to accomplish the automatic generation of these PS built commands to execute MJE Actions. Above is critical to accomplish since it would necessitate plenty of resources to manually implement the PS commands and then maintain them. And if adding and combining other features of the MJE framework with MJE Actions, the CLI can become an even more powerful tool that test engineers can use to a very great extent. Further investigation would, therefore, be necessary to decide what these other MJE features would be and how to implement these into the CLI. However, before that, it is crucial to get the prototype out to the hands of the Ericsson test engineers and make them try it

out to see if their experiences are positive or not, which can tell if further work would pay off.

References

[1] A. Dwarakanath, D. Era, A. Priyadarshi, N. Dubash and S. Podder, "Accelerating Test Automation through a Domain Specific Language," 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), Tokyo, 2017, pp. 460-467. doi: 10.1109/ICST.2017.52 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7928002&isnumb er=7927908

[2] "TestNG - Welcome", Testng.org, 2020. [Online]. Available: https://testng.org/doc/. [Accessed: 06- Feb- 2020].

[3] "Spring - Bean Definition - Tutorialspoint", Tutorialspoint.com, 2020. [Online]. Available: https://www.tutorialspoint.com/spring/spring_bean_definition.htm. [Accessed: 06- Feb- 2020].

[4] W. Bland, T. Naughton, G. Vallee and S. L. Scott, "Design and Implementation of a Menu Based OSCAR Command Line Interface," 21st International Symposium on High Performance Computing Systems and Applications (HPCS'07), Saskatoon, SK, 2007, pp. 25-25. doi: 10.1109/HPCS.2007.14 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4215574&isnumb er=4215546

[5] M. Fowler, "A Pedagogical Framework for Domain-Specific Languages," in IEEE Software, vol. 26, no. 4, pp. 13-14, July-Aug. 2009. doi: 10.1109/MS.2009.85 URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5076452&isnumb er=5076443

[6] "picocli - a mighty tiny command line interface", Picocli.info, 2020. [Online]. Available: https://picocli.info/. [Accessed: 06- Feb- 2020].

[7] "remkop/picocli", GitHub, 2020. [Online]. Available: https://github.com/remkop/picocli. [Accessed: 06- Feb- 2020].

[8] "jline/jline3", GitHub, 2020. [Online]. Available:

https://github.com/jline/jline3. [Accessed: 06- Feb- 2020].

[9] "remkop/picocli", GitHub, 2020. [Online]. Available: https://github.com/remkop/picocli/tree/master/picocli-shell-jline3. [Accessed: 06- Feb- 2020].

[10] "System under test", En.wikipedia.org, 2020. [Online]. Available: https://en.wikipedia.org/wiki/System_under_test. [Accessed: 06- Feb- 2020].

[11] "Exploratory testing", En.wikipedia.org, 2020. [Online]. Available: https://en.wikipedia.org/wiki/Exploratory_testing. [Accessed: 06- Feb- 2020].

[12] Ken Peffers, Tuure Tuunanen, Marcus A. Rothenberger & Samir Chatterjee (2007) A Design Science Research Methodology for Information Systems Research, Journal of Management Information Systems, 24:3, 45-77, DOI: 10.2753/MIS0742-1222240302

[13] "Method chaining", En.wikipedia.org, 2020. [Online]. Available: https://en.wikipedia.org/wiki/Method_chaining. [Accessed: 25- Mar- 2020].

[14] "Fluent interface", En.wikipedia.org, 2020. [Online]. Available: https://en.wikipedia.org/wiki/Fluent_interface. [Accessed: 05- May- 2020].

[15] "Trail: The Reflection API (The Java™ Tutorials)", Docs.oracle.com, 2020. [Online]. Available: https://docs.oracle.com/javase/tutorial/reflect/index.html. [Accessed: 05- May- 2020].

Appendix A - The requirements of the prototype R1: The prototype shall be executable from a terminal on an Ericsson server with the permissions to execute MJE Actions.

R2: The prototype shall interpret each read line as one single command.

R3: The prototype shall, by default, execute a command when the user presses enter.

R4: The commands that are based on underlying MJE Actions in the prototype shall follow a specific design pattern.

R5: The prototype shall support the execution of multiple commands together.

R6: The prototype shall provide a command to retrieve all the active alarms of the specified nodes and optionally assign them to a variable.

R7: The prototype shall provide a command to execute MoShell commands over the MJE CLI towards MoShell.

R8: The prototype shall provide the necessary commands to verify that there are no new alarms after the state has been changed on the specified nodes.

R9: When the user presses tab, the prototype shall suggest matching commands based on what the user has already entered as input.

R10: When the user presses tab and there is only one command matching what the user has already entered as input, the prototype shall autocomplete that one command.

R11: The prototype shall provide a command that, after it has been run, records all commands that were successfully executed.

R12: The prototype shall provide a command that stops the recording of all commands that were executed successfully and lets the user save what was recorded into a file.

R13: The prototype shall provide a command that executes an executable file, consisting of executable commands.

Appendix B - The test cases for the prototype

Test id: T1 Description: This manual test case tests so that you can start the prototype. Tested requirements: R1 Prerequisites: ● Intellij version 2019.3 is running with the project source code on an Ericsson server with the permissions to execute MJE Actions. Test execution steps: 1. Click the run button in Intellij. 2. Copy the command line that Intellij uses to execute the source code located as the first line in the Main run-window. 3. Open a standard desktop-terminal on the server. 4. Paste the copied execution command line and press enter. Expected result: ● A terminal prompt appears in the terminal with the text MjeShell>.

Test id: T2 Description: This manual test case tests so that we can retrieve all the active alarms on all available nodes. Tested requirements: R2, R3, R4, R6 Prerequisites: ● Test case T1 has been executed with its expected result. Test execution steps: 1. Enter fmGetActiveAlarm retrieve into the prototype and press enter. Expected result: ● A table with information regarding all the active alarms gets outputted for each node that was operated on by the prototype.

Test id: T3 Description: This manual test case tests so that we can retrieve all the active alarms on all available nodes that matches the node type Rbs or G2Rbs. Tested requirements: R2, R3, R4 R6 Prerequisites: ● Test case T1 has been executed with its expected result. Test execution steps: 1. Enter fmGetActiveAlarm --byNodeType Rbs --byNodeType G2Rbs retrieve into the prototype and press enter. Expected result:

● A table with information regarding all the active alarms gets outputted for each node that was operated on by the prototype.

Test id: T4 Description: This manual test case tests so that we can execute Moshell commands in the prototype. Tested requirements: R2, R3, R7 Prerequisites: ● Test case T1 has been executed with its expected result. Test execution steps: 1. Enter cliSendMoshellCommand command --name cvls perform into the prototype and press enter. Expected result: ● A line in the format of INFO Response from Moshell command ‘cvls’ command response file: ‘’ gets outputted for each node that was operated on by the prototype.

Test id: T5 Description: This manual test case tests so that we can verify that there are no new variables on all available nodes after that the state has been changed. It also tests so that it is possible to execute multiple commands together. Tested requirements: R2, R3, R5, R6, R7, R8 Prerequisites: ● Test case T1 has been executed with its expected result. ● Test case T2 and T4 gets their expected result when executed. Test execution steps: 1. Enter fmGetActiveAlarm retrieve --toVariable alarmsBefore; into the prototype and press enter. 2. Enter cliSendMoshellCommand command --name cvls perform; into the prototype and press enter. 3. Enter fmGetActiveAlarm retrieve --toVariable alarmsAfter; into the prototype and press enter. 4. Enter fmGetActiveAlarm verifyNoNewAlarms --before alarmsBefore --after alarmsAfter and press enter. Expected result: ● For each available node a line in the format of INFO Verify successful - no new alarms found gets outputted by the prototype.

Test id: T6 Description: This manual test case tests so that the user gets suggestions on possible commands based on what has been already entered by the user. Tested requirements: R9 Prerequisites: ● Test case T1 has been executed with its expected result. ● T5 gets its expected result when executed. Test execution steps: ● Enter fmGetActiveAlarm verifyNoNewAlarms and press tab. Expected result: ● The prototype suggests the following: ○ --after ○ --before

Test id: T7 Description: This manual test case tests so that the user gets autocompletion when there is only one single command matching what the user has already entered as input. Tested requirements: R10 Prerequisites: ● Test case T1 has been executed with its expected result. ● T2 gets its expected result when executed. Test execution steps: 1. Enter fmG into the prototype and press tab. Expected result: ● The prototype auto completes the fmG into fmGetActiveAlarm and waits for further input.

Test id: T8 Description: This manual test case tests so that you can start to record without verifying what has been recorded. Tested requirements: R11 Prerequisites: ● Test case T1 has been executed with its expected result. Test execution steps: ● Enter r ecord into the prototype and press enter. Expected result: ● System is recording… is being outputted by the prototype.

Test id: T9 Description: This manual test case tests so that we can record and then stop the recording to save what has been recorded. Tested requirements: R11, R12 Prerequisites: ● Test case T8 has been executed with its expected result and then immediately after that T5 has been executed with its expected result. Test execution steps: ● Enter stop --saveTo /tmp/myProgram.mjes into the prototype and press enter . Expected result: ● The file /tmp/myProgram.mjes should contain the following lines in the specified order: ○ fmGetActiveAlarm retrieve --toVariable alarmsBefore; ○ cliSendMoshellCommand command --name cvls perform; ○ fmGetActiveAlarm retrieve --toVariable alarmsAfter; ○ fmGetActiveAlarm verifyNoNewAlarms --before alarmsBefore --after alarmsAfter

Test id: T10 Description: This manual test case tests so that we can execute executable files from the prototype. Tested requirements: R8, R13 Prerequisites: ● The file /tmp/myProgram.mjes should contain the following lines in the specified order: ○ fmGetActiveAlarm retrieve --toVariable alarmsBefore; ○ cliSendMoshellCommand command --name cvls perform; ○ fmGetActiveAlarm retrieve --toVariable alarmsAfter; ○ fmGetActiveAlarm verifyNoNewAlarms --before alarmsBefore --after alarmsAfter

Test execution steps: ● Enter execute /tmp/myProgram.mjes into the prototype and press enter. Expected result: ● For each available node a line in the format of INFO Verify successful - no new alarms found gets outputted by the prototype.