Requirements ambiguousness pitfalls Exploring the concept of a lightweight ambiguity tactile check, based on a finite dictionary and non-context awareness, to limit natural language ambiguousness pitfalls in business requirements The Good, the Bad….and the Ugly Student: Martin Wilmink Identiteitsnummer: 835864893 Datum rapport: 25 July 2016 Datum presentatie: Planned 6 sept 2016 Requirements ambiguousness pitfalls RT Exploring the concept of a lightweight ambiguity tactile check, based on a finite dictionary and non-context awareness, to limit natural language ambiguousness pitfalls in business requirements The Good, the Bad….and the Ugly. Opleiding: Open Universiteit, faculteit Management, Science & Technology Masteropleiding Business Process Management & IT Open University of the Netherlands, faculty of Management, Programme: Science & Technology Master Business Process Management & IT Cursus: IM9806 Afstudeertraject Business Process Management and IT Student: Martin Wilmink Identiteitsnummer: 835864893 Datum rapport: 25 July 2016 Afstudeerbegeleider: Christoph Bockisch Meelezer: Stef Joosten Examiner: Stef Joosten i Abstract Requirement documentation form one of the important artefacts of the requirements engineering process. One of the downturns of using natural language in the context of requirement specifications is its natural tendency to ambiguousness. A simplified assessment on phrases that influence ambiguity has the potential to be beneficent for the analyst writing requirements documentation to stay away from the obvious pitfalls. The paper introduces the idea of a lightweight tactile check that annotates weak and strong phrases that are know to influence ambiguity. The performed assessment validated the ability to identify these phrases without any context analysis. The used finitie dictionarys that contain the weak and strong phrases are based on previous work (Carlson & Laplante, 2014; Femmer et al., 2014; Wyatt, DiStefano, Chapman, & Aycoth, 2003). The annotated phrases are evaluated by field experts and labeled to indicate if ambiguity is influenced and if the annotation is perceived as helpful to further reduce ambiguity. Finaly discussing the the quality of detection and the perceived helpfulness of annotating weak and strong phrases leading to the research conclusions and proposed future research. 1.1 Categories and Subject Descriptors [Software Engineering] Requirements specification 1.2 General terms Requirements Engineering, Quality Assurance, Natural language, Dumb Tooling 1.3 Keywords Business requirements; Requirements engineering; Software quality; Natural language; Ambiguity; Dumb tooling; Tactile Check ii Summary Requirement documentation form one of the important artefacts of the requirements engineering process. One of the downturns of using natural language in the context of requirement specifications is its natural tendency to ambiguousness. A simplified assessment on phrases that influence ambiguity has the potential to be beneficent for the analyst writing requirements documentation to stay away from the obvious pitfalls. The objective of this research is to understand the ability of a light tactile check approach to give useful feedback about the influence of weak and strong phrases on the ambiguity in requirement documents. The main research question is worded as follows: How do business analysts perceive the effectiveness of a lightweight tactile check used to evade the most common natural language ambiguousness pitfalls in business requirements? The research in this thesis is performed as an embedded single case study. Where the case subject is KLM Engineering & Maintenance with two different, Business IT related projects as subject of analysis. Based on the defined dictionary with weak and strong phrases, a ‘lightweight tactile check’ tool is designed to identify and tag these phrases in a given requirements set. Three domain experts assessed the annotated phrases to determine if the annotation is performed correctly, what the effect on the ambiguousness is in the given context is and if the assessment is perceived useful. This resulted in data to calculate the precision, recall and F-score values as an indication of the reliability of the annotation process. The assessment on ambiguousness influence and perceived helpfulness provided data to assess the overall usability of the tactile check method. In this case study two requirements sets where used containing a total number of 293 requirements. With 87 weak phrases and 367 strong phrases annotated. All weak phrases and a selection of the strong phrases where assessed. With an average F-score of 84% for weak phrase annotation and 94% for strong phrase annotation this is considered sufficient to assert that, the tactile check is reliable enough to be practical in annotating the weak and strong phrases. Annotating the weak phrases is perceived as helpful in further reducing the ambiguity level of the requirement. And although the strong phrases were recognized with high accuracy, the experts considered the annotation of strong phrases as not helpful and without potential of further reducing the ambiguity. The fact that the experts rated the annotation of strong phrases as little helpful was a slight surpise: We expected that the annotation of strong phrases would indicate maturity of requirements and help to focus on improving the less mature ones. One reason for this outcome may be that our study is based on final versions of requirements documents after the requirements engineering phase, instead of during the requirements engineering. As future work, we therefore propose to repeat this study in a setting with a reference measurement performed on a document in an early stage of requirements engineering, which contains more ambiguity. Comparing the results of this reference measurement with the same measurement taken from a later version of the requirements document could establish whether applying the tactile check approach throughout the requirements engineering phase can be more efficient. iii Table of contents 1 Table of Contents Abstract ...................................................................................................................................... ii 1.1 Categories and Subject Descriptors .......................................................................................... ii 1.2 General terms ........................................................................................................................... ii 1.3 Keywords .................................................................................................................................. ii Summary ................................................................................................................................... iii Table of contents ....................................................................................................................... iv Table of figures .......................................................................................................................... vi 1 Introduction ....................................................................................................................... 1 1.1 Research context ...................................................................................................................... 2 1.2 Relevance ................................................................................................................................. 3 1.3 Research scope ......................................................................................................................... 4 1.4 Main research theme and objective ......................................................................................... 5 2 Research, philosophy, approach and strategy ................................................................... 7 2.1 Research Philosophy................................................................................................................. 7 2.2 Research approach ................................................................................................................... 7 2.3 Research Methodological choice .............................................................................................. 8 2.4 Research strategy ..................................................................................................................... 8 2.5 Time horizon. ............................................................................................................................ 8 3 Research design ................................................................................................................. 9 3.1 Global design ............................................................................................................................ 9 3.2 Detailed Research approach ..................................................................................................... 9 3.3 Reliability and validity aspects................................................................................................ 14 3.4 Data access and resources ..................................................................................................... 16 4 Research execution .......................................................................................................... 17 4.1 Data access and resources ..................................................................................................... 17