IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version htuesntol hr hi pnosaotsfwr but software about opinions their share A only managed? shows not Amazon and on users reviews captured “user that software the be a through there should browse quick Is thus work: requirements and and this software engineering for behind observation valuable be question can This which main software rationale” [16]. about the decisions evolution to making led and when development, users of design, attention more input giving the started to vendors software stores, app and [12]. in sketches found or conversations often team also as is such to artifacts Rationale criteria informal the [2]. and alternatives issues, the the includes evaluate analysts, solve and to This designers explored [7]. by alternatives document the encountered taken issues to be were or questions artifacts should decisions the design project rationale certain and the why requirements Ideally, software in [10]. in captured concern [6], major [3], a and engineering been design has managing rationale decades, last requirements justifications the and Over reasons decisions. the behind or sharing and practice, capturing management on belief, Rationale reason.”. focuses opinion, underlying an of or principles phenomena, controlling of tion user in developers. and deliberation for higher findings reviews supporting 13% the the synthesizing to for discuss and up We importance communities was 99%. rationale precision reaching level, the values values review recall recall the top with On with 98%. decisions, and of predicting alternatives for 80% with and around text, sentences precision review a the achieved of terms, tree explore Regression, syntax influential metadata, Logistic to review the and reviews. on sentences the Bayes, trained from Naive review used concepts Classifier, Vector rationale labeled then mine Support manually can We we of concept. accurately and how set pervasive compatibility, truth most performance, the the like issues represent found as criteria and usability reviews such assessment the concepts in studied that considered upgrading, rationale also alternatives about of or We encountered frequency e.g. applications. occurrence decisions, software the their switching how 52 or justify investigated for installing, and we reviews analysis, argue grounded content 32,414 users a peer Through studied and Store. approach We Amazon theory the reviews. in online applications written software rationale in on users view software different by a In takes project paper reusing This and design beliefs. organizing knowledge. capturing on and and on decisions requirements focuses and opinions, management rationale decisions, engineering, human behind ihteicesn ouaiyo oilmda srforums, user media, social of popularity increasing the With Merriam-Webster to According Terms Index Abstract iigUe ainl rmSfwr Reviews Software from Rationale User Mining Rtoaerfr oteraoigadjustification and reasoning the to refers —Rationale ApAayis ainl,Rve Mining Review Rationale, Analytics, —App [email protected] .I I. NTRODUCTION nvriyo Hamburg of University abr,Germany Hamburg, ia Kurtanovi Zijad rationale c ´ s“h explana- “the is h eerhmtosaddt sdi hswork. this in used data and methods research the in work IX. related Section in sketch paper VII, the conclude Section and in VIII, Section Section in limitations include results that and research the sentences VI discuss and We concepts. reviews rationale detect to configurations we Third, accuracy V). the (Section learning rationale machine how supervised discuss be we automatically can Second, in concepts criteria IV). and (Section alternatives as reviews such user concepts its and rationale and of III) (Section rationale denote We users engineering. requirements rationale on in perspective user-data-driven grounded, new a duce work. empirical introduces this automatically II underlying framework Section to reviews. methodological the approaches the from evaluate store information for such and mine software rationale theory reviews grounded user Amazon’s a develop the for to on is studying goal available Our (www.amazon.com). empirically products by software rationale users crowd- documentation. a requirements and considered design be of might extension require- rationale based new even user and Finally, arguments, derive ments. criteria, to decision testers a own and designers, performance): their analysts, usability, for (e.g. knowledge used alternatives useful criteria these the other evaluate and (e.g. to improved workarounds) considered user alternatives an configurations, Second, the products, reasoning. on reveal their decisions might and would rationale users their the This base of transparent. understanding teams more knowledge software needs tacit help and users’ make preferences First, would about benefits. rationale several bring user product”). might capturing the feedback or purchased have debates negative, not users’ would huge I a that that’s knew me I without For if their support? and – phone describe more offers or longer and speeds”), did processor conclusive norton your everything down does weighting “it (e.g. bu the about insights provide CLX-3160FN”), Samsung printer/scanne laser report also efis nrdc h eerhqetosadte describe then and questions research the introduce first We intro- we First, threefold. is work our of contribution The understanding towards step first a takes paper This in rationale the managing and identifying Systematically [email protected] alternatives issues decisions fvrosmcielann prahsand approaches learning machine various of nvriyo Hamburg of University abr,Germany Hamburg, eg teesn rvrfrm multifunction my for driver no “there’s (e.g. I R II. ai Maalej Walid eg ddyuko htQiknno Quicken that know you “did (e.g. osdrdwt hi evaluation their with considered ESEARCH detected D qualitatively ESIGN quantify nue etne with sentences user in h occurrence the eciehow describe compare criteria IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version p trssc sGol lyo p trs Amazon Stores, App or Play Google as such stores GitHub using on app 2016, crawler May available and freely 2014 the November between Amazon I). from (Table categories 17 from applications The it 52 store. includes the of for in list to list category final vote software three a each selected or from We compiled applications reviews. review four manually many a We with applications on helpfulness. popular its comment reviewer indicate to the Other to allowed date. whether submission are review reviewer, the metadata users the and and review product, rating, of user the from the The purchased name text, scale feedback. review the a text the a including on title, write the products research and of rate the consists stars five can build to users to one Amazon, applications the In these crawled data. then for We study. reviews to online applications the store software below. described phases main four includes Method Research B. heterogeneous and to large a aims input. from techniques user RQ3 concepts learning rationale distributed. the machine mine and different to provide combined compare to and are evaluate aims rationale they user RQ2 how of concepts and criteria). various about decisions assessment evidence explain quantitative or and argue issues which to rationale, (e.g. apply user of users theory concepts a develop includes to aim we RQ1, With questions: research three following information. the rationale on mine focus We automatically investi- and to reviews approaches software gate in rationale user explore tatively Questions Research A. Select&applica*ons& 1.#Data#Collec,on# 1 hnw rwe h eiw o h eetdapplications selected the for reviews the crawled we Then Collection: Data 1) which framework, methodological our overviews 1 Figure ma- of configurations various different the can of accurately How distribution frequency RQ3 the is How and reviews, online RQ2 in rationale denote users do How RQ1 quanti- and qualitatively to is paper the of goal overall The Crawl&reviews& tp:/ihbcmasl/mzndwlae,acse nMrh2016 March on accessed https://github.com/aesuli/Amazon-downloader, Applica'on*list* dataset* Reviews/app** So9ware*Store* Amazon* i.1 vriwo u eerhmtoooywt orphases. four with methodology research our of Overview 1. Fig. rmteue eiw n sentences? concepts and reviews rationale user mine the from techniques learning chine reviews? the over concepts rationale include? they do types) (information concepts which Open,&axial&&&selec*ve& Theore*cal&sampling& Update&coding&guide& 2.#Grounded#Theory# Ra'onale*concepts* Review*samples* coding& Coding*guide* efis eetdfo h Amazon’s the from selected first We 3.#Content#Analysis# Manual&peer&coding& Create&coding&form& Conflict&resolu*on& Stra*fied&sampling& Coding*sample* Coding*form* Coding*results* Truth*set* 1 oprdt mobile to Compared . 4.Automated#Classifica,on# Comparison&of&results& features&&&classifiers& Experiments&with& Cross&valida*on& Indica've* F1Escores* Precision,*recall,** ra'onale* mining*user* Insights*on** configs .* ua oes h olwst nwrR2adt raea create to and RQ2 RQ3. answer answering by to for involving sample set was document truth goal [17], a The Robillard by of coders. assessment human described and manual as Maalej systematic techniques the and analysis [20] content Neuendorf manual the of sentences. we example extracted iteration, and each definitions After particularly code syntax. the observations, and refined our keywords about special notes those about to made codes and assigned continuously im- aspects We an to. related refer getting and they users on aspects of argumentation focused of the of we new sample understanding coding, proved potential theoretical selective open with a data During and only insights. used including axial, we sentences and iteration, (open, reviews each iterations In multiple coding). in dry the analysis in disagreements. the coders the discussed four from and involved guide them, We the interviewed coders. runs, refined the and of runs feedback dry we suggests, coding [20] 8 Neuendorf As conducted coding. for guidelines as concepts, additional hints the and counter-examples, of and definitions examples clear review the detailed tasks, on the instructions and includes process It coding [20]. [17], website peer-coders between project creation guide the coding the The from for Guide analysis. downloaded used content the guide be was during coding can and that set a goal truth and of the The creation theory of the data. the was describe in phase that evidence this answer of on to output to based method final qualitative theory [27] systematic, a Corbin a develop Grounded is and theory a Strauss Grounded following RQ1. after sample approach review Theory a analyzed manually more dataset. potentially our and summarizes longer I Table culture, reviews. reviewing informative older an has 7Vdo31186,185 1,198 10,119 3 2,330 3,322 667 3 2,704 5,384 3 497 905 3,439 Video 512 4,531 17 3 Preparation 9,585 16 3 Tax 1,030 15 2,224 14 3 Photography 10,161 13 3 Servers 2,396 & 12 Networking 3 11 Hobbies & Lifestyle 10 9 Reference #Sentences & Education 3 8 #Reviews 7 #Apps Illustration & Design 6 5 Office & Business 4 3 Finance & Accounting 2 1 Category Software No. 2 )Mna otn Analysis: Content Manual 3) the during codes the refined and related, categorized, We Approach: Theory Grounded 2) https://mast.informatik.uni-hamburg.de/app-review-analysis/ oa 23,1 135,395 32,414 52 Total Utilities Develop- Web ment & Programming Systems Operating Music Games Software Digital Children’s Security & Antivirus ytmtzstecdn akadrdcsdisagreements reduces and task coding the systematizes O EVE FTECLETDDATASET COLLECTED THE OF VERVIEW AL I TABLE 3 4 3 3 3 3 3 3 h hr hs consists phase third The ntescn hs,we phase, second the In 11,752 1,696 2,100 1,872 1,817 435 440 543 2 . The . 7,121 2,242 11,823 2,206 12,990 3,394 6,427 33,762 Coding IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version rt e o hi riigadevaluation. and training classifier, their supervised scikit for set library built truth learning then machine We the extraction employing feature classifiers. and concept preprocessing text for for tools the applied Parser of level.Stanford concepts sentence detect and can review we the whether on assessed rationale user we study the solvers. of disagreement the parts of three one into by split the recoded were peer-solving and half and remaining discussing The concepts, disagreements. rationale in the understand- coder of in their improved second ing involved solvers a were disagreement The solvers and step. this disagreement author All first iterations. the several disagreements by as all to peer-solved of (referred Half were phases. coders two four in by solvers) done disagreement resolu- was merged The disagreements disagreements. were the of results resolve tion to coding form The coding one [5]. agreement- into kappa coder Cohen’s the and calculating rate by reliability, their assessed task. this They for paper. development trained software this had were of English, and of author experience, command first high a the had student, and all Ph.D. professional, one science, IT were computer one were coder of coders The that students coders. master of six three remaining assignments the coding of two the to distributed time, unable on was coder finish one to Since equally the coders. and remaining random of peered all to each were contained distributed assignments For coding form sample. the coding coders, coding reviews seven the the The from with assignments guide. form coding coding coding a the received and coders analysis. seven content the fine-grained of more unit a information enable to basic coded the be as to sentence choose all. We at document. represented be not might ratings applications uniform certain of for a number categories or small adopt underrepresented from relatively not a reviews did only because with we strategy, [17], sampling Robillard suggested random As and rating. star Maalej each for by sentences to- and resulting reviews the of the shows tals to II Table proportion categories. in software reviews, and ratings 1020 of sample random stratified O 5 4 3 eue h aua agaeToktNLTK Toolkit Language Natural the used We Classification: Automated 4) and results coding the merged we peer-coding, the After Each reviews. the peer-coding of consisted step Excel third an The as form coding a created we step, second the In a generated we First, steps. main three included phase This oa ,2 ,6 7,288 1427 1638 6,268 1223 1434 1,020 204 #Sentences+titles 204 Total star 5 #Sentences star 4 star 3 star #Reviews 2 star 1 rating Star tp/sii-er.r/ cesdMrh2017 March accessed http://scikit-learn.org/, 2017 June accessed http://nlp.stanford.edu/software/lex-parser.shtml, 2017 June accessed http://www.nltk.org/, EVE FTECDN SAMPLE CODING THE OF VERVIEW 4 204 204 204 oaayeterves npriua,we particular, In reviews. the analyze to AL II TABLE 799 1169 1643 ( ntefut n nlphase final and fourth the In SDFRCNETANALYSIS CONTENT FOR USED 1003 1373 1847 5 eue our used We . 3 n the and ). amncma nw steFMaue(locle F1). called (also F-Measure their the and as recall, known mean precision, and harmonic metrics: collocations, common of tag accuracy using POS prediction the classifiers tags, evaluated We (POS) height. are: tree speech features syntax and text syntax of of part neutral, Examples of positive, text. bag a the of capture emotions metadata negative a sentiments of example while textual an of is feature, length examples text are and collo- rating trigrams, Star word and features. and bigrams words as of such Bag cations features. syntax and sentiment, otae nipoe eso fafaue isn or missing another a of feature, feature a a of mentions version user improved the an software, when code powerful The more assigned using”. a is been certainly I’ve is than Ultimate program X6 Paintshop, Pro Corel Paintshop about things and good heard I’ve example “But An is: mentioned. sentence is application software code reviewed the The of Webroot”. for Pro to Defender switching version using before “Was as: years (not such software 5 review, alternative the an in mentions version) code user The the user. when alternative a assigned concrete a by reports it described if option sentence, a to assigned issue. be worked concrete can a was lacks it sentence but this first, counter and as at satisfaction”, A using it my with to help”. to problems out much use had “I isn’t I’m is: that instructions example usability controls for any searches a time find doing reporting a “Can’t sentence is: at in issue user work example NOT one Another An does strictly solved. ONLY!!”. “It environment, be is: should server/multi-user concept an or a this can or for that issues), sentence quality, user example the driver software’s product by or with (e.g. described software, auxiliary issue licensing, issue products’s the support, an software in the documentation, feature, bug with a issue a an with denote issue can an problem a The reports software. it if sentence, the details). for from guide counter-examples coding and (see rational examples dataset the yet give describe not we and “others” following, concepts for the The options In criteria. concepts. open concept identified includes the also of software guide names chose classifying coding code for We for model attributes merged. FURPS quality taking or the example, for from removed codes, inspirations the either for names was unrelated to transparent found common, merged it and was discussed rationales it code was coding, code a to a open if If the instance, concept. For in related goal. times a study few meaning, the a their to occurs related prevalence, only the is their in it by defined how driven as and codes was the decision, guide of criteria, coding necessity alternative, The issue, justification. guide: and coding a final forming the finally in until coding), of (axial theory concepts into grouped eeprmne ihdfeetfaue:tx,metadata, text, features: different with experimented We eae oe hteegddrn pncdn were coding open during emerged that codes Related )Alternative: b) Issue: a) I.G III. ple hna lentv dto,vrin rrelease or version, edition, alternative an when applies srRationale User ROUNDED h oefrti ocp sasge oa to assigned is concept this for code The hscnetcnit ftrecdsthat codes three of consists concept This T h olwn ocpsaedefined are concepts following The . concrete ER OF HEORY su rpolmwt the with problem or issue U SER lentv software alternative R lentv feature alternative ATIONALE alternative is IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version eoegvn paduigaohrbcu rga” The program”. ... backup support, another tech using contacted and “I up shows: de- giving similar excerpt before a the through or As click uninstalling, cision. code to definite The returning, had update”. abandoning, I the X6 download software updated actually relinquish to I screens only when “My several that example renewal, in is upgrade, as decision, quibble update, renewal an software report similar or that sentences it code to The got friend”. applies but a of software, recommendation editing the on photography particular skeptical somewhat this software was “I about on is decision sentence example similar An or acquisition. install, download, purchase, planned. a and or conclusive, taken, single, considered, a is reporting that decision sentence actionable a to assigned be use”. to features you system for advanced of settings plenty customizable has and it if user, “Even power instance: a For are software. you the of supportability to related criterion similar or reusability portability), interoperability, accessibility, (compatibility, currency), language, (e.g. lo- calizability modularity), extensibility, adaptability, configurability, ity, (modifiabil- flexibility maintainability, adaptability, viceability, 7”. Windows than pulling faster screen much login the so and time is “The computer up is: the example on Am turning software. related between the criterion of similar performance a the or to scalability, battery, capacity, memory, time, etc.), (power, cache, recovery consumption resource time, time, response start-up throughput, are inappro- efficiency, transactions and speed, incorrect when totally code change are The all names priate”. that of “Payee names best to or and downloaded, free!” regularly the product is updates of “This automatically it is reliability code it the this server), reliable, to for is a sentence related candidate of criterion A similar (e.g. software. a availability or stability, safety, recoverability crashes), failure, after the of (e.g. severity and is frequency effects accuracy, special a X7”. of me to number take X3 “The comparing will “The and impressive or which to” are also used software interface-like, code getting web the this little more of with is sentences usability interface Example to the auxiliaries. related using its material, criterion learn training to similar documentation, is or help, it aesthetics, integrated easy attractiveness, software), (how layout, interface learnability user consistency, usage, assess- the concrete to code sentences The a such requirements. Typically, reports usability non-functional user. it of a assessments if by report sentence, reported criterion a ment to assigned be rmte”a hssnec ee ouceralternatives. unclear to away refer stay sentence to this decided and as items them” is: cheaper from features example on reviews counter alternative the A more read compared. “I are or software two reviewed the when of or feature, requested h code The Finally, code The )Decision: d) )Criteria: c) eest etne htadeshmnfcosrelated factors human address that sentences to refers supportability cur software acquire reliability performance hscnetcnit ffu oe htcan that codes four of consists concept This hscnetcnit ffu oe htcan that codes four of consists concept This eest etne htrpr software report that sentences to refers sasge osnecsta address that sentences to assigned is eest etne htadesser- address that sentences to refers ple osnecsta report that sentences to applies eest etne htaddress that sentences to refers paesoftware update otaesrqieet eoemkn ucaedecision”. purchase a making “Capturing before : requirements e.g. computer, software’s any result, of a capabilities as/with the so challenges or video that’, editing ... and ’so ’such ’so’, as that’, such conjunctions ... by introduced are They situation. ’since’ adverb. word or The preposition ’since’. as used like be meaning that also in marker can argumentation ambiguous typical not the is is again or ’Because’ updating 8.1”. ’as’, am with ’because’, “I conjunctions e.g: This with clause. ’since’, independent introduced the typically in statement is a for explanation an features. mentioned X6 explicitly product new a to upgrading of upgrade for because support to gives example reason sentence the This An enough plug-ins”. ’for than ways. that’, more other is Ultimate order many there and ’in “But to’, that’, to- is order ’so the ’in with with purpose’, or introduced clause commonly infinitive clause, independent the purpose, a clauses: clause. of result types and different three reason, state with We justification). sentences the statement), review (i.e. the clause (i.e. dependent clause one independent can and one instance, has we For that rules, sentences. sentence two compound linguistic a in common arguments between text find the to unstructured transition to expect write adhere a humans typically Since as and sentence. the a acts of in and A thoughts importance clauses. clause the dependent independent emphasizes into conjunction subordination clauses subordinate in- while independent sentences, short single transforms combines into Coordination clauses dependent subordination. and nation like (argument)”. It’s overnight (statement). calculus to software math this basic person purchase from IT don’t going an aren’t nerd, you a “If argument relation: or an (a) illustrates implicit example an in following subordinate involved The a or comma. in), a after, by (e.g. just prepositions typically until), because, separated (e.g. Explicitly conjunction [24]. are inference and arguments base knowledge related only a detected of basis be the syntactic can on or relations connectives implicit as whereas such constructions, lexemes certain by statement or statement A argument. a sentence. an behind another as reasons in acts the contained made, explains a statement is either of a to justification to part refers relate be it can can statement or justification the where a justification sentence is, a as compound That acts sentence. or justification another a for contains it if sentence, a GoBack”. for replacement a to for upgraded looking I’ve been “Since I’ve instance: 7, For Windows software. a changing on code ial,rsl lue niaetersl fa cino a or action an of result the indicate clauses result Finally, give They why-questions. to answers contain clauses Reason in action an for purpose the expresses clause purpose A coordi- using combined are clauses and phrases, Words, a to related explicitly be can argument or explanation An o hudass orcmue’ aaiiisadthis and capabilities computer’s your assess should you )Justification: e) wthsoftware switch nodrto order in sue o etne htrpr decisions report that sentences for used is e h efcl la n aeFle 3 Filter Face and Clear Perfectly the get h oefrti ocp sasge to assigned is concept this for code The because tde o work not does it IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version utfiain iue2sosasreso ftecdn form. coding named the columns indicating of two or review, screenshot first decision, a The the shows criteria, 2 alternative, of Figure issue, coders justification. sentence an The contains and sample. it whether title coding the each from assessed reviews all containing sr n3%o l eiw.TbeIIsosa vriwof overview an shows III Table reviews. all by of reported 39% are Justifications in concept. users rationale the user From one set. least agreed. truth review-level coders the derived three we set, of truth sentence-level two where assignments the while kappa 80%), (average: 94% - Cohen’s inter-coder 65% the level, from review ranged the - On agreement 0.36). 83% Cohen’s (average: 0.55 from the - ranged 0.17 while level 91%), sentence (average: inter-coder the 99% The review. on that peers in both agreement code if that percent code, on a once if least on review at level, agree the agreed to On sentence sentence. considered the that were for on coders code level, code that on a agreed on peers both agree Coders to agreement. considered inter-coder time. the were any assessed at and tasks results coding coding their resume Coders allowed. and were stop review a to of Multiple able sentence review. a reviews.were each or the title for a code repeated for codes is to application’s name starting read the application’s to before at The coder is Amazon a application on allowed each It description to group. a link appear of The assignment beginning order. coding consecutive the of a part in are that application an of denote columns comments. to The add columns code. or the the completed of reports any title to or and x sentence an a assign that other. could each coder adjoin concept A Columns rationale codes. user rationale same 16 user next the 16 The representing the sentence. of or one rating, represent title, as columns such row) the (i.e. item ahcdrrcie h oiggieadtecdn form coding the and guide coding the received coder Each vrl 8 fsnecsad8%o eiw oti at contain reviews of 86% and sentences of 48% Overall coding only contained sets truth conflict-free resulting The all merged we completed, were peer-codings the After reviews all i.e. application, the by grouped were reviews The Comment ausaesaitclysgicn with significant statistically are values kappa i.2 cenhtprl hwn h oigform. coding the showing partly screenshot A 2. Fig. V U IV. r sdt aktecdn fasnec as sentence a of coding the mark to used are agdfo .7-06 aeae .3.The 0.43). (average: 0.66 - 0.27 from ranged SER R ATIONALE Key and F REQUENCIES Value kappa eoeareview a denote < p agdfrom ranged 0 . 01 . Done utfiain 7%.Jsictosapa nls hn2 of 2% a concept. than other of less with any fraction in without reviews, the considerable appear reviews all 12%) a Justifications (or with (71%). of reviews together, justifications 13% 123 all In or appear justifications). concepts 131 compared 21% (in of alone fraction concepts often most other the appear to as concept, Criteria density. pervasive justification and the most alternatives, is criteria, are higher reviews the issues, the decision), of informative more co-occurrence the from higher that, ranging (i.e. seems justifications It of 71%. to in fraction 21 co-occur notable to a tend with standardized decisions independence. reviews the and of alternatives, the model of criteria, assumed of magnitude Issues, the shading and under residuals sign The Pearson the rationale. highlights with tiles reviews in collocations sample. our in frequencies concepts’ rationale the htrpr suswstelws oprdt te concepts other to compared more lowest appear the ( to was reviews for issues tend (2.2) report Issues rating that mean ratings. The reviews. star lower-rated in 1-5 often to respect with presence. concept’s a indicate Parenthesis side-bars concepts. rationale black user ratio, of justification co-occurrences the of hold Frequencies 3. Fig. frvesicuigatraie mut to to amounts compared verbosity 108), alternatives more mean (median: including The be rationale reviews criteria. to reported reporting of seem reviews the alternatives than to verbose respect reporting with Reviews count) concepts. word reviews. in rated 3-star sured in often more justifi- often appear and more alternatives, to criteria, occur seem while cations to reviews, seem rated decisions lower in Also, [22]. [15], studies < p

HasIssue 785 570 1981 794 1182 with Reviews with Sentences h oacpo nFgr ie noeve fconcept of overview an gives 3 Figure in plot mosaic The esuidtedsrbto fteue ainl concepts rationale user the of distribution the studied We eas tde h ifrneo eiw’vroiy(mea- verbosity reviews’ of difference the studied also We 16 (100%) T UHSET RUTH 1 15 (13%) e − (42%) 148 (21%) 06 131 : tdn’ -et.Ti si iewt earlier with line in is This t-test). Student’s , NTESNEC EE N H EIWLEVEL REVIEW THE AND LEVEL SENTENCE THE ON susAtraie rtraDcsosJustifications 484 Decisions Criteria Alternatives Issues 40 (38%) 16 (38%) HasAlternative (49%) HasDecision 390 84 (45%) AL III TABLE 40 ∼ 2 od mda:7)for 74) (median: words 128 (51%) 8 (38%) 82 726 (44%) 59 34 (24%) 387 (71%) 123 8 (50%) (54%) 59 (35%) 17 ∼ 7 words 171 396

HasCriteria < 2.22e−16 p−v residuals: P earson . alue = −4.5 −4.0 −2.0 0.0 2.0 4.0 5.6 IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version eecsbtenteemasaesaitclysgicn (with significant < statistically p are means these between ferences (mean: verbose more ∼ be to tend justifications reporting reviews (mean: criteria reporting (mean: verbose to more seem be decisions reporting Reviews criteria. reporting reviews nymtdt etrs u iileprmnssoe that showed experiments nitial Our containing features. those exclud- excluded metadata Besides also only classifiers. we the configurations, test invalid and ing train features to and used steps were that preprocessing compatible of combinations Accuracy Classifier A. +lemma). (abbreviated: lemmatization and (abbreviated:-punct), re- removal stopwords punctuation were -stops), steps (abbreviated: preprocessing moval The features IV. Table and employed in classifiers height, Penn The listed tree phrases. the syntax and clauses text on of length, text count based text measure – as to features such level studied sophistication phrase also we and [18], Tagset clause, Treebank word, the on configurations. possible all evaluating rather systematically classifiers on accurate an to than lead was with that goal configurations The [11], items. identify non-candidate validation to and candidate cross of ratio Vector 10-fold equal Regression conducted Support Logistic We and algorithms (NB), (LR). Bayes classification We Naive (SVC), classifiers. popular Classifier review-level the 5 used and classifiers classifier. review sentence-level for dataset a and classifier sentence for decisions. reporting those than longer be words to 5 tend (mean: alternatives justifications to report 3 that be those ∼ to than average tend on decisions shorter reported and concepts criteria, the alternatives, to respect with ( sentences of verbosity Puirm ngaso ato peh(O)tg ntecas and clause the on tags (POS) speech of part of Unigrams unigrams only) CP classification review (for sentiment Title T-sentiment for title, review the of words of N-grams n-grams Title rehih etnesna rehih rve lsie:ma value) mean classifier: (review height tree syntax Sentence height Tree count Subtree length Text n-grams POS Sentiment Length Rating Description n-grams Body Feature < p 6,mda:15 hnrvesrprigciei.Tedif- The criteria. reporting reviews than 105) median: 163, 4wrs ein 0wrs.As,snecsreporting sentences Also, words). 20 median: words, 24 eatmtclygnrtdasto lsie configurations: classifier of set a generated automatically We – tags (POS) speech of part with experimenting Besides 5 is that concept: each for classifier one dataset a evaluated set: We truth the represent that datasets two built We the in differences significant statistically also were There 0 . F 05 AUE SDT RDC AINL NUE REVIEWS USER IN RATIONALE PREDICT TO USED EATURES 0 . tdn’ t-test). Student’s , 05 tdn’ -et.Snecsta eotissues, report that Sentences t-test). Student’s , .A V. frrve lsicto only) classification review (for haelvl(P,bsdo h enTebn ops[19] corpus Treebank Penn the on based (CP), level phrase etnesna u-re on rve lsie:mean classifier: (review count value) sub-trees syntax Sentence level, word [19], the Corpus on Treebank tags Penn (POS) the speech on of based part of N-grams score sentiment lexical body Review rating Review for body, review the of words of N-grams UTOMATED ∼ 5,mda:9)cmae othose to compared 97) median: 150, AL IV TABLE ∼ 2,mda:7) Furthermore, 74). median: 128, C LASSIFICATION {− ∈ n n 5 n , { ∈ { ∈ 5 { ∈ } 1 1 , , 1 2 . 2 , , , 2 3 3 , } } 3 } hl h eodms infiatwslnt.Tefurther The length. was rating, significant was most Issue concept second the the for while features informative most features. the informative selected most impor- then We 10 all classifiers. over top these sum by the obtained averaged scores Random we tance and feature, each Boosting, were the For Gradient Forest. classifiers Tree, for The Extra importance Boost, classification. feature Adaptive level the sentence assess and to review dataset, review and best the Analysis Feature (0.82). achieved issues B. We for 81%. classifier the and F1-score with 99%, and F1-score 87%, recall, precision, respectively precision for of best values overall reaching review achieved recall, criteria the and for On overall classifier but justifications. the (0.98), classifying level, values for recall precision decisions de- highest lowest and classifying the alternatives for overall for achieved (0.83) classifier sentence F1-score The and cisions. (0.80) precision results for 0.6. all below in F1-score filtered rationale the or We user recall, respectively. mining precision, in having reviews performance and top sentences the to leading 20,433. - 9,895 between config- was classifier urations number different using the configurations evaluations classifier, cross-validated we review different of each classifier, 17,457 For cross-validation. sentence - using each 10,109 between For evaluated randomized. [15]. was al. et urations Maalej of experiments accuracy, classifier earlier high the a to achieve similar to sufficient not is metadata M Decision Criteria Alternative Configuration F1 Recall Issue Prec. concept R. Justification S CUAECASFEST IEUE AINL NSENTENCES IN RATIONALE USER MINE TO CLASSIFIERS ACCURATE OST )Ms nomtv etrsfrSnec Classifier: Sentence for features informative Most 1) sentence the using classifiers tree of ensemble an used We values highest the achieved we classifier, sentence the For configurations classifier of overview an give VI and V Table config- classifier of list the classifier, a running to Prior .60.93 0.76 0.80 0.84 0.71 0.76 0.88 0.77 0.80 0.86 0.70 0.74 .60.86 0.66 0.69 0.63 0.60 0.60 0.60 0.60 .008 R oy1gas sos aig sentiment rating, -stops, 1-grams, Body LR: 0.80 0.80 1-grams CP rating, -stops, 1-grams, Body SVC: 0.75 0.75 0.84 rating, -punct, -stops, lemma, 1-grams, Body NB: 0.71 0.70 .407 R oy1gas O -rm sentiment 1-gram, POS 1-grams, Body LR: 0.71 0.74 0.98 0.96 0.98 0.93 0.93 0.83 0.76 0.77 0.74 0.82 0.75 0.82 0.77 0.73 0.74 0.73 V:Bd &-rm,+em,rtn,subt. rating, +lemma, count 1&2-grams, count Body subt. SVC: length, 2-grams, Body rating, NB: 1-grams, POS height 1-grams, t. length, Body 1- CP SVC: sentiment, height t. length, +lemma, grams, rating, 1&2-grams, -punct, POS -stops, 2-grams, Body NB: -stops, count 1&2-gram, 1&2-grams, subt. POS 1&2-grams, height Body t. POS SVC: count, subt. 1-grams, sentiment, 1&2-grams, length, CP rating, 1&2-grams, -punct, -stops, Body +lemma, POS NB: 1&2-grams, -stops +lemma, Body SVC: sentiment - length, -stops, rating, 1&2grams, punct, POS 1&2-grams, -punct Body 1-2grams, LR: POS height 1-grams, tree Body syn. NB: count, subtree syn. sentiment, V:Bd -rm,sniet .height t. sentiment, 2-grams, Body SVC: height t. length, 1-grams, -punct, CP 1-grams, POS 1-grams, Body NB: AL V TABLE The IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version odadPSuirm a iia infiac,word significance, similar a had unigrams bigrams. POS POS and and count, Word subtree syntactic unigrams, POS and NNS-IN- determiner). trigram and POS preposition, the plural, and personal was (noun ’easy’, feature and DT ’are’, placed infinitive 10th were The words to ’not’. significant (i.e. adverb informative most (i.e. The TO-PRP 10 RB-JJ pronoun). and were top adjective) bigrams the scoring and word the best by than The better half achieved slightly unigrams. scored almost bigrams score for POS The overall account features. They the features. of significant most the was significance. Length unigram of ’versions’. rank word 10th and the ’version’ scoring and at word best placed best noun the singular the the were (i.e. while feature Among NNP-CD noun), number). was singular cardinal bigram was them (proper POS among NNP scoring scoring best unigram The overall the score. the significance of 10 half almost top for accounting features, informative using “after it in times found is of i.e. feature couple MD-RB-VB, This a verb. was and trigram adverb, POS random modal, ranked occasional best only The with loaded success”. “... transactions is bank feature my excerpt this get example with An sentence verb. a and adverb, of was i.e. third RB-VB, overall bigram ranked feature the POS remaining scoring the best to- than The informative were features. more length 2 times and three bigrams, Rating almost unigram. POS gether word 2 a unigrams, and trigrams POS POS 2 were features significant M Justification Decision Criteria Alternative Configuration F1 Recall Issue Prec. Concept R. h o 0faue o h ocp eiinwr word were Decision concept the for features 10 top The among were sentiment and length Criteria, concept the For most the were unigrams POS Alternative, concept the For S CUAECASFEST IEUE AINL NREVIEWS IN RATIONALE USER MINE TO CLASSIFIERS ACCURATE OST .10.87 0.71 0.61 0.82 0.71 0.60 0.80 0.92 0.73 0.60 0.87 0.74 0.62 0.81 0.86 0.78 0.60 0.83 .007 B O &-rm,T -rm,-uc,rating, -punct, 1-grams, T. 1&2-grams, POS NB: 0.70 0.60 0.96 senti- length, 2-grams, T. 1&2-grams, POS NB: 0.71 0.61 0.98 rating, -punct, +lemma, 1-gram, Body NB: 0.70 0.61 0.96 0.89 0.67 0.99 0.91 0.70 0.98 o’ e eproceed me let won’t 0.78 1- CP length, rating, -punct, 1-grams, Body NB: 0.73 0.81 title, 1&2-grams, POS 1&2grams, Body SVC: 0.74 0.82 0.74 0.78 0.72 0.74 0.81 0.72 0.74 V:Bd -rm,-tp,lnt,sentiment, height length, t. -stops, 1-grams, CP 1-grams, Body SVC: sentiment count subt. grams, rating, -punct, +lemma, 2-grams, sentiment Body SVC: height t. , ment height t. count, subt. -stops, +lemma, length, 1-grams rating, CP -punct, sentiment, +lemma, 1-grams, B SVC: height t. sentiment, length, V:Bd &gas lma sos senti- -stops, height +lemma, t. ment, 1&2grams, Body SVC: t- length, +lemma, height t. 1-grams, count, subt. sentiment, Body SVC: t-sentiment rating, -stops, 1-ngrams, Body NB: +lemma,- sentiment 1&2-grams, length, POS punct, 2-grams, Body NB: senti- height length, t. rating, t-sentiment, -stops, ment, 1-grams, Body NB: sentiment, rating, height -stops, t. count, 1-grams, subt. Body NB: height 1- t. CP count, length, subt. -punct, grams, -stops, 1-grams, Body NB: AL VI TABLE eetdyattempting repeatedly ni register”. I until to . ocniu sn arl ecatsrie” mn the VBD ’upgrade’. Among and ’bought’ services”. verbs the merchant were & words placed order payroll best in 2014) using (Pro “ continue version is newest to the feature tense). to this upgrade past to having the FORCED features sentence in was review verb 10 feature example and best top An pronoun single personal the overall (i.e. The among PRP-VBD Decision. were concept the trigrams for adjective). POS (i.e. and JJ is rating, unigram POS the for scoring use”, top to The easy use”. and to me, dependable “For very with PRP-VBD is givenfeature it is me, RB-JJ feature “For the with having was sentence bigram review conjunction). POS example coordinating and places An and best pronoun pronoun third personal personal The (i.e. tense). PRP-CC (i.e. past PRP-VBD the in was adjective).verb and best adverb second (i.e. The RB-JJ The ’user’. was feature and bigram the ’easy’, POS of ’not’, placed ’are’, significance best were the words of top half The than length. bigrams. POS less ranked had unigrams three each word top the They ranked as top significance informative four similar more a The had times score. sentiment three the approximately than was started It “I Criteria. with given is A feature the 10th. a with placed such playing with was present sentence in determiner) review verb and best (i.e. preposition, The VBG-IN-DT bigram participle, POS trigram number). ranked The cardinal best TO-NNP. the (i.e. was while ’version’, CD was than ranked word unigram informative top ranked more POS all times one than two more the almost scored and It unigrams, 10. word top the among saved”. it’s feature though even password online a my RB-VB enter “Syncing to was in me present bigram in is does POS ’not’ just that placed accounts was verb), informative and best word as adverb The informative (i.e. times title. most three review The top the than all bigrams. than more POS informative as and more words, times Rating three scoring sentiment. and almost length was also were alone features Among best Issue. concept 10 the top for the feature informative most feature. the informative was most most 7th The ranked The was unigram. infinite). feature to POS height and past noun tree informative verb (i.e. NN-TO (i.e. most as was bigram VBD informative third the The a was unigrams. approximately word tense) POS each the The as were informative best. bigrams with scoring and importance, ’because’ unigrams similar marker of argumentation were the unigrams Word bigrams. tification. POS the that as thing informative the as is is “This feature past is verb feature word make this and will the with pronoun sentence while personal significant A most (i.e. tense). tag The PRP-VBD word. POS was significant bigram significant most the past most was verb ’bought’ the (i.e. VBD was important. tense) more slightly being unigrams O irm,floe ywr ngas O unigrams, POS unigrams, word by followed bigrams, POS concept the for feature informative most the was Length informative most the was length Alternative, concept the For Classifier: Review for features informative Most 2) Jus- concept the for feature ranked best the was Length ereturn me rgnlSm n ogtSm 2”. Sims bought and Sims original o work not hsi 5mr as.Tesbrecount subtree The days”. more 55 in this orcl eas tkesasking keeps it because correctly tis it eydpnal n easy and dependable very ewere We Rating IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version rvd edako e novdit neitn discussion. existing an the easily into to survey involved and get quickly feedback or to user feedback users of provide number the growing challenges and At it emerging discussion. public time, spec- the same broad in perspectives the the and and soft- rationale polarization of for Google trum investigate resource store, to important software vendors an Amazon’s ware provide the uservoice.com as Play, such platforms on users for support Deliberation A. analysts. synthesis and and developers users for for management reviews support and of deliberation mining scenarios: its two of in potentials the and engineering reviews. the rational within for user searched Since sentences, or highlighted tool. single be a and can such concepts reviews our towards whole see step studied We first we a evalu- tools. as summarization and results and design classification analytics to rationale and ate performance inter-dependencies, to (e.g. researchers their concepts for criteria), useful rationale are fine-grained results more and study dataset collect We our information. to that relevant In think positive) used true information. be (i.e. relevant representative might any missing classifiers of rationale high-precision user positives, chance contrary, mining false the in than minimize used worse to be are might negatives classifiers false high-recall workload. when the reducing case in assist In can labeling classifiers the manual laborious, Since is limitations. and challenges have review (relinquish software, decision issue). a this (compatibility reports argument to sentence an TAXACT The and from software) it”. fast, returned import “Delivered I to is so unable sentence An a review was 71%. with justified to but reviews a 21 from of in ranging illustration co-occur ap- justifications justifications. of to that contain fraction tend notable found reviews concepts We studied rationale User engineering. of software 39% proximately to relevance its 10) to (up several seen takes as “It switch adverbs, sentence 5th. to or review seconds ranked adjectives example was the as phrase) in act prepositional The phrases noun). (i.e. Prepositional and PP (i.e. determiner, unigram to, IN-PRP POS CP (i.e. placed and TO-DT-NN best was noun) the while trigram and pronoun), bigrams personal preposition and POS ’bought’, preposition (i.e. ranked were best IN-NN words The were scoring ’since’. and top ’software’, the Other was ’you’, word. and 2nd scoring ranked The best unigram. was CP ’because’ one marker and trigram, features argumentation one top bigrams, remaining The POS words. two 5 were 10. top the top than the better among slightly ranking trigram determiner, POS unigram only participle, POS the present ranked noun) in best and the verb was (i.e. tense) VBG-DT-NN past and the in verb (i.e. ag opr fue omns atclryue reviews user particularly comments, user of corpora Large software for rationale user of importance the discuss We in rationale of labeling automated and manual the Both and rationale user of importance the stress findings Our was It Justification. concept the for best scored Length ewe conso screens or accounts between I D VI. ISCUSSION ”. hnavctn o oersucs nteln em we term, long the reused In resources. be more might for support advocating when technical when users the by assessing provided negatively arguments The be stakeholders. comments). also media among social can (e.g. inputs approach of different types Our other their usability). to understand applied better on and (e.g. insights perspectives get to can justifications help their and of also requirements spectrum broad of The documentation of decisions. trade- design existing justifications on enrich Furthermore, can deciding features). users in alternative stakeholders (e.g. offs support these quantifying might and considered. concepts clustering be or together duplicates should used of requirements are Identification features compatibility software which how and on Alterna- missing”.) insight are give users tives that apps or other features by software provided important issues most alter- does the software are “How mentioned “What most and (e.g. native?” the others to to fre- compare application compared is my performs it tool out their A finding when how in software. practitioners prioritized, the support abandoning might higher for analysis between-app reason be a as can mentioned quently issue an best users. ample, of take justifications to reported the stakeholders to allows tailored actions on This possible decisions software. conclusive mention a Furthermore, that switching selected filtered. those be e.g. might be explored, interest and might special of density reviews have rationale-backed justification that instance low for a Those reviews. informative low/highly requirements during communication. (e.g. and decisions documentation, prioritization), better make reviews, ing practitioners SE for reviews software of Synthesis B. automated reviews. the their of challenges sense This and review. making write, and their do they the processing in also when biased so They rules be vary. grammatical and can to strongly professionals, adhere can always not reviews not are their of Users processing the usefulness feedback. and their users are involving there of However, when stances). issues usability general contra some requirements and non-functional pro e.g. as con- (such on highlight perceptions that stances user user for or Voice, trasting contra User and criteria as pro such of a platforms visualization feedback include useful user by on be , debates also (e.g. for can text e.g. rationale the User should extend functionality). auto-complete to review support recommendations a and get justification also if might might identify This Users about. to and rating. users thought application complexity have help the its not might improve might understand rationale users and User tradeoffs software the [28]. the [4], about review useful- learn the a indicate reported of significantly of ness can justifications. overview density users’ justification an or The decisions, provide alternatives, might criteria, issues, reviews. statistics existing simple in discussion/debate A the structuring by ment ial,ue ainl lsie a mrv communication improve can classifier rationale user Finally, ex- for negotiation, and prioritization requirements During potentially filter and mark to used be can rationale User filter- in analysts and developers support can rationale User involve- user support to employed be might rationale User IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version is,ordtsticue eiw rmpplrapplications popular from moderate reviews a included have dataset Amazon. results our on our applications First, popular that for generalizability think of we degree avail- applications Amazon, 300,000 on than able more overall indicative. the as to compared primary be results to the designed see We not Store. such was App including markets as software study review, other to our representative a However, nor generalizable validity writing results. external the the of limit of might effort This reviews. the fake potentially make will users number large a cross-validation. employed with after we experiments results ratio classifiers. of the reliable the of more chose curves We obtain learning To set. the of testing items investigation and initial non-candidate be an training and also the candidate in of might on used ratio results performance the improve The by to influenced datasets. used and high-dimensional be information redundant can large much removing lose methods to These by They not [9]. space [8]. trying feature while Chi-squared features, as the irrelevant such reduce tests, to statistical try dif- functions the scoring of to on use lead application based methods have selection the Feature might as results. methods, ferent well selection as feature Machine), statistical for Vector parameter Support optimization (e.g, the of all algorithms evaluate classifier parameterization used not Different the did combinations. applicability We feature the approach. able possible our facilitates still of This were reproducibility results. and and accurate features achieve on learning focused to We claiming machine of results. from simple instead classification refrain rather our features of we informative completeness Thus, most the results. the complete as obtaining well agreed. as coders sets two tionale least truth at final which coding The on codings conducted reliability. the only we their in Fourth, contained increase coding. involved the to were doing peer-coding volunteer to that the prior coders reduce trained were to human coders The hired guide bias. with the refined iterations, non- we eight Third, and in code. candidate each included for we examples from candidate Second, downloaded be website. can project guide the The “other” codes. option new free identifying the codes including for all concepts, for rationale definitions user and the process, of the describes tasks, that coding guide main coding the detailed a created we First, that validity. theory plausible a for grounded. aimed of empirically we completeness is Instead, a codes. claim to not final due can the we and by biases Therefore affected indicated sampling agreement. partly potential been inter-coder is have moderate This the might coding. by study during assessment our human of the validity the ysis, orientation pro/contra their and topics. arguments reference user towards study to aim ept htw rwe nyasalfato fapplications of fraction small a only crawled we that Despite software of subset a only that assume reasonably can We ra- user identifying of feasibility the study to aimed We to threats these mitigate to measures several took We anal- content manual and approach grounded every for As I.T VII. RAST VALIDITY TO HREATS aaadrpr nF-cr faon 0 o recognizing of for corpus a 70% studied [29] around meta- al. of et Wyner and conclusions. F1-Score features and an premises linguistic report employ as and They well texts. data as legal keywords, in as conclusions n-grams, counts and what premises define detect data. to empirical [23]. to is regard [14], with field not arguments does this what and between in argumentation challenge relationships major the A as a well arguments, within whole as and structures conclusion, premises, the argumentative in e.g. document, identify arguments automatically many mining aims mining of to Argumentation as text. potential journalistic such the or political, with legal, [24] area applications research new interesting relatively a data. is in information. evidence on and of based approach type concepts grounded up any rationale button contain the a developed can from we that rationale Furthermore, reviews focusing mining perspective user different on a on had focused We [26] Reports. but Bug another al. Chrome In precision et mining. classifier text Rogers to study, improve compared features recall auxiliaries lower linguistic significantly of modal use as the that such suggest, documents. results existing parsing initial from and Their rationale mining identifying text studied for [25] techniques al. patent and et from Rogers solution, rationale documents. design issue, learning approach automatically an Their on on focuses modeling. rationale based design layer-based approach artifact an propose [13] automatically information to approaches also propose They developers. to for started advices actionable into soft- artifacts synthesize ware and filter, analyze, automatically to reviews.approaches the from our motivates concepts also rationale which mine maintain, to and goal capture to hard ratio- [6] is model-based current nale al. formal, the that et conclude and Dutoit They decision approaches state-of-the-art. cycle. for management life used of rationale software be use overviews the can and rationale throughout capture how the making and and describe support rationale [3] decision design on al. issues focus et a Burge the negotiation. with and rationale models, managing rationale creating, on of accessing activities and and the design maintaining, discuss describe [6] and al. rationale et Dutoit requirements and [2] integration. cost- Dutoit and and as Bruegge generation, such systems domain-knowledge rationale use, design technical effective for seven issues identifies business author and The human-computer- the perspective. [10]. from [6], interaction rationale [3], design decades discusses [12] for Lee engineering requirements and ware mining. argumentation and mining, rationale agement, of significance influence. the hazard for Second, excluding check results ratings. to the tests and statistical categories conducted software we Amazon all across aa ta.[3 rpsda praht automatically to approach an proposed [23] al. et Palau Finally, suggesting started vendors tool and researchers Recently, man- studied rationale have areas: Researchers three from work related discuss We ruetto mining argumentation rmeitn ouetto 2] in tal. et Liang [25]. documentation existing from II R VIII. ELATED ainl management rationale rmntrllnug text language natural from W ORK ierationale mine nsoft- in IEEE copyrighted paper - Accepted for publication at IEEE RE 2017 - Authors’ preprint version epwt h oig hswr sprl uddb h H2020 732463). the (ID by OPENREQ funded project partly is research work EU This coding. the with help Kurtanovi E. Hennings, users. other reviews or analysts, rationale-backed users’ developers, as filter software stakeholders, and certain structure for synthesize a) and b) and helps 80%-99% debates all approach between for Our respectively from recall 69%-98%. ranging sentences – and and encouraging precision were reviews concepts The rationale sentences. user five and rationale user reviews predict Re- to in Logistic configurations and different Machine, and Vector gression) Support Bayes, algorithms sentiments, Naive classifier data, three (i.e. evaluated meta We features. text, syntactic ma- using and supervised techniques, with learning predicted chine automatically be can concepts. concepts other to appear compared sentences justifications review that longer to in found 21 higher rather also a from We indicate density. ranging co-occurrence higher justification justifications that of observed issues, and fraction that 70% reviews notable found in We co-occur a to reviews. with tend the decisions and in criteria, found alternatives, alternatives are and in as criteria reviews such and and software concepts rationale how in pervasive rationale how describe quantify report qualitatively users We context assessment, which decisions. of - criteria their users alternatives, of and encounter, why justifications they reasons the issues around on the user developed by and encompassed were concepts rationale decisions the made, related were decisions those design rationale design on While engineering. focuses software in rationale user targets analysts. stakeholders work or for developers, Our impact users, potential contexts. as a argument with the engineering user as software in well reported diversity as arguments reviews on focus argumen- rather of a we validation have tation, and works conclusiveness improved of these emphasis an While strong for justifications. user features inspired of syntactical [21]) classification with (e.g. experiment and mining to 70 argumentation ours between from F1-Scores Works achieved vector 80%. support they Using classifier similarity features. text machine alignment semantic stance features, and different entailment features, employed as They such pairs. features comment-argument prop- of identifying on studied erties focussing [1] discussions, Najder online and in Boltui reasoning others. nego- and persuasion, deliberation, e.g. tiation, examine, and to camera, activities a dialogical purchasing about various forum Internet an in comments F Boltu F. [1] h uhr hn h oes atclryA lzdh J. Alizadeh, A. particularly coders, the thank authors The rationale user how discuss we experiments, of series a In of theory grounded novel a introduce we paper this In opttoa Linguistics. In Computational Mining discussions. Argumentation on online in ments zi ˇ n J. and c ´ A nje.Bc pyu tne eonzn argu- Recognizing stance: your up Back Snajder. ˇ X C IX. CKNOWLEDGMENT ,D atn,adM ie o their for Ziaei M. and Martens, D. c, ´ R atmr,Mrln,21.Ascainfor Association 2014. Maryland, Baltimore, , EFERENCES ONCLUSION rceig fteFrtWorkshop First the of Proceedings A ye,J cnie,K tisn n .J .Bench-Capon. M. J. T. and Atkinson, K. Schneider, J. Wyner, A. [29] Highly Ridder. De A. J. and Bronner, F. Neijens, C. P. Willemsen, M. L. [28] Text Using Burge. E. J. and Mathur, T. Gung, J. for Qiao, techniques Y. Exploring Rogers, B. Burge. J. [26] and Qiao, Y. Gung, J. Rogers, B. Argumentation[25] to Diagrams Argument From Stede. M. and Peldszus A. [24] Detection, The Mining: Argumentation Moens. M.-F. empirical and An Palau appstore: M. the R. in feedback [23] User Maalej. W. and Pagano D. words domain [22] and argument Extracting Litman. J. D. and Nguyen V. H. [21] Neuendorf. A. large K. a Building [20] Santorini. B. a and Building Marcinkiewicz, A. M. Santorini. Marcus, P. B. M. and [19] ref- Marcinkiewicz, api A. in M. knowledge Marcus, of P. Patterns M. [18] Robillard. data-driven P. Toward M. Ruhe. and G. Maalej and W. Johann, T. [17] Nayebi, M. Maalej, and W. art [16] the of Kurtanovi State Z. Maalej, mining: W. Argumentation whys: the [15] Torroni. P. Learning and Lee. Lippi B. M. W. [14] and Kwong, K. Issues. C. Liu, the Y. Understanding Liang, Systems: Y. Rationale [13] accuracy Design for bootstrap Lee. and cross-validation J. of study [12] A al. software et Kohavi for R. rationale Design [11] Shipman. F. and Loffler, P. Jarczyk, A. [10] A tas n .Corbin. J. and Strauss A. [27] I uo n .Eisef nitouto ovral n feature and variable to introduction An Elisseeff. A. and Guyon specification. Nikulin. I. case S. use [9] M. and Rationale-based Greenwood E. Paech. P. B. [8] and Dutoit Paech. B. H. and A. Mistrik, I. [7] McCall, R. Dutoit, H. A. scaled for [6] provision agreement scale Nominal kappa: Weighted usefulness Cohen. the J. influencing factors [5] read? to one Mistrk. Which I. and Charrada. McCall, B. R. E. Carroll, [4] M. J. Burge, E. J. Dutoit. [3] A. A. and Bruegge B. [2] rnir nAtfiilItliec n Applications and Intelligence In editors, Artificial Woltran, in reviews. S. Frontiers and product Szeider, online S. Verheij, of B. analysis argumentative Semi-automated of usefulness tion perceived reviews. and consumer characteristics online content the recommended! Theory Grounded Developing for Procedures and editors, Hanna, S. and Documentation. ’14 Gero Existing J. from S. Rationale In Extract to Techniques Mining on Conference International In 34th 2012 documents. (ICSE), existing from extraction rationale (IJCINI) Intelligence Natural Survey. and A matics Texts: ACM. in 2009. Law Mining USA, and NY, York, Intelligence New Artificial 98–107, In pages on ’09, Text. Conference ICAIL in International Arguments 12th of the Structure and Classification In study. In Mining texts. Argumentation in on Workshop components Second argument Paperback. identifying Published: for 2001. edition, 1st CA, Oaks, Thousand Inc, treebank. penn The 1993. June english: 19(2):313–330, of corpus annotated treebank. penn The english: of linguistics corpus annotated large 2013. 39(9):1264–1282, documentation. erence engineering. requirements reviews. app of classification trends. emerging perspective. algorithm miningan Design text Computer-Aided using rationale design Discovering Applications Their and Systems Intelligent Expert: In selection. model and estimation on Conference International In Hawaii Twenty-Fifth survey. a engineering: selection. 1996. Sons, & Wiley John 280. volume Engineering Requirements Engineering Software in ment credit. partial or disagreement (REW) Workshops In Conference re. Engineering for reviews online of Engineering Software Systems Changing and Complex Conquering ae 5–7.Srne,2015. Springer, 457–474. pages , 71:93,2011. 17(1):19–38, , RE 92:1–3,1993. 19(2):313–330, , h ora fMcieLann Research Learning Machine of Journal The ae 2–3.IE optrScey 2013. Society, Computer IEEE 125–134. pages , C rnatoso nentTechnology Internet on Transactions ACM h otn nlssGuidebook Analysis Content The pigr 2008. Springer, . 41)9690 2012. 44(10):916–930, , ,H ai,adC tnk nteautomatic the On Stanik. C. and Nabil, H. c, EETascin nSfwr Engineering Software on Transactions IEEE ´ ()31,2002. 7(1):3–19, , ora fCmue-eitdCommunica- Computer-Mediated of Journal EESoftware IEEE aiso ulttv eerh Techniques Research: Qualitative of Basics ytmSine,19.Poednso the of Proceedings 1992. Sciences, System 06IE 4hItrainlRequirements International 24th IEEE 2016 nentoa ora fCgiieInfor- Cognitive of Journal International pigr eln edleg 2006. Heidelberg, Berlin, Springer, . eurmnsEngineering Requirements scooia bulletin Psychological betOine otaeEngineering; Software Object-Oriented Ijcai einCmuigadCognition and Computing Design ud ocisurdtesting chi-squared to guide A oue1,1995. 14, volume , ()13,2013. 7(1):1–31, , 31:85,2016. 33(1):48–54, , ae 65,Sp 2016. Sept 46–52, pages , . rnieHl,1999. Hall, Prentice . COMMA ue2012. June , . otaeEngineering Software AE 1998. SAGE, . 23,My1997. May 12(3), , O rs,2012. Press, IOS . aePublications, Sage . rceig fthe of Proceedings ainl Manage- Rationale opt Linguist. Comput. 04:1,1968. 70(4):213, , ,2003. 3, , Rationale-Based oue25of 245 volume , rceig of Proceedings 2016. , Computational 2016. , IEEE , , , ,