Universiteit Antwerpen Research Methods in Computer Science Ansymo

Helicopter View

Research Methods in Computer Science (Ph.D.) (Serge Demeyer — University of Antwerp) Research AnSyMo

Antwerp Systems and software Modelling How to perform research ? How to write research ? http://ansymo.ua.ac.be/ (and get “empirical” results) (and get papers accepted)

How many of you have a systematic way of writing an abstract ? Universiteit Antwerpen

2. Reporting & Reviewing Research !2

Personal Opinion Disclaimer 2. Reporting & Reviewing Research

Introduction • The Publication Process + Publication Categories + Quality indicators Personal Opinion The Review Process • Identify the Champion • Implications for Authors + The 4-line abstract rule + The fish model + Natural emphasis of paragraphs • Things to avoid + Method vs. Methodology Sometimes I will give advice based on personal experience or The Task of the referee representing a particular school of thought. These do not • Questions to answer Review Template necessarily confirm with what your supervisor says ! ⇒ Once Accepted … Such opinions will be flagged with the Personal Opinion Disclaimer. • Tips and Tricks Conclusion

2. Reporting & Reviewing Research !3 2. Reporting & Reviewing Research !4 Publications: Output Measure What is (Ph.d.) Research ?

http://gizmodo.com/5613794/what-is-exactly-a-doctorate “If I have seen a little further it is “Are We Polishing a Round Ball?” by standing on the shoulders of Giants.” (Michael Stonebraker; Panel abstract — Proceedings of the Ninth International (Isaac newton) Conference on Data Engineering)

Elementary Human Sceptic perspective: School • the quest for the “least publishable unit” Knowledge High School Bachelor • “publish or perish”

"And since dissertations can be written about everything under the sun, the number of topics is infinite. Sheets of paper covered up with words pile up in archives sadder than cemeteries, because no one ever visits them, not even on All Souls' Day. Culture is perishing in overproduction, in an avalanche of words, in the madness of quantity. That's why one banned book in your former country means infinitely more than the Ph.D. Ph.D. billions of words spewed out by our universities." Master (early stages) (finished)

(Milan Kundera, The Unbearable Lightness of Being; Part Three: Words Misunderstood — Sabina's Country)

2. Reporting & Reviewing Research !5 1. Research Methods !6

Publication Categories Publication Categories — Computer Science

source: guidelines for project reports Journal Publications FWO (Research Fund Flanders) • a1) citation index (ISI web of science) Journal Publications • a2) international; peer reviewed • citation index (ISI web of science) • a3) national; peer reviewed • international; peer reviewed • a4) other

Books Books Conference Publications • b1) book • book • peer reviewed • b2) chapter • editor (incl. proceedings) (acceptance ratio) • a3) editor (incl. proceedings) • chapter

Other Comparing apples and oranges Other International vs. National • c1) articles in proceedings Artifacts • workshops • inherently regional research (law, politics, …) • c2) technical reports; • tools • technical reports; extended • vulgarizing research extended abstracts; thesis • patents abstracts; thesis • scientists taking position in society debates • c3) patents

Publication Culture Artifacts ??? • co-authorship (e.g. alphabetical sorting) •Computer Science and Telecommunications Board, C. • citation behavior 1994. Academic careers for experimental computer • half-life time of ideas scientists and engineers. Communications of the ACM 37, 4 (Apr. 1994), 87-90.

2. Reporting & Reviewing Research !7 2. Reporting & Reviewing Research !8 The Pipeline Model Personal Opinion Quality Indicators

Proceedings: Acceptance Ratio Journal Publications: Impact factor • Andy Zaidman, Bart Van Rompaey, Serge • Bart Van Rompaey, Bart Du Bois, Serge Demeyer, and Arie van Deursen. Mining Demeyer, and Matthias Rieger. On the software reposito- ries to study co- detection of test smells: A metrics-based evolution of production and test code. In approach for general fixture and eager Proceedings ICST’08 (The 1st International test. Transactions on Software new, unpolished idea good, valuable idea archival reference Conference on Software Testing, Engineering, 33(12):800–817, 2007. Verification and Validation), pages 220– [SCI impact factor 1.967, ranked 7 / 79] 229. IEEE, 2008. [Acceptance ratio: 37/147 = 25%]

workshops conferences journals • Andy Zaidman, Bram Adams, Kris De Schutter, Serge Demeyer, Ghislain Hoffman, and Bernard De Ruyck. Regaining lost knowledge through dynamic analysis and aspect orientation - an industrial experience report. In Proceedings CSMR’06 (the 10th Conference on Software Maintenance and Reengi- neering), pages 89–98. IEEE Computer Society, 2006. [Acceptance ratio: 27+4/65 = 42%] Typical for computer science. Not in other scientific disciplines.

Short Papers

2. Reporting & Reviewing Research !9 2. Reporting & Reviewing Research !10

Acceptance Rates Ranking of Conferences Source http://people.engr.ncsu.edu/txie/seconferences.htm

B http://portal.core.edu.au/conf-ranks/ C

Unranked Personal Opinion

• [100% - 50%[: not selective • [30% - 15%[: selective http://valutazione.unibas.it/gii-grin-scie-rating/ • [50% - 30%[: reasonably selective • [15% - 0%[: too selective !?

2. Reporting & Reviewing Research !11 Impact Factor — Citation Index The h-index

Represent both • scientific productivity • scientific impact ⇒ in a single number (measurement)

A scientist has index h if • h of [his/her] Np papers have at least h citations each, and • the other (Np − h) papers have at most h citations each.

Sources to calculate • ISI web of knowledge http://isiknowledge.com/ • UAD - Search http://quadsearch.csd.auth.gr/

2. Reporting & Reviewing Research !14

Quality Indicators — Beware The Reviewer

• impact factor of journal ≠ impact factor of article • volunteer + Seglen PO (1997). "Why the impact factor of journals should not be + don’t waste his/her time used for evaluating research". BMJ 314 (7079): 498–502. + Joint Committee on Quantitative Assessment of Research (June 12, • curious 2008). "Citation Statistics". International Mathematical Union. + catch his/her interest

• #citations ≠ impact • constructive + Carlo Ghezzi; Reflections on 40+ years of software engineering + supervises other Ph.D. research and beyond an insider's view (ICSE 2009, keynote) • influential • “The widespread practice of counting publications without reading and + wants to support “valuable” papers judging them is fundamentally flawed.” + Parnas, D. L. 2007. Stop the numbers game. Commun. ACM 50, 11 • anonymous (Nov. 2007) + avoid tampering

• “If used unwisely, as is increasingly the case, they discourage people … unfortunately … (young ones in particular) right from the outset from daring to think, from exploring new paths […]” • busy + Math. Struct. in Comp. Science Editorial Board; Math. Struct. in + read’s on train, bus, air-plane, … Comp. Science (2009), vol. 19, pp. 1–4.

2. Reporting & Reviewing Research !15 2. Reporting & Reviewing Research !16 Review Process Steps Bidding for Abstracts Providing Keywords abstracts + key-words

= “first date” with your reviewer ■Automated reasoning techniques

■Component-based systems

■Computer-supported cooperative work

■Conﬁguration management

■Domain modelling and meta-modelling

■Empirical software engineering

■Human-computer interaction

■Knowledge acquisition and management

■Maintenance and evolution As many as possible ? ■Model-based software development ■Model-driven engineering and model transformation

vs. As few as possible ? ■Modeling language semantics

■Open systems development

■Product line architectures

■Program understanding

■Program synthesis

■Program transformation

■Re-engineering

■Requirements engineering

■Speciﬁcation languages Identify the Champion ■Software architecture and design your reviewer needs arguments ■Software visualization source: CyberChair (http://www.CyberChair.org) to support your paper ■Testing, veriﬁcation, and validation ■Tutoring, help, and documentation systems

2. Reporting & Reviewing Research !17 2. Reporting & Reviewing Research !18

Writing Abstracts 4-line abstract guideline

• source: Kent Beck “How to Get a Paper Accepted at OOPSLA” [ http://lore.ua.ac.be/Teaching/ThesisMaster/BeckAbstract.html ]

• 1) states the problem + WHO is suffering the problem ? + Connect with your target audience Descriptive Abstract Informative Abstract • 2) why the problem is a problem • outlines the topics covered in a • provides detail about the + WHY is it a problem ? piece of writing substance of a piece of writing + Cost / Art rather than a science / … + reader can decide whether + readers remember key • 3) startling sentence to read entire document findings + WHAT is the claimed solution ? + reviewers find the claims + the one thing to say that will catch interest • ≈ table of contents in • ≈ claim and supporting … and that you will actually demonstrate in the paper paragraph form. evidence in paragraph form ➡ must be falsifiable • 4) the implication of my startling sentence + WHERE can we use this solution ? + implications for society, community, other researchers, … ≠ executive summary (abstracts use the same level of technical language)

2. Reporting & Reviewing Research !19 2. Reporting & Reviewing Research !20 Identify The Champion (1/2) Identify The Champion (2/2)

• source: Oscar Nierstrasz, “Identify the Champion,” in Pattern Languages • Identify the Conflicts (classify according to extreme reviews) of Program Design 4 + AA, AB: All reviews are positive, at least one champion. + AC: Likely accept; at least one champion, and no strong detractor. • Make Champions Explicit + AD: This is a serious conflict, and will certainly lead to debate. + A: Good paper. I will champion it at the PC meeting. + BC: Borderline papers, no strong advocate nor a detractor. + B: OK paper, but I will not champion it. + BD: Likely to be rejected. + C: Weak paper, though I will not fight strongly against it. + CC, CD, DD: Almost certain rejects. + D: Serious problems. I will argue to reject this paper. • inexpert champion ➡ “The most important thing for a reviewer to decide is whether + If all champions are Y (or Z) he or she thinks that the paper is worth defending at the PC + If all reviews are Y or Z meeting, not whether it is a great paper or not.” ➡ solicit extra review

• Make Experts Explicit • expert fence-sitters + X: I am an expert in the subject area of this paper. + Experts tend to be more critical + Y: I am knowledgeable in the area, though not an expert. ➡ B or even C ratings by X may turn out to be champions + Z: My evaluation is that of an informed outsider. (remember: PC members want to influence the research) ➡ detect inexpert champion — expert fence-sitter

These scores are *not* revealed to the authors

2. Reporting & Reviewing Research !21 2. Reporting & Reviewing Research !22

Example: Easychair Make it Easy for your Champion

• Select appropriate keywords + Why are you in the scope of the conference/journal/… ?

• Test the abstract + Start early with the abstract + Ask for early (external) feedback • Clear accept at top • Clear reject at the bottom • Visible claims (not shown) + Abstract + intro + conclusion have have visible claim(s) • middle area: to discuss + Ask early feedback to summarize what reviewers think the claim is

• Clear validation + Champion is then able to defend it against detractors

• Write to the Program Committee + Target a PC member + Have a clear picture of your champion

2. Reporting & Reviewing Research !23 2. Reporting & Reviewing Research !24 Personal Opinion The Fish Model Role of “Related Work” Personal Opinion (5) The conclusion • quick summary of solution (1) The problem + empirical evidence • who has the problem ? • implications of the solution • why is it a problem ? • future work for the community Related • what is the (sketch of) the solution ? (long term) Work

(3) The solution • detailed description Problem Statement Problem Context (sufficient for replication) (beginning of paper) (end of paper) • empirical evidence

Other researchers define Other researchers do the research agenda complimentary work (2) The problem context (4) The problem context revisited • why is it a difficult problem ? [a.k.a. “Related Work”] ⇒ high entry barrier ⇒ crisp problem statement (few bibliographical details) • I only solved one aspect of problem • which aspect of the problem do you tackle ? (for experts only) (difficult to write) • others have worked on it • how can you show that you solved (many bibliographical details) the problem ? • future work (short term) (criteria / units of analysis / …) ⇒ together we made progress

2. Reporting & Reviewing Research !25 2. Reporting & Reviewing Research !26

Target Audience Personal Opinion Books on writing

• The Elements of Style • Style: Toward Clarity and William Strunk Jr., E. B. White Grace Target Joseph M. Williams, Gregory G. Audience Colomb

Experts in sub-domain Broader Audience (in-crowd) (informed outsider)

= arguing the problem and = preaching to the quire inviting others to contribute • 18 simple guidelines + elementary rules of usage • guidelines • Conferences: ICSE, ESEC/FSE + elementary rules of composition + refactoring rules • Journals: TSE, TOSEM • magazines: IEEE Software, IEEE • You have to know the rules before you can Computer, Communications of the ACM break them • Give a man a fish and you feed him for a day. Teach a man to fish and you feed him

2. Reporting & Reviewing Research !27 2. Reporting & Reviewing Research !28 Skimming texts — Emphasis How to structure your writing

The last thing one discovers in writing a book is what to put first [Blaise Pascal]

all of us … must understand three things about complex writing: • it may precisely reflect complex ideas “natural” emphasis of paragraphs • it may gratuitously complicate complex ideas • 1rst 1/2 of last sentence (most) • it may gratuitously complicate simple ideas • 2nd 1/2 of first sentence © Joseph M. Williams, “Style: Toward Clarity and Grace”

FIXED Issue Discussion

VARIABLE Point —

On section/chapter level FIXED Topic Stress • say what you gonna say • say it VARIABLE Old/Familiar New/Unfamiliar • say what you have said FIXED Subject Verb Complement

Source: Joseph M. Williams, “Style: Toward Clarity VARIABLE Characters Action — and Grace” The University of Chicago Press 1990

2. Reporting & Reviewing Research !29 2. Reporting & Reviewing Research !30

Things to Avoid Things to Avoid: Methodology Personal Opinion

• report order ≠ investigate order • “In this paper we propose a methodology for XXX” + arguments should appear in order that bests support the claim + My favorite reviewing sentence: - Do not use the word "Methodology" for something • unsubstantiated claims, hopes, assumptions simple like a technique, algorithm or even method; + XXX will make it easy/fast/better/integrate with other tools ... this is inflation of words - do you actually demonstrate these claims in your paper ? + We believe ..., We hope ... • the postfix -OLOGY - My favorite reviewing sentence: - biology = the study of the living organisms Personal Opinion “We are doing science, not religion …” - psychology = is the study of the human mind + XXX is valuable ...‚ XXX can help ...‚ - cosmology = is the study of the cosmos XXX is an excellent ... ➡ methodology = the study of the methods. - My favorite reviewing sentence: METHOD "Are these opinions? Hypotheses? Proven facts? Please add references." • method = a series of steps or acts taken to achieve a goal • tackling a non-problem, a problem which you cannot solve + substeps of method + A software engineering example remain a method METHODOLOGY STEP - papers citing “Software Crisis” + cfr. Composite design pattern

2. Reporting & Reviewing Research !31 2. Reporting & Reviewing Research !32 The Task of a referee (1/2) The Task of a referee (2/2)

• source: Alan Jay Smith, "The Task of the Referee," Computer, vol. 23, Categories no. 4, pp. 65-71, Apr. 1990 • (1) Major results; very significant (fewer than 1% of all papers). • (2) Good, solid, interesting work; Decide a definite contribution (≤ 10 %) • Makes sufficient contribution ? • (3) Minor, but positive, contribution to knowledge + depends on the standards of the journal/conference/workshop/… (perhaps 10-30 %).

Questions to answer • What is the purpose of this paper ? • Is the paper appropriate? (for computer science / software engineering / reengineering / …) • Is the goal significant ? • (4) Elegant and technically correct but useless. • Is the method of approach valid ? This category includes sophisticated • Is the actual execution of research correct ? analyses of flying pigs. • Are the correct conclusions drawn from the results ? • (5) Neither elegant nor useful, but not actually wrong. • Is the presentation satisfactory ? • What did you learn ? • (6) Wrong and misleading. • (7) So badly written that technical evaluation is impossible.

2. Reporting & Reviewing Research !33 2. Reporting & Reviewing Research !34

Reviewing Template Time estimation Personal Opinion

1 paper = ± 4 hours • 1,5 hour reading + annotating + read on paper ➡ submission for review incl. page numbers & white-space • 1 hour writing review Review • 1 hour discussion + adapting reviews • Strong accept / weak accept / weak reject / strong reject + over mailing lists etc. ➡ Including a solid motivation for your recommendation • 0,5 hour overhead • Template + print papers (write numbers on them !!!) + summary (neutral) ➡ “first contact” with the papers + strong points (bullet points) + managing conference reviewing system + points to improve (bullet points) + distribute among co-reviewers + details + … Important for champion/detractor + PC-only comments

Ph.d. students as Co-reviewer • 2nd opinion (reduces time spent for “reading” and “writing review”) • Ph.d. students experience “the other side of the fence” • Mentioned in the proceedings (CV)

2. Reporting & Reviewing Research !35 2. Reporting & Reviewing Research !36 Once Accepted … Conclusion

… at the Conference Introduction • The Publication Process • prepare an elevator-pitch + Publication Categories + based around “startling sentence” from your abstract + Quality indicators The Review Process • approach gurus • Identify the Champion + they like it, it's good for their ego • Implications for Authors • "explain your Ph.d. topic to at least 3 persons each day" + The 4-line abstract rule + = advice from ICSM 2009 Ph.d. symposium + The fish model + Natural emphasis of paragraphs • submit to Ph.d. symposium • Things to avoid + receive valuable feedback + Method vs. Methodology + network with future peers The Task of the referee • Questions to answer ⇒ Review Template • participate in workshops + test how the community reacts to research questions Once Accepted … + the gurus struggle too ! • Tips and Tricks Conclusion

2. Reporting & Reviewing Research !37 2. Reporting & Reviewing Research !38