INFOMSCIP 2018-2019 lecture 2 Sep 13, 2018
Disclaimer: These slides may contain copyrighted material that is used for pure educational purposes based on the Fair Use policy. http://en.wikipedia.org/wiki/Fair_use Scientific perspectives on GMT (INFOMSCIP) Introduction (cont.)
§ General aspects Why publish & search for related literature? § The academic world Organizations & people, quality measures (for people, publications, events) § Publishing Where & how (review process) § Literature research Where & how § Ethics, risks, and integrity
2 Mon, Sep 10 Introduction INFOMSCIP Thu, Sep 13 Introduction
Mon, Sep 17 Basics Thu, Sep 20 Basics • Handout for next week Fri, Sep 21 Deadline assignment 1 st • 1 assignment Mon, Sep 24 Experimental research st • Groups for 1 assignment Thu, Sep 27 Experimental research Fri, Sep 28 Deadline assignment 2
• Attendance list: check for Mon, Oct 01 Experimental research typos in your names & Thu, Oct 04 Experimental research apologize and correct Fri, Oct 05 Deadline assignment 3
mistakes, please Mon, Oct 08 User studies Thu, Oct 11 User studies Fri, Oct 12 Deadline assignment 4
Mon, Oct 15 Fundamental research Thu, Oct 18 Fundamental research Fri, Oct 19 Deadline assignment 5
Mon, Oct 22 Guest lecture Thu, Oct 25 TBD Fri, Oct 26 Deadline assignment 6
Mon, Oct 29 Engineering, design, systems Thu, Nov 01 Engineering, design, systems Fri, Nov 02 Deadline assignment 7
Fri, Nov 09 Deadline assignment 8 In GMT, mostly: Research question & hypothesis (can be informal) Searching for related research, week 2 Specifying a measurable Observation, framework, week 3 problem, idea
Publishing research • Experimental research, week 4 & 5 results, week 2 • User studies, week 6 • Fundamental research, week 7 • Other (design, engin., systems), week 8
Important characteristics of scientific research Þ Verifiable Þ Replicable Þ E.g., observations have to be measurable Þ Objective (e.g., via operational definitions) Þ Explainable
4 A linearized, pragmatic scheme of the four points above is sometimes offered as a guideline for proceeding: 1. Define a question 2. Gather information and resources (observe) 3. Form an explanatory hypothesis 4. Test the hypothesis by performing an experiment and collecting data in a reproducible manner 5. Analyze the data 6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis 7. Publish results 8. Retest (frequently done by other scientists) The iterative cycle inherent in this step-by-step method goes from point 3 to 6 back to 3 again.
From Wikipedia, “Scientific method”
5 Gathering information and resources (observe) Þ Normally done by literature review
Why literature review? • To know and understand the existing knowledge (state of the art) • To identify lacks and shortcoming in the state of the art Þ Motivation for your research
• To base your research on, e.g.: Þ Established evaluation methods Þ Results that you can take for granted Þ …
Verifying the references of a paper is an important part of the review process. Lack of essential citations can lead to rejection.
Important: When writing the related work section of a paper or your thesis, do not just address the work, but also the relation to your research! Hint: Check the related work section of good papers. They usually do not just discuss the related work, but how, e.g., their own differs, builds on it, etc.
6 Some good examples from thesis projects (that have also been published): Nina Rosa (thesis published as paper at ICMI 2015) According to the model by Steuer [30], presence in VR is a human experience and a consequence of immersive technologies. It has two determining dimensions: vividness with the two contributing factors breadth and depth of included modalities; and interactivity with the three contributing factors speed, range, and mapping. Many studies target increasing interactivity by focusing on improving task performance [2, 24, 5]. Yet, when the goal is to improve experiences that users can take part in passively, vividness becomes a vital part of the system, even more urgent than interactivity. Here, we focus on this less studied aspect by specifying the sub-goal of our research as investigating the experience of passive touch under varying visual and auditory conditions. … It has been shown that a first person perspective of a life-sized virtual human body alone is sufficient to generate a body transfer illusion [27]. In our experiments we thus apply this methodology focusing on first person body experiences. … Wendy Bolier (thesis published as paper at ACM MM 2018)
Since the emergence of head-mounted displays (HMDs) on the consumer market, many VR painting tools have been released. Some well-known examples are Tilt Brush [3], Quill [2] and A-Painter [1]. For this study, the choice was made to use A-Painter, as it is open source and thus allows us to make the changes necessary for our research. … Although many studies suggest that the drawing of 3D objects is beneficial for the improvement of spatial visualization skills, more research is needed to prove the exact influence of S&D by itself. Furthermore, the existing studies all apply to drawing 3D objects onto a 2D surface using graphical projection. No research exists yet evaluating the benefits of drawing 3D objects in a 3D space. … Finally it needs to be emphasized that the VR applications in these studies are completely focused on the training of spatial skills. Concerning spatial visualization and mental rotation, no proof exists that an arbitrary VR application will have a positive influence on these abilities. … Extend existing knowledge with new findings Þ Normally done by scientific publications
Why publishing research results? • To make your newly created knowledge know to everyone Þ It is only useful for society, if it is made available
• To get promoted Þ Notice that this should not be the purpose of research, but a means to spread your results
Like research, quality of research needs to be quantified and measurable. Publications have become the main standard to evaluate one’s research quality.
Phrases every PhD student will hear from his/her supervisor at some time: • “Publish or perish” • “Demo or die” (The latter mostly in applied domains)
8 Who does (and publishes) scientific research? • Universities • Research labs • E.g. TNO, CWI, .. • Companies • E.g., Xerox PARC, Microsoft Research, …
9 Xerox PARC has been the inventor and Who does (and publishes) scientific research? incubator of many elements of modern • Universities computing in the contemporary office work • Research labs place: • Laser printers • E.g. TNO, CWI, .. • Computer-generated bitmap graphics • Companies • The graphical user interface, featuring • E.g., Xerox PARC, Microsoft Research, … windows and icons, operated with a mouse • The WYSIWYG text editor • Interpress, a resolution-independent PARC (Palo Alto Research Center; graphical page-description language and formerly Xerox PARC) is a research and the precursor to PostScript development company in Palo Alto, • Ethernet as a local-area computer network California, with a distinguished reputation • Fully formed object-oriented programming (with class-based inheritance, for its contributions to information the most popular OOP model to this day) in technology and hardware systems. the Smalltalk programming language and integrated development environment • Prototype-based programming (the second most popular inheritance model in OOP) in the Self programming language • Model-view-controller software architecture • AspectJ an aspect-oriented programming (AOP) extension for the Java programming language
From Wikipedia (“PARC (company)”)10 Who does (and publishes) scientific research? • Universities • Research labs • E.g. TNO, CWI, .. Microsoft Research is the research subsidiary • Companies of Microsoft. It was formed • E.g., Xerox PARC, Microsoft Research, … in 1991, with the intent to advance state-of-the-art computing and solve difficult world problems through technological innovation in collaboration with academic, government, and industry researchers. The Microsoft Research team employs more than 1,000 computer scientists, physicists, engineers, and mathematicians, including Turing Award winners, Fields Medal winners, MacArthur fellows, and Dijkstra price winners.
11 An analysis of top research laboratories in Human- Computer Interaction (HCI) over the last decades.
Source: https://www.nngroup.com/articles/to p-research-laboratories-in-human- computer-interaction-hci/
12 An analysis of top research laboratories in Human-Computer Interaction (HCI) over the last decades (by Jakob Nielsen, 2002, updated 2013).
The Dawn of Time: 1945-1979 § Gold: Stanford Research Institute (SRI) § Silver: Xerox PARC § Bronze: Bell Laboratories The 1980s § Gold: Xerox PARC § Silver: IBM T.J. Watson Research Center, Yorktown Heights § Bronze: MIT Media Lab The 1990s § Gold: Bell Communications Research (Bellcore) § Silver: Apple Computer Advanced Technology Group § Bronze: Xerox PARC A First Look: 2000-2010 It's early yet to truly evaluate research labs' contribution to this decade, so check back in 2010 for the final score. Currently, my assessment of the best HCI research labs is: § Gold: Microsoft Research § Silver: Xerox PARC § Bronze: Carnegie Mellon University (Update 2013: I think my assessment in 2002 proved fairly predictive for the decade, because now with the benefit of hindsight I would still give out the same "medals.")
Source: https://www.nngroup.com/articles/top-research-laboratories-in-human-computer-interaction-hci/
13 Unpublished research from industry
Examples from Apple’s announcement of new iPhones on September 12, 2018
See https://www.apple.com/lae/ apple-events/september-2018/
Fall detection
Material testing Question: is this really research? 14 Who does (and publishes) scientific research? • Universities • Research labs • E.g. TNO, CWI, .. • Companies • E.g., Xerox PARC, Microsoft Research, …
Nederlandse Organisatie voor Toegepast Natuurwetenschappelijk Onderzoek (TNO; English: Netherlands Organisation for Applied Scientific Research) is an independent research organisation in the Netherlands that focuses on applied science. The organisation also conducts contract research, offers specialist consulting services, and grants licences for patents and specialist software. TNO tests and certifies products and services, and issues an independent evaluation of quality. Moreover, TNO sets up new companies to market innovations.
15 Who does (and publishes) scientific research? • Universities • Research labs • E.g. TNO, CWI, .. • Companies • E.g., Xerox PARC, Microsoft Research, …
The Centrum Wiskunde & Informatica (abbr. CWI; English: "National Research Institute for Mathematics and Computer Science") is a research center in the field of mathematics and theoretical computer science. It is part of the Netherlands Organization of Scientific Research (NWO) and is located at the Amsterdam Science Park. This institute is famous as the creation ground of the Python Programming Language. It was a founding member of the European Research Consortium for Informatics and Mathematics (ERCIM). 16 • BSc students / Bachelor thesis (but not necessary undergraduate students (publishable) research)
• MSc students / GMT program: ”thesis results are publishable” graduate students is a criteria for a cum laude degree (min. 8.5)
• PhD students (NL: AIOs, assistent in opleiding)
• Post Docs
• Assistant professors (NL: UDs, universitair docent)
• Associate professors (NL: UHDs, universitair hoofddocent)
• Full professors (NL: Hoogleraar)
17 From http://www.cs.uu.nl/education/vak.php?stijl=2&vak=INFOMGMT2 Some comments on your GMT MSc thesis The first part comprises 1 period of full-time work. The student will complete (at least) the following deliverables: • Paper & annotated appendix • The MSc research application form (asap) versus full report • A completed literature research for the project • A clear (list of) research question(s) • Digital library & examples • A skeleton of the thesis • A time plan for the second part of the thesis • GMT project site • A description of the research methodology • A plan for the evaluation of the results/outcome
The second part comprises 25 EC (2 periods). You will complete (at least) the following items: • Perform and complete scientific research according to the predefined plan; • Write a scientific report about this research. You may choose between a regular thesis (plus optional appendix), or a scientific conference/journal paper, plus mandatory appendix; • Give a presentation about the work; • Produce a dissemination (see below).
18 Some comments on your You find UU theses (since 2012) here: http://studenttheses.library.uu.nl/search.php?language=en GMT MSc thesis The ones from Computer Science are here: http://studenttheses.library.uu.nl/search.php?m=course&co • Paper & annotated appendix urse=Computing%20Science&language=en versus full report • Digital library & examples Example for a good thesis report: Marries van de Hoef, “Real-Time Dynamic Radiosity for High • GMT project site Quality Global Illumination”, 2014 https://dspace.library.uu.nl/handle/1874/288879 Marries’ thesis won the Science Faculty’s thesis prize in 2014.
Examples for good papers with appendix: Nina Rosa, “Immersive Multimodal Virtual Reality Experiences - Using Visual and Auditory Stimuli to Improve Tactile Experiences”, 2015 https://dspace.library.uu.nl/handle/1874/317772 Nina’s thesis won the Ngi-NGN Informatie Scriptieprijs in 2015 and has been published at ACM ICMI 2015.
Wendy Bolier, “Drawing in a Virtual 3D Space - Introducing VR Hmm, but if it is my thesis, Drawing in Elementary School Art Education”, 2017 https://dspace.library.uu.nl/handle/1874/353003 who is going to be the author Wendy’s thesis will be published at ACM Multimedia 2018 (A* of the published paper? event in the multimedia community)
19 Fun fact: • The scientific paper with the longest Robert Garisto, an editor of Physical Review author list has 5154 co-authors. Letters, says that publishing the paper presented challenges above and beyond the • It has 33 pages, of which only about already Sisyphean task of dealing with 7.5 are actual content, 1 is teams that have thousands of members. references, 15.5 list the authors’ “The biggest problem was merging the names, and 9 their institutions. author lists from two collaborations with • It is a first joint paper from the two their own slightly different styles,” Garisto teams that operate ATLAS and says. “I was impressed at how well the pair CMS, two massive detectors at the of huge collaborations worked together in Large Hadron Collider (LHC) at responding to referee and editorial CERN, Europe’s particle-physics lab comments,” he adds. near Geneva, Switzerland. Each team is a sprawling collaboration involving researchers from dozens of institutions and countries.
20 21 Some comments on your GMT MSc thesis
• Paper & annotated appendix versus full report • Digital library & examples • GMT project site
From http://www.cs.uu.nl/education/vak.php?stijl=2 &vak=INFOMGMT2
The second part comprises 25 EC (2 periods). You will complete (at least) the following items: • … • Produce a dissemination (see below).
Dissemination The dissemination should be targeted at a broad audience, i.e. people who do not have background in computer science. … The dissemination needs to be uploaded to 22 the GMT thesis projects site. https://www.projects.science.uu.nl/cs-gmt/ • BSc students / Bachelor thesis (but not necessary undergraduate students (publishable) research)
• MSc students / GMT program: ”thesis results are publishable” graduate students is a criteria for a cum laude degree (min. 8.5)
• PhD students A good publication record is a requirement (NL: AIOs, assistent in opleiding) for a PhD degree (“publish or perish”)
• Post Docs
• Assistant professors (NL: UDs, universitair docent)
• Associate professors (NL: UHDs, universitair hoofddocent)
• Full professors (NL: Hoogleraar)
23 Some comments doing a PhD thesis in the area of GMT @ UU
Some interesting information:
PhD regulations https://www.uu.nl/en/organisation/ph d-programmes/practical- matters/regulations-and-forms
Doctoral Degree Regulations https://www.uu.nl/en/files/phd- regulations-2017pdf-1
From the “Instructions to the PhD candidate” for the Doctoral Thesis Defence Ceremony:
21. Dress code PhD candidates are to dress in accordance with the importance Utrecht University attaches to the doctoral thesis defence ceremony. A dark suit with tie or dress suit for male PhD candidates and male paranimfs, and a woman’s suit, suitable dress, or the equivalent of a dress suit for female PhD candidates and female paranimfs are deemed appropriate. 24 • BSc students / Bachelor thesis (but not necessary undergraduate students (publishable) research)
• MSc students / GMT program: ”thesis results are publishable” graduate students is a criteria for a cum laude degree (min. 8.5)
• PhD students A good publication record is a requirement (NL: AIOs, assistent in opleiding) for a PhD degree (“publish or perish”)
• Post Docs
• Assistant professors (NL: UDs, universitair docent) Publications = measure of one’s research contributions, quality, • Associate professors and qualification (NL: UHDs, universitair hoofddocent)
• Full professors (NL: Hoogleraar) Question: What are we commonly measuring here? 1. Number 2. Quality 3. Citations
25 Measuring scientific quality and impact via publications 1. Number 2. Quality 3. Citations
Measuring quality by number of publications and citations Example: h-index
Advantage: • Combines productivity (number of papers) and impact (number of citations) in a single number
Limitations: From https://bitesizebio.com/13614/does-your- • Does not take into account h-index-measure-up/ the number of authors of a paper Hirsch reckons that after 20 years of research, • Does penalize early career scientistsan h index of 20 is good, 40 is outstanding, and (it takes time till your work gets cited)60 is truly exceptional. The advantage of the h- • Gives higher impact to review articlesindex is that it combines productivity (i.e., compared to original papers number of papers produced) and impact (first are generally cited more often)(number of citations) in a single number.
26 Questions: What makes a good paper? What makes a good conference? What makes a good journal?
1. References / citations Þ Impact factor From http://phdcomics.com/, a fun website most PhD students will read sometimes during their studies. 2. Acceptance rates Þ Pros & cons? The Impact Factor is calculated by dividing 3. Organizers the number of citations in the JCR year by the a. publisher, financial sponsor, total number of articles published in the two supporting groups previous years. An Impact Factor of 1.0 b. organizing committee, means that, on average, the articles published program committee) one or two year ago have been cited one time. Þ Again pros & cons From https://www.researchgate.net/post/ How_is_impact_factor_calculated
27 Who organizes conferences, publishes scientific papers, etc.?
Different options exist, but mostly: • Publishing houses (specialized on scientific literature), e.g., Springer, Elsevier, … • Research organizations, e.g., IEEE, ACM, …
Two (most?) important organizations in Computer Science:
IEEE Computer Society (sometimes abbreviated Computer Society or CS) is a The Association for Computing Machinery (ACM) is professional society of IEEE. Its purpose an international learned society for computing. It was and scope is "to advance the theory, founded in 1947, and is the world's largest scientific practice, and application of computer and and educational computing society. The ACM is a non- information processing science and profit professional membership group, with more than technology" and the "professional 100,000 members as of 2011. standing of its members.” The CS is the The ACM is an umbrella organization for academic largest of 39 technical societies organized and scholarly interests in computer science. Its motto under the IEEE Technical Activities Board. is "Advancing Computing as a Science & Profession". https://www.computer.org/ https://www.acm.org/
28 Example ACM SIGCHI is the premier international society for professionals, academics and students who are interested in • Special Interest Groups (SIGs) human-technology and human- E.g. SIGGRAPH, SIGCHI, SIGMM, SIGAI, … computer interaction (HCI). • ACM computing classification For publications, see last time Major journals & magazines:
• Publications ACM Transactions on Computer-Human Interaction Journals & Magazines • Events ACM Interactions magazine Conferences, symposia, workshops
Major event: ACM CHI Conference on You are very much encouraged to go to Human Factors in the IEEE Computer or ACM websites Computing Systems and browse, e.g., special interest groups that you are interested in, look Plus various smaller ones, in-cooperation events,… at what journals they publish, what events they organize or co-sponsor, etc.
You can also browse the ACM ACM SIGCHI publications in the ACM DL: Digital Library which is freely https://dl.acm.org/sig.cfm?id=SP923 accessible within the UU network. 29 Publications: types, levels, rankings, …
“Newness” “Deepness”, Quantity Quality thoroughness • Books • Encyclopedias, book chapters • Editorials (e.g., LNCS)
• Journal articles Rules of thumb • Magazine articles
• Conference papers • Symposia papers • Workshop papers
Note that these are just rules of thumb. Individual performance indicators are way more important!
30 Publications: types, levels, rankings, …
Examples for the relevance of individual performance indicators
• Books Journal with low impact factor or no-name • Encyclopedias, book chapters publisher versus established, high ranked • Editorials (e.g., LNCS) conference? High ranked symposium versus low ranked • Journal articles conference? • Magazine articles E.g.: IEEE/ACM International Symposium on • Conference papers Mixed and Augmented Reality (ISMAR) has an • Symposia papers A* CORE rating (despite being called symposium) • Workshop papers
Then why do low-ranked events even exist and survive? Some are just to make money or even scams! But others do have true value!
31 Publications: types, levels, rankings, …
• Books Also notice different • Encyclopedias, book chapters contribution types, e.g.: • Editorials (e.g., LNCS) • Survey, • Journal articles • Research paper, • Magazine articles • Letters, • … • Conference papers • Symposia papers • Workshop papers • Full paper • Short paper • Poster • WIP / late breaking • Demos • Videos • …
32 Peer-review organization
• Conference papers • Journal articles • Symposia papers • Magazine articles • Workshop papers
• Steering committee • General chairs • Editor in chief • Technical program chairs, • Editorial board Poster chairs, • (Reviewers) invited, not fixed Demo chairs, … • Technical program committee (reviewers)
Note that this can vary. That’s why a look at the organization and review procedure can tell you something about the quality of the publication / event. Indications for good conferences (and journals): • Review procedure • Reviewer discussion (number, length of reviews, …) • Author rebuttal • Meta reviews • In-person meeting, …
33 Typical review procedure
Authors Organizers (authors) Submit paper Assign to reviewers (program chairs) Reviewing (min. 2) (program committee) (authors) Rebuttal Discussion (PC members) Meta review (area or PC chairs) PC meeting (PC or chairs) Decisions sent (program chairs) (authors) Revise if accepted (authors) Submit final version (one author) Present at event Publish proceedings (publication chair)
Note that not all these steps may apply. Again, this can be an indication for quality (or lack thereof) of an event.
34 Peer review process: problems, pitfalls, what can go wrong, … (or: why can you find bad papers, even at top events?)
• Not all review processes are done that thorough. • Fairness? Double-blindness cannot always solve this. • Reviewer bias (remember the importance of citations for one’s career!) • Sloppy, low quality work. • Mistakes happen. • … Other reasons?
Note: While the scientific peer-reviewing process is well established (and works to some degree) it has many flaws, too. Thus, criticism and alternative approaches exist (see, e.g., the open publication initiatives mentioned in the handout).
35 Literature review
General comments: Approaches: • See handout for informal example • Skim journals and some sources of relevant communities (e.g., Google Scholar, DBLP) • Skim proceedings • Keep things in mind discussed today of relevant conferences • Communities & organizers • Keyword searches to get some of events, publications • The relevance of citations papers as starting points • ”Snowballing” • Forward snowballing: Starting points: look at references from • (Dedicated) search engines relevant papers Examples: Google Scholar, DBLP, Citeseer, .. • Backward snowballing: • Publications of the related look at papers citing them community (or communities) Examples: ACM DL, IEEE Computer, … • Known articles and papers
36 Some comments on access to papers via UU resources https://www.uu.nl/en/university-library/help-in-searching/online-access
37 In GMT, mostly: Research question & hypothesis (can be informal) Searching for related research, week 2 Specifying a measurable Observation, framework, week 3 problem, idea
Publishing research • Experimental research, week 4 & 5 results, week 2 • User studies, week 6 • Fundamental research, week 7 • Other (design, engin., systems), week 8
Important characteristics of scientific research Þ Verifiable Þ Replicable Þ E.g., observations have to be measurable Þ Objective (e.g., via operational definitions) Þ Explainable
38 Ethics, integrity, risks
Risks and ethics related to your own research: Scientific integrity refers to honest and transparent methods and • Unintentional mistakes & problems reporting of research. In other words, • Plagiarism there is no cooking the books or even • Implementation bugs subtly influences of bias to privilege (avoid, e.g., by unit testing) some lines of investigation or results • Erroneous conclusions over others. There is no falsification or fabrication or plagiarism. • Reproducibility of results • … https://en.wikiversity.org/wiki/Scientific_integrity • Fraud, misconduct • Leaving out results These issues are important! We will address them throughout the course • Honesty in reporting with concrete examples when • Reproducibility of results discussing the different methods.
Risks and ethics involving others: Question: Why is “Reproducibility of • Research with human test subjects results” listed under intentional and (or animals) unintentional mistakes?
39