QUARTERLY CHECK-IN Technology (Services) TECH GOAL QUADRANT C Features that we build to improve our technology A Foundation level goals offering B Features we build for others D Modernization, renewal and tech debt goals The goals in each team pack are annotated using this scheme illustrate the broad trends in our priorities Agenda ● CTO Team ● Research and Data ● Design Research ● Performance ● Release Engineering ● Security ● Technical Operations Photos (left to right) Technology (Services) CTO July 2017 quarterly check-in All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. CTO Team ● Victoria Coleman - Chief Technology Officer ● Joel Aufrecht - Program Manager (Technology) ● Lani Goto - Project Assistant ● Megan Neisler - Senior Project Coordinator ● Sarah Rodlund - Senior Project Coordinator ● Kevin Smith - Program Manager (Engineering) Photos (left to right) CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 CTO 4.5 [LINK] ANNUAL PLAN GOAL: expand and strengthen our technical communities What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow? Program 4: Technical LAST QUARTER community building (none) Outcome 5: Organize Wikimedia Developer Summit NEXT QUARTER Objective 1: Developer Technical Collaboration Decide on event location, dates, theme, deadlines, etc. Summit web page and publicize the information published four months before the event (B) STATUS: OBJECTIVE IN PROGRESS Technology (Services) Research and Data July, 2017 quarterly check-in <Confidentiality notice if applicable> All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. 2 research scientist positions we’re actively hiring for Principal Research Scientist Data Analyst Senior Research Scientist Director, Head of Research Software Engineer Research Fellow Research Fellow 5 f/t staff • 2 fellows • 3 contractors • 16 collaborators We use research methods to design new technology and produce knowledge to understand and empower our communities We act as the bridge between the organization, the Wikimedia movement and the academic community CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Research <2> [LINK] ANNUAL PLAN OUTCOME: Annual Workshops and outreach What is your objective / Who is working on this? What impact / deliverables are you expecting? workflow? Annual LAST QUARTER workshops and ● Workshop coorganizers at ● Hosted the annual Wiki Research Workshop at Stanford and EPFL WWW '17 in Perth, Australia outreach ● 6 WikiCite organizers + ● Hosted WikiCite 2017 in Vienna, Austria substantial support from ● Attended AI for Good Summit in Geneva, Legal, Eng-Admin, Switzerland (B) Developer Relations ● Attended the Wikimedia Hackathon in Vienna NEXT QUARTER ● Write up a report from WikiCite STATUS: OBJECTIVE COMPLETE Annual Workshops and outreach (1/2) ● More than 60 researchers attended our 4th annual Wiki Research Workshop at the WWW ‘17 conference in Perth ● 10 papers were accepted and the authors presented their work as part of the poster presentation Annual Workshops and outreach (2/2) ● Significant progress in laying the foundation for citations as structured data to support free knowledge ● Nearly 100 attendees from 22 countries attended a 3-day event, with 16 conference presentations, 17 summit sessions, 38 lightning talks, over 20 hackathon demos. ● Building technical partnerships with Internet Archive, Zotero, Crossref, DBLP, OCLC and relationships with funders CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Research → Scoring Platform <5> [LINK] ANNUAL PLAN OUTCOME: 1: Innovate tools that use AI; make wiki-work more efficient. What is your objective? Who are you working with? What impact / deliverables are you expecting? AI as a service LAST QUARTER Scoring Platform (Amir) ● Wikidata item quality model in ORES. (Done) Research contractors (Morten ● Complete a research study to characterize and model (B,C) Warncke-Wang, Andrew Hall, article importance. (Ongoing) Meen Chul Kim) ● Conduct research to characterize the value of statements in Wikidata. (Done) Lots of volunteers from various communities ● Design machine-learning methods to extract and analyze citations and their context. (Ongoing) NEXT QUARTER Scoring Platform ● Deploy thresholds selection system (1.1) Community Engagement ● Advanced support for Albanian and Romanian WP, basic support for Greek & Tamil WP (1.1) ● Design schema and outreach for meta ORES (2.1) STATUS: OBJECTIVE IN PROGRESS CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Research → Scoring Platform <5> [LINK] ANNUAL PLAN Deliverable 1: Wikidata item quality model in ORES. (DONE) Completed outreach and labeling campaign. Trained and deployed model. Model shows a high level of fitness. Deliverable 2: Complete a research study to characterize and model article importance. (ONGOING) Literature review and modeling work complete. Outreach to WikiProjects (the owners of “importance”) has been substantial. Many lessons learned about the meaning of “importance”. Missing infrastructure for ORES deployment identified and tasked. Deliverable 3: Conduct research to characterize the value of statements in Wikidata. (DONE) Analysis of value of entire entities (more coarse than *statements*) complete. Dataset release complete. Statement tracking blocked on Wikidata engineering. Also, the contract was substantially delayed so work started ~1 month late. Deliverable 4: Design machine-learning methods to extract and analyze citations and their context (ONGOING) Citation extraction complete and extraction schema standardized. Report in progress for GROUP’18. Contract delayed for almost the entire quarter. Machine learning work is delayed but in progress. ● The Keilana Effect (blog and paper accepted) ● New models for: ○ English Wikipedia (Draft quality) ○ Estonian Wikipedia ○ Finnish Wikipedia ○ Hebrew Wikipedia ○ Korean Wikipedia ○ Wikidata (Item quality) ● Mentorship @ Wikimedia Hackathon brought in new volunteers from: ○ Tamil Wikipedia ○ Greek Wikipedia ○ German Wikipedia ○ Finnish Wikipedia ● Worked with WMF Product to support new RC Filters ○ E.g. a study of overlap between “damaging” and “goodfaith” predictions for newcomers Other Q4 accomplishments ● Initiated research to expand the results of Why We Read Wikipedia to 14 languages. We worked with the community to prepare the surveys for their languages and ran them. The result is a collection of 254,000 responses that we are analyzing in Q1 and Q2. ● Continued research on building recommendation systems for helping editathon organizers and newcomers with automatic template generation. The focus has been on deriving an algorithm that can turn the category graph of Wikipedia to a hierarchical graph that can be read by machines. First results are available, but much more improvement needed for the algorithm to be usable. ● Nearly completed productization of the Article Recommendation API, to be completed in Q1. ● Hosted an AMA on Reddit on AI and community dynamics at Wikimedia ● Rebuttal to “Even Good Bots Fight” submitted to CSCW (positive initial reviews). Blog post in progress. See inane media coverage (e.g. The Growing Problem of Bots that Fight Online) Technology (Services) Design Research July 2017 quarterly check-in All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. 1 Manager/ Lead Design Researcher 1 Senior Design Researcher CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 1 [LINK] ANNUAL PLAN GOAL: Use research-centered approach to drive product development What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow? New Editor Experiences LAST QUARTER Research ● ● Lit review of prior WMF Team (Editing and Design Lit review, analysis of prior research about new research about Research (two people) editors: done ● new editors Reboot Team (two people Contextual inquiry in South Korea: done ● ● Contextual inquiry: Four local researchers (two from Contextual inquiry in Czech Republic: done new editor South Korea, two from Czech retention in South Republic) Korea and Czech NEXT QUARTER Republic People on Editing, Communications, ● Synthesis workshop with Reboot and sharing -done Community Engagement and ● Complete report from both contextual inquiries Research teams. ● Begin collaboratively applying findings ● Begin Open Access release of corpus STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 1 [LINK] ANNUAL PLAN GOAL: Use research-centered approach to drive product development What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow? New Readers: LAST QUARTER ● Heuristic evaluation of ● Kiwix App (WikiMed) New Readers Reading Completed Heuristic evaluation of Kiwix -done ● ● Provide (Anne, Toby, Nirzar) Provide recommendations for improvements -done ● recommendations for Kiwix (Emmanuel) Clarification of findings with Emmanuel at Kiwix - improvement to Kiwix Communication (Zack) done ● team Partnerships (Jack, Jorge, Collaborated with Awareness team on choosing media ● Contribute Design Ravi) partners in Nigeria and India - done Research perspectives to Affordability and NEXT QUARTER Awareness tracks as TBD needed STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 2. Expand Research capabilities ANNUAL PLAN GOAL 1: Build an open infrastructure What