QUARTERLY CHECK-IN Technology (Services) TECH GOAL QUADRANT
C Features that we build to improve our technology A Foundation level goals offering
B Features we build for others D Modernization, renewal and tech debt goals
The goals in each team pack are annotated using this scheme illustrate the broad trends in our priorities Agenda ● CTO Team ● Research and Data ● Design Research ● Performance ● Release Engineering ● Security ● Technical Operations
Photos (left to right) Technology (Services) CTO
July 2017 quarterly check-in
All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. CTO Team ● Victoria Coleman - Chief Technology Officer ● Joel Aufrecht - Program Manager (Technology) ● Lani Goto - Project Assistant ● Megan Neisler - Senior Project Coordinator ● Sarah Rodlund - Senior Project Coordinator ● Kevin Smith - Program Manager (Engineering)
Photos (left to right) CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 CTO 4.5 [LINK] ANNUAL PLAN
GOAL: expand and strengthen our technical communities
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Program 4: Technical LAST QUARTER community building (none) Outcome 5: Organize Wikimedia Developer
Summit NEXT QUARTER
Objective 1: Developer Technical Collaboration Decide on event location, dates, theme, deadlines, etc. Summit web page and publicize the information published four months before the event (B)
STATUS: OBJECTIVE IN PROGRESS Technology (Services) Research and Data
July, 2017 quarterly check-in
Principal Research Scientist Data Analyst Senior Research Scientist
Director, Head of Research Software Engineer Research Fellow Research Fellow
5 f/t staff • 2 fellows • 3 contractors • 16 collaborators We use research methods to design new technology and produce knowledge to understand and empower our communities
We act as the bridge between the organization, the Wikimedia movement and the academic community CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Research <2> [LINK] ANNUAL PLAN
OUTCOME: Annual Workshops and outreach
What is your objective / Who is working on this? What impact / deliverables are you expecting? workflow?
Annual LAST QUARTER workshops and ● Workshop coorganizers at ● Hosted the annual Wiki Research Workshop at Stanford and EPFL WWW '17 in Perth, Australia outreach ● 6 WikiCite organizers + ● Hosted WikiCite 2017 in Vienna, Austria substantial support from ● Attended AI for Good Summit in Geneva, Legal, Eng-Admin, Switzerland (B) Developer Relations ● Attended the Wikimedia Hackathon in Vienna
NEXT QUARTER
● Write up a report from WikiCite
STATUS: OBJECTIVE COMPLETE Annual Workshops and outreach (1/2)
● More than 60 researchers attended our 4th annual Wiki Research Workshop at the WWW ‘17 conference in Perth ● 10 papers were accepted and the authors presented their work as part of the poster presentation Annual Workshops and outreach (2/2)
● Significant progress in laying the foundation for citations as structured data to support free knowledge ● Nearly 100 attendees from 22 countries attended a 3-day event, with 16 conference presentations, 17 summit sessions, 38 lightning talks, over 20 hackathon demos. ● Building technical partnerships with Internet Archive, Zotero, Crossref, DBLP, OCLC and relationships with funders CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Research → Scoring Platform <5> [LINK] ANNUAL PLAN
OUTCOME: 1: Innovate tools that use AI; make wiki-work more efficient.
What is your objective? Who are you working with? What impact / deliverables are you expecting?
AI as a service LAST QUARTER Scoring Platform (Amir) ● Wikidata item quality model in ORES. (Done) Research contractors (Morten ● Complete a research study to characterize and model (B,C) Warncke-Wang, Andrew Hall, article importance. (Ongoing) Meen Chul Kim) ● Conduct research to characterize the value of statements in Wikidata. (Done) Lots of volunteers from various communities ● Design machine-learning methods to extract and
analyze citations and their context. (Ongoing)
NEXT QUARTER
Scoring Platform ● Deploy thresholds selection system (1.1) Community Engagement ● Advanced support for Albanian and Romanian WP, basic support for Greek & Tamil WP (1.1) ● Design schema and outreach for meta ORES (2.1)
STATUS: OBJECTIVE IN PROGRESS CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Research → Scoring Platform <5> [LINK] ANNUAL PLAN
Deliverable 1: Wikidata item quality model in ORES. (DONE)
Completed outreach and labeling campaign. Trained and deployed model. Model shows a high level of fitness.
Deliverable 2: Complete a research study to characterize and model article importance. (ONGOING)
Literature review and modeling work complete. Outreach to WikiProjects (the owners of “importance”) has been substantial. Many lessons learned about the meaning of “importance”. Missing infrastructure for ORES deployment identified and tasked.
Deliverable 3: Conduct research to characterize the value of statements in Wikidata. (DONE)
Analysis of value of entire entities (more coarse than *statements*) complete. Dataset release complete. Statement tracking blocked on Wikidata engineering. Also, the contract was substantially delayed so work started ~1 month late.
Deliverable 4: Design machine-learning methods to extract and analyze citations and their context (ONGOING)
Citation extraction complete and extraction schema standardized. Report in progress for GROUP’18. Contract delayed for almost the entire quarter. Machine learning work is delayed but in progress. ● The Keilana Effect (blog and paper accepted) ● New models for: ○ English Wikipedia (Draft quality) ○ Estonian Wikipedia ○ Finnish Wikipedia ○ Hebrew Wikipedia ○ Korean Wikipedia ○ Wikidata (Item quality) ● Mentorship @ Wikimedia Hackathon brought in new volunteers from: ○ Tamil Wikipedia ○ Greek Wikipedia ○ German Wikipedia ○ Finnish Wikipedia ● Worked with WMF Product to support new RC Filters ○ E.g. a study of overlap between “damaging” and “goodfaith” predictions for newcomers Other Q4 accomplishments
● Initiated research to expand the results of Why We Read Wikipedia to 14 languages. We worked with the community to prepare the surveys for their languages and ran them. The result is a collection of 254,000 responses that we are analyzing in Q1 and Q2. ● Continued research on building recommendation systems for helping editathon organizers and newcomers with automatic template generation. The focus has been on deriving an algorithm that can turn the category graph of Wikipedia to a hierarchical graph that can be read by machines. First results are available, but much more improvement needed for the algorithm to be usable. ● Nearly completed productization of the Article Recommendation API, to be completed in Q1. ● Hosted an AMA on Reddit on AI and community dynamics at Wikimedia ● Rebuttal to “Even Good Bots Fight” submitted to CSCW (positive initial reviews). Blog post in progress. See inane media coverage (e.g. The Growing Problem of Bots that Fight Online) Technology (Services) Design Research
July 2017 quarterly check-in
All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. 1 Manager/ Lead Design Researcher 1 Senior Design Researcher CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 1 [LINK] ANNUAL PLAN
GOAL: Use research-centered approach to drive product development
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
New Editor Experiences LAST QUARTER Research ● ● Lit review of prior WMF Team (Editing and Design Lit review, analysis of prior research about new research about Research (two people) editors: done ● new editors Reboot Team (two people Contextual inquiry in South Korea: done ● ● Contextual inquiry: Four local researchers (two from Contextual inquiry in Czech Republic: done new editor South Korea, two from Czech retention in South Republic)
Korea and Czech NEXT QUARTER Republic People on Editing, Communications, ● Synthesis workshop with Reboot and sharing -done Community Engagement and ● Complete report from both contextual inquiries Research teams. ● Begin collaboratively applying findings ● Begin Open Access release of corpus
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 1 [LINK] ANNUAL PLAN
GOAL: Use research-centered approach to drive product development
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
New Readers: LAST QUARTER ● Heuristic evaluation of ● Kiwix App (WikiMed) New Readers Reading Completed Heuristic evaluation of Kiwix -done ● ● Provide (Anne, Toby, Nirzar) Provide recommendations for improvements -done ● recommendations for Kiwix (Emmanuel) Clarification of findings with Emmanuel at Kiwix - improvement to Kiwix Communication (Zack) done ● team Partnerships (Jack, Jorge, Collaborated with Awareness team on choosing media ● Contribute Design Ravi) partners in Nigeria and India - done Research perspectives to Affordability and NEXT QUARTER Awareness tracks as TBD needed
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 2. Expand Research capabilities ANNUAL PLAN
GOAL 1: Build an open infrastructure
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Understand editing LAST QUARTER culture Structured Data Program, External ● Analyze GLAM user workshop data (NOT DONE) Collaborator (Carnegie Mellon) ● Publish Teahouse study results (NOT DONE)
NEXT QUARTER
Structured Data Program, External ● Develop interview protocol; interview 6-8 GLAM Collaborator (Carnegie Mellon) stakeholders; draft user stories; share initial findings at Structured Data offsite ● Publish Teahouse study results and data in peer-reviewed venue and open repos; share with anti-harassment team
STATUS: OBJECTIVE IN PROGRESS (Delayed) CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 2. Expand Research capabilities ANNUAL PLAN
GOAL 3: Perform evaluative design research
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Conduct evaluative design LAST QUARTER research; Expand capacity for ● standardized iterative user Reading (Apps) Complete a user study with crowdworkers to testing compare ‘top read’ and ‘trending’ lists of article recommendations on mobile app
NEXT QUARTER
Reading (Apps/Services) ● Complete 1-2 additional user studies of ‘trending’ edits model in mobile features ● Reduce per-study overhead by automating data cleaning process ● Document the method
STATUS: OBJECTIVE IN PROGRESS CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Design Research 2. Expand Research capabilities ANNUAL PLAN
GOAL 4: Improve communication and access of all research projects
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Create a research LAST QUARTER minisite that would act as a landing page for all External contractor Scope the project, hire a content writer, and draft native resources related to content for the Research Landing Page research at the organization NEXT QUARTER
External contractor, Reading Finalize all native content, build the Research landing design, Communication page, write blog posts
STATUS : OBJECTIVE IN PROGRESS Technology (Services) Performance
July 2017 quarterly check-in
All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. Performance Team
Gilles Dubuc Aaron Schulz Peter Hedenskog Timo Tijhof
Photos (left to right) Myleen Hollero / CC BY-SA-3.0 Myleen Hollero / CC BY-SA-3.0 Myleen Hollero / CC BY-SA-3.0 Sage Ross / CC BY-SA 2.0 CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 4.1.1 [LINK] ANNUAL PLAN
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Improve page LAST QUARTER delivery (D) We can’t look at First Paint over the course of a year (historical data is tainted). We didn’t improve it by the targeted 15%, however, it has remained stable since the metrics have been fixed.
Save Timing was improved by 25% for the 99th percentile, and by 15% for the median. Annual objective was 25%.
NEXT QUARTER
We won’t be committing to arbitrary performance targets like this anymore, since it’s impossible to know ahead of time what is actually feasible.
Efforts will continue as part of FY17-18 Program 1.
STATUS: OBJECTIVE PARTIALLY COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 4.1.3 [LINK] ANNUAL PLAN
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Improve site LAST QUARTER availability (B) Technical Operations Evaluated Dynomite as a potential replacement for Mcrouter for active/active datacenter support of our caching. (T156938)
NEXT QUARTER
Will continue as part of FY17-18 Program 1.
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 4.1 [LINK] ANNUAL PLAN
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Migrate media LAST QUARTER thumbnailing from Technical Operations Thumbor now handles all production traffic of MediaWiki to a new thumbnails for all public wikis. (T121388) dedicated open-source NEXT QUARTER service (D) Will continue as part of FY17-18 Program 1.
STATUS: OBJECTIVE COMPLETE
CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 4.1 [LINK] ANNUAL PLAN
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Set up monitoring LAST QUARTER of performance Performance alerts extended to real user metrics metrics with alerts (Navigation Timing) and article Save Timing. and notifications Caught several incidents already. (C) NEXT QUARTER
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 4.1 [LINK] ANNUAL PLAN
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Research isolated LAST QUARTER browser testing Found a synthetic software stack that allows us to environment and achieve very small variance between runs. proxy (C)
NEXT QUARTER
Work will continue in Q2 of FY 17-18, once the fork of an open source project we’ve been working with (created by a PHD student) is publicly released. Will continue as part of FY17-18 Program 1.
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 4.1 [LINK] ANNUAL PLAN
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Communicate LAST QUARTER about our Communications Published multiple blog posts, one of which was picked performance work up by Communications for the main blog. (B) Improved the performance.wikimedia.org portal.
NEXT QUARTER
Will continue as part of FY17-18 Program 1.
STATUS: OBJECTIVE COMPLETE Screenshot from twitter.com Screenshot from twitter.com Other tasks and ● 4 blog posts ● 10 incident investigations ● 1 extension review unplanned ● 2 org-wide perf audits work
FY 2017-2018
NASA CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 1.12 [LINK] ANNUAL PLAN
OUTCOME: All production sites and services maintain current levels of availability or better
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Assist in the LAST QUARTER architectural design of new services and making them NEXT QUARTER operate at scale (B) Technical Operations Follow-up bugfixes and improvements after Thumbor deployment to production
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 1.22 [LINK] ANNUAL PLAN
OUTCOME: All our users consistently experience systems that perform well
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Catch and address LAST QUARTER performance regressions in a timely fashion through NEXT QUARTER automation (B) Technical Operations Test user performance from Asia to validate changes when the Asia Cache PoP goes live
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 1.23 [LINK] ANNUAL PLAN
OUTCOME: All our users consistently experience systems that perform well
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Modernize our LAST QUARTER performance toolset. We will measure performance NEXT QUARTER metrics that are closer to what Rework Navigation Timing metrics to make them users experience stackable
(C) Add metrics for master queries on HTTP GET/HEAD
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 1.2 [LINK] ANNUAL PLAN
OUTCOME: All our users consistently experience systems that perform well
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Infrastructural LAST QUARTER improvements aiming for performance gains
(D) NEXT QUARTER
Preload base module requests
Upgrade to jQuery 3
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Performance 8.11 [LINK] ANNUAL PLAN
OUTCOME: Our audiences enjoy improved MediaWiki and REST API availability
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
MediaWiki support LAST QUARTER for having read-only “read” requests
(GET/HEAD) be NEXT QUARTER routed to other data Technical Operations Enable HTTPS for Swift and MariaDB clients centers (B) Deploy mcrouter on Beta
STATUS: OBJECTIVE ON TRACK Technology (Services) Release Engineering
July 2017 quarterly check-in
All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise.
CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA Streamlined Services FOUNDATION July 2017 6 ANNUAL PLAN Delivery Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment.
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Build a runnable image LAST QUARTER from tested and ● define 'tested and confirmed' / determine the confirmed code that is Operations, Services suitable for promotion. entry-point for building a container (C - features) ● generate or consume a docker file for building ● build image, upload and tag it in docker registry
Set up a continuous NEXT QUARTER integration and deployment pipeline (C Operations, Services ● Define functional tests for Mathoid running on the - features) staging Kubernetes cluster for use in future gating decisions ● Define method for monitoring and to the above functional tests
STATUS: OBJECTIVE IN PROGRESS, ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Release Engineering None ANNUAL PLAN
GOAL: Train 2.0 / Unified deployment tooling
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Move MW+Extension LAST QUARTER deploys to scap3 (D - tech debt) Operations Integrate scap with etcd/pybal to automatically depool and repool servers
Not complete - pending cross-team code review. ETA of early July.
Deprecate use of NEXT QUARTER Trebuchet across production Operations, Security, Discovery, Trebuchet is our deprecated deployment tool and is no (D - tech debt) Cloud Services longer supported, it should be removed everywhere
STATUS: OBJECTIVE IN PROGRESS, ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Release Engineering None ANNUAL PLAN
GOAL: Consistent MediaWiki third-party releases
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Release MediaWiki 1.29 LAST QUARTER (D - Tech debt) All MW developers A quality MediaWiki 1.29 is successfully released
Late - Unexpected blocker issues arose late in the process. Released July 13th.
Improve release NEXT QUARTER process (D - Tech debt) Security, All MW developers Finish implementation of nightly build server
STATUS: OBJECTIVE IN PROGRESS, ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Release Engineering 1 ANNUAL PLAN
GOAL: Effective and easy-to-use testing infrastructure tooling
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Develop and migrate to LAST QUARTER a Javascript-based browser testing stack
(C - improve our own NEXT QUARTER feature set; D - Tech debt) Wikidata, CirrusSearch, All MW Migrate majority of developers to Javascript-based developers browser test framework (webdriver.io)
STATUS: OBJECTIVE IN PROGRESS, ON TRACK Other Successes
● free some repositories from their ● Deploy logstash/plugins with scap3 unintended chain to mediawiki/core ● Set up a warm failover in Dallas/codfw for ● Add support for skin and extension Gerrit. dependencies in new skin unit test ● Automatically clean up unused wmfXX ● Convert all repos to use npm Jenkins job versions with jsonlint and eslint ● setup automatic deletion of old l10nupdate ● Raise priority for ● Release MediaWiki 1.28.1/1.27.2/1.23.16 operations-mw-config-composer-hhvm-jess ● Building Better Software (Hack-a-thon session) ie from the gate-and-submit pipeline ● Improved scap rollback behavior ● Depool precise jenkins instances ● Migrate NodeJS Nodepool jobs from Trusty to ● Enable embedding of videos from Jessie Commons on Phabricator ● Alert when Zuul/Gearman queue is stalled ● scap now allows restarting multiple ● WebdriverIO CREDIT showcase services Technology (Services) Security
July 2017 quarterly check-in
All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. Darian Anthony Brian Wolff Sam Reed Patrick CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Security 3 & 4 [LINK] ANNUAL PLAN
GOAL: Improve wiki account authentication
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Improve wiki account LAST QUARTER authentication (C, D) Community Tech - Improve security messages to users (LoginNotify extension being tested)
NEXT QUARTER
Editing - Support Community Tech in deployment of LoginNotify - Provide insight into active sessions
STATUS: OBJECTIVE IN PROGRESS, ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION July 2017 Security 2 & 3 [LINK] ANNUAL PLAN
GOAL: Improve MediaWiki security release process
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Improve MediaWiki NEXT QUARTER security release process (C) Release Engineering, Tech. Ops. - Adherence to a stable, regular security release timeline - Improved ability to make emergency releases with lower impact to Security and RelEng teams - Earlier detection of unreleased patch inapplicability as code development progresses
STATUS: OBJECTIVE IN PROGRESS, ON TRACK Technology (Services) Technical Operations
July 2017 quarterly check-in
All content is © Wikimedia Foundation & available under CC BY-SA 4.0, unless noted otherwise. Mark Bergsma Riccardo Coccioli Jaime Crespo Filippo Giunchendi Ariel Glenn Keith Herron
Alexandros Kosiaris Giuseppe Lavagetto Faidon Liambotis Manuel Arostegui Moritz Mühlenhoff Daniel Zahn
Rob Halsell Chris Johnson Papaul Tshibamba Brandon Black Emanuele Rocca Arzhel Younsi CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 8.4.3 Improve site FOUNDATION ANNUAL PLAN July 2017 TechOps / Core Ops performance & reliability
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Perform a datacenter LAST QUARTER switchover (A) ● MediaWiki ● Successfully switch backend traffic (MediaWiki, ● Performance Swift, ElasticSearch, RESTBase, and Parsoid) to ● Release Engineering be served from codfw with no downtime and ● Services (RESTbase) reduced read-only time ● Discovery (Search) ● Serve the site from codfw for a longer period of ● Parsing (Parsoid) time than the last switchover (at least 2 weeks) ● Analytics (EventBus) ● Community Liaisons
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 8.3.1.1 Operational FOUNDATION ANNUAL PLAN July 2017 TechOps / Core Ops excellence
GOAL: Keep Wikimedia's sites and services running responsibly, reliably, and securely, including protection against and response to outages, data loss, and breaches
What is your objective / Who are you working What impact / deliverables are you expecting? workflow? with?
Reduce Swift technical LAST QUARTER debt (D) ● Support diversified storage policies such as different levels of replication and low-latency containers ● Upgrade the Swift infrastructure to run using Debian jessie (or later) ● Allow rate-limited container operations for better isolation among Swift users and use cases ● Delete non-used/non-requested thumbnail sizes periodically to reclaim space
STATUS: OBJECTIVE COMPLETE CHECK IN TEAM/DEPT July 2017 Technical Operations
● Thumbnail disk space reduced by ~20 TB ● Replication factor 6 ● 120 TB raw space saved ● 3 storage backend machines CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA FOUNDATION (none in FY16-17) ANNUAL PLAN July 2017 TechOps / Core Ops 6: Streamlined service delivery
GOAL: Streamlined service delivery
What is your objective Who are you What impact / deliverables are you expecting? / workflow? working with?
Prepare to serve LAST QUARTER applications from Kubernetes (C) Services, Release ● Eliminate SPOFs in the existing eqiad infrastructure Engineering ● Expand the infrastructure to codfw ● Prepare and maintain base container images ● Define a production process to keep images up-to-date ● Design/implement a Kubernetes-based staging environment (s)
NEXT QUARTER
Services, Release ● Implement a pod networking policy approach Engineering ● Upgrade to Kubernetes >= 1.5 ● Standardize on a "default" pod setup ● Experiment with ingress solutions (stretch)
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 8.4.1.4 Improve FOUNDATION ANNUAL PLAN July 2017 TechOps / Traffic site performance
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Deploy Asian caching LAST QUARTER POP (A) Legal, Finance, DC Ops ● Finalize and sign contract with the data center vendor ● Finalize system specifications and purchase the necessary hardware ● Prepare and deploy configuration changes needed for the new systems ● Allocate the data center's IPv4/IPv6 address space A vendor & site recommendation along with selection criteria has now been provided to and approved by the C-team, but no contract has been signed yet.
STATUS: OBJECTIVE AT RISK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 8.4.1.4 Improve FOUNDATION ANNUAL PLAN July 2017 TechOps / Traffic site performance
GOAL: Make access to Wikipedia faster and more reliable, especially for mobile and global users
What is your objective / Who are you working with? What impact / deliverables are you expecting? workflow?
Deploy Asian caching NEXT QUARTER POP (A) Legal, Finance, DC Ops, Continue Asia Cache PoP procurement, installation, Partnerships and configuration tasks ● Finish up trailing purchasing tasks from previous quarter (DC, hardware, network links, etc). ● Procure at least one transit or peering link to help advance address space issue ● Physically install all hardware ● Acquire address space & communicate it to Wikipedia Zero partners (via the Zero team) ● Turn up network links (stretch) ● Configure network devices and hosts (stretch)
STATUS: OBJECTIVE AT RISK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 1 Availability, performance, FOUNDATION ANNUAL PLAN July 2017 TechOps / Core Ops and maintenance
GOAL: We will maintain the availability of Wikimedia’s sites and services for our global audiences and ensure they’re running reliably, securely, and with high performance.
What is your objective / Who are you working What impact / deliverables are you expecting? workflow? with?
Outcome 1: All NEXT QUARTER production sites and services maintain current Release Engineering, Remove Salt from our infrastructure levels of availability or WMCS ● Port debdeploy to Cumin better. ● Migrate the reimage script to Cumin Objective 1: Deploy, ● Remove support for the Trebuchet deployment system update, configure, and ● maintain production Remove Salt from production & WMCS services (Traffic infrastructure, databases & storage, MediaWiki application servers, (micro)services, network, and miscellaneous sites & services) (C, D)
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 1 Availability, performance, FOUNDATION ANNUAL PLAN July 2017 TechOps / Core Ops and maintenance
GOAL: We will maintain the availability of Wikimedia’s sites and services for our global audiences and ensure they’re running reliably, securely, and with high performance.
What is your objective / Who are you working What impact / deliverables are you expecting? workflow? with?
Outcome 1: All NEXT QUARTER production sites and services maintain current Release Engineering Prepare for Puppet 4 levels of availability or ● Support directory environments in our Puppet infrastructure better. and add an environment that uses Puppet's future parser Objective 1: Deploy, ● Switch at least 3 node groups to the future parser update, configure, and environment maintain production ● Force both current and future parser for every test in the services (Traffic puppet-compiler infrastructure, databases ● Integrate puppet-compiler with the Continuous Integration & storage, MediaWiki infrastructure (task T166066) (stretch) application servers, (micro)services, network, ● Speed up CI for operations/puppet (task T166888) and add and miscellaneous sites & future parser validation services) (C, D)
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM WIKIMEDIA 1 Availability, performance, FOUNDATION ANNUAL PLAN July 2017 TechOps / Core Ops and maintenance
GOAL: We will maintain the availability of Wikimedia’s sites and services for our global audiences and ensure they’re running reliably, securely, and with high performance.
What is your objective / Who are you working What impact / deliverables are you expecting? workflow? with?
Outcome 1: All NEXT QUARTER production sites and services maintain current Improve database backups' coverage, monitoring and data recovery levels of availability or time (part 1) better. ● Adjust configuration management manifests to support Objective 1: Deploy, MariaDB multi-instances update, configure, and ● Migrate at least 2 instances on 1 dbstore host to the new maintain production multi-instance setup services (Traffic ● Research backup storage options and prepare a design infrastructure, databases document & storage, MediaWiki ● Investigate and experiment with replacements of mysqldump application servers, (micro)services, network, and miscellaneous sites & services) (D)
STATUS: OBJECTIVE ON TRACK CHECK IN TEAM/DEPT PROGRAM July 2017 Technical Operations 8.3.1.1 Operational excellence
● Migrated application servers to HHVM ● Refactored mediawiki comment/revision 3.18 table ● Removed torrus, replaced with ● Reduced mediawiki max query to 60s prometheus ● Rebuilt pc hosts with file-per-table ● Expanded mail server monitoring ● Migrated tendril to dedicated VMs ● Migrated eqiad row D to new switches ● Migrated Codfw mediawiki db to jessie ● Upgraded switch firmware in data center ● Ganeti installations became multi-row rows increasing availability ● mariadb-systemd integration ● poolcounter hardware VMized reducing ● stretch support for mariadb costs ● MariaDB 10.1 compatibility ● url-downloader made redundant ● ~200000 schema changes adding PKs ● Patched an etherpad vulnerability ● All core servers/servers with core data ● Upgraded IRC bots to newer versions of support TLS connections and use it for supporting libraries replication ● Migrated multiple puppet roles to the ● Backup server (dbstore1001) migrated to profile pattern jessie and upgraded its MariaDB version. ● Started work on the puppet future parser CHECK IN TEAM/DEPT PROGRAM July 2017 Technical Operations 8.3.1.1 Operational excellence
● Enabled structured facts in puppet ● Production load balancer (pybal): added ● Upgraded apertium with Translation team unit tests, solidified BGP ● Migrated various deployed softwares to ● Improved network monitoring scap3 (with services, releng) ● Upgraded Prometheus to bare metal ● Upgraded OTRS to 5.0.19 and then to ● Improved investigation of HTTP errors 5.0.20 with Logstash ● Upgraded librenms to latest version ● Made icinga redundant in a master/slave pattern ● 2/3 of the jessie servers were migrated to Linux 4.9 (Stretch kernel) ● Reconfigured wikidata xml dumps to run much faster ● Enabled TCP BBR on cache nodes ● Implemented rate limiting of HTTP requests ● Prometheus monitoring of cache systems and load balancers