Zulip Documentation Release 1.4.0

1 Zulip overview 3 1.1 Community...... 3 1.2 Installing the Zulip Development environment...... 4 1.3 Running Zulip in production...... 4 1.4 Ways to contribute...... 4 1.5 Google Summer of Code...... 5 1.6 How to get involved with contributing to Zulip...... 5 1.7 License...... 6

2 Zulip architectural overview 7 2.1 Key Codebases...... 7 2.2 Usage assumptions and concepts...... 7 2.3 Components...... 8 2.4 Glossary...... 11

3 Directory structure 13 3.1 Core Python files...... 13 3.2 HTML Templates...... 13 3.3 JavaScript and other static assets...... 14 3.4 Tests...... 14 3.5 Management commands...... 14 3.6 Scripts...... 14 3.7 API and Bots...... 15 3.8 Production puppet configuration...... 15 3.9 Additional Django apps...... 15 3.10 Jinja2 Compatibility Files...... 15 3.11 Translation files...... 16 3.12 Documentation...... 16

4 Zulip Roadmap 17 4.1 Introduction...... 17 4.2 Major projects...... 17 4.3 Core User Experience...... 18 4.4 Social features...... 18 4.5 Real-time sync...... 18 4.6 Onboarding issues...... 18 4.7 Production installation issues...... 19

i 4.8 Administration and management...... 19 4.9 Scalability and performance...... 19 4.10 Technology improvements...... 19 4.11 Technical Debt...... 20 4.12 Security...... 20 4.13 Testing...... 20 4.14 Documentation...... 20 4.15 Integrations and bots...... 20 4.16 Android app...... 21 4.17 iOS app...... 21 4.18 Server/webapp support for mobile...... 21 4.19 Desktop apps...... 21 4.20 Community...... 21

5 Version History 23 5.1 Unreleased...... 23 5.2 1.6.0 – 2017-06-06...... 23 5.3 1.5.2 – 2017-06-01...... 26 5.4 1.5.1 – 2017-02-07...... 26 5.5 1.5.0 – 2017-02-06...... 26 5.6 1.4.3 - 2017-01-29...... 29 5.7 1.4.2 - 2016-09-27...... 29 5.8 1.4.1 - 2016-09-03...... 29 5.9 1.4.0 - 2016-08-25...... 29 5.10 1.3.13 - 2016-06-21...... 32 5.11 1.3.12 - 2016-05-10...... 33 5.12 1.3.11 - 2016-05-02...... 33 5.13 1.3.10 - 2016-01-21...... 34 5.14 1.3.9 - 2015-11-16...... 34 5.15 1.3.8 - 2015-11-15...... 34 5.16 1.3.7 - 2015-10-19...... 35

7 Production Installation 39 7.1 Step 0: Subscribe...... 39 7.2 Step 1: Install SSL Certiﬁcates...... 39 7.3 Step 2: Download and install latest release...... 39 7.4 Step 3: Conﬁgure Zulip...... 40 7.5 Step 4: Initialize Zulip database...... 40 7.6 Step 5: Create a Zulip organization and login...... 41 7.7 Troubleshooting...... 41

8 Troubleshooting 43 8.1 Using supervisorctl...... 43 8.2 Troubleshooting services...... 45

9 Customize Zulip 47 9.1 Mobile and desktop apps...... 47 9.2 Terms of service and Privacy policy...... 47 9.3 Miscellaneous server settings...... 48 9.4 Zulip announcement list...... 48 9.5 Enjoy your Zulip installation!...... 48 ii 10 Mobile push notiﬁcation service 49 10.1 How to sign up...... 49 10.2 Why this is necessary...... 49 10.3 Security and privacy implications...... 50

11 Secure, maintain, and upgrade 51 11.1 Upgrading...... 51 11.2 Upgrading from a git repository...... 53 11.3 Backups...... 53 11.4 Monitoring...... 55 11.5 Scalability...... 55 11.6 Securing your Zulip server...... 56 11.7 Management commands...... 56

12 Security Model 59 12.1 Secure your Zulip server like your email server...... 59 12.2 Encryption and Authentication...... 59 12.3 Messages and History...... 60 12.4 Users and Bots...... 61 12.5 User-uploaded content...... 61 12.6 Final notes and security response...... 62

13 Authentication methods 63 13.1 Adding additional methods using python-social-auth...... 63 13.2 Remote User SSO Authentication...... 64

14 Postgres database details 67 14.1 Remote Postgres database...... 67 14.2 Debugging postgres database issues...... 68 14.3 Stopping the Zulip postgres database...... 68 14.4 Debugging issues starting postgres...... 68 14.5 Postgres Vacuuming alerts...... 69

15 Development environment installation 71 15.1 Requirements...... 71 15.2 Recommended setup (Vagrant)...... 71 15.3 Advanced setup (non-Vagrant)...... 71 15.4 Slow internet connections...... 72 15.5 Installing remotely...... 72 15.6 Next steps...... 72

16 Vagrant environment setup tutorial 73 16.1 Requirements...... 74 16.2 Step 1: Install Prerequisites...... 74 16.3 Step 2: Get Zulip Code...... 77 16.4 Step 3: Start the development environment...... 77 16.5 Step 4: Developing...... 81 16.6 Next Steps...... 83 16.7 Troubleshooting & Common Errors...... 83 16.8 Specifying a proxy...... 88

17 Zulip development environment setup without Vagrant 91 17.1 Installing directly on Ubuntu...... 91 17.2 Installing manually on Linux...... 91 17.3 Using Docker (experimental)...... 97

19 Developing on a remote machine 101 19.1 Connecting to the remote environment...... 101 19.2 Setting up the development environment...... 101 19.3 Running the development server...... 102 19.4 Making changes to code on your remote development server...... 103

20 Writing a new integration 107 20.1 Types of integrations...... 107 20.2 General advice...... 108 20.3 Webhook integrations...... 108 20.4 Python script and plugin integrations...... 109 20.5 Documenting your integration...... 110

21 Webhook walkthrough 111 21.1 Step 0: Create ﬁxtures...... 111 21.2 Step 1: Initialize your webhook python package...... 112 21.3 Step 2: Create main webhook code...... 112 21.4 Step 3: Create an api endpoint for the webhook...... 113 21.5 Step 4: Create tests...... 114 21.6 Step 5: Create documentation...... 116 21.7 Step 5: Preparing a pull request to zulip/zulip...... 117 21.8 Advanced topics...... 117

22 Writing a new application feature 121 22.1 General Process in brief...... 121 22.2 Example Feature...... 122

23 Writing views in Zulip 129 23.1 What this covers...... 129 23.2 What is a view?...... 129 23.3 Modifying urls.py...... 129 23.4 Writing human-readable views...... 130 23.5 Writing API REST endpoints...... 130 23.6 Legacy endpoints used by the web client...... 134 23.7 Webhook integration endpoints...... 134

24 Life of a Request 135 24.1 A request is sent to the server, and handled by Nginx...... 135 24.2 Nginx secures traffic with SSL...... 135 24.3 Static files are served directly by Nginx...... 135 24.4 Nginx routes other requests between django and tornado...... 136 24.5 Django routes the request to a view in urls.py files...... 136 24.6 Views serving HTML are internationalized by server path...... 136 24.7 API endpoints use REST...... 136 24.8 Django calls rest_dispatch for REST endpoints, and authenticates...... 137 24.9 The view will authorize the user, extract request variables, and validate them...... 138 24.10 Results are given as JSON...... 138

25 Reading list 139 25.1 General programming/IT...... 139 25.2 Python...... 140 25.3 Java/Android...... 140 25.4 JavaScript/ECMAScript...... 140 iv 25.5 Git/Version Control Systems (VCS)...... 140 25.6 Computer Science/Algorithms...... 140 25.7 Community experience...... 141 25.8 Competitions/Camps...... 141 25.9 Massive Open Online Courses (MOOC) Platforms...... 141

26 Git & GitHub Guide 143 26.1 Quick start: How Zulip uses Git and GitHub...... 143 26.2 Set up Git...... 144 26.3 How Git is different...... 144 26.4 Important Git terms...... 145 26.5 Get Zulip code...... 147 26.6 Using Git as you work...... 148 26.7 Create a pull request...... 155 26.8 Update a pull request...... 157 26.9 Collaborate...... 157 26.10 Review changes...... 158 26.11 Get and stay out of trouble...... 158 26.12 Zulip-speciﬁc tools...... 162

27 Version control 165 27.1 Commit Discipline...... 165 27.2 Commit Messages...... 166

28 Code style and conventions 169 28.1 Be consistent!...... 169 28.2 Lint tools...... 169 28.3 Secrets...... 170 28.4 Dangerous constructs...... 170 28.5 JS array/object manipulation...... 172 28.6 More arbitrary style things...... 173

29 Python static type checker (mypy) 177 29.1 type_debug.py ...... 177 29.2 Zulip goals...... 178 29.3 Installing mypy...... 178 29.4 Running mypy on Zulip’s code locally...... 178 29.5 Excluded ﬁles...... 179 29.6 Mypy is there to ﬁnd bugs in Zulip before they impact users...... 179 29.7 Annotating strings...... 179

30 Reviewing Zulip server code 181 30.1 Things to look for...... 181 30.2 Tooling...... 182 30.3 Additional Resources...... 182

31 The chat.zulip.org community 183 31.1 This is a bleeding edge development server...... 183 31.2 Community conventions...... 183 31.3 Streams...... 184 31.4 Chat meetings...... 184

v 33 Accessibility 189 33.1 Guidelines...... 189 33.2 Tools...... 189 33.3 GitHub Issues...... 190 33.4 Additional Resources...... 190

34 Testing and writing tests 191 34.1 Overview...... 191 34.2 Running tests...... 191 34.3 Schema and initial data changes...... 192 34.4 Wiping the test databases...... 192 34.5 Local browser testing (local app + web browser)...... 193 34.6 Python 3 Compatibility...... 193

35 Linters 195 35.1 Overview...... 195 35.2 Running the linters...... 195 35.3 General considerations...... 196 35.4 Lint checks...... 196 35.5 lint...... 197 35.6 Philosophy...... 198

36 Backend Django tests 201 36.1 Overview...... 201 36.2 Running tests...... 201 36.3 How to write tests...... 202 36.4 Zulip Testing Philosophy...... 202 36.5 Testing considerations...... 205

37 JavaScript unit tests 207 37.1 HTML output...... 207 37.2 Coverage reports...... 208 37.3 Handling dependencies in unit tests...... 208 37.4 Creating new test modules...... 209

38 Web frontend black-box casperjs tests 211 38.1 Debugging Casper.JS...... 211 38.2 Writing Casper tests...... 212

39 Travis CI 215 39.1 Goals...... 215 39.2 Conﬁguration...... 215 39.3 Useful debugging tips and tools...... 216 39.4 Performance optimizations...... 216

41 Settings system 231 41.1 Server settings...... 231 41.2 Realm settings...... 232

42 Real-time Push and Events 235 42.1 Generation system...... 236 42.2 Delivery system...... 236

43 Queue processors 239 43.1 Adding a new queue processor...... 239 43.2 Publishing events into a queue...... 240 43.3 Clearing a RabbitMQ queue...... 240

44 Zulip bot system 241 44.1 The bots system...... 241 44.2 How to run a bot...... 242 44.3 How to develop a bot...... 243 44.4 Bot API...... 244 44.5 Common problems...... 246 44.6 Future direction...... 247

45 Custom Apps 249 45.1 Deﬁnition...... 249 45.2 Problem statement...... 249 45.3 A quick note on bots/integrations...... 249 45.4 Categories of custom apps...... 250 45.5 World Reader...... 251 45.6 Zulip Reader...... 253 45.7 Deployment issues...... 253

46 Unread counts and the pointer 257 46.1 Pointer logic...... 257 46.2 Unread count logic...... 258 46.3 Testing and development...... 259

47 Markdown implementation 261 47.1 Testing...... 261 47.2 Changing Zulip’s markdown processor...... 262 47.3 Zulip’s Markdown philosophy...... 262 47.4 Zulip’s Changes to Markdown...... 263

49 Static asset pipeline 267 49.1 Primary build process...... 267 49.2 Adding static ﬁles...... 267 49.3 How it works in production...... 268 49.4 Webpack/CommonJS/ES6/Typescript modules...... 268

51 HTML and CSS 271 51.1 Zulip CSS organization...... 271 51.2 Editing Zulip CSS...... 271 51.3 CSS Style guidelines...... 272 51.4 Validating CSS...... 272

52 URL hashes and deep linking 273 52.1 Hashchange...... 273 52.2 Server-initiated reloads...... 274

54 Full-text search 277 54.1 The default full-text search implementation...... 277 54.2 An optional full-text search implementation...... 277

55 Analytics 279 55.1 Analytics backend overview...... 279 55.2 The *Count database tables...... 280 55.3 CountStats...... 280 55.4 The FillState table...... 280 55.5 Performance strategy...... 280 55.6 Backend Testing...... 281 55.7 LoggingCountStats...... 281 55.8 Analytics UI development and testing...... 282

56 Translating Zulip 283 56.1 Translation style guides...... 283 56.2 Translation process...... 284 56.3 Translators’ workflow...... 284 56.4 Testing translations...... 285 56.5 Setting the default language in Zulip...... 286 56.6 Translation resource files...... 286 56.7 Backend translations...... 286 56.8 Frontend translations...... 287 56.9 Transifex config...... 288 56.10 Transifex CLI setup...... 288

59 Zulip server release checklist 293 59.1 A week before the release...... 293 59.2 Final release preparation...... 293 59.3 Executing the release...... 294 59.4 Post-release...... 294

61 Documentation 297 61.1 Developer and sysadmin documentation...... 297 61.2 Core website documentation...... 298 61.3 General user documentation...... 298

62 General user guide documentation 299 62.1 Editing and testing...... 299 62.2 Writing documentation...... 299 62.3 Features...... 301 62.4 Documentation template...... 308 viii Zulip Documentation, Release 1.4.0

Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip supports both private messaging and group chats via conversation streams. Zulip also supports fast search, drag-and-drop ﬁle uploads, image previews, group private messages, audible notiﬁca- tions, missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org and in these docs. Our code is available at our GitHub repository. This set of documents covers installation and contribution instructions. Contents: • Overview • Zulip in production • Development environment • Developer tutorials • Code contribution guide • Code testing • Subsystem documentation Zulip overview | Community | Installing for dev | Installing for production | Ways to contribute | How to get involved | License

There are several places online where folks discuss Zulip. • The primary place is the Zulip development community Zulip server. • For Google Summer of Code students and applicants, we have a mailing list for help, questions, and announcements. But it’s often simpler to visit chat.zulip.org instead. • We have a public mailing list that is currently pretty low trafﬁc because most discussions happen in our public Zulip instance. We use it to announce Zulip developer community gatherings and ask for feedback on major technical or design decisions. It has several hundred subscribers, so you can use it to ask questions about features or possible bugs, but please don’t use it ask for generic help getting started as a contributor (e.g. because you want to do Google Summer of Code). The rest of this page covers how to get involved in the Zulip project in detail. • Zulip also has a blog and twitter account. • Last but not least, we use GitHub to track Zulip-related issues (and store our code, of course). Anybody with a GitHub account should be able to create Issues there pertaining to bugs or enhancement requests. We also use Pull Requests as our primary mechanism to receive code contributions. The Zulip community has a Code of Conduct.

The Zulip development environment is the recommended option for folks interested in trying out Zulip. This is documented in the developer installation guide.

Zulip in production supports Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial. Work is ongoing on adding support for additional platforms. The installation process is documented at https://zulip.org/server.html and in more detail in the documentation.

Zulip welcomes all forms of contributions! This page documents the Zulip development process. • Pull requests. Before a pull request can be merged, you need to sign the Dropbox Contributor License Agree- ment. Also, please skim our commit message style guidelines. We encourage early pull requests for work in progress. Preﬁx the title of your pull request with [WIP] and reference it when asking for community feedback. When you are ready for ﬁnal review, remove the [WIP]. • Testing. The Zulip automated tests all run automatically when you submit a pull request, but you can also run them all in your development environment following the instructions in the testing docs. You can also try out our new desktop client, which is in alpha; we’d appreciate testing and feedback. • Developer Documentation. Zulip has a growing collection of developer documentation on Read The Docs. Recommended reading for new contributors includes the directory structure and new feature tutorial. You can also improve Zulip.org. • Mailing lists and bug tracker. Zulip has a development discussion mailing list and uses GitHub issues . There are also lists for the Android and iOS apps. Feel free to send any questions or suggestions of areas where you’d love to see more documentation to the relevant list! Please report any security issues you discover to [email protected]. • App codebases. This repository is for the Zulip server and web app (including most integrations); the React Native Mobile iOS app, Android app, new Electron desktop app, and legacy QT-based desktop app are all separate repositories. • Glue code. We maintain a Hubot adapter and several integrations (Phabricator, Jenkins, Puppet, Redmine, and Trello), plus node.js API bindings, an isomorphic JavaScript library, and a full-text search PostgreSQL extension, as separate repos. • Translations. Zulip is in the process of being translated into 10+ languages, and we love contributions to our translations. See our translating documentation if you’re interested in contributing! • Code Reviews. Zulip is all about community and helping each other out. Check out #code review on chat.zulip.org to help review PRs and give comments on other people’s work. Everyone is welcome to participate, even those new to Zulip! Even just checking out the code, manually testing it, and posting on whether or not it worked is valuable.

We participated in GSoC in 2016 (with great results) and are participating in 2017 as well. For guidance, please read our GSoC instructions and ideas page and feel free to email our GSoC mailing list. Note: For GSoC 2017, we will be focused on making our React Native app better rather than developing the Java Android app and React Native app in parallel. You can review our detailed plan for further details on the motivation and logistics.

First, subscribe to the Zulip development discussion mailing list. The Zulip project uses a system of labels in our issue tracker to make it easy to find a project if you don’t have your own project idea in mind or want to get some experience with working on Zulip before embarking on a larger project you have in mind: • Integrations. Integrate Zulip with another piece of software and contribute it back to the community! Writing an integration can be a great first contribution. There’s detailed documentation on how to write integrations in the Zulip integration writing guide. • Bite Size: Smaller projects that might be a great first contribution. • Documentation: The Zulip project loves contributions of new documentation. • Help Wanted: A broader list of projects that nobody is currently working on. • Platform support: These are open issues about making it possible to install Zulip on a wider range of platforms. • Bugs: Open bugs. • Feature requests: Browsing this list can be a great way to find feature ideas to implement that other Zulip users are excited about. • 2016 roadmap milestone: The projects that are priorities for the Zulip project. These are great projects if you’re looking to make an impact. Another way to find issues in Zulip is to take advantage of our “area:” convention in separating out issues. We partition all of our issues into areas like admin, compose, emoji, hotkeys, i18n, onboarding, search, etc. You can see this here: https://github.com/zulip/zulip/labels Click on any of the “area:” labels and you will see all the tickets related to your area of interest. If you’re excited about helping with an open issue, make sure to claim the issue by commenting the following in the comment section: “@zulipbot claim”. @zulipbot will assign you to the issue and label the issue as in progress. For more details, check out @zulipbot. You’re encouraged to ask questions on how to best implement or debug your changes – the Zulip maintainers are excited to answer questions to help you stay unblocked and working efficiently. It’s great to ask questions in comments on GitHub issues and pull requests, or on chat.zulip.org. We’ll direct longer discussions to Zulip chat, but please post a summary of what you learned from the chat, or link to the conversation, in a comment on the GitHub issue. We also welcome suggestions of features that you feel would be valuable or changes that you feel would make Zulip a better open source project, and are happy to support you in adding new features or other user experience improvements to Zulip. If you have a new feature you’d like to add, we recommend you start by opening a GitHub issue about the feature idea explaining the problem that you’re hoping to solve and that you’re excited to work on it. A Zulip maintainer will usually reply within a day with feedback on the idea, notes on any important issues or concerns, and and often tips on

1.5. Google Summer of Code 5 Zulip Documentation, Release 1.4.0 how to implement or test it. Please feel free to ping the thread if you don’t hear a response from the maintainers – we try to be very responsive so this usually means we missed your message. For signiﬁcant changes to the visual design, user experience, data model, or architecture, we highly recommend posting a mockup, screenshot, or description of what you have in mind to zulip-devel@ to get broad feedback before you spend too much time on implementation details. Finally, before implementing a larger feature, we highly recommend looking at the new feature tutorial and coding style guidelines on ReadTheDocs. Feedback on how to make this development process more efﬁcient, fun, and friendly to new contributors is very welcome! Just send an email to the Zulip Developers list with your thoughts. When you feel like you have completed your work on an issue, post your PR to the #code review stream on chat.zulip.org. This is our lightweight process that gives other developers the opportunity to give you comments and suggestions on your work.

Copyright 2011-2017 Dropbox, Inc., Kandra Labs, Inc., and contributors Licensed under the Apache License, Version 2.0 (the “License”); you may not use this ﬁle except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the speciﬁc language governing permissions and limitations under the License. The software includes some works released by third parties under other free and open source licenses. Those works are redistributed under the license terms under which the works were received. For more details, see the docs/ THIRDPARTY ﬁle included with this distribution.

The core Zulip application is at https://github.com/zulip/zulip and is a web application written in Python 2.7 (soon to also support Python 3) and using the Django framework. That codebase includes server-side code and the web client, as well as Python API bindings and most of our integrations with other services and applications (see the directory structure guide). We maintain several separate repositories for integrations and other glue code: a Hubot adapter; integrations with Phabricator, Jenkins, Puppet, Redmine, and Trello; node.js API bindings; and our full-text search PostgreSQL extension. Our mobile clients are separate code repositories: Android and React Native iOS app. Our legacy desktop application (implemented in QT/WebKit) and our new, alpha cross-platform desktop app (implemented in Electron) are also separate repositories. We use Transifex to do translations. In this overview, we’ll mainly discuss the core Zulip server and web application.

Zulip is a real-time web-based chat application meant for companies and similar groups ranging in size from a small team to more than a thousand users. It features real-time notiﬁcations, message persistence and search, public group conversations (streams), invite-only streams, private one-on-one and group conversations, inline image previews, team presence/buddy lists, a rich API, Markdown message support, and numerous integrations with other services. The maintainer team aims to support users who connect to Zulip using dedicated iOS, Android, Linux, Windows, and macOS clients, as well as people using modern web browsers or dedicated Zulip API clients. A server can host multiple Zulip realms (organizations) at the same domain, each of which is a private chamber with its own users, streams, customizations, and so on. This means that one person might be a user of multiple Zulip realms. The administrators of a realm can choose whether to allow anyone to register an account and join, or only allow people who have been invited, or restrict registrations to members of particular groups (using email domain

7 Zulip Documentation, Release 1.4.0 names or corporate single-sign-on login for verification). For more on security considerations, see the security model section. The default Zulip home screen is like a chronologically ordered inbox; it displays messages, starting at the oldest message that the user hasn’t viewed yet (for more on that logic, see the guide to the pointer and unread counts). The home screen displays the most recent messages in all the streams a user has joined (except for the streams they’ve muted), as well as private messages from other users, in strict chronological order. A user can narrow to view only the messages in a single stream, and can further narrow to focus on a topic (thread) within that stream. Each narrow has its own URL. The user can quickly see what conversation they’re in – the stream and topic, or the names of the user(s) they’re private messaging with – using the recipient bar displayed atop each conversation. Zulip’s philosophy is to provide sensible defaults but give the user fine-grained control over their incoming information flow; a user can mute topics and streams, and can make fine-grained choices to reduce real-time notifications they find irrelevant.

Zulip is primarily implemented in the Django Python web framework. We also make use of Tornado for the real-time push system. Django is the main web application server; Tornado runs the server-to-client real-time push system. The app servers are configured by the Supervisor configuration (which explains how to start the server processes; see “Supervisor” below) and the nginx configuration (which explains which HTTP requests get sent to which app server).

Tornado is an asynchronous server and is meant speciﬁcally to hold open tens of thousands of long-lived (long-polling or websocket) connections – that is to say, routes that maintain a persistent connection from every running client. For this reason, it’s responsible for event (message) delivery, but not much else. We try to avoid any blocking calls in Tornado because we don’t want to delay delivery to thousands of other connections (as this would make Zulip very much not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since those blocking requests carry a very high performance penalty for a single-threaded, asynchronous server. The parts that are activated relatively rarely (e.g. when people type or click on something) are processed by the Django application server. One exception to this is that Zulip uses websockets through Tornado to minimize latency on the code path for sending messages. There is detailed documentation on the real-time push and event queue system; most of the code is in zerver/ tornado.

nginx is the front-end web server to all Zulip traffic; it serves static assets and proxies to Django and Tornado. It handles HTTP requests according to the rules laid down in the many config files found in zulip/puppet/zulip/ files/nginx/. zulip/puppet/zulip/files/nginx/zulip-include-frontend/app is the most important of these files. It explains what happens when requests come in from outside. • In production, all requests to URLs beginning with /static/ are served from the corresponding files in / home/zulip/prod-static/, and the production build process (tools/build-release-tarball) compiles, minifies, and installs the static assets into the prod-static/ tree form. In development, files are served directly from /static/ in the git repository. • Requests to /json/events, /api/v1/events, and /sockjs are sent to the Tornado server. These are requests to the real-time push system, because the user’s web browser sets up a long-lived TCP connection with Tornado to serve as a channel for push notifications. nginx gets the hostname for the Tornado server via puppet/zulip/files/nginx/zulip-include-frontend/upstreams. • Requests to all other paths are sent to the Django app via the UNIX socket unix:/home/ zulip/deployments/uwsgi-socket (defined in puppet/zulip/files/nginx/ zulip-include-frontend/upstreams). We use zproject/wsgi.py to implement uWSGI here (see django.core.wsgi).

We use supervisord to start server processes, restart them automatically if they crash, and direct logging. The conﬁg ﬁle is zulip/puppet/zulip/templates/supervisor/zulip.conf.template.erb. This is where Tornado and Django are set up, as well as a number of background processes that process event queues. We use event queues for the kinds of tasks that are best run in the background because they are expensive (in terms of performance) and don’t have to be synchronous — e.g., sending emails or updating analytics. Also see the queuing guide.

memcached is used to cache database model objects. zerver/lib/cache.py and zerver/lib/ cache_helpers.py manage putting things into memcached, and invalidating the cache when values change. The memcached conﬁguration is in puppet/zulip/files/memcached.conf.

Redis is used for a few very short-term data stores, such as in the basis of zerver/lib/rate_limiter.py, a per-user rate limiting scheme example), and the email-to-Zulip integration. Redis is conﬁgured in zulip/puppet/zulip/files/redis and it’s a pretty standard conﬁguration except for the last line, which turns off persistence:

memcached was used ﬁrst and then we added Redis speciﬁcally to implement rate limiting. We’re discussing switching everything over to Redis.

RabbitMQ is a queueing system. Its config files live in zulip/puppet/zulip/files/rabbitmq. Initial configuration happens in zulip/scripts/setup/configure-rabbitmq. We use RabbitMQ for queuing expensive work (e.g. sending emails triggered by a message, push notifications, some analytics, etc.) that require reliable delivery but which we don’t want to do on the main thread. It’s also used for communication between the application server and the Tornado push system. Two simple wrappers around pika (the Python RabbitMQ client) are in zulip/zerver/lib/queue.py. There’s an asynchronous client for use in Tornado and a more general client for use elsewhere. Most of the processes started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handle them; they are defined in zerver/worker/queue_processors.py. Also see the queuing guide.

PostgreSQL (also known as Postgres) is the database that stores all persistent data, that is, data that’s expected to live beyond a user’s current session. In production, Postgres is installed with a default configuration. The directory that would contain configuration files (puppet/zulip/files/postgresql) has only a utility script and a custom list of stopwords used by a Post- gresql extension. In a development environment, configuration of that postgresql extension is handled by tools/ postgres-init-dev-db (invoked by tools/provision). That file also manages setting up the development postgresql user. tools/provision also invokes tools/do-destroy-rebuild-database to create the actual database with its schema.

Nagios is an optional component used for notiﬁcations to the system administrator, e.g., in case of outages. zulip/puppet/zulip/manifests/nagios.pp installs Nagios plugins from puppet/zulip/files/ nagios_plugins/. This component is intended to install Nagios plugins intended to be run on a Nagios server; most of the Zulip Nagios plugins are intended to be run on the Zulip servers themselves, and are included with the relevant component of

the Zulip server (e.g. puppet/zulip/manifests/postgres_common.pp installs a few under /usr/lib/ nagios/plugins/zulip_postgres_common).

This section gives names for some of the elements in the Zulip UI used in Zulip development conversations. Contri- butions to extend this list are welcome! • chevron: A small downward-facing arrow next to a message’s timestamp, offering contextual options, e.g., “Reply”, “Mute [this topic]”, or “Link to this conversation”. To avoid visual clutter, the chevron only appears in the web UI upon hover. • huddle: What the codebase calls a “group private message”. • message editing: If the realm admin allows it, then after a user posts a message, the user has a few minutes to click “Edit” and change the content of their message. If they do, Zulip adds a marker such as “(EDITED)” at the top of the message, visible to anyone who can see the message. • realm: What the codebase calls an “organization” in the UI. • recipient bar: A visual indication of the context of a message or group of messages, displaying the stream and topic or private message recipient list, at the top of a group of messages. A typical 1-line message to a new recipient shows to the user as three lines of content: ﬁrst the recipient bar, second the sender’s name and avatar alongside the timestamp (and, on hover, the star and the chevron), and third the message content. The recipient bar is or contains hyperlinks to help the user narrow. • star: Zulip allows a user to mark any message they can see, public or private, as “starred”. A user can easily access messages they’ve starred through the “Starred messages” link in the menu near “Home”, or use “is:starred” as a narrow or a search constraint. Whether a user has or has not starred a particular message is private; other users and realm admins don’t know whether a message has been starred, or by whom. • subject: What the codebase calls a “topic” in many places. • bankruptcy: When a user has been off Zulip for several days and has hundreds of unread messages, they are prompted for whether they want to mark all their unread messages as read. This is called “declaring bankruptcy” (in reference to the concept in ﬁnance).

This page documents the Zulip directory structure, where to find things, and how to decide where to put a file. You may also find the new application feature tutorial helpful for understanding the flow through these files.

Zulip uses the Django web framework, so a lot of these paths will be familiar to Django developers. • zproject/urls.py Main Django routes file. Defines which URLs are handled by which view functions or templates. • zerver/models.py Main Django models file. Defines Zulip’s database tables. • zerver/lib/actions.py Most code doing writes to user-facing database tables. • zerver/views/*.py Most Django views. • zerver/webhooks/ Webhook views and tests for Zulip webhook integrations. • zerver/tornado/views.py Tornado views. • zerver/worker/queue_processors.py Queue workers. • zerver/lib/*.py Most library code. • zerver/lib/bugdown/ Backend Markdown processor. • zproject/backends.py Authentication backends.

• templates/zerver/ For Jinja2 templates for the backend (for zerver app). • static/templates/ Handlebars templates for the frontend.

• static/js/ Zulip’s own JavaScript. • static/styles/ Zulip’s own CSS. • static/images/ Zulip’s images. • static/third/ Third-party JavaScript and CSS that has been vendored. • node_modules/ Third-party JavaScript installed via npm. • static/assets/ For assets not to be served to the web (e.g. the system to generate our favicons).

• zerver/tests/ Backend tests. • frontend_tests/node_tests/ Node Frontend unit tests. • frontend_tests/casper_tests/ Casper frontend tests. • tools/test-* Developer-facing test runner scripts.

These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access). • zerver/management/commands/ Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly).

• scripts/ Scripts that production deployments might run manually (e.g., restart-server). • scripts/lib/ Scripts that are needed on production deployments but humans should never run directly. • scripts/setup/ Scripts that production deployments will only run once, during installation. • tools/ Scripts used only in a Zulip development environment. These are not included in production release tarballs for Zulip, so that we can include scripts here one wouldn’t want someone to run in production accidentally (e.g. things that delete the Zulip database without prompting).

• tools/setup/ Subdirectory of tools/ for things only used during the development environment setup process. • tools/travis/ Subdirectory of tools/ for things only used to setup and run our tests in Travis CI. Actual test suites should go in tools/.

• api/ Zulip’s Python API bindings (released separately). • api/examples/ API examples. • api/integrations/ Integrations distributed as part of the Zulip API bundle. • api/bots/ Bots distributed as part of the Zulip API bundle. • api/bots_api/ Framework for running and testing bots in api/bots/.

This is used to deploy essentially all conﬁguration in production. • puppet/zulip/ For conﬁguration for production deployments. • puppet/zulip/manifests/voyager.pp Main manifest for Zulip standalone deployments.

• confirmation Email conﬁrmation system. • analytics Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites). • corporate The old Zulip.com website. Not included in production distribution. • zilencer Primarily used to hold management commands that aren’t used in production. Not included in production distribution.

• zproject/jinja2/__init__.py Jinja2 environment. • zproject/jinja2/backends.py Jinja2 backend. • zproject/jinja2/compressors.py Jinja2 compatible functions of Django-Pipeline.

• locale/ Backend (Django) translations data ﬁles. • static/locale/ Frontend translations data ﬁles.

You can consult the repository’s .gitattributes ﬁle to see exactly which components are excluded from production releases (release tarballs are generated using tools/build-release-tarball).

Zulip has received a great deal of interest and attention since it was released as free and open source software by Dropbox. That attention has come with a lot of active development work from members of the Zulip community. From when Zulip was released as open source in late September 2015 through today (early November, 2016), more than 150 people have contributed over 1000 pull requests to the various Zulip repositories, the vast majority of which were submitted by Zulip’s users around the world (as opposed to the small core team that reviews and merges the pull requests). In any project, there can be a lot of value in periodically putting together a roadmap detailing the major areas where the project is hoping to improve. This can be especially important in an open source project like Zulip, where development is distributed across many people around the world. This roadmap is intended to organize a list of the most important improvements that should be made to Zulip in the relatively near future. Our aim is to complete most of these improvements by February 2017 and then prepare a new roadmap then. This document is not meant to constrain in any way what contributions to Zulip will be accepted; instead, it will be used by the Zulip core team to prioritize our efforts, measure progress on improving the Zulip product and hold ourselves accountable for making Zulip improve rapidly. This roadmap is the best place for contributors to look for substantial projects that will deﬁnitely be of value to the community (if you’re looking for a starter project, see the guide to getting involved with Zulip). We periodically update this roadmap by adding strikethrough to issues that have been resolved, but the linked GitHub issues are the most up-to-date source for that information. Update: As of May 2017, we are approaching the point where we need to update the roadmap due to much of it being completed. Without further ado, below is the current Zulip roadmap.

There are 2 huge projects that Zulip is working on right now that are too big to have a coherent GitHub issue:

• We are working with a world-class designer on a major visual redesign of the Zulip webapp. Already complete is completely redesining the streams and settings UIs, logged-out pages, and various other major components. • We are writing a new React Native iOS app for Zulip to replace the old iOS app. The new app is progressing rapidly, but is not yet feature complete. We expect it to be in the app store in May 2017.

• Provide shorter UI/Keyboard sequence to edit the last message • Better drafts management • Make clicking on desktop notiﬁcations renarrow properly • Add pretty bubbles for recipients in the compose box • Make right sidebar buddy list UI scale well to large teams • Display stream descriptions more prominently • Add support for managing uploaded ﬁles

• Add support for showing “user is typing” notiﬁcations, at least for private messages • Support lightweight emoji “reactions” • Open graph previews of generic websites • Add a “join Zulip chat” badge for projects that use Zulip to document that nicely

The overall goal is to eliminate the few known issues where Zulip does not provide a seamless real-time sync experience. • Notiﬁcation bot advertisements for new streams don’t handle stream renames • Avatar/name changes don’t propagate to already-sent messages • Advance the pointer / where we load the user to based on unread counts in home view • Fix the known bug where messages could be incorrectly marked as read

This category focuses on issues users experience when installing a new Zulip server, setting up a new Zulip realm, or starting to use Zulip. • Move Zulip’s prompt for permission to display notiﬁcations to be manually triggered • Add a mechanism for deleting early test messages (e.g., administrators can hard-delete messages) • Allow customizing emails when inviting new users

• Document or better script solution to rabbitmq startup issues • Merge a supported way to use Zulip in Docker in production implementation.

• Make list of allowed domains web-configurable • Statistics display for realm and server administrators • Keep track of which users added which realm emoji • Add setting to enable any user to add new realm emoji • Make realm filters web-configurable • Improve administrative controls for managing streams • Enhance the LDAP integration and make it web-configurable • Add a SAML integration for Zulip

Scalability and performance are not currently major problems for Zulip; it already scales well to thousands of users and is signiﬁcantly faster than proprietary alternatives. So, this is not a major focus area for the project. • Make the Zulip Tornado service support horizontal scaling • Make presence system scale well to 10000 users in a realm. • Support running queue workers multithreaded in production to decrease minimum memory footprint • Improve @-mentioning syntax based on stronger unique identiﬁers

• Add support for Zulip running purely on Python 3 • Automatic thumbnailing of uploaded images’ previews to save bandwidth • Upgrade Zulip to use Django 1.10. The patches needed to run Zulip were merged into mainline Django in Django 1.10, so this will mean we don’t need to use a fork of Django anymore. • Upgrade and remove from codebase all unnecessarily vendored JS libraries • Add support for changing users’ email addresses • Migrate from jslint to eslint • Replace the slow closure-compiler based static asset toolchain • Use a modern JavaScript bundler like webpack • Add support for building frontend features in something like React

While the Zulip server has a great codebase compared to most projects of its size, it takes work to keep it that way. • Migrate most web routes to REST API • Split Tornado subsystem into a separate Django app • Refactor zulip.css to be broken into components

• Add support for 2-factor authentication on all platforms • Add support for stronger security controls for uploaded ﬁles (The LOCAL_UPLOADS_DIR ﬁle uploads backend only supports world-readable uploads) • Fix requirement to set a password when creating account via Google • Add a retention policy feature that automatically deletes old messages • Add UI for viewing and cancelling open Zulip invitations

• Extend Zulip’s automated test coverage to include all API endpoints • Build automated tests for the client API bindings • Add automated tests for the production upgrade process

• Add an in-app mechanism for updating users about new Zulip features • Signiﬁcantly expand documentation of the Zulip API and integrating with Zulip. • Write a visual design / frontend style guide for Zulip • Update all screenshots to show the current Zulip UI

• Expand library of documentation on Zulip’s feature set. Currently most documentation is for either developers or system administrators.

Integrations are essential to Zulip. While we currently have a reasonably good framework for writing new webhook integrations for getting notiﬁcations into Zulip, it’d be great to streamline that process and make bots that receive messages just as easy to build.

• Add an outgoing webhook integration system • Make setting up a new integration a smooth ﬂow • Default new incoming webhooks to permissions-limited incoming webhook bots • Change how Zulip displays bot names to distinguish them from human users

• Add support for narrowing to @-mentions • Support having multiple Zulip realms open simultaneously iOS app

For the new React Native iOS app, the major goal for it is to be released into the app store. Since it is moving quickly, we’re tracking its roadmap via GitHub milestones.

To support a great mobile experiences, we need to make some improvements in the Zulip server. • Push notiﬁcations bouncer service for GCM and APNS • A slick process for doing mobile login without typing your password on your phone • @here mention support (that doesn’t spam people not currently online, i.e. no email/push notiﬁcations) • Fix sending messages from mobile web

The new cross-platform desktop app is implemented in Electron, and primarily needs work on installer tooling to ﬁnish replacing the old app. • Finish releasing the Electron app to replace the old desktop app • Support having multiple Zulip realms open simultaneously

These don’t get GitHub issues since they’re not technical projects, but they are important goals for the project. • Expand the number of core developers able to do code reviews • Have a successful season with Zulip’s Outreachy participants • Have a successful season with Google Code In.

Highlights: • A complete visual redesign of the logged-out pages, including login, registration, integrations, etc. • New visual designs for numerous UI elements, including the emoji picker, user profile popovers, sidebars, compose, and many more. • A complete redesign of the Zulip settings interfaces to look a lot nicer and be easier to navigate. • Organization admins can now configure the login and registration pages to show visitors a nice organization profile with custom text and images, written in Markdown. • Massively improved performance for presence and settings pages, especially for very large organizations (1000+ users). • A dozen useful new keyboard shortcuts, from editing messages to emoji reactions to drafts and managing streams. • Typing notifications for private message threads. • Users can now change their own email address. • New saved-drafts feature. • The server can now run on a machine with as little as 2GB of RAM. • The new Electron desktop app and new React Native mobile app for iOS are now the recommended Zulip apps. • Mobile web now works much better, especially on iOS.

• Support for sending mobile push notifications via a new forwarding service • Complete translations for Spanish, German, and Czech (and expanded partial translations for Japanese, Chinese, French, Hungarian, Polish, Dutch, Russian, Bulgarian, Portuguese, Serbian, Malayalam, Korean, and Italian). Full feature Changelog: • Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and WordPress webhook integrations. • Added LaTeX support to the markdown processor. • Added support for filtering branches to all Git integrations. • Added read-only access to organization-level settings for all users. • Added UI for managing muted topics and uploaded files. • Added UI for displaying message edit history. • Added support for various features needed by new mobile app. • Added deep links for settings/subscriptions interfaces. • Added an animation when messages are edited. • Added support for registration with GitHub auth (not just login). • Added tracking of uploaded file quotas. • Added option to display emoji as their alt codes. • Added new audit log table, to eventually support an auditing UI. • Added several new permissions-related organization settings. • Added new endpoint for fetching presence data, useful in employee directories. • Added typeahead for language for syntax highlighting in code blocks. • Added support for basic markdown in stream descriptions. • Added email notifications on new Zulip logins. • Added security hardening before serving uploaded files. • Added new PRIVACY_POLICY setting to provide a Markdown privacy policy. • Added an icon to distinguish bot users as message senders. • Added a command-line Slack importer tool using the API. • Added new announcement notifications on stream creation. • Added support for some newer unicode emoji code points. • Added support for users deleting realm emoji they themselves uploaded. • Added support for organization administrators deleting messages. • Extended data available to mobile apps to cover the entire API. • Redesigned bots UI. Now can change owners and reactivate bots. • Redesigned the visuals of code blocks to be prettier. • Changed right sidebar presence UI to only show recently active users in large organizations. This has a huge performance benefit. • Changed color for private messages to look better. • Converted realm emoji to be uploaded, not links, for better robustness.

• Switched the default password hasher for new passwords to Argon2. • Increased the paragraph spacing, making multi-paragraph. • Improved formatting of all Git integrations. • Improved the UI of the /stats analytics pages. • Improved search typeahead to support group private messages. • Improved logic for when the compose box should open/close. • Improved lightbox to support scrolling through images. • Improved markdown support for bulleted lists. • Improved copy-to-clipboard support in various places. • Improved subject lines of missed message emails. • Improved handling of users trying to login with Oauth without an account. • Improved UI of off-the-Internet errors to not be hidden in narrow windows. • Improved rate-limiting errors to be more easily machine-readable. • Parallelized the backend test suite; now runs 1600 tests in <30s. • Fixed numerous bugs and performance issues with stream management. • Fixed an issue with the fake emails assigned to bot users. • Fixed a major performance issue in stream creation. • Fixed numerous minor accessibility issues. • Fixed a subtle interaction between click-to-reply and copy-paste. • Fixed various formatting issues with /me messages. • Fixed numerous real-time sync issues involving users changing their name, avatar, or email address and streams being renamed. • Fixed numerous performance issues across the project. • Fixed various left sidebar ordering and live-updated bugs. • Fixed numerous bugs with the message editing widget. • Fixed missing logging / rate limiting on browser endpoints. • Fixed regressions in Zulip’s browser state preservation on reload logic. • Fixed support for unicode characters in the email mirror system. • Fixed load spikes when email mirror is receiving a lot of traffic. • Fixed the ugly grey flicker when scrolling fast on Macs. • Fixed previews of GitHub image URLs. • Fixed narrowing via clicking on desktop notifications. • Fixed Subscribed/Unsubscribed bookends appearing incorrectly. • Eliminated the idea of a realm having a canonical domain; now there’s simply the list of allowed domains for new users. • Migrated avatars to a user-id-based storage setup (not email-based). • Trailing whitespace is now stripped in code blocks, avoiding unnecessary scrollbars.

• Most API payloads now refer to users primarily by user ID, with email available for backwards-compatibility. In the future, we may remove email support. • Cleaned up Zulip’s supervisord conﬁguration. A side effect is the names of the log ﬁles have changed for all the queue workers. • Refactored various endpoints to use a single code path for security hardening. • Removed support for the MANDRILL_CLIENT setting. It hadn’t been used in years. • Changed NOREPLY_EMAIL_ADDRESS setting to Name format. • Disabled the web tutorial on mobile. • Backend test coverage is now 93%, with 100% in views code.

• CVE-2017-0896: Restricting inviting new users to admins was broken. • CVE-2015-8861: Insecure old version of handlebars templating engine.

• Fix exception trying to copy node_modules during upgrade process. • Improved styling of /stats page to remove useless login/register links.

Highlights: • Completely redesigned the Manage streams interface. • Added support for emoji reactions to messages. • Added a lightbox for viewing images and videos. • Added an extensive user documentation site at /help/. • Added admin setting to auto-linkify certain strings (useful for issue numbers and Git commit IDs). • Upgraded how the main application runs from FastCGI on Django 1.8 to uwsgi and Django 1.10. • Added preliminary support for open graph previews of links (the setting, INLINE_URL_EMBED_PREVIEW, is disabled by default in this release). Full feature Changelog: • Added an emoji picker/browser to the compose box. • Added markdown preview support to the compose box. • Added a new analytics system to track interesting usage statistics. • Added a /stats page with graphs of the analytics data. • Added display of subscriber counts in Manage streams. • Added support for ﬁltering streams in Manage streams.

• Added support for setting a stream description on creation. • Added support for copying subscribers from existing streams on creation. • Added several new search/filtering UI elements. • Added UI for deactivating your own Zulip account. • Added support for viewing the raw markdown content of a message. • Added support for deploying Zulip with subdomains for each realm. This entailed numerous changes to ensure a consistent experience. • Added support for (optionally) using PGRoonga to support full-text search in all languages (not just English). • Added AppFollow, GitLab, Google Calendar, GoSquared, HelloSign, Heroku, Librato, MailChimp, Mention, Papertrail, Sentry, Solano Labs, Stripe and Zapier integrations. • Added a webhook integration for GitHub, replacing the deprecated github-services hook. • Normalized the message formatting for all the Zulip Git integrations. • Added support for VMWare Fusion Vagrant provider for faster OSX development. • Added a shields.io style badge for joining a Zulip server. • Added admin setting for which email domains can join a realm. • Added admin setting for controlling who can create streams. • Added admin setting to limit stream creation to older users. • Added a notification when you muted a topic. • Added a new hotkey for muting/unmuting topics. • Added support for testing websockets to the Nagios plugins. • Added a configuration option to disable websockets. • Added support for removing one’s own Zulip account. • Added support for realm admins which auth backends are supported. • Added new organization type concept. This will be used to control whether Zulip is optimized around protecting user privacy vs. administrative control. • Added #streamName syntax for linking to a stream. • Added support for viewing markdown source of messages. • Added setting to always send push notifications. • Added setting to hide private message content in desktop notifications. • Added buttons to download .zuliprc files. • Added italics and strikethrough support in markdown implementation. • Added errors for common installations mistakes (e.g. too little RAM). • Added a new /authors page showing the contributors to the current Zulip version. • Upgraded all Python dependencies to modern versions, including Django 1.10 (all of Zulip’s patches have been merged into mainline). • Increased backend test coverage of Python codebase to 90%. • Increased mypy static type coverage of Python code to 100%.

• Added several new linters (eslint, pep8) and cleaned the codebase. • Optimized the speed of the Zulip upgrade process, especially with Git. • Have peer_add events send user_id, not email. • Fixed problems with rabbitmq when installing Zulip. • Fixed JavaScript not being gzip-compressed properly. • Fixed a major performance bug in the Tornado service. • Fixed a frontend performance bug creating streams in very large realms. • Fixed numerous bugs where strings were not properly tagged for translation. • Fixed several real-time sync bugs, and removed several AJAX calls. Zulip should be more performant than ever before. • Fixed Zulip Tornado service not working with http_proxy set in environment. • Fixed text overflow in stream subscriptions. • Fixed CSS issues with message topic editing. • Fixed several transactionality bugs (e.g. in Huddle creation). • Fixed missed-message email configuration error handling. • Fixed annoying @-mentions in Jira integration. • Fixed various mismatches between frontend and backend markdown implementations. • Fixed various popover-related UI bugs. • Fixed duplicate notifications with multiple open Zulip tabs. • Fixed support for emailing the server administrator about backend exceptions. • Cleaned up the “edit message” form. • Eliminated most of the legacy API endpoints. • Improved typeahead and autocomplete across the application. Highlights include much better handling of many users with similar names. • Improved the color scheme for code blocks. • Improved the message editing UI in several ways. • Improved how dates are displayed in the UI. • Improved default settings for zxcvbn password strength checker. • Upgraded jQuery to the latest 1.12 release. • Made numerous improvements to the development tooling. • Made extensive improvements to code organization. • Restyled all the registration pages to look nicer and be responsive. • Extensively refactored views to use common functions for fetching stream and message objects. • Suppressed @-all mentions being treated as mentions on muted streams. • Documented preliminary design for interactive bot system.

• Upgraded Django to version 1.8.15 (with the Zulip patches applied), ﬁxing a CSRF vulnerability in Django (see https://www.djangoproject.com/weblog/2016/sep/26/security-releases/), and a number of other Django bugs from past Django stable releases that largely affects parts of Django that are not used by Zulip. • Fixed buggy logrotate conﬁguration.

• Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0. • Fixed local ﬁle uploads integration being broken for new 1.4.0 installations.

• Migrated Zulip’s python dependencies to be installed via a virtualenv, instead of the via apt. This is a major change to how Zulip is installed that we expect will simplify upgrades in the future. • Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncached network transfer for loading Zulip. • Added support for using Ubuntu Xenial in production. • Added a powerful and complete realm import/export tool. • Added nice UI for selecting a default language to display settings. • Added UI for searching streams in left sidebar with hotkeys. • Added Semaphore, Bitbucket, and HelloWorld (example) integrations. • Added new webhook-based integration for Trello. • Added management command for creating realms through web UI. • Added management command to send password reset emails. • Added endpoint for mobile apps to query available auth backends. • Added LetsEncrypt documentation for getting SSL certiﬁcates. • Added nice rendering of unicode emoji. • Added support for pinning streams to the top of the left sidebar. • Added search box for ﬁltering user list when creating a new stream. • Added realm setting to disable message editing. • Added realm setting to time-limit message editing. Default is 10m. • Added realm setting for default language.

• Added year to timestamps in message interstitials for old messages. • Added GitHub authentication (and integrated python-social-auth, so it’s easy to add additional social authentication methods). • Added TERMS_OF_SERVICE setting using markdown formatting to configure the terms of service for a Zulip server. • Added numerous hooks to puppet modules to enable more configurations. • Moved several useful puppet components into the main puppet manifests (setting a redis password, etc.). • Added automatic configuration of postgres/memcached settings based on the server’s available RAM. • Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo. • Added preliminary support for Python 3. All of Zulip’s test suites now pass using Python 3.4. • Added support for Name format when inviting users. • Added numerous special-purpose settings options. • Added a hex input field in color picker. • Documented new Electron beta app and mobile apps in the /apps/ page. • Enabled Android Google authentication support. • Enhanced logic for tracking origin of user uploads. • Improved error messages for various empty narrows. • Improved missed message emails to better support directly replying. • Increased backend test coverage of Python code to 85.5%. • Increased mypy static type coverage of Python code to 95%. Also fixed many string annotations to properly handle unicode. • Fixed major i18n-related frontend performance regression on /#subscriptions page. Saves several seconds of load time with 1k streams. • Fixed Jinja2 migration bug when trying to register an email that already has an account. • Fixed narrowing to a stream from other pages. • Fixed various frontend strings that weren’t marked for translation. • Fixed several bugs around editing status (/me) messages. • Fixed queue workers not restarting after changes in development. • Fixed Casper tests hanging while development server is running. • Fixed browser autocomplete issue when adding new stream members. • Fixed broken create_stream and rename_stream management commands. • Fixed zulip-puppet-apply exit code when puppet throws errors. • Fixed EPMD restart being attempted on every puppet apply. • Fixed message cache filling; should improve perf after server restart. • Fixed caching race condition when changing user objects. • Fixed buggy puppet configuration for supervisord restarts. • Fixed some error handling race conditions when editing messages.

• Fixed fastcgi_params to protect against the httpoxy attack. • Fixed bug preventing users with mit.edu emails from registering accounts. • Fixed incorrect settings docs for the email mirror. • Fixed APNS push notification support (had been broken by Apple changing the APNS API). • Fixed some logic bugs in how attachments are tracked. • Fixed unnecessarily resource-intensive rabbitmq cron checks. • Fixed old deployment directories leaking indefinitely. • Fixed need to manually add localhost in ALLOWED_HOSTS. • Fixed display positioning for the color picker on subscriptions page. • Fixed escaping of Zulip extensions to markdown. • Fixed requiring a reload to see newly uploaded avatars. • Fixed @all warning firing even for @all. • Restyled password reset form to look nice. • Improved formatting in reset password links. • Improved alert words UI to match style of other settings. • Improved error experience when sending to nonexistent users. • Portions of integrations documentation are now automatically generated. • Restructured the URLs files to be more readable. • Upgraded almost all Python dependencies to current versions. • Substantially expanded and reorganized developer documentation. • Reorganized production documentation and moved to ReadTheDocs. • Reorganized .gitignore type files to be written under var/ • Refactored substantial portions of templates to support subdomains. • Renamed local_settings.py symlink to prod_settings.py for clarity. • Renamed email-mirror management command to email_mirror. • Changed HTTP verb for create_user_backend to PUT. • Eliminated all remaining settings hardcoded for zulip.com. • Eliminated essentially all remaining hardcoding of mit.edu. • Optimized the performance of all the test suites. • Optimized Django memcached configuration. • Removed old prototype data export tool. • Disabled insecure RC4 cipher in nginx configuration. • Enabled shared SSL session cache in nginx configuration. • Updated header for Zulip static assets to reflect Zulip being open source.

• Added nearly complete internationalization of the Zulip UI. • Added warning when using @all/@everyone. • Added button offering to subscribe at bottom of narrows to streams the user is not subscribed to. • Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT, Transifex, and Updown.io. • Added menu option to mark all messages in a stream or topic as read. • Added new Attachment model to keep track of uploaded files. • Added caching of virtualenvs in development. • Added mypy static type annotations to about 85% of the Zulip Python codebase. • Added automated test of backend templates to test for regressions. • Added lots of detailed documentation on the Zulip development environment. • Added setting allowing only administrators to create new streams. • Added button to exit the Zulip tutorial early. • Added web UI for configuring default streams. • Added new OPEN_REALM_CREATION setting (default off), providing a UI for creating additional realms on a Zulip server. • Fixed email_gateway_password secret not working properly. • Fixed missing helper scripts for RabbitMQ Nagios plugins. • Fixed skipping forward to latest messages (“More messages below” button). • Fixed netcat issue causing Zulip installation to hang on Scaleway machines. • Fixed rendering of /me status messages after message editing. • Fixed case sensitivity of right sidebar fading when compose is open. • Fixed error messages when composing to invalid PM recipients. • Fixed LDAP auth backend not working with Zulip mobile apps. • Fixed erroneous WWW-Authenticate headers with expired sessions. • Changed “coworkers” to “users” in the Zulip UI. • Changed add_default_stream REST API to correctly use PUT rather than PATCH. • Updated the Zulip emoji set (the Android Emoji) to a modern version. • Made numerous small improvements to the Zulip development experience. • Migrated backend templates to the faster Jinja2 templating system. • Migrated development environment setup scripts to tools/setup/. • Expanded test coverage for several areas of the product. • Simplified the API for writing new webhook integrations. • Removed most of the remaining JavaScript global variables.

• CVE-2016-4426: Bot API keys were accessible to other users in the same realm. • CVE-2016-4427: Deactivated users could access messages if SSO was enabled. • Fixed a RabbitMQ conﬁguration bug that resulted in reordered messages. • Added expansive test suite for authentication backends and decorators. • Added an option to logout_all_users to delete only sessions for deactivated users.

• Moved email digest support into the default Zulip production configuration. • Added options for configuring Postgres, RabbitMQ, Redis, and memcached in settings.py. • Added documentation on using Hubot to integrate with useful services not yet integrated with Zulip directly (e.g. Google Hangouts). • Added new management command to test sending email from Zulip. • Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations. • Added Nagios plugins to the main distribution. • Added ability for realm administrators to manage custom emoji. • Added guide to writing new integrations. • Enabled camo image proxy to fix mixed-content warnings for http images. • Refactored the Zulip puppet modules to be more modular. • Refactored the Tornado event system, fixing old memory leaks. • Removed many old-style /json API endpoints • Implemented running queue processors multithreaded in development, decreasing RAM requirements for a Zulip development environment from ~1GB to ~300MB. • Fixed rerendering the complete buddy list whenever a user came back from idle, which was a significant performance issue in larger realms. • Fixed the disabling of desktop notifications from 1.3.7 for new users. • Fixed the (admin) create_user API enforcing restricted_to_domain, even if that setting was disabled for the realm. • Fixed bugs changing certain settings in administration pages. • Fixed collapsing messages in narrowed views. • Fixed 500 errors when uploading a non-image file as an avatar. • Fixed Jira integration incorrectly not @-mentioning assignee.

• Added new integration for Travis CI. • Added settings option to control maximum file upload size. • Added support for running Zulip development environment in Docker. • Added easy configuration support for a remote postgres database. • Added extensive documentation on scalability, backups, and security. • Recent private message threads are now displayed expanded similar to the pre-existing recent topics feature. • Made it possible to set LDAP and EMAIL_HOST passwords in /etc/zulip/secrets.conf. • Improved the styling for the Administration page and added tabs. • Substantially improved loading performance on slow networks by enabling GZIP compression on more assets. • Changed the page title in narrowed views to include the current narrow. • Fixed several backend performance issues affecting very large realms. • Fixed bugs where draft compose content might be lost when reloading site. • Fixed support for disabling the “zulip” notifications stream. • Fixed missing step in postfix_localmail installation instructions. • Fixed several bugs/inconveniences in the production upgrade process. • Fixed realm restrictions for servers with a unique, open realm. • Substantially cleaned up console logging from run-dev.py.

• Added options to the Python api for working with untrusted server certiﬁcates. • Added a lot of documentation on the development environment and testing. • Added partial support for translating the Zulip UI. • Migrated installing Node dependencies to use npm. • Fixed LDAP integration breaking autocomplete of @-mentions. • Fixed admin panel reactivation/deactivation of bots. • Fixed inaccurate documentation for downloading the desktop apps. • Fixed various minor bugs in production installation process. • Fixed security issue where recent history on private streams might be visible to new users (to the Zulip team) who were invited with that private stream as one of their initial streams (https://github.com/zulip/zulip/issues/230). • Major preliminary progress towards supporting Python 3.

• Turn off desktop and audible notiﬁcations for streams by default. • Added support for the LDAP authentication integration creating new users. • Added new endpoint to support Google auth on mobile. • Fixed desktop notiﬁcations in modern Firefox. • Fixed several installation issues for both production and development environments. • Improved documentation for outgoing SMTP and the email mirror integration.

Note that if you just want to play around with Zulip and see what it looks like, it is easier to install it in a development environment following these instructions, since then you don’t need to worry about setting up SSL certiﬁcates and an authentication mechanism. Or, you can check out the Zulip development community server.

• CPU and Memory: For installations with 100+ users you’ll need a minimum of 2 CPUs and 4GB RAM. For installations with fewer users, 1 CPU and 2GB RAM is sufﬁcient. We strongly recommend against installing with less than 2GB of RAM, as you will likely experience out of memory issues installing dependencies. We recommend against using highly CPU-limited servers like the AWS t2 style instances for organizations with a hundreds of users (active or no). See our documentation on scalability for advice on hardware requirements for larger organizations. • Disk space: You’ll need at least 10GB of free disk space. If you intend to store uploaded ﬁles locally rather than on S3 you will likely need more.

• Incoming HTTPS access (usually port 443, though this is conﬁgurable) from the networks where your users are (usually, the public Internet). If you also open port 80, Zulip will redirect users to HTTPS rather than not working when users type e.g. http://zulip.example.com in their browser. If you are using Zulip’s incoming email integration you may also need incoming port 25 open. • Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so that Zulip can properly manage inline image previews. You’ll also need outgoing SMTP access to your SMTP server (the standard port for this is 587) so that Zulip can send email.

Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are supported for running Zulip in production. 64-bit is recommended.

You should already have a domain name available for your Zulip production instance. In order to generate valid SSL certiﬁcates with Let’s Encrypt, and to enable other services such as Google Authentication, you’ll need to update the domains A record to point to your production server.

• SSL Certiﬁcate for the host you’re putting this on (e.g. zulip.example.com). The installation instructions contain documentation for how to get an SSL certiﬁcate for free using LetsEncrypt.

• Outgoing email (SMTP) credentials that Zulip can use to send outgoing emails to users (e.g. email address conﬁrmation emails during the signup process, missed message notiﬁcations, password reset, etc.). If you don’t have an existing outgoing SMTP solution, read about free outgoing SMTP options and options for prototyping. Once you have met these requirements, see full instructions for installing Zulip in production.

Make sure you want to install a Zulip production server; if you’d instead like to test or develop a new feature, we recommend the Zulip server development environment instead. You will need an Ubuntu system that satisﬁes the installation requirements. In short, you need: • A Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial 64-bit server • At least 2GB RAM and 10 GB disk space (4GB and 2 CPUs recommended for 100+ users). • A DNS name, an SSL certiﬁcate, and credentials for sending email.

Please subscribe to low-trafﬁc the Zulip announcements Google Group to get announcements about new releases, security issues, etc.

Zulip runs over https only and requires ssl certificates in order to work. It looks for the certificates in /etc/ ssl/private/zulip.key and /etc/ssl/certs/zulip.combined-chain.crt. Note that Zulip uses nginx as its webserver and thus expects a chained certificate bundle If you need an SSL certificate, see our SSL certificate documentation. If you already have an SSL certificate, just install (or symlink) them into place at the above paths, and move on to the next step.

If you haven’t already, download and unpack the latest built server tarball with the following commands:

sudo-i # If not already root cd/root wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz rm-rf/root/zulip&& mkdir/root/zulip tar-xf zulip-server-latest.tar.gz--directory=/root/zulip--strip-components=1

This may take a while to run, since it will install a large number of dependencies. It also creates zulip user, which will be used to run the various Zulip servers. The Zulip install script is designed to be idempotent, so if it fails, you can just rerun it after correcting the issue that caused it to fail. Also note that it automatically logs a transcript to /var/log/zulip/install.log; please include a copy of that ﬁle in any bug reports.

Configure the Zulip server instance by editing /etc/zulip/settings.py and providing values for the mandatory settings, which are all found under the heading ### MANDATORY SETTINGS. These settings include: • EXTERNAL_HOST: the user-accessible Zulip domain name for your Zulip installation (aka what users will type in their web browser). This should of course match the DNS name you configured to point to your server and for which you configured SSL certificates. If you plan to use multiple domains, add the others to ALLOWED_HOSTS. • ZULIP_ADMINISTRATOR: the email address of the person or team maintaining this installation and who will get support and error emails. • EMAIL_*, DEFAULT_FROM_EMAIL, and NOREPLY_EMAIL_ADDRESS: credentials for an outgoing SMTP server so Zulip can send emails when needed (don’t forget to set email_password in the zulip-secrets.conf file!). We highly recommend reading our production email docs and following the test procedure discussed there to make sure you’ve setup outgoing email correctly, since outgoing email is the most common configuration problem. • AUTHENTICATION_BACKENDS: a list of enabled authentication mechanisms. You’ll need to enable at least one authentication mechanism by uncommenting its corresponding line, and then also do any additional configuration required for that backend as documented in the settings.py file (the email backend requires no extra configuration). See the section on Authentication for more detail on the available authentication backends and how to configure them.

At this point, you are done doing things as root. The remaining commands are run as the zulip user using su zulip. To initialize the Zulip database for your production install, run: su zulip-c/home/zulip/deployments/current/scripts/setup/initialize-database

The initialize-database script will report an error if you did not ﬁll in all the mandatory settings from /etc/ zulip/settings.py. It is safe to rerun it after correcting the problem if that happens.

This completes the process of installing Zulip on your server. However, in order to use Zulip, you’ll need to create an organization in your Zulip installation.

• If you haven’t already, verify that your outgoing email conﬁguration works. The organization creation process will fail if outgoing email is not conﬁgured properly. • Run the organization (realm) creation management command :

su zulip # If you weren't already the zulip user /home/zulip/deployments/current/manage.py generate_realm_creation_link

This will print out a secure 1-time use link that allows creation of a new Zulip organization on your server. For most servers, you will only ever do this once, but you can run manage.py generate_realm_creation_link again if you want to host another organization on your Zulip server. • Open the link generated with your web browser. You’ll see the create organization page (screenshot here). Enter your email address and click Create organization. • Check your email to find the confirmation email and click the link. You’ll be prompted to finish setting up your organization and initial administrator user (screenshot here). Complete this form and log in! Congratulations! You are logged in as an organization administrator for your new Zulip organization. After getting oriented, we recommend visiting the “Organization settings” UI (linked from the upper-right gear menu in the Zulip webapp) to configure important policy settings like how users can join your new organization. By default, your organization will be configured as follows depending on what type of organization you selected: Community Organization: • restricted_to_domain=False: No restriction on user email addresses. • invite_required=True: A user must be invited to join. Corporate Organization: • restricted_to_domain=True: New users must have an email address in the same domain (e.g. @acme.com) as yours. • invite_required=False: No invitation is required to join. Next, you’ll likely want to do one of the following: • Customize your Zulip organization. • Learn about managing a production Zulip server.

If you get an error after scripts/setup/install completes, check /var/log/zulip/errors.log for a traceback, and consult the troubleshooting section for advice on how to debug. If that doesn’t help, please visit #production help in the Zulip developerment community server for realtime help or email [email protected] with the full traceback and we’ll try to help you out!

7.6. Step 5: Create a Zulip organization and login 41 Zulip Documentation, Release 1.4.0

Zulip uses Supervisor to monitor and control its many Python services. Read the next section, Using supervisorctl, to learn how to use the Supervisor client to monitor and manage services. If you haven’t already, now might be a good time to read Zulip’s architectural overview, particularly the Components section. This will help you understand the many services Zulip uses. If you encounter issues while running Zulip, take a look at Zulip’s logs, which are located in /var/log/zulip/. That directory contains one log ﬁle for each service, plus errors.log (has all errors), server.log (has logs from the Django and Tornado servers), and workers.log (has combined logs from the queue workers). The section troubleshooting services on this page includes details about how to ﬁx common issues with Zulip services. If you run into additional problems, please report them so that we can update this page! The Zulip installation scripts logs its full output to /var/log/zulip/install.log, so please include the context for any tracebacks from that log.

To see what Zulip-related services are conﬁgured to use Supervisor, look at /etc/supervisor/conf.d/ zulip.conf and /etc/supervisor/conf.d/zulip-db.conf. Use the supervisor client supervisorctl to list the status of, stop, start, and restart various services.

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-0 RUNNING pid 2209,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-1 RUNNING pid 2210,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-2 RUNNING pid 2211,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-3 RUNNING pid 2212,

˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-4 RUNNING pid 2208,

˓→uptime1:13:11 zulip-workers:zulip-deliver-enqueued-emails STARTING zulip-workers:zulip-events-confirmation-emails RUNNING pid 2199,

˓→uptime1:13:11 zulip-workers:zulip-events-feedback_messages RUNNING pid 2207,

˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_mobile_notifications RUNNING pid 2204,

˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_reminders RUNNING pid 2206,

˓→uptime1:13:11 zulip-workers:zulip-events-user-activity-interval RUNNING pid 2196,

After you change configuration in /etc/zulip/settings.py or fix a misconfiguration, you will often want to restart the Zulip application. You can restart Zulip using: supervisorctl restart all

The Zulip application uses several major open source services to store and cache data, queue messages, and otherwise support the Zulip application: • postgresql • rabbitmq-server • nginx • redis • memcached If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some common conﬁguration problems and how to resolve them: • An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running; to ﬁx this, try:

If RabbitMQ fails to start, the problem is often that you are using a virtual machine with broken DNS configuration; you can often correct this by configuring /etc/hosts properly. • If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus failed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. To fix this, configure them properly and then run:

• If your host is being port scanned by unauthorized users, you may see messages in /var/log/zulip/ server.log like

Django uses the hostnames conﬁgured in ALLOWED_HOSTS to identify legitimate requests and block others. When an incoming request does not have the correct HTTP Host header, Django rejects it and logs the attempt. For more on this issue, see the Django release notes on Host header poisoning Next: Making your Zulip instance awesome.

Once you’ve got Zulip setup, you’ll likely want to configure it the way you like. Most configuration can be done by a realm administrator on the web (see the documentation for realm administrators; this page discusses those additional configuration items that can be done by a system administrator.

The Zulip apps expect to be talking to to servers with a properly signed SSL certificate, in most cases and will not accept a self-signed certificate. You should get a proper SSL certificate before testing the apps. Because of how Google and Apple have architected the security model of their push notification protocols, the Zulip mobile apps for iOS and Android can only receive push notifications from a single Zulip server. We have configured that server to be push.zulipchat.com, and offer a push notification forwarding service that forwards push notifications through our servers to mobile devices. Read the linked documentation for instructions on how to register for and configure this service. By the end of summer 2017, all of the Zulip apps will have full support for multiple accounts, potentially on different Zulip servers, with a convenient UI for switching between them.

Zulip allows you to configure your server’s Terms of Service and Privacy Policy pages (/terms and /privacy, respectively). You can use the TERMS_OF_SERVICE and PRIVACY_POLICY settings to configure the path to your server’s policies. The syntax is Markdown (with support for included HTML). A good approach is to use paths like /etc/zulip/terms.md, so that it’s easy to back up your policy configuration along with your other Zulip server configuration.

Zulip has dozens of settings documented in the comments in /etc/zulip/settings.py; you can review the latest version of the settings.py template if you’ve deleted the comments or want to check if new settings have been added in more recent versions of Zulip. Since Zulip’s settings file is a Python script, there are a number of other things that one can configure that are not documented; ask on chat.zulip.org if there’s something you’d like to do but can’t figure out how to.

If you haven’t already, subscribe to the zulip-announce list so that you can receive important announces like new Zulip releases or major changes to the app ecosystem..

If you discover things that you wish had been documented, please contribute documentation suggestions either via a GitHub issue or pull request; we love even small contributions, and we’d love to make the Zulip documentation cover everything anyone might want to know about running Zulip in production. Next: Maintaining and upgrading Zulip in production.

Zulip’s iOS and Android mobile apps support receiving push notifications from Zulip servers to let users know when new messages have arrived. This is an important feature to having a great experience using the Zulip mobile apps. For technical reasons (explained below), in order to deliver mobile push notifications in the app store versions of our mobile apps, you will need to register your Zulip server with the Zulip mobile push notification service. This service will forward push notifications generated by your server to the Zulip mobile app automatically.

Starting with Zulip 1.6, Zulip servers support forwarding push notifications to a central push notification forwarding service. You can enable this for your Zulip server as follows: 1. First, contact [email protected] for instructions on registering your server (we hope to have a nice web flow available for this soon). 2. Then, uncomment the PUSH_NOTIFICATION_BOUNCER_URL = https://push.zulipchat.com line in your /etc/zulip/settings.py file, and restart your Zulip server. That should be all you need to do! To test whether you’ve configured it properly, just configure mobile push notifications to always be sent, start the Zulip Android app, in the background, and then have another user send you private messages. They should appear in the Android notification area.

Both Google’s and Apple’s push notification services have a security model that does not support mutually untrusted self-hosted servers sending push notifications to the same app. In particular, when an app is published to their respec- tive app stores, one must compile into the app a secret corresponding to the server that will be able to publish push notifications for the app. This means that it is impossible for a single app in their stores to receive push notifications from multiple, mutually untrusted, servers.

Zulip’s solution to this problem is to provide a central push notiﬁcation forwarding service, which allows registered Zulip servers to send push notiﬁcations to the Zulip app indirectly (through the forwarding service).

We’ve designed this push notification bouncer service with security in mind: • All of the network requests (both from Zulip servers to the Push Notification Service and from the Push Notifi- cation Service to the relevant Google and Apple services) are encrypted over the wire with SSL/TLS. • The code for the push notification forwarding service is 100% open source and available as part of the Zulip server project on GitHub. The service’s logging is designed to protect the privacy of users of Zulip servers that are using the forwarding service. • The push notification forwarding servers are professionally managed by a small team. If you have any questions about the security model, contact [email protected].

This page covers topics that will help you maintain a healthy, up-to-date, and secure Zulip installation, including: • Upgrading • Upgrading from a git repository • Backups • Monitoring • Scalability • Management commands You may also want to read this related content: • Security Model

We recommend reading this entire section before doing your ﬁrst upgrade. To upgrade to a new version of the zulip server, download the appropriate release tarball from https://www.zulip.org/ dist/releases/. You also have the option of creating your own release tarballs from a copy of the zulip.git repository using tools/ build-release-tarball. And, starting with Zulip version 1.4, you can upgrade Zulip to a version in a Git repository directly. Next, run as root:

/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz

The upgrade process will shut down the Zulip service and then run apt-get upgrade, a puppet apply, any database migrations, and then bring the Zulip service back up. Upgrading will result in some brief downtime for the service,

which should be under 30 seconds unless there is an expensive transition involved. Unless you have tested the upgrade in advance, we recommend doing upgrades at off hours. Note that upgrading an existing Zulip production server from Ubuntu 14.04 Trusty to Ubuntu 16.04 Xenial will require signiﬁcant manual intervention on your part to migrate the data in the database from Postgres 9.3 to Postgres 9.5. Contributions on testing and documenting this process are welcome!

Warning: If you have modified configuration files installed by Zulip (e.g. the nginx configuration), the Zulip upgrade process will overwrite your configuration when it does the puppet apply. You can test whether this will happen assuming no upstream changes to the configuration using scripts/ zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changes it would make. Using this list, you can save a copy of any files that you’ve modified, do the upgrade, and then restore your configuration. If you need to do this, please report the issue so that we can make the Zulip puppet configuration flexible enough to handle your setup.

The Zulip upgrade script automatically logs output to /var/log/zulip/upgrade.log. Please use those logs to include output that shows all errors in any bug reports. After the upgrade, we recommend checking /var/log/zulip/errors.log to conﬁrm that your users are not experiencing errors after the upgrade.

The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/ containing a complete copy of the Zulip server code, and then moving the symlinks at /home/zulip/deployments/ current and /root/zulip as part of the upgrade process. This means that if the new version isn’t working, you can quickly downgrade to the old version by using /home/ zulip/deployments//scripts/restart-server to return to a previous version that you’ve deployed (the version is speciﬁed via the path to the copy of restart-server you call).

If required, you can update your settings by editing /etc/zulip/settings.py and then run /home/zulip/ deployments/current/scripts/restart-server to restart the server.

While the Zulip upgrade script runs apt-get upgrade, you are responsible for running this on your system on a regular basis between Zulip upgrades to ensure that it is up to date with the latest security patches.

To use the Zulip API with your Zulip server, you will need to use the API endpoint of e.g. https:// zulip.example.com/api. Our Python API example scripts support this via the --site=https://zulip. example.com argument. The API bindings support it via putting site=https://zulip.example.com in your .zuliprc. Every Zulip integration supports this sort of argument (or e.g. a ZULIP_SITE variable in a zuliprc ﬁle or the environment), but this is not yet documented for some of the integrations (the included integration documentation on /integrations will properly document how to do this for most integrations). We welcome pull requests for integrations that don’t discuss this! Similarly, you will need to instruct your users to specify the URL for your Zulip server when using the Zulip desktop and mobile apps.

As a measure to mitigate the impact of potential memory leaks in one of the Zulip daemons, the service automatically restarts itself every Sunday early morning. See /etc/cron.d/restart-zulip for the precise conﬁguration.

Starting with version 1.4, the Zulip server supports doing deployments from a Git repository. To conﬁgure this, you will need to add zulip::static_asset_compiler to your /etc/zulip/zulip.conf ﬁle’s puppet_classes entry, like this:

Then, run scripts/zulip-puppet-apply to install the dependencies for building Zulip’s static assets. You can conﬁgure the git repository that you’d like to use by adding a section like this to /etc/zulip/zulip.conf; by default it uses the main zulip repository (shown below).

Once that is done (and assuming the currently installed version of Zulip is new enough that this script exists), you can do deployments by running as root:

and Zulip will automatically fetch the relevant branch from the speciﬁed repository, build the static assets, and deploy that version. Currently, the upgrade process is slow, but it doesn’t need to be; there is ongoing work on optimizing it.

There are several pieces of data that you might want to back up: • The postgres database. That you can back up like any postgres database; we have some example tooling for doing that incrementally into S3 using wal-e in puppet/zulip_internal/manifests/ postgres_common.pp (that’s what we use for zulip.com’s database backups). Note that this module isn’t part of the Zulip server releases since it’s part of the zulip.com conﬁguration (see https://github.com/zulip/zulip/ issues/293 for a ticket about ﬁxing this to make life easier for running backups).

• Any user-uploaded files. If you’re using S3 as storage for file uploads, this is backed up in S3, but if you have instead set LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars) will be stored in that directory and you’ll want to back it up. • Your Zulip configuration including secrets from /etc/zulip/. E.g. if you lose the value of secret_key, all users will need to login again when you setup a replacement server since you won’t be able to verify their cookies; if you lose avatar_salt, any user-uploaded avatars will need to be re-uploaded (since avatar filenames are computed using a hash of avatar_salt and user’s email), etc. • The logs under /var/log/zulip can be handy to have backed up, but they do get large on a busy server, and it’s definitely lower-priority. If you are interested in backups because you are moving from one Zulip server to another server and can’t transfer a full postgres dump (which is definitely the simplest approach), our draft conversion and export design document may help. The tool is well designed and was tested carefully with dozens of realms as of mid-2016 but is not integrated into Zulip’s regular testing process, and thus it is worth asking on the Zulip developers mailing list whether it needs any minor updates to do things like export newly added tables.

To restore from backups, the process is basically the reverse of the above: • Install new server as normal by downloading a Zulip release tarball and then using scripts/setup/ install, you don’t need to run the initialize-database second stage which puts default data into the database. • Unpack to /etc/zulip the settings.py and zulip-secrets.conf files from your backups. • Restore your database from the backup using wal-e; if you ran initialize-database anyway above, you’ll want to first scripts/setup/postgres-init-db to drop the initial database first. • Reconfigure rabbitmq to use the password from secrets.conf by running, as root, scripts/setup/ configure-rabbitmq. • If you’re using local file uploads, restore those files to the path specified by settings. LOCAL_UPLOADS_DIR and (if appropriate) any logs. • Start the server using scripts/restart-server. This restoration process can also be used to migrate a Zulip installation from one server to another. We recommend running a disaster recovery after you setup backups to confirm that your backups are working; you may also want to monitor that they are up to date using the Nagios plugin at: puppet/zulip_internal/files/ nagios_plugins/check_postgres_backup. Contributions to more fully automate this process or make this section of the guide much more explicit and detailed are very welcome!

Zulip has database configuration for using Postgres streaming replication; you can see the configuration in these files: • puppet/zulip_internal/manifests/postgres_slave.pp • puppet/zulip_internal/manifests/postgres_master.pp • puppet/zulip_internal/files/postgresql/* Contribution of a step-by-step guide for setting this up (and moving this configuration to be available in the main puppet/zulip/ tree) would be very welcome!

The complete Nagios configuration (sans secret keys) used to monitor zulip.com is available under puppet/ zulip_internal in the Zulip Git repository (those files are not installed in the release tarballs). The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirec- tories under /usr/lib/nagios/plugins/. The following is a summary of the various Nagios plugins included with Zulip and what they check: Application server and queue worker monitoring: • check_send_receive_time (sends a test message through the system between two bot users to check that end-to-end message sending works) • check_rabbitmq_consumers and check_rabbitmq_queues (checks for rabbitmq being down or the queue workers being behind) • check_queue_worker_errors (checks for errors reported by the queue workers) • check_worker_memory (monitors for memory leaks in queue workers) • check_email_deliverer_backlog and check_email_deliverer_process (monitors for whether outgoing emails are being sent) Database monitoring: • check_postgres_replication_lag (checks streaming replication is up to date). • check_postgres (checks the health of the postgres database) • check_postgres_backup (checks backups are up to date; see above) • check_fts_update_log (monitors for whether full-text search updates are being processed) Standard server monitoring: • check_website_response.sh (standard HTTP check) • check_debian_packages (checks apt repository is up to date) If you’re using these plugins, bug reports and pull requests to make it easier to monitor Zulip and maintain it in production are encouraged!

This section attempts to address the considerations involved with running Zulip with larger teams (especially >1000 users). • For an organization with 100+ users, it’s important to have more than 4GB of RAM on the system. Zulip will install on a system with 2GB of RAM, but with less than 3.5GB of RAM, it will run its queue processors multithreaded to conserve memory; this creates a signiﬁcant performance bottleneck. • chat.zulip.org, with thousands of user accounts and thousands of messages sent every week, has 8GB of RAM, 4 cores, and 80GB of disk. The CPUs are essentially always idle, but the 8GB of RAM is important. • We recommend using a remote postgres database for isolation, though it is not required. In the following, we discuss a relatively simple conﬁguration with two types of servers: application servers (running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and database servers. • You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) with just a single reasonably large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB of RAM) sitting mostly idle (<10% CPU used and only 4GB of the 16GB RAM actively in use). You can probably get away

with half that (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale. Beyond a 1000 active users, you will eventually want to increase the memory cap in memcached.conf from the default 512MB to avoid high rates of memcached misses. • For the database server, we highly recommend SSD disks, and RAM is the primary resource limitation. We have not aggressively tested for the minimum resources required, but 8 cores with 30GB of RAM (e.g. AWS’s m3.2xlarge) should suffice; you may be able to get away with less especially on the CPU side. The database load per user is pretty optimized as long as memcached is working correctly. This has not been tested, but from extrapolating the load profile, it should be possible to scale a Zulip installation to 10,000s of active users using a single large database server without doing anything complicated like sharding the database. • For reasonably high availability, it’s easy to run a hot spare application server and a hot spare database (using Postgres streaming replication; see the section on configuring this). Be sure to check out the section on backups if you’re hoping to run a spare application server; in particular you probably want to use the S3 backend for storing user-uploaded files and avatars and will want to make sure secrets are available on the hot spare. • Zulip does not support dividing traffic for a given Zulip realm between multiple application servers. There are two issues: you need to share the memcached/Redis/RabbitMQ instance (these should can be moved to a network service shared by multiple servers with a bit of configuration) and the Tornado event system for pushing to browsers currently has no mechanism for multiple frontend servers (or event processes) talking to each other. One can probably get a factor of 10 in a single server’s scalability by supporting multiple tornado processes on a single server, which is also likely the first part of any project to support exchanging events amongst multiple servers. Questions, concerns, and bug reports about this area of Zulip are very welcome! This is an area we are hoping to improve.

Zulip has a large library of Django management commands. To use them, you will want to be logged in as the zulip user and for the purposes of this documentation, we assume the current working directory is /home/zulip/ deployments/current. Below, we show several useful examples, but there are more than 100 in total. We recommend skimming the usage docs (or if there are none, the code) of a management command before using it, since they are generally less polished and more designed for expert use than the rest of the Zulip system.

You can get an iPython shell with full access to code within the Zulip project using manage.py shell, e.g., you can do the following to change an email address:

$ /home/zulip/deployments/current/manage.py shell In [1]: user_profile = get_user_profile_by_email("[email protected]") In [2]: do_change_user_email(user_profile, "[email protected]")

56 Chapter 11. Secure, maintain, and upgrade Zulip Documentation, Release 1.4.0 manage.py dbshell

You can make any user a realm administrator on the command line with the knight management command:

If you need to manage the IRC, Jabber, or Zephyr mirrors, you will need to create API super users. To do this, use ./manage.py knight with the --permission=api_super_user argument. See api/integrations/ irc-mirror.py and api/integrations/jabber_mirror.py for further detail on these.

If you need to do an export of a single user or of an entire realm, we have tools in management/ that essentially export Zulip data to the ﬁle system. export_single_user.py exports the message history and realm-public metadata for a single Zulip user (including that user’s received messages as well as their sent messages). A good overview of the process for exporting a single realm when moving a realm to a new server (without moving a full database dump) is in management/export.py. We recommend you read the comment there for words of wisdom on speed, what is and is not exported, what will break upon a move to a new server, and suggested procedure.

There are a large number of useful management commands under zerver/management/commands/; you can also see them listed using ./manage.py with no arguments.

This section attempts to document the Zulip security model. Since this is new documentation, it likely does not cover every issue; if there are details you’re curious about, please feel free to ask questions on the Zulip development mailing list (or if you think you’ve found a security bug, please report it to [email protected] so we can do a responsible security announcement).

• It’s reasonable to think about security for a Zulip server like you do security for a team email server – only trusted administrators within an organization should have shell access to the server. In particular, anyone with root access to a Zulip application server or Zulip database server, or with access to the zulip user on a Zulip application server, has complete control over the Zulip installation and all of its data (so they can read messages, modify history, etc.). It would be difﬁcult or impossible to avoid this, because the server needs access to the data to support features expected of a group chat system like the ability to search the entire message history, and thus someone with control over the server has access to that data as well.

• Trafﬁc between clients (web, desktop and mobile) and the Zulip is encrypted using HTTPS. By default, all Zulip services talk to each other either via a localhost connection or using an encrypted SSL connection. • Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks. • The preferred way to login to Zulip is using an SSO solution like Google Auth, LDAP, or similar, but Zulip also supports password authentication. See the authentication methods documentation for details on Zulip’s available authentication methods.

Zulip stores user passwords using the standard PBKDF2 algorithm. Password strength is checked and weak passwords are visually discouraged using the popular zxcvbn library. The minimum password strength allowed is controlled by two settings in /etc/zulip/settings.py; PASSWORD_MIN_LENGTH and PASSWORD_MIN_ZXCVBN_QUALITY. The former is self-explanatory; we will explain the latter. Password strength estimation is a complicated topic that we can’t go into great detail on here; we recommend reading the zxvcbn website for details if you are not familiar with password strength analysis. In Zulip’s configuration, a password has quality X if zxcvbn estimates that it would take e^(X * 22) seconds to crack the password with a specific attack scenario. The scenario Zulip uses is one where an the attacker breaks into the Zulip server and steals the hashed passwords; in that case, with a slow hash, the attacker would be able to make roughly 10,000 attempts per second. E.g. a password with quality 0.5 (the default), it would take an attacker about 16 hours to crack such a password in this sort of offline attack. Another important attack scenario is the online attacks (i.e. an attacker sending tons of login requests guessing different passwords to a Zulip server over the web). Those attacks are much slower (more like 10/second without rate limiting), and you should estimate the time to guess a password as correspondingly longer. As a server administrators, you must balance the security risks associated with attackers guessing weak passwords against the usability challenges associated with requiring strong passwords in your organization.

• Zulip message content is rendered using a specialized Markdown parser which escapes content to protect against cross-site scripting attacks. • Zulip supports both public streams and private (“invite-only”) streams. Any Zulip user can join any public stream in the realm, and can view the complete message history of any public stream without joining the stream. • A private (“invite-only”) stream is hidden from users who are not subscribed to the stream. Users who are not members of a private stream cannot read messages on the stream, send messages to the stream, or join the stream, even if they are a Zulip realm administrator. Users can join private streams only when they are invited. However, any member of a private stream can invite other users to the stream. When a new user joins a private stream, they can see future messages sent to the stream, but they do not receive access to the stream’s message history. • Zulip supports editing the content and topics of messages that have already been sent. As a general philosophy, our policies provide hard limits on the ways in which message content can be changed or undone. In contrast, our policies around message topics favor usefulness (e.g. for conversational organization) over faithfulness to the original. The message editing policy can be configured on the /#organization page. There are three configurations provided out of the box: (i) users cannot edit messages at all, (ii) users can edit any message they have sent, and (iii) users can edit the content of any message they have sent in the last N minutes, and the topic of any message they have sent. In (ii) and (iii), topic edits can also be propagated to other messages with the same original topic, even if those messages were sent by other users. The default setting is (iii), with N = 10. In addition, and regardless of the configuration above, messages with no topic can always be edited to have a topic, by anyone in the organization, and the topic of any message can also always be edited by a realm administrator. Also note that while edited messages are synced immediately to open browser windows, editing messages is not a safe way to redact secret content (e.g. a password) shared unintentionally. Other users may have seen and saved the content of the original message, or have an integration (e.g. push notifications) forwarding all

messages they receive to another service. Zulip also stores and sends to clients the content of every historical version of a message.

• There are three types of users in a Zulip realm: Administrators, normal users, and bots. Administrators have the ability to deactivate and reactivate other human and bot users, delete streams, add/remove administrator privileges, as well as change conﬁguration for the overall realm (e.g. whether an invitation is required to join the realm). Being a Zulip administrator does not provide the ability to interact with other users’ private messages or the messages sent to private streams to which the administrator is not subscribed. However, a Zulip administrator subscribed to a stream can toggle whether that stream is public or private. Also, Zulip realm administrators have administrative access to the API keys of all bots in the realm, so a Zulip administrator may be able to access messages sent to private streams that have bots subscribed, by using the bot’s credentials. In the future, Zulip’s security model may change to allow realm administrators to access private messages (e.g. to support auditing functionality). • Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially everything the user can do; for that reason, users should keep their API key safe. Users can rotate their own API key if it is accidentally compromised. • To properly remove a user’s access to a Zulip team, it does not sufﬁce to change their password or deactivate their account in the SSO system, since neither of those prevents authenticating with the user’s API key or those of bots the user has created. Instead, you should deactivate the user’s account in the “Organization settings” interface (/#organization); this will automatically also deactivate any bots the user had created. • The Zulip mobile apps authenticate to the server by sending the user’s password and retrieving the user’s API key; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user’s phone is lost, in addition to changing passwords, you should rotate the user’s Zulip API key. • Zulip bots are used for integrations. A Zulip bot can do everything a normal user in the realm can do including reading other, with a few exceptions (e.g. a bot cannot login to the web application or create other bots). In particular, with the API key for a Zulip bot, one can read any message sent to a public stream in that bot’s realm. A likely future feature for Zulip is limited bots that can only send messages. • Certain Zulip bots can be marked as “API super users”; these special bots have the ability to send messages that appear to have been sent by another user (an important feature for implementing integrations like the Jabber, IRC, and Zephyr mirrors).

• Zulip supports user-uploaded files; ideally they should be hosted from a separate domain from the main Zulip server to protect against various same-domain attacks (e.g. zulip-user-content.example.com) using the S3 integration. The URLs of user-uploaded files are secret; if you are using the “local file upload” integration, anyone with the URL of an uploaded file can access the file. This means the local uploads integration is vulnerable to a subtle attack where if a user clicks on a link in a secret .PDF or .HTML file that had been uploaded to Zulip, access to the file might be leaked to the other server via the Referrer header (see the “Uploads world readable” issue on GitHub). The Zulip S3 file upload integration is relatively safe against that attack, because the URLs of files presented to users don’t host the content. Instead, the S3 integration checks the user has a valid Zulip session in the relevant realm, and if so then redirects the browser to a one-time S3 URL that expires a short time later. Keeping the URL secret is still important to avoid other users in the Zulip realm from being able to access the file.

• Zulip supports using the Camo image proxy to proxy content like inline image previews that can be inserted into the Zulip message feed by other users over HTTPS. • By default, Zulip will provide image previews inline in the body of messages when a message contains a link to an image. You can control this using the INLINE_IMAGE_PREVIEW setting.

If you ﬁnd some aspect of Zulip that seems inconsistent with this security model, please report it to zulip- [email protected] so that we can investigate and coordinate an appropriate security release if needed. Zulip security announcements will be sent to [email protected], so you should subscribe if you are running Zulip in production.

Zulip supports several different authentications methods: • EmailAuthBackend - Email/password authentication. • ZulipLDAPAuthBackend - LDAP username/password authentication. • GoogleMobileOauth2Backend - Google authentication. • GitHubAuthBackend - GitHub authentication. • ZulipRemoteUserBackend - Authentication using an existing Single-Sign-On (SSO) system that can set REMOTE_USER in Apache. • DevAuthBackend - Only for development, passwordless login as any user. It’s easy to add more; see the docs on python-social-auth below. The setup documentation for most of these is simple enough that we’ve included it inline in /etc/zulip/ settings.py, right above to the settings used to conﬁgure them. The remote user authentication backend is more complex since it requires interfacing with a generic third-party authentication system, and so we’ve documented it in detail below.

The implementation for GitHubAuthBackend is a small wrapper around the popular python-social-auth library. So if you’d like to integrate Zulip with another authentication provider (e.g. Facebook, Twitter, etc.), you can do this by writing a class similar to GitHubAuthBackend in zproject/backends.py and adding a few settings. Pull requests to add new backends are welcome; they should be tested using the framework in test_auth_backends. py.

Zulip supports integrating with a Single-Sign-On solution. There are a few ways to do it, but this section documents how to conﬁgure Zulip to use an SSO solution that best supports Apache and will set the REMOTE_USER variable: (0) Check that /etc/zulip/settings.py has zproject.backends.ZulipRemoteUserBackend as the only enabled value in the AUTHENTICATION_BACKENDS list, and that SSO_APPEND_DOMAIN is correct set depending on whether your SSO system uses email addresses or just usernames in REMOTE_USER. Make sure that you’ve restarted the Zulip server since making this conﬁguration change. (1) Edit /etc/zulip/zulip.conf and change the puppet_classes line to read:

(2) As root, run /home/zulip/deployments/current/scripts/zulip-puppet-apply to install our SSO integration. (3) To configure our SSO integration, edit /etc/apache2/sites-available/zulip-sso.example and fill in the configuration required for your SSO service to set REMOTE_USER and place your completed configuration file at /etc/apache2/sites-available/zulip-sso.conf zulip-sso.example is correct configuration for using an htpasswd file for REMOTE_USER authentication, which is useful for testing quickly. You can set it up by doing the following:

/home/zulip/deployments/current/scripts/restart-server cd/etc/apache2/sites-available/ cp zulip-sso.example zulip-sso.conf htpasswd-c/home/zulip/zpasswd username @example.com # prompts for a password

and then continuing with the steps below. (4) Run a2ensite zulip-sso to enable the Apache integration site. (5) Run service apache2 reload to use your new conﬁguration. If Apache isn’t already running, you may need to run service apache2 start instead. Now you should be able to visit https://zulip.example.com/ and login via the SSO solution.

This system is a little finicky to networking setup (e.g. common issues have to do with /etc/hosts not mapping settings.EXTERNAL_HOST to the Apache listening on 127.0.0.1/localhost, for example). It can often help while debugging to temporarily change the Apache config in /etc/apache2/sites-available/ zulip-sso to listen on all interfaces rather than just 127.0.0.1 as you debug this. It can also be helpful to change /etc/nginx/zulip-include/app.d/external-sso.conf to proxy_pass to a more explicit URL possibly not over HTTPS when debugging. The following log files can be helpful when debugging this setup: • /var/log/zulip/{errors.log,server.log} (the usual places) • /var/log/nginx/access.log (nginx access logs) • /var/log/apache2/zulip_auth_access.log (you may want to change LogLevel to “debug” in the Apache config file to make this more verbose) Here’s a summary of how the remote user SSO system works assuming you’re using HTTP basic auth; this summary should help with understanding what’s going on as you try to debug:

• Since you’ve configured /etc/zulip/settings.py to only define the zproject.backends. ZulipRemoteUserBackend, zproject/settings.py configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makes https://zulip.example.com/ (a.k.a. the homepage for the main Zulip Django app running behind nginx) redirect to /accounts/login/sso if you’re not logged in. • nginx proxies requests to /accounts/login/sso/ to an Apache instance listening on localhost:8888 apache via the config in /etc/nginx/zulip-include/app.d/external-sso.conf (using the upstream localhost:8888 defined in /etc/nginx/zulip-include/upstreams). • The Apache zulip-sso site which you’ve enabled listens on localhost:8888 and presents the htpasswd dialogue; you provide correct login information and the request reaches a second Zulip Django app instance that is running behind Apache with with REMOTE_USER set. That request is served by zerver. views.remote_user_sso, which just checks the REMOTE_USER variable and either logs in (sets a cookie) or registers the new user (depending whether they have an account). • After succeeding, that redirects the user back to / on port 443 (hosted by nginx); the main Zulip Django app sees the cookie and proceeds to load the site homepage with them logged in (just as if they’d logged in normally via username/password). Again, most issues with this setup tend to be subtle issues with the hostname/DNS side of the configuration. Sugges- tions for how to improve this SSO setup documentation are very welcome!

This is a bit annoying to setup, but you can configure Zulip to use a dedicated postgres server by setting the REMOTE_POSTGRES_HOST variable in /etc/zulip/settings.py, and configuring Postgres certificate authentication (see http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for documentation on how to set this up and deploy the certificates) to make the DATABASES configuration in zproject/settings.py work (or override that configuration). If you want to use a remote Postgresql database, you should configure the information about the connection with the server. You need a user called “zulip” in your database server. You can configure these options in /etc/zulip/settings.py (the below descriptions are from the Postgresql documentation): • REMOTE_POSTGRES_HOST: Name or IP address of the remote host • REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are: – disable: I don’t care about security, and I don’t want to pay the overhead of encryption. – allow: I don’t care about security, but I will pay the overhead of encryption if the server insists on it. – prefer: I don’t care about encryption, but I wish to pay the overhead of encryption if the server supports it. – require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want. – verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust. – verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it’s the one I specify. Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf: postgres_password= xxxx

Finally, you can stop your database on the Zulip server via: sudo service postgresql stop sudo update-rc.d postgresql disable

In future versions of this feature, we’d like to implement and document how to the remote postgres database server itself automatically by using the Zulip install script with a different set of puppet manifests than the all-in-one feature; if you’re interested in working on this, post to the Zulip development mailing list and we can give you some tips.

When debugging postgres issues, in addition to the standard pg_top tool, often it can be useful to use this query:

˓→procpid; which shows the currently running backends and their activity. This is similar to the pg_top output, with the added advantage of showing the complete query, which can be valuable in debugging. To stop a runaway query, you can run SELECT pg_cancel_backend(pid int) or SELECT pg_terminate_backend(pid int) as the ‘postgres’ user. The former cancels the backend’s current query and the latter terminates the backend process. They are implemented by sending SIGINT and SIGTERM to the processes, respectively. We recommend against sending a Postgres process SIGKILL. Doing so will cause the database to kill all current connections, roll back any pending transactions, and enter recovery mode.

To start or stop postgres manually, use the pg_ctlcluster command: pg_ctlcluster 9.1[--force] main {start|stop|restart|reload}

By default, using stop uses “smart” mode, which waits for all clients to disconnect before shutting down the database. This can take prohibitively long. If you use the –force option with stop, pg_ctlcluster will try to use the “fast” mode for shutting down. “Fast” mode is described by the manpage thusly: With the –force option the “fast” mode is used which rolls back all active transactions, disconnects clients immediately and thus shuts down cleanly. If that does not work, shutdown is attempted again in “immediate” mode, which can leave the cluster in an inconsistent state and thus will lead to a recovery run at the next start. If this still does not help, the postmaster process is killed. Exits with 0 on success, with 2 if the server is not running, and with 1 on other failure conditions. This mode should only be used when the machine is about to be shut down. Many database parameters can be adjusted while the database is running. Just modify /etc/postgresql/9.1/main/postgresql.conf and issue a reload. The logs will note the change.

Debugging issues starting postgres pg_ctlcluster often doesn’t give you any information on why the database failed to start. It may tell you to check the logs, but you won’t ﬁnd any information there. pg_ctlcluster runs the following command underneath when it actually goes to start Postgres:

/usr/lib/postgresql/9.1/bin/pg_ctl start-D/var/lib/postgresql/9.1/main-s-o \ '-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'

Since pg_ctl doesn’t redirect stdout or stderr, running the above can give you better diagnostic information. However, you might want to stop Postgres and restart it using pg_ctlcluster after you’ve debugged with this approach, since it does bypass some of the work that pg_ctlcluster does.

The autovac_freeze postgres alert from check_postgres is particularly important. This alert indicates that the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to the autovacuum_freeze_max_age setting. When the oldest XID hits that age, Postgres will force a VACUUM operation, which can often lead to sudden downtime until the operation ﬁnishes. If it did not do this and the age of the oldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagios alert, perform a VACUUM in each indicated database as a database superuser (postgres). See http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND for more details on postgres vacuuming.

The Zulip development environment can be installed on macOS, Windows, and Linux (Ubuntu recommended). You’ll need at least 2GB of available RAM. Installing the Zulip development environment requires downloading several hundred megabytes of dependencies, so you will need an active, reasonably fast, internet connection throughout the entire installation processes. You can conﬁgure a proxy if you need one.

For ﬁrst-time contributors on macOS, Windows, and Ubuntu, we recommend using the Vagrant development environment. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Vagrant adds a bit of overhead to using the Zulip development server, but provides an isolated environment that is easy to install, update, and uninstall. It has been well-tested and performs well.

For more experienced contributors, or for ﬁrst-time contributors who don’t want to or can’t use Vagrant, Zulip supports a wide range of ways to install the Zulip development environment on macOS and Linux (Ubuntu recommended): • On Ubuntu 16.04 Xenial and 14.04 Trusty, you can easily install without using Vagrant. • On other Linux distributions, you’ll need to follow slightly different instructions to install manually. • On macOS and Linux (Ubuntu recommended), you can install using Docker, though support for this remains experimental.

Unfortunately, the only supported method to install on Windows is the Vagrant method.

If you have a very slow network connection, however, you may want to avoid using Vagrant (which involves downloading an Ubuntu virtual machine or Linux Container) and either install directly on Ubuntu 16.04 Xenial or 14.04 Trust, or use the manual install process instead. These options only support Linux. An alternative option if you have poor network connectivity is to rent a cloud server and install the Zulip development environment for remote use. See next section for details.

The Zulip development environment works well on remote virtual machines. This can be a good alternative for those with poor network connectivity or who have limited storage/memory on their local machines. We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04, with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to. If you want to run a non-Ubuntu distribution, follow the generic Linux directions.

Once you’ve installed the Zulip development environment, you’ll want to read these documents to learn how to use it: • Using the Development Environment • Testing And if you’ve setup the Zulip development environment on a remote machine, take a look at our tips for developing remotely.

This section guides ﬁrst-time contributors through installing the Zulip development environment on Windows, macOS, and Ubuntu. The recommended method for installing the Zulip development environment is to use Vagrant with VirtualBox on Windows and macOS, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Contents: • Requirements • Step 1: Install Prerequisites • Step 2: Get Zulip code • Step 3: Start the development environment • Step 4: Developing • Troubleshooting & Common Errors • Specifying a proxy If you encounter errors installing the Zulip development environment, check Troubleshooting & Common Errors. If that doesn’t help, please visit #provision help in the Zulip development community server for real-time help, send a note to the Zulip-devel Google group or ﬁle an issue. When reporting your issue, please include the following information: • host operating system • installation method (Vagrant or direct) • whether or not you are using a proxy • a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual machine

Installing the Zulip development environment requires downloading several hundred megabytes of dependencies. You will need an active internet connection throughout the entire installation processes. (See Specifying a proxy if you need a proxy to access the internet.) • All: 2GB available RAM, Active broadband internet connection. • macOS: macOS (10.11 El Capitan or 10.12 Sierra recommended), Git, VirtualBox, Vagrant. • Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc. – or Debian: 9.0 “stretch” 64-bit • Windows: Windows 64-bit (Win 10 recommended), hardware virtualization enabled (VT-X or AMD-V), administrator access, Git for Windows (which installs Git BASH), VirtualBox, Vagrant. Don’t see your system listed above? See Advanced setup for details about installing for other Linux and UNIX platforms.

1. Install Vagrant (1.8.4-1.8.6, do not use 1.8.7). 2. Install VirtualBox (>= 5.1.8). (For a non-free option, but better performance, you can also use VMWare Fusion with the VMWare Fusion Vagrant plugin.) Now you are ready for Step 2: Get Zulip Code..

The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same. If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code: sudo apt-get-y purge vagrant&&\ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb&&\ sudo dpkg-i vagrant *.deb&&\ sudo apt-get-y install build-essential git ruby lxc lxc-templates cgroup-lite redir&

74 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in the ofﬁcial Ubuntu repositories. First uninstall any vagrant package you may have installed from the Ubuntu repository:

christie@ubuntu-desktop:~ $ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb

christie@ubuntu-desktop:~ $ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir

christie@ubuntu-desktop:~ $ vagrant plugin install vagrant-lxc Installing the 'vagrant-lxc' plugin. This can take a few minutes... Installed the plugin 'vagrant-lxc (1.2.1)'!

If you encounter an error when trying to install the vagrant-lxc plugin, see this.

If you encounter an error running vagrant lxc sudoers, see this. Now you are ready for Step 2: Get Zulip Code.

The setup for Debian 9.0 “stretch” is just like for Ubuntu 16.04, with one difference.

If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code:

sudo apt-get-y purge vagrant&&\ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb&&\ sudo dpkg-i vagrant *.deb&&\ sudo apt-get-y install build-essential git ruby lxc redir&&\ vagrant plugin install vagrant-lxc&&\ vagrant lxc sudoers

For a step-by-step explanation, follow the Ubuntu instructions above, with the following difference: in “2. Install remaining dependencies”, the command is

1. Install Git for Windows, which installs Git BASH. 2. Install VirtualBox (version >= 5.1.6). 3. Install Vagrant (version 1.8.4-1.8.6, do not use 1.8.7). (Note: While Git BASH is recommended, you may also use Cygwin. If you do, make sure to install default required packages along with git, curl, openssh, and rsync binaries.) After installing, you must run Git BASH as an administrator. Also, you must have hardware virtualization enabled (VT-X or AMD-V) in your computer’s BIOS.

The Zulip code includes symbolic links (symlinks). By default, native Windows symlinks are not enabled in either Git BASH or Cygwin, so you need to do a bit of conﬁguration. You must do this before you clone the Zulip code. In Git for BASH: Open Git BASH as an administrator and run:

If you see true, you are ready for Step 2: Get Zulip Code. Otherwise, if the above command prints false or nothing at all, then symlinks have not been enabled. In Cygwin: Open a Cygwin window as an administrator and do this:

christie@win10 ~ $ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile

Next, close that Cygwin window and open another. If you echo $CYGWIN you should see:

76 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

If you haven’t already created an ssh key and added it to your GitHub account, you should do that now by following these instructions. 1. In your browser, visit https://github.com/zulip/zulip and click the fork button. You will need to be logged in to GitHub to do this. 2. Open Terminal (macOS/Ubuntu) or Git BASH (Windows; must run as an Administrator). 3. In Terminal/Git BASH, clone your fork: git clone [email protected]:YOURUSERNAME/zulip.git

This will create a ‘zulip’ directory and download the Zulip code into it. Don’t forget to replace YOURUSERNAME with your git username. You will see something like: christie@win10 ~ $ git clone [email protected]:YOURUSERNAME/zulip.git Cloning into 'zulip'... remote: Counting objects: 73571, done. remote: Compressing objects: 100% (2/2), done. remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569 Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done. Resolving deltas: 100% (51448/51448), done. Checking connectivity... done. Checking out files: 100% (1912/1912), done.`

Change into the zulip directory and tell vagrant to start the Zulip development environment with vagrant up. christie@win10 ~ $ cd zulip christie@win10 ~/zulip $ vagrant up

The ﬁrst time you run this command it will take some time because vagrant does the following: • downloads the base Ubuntu 14.04 virtual machine image (for macOS and Windows) or container (for Ubuntu) • conﬁgures this virtual machine/container for use with Zulip, • creates a shared directory mapping your clone of the Zulip code inside the virtual machine/container at ~/ zulip

• runs the tools/provision script inside the virtual machine/container, which downloads all required dependencies, sets up the python environment for the Zulip development server, and initializes a default test database. We call this process “provisioning”. You will need an active internet connection during the entire process. (See Specifying a proxy if you need a proxy to access the internet.) vagrant up can fail while provisioning if your Internet connection is unreliable. To retry, you can use vagrant provision (vagrant up will just boot the guest without provisioning after the ﬁrst time). Other common issues are documented in the Troubleshooting & Common Errors section. If that doesn’t help, please visit #provision help in the Zulip development community server for real-time help. On Windows, you will see The system cannot find the path specified. message several times. This is expected behavior and is not an error. Once vagrant up has completed, connect to the development environment with vagrant ssh: christie@win10 ~/zulip $ vagrant ssh

System load: 0.61 Processes: 88 Usage of/: 3.5% of 39.34GB Users logged in:0 Memory usage:7% IP address for eth0: 10.0.2.15 Swap usage:0%

Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud

Congrats, you’re now inside the Zulip development environment! You can conﬁrm this by looking at the command prompt, which starts with (zulip-venv)vagrant@. If it just starts with vagrant@, your provisioning failed and you should look at the troubleshooting section. Next, start the Zulip server:

78 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

2016-05-04 22:20:33,895 INFO: process_fts_updates starting Recompiling templates 2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates done Validating Django models.py... System check identified no issues (0 silenced).

Django version 1.8 Tornado server is running at http://localhost:9993/ Quit the server with CTRL-C. 2016-05-04 18:20:40,716 INFO Tornado loaded0 event queues in 0.001s 2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 seconds Performing system checks...

And ending with something similar to: http://localhost:9994/webpack-dev-server/ webpack result is served from http://localhost:9991/webpack/ content is served from /srv/zulip webpack: bundle is now VALID. 2016-05-06 21:43:29,553 INFO Tornado 31.6% busy over the past 10.6 seconds 2016-05-06 21:43:35,007 INFO Tornado 23.9% busy over the past 16.0 seconds

Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in the browser on your main machine. You should see something like this:

16.4. Step 3: Start the development environment 79 Zulip Documentation, Release 1.4.0

The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in your browser, check your terminal and you should see something like:

˓→(unauth via ?) [04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0 2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth via

˓→?) [04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0 2016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /

80 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

You’ll work by editing ﬁles on your host machine, in the directory where you cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++, etc.). When you save changes they will be synced automatically to the Zulip development environment on the virtual machine/container. Each component of the Zulip development server will automatically restart itself or reload data appropriately when you make changes. So, to see your changes, all you usually have to do is reload your browser. More details on how this works are available below. Don’t forget to read through the code style guidelines for details about how to conﬁgure your editor for Zulip. For example, indentation should be set to 4 spaces rather than tabs.

It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along with every backend request will be printed there. See Logging for further details on the run-dev.py console output.

When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (ma- cOS/Ubuntu) or Git BASH (Windows) in the directory where you cloned Zulip on your main machine. If you’re new to working with Git/GitHub, check out our Git & GitHub Guide.

If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server or running tests, this is probably not because Zulip’s master branch is broken. Instead, this is likely because we’ve recently merged changes to the development environment provisioning process that you need to apply to your development environment. To update your environment, you’ll need to re-provision your vagrant machine using vagrant provision (this just runs tools/provision from your Zulip checkout inside the Vagrant guest); this should complete in about a minute. After provisioning, you’ll want to (re)start the Zulip development server. If you run into any trouble, the #provision help in the Zulip development community server for is a great place to ask for help.

If you ever want to recreate your development environment again from scratch (e.g. to test as change you’ve made to the provisioning process, or because you think something is broken), you can do so using vagrant destroy and then vagrant up. This will usually be much faster than the original vagrant up since the base image is already cached on your machine (it takes about 5 minutes to run with a fast Internet connection).

Any additional programs (e.g. Zsh, emacs, etc.) or conﬁguration that you may have installed in the development environment will be lost when you recreate it. To address this, you can create a script called tools/ custom_provision in your Zulip Git checkout; and place any extra setup commands there. Vagrant will run tools/custom_provision every time you run vagrant provision (or create a Vagrant guest via vagrant up).

To shut down but preserve the development environment so you can use it again later use vagrant halt or vagrant suspend. You can do this from the same Terminal/Git BASH window that is running run-dev.py by pressing ^C to halt the server and then typing exit. Or you can halt vagrant from another Terminal/Git BASH window. From the window where run-dev.py is running:

˓→via ?) ^C KeyboardInterrupt (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip$ exit logout Connection to 127.0.0.1 closed. christie@win10 ~/zulip

christie@win10 ~/zulip $ vagrant suspend ==> default: Saving VM state and suspending execution...

christie@win10 ~/zulip $ vagrant halt ==> default: Attempting graceful shutdown of VM...

When you’re ready to work on Zulip again, run vagrant up. You will also need to connect to the virtual machine with vagrant ssh and re-start the Zulip server:

82 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

Next, read the following to learn more about developing for Zulip: • Git & GitHub Guide • Using the Development Environment • Testing

Below you’ll ﬁnd a list of common errors and their solutions. Most issues are resolved by just provisioning again (via tools/provision.py inside the Vagrant guest or equivalently vagrant provision from outside). If these solutions aren’t working for you or you encounter an issue not documented below, there are a few ways to get further help: • Ask in #provision help in the Zulip development community server, • send a note to the Zulip-devel Google group, or • File an issue. When reporting your issue, please include the following information: • host operating system • installation method (Vagrant or direct) • whether or not you are using a proxy • a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual machine. If you choose to post just the error output, please include the beginning of the error output, not just the last few lines. The output of tools/diagnose run inside the Vagrant guest is also usually helpful.

This is caused by provisioning failing to complete successfully. You can see the errors in var/log/provision. log; it should end with something like this:

The ESC stuff are the terminal color codes that make it show as a nice blue in the terminal, which unfortunately looks ugly in the logs. If you encounter an incomplete /var/log/provision.log file, you need to update your environment. Re- provision your vagrant machine; if the problem persists, please come chat with us (see instructions above) for help. After you provision successfully, you’ll need to exit your vagrant ssh shell and run vagrant ssh again to get the virtualenv setup properly.

The box 'ubuntu/trusty64' could not be found or could not be accessed in the remote catalog. If this is a private box on HashiCorp's Atlas, please verify you're logged in via `vagrant login`. Also, please double-check the name. The expanded URL and error message are shown below: URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]

Then the version of curl that ships with Vagrant is not working on your machine. You are most likely to encounter this error on Windows/Cygwin and macOS. On macOS this error is most likely to occur with Vagrant version 1.8.7 and is a known issue. The solution is to downgrade Vagrant to version 1.8.6 (available here), or to use your system’s version of curl instead of the one that ships with Vagrant: sudo ln-nsf/usr/bin/curl/opt/vagrant/embedded/bin/curl

On Windows/Cygwin, the ﬁx is simple: replace it with the version from Cygwin. First, determine the location of Cygwin’s curl with which curl: christie@win10 ~/zulip $ which curl /usr/bin/curl

Now determine the location of Vagrant with which vagrant: christie@win10 ~/zulip $ which vagrant /cygdrive/c/HashiCorp/Vagrant/bin/vagrant

The path up until /bin/vagrant is what you need to know. In the example above it’s /cygdrive/c/ HashiCorp/Vagrant. Finally, copy Cygwin’s curl to Vagrant embedded/bin directory: christie@win10 ~/zulip $ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/

Now re-run vagrant up and vagrant should be able to fetch the required box ﬁle.

Vagrant was unable to mount VirtualBox shared folders. This is usually because the filesystem"vboxsf" is not available. This filesystem is made available via the VirtualBox Guest Additions and kernel module. Please verify that these guest additions are properly installed in the guest. This is not a bug in Vagrant and is usually caused by a faulty Vagrant box. For context, the command attempted was:

84 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

This is equivalent of running a halt followed by an up (aka rebooting the guest). After this, you can do vagrant provision and vagrant ssh. os.symlink error

==> default: Traceback (most recent call last): ==> default: File"./emoji_dump.py", line 75, in ==> default: ==> default: os.symlink('unicode/{}.png'.format(code_point),'out/ {}.png'.

˓→format(name)) ==> default: OSError ==> default: : ==> default: [Errno 71] Protocol error

Then Vagrant was not able to create a symbolic link. First, if you are using Windows, make sure you have run Git BASH (or Cygwin) as an administrator. By default, only administrators can create symbolic links on Windows. Second, VirtualBox does not enable symbolic links by default. Vagrant starting with version 1.6.0 enables symbolic links for VirtualBox shared folder. You can check to see that this is enabled for your virtual machine with vboxmanage command. Get the name of your virtual machine by running vboxmanage list vms and then print out the custom settings for this virtual machine with vboxmanage getextradata YOURVMNAME enumerate:

christie@win10 ~/zulip $ vboxmanage list vms "zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}

christie@win10 ~/zulip $ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1 Key: supported, Value: false

If you see “command not found” when you try to run VBoxManage, you need to add the VirtualBox directory to your path. On Windows this is mostly likely C:\Program Files\Oracle\VirtualBox\. If vboxmanage enumerate prints nothing, or shows a value of 0 for VBoxInter- nal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable symbolic links by running this command in Terminal/Git BASH/Cygwin:

vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/

default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying...

A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’s BIOS settings. Look for a setting called VT-x (Intel) or (AMD-V). If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system. For further information about troubleshooting vagrant timeout errors see this post.

The vagrant up command basically does the following: • Downloads an Ubuntu image and starts it using a Vagrant provider. • Uses vagrant ssh to connect to that Ubuntu guest, and then runs tools/provision, which has a lot of subcommands that are executed via Python’s subprocess module. These errors mean that one of those subcommands failed. To debug such errors, you can log in to the Vagrant guest machine by running vagrant ssh, which should present you with a standard shell prompt. You can debug interactively by using e.g. cd zulip && ./tools/ provision, and then running the individual subcommands that failed. Once you’ve resolved the problem, you can rerun tools/provision to proceed; the provisioning system is designed to recover well from failures. The zulip provisioning system is generally highly reliable; the most common cause of issues here is a poor network connection (or one where you need a proxy to access the Internet and haven’t conﬁgured the development environment to use it. Once you’ve provisioned successfully, you’ll get output like this:

Zulip development environment setup succeeded! (zulip-venv) vagrant@vagrant-base-trusty-amd64:~/zulip$

If the (zulip-venv) part is missing, this is because your installation failed the ﬁrst time before the Zulip virtualenv was created. You can ﬁx this by just closing the shell and running vagrant ssh again, or using source /srv/ zulip-venv/bin/activate. Finally, if you encounter any issues that weren’t caused by your Internet connection, please report them! We try hard to keep Zulip development environment provisioning free of bugs. pip install fails during vagrant up on Ubuntu

Likely causes are: 1. Networking issues 2. Insufﬁcient RAM. Check whether you’ve allotted at least two gigabytes of RAM, which is the minimum Zulip requires. If not, go to your VM settings and increase the RAM, then restart the VM.

86 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0 npm install errors

The tools/provision script may encounter an error related to npm install that looks something like:

==> default:+ npm install ==> default: Traceback (most recent call last): ==> default: File"/srv/zulip/tools/provision", line 195, in ==> default: ==> default: sys.exit(main()) ==> default: File"/srv/zulip/tools/provision", line 191, in main ==> default: ==> default: run(["npm","install"]) ==> default: File"/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run ==> default: ==> default: raise subprocess.CalledProcessError(rc, args) ==> default: subprocess ==> default:. ==> default: CalledProcessError ==> default: : ==> default: Command'['npm','install']' returned non-zero exit status 34 The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed. The output for this command should be in the log above. Please read the output to determine what went wrong.

Usually this error is not fatal. Try connecting to the development environment and re-trying the command from withing the virtual machine: christie@win10 ~/zulip $ vagrant ssh (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ npm install npm WARN optional Skipping failed optional dependency /chokidar/fsevents: npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.

These are just warnings so it is okay to proceed and start the Zulip server. vagrant-lxc errors

When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.

/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_

˓→by' for nil:NilClass (NoMethodError) from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_

˓→plugin' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin' from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run' from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run' from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in

˓→execute' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in èach' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in èxecute' from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in èxecute' from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in èxecute' from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli' from /usr/bin/vagrant:173:in `

And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of the issue, which should be ﬁxed when Vagrant 1.8.2 is released. In the meantime, read this post for how to create and apply the patch. It will look something like this: christie@xenial:~ $ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch patching file bundler.rb

Permissions errors when running the test suite in LXC

See “Possible testing issues”.

Specifying a proxy

If you need to use a proxy server to access the Internet, you will need to specify the proxy settings before running Vagrant up. First, install the Vagrant plugin vagrant-proxyconf: vagrant plugin install vagrant-proxyconf.

Then create ~/.zulip-vagrant-config and add the following lines to it (with the appropriate values in it for your proxy):

HTTP_PROXY http://proxy_host:port HTTPS_PROXY http://proxy_host:port NO_PROXY localhost,127.0.0.1,.example.com

Now run vagrant up in your terminal to install the development server. If you ran vagrant up before and failed, you’ll need to run vagrant destroy ﬁrst to clean up the failed installation.

88 Chapter 16. Vagrant environment setup tutorial Zulip Documentation, Release 1.4.0

You can also change the port on the host machine that Vagrant uses by adding to your ~/. zulip-vagrant-config ﬁle. E.g. if you set:

HOST_PORT 9971

(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your development server. If you’d like to be able to connect to your development environment from other machines than the VM host, you can manually set the host IP address in the ‘~/.zulip-vagrant-conﬁg’ ﬁle as well. For example, if you set:

HOST_IP_ADDR 0.0.0.0

(and restart the Vagrant guest), your host IP would be 0.0.0.0, a special value for the IP address that means any IP address can connect to your development server.

16.8. Specifying a proxy 89 Zulip Documentation, Release 1.4.0

90 Chapter 16. Vagrant environment setup tutorial CHAPTER 17

Zulip development environment setup without Vagrant

Contents: • Installing directly on Ubuntu • Installing manually on Linux • Using Docker (experimental)

Installing directly on Ubuntu

Start by cloning this repository: git clone https://github.com/zulip/zulip.git If you’d like to install a Zulip development environment on a computer that’s already running Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial, you can do that by just running:

# From a clone of zulip.git ./tools/provision source/srv/zulip-venv/bin/activate ./tools/run-dev.py # starts the development server

Note that there is no supported uninstallation process without Vagrant (with Vagrant, you can just do vagrant destroy to clean up the development environment). Once you’ve done the above setup, you can pick up the documentation on using the Zulip development environment, ignoring the parts about vagrant (since you’re not using it).

Installing manually on Linux

• Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental)

91 Zulip Documentation, Release 1.4.0

• OpenBSD 5.8 (experimental) • Fedora/CentOS common steps • Steps for all systems If you really want to install everything manually, the below instructions should work. Install the following non-Python dependencies: • libfﬁ-dev — needed for some Python extensions • postgresql 9.1 or later — our database (client, server, headers) • nodejs 0.10 (and npm) • memcached (and headers) • rabbitmq-server • libldap2-dev • python-dev • python3-dev • python-virtualenv • redis-server — rate limiting • tsearch-extras — better text search • libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs

On Debian or Ubuntu systems:

Using the ofﬁcial Ubuntu repositories, PGroonga PPA and tsearch-extras deb package:

Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python-dev \ python3-dev python-virtualenv hunspell-en-us nodejs \ nodejs-legacy npm git yui-compressor puppet gettext postgresql

# If using Ubuntu, install PGroonga from its PPA sudo add-apt-repository-ys ppa:groonga/ppa sudo apt-get update # On 14.04 sudo apt-get install postgresql-9.3-pgroonga # On 16.04 sudo apt-get install postgresql-9.5-pgroonga

# If using Debian, follow the instructions here: http://pgroonga.github.io/install/

˓→debian.html

# Next, install Zulip's tsearch-extras postgresql extension # If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras: cd zulip sudo apt-add-repository-ys ppa:tabbott/zulip sudo apt-get update # On 14.04

92 Chapter 17. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.4.0

sudo apt-get install postgresql-9.3-tsearch-extras # On 16.04 sudo apt-get install postgresql-9.5-tsearch-extras

# Otherwise, you can download a .deb directly # If on 12.04 or wheezy: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-

˓→extras_0.1.2_amd64.deb sudo dpkg-i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb

# If on 14.04: wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3-

˓→tsearch-extras_0.1.3_amd64.deb sudo dpkg-i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb

# If on 15.04 or jessie: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-

˓→extras_0.1_amd64.deb sudo dpkg-i postgresql-9.4-tsearch-extras_0.1_amd64.deb

# If on 16.04 or stretch wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5-

˓→tsearch-extras_0.2_amd64.deb sudo dpkg-i postgresql-9.5-tsearch-extras_0.2_amd64.deb

Alternatively, you can always build the package from tsearch-extras git. Now continue with the All Systems instructions below.

Using the ofﬁcial Zulip PPA (for 14.04 Trusty or 16.04 Xenial):

Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo add-apt-repository ppa:tabbott/zulip sudo apt-get update sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python-dev \ hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \ puppet gettext tsearch-extras

Now continue with the All Systems instructions below.

On Fedora 22 (experimental):

These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo dnf install libffi-devel memcached rabbitmq-server \ openldap-devel python-devel redis postgresql-server \ postgresql-devel postgresql libmemcached-devel freetype-devel \ nodejs npm yuicompressor closure-compiler gettext

Now continue with the Common to Fedora/CentOS instructions below.

17.2. Installing manually on Linux 93 Zulip Documentation, Release 1.4.0

On CentOS 7 Core (experimental):

These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git

# Add user zulip to the system (not necessary if you configured zulip # as the administrator user during the install process of CentOS 7). useradd zulip

# Create a password for zulip user passwd zulip

# Allow zulip to sudo visudo # Add this line after line `root ALL=(ALL) ALL` zulip ALL=(ALL) ALL

# Switch to zulip user su zulip

# Enable EPEL 7 repo so we can install rabbitmq-server, redis and # other dependencies sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.

˓→rpm

# Install dependencies sudo yum install libffi-devel memcached rabbitmq-server openldap-devel \ python-devel redis postgresql-server postgresql-devel postgresql \ libmemcached-devel wget python-pip openssl-devel freetype-devel \ libjpeg-turbo-devel zlib-devel nodejs yuicompressor \ closure-compiler gettext

# We need these packages to compile tsearch-extras sudo yum groupinstall"Development Tools"

# clone Zulip's git repo and cd into it cd&& git clone https://github.com/zulip/zulip&& cd zulip/

## NEEDS TESTING: The next few DB setup items may not be required at all. # Initialize the postgres db sudo postgresql-setup initdb

# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf

# Change these lines: host all all 127.0.0.1/32 ident host all all::1/128 ident # to this: host all all 127.0.0.1/32 md5 host all all::1/128 md5

Now continue with the Common to Fedora/CentOS instructions below.

On OpenBSD 5.8 (experimental):

These instructions are experimental and may have bugs; patches welcome!

94 Chapter 17. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.4.0

Start by cloning this repository: git clone https://github.com/zulip/zulip.git doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \ memcached node libmemcached py-Pillow py-cryptography py-cffi

# Get tsearch_extras and build it (using a modified version which # aliases int4 on OpenBSD): git clone https://github.com/blablacio/tsearch_extras cd tsearch_extras gmake&& sudo gmake install

# Point environment to custom include locations and use newer GCC # (needed for Node modules): export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl" export CXX=eg++

# Create tsearch_data directory: sudo mkdir/usr/local/share/postgresql/tsearch_data

# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/local/share/postgresql/tsearch_data/english.stop sudo touch/usr/local/share/postgresql/tsearch_data/en_us.dict sudo touch/usr/local/share/postgresql/tsearch_data/en_us.affix

Finally continue with the All Systems instructions below.

Common to Fedora/CentOS instructions

Start by cloning this repository: git clone https://github.com/zulip/zulip.git

# Build and install postgres tsearch-extras module wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.

˓→tar.gz tar xvzf tsearch-extras_0.1.3.tar.gz cd ts2 make sudo make install

# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/share/pgsql/tsearch_data/english.stop sudo touch/usr/share/pgsql/tsearch_data/en_us.dict sudo touch/usr/share/pgsql/tsearch_data/en_us.affix

# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf

# Add this line before the first uncommented line to enable password # auth: host all all 127.0.0.1/32 md5

# Start the services sudo systemctl start redis memcached rabbitmq-server postgresql

17.2. Installing manually on Linux 95 Zulip Documentation, Release 1.4.0

# Enable automatic service startup after the system startup sudo systemctl enable redis rabbitmq-server memcached postgresql

Finally continue with the All Systems instructions below.

All Systems:

Make sure you have followed the steps speciﬁc for your platform: • Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental) • OpenBSD 5.8 (experimental) • Fedora/CentOS For managing Zulip’s python dependencies, we recommend using virtualenvs. You must create two virtualenvs. One for Python 2 and one for Python 3. You must also install appropriate python packages in them. You should either install the virtualenvs in /srv, or put symlinks to them in /srv. If you don’t do that, some scripts might not work correctly. You can run tools/setup/setup_venvs.py to do this. This script will create two virtualenvs - /srv/zulip-venv and /srv/zulip-py3-venv. If you want to do it manually, here are the steps: sudo virtualenv /srv/zulip-venv -p python2 # Create a python2 virtualenv sudo chown -R `whoami`:`whoami` /srv/zulip-venv source /srv/zulip-venv/bin/activate # Activate python2 virtualenv pip install --upgrade pip # upgrade pip itself because older versions have known

˓→issues pip install --no-deps -r requirements/py2_dev.txt # install python packages required

˓→for development sudo virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenv sudo chown -R `whoami`:`whoami` /srv/zulip-py3-venv source /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv pip install --upgrade pip # upgrade pip itself because older versions have known

˓→issues pip install --no-deps -r requirements/py3_dev.txt # install python packages required

˓→for development

Now run these commands:

./tools/install-mypy sudo mkdir /srv/zulip-emoji-cache sudo chown -R `whoami`:`whoami` /srv/zulip-emoji-cache ./tools/setup/emoji/download-emoji-data ./tools/setup/emoji/build_emoji ./tools/setup/build_pygments_data.py ./scripts/setup/generate_secrets.py --development if [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/

˓→tsearch_data/

96 Chapter 17. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.4.0

else sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/ ˓→9.*/tsearch_data/ fi ./scripts/setup/configure-rabbitmq ./tools/setup/postgres-init-dev-db ./tools/do-destroy-rebuild-database ./tools/setup/postgres-init-test-db ./tools/do-destroy-rebuild-test-database ./manage.py compilemessages sudo ./scripts/lib/install-node npm install

To start the development server:

./tools/run-dev.py

. . . and visit http://localhost:9991/.

Proxy setup for by-hand installation

If you are building the development environment on a network where a proxy is required to access the Internet, you will need to set the proxy in the environment as follows: • On Ubuntu, set the proxy environment variables using: export https_proxy=http://proxy_host:port export http_proxy=http://proxy_host:port

• And set the npm proxy and https-proxy using: npm config set proxy http://proxy_host:port npm config set https-proxy http://proxy_host:port

Using Docker (experimental)

Start by cloning this repository: git clone https://github.com/zulip/zulip.git The docker instructions for development are experimental, so they may have bugs. If you try them and run into any issues, please report them! You can also use Docker to run a Zulip development environment. First, you need to install Docker in your development machine following the instructions. Some other interesting links for somebody new in Docker are: • Get Started • Understand the architecture • Docker run reference • Dockerﬁle reference Then you should create the Docker image based on Ubuntu Linux, ﬁrst go to the directory with the Zulip source code: docker build-t user/zulipdev.

17.3. Using Docker (experimental) 97 Zulip Documentation, Release 1.4.0

Commit and tag the provisioned images. The below will install Zulip’s dependencies: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash $ /bin/bash sudo chown -R zulip:zulip /srv/zulip $ /bin/bash /srv/zulip/tools/provision --docker docker ps -af ancestor=user/zulipdev docker commit -m "Zulip installed" user/zulipdev:v2

Now you can run the docker server with: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \ /srv/zulip/tools/start-dockers

You’ll want to read the guide for Zulip development to understand how to use the Zulip development. Note that start-dockers automatically runs tools/run-dev.py inside the container; you can then visit http://localhost:9991 to connect to your new Zulip Docker container. To view the container’s run-dev.py console logs to get important debugging information (and e.g. outgoing emails) printed by the Zulip development environment, you can use: docker logs--follow

To restart the server use: docker ps docker restart

To stop the server use: docker ps docker kill

If you want to connect to the Docker instance to run commands (e.g. build a release tarball), you can use: docker ps docker exec -it /bin/bash $ source /home/zulip/.bash_profile $ $ exit

If you want to run all the tests you need to start the servers ﬁrst, you can do it with: docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash $ tools/test-all-docker

You can modify the source code in your development machine and review the results in your browser. Currently, the Docker workﬂow is substantially less convenient than the Vagrant workﬂow and less documented; please contribute to this guide and the Docker tooling if you are using Docker to develop Zulip!

98 Chapter 17. Zulip development environment setup without Vagrant CHAPTER 18

Using the Development Environment

Once the development environment is running, you can visit http://localhost:9991/ in your browser. By default, the development server homepage just shows a list of the users that exist on the server and you can login as any of them by just clicking on a user. This setup saves time for the common case where you want to test something other than the login process; to test the login process you’ll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTION case of zproject/settings.py from zproject.backends.DevAuthBackend to use the auth method(s) you’d like to test. While developing, it’s helpful to watch the run-dev.py console output, which will show any errors your Zulip development server encounters. To manually query the Postgres database, run psql zulip for an interactive console. When you make a change, here’s a guide for what you need to do in order to see your change take effect in Develop- ment: • If you change JavaScript, CSS, or Jinja2 backend templates (under templates/), you’ll just need to reload the browser window to see changes take effect. The Handlebars frontend HTML templates (static/ templates) are automatically recompiled by the tools/compile-handlebars-templates job, which runs as part of tools/run-dev.py. • If you change Python code used by the main Django/Tornado server processes, these services are run on top of Django’s manage.py runserver which will automatically restart the Zulip Django and Tornado servers whenever you save changes to Python code. You can watch this happen in the run-dev.py console to make sure the backend has reloaded. • The Python queue workers will also automatically restart when you save changes. However, you may need to ctrl-C and then restart run-dev.py manually if a queue worker has crashed. • If you change the database schema, you’ll need to use the standard Django migrations process to create and then run your migrations; see the new feature tutorial for an example. Additionally you should check out the detailed testing docs for how to run the tests properly after doing a migration. (In production, everything runs under supervisord and thus will restart if it crashes, and upgrade-zulip will take care of running migrations and then cleanly restaring the server for you).

99 Zulip Documentation, Release 1.4.0

100 Chapter 18. Using the Development Environment CHAPTER 19

Developing on a remote machine

The Zulip developer environment works well on remote virtual machines. This can be a good alternative for those with poor network connectivity or who have limited storage/memory on their local machines. We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04, with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.

Connecting to the remote environment

The best way to connect to your server is using the command line tool ssh. • On macOS and Linux/UNIX, ssh is a part of Terminal. • On Windows, ssh comes with Bash for Git. Open Terminal or Bash for Git, and connect with the following:

$ ssh username@host

If you have poor internet connectivity, we recommend using Mosh as it is more reliable over slow or unreliable networks.

Setting up the development environment

After you have connected to your remote server, you need to install the development environment. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.

101 Zulip Documentation, Release 1.4.0

Running the development server

Once you have set up the development environment, you can start up the development instance of Zulip with the following command in the directory where you cloned Zulip:

./tools/run-dev.py--interface=''

This will start up the Zulip server on port 9991. You can then navigate to http://:9991 and you should see something like this screenshot of the Zulip development environment:

You can port forward using ssh instead of running the development environment on an exposed interface. For more information, see Using the development environment.

102 Chapter 19. Developing on a remote machine Zulip Documentation, Release 1.4.0

Making changes to code on your remote development server

To see changes on your remote development server, you need to do one of the following: • Edit locally: Clone Zulip code to your computer and then use your favorite editor to make changes. When you want to see changes on your remote Zulip development instance, sync with Git. • Edit remotely: Edit code directly on your remote Zulip development instance using a Web-based IDE (recommended for beginners) or a command line editor.

Editing locally

If you want to edit code locally install your favorite text editor. If you don’t have a favorite, here are some suggestions: • atom • emacs • vim • spacemacs • sublime Next, follow our Git and GitHub Guide to clone and conﬁgure your fork of zulip on your local computer. Once you have cloned your code locally, you can get to work.

Syncing changes

The easiest way to see your changes on your remote development server is to push them to GitHub and them fetch and merge them from the remote server. For more detailed instructions about how to do this, see our Git & GitHub Guide. In brief, the steps are as follows. On your local computer: 1. Open Terminal (macOS/Linux) or Git for BASH. 2. Change directory to where you cloned Zulip (e.g. cd zulip). 3. Use git add and git commit to stage and commit your changes (if you haven’t already). 4. Push your commits to GitHub with git push origin branchname. Be sure to replace branchname with the name of your actual feature branch. Once git push has completed successfully, you are ready to fetch the commits from your remote development instance: 1. In Terminal or Git BASH, connect to your remote development instance with ssh user@host. 2. Change to the zulip directory (e.g., cd zulip). 3. Fetch new commits from GitHub with git fetch origin. 4. Change to the branch you want to work on with git checkout branchname. 5. Merge the new commits into your branch with git merge origin/branchname.

19.4. Making changes to code on your remote development server 103 Zulip Documentation, Release 1.4.0

Editing remotely

Web-based IDE

If you are relatively new to working on the command line, or just want to get started working quickly, we recommend web-based IDE Codeanywhere. To setup Codeanywhere for Zulip: 1. Create a Codeanywhere account and log in. 2. Create a new SFTP-SSH project. Use Public key for authentication. 3. Click GET YOUR PUBLIC KEY to get the new public key that Codeanywhere generates when you create a new project. Add this public key to ~/.ssh/authorized_keys on your remote development instance. 4. Once you’ve added the new public key to your remote development instance, click CONNECT.

Now your workspace should look similar this:

Command line editors

Another way to edit directly on the remote development server is with a command line text editor on the remote machine. Two editors often available by default on Linux systems are: • Nano: A very simple, beginner-friendly editor. However, it lacks a lot of features useful for programming, such as syntax highlighting, so we only recommended it for quick edits to things like conﬁguration ﬁles. Launch by running command nano . Exit by pressing control-X.

104 Chapter 19. Developing on a remote machine Zulip Documentation, Release 1.4.0

• Vim: A very powerful editor that can take a while to learn. Launch by running vim . Quit Vim by pressing escape, typing :q, and then pressing return. Vim comes with a program to learn it called vimtutor (just run that command to start it). Other options include: • emacs • spacemacs

Next steps

Next, read the following to learn more about developing for Zulip: • Git & GitHub Guide • Using the Development Environment • Testing

19.4. Making changes to code on your remote development server 105 Zulip Documentation, Release 1.4.0

106 Chapter 19. Developing on a remote machine CHAPTER 20

Writing a new integration

Integrations are one of the most important parts of a group chat tool like Zulip, and we are committed to making integrating with Zulip and getting you integration merged upstream so everyone else can beneﬁt from it as easy as possible while maintaining the high quality of the Zulip integrations library. On this page you’ll ﬁnd: • An overview of the different types of integrations possible with Zulip. • General advice for writing integrations. • Details about writing webhook integrations. • Details about writing Python script and plugin integrations. • A guide to documenting your integration. A detailed walkthrough of a simple “Hello World” integration can be found in the webhook walkthrough. Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up with any tips or tools that help writing integration, please email [email protected], open an issue, or submit a pull request to share your ideas!

Types of integrations

We have several different ways that we integrate with 3rd party products, ordered here by which types we prefer to write: 1. Webhook integrations (examples: Freshdesk, GitHub), where the third-party service supports posting content to a particular URI on our site with data about the event. For these, you usually just need to create a new python package in the zerver/webhooks/ directory. You can easily ﬁnd recent commits adding new integrations to crib from via git log zerver/webhooks/. 2. Python script integrations (examples: SVN, Git), where we can get the service to call our integration (by shelling out or otherwise), passing in the required data. Our preferred model for these is to ship these integrations in our API release tarballs (by writing the integration in api/integrations).

107 Zulip Documentation, Release 1.4.0

3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user needs to install a plugin into their existing software. These are often more work, but for some products are the only way to integrate with the product at all.

General advice

• Consider using our Zulip markup to make the output from your integration especially attractive or useful (e.g. emoji, markdown emphasis, @-mentions, or !avatar(email)). • Use topics effectively to ensure sequential messages about the same thing are threaded together; this makes for much better consumption by users. E.g. for a bug tracker integration, put the bug number in the topic for all messages; for an integration like Nagios, put the service in the topic. • Integrations that don’t match a team’s workflow can often be uselessly spammy. Give careful thought to providing options for triggering Zulip messages only for certain message types, certain projects, or sending different messages to different streams/topics, to make it easy for teams to configure the integration to support their workflow. • Consistently capitalize the name of the integration in the documentation and the Client name the way the vendor does. It’s OK to use all-lower-case in the implementation. • Sometimes it can be helpful to contact the vendor if it appears they don’t have an API or webhook we can use – sometimes the right API is just not properly documented. • A helpful tool for testing your integration is UltraHook, which allows you to receive webhook calls via your local Zulip development environment. This enables you to do end-to-end testing with live data from the service you’re integrating and can help you spot why something isn’t working or if the service is using custom HTTP headers.

Webhook integrations

A webhook allows a third-party service to push data to you when something happens. It’s different from making a REST API call, where you send a request to the service’s API and wait for a response. With a webhook, the third- party service sends you an HTTP POST when it has something for you. Your webhook integration defines the URI the service uses to communicate with Zulip, and handles that incoming data. New Zulip webhook integrations can take just a few hours to write, including tests and documentation, if you use the right process. Here’s how we recommend doing it: • First, use http://requestb.in/ or a similar site to capture an example webhook payload from the service you’re integrating. You can use these captured payloads to create a set of test fixtures for your integration under zerver/webhooks/mywebhook/fixtures/. • Then write a draft webhook handler under zerver/webhooks/; there are a lot of examples in that directory. We recommend templating off a short one (like stash or zendesk), since the longer ones usually just have more complex parsing which can obscure what’s common to all webhook integrations. In addition to writing the integration itself, you’ll need to create Integration object and add it to WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py; search for webhook in that file to find the existing ones (and please add yours in the alphabetically correct place). • Then write a test for your fixture in the tests.py file in the zerver/webhooks/mywebhook directory. You can now iterate on debugging the tests and webhooks handler until they work, all without ever needing to post directly from the service you’re integrating with to your Zulip development machine. You can run just the tests for one integration like this:

108 Chapter 20. Writing a new integration Zulip Documentation, Release 1.4.0

test-backend zerver/webhooks/pagerduty/

See this guide for more details on the Zulip test runner. • Once you’ve gotten your webhook working and passing a test, capture payloads for the other common types of posts the service’s webhook will make, and add tests for them; usually this part of the process is pretty fast. Webhook integration tests should all use ﬁxtures (as opposed to contacting the service), since otherwise the tests can’t run without Internet access and some sort of credentials for the service. • Finally, write documentation for the integration; there’s a detailed guide below. See the “Hello World” example in the webhook walkthrough for a detailed look at how to write a simple webhook.

Files that need to be created

Select a name for your webhook and use it consistently. The examples below are for a webhook named ‘MyWebHook’. • static/images/integrations/logos/mywebhook.png: An image to represent your integration in the user interface. Generally this Should be the logo of the platform/server/product you are integrating. See Documenting your integration for details. • static/images/integrations/mywebbook/001.png: A screen capture of your integration for use in the user interface. You can add as many images as needed to effectively document your webhook integration. See Documenting your integration for details. • zerver/webhooks/mywebhook/fixtures/messagetype.json: Sample json payload data used by tests. Add one fixture file per type of message supported by your integration. See Testing and writing tests for details. • zerver/webhooks/mywebhook/__init__.py: Empty file that is obligatory part of every python package. Remember to git add it. • zerver/webhooks/mywebhook/view.py: Includes the main webhook integration function including any needed helper functions. • zerver/webhooks/mywebhook/tests.py: Add tests for your webbook. See Testing and writing tests for details. • zerver/webhooks/mywebhook/doc.html: Add end-user documentation. See Documenting your integration for details.

Files that need to be updated

• zerver/lib/integrations.py: Add your integration to WEBHOOK_INTEGRATIONS to register it. This will automatically register a url for the webhook of the form api/v1/external/mywebhook and associate with the function called api_mywebhook_webhook in zerver/webhooks/mywebhook/ view.py.

Python script and plugin integrations

For plugin integrations, usually you will need to consult the documentation for the third party software in order to learn how to write the integration. But we have a few notes on how to do these: • You should always send messages by POSTing to URLs of the form https://zulip.example.com/v1/ messages/.

20.4. Python script and plugin integrations 109 Zulip Documentation, Release 1.4.0

• We usually build Python script integration with (at least) 2 files: zulip_foo_config.py containing the configuration for the integration including the bots’ API keys, plus a script that reads from this configuration to actually do the work (that way, it’s possible to update the script without breaking users’ configurations). • Be sure to test your integration carefully and document how to install it (see notes on documentation below). • You should specify a clear HTTP User-Agent for your integration. The user agent should at a minimum identify the integration and version number, separated by a slash. If possible, you should collect platform information and include that in ()s after the version number. Some examples of ideal UAs are:

ZulipDesktop/0.7.0 (Ubuntu; 14.04) ZulipJenkins/0.1.0 (Windows; 7.2) ZulipMobile/0.5.4 (Android; 4.2; maguro)

Documenting your integration

Every Zulip integration must be documented in zerver/webhooks/mywebhook/doc.html (or templates/ zerver/integrations/.html, for non-webhook integrations). Usually, this involves a few steps: • Add text explaining all of the steps required to setup the integration, including what URLs to use, etc. If there are any screens in the product involved, take a few screenshots with the input fields filled out with sample values in order to make the instructions really easy to follow. For the screenshots, use a bot with a name like “GitHub Bot”, an email address for the bot like [email protected], and an obviously fake API key like abcdef123456790. • Make sure you’ve added your integration to zerver/lib/integrations.py; this results in your integration appearing on the /integrations page. • You’ll need to add a SVG graphic of your integration’s logo under the static/images/integrations/ logos/.svg, where is the name of the integration, all in lower case; you can usually find them in the product branding or press page. Make sure to optimize the SVG graphic by running svgo -f path-to-file. If you cannot find a SVG graphic of the logo, please find and include a PNG image of the logo instead. • Finally, generate a message sent by the integration and take a screenshot of the message to provide an example message in the documentation. If your new integration is a webhook integration, you can generate such a message from your test fixtures using send_webhook_fixture_message:

./manage.py send_webhook_fixture_message \ --fixture=zerver/webhooks/pingdom/fixtures/imap_down_to_up.json \ '--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'

When generating the screenshot of a sample message, give your test bot a nice name like “GitHub Bot”, use the project’s logo as the bot’s avatar, and take the screenshots showing the stream/topic bar for the message, not just the message body. When writing documentation for your integration, be sure to use the {{ external_api_uri }} template variable, so that your integration documentation will provide the correct URL for whatever server it is deployed on. If special conﬁguration is required to set the SITE variable, you should document that too, inside an {% if api_site_required %} check.

110 Chapter 20. Writing a new integration CHAPTER 21

Webhook walkthrough

Below explains each part of a simple webhook integration, called Hello World. This webhook sends a “hello” message to the test stream and includes a link to the Wikipedia article of the day, which it formats from json data it receives in the http request. Use this walkthrough to learn how to write your ﬁrst webhook integration.

Step 0: Create ﬁxtures

The first step in creating a webhook is to examine the data that the service you want to integrate will be sending to Zulip. You can use http://requestb.in or a similar tool to capture webhook payload(s) from the service you are integrating. Examining this data allows you to do two things: 1. Determine how you will need to structure your webhook code, including what message types your integration should support and how; and, 2. Create fixtures for your webhook tests. A test fixture is a small file containing test data, one for each test. Fixtures enable the testing of webhook integration code without the need to actually contact the service being integrated. Because Hello World is a very simple webhook that does one thing, it requires only one fixture, zerver/ webhooks/helloworld/fixtures/hello.json:

{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe", }

When writing your own webhook integration, you’ll want to write a test function for each distinct message condition your webhook supports. You’ll also need a corresponding ﬁxture for each of these tests. Depending on the type of data the 3rd party service sends, your ﬁxture may contain JSON, URL encoded text, or some other kind of data. See Step 4: Create tests or Testing for further details.

111 Zulip Documentation, Release 1.4.0

Step 1: Initialize your webhook python package

In the zerver/webhooks/ directory, create new subdirectory that will contain all of corresponding code. In our example it will be helloworld. The new directory will be a python package, so you have to create an empty __init__.py ﬁle in that directory via e.g. touch zerver/webhooks/helloworld/__init__.py.

Step 2: Create main webhook code

The majority of the code for your webhook integration will be in a single python ﬁle, zerver/webhooks/ mywebhook/view.py. The Hello World integration is in zerver/webhooks/helloworld/view.py: from __future__ import absolute_import from django.utils.translation import ugettext as _ from zerver.lib.actions import check_send_message from zerver.lib.response import json_success, json_error from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view from zerver.lib.validator import check_dict, check_string from zerver.models import Client, UserProfile from django.http import HttpRequest, HttpResponse from typing import Dict, Any, Iterable, Optional, Text

@api_key_only_webhook_view('HelloWorld') @has_request_variables def api_helloworld_webhook(request, user_profile, payload=REQ(argument_type='body'), stream=REQ(default='test'), topic=REQ(default='Hello World')): # type: (HttpRequest, UserProfile, Dict[str, Iterable[Dict[str, Any]]], Text,

˓→Optional[Text]) -> HttpResponse

# construct the body of the message body='Hello! I am happy to be here! :smile:'

# try to add the Wikipedia article of the day # return appropriate error if not successful try: body_template=' \nThe Wikipedia featured article for today is **[{featured_ ˓→title}]({featured_url})**' body+= body_template.format( **payload) except KeyError as e: return json_error(_("Missing key {} in JSON").format(str(e)))

# send the message check_send_message(user_profile, request.client,'stream', [stream], topic, body)

# return json result return json_success()

The above code imports the required functions and deﬁnes the main webhook function api_helloworld_webhook, decorating it with api_key_only_webhook_view and

112 Chapter 21. Webhook walkthrough Zulip Documentation, Release 1.4.0

has_request_variables. The has_request_variables decorator allows you to access request variables with REQ(). You can find more about REQ and request variables in Writing views. You must pass the name of your webhook to the api_key_only_webhook_view decorator so your webhook can access the user_profile and request.client (Zulip’s analogue of UserAgent) fields from the request. Here we have used HelloWorld. To be consistent with Zulip code style, use the name of the product you are integrating in camel case, spelled as the product spells its own name (except always first letter upper-case). The api_key_only_webhook_view decorator indicates that the 3rd party service will send the authorization as an API key in the query parameters. If your service uses HTTP Basic authentication, you would instead use the authenticated_rest_api_view decorator. You should name your webhook function as such api_webhookname_webhook where webhookname is the name of your webhook and is always lower-case. At minimum, the webhook function must accept request (Django HttpRequest object), and user_profile (Zulip’s user object). You may also want to define additional parameters using the REQ object. In the example above, we have defined payload which is populated from the body of the http request, stream with a default of test (available by default in the Zulip development environment), and topic with a default of Hello World. If your webhook uses a custom stream, it must exist before a message can be created in it. (See Step 4: Create tests for how to handle this in tests.) The line that begins # type is a mypy type annotation. See this page for details about how to properly annotate your webhook functions. In the body of the function we define the body of the message as Hello! I am happy to be here! :smile:. The :smile: indicates an emoji. Then we append a link to the Wikipedia article of the day as provided by the json payload. If the json payload does not include data for featured_title and featured_url we catch a KeyError and use json_error to return the appropriate information: a 400 http status code with relevant details. Then we send a public (stream) message with check_send_message which will validate the message and then send it. Finally, we return a 200 http status with a JSON format success message via json_success().

Step 3: Create an api endpoint for the webhook

In order for a webhook to be externally available, it must be mapped to a url. This is done in zerver/lib/ integrations.py. Look for the lines beginning with:

WEBHOOK_INTEGRATIONS=[

And you’ll ﬁnd the entry for Hello World:

WebhookIntegration('helloworld', display_name='Hello World'),

This tells the Zulip api to call the api_helloworld_webhook function in zerver/webhooks/ helloworld/view.py when it receives a request at /api/v1/external/helloworld. This line also tells Zulip to generate an entry for Hello World on the Zulip integrations page using static/images/ integrations/logos/helloworld.png as its icon. At this point, if you’re following along and/or writing your own Hello World webhook, you have written enough code to test your integration.

21.4. Step 3: Create an api endpoint for the webhook 113 Zulip Documentation, Release 1.4.0

First, get an API key from the Your bots section of your Zulip user’s Settings page. If you haven’t created a bot already, you can do that there. Then copy its API key and replace the placeholder “” in the examples with your real key. This is how Zulip knows the request is from an authorized user. Now you can test using Zulip itself, or curl on the command line. Using manage.py from within the Zulip development environment:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/webhooks/helloworld/fixtures/hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key='

After which you should see something similar to:

2016-07-07 15:06:59,187 INFO 127.0.0.1 POST 200 143ms (mem:6ms/13) (md:

˓→43ms/1) (db: 20ms/9q) (+start: 147ms)/api/v1/external/helloworld (helloworld-

˓→[email protected] via ZulipHelloWorldWebhook)

Using curl: curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn

˓→Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://

˓→localhost:9991/api/v1/external/helloworld\?api_key\=

After which you should see:

{"msg":"","result":"success"}

Using either method will create a message in Zulip:

Step 4: Create tests

Every webhook integration should have a corresponding test ﬁle: zerver/webhooks/mywebhook/tests.py. The Hello World integration’s tests are in zerver/webhooks/helloworld/tests.py You should name the class HookTests and have it inherit from the base class WebhookTestCase. For our HelloWorld webhook, we name the test class HelloWorldHookTests: class HelloWorldHookTests(WebhookTestCase): STREAM_NAME='test' URL_TEMPLATE="/api/v1/external/helloworld?&api_key= {api_key}" FIXTURE_DIR_NAME='helloworld'

# Note: Include a test function per each distinct message condition your

˓→integration supports def test_hello_message(self): # type: () -> None expected_subject=u"Hello World"; expected_message=u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/ ˓→Marilyn_Monroe)**";

114 Chapter 21. Webhook walkthrough Zulip Documentation, Release 1.4.0

# use fixture named helloworld_hello self.send_and_test_stream_message('hello', expected_subject, expected_message, content_type="application/x-www-form-

˓→urlencoded")

def get_body(self, fixture_name): # type: (Text) -> Text return self.fixture_data("helloworld", fixture_name, file_type="json")

In the above example, STREAM_NAME, URL_TEMPLATE, and FIXTURE_DIR_NAME refer to class attributes from the base class, WebhookTestCase. These are needed by the helper function send_and_test_stream_message to determine how to execute your test. STREAM_NAME should be set to your default stream. If it doesn’t exist, send_and_test_stream_message will create it while executing your test. If your test expects a stream name from a test fixture, the value in the fixture and the value you set for STREAM_NAME must match. The test helpers use STREAM_NAME to create the destination stream, and then create the message to send using the value from the fixture. If these don’t match, the test will fail. URL_TEMPLATE defines how the test runner will call your webhook, in the same way you would provide a webhook URL to the 3rd party service. api_key={api_key} says that an API key is expected. In get_body, the first argument in the call to self.fixture_data specifies the prefix of your fixture file names, and file_type their type. Common types are json and txt. When writing tests for your webhook, you’ll want to include one test function (and corresponding fixture) per each distinct message condition that your integration supports. If, for example, we added support for sending a goodbye message to our Hello World webhook, we would add another test function to HelloWorldHookTests class called something like test_goodbye_message:

def test_goodbye_message(self): # type: () -> None expected_subject=u"Hello World"; expected_message=u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";

# use fixture named helloworld_goodbye self.send_and_test_stream_message('goodbye', expected_subject, expected_

˓→message, content_type="application/x-www-form-

˓→urlencoded")

As well as a new ﬁxture goodbye.json in zerver/webhooks/helloworld/fixtures/:

{ "featured_title":"Goodbye", "featured_url":"https://en.wikipedia.org/wiki/Goodbye", }

Also consider if your integration should have negative tests, a test where the data from the test ﬁxture should result in an error. For details see Negative tests, below. Once you have written some tests, you can run just these new tests from within the Zulip development environment with this command:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend zerver/webhooks/helloworld

21.5. Step 4: Create tests 115 Zulip Documentation, Release 1.4.0

(Note: You must run the tests from the top level of your development directory. The standard location in a Vagrant environment is /srv/zulip. If you are not using Vagrant, use the directory where you have your development environment.) You will see some script output and if all the tests have passed, you will see:

Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_goodbye_message Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_hello_message DONE!

Step 5: Create documentation

Next, we add end-user documentation for our webhook integration. You can see the existing examples at https: //zulipchat.com/integrations or by accessing /integrations in your Zulip development environment. There are two parts to the end-user documentation on this page. The ﬁrst is the lozenge in the grid of integrations, showing your integration logo and name, which links to the full documentation. This is generated automatically once you’ve registered the integration in WEB- HOOK_INTEGRATIONS in zerver/lib/integrations.py, and supports some customization via options to the WebhookIntegration class. Second, you need to write the actual documentation content in zerver/webhooks/mywebhook/doc.html.

Learn how Zulip integrations work with this simple Hello World example!

The Hello World webhook will use the test stream, which is created by default in the Zulip development environment. If you are running Zulip in production, you should make sure this stream exists.

Next, on your {{ settings_html|safe }}, create a Hello World bot. Construct the URL for the Hello World bot using the API key and stream name: {{ external_api_uri_subdomain }}/v1/external/helloworld?api_key=abcdefgh&

˓→amp;stream=test

To trigger a notication using this webhook, use `send_webhook_fixture_message` from the Zulip command line:

(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/webhooks/helloworld/fixtures/hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'

Or, use curl:

curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":

˓→"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }'

˓→http://localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>

116 Chapter 21. Webhook walkthrough Zulip Documentation, Release 1.4.0

Congratulations! You're done!
Your messages may look like:

See Documenting your integration for further details, including how to easily create the message screenshot.

Step 5: Preparing a pull request to zulip/zulip

When you have ﬁnished your webhook integration and are ready for it to be available in the Zulip product, follow these steps to prepare your pull request: 1. Run tests including linters and ensure you have addressed any issues they report. See Testing and Linters for details. 2. Read through Code styles and conventions and take a look through your code to double-check that you’ve followed Zulip’s guidelines. 3. Take a look at your git history to ensure your commits have been clear and logical (see Version Control for tips). If not, consider revising them with git rebase --interactive. For most webhooks, you’ll want to squash your changes into a single commit and include a good, clear commit message. 4. Push code to your fork. 5. Submit a pull request to zulip/zulip. If you would like feedback on your integration as you go, feel free to post a message on the public Zulip instance. You can also create a [WIP] pull request while you are still working on your integration. See the Git guide for more on Zulip’s pull request process.

Advanced topics

More complex implementation or testing needs may require additional code, beyond what the standard helper functions provide. This section discusses some of these situations.

Negative tests

A negative test is one that should result in an error, such as incorrect data. The helper functions may interpret this as a test failure, when it should instead be a successful test of an error condition. To correctly test these cases, you must explicitly code your test’s execution (using other helpers, as needed) rather than call the usual helper function. Here is an example from the WordPress webhook:

def test_unknown_action_no_data(self): # type: () -> None

# Mimic send_and_test_stream_message() to manually execute a negative test. # Otherwise its call to send_json_payload() would assert on the non-success # we are testing. The value of result is the error message the webhook should # return if no params are sent. The fixture for this test is an empty file.

# subscribe to the target stream self.subscribe_to_stream(self.TEST_USER_EMAIL, self.STREAM_NAME)

21.7. Step 5: Preparing a pull request to zulip/zulip 117 Zulip Documentation, Release 1.4.0

# post to the webhook url post_params={'stream_name': self.STREAM_NAME, 'content_type':'application/x-www-form-urlencoded'} result= self.client_post(self.url,'unknown_action', **post_params)

# check that we got the expected error message self.assert_json_error(result,"Unknown WordPress webhook action: WordPress Action

˓→")

In a normal test, send_and_test_stream_message would handle all the setup and then check that the webhook’s response matches the expected result. If the webhook returns an error, the test fails. Instead, explicitly do the setup it would have done, and check the result yourself. Here, subscribe_to_stream is a test helper that uses TEST_USER_EMAIL and STREAM_NAME (attributes from the base class) to register the user to receive messages in the given stream. If the stream doesn’t exist, it creates it. client_post, another helper, performs the HTTP POST that calls the webhook. As long as self.url is correct, you don’t need to construct the webhook URL yourself. (In most cases, it is.) assert_json_error then checks if the result matches the expected error. If you had used send_and_test_stream_message, it would have called send_json_payload, which checks the result with assert_json_success.

Custom query parameters

Custom arguments passed in URL query parameters work as expected in the webhook code, but require special handling in tests. For example, here is the deﬁnition of a webhook function that gets both stream and topic from the query parameters:

def api_querytest_webhook(request, user_profile, client, payload=REQ(argument_type='body'), stream=REQ(default='test

˓→'), topic=REQ(default='Default Alert')):

In actual use, you might conﬁgure the 3rd party service to call your Zulip integration with a URL like this:

http://myhost/api/v1/external/querytest?api_key=abcdefgh&stream=alerts&topic=queries

It provides values for stream and topic, and the webhook can get those using REQ without any special handling. How does this work in a test? The new attribute TOPIC exists only in our class so far. In order to construct a URL with a query parameter for topic, you can pass the attribute TOPIC as a keyword argument to build_webhook_url, like so:

class QuerytestHookTests(WebhookTestCase):

STREAM_NAME='querytest' TOPIC="Default Topic" URL_TEMPLATE="/api/v1/external/querytest?api_key= {api_key}&stream={stream}" FIXTURE_DIR_NAME='querytest'

def test_querytest_test_one(self): # type: () -> None

118 Chapter 21. Webhook walkthrough Zulip Documentation, Release 1.4.0

# construct the URL used for this test self.TOPIC=u"Query Test" self.url= self.build_webhook_url(topic=self.TOPIC)

# define the expected message contents expected_subject=u"Query Test" expected_message=u"This is a test of custom query parameters."

self.send_and_test_stream_message('test_one', expected_subject, expected_

˓→message, content_type="application/x-www-form-

˓→urlencoded")

def get_body(self, fixture_name): # type: (Text) -> Text return self.fixture_data("querytest", fixture_name, file_type="json")

You can also override get_body if your test data needs to be constructed in an unusual way. For more, see the deﬁnition for the base class, WebhookTestCase in zerver/lib/test_classes.py.

21.8. Advanced topics 119 Zulip Documentation, Release 1.4.0

120 Chapter 21. Webhook walkthrough CHAPTER 22

Writing a new application feature

The changes needed to add a new feature will vary, of course, but this document provides a general outline of what you may need to do, as well as an example of the speciﬁc steps needed to add a new feature: adding a new option to the application that is dynamically synced through the data system in real-time to all browsers the user may have open. As you read this, you may ﬁnd you need to learn about Zulip’s real-time push system; the real-time push and events documentation has a detailed explanation of how everything works.

General Process in brief

Adding a ﬁeld to the database

Update the model: The server accesses the underlying database in zerver/ models.py. Add a new ﬁeld in the appropriate class. Create and run the migration: To create and apply a migration, run:

./manage.py makemigrations ./manage.py migrate

Test your changes: Once you’ve run the migration, restart memcached on your development server (/etc/init. d/memcached restart) and then restart run-dev.py to avoid interacting with cached objects.

Backend changes

For most new features/settings, the property_types framework handles most of this automatically, but it’s valuable to understand the ﬂow even if the property_types framework means you don’t have to write any code. Database interaction: Add any necessary code for updating and interacting with the database in zerver/lib/ actions.py. It should update the database and send an event announcing the change. Application state: Modify the fetch_initial_state_data and apply_event functions in zerver/lib/ events.py to update the state based on the event you just created.

121 Zulip Documentation, Release 1.4.0

Backend implementation: Make any other modiﬁcations to the backend required for your feature to do what it’s supposed to do. New views: Add any new application views to zerver/urls.py. This includes both views that serve HTML (new pages on Zulip) as well as new API endpoints that serve JSON-formatted data. Testing: At the very least, add a test of your event data ﬂowing through the system in test_events.py and an API test in (e.g. for a Realm setting, in test_realm.py).

Frontend changes

JavaScript: Zulip’s JavaScript is located in the directory static/js/. The exact files you may need to change depend on your feature. If you’ve added a new event that is sent to clients, be sure to add a handler for it to static/ js/server_events_dispatch.js. CSS: The primary CSS file is static/styles/zulip.css. If your new feature requires UI changes, you may need to add additional CSS to this file. Templates: The initial page structure is rendered via Jinja2 templates located in templates/zerver. For JavaScript, Zulip uses Handlebars templates located in static/templates. Templates are precompiled as part of the build/deploy process. Zulip is fully internationalized, so when writing both HTML templates or JavaScript code that generates user-facing strings, be sure to tag those strings for translation. Testing: There are two types of frontend tests: node-based unit tests and blackbox end-to-end tests. The blackbox tests are run in a headless browser using Casper.js and are located in frontend_tests/casper_tests/. The unit tests use Node’s assert module are located in frontend_tests/node_tests/. For more information on writing and running tests see the testing documentation.

Documentation changes

After implementing the new feature, you should document it and update any existing documentation that might be relevant to the new feature. For more information on the kinds of documentation Zulip has, see Documentation.

Example Feature

This example describes the process of adding a new setting to Zulip: a flag that restricts inviting new users to admins only (the default behavior is that any user can invite other users). This flag is an actual Zulip feature. You can review the original commit in the Zulip repo. Note that the code involved in adding a realm feature has been refactored significantly since this feature was created, and Zulip has since been upgraded from Django 1.6 to 1.10.

Update the model

First, update the database and model to store the new setting. Add a new boolean ﬁeld, invite_by_admins_only, to the Realm model in zerver/models.py.

--- a/zerver/models.py +++ b/zerver/models.py @@ -108,6 +108,7 @@ class Realm(ModelReprMixin, models.Model): restricted_to_domain = models.BooleanField(default=True) # type: bool invite_required = models.BooleanField(default=False) # type: bool + invite_by_admins_only = models.BooleanField(default=False) # type: bool

122 Chapter 22. Writing a new application feature Zulip Documentation, Release 1.4.0

create_stream_by_admins_only = models.BooleanField(default=False) # type: bool mandatory_topics = models.BooleanField(default=False) # type: bool

The Realm model also contains an attribute, property_types, which other functions use to handle most realm settings without any custom code for the setting (more on this process below). The attribute is a dictionary, where the key is the name of the realm field and the value is the field’s type. Add the new field to the property_types dictionary.

# Define the types of the various automatically managed properties property_types= dict( # ... invite_by_admins_only=bool, # ...

Note: the majority of realm settings can be included in property_types. However, there are some properties that need custom logic and thus cannot use the property_types framework. For example: • The realm authentication_methods attribute is a bitfield and needs additional code for validation and updating. • The allow_message_editing and message_content_edit_limit_seconds fields depend on one another, so they are also handled separately and not included in property_types. When creating a realm property that is not a boolean, Text or integer field, or when adding a field that is dependent on other fields, handle it separately and do not add the field to the property_types dictionary. The steps below will point out where to write code for these cases.

Create the migration

Create the migration ﬁle: ./manage.py makemigrations. Make sure to commit the generated ﬁle to git: git add zerver/migrations/NNNN_realm_invite_by_admins_only.py (NNNN is a number that is equal to the number of migrations.) If you run into problems, the Django migration documentation is helpful.

Test your migration changes

Apply the migration: ./manage.py migrate Output: shell $ ./manage.py migrate Operations to perform: Synchronize unmigrated apps: staticfiles, analytics, pipeline Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites,

˓→auth, contenttypes Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying zerver.0026_realm_invite_by_admins_only... OK

22.2. Example Feature 123 Zulip Documentation, Release 1.4.0

Handle database interactions

Next, we will move on to implementing the backend part of this feature. Like typical apps, we will need our backend to update the database and send some response to the client that made the request. Beyond that, we need to orchestrate notifications to other clients (or other users, if you will) that our setting has changed. Clients find out about settings through two closely related code paths. When a client first contacts the server, the server sends the client its initial state. Subsequently, clients subscribe to “events,” which can (among other things) indicate that settings have changed. For the backend piece, we will need our action to make a call to send_event to send the event to clients that are active. We will also need to modify fetch_initial_state_data so that the new field is passed to clients. See our event system docs for all the gory details. Anyway, getting back to implementation details... In zerver/lib/actions.py, the function do_set_realm_property takes in the name of a realm property to update and the value it should have. This function updates the database and triggers an event to notify clients about the change. It uses the field’s type, specified in the Realm.property_types dictionary, to validate the type of the value before updating the property; this is primarily an assertion to help catch coding mistakes, not to check for bad user input. After updating the given realm field, do_set_realm_property creates an ‘update’ event with the name of the property and the new value. It then calls send_event, passing the event and the list of users whose browser sessions should be notified as the second argument. The latter argument can be a single user (if the setting is a personal one, like time display format), members in a particular stream only or all active users in a realm.

# zerver/lib/actions.py

def do_set_realm_property(realm, name, value): # type: (Realm, str, Union[Text, bool, int]) -> None """Takes in a realm object, the name of an attribute to update, and the value to update. """ property_type= Realm.property_types[name] assert isinstance(value, property_type), ( 'Cannot update %s: %s is not an instance of %s'%( name, value, property_type,))

setattr(realm, name, value) realm.save(update_fields=[name]) event= dict( type='realm', op='update', property=name, value=value, ) send_event(event, active_user_ids(realm))

If the new realm property being added does not fit into the do_set_realm_property framework (such as the authentication_methods field), you’ll need to create a new function to explicitly update this field and send an event.

# zerver/lib/actions.py

def do_set_realm_authentication_methods(realm, authentication_methods): # type: (Realm, Dict[str, bool]) -> None for key, value in list(authentication_methods.items()): index= getattr(realm.authentication_methods, key).number realm.authentication_methods.set_bit(index, int(value))

124 Chapter 22. Writing a new application feature Zulip Documentation, Release 1.4.0

realm.save(update_fields=['authentication_methods']) event= dict( type="realm", op="update_dict", property='default', data=dict(authentication_methods=realm.authentication_methods_dict()) ) send_event(event, active_user_ids(realm))

Update application state

You then need to add code to ensure that your new setting is included in the data sent down to clients, both when a new client is loaded, and when changes happen. The fetch_initial_state_data function is responsible for the former (data added to the state here will be available both in page_params in the browser, as well as to API clients like the mobile apps). The apply_event function in zerver/lib/events.py is important for making sure the state is always correct, even in the event of rare race conditions.

# zerver/lib/events.py def fetch_initial_state_data(user_profile, event_types, queue_id, include_

˓→subscribers=True): # ... if want('realm'): for property_name in Realm.property_types: state['realm_'+ property_name]= getattr(user_profile.realm, property_name) state['realm_authentication_methods']= user_profile.realm.authentication_methods_

˓→dict() state['realm_allow_message_editing']= user_profile.realm.allow_message_editing # ... def apply_event(state, events, user_profile, include_subscribers): for event in events: # ... elif event['type'] =='realm': field='realm_'+ event['property'] state[field]= event['value'] # ...

If you are adding a realm property that ﬁts the property_types framework, you don’t need to change fetch_initial_state_data or apply_event because there is already code to get the initial data and handle the realm update event type. However, if you are adding a property that is handled separately, you will need to explicitly add the property to the state dictionary in the fetch_initial_state_data function. E.g., for authentication_methods: def fetch_initial_state_data(user_profile, event_types, queue_id, include_

˓→subscribers=True): # ... if want('realm'): # ... state['realm_authentication_methods']= user_profile.realm.authentication_

˓→methods_dict() # ...

For this setting, one won’t need to change apply_event since its default code for realm event types handles this case correctly, but for a totally new type of feature, a few lines in that function may be needed.

22.2. Example Feature 125 Zulip Documentation, Release 1.4.0

Add a new view

You will need to add a view for clients to access that will call the actions.py code to update the database. This example feature adds a new parameter that will be sent to clients when the application loads and should be accessible via JavaScript. There is already a view that does this for related flags: update_realm in zerver/views/ realm.py. So in this case, we can add our code to the existing view instead of creating a new one. Since this feature adds a checkbox to the admin page and a new property to the Realm model that can be modified from there, you need to add a parameter for the new field to the update_realm function in zerver/views/realm. py.

def update_realm(request, user_profile, name=REQ(validator=check_string,

˓→default=None), # ..., invite_by_admins_only=REQ(validator=check_bool, default=None), # ...): # type: (HttpRequest, UserProfile, ..., Optional[bool], ... # ...

If this feature ﬁts the do_set_realm_property framework and does not require additional validation, this is the only change to make to zerver/views/realm.py. Text ﬁelds or other realm properties that need additional validation can be handled at the beginning of update_realm.

# Additional validation/error checking beyond types go here, so # the entire request can succeed or fail atomically. if default_language is not None and default_language not in get_available_language_

˓→codes(): raise JsonableError(_("Invalid language' %s'"% (default_language,))) if description is not None and len(description)> 100: return json_error(_("Realm description cannot exceed 100 characters.")) # ...

Then, the code in update_realm loops through the property_types dictionary and calls do_set_realm_property on any property to be updated from the request. However, if the new feature is not in property_types, you will need to write the code to speciﬁcally handle it. Ex, for authentication_methods:

# zerver/views/realm.py

# ... if authentication_methods is not None and realm.authentication_methods_dict() !=

˓→authentication_methods: do_set_realm_authentication_methods(realm, authentication_methods) data['authentication_methods']= authentication_methods # ...

This completes the backend implementation. A great next step is to write the backend tests. With the property_types framework, one just needs to add a line in test_events.py and test_realm.py with a list of values to switch between in the test.

Update the front end

Then make the required front end changes: in this case a checkbox needs to be added to the admin page (and its value added to the data sent back to server when a realm is updated) and the change event needs to be handled on the client.

126 Chapter 22. Writing a new application feature Zulip Documentation, Release 1.4.0

To add the checkbox to the admin page, modify the relevant template, static/templates/admin_tab. handlebars (omitted here since it is relatively straightforward). Then add code to handle changes to the new form control in static/js/admin.js. var url = "/json/realm"; var new_invite_by_admins_only = $("#id_realm_invite_by_admins_only").prop("checked"); data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only); channel.patch({ url: url, data: data, success: function (data) { # ... if (data.invite_by_admins_only) { ui_report.success("New users must be invited by an admin!", invite_by_admins_

˓→only_status); } else { ui_report.success("Any user may now invite new users!", invite_by_admins_only_

˓→status); } # ... } });

Finally, update server_events_dispatch.js to handle related events coming from the server.

# static/js/server_events_dispatch.js function dispatch_normal_event(event) { switch (event.type) { # ... case'realm': if (event.op ==='update'&& event.property ==='invite_by_admins_only'){ page_params.realm_invite_by_admins_only= event.value; } } }

Any code needed to update the UI should be placed in dispatch_normal_event callback (rather than the channel.patch) function. This ensures the appropriate code will run even if the changes are made in another browser window. In this example most of the changes are on the backend, so no UI updates are required.

Update documentation

After you add a new view, you should document your feature. This feature adds new functionality that restricts inviting new users to admins only. A recommended way to document this feature would be to update and/or augment Zulip’s user documentation to reflect your changes and additions. At the very least, this will involve adding (or modifying) a Markdown file documenting the feature to templates/ zerver/help/ in the main Zulip server repository, where the source for Zulip’s user documentation is stored. For information on writing user documentation, see Zulip’s general user guide documentation. For a more concrete example of writing documentation for a new feature, see the original commit in the Zulip repo that documented this feature, the current source, and the final rendered documentation.

22.2. Example Feature 127 Zulip Documentation, Release 1.4.0

128 Chapter 22. Writing a new application feature CHAPTER 23

Writing views in Zulip

What this covers

This page documents how views work in Zulip. You may want to read the new feature tutorial or the integration guide, and treat this as a reference. If you have experience with Django, much of this will be familiar, but you may want to read about how REST requests are dispatched, and how request authentication works. This document supplements the new feature tutorial and the testing documentation.

What is a view?

A view in Zulip is everything that helps implement a server endpoint. Every path that the Zulip server supports (doesn’t show a 404 page for) is a view. The obvious ones are those you can visit in your browser, for example /integrations, which shows the integration documentation. These paths show up in the address bar of the browser. There are other views that are only seen by software, namely the API views. They are used to build the various clients that Zulip has, namely the web client (which is also used by the desktop client) and the mobile clients.

Modifying urls.py

A view is anything with an entry in the appropriate urls.py, usually zproject/urls.py. Zulip views either serve HTML (pages for browsers) or JSON (data for Zulip clients on all platforms, custom bots, and integrations). The format of the URL patterns in Django is documented here, and the Zulip speciﬁc details for these are discussed in detail in the life of a request doc. We have two Zulip-speciﬁc conventions we use for internationalization and for our REST API, respectively.

129 Zulip Documentation, Release 1.4.0

Writing human-readable views

If you’re writing a new page for the website, make sure to add it to i18n_urls in zproject/urls.py

i18n_urls = [ ... + url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html

˓→')), + url(r'^postcards/$', 'zerver.views.postcards'), ]

As an example, if a request comes in for Spanish, language code es, the server path will be something like: es/ features/.

Decorators used for webpage views

This section documents a few simple decorators that we use for webpage views, as an introduction to view decorators. require_post:

@require_post def accounts_register(request): # type: (HttpRequest) -> HttpResponse

This decorator ensures that the requst was a POST–here, we’re checking that the registration submission page is requested with a post, and inside the function, we’ll check the form data. If you request this page with GET, you’ll get a HTTP 405 METHOD NOT ALLOWED error. zulip_login_required: This decorator veriﬁes that the browser is logged in (i.e. has a valid session cookie) before providing the view for this route, or redirects the browser to a login page. This is used in the root path (/) of the website for the web client. If a request comes from a browser without a valid session cookie, they are redirected to a login page. It is a small fork of Django’s login_required, adding a few extra checks speciﬁc to Zulip.

@zulip_login_required def home(request): # type: (HttpRequest) -> HttpResponse

Writing a template

Templates for the main website are found in templates/zerver.

Writing API REST endpoints

These are code-parseable views that take x-www-form-urlencoded or JSON request bodies, and return JSON-string responses. Almost all Zulip view code is in the implementations of API REST endpoints. The REST API does authentication of the user through rest_dispatch, which is documented in detail at zerver/lib/rest.py. This method will authenticate the user either through a session token from a cookie on the browser, or from a base64 encoded email:api-key string given via HTTP Basic Auth for API clients.

130 Chapter 23. Writing views in Zulip Zulip Documentation, Release 1.4.0

>>> import requests >>> r= requests.get('https://api.github.com/user', auth=('[email protected]',

˓→'0123456789abcdeFGHIJKLmnopQRSTUV')) >>> r.status_code -> 200

Request variables

Most API views will have some arguments that are passed as part of the request to control the behavior of the view. In any well-engineered view, you need to write code to parse and validate that the arguments exist and have the correct form. For many applications, this leads to one of several bad outcomes: • The code isn’t written, so arguments aren’t validated, leading to bugs and confusing error messages for users of the API. • Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent error messages. • Every view function comes with another function that does the validation that has the problems from the last bullet point. In Zulip, we solve this problem with a the special decorator called has_request_variables which allows a developer to declare the arguments a view function takes and validate their types all within the def line of the function. We like this framework because we have found it makes the validation code compact, readable, and conveniently located in the same place as the method it is validating arguments for. Here’s an example:

from zerver.decorator import has_request_variables, REQ, JsonableError, \ require_realm_admin

@require_realm_admin @has_request_variables def create_user_backend(request, user_profile, email=REQ(), password=REQ(), full_name=REQ(), short_name=REQ()): # ... code here

You will notice the special REQ() in the keyword arguments to create_user_backend. has_request_variables parses the declared keyword arguments of the decorated function, and for each that has an instance of REQ as the default value, it extracts the HTTP parameter with that name from the request, parses it as JSON, and passes it to the function. It will return an nicely JSON formatted HTTP 400 error in the event that an argument is missing, doesn’t parse as JSON, or otherwise is invalid. require_realm_admin is another decorator which checks the authorization of the given user_profile to make sure it belongs to a realm administrator (and thus has permission to create a user); we show it here primarily to show how has_request_variables should be the inner decorator. The implementation of has_request_variables is documented in detail in zerver/lib/request.py) REQ also helps us with request variable validation. For example: • msg_ids = REQ(validator=check_list(check_int)) will check that the msg_ids HTTP parameter is a list of integers, marshalled as JSON, and pass it into the function as the msg_ids Python keyword argument. • streams_raw = REQ("subscriptions", validator=check_list(check_string)) will check that the “subscriptions” HTTP parameter is a list of strings, marshalled as JSON, and pass it into the function with the Python keyword argument streams_raw.

23.5. Writing API REST endpoints 131 Zulip Documentation, Release 1.4.0

• message_id=REQ(converter=to_non_negative_int) will check that the message_id HTTP parameter is a string containing a non-negative integer (converter differs from validator in that it does not automatically marshall the input from JSON). See zerver/lib/validator.py for more validators and their documentation.

Deciding which HTTP verb to use

When writing a new API view, you should writing a view to do just one type of thing. Usually that’s either a read or write operation. If you’re reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if a resource is available to be read with GET, without the expense of the full GET. OPTIONS is also read-only, and used by clients to determine which HTTP verbs are available for a given path. This isn’t something you need to write, as it happens automatically in the implementation of rest_dispatch–see zerver/lib/rest.py for more. If you’re creating new data, try to figure out if the thing you are creating is uniquely identifiable. For example, if you’re creating a user, there’s only one user per email. If you can find a unique ID, you should use PUT for the view. If you want to create the data multiple times for multiple requests (for example, requesting the send_message view multiple times with the same content should send multiple messages), you should use POST. If you’re updating existing data, use PATCH. If you’re removing data, use DELETE.

Idempotency

When writing a new API endpoint, with the exception of things like sending messages, requests should be safe to repeat, without impacting the state of the server. This is idempotency. You will often want to return an error if a request to change something would do nothing because the state is already as desired, to make debugging Zulip clients easier. This means that the response for repeated requests may not be the same, but the repeated requests won’t change the server more than once or cause unwanted side effects.

Making changes to the database

If the view does any modiﬁcation to the database, that change is done in a helper function in zerver/lib/ actions.py. Those functions are responsible for doing a complete update to the state of the server, which often entails both updating the database and sending any events to notify clients about the state change. When possible, we prefer to design a clean boundary between the view function and the actions function is such that all user input validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsible for atomically executing the change (this is usually signalled by having the actions function have a name starting with do_. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connection to the database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have a name starting with check_. For example, in zerver/views/realm.py:

@require_realm_admin @has_request_variables def update_realm(request, user_profile, name=REQ(validator=check_string,

˓→default=None),...)): # type: (HttpRequest, UserProfile, ...) -> HttpResponse realm= user_profile.realm data={} # type: Dict[str, Any]

132 Chapter 23. Writing views in Zulip Zulip Documentation, Release 1.4.0

if name is not None and realm.name != name: do_set_realm_name(realm, name) data['name']='updated' and in zerver/lib/actions.py: def do_set_realm_name(realm, name): # type: (Realm, Text) -> None realm.name= name realm.save(update_fields=['name']) event= dict( type="realm", op="update", property='name', value=name, ) send_event(event, active_user_ids(realm)) realm.save() actually saves the changes to the realm to the database, and send_event sends the event to active clients belonging to the provided list of users (in this case, all altive users in the Zulip realm).

Calling from the web application

You should always use channel. to make an HTTP call to the Zulip JSON API. As an example, in static/js/admin.js var url= "/json/realm"; var data={ name: JSON.stringify(new_name), } channel.patch({ url: url, data: data, success: function (response_data) { if (response_data.name !== undefined){ ui_report.success(i18n.t("Name changed!"), name_status); } ...

Calling from an API client

Here’s how you might manually make a call from python: payload={'name': new_name}

# email and API key api_auth=('[email protected]','0123456789abcdeFGHIJKLmnopQRSTUV') r= requests.patch(SERVER_URL+'api/v1/realm', data=json.dumps(payload), auth=api_auth, )

This is simply an illustration; we recommend making use of the Zulip Python API bindings since they provide a nice interface for accessing the API.

23.5. Writing API REST endpoints 133 Zulip Documentation, Release 1.4.0

Legacy endpoints used by the web client

New features should conform the REST API style. The legacy, web-only endpoints can’t effectively enforce usage of a browser, so they aren’t preferable from a security perspective, and it is generally a good idea to make your feature available to other clients, especially the mobile clients. These endpoints make use of some older authentication decorators, authenticated_json_api_view, authenticated_json_post_view, and authenticated_json_view, so you may see them in the code.

Webhook integration endpoints

Webhooks are called by other services, often to send a message as part of those services’ integrations. They are most often POST requests, and often there is very little you can customize about them. Usually you can expect that the webhook for a service will allow specification for the target server for the webhook, and an API key. If the webhook does not have an option to provide a bot email, use the api_key_only_webhook_view decorator, to fill in the user_profile and request.client fields of a request:

@api_key_only_webhook_view('PagerDuty') @has_request_variables def api_pagerduty_webhook(request, user_profile, payload=REQ(argument_type='body'), stream=REQ(default='pagerduty'), topic=REQ(default=None)): request.client will be the result of get_client("ZulipPagerDutyWebhook") in this example and it will be passed to check_send_message. For more information, see Clients in Zulip.

134 Chapter 23. Writing views in Zulip CHAPTER 24

Life of a Request

It can sometimes be confusing to ﬁgure out how to write a new feature, or debug an existing one. Let us try to follow a request through the Zulip codebase, and dive deep into how each part works. We will use as our example the creation of users through the API, but we will also highlight how alternative requests are handled.

A request is sent to the server, and handled by Nginx

When Zulip is deployed in production, all requests go through nginx. For the most part we don’t need to know how this works, except for when it isn’t working. Nginx does the first level of routing–deciding which application will serve the request (or deciding to serve the request itself for static content). In development, tools/run-dev.py fills the role of nginx. Static files are in your git checkout under static, and are served unminified.

Nginx secures trafﬁc with SSL

If you visit your Zulip server in your browser and discover that your trafﬁc isn’t being properly encrypted, an nginx misconﬁguration is the likely culprit.

Static ﬁles are served directly by Nginx

Static ﬁles include JavaScript, css, static assets (like emoji, avatars), and user uploads (if stored locally and not on S3). location/static/{ alias/home/zulip/prod-static/; error_page 404/static/html/404.html; }

135 Zulip Documentation, Release 1.4.0

Nginx routes other requests between django and tornado

All our connected clients hold open long-polling connections so that they can receive events (messages, presence notiﬁcations, and so on) in real-time. Events are served by Zulip’s tornado application. Nearly every other kind of request is served by the zerver Django application. Here is the relevant nginx routing conﬁguration.

Django routes the request to a view in urls.py ﬁles

There are various urls.py ﬁles throughout the server codebase, which are covered in more detail in the directory structure doc. The main Zulip Django app is zerver. The routes are found in

zproject/urls.py zproject/legacy_urls.py

There are HTML-serving, REST API, legacy, and webhook url patterns. We will look at how each of these types of requests are handled, and focus on how the REST API handles our user creation example.

Views serving HTML are internationalized by server path

If we look in zproject/urls.py, we can see something called i18n_urls. These urls show up in the address bar of the browser, and serve HTML. For example, the /hello page (preview here) gets translated in Chinese at zh-cn/hello/ (preview here). Note the zh-cn preﬁx–that url pattern gets added by i18n_patterns.

API endpoints use REST

Our example is a REST API endpoint. It’s a PUT to /users. With the exception of Webhooks (which we do not usually control the format of), legacy endpoints, and logged-out endpoints, Zulip uses REST for its API. This means that we use: • POST for creating something new where we don’t have a unique ID. Also used as a catch-all if no other verb is appropriate. • PUT for creating something for which we have a unique ID. • DELETE for deleting something • PATCH for updating or editing attributes of something. • GET to get something (read-only) • HEAD to check the existence of something to GET, without getting it; useful to check a link without downloading a potentially large link • OPTIONS (handled automatically, see more below)

136 Chapter 24. Life of a Request Zulip Documentation, Release 1.4.0

Of these, PUT, DELETE, HEAD, OPTIONS, and GET are idempotent, which means that we can send the request multiple times and get the same state on the server. You might get a different response after the ﬁrst request, as we like to give our clients an error so they know that no new change was made by the extra requests. POST is not idempotent–if I send a message multiple times, Zulip will show my message multiple times. PATCH is special–it can be idempotent, and we like to write API endpoints in an idempotent fashion, as much as possible. This cookbook and tutorial can be helpful if you are new to REST web applications.

PUT is only for creating new things

If you’re used to using PUT to update or modify resources, you might find our convention a little strange. We use PUT to create resources with unique identifiers, POST to create resources without unique identifiers (like sending a message with the same content multiple times), and PATCH to modify resources. In our example, create_user_backend uses PUT, because there’s a unique identifier, the user’s email.

OPTIONS

The OPTIONS method will yield the allowed methods. This request: OPTIONS https://chat.zulip.org/api/v1/users yields a response with this HTTP header: Allow: PUT, GET We can see this reﬂected in zproject/urls.py:

url(r'^users$','zerver.lib.rest.rest_dispatch', {'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'}),

In this way, the API is partially self-documenting.

Legacy endpoints are used by the web client

The endpoints from the legacy JSON API are written without REST in mind. They are used extensively by the web client, and use POST. You can see them in zproject/legacy_urls.py.

Webhook integrations may not be RESTful

Zulip endpoints that are called by other services for integrations have to conform to the service’s request format. They are likely to use only POST.

Django calls rest_dispatch for REST endpoints, and authenticates

For requests that correspond to a REST url pattern, Zulip conﬁgures its url patterns (see zerver/lib/rest.py) so that the action called is rest_dispatch. This method will authenticate the user, either through a session token from a cookie, or from an email:api-key string given via HTTP Basic Auth for API clients. It will then look up what HTTP verb was used (GET, POST, etc) to make the request, and then ﬁgure out which view to show from that.

24.8. Django calls rest_dispatch for REST endpoints, and authenticates 137 Zulip Documentation, Release 1.4.0

In our example,

{'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'} is supplied as an argument to rest_dispatch, along with the HTTPRequest. The request has the HTTP verb PUT, which rest_dispatch can use to ﬁnd the correct view to show: zerver.views.users. create_user_backend.

The view will authorize the user, extract request variables, and validate them

This is covered in good detail in the writing views doc.

Results are given as JSON

Our API works on JSON requests and responses. Every API endpoint should return json_error in the case of an error, which gives a JSON string: {'result': 'error', 'msg': } in a HTTP Response with a content type of ‘application/json’. To pass back data from the server to the calling client, in the event of a successfully handled request, we use json_success(data=. This will result in a JSON string: {'result': 'success', 'msg': '', 'data'='{'var_name1': 'var_value1', 'var_name2': 'var_value2'...} with a HTTP 200 status and a content type of ‘application/json’. That’s it!

138 Chapter 24. Life of a Request CHAPTER 25

Reading list

Here you can ﬁnd a list of relevant resources that you may ﬁnd useful for learning new skills, or polishing the ones you already have. The topics cover a wide variety of topics, from basic Python coding to general developing guidelines. Feel free to create a pull request in Zulip’s GitHub repository with any interesting books, articles or videos you would like to see in this list. Some titles have been shortened for organizational purposes.

General programming/IT

Book - Clean Code: A Handbook of Agile Software Craftsmanship (Not free!) Books - Free programming books list Blog - Free Code Camp blog Blog - Idle Words talks transcripts Tutorial - HTTP Can Do That?!, by Sumana Harihareswara (PyCon 2016) Video - Minimum Viable Documentation, by Matthew Lyon (WriteTheDocs 2014) Video - NoOps, by Kelsey Hightower (DepOpsDays 2016) Video - The mind behind Linux (TED interview) Tutorial - Learn code the hard way Tutorial - What happens when... Article - An Interview With Linus Torvalds Article - Effective Learning Strategies for Programmers Article - Readme Driven Development

139 Zulip Documentation, Release 1.4.0

Article - Systematic Debugging Paper - Floating-Point Arithmetic

Python

Video - Intro to Python for beginners, by Jessica McKellar (PyCon 2013) Video - Breaking the rules, by Jessica McKellar (PyCon Sweden) Video - Build & break a Python sandbox, by Jessica McKellar (PyCon 2014) Video - Cache me if you can, by Guillaume Ardaud (PyCon 2014) Video - Loop like a native, by Ned Batchelder (PyCon 2013) Video - Modern Dictionaries, by Raymond Hettinger (SF Python) Video - Python Language, by Guido van Rossum (PyCon 2016) Video - The Mighty Dictionary, by Brandon Rhodes (PyCon 2010) Tutorial - Code Like a Pythonista: Idiomatic Python Article - Static types in Python, oh my(py)! Guide - The Hitchhiker’s Guide to Python!

Java/Android

Course - Android Development for Beginners Blog - Java Tutorials for Beginners

JavaScript/ECMAScript

Tutorial - clean-code-javascript Software engineering principles Course - React native and redux course( Not free!) Slides - TypeScript vs. CoffeeScript vs. ES6

Git/Version Control Systems (VCS)

You may want to take a look ﬁrst at our Git and GitHub guide. Article - Git tips

Computer Science/Algorithms

Blog - GeeksforGeeks Book Introduction to Algorithms( Not free!)

140 Chapter 25. Reading list Zulip Documentation, Release 1.4.0

Blog - Setosa data visualization and visual explanations Course - Algorithms, Part I Course - Open Source Society University Course - MIT CSAIL 6.828: Operative Systems Engineering

Community experience

Book - Producing Open Source Software Article - Advice on Starting And Running A New Open Source Project Article - How to ask good questions Article - Notes for New FLOSS Contributors Article - To be mentored Article - To mentor List of good projects for new contributors

Competitions/Camps

CodeForces Free Code Camp

Massive Open Online Courses (MOOC) Platforms

Coursera edX MIT OpenCourseWare Udacity

25.7. Community experience 141 Zulip Documentation, Release 1.4.0

142 Chapter 25. Reading list CHAPTER 26

Git & GitHub Guide

Quick start: How Zulip uses Git and GitHub

This quick start provides a brief overview of how Zulip uses Git and GitHub. Those who are familiar with Git and GitHub should be able to start contributing with these details in mind: • We use GitHub for source control and code review. To contribute, fork zulip/zulip (or the appropriate repository, if you are working on something else besides Zulip server) to your own account and then create feature/issue branches. When you’re ready to get feedback, submit a work-in-progress (WIP) pull request. We encourage you to submit WIP pull requests early and often. • We use a rebase-oriented workflow. We do not use merge commits. This means you should use git fetch followed by git rebase rather than git pull (or you can use git pull --rebase). Also, to prevent pull requests from becoming out of date with the main line of development, you should rebase your feature branch prior to submitting a pull request, and as needed thereafter. If you’re unfamiliar with how to rebase a pull request, read this excellent guide. We use this strategy in order to avoid the extra commits that appear when another branch is merged, that clutter the commit history (it’s popular with other large projects such as Django). This makes Zulip’s commit history more readable, but a side effect is that many pull requests we merge will be reported by GitHub’s UI as closed instead of merged, since GitHub has poor support for rebase-oriented workflows. • We have a code style guide, a commit message guide, and strive for each commit to be a minimal coherent idea (see commit discipline for details). • We provide many tools to help you submit quality code. These include linters, tests, continuous integration with TravisCI, and mypy. • We use zulipbot to manage our issues and pull requests to create a better GitHub workflow for contributors. Finally, take a quick look at Zulip-specific Git scripts, install the Zulip developer environment, and then configure your fork for use with TravisCI.

The following sections will help you be awesome with Zulip and Git/GitHub in a rebased-based workﬂow. Read through it if you’re new to git, to a rebase-based git workﬂow, or if you’d like a git refresher.

143 Zulip Documentation, Release 1.4.0

Set up Git

If you’re already using Git, have a client you like, and a GitHub account, you can skip this section. Otherwise, read on!

Install and conﬁgure Git, join GitHub

If you’re not already using Git, you might need to install and conﬁgure it. You’ll also need a GitHub account, which you can sign up for here. We also recommend you create an ssh key if you don’t already have one and add it to your GitHub account. If you plan on using Git from the command line, the following tips can make your experience better: • Install the command auto-completion and/or git-prompt plugins available for Bash and Zsh. • Conﬁgure Git with your user details and aliases for commands you’ll use often.

Get a graphical client

Even if you’re comfortable using git on the command line, having a graphic client can be useful for viewing your repository. This is especially when doing a complicated rebases and similar operations because you can check the state of your repository after each command to see what changed. If something goes wrong, this helps you ﬁgure out when and why. If you don’t already have one installed, here are some suggestions: • macOS: GitX-dev • Ubuntu/Linux: git-cola, gitg, gitk • Windows: SourceTree If you like working on the command line, but want better visualization and navigation of your git repo, try Tig, a cross-platform ncurses-based text-mode interface to Git. And, if none of the above are to your liking, try one of these.

How Git is different

Whether you’re new to Git or have experience with another version control system (VCS), it’s a good idea to learn a bit about how Git works. We recommend this excellent presentation Understanding Git from Nelson Elhage and Anders Kaseorg and the Git Basics chapter from Pro Git by Scott Chacon and Ben Straub. Here are the top things to know: • Git works on snapshots: Unlike other version control systems (e.g., Subversion, Perforce, Bazaar), which track files and changes to those files made over time, Git tracks snapshots of your project. Each time you commit or otherwise make a change to your repository, Git takes a snapshot of your project and stores a reference to that snapshot. If a file hasn’t changed, Git creates a link to the identical file rather than storing it again. • Most Git operations are local: Git is a distributed version control system, so once you’ve cloned a repository, you have a complete copy of that repository’s entire history. Staging, committing, branching, and browsing history are all things you can do locally without network access and without immediately affecting any remote repositories. To make or receive changes from remote repositories, you need to git fetch, git pull, or git push.

144 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

• Nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to make Git perform actions that you can’t undo. However, Git can’t undo what it doesn’t know about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote repository. • Git is designed for lightweight branching and merging. Branches are simply references to snapshots. It’s okay and expected to make a lot of branches, even throwaway and experimental ones. • Git stores all data as objects, of which there are four types: blob (file), tree (directory), commit (revision), and tag. Each of these objects is named by a unique hash, the SHA-1 has of its contents. Most of the time you’ll refer to objects by their truncated hash or more human-readable reference like HEAD (the current branch). Blobs and trees represent files and directories. Tags are named references to other objects. A commit object includes: tree id, zero or more parents as commit ids, an author (name, email, date), a committer (name, email, date), and a log message. A Git repository is a collection of mutable pointers to these objects called refs. • Cloning a repository creates a working copy. Every working copy has a .git subdirectory, which contains its own Git repository. The .git subdirectory also tracks the index, a staging area for changes that will become part of the next commit. All files outside of .git is the working tree. • Files tracked with Git have possible three states: committed, modified, and staged. Committed files are those safely stored in your local .git repository/database. Staged files have changes and have been marked for inclusion in the next commit; they are part of the index. Modified files have changes but have not yet been marked for inclusion in the next commit; they have not been added to the index. • Git commit workflow is as follows: Edit files in your working tree. Add to the index (that is stage) with git add. Commit to the HEAD of the current branch with git commit.

Important Git terms

When you install Git, it adds a manual entry for gitglossary. You can view this glossary by running man gitglossary. Below we’ve included the git terms you’ll encounter most often along with their deﬁnitions from gitglossary. branch

A “branch” is an active line of development. The most recent commit on a branch is referred to as the tip of that branch. The tip of the branch is referenced by a branch head, which moves forward as additional development is done on the branch. A single Git repository can track an arbitrary number of branches, but your working tree is associated with just one of them (the “current” or “checked out” branch), and HEAD points to that branch. cache

Obsolete for: index checkout

The action of updating all or part of the working tree with a tree object or blob from the object database, and updating the index and HEAD if the whole working tree has been pointed at a new branch.

26.4. Important Git terms 145 Zulip Documentation, Release 1.4.0 commit

As a noun: A single point in the Git history; the entire history of a project is represented as a set of interrelated commits. The word “commit” is often used by Git in the same places other revision control systems use the words “revision” or “version”. Also used as a short hand for commit object. As a verb: The action of storing a new snapshot of the project’s state in the Git history, by creating a new commit representing the current state of the index and advancing HEAD to point at the new fast-forward

A fast-forward is a special type of merge where you have a revision and you are “merging” another branch’s changes that happen to be a descendant of what you have. In such these cases, you do not make a new mergecommit but instead just update to his revision. This will happen frequently on a remote-tracking branch of a remote repository. fetch

Fetching a branch means to get the branch’s head ref from a remote repository, to ﬁnd out which objects are missing from the local object database, and to get them, too. See also git-fetch(1). hash

In Git’s context, synonym for object name. head

A named reference to the commit at the tip of a branch. Heads are stored in a ﬁle in $GIT_DIR/refs/heads/ directory, except when using packed refs. (See git-pack-refs(1).)

HEAD

The current branch. In more detail: Your working tree is normally derived from the state of the tree referred to by HEAD. HEAD is a reference to one of the heads in your repository, except when using a detached HEAD, in which case it directly references an arbitrary commit. index

A collection of ﬁles with stat information, whose contents are stored as objects. The index is a stored version of your working tree. Truth be told, it can also contain a second, and even a third version of a working tree, which are used when merging. pull

Pulling a branch means to fetch it and merge it. See also git- pull(1).

146 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

push

Pushing a branch means to get the branch’s head ref from a remote repository, ﬁnd out if it is a direct ancestor to the branch’s local head ref, and in that case, putting all objects, which are reachable from the local head ref, and which are missing from the remote repository, into the remote object database, and updating the remote head ref. If the remote head is not an ancestor to the local head, the push fails.

rebase

To reapply a series of changes from a branch to a different base, and reset the head of that branch to the result.

Get Zulip code

Zulip uses a forked-repo and rebase-oriented workﬂow.. This means that all contributors create a fork of the Zulip repository they want to contribute to and then submit pull requests to the upstream repository to have their contributions reviewed and accepted. We also recommend you work on feature branches.

Step 1a: Create your fork

The following steps you’ll only need to do the ﬁrst time you setup a machine for contributing to a given Zulip project. You’ll need to repeat the steps for any additional Zulip projects (list) that you work on. The ﬁrst thing you’ll want to do to contribute to Zulip is fork (see how) the appropriate Zulip repository. For the main server app, this is zulip/zulip.

Step 1b: Clone to your machine

Next, clone your fork to your local machine:

$ git clone [email protected]:christi3k/zulip.git Cloning into 'zulip' remote: Counting objects: 86768, done. remote: Compressing objects: 100% (15/15), done. remote: Total 86768 (delta 5), reused 1 (delta 1), pack-reused 86752 Receiving objects: 100% (86768/86768), 112.96 MiB | 523.00 KiB/s, done. Resolving deltas: 100% (61106/61106), done. Checking connectivity... done.

Note: If you receive an error while cloning, you may not have added your ssh key to GitHub.

Step 1c: Connect your fork to Zulip upstream

Next you’ll want to conﬁgure an upstream remote repository for your fork of Zulip. This will allow you to sync changes from the main project back into your fork. First, show the currently conﬁgured remote repository:

$ git remote -v origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push)

26.5. Get Zulip code 147 Zulip Documentation, Release 1.4.0

Note: If you’ve cloned the repository using a graphical client, you may already have the upstream remote repository conﬁgured. For example, when you clone zulip/zulip with the GitHub desktop client it conﬁgures the remote repository zulip and you see the following output from git remote -v: origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push) zulip https://github.com/zulip/zulip.git (fetch) zulip https://github.com/zulip/zulip.git (push)

If your client hasn’t automatically conﬁgured a remote for zulip/zulip, you’ll need to with:

$ git remote add upstream https://github.com/zulip/zulip.git

Finally, conﬁrm that the new remote repository, upstream, has been conﬁgured:

$ git remote -v origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push) upstream https://github.com/zulip/zulip.git (fetch) upstream https://github.com/zulip/zulip.git (push)

Step 2: Set up the Zulip development environment

If you haven’t already, now is a good time to install the Zulip development environment (overview). If you’re new to working on Zulip or open source projects in general, we recommend following our detailed guide for ﬁrst-time contributors.

Step 3: Conﬁgure Travis CI (continuous integration)

This step is optional, but recommended. Zulip Server is configured to use Travis CI to test and create builds upon each new commit and pull request. Travis CI is free for open source projects and it’s easy to configure for your own fork of Zulip. After doing so, TravisCI will run tests for new refs you push to GitHub and email you the outcome (you can also view the results in the web interface). First, sign in to Travis CI with your GitHub account and authorize Travis CI to access your GitHub account and repositories. Once you’ve done this, Travis CI will fetch your repository information and display it on your profile page. From there you can enable integration with Zulip. (See screen cast.)

Using Git as you work

Know what branch you’re working on

When using Git, it’s important to know which branch you currently have checked out because most git commands implicitly operate on the current branch. You can determine the currently checked out branch several ways. One way is with git status:

$ git status On branch issue-demo nothing to commit, working directory clean

Another is with git branch which will display all local branches, with a star next to the current branch:

148 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

$ git branch * issue-demo master

To see even more information about your branches, including remote branches, use git branch -vva:

$ git branch -vva * issue-123 517468b troubleshooting tip about provisioning master f0eaee6 [origin/master] bug: Fix traceback in get_missed_

˓→message_token_from_address(). remotes/origin/HEAD -> origin/master remotes/origin/issue-1234 4aeccb7 Another test commit, with longer message. remotes/origin/master f0eaee6 bug: Fix traceback in get_missed_message_token_

˓→from_address(). remotes/upstream/master dbeab6a Optimize checks of test database state by moving

˓→into Python.

You can also conﬁgure Bash and Zsh to display the current branch in your prompt.

Keep your fork up to date

You’ll want to keep your fork up-to-date with changes from Zulip’s main repositories. Note about git pull: You might be used to using git pull on other projects. With Zulip, because we don’t use merge commits, you’ll want to avoid it. Rather that using git pull, which by default is a shortcut for git fetch && git merge FETCH_HEAD (docs), you should use git fetch and then git rebase. First, fetch changes from Zulip’s upstream repository you conﬁgured in the step above:

$ git fetch upstream

Next, checkout your master branch and rebase it on top of upstream/master:

$ git checkout master Switched to branch 'master'

$ git rebase upstream/master

This will rollback any changes you’ve made to master, update it from upstream/master, and then re-apply your changes. Rebasing keeps the commit history clean and readable. When you’re ready, push your changes to your remote fork. Make sure you’re in branch master and the run git push:

$ git checkout master $ git push origin master

You can keep any branch up to date using this method. If you’re working on a feature branch (see next section), which we recommend, you would change the command slightly, using the name of your feature-branch rather than master:

$ git checkout feature-branch Switched to branch 'feature-branch'

$ git rebase upstream/master

$ git push origin feature-branch

26.6. Using Git as you work 149 Zulip Documentation, Release 1.4.0

Work on a feature branch

One way to keep your work organized is to create a branch for each issue or feature. Recall from how Git is different that Git is designed for lightweight branching and merging. You can and should create as many branches as you’d like. First, make sure your master branch is up-to-date with Zulip upstream (see how). Next, from your master branch, create a new tracking branch, providing a descriptive name for your feature branch:

$ git checkout master Switched to branch 'master'

$ git checkout -b issue-1755-fail2ban Switched to a new branch 'issue-1755-fail2ban'

Alternatively, you can create a new branch explicitly based off upstream/master:

$ git checkout -b issue-1755-fail2ban upstream/master Switched to a new branch 'issue-1755-fail2ban'

Now you’re ready to work on the issue or feature.

Run linters and tests locally

In addition to having Travis run tests and linters each time you push a new commit, you can also run them locally. See testing for details.

Stage changes

Recall that files tracked with Git have possible three states: committed, modified, and staged. To prepare a commit, first add the files with changes that you want to include in your commit to your staging area. You add both new files and existing ones. You can also remove files from staging when necessary.

Get status of working directory

To see what ﬁles in the working directory have changes that have not been staged, use git status. If you have no changes in the working directory, you’ll see something like this:

$ git status On branch issue-123 nothing to commit, working directory clean

If you have unstaged changes, you’ll see something like this:

On branch issue-123 Untracked files: (use"git add ..." to include in what will be committed)

newfile.py

nothing added to commit but untracked files present (use"git add" to track)

150 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

Stage additions with git add

To add changes to your staging area, use git add . Because git add is all about staging the changes you want to commit, you use it to add new ﬁles as well as ﬁles with changes to your staging area. Continuing our example from above, after we run git add newfile.py, we’ll see the following from git status:

On branch issue-123 Changes to be committed: (use"git reset HEAD ..." to unstage)

new file: newfile.py

You can view the changes in files you have staged with git diff --cached. To view changes to files you haven’t yet staged, just use git diff. If you want to add all changes in the working directory, use git add -A (documentation). You can also stage changes using your graphical Git client. If you stage a file, you can undo it with git reset HEAD . Here’s an example where we stage a file test3.txt and then unstage it:

$ git add test3.txt On branch issue-1234 Changes to be committed: (use "git reset HEAD ..." to unstage)

new file: test3.txt

$ git reset HEAD test3.txt $ git status On branch issue-1234 Untracked files: (use "git add ..." to include in what will be committed)

test3.txt

nothing added to commit but untracked files present (use "git add" to track)

Stage deletions with git rm

To remove existing files from your repository, use git rm (documentation). This command can either stage the file for removal from your repository AND delete it from your working directory or just stage the file for deletion and leave it in your working directory. To stage a file for deletion and remove it from your working directory, use git rm :

$ git rm test.txt rm 'test.txt'

$ git status On branch issue-1234 Changes to be committed: (use "git reset HEAD ..." to unstage)

deleted: test.txt

26.6. Using Git as you work 151 Zulip Documentation, Release 1.4.0

$ ls test.txt ls: No such file or directory

To stage a ﬁle for deletion and keep it in your working directory, use git rm --cached :

$ git rm --cached test2.txt rm 'test2.txt'

$ git status On branch issue-1234 Changes to be committed: (use "git reset HEAD ..." to unstage)

deleted: test2.txt

$ ls test2.txt test2.txt

If you stage a ﬁle for deletion with the --cached option, and haven’t yet run git commit, you can undo it with git reset HEAD :

$ git reset HEAD test2.txt

Unfortunately, you can’t restore a ﬁle deleted with git rm if you didn’t use the --cache option. However, git rm only deletes ﬁles it knows about. Files you have never added to git won’t be deleted.

Commit changes

When you’ve staged all your changes, you’re ready to commit. You can do this with git commit -m "My commit message." to include a commit message. Here’s an example of committing with the -m for a one-line commit message:

$ git commit -m "Add a test commit for docs." [issue-123 173e17a] Add a test commit for docs. 1 file changed, 1 insertion(+) create mode 100644 newfile.py

You can also use git commit without the -m option and your editor to open, allowing you to easily draft a multi-line commit message. How long your commit message should be depends on where you are in your work. Using short, one-line messages for commits related to in-progress work makes sense. For a commit that you intend to be ﬁnal or that encompasses a signiﬁcant amount or complex work, you should include a longer message. Keep in mind that your commit should contain a ‘minimal coherent idea’ and have a quality commit message. See Zulip docs Commit Discipline and Commit messages for details. Here’s an example of a longer commit message that will be used for a pull request:

Integrate Fail2Ban.

Updates Zulip logging to put an unambiguous entry into the logs such that fail2ban can be configured to look for these entries.

Tested on my local Ubuntu development server, but would appreciate

152 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

someone testing on a production install with more users.

Fixes #1755.

The first line is the summary. It’s a complete sentence, ending in a period. It uses a present-tense action verb, “Integrate”, rather than “Integrates” or “Integrating”. The following paragraphs are full prose and explain why and how the change was made. It explains what testing was done and asks specifically for further testing in a more production-like environment. The final paragraph indicates that this commit addresses and fixes issue #1755. When you submit your pull request, GitHub will detect and link this reference to the appropriate issue. Once your commit is merged into zulip/master, GitHub will automatically close the referenced issue. See Closing issues via commit messages for details. Make as many commits as you need to to address the issue or implement your feature.

Push your commits to GitHub

As you’re working, it’s a good idea to frequently push your changes to GitHub. This ensures your work is backed up should something happen to your local machine and allows others to follow your progress. It also allows you to work from multiple computers without losing work. Pushing to a feature branch is just like pushing to master:

$ git push origin Counting objects: 6, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git * [new branch] issue-demo -> issue-demo

If you want to see what git will do without actually performing the push, add the -n (dry-run) option: git push -n origin . If everything looks good, re-run the push command without -n. If the feature branch does not already exist on GitHub, it will be created when you push and you’ll see * [new branch] in the command output.

Examine and tidy your commit history

Examining your commit history prior to submitting your pull request is a good idea. Is it tidy such that each commit represents a minimally coherent idea (see commit discipline)? Do your commit messages follow Zulip’s style? Will the person reviewing your commit history be able to clearly understand your progression of work? On the command line, you can use the git log command to display an easy to read list of your commits:

$ git log --all --graph --oneline --decorate

* 4f8d75d (HEAD -> 1754-docs-add-git-workflow) docs: Add details about configuring ˓→Travis CI. * bfb2433 (origin/1754-docs-add-git-workflow) docs: Add section for keeping fork up- ˓→to-date to Git Guide. * 4fe10f8 docs: Add sections for creating and configuring fork to Git Guide. * 985116b docs: Add graphic client recs to Git Guide. * 3c40103 docs: Add stubs for remaining Git Guide sections.

26.6. Using Git as you work 153 Zulip Documentation, Release 1.4.0

* fc2c01e docs: Add git guide quickstart. | * f0eaee6 (upstream/master) bug: Fix traceback in get_missed_message_token_from_ ˓→address().

Alternatively, use your graphical client to view the history for your feature branch. If you need to update any of your commits, you can do so with an interactive rebase. Common reasons to use an interactive rebase include: • squashing several commits into fewer commits • splitting a single commit into two or more • rewriting one or more commit messages There is ample documentation on how to rebase, so we won’t go into details here. We recommend starting with GitHub’s help article on rebasing and then consulting Git’s documentation for git-rebase if you need more details. If all you need to do is edit the commit message for your last commit, you can do that with git commit --amend. See Git Basics - Undoing Things for details on this and other useful commands.

Force-push changes to GitHub after you’ve altered your history

Any time you alter history for commits you have already pushed to GitHub, you’ll need to preﬁx the name of your branch with a +. Without this, your updates will be rejected with a message such as:

$ git push origin 1754-docs-add-git-workflow To [email protected]:christi3k/zulip.git ! [rejected] 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (non-fast-

˓→forward) error: failed to push some refs to '[email protected]:christi3k/zulip.git' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Re-running the command with + allows the push to continue by re-writing the history for the remote repository:

$ git push origin +1754-docs-add-git-workflow Counting objects: 12, done. Delta compression using up to 4 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 3.71 KiB | 0 bytes/s, done. Total 12 (delta 8), reused 0 (delta 0) remote: Resolving deltas: 100% (8/8), completed with 2 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (forced

˓→update)

This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the branch. If others are working along with you, they might run into complications when they retrieve your changes because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.

154 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

Create a pull request

When you’re ready for feedback, submit a pull request. At Zulip we recommend submitting work-in-progress pull requests early and often. This allows you to get feedback and help with your bugfix or feature. Prefix work-in-progress pull requests with [WIP]. Pull requests are a feature specific to GitHub. They provide a simpler, web-based way to submit your work (often called “patches”) to a project. It’s called a pull request because you’re asking the project to pull changes from your fork. If you’re unfamiliar with how to create a pull request, checkout GitHub’s documentation on creating a pull request from a fork. You might also find GitHub’s article about pull requests helpful.

Step 1: Update your branch with git rebase

The best way to update your branch is with git fetch and git rebase. Do not use git pull or git merge as this will create merge commits. See keep your fork up to date for details. Here’s an example (you would replace issue-123 with the name of your feature branch):

$ git checkout issue123 Switched to branch 'issue-123'

$ git fetch upstream remote: Counting objects: 69, done. remote: Compressing objects: 100% (23/23), done. remote: Total 69 (delta 49), reused 39 (delta 39), pack-reused 7 Unpacking objects: 100% (69/69), done. From https://github.com/zulip/zulip 69fa600..43e21f6 master -> upstream/master

$ git rebase upstream/master

First, rewinding head to replay your work on top of it... Applying: troubleshooting tip about provisioning

Step 2: Push your updated branch to your remote fork

Once you’ve updated your local feature branch, push the changes to GitHub:

$ git push origin issue-123 Counting objects: 6, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 issue-123 -> issue-123

If your push is rejected with error failed to push some refs then you need to preﬁx the name of your branch with a +:

$ git push origin +issue-123 Counting objects: 6, done. Delta compression using up to 4 threads.

26.7. Create a pull request 155 Zulip Documentation, Release 1.4.0

Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 issue-123 -> issue-123 (forced update)

Step 3: Open the pull request

If you’ve never created a pull request or need a refresher, take a look at GitHub’s article creating a pull request from a fork. We’ll brieﬂy review the process here. The ﬁrst step in creating a pull request is to use your web browser to navigate to your fork of Zulip. Sign in to GitHub if you haven’t already. Next, navigate to the branch you’ve been working on. Do this by clicking on the Branch button and selecting the relevant branch. Finally, click the New pull request button. Alternatively, if you’ve recently pushed to your fork, you will see a green Compare & pull request button. You’ll see the Open a pull request page:

156 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

Provide a title and ﬁrst comment for your pull request. When ready, click the green Create pull request to submit the pull request. Note: Pull request titles are different from commit messages. Commit messages can be edited with git commit --amend, git rebase -i, etc., while the title of a pull request can only be edited via GitHub.

Update a pull request

As you get make progress on your feature or bugfix, your pull request, once submitted, will be updated each time you push commits to your remote branch. This means you can keep your pull request open as long as you need, rather than closing and opening new ones for the same feature or bugfix. It’s a good idea to keep your pull request mergeable with Zulip upstream by frequently fetching, rebasing, and pushing changes. See keep your fork up to date for details. You might also find this excellent article How to Rebase a Pull Request helpful. And, as you address review comments others have made, we recommend posting a follow-up comment in which you: a) ask for any clarifications you need, b) explain to the reviewer how you solved any problems they mentioned, and c) ask for another review.

Collaborate

Fetch another contributor’s branch

What happens when you would like to collaborate with another contributor and they have work-in-progress on their own fork of Zulip? No problem! Just add their fork as a remote and pull their changes.

$ git remote add https://github.com//zulip.git $ git fetch

Now you can checkout their branch just like you would any other. You can name the branch anything you want, but using both the username and branch name will help you keep things organized.

$ git checkout -b -

Checkout a pull request locally

Just as you can checkout any user’s branch locally, you can also checkout any pull request locally. GitHub provides a special syntax (details) for this since pull requests are speciﬁc to GitHub rather than Git. First, fetch and create a branch for the pull request, replacing ID and BRANCHNAME with the ID of the pull request and your desired branch name:

$ git fetch upstream pull/ID/head:BRANCHNAME

Now switch to the branch:

$ git checkout BRANCHNAME

Now you work on this branch as you would any other.

26.8. Update a pull request 157 Zulip Documentation, Release 1.4.0

Review changes

Changes on (local) working tree

Display changes between index and working tree (what is not yet staged for commit):

$ git diff

Display changes between index and last commit (what you have staged for commit):

$ git diff --cached

Display changes in working tree since last commit (changes that are staged as well as ones that are not):

$ git diff HEAD

Changes within branches

Use any git-ref to compare changes between two commits on the current branch. Display changes between commit before last and last commit:

$ git diff HEAD^ HEAD

Display changes between two commits using their hashes:

$ git diff e2f404c 7977169

Changes between branches

Display changes between tip of topic branch and tip of master branch:

$ git diff topic master

Display changes that have occurred on master branch since topic branch was created:

$ git diff topic...master

Display changes you’ve committed so far since creating a branch from upstream/master:

$ git diff upstream/master...HEAD

Get and stay out of trouble

Git is a powerful yet complex version control system. Even for contributors experienced at using version control, it can be confusing. The good news is that nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to make Git perform actions that you can’t undo. However, git can’t undo what it doesn’t know about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote repository.

158 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

Undo a merge commit

A merge commit is a special type of commit that has two parent commits. It’s created by Git when you merge one branch into another and the last commit on your current branch is not a direct ancestor of the branch you are trying to merge in. This happens quite often in a busy project like Zulip where there are many contributors because upstream/zulip will have new commits while you’re working on a feature or bugfix. In order for Git to merge your changes and the changes that have occurred on zulip/upstream since you first started your work, it must perform a three-way merge and create a merge commit. Merge commits aren’t bad, however, Zulip don’t use them. Instead Zulip uses a forked-repo, rebase-oriented workflow. A merge commit is usually created when you’ve run git pull or git merge. You’ll know you’re creating a merge commit if you’re prompted for a commit message and the default is something like this:

Merge branch'master' of https://github.com/zulip/zulip

# Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

And the ﬁrst entry for git log will show something like: commit e5f8211a565a5a5448b93e98ed56415255546f94 Merge: 13bea0e e0c10ed Author: Christie Koehler Date: Mon Oct 10 13:25:51 2016-0700

Merge branch'master' of https://github.com/zulip/zulip

Some graphical Git clients may also create merge commits. To undo a merge commit, ﬁrst run git reflog to identify the commit you want to roll back to:

$ git reflog e5f8211 HEAD@{0}: pull upstream master: Merge made by the 'recursive' strategy. 13bea0e HEAD@{1}: commit: test commit for docs.

Reﬂog output will be long. The most recent git refs will be listed at the top. In the example above e5f8211 HEAD@{0}: is the merge commit made automatically by git pull and 13bea0e HEAD@{1}: is the last commit I made before running git pull, the commit that I want to rollback to. Once you’d identiﬁed the ref you want to revert to, you can do so with git reset:

$ git reset --hard 13bea0e HEAD is now at 13bea0e test commit for docs.

Important: git reset --hard will discard all changes in your working directory and index since the commit you’re resetting to with . This is the main way you can lose work in Git. If you need to keep any changes that are in your working directory or that you have committed, use git reset --merge instead. You can also use the relative reﬂog HEAD@{1} instead of the commit hash, just keep in mind this changes as you run git commands. Now when I look at the git reﬂog, I see the tip of my branch is pointing to my last commit 13bea0e before the merge:

26.11. Get and stay out of trouble 159 Zulip Documentation, Release 1.4.0

$ git reflog

13bea0e HEAD@{2}: reset: moving to HEAD@{1} e5f8211 HEAD@{3}: pull upstream master: Merge made by the 'recursive' strategy. 13bea0e HEAD@{4}: commit: test commit for docs.

And the ﬁrst entry git log shows is this: commit 13bea0e40197b1670e927a9eb05aaf50df9e8277 Author: Christie Koehler Date: Mon Oct 10 13:25:38 2016-0700

test commit for docs.

Restore a lost commit

We’ve mentioned you can use git reset --hard to rollback to a previous commit. What if you run git reset --hard and then realize you actually need one or more of the commits you just discarded? No problem, you can restore them with git cherry-pick (docs). For example, let’s say you just committed “some work” and your git log looks like this:

* 67aea58 (HEAD-> master) some work * 13bea0e test commit for docs.

You then mistakenly run git reset --hard 13bea0e:

$ git reset --hard 13bea0e HEAD is now at 13bea0e test commit for docs.

$ git log * 13bea0e (HEAD -> master) test commit for docs.

And then realize you actually needed to keep commit 67aea58. First, use git reflog to conﬁrm that commit you want to restore and then run git cherry-pick :

$ git reflog 13bea0e HEAD@{0}: reset: moving to 13bea0e 67aea58 HEAD@{1}: commit: some work

$ git cherry-pick 67aea58 [master 67aea58] some work Date: Thu Oct 13 11:51:19 2016 -0700 1 file changed, 1 insertion(+) create mode 100644 test4.txt

Recover from a git rebase failure

One situation in which git rebase will fail and require you to intervene is when your change, which git will try to re-apply on top of new commits from which ever branch you are rebasing on top of, is to code that has been changed by those new commits. For example, while I’m working on a ﬁle, another contributor makes a change to that ﬁle, submits a pull request and has their code merged into master. Usually this is not a problem, but in this case the other contributor made a change

160 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0 to a part of the ﬁle I also want to change. When I try to bring my branch up to date with git fetch and then git rebase upstream/master, I see the following:

First, rewinding head to replay your work on top of it... Applying: test change for docs Using index info to reconstruct a base tree... M README.md Falling back to patching base and 3-way merge... Auto-merging README.md CONFLICT (content): Merge conflict in README.md error: Failed to merge in the changes. Patch failed at 0001 test change for docs The copy of the patch that failed is found in:.git/rebase-apply/patch

When you have resolved this problem, run"git rebase --continue". If you prefer to skip this patch, run"git rebase --skip" instead. To check out the original branch and stop rebasing, run"git rebase --abort".

This message tells me that Git was not able to apply my changes to README.md after bringing in the new commits from upstream/master. Running git status also gives me some information: rebase in progress; onto5ae56e6 You are currently rebasing branch'docs-test' on'5ae56e6'. (fix conflicts and then run"git rebase --continue") (use"git rebase --skip" to skip this patch) (use"git rebase --abort" to check out the original branch)

Unmerged paths: (use"git reset HEAD ..." to unstage) (use"git add ..." to mark resolution)

both modified: README.md no changes added to commit (use"git add" and/or "git commit -a")

To fix, open all the files with conflicts in your editor and decide which edits should be applied. Git uses standard conflict-resolution (<<<<<<<, ======, and >>>>>>>) markers to indicate where in files there are conflicts. Once you’ve done that, save the file(s), stage them with git add and then continue the rebase with git rebase --continue:

$ git add README.md

$ git rebase --continue Applying: test change for docs

For help resolving merge conflicts, see basic merge conflicts, advanced merging, and/or GitHub’s help on how to resolve a merge conflict.

Working from multiple computers

Working from multiple computers with Zulip and Git is ﬁne, but you’ll need to pay attention and do a bit of work to ensure all of your work is readily available. Recall that most Git operations are local. When you commit your changes with git commit they are safely stored in your local Git database only. That is, until you push the commits to GitHub, they are only available on the computer

26.11. Get and stay out of trouble 161 Zulip Documentation, Release 1.4.0 where you committed them. So, before you stop working for the day, or before you switch computers, push all of your commits to GitHub with git push:

$ git push origin

When you ﬁrst start working on a new computer, you’ll clone the Zulip repository and connect it to Zulip upstream. A clone retrieves all current commits, including the ones you pushed to GitHub from your other computer. But if you’re switching to another computer on which you have already cloned Zulip, you need to update your local Git database with new refs from your GitHub fork. You do this with git fetch:

$ git fetch

Ideally you should do this before you have made any commits on the same branch on the second computer. Then you can git merge on whichever branch you need to update:

$ git checkout Switched to branch ''

$ git merge origin/master

If you have already made commits on the second computer that you need to keep, you’ll need to use git log FETCH_HEAD to identify that hashes of the commits you want to keep and then git cherry-pick those commits into whichever branch you need to update.

Zulip-speciﬁc tools

This section will document the zulip-speciﬁc git tools contributors will ﬁnd helpful.

Set up git repo script

In the tools directory of zulip/zulip you’ll ﬁnd a bash script setup-git-repo. This script installs the Zulip pre-commit hook. This hook will run each time you git commit to automatically run linters, etc. The hook passes no matter the result of the linter, but you should still pay attention to any notices or warnings it displays. It’s simple to use. Make sure you’re in the clone of zulip and run the following:

$ ./tools/setup-git-repo

The script doesn’t produce any output if successful. To check that the hook has been installed, print a directory listing for .git/hooks and you should see something similar to:

$ ls -l .git/hooks pre-commit -> ../../tools/pre-commit

Reset to pull request tools/reset-to-pull-request is a short-cut for checking out a pull request locally. It works slightly differ- ently from the method described above in that it does not create a branch for the pull request checkout. This tool checks for uncommitted changes, but it will move the current branch using git reset --hard. Use with caution.

162 Chapter 26. Git & GitHub Guide Zulip Documentation, Release 1.4.0

First, make sure you are working in a branch you want to move (in this example, we’ll use the local master branch). Then run the script with the ID number of the pull request as the ﬁrst argument.

$ git checkout master Switched to branch 'master' Your branch is up-to-date with 'origin/master'.

$ ./tools/reset-to-pull-request 1900 + request_id=1900 + git fetch upstream pull/1900/head remote: Counting objects: 159, done. remote: Compressing objects: 100% (17/17), done. remote: Total 159 (delta 94), reused 91 (delta 91), pack-reused 51 Receiving objects: 100% (159/159), 55.57 KiB | 0 bytes/s, done. Resolving deltas: 100% (113/113), completed with 54 local objects. From https://github.com/zulip/zulip * branch refs/pull/1900/head -> FETCH_HEAD + git reset --hard FETCH_HEAD HEAD is now at 2bcd1d8 troubleshooting tip about provisioning

Fetch a pull request and rebase tools/fetch-rebase-pull-request is a short-cut for checking out a pull request locally in its own branch and then updating it with any changes from upstream/master with git rebase. First, make sure you are working in branch master. Then run the script with the ID number of the pull request as the ﬁrst argument.

$ tools/fetch-rebase-pull-request 1913 + request_id=1913 + git fetch upstream pull/1913/head remote: Counting objects: 4, done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (4/4), done. From https://github.com/zulip/zulip * branch refs/pull/1913/head -> FETCH_HEAD + git checkout upstream/master -b review-1913 Branch review-1913 set up to track remote branch master from upstream. Switched to a new branch 'review-1913' + git reset --hard FETCH_HEAD HEAD is now at 99aa2bf Add provision.py fails issue in common erros + git pull --rebase Current branch review-1913 is up to date.

26.12. Zulip-speciﬁc tools 163 Zulip Documentation, Release 1.4.0

164 Chapter 26. Git & GitHub Guide CHAPTER 27

Version control

Commit Discipline

We follow the Git project’s own commit discipline practice of “Each commit is a minimal coherent idea”. This discipline takes a bit of work, but it makes it much easier for code reviewers to spot bugs, and makes the commit history a much more useful resource for developers trying to understand why the code works the way it does, which also helps a lot in preventing bugs. Coherency requirements for any commit: • It should pass tests (so test updates needed by a change should be in the same commit as the original change, not a separate “fix the tests that were broken by the last commit” commit). • It should be safe to deploy individually, or comment in detail in the commit message as to why it isn’t (maybe with a [manual] tag). So implementing a new API endpoint in one commit and then adding the security checks in a future commit should be avoided – the security checks should be there from the beginning. • Error handling should generally be included along with the code that might trigger the error. • TODO comments should be in the commit that introduces the issue or functionality with further work required. When you should be minimal: • Significant refactorings should be done in a separate commit from functional changes. • Moving code from one file to another should be done in a separate commits from functional changes or even refactoring within a file. • 2 different refactorings should be done in different commits. • 2 different features should be done in different commits. • If you find yourself writing a commit message that reads like a list of somewhat dissimilar things that you did, you probably should have just done 2 commits. When not to be overly minimal: • For completely new features, you don’t necessarily need to split out new commits for each little subfeature of the new feature. E.g. if you’re writing a new tool from scratch, it’s fine to have the initial tool have plenty of

165 Zulip Documentation, Release 1.4.0

options/features without doing separate commits for each one. That said, reviewing a 2000-line giant blob of new code isn’t fun, so please be thoughtful about submitting things in reviewable units. • Don’t bother to split back end commits from front end commits, even though the backend can often be coherent on its own. Other considerations: • Overly fine commits are easily squashed, but not vice versa, so err toward small commits, and the code reviewer can advise on squashing. • If a commit you write doesn’t pass tests, you should usually fix that by amending the commit to fix the bug, not writing a new “fix tests” commit on top of it. Zulip expects you to structure the commits in your pull requests to form a clean history before we will merge them; it’s best to write your commits following these guidelines in the first place, but if you don’t, you can always fix your history using git rebase -i. Never mix multiple changes together in a single commit, but it’s great to include several related changes, each in their own commit, in a single pull request. If you notice an issue that is only somewhat related to what you were working on, but you feel that it’s too minor to create a dedicated pull request, feel free to append it as an additional commit in the pull request for your main project (that commit should have a clear explanation of the bug in its commit message). This way, the bug gets fixed, but this independent change is highlighted for reviewers. Or just create a dedicated pull request for it. Whatever you do, don’t squash unrelated changes together in a single commit; the reviewer will ask you to split the changes out into their own commits. It can take some practice to get used to writing your commits with a clean history so that you don’t spend much time doing interactive rebases. For example, often you’ll start adding a feature, and discover you need to a refactoring part- way through writing the feature. When that happens, we recommend stashing your partial feature, do the refactoring, commit it, and then finish implementing your feature.

Commit Messages

First, check out these examples of commits with good commit messages. There are a few mistakes we see often that make commit messages less useful than they could be. • The ﬁrst line of commit messages should be written in the imperative and be kept relatively short while concisely explaining what the commit does. And it should be clear about what part of the code is affected. For example: Good: gather_subscriptions: Fix exception handling bad input. zjsunit: Fix running stream_data and node tests individually. Add GitLab integration.

Bad: bugfix gather_subscriptions was broken fixing exception when given bad input fix bug #234.

• Use present-tense action verbs in your commit messages. Bad: gather_subscriptions: Fixing exception when given bad input. gather_subscriptions: Fixed exception when given bad input.

166 Chapter 27. Version control Zulip Documentation, Release 1.4.0

• Please use a complete sentence in the summary, ending with a period. • The rest of the commit message should be written in full prose and explain why and how the change was made. If the commit makes performance improvements, you should generally include some rough benchmarks showing that it actually improves the performance. • When you fix a GitHub issue, mark that you’ve fixed the issue in your commit message so that the issue is automatically closed when your code is merged. Zulip’s preferred style for this is to have the final paragraph of the commit message read e.g. “Fixes: #123.” • Any paragraph content in the commit message should be line-wrapped to less than 76 characters per line, so that your commit message will be reasonably readable in git log in a normal terminal. • In your commit message, you should describe any manual testing you did in addition to running the automated tests, and any aspects of the commit that you think are questionable and you’d like special attention applied to.

27.2. Commit Messages 167 Zulip Documentation, Release 1.4.0

168 Chapter 27. Version control CHAPTER 28

Code style and conventions

Be consistent!

Look at the surrounding code, or a similar part of the project, and try to do the same thing. If you think the other code has actively bad style, ﬁx it (in a separate commit). When in doubt, send an email to [email protected] with your question.

Lint tools

You can run them all at once with

./tools/lint

You can set this up as a local Git commit hook with

``tools/setup-git-repo``

The Vagrant setup process runs this for you. lint runs many lint checks in parallel, including • JavaScript (ESLint) • Python (Pyﬂakes) • templates • Puppet conﬁguration • custom checks (e.g. trailing whitespace and spaces-not-tabs)

169 Zulip Documentation, Release 1.4.0

Secrets

Please don’t put any passwords, secret access keys, etc. inline in the code. Instead, use the get_secret function in zproject/settings.py to read secrets from /etc/zulip/secrets.conf.

Dangerous constructs

Misuse of database queries

Look out for Django code like this:

[Foo.objects.get(id=bar.x.id) for bar in Bar.objects.filter(...) if bar.baz<7]

This will make one database query for each Bar, which is slow in production (but not in local testing!). Instead of a list comprehension, write a single query using Django’s QuerySet API. If you can’t rewrite it as a single query, that’s a sign that something is wrong with the database schema. So don’t defer this optimization when performing schema changes, or else you may later ﬁnd that it’s impossible.

UserProﬁle.objects.get() / Client.objects.get / etc.

In our Django code, never do direct UserProfile.objects.get(email=foo) database queries. Instead always use get_user_profile_by_{email,id}. There are 3 reasons for this: 1. It’s guaranteed to correctly do a case-inexact lookup 2. It fetches the user object from remote cache, which is faster 3. It always fetches a UserProﬁle object which has been queried using .select_related(), and thus will perform well when one later accesses related models like the Realm. Similarly we have get_client and get_stream functions to fetch those commonly accessed objects via remote cache.

Using Django model objects as keys in sets/dicts

Don’t use Django model objects as keys in sets/dictionaries – you will get unexpected behavior when dealing with objects obtained from different database queries: For example, UserProfile.objects.only("id").get(id=17) in set([UserProfile. objects.get(id=17)]) is False You should work with the IDs instead. user_proﬁle.save()

You should always pass the update_fields keyword argument to .save() when modifying an existing Django model object. By default, .save() will overwrite every value in the column, which results in lots of race conditions where unrelated changes made by one thread can be accidentally overwritten by another thread that fetched its UserProfile object before the first thread wrote out its change.

170 Chapter 28. Code style and conventions Zulip Documentation, Release 1.4.0

Using raw saves to update important model objects

In most cases, we already have a function in zephyr/lib/actions.py with a name like do_activate_user that will correctly handle lookups, caching, and notifying running browsers via the event system about your change. So please check whether such a function exists before writing new code to modify a model object, since your new code has a good chance of getting at least one of these things wrong.

Naive datetime objects

Python allows datetime objects to not have an associated timezone, which can cause time-related bugs that are hard to catch with a test suite, or bugs that only show up during daylight savings time. Good ways to make timezone-aware datetimes are below. We import timezone function as from django. utils.timezone import now as timezone_now and from django.utils.timezone import utc as timezone_utc. When Django is not available, timezone_utc should be replaced with pytz.utc below. • timezone_now() when Django is available, such as in zerver/. • datetime.now(tz=pytz.utc) when Django is not available, such as for bots and scripts. • datetime.fromtimestamp(timestamp, tz=timezone_utc) if creating a datetime from a timestamp. This is also available as zerver.lib.timestamp.timestamp_to_datetime. • datetime.strptime(date_string, format).replace(tzinfo=timezone_utc) if creating a datetime from a formatted string that is in UTC. Idioms that result in timezone-naive datetimes, and should be avoided, are datetime.now() and datetime. fromtimestamp(timestamp) without a tz parameter, datetime.utcnow() and datetime. utcfromtimestamp(), and datetime.strptime(date_string, format) without replacing the tzinfo at the end. Additional notes: • Especially in scripts and puppet conﬁguration where Django is not available, using time.time() to get timestamps can be cleaner than dealing with datetimes. • All datetimes on the backend should be in UTC, unless there is a good reason to do otherwise. x.attr('zid') vs. rows.id(x)

Our message row DOM elements have a custom attribute zid which contains the numerical message ID. Don’t access this directly as x.attr('zid') ! The result will be a string and comparisons (e.g. with <=) will give the wrong result, occasionally, just enough to make a bug that’s impossible to track down. You should instead use the id function from the rows module, as in rows.id(x). This returns a number. Even in cases where you do want a string, use the id function, as it will simplify future code changes. In most contexts in JavaScript where a string is needed, you can pass a number without any explicit conversion.

JavaScript var

Always declare JavaScript variables using var: var x=...;

28.4. Dangerous constructs 171 Zulip Documentation, Release 1.4.0

In a function, var is necessary or else x will be a global variable. For variables declared at global scope, this has no effect, but we do it for consistency. JavaScript has function scope only, not block scope. This means that a var declaration inside a for or if acts the same as a var declaration at the beginning of the surrounding function. To avoid confusion, declare all variables at the top of a function.

JavaScript for (i in myArray)

Don’t use it: [1], [2], [3] jQuery global state

Don’t mess with jQuery global state once the app has loaded. Code like this is very dangerous:

$.ajaxSetup({ async: false }); $.get(...); $.ajaxSetup({ async: true }); jQuery and the browser are free to run other code while the request is pending, which could perform other Ajax requests with the altered settings. Instead, switch to the more general $.ajax_ function, which can take options like async.

Translation tags

Remember to tag all user-facing strings for translation, whether they are in HTML templates or JavaScript editing the HTML (e.g. error messages).

State and logs ﬁles

Do not write state and logs ﬁles inside the current working directory in the production environment. This will not do what you expect, because the current working directory for the app changes every time we do a deploy. Instead, hardcode a path in settings.py – see SERVER_LOG_PATH in settings.py for an example.

JS array/object manipulation

For generic functions that operate on arrays or JavaScript objects, you should generally use Underscore. We used to use jQuery’s utility functions, but the Underscore equivalents are more consistent, better-behaved and offer more choices. A quick conversion table:

$.each → _.each (parameters to the callback reversed) $.inArray → _.indexOf (parameters reversed) $.grep → _.filter $.map → _.map $.extend → _.extend

There’s a subtle difference in the case of _.extend; it will replace attributes with undeﬁned, whereas jQuery won’t:

172 Chapter 28. Code style and conventions Zulip Documentation, Release 1.4.0

$.extend({foo: 2}, {foo: undefined}); // yields {foo: 2}, BUT... _.extend({foo: 2}, {foo: undefined}); // yields {foo: undefined}!

Also, _.each does not let you break out of the iteration early by returning false, the way jQuery’s version does. If you’re doing this, you probably want _.find, _.every, or _.any, rather than ‘each’. Some Underscore functions have multiple names. You should always use the canonical name (given in large print in the Underscore documentation), with the exception of _.any, which we prefer over the less clear ‘some’.

More arbitrary style things

General

Indentation is four space characters for Python, JS, CSS, and shell scripts. Indentation is two space characters for HTML templates. We never use tabs anywhere in source code we write, but we have some third-party files which contain tabs. Keep third-party static files under the directory zephyr/static/third/, with one subdirectory per third-party project. We don’t have an absolute hard limit on line length, but we should avoid extremely long lines. A general guideline is: refactor stuff to get it under 85 characters, unless that makes the code a lot uglier, in which case it’s fine to go up to 120 or so. Whitespace guidelines: • Put one space (or more for alignment) around binary arithmetic and equality operators. • Put one space around each part of the ternary operator. • Put one space between keywords like if and while and their associated open paren. • Put one space between the closing paren for if and while-like constructs and the opening curly brace. Put the curly brace on the same line unless doing otherwise improves readability. • Put no space before or after the open paren for function calls and no space before the close paren for function calls. • For the comma operator and colon operator in languages where it is used for inline dictionaries, put no space before it and at least one space after. Only use more than one space for alignment.

JavaScript

Don’t use == and != because these operators perform type coercions, which can mask bugs. Always use === and !==. End every statement with a semicolon. if statements with no braces are allowed, if the body is simple and its extent is abundantly clear from context and formatting. Anonymous functions should have spaces before and after the argument list: var x= function (foo, bar) {//...

When calling a function with an anonymous function as an argument, use this style:

28.6. More arbitrary style things 173 Zulip Documentation, Release 1.4.0

$.get('foo', function (data) { var x = ...; // ... });

The inner function body is indented one level from the outer function call. The closing brace for the inner function and the closing parenthesis for the outer call are together on the same line. This style isn’t necessarily appropriate for calls with multiple anonymous functions or other arguments following them. Use

$(function () { ... rather than

$(document).ready(function () { ... and combine adjacent on-ready functions, if they are logically related. The best way to build complicated DOM elements is a Mustache template like zephyr/static/templates/ message.handlebars. For simpler things you can use jQuery DOM building APIs like so: var new_tr = $('').attr('id', zephyr.id);

Passing a HTML string to jQuery is ﬁne for simple hardcoded things: foo.append('

foo

'); but avoid programmatically building complicated strings. We used to favor attaching behaviors in templates like so:

but there are some reasons to prefer attaching events using jQuery code: • Potential huge performance gains by using delegated events where possible • When calling a function from an onclick attribute, this is not bound to the element like you might think • jQuery does event normalization Either way, avoid complicated JavaScript code inside HTML attributes; call a helper function instead.

HTML / CSS

Don’t use the style= attribute. Instead, deﬁne logical classes and put your styles in external ﬁles such as zulip. css. Don’t use the tag name in a selector unless you have to. In other words, use .foo instead of span.foo. We shouldn’t have to care if the tag type changes in the future. Don’t use inline event handlers (onclick=, etc. attributes). Instead, attach a jQuery event handler ($('#foo'). on('click', function () {...})) when the DOM is ready (inside a $(function () {...}) block). Use this format when you have the same block applying to multiple CSS styles (separate lines for each selector): selector1, selector2 { };

174 Chapter 28. Code style and conventions Zulip Documentation, Release 1.4.0

Python

• Scripts should start with #!/usr/bin/env python and not #/usr/bin/python (the right Python may not be installed in /usr/bin) or #/usr/bin/env python2.7 (bad for Python 3 compatibility). Don’t put a shebang line on a Python file unless it’s meaningful to run it as a script. (Some libraries can also be run as scripts, e.g. to run a test suite.) • Scripts should be executed directly (./script.py), so that the interpreter is implicitly found from the shebang line, rather than explicitly overridden (python script.py). • The first import in a file should be from __future__ import absolute_import, per PEP 328 • Put all imports together at the top of the file, absent a compelling reason to do otherwise. • Unpacking sequences doesn’t require list brackets:

[x, y]= xs # unnecessary x, y= xs # better

• For string formatting, use x % (y,) rather than x % y, to avoid ambiguity if y happens to be a tuple. • When selecting by id, don’t use foo.pk when you mean foo.id. E.g.

recipient= Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)

should be written as

recipient= Recipient(type_id=huddle.id, type=Recipient.HUDDLE)

in case we ever change the primary keys.

Tests

All signiﬁcant new features should come with tests. See testing.

Third party code

When adding new third-party packages to our codebase, please include “[third]” at the beginning of the commit message. You don’t necessarily need to do this when patching third-party code that’s already in tree.

28.6. More arbitrary style things 175 Zulip Documentation, Release 1.4.0

176 Chapter 28. Code style and conventions CHAPTER 29

Python static type checker (mypy)

mypy is a compile-time static type checker for Python, allowing optional, gradual typing of Python code. Zulip is using mypy’s Python 2 compatible syntax for type annotations, which means that type annotations are written inside comments that start with # type:. Here’s a brief example of the mypy syntax we’re using in Zulip: user_dict={} # type: Dict[str, UserProfile] def get_user_profile_by_email(email): # type: (str) -> UserProfile ... # Actual code of the function here

You can learn more about it at: • The mypy cheat sheet is the best resource for quickly understanding how to write the PEP 484 type annotations used by mypy correctly. • The Python 2 type annotation syntax spec in PEP 484 • Using mypy with Python 2 code The mypy type checker is run automatically as part of Zulip’s Travis CI testing process in the ‘static-analysis’ build. type_debug.py zerver/lib/type_debug.py has a useful decorator print_types. It prints the types of the parameters of the decorated function and the return type whenever that function is called. This can help ﬁnd out what parameter types a function is supposed to accept, or if parameters with the wrong types are being passed to a function. Here is an example using the interactive console:

>>> from zerver.lib.type_debug import print_types >>> >>> @print_types ... def func(x, y): ... return x+y

177 Zulip Documentation, Release 1.4.0

... >>> func(1.0,2) func(float, int) -> float 3.0 >>> func('a','b') func(str, str) -> str 'ab' >>> func((1,2), (3,)) func((int, int), (int,)) -> (int, int, int) (1, 2, 3) >>> func([1,2,3], [4,5,6,7]) func([int, ...], [int, ...]) -> [int, ...] [1, 2, 3, 4, 5, 6, 7] print_all prints the type of the ﬁrst item of lists. So [int, ...] represents a list whose ﬁrst element’s type is int. Types of all items are not printed because a list can have many elements, which would make the output too large. Similarly in dicts, one key’s type and the corresponding value’s type are printed. So {1: 'a', 2: 'b', 3: 'c'} will be printed as {int: str, ...}.

Zulip goals

Zulip is hoping to reach 100% of the codebase annotated with mypy static types, and then enforce that it stays that way. Our current coverage is shown in Coveralls.

Installing mypy

If you installed Zulip’s development environment correctly, mypy should already be installed inside the Python 3 virtualenv at zulip-py3-venv (mypy only supports Python 3). If it isn’t installed (e.g. because you haven’t reprovisioned recently), you can run tools/install-mypy to install it.

Running mypy on Zulip’s code locally

To run mypy on Zulip’s python code, run the command: tools/run-mypy

It will output errors in the same style as a compiler would. For example, if your code has a type error like this: foo=1 foo='1' you’ll get an error like this: test.py: note: In function"test": test.py:200: error: Incompatible types in assignment (expression has type"str",

˓→variable has type"int")

If you need help interpreting or debugging mypy errors, please feel free to mention @sharmaeklavya2 or @timabbott on your pull request (or email [email protected]) to get help; we’d love to both build a great troubleshooting guide in this doc and also help contribute improvements to error messages upstream.

178 Chapter 29. Python static type checker (mypy) Zulip Documentation, Release 1.4.0

Since mypy is a new tool under rapid development and occasionally makes breaking changes, Zulip is using a pinned version of mypy from its git repository rather than tracking the (older) latest mypy release on PyPI.

Excluded ﬁles

Since several Python files in Zulip’s code don’t pass mypy’s checks (even for unannotated code) right now, a list of files to be excluded from the check for CI is present in tools/run-mypy. To run mypy on all Python files, ignoring the exclude list, you can pass the --all option to tools/run-mypy. tools/run-mypy--all

If you type annotate some of those ﬁles so that they pass without errors, please remove them from the exclude list.

Mypy is there to ﬁnd bugs in Zulip before they impact users

For the purposes of Zulip development, you can treat mypy like a much more powerful linter that can catch a wide range of bugs. If, after running tools/run-mypy on your Zulip branch, you get mypy errors, it’s important to get to the bottom of the issue, not just do something quick to silence the warnings. Possible explanations include: • A bug in any new type annotations you added. • A bug in the existing type annotations. • A bug in Zulip! • Some Zulip code is correct but confusingly reuses variables with different types. • A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project). Each explanation has its own solution, but in every case the result should be solving the mypy warning in a way that makes the Zulip codebase better. If you need help understanding an issue, please feel free to mention @shar- maeklavya2 or @timabbott on the relevant pull request or issue on GitHub. If you think you have found a bug in Zulip or mypy, inform the zulip developers by opening an issue on Zulip’s GitHub repository or posting on zulip-devel. If it’s indeed a mypy bug, we can help with reporting it upstream.

Annotating strings

In Python 3, strings can have non-ASCII characters without any problems. Such characters are required to support languages which use non-latin scripts like Japanese and Hindi. They are also needed to support special characters like mathematical symbols, musical symbols, etc. In Python 2, however, str generally doesn’t work well with non-ASCII characters. That’s why unicode was introduced in Python 2. But there are problems with the unicode and str system. Implicit conversions between str and unicode use the ascii codec, which fails on strings containing non-ASCII characters. Such errors are hard to detect by people who always write in English. To minimize such implicit conversions, we should have a strict separation between str and unicode in Python 2. It might seem that using unicode everywhere will solve all problems, but unfortunately it doesn’t. This is because some parts of the standard library and the Python language (like keyword argument unpacking) insist that parameters passed to them are str. To make our code work correctly in Python 2, we have to identify strings which contain data which could come from non-ASCII sources like stream names, people’s names, domain names, content of messages, emails, etc. These strings

29.5. Excluded files 179 Zulip Documentation, Release 1.4.0 should be unicode. We also have to identify strings which should be str like Exception names, attribute names, parameter names, etc. Mypy can help with this. We just have to annotate each string as either str or unicode and mypy’s static type checking will tell us if we are incorrectly mixing the two. However, unicode is not defined in Python 3. We want our code to be Python 3 compatible in the future. This can be achieved using ‘typing.Text’, a Python 2 and 3 compatibility type. typing.Text is defined as str in Python 3 and as unicode in Python 2. We’ll be using Text (instead of unicode) and str to annotate strings in Zulip’s code. We follow the style of doing from typing import Text and using Text for annotation instead of doing import typing and using typing.Text for annotation, because Text is used so extensively for type annotations that we don’t need to be that verbose. Sometimes you’ll find that you have to convert strings from one type to another. zerver/lib/str_utils.py has utility functions to help with that. It also has documentation (in docstrings) which explains the right way to use them.

180 Chapter 29. Python static type checker (mypy) CHAPTER 30

Reviewing Zulip server code

This document is a brief discussion of what we look for when reviewing contributions to Zulip. It’s meant partially for developers who want to get their code merged faster, and partially for developers who have made successful pull requests already and would like to start participating in code review.

Things to look for

• The Travis CI build. The tests need to pass. One can investigate any failures and figure out what to fix by clicking on a red X next to the commit hash or the Detail links on a pull request. (Example: in #1219, click the red X next to f1f474e to see the build jobs for that commit, at least one of which has failed. Click on the link for Travis continuous integrations details to see the tests Travis ran on that commit, at least one of which failed, and go to one of the failing tests to see the error.) Since Coveralls’s data on changes isn’t always accurate, one should look at the test coverage situation directly. • Technical design. There are a lot of considerations here: security, migration paths/backwards compatibility, cost of new dependencies, interactions with features, speed of performance, API changes. Security is especially important and worth thinking about carefully with any changes to security-sensitive code like views. • User interface and visual design. If frontend changes are involved, the reviewer will check out the code, play with the new UI, and verify it for both quality and consistency with the rest of the Zulip UI. We highly encourage posting screenshots to save reviewers time in getting a feel for what the feature looks like – you’ll get a quicker response that way. • Error handling. The code should always check for invalid user input. User-facing error messages should be clear and when possible be actionable (it should be obvious to the user what they need to do in order to correct the problem). • Testing. The tests should validate that the feature works correctly, and specifically test for common error conditions, bad user input, and potential bugs that are likely for the type of change being made. Tests that exclude whole classes of potential bugs are preferred when possible (e.g., the common test suite test_bugdown.py between the frontend and backend Markdown processors or the GetEventsTest test for buggy race condition handling).

181 Zulip Documentation, Release 1.4.0

Backend: we are trying to maintain ~100% test coverage on the backend, so backend changes should have negative tests for the various error conditions. Frontend: If the feature involves frontend changes, there should be frontend tests. See the test writing documentation for more details. • mypy annotations. New functions should be annotated using mypy and existing annotations should be updated. Use of Any, ignore, and unparameterized containser should be limited to cases where a more precise type cannot be specified. • Translation. Make sure that the strings are marked for translation. • Clear function, argument, variable, and test names. Every new piece of Zulip code will be read many times by other developers, and future developers will grep for relevant terms when researching a problem, so it’s important that variable names communicate clearly the purpose of each piece of the codebase. • Duplicated code. Code duplication is a huge source of bugs in large projects and makes the codebase difficult to understand, so we avoid significant code duplication wherever possible. Sometimes avoiding code duplication involves some refactoring of existing code; if so, that should usually be done as its own series of commits (not squashed into other changes or left as a thing to do later). That series of commits can be in the same pull request as the feature that they support, and we recommend ordering the history of commits so that the refactoring comes before the feature. That way, it’s easy to merge the refactoring (and minimize risk of merge conflicts) if there are still user experience issues under discussion for the feature itself. • Completeness. For refactorings, verify that the changes are complete. Usually one can check that efficiently using git grep, and it’s worth it, as we very frequently find issues by doing so. • Documentation updates. If this changes how something works, does it update the documentation in a corresponding way? If it’s a new feature, is it documented, and documented in the right place? • Good comments. It’s often worth thinking about whether explanation in a commit message or pull request discussion should be included in a comment, /docs, or other documentation. But it’s better yet if verbose explanation isn’t needed. We prefer writing code that is readable without explanation over a heavily commented codebase using lots of clever tricks. • Coding style. See the Zulip code-style documentation for details. Our goal is to have as much of this as possible verified via the linters and tests, but there’s always going to be unusual forms of Python/JavaScript style that our tools don’t check for. • Clear commit messages. See the Zulip version control documentation for details on what we look for.

Tooling

To make it easier to review pull requests, use our git tool tools/fetch-rebase-pull-request to check out a pull request locally and rebase it against master. If a pull request just needs a little ﬁxing to make it mergeable, feel free to do that in a new commit, then push your branch to GitHub and mention the branch in a comment on the pull request. That’ll save the maintainer time and get the PR merged quicker.

Additional Resources

We also strongly recommend reviewers to go through the following resources. • The Gentle Art of Patch Review article by Sarah Sharp • Zulip & Good Code Review article by Sumana Harihareswara • Code Review - A consolidation of advice and stuff from the sinternet article by James J. Porter • Zulip Code of Conduct

182 Chapter 30. Reviewing Zulip server code CHAPTER 31

The chat.zulip.org community

chat.zulip.org is the primary communication forum for the Zulip community. You can go through the simple signup process at that link, and then you will soon be talking to core Zulip developers and other users. To get help in real time, you will have the best luck ﬁnding core developers roughly between 17:00 UTC and 2:00 UTC or during ofﬁce hours and sprints, but the sun never sleeps on the Zulip community. Most questions get a reply within minutes to a few hours, depending on the time of day.

This is a bleeding edge development server

The chat.zulip.org server is frequently deployed off of master from the Zulip Git repository, so please point out anything you notice that seems wrong! We catch many bugs that escape code review this way. The chat.zulip.org server is a development and testing server, not a production service, so don’t use it for anything mission-critical, secret/embarrassing, etc.

Community conventions

• Send any test messages to #test here or as a PM to yourself to avoid disrupting others. • When asking for help, provide the details needed for others to help you. E.g. include the full traceback in a code block (not a screenshot), a link to the code or a WIP PR you’re having trouble debugging, etc. • Ask questions on streams rather than PMing core contributors. You’ll get answers faster since other people can help, and it makes it possible for other developers to learn from reading the discussion. • Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it’s usually easy to see which message you’re replying to, so you don’t need to mention your conversation partner in every reply. Mentioning other users is great for timely questions or making sure someone who is not online sees your message. • Converse informally; there’s no need to use titles like “Sir” or “Madam”. • Use gender-neutral language.

183 Zulip Documentation, Release 1.4.0

• Follow the community code of conduct. • Participate! Zulip is a friendly and welcoming community, and we love meeting new people, hearing about what brought them to Zulip, and getting their feedback. If you’re not sure where to start, introduce yourself and your interests in #new members, using your name as the topic.

Streams

There are a few streams worth highlighting that are relevant for everyone, even non-developers: • #announce is for announcements and discussions thereof; we try to keep trafﬁc there to a minimum. • #design is where we discuss the UI design and collect feedback on potential design changes. We love feedback, so don’t hesitate to speak up! • #documentation is where we discuss improving Zulip’s user, sysadmin, and developer documentation. • #production help is for production environment related discussions. • #test here is for sending test messages without inconveniencing other users :). We recommend muting this stream when not using it. • #translation is for discussing Zulip’s translations. There are dozens of streams for development discussions in the Zulip community (e.g. one for each app, etc.); check out the Streams page to see the descriptions for all of them. Relevant to almost everyone are these: • #checkins is for progress updates on what you’re working on and its status; usually folks post with their name as the topic. Everyone is welcome to participate! • #development help is for asking for help with any Zulip server/webapp development work (use the app streams for help working on one of the apps). • #code review is for getting feedback on your work. We encourage all developers to comment on work posted here, even if you’re new to the Zulip project; reviewing other PRs is a great way to develop experience, and even just manually testing a proposed new feature and posting feedback is super helpful. • #learning is for posting great learning resources one comes across.

Chat meetings

We have regular chat meetings on Zulip to coordinate work on various parts of the Zulip project. While most developer discussions happen asynchonrously, these meetings are used mainly to coordinate work within a major area of Zulip. These meetings are usually scheduled in Pacific time mornings, since that seems to be the best time for our global contributor base (the part of the world where it’s the deep middle of the night is the Pacific Ocean). Anyone is welcome to attend and contribute to the discussions in these meetings, and they’re a great time to stop by and introduce yourself if you’d like to get involved (though really, any time is, so). Here are the regular meetings that exist today along with their usual times (actual times are listed in the linked agenda documents): • Mobile team on #mobile, generally Wednesdays at 10AM Pacific time. Agendas. • Backend/infrastructure team on #backend, generally Fridays at 10AM Pacific time. Agendas. • Bots and integrations team on #integrations, generally Thursdays at 9AM Pacific time. Agendas.

184 Chapter 31. The chat.zulip.org community Zulip Documentation, Release 1.4.0

Ofﬁce hours and sprints

We also do project-wide ad-hoc “office hours” and remote sprints irregularly, about once a month. Anyone can schedule one: announce it in #announce and on the zulip-devel mailing list a few days ahead of time, and ideally, tell Sumana so she can put it on the public Zulip meetings calendar. Office hours are simply times for us to informally discuss current global project priorities, find out what questions people have, and so on. We set them up so people know there’ll be more people around at a particular time to chat. You don’t need to RSVP and you don’t need to show up on time or stop conversations when the “hour” stops. They start in #general and conversations move into other streams and topics as they come up. Sprints are times when Zulip developers get together in chat, and sometimes in person, to work on related issues at the same time.

31.4. Chat meetings 185 Zulip Documentation, Release 1.4.0

186 Chapter 31. The chat.zulip.org community CHAPTER 32

Using zulipbot

Zulip uses @zulipbot, a GitHub workflow bot deployed on all Zulip repositories, to handle issues and pull requests in our repositories in order to create a better workflow for Zulip contributors. Its purpose is to work around various limitations in GitHub’s permissions and notifications systems to make it possible to have a much more democractic workflow for our contributors. It allows anyone to self-assign or label an issue, not just the core contributors trusted with full write access to the repository (which is the only model GitHub supports).

Usage

• Claim an issue — Comment @zulipbot claim on the issue you want to claim; @zulipbot will assign you to the issue and label the issue as in progress. – If you’re a new contributor, @zulipbot will give you read-only collaborator access to the repository and leave a welcome message on the issue you claimed. – You can also claim an issue that you’ve opened by including @zulipbot claim in the body of your issue. – If you accidentally claim an issue you didn’t want to claim, comment @zulipbot abandon to abandon an issue. • Label your issues — Add appropriate labels to issues that you opened by including @zulipbot label in an issue comment or the body of your issue followed by the desired labels enclosed within double quotes (""). – For example, to add the bug and help wanted labels to your issue, comment or include @zulipbot label "bug" "help wanted" in the issue body. – You’ll receive an error message if you try to add any labels to your issue that don’t exist in your repository. – If you accidentally added the wrong labels, you can remove them by commenting @zulipbot remove followed by the desired labels enclosed with double quotes (""). • Find unclaimed issues - Use the GitHub search feature to ﬁnd unclaimed issues by adding one of the following ﬁlters to your search:

187 Zulip Documentation, Release 1.4.0

– -label: "in progress" (excludes issues labeled with the in progress label) – no:assignee (shows issues without assignees) Issues labeled with the in progress label and/or assigned to other users have already been claimed. • Collaborate in area label teams - Receive notifications on issues and pull requests within your fields of ex- pertise on the Zulip server repository by joining the Zulip server area label teams. These teams correspond to the repository’s area labels, although some teams are associated with multiple labels; for example, the area: message-editing and area: message view labels are both related to the Server message view team. Feel free to join as many area label teams as as you’d like! After your request to join an area label team is approved, you’ll receive notifications for any issues labeled with the team’s corresponding area label as well as any pull requests that reference issues labeled with your team’s area label. • Track inactive claimed issues — If a claimed issue has not been updated for a week, @zulipbot will post a comment on the inactive issue to ask the assignee(s) if they are still working on the issue. If you see this comment on an issue you claimed, you should post a comment on the issue to notify @zulipbot that you’re still working on it. If @zulipbot does not receive a response from the assignee within 3 days of an inactive issue prompt, @zulipbot will automatically remove the issue’s current assignee(s) and the “in progress” label to allow others to work on an inactive issue. • Receive Travis build status notifications — If you would like to receive a notification whenever the build status of your pull request is updated, label your pull request with the “travis updates” label using the command @zulipbot label "travis updates", and @zulipbot will let you know the build status (e.g. passed, failed, errored) of your pull request once all tests finish.

Contributing

If you wish to help develop and contribute to @zulipbot, check out the zulip/zulipbot repository on GitHub and read the project’s contributing guidelines for more information.

188 Chapter 32. Using zulipbot CHAPTER 33

Accessibility

Guidelines

In order to accommodate all users, Zulip strives to implement accessibility best practices in its user interface. There are many aspects to accessibility; here are some of the more important ones to keep in mind. • All images should have alternative text attributes for the beneﬁt of users who cannot see them (this includes users who are utilizing a voice interface to free up their eyes to look at something else instead). • The entire application should be usable via a keyboard (many users are unable to use a mouse, and many accessibility aids emulate a keyboard). • Text should have good enough contrast against the background to enable even users with moderate visual im- pairment to be able to read it. • ARIA (Accessible Rich Internet Application) attributes should be used appropriately to enable screen readers and other alternative interfaces to navigate the application effectively. There are many different standards for accessibility, but the most relevant one for Zulip is the W3C’s WCAG (Web Content Accessibility Guidelines), currently at version 2.0. Whenever practical, we should strive for compliance with the AA level of this speciﬁcation. (The W3C itself recommends not trying to comply with the highest AAA level for an entire web site or application, as it is not possible for some content.)

Tools

There are tools available to automatically audit a web page for compliance with many of the WCAG guidelines. Here are some of the more useful ones: • Accessibility Developer Tools This open source Chrome extension from Google adds an accessibility audit to the “Audits” tab of the Chrome Developer Tools. The audit is performed against the page’s DOM via JavaScript, allowing it to identify some issues that a static HTML inspector would miss. • aXe An open source Chrome and Firefox extension which runs a somewhat different set of checks than Google’s Chrome extension.

189 Zulip Documentation, Release 1.4.0

• Wave This web application takes a URL and loads it in a frame, reporting on all the issues it ﬁnds with links to more information. Has the advantage of not requiring installation, but requires a URL which can be directly accessed by an external site. • Web Developer This browser extension has many useful features, including a convenient link for opening the current URL in Wave to get an accessibility report. Note that these tools cannot catch all possible accessibility problems, and sometimes report false positives as well. They are a useful aid in quickly identifying potential problems and checking for regressions, but their recommendations should not be blindly obeyed.

GitHub Issues

Problems with Zulip’s accessibility should be reported as GitHub issues with the “accessibility” label. This label can be added by entering the following text in a separate comment on the issue:

@zulipbot label"accessibility"

If you want to help make Zulip more accessible, here is a list of the currently open accessibility issues.

Additional Resources

For more information about making Zulip accessible to as many users as possible, the following resources may be useful. • Web Content Accessibility Guidelines (WCAG) 2.0 • WAI-ARIA - Web Accessibility Initiative Accessible Rich Internet Application Suite • WebAIM - Web Accessibility in Mind • The MDN page on accessibility • The Open edX Accessibility Guidelines for developers

190 Chapter 33. Accessibility CHAPTER 34

Testing and writing tests

Overview

Zulip has a full test suite that includes many components. The most important components are documented in depth in their own sections: • Django: backend Python tests • Casper: end-to-end UI tests • Node: unit tests for JS front end code • Linters: Our parallel linter suite • Travis CI details: How all of these run in Travis CI This document covers more general testing issues, such as how to run the entire test suite, how to troubleshoot database issues, how to manually test the front end, and how to plan for the future upgrade to Python3. We also document how to manually test the app.

Running tests

Zulip tests must be run inside a Zulip development environment; if you’re using Vagrant, you will need to enter the Vagrant environment before running the tests: vagrant ssh cd/srv/zulip

Then, to run the full Zulip test suite, do this:

./tools/test-all

191 Zulip Documentation, Release 1.4.0

This runs the linter (tools/lint) plus all of our test suites; they can all be run separately (just read tools/ test-all to see them). You can also run individual tests which can save you a lot of time debugging a test failure, e.g.:

./tools/lint # Runs all the linters in parallel ./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube ./tools/test-backend BugdownTest # Run `test-backend --help` for more options ./tools/test-js-with-casper 09-navigation.js ./tools/test-js-with-node utils.js

The above setup instructions include the ﬁrst-time setup of test databases, but you may need to rebuild the test database occasionally if you’re working on new database migrations. To do this, run:

./tools/do-destroy-rebuild-test-database

Possible testing issues

• When running the test suite, if you get an error like this:

sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_

˓→array(unknown, text, tsquery) does not exist LINE2:...ECT message_id, flags, subject, rendered_content, ts_match_l... ^

. . . then you need to install tsearch-extras, described above. Afterwards, re-run the init*-db and the do-destroy-rebuild*-database scripts. • When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.

Schema and initial data changes

If you change the database schema or change the initial test data, you have to regenerate the pristine test database by running tools/do-destroy-rebuild-test-database.

Wiping the test databases

You should ﬁrst try running: tools/do-destroy-rebuild-test-database If that fails you should try to do:

sudo-u postgres psql > DROP DATABASE zulip_test; > DROP DATABASE zulip_test_template;

and then run tools/do-destroy-rebuild-test-database

192 Chapter 34. Testing and writing tests Zulip Documentation, Release 1.4.0

Recreating the postgres cluster

warning This is irreversible, so do it with care, and never do this anywhere in production. If your postgres cluster (collection of databases) gets totally trashed permissions-wise, and you can’t otherwise repair it, you can recreate it. On Ubuntu: sudo pg_dropcluster--stop 9.1 main sudo pg_createcluster--locale=en_US.utf8--start 9.1 main

Local browser testing (local app + web browser)

This section is about troubleshooting your local development environment. There is a separate manual testing doc that enumerates things you can test as part of manual QA.

Clearing the development database

You can use:

./tools/do-destroy-rebuild-database to drop the database on your development environment and repopulate your it with the Shakespeare characters and some test messages between them. This is run automatically as part of the development environment setup process, but is occasionally useful when you want to return to a clean state for testing.

JavaScript manual testing debug.js has some tools for profiling JavaScript code, including: • ‘print_elapsed_time‘: Wrap a function with it to print the time that function takes to the JavaScript console. • ‘IterationProfiler‘: Profile part of looping constructs (like a for loop or $.each). You mark sections of the iteration body and the IterationProfiler will sum the costs of those sections over all iterations. Chrome has a very good debugger and inspector in its developer tools. Firebug for Firefox is also pretty good. They both have profilers, but Chrome’s is a sampling profiler while Firebug’s is an instrumenting profiler. Using them both can be helpful because they provide different information.

Python 3 Compatibility

Zulip is working on supporting Python 3, and all new code in Zulip should be Python 2+3 compatible. We have converted most of the codebase to be compatible with Python 3 using a suite of 2to3 conversion tools and some manual work. In order to avoid regressions in that compatibility as we continue to develop new features in Zulip, we have a special tool, tools/check-py3, which checks all code for Python 3 syntactic compatibility by running a subset of the automated migration tools and checking if they trigger any changes. tools/check-py3 is run automatically in Zulip’s Travis CI tests (in the ‘static-analysis’ build) to avoid any regressions, but is not included in test-all since it is quite slow.

34.5. Local browser testing (local app + web browser) 193 Zulip Documentation, Release 1.4.0

To run tools/check-py3, you need to install the modernize and future Python packages (which are included in requirements/py3k.txt, which itself is included in requirements/dev.txt, so you probably already have these packages installed). To run check-py3 on just the Python ﬁles in a particular directory, you can change the current working directory (e.g. cd zerver/) and run check-py3 from there. Also, if you’re using Vagrant or if you set up virtualenvs in the non-Vagrant setup, you should have two different virtualenvs: one for Python 2 (which is the one used by default), and another one for Python 3. This is useful if you want to do some manual testing using either version. To switch between both virtualenvs, run: • Use Python 2: source /srv/zulip-venv/bin/activate • Use Python 3: source /srv/zulip-py3-venv/bin/activate

194 Chapter 34. Testing and writing tests CHAPTER 35

Linters

Overview

Zulip does extensive linting of much of its source code, including Python/JavaScript files, HTML templates (Django/handlebars), CSS files, JSON fixtures, Markdown documents, puppet manifests, and shell scripts. For some files we simply check for small things like trailing whitespace, but for other files, we are quite thorough about checking semantic correctness. Obviously, a large reason for linting code is to enforce the Zulip coding standards. But we also use the linters to prevent common coding errors. We borrow some open source tools for much of our linting, and the links below will direct you to the official documentation for these projects. • eslint • mypy • puppet (puppet provides its own mechanism for validating manifests) • pyflakes Zulip also uses some home-grown code to perform tasks like validating indentation in template files, enforcing coding standards that are unique to Zulip, allowing certain errors from third party linters to pass through, and exempting legacy files from lint checks.

Running the linters

If you run ./tools/test-all, it will automatically run the linters (with one small exception: it does not run mypy against scripts). You can also run them individually:

195 Zulip Documentation, Release 1.4.0

./tools/lint ./tools/run-mypy ./tools/run-mypy--scripts-only

Finally, you can rely on our Travis CI setup to run linters for you, but it is good practice to run lint checks locally. Note: The linters only check files that git tracks. Remember to git add new files before running lint checks. Our linting tools generally support the ability to lint files individually–with some caveats–and those options will be described later in this document. We may eventually bundle run-mypy into lint, but mypy is pretty resource intensive compared to the rest of the linters, because it does static code analysis. So we keep mypy separate to allow folks to quickly run the other lint checks.

General considerations

Once you have read the Zulip coding guidelines, you can be pretty confident that 99% of the code that you write will pass through the linters fine, as long as you are thorough about keeping your code clean. And, of course, for minor oversights, lint is your friend, not your foe. Occasionally, our linters will complain about things that are more of an artifact of the linter limitations than any actual problem with your code. There is usually a mechanism where you can bypass the linter in extreme cases, but often it can be a simple matter of writing your code in a slightly different style to appease the linter. If you have problems getting something to lint, you can submit an unfinished PR and ask the reviewer to help you work through the lint problem, or you can find other people in the Zulip Community to help you. Also, bear in mind that 100% of the lint code is open source, so if you find limitations in either the Zulip home-grown stuff or our third party tools, feedback will be highly appreciated. Finally, one way to clean up your code is to thoroughly exercise it with tests. The Zulip test documentation describes our test system in detail.

Lint checks

Most of our lint checks get performed by ./tools/lint. These include the following checks: • Check Python code with pyﬂakes. • Check JavaScript code with eslint. • Check Python code for custom Zulip rules. • Check non-Python code for custom Zulip rules. • Check puppet manifests with the puppet validator. • Check HTML templates for matching tags and indentations. • Check CSS for parsability. • Check JavaScript code for addClass calls. The remaining lint checks occur in ./tools/run-mypy. It is probably somewhat of an understatement to call “mypy” a “linter,” as it performs static code analysis of Python type annotations throughout our Python codebase. Our documentation on using mypy covers mypy in more detail. The rest of this document pertains to the checks that occur in ./tools/lint.

196 Chapter 35. Linters Zulip Documentation, Release 1.4.0

lint

Zulip has a script called lint that lives in our “tools” directory. It is the workhorse of our linting system, although in some cases it dispatches the heavy lifting to other components such as pyflakes, eslint, and other home grown tools. You can find the source code here. In order for our entire lint suite to run in a timely fashion, the lint script performs several lint checks in parallel by forking out subprocesses. This mechanism is still evolving, but you can look at the method run_parallel to get the gist of how it works. Note that our project does custom regex-based checks on the code, and we also customize how we call pyflakes and pycodestyle (pep8). The code for these types of checks mostly lives here.

Special options

You can use the -h option for lint to see its usage. One particular flag to take note of is the --modified flag, which enables you to only run lint checks against files that are modified in your git repo. Most of the “sub-linters” respect this flag, but some will continue to process all the files. Generally, a good workflow is to run with --modified when you are iterating on the code, and then run without that option right before commiting new code. If you need to troubleshoot the linters, there is a --verbose option that can give you clues about which linters may be running slow, for example.

Lint checks

The next part of this document describes the lint checks that we apply to various ﬁle types.

Generic source code checks

We check almost our entire codebase for trailing whitespace. Also, we disallow tab (\t) characters in all but two files. We also have custom regex-based checks that apply to specific file types. For relatively minor files like Markdown files and JSON fixtures, this is the extent of our checking. Finally, we’re checking line length in Python code (and hope to extend this to other parts of the codebase soon). You can use #ignorelinelength for special cases where a very long line makes sense (e.g. a link in a comment to an extremely long URL).

Python code

The bulk of our Python linting gets outsourced to the “pyflakes” tool. We call “pyflakes” in a fairly vanilla fashion, and then we post-process its output to exclude certain types of errors that Zulip is comfortable ignoring. (One notable class of error that Zulip currently tolerates is unused imports–because of the way mypy type annotations work in Python 2, it would be inconvenient to enforce this too strictly.) Zulip also has custom regex-based rules that it applies to Python code. Look for python_rules in the source code for lint. Note that we provide a mechanism to exclude certain lines of codes from these checks. Often, it is simply the case that our regex approach is too crude to correctly exonerate certain valid constructs. In other cases, the code that we exempt may be deemed not worthwhile to fix.

35.5. lint 197 Zulip Documentation, Release 1.4.0

JavaScript code

We check our JavaScript code in a few different ways: • We run eslint. • We perform custom Zulip regex checks on the code. • We verify that all addClass calls, with a few exceptions, explicitly contain a CSS class. The last check happens via a call to ./tools/find-add-class. This particular check is a work in progress, as we are trying to evolve a more rigorous system for weeding out legacy CSS styles, and the ability to quickly introspect our JS code for addClass calls is part of our vision.

Puppet manifests

We use Puppet as our tool to manage conﬁguration ﬁles, using puppet “manifests.” To lint puppet manifests, we use the “parser validate” option of puppet.

HTML Templates

Zulip uses two HTML templating systems: • Django templates • handlebars Zulip has an internal tool that validates both types of templates for correct indentation and matching tags. You can find the code here: • driver: check-templates • engine: lib/template_parser.py We exempt some legacy files from indentation checks, but we are hoping to clean those files up eventually.

CSS

Zulip does not currently lint its CSS for any kind of semantic correctness, but that is definitely a goal moving forward. We do ensure that our home-grown CSS parser can at least parse the CSS code. This is a slightly more strict check than checking that the CSS is compliant to the official spec, as our parser will choke on unusual constructs that we probably want to avoid in our code, anyway. (When the parser chokes, the lint check will fail.) You can find the code here: • driver: check-css • engine: lib/css_parser.py

Markdown, shell scripts, JSON ﬁxtures

We mostly validate miscellaneous source ﬁles like .sh, .json, and .md ﬁles for whitespace issues.

Philosophy

If you want to help improve Zulip’s system for linting, here are some considerations.

198 Chapter 35. Linters Zulip Documentation, Release 1.4.0

Speed

We want our linters to be fast enough that most developers will feel comfortable running them in a pre-commit hook, so we run our linters in parallel and support incremental checks.

Accuracy

We try to catch as many common mistakes as possible, either via a linter or an automated test.

Completeness

Our goal is to have most common style issues by caught by the linters, so new contributors to the codebase can efﬁciently ﬁx produce code with correct style without needing to go back-and-forth with a reviewer.

35.6. Philosophy 199 Zulip Documentation, Release 1.4.0

200 Chapter 35. Linters CHAPTER 36

Backend Django tests

Overview

Zulip uses the Django framework for its Python back end. We use the testing framework from django.test to test our code. We have over a thousand automated tests that verify that our backend works as expected. All changes to the Zulip backend code should be supported by tests. We enforce our testing culture during code review, and we also use coverage tools to measure how well we test our code. We mostly use tests to prevent regressions in our code, but the tests can have ancillary benefits such as documenting interfaces and influencing the design of our software. If you have worked on other Django projects that use unit testing, you will probably find familiar patterns in Zulip’s code. This document describes how to write tests for the Zulip back end, with a particular emphasis on areas where we have either wrapped Django’s test framework or just done things that are kind of unique in Zulip.

Running tests

Our tests live in zerver/tests/. You can run them with ./tools/test-backend. The tests run in parallel using multiple threads in your development environment, and can ﬁnish in under 30s on a fast machine. When you are in iterative mode, you can run individual tests or individual modules, following the dotted.test.name convention below:

cd/srv/zulip ./tools/test-backend zerver.tests.test_queue_worker.WorkerTest

There are many command line options for running Zulip tests, such as a --verbose option. The best way to learn the options is to use the online help:

./tools/test-backend-h

We also have ways to instrument our tests for ﬁnding code coverage, URL coverage, and slow tests. Use the -h option to discover these features. We also have a --profile option to facilitate proﬁling tests.

201 Zulip Documentation, Release 1.4.0

Another thing to note is that our tests generally “fail fast,” i.e. they stop at the ﬁrst sign of trouble. This is generally a good thing for iterative development, but you can override this behavior with the --nonfatal-errors option. A useful option to combine with that is the --rerun option, which will rerun just the tests that failed in the last test run.

How to write tests.

Before you write your ﬁrst tests of Zulip, it is worthwhile to read the rest of this document, and you can also read some of the existing tests in zerver/tests to get a feel for the patterns we use. A good practice is to get a “failing test” before you start to implement your feature. First, it is a useful exercise to understand what needs to happen in your tests before you write the code, as it can help drive out simple design or help you make incremental progress on a large feature. Second, you want to avoid introducing tests that give false positives. Ensuring that a test fails before you implement the feature ensures that if somebody accidentally regresses the feature in the future, the test will catch the regression. Another important ﬁles to skim are zerver/lib/test_helpers.py, which contains test helpers. zerver/lib/test_classes.py, which contains our ZulipTestCase and WebhookTestCase classes.

Setting up data for tests

All tests start with the same fixture data. (The tests themselves update the database, but they do so inside a transaction that gets rolled back after each of the tests complete. For more details on how the fixture data gets set up, refer to tools/setup/generate-fixtures.) The fixture data includes a few users that are named after Shakesepeare characters, and they are part of the “zulip.com” realm. Generally, you will also do some explicit data setup of your own. Here are a couple useful methods in ZulipTestCase: • common_subscribe_to_streams • send_message • make_stream • subscribe_to_stream More typically, you will use methods directly from the backend code. (This ensures more end-to-end testing, and avoids false positives from tests that might not consider ancillary parts of data setup that could influence tests results.) Here are some example action methods that tests may use for data setup: • check_send_message • do_change_is_admin • do_create_user • do_make_stream_private

Zulip Testing Philosophy

If there is one word to describe Zulip’s philosophy for writing tests, it is probably “ﬂexible.” (Hopefully “thorough” goes without saying.)

202 Chapter 36. Backend Django tests Zulip Documentation, Release 1.4.0

When in doubt, unless speed concerns are prohibitive, you usually want your tests to be somewhat end-to-end, particularly for testing endpoints. These are some of the testing strategies that you will see in the Zulip test suite...

Endpoint tests

We strive to test all of our URL endpoints. The vast majority of Zulip endpoints support a JSON interface. Regardless of the interface, an endpoint test generally follows this pattern: • Set up the data. • Login with self.login() or set up an API key. • Use a Zulip test helper to hit the endpoint. • Assert that the result was either a success or failure. • Check the data that comes back from the endpoint. Generally, if you are doing endpoint tests, you will want to create a test class that is a subclass of ZulipTestCase, which will provide you helper methods like the following: • api_auth • assert_json_error • assert_json_success • client_get • client_post • get_api_key • get_streams • login • send_message

Library tests

For certain Zulip library functions, especially the ones that are not intrinsically tied to Django, we use a classic unit testing approach of calling the function and inspecting the results. For these types of tests, you will often use methods like self.assertEqual(), self.assertTrue(), etc., which come with unittest via Django.

Fixture-driven tests

Particularly for testing Zulip’s integrations with third party systems, we strive to have a highly data-driven approach to testing. To give a specific example, when we test our GitHub integration, the test code reads a bunch of sample inputs from a JSON fixture file, feeds them to our GitHub integration code, and then verifies the output against expected values from the same JSON fixture file. Our fixtures live in zerver/fixtures.

36.4. Zulip Testing Philosophy 203 Zulip Documentation, Release 1.4.0

Mocks and stubs

We use mocks and stubs for all the typical reasons: • to more precisely test the target code • to stub out calls to third-party services • to make it so that you can run your tests on the airplane without wiﬁ For mocking we generally use the “mock” library and use mock.patch as a context manager or decorator. We also take advantage of some context managers from Django as well as our own custom helpers. Here is an example:

with self.settings(RATE_LIMITING=True): with mock.patch('zerver.decorator.rate_limit_user') as rate_limit_mock: api_result= my_webhook(request)

self.assertTrue(rate_limit_mock.called)

Follow this link for more information on the “settings” context manager.

Template tests

In zerver/tests/test_templates.py we have a test that renders all of our back end templates with a “dummy” context, to make sure the templates don’t have obvious errors. (These tests won’t catch all types of errors; they are just a ﬁrst line of defense.)

SQL performance tests

A common class of bug with Django systems is to handle bulk data in an inefﬁcient way, where the back end populates objects for join tables with a series of individual queries that give O(N) latency. (The remedy is often just to call select_related(), but sometimes it requires a more subtle restructuring of the code.) We try to prevent these bugs in our tests by using a context manager called queries_captured() that captures the SQL queries used by the back end during a particular operation. We make assertions about those queries, often simply asserting that the number of queries is below some threshold.

Event-based tests

The Zulip back end has a mechanism where it will fetch initial data for a client from the database, and then it will subsequently apply some queued up events to that data to the data structure before notifying the client. The EventsRegisterTest.do_test() helper helps tests verify that the application of those events via ap- ply_events() produces the same data structure as performing an action that generates said event. This is a bit esoteric, but if you read the tests, you will see some of the patterns. You can also learn more about our event system in the new feature tutorial.

Negative tests

It is important to verify error handling paths for endpoints, particularly situations where we need to ensure that we don’t return results to clients with improper authentication or with limited authorization. A typical test will call the endpoint with either a non-logged in client, an invalid API key, or missing input ﬁelds. Then the test will call assert_json_error() to verify that the endpoint is properly failing.

204 Chapter 36. Backend Django tests Zulip Documentation, Release 1.4.0

Testing considerations

Here are some things to consider when writing new tests: • Duplication We try to avoid excessive duplication in tests. If you have several tests repeating the same type of test setup, consider making a setUp() method or a test helper. • Network independence Our tests should still work if you don’t have an internet connection. For third party clients, you can simulate their behavior using fixture data. For third party servers, you can typically simulate their behavior using mocks. • Coverage We have 100% line coverage on several of our backend modules. You can use the --coverage option to generate coverage reports, and new code should have 100% coverage, which generally requires testing not only the “happy path” but also error handling code and edge cases. It will generate a nice HTML report that you can view right from your browser (the tool prints the URL where the report is exposed in your development environment). Note that test-backend --coverage will assert that various specific files in the project have 100% test coverage and throw an error if their coverage has fallen. One of our project goals is to expand that checking to ever-larger parts of the codebase.

36.5. Testing considerations 205 Zulip Documentation, Release 1.4.0

206 Chapter 36. Backend Django tests CHAPTER 37

JavaScript unit tests

As an alternative to the black-box whole-app testing, you can unit test individual JavaScript ﬁles. If you are writing JavaScript code that manipulates data (as opposed to coordinating UI changes), then you probably modify existing unit test modules to ensure the quality of your code and prevent regressions. The JS unit tests are written to work with node. You can ﬁnd them in frontend_tests/node_tests. Here is an example test from frontend_tests/node_tests/stream_data.js:

(function test_get_by_id() { stream_data.clear_subscriptions(); var id= 42; var sub={ name:'Denmark', subscribed: true, color:'red', stream_id: id }; stream_data.add_sub('Denmark', sub); sub= stream_data.get_sub('Denmark'); assert.equal(sub.color,'red'); sub= stream_data.get_sub_by_id(id); assert.equal(sub.color,'red'); }());

The names of the node tests generally align with the names of the modules they test. If you modify a JS module in static/js you should see if there are corresponding test in frontend_tests/node_tests. If there are, you should strive to follow the patterns of the existing tests and add your own tests.

HTML output

The JavaScript unit tests can generate output to be viewed in the browser. The best examples of this are in frontend_tests/node_tests/templates.js.

207 Zulip Documentation, Release 1.4.0

The main use case for this mechanism is to be able to unit test templates and see how they are rendered without the complications of the surrounding app. (Obviously, you still need to test the app itself!) The HTML output can also help to debug the unit tests. Each test calls a method named write_handlebars_output after it renders a template with similar data. This API is still evolving, but you should be able to look at existing code for patterns. When you run tools/test-js-with-node, it will present you with a message like “To see more output, open var/test-js-with-node/index.html.” Basically, you just need to open the ﬁle in the browser. (If you are running a VM, this might require switching to another terminal window to launch the open command.)

Coverage reports

You can automatically generate coverage reports for the JavaScript unit tests like this:

tools/test-js-with-node--coverage

Then open coverage/lcov-report/js/index.html in your browser. Modules we don’t test at all aren’t listed in the report, so this tends to overstate how good our overall coverage is, but it’s accurate for individual files. You can also click a filename to see the specific statements and branches not tested. 100% branch coverage isn’t necessarily possible, but getting to at least 80% branch coverage is a good goal.

Handling dependencies in unit tests

The following scheme helps avoid tests leaking globals between each other. First, if you can avoid globals, do it, and the code that is directly under test can simply be handled like this:

var search= require('js/search_suggestion.js');

For deeper dependencies, you want to categorize each module as follows: • Exercise the module’s real code for deeper, more realistic testing? • Stub out the module’s interface for more control, speed, and isolation? • Do some combination of the above? For all the modules where you want to run actual code, add a statement like the following to the top of your test ﬁle:

add_dependencies({ _:'node_modules/underscore/underscore.js', util:'js/util.js', Dict:'js/dict.js', Handlebars:'handlebars', Filter:'js/filter.js', typeahead_helper:'js/typeahead_helper.js', stream_data:'js/stream_data.js', narrow:'js/narrow.js' });

For modules that you want to completely stub out, please use a pattern like this:

set_global('page_params',{ email:'[email protected]' });

208 Chapter 37. JavaScript unit tests Zulip Documentation, Release 1.4.0

// then maybe further down global.page_params.email='[email protected]';

Finally, there’s the hybrid situation, where you want to borrow some of a module’s real functionality but stub out other pieces. Obviously, this is a pretty strong smell that the other module might be lacking in cohesion, but that code might be outside your jurisdiction. The pattern here is this:

// Use real versions of parse/unparse var narrow= require('js/narrow.js'); set_global('narrow',{ parse: narrow.parse, unparse: narrow.unparse });

// But later, I want to stub the stream without having to call super-

˓→expensive // real code like narrow.activate(). global.narrow.stream= function () { return 'office'; };

Creating new test modules

The nodes tests rely on JS files that use the module pattern. For example, to test the foobar.js file, you would first add the following to the bottom of foobar.js:

if (typeof module !=='undefined'){ module.exports= foobar; }

This makes foobar.js follow the CommonJS module pattern, so it can be required in Node.js, which runs our tests. Now create frontend_tests/node_tests/foobar.js. At the top, require the Node.js assert module, and the module you’re testing, like so:

var assert = require('assert'); var foobar= require('js/foobar.js');

And of course, if the module you’re testing depends on other modules, or modifies global state, you may need to review the section on handling dependencies above. Define and call some tests using the assert module. Note that for “equal” asserts, the actual value comes first, the expected value second.

(function test_somefeature() { assert.strictEqual(foobar.somefeature('baz'),'quux'); assert.throws(foobar.somefeature('Invalid Input')); }());

The test runner (index.js) automatically runs all .js ﬁles in the frontend_tests/node directory.

37.4. Creating new test modules 209 Zulip Documentation, Release 1.4.0

210 Chapter 37. JavaScript unit tests CHAPTER 38

Web frontend black-box casperjs tests

These live in frontend_tests/casper_tests/. This is a “black box” integration test; we load the frontend in a real (headless) browser, from a real (development) server, and simulate UI interactions like sending messages, narrowing, etc., by actually clicking around the UI and waiting for things to change before doing the next step. These tasks are fantastic for ensuring the overall health of the project, but are also costly to maintain and keep free of nondeterministic failures, so we usually prefer to write a Node test instead when possible. Since the Casper tests interact with a real dev server, they can often catch backend bugs as well. You can run the casper tests with ./tools/test-js-with-casper or as ./tools/ test-js-with-casper 06-settings.js to run a single test ﬁle from frontend_tests/ casper_tests/.

Debugging Casper.JS

Casper.js (via PhantomJS) has support for remote debugging. However, it is not perfect. Here are some steps for using it and gotchas you might want to know; you’ll likely also want to read the section on writing tests (below) if you get stuck, since the advice on how to write correct Casper selectors will likely be relevant. The ﬁrst thing to do when debugging Casper tests is to check the additional debug output that our framework provides: • You can check the screenshots of what the UI looked like at the time of failures at var/casper/ casper-failure*.png. • If it’s possible there’s a backend exception involved, var/casper/server.log will contain the server logs from the casper run; it’s worth looking there for tracebacks if you get stuck.

Print debugging

If you need to use print debugging in casper, you can do using casper.log; see http://docs.casperjs.org/en/latest/ logging.html for details. An additional debugging technique is to enable verbose mode in the Casper tests; you can do this by adding to the top of the relevant test ﬁle the following:

211 Zulip Documentation, Release 1.4.0

var casper= require('casper').create({ verbose: true, logLevel:"debug" });

This can sometimes give insight into exactly what’s happening.

Remote debugging

This is a pain to setup with Vagrant because port 7777 and 9981 aren’t forwarded to the host by default, but can be pretty useful in rare difﬁcult cases. To turn on remote debugging, pass --remote-debug to the ./frontend_tests/run-casper script. This will run the tests with port 7777 open for remote debugging. You can now connect to localhost:7777 in a Webkit browser. Somewhat recent versions of Chrome or Safari might be required. • When connecting to the remote debugger, you will see a list of pages, probably 2. One page called about:blank is the headless page in which the CasperJS test itself is actually running in. This is where your test code is. • The other page, probably localhost:9981, is the Zulip page that the test is testing—that is, the page running our app that our test is exercising. Since the tests are now running, you can open the about:blank page, switch to the Scripts tab, and open the running 0x-foo.js test. If you set a breakpoint and it is hit, the inspector will pause and you can do your normal JS debugging. You can also put breakpoints in the Zulip webpage itself if you wish to inspect the state of the Zulip frontend.

Reproducing races only seen in Travis CI

We’ve sometimes found it useful for reproducing Casper race conditions in Casper tests that mostly only happen in Travis CI with really cheap VPS servers (e.g. Scaleway’s 2GB x86). This works because an ultra slow machine is more likely to have things happen in an order similar to what happens in Travis CI’s very slow containers.

Writing Casper tests

Probably the easiest way to learn how to write Casper tests is to study some of the existing test ﬁles. There are a few tips that can be useful for writing Casper tests in addition to the debugging notes below: • Run just the ﬁle containing your new tests as described above to have a fast debugging cycle. • With frontend tests in general, it’s very important to write your code to wait for the right events. Before essentially every action you take on the page, you’ll want to use waitUntilVisible, waitWhileVisible, or a similar function to make sure the page or elemant is ready before you interact with it. For instance, if you want to click a button that you can select via #btn-submit, and then check that it causes success-elt to appear, you’ll want to write something like:

casper.waitUntilVisible("#btn-submit", function () { casper.click('#btn-submit') casper.test.assertExists("#success-elt"); });

212 Chapter 38. Web frontend black-box casperjs tests Zulip Documentation, Release 1.4.0

In many cases, you will actually need to wait for the UI to update clicking the button before doing asserts or the next step. This will ensure that the UI has ﬁnished updating from the previous step before Casper attempts to next step. The various wait functions supported in Casper are documented in the Casper here: http://docs.casperjs. org/en/latest/modules/casper.html#waitforselector and the various assert statements available are documented here: http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype • The casper.wait style functions (waitWhileVisible, waitUntilVisible, etc.) cannot be chained together in certain conditions without creating race conditions where the test may fail nondeterministically. For example, don’t do this:

casper.waitUntilVisible('tag 1'); casper.click('button'); casper.waitUntilVisible('tag 2');

Instead, if you want to avoid race condition, wrap the second waitFor in a then function like this:

casper.then(function () { casper.waitUntilVisible('tag 1', function () { casper.click('#btn-submit'); }); }); casper.then(function () { casper.waitUntilVisible('tag 2', function () { casper.test.assertExists('#success-elt'); }); });

(You’ll also want to use selectors that are as explicit as possible, to avoid accidentally clicking multiple buttons or the wrong button in your test, which can cause nondeterministic failures) • Generally casper.waitUntilVisible is preferable to e.g. casper.waitForSelector, since the former will confirm the thing is actually on screen. E.g. if you’re waiting to switch from one panel of the the settings overlay to another by waiting for a particular widget to appear, casper.waitForSelector may not actually wait (since the widget is probably in the DOM, just not visible), but casper.waitUntilVisible will wait until it’s actually shown. • The selectors (i.e. things you put inside casper.waitUntilVisible() and friends) appearing in Casper tests are CSS3 selectors, which is a slightly different syntax from the jQuery selectors used in the rest of the Zulip codebase; in particular, some expressions that work with jQuery (and thus normal Zulip JavaScript code) won’t work with CSS3. It’s often helpful to debug selectors interactively, which you can do in the Chrome JavaScript console. The way to do it is $$("#settings-dropdown"); that queries CSS3 selectors, so you can debug your selector in the console and then paste it into your Casper test once it’s working. For other browsers like Firefox, you can use querySelectorAll("#settings-dropdown"), syntax which is only available in the browser’s JavaScript console. You can learn more about these selectors and other JavaScript console tools here. • The test suite uses a smaller set of default user accounts and other data initialized in the database than the development environment; to see what differs check out the section related to options["test_suite"] in zilencer/management/commands/populate_db.py. • Casper effectively runs your test file in two phases – first it runs the code in the test file, which for most test files will just collect a series of steps (each being a casper.then or casper.wait... call). Then, usually at the end of the test file, you’ll have a casper.run call which actually runs that series of steps. This means that if you write code in your test file outside a casper.then or casper.wait... method, it will actually run before all the Casper test steps that are declared in the file, which can lead to confusing failures where the new code you write in between two casper.then blocks actually runs before either of them. See this for more details about how Casper works: http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work

38.2. Writing Casper tests 213 Zulip Documentation, Release 1.4.0

214 Chapter 38. Web frontend black-box casperjs tests CHAPTER 39

Travis CI

The Zulip server uses Travis CI for its continuous integration. This page documents useful tools and tips to know about when using Travis CI and debugging issues with it.

Goals

The overall goal of our Travis CI setup is to avoid regressions and minimize the total time spent debugging Zulip. We do that by trying to catch as many possible future bugs as possible, while minimizing both latency and false positives, both of which can waste a lot of developer time. There are a few implications of this overall goal: • If a test is failing nondeterministically in Travis CI, we consider that to be an urgent problem. • If the tests become a lot slower, that is also an urgent problem. • Everything we do in CI should also have a way to run it quickly (under 1 minute, preferably under 3 seconds), in order to iterate fast in development. Except when working on the Travis CI conﬁguration itself, a developer should never have to repeatedly wait 10 minutes for a full Travis run to iteratively debug something.

Conﬁguration

The main Travis configuration file is .travis.yml. The specific test suites we have are listed in the matrix section, which has a matrix of Python versions and test suites ($TEST_SUITE). We’ve configured it to use a few helper scripts for each job: • tools/travis/setup-$TEST_SUITE: The script that sets up the test environment for that suite (E.g., installing dependencies). – For the backend and frontend suites, this is a thin wrapper around tools/provision, aka the development environment provision script. – For the production suite, this is a more complicated process because of all the packages Travis installs. See the comments in tools/travis/setup-production for details.

215 Zulip Documentation, Release 1.4.0

• tools/travis/$TEST_SUITE: The script that runs the actual test suite. The main purpose of the distinction between the two is that if the setup-backend job fails, Travis CI will report it as the suite having “Errored” (grey in their emails), whereas if the backend job fails, it’ll be reported as “Failed” failure (red in their emails). Note that Travis CI’s web UI seems to make no visual distinction between these. An important detail is that Travis CI will by default hide most phases other than the actual test; you can see this easily by looking at the line numbers in the Travis CI output. There are actually a bunch of phases (e.g. the project’s setup job, downloading caches near the beginning, uploading caches at the end, etc.), and if you’re debugging our conﬁguration, you’ll want to look at these closely.

Useful debugging tips and tools

• For performance issues, this statistics tool can give you test runtime history data that can help with determining when a performance issue was introduced and whether it was ﬁxed. Note you need to click the “Run” button for it to do anything. • You can sign up your personal repo for Travis CI so that every remote branch you push will be tested, which can be helpful when debugging something complicated. • You can use the ts tool to get timing for every line of your Travis scripts (which can help with performance debugging). Note, however, that ts always return exit code 0, so with that patch, Travis CI will always report success.

Performance optimizations

Caching

An important element of making Travis CI perform effectively is caching the provisioning of a Zulip development environment. In particular, we cache the following across jobs: • Python virtualenvs • node_modules directories • Built/downloaded emoji sprite sheets and data This has a huge impact on the performance of running tests in Travis CI; without these caches, the average test time would be several times longer. We have designed these caches carefully (they are also used in production and the Zulip development environment) to ensure that each is named by a hash of its dependencies, so Zulip should always be using the same version of dependencies it would have used had the cache not existed. In practice, bugs are always possible, so be mindful of this possibility. A consequence of this caching is that test jobs for branches which modify package.json, requirements/, and other key dependencies will be signiﬁcantly slower than normal, because they won’t get to beneﬁt from the cache.

Uninstalling packages

In the production suite, we run apt-get upgrade at some point (effectively, because the Zulip installer does). This carries a huge performance cost in Travis CI, because (1) they don’t keep their test systems up to date and (2) literally everything is installed in their build workers (e.g. several copies of Postgres, Java, MySQL, etc.).

216 Chapter 39. Travis CI Zulip Documentation, Release 1.4.0

In order to make Zulip’s tests performance reasonably well, we uninstall (or mark with apt-mark hold) many of these dependencies that are irrelevant to Zulip in tools/travis/setup-production.

39.4. Performance optimizations 217 Zulip Documentation, Release 1.4.0

218 Chapter 39. Travis CI CHAPTER 40

Manual testing

As a general rule, we like to have automated tests for everything that can be practically tested. However, there are certain types of bugs that are best caught with old fashioned manual testing (also called manual QA). Manual testing not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics of a feature they’re working on. This doc assumes you know how to set up a local development server and open the Zulip app in the browser. It also assumes a basic knowledge of how to use Zulip.

Basic Stuff

When testing Zulip manually, here are things to focus on: • The best bugs to catch are security/permissions bugs. • Don’t rush manual testing. Look for small details like display glitches. • Always test with multiple users (you can use incognito windows to facilitate this). • Always keep the inspector console open and watch for warnings or errors. • Be methodical about collecting information on bugs. (You will eventually want to create tickets, but you may want to consolidate your own notes before ﬁling tickets.) You generally want to test with Cordelia as the primary user, and use Hamlet as her primary conversation partner. Use Iago when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things such as Cordelia not being able to receive messages not intended for her. The rest of this document groups tasks into basic areas of functionality of the system. If you have multiple people testing at once, you can divvy up QA tasks by these sections in the doc.

Message view

We mostly test the message view as part of testing everything else, but there are few things to specially test here.

219 Zulip Documentation, Release 1.4.0

Try using all the navigation hotkeys: • Up/k • Down/j • PgUp/K • PgDn/J/Spacebar • End (or fn-right-arrow on OSX) • also try scrolling aggressively with the mouse Try narrowing from the message view: • Hotkeys – use Esc to go to home – use s to narrow to a stream (select message ﬁrst and verify in sidebar) – use S to narrow to the topic (and verify in sidebar) – use v to navigate to private messages • Click on the recipient bar – narrow to a stream – narrow to a topic – narrow to PMs with one user – narrow to a group PM • Click on the Zulip logo – narrow to a topic – click on the Zulip logo (and verify you’re in the home view)

Message editing

With message editing we mainly want to exercise topic changes. Here are some tasks: • Do lots of editing – send a message to the topic “original” – edit the message content – send two messages to the “original” stream – start to edit a message but then cancel – change the topic for the ﬁrst message to “change1” (just this message) – narrow back to “original” – send one more message to the stream – change the topic for the last two messages to “change2” – narrow back to “original” – send two more messages to the stream

220 Chapter 40. Manual testing Zulip Documentation, Release 1.4.0

– edit the 2nd message on topic and change all messages to “change3” • Test UI entry points – hit “i” then down arrow to edit with the popup – use the popup using the mouse – enter edit mode using the pencil icon

Narrowing

Zulip uses the term “narrowing” to refer to opening different views of your messages, whether by clicking on sidebar options, recipient bars, or by using search. The main focus of these tasks should be watching unread counts. Of course, you also want to see messages show up in the message pane. And, ﬁnally, you should make sure that no messages outside the narrow show up in Cordelia’s view. Important: Make sure that Cordelia is subscribed to Verona but not subscribed to Denmark; if not, you should use different streams for your testing. When testing narrows, you want to have Hamlet send the same message several times in a row, while cycling Cordelia through various narrows. Here are the main tasks for Hamlet (and each message gets sent several times): • Send Cordelia/Othello a PM. • Send Cordelia a PM. • Send Othello a PM. • Post to Verona/foo. • Post to Verona/bar. • Post to Denmark/foo. • Post to Denmark/foo and mention Cordelia. For each of the above types of messages, you will want to cycle through the following views for Cordelia (and have Hamlet send new messages after each narrow): • Go to Home view. • Go to Private Messages view. • Go to Private Messages w/Hamlet. • Go to Private Messages w/Hamlet and Othello. • Go to Verona view. • Go to Verona/bar view. • Go to Verona/foo view. • Go to Denmark view. • Go to Denmark/foo view. There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then the whole exercise is about 30 minutes, assuming no bugs.

40.1. Basic Stuff 221 Zulip Documentation, Release 1.4.0

Composing messages

We have pretty good automated tests for our markdown processor, so manual testing is targeted more to other interactions. For composing a message, pay attention to details like what is automatically populated and where the focus is placed. • Hotkeys – use r to reply to a stream message – use r to reply to a PM – use R to reply to the author of a PM – use R to reply to the author of a PM stream – use c to compose a stream message – use C to compose a new PM • Buttons – Narrow to a stream and click on “New topic” – Narrow “Private Messages” and click on “New topic” – Narrow to a stream and click on “New private message” – Narrow “Private Messages” and click on “New private message” • Topics – Compose/send a message to a stream with no topic. – Compose/send a message to a stream with a new topic. – Compose/send a message to a stream with autocomplete. – Compose/send a message to a stream manually typing an existing topic. • Formatting stuff – Use the “A” icon to get markdown help. – Use the eyeball icon to show a preview and send from preview mode. – Toggle in and out of preview before sending a message. – Use @-mention to mention Hamlet (and send him a message). – Use #**devel** syntax and send to Hamlet, then follow the link. – Create a bulleted list. – Use the emoji icon to ﬁnd an emoji in the picker. • Attachments – Send a message with an attachment using the paperclip icon. – Send a message with multiple attachments. – Copy an image from the clipboard. – Use drag/drop from the desktop to upload an image. • Drafts – Start composing a message then click outside the compose box.

222 Chapter 40. Manual testing Zulip Documentation, Release 1.4.0

– Use “restore drafts” to restore the draft. – Start composing then use “Esc” to abort the message. – Use “restore drafts” to restore the draft. – Start composing a stream message and then abort using the little “x” icon in the compose box. – Click on “New private message” and restore the draft. (You should now be sending to a stream.) • Click to send – Turn off enter-to-send.

* Send a two-paragraph message using tab and enter. * Send a two-paragraph message using control-enter or command-enter. – Turn on enter-to-send.

* Hit enter to send.

Popover menus

For this task you just want to go through all of our popover menus and exercise them. The main nuance here is that you occasionally want to click somewhere on the UI outside of an existing popover to see if the popover menu is “too sticky.” Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view, then the message will disappear from the view. Here are the things to test: • Stream sidebar menus – Stream settings (just make sure it goes there) – Narrow (and then have Hamlet send a message) – Pin/unpin (do both) – Compose (send a message to the stream) – Mark as read (scroll back and then have Hamlet send you a message) – Mute/unmute (do both) – Unsubscribe (and then go to Stream settings in the gear menu to resubscribe) – Choose custom color (play around with this) • Topic sidebar menus – Narrow (and then have Hamlet send a message) – Mute/unmute (try both) – Mark as read (scroll back and then have Hamlet send you a message) • Left-message-pane menus (click on person’s name) – Verify email – Verify date message sent – Send a PM (make sure compose box is ﬁlled out ok) – Narrow to PMs with – Narrow to PMs sent by

40.1. Basic Stuff 223 Zulip Documentation, Release 1.4.0

• Right-pane-pane menus (click on chevron when hovering) – use “i” hotkey to open the menu – Edit a message you sent (using the down-arrow key to navigate the popup) – View Source for somebody else’s message (make sure it’s not editable) – Reply (send a message) – Collapse/uncollapse (try both) – Mute/unmute (try both, watch left sidebar) – Link to this conversation • Buddy list chevron menus – Narrow to PMs with – Narrow to message sent by – Compose a message to

Sidebar ﬁltering

This is a fairly quick task where we test the search filters on the left sidebar and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to that stream. • Streams filtering – Use “w” hotkey to open the search. – Filter on “d”. – Pin/unpin Denmark. – Clear filter. – Use “A” and “D” hotkeys to cycle through the streams. – Filter again and then click somewhere else. • Buddy list filtering – Use “q” hotkey to open the search. – Filter for Hamlet, Prospero, Othello, etc. – Log on Hamlet and log off Hamlet while filtering for Hamlet. – Log on/log off Hamlet while filtering for Othello. – Log on/log off Hamlet while not filtering at all. – Filter again and then click somewhere else.

Stream permissions

This is an important category to test, because we obviously do not want to have bugs where people can read messages on streams they should not have access to. The general ﬂow here is for Hamlet to create the streams and verify that Cordelia has the correct visibility to them. First, we start off with “positive” tests.

224 Chapter 40. Manual testing Zulip Documentation, Release 1.4.0

• Positive tests – Have Hamlet create a public stream w/Cordelia subscribed and have him post a message to the stream. – Have Hamlet create a public stream without Cordelia and then...

* Have Hamlet post to the stream. * Have Cordelia subscribe to the stream. * Verify Cordelia can see the previous message. * Have Cordelia post a message to the stream. – Have Hamlet create an invite-only stream with Cordelia invited and test a two-way conversation between the two users. For negative tests, we want to dig a little deeper to ﬁnd back doors for Cordelia to access the stream. Here are some techniques to try: • Try to have her compose a message to the stream by circumventing autocomplete. • Try to have her narrow to the stream using stream:foo in search. • Go to stream settings and see if the stream shows up. For public streams, it’s ok for Cordelia to know the stream exists, and she can subsequently subscribe. For private streams, she should not even know they exist (until she’s invited, of course). • Negative tests – Have Hamlet create a public stream without inviting Cordelia.

* Verify Cordelia can see the stream in her settings. * Verify Cordelia can’t compose a message to the stream. * Verify that Cordelia sees nothing when Hamlet posts to the stream. – Have Hamlet create a public stream with Cordelia, but then have Iago revoke her subscription using the admin page.

* Verify that the stream appears in Cordelia’s left sidebar and then goes away. * Try to have Cordelia view the stream using a sneaky search along the lines of stream:foo. – Have Hamlet create a private stream without inviting Cordelia.

* Verify Cordelia can’t compose a message to the stream.

The main task for testing search is to play around with search suggestions (autocomplete). Once you select an option, verify the message view is consistent with the search and that the left sidebar reﬂects the current narrow. If a search comes up legitimately empty, have Hamlet send a message that matches the search. Here are searches you should be able to do with autocomplete: • stream:design • stream:Verona topic:Verona1 • stream:Verona keyword • sent by me • @-mentions

40.1. Basic Stuff 225 Zulip Documentation, Release 1.4.0

• starred messages • messages sent by Hamlet • PMs with Hamlet • PMs with Hamlet matching keyword “foo” There are some things you can try that don’t come up in autocomplete: • -stream:Verona (exclude Verona) • stream:Verona stream:devel (should return no results) Miscellaneous: • Use the “/” hotkey to start a search. • Use the “x” icon to clear a search. • Use the “Esc” hotkey to clear a search.

Stream settings

Test various UI entry points into stream settings: • Use small gear menu in left sidebar, then filter to “devel”. • Use popover menu in left sidebar next to “devel”. • Use gear menu above buddy list and filter to “devel”. • Use gear menu and click on “devel.” • Use gear menu and then click on chevron menu next to “devel.” (I’m not sure why we still have the chevron at this writing.) Create new public stream “public1” and add Hamlet: • Type “public1” in the text box and then click “Create new stream.” • Select “People must be invited” and then verify you can’t select “Announce stream”. • Select “Anyone can join” again to make it be public. • Check the checkbox for Hamlet. • Hit the “Create” button. Test subscribe/unsubscribe: • Log in as Hamlet and go to his stream settings. • As Cordelia, unsubscribe from “public1” using the checkmark in the streams settings page. • Verify that Hamlet sees that Cordelia has unsubscribed (and the subscriber count should decrement). • As Cordelia, resubscribe to “public1.” • Verify Hamlet sees that change. As Cordelia, exercise different options in Create Stream dialog by creating streams s1, s2, s3, etc.: • s1: anyone can join, announce it, and add Hamlet using filter feature • s2: people must be invited • s3: anyone can join, don’t announce

226 Chapter 40. Manual testing Zulip Documentation, Release 1.4.0

• s4: check all, then uncheck all, then invite only Hamlet • s5: invite everybody but Hamlet • s6: – create the stream as public, but don’t subscribe anybody initially – then click on stream options to add Hamlet using “Add” button Test per-stream options: • Use “devel” stream and send a message to it • Do mute and unmute, have Hamlet send messages • Test notiﬁcations on/off, have Hamlet send messages • Test pin and unpin, view left sidebar • Change stream color, and then view the left sidebar and the Home message view • Verify stream subscriber counts in the main stream view

User Settings

You can modify per-user settings by choosing “Settings” in the gear menu. Do these tasks as Cordelia. • Your account – Change full name (Hamlet should see the name change) – Customize avatar – Deactivate account (and then log in as Iago to re-activate Cordelia) • Display settings – Right now, these unfortunately require reloads to take effect. – Default language (change to Spanish) – Display user list in narrow windows (verify by making window thinner) – Display time in 24-hour format (and then test going back to AM/PM) • Notiﬁcations – Stream Message

* turn off notiﬁcations at user level · create a new stream · have Hamlet send a message

* turn on notiﬁcations at user level · create a new stream · have Hamlet send a message · then turn off notiﬁcations for that stream · have Hamlet send another message – Private Messages and @-mentions

* Test Desktop/Audible options

40.1. Basic Stuff 227 Zulip Documentation, Release 1.4.0

* You can ignore other stuff for now • Bots/API key – Create a bot with a generic avatar and send it a PM – Create a bot with a custom avatar and send it a PM – Change your API key • Alert words – Create an alert word – Have Hamlet send you a message that includes the alert word • Zulip labs – Turn on auto-scroll to new messages (and have Hamlet send you one) – Turn on/off “Enable desktop notiﬁcations for new streams” and test. (We may eliminate this option soon.)

Keyboard Shorcuts

We mostly test keyboard shortcuts as part of other tasks. Here are the tasks for this section: • Use the ”?” hotkey to open the keyboard help • Proofread the dialog for typos. • Close the dialog. • Re-open the keyboard help using the gear menu. • Find a hotkey that you don’t frequently use and experiment with its usage.

Miscellaneous menu options

Make sure that these options launch appropriate help screens: • Proofread and try a couple random options: – Message formatting – Search operators • Make sure help launches in a separate browser tab: – Desktop and mobile apps – Integrations – API documentation

Inviting users/tutorial

Here are the tasks: • Invite [email protected] using the link beneath the buddy list but then don’t take further action. • Fully invite [email protected] using the gear menu.

228 Chapter 40. Manual testing Zulip Documentation, Release 1.4.0

• Go to the development console to get the login link for [email protected]. • Go through the signup ﬂow. • Follow the tutorial. • Use the gear menu to log out. • Log back in as Cordelia (admittedly, this step doesn’t really QA much of our production code, since the login ﬂow is customized for the development environment).

To be continued...

This document does not cover settings/admin options yet. The main things to do when testing the settings system are: • Verify that changes are synced to other users. • Verify error messages appear if you do something wrong and look right. • For organization settings, verify that they look right in read-only mode (i.e. when not logged into an administrator account).

40.1. Basic Stuff 229 Zulip Documentation, Release 1.4.0

230 Chapter 40. Manual testing CHAPTER 41

Settings system

The page documents the Zulip settings system, and hopefully should help you decide how to correctly implement new settings you’re adding to Zulip. We have two types of administrative settings in Zulip: server settings (which are set via configuration files are apply to the whole Zulip installation), and realm settings (which are usually set via the /#organization page in the Zulip web application) and apply to a single Zulip realm/organization (which for most Zulip servers is the only realm on the server). Philosophically, the goals of the settings system are to make it convenient for: • Zulip server administrators to configure Zulip’s featureset for their server without needing to patch Zulip • Realm administrators to configure settings for their organization independently without needing to talk with the server administrator. • Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable configuration.

Server settings

Zulip uses the Django settings system, which means that the settings ﬁles are Python programs that set a lot of variables with all-capital names like EMAIL_GATEWAY_PATTERN. You can access these anywhere in the Zulip Django code using e.g.: from django.conf import settings print(settings.EMAIL_GATEWAY_PATTERN)

Additionally, if you need to access a Django setting in a shell script (or just on the command line for debugging), you can use e.g.:

$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN %s@localhost:9991

Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the ~1000 lines of settings needed by the Zulip Django app. As a result, there are a few ﬁles involved in the Zulip settings for server administrators. In a production environment, we have:

231 Zulip Documentation, Release 1.4.0

• /etc/zulip/settings.py (the template is in the Zulip repo at zproject/ prod_settings_template.py) is the main system administrator-facing settings file for Zulip. It contains all the server-specific settings, such as how to send outgoing email, the hostname of the Postgres database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys, etc.). The way we generally do settings that can be controlled with shell access to a Zulip server is to put a default in zproject/settings.py, and then override it here. • /etc/zulip/zulip-secrets.conf (generated by scripts/setup/generate_secrets.py as part of installation) contains secrets used by the Zulip installation. These are read using the standard Python ConfigParser, and accessed in zproject/settings.py by the get_secret function. All secrets/API keys/etc. used by the Zulip Django application should be stored here, and read using the get_secret function in zproject/settings.py. • zproject/settings.py is the main Django settings file for Zulip. It contains all the settings that are constant for all Zulip installations (e.g. configuration for logging, static assets, middleware, etc.), as well as default values for the settings the user would set in /etc/zulip/settings.py (you can look at the DEFAULT_SETTINGS dictionary to easily review the settings available). zproject/settings.py has a line from prod_settings import *, which has the effect of importing /etc/zulip/settings. py in a prod environment (via a symlink). In a development environment, we have zproject/settings.py, and additionally: • zproject/dev_settings.py has the settings for the Zulip development environment; it mostly just imports prod_settings_template.py. • zproject/dev-secrets.conf replaces /etc/zulip/zulip-secrets.conf. • zproject/test_settings.py has the (default) settings used for the Zulip tests (both backend and Casper), which are applied on top of the development environment settings. When adding a new server setting to Zulip, you will typically add it in two or three places: • In DEFAULT_SETTINGS in zproject/settings.py, with a default value for production environments. If the settings has a secret key, you’ll add a get_secret call in zproject/settings.py (and the user will add the value when they configure the feature). • In an appropriate section of zproject/prod_settings_template.py, with documentation in the comments explaining the settings’s purpose and effect. • Possibly also zproject/dev_settings.py, if the desired value of the setting for Zulip development environments is different from the default for production (and similarly for zproject/test_settings. py). Most settings should be enabled in the development environment, to maximize convenience of testing all of Zulip’s features; they should be enabled by default in production if we expect most Zulip sites to want those settings.

Testing non-default settings

You can write tests for settings using e.g. with self.settings(GOOGLE_CLIENT_ID=None). However, this only works for settings which are checked at runtime, not settings which are only accessed in initialization of Django (or Zulip) internals (e.g. DATABASES). See the Django docs on overriding settings in tests for more details.

Realm settings

Realm settings are preferred for any conﬁguration that is a matter of organizational policy (as opposed to technical capabilities of the server). As a result, conﬁguration options for user-facing functionality is almost always added as a

232 Chapter 41. Settings system Zulip Documentation, Release 1.4.0 new realm setting, not a server setting. The new feature tutorial documents the process for adding a new realm setting to Zulip. So for example, the following server settings will eventually be replaced with realm settings: • NAME_CHANGES_DISABLED • INLINE_IMAGE_PREVIEW • ENABLE_GRAVATAR • Which authentication methods are allowed should probably appear in both places; in server settings indicating the capabilities of the server, and in the realm settings indicating which methods the realm administrator wants to allow users to login with.

41.2. Realm settings 233 Zulip Documentation, Release 1.4.0

234 Chapter 41. Settings system CHAPTER 42

Real-time Push and Events

Zulip’s “events system” is the server-to-client push system that powers our real-time sync. This document explains how it works; to read an example of how a complete feature using this system works, check out the new application feature tutorial. Any single-page web application like Zulip needs a story for how changes made by one client are synced to other clients, though having a good architecture for this is particularly important for a chat tool like Zulip, since the state is constantly changing. When we talk about clients, think a browser tab, mobile app, or API bot that needs to receive updates to the Zulip data. The simplest example is a new message being sent by one client; other clients must be notified in order to display the message. But a complete application like Zulip has dozens of different types of data that need to be synced to other clients, whether it be new streams, changes in a user’s name or avatar, settings changes, etc. In Zulip, we call these updates that need to be sent to other clients events. An important thing to understand when designing such a system is that events need to be synced to every client that has a copy of the old data if one wants to avoid clients displaying inaccurate data to users. So if a user has two browser windows open and sends a message, every client controlled by that user as well as any recipients of the message, including both of those two browser windows, will receive that event. (Technically, we don’t need to send events to the client that triggered the change, but this approach saves a bunch of unnecessary duplicate UI update code, since the client making the change can just use the same code as every other client, maybe plus a little notification that the operation succeeded). Architecturally, there are a few things needed to make a successful real-time sync system work: • Generation. Generating events when changes happen to data, and determining which users should receive each event. • Delivery. Efficiently delivering those events to interested clients, ideally in an exactly-once fashion. • UI updates. Updating the UI in the client once it has received events from the server. Reactive JavaScript libraries like React and Vue can help simplify the last piece, but there aren’t good standard systems for doing generation and delivery, so we have to build them ourselves. This document discusses how Zulip solves the generation and delivery problems in a scalable, correct, and predictable way.

235 Zulip Documentation, Release 1.4.0

Generation system

Zulip’s generation system is built around a Python function, send_event(event, users). It accepts an event data structure (just a Python dictionary with some keys and value; type is always one of the keys but the rest depends on the specific event) and a list of user IDs for the users whose clients should receive the event. In special cases such as message delivery, the list of users will instead be a list of dicts mapping user IDs to user-specific data like whether that user was mentioned in that message. The data passed to send_event are simply marshalled as JSON and placed in the notify_tornado RabbitMQ queue to be consumed by the delivery system. Usually, this list of users is one of 3 things: • A single user (e.g. for user-level settings changes). • Everyone in the realm (e.g. for organization-level settings changes, like new realm emoji). • Everyone who would receive a given message (for messages, emoji reactions, message editing, etc.); i.e. the subscribers to a stream or the people on a private message thread. It is the responsibility of the caller of send_event to choose the list of user IDs correctly. There can be security problems if e.g. an event containing private message content is sent to the entire organization. However, if an event isn’t sent to enough clients, there will likely be user-visible real-time sync bugs. Most of the hard work in event generation is about defining consistent event dictionaries that are clear, readable, will be useful to the wide range of possible clients, and make it easy for developers.

Delivery system

Zulip’s event delivery (real-time push) system is based on Tornado, which is ideal for handling a large number of open requests. Details on Tornado are available in the architecture overview, but in short it is good at holding open a large number of connections for a long time. The complete system is about 1500 lines of code in zerver/tornado/, primarily zerver/tornado/event_queue.py. Zulip’s event delivery system is based on “long-polling”; basically clients make GET /json/events calls to the server, and the server doesn’t respond to the request until it has an event to deliver to the client. This approach is reasonably efﬁcient and works everywhere (unlike websockets, which have a decreasing but nonzero level of client compatibility problems). For each connected client, the Event Queue Server maintains an event queue, which contains any events that are to be delivered to that client which have not yet been acknowledged by that client. Ignoring the subtle details around error handling, the protocol is pretty simple; when a client does a GET /json/events call, the server checks if there are any events in the queue. If there are, it returns the events immediately. If there aren’t, it records that queue as having a waiting client (often called a handler in the code). When it pulls an event off the notify_tornado RabbitMQ queue, it simply delivers the event to each queue associated with one of the target users. If the queue has a waiting client, it breaks the long-poll connection by returning an HTTP response to the waiting client request. If there is no waiting client, it simply pushes the event onto the queue. When starting up, each client makes a POST /json/register to the server, which creates a new event queue for that client and returns the queue_id as well as an initial last_event_id to the client (it can also, optionally, fetch the initial data to save an RTT and avoid races; see the below section on initial data fetches for details on why this is useful). Once the event queue is registered, the client can just do an inﬁnite loop calling GET /json/ events with those parameters, updating last_event_id each time to acknowledge any events it has received (see call_on_each_event in the Zulip Python API bindings for a complete example implementation). When handling each GET /json/events request, the queue server can safely delete any events have have an event ID less than or equal to the client’s last_event_id (event IDs are just a counter for the events a given queue has received.)

236 Chapter 42. Real-time Push and Events Zulip Documentation, Release 1.4.0

If network failures were impossible, the last_event_id parameter in the protocol would not be required, but it is important for enabling exactly-once delivery in the presence of potential failures. (Without it, the queue server would have to delete events from the queue as soon as it attempted to send them to the client; if that specific HTTP response didn’t reach the client due to a network TCP failure, then those events could be lost). The queue servers are a very high-traffic system, processing at a minimum a request for every message delivered to every Zulip client. Additionally, as a workaround for low-quality NAT servers that kill HTTP connections that are open without activity for more than 60s, the queue servers also send a heartbeat event to each queue at least once every 45s or so (if no other events have arrived in the meantime). To avoid a large memory and other resource leak, the queues are garbage collected after (by default) 10 minutes of inactivity from a client, under the theory that the client has likely gone off the Internet (or no longer exists) access; this happens constantly. If the client returns, it will receive a “queue not found” error when requesting events; it’s handler for this case should just restart the client / reload the browser so that it refetches initial data the same way it would on startup. Since clients have to implement their startup process anyway, this approach adds minimal technical complexity to clients. A nice side effect is that if the Event Queue Server server (which stores queues in memory) were to crash and lose its data, clients would recover, just as if they had lost Internet access briefly (there is some DoS risk to manage, though). (The Event Queue Server is designed to save any event queues to disk and reload them when the server is restarted, and catches exceptions carefully, so such incidents are very rare, but it’s nice to have a design that handles them without leaving broken out-of-date clients anyway).

The initial data fetch

When a client starts up, it usually wants to get 2 things from the server: • The “current state” of various pieces of data, e.g. the current settings, set of users in the organization (for typeahead), stream, messages, etc. (aka the “initial state”). • A subscription to receive updates to those data when they are changed by a client (aka an event queue). Ideally, one would get those two things atomically, i.e. if some other user changes their name, either the name change happens before the fetch (and thus the old name is in the initial state and there will be an event in the queue for the name change) or after (the new name is in the initial state, and there is no event for that name change in the queue). Achieving this atomicity goals means we save a huge amount of work that the N clients for Zulip don’t need to worry about a wide range of potential rare and hard to reproduce race conditions; we just have to implement things correctly once in the Zulip server. This is quite challenging to do technically, because fetching the initial state for a complex web application like Zulip might involve dozens of queries to the database, caches, etc. over the course of 100ms or more, and it is thus nearly impossible to do all of those things together atomically. So instead, we use a more complicated algorithm that can produce the atomic result from non-atomic subroutines. Here’s how it works when you make a register API request; the logic is in zerver/views/events_register.py and zerver/lib/events.py. The request is directly handled by Django: • Django makes an HTTP request to Tornado, requesting that a new event queue be created, and records its queue ID. • Django does all the various database/cache/etc. queries to fetch the data, non-atomically, from the various data sources (see the fetch_initial_state_data function). • Django makes a second HTTP request to Tornado, requesting any events that had been added to the Tornado event queue since it was created.

42.3. The initial data fetch 237 Zulip Documentation, Release 1.4.0

• Finally, Django “applies” the events (see the apply_events function) to the initial state that it fetched. E.g. for a name change event, it finds the user data in the realm_user data struture, and updates it to have the new name. This achieves everything we desire, at the cost that we need to write the apply_events function. This is a difficult function to implement correctly, because the situations that it tests for almost never happen (being race conditions). So we have a special test class, EventsRegisterTest, that is specifically designed to test this function by ensuring the possible race condition always happens. In particular, it does the following: • Call fetch_initial_state_data to get the current state. • Call a state change function that issues an event, e.g. do_change_full_name, and capture any events that are generated. • Call apply_events(state, events), and compare the result to calling fetch_initial_state_data again now. The apply_events code is correct if those two results are identical. The final detail we need to ensure that apply_events always works correctly is to make sure that we have EventsRegisterTest tests for every event type that can be generated by Zulip. This can be tested manually using test-backend --coverage EventsRegisterTest and then checking that all the calls to send_event are covered. Someday we’ll add automation that verifies this directly by inspecting the coverage data. In the Zulip webapp, the data returned by the register API is available via the page_params parameter.

Messages

One exception to the protocol described in the last section is the actual messages. Because Zulip clients usually fetch them in a separate AJAX call after the rest of the site is loaded, we don’t need them to be included in the initial state data. To handle those correctly, clients are responsible for discarding events related to messages that the client has not yet fetched.

238 Chapter 42. Real-time Push and Events CHAPTER 43

Queue processors

Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes: • Asynchronously doing expensive operations like sending email notifications which can take seconds per email and thus would otherwise timeout when 100s are triggered at once (E.g. inviting a lot of new users to a realm). • Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g. UserActivityInternal) which don’t have any immediate runtime effect. • Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the Tornado-based events system. Example events might be that a new message was sent, a user has changed their subscriptions, etc. • Processing mobile push notifications and email mirroring system messages. • Processing various errors, frontend tracebacks, and slow database queries in a batched fashion. • Doing markdown rendering for messages delivered to the Tornado via websockets. Needless to say, the RabbitMQ-based queuing system is an important part of the overall Zulip architecture, since it’s in critical code paths for everything from signing up for account, to rendering messages, to delivering updates to clients. We use the pika library to interface with RabbitMQ, using a simple custom integration defined in zerver/lib/ queue.py.

Adding a new queue processor

To add a new queue processor: • Deﬁne the processor in zerver/worker/queue_processors.py using the @assign_queue decorator; it’s pretty easy to get the template for an existing similar queue processor. This sufﬁces to test your queue worker in the Zulip development environment (tools/run-dev.py will automatically restart the queue processors and start running your new queue processor code). You can also run a single queue processor manually using e.g. ./manage.py process_queue --queue=user_activity.

239 Zulip Documentation, Release 1.4.0

• So that supervisord will known to run the queue processor in production, you will need to add to to normal_queues in puppet/zulip/manifests/base.pp; the list there is used to generate /etc/ supervisor/conf.d/zulip.conf via a puppet template in app_frontend.pp. The queue will automatically be added to the list of queues tracked by scripts/nagios/ check-rabbitmq-consumers, so Nagios can properly check whether a queue processor is running for your queue. You still need to update the sample Nagios conﬁguration in puppet/zulip_ops manually.

Publishing events into a queue

You can publish events to a RabbitMQ queue using the queue_json_publish function deﬁned in zerver/ lib/queue.py.

Clearing a RabbitMQ queue

If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue , for example:

./manage.py purge_queue user_activity

You can also use the amqp tools directly. Install amqp-tools from apt and then run: amqp-delete-queue--username=zulip--password='...'--server=localhost \ --queue=user_presence with the RabbitMQ password from /etc/zulip/zulip-secrets.conf.

240 Chapter 43. Queue processors CHAPTER 44

Zulip bot system

Zulip’s features can be extended by the means of bots and integrations. • Integrations are used to connect Zulip with different chat, scheduling and workﬂow software. If this is what you are looking for, please check out the integrations guide. • Bots, as a more general concept, intercept and react to messages. If this is what you are looking for, read on! The purpose of this documentation is to provide you with information about Zulip’s bot system. On this page you’ll ﬁnd: • A step-by-step tutorial on how to run a bot. • A step-by-step tutorial on how to develop a bot. •A documentation of the bot API. • Common problems when developing/running bots and their solutions. Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up with any tips or tools that help with writing bots, please visit #bots on the Zulip development community server, open an issue, or submit a pull request to share your ideas!

The bots system

Zulip’s bot system resides in the api directory. The structure of the bots ecosystem in the api directory looks like the following: api --bots --bot1 --bot2 | --readme.md --bot2.py

241 Zulip Documentation, Release 1.4.0

Each subdirectory in bots contains a bot. When developing bots, try to use the structure outlined above as an orientation.

How to run a bot

This guide will show you how to run a bot on a running Zulip server. It assumes you want to use one of the existing api/bots bots in your Zulip organization. If you want to write a new one, you just need to write the .py script and put it into /api/bots/ directory. You need: • An account in an organization on a Zulip server (e.g. chat.zulip.org or yourSubdomain.zulipchat.com, or your own development server). Within that Zulip organization, users will be able to interact with your bot. • A computer where you’re running the bot from, with a clone of the Zulip repository, which contains the bot library code in its api/bots_api/ subdirectory. This is required to run your bot. The following instructions assume this repository to be located in ~/zulip/. Note: Please be considerate when testing experimental bots on public servers such as chat.zulip.org. 1. Register a new bot user on the Zulip server’s web interface. • Log in to the Zulip server. • Navigate to Settings -> Your bots -> Add a new bot, fill out the form and click on Create bot. • A new bot user should appear in the Your bots panel. 2. Download the bot’s zuliprc configuration file to your computer. • In the Your bots panel, click on the green icon to download its configuration file zuliprc (the structure of this file is explained here.) • Copy the file to a destination of your choice, e.g. to ~/.zuliprc or ~/zuliprc-test. Note that the destination should be accessible from your Zulip dev environment (e.g. Vagrant or Digital Ocean). 3. Subscribe the bot to the streams that the bot needs to interact with. • To subscribe your bot to streams, navigate to Manage Streams. Select a stream and add your bot by its email address (the address you assigned in step 1). • Now, the bot can do its job on the streams you subscribed it to. • (In future versions of the API, this step may not be required). 4. Run the bot.

242 Chapter 44. Zulip bot system Zulip Documentation, Release 1.4.0

• In your Zulip repository, navigate to ~/zulip/api/bots_api/ • Run

python run.py ../bots//.py --config-file ~/.zuliprc`

(using the path to the .zuliprc ﬁle from step 2). • Check the output of the command. It should start with the text the usage function returns, followed by logging output similar to this:

INFO:root:starting message handling... INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection

˓→(1): localhost

• Congrats! Now, your bot should be ready to test on the streams you’ve subscribed it to.

Testing the helloworld bot

• The helloworld bot is a simple bot that responds with a ‘beep boop’ when queried. It can be used as a template to build more complex bots. • Go to a stream your bot is subscribed to. Talk to the bot by typing @ followed by some commands. If the bot is the helloworld bot, you should expect the bot to respond with “beep boop”.

How to develop a bot

The tutorial below explains the structure of a bot .py, which is the only ﬁle you need to create to develop a new bot. You can use this as boilerplate code for developing your own bot. Every bot is built upon this structure:

class MyBotHandler(object): ''' A docstring documenting this bot. '''

def usage(self): return '''Your description of the bot'''

def handle_message(self, message, client, state_handler): # add your code here

handler_class= MyBotHandler

• The class name (in this case MyBotHandler) can be deﬁned by you and should match the name of your bot. To register your bot’s class, adjust the last line handler_class = MyBotHandler to match your class name. • Every bot needs to implement the functions – usage(self) – handle_message(self, message, client) • These functions are documented in the next section.

44.3. How to develop a bot 243 Zulip Documentation, Release 1.4.0

Bot API

This section documents functions available to the bot and the structure of the bot’s conﬁg ﬁle. With this API, you can • intercept, view, and process messages sent by users on Zulip. • send out new messages as replies to the processed messages. With this API, you cannot • modify an intercepted message (you have to send a new message). • send messages on behalf of or impersonate other users. • intercept private messages (except for PMs with the bot as an explicit recipient). usage usage(self) is called to retrieve information about the bot.

Arguments

• self - the instance the method is called on.

Return values

• A string describing the bot’s functionality

Example implementation def usage(self): return ''' This plugin will allow users to flag messages as being follow-up items. Users should preface messages with"@followup". Before running this, make sure to create a stream called"followup" that your API user can send to. ''' handle_message handle_message(self, message, client) handles user message.

244 Chapter 44. Zulip bot system Zulip Documentation, Release 1.4.0

Arguments

• self - the instance the method is called on. • message - a dictionary describing a Zulip message • client - used to interact with the server, e.g. to send a message • state_handler - used to save states/information of the bot beta – use state_handler.set_state(state) to set a state (any object) – use state_handler.get_state() to retrieve the state set; returns a NoneType object if no state is set

Return values

None.

Example implementation

def handle_message(self, message, client, state_handler): original_content= message['content'] original_sender= message['sender_email'] new_content= original_content.replace('@followup', 'from %s:'% (original_sender,))

client.send_message(dict( type='stream', to='followup', subject=message['sender_email'], content=new_content, )) client.send_message client.send_message(message) will send a message as the bot user. Generally, this is less convenient than send_reply, but it offers additional ﬂexibility about where the message is sent to.

Arguments

• message - a dictionary describing the message to be sent by the bot

Example implementation client.send_message(dict( type='stream', # can be 'stream' or 'private' to=stream_name, # either the stream name or user's email subject=subject, # message subject content=message, # content of the sent message ))

44.4. Bot API 245 Zulip Documentation, Release 1.4.0 client.send_reply client.send_reply(message, response) will reply to the triggering message to the same place the original message was sent to, with the content of the reply being response.

Arguments

• message - Dictionary containing information on message to respond to (provided by handle_message). • response - Response message from the bot (string). client.update_message client.update_message(message) will edit the content of a previously sent message.

Arguments

• message - dictionary deﬁning what message to edit and the new content

Example

From /zulip/api/bots/incrementor/incrementor.py: client.update_message(dict( message_id=self.message_id, # id of message to be updated content=str(self.number), # string with which to update message with ))

Conﬁguration ﬁle

[api] key= email= site=

• key - the API key you created for the bot; this is how Zulip knows the request is from an authorized user. • email - the email address of the bot, e.g. [email protected] • site - your development environment URL; if you are working on a development environment hosted on your computer, use localhost:9991

Common problems

• I modiﬁed my bot’s code, yet the changes don’t seem to have an effect. – Ensure that you restarted the run.py script.

246 Chapter 44. Zulip bot system Zulip Documentation, Release 1.4.0

• My bot won’t start – Ensure that your API config file is correct (download the config file from the server). – Ensure that you bot script is located in zulip/api/bots// – Are you using your own Zulip development server? Ensure that you run your bot outside the Vagrant environment. – Some bots require Python 3. Try switching to a Python 3 environment before running your bot:

source/srv/zulip-py3-venv/bin/activate

Note that you can switch back to a Python 2 environment as follows:

source/srv/zulip-venv/bin/activate

• My bot works only on some streams. – Subscribe your bot to other streams, as described here.

Future direction

The long-term plan for this bot system is to allow the same BotHandler code to eventually be usable in several contexts: • Run directly using the Zulip call_on_each_message API, which is how the implementation above works. This is great for quick development with minimal setup. • Run in a simple Python webserver server, processing messages received from Zulip’s outgoing webhooks integration. • For bots merged into the mainline Zulip codebase, enabled via a button in the Zulip web UI, with no code deployment effort required.

44.6. Future direction 247 Zulip Documentation, Release 1.4.0

248 Chapter 44. Zulip bot system CHAPTER 45

Custom Apps

Deﬁnition

Zulip deﬁnes a “custom app” to be a piece of code that runs in the Zulip ecosystem, but which is not part of the core Zulip codebase. Custom apps are mostly synonymous with “bots” and “integrations” in the Zulip ecosystem. We currently do not support any kind of browser plugin model.

Problem statement

Zulip wants to enable people in the world to author custom apps with the following goals in mind: • Simple custom apps should be simple to write and deploy. • Custom app authors should be able to easily distribute their work. • Zulip should provide deployment support for mature, general-purpose bots, ideally either within organizations (a Zulip admin can vet her own custom apps and easily deploy them across upgrade cycles) or across organizations (custom apps get distributed with the Zulip tarball). This document describes Zulip’s current infrastructure, as well as laying out a roadmap for some future features.

A quick note on bots/integrations

As noted earlier, a custom app is just a generic term for what we often call bots or integrations. We recognize that bots and integrations can have different connotations. A bot typically spends most of its time responding to Zulip messages. An integration usually represents an app that interacts with some large third party system like an issue tracker. We will use both terms in this document in an informal sense, but from an architecture standpoint, we treat bots and integrations as essentially two shades of the same color. Many integrations are implemented as “bots.” Likewise, any bot that does stuff outside of Zulip acts as an “integration.”

249 Zulip Documentation, Release 1.4.0

Since the line between what a “bot” is and what an “integration” is can get very blurry, we try to be informal about “bots/integrations” and more formal about how “custom apps” actually function within the system.

Categories of custom apps

Stimulus/response and read/write

At the end of the day, most useful apps respond to some stimulus and produce a response. In the Zulip universe, Zulip can be the source of the stimulus, or the target of the response, or both. Along those lines, we divide custom apps into these three types: •A Zulip Reader uses activity on Zulip to stimulate an external response. An example here would be a follow-up bot that sees messages with the alert word “@todo” on a stream and then adds a task to a third party todo-list tool. •A Zulip Writer reacts to external stimuli and generates Zulip responses. An example here might be a build bot that gets triggered by an automated code build finishing and then writes “build finished” to a Zulip stream. •A Zulip Read/Writer reacts to a stimulus from Zulip by responding to Zulip. An example here would be a math bot that sees a message saying “compute 2+2” and responds with “2+2=4” on the same stream or back to the user in a PM. The above three classifications represent kind of a Zulip-centric view of the universe, but we should put ourselves in the shoes of somebody “out in the world.” •A World Reader is an app that gets some stimulus from the outside world and produces a response in Zulip. (So, a world reader is a Zulip writer.) •A World Writer is an app that gets some stimulus from Zulip and produces a response in the outside world. (So, a world writer is a Zulip reader.) Some things are a little outside of the scope of this document. We could plausibly extend Zulip some day to host World Reader/Writer apps that don’t even write Zulip messages but simply use Zulip as a kind of middleware platform. More in the short term, we will have custom apps that may read/write from multiple sources. For example, a meeting bot may take input from both a cron job and a Zulip stream, and it may write to both a Zulip stream and a third party calendar tool. For the scope of this document, we won’t spend a lot of time talking about how to build these types of apps, but we are aware that any solution needs to accommodate multiple sources and targets.

World Reader/Zulip Reader

Finally, we set the stage for how we talk about custom apps in terms of these two broad categories: •A World Reader responds to stimuli from the outside world (and typically produces a response in Zulip). •A Zulip Reader responds to stimuli from Zulip conversations (and typically produces a response in the outside world). Again, we recognize that there can be overlap between those two categories for complex custom apps, but we mostly leave it as an exercise for the reader how to implement those apps.

Other classiﬁcations

We discussed one dimension for classifying custom apps, which is whether they are world-readers or Zulip-readers. Here we cover a few other classiﬁcation schemes brieﬂy:

250 Chapter 45. Custom Apps Zulip Documentation, Release 1.4.0

• Generality Does the custom app have a specific use case or a general one? The spectrum here could run from a bot that Alice runs to update a text file on her laptop (specific) to a Twitter Bot that is optionally deployed on all Zulip realms (general). • Authorship Who wrote the custom app? Was it written by contributors to the Zulip project? • Maturity How well tested is the custom app? Is it just a prototype? Has it been sanctioned by an open source community? Has it been vetted by Zulip developers? • Deployment Where does the custom app run? Does it run on Alice’s laptop? Does it run on a Zulip server? Does it run as a plugin on third party infrastructure? • Authorization What streams are the custom app allowed to read and write from? Which users can the custom app interact with? • Identity How does the custom app identify itself on Zulip? How does it identify itself to the outside world? • Third party We call the non-Zulip target or source of a custom app the “world.” The “world” could be almost anything, ranging from an electronic device or text file to a large third-party system like Twitter or GitHub. A lot of the classification schemes are interrelated. Here are some examples: • For specific-purpose custom apps, authors may be happy to just deploy them on their own hardware. For general- use custom apps, authors may want to have them deployed on the Zulip server with super-user capabilities. • As a custom app becomes more well-tested and well-vetted, the author will likely upgrade its deployment over time. At first the author may run the custom app on their laptop, then they may find dedicated hardware, and then finally they contribute the app to the Zulip project so that Zulip admins can deploy the app on Zulip servers. • The nature of the third party will influence the deployment strategy. If I have a little home-grown gadget that can turn off the lights in my kitchen, I may run a custom app on my laptop that reads my PMs for “turn-off-the-light” messages. If I write a generic custom app that needs to update a third party corporate system based on Zulip events, I may want to deploy code to a public webserver or try to get my code to be part of the Zulip project itself.

World Reader

A World Reader custom app is an app that responds to stimuli from the world outside of Zulip. It typically functions as a Zulip Writer and posts some kind of message to a Zulip stream or user to alert people of world events. Here are some example stimuli: • A Travis build ﬁnishes. • Somebody tweets on Twitter. • A hardware sensor notices a temperature increase. • A pull request is submitted to GitHub. • A cron job gets started on your laptop to send a reminder. • Nagios detects a system anomaly. Setting aside issues of how a custom app is constructed or deployed, you basically have to solve these problems: • Detect events. • Translate events into Zulip messages. • Post the messages to Zulip.

45.5. World Reader 251 Zulip Documentation, Release 1.4.0

Zulip integrations

Zulip actually supports a bunch of integrations out-of-the-box that perform as World Readers. The three different integration models basically differ in where they perform the main functions of a World Reader.

Webhook integrations

In a webhook integration, the deployment model is usually this:: 3rd party hardware: • detect event • send data to Zulip webhook Zulip: • support webhook endpoint • translate event to messages • internally post messages One current limitation of our system is that we don’t have a great way to deploy prototypes of webhook-based custom apps before Zulip has vetted the translation and added an ofﬁcial endpoint. Maybe we could set up some kind of webserver that can run translation code outside of Zulip and externally post the messages, and we could think about how to structure the code so that it is easy to eventually turn it into a Zulip-hosted integration.

Python scripts

In script integrations, the deployment model is usually this: Custom app author’s hardware: • detect event by polling a third party system • translate event in the script • externally post messages These type of integrations are typically easy to prototype, but they can be harder to deploy in production settings, since we rely on the authors to run their own scripts. In some cases authors might want to at least move the translation/posting code to live on Zulip, by contributing that code to Zulip as a server-side integration. Then, there would still be the challenge of detecting events in the third party system, where maybe the user submits a patch to the third party as well.

Plugin integrations

In plugin integrations, the deployment model is usually this: Third party system (driver): • detect event Third party system (plugin): • further detect/triage event • translate event

252 Chapter 45. Custom Apps Zulip Documentation, Release 1.4.0

• externally post to Zulip For third parties that have a plugin model, there are often other issues at play, like the plugins may need to be written in a non-Python language like Ruby. There are probably still some scenarios, however, where a lot of the logic for translation could be moved to a Zulip-side integration, and then we supply very thin client code for the plugin.

Zulip Reader

A Zulip Reader custom app gets stimuli from Zerver itself. Most Zulip Reader apps are packaged/advertised more as what people commonly call “bots” than as “integrations.” (But sometimes what is currently a “bot” should really be deployed more like an “integration” in an ideal Zulip universe.) Example custom Zulip Reader apps can be serious or whimsical. Serious • A user tags a message with an alert word like @followup or @ticket. • A user needs help computing something, like a simple math expression or a timezone conversion. •A World Reader custom app posts something to a Zulip stream that we want to cross-post to another external system. • A user wants the custom app to query the outside world, like look up the weather or search Wikipedia. • A bot collects RSVPs for an event. • A bot conducts a user survey. Whimsical • A user wants to see a random quote of the day or a random cat fact. • A user wants to tell the ofﬁce telepresence robot to “turn left.” Setting aside whether a custom app is performing a serious or whimsical function, there are a few different types of Zulip Readers: • Some readers will do simple local computations and post right back to Zulip. • Some readers will do more expensive/web-related computations like searching Wikipedia, but then post right back to Zulip. • Some readers will mutate the outside world in some way, like posting messages to third party APIs or controlling hardware. • Some readers will do some combination of the prior bullets.

Deployment issues

Zulip currently provides only minimal deployment support for Zulip Reader custom apps: • It ships with a few native server-side bots like the welcome bot and the notiﬁcations bot. (These are nice to have, but they are so tightly integrated into the Zulip core that they don’t act as great examples for future app authors, and they are not easy to extend/customize.) • Zulip does ship an API client that can conveniently read a .zuliprc ﬁle, poll for incoming messages/events, and post new messages to the Zulip server.

45.6. Zulip Reader 253 Zulip Documentation, Release 1.4.0

Local deployment

If you download the API client and write a bot that reads from Zulip, you face the following challenges if you deploy your code on your own devices: • It can be difficult to keep the app running 24/7. • You may have latency issues connecting to the server. • If you want super-user permissions, you have to secure the API key. • Without integration to the Zulip server, the app may spin needlessly during upgrades. • If you’ve written a personal-use bot, it can be difficult to distribute code to your friends and have them be able to deploy it. • If you’ve written a general-use bot, it may be difficult to persuade your admin to give you a superuser account. We want to make it easier to deploy Zulip Readers on Zulip hardware. The following document talks about how we want to enable this from a code structuring standpoint: Writing contrib bots This document, on the other hand, is more about designing the Zulip backend system to support eventual deployment of reader apps on the Zulip server. Before we talk about server-side apps, we should consider an intermediate solution.

Non-Zulip dedicated hardware

There are some scenarios, mostly with general-purpose “serious” custom apps, where an app author might use the following development process: • Create a prototype and deploy it locally. • Publicize the app and deploy it on non-Zulip hardware. • Contribute the app to the Zulip distribution, so that admins can run it Zulip-side. To give a concrete example, let’s say that I work for a company that is building an issue tracker, and we want to offer Zulip support. I would start by writing a Zulip Reader that scans for the alert word @ticket on certain public Zulip streams, and part of that app would have logic to post to my company’s issue-tracking API. Once I’m confident in my prototype, I will probably run it on dedicated company hardware that might already have tight physical security, 24/7 IT monitoring, etc. But what if I don’t have this kind of infrastructure available to me? Typically what I will do instead is rent time on some kind of hosting service. Some hosting platforms are basically just remote Unix systems, but others are more oriented toward hosting web apps. Zulip’s current roadmap assumes that authors will likely gravitate toward web-based solutions (even if it’s just running a web server on their own Unix host in the cloud). Zulip intends to offer support for “outgoing webhooks.” The term “outgoing webhook” can be confusing, depending on your perspective, but it simply means that an HTTP request is outgoing from Zulip, so that it will hit a web endpoint that runs a third-party custom app. Zulip will allow the custom app author, probably with the help of a Zulip admin, to configure Zulip to send a subset of Zulip messages to the author’s web endpoint, and then the protocol for the custom app will to read the HTTP request and send some kind of HTTP response that optionally results in a message being written to Zulip. Meanwhile, the custom app can mutate the “world” as it sees fit.

254 Chapter 45. Custom Apps Zulip Documentation, Release 1.4.0

Zulip-side support for reader apps

Even for app authors that have access to dedicated hardware, there would be several advantages to running Zulip Readers under the same umbrella as the core Zulip system. • Your app will automatically inherit the uptime of the Zulip server itself (in terms of hardware availability). • There will be no network latency between the app and the server. • Securing apps to have superuser permissions will be less problematic. • Keeping your app in sync with Zulip upgrades could become more automatic. • Allowing multiple users in your realm to run their own copies of personal-use bots would be easier to administer. The only problem with the above bullets is that we haven’t built out any of that infrastructure yet. We do have pending PR #1393, which addresses some of the issues that might come up. In order to run apps inside the Zulip server, we basically need to solve the problems below. (One assumption is that we don’t run apps truly in-process.) • Contributions: We need a process for users to contribute code. • Configuration/Discovery: We need Zulip to be able to find which apps are allowed to run for a particular deployment. (The admin may choose to run only a subset of contributed apps.) • Queuing: We need to queue up events for readers, with some possible optimizations to scan for alert words during the in-process part of the call. • Drivers: We need a generic driver that can pull events off of a queue and hand them off to our specific reader objects. • Nannying: We need to launch readers with some kind of supervisord-like nannying. • Pausing: We probably need a way to pause/stop readers without stopping the Zulip main processes. (At first this may just be part of solving the nanny problem.) • Identity: We need to identify reader instances as specific Zulip users (non-owned bot, human-owned bot, or human). • Superusers: We may need some readers to have users with special privileges like being auto-subscribed to all public streams. • Read-only: We may need some readers at the other end of the spectrum to be highly locked down, e.g. enforce that they truly only have read access to Zulip messages. • UI: We will want to provide some UI features that give admins and/or regular users visibility into which server- side apps are running.

45.7. Deployment issues 255 Zulip Documentation, Release 1.4.0

256 Chapter 45. Custom Apps CHAPTER 46

Unread counts and the pointer

When you’re using Zulip and you reload, or narrow to a stream, how does Zulip decide where to place you? Conceptually, Zulip takes you to the place where you left off (e.g. the ﬁrst unread message), not the most recent messages, to facilitate reviewing all the discussions that happened while you were away from your computer. The scroll position is then set to keep that message in view and away from both the top and bottom of the visible section of messages. But there a lot of details around doing this right, and around counting unread messages. Here’s how Zulip currently decides which message to select, along with some notes on improvements we’d like to make to the model. First a bit of terminology: • “Narrowing” is the process of ﬁltering to a particular subset of the messages the user has access to. • The blue cursor box (the “pointer”) is around is called the “selected” message. Zulip ensures that the currently selected message is always in-view.

Pointer logic

Recipient bar: message you clicked

If you enter a narrow by clicking on a message group’s recipient bar (stream/topic or private message recipient list at the top of a group of messages), Zulip will select the message you clicked on. This provides a nice user experience where you get to see the stuff near what you clicked on, and in fact the message you clicked on stays at exactly the same scroll position in the window after the narrowing as it was at before.

Search or sidebar click: unread/recent matching narrow

If you instead narrow by clicking on something in the left sidebar or typing some terms into the search box, Zulip will instead select the ﬁrst unread message matching that narrow, or if there are none, the most recent messages matching that narrow. This provides the nice user experience of taking you to the start of the new stuff (with enough messages you’ev seen before still in view at the top to provide you with context), which is usually what you want.

257 Zulip Documentation, Release 1.4.0

(When ﬁnding the “ﬁrst unread message”, Zulip ignores unread messages in muted streams or in muted topics within non-muted streams.)

Unnarrow: previous sequence

When you unnarrow using e.g. the escape key, you will automatically be taken to the same message that was selected in the home view before you narrowed, unless in the narrow you read new messages, in which case you will be jumped forward to the ﬁrst unread and non-muted message in the home view (or the bottom of the feed if there is none). This makes for a nice experience reading threads via the home view in sequence.

New home view: “high watermark”

When you open a new browser window or tab to the home view (a.k.a. the interleaved view you get if you visit /), Zulip will select the furthest down that your cursor has ever reached in the home view. Because of the logic around unnarrowing in the last bullet, this is usually just before the ﬁrst unread message in the home view, but if you never go to the home view, or you leave messages unread on some streams in your home view, this can lag. We plan to change this to automatically advance the pointer in a way similar to the unnarrow logic.

Narrow in a new tab: closest to pointer

When you load a new browser tab or window to a narrowed view, Zulip will select the message closest to your pointer, which is what you would have got had you loaded the browser window to your home view and then clicked on the nearest message matching your narrow (which might have been offscreen). We plan to change this to match the Search/sidebar behavior.

Forced reload: state preservation

When the server forces a reload of a browser that’s otherwise caught up (which happens within 30 minutes when a new version of the server is deployed, usually at a type when the user isn’t looking at the browser), Zulip will preserve the state – what (if any) narrow the user was in, the selected message, and even exact scroll position! For more on the user experience philosophy guiding these decisions, see the architectural overview.

Unread count logic

How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range of ways people might use the product. The algorithm is as follows: • Any message which is selected or above a message which is selected is marked as read. So messages are marked as read as you scroll down the keyboard when the pointer passes over them. • If the whitespace at the very bottom of the feed is in view, all messages in view are marked as read. These two simple rules, combined with the pointer logic above, end up matching user expectations well for whether the product should treat them as having read a set of messages (or not).

258 Chapter 46. Unread counts and the pointer Zulip Documentation, Release 1.4.0

Testing and development

In a Zulip development environment, you can use manage.py mark_all_messages_unread to set every user’s pointer to 0 and all messages as unread, for convenience in testing unread count related logic. It can be useful to combine this with manage.py populate_db -n 3000 (which rebuilds the database with 3000 initial messages) to ensure a large number of messages are present.

46.3. Testing and development 259 Zulip Documentation, Release 1.4.0

260 Chapter 46. Unread counts and the pointer CHAPTER 47

Markdown implementation

Zulip has a special ﬂavor of Markdown, currently called ‘bugdown’ after Zulip’s original name of “humbug”. End users are using Bugdown within the client, not original Markdown. Zulip has two implementations of Bugdown. The backend implementation at zerver/lib/bugdown/ is based on Python-Markdown and is used to authoritatively render messages to HTML (and implements slow/expensive/complex features like querying the Twitter API to render tweets nicely). The frontend implementation is in JavaScript, based on marked.js( static/js/echo.js), and is used to preview and locally echo messages the moment the sender hits enter, without waiting for round trip from the server. Those frontend renderings are only shown to the sender of a message, and they are (ideally) identical to the backend rendering. The JavaScript markdown implementation has a function, markdown.contains_bugdown, that is used to check whether a message contains any syntax that needs to be rendered to HTML on the backend. If markdown. contains_bugdown returns true, the frontend simply won’t echo the message for the sender until it receives the rendered HTML from the backend. If there is a bug where markdown.contains_bugdown returns false incorrectly, the frontend will discover this when the backend returns the newly sent message, and will update the HTML based on the authoritative backend rendering (which would cause a change in the rendering that is visible only to the sender shortly after a message is sent). As a result, we try to make sure that markdown.contains_bugdown is always correct.

Testing

The Python-Markdown implementation is tested by zerver/tests/test_bugdown.py, and the marked.js implementation and markdown.contains_bugdown are tested by frontend_tests/node_tests/ markdown.js. A shared set of ﬁxed test data (“test ﬁxtures”) is present in zerver/fixtures/ bugdown-data.json, and is automatically used by both test suites; as a result, it the preferred place to add new tests for Zulip’s markdown system. If you’re going to manually test some changes in the frontend Markdown implementation, the easiest way to do this is as follows: 1. Login to your development server. 2. Stop your Zulip server with ctrl-C, leaving the browser open.

261 Zulip Documentation, Release 1.4.0

3. Compose and send the messages you’d like to test. They will be locally echoed using the frontend rendering. This procedure prevents any server-side rendering. If you don’t do this, backend will likely render the Markdown you’re testing and swap it in before you can see the frontend’s rendering.

Changing Zulip’s markdown processor

When changing Zulip’s markdown syntax, you need to update several places: • The backend markdown processor (zerver/lib/bugdown/__init__.py). • The frontend markdown processor (static/js/markdown.js and sometimes static/third/ marked/lib/marked.js), or markdown.contains_bugdown if your changes won’t be supported in the frontend processor. • If desired, the typeahead logic in static/js/composebox_typeahead.js. • The test suite, probably via adding entries to zerver/fixtures/bugdown-data.json. • The in-app markdown documentation (templates/zerver/markdown_help.html). • The list of changes to markdown at the end of this document. Important considerations for any changes are: • Security: A bug in the markdown processor can lead to XSS issues. For example, we should not insert unsani- tized HTML from a third-party web application into a Zulip message. • Uniqueness: We want to avoid users having a bad experience due to accidentally triggering markdown syntax or typeahead that isn’t related to what they are trying to express. • Performance: Zulip can render a lot of messages very quickly, and we’d like to keep it that way. New regular expressions similar to the ones already present are unlikely to be a problem, but we need to be thoughtful about expensive computations or third-party API requests. • Database: The backend markdown processor runs inside a Python thread (as part of how we implement timeouts for third-party API queries), and for that reason we currently should avoid making database queries inside the markdown processor. This is a technical implementation detail that could be changed with a few days of work, but is important detail to know about until we do that work. • Testing: Every new feature should have both positive and negative tests; they’re easy to write and give us the ﬂexibility to refactor frequently.

Zulip’s Markdown philosophy

Note that this discussion is based on a comparison with the original Markdown, not newer Markdown variants like CommonMark. Markdown is great for group chat for the same reason it’s been successful in products ranging from blogs to wikis to bug trackers: it’s close enough to how people try to express themselves when writing plain text (e.g. emails) that it helps more than getting in the way. The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot of wikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to fix the formatting after typing it the first time. While that’s basically fine when writing a blog, it gets annoying very fast in a chat product; even though you can edit messages to fix formatting mistakes, you don’t want to be doing that often. There are basically 2 types of error rates that are important for a product like Zulip:

262 Chapter 47. Markdown implementation Zulip Documentation, Release 1.4.0

• What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through your Markdown implementation, would you need to change the text of your email for it to render in a reasonable way? This is the “accidental Markdown syntax” problem, common with Markdown syntax like the italics syntax interacting with talking about char *s. • What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doing so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this ﬁgure substantially. Both of these are minor issues for most products using Markdown, but they are major problems in the instant messaging context, because one can’t edit a message that has already been sent and users are generally writing quickly. Zulip’s Markdown strategy is based on the principles of giving users the power they need to express complicated ideas in a chat context while minimizing those two error rates.

Zulip’s Changes to Markdown

Below, we document the changes that Zulip has against stock Python-Markdown; some of the features we modify / disable may already be non-standard.

Basic syntax

• Enable nl2br extension: this means one newline creates a line break (not paragraph break). • Allow only * syntax for italics, not _. This resolves an issue where people were using _ and hitting it by mistake too often. Asterisks surrounded by spaces won’t trigger italics, either (e.g. with stock Markdown You should use char * instead of void * there would produce undesired results). • Allow only ** syntax for bold, not __ (easy to hit by mistake if discussing Python __init__ or something). • Add ~~ syntax for strikethrough. • Disable special use of \ to escape other syntax. Rendering \\ as \ was hugely controversial, but having no escape syntax is also controversial. We may revisit this. For now you can always put things in code blocks.

Lists

• Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it. • Allow only * for bulleted lists, not + or - (previously created confusion with diff-style text sloppily not included in a code block). • Disable ordered list syntax: stock Markdown automatically renumbers, which can be really confusing when sending a numbered list across multiple messages.

Links

• Enable auto-linkiﬁcation, both for http://... and guessing at things like t.co/foo. • Force links to be absolute. [foo](google.com) will go to http://google.com, and not http:// zulip.com/google.com which is the default behavior. • Set target="_blank" and title=(the url) on every link tag so clicking always opens a new window. • Disable link-by-reference syntax, [foo][bar] ... [bar]: http://google.com. • Enable linking to other streams using #**streamName**.

47.4. Zulip’s Changes to Markdown 263 Zulip Documentation, Release 1.4.0

Code

• Enable fenced code block extension, with syntax highlighting. • Disable line-numbering within fenced code blocks – the

output confused our web client code.

Other

• Disable headings, both # foo and == foo == syntax: they don’t make much sense for chat messages. • Disabled images with ![]() (images from links are shown as an inline preview). • Allow embedding any avatar as a tiny (list bullet size) image. This is used primarily by version control integrations. • We added the ~~~ quote block quote syntax.

264 Chapter 47. Markdown implementation CHAPTER 48

Realms in Zulip

Zulip allows multiple realms to be hosted on a single instance. Realms are the Zulip codebases’s internal name for what we refer to in user documentation as an organization (the name “realm” comes from Kerberos).

Creating Realms

There are two main methods for creating realms. • Using unique link generator • Enabling open realm creation

Using Unique Link Generator

./manage.py generate_realm_creation_link

The above command will output a URL which can be used for creating a new realm and an administrator user for that realm. The link expires after the creation of the realm. The link also expires if not used within 7 days. The expiration period can be changed by modifying REALM_CREATION_LINK_VALIDITY_DAYS in settings.py.

Enabling Open Realm Creation

If you want anyone to be able to create new realms on your server, you can enable Open Realm Creation. This will add a Create new organization link to your Zulip homepage footer, and anyone can create a new realm by visiting this link (/create_realm). This feature is disabled by default in production instances, and can be enabled by setting OPEN_REALM_CREATION = True in settings.py.

265 Zulip Documentation, Release 1.4.0

Subdomains

A reasonable way to deploy a multi-realm Zulip server in production is to give each realm a unique subdomain. For example if the Zulip instance is hosted at zulip.example.com and the subdomain of your organization is acme you can use acme.zulip.example.com for accessing the organization. This subdomain feature is not enabled by default, since it requires additional DNS conﬁguration. You can enable subdomains by setting the value of REALMS_HAVE_SUBDOMAINS to True in settings.py. For subdomains to work properly, you also have to change your DNS records so that the subdomains point to your Zulip installation IP. An A record with host name value * pointing to your IP should do the job. Converting a production Zulip server from not using subdomains to using subdomains requires some setup work; contact the Zulip development community for help with this.

Working With Subdomains In Development Environment

By default, Linux does not provide a convenient way to use subdomains in your local development environment. To solve this problem, we use the zulipdev.com domain, which has a wildcard A record pointing to 127.0.0.1. When REALMS_HAVE_SUBDOMAINS = True in zproject/dev_settings.py, you should use zulipdev.com to connect to your Zulip development server instead of localhost. The default realm with the Shakespeare users has the subdomain zulip and can be accessed by visiting zulip.zulipdev.com. If you are behind a proxy server, this method won’t work. When you make a request to load zulipdev.com in your browser, the proxy server will try to get the page on your behalf. Since zulipdev.com points to 127.0.0.1 the proxy server is likely to give you a 503 error. The workaround is to disable your proxy for *.zulipdev.com. The DNS lookup should still work even if you disable proxy for *.zulipdev.com. If it doesn’t you can add zulipdev.com records in /etc/hosts ﬁle. The ﬁle should look something like this.

127.0.0.1 localhost

127.0.0.1 zulipdev.com

127.0.0.1 zulip.zulipdev.com

127.0.0.1 testsubdomain.zulipdev.com

These records are also useful if you want to e.g. run the casper tests when you are not connected to the Internet.

266 Chapter 48. Realms in Zulip CHAPTER 49

Static asset pipeline

This page documents additional information that may be useful when developing new features for Zulip that require front-end changes, especially those that involve adding new ﬁles. For a more general overview, see the new feature tutorial.

Primary build process

Most of the existing JS in Zulip is written in IIFE-wrapped modules, one per file in the static/js directory. We will over time migrate this to Typescript modules. In development mode files are loaded using webpack eval with sourcemaps. In production mode (and when creating a release tarball using tools/build-release-tarball), JavaScript files are concatenated and minified. We use the django pipeline extension to manage our static assets, webpack, and typescript.

Adding static ﬁles

To add a static file to the app (JavaScript, CSS, images, etc), first add it to the appropriate place under static/. • Third-party files that we haven’t patched should be installed via npm, so that it’s easy to upgrade them and third-party code doesn’t bloat the Zulip repository. You can then access them in webpack.assets.json via their paths under node_modules. You’ll want to add these to the package.json in the root of the repository, and then provision (to have npm download them) before continuing. Your commit should also update PROVISION_VERSION in version.py. When adding modules to package.json, please pin specific versions of them (don’t using carets ^, tildes ~, etc). We prefer fixed versions so that when the upstream providers release new versions with incompatible APIs, it can’t break Zulip. We update those versions periodically to ensure we’re running a recent version of third-party libraries. • Third-party files that we have patched should all go in static/third/. Tag the commit with “[third]” when adding or modifying a third-party package. Our goal is to the extent possible to eliminate patched third-party code from the project.

267 Zulip Documentation, Release 1.4.0

– Our own JavaScript lives under static/js; Typescript files live under static/ts; CSS lives under static/styles. Portico JavaScript (“portico” means for logged-out pages) lives under static/ js/portico. After you add a new JavaScript file, it needs to be specified in the entries dictionary defined in tools/webpack. assets.json to be included in the concatenated file; this will magically ensure it is available both in development and production. CSS should be added to the STYLESHEETS section of PIPELINE in zproject/settings.py. A few notes on doing this: • If you plan to only use the JS/CSS within the app proper, and not on the login page or other standalone pages, put it in the app bundle. • If you plan to use it in both, put it in the common bundle. • If it’s just used on a single standalone page (e.g. /stats), give it its own bundle. To load a bundle in the relevant Jinja2 template for that page, use render_bundle and stylesheet for JS and CSS, respectively. If you want to test minified files in development, look for the PIPELINE_ENABLED = line in zproject/ settings.py and set it to True – or just set DEBUG = False. Note that static/html/{400,5xx}.html will only render properly if minification is enabled, since they, by nature, hardcode the path static/min/portico.css.

How it works in production

You can learn a lot from reading about django-pipeline, but a few useful notes are: • Zulip installs static assets in production in /home/zulip/prod-static. When a new version is deployed, before the server is restarted, files are copied into that directory. • We use the VFL (Versioned File Layout) strategy, where each file in the codebase (e.g. favicon.ico) gets a new name (e.g. favicon.c55d45ae8c58.ico) that contains a hash in it. Each deployment, has a manifest file (e.g. /home/zulip/deployments/current/staticfiles.json) that maps codebase filenames to serving filenames for that deployment. The benefit of this VFL approach is that all the static files for past deployments can coexist, which in turn eliminates most classes of race condition bugs where browser windows opened just before a deployment can’t find their static assets. It also is necessary for any incremental rollout strategy where different clients get different versions of the site. • Some paths for files (e.g. emoji) are stored in the rendered_content of past messages, and thus cannot be removed without breaking the rendering of old messages (or doing a mass-rerender of old messages).

Webpack/CommonJS/ES6/Typescript modules

New JS written for Zulip can be written as Typescript or if a more incremental migration is required, CommonJS modules (bundled using webpack, though this will be taken care of automatically whenever run-dev.py is running). (CommonJS is the same module format that Node uses, so see the Node documentation for more information on the syntax.) All JavaScript we provide will eventually be migrated to Typescript, which will make refactoring the frontend code easier and allow static analyzers to reason about our code more easily. Declare entry points in webpack.assets.json. Any modules you add will need to be required or imported from this ﬁle (or one of its dependencies) in order to be included in the script bundle.

268 Chapter 49. Static asset pipeline CHAPTER 50

Schema Migrations

Zulip uses the standard Django system for doing schema migrations. There is some example usage in the new feature tutorial. This page documents some important issues related to writing schema migrations. • Large tables: For large tables like Message and UserMessage, you want to take precautions when adding columns to the table, performing data backfills, or building indexes. We have a zerver/lib/migrate.py library to help with adding columns and backfilling data. For building indexes on these tables, we should do this using SQL with postgres’s CONCURRENTLY keyword. • Numbering conflicts across branches: If you’ve done your schema change in a branch, and meanwhile another schema change has taken place, Django will now have two migrations with the same number. To fix this, you can either run ./tools/renumber-migrations which renumbers your migration(s) and fixes up the “dependencies” entries in your migration(s), and then rewrite your git history as needed, or you can do it manually. There is a tutorial here that walks you though that process. • Atomicity. By default, each Django migration is run atomically inside a transaction. This can be problematic if one wants to do something in a migration that touches a lot of data and would best be done in batches of e.g. 1000 objects (e.g. a Message or UserMessage table change). There is a new Django feature added in Django 1.10 that makes it possible to add atomic=False at the top of a Migration class and thus not have the entire migration in a transaction. This should make it possible to use the batch update tools in zerver/lib/migrate.py (originally written to work with South) for doing larger database migrations. • Accessing code and models in RunPython migrations. When writing a migration that includes custom python code (aka RunPython), you almost never want to import code from zerver or anywhere else in the codebase. If you imagine the process of upgrading a Zulip server, it goes as follows: first a server admin checks out a recent version of the code, and then runs any migrations that were added between the last time they upgraded and the current check out. Note that for each migration, this means the migration is run using the code in the server admin’s check out, and not the code that was there at the time the migration was written. This can be a difference of thousands of commits for installations that are only upgraded occasionally. It is hard to reason about the effect of a code change on a migration that imported it so long ago, so we recommend just copying any code you’re tempted to import into the migration file directly, and have a linter rule enforcing this. There is one special case where this doesn’t work: you can’t copy the definition of a model (like Realm) into a migration, and you can’t import it from zerver.models for the reasons above. In this situation you should

269 Zulip Documentation, Release 1.4.0

use Django’s apps.get_model to get access to a model as it is at the time of a migration. Note that this will work for doing something like Realm.objects.filter(..), but shouldn’t be used for accessing Realm.subdomain or anything not related to the Django ORM. • Making large migrations work. Major migrations should have a few properties: – Unit tests. You’ll want to carefully test these, so you might as well write some unit tests to verify the migration works correctly, rather than doing everything by hand. This often saves a lot of time in re- testing the migration process as we make adjustments to the plan. – Run in batches. Updating more than 1K-10K rows (depending on type) in a single transaction can lock up a database. It’s best to do lots of small batches, potentially with a brief sleep in between, so that we don’t block other operations from finishing. – Rerunnability/idempotency. Good migrations are ones where if operational concerns (e.g. it taking down the Zulip server for users) interfere with it finishing, it’s easy to restart the migration without doing a bunch of hand investigation. Ideally, the migration can even continue where it left off, without needing to redo work. – Multi-step migrations. For really big migrations, one wants to split the transition into into several commits that are each individually correct, and can each be deployed independently: 1. First, do a migration to add the new column to the Message table and start writing to that column (but don’t use it for anything) 2. Second, do a migration to copy values from the old column to the new column, to ensure that the two data stores agree. 3. Third, a commit that stops writing to the old field. 4. Any cleanup work, e.g. if the old field were a column, we’d do a migration to remove it entirely here. This multi-step process is how most migrations on large database tables are done in large-scale systems, since it ensures that the system can continue running happily during the migration.

270 Chapter 50. Schema Migrations CHAPTER 51

HTML and CSS

Zulip CSS organization

The Zulip application’s CSS can be found in the static/styles/ directory. Zulip uses Bootstrap as its main third-party CSS library. Zulip currently does not use any CSS preprocessors, and is organized into several files. For most pages, the CSS is combined into a single CSS file by the static asset pipeline, controlled by the PIPELINE_CSS code in zproject/ settings.py. The CSS files are: • portico.css - Main CSS for logged-out pages • pygments.css - CSS for Python syntax highlighting • activity.css - CSS for the activity app • fonts.css - Fonts for text in the Zulip app • static/third/thirdparty-fonts.css - Font Awesome (used for icons) The CSS for the Zulip web application UI is primarily here: • settings.css - CSS for the Zulip settings (including organization settings) pages • zulip.css - CSS for the rest of the Zulip logged-in app • media.css - CSS for media queries (particularly related to screen width) We are in the process of splitting zulip.css into several more files; help with that project is very welcome!

Editing Zulip CSS

If you aren’t experienced with doing web development and want to make CSS changes, we recommend reading the excellent Chrome web inspector guide on editing HTML/CSS, especially the section on CSS to learn about all the great

271 Zulip Documentation, Release 1.4.0 tools that you can use to modify and test changes to CSS interactively in-browser (without even having the reload the page!).

CSS Style guidelines

Avoid duplicated code

Without care, it’s easy for a web application to end up with thousands of lines of duplicated CSS code, which can make it very difﬁcult to understand the current styling or modify it. We would very much like to avoid such a fate. So please make an effort to reuse existing styling, clean up now-unused CSS, etc., to keep things maintainable.

Be consistent with existing similar UI

Ideally, do this by reusing existing CSS declarations, so that any improvements we make to the styling can improve all similar UI elements.

Use clear, unique names for classes and object IDs

This makes it much easier to read the code and use git grep to ﬁnd where a particular class is used.

Validating CSS

When changing any part of the Zulip CSS, it’s important to check that the new CSS looks good at a wide range of screen widths, from very wide screen (e.g. 1920px) all the way down to narrow phone screens (e.g. 480px). For complex changes, it’s deﬁnitely worth testing in a few different browsers to make sure things look the same.

272 Chapter 51. HTML and CSS CHAPTER 52

URL hashes and deep linking

Hashchange

The Zulip web application has a nice system of hash (#) URLs that can be used to deep-link into the application and allow the browser’s “back” functionality to let the user navigate between parts of the UI. Some examples are: • /#settings/your-bots: Bots section of the settings overlay. • /#streams: Streams overlay, where the user manages streams (subscription etc.) • /#streams/11/announce: Streams overlay with stream ID 11 (called “announce”) selected. • /#narrow/stream/android/subject/fun: Message feed showing stream “android” and topic “fun”. The main module in the frontend that manages this all is static/js/hashchange.js (plus hash_util.js for all the parsing code), which is unfortunately one of our thorniest modules. Part of the reason that it’s thorny is that it needs to support a lot of different ﬂows: • The user clicking on an in-app link, which in turn opens an overlay. For example the streams overlay opens when the user clicks the small cog symbol on the left sidebar, which is in fact a link to /#streams. This makes it easy to have simple links around the app without custom click handlers for each one. • The user uses the “back” button in their browser (basically equivalent to the previous one, as a link out of the browser history will be visited). • The user clicking some in-app click handler (e.g. “Stream settings” for an invidual stream), that potentially does several UI-manipulating things including e.g. loading the streams overlay, and needs to update the hash without re-triggering the open animation (etc.). • Within an overlay like the streams overlay, the user clicks to another part of the overlay, which should update the hash but not re-trigger loading the overlay (which would result in a confusing animation experience). • The user is in a part of the webapp, and reloads their browser window. Ideally the reloaded browser window should return them to their original state. • A server-initiated browser reload (done after a new version is deployed, or when a user comes back after being idle for a while, see notes below), where we try to preserve extra state (e.g. content of compose box, scroll position within a narrow) using the /#reload hash preﬁx.

273 Zulip Documentation, Release 1.4.0

When making changes to the hashchange system, it is essential to test all of these ﬂows, since we don’t have great automated tests for all of this (would be a good project to add them to the Casper suite) and there’s enough complexity that it’s easy to accidentally break something. Here’s some notes on how we handle these cases: • hashchange.hashchanged is the function used to handle the hash, when it’s changed by the browser (e.g. by clicking on a link to a hash or using the back button). • hashchange.should_ignore is the function hashchange.hashchanged calls to make it possible for clicking on links within a given overlay to just be managed by code within that overlay, without reloading the overlay. It primarily checks whether the “main hash” (i.e. the ﬁrst piece like settings for #settings/ your-account) is an overlay. • hashchange.do_hashchange is what is called when the user reloads the browser. If the hash is nonempty, it ensures the relevant overlay is opened or the user is narrowed as part of the page load process. It is also is called by hashchange.hashchanged when the hash changes outside the should_ignore boundaries, since the logic for that case is identical. • reload.preserve_state is called when a server-initiated browser reload happens, and encodes a bunch of data like the current scroll position into the hash. • reload.initialize handles restoring the preserved state after a reload where the hash starts with / #reload.

Server-initiated reloads

There are a few circumstances when the Zulip browser window needs to reload itself: • If the browser has been ofﬂine for more than 10 minutes, the browser’s event queue will have been garbage- collected by the server, meaning the browser can no longer get real-time updates altogether. In this case, the browser auto-reloads immediately in order to reconnect. We have coded an unsuspend trigger (based on some clever time logic) that ensures we check immediately when a client unsuspends; grep for unsuspend to see the code. • If a new version of the server has been deployed, we want to reload the browser so that it will start running the latest code. However, we don’t want server deploys to be disruptive. So, the backend preserves user-side event queues (etc.) and just pushes a special restart event to all clients. That event causes the browser to start looking for a good time to reload, based on when the user is idle (ideally, we’d reload when they’re not looking and restore state so that the user never knew it happened!). The logic for doing this is in static/js/ reload.js; but regardless we’ll reload within 30 minutes unconditionally. An important detail in server-initiated reloads is that we desynchronize when browsers start attempting them randomly, in order to avoid a thundering herd situation bringing down the server.

All reloads

In addition to saving state as described above when reloading the browser, Zulip also does a few bookkeeping things on page reload (like cleaning up its event queue, and saving any text in an open compose box as a draft).

274 Chapter 52. URL hashes and deep linking CHAPTER 53

Emoji

Emoji seem like a simple idea, but there’s actually a ton of complexity that goes into an effective emoji implementation. This document discusses a number of these issues. Currently, Zulip uses the Noto (Android) emoji set. We are considering adding additional emoji sets as options.

Emoji codes

The Unicode standard has various ranges of characters set aside for emoji. So you can put emoji in your terminal using actual unicode characters like and . If you paste those into Zulip, Zulip will render them as the corresponding emoji image. However, the Unicode committee did not standardize on a set of human-readable names for emoji. So, for example, when using the popular : based style for entering emoji from the keyboard, we have to decide whether to use :angry: or :angry_face: to represent an angry face. Different products use different approaches, but for purposes like emoji pickers or autocomplete, you deﬁnitely want to pick exactly one of these names, since otherwise users will always be seeing duplicates of a given emoji next to each other. Picking which emoji name to use is surprisingly complicated! Zulip has a nice library, tools/setup/emoji/ emoji_setup_utils.py, which we use to make sense of these decisions with a relatively small list of hand-coded exceptions.

Tooling

Zulip has a tool, tools/setup/emoji/build_emoji, that combines emoji-map.json (an open source mapping of emoji names to short codes that is fairly liberal about including duplicates) and the Noto emoji font to extract the emoji that we use in the product. This tool generates a set of ﬁles under static/generated/emoji (or really, it generates the /srv/ zulip-emoji-cache//emoji tree, and static/generated/emoji is a symlink to that tree; we do this in order to cache old versions to make provisioning and production deployments super fast in the common case that we haven’t changed the emoji tooling).

275 Zulip Documentation, Release 1.4.0

The emoji tree generated by this process contains several import elements: • emoji_codes.js: A set of mappings used by the Zulip frontend to understand what unicode emoji exist and what their shortnames are, used for autocomplete, emoji pickers, etc. This has been deduplicated using the logic in tools/setup/emoji/emoji_setup_utils.py to generally only have :angry: and not also :angry_face:, since having both is ugly and pointless for purposes like autocomplete and emoji pickers. • images/emoji/unicode/*.png: A farm of emoji • images/emoji/*.png: A farm of symlinks from emoji names to the images/emoji/unicode/ tree. This is used to serve individual emoji images, as well as for the backend markdown processor to know which emoji names exist and what unicode emoji / images they map to. In this tree, we currently include all of the emoji in emoji-map.json; this means that if you send :angry_face:, it won’t autocomplete, but will still work (but not in previews). • Some CSS and a PNG for an emoji spritesheet, used in Zulip for emoji pickers where we would otherwise need to download over 1000 of individual emoji images (which would cause a browser performance problem). We will likely eventually replace the images/emoji/unicode/ tree with using this spritesheet as well.

276 Chapter 53. Emoji CHAPTER 54

Full-text search

Zulip supports full-text search, which can be combined arbitrarily with Zulip’s full suite of narrowing operators. By default, it only supports English text, but there is an experimental PGroonga integration that provides full-text search for all languages. The user interface and feature set for Zulip’s full-text search is documented in the “Search operators” documentation section in the Zulip app’s gear menu.

The default full-text search implementation

Zulip’s uses PostgreSQL’s built-in full-text search feature, with a custom set of English stop words to improve the quality of the search results. We use a small extension, tsearch_extras, for highlighting of the matching words. There is some discussion of removing this extension, at least as an option, so that Zulip can be used with database-as-a-service platforms. In order to optimize the performance of delivering messages, the full-text search index is updated for newly sent messages in the background, after the message has been delivered. This background updating is done by puppet/ zulip/files/postgresql/process_fts_updates, which is usually deployed on the database server, but could be deployed on an application server instead.

An optional full-text search implementation

Zulip now supports using PGroonga for full-text search. PGroonga is a PostgreSQL extension that provides full-text search feature. PostgreSQL’s built-in full-text search feature supports only one language at a time (in Zulip’s case, English). PGroonga supports all languages simultaneously, including Japanese, Chinese and so on, all at once. We expect to migrate Zulip’s full-text search to only support PGroonga once we have tested this new extension fully. The following processes should be executed as the root user. Run:

sudo-i

277 Zulip Documentation, Release 1.4.0

How to enable full-text search across all languages

This section describes how to enable using PGroonga to back the full-text search feature. To install PGroonga, add pgroonga = enabled in the [machine] section in /etc/zulip/zulip.conf:

[machine] ... pgroonga= enabled

And then run as root:

/home/zulip/deployments/current/scripts/zulip-puppet-apply

Then, add USING_PGROONGA = true in /etc/zulip/settings.py:

USING_PGROONGA= True

And apply the PGroonga migrations: cd/srv/zulip ./manage.py migrate pgroonga

Note that the migration may take a long time, and you can’t send new messages until the migration ﬁnishes. Once the migrations are complete, restart Zulip: su zulip-c/home/zulip/deployments/current/scripts/restart-server

Now, you can use full-text search across all languages.

How to disable full-text search across all languages

This section describes how to disable full-text search feature based on PGroonga. If you want to fully remove PGroonga, ﬁrst you need to remove the PGroonga column (as above, this will take a long time and no messages can be sent while it is running). If you intend to re-enable PGroonga later, you can skip this step (at the cost of your Message table being slightly larger than it would be otherwise).

/home/zulip/deployments/current/manage.py migrate pgroonga zero

Then, set USING_PGROONGA = False in /etc/zulip/settings.py:

USING_PGROONGA= False

And, restart Zulip: su zulip-c/home/zulip/deployments/current/scripts/restart-server

Now, full-text search feature based on PGroonga is disabled. If you’d like, you can also remove the pgroonga = enabled line in /etc/zulip/zulip.conf and uninstall the pgroonga packages.

278 Chapter 54. Full-text search CHAPTER 55

Analytics

Zulip has a cool analytics system for tracking various useful statistics that currently power the /stats page, and over time will power other features, like showing usage statistics for the various streams. It is designed around the following goals: • Minimal impact on scalability and service complexity. • Well-tested so that we can count on the results being correct. • Efﬁcient to query so that we can display data in-app (e.g. on the streams page) with minimum impact on the overall performance of those pages. • Storage size smaller than the size of the main Message/UserMessage database tables, so that we can store the data in the main postgres database rather than using a specialized database platform. There are a few important things you need to understand in order to effectively modify the system.

Analytics backend overview

There are three main components: • models: The UserCount, StreamCount, RealmCount, and InstallationCount tables (analytics/models.py) collect and store time series data. • stat deﬁnitions: The CountStat objects in the COUNT_STATS dictionary (analytics/lib/counts.py) deﬁne the set of stats Zulip collects. • accounting: The FillState table (analytics/models.py) keeps track of what has been collected for which CountStats. The next several sections will dive into the details of these components.

279 Zulip Documentation, Release 1.4.0

The *Count database tables

The Zulip analytics system is built around collecting time series data in a set of database tables. Each of these tables has the following fields: • property: A human readable string uniquely identifying a CountStat object. Example: “ac- tive_users:is_bot:hour” or “messages_sent:client:day”. • subgroup: Almost all CountStats are further sliced by subgroup. For “active_users:is_bot:day”, this column will be False for measurements of humans, and True for measurements of bots. For “messages_sent:client:day”, this column is the client_id of the client under consideration. • end_time: A datetime indicating the end of a time interval. It will be on an hour (or UTC day) boundary for stats collected at hourly (or daily) frequency. The time interval is determined by the CountStat. • various “id” fields: Foreign keys into Realm, UserProfile, Stream, or nothing. E.g. the RealmCount table has a foreign key into Realm. • value: The integer counts. For “active_users:is_bot:hour” in the RealmCount table, this is the number of active humans or bots (depending on subgroup) in a particular realm at a particular end_time. For “mes- sages_sent:client:day” in the UserCount table, this is the number of messages sent by a particular user, from a particular client, on the day ending at end_time. • anomaly: Currently unused, but a key into the Anomaly table allowing someone to indicate a data irregularity. There are four tables: UserCount, StreamCount, RealmCount, and InstallationCount. Every CountStat is initially collected into UserCount, StreamCount, or RealmCount. Every stat in UserCount and StreamCount is aggregated into RealmCount, and then all stats are aggregated from RealmCount into InstallationCount. So for example, “mes- sages_sent:client:day” has rows in UserCount corresponding to (user, end_time, client) triples. These are summed to rows in RealmCount corresponding to triples of (realm, end_time, client). And then these are summed to rows in InstallationCount with totals for pairs of (end_time, client). Note: In most cases, we do not store rows with value 0. See Performance Strategy below.

CountStats

CountStats declare what analytics data should be generated and stored. The CountStat class deﬁnition and instances live in analytics/lib/counts.py. These declarations specify at a high level which tables should be populated by the system and with what data.

The FillState table

The default Zulip production conﬁguration runs a cron job once an hour that updates the *Count tables for each of the CountStats in the COUNT_STATS dictionary. The FillState table simply keeps track of the last end_time that we successfully updated each stat. It also enables the analytics system to recover from errors (by retrying) and to monitor that the cron job is running and running to completion.

Performance strategy

An important consideration with any analytics system is performance, since it’s easy to end up processing a huge amount of data inefﬁciently and needing a system like Hadoop to manage it. For the built-in analytics in Zulip, we’ve

280 Chapter 55. Analytics Zulip Documentation, Release 1.4.0 designed something lightweight and fast that can be available on any Zulip server without any extra dependencies through the carefully designed set of tables in Postgres. This requires some care to avoid making the analytics tables larger than the rest of the Zulip database or adding a ton of computational load, but with careful design, we can make the analytics system very low cost to operate. Also, note that a Zulip application database has 2 huge tables: Message and UserMessage, and everything else is small and thus not performance or space-sensitive, so it’s important to optimize how many expensive queries we do against those 2 tables. There are a few important principles that we use to make the system efﬁcient: • Not repeating work to keep things up to date (via FillState) • Storing data in the *Count tables to avoid our endpoints hitting the core Message/UserMessage tables is key, because some queries could take minutes to calculate. This allows any expensive operations to run ofﬂine, and then the endpoints to server data to users can be fast. • Doing expensive operations inside the database, rather than fetching data to Python and then sending it back to the database (which can be far slower if there’s a lot of data involved). The Django ORM currently doesn’t support the “insert into .. select” type SQL query that’s needed for this, which is why we use raw database queries (which we usually avoid in Zulip) rather than the ORM. • Aggregating where possible to avoid unnecessary queries against the Message and UserMessage tables. E.g. rather than querying the Message table both to generate sent message counts for each realm and again for each user, we just query for each user, and then add up the numbers for the users to get the totals for the realm. • Not storing rows when the value is 0. An hourly user stat would otherwise collect 24 * 365 * roughly .5MB per db row = 4GB of data per user per year, most of whose values are 0. A related note is to be cautious about adding queries that are typically non-0 instead of being typically 0.

Backend Testing

There are a few types of automated tests that are important for this sort of system: • Most important: Tests for the code path that actually populates data into the analytics tables. These are most important, because it can be very expensive to ﬁx bugs in the logic that generates these tables (one basically needs to regenerate all of history for those tables), and these bugs are hard to discover. It’s worth taking the time to think about interesting corner cases and add them to the test suite. • Tests for the backend views code logic for extracting data from the database and serving it to clients. For manual backend testing, it sometimes can be valuable to use ./manage.py dbshell to inspect the tables manually to check that things look right; but usually anything you feel the need to check manually, you should add some sort of assertion for to the backend analytics tests, to make sure it stays that way as we refactor.

LoggingCountStats

The system discussed above is designed primarily around the technical problem of showing useful analytics about things where the raw data is already stored in the database (e.g. Message, UserMessage). This is great because we can always backﬁll that data to the beginning of time, but of course sometimes one wants to do analytics on things that aren’t worth storing every data point for (e.g. activity data, request performance statistics, etc.). There is currently a reference implementation of a “LoggingCountStat” that shows how to handle such a situation.

55.6. Backend Testing 281 Zulip Documentation, Release 1.4.0

Analytics UI development and testing

Setup and Testing

The main testing approach for the /stats page UI is manual testing. For UI testing, you want a comprehensive initial data set; you can use manage.py populate_analytics_db to set up, login as the shylock user, and then go to /stats.

Adding or editing /stats graphs

The relevant files are: • analytics/views.py: All chart data requests from the /stats page call get_chart_data in this file. The bottom half of this file (with all the raw sql queries) is for a different page (/activity), not related to /stats. • static/js/stats/stats.js: The JavaScript and Plotly code. • templates/analytics/stats.html • static/styles/stats.css and static/styles/portico.css: We are in the process of re-styling this page to use in-app css instead of portico css, but there is currently still a lot of portico influence. • analytics/urls.py: Has the URL routes; it’s unlikely you will have to modify this, including for adding a new graph. Most of the code is self-explanatory, and for adding say a new graph, the answer to most questions is to copy what the other graphs do. It is easy when writing this sort of code to have a lot of semi-repeated code blocks (especially in stats.js); it’s good to do what you can to reduce this. Tips and tricks: • Use $.get to fetch data from the backend. You can grep through stats.js to find examples of this. • The Plotly documentation is at https://plot.ly/javascript/ (check out the full reference, event reference, and function reference). The documentation pages seem to work better in Chrome than in Firefox, though this hasn’t been extensively verified. • Unless a graph has a ton of data, it is typically better to just redraw it when something changes (e.g. in the various aggregation click handlers) rather than to use retrace or relayout or do other complicated things. Performance on the /stats page is nice but not critical, and we’ve run into a lot of small bugs when trying to use Plotly’s retrace/relayout. • There is a way to access raw d3 functionality through Plotly, though it isn’t documented well. • ‘paper’ as a Plotly option refers to the bounding box of the graph (or something related to that). • You can’t right click and inspect the elements of a Plotly graph (e.g. the bars in a bar graph) in your browser, since there is an interaction layer on top of it. But if you hunt around the document tree you should be able to find it.

/activity page

• There’s a somewhat less developed /activity page, for server administrators, showing data on all the realms on a server. To access it, you need to have the is_staff bit set on your UserProﬁle object. You can set it using manage.py shell and editing the UserProﬁle object directly. A great future project is to clean up that page’s data sources, and make this a documented interface.

282 Chapter 55. Analytics CHAPTER 56

Translating Zulip

To make Zulip even better for users around the world, the Zulip UI is being translated into a number of major languages, including Spanish, German, French, Chinese, Russian, and Japanese, with varying levels of progress. If you speak a language other than English, your help with translating Zulip would be greatly appreciated! If you’re interested in contributing translations to Zulip, please join #translation in the Zulip development community server, and say hello. And please join the Zulip project on Transifex and ask to join any languages you’d like to contribute to (or add new ones). Transifex’s notiﬁcation system sometimes fails to notify the maintainers when you ask to join a project, so please send a quick email to [email protected] when you request to join the project or add a language so that we can be sure to accept your request to contribute. Zulip has full support for Unicode, so you can already use your preferred language everywhere in Zulip.

Translation style guides

We are building a collection of translation style guides for Zulip, giving guidance on how Zulip should be translated into speciﬁc languages (e.g. what word to translate words like “home” to): • Chinese • German • Polish • Russian • Spanish A great ﬁrst step when getting started translating Zulip into a new language is to write a style guide, since it greatly increases the ability of future translators to translate in a way that’s consistent with what your work.

Capitalization

We expect that all the English translatable strings in Zulip are properly capitalized in a way consistent with how Zulip does capitalization in general. This means that:

283 Zulip Documentation, Release 1.4.0

• The ﬁrst letter of a sentence or phrase should be capitalized. – Correct: “Manage streams” – Incorrect: “Manage Streams” • All proper nouns should be capitalized. – Correct: “This is Zulip” – Incorrect: “This is zulip” • All common words like URL, HTTP, etc. should be written in their standard forms. – Correct: “URL” – Incorrect: “Url” We have a tool to check for the correct capitalization of the translatable strings; this tool will not allow the Travis builds to pass in case of errors. You can use our capitalization checker to validate your code by running ./tool/check-capitalization. If you think that you have a case where our capitalization checker tool wrongly categorizes a string as not capitalized, you can add an exception in the tools.lib.capitalization. IGNORED_PHRASES list to make the tool pass. Please, stick to these while translating, and feel free to point out any strings that should be improved or ﬁxed.

Translation process

The end-to-end process to get the translations working is as follows. Please note that you don’t need to do this if you’re translating; this is only to describe how the whole process is. If you’re interested in translating, you should check out the translators’ workflow. 1. The strings are marked for translation (see sections for backend and frontend translations for details on this). 2. Translation resource files are created using the ./manage.py makemessages command. This command will create, for each language, a resource file called translations.json for the frontend strings and django.po for the backend strings. The makemessages command is idempotent in that: • It will only delete singular keys in the resource file when they are no longer used in Zulip code. • It will only delete plural keys (see below for the documentation on plural translations) when the corresponding singular key is absent. • It will not override the value of a singular key if that value contains a translated text. 3. Those resource files are uploaded to Transifex by a maintainer using the tx push -s -a command. 4. Translators translate the strings in Transifex. 5. The translations are downloaded back into the codebase by a maintainer, using tx pull -a -f --mode=developer (since we don’t use the Transifex review flow).

Translators’ workﬂow

These are the steps you should follow if you want to help to translate Zulip: 1. Join us on Zulip and ask for access to the organization, as described at the beginning. 2. Make sure you have access to Zulip’s dashboard in Transifex.

284 Chapter 56. Translating Zulip Zulip Documentation, Release 1.4.0

3. Ask a maintainer to update the strings. 4. Translate the strings for your language in Transifex. Some useful tips for your translating journey: • Follow your language’s translation guide. Keeping it open in a tab while translating is very handy. If one doesn’t exist one, write one as you go; they’re easiest to write as you go along and will help any future translators a lot. • Don’t translate variables or code (usually preceded by a %, or inside HTML tags <...>). • When in doubt, ask for context in #translation in the Zulip development community server. • If there are multiple possible translations for a term, search for it in the Concordance tool (the button with a magnet in the top right corner). It will show if anyone translated that term before, so we can achieve good consistency with all the translations, no matter who makes them. • Pay attention to capital letters and punctuation. Details make the difference! • Take advantage of the hotkeys the Transifex Web Editor provides, such as Tab for saving and going to the next string.

Testing translations

This section assumes you have a Zulip development environment setup. First of all, download the updated resource files from Transifex using the tx pull -a --mode=developer command (it will require some initial setup). This command will download the resource files from Transifex and replace your local resource files with them. Then, make sure that you have compiled the translation strings using ./manage.py compilemessages. Django figures out the effective language by going through the following steps: 1. It looks for the language code in the url (e.g. /de/). 2. It looks for the LANGUAGE_SESSION_KEY key in the current user’s session. 3. It looks for the cookie named ‘django_language’. You can set a different name through the LANGUAGE_COOKIE_NAME setting. 4. It looks for the Accept-Language HTTP header in the HTTP request. Normally your browser will take care of this. The easiest way to test translations is through the i18n URLs, e.g., if you have German translations available, you can access the German version of a page by going to /de/path_to_page in your browser. To test translations using other methods you will need an HTTP client library like requests, cURL or urllib. Here is some sample code to test Accept-Language header using Python and requests:

import requests headers={"Accept-Language":"de"} response= requests.get("http://localhost:9991/login/", headers=headers) print(response.content)

56.4. Testing translations 285 Zulip Documentation, Release 1.4.0

Setting the default language in Zulip

Zulip allows you to set the default language through the settings page, in the ‘Display settings’ section. The URL will be /#settings/display-settings on your realm. Organizations can set the default language for new users in their organization on the /#organization page.

Translation resource ﬁles

All the translation magic happens through resource files which hold the translated text. Backend resource files are located at static/locale//LC_MESSAGES/django.po, while frontend resource files are located at static/locale//translations.json. These files are uploaded to Transifex, where they can be translated.

Backend translations

All user-facing text in the Zulip UI should be generated by an HTML template so that it can be translated. Zulip uses two types of templates: backend templates (powered by the Jinja2 template engine, though the original Django template engine is still supported) and frontend templates (powered by Handlebars). To mark a string for translation in the Jinja2 and Django template engines, you can use the _() function in the templates like this:

If a string contains both a literal string component and variables, you can use a block translation, which makes use of placeholders to help translators to translate an entire sentence. To translate a block, Jinja2 uses the trans tag while Django uses the blocktrans tag. So rather than writing something ugly and confusing for translators like this:

# Don't do this! {{ _("This string will have") }} {{ value }} {{ _("inside") }}

You can instead use:

# Jinja2 style {% trans%}This string will have {{ value }} inside.{% endtrans%} # Django style {% blocktrans%}This string will have {{ value }} inside.{% endblocktrans%}

Zulip expects all the error messages to be translatable as well. To ensure this, the error message passed to json_error and JsonableError should always be a literal string enclosed by _() function, e.g.:

json_error(_('English Text')) JsonableError(_('English Text'))

To ensure we always internationalize our JSON errors messages, the Zulip linter (tools/lint) checks for correct usage.

286 Chapter 56. Translating Zulip Zulip Documentation, Release 1.4.0

Frontend translations

Zulip uses the i18next library for frontend translations. There are two types of files in Zulip frontend which can hold translatable strings: JavaScript code files and Handlebar templates. To mark a string translatable in JavaScript files, pass it to the i18n.t function. i18n.t('English Text', context); i18n.t('English text with a __variable__',{'variable':'Variable value'});

Note: In the second example above, instead of enclosing the variable with handlebars, {{ }}, we enclose it with __ because we need to differentiate the variable from the Handlebar tags. The symbol which is used to enclose the variables can be changed in /static/js/src/main.js. i18next also supports plural translations. To support plurals make sure your resource ﬁle contains the related keys:

{ "en":{ "translation":{ "key":"item", "key_plural":"items", "keyWithCount":"__count__ item", "keyWithCount_plural":"__count__ items" } } }

With this resource you can show plurals like this: i18n.t('key', {count:0});// output:'items' i18n.t('key', {count:1});// output:'item' i18n.t('key', {count:5});// output:'items' i18n.t('key', {count: 100});// output:'items' i18n.t('keyWithCount', {count:0});// output:'0 items' i18n.t('keyWithCount', {count:1});// output:'1 item' i18n.t('keyWithCount', {count:5});// output:'5 items' i18n.t('keyWithCount', {count: 100});// output:'100 items'

For further reading on plurals, read the ofﬁcial documentation. To mark the strings as translatable in the Handlebar templates, Zulip registers two Handlebar helpers. The syntax for simple strings is:

The syntax for block strings or strings containing variables is:

{{#tr context}} Block of English text. {{/tr}} var context={'variable':'variable value'}; {{#tr context}} Block of English text with a __variable__. {{/tr}}

The rules for plurals are same as for JavaScript ﬁles. You just have to declare the appropriate keys in the resource ﬁle and then include the count in the context.

56.8. Frontend translations 287 Zulip Documentation, Release 1.4.0

Transifex conﬁg

The conﬁg ﬁle that maps the resources from Zulip to Transifex is located at .tx/config.

Transifex CLI setup

In order to be able to run tx pull (and tx push as well, if you’re a maintainer), you have to specify your Transifex credentials in a config file, located at ~/.transifexrc. You can find details on how to set it up here, but it should look similar to this (with your credentials):

[https://www.transifex.com] username= user token= password=p @ssw0rd hostname= https://www.transifex.com

This basically identiﬁes you as a Transifex user, so you can access your organizations from the command line.

288 Chapter 56. Translating Zulip CHAPTER 57

Clients in Zulip

zerver.models.Client is Zulip’s analogue of the HTTP User-Agent header (and is populated from User- Agent). It exists for use in analytics and other places to provide human-readable summary data about “which Zulip client” was used for an operation (e.g. was it the Android app, the desktop app, or a bot?). In general, it shouldn’t be used for anything controlling the behavior of Zulip; it’s primarily intended to assist debugging.

Analytics

A Client is used to sort messages into client categories such as ZulipElectron on the /stats page. For more information see, Analytics.

Integrations

Generally, integrations in Zulip should declare a unique User-Agent, so that it’s easy to ﬁgure out which integration is involved when debugging an issue. For incoming webhook integrations, we do that convenentialy via the auth decorators (as we will describe shortly); other integrations generally should set the ﬁrst User-Agent element on their HTTP requests to something of the form ZulipIntegrationName/1.2 so that they are categorized properly. The api_key_only_webhook_view auth decorator, used for most incoming webhooks, accepts the name of the integration as an argument and uses it to generate a client name that it adds to the request (Django HttpRequest) object as request.client. In most integrations, request.client is then passed to check_send_message, where it is used to keep track of which client sent the message (which in turn is used by analytics). For more information, see the webhook walkthrough.

289 Zulip Documentation, Release 1.4.0

290 Chapter 57. Clients in Zulip CHAPTER 58

Logging and Performance Debugging

It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along with every backend request will be printed there. The messages will look similar to:

2016-05-20 14:50:22,056 INFO 127.0.0.1 GET 302 528ms (db: 1ms/1q)

˓→(+start: 123ms) / (unauth via ?) [20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0 2016-05-20 14:50:22,272 INFO 127.0.0.1 GET 200 124ms (db: 3ms/2q) /

˓→login/ (unauth via ?) 2016-05-20 14:50:26,333 INFO 127.0.0.1 POST 302 37ms (db: 6ms/7q) /

˓→accounts/login/local/ (unauth via ?) [20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 0 2016-05-20 14:50:26,538 INFO 127.0.0.1 GET 200 12ms (db: 1ms/2q)

˓→(+start: 53ms) /api/v1/events [1463769771:0/0] ([email protected] via internal) 2016-05-20 14:50:26,657 INFO 127.0.0.1 GET 200 10ms (+start: 8ms) /api/

˓→v1/events [1463769771:0/0] ([email protected] via internal) 2016-05-20 14:50:26,959 INFO 127.0.0.1 GET 200 588ms (db: 26ms/21q) /

˓→[1463769771:0] ([email protected] via website)

The format of this output is: timestamp, loglevel, IP, HTTP Method, HTTP status code, time to process, (optional perf data details, e.g. database time/queries, memcached time/queries, Django process startup time, markdown processing time, etc.), URL, and “email via client” showing user account involved (if logged in) and the type of client they used (“web”, “Android”, etc.).

291 Zulip Documentation, Release 1.4.0

292 Chapter 58. Logging and Performance Debugging CHAPTER 59

Zulip server release checklist

This document has reminders of things one might forget to do when preparing a new release.

A week before the release

• Upgrade all Python dependencies in requirements to latest upstream versions so they can burn in (use pip list --outdated). • Update all the strings on Transifex and notify translators that they should translate the new strings to get them in for the next release. • Update changelog.md with major changes going into the release. • Create a burndown list of bugs that need to be ﬁxed before we can release, and make sure all of them are being worked on.

Final release preparation

• Update changelog.md with any ﬁnal changes since the last update. • Draft the release notes; see previous zulip-announce emails for the tooling needed. • Download updated translation strings from Transifex and commit them. • Use build-release-tarball to generate a release tarball. • Test the new tarball extensively, both new install and upgrade from last release, on both Trusty and Xenial. • Repeat until release is ready.

293 Zulip Documentation, Release 1.4.0

Executing the release

• Do ﬁnal updates to changelog.md. • Update ZULIP_VERSION in version.py. • Use build-release-tarball to generate a ﬁnal release tarball. • Post the release tarball on zulip.org and update zulip.org. • Create a git tag and push the tag. • Upload the release on GitHub so it doesn’t provide a broken release tarball. • Email zulip-announce with the release notes • For a major release, post on the blog, tweet, etc.

Post-release

• Update ZULIP_VERSION in version.py to e.g. 1.6.0+git.

294 Chapter 59. Zulip server release checklist CHAPTER 60

Zulip PyPI package release checklist

This document describes the steps to be followed when preparing a new release of the PyPI package for our API bindings. While performing the steps outlined below, we should adhere to the guidelines presented in the Python Packaging User Guide. The steps below assume that you are familiar with the material presented here. 1. Reconfigure the package, if need be (upgrade version number, development status, and so on). 2. Create a source distribution. 3. Create a pure Python Wheel. 4. Upload the distribution file(s) to zulip-beta. 5. Post about the beta release in #general and test the zulip-beta package extensively. 6. Respond to the feedback received in Step 5. 7. Make final changes, upload the distribution file(s) to the main zulip package. 8. Post in #general about the new release. Note: We may upload directly to the main zulip package without beta-testing on zulip-beta, if we feel that the changes made in the new release are minor and not disruptive enough to warrant extensive pretesting.

295 Zulip Documentation, Release 1.4.0

296 Chapter 60. Zulip PyPI package release checklist CHAPTER 61

Documentation

Zulip has three major documentation systems: • Developer and sysadmin documentation: Documentation for people actually interacting with the Zulip codebase (either by developing it or installing it), and written in Markdown. • Core website documentation: Complete webpages for complex topics, written in HTML, JavaScript, and CSS (using the Django templating system). These roughly correspond to the documentation someone might look at when deciding whether to use Zulip. We don’t expect to ever have more than about 10 pages written using this system. • General user documentation: Our scalable system for documenting Zulip’s huge collection of speciﬁc features without a lot of overhead or duplicated code/syntax, written in Markdown. We expect to eventually have around 100 pages written using this system. The target audience for this system is individual Zulip users. These three systems are documented in detail.

Developer and sysadmin documentation

What you are reading right now is part of the collection of documentation targeted at developers and people running their own Zulip servers. These docs are written in Commonmark Markdown with a small bit of rST. We’ve chosen Markdown because it is easy to write. The source for Zulip’s developer documentation is at docs/ in the Zulip git repository, and they are served in production at zulip.readthedocs.io. If you want to build the developer documentation locally (e.g. to test your changes), the dependencies are automatically installed as part of Zulip development environment provisioning, and you can build the documentation using: cd docs/ make html and then opening http://127.0.0.1:9991/docs/index.html in your browser. The raw ﬁles are available at file:///path/to/zulip/docs/_build/html/index.html in your browser (so you can also use e.g. firefox docs/_build/html/index.html from the root of your Zulip checkout).

297 Zulip Documentation, Release 1.4.0

If you are adding a new page to the table of contents, you will want to modify docs/index.rst and run make clean before make html, so that other docs besides your new one also get the new entry in the table of contents. You can also usually test your changes by pushing a branch to GitHub and looking at the content on the GitHub web UI, since GitHub renders Markdown, though that won’t be as faithful as the make html approach. When editing dependencies for the Zulip documentation, you should edit requirements/docs.txt (which is used by ReadTheDocs to build the Zulip developer documentation, without installing all of Zulip’s dependencies).

Core website documentation

Zulip has around 10 HTML documentation pages under templates/zerver for speciﬁc major topics, like the features list, client apps, integrations, hotkeys, API bindings, etc. These documents often have somewhat complex HTML and JavaScript, without a great deal of common pattern between them other than inheriting from the portico.html template. We generally avoid adding new pages to this collection unless there’s a good reason, but we don’t intend to migrate them, either, since this system gives us the ﬂexibility to express these important elements of the product clearly.

General user documentation

To learn more about Zulip’s general user documentation, visit our guide on writing user documentation here.

298 Chapter 61. Documentation CHAPTER 62

General user guide documentation

Our goal is for Zulip to have complete, high-quality user-appealing documentation about use Zulip’s features and perform certain tasks, such as setting up an organization. This system is designed to make writing and maintaining such documentation highly efﬁcient.

Editing and testing

The user documentation is available under /help/ on any Zulip server; (e.g. https://chat.zulip.org/help/ or http:/ /localhost:9991/help/ in the Zulip development environment). The user documentation is not hosted on ReadTheDocs, since Zulip supports running a server completely disconnected from the Internet, and we’d like the documentation to be available in that environment. The source for this user documentation is the Markdown files under templates/zerver/help/ in the main Zulip server repository. The file foo.md is automatically rendered by the render_markdown_path function in zerver/templatetags/app_filters.py when the user accesses a URL of the form /help/foo; with special cases for /help/ going to index.md and /help/unknown_article going to missing.md (with a 404 response). Images are usually linked from static/images/help/. This means that you can contribute to the Zulip user documentation by just adding to or editing the collection of markdown files under templates/zerver/help. If you have the Zulip development environment setup, you simply need to reload your browser on http://localhost:9991/help/foo to see the latest version of foo. md rendered.

Writing documentation

Writing documentation is a different form of writing than most people have experience with. When you write user-appealing documentation, keep in mind that most users don’t care about how the underlying code works; they simply want to know how a feature works or how they can do something.

299 Zulip Documentation, Release 1.4.0

Think of writing user documentation as writing a recipe; you’re trying to direct your readers how to cook a dish while introducing them to the dish itself, yet you shouldn’t go into detail about the scientiﬁc processes that allow you to cook your dish. By keeping these thoughts in mind, you’ll be able to write better user guide documentation.

Title

The title of your documentation should be a brief summary of your documentation. Your title should be formatted as a heading by prepending your title with a #. The title of your documentation needs to be included under a ﬁtting section in templates/zerver/help/ index.md so users can access it from the user documentation index.

Introduction

By writing a brief introduction of what you will be discussing in your documentation, users will gain a better understanding of your document. For example, if you are writing about a feature, you could describe what the feature allows users to do. Your introduction should be no longer than two paragraphs.

Directions

If you are guiding a user on how to perform a certain task in Zulip, it is best to explain the process through a series of numbered steps. Your documentation’s goal is to direct users on how to do something so that they achieve their goals. Thus, steps should be as detailed yet concise as possible. For example, if you tried to explain copying and pasting text, you could describe it in the following steps:

1. Highlight the text you want to copy.

2. Press CTRL+C to copy the highlighted text.

3. Move your cursor to the location where you want to paste the text.

4. Press CTRL+V to paste the copied text.

You can continue steps with additional notes or instructions by breaking the line and indenting the next line, which is especially useful for including images within a single step.

1. Do something.

You can also do this.

A general tip for breaking up a process is whenever a user interacts with the Zulip UI (e.g.: click a button, refresh the page), a new step should be created.

Sections

If you are documenting multiple processes in your documentation, such as how to perform a task by using several different features, you should differentiate the processes by dividing them into sections using headings. Like the title, your section headings should give a brief description of the processes you describe. All subsequent subsections should be appended with an increasing number of #‘s to make the headings smaller.

300 Chapter 62. General user guide documentation Zulip Documentation, Release 1.4.0

# Title

## Section 1

### Subsection 1

### Subsection 2

## Section 2

### Subsection 1

Always leave a blank line on either side of a section heading.

User interface

When you refer to the features in the Zulip UI, you should bold the feature’s name followed by the feature itself (e.g. Settings page, Change password button, Email ﬁeld). No quotation marks should be used. Keep in mind that the UI may change — don’t describe it in more detail than is needed. Never identify or refer to a button by its color.

Features

Zulip’s Markdown processor allows you to include several special features in your documentation to help improve its readibility: • Since raw HTML is supported in Markdown, you can include arbitrary HTML/CSS in your documentation as needed. • Code blocks allow you to highlight syntax, similar to Zulip’s own markdown. • Anchor tags can be used to link to headers in other documents. • Images of Zulip UI can be added to documentation. • Inline icons used to refer to features in the Zulip UI. • You can utilize macros to limit repeated content in the documentation. • You can create special highlight warning blocks using tips and warnings.

Images

Images and screenshots should be included in user documentation only if it will help guide the user in how to do something (e.g. if the image will make it much clearer which element on the page the user should interact with). For instance, an image of an element should not be included if the element the user needs to interact with is the only thing on the page, but images can be included to show the end result of an interaction with the UI. Using too many screenshots creates maintainability problems (we have to update them every time the UI is changed) and also can make the instructions for something simple look long and complicated. When taking screenshots, the image should never include the whole Zulip browser window in a screenshot; instead, it should only show relevant parts of the app. In addition, the screenshot should always come after the text that describes it, never before. Images are often formatted as continuations of steps and must be indented on a new line to be formatted this way.

62.3. Features 301 Zulip Documentation, Release 1.4.0

Icons

You can refer to features in the Zulip UI by referencing their names and their FontAwesome (version 4.7.0) text icons within parentheses. The source for the text icons is located in static/third/thirdparty-fonts.css. Note: It is strongly recommended to use the new base class fa instead of the older base class icon-vector when specifying icons. In future we will be removing support for the icons with base class icon-vector. • cog () icon — cog () icon • down chevron () icon — down chevron () icon • eye () icon — eye () icon • file () icon — file () icon • filled star () icon — filled star () icon • formatting () icon — formatting () icon • menu () icon — menu () icon • overflow ( ) icon — overflow ( ) icon • paperclip () icon — paperclip () icon • pencil () icon — pencil () icon • pencil and paper () icon — pencil and paper () icon • plus () icon — plus () icon • smiley face () icon — smiley face () icon • star () icon — star () icon • trash () icon — trash () icon • x () icon — x () icon

Macros

Macros are elements in the format of {!macro.md!} that insert common phrases and steps at the location of the macros. Macros help eliminate repeated content in our documentation. The source for macros is the Markdown files under templates/zerver/help/include in the main Zulip server repository. If you find multiple instances of particular content in the documentation, you can always create a new macro by adding a new file to that folder.

Organization settings {!admin.md!} macro

• About: Links to the Organization settings documentation. Usually preceded by the Go to the macro and a link to a particular section on the Organization settings page. • Contents:

tab of the [Organization settings](/help/edit-administrator-settings) page.

• Example usage and rendering:

{!go-to-the.md!} [Organization settings](/#organization/organization-settings) {!admin.md!}

302 Chapter 62. General user guide documentation Zulip Documentation, Release 1.4.0

1. Go to the [Organization settings](/#organization/organization-settings) tab of

˓→the [Organization](/help/edit-administrator-settings) page.

Administrator only feature {!admin-only.md!} macro

• About: Creates a note the feature discussed in the documentation is only available to organization administrators. It should be placed immediately after the title. • Contents:

!!! warn "" **Note:** This feature can only be controlled by organization administrators.

• Example usage and rendering:

{!admin-only.md!}

{!follow-steps.md!} change who can join your stream by changing the stream's accessibility.

!!! warn "" **Note:** This feature can only be controlled by organization administrators.

Follow the following steps to change who can join your stream by changing the stream's accessibility.

All streams {!all-streams.md!} macro

• About: Explains how to view all streams in the organization on the Streams page. Usually formatted as a tip and preceded by the Streams macro and the Filter streams macro. • Contents:

If you wish to see streams that you aren't subscribed to, click on the **All streams** tab; the tab will turn gray upon doing so.

• Example usage and rendering:

{!subscriptions.md!} {!filter-streams.md!} !!! tip "" {!all-streams.md!}

1. [Find the relevant stream](/help/browse-and-join-streams#browse-streams) on the [Streams](/#streams) page. You can search for specific streams by entering the name of the stream in the **Filter streams** input. !!! tip "" If you wish to see streams that you aren't subscribed to, click on the **All streams** tab; the tab will turn gray upon doing so.

62.3. Features 303 Zulip Documentation, Release 1.4.0

Down chevron {!down-chevron.md!} macro

• About: Instructs readers to click on the down chevron () icon to reveal an actions dropdown; usually preceded by an command, such as the Message actions macro. • Contents:

down chevron () icon to reveal an actions

˓→dropdown.

• Example usage and rendering:

{!message-actions.md!} {!down-chevron.md!}

1. Hover over a message to replace the message's timestamp with its message actions, represented by three icons. From the icons that appear, select the down chevron () icon to reveal an actions

˓→dropdown.

Go to the {!go-to-the.md} macro

• About: Usually precedes the Settings macro or the Organization settings macro. Transforms following content into a step. • Contents:

1. Go to the

• Example usage and rendering:

{!go-to-the.md!} [Notifications](/#settings/notifications) {!settings.md!}

1. Go to the [Notifications](/#settings/notifications) tab on the [Settings](/help/edit-settings) page.

Filter streams {!filter-streams.md!} macro

• About: Explains how to search for speciﬁc streams in the Streams page using the Filter streams input. Usually preceded by the Streams macro. • Contents:

You can search for specific streams by entering the name of the stream in the **Filter streams** input.

• Example usage and rendering:

{!subscriptions.md!} {!filter-streams.md!}

304 Chapter 62. General user guide documentation Zulip Documentation, Release 1.4.0

Follow steps {!follow-steps.md!} macro

• About: Prepends phrases with instructions to follow the following steps. • Contents:

Follow the following steps to

• Example usage and rendering:

{!follow-steps.md!} change your mobile notification settings.

Follow the following steps to change your mobile notification settings.

Message actions {!message-actions.md!} macro

• About: Explains how to view the actions of message. Usually followed by an instruction to click a speciﬁc icon, such as the Down chevron macro. • Contents:

1. Hover over a message to replace the message's timestamp with its message actions, represented by three icons. From the icons that appear, select the

• Example usage and rendering:

{!message-actions.md!} {!down-chevron.md!}

1. Hover over a message to replace the message's timestamp with its message actions, represented by three icons. From the icons that appear, select the down chevron () icon to reveal an actions

˓→dropdown.

Save changes {!save-changes.md!} macro

• About: Used in documentation that requests users to save their changes by pressing the Save changes button. Usually followed by a phrase describing the settings they modiﬁed. • Contents:

1. Click on the **Save changes** button to save any changes you made to your

• Example usage and rendering:

{!save-changes.md!} notification settings.

1. Click on the **Save changes** button to save any changes you made to your notification settings.

62.3. Features 305 Zulip Documentation, Release 1.4.0

Settings {!settings.md!} macro

• About: Links to the Edit Settings documentation. Usually preceded by the Go to the macro and a link to a particular section on the Settings page. • Contents:

tab on the [Settings](/help/edit-settings) page.

• Example usage and rendering:

{!go-to-the.md!} [Notifications](/#settings/notifications) {!settings.md!}

1. Go to the [Notifications](/#settings/notifications) tab on the [Settings](/help/edit-settings) page.

Stream actions {!stream-actions.md!} macro

• About: Explains how to view the actions of stream. Usually followed by the an instruction and the Down chevron macro. • Contents:

1. On the left sidebar in the **Streams** section, hover over a stream to reveal a down chevron () icon to the right of the stream name.

• Example usage and rendering:

{!stream-actions.md!}

1. Click on the {!down-chevron.md!}

1. On the left sidebar in the **Streams** section, hover over a stream to reveal a down chevron () icon to the right of the stream name.

2. Click on the down chevron () icon to reveal an actions dropdown.

Stream settings {!stream-settings.md!} macro

• About: Notiﬁes readers about the changes in the Streams page when a stream is selected; usually followed by an instruction. • Contents:

the right side of the [Streams](/#streams) page, labeled **Stream settings**, will now display the selected stream's settings.

• Example usage and rendering:

1. Click on the stream you want to edit; {!stream-settings.md!}

306 Chapter 62. General user guide documentation Zulip Documentation, Release 1.4.0

1. Click on the stream you want to edit; the right side of the [Streams](/#streams) page, labeled **Stream settings**, will now display the selected stream's settings.

Stream settings scroll {!stream-settings.md!} macro

• About: Instructs readers to scroll down to a particular section on the Streams page after making sure their cursors are hovering above the Streams Settings section. • Contents:

1. After making sure that your cursor is hovering over the **Streams Settings** section, scroll down to the

• Example usage and rendering:

{!stream-settings-scroll.md!} **Stream membership** section. This section shows the usernames and emails of all users that are currently subscribed to the selected stream.

1. After making sure that your cursor is hovering over the **Streams Settings** section, scroll down to the **Stream membership** section. This section shows the usernames and emails of all users that are currently subscribed to the selected stream.

Streams {!subscriptions.md!} macro

• About: Used in documentation that direct users to the Streams page. Often followed by the Filter streams macro. • Contents:

1. [Find the relevant stream](/help/browse-and-join-streams#browse-streams) on the [Streams](/#streams) page.

• Example usage and rendering:

{!subscriptions.md!} {!filter-streams.md!}

Tips and warnings

A tip is any suggestion for the user that is not part of the main set of instructions. For instance, it may address a common problem users may encounter while following the instructions, or point to an option for power users.

!!! tip "" If you've forgotten your password, see the [Change your password](/help/change-your-password) page for instructions on how to reset it.

62.3. Features 307 Zulip Documentation, Release 1.4.0

A warning is a note on what happens when there is some kind of problem. Tips are more common than warnings.

!!! warn "" **Note:** If you attempt to input a nonexistent stream name, an error message will appear.

All tips/warnings should appear inside tip/warning blocks. There should be only one tip/warning inside each block.They usually be formatted as a continuation of a numbered step.

Documentation template

Here is a template for writing user documentation that follows the guidelines listed in this document. It demonstrates the proper formatting of several features discussed above. You can always look at previously-written documentation in the templates/zerver/help folder as a reference for writing your documentation.

# Title of documentation

This paragraph briefly summarizes the documentation. Please remember to replace this paragraph with an appropriate description of the documentation.

## Writing documentation as a series of steps

1. This is the first step of the documentation.

![First step](/static/images/help/image1.png)

2. This is the second step of the documentation.

Here's an additional note about the second step. Click on the star () icon to learn more about it.

3. This is the third step of the documentation.

!!! tip "" You can always add more steps to your documentation!

## Writing documentation to introduce a feature

If necessary, you can add another section to your documentation. Sections can be used to differentiate different methods of performing a task or describing a related task.

![Feature](/static/images/help/feature.png)

You can also conclude your documentation with some final notes.

!!! warn "" **Note:** This is only a documentation template; you can always deviate **from this template to suit your documentation needs.

308 Chapter 62. General user guide documentation