Zulip Documentation Release 1.7.0
The Zulip Team
Oct 25, 2017
Overview
1 Zulip overview 3 1.1 Community...... 3 1.2 Installing the Zulip Development environment...... 4 1.3 Running Zulip in production...... 4 1.4 Ways to contribute...... 4 1.5 Google Summer of Code...... 4 1.6 How to get involved with contributing to Zulip...... 5 1.7 License...... 6
2 Zulip architectural overview 7 2.1 Key Codebases...... 7 2.2 Usage assumptions and concepts...... 7 2.3 Components...... 8 2.4 Glossary...... 11
3 Directory structure 13 3.1 Core Python files...... 13 3.2 HTML Templates...... 13 3.3 JavaScript and other static assets...... 14 3.4 Tests...... 14 3.5 Management commands...... 14 3.6 Scripts...... 14 3.7 API and Bots...... 15 3.8 Production puppet configuration...... 15 3.9 Additional Django apps...... 15 3.10 Jinja2 Compatibility Files...... 15 3.11 Translation files...... 16 3.12 Documentation...... 16
4 Zulip Roadmap 17 4.1 Introduction...... 17 4.2 Major projects...... 17 4.3 Core User Experience...... 18 4.4 Social features...... 18 4.5 Real-time sync...... 18 4.6 Onboarding issues...... 18 4.7 Production installation issues...... 19
i 4.8 Administration and management...... 19 4.9 Scalability and performance...... 19 4.10 Technology improvements...... 19 4.11 Technical Debt...... 20 4.12 Security...... 20 4.13 Testing...... 20 4.14 Documentation...... 20 4.15 Integrations and bots...... 20 4.16 Android app...... 21 4.17 iOS app...... 21 4.18 Server/webapp support for mobile...... 21 4.19 Desktop apps...... 21 4.20 Community...... 21
5 Version History 23 5.1 Unreleased...... 23 5.2 1.7.0 – 2017-10-25...... 23 5.3 1.6.0 – 2017-06-06...... 27 5.4 1.5.2 – 2017-06-01...... 30 5.5 1.5.1 – 2017-02-07...... 30 5.6 1.5.0 – 2017-02-06...... 30 5.7 1.4.3 - 2017-01-29...... 32 5.8 1.4.2 - 2016-09-27...... 32 5.9 1.4.1 - 2016-09-03...... 33 5.10 1.4.0 - 2016-08-25...... 33 5.11 1.3.13 - 2016-06-21...... 35 5.12 1.3.12 - 2016-05-10...... 36 5.13 1.3.11 - 2016-05-02...... 36 5.14 1.3.10 - 2016-01-21...... 37 5.15 1.3.9 - 2015-11-16...... 38 5.16 1.3.8 - 2015-11-15...... 38 5.17 1.3.7 - 2015-10-19...... 38
6 Zulip in production 39 6.1 Requirements...... 39 6.2 Install...... 39 6.3 Running...... 39
7 Requirements 41 7.1 Server...... 41 7.2 Credentials needed...... 42
8 Production Installation 43 8.1 Step 1: Install SSL Certificates...... 43 8.2 Step 2: Download and install latest release...... 43 8.3 Step 3: Configure Zulip...... 44 8.4 Step 4: Test email configuration...... 44 8.5 Step 5: Initialize Zulip database...... 44 8.6 Step 6: Create a Zulip organization and login...... 45 8.7 Step 7: Next steps...... 45 8.8 Troubleshooting...... 45
9 Troubleshooting 47 9.1 Using supervisorctl...... 47 9.2 Troubleshooting services...... 49 ii 10 Customize Zulip 51 10.1 Making changes...... 51 10.2 Specific settings...... 51 10.3 Zulip announcement list...... 52 10.4 Enjoy your Zulip installation!...... 52
11 Mobile push notification service 53 11.1 How to sign up...... 53 11.2 Why this is necessary...... 54 11.3 Security and privacy implications...... 54
12 Secure, maintain, and upgrade 55 12.1 Upgrading...... 55 12.2 Upgrading from a git repository...... 57 12.3 Backups...... 58 12.4 Monitoring...... 59 12.5 Scalability...... 59 12.6 Securing your Zulip server...... 60 12.7 Management commands...... 60 12.8 Hosting multiple Zulip organizations...... 61
13 Security Model 63 13.1 Secure your Zulip server like your email server...... 63 13.2 Encryption and Authentication...... 63 13.3 Messages and History...... 64 13.4 Users and Bots...... 65 13.5 User-uploaded content...... 65 13.6 Final notes and security response...... 66
14 Authentication methods 67 14.1 Adding additional methods using python-social-auth...... 67 14.2 Remote User SSO Authentication...... 68
15 Postgres database details 71 15.1 Remote Postgres database...... 71 15.2 Debugging postgres database issues...... 72 15.3 Stopping the Zulip postgres database...... 72 15.4 Debugging issues starting postgres...... 72 15.5 Postgres Vacuuming alerts...... 73
16 Development environment installation 75 16.1 Requirements...... 75 16.2 Recommended setup (Vagrant)...... 75 16.3 Advanced setup (non-Vagrant)...... 75 16.4 Slow internet connections...... 76 16.5 Installing remotely...... 76 16.6 Next steps...... 76
17 Vagrant environment setup tutorial 77 17.1 Requirements...... 78 17.2 Step 0: Set up Git & GitHub...... 78 17.3 Step 1: Install Prerequisites...... 78 17.4 Step 2: Get Zulip Code...... 81 17.5 Step 3: Start the development environment...... 82 17.6 Step 4: Developing...... 85
iii 17.7 Next Steps...... 86 17.8 Troubleshooting and Common Errors...... 87 17.9 Specifying a proxy...... 93
18 Zulip development environment setup without Vagrant 95 18.1 Installing directly on Ubuntu...... 95 18.2 Installing manually on Linux...... 95 18.3 Using Docker (experimental)...... 101
19 Using the Development Environment 103
20 Developing on a remote machine 105 20.1 Connecting to the remote environment...... 105 20.2 Setting up the development environment...... 105 20.3 Running the development server...... 106 20.4 Making changes to code on your remote development server...... 107
21 Writing a new integration 111 21.1 Types of integrations...... 111 21.2 General advice...... 112 21.3 Webhook integrations...... 112 21.4 Python script and plugin integrations...... 113 21.5 Documenting your integration...... 114
22 Documenting an integration 115 22.1 Markdown macros...... 116
23 Webhook walkthrough 121 23.1 Step 0: Create fixtures...... 121 23.2 Step 1: Initialize your webhook python package...... 122 23.3 Step 2: Create main webhook code...... 122 23.4 Step 3: Create an api endpoint for the webhook...... 123 23.5 Step 4: Create tests...... 124 23.6 Step 5: Create documentation...... 126 23.7 Step 5: Preparing a pull request to zulip/zulip...... 127 23.8 Advanced topics...... 127
24 Interactive bots 131 24.1 Installing the zulip_bots package...... 131 24.2 Running a bot...... 131 24.3 Zulip Botserver...... 132 24.4 Common problems...... 134
25 Writing interactive bots 135 25.1 Installing a development version of the zulip_bots package...... 135 25.2 Writing a bot...... 136 25.3 Adding a bot to Zulip...... 136 25.4 Testing a bot’s output...... 137 25.5 Bot API...... 137 25.6 Writing tests for bots...... 140 25.7 Common problems...... 142 25.8 Future direction...... 142
26 Writing a new application feature 143 26.1 General Process...... 143
iv 26.2 Example Feature...... 145
27 Writing views in Zulip 153 27.1 What this covers...... 153 27.2 What is a view?...... 153 27.3 Modifying urls.py...... 153 27.4 Writing human-readable views...... 154 27.5 Writing API REST endpoints...... 154 27.6 Legacy endpoints used by the web client...... 158 27.7 Webhook integration endpoints...... 158
28 Life of a Request 159 28.1 A request is sent to the server, and handled by Nginx...... 159 28.2 Nginx secures traffic with SSL...... 159 28.3 Static files are served directly by Nginx...... 159 28.4 Nginx routes other requests between django and tornado...... 160 28.5 Django routes the request to a view in urls.py files...... 160 28.6 Views serving HTML are internationalized by server path...... 160 28.7 API endpoints use REST...... 160 28.8 Django calls rest_dispatch for REST endpoints, and authenticates...... 162 28.9 The view will authorize the user, extract request variables, and validate them...... 162 28.10 Results are given as JSON...... 162
29 Reading list 163 29.1 General programming/IT...... 163 29.2 Python...... 164 29.3 Java/Android...... 164 29.4 JavaScript/ECMAScript...... 164 29.5 Git/Version Control Systems (VCS)...... 164 29.6 Computer Science/Algorithms...... 164 29.7 Community experience...... 165 29.8 Competitions/Camps...... 165 29.9 Massive Open Online Courses (MOOC) Platforms...... 165
30 Screenshot and GIF software 167 30.1 Screenshot tools by platform...... 167 30.2 GIF tools by platform...... 168
31 Fixing Commits 169 31.1 Fixing the last commit...... 169 31.2 Fixing older commits...... 169 31.3 Squashing commits...... 170 31.4 Reordering commits...... 170 31.5 Pushing commits after tidying them...... 170
32 Git Cheat Sheet (Detailed) 171
33 Git Cheat Sheet 173
34 Shell tips 175 34.1 The prompt ($)...... 176 34.2 Tilde character (~)...... 176 34.3 Change directory (cd)...... 176 34.4 Running commands as root (sudo)...... 177 34.5 Escaping characters...... 177
v 34.6 Sequencing commands...... 177 34.7 Splitting commands into multiple lines...... 178 34.8 Arguments...... 178 34.9 Shebang...... 179 34.10 Understanding commands...... 180 34.11 Cheatsheet...... 180 34.12 Git...... 180
35 Working copies 183 35.1 Workflows...... 183 35.2 Names...... 184 35.3 Relevant git commands...... 184
36 Git & GitHub Guide 185 36.1 Quick start: How Zulip uses Git and GitHub...... 185 36.2 Set up Git...... 186 36.3 How Git is different...... 186 36.4 Important Git terms...... 187 36.5 Get Zulip code...... 189 36.6 Using Git as you work...... 190 36.7 Create a pull request...... 197 36.8 Update a pull request...... 199 36.9 Collaborate...... 200 36.10 Review changes...... 200 36.11 Get and stay out of trouble...... 201 36.12 Zulip-specific tools...... 205
37 Version control 209 37.1 Commit Discipline...... 209 37.2 Commit Messages...... 210
38 Code style and conventions 213 38.1 Be consistent!...... 213 38.2 Lint tools...... 213 38.3 Secrets...... 214 38.4 Dangerous constructs...... 214 38.5 JS array/object manipulation...... 216 38.6 More arbitrary style things...... 216
39 Python static type checker (mypy) 219 39.1 type_debug.py ...... 219 39.2 Zulip goals...... 220 39.3 Installing mypy...... 220 39.4 Running mypy on Zulip’s code locally...... 220 39.5 Excluded files...... 221 39.6 Mypy is there to find bugs in Zulip before they impact users...... 221 39.7 Annotating strings...... 221
40 Reviewing Zulip code 223 40.1 Principles of code review...... 223 40.2 Things to look for...... 224 40.3 Tooling...... 225 40.4 Additional Resources...... 225
41 The chat.zulip.org community 227 vi 41.1 This is a bleeding edge development server...... 227 41.2 Community conventions...... 227 41.3 High traffic community...... 228 41.4 Streams...... 228 41.5 Chat meetings...... 229
42 Using zulipbot 231 42.1 Usage...... 231
43 Accessibility 233 43.1 Guidelines...... 233 43.2 Tools...... 233 43.3 GitHub Issues...... 234 43.4 Additional Resources...... 234
44 Bug report guidelines 235
45 Testing and writing tests 237 45.1 Overview...... 237 45.2 Running tests...... 237 45.3 Schema and initial data changes...... 239 45.4 Wiping the test databases...... 239 45.5 Local browser testing (local app + web browser)...... 240
46 Linters 241 46.1 Overview...... 241 46.2 Running the linters...... 241 46.3 General considerations...... 242 46.4 Lint checks...... 242 46.5 lint...... 243 46.6 Philosophy...... 244
47 Backend Django tests 247 47.1 Overview...... 247 47.2 Running tests...... 247 47.3 Writing tests...... 248 47.4 Zulip Testing Philosophy...... 251 47.5 Testing considerations...... 253
48 JavaScript unit tests 255 48.1 HTML output...... 255 48.2 Coverage reports...... 256 48.3 Handling dependencies in unit tests...... 256 48.4 Creating new test modules...... 257
49 Web frontend black-box casperjs tests 259 49.1 Debugging Casper.JS...... 259 49.2 Writing Casper tests...... 260
50 Travis CI 263 50.1 Goals...... 263 50.2 Configuration...... 263 50.3 Useful debugging tips and tools...... 264 50.4 Performance optimizations...... 264
vii 51 Manual testing 265 51.1 Basic Stuff...... 265
52 Provisioning and third-party dependencies 277 52.1 Provisioning...... 277 52.2 Philosophy on adding third-party dependencies...... 277 52.3 System packages...... 278 52.4 Python packages...... 278 52.5 JavaScript and other frontend packages...... 279 52.6 Node and Yarn...... 280 52.7 Other third-party and generated files...... 280 52.8 Modifying provisioning...... 281
53 Settings system 283 53.1 Server settings...... 283 53.2 Realm settings...... 285
54 Real-time Push and Events 287 54.1 Generation system...... 288 54.2 Delivery system...... 288 54.3 The initial data fetch...... 289
55 Queue processors 291 55.1 Adding a new queue processor...... 291 55.2 Publishing events into a queue...... 292 55.3 Clearing a RabbitMQ queue...... 292
56 Custom Apps 293 56.1 Definition...... 293 56.2 Problem statement...... 293 56.3 A quick note on bots/integrations...... 293 56.4 Categories of custom apps...... 294 56.5 World Reader...... 295 56.6 Zulip Reader...... 297 56.7 Deployment issues...... 297
57 Unread counts and the pointer 301 57.1 Pointer logic...... 301 57.2 Unread count logic...... 302 57.3 Testing and development...... 303
58 Markdown implementation 305 58.1 Testing...... 305 58.2 Changing Zulip’s markdown processor...... 306 58.3 Zulip’s Markdown philosophy...... 307 58.4 Zulip’s Changes to Markdown...... 307
59 Realms in Zulip 309 59.1 Creating Realms...... 309 59.2 Subdomains...... 310
60 Management commands 311 60.1 Writing management commands...... 311
61 Static asset pipeline 313 61.1 Primary build process...... 313 viii 61.2 Adding static files...... 313 61.3 How it works in production...... 314 61.4 Webpack/CommonJS/ES6/Typescript modules...... 314
62 Schema Migrations 317
63 HTML and CSS 319 63.1 Zulip CSS organization...... 319 63.2 Editing Zulip CSS...... 319 63.3 CSS Style guidelines...... 320 63.4 Validating CSS...... 320
64 URL hashes and deep linking 321 64.1 Hashchange...... 321 64.2 Server-initiated reloads...... 322 64.3 All reloads...... 322
65 Emoji 323 65.1 Emoji codes...... 323 65.2 Tooling...... 324
66 Hotspots 325 66.1 Adding a new hotspot...... 325
67 Full-text search 327 67.1 The default full-text search implementation...... 327 67.2 An optional full-text search implementation...... 327
68 Email 329 68.1 Development and testing...... 330
69 Analytics 331 69.1 Analytics backend overview...... 331 69.2 The *Count database tables...... 332 69.3 CountStats...... 332 69.4 The FillState table...... 332 69.5 Performance strategy...... 332 69.6 Backend Testing...... 333 69.7 LoggingCountStats...... 333 69.8 Analytics UI development and testing...... 334
70 Translating Zulip 337 70.1 Translation style guides...... 337 70.2 Translation process...... 338 70.3 Translators’ workflow...... 339 70.4 Testing translations...... 339 70.5 Setting the default language in Zulip...... 340 70.6 Translation resource files...... 340 70.7 HTML Templates...... 340 70.8 Backend translations...... 340 70.9 Frontend translations...... 341 70.10 Transifex config...... 342 70.11 Transifex CLI setup...... 342
71 HTML templates 343 71.1 Behavior...... 343
ix 71.2 Backend...... 343 71.3 Frontend...... 344
72 Clients in Zulip 345 72.1 Analytics...... 345 72.2 Integrations...... 345
73 Logging and Error reporting 347 73.1 Backend error reporting...... 347 73.2 Blueslip frontend error reporting...... 349 73.3 Frontend performance reporting...... 350
74 Typing indicators 351 74.1 Writing user...... 351 74.2 Server...... 352 74.3 Receiving user...... 352 74.4 Ecosystem...... 353 74.5 Roadmap...... 353
75 Zulip server release checklist 355 75.1 A week before the release...... 355 75.2 Final release preparation...... 355 75.3 Executing the release...... 356 75.4 Post-release...... 356
76 Zulip PyPI package release checklist 357
77 Swagger API documentation 359 77.1 Working with the zulip.yaml file...... 359 77.2 Zulip Swagger YAML style:...... 361 77.3 Tips for working with YAML:...... 361
78 Documentation 363 78.1 Developer and sysadmin documentation...... 363 78.2 Core website documentation...... 364 78.3 General user documentation...... 364
79 General user guide documentation 365 79.1 Editing and testing...... 365 79.2 Writing documentation...... 366 79.3 Features...... 367 79.4 Documentation template...... 374
x Zulip Documentation, Release 1.7.0
Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip supports both private messaging and group chats via conversation streams. Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifica- tions, missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org and in these docs. Our code is available at our GitHub repository. This set of documents covers installation and contribution instructions. Contents: • Overview • Zulip in production • Development environment • Developer tutorials • Code contribution guide • Code testing • Subsystem documentation Zulip overview | Community | Installing for dev | Installing for production | Ways to contribute | How to get involved | License
Overview 1 Zulip Documentation, Release 1.7.0
2 Overview CHAPTER 1
Zulip overview
Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip supports both private messaging and group chats via conversation streams. Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifica- tions, missed-message emails, desktop apps, and much more. Further information on the Zulip project and its features can be found at https://www.zulip.org.
1.1 Community
There are several places online where folks discuss Zulip. • The primary place is the Zulip development community Zulip server at chat.zulip.org. • For Google Summer of Code students and applicants, we have a mailing list for help, questions, and announce- ments. But it’s often simpler to visit chat.zulip.org instead. • We have a public development discussion mailing list, zulip-devel, which is currently pretty low traffic because most discussions happen in our public Zulip instance. We use it to announce Zulip developer community gatherings and ask for feedback on major technical or design decisions. It has several hundred subscribers, so you can use it to ask questions about features or possible bugs, but please don’t use it ask for generic help getting started as a contributor (e.g. because you want to do Google Summer of Code). The rest of this page covers how to get involved in the Zulip project in detail. • Zulip also has a blog and twitter account. • Last but not least, we use GitHub to track Zulip-related issues (and store our code, of course). Anybody with a GitHub account should be able to create Issues there pertaining to bugs or enhancement requests. We also use Pull Requests as our primary mechanism to receive code contributions. The Zulip community has a Code of Conduct.
3 Zulip Documentation, Release 1.7.0
1.2 Installing the Zulip Development environment
The Zulip development environment is the recommended option for folks interested in trying out Zulip, since it is very easy to install. This is documented in the developer installation guide.
1.3 Running Zulip in production
Zulip in production supports Ubuntu 16.04 Xenial and Ubuntu 14.04 Trusty. We’re happy to support work to enable Zulip to run on additional platforms. The installation process is documented here.
1.4 Ways to contribute
Zulip welcomes all forms of contributions! This page documents the Zulip development process. • Pull requests. Before a pull request can be merged, you need to sign the Dropbox Contributor License Agree- ment. Also, please skim our commit message style guidelines. We encourage early pull requests for work in progress. Prefix the title of your pull request with [WIP] and reference it when asking for community feedback. When you are ready for final review, remove the [WIP]. • Testing. The Zulip automated tests all run automatically when you submit a pull request, but you can also run them all in your development environment following the instructions in the testing docs. You can also try out our new desktop client, which is in alpha; we’d appreciate testing and feedback. • Developer Documentation. Zulip has a growing collection of developer documentation on Read The Docs. Recommended reading for new contributors includes the directory structure and new feature tutorial. You can also improve Zulip.org. • Mailing lists and bug tracker. Zulip has a development discussion mailing list and uses GitHub issues . There are also lists for the Android and iOS apps. Feel free to send any questions or suggestions of areas where you’d love to see more documentation to the relevant list! Check out our bug report guidelines before submitting. Please report any security issues you discover to [email protected]. • App codebases. This repository is for the Zulip server and web app (including most integrations). The beta React Native mobile app, Java Android app (see our mobile strategy), new Electron desktop app, and legacy Qt-based desktop app are all separate repositories. • Glue code. We maintain a Hubot adapter and several integrations (Phabricator, Jenkins, Puppet, Redmine, and Trello), plus node.js API bindings, an isomorphic JavaScript library, and a full-text search PostgreSQL extension, as separate repos. • Translations. Zulip is in the process of being translated into 10+ languages, and we love contributions to our translations. See our translating documentation if you’re interested in contributing! • Code Reviews. Zulip is all about community and helping each other out. Check out #code review on chat.zulip.org to help review PRs and give comments on other people’s work. Everyone is welcome to par- ticipate, even those new to Zulip! Even just checking out the code, manually testing it, and posting on whether or not it worked is valuable.
1.5 Google Summer of Code
We participated in GSoC in 2016 (with great results) and are participating in 2017 as well.
4 Chapter 1. Zulip overview Zulip Documentation, Release 1.7.0
1.6 How to get involved with contributing to Zulip
First, subscribe to the Zulip development discussion mailing list. The Zulip project uses a system of labels in our issue tracker to make it easy to find a project if you don’t have your own project idea in mind or want to get some experience with working on Zulip before embarking on a larger project you have in mind: • Integrations. Integrate Zulip with another piece of software and contribute it back to the community! Writing an integration can be a great first contribution. There’s detailed documentation on how to write integrations in the Zulip integration writing guide. • Good first issue: Smaller projects that might be a great first contribution. • Documentation: The Zulip project loves contributions of new documentation. • Help Wanted: A broader list of projects that nobody is currently working on. • Platform support: These are open issues about making it possible to install Zulip on a wider range of platforms. • Bugs: Open bugs. • Feature requests: Browsing this list can be a great way to find feature ideas to implement that other Zulip users are excited about. • 2016 roadmap milestone: The projects that are priorities for the Zulip project. These are great projects if you’re looking to make an impact. Another way to find issues in Zulip is to take advantage of our area:
1.6. How to get involved with contributing to Zulip 5 Zulip Documentation, Release 1.7.0
When you feel like you have completed your work on an issue, post your PR to the #code review stream on chat.zulip.org. This is our lightweight process that gives other developers the opportunity to give you comments and suggestions on your work.
1.7 License
Copyright 2011-2017 Dropbox, Inc., Kandra Labs, Inc., and contributors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. The software includes some works released by third parties under other free and open source licenses. Those works are redistributed under the license terms under which the works were received. For more details, see the docs/ THIRDPARTY file included with this distribution.
6 Chapter 1. Zulip overview CHAPTER 2
Zulip architectural overview
2.1 Key Codebases
The core Zulip application is at https://github.com/zulip/zulip and is a web application written in Python 2.7 (soon to also support Python 3) and using the Django framework. That codebase includes server-side code and the web client, as well as Python API bindings and most of our integrations with other services and applications (see the directory structure guide). Zulip Mobile is the official mobile Zulip client supporting both iOS and Android, written in JavaScript with React Native, and Zulip Desktop is the official Zulip desktop client for macOS, Linux, and Windows. We also maintain several separate repositories for integrations and other glue code: a Hubot adapter; integrations with Phabricator, Jenkins, Puppet, Redmine, and Trello; node.js API bindings; and our full-text search PostgreSQL extension. We use Transifex to do translations. In this overview, we’ll mainly discuss the core Zulip server and web application.
2.2 Usage assumptions and concepts
Zulip is a real-time web-based chat application meant for companies and similar groups ranging in size from a small team to more than a thousand users. It features real-time notifications, message persistence and search, public group conversations (streams), invite-only streams, private one-on-one and group conversations, inline image previews, team presence/buddy lists, a rich API, Markdown message support, and numerous integrations with other services. The maintainer team aims to support users who connect to Zulip using dedicated iOS, Android, Linux, Windows, and macOS clients, as well as people using modern web browsers or dedicated Zulip API clients. A server can host multiple Zulip realms (organizations) at the same domain, each of which is a private chamber with its own users, streams, customizations, and so on. This means that one person might be a user of multiple Zulip realms. The administrators of a realm can choose whether to allow anyone to register an account and join, or only allow people who have been invited, or restrict registrations to members of particular groups (using email domain
7 Zulip Documentation, Release 1.7.0 names or corporate single-sign-on login for verification). For more on security considerations, see the security model section. The default Zulip home screen is like a chronologically ordered inbox; it displays messages, starting at the oldest message that the user hasn’t viewed yet (for more on that logic, see the guide to the pointer and unread counts). The home screen displays the most recent messages in all the streams a user has joined (except for the streams they’ve muted), as well as private messages from other users, in strict chronological order. A user can narrow to view only the messages in a single stream, and can further narrow to focus on a topic (thread) within that stream. Each narrow has its own URL. The user can quickly see what conversation they’re in – the stream and topic, or the names of the user(s) they’re private messaging with – using the recipient bar displayed atop each conversation. Zulip’s philosophy is to provide sensible defaults but give the user fine-grained control over their incoming information flow; a user can mute topics and streams, and can make fine-grained choices to reduce real-time notifications they find irrelevant.
2.3 Components
2.3.1 Django and Tornado
Zulip is primarily implemented in the Django Python web framework. We also make use of Tornado for the real-time push system. Django is the main web application server; Tornado runs the server-to-client real-time push system. The app servers are configured by the Supervisor configuration (which explains how to start the server processes; see "Supervisor" below) and the nginx configuration (which explains which HTTP requests get sent to which app server).
8 Chapter 2. Zulip architectural overview Zulip Documentation, Release 1.7.0
Tornado is an asynchronous server and is meant specifically to hold open tens of thousands of long-lived (long-polling or websocket) connections – that is to say, routes that maintain a persistent connection from every running client. For this reason, it’s responsible for event (message) delivery, but not much else. We try to avoid any blocking calls in Tornado because we don’t want to delay delivery to thousands of other connections (as this would make Zulip very much not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since those blocking requests carry a very high performance penalty for a single-threaded, asynchronous server. The parts that are activated relatively rarely (e.g. when people type or click on something) are processed by the Django application server. One exception to this is that Zulip uses websockets through Tornado to minimize latency on the code path for sending messages. There is detailed documentation on the real-time push and event queue system; most of the code is in zerver/ tornado.
HTML templates, JavaScript, etc.
Zulip’s HTML is primarily implemented using two types of HTML templates: backend templates (powered by the Jinja2 template engine used for logged-out ("portico") pages and the webapp’s base content) and frontend templates (powered by Handlebars) used for live-rendering HTML from JavaScript for things like the main message feed. For more details on the frontend, see our documentation on translation, templates, directory structure, and the static asset pipeline.
2.3.2 nginx
nginx is the front-end web server to all Zulip traffic; it serves static assets and proxies to Django and Tornado. It handles HTTP requests according to the rules laid down in the many config files found in zulip/puppet/zulip/ files/nginx/. zulip/puppet/zulip/files/nginx/zulip-include-frontend/app is the most important of these files. It explains what happens when requests come in from outside. • In production, all requests to URLs beginning with /static/ are served from the corresponding files in / home/zulip/prod-static/, and the production build process (tools/build-release-tarball) compiles, minifies, and installs the static assets into the prod-static/ tree form. In development, files are served directly from /static/ in the git repository. • Requests to /json/events, /api/v1/events, and /sockjs are sent to the Tornado server. These are requests to the real-time push system, because the user’s web browser sets up a long-lived TCP connection with Tornado to serve as a channel for push notifications. nginx gets the hostname for the Tornado server via puppet/zulip/files/nginx/zulip-include-frontend/upstreams. • Requests to all other paths are sent to the Django app via the UNIX socket unix:/home/ zulip/deployments/uwsgi-socket (defined in puppet/zulip/files/nginx/ zulip-include-frontend/upstreams). We use zproject/wsgi.py to implement uWSGI here (see django.core.wsgi). • By default (i.e. if LOCAL_UPLOADS_DIR is set), nginx will serve user-uploaded content like avatars, custom emoji, and uploaded files. However, one can configure Zulip to store these in a cloud storage service like Amazon S3 instead.
2.3.3 Supervisor
We use supervisord to start server processes, restart them automatically if they crash, and direct logging.
2.3. Components 9 Zulip Documentation, Release 1.7.0
The config file is zulip/puppet/zulip/templates/supervisor/zulip.conf.template.erb. This is where Tornado and Django are set up, as well as a number of background processes that process event queues. We use event queues for the kinds of tasks that are best run in the background because they are expensive (in terms of performance) and don’t have to be synchronous — e.g., sending emails or updating analytics. Also see the queuing guide.
2.3.4 memcached
memcached is used to cache database model objects. zerver/lib/cache.py and zerver/lib/ cache_helpers.py manage putting things into memcached, and invalidating the cache when values change. The memcached configuration is in puppet/zulip/files/memcached.conf.
2.3.5 Redis
Redis is used for a few very short-term data stores, such as in the basis of zerver/lib/rate_limiter.py, a per-user rate limiting scheme example), and the email-to-Zulip integration. Redis is configured in zulip/puppet/zulip/files/redis and it’s a pretty standard configuration except for the last line, which turns off persistence:
# Zulip-specific configuration: disable saving to disk. save""
memcached was used first and then we added Redis specifically to implement rate limiting. We’re discussing switching everything over to Redis.
2.3.6 RabbitMQ
RabbitMQ is a queueing system. Its config files live in zulip/puppet/zulip/files/rabbitmq. Initial configuration happens in zulip/scripts/setup/configure-rabbitmq. We use RabbitMQ for queuing expensive work (e.g. sending emails triggered by a message, push notifications, some analytics, etc.) that require reliable delivery but which we don’t want to do on the main thread. It’s also used for communication between the application server and the Tornado push system. Two simple wrappers around pika (the Python RabbitMQ client) are in zulip/zerver/lib/queue.py. There’s an asynchronous client for use in Tornado and a more general client for use elsewhere. Most of the pro- cesses started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handle them; they are defined in zerver/worker/queue_processors.py. Also see the queuing guide.
2.3.7 PostgreSQL
PostgreSQL (also known as Postgres) is the database that stores all persistent data, that is, data that’s expected to live beyond a user’s current session. In production, Postgres is installed with a default configuration. The directory that would contain configuration files (puppet/zulip/files/postgresql) has only a utility script and a custom list of stopwords used by a Post- gresql extension. In a development environment, configuration of that postgresql extension is handled by tools/ postgres-init-dev-db (invoked by tools/provision). That file also manages setting up the development postgresql user.
10 Chapter 2. Zulip architectural overview Zulip Documentation, Release 1.7.0 tools/provision also invokes tools/do-destroy-rebuild-database to create the actual database with its schema.
2.3.8 Nagios
Nagios is an optional component used for notifications to the system administrator, e.g., in case of outages. zulip/puppet/zulip/manifests/nagios.pp installs Nagios plugins from puppet/zulip/files/ nagios_plugins/. This component is intended to install Nagios plugins intended to be run on a Nagios server; most of the Zulip Nagios plugins are intended to be run on the Zulip servers themselves, and are included with the relevant component of the Zulip server (e.g. puppet/zulip/manifests/postgres_common.pp installs a few under /usr/lib/ nagios/plugins/zulip_postgres_common).
2.4 Glossary
This section gives names for some of the elements in the Zulip UI used in Zulip development conversations. Contri- butions to extend this list are welcome! • chevron: A small downward-facing arrow next to a message’s timestamp, offering contextual options, e.g., "Reply", "Mute [this topic]", or "Link to this conversation". To avoid visual clutter, the chevron only appears in the web UI upon hover. • huddle: What the codebase calls a "group private message". • message editing: If the realm admin allows it, then after a user posts a message, the user has a few minutes to click "Edit" and change the content of their message. If they do, Zulip adds a marker such as "(EDITED)" at the top of the message, visible to anyone who can see the message. • realm: What the codebase calls an "organization" in the UI. • recipient bar: A visual indication of the context of a message or group of messages, displaying the stream and topic or private message recipient list, at the top of a group of messages. A typical 1-line message to a new recipient shows to the user as three lines of content: first the recipient bar, second the sender’s name and avatar alongside the timestamp (and, on hover, the star and the chevron), and third the message content. The recipient bar is or contains hyperlinks to help the user narrow. • star: Zulip allows a user to mark any message they can see, public or private, as "starred". A user can easily ac- cess messages they’ve starred through the "Starred messages" link in the menu near "Home", or use "is:starred" as a narrow or a search constraint. Whether a user has or has not starred a particular message is private; other users and realm admins don’t know whether a message has been starred, or by whom. • subject: What the codebase calls a "topic" in many places. • bankruptcy: When a user has been off Zulip for several days and has hundreds of unread messages, they are prompted for whether they want to mark all their unread messages as read. This is called "declaring bankruptcy" (in reference to the concept in finance).
2.4. Glossary 11 Zulip Documentation, Release 1.7.0
12 Chapter 2. Zulip architectural overview CHAPTER 3
Directory structure
This page documents the Zulip directory structure, where to find things, and how to decide where to put a file. You may also find the new application feature tutorial helpful for understanding the flow through these files.
3.1 Core Python files
Zulip uses the Django web framework, so a lot of these paths will be familiar to Django developers. • zproject/urls.py Main Django routes file. Defines which URLs are handled by which view functions or templates. • zerver/models.py Main Django models file. Defines Zulip’s database tables. • zerver/lib/actions.py Most code doing writes to user-facing database tables. • zerver/views/*.py Most Django views. • zerver/webhooks/ Webhook views and tests for Zulip webhook integrations. • zerver/tornado/views.py Tornado views. • zerver/worker/queue_processors.py Queue workers. • zerver/lib/*.py Most library code. • zerver/lib/bugdown/ Backend Markdown processor. • zproject/backends.py Authentication backends.
3.2 HTML Templates
See our docs for details on Zulip’s templating systems.
13 Zulip Documentation, Release 1.7.0
• templates/zerver/ For Jinja2 templates for the backend (for zerver app). • static/templates/ Handlebars templates for the frontend.
3.3 JavaScript and other static assets
• static/js/ Zulip’s own JavaScript. • static/styles/ Zulip’s own CSS. • static/images/ Zulip’s images. • static/third/ Third-party JavaScript and CSS that has been vendored. • node_modules/ Third-party JavaScript installed via yarn. • static/assets/ For assets not to be served to the web (e.g. the system to generate our favicons).
3.4 Tests
• zerver/tests/ Backend tests. • frontend_tests/node_tests/ Node Frontend unit tests. • frontend_tests/casper_tests/ Casper frontend tests. • tools/test-* Developer-facing test runner scripts.
3.5 Management commands
These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access). • zerver/management/commands/ Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly).
3.6 Scripts
• scripts/ Scripts that production deployments might run manually (e.g., restart-server). • scripts/lib/ Scripts that are needed on production deployments but humans should never run directly. • scripts/setup/ Scripts that production deployments will only run once, during installation. • tools/ Scripts used only in a Zulip development environment. These are not included in production release tarballs for Zulip, so that we can include scripts here one wouldn’t want someone to run in production acciden- tally (e.g. things that delete the Zulip database without prompting).
14 Chapter 3. Directory structure Zulip Documentation, Release 1.7.0
• tools/setup/ Subdirectory of tools/ for things only used during the development environment setup process. • tools/travis/ Subdirectory of tools/ for things only used to setup and run our tests in Travis CI. Actual test suites should go in tools/.
3.7 API and Bots
• See the Zulip API repository. Zulip’s Python API bindings, a number of Zulip integrations and bots, and a framework for running and testing Zulip bots, used to be developed in the main Zulip server repo but are now in their own repo. • templates/zerver/integrations/ (within templates/zerver/, above). Documentation for these integrations.
3.8 Production puppet configuration
This is used to deploy essentially all configuration in production. • puppet/zulip/ For configuration for production deployments. • puppet/zulip/manifests/voyager.pp Main manifest for Zulip standalone deployments.
3.9 Additional Django apps
• confirmation Email confirmation system. • analytics Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites). • corporate The old Zulip.com website. Not included in production distribution. • zilencer Primarily used to hold management commands that aren’t used in production. Not included in production distribution.
3.10 Jinja2 Compatibility Files
• zproject/jinja2/__init__.py Jinja2 environment. • zproject/jinja2/backends.py Jinja2 backend. • zproject/jinja2/compressors.py Jinja2 compatible functions of Django-Pipeline.
3.7. API and Bots 15 Zulip Documentation, Release 1.7.0
3.11 Translation files
• locale/ Backend (Django) translations data files. • static/locale/ Frontend translations data files.
3.12 Documentation
• docs/ Source for this documentation.
You can consult the repository’s .gitattributes file to see exactly which components are excluded from produc- tion releases (release tarballs are generated using tools/build-release-tarball).
16 Chapter 3. Directory structure CHAPTER 4
Zulip Roadmap
4.1 Introduction
Zulip has received a great deal of interest and attention since it was released as free and open source software by Dropbox. That attention has come with a lot of active development work from members of the Zulip community. From when Zulip was released as open source in late September 2015 through today (early November, 2016), more than 150 people have contributed over 1000 pull requests to the various Zulip repositories, the vast majority of which were submitted by Zulip’s users around the world (as opposed to the small core team that reviews and merges the pull requests). In any project, there can be a lot of value in periodically putting together a roadmap detailing the major areas where the project is hoping to improve. This can be especially important in an open source project like Zulip, where development is distributed across many people around the world. This roadmap is intended to organize a list of the most important improvements that should be made to Zulip in the relatively near future. Our aim is to complete most of these improvements by February 2017 and then prepare a new roadmap then. This document is not meant to constrain in any way what contributions to Zulip will be accepted; instead, it will be used by the Zulip core team to prioritize our efforts, measure progress on improving the Zulip product and hold ourselves accountable for making Zulip improve rapidly. This roadmap is the best place for contributors to look for substantial projects that will definitely be of value to the community (if you’re looking for a starter project, see the guide to getting involved with Zulip). We periodically update this roadmap by adding strikethrough to issues that have been resolved, but the linked GitHub issues are the most up-to-date source for that information. Update: As of May 2017, we are approaching the point where we need to update the roadmap due to much of it being completed. Without further ado, below is the current Zulip roadmap.
4.2 Major projects
There are 2 huge projects that Zulip is working on right now that are too big to have a coherent GitHub issue:
17 Zulip Documentation, Release 1.7.0
• We are working with a world-class designer on a major visual redesign of the Zulip webapp. Already complete is completely redesining the streams and settings UIs, logged-out pages, and various other major components. • We are writing a new React Native iOS app for Zulip to replace the old iOS app. The new app is progressing rapidly, but is not yet feature complete. We expect it to be in the app store in May 2017.
4.3 Core User Experience
• Provide shorter UI/Keyboard sequence to edit the last message • Better drafts management • Make clicking on desktop notifications renarrow properly • Add pretty bubbles for recipients in the compose box • Make right sidebar buddy list UI scale well to large teams • Display stream descriptions more prominently • Add support for managing uploaded files
4.4 Social features
• Add support for showing "user is typing" notifications, at least for private messages • Support lightweight emoji "reactions" • Open graph previews of generic websites • Add a "join Zulip chat" badge for projects that use Zulip to document that nicely
4.5 Real-time sync
The overall goal is to eliminate the few known issues where Zulip does not provide a seamless real-time sync experi- ence. • Notification bot advertisements for new streams don’t handle stream renames • Avatar/name changes don’t propagate to already-sent messages • Advance the pointer / where we load the user to based on unread counts in home view • Fix the known bug where messages could be incorrectly marked as read
4.6 Onboarding issues
This category focuses on issues users experience when installing a new Zulip server, setting up a new Zulip realm, or starting to use Zulip. • Move Zulip’s prompt for permission to display notifications to be manually triggered • Add a mechanism for deleting early test messages (e.g., administrators can hard-delete messages) • Allow customizing emails when inviting new users
18 Chapter 4. Zulip Roadmap Zulip Documentation, Release 1.7.0
4.7 Production installation issues
• Document or better script solution to rabbitmq startup issues • Merge a supported way to use Zulip in Docker in production implementation.
4.8 Administration and management
• Make list of allowed domains web-configurable • Statistics display for realm and server administrators • Keep track of which users added which realm emoji • Add setting to enable any user to add new realm emoji • Make realm filters web-configurable • Improve administrative controls for managing streams • Enhance the LDAP integration and make it web-configurable • Add a SAML integration for Zulip
4.9 Scalability and performance
Scalability and performance are not currently major problems for Zulip; it already scales well to thousands of users and is significantly faster than proprietary alternatives. So, this is not a major focus area for the project. • Make the Zulip Tornado service support horizontal scaling • Make presence system scale well to 5000 users in a realm. • Support running queue workers multithreaded in production to decrease minimum memory footprint • Improve @-mentioning syntax based on stronger unique identifiers
4.10 Technology improvements
• Add support for Zulip running purely on Python 3 • Automatic thumbnailing of uploaded images’ previews to save bandwidth • Upgrade Zulip to use Django 1.10. The patches needed to run Zulip were merged into mainline Django in Django 1.10, so this will mean we don’t need to use a fork of Django anymore. • Upgrade and remove from codebase all unnecessarily vendored JS libraries • Add support for changing users’ email addresses • Migrate from jslint to eslint • Replace the slow closure-compiler based static asset toolchain • Use a modern JavaScript bundler like webpack • Add support for building frontend features in something like React
4.7. Production installation issues 19 Zulip Documentation, Release 1.7.0
4.11 Technical Debt
While the Zulip server has a great codebase compared to most projects of its size, it takes work to keep it that way. • Migrate most web routes to REST API • Split Tornado subsystem into a separate Django app • Refactor zulip.css to be broken into components
4.12 Security
• Add support for 2-factor authentication on all platforms • Add support for stronger security controls for uploaded files (The LOCAL_UPLOADS_DIR file uploads back- end only supports world-readable uploads) • Fix requirement to set a password when creating account via Google • Add a retention policy feature that automatically deletes old messages • Add UI for viewing and cancelling open Zulip invitations
4.13 Testing
• Extend Zulip’s automated test coverage to include all API endpoints • Build automated tests for the client API bindings • Add automated tests for the production upgrade process
4.14 Documentation
• Add an in-app mechanism for updating users about new Zulip features • Significantly expand documentation of the Zulip API and integrating with Zulip. • Write a visual design / frontend style guide for Zulip • Update all screenshots to show the current Zulip UI
4.14.1 Nice to have
• Expand library of documentation on Zulip’s feature set. Currently most documentation is for either developers or system administrators.
4.15 Integrations and bots
Integrations are essential to Zulip. While we currently have a reasonably good framework for writing new webhook integrations for getting notifications into Zulip, it’d be great to streamline that process and make bots that receive messages just as easy to build.
20 Chapter 4. Zulip Roadmap Zulip Documentation, Release 1.7.0
• Add an outgoing webhook integration system • Make setting up a new integration a smooth flow • Default new incoming webhooks to permissions-limited incoming webhook bots • Change how Zulip displays bot names to distinguish them from human users
4.16 Android app
• Add support for narrowing to @-mentions • Support having multiple Zulip realms open simultaneously
4.17 iOS app
For the new React Native iOS app, the major goal for it is to be released into the app store. Since it is moving quickly, we’re tracking its roadmap via GitHub milestones.
4.18 Server/webapp support for mobile
To support a great mobile experiences, we need to make some improvements in the Zulip server. • Push notifications bouncer service for GCM and APNS • A slick process for doing mobile login without typing your password on your phone • @here mention support (that doesn’t spam people not currently online, i.e. no email/push notifications) • Fix sending messages from mobile web
4.19 Desktop apps
The new cross-platform desktop app is implemented in Electron, and primarily needs work on installer tooling to finish replacing the old app. • Finish releasing the Electron app to replace the old desktop app • Support having multiple Zulip realms open simultaneously
4.20 Community
These don’t get GitHub issues since they’re not technical projects, but they are important goals for the project. • Expand the number of core developers able to do code reviews • Have a successful season with Zulip’s Outreachy participants • Have a successful season with Google Code In.
4.16. Android app 21 Zulip Documentation, Release 1.7.0
22 Chapter 4. Zulip Roadmap CHAPTER 5
Version History
All notable changes to the Zulip server are documented in this file.
5.1 Unreleased
This section lists notable unreleased changes; it is generally updated in bursts.
5.2 1.7.0 – 2017-10-25
Highlights: Web • We’ve completely redesigned our onboarding process to explain Zulip, and especially topics, to new users. • We’ve built a beautiful new emoji picker with categories, a showcase, and much better data. Note the clean, underscore-free display! • The emails sent by Zulip are more consistent, readable, and visually interesting. • Chinese (Simplified) and Japanese join Spanish, German, and Czech in having the user interface fully translated, in addition to partial translations for many other languages. We also fixed many small issues where strings weren’t tagged for translation. • Many pages have been redesigned to be easier to use and visually cleaner, including the settings pages and the user documentation at /help, /integrations, and /apps. Mobile and Desktop support • Zulip Server 1.7 adds several new APIs that are critical for mobile app performance and that let the app track unread messages. If you’re using the mobile apps at all (iOS or Android), you will definitely want to upgrade to Zulip 1.7.
23 Zulip Documentation, Release 1.7.0
• The iOS and Android apps can receive push notifications (configurable, naturally) for events like PMs and @- mentions. While Zulip Server 1.6 has basic support for these, 1.7 brings a new, clearer format to notifications, and gives each user more options for finer-grained control. • The new Electron desktop app is out of beta and replaces our legacy desktop apps. Backend and scaling • Zulip now runs exclusively on Python 3. This is the culmination of an 18-month migration effort. We are very excited about this! • We’ve added an automatic "soft deactivation" process, which dramatically improves performance for organiza- tions with a large number of inactive users, without any impact on those users’ experience if they later come back. • Zulip’s performance at scale has improved significantly. Performance now scales primarily with number of active users (not total users). As an example, chat.zulip.org serves 400 monthly active users and about 3500 total users, on one VM with just 8GB of RAM and a CPU consistently over 90% idle. Upgrade notes: • Zulip 1.7 contains some significant database migrations that can take several minutes to run. The upgrade process automatically minimizes disruption by running these first, before beginning the user-facing downtime. However, if you’d like to watch the downtime phase of the upgrade closely, we recommend running them first manually and as well as the usual trick of doing an apt upgrade first. • We’ve removed support for an uncommon legacy deployment model where a Zulip server served multiple or- ganizations on the same domain. Installs with multiple organizations now require each organization to have its own subdomain. This change should have no effect for the vast majority of Zulip servers that only have one organization. If you manage a server that hosts multiple organizations, you’ll want to read our guide on multiple organizations. • We simplified the configuration for our password strength checker to be much more intuitive. If you were using the PASSWORD_MIN_ZXCVBN_QUALITY setting, it has been replaced by the more intuitive PASSWORD_MIN_GUESSES. Full feature changelog: • Simplified the process for installing a new Zulip server, as well as fixing the most common roadbumps and confusing error messages. • Added a new "incoming webhook" bot type, limited to only sending messages into Zulip, for better security. • Added experimental support for outgoing webhooks. • Added support for changing the notifications stream. • Added ’u’ hotkey to show a user’s profile. • Added ’-’ hotkey to toggle collapsing a message. • Added an organization setting to require topics in stream messages. • Added an organization setting to control whether edit history is available. • Added a confirmation dialogue when inviting many users to a new stream. • Added new notification setting to always get push notifications on a stream. • Added new "getting started" guides to the user documentation. • Added support for installing a Zulip server from a Git checkout. • Added support for mentioning a user when editing a message.
24 Chapter 5. Version History Zulip Documentation, Release 1.7.0
• Added OpsGenie, Google Code-In, Google Search, and xkcd integrations. • Added support for organization administrators deleting private streams. • Added support for using any LDAP attribute for login username. • Added support for searching by group-pm-with. • Added support for mentioning users when editing messages. • Added a much prettier prompt for enabling desktop notifications. • Added a new PHYSICAL_ADDRESS setting to be used in outgoing emails to support compliance with anti- spam regulations. • Dramatically improved the search typeahead experience when using multiple operators. • Improved design for /stats page and added a link to it in the gear menu. • Improved how timestamps are displayed across the product. • Improved the appearance of mention/compose typeahead. • Improved lightbox to support panning and zooming on images. • Improved "more topics" to fetch all historical topics from the server. • Improved scrollbars across the site to look good on Windows and Linux. • Improved visual design of stream management UI. • Improved management of disk space, especially when deploying with Git frequently. • Improve mention typeahead sort order to prioritize recent senders in a stream. • Swapped the ’q’ and ’w’ hotkeys to better match the UI. • Fixed most issues with the registration flow, including adding Oauth support for mobile and many corner case problems. • Significantly improved sort ordering for the emoji picker. • Fixed most accessibility errors detected by major accessibility checker tools. • Extracted Zulip’s Python API and bots ecosystem into its own repository, zulip/python-zulip-api. • Enter hotkey now opens compose in empty narrows. • Significantly improved performance of "starred messages" and "mentions" database queries through new in- dexes. • Upgraded to Django 1.11.x. • Upgraded to a more modern version of the SourceSansPro font. • Redesigned several settings subpages to be visually cleaner. • Redesigned Zulip’s error pages to feature cute illustrations. • Dramatically improved the user typeahead algorithm to suggest relevant users even in large organizations with 1000s of accounts. • Fixed log rotation structural issues which wasted a lot of disk. • Updated notification settings to not require a "save changes" button. • Rewrote the documentation for almost all of our integrations to be much clearer and more consistent through use of Markdown and macros. • Restructured Zulip’s management commands to use a common system for accessing realms and users.
5.2. 1.7.0 – 2017-10-25 25 Zulip Documentation, Release 1.7.0
• Made starting editing a message you just sent not require a round trip. • Dramatically increased test coverage of the frontend codebase. • Dramatically improved the responsive mobile user experience. • Changed the right sidebar search to ignore diacritics. • Overhauled error handling in the new user registration flows. • Fixed minor bugs in several webhook integrations. • Fixed several local echo bugs involving mentions and line-wrapping. • Fixed various inconsistent old-style buttons in settings pages. • Fixed some obscure bugs with uploading files. • Fixed issues with deactivating realm emoji. • Fixed rendering of emoji in tweet previews. • Fixed buggy translation caching which filled local storage. • Fixed handling of desktop and mobile apps in new-login emails. • Fixed caching of source repository in upgrade-zulip-from-git. • Fixed numerous minor internationalization bugs. • Fixed several bugs with the LDAP authentication backend. • Fixed several corner case bugs with push notification. • Fixed rendering of realm emoji in missed-message emails. • Fixed various endpoints incorrectly using the PUT HTTP method. • Fixed bugs in scrolling up using the home key repeatedly. • Fixed a bug where private messages from multiple users could be included in a single missed-message email. • Fixed issues with inconsistent visual display of @-all mentions. • Fixed zombie process leaks on servers with <4GB of RAM. • Fixed markdown previews of /me messages. • Fixed a subtle bug involving timestamps of locally echoed messages. • Fixed the behavior of key combintions like Ctrl+Enter in the compose box. • Worked around Google Compute Engine’s default boto configuration, which broke Zulip (and any other app using boto). • Zulip now will gracefully handle the Postgres server being restarted. • Optimized marking an entire topic as read. • Switched from npm to yarn for downloading JS packages. • Switched the function of the ’q’ and ’w’ search hotkeys. • Simplified the settings for configuring senders for our emails. • Emoji can now be typed with spaces, e.g. entering "robot face" in the typeahead as well as "robot_face". • Improved title and alt text for unicode emoji. • Added development tools to make iterating on emails and error pages easy.
26 Chapter 5. Version History Zulip Documentation, Release 1.7.0
• Added backend support for multi-use invite links (no UI for creating yet). • Added a central debugging log for attempts to send outgoing emails. • Added a deprecation notice for the legacy QT-based desktop app. • Removed most remaining legacy API format endpoints. • Removed the obsolete shortname-based syntax. • Removed the old django-guardian dependency. • Removed several obsolete settings. • Partially completed migration to webpack as our static asset bundler.
5.3 1.6.0 – 2017-06-06
Highlights: • A complete visual redesign of the logged-out pages, including login, registration, integrations, etc. • New visual designs for numerous UI elements, including the emoji picker, user profile popovers, sidebars, compose, and many more. • A complete redesign of the Zulip settings interfaces to look a lot nicer and be easier to navigate. • Organization admins can now configure the login and registration pages to show visitors a nice organization profile with custom text and images, written in Markdown. • Massively improved performance for presence and settings pages, especially for very large organizations (1000+ users). • A dozen useful new keyboard shortcuts, from editing messages to emoji reactions to drafts and managing streams. • Typing notifications for private message threads. • Users can now change their own email address. • New saved-drafts feature. • The server can now run on a machine with as little as 2GB of RAM. • The new Electron desktop app and new React Native mobile app for iOS are now the recommended Zulip apps. • Mobile web now works much better, especially on iOS. • Support for sending mobile push notifications via a new forwarding service • Complete translations for Spanish, German, and Czech (and expanded partial translations for Japanese, Chinese, French, Hungarian, Polish, Dutch, Russian, Bulgarian, Portuguese, Serbian, Malayalam, Korean, and Italian). Full feature changelog: • Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and WordPress webhook integrations. • Added LaTeX support to the markdown processor. • Added support for filtering branches to all Git integrations. • Added read-only access to organization-level settings for all users. • Added UI for managing muted topics and uploaded files. • Added UI for displaying message edit history.
5.3. 1.6.0 – 2017-06-06 27 Zulip Documentation, Release 1.7.0
• Added support for various features needed by new mobile app. • Added deep links for settings/subscriptions interfaces. • Added an animation when messages are edited. • Added support for registration with GitHub auth (not just login). • Added tracking of uploaded file quotas. • Added option to display emoji as their alt codes. • Added new audit log table, to eventually support an auditing UI. • Added several new permissions-related organization settings. • Added new endpoint for fetching presence data, useful in employee directories. • Added typeahead for language for syntax highlighting in code blocks. • Added support for basic markdown in stream descriptions. • Added email notifications on new Zulip logins. • Added security hardening before serving uploaded files. • Added new PRIVACY_POLICY setting to provide a Markdown privacy policy. • Added an icon to distinguish bot users as message senders. • Added a command-line Slack importer tool using the API. • Added new announcement notifications on stream creation. • Added support for some newer unicode emoji code points. • Added support for users deleting realm emoji they themselves uploaded. • Added support for organization administrators deleting messages. • Extended data available to mobile apps to cover the entire API. • Redesigned bots UI. Now can change owners and reactivate bots. • Redesigned the visuals of code blocks to be prettier. • Changed right sidebar presence UI to only show recently active users in large organizations. This has a huge performance benefit. • Changed color for private messages to look better. • Converted realm emoji to be uploaded, not links, for better robustness. • Switched the default password hasher for new passwords to Argon2. • Increased the paragraph spacing, making multi-paragraph. • Improved formatting of all Git integrations. • Improved the UI of the /stats analytics pages. • Improved search typeahead to support group private messages. • Improved logic for when the compose box should open/close. • Improved lightbox to support scrolling through images. • Improved markdown support for bulleted lists. • Improved copy-to-clipboard support in various places.
28 Chapter 5. Version History Zulip Documentation, Release 1.7.0
• Improved subject lines of missed message emails. • Improved handling of users trying to login with Oauth without an account. • Improved UI of off-the-Internet errors to not be hidden in narrow windows. • Improved rate-limiting errors to be more easily machine-readable. • Parallelized the backend test suite; now runs 1600 tests in <30s. • Fixed numerous bugs and performance issues with stream management. • Fixed an issue with the fake emails assigned to bot users. • Fixed a major performance issue in stream creation. • Fixed numerous minor accessibility issues. • Fixed a subtle interaction between click-to-reply and copy-paste. • Fixed various formatting issues with /me messages. • Fixed numerous real-time sync issues involving users changing their name, avatar, or email address and streams being renamed. • Fixed numerous performance issues across the project. • Fixed various left sidebar ordering and live-updated bugs. • Fixed numerous bugs with the message editing widget. • Fixed missing logging / rate limiting on browser endpoints. • Fixed regressions in Zulip’s browser state preservation on reload logic. • Fixed support for unicode characters in the email mirror system. • Fixed load spikes when email mirror is receiving a lot of traffic. • Fixed the ugly grey flicker when scrolling fast on Macs. • Fixed previews of GitHub image URLs. • Fixed narrowing via clicking on desktop notifications. • Fixed Subscribed/Unsubscribed bookends appearing incorrectly. • Eliminated the idea of a realm having a canonical domain; now there’s simply the list of allowed domains for new users. • Migrated avatars to a user-id-based storage setup (not email-based). • Trailing whitespace is now stripped in code blocks, avoiding unnecessary scrollbars. • Most API payloads now refer to users primarily by user ID, with email available for backwards-compatibility. In the future, we may remove email support. • Cleaned up Zulip’s supervisord configuration. A side effect is the names of the log files have changed for all the queue workers. • Refactored various endpoints to use a single code path for security hardening. • Removed support for the MANDRILL_CLIENT setting. It hadn’t been used in years. • Changed NOREPLY_EMAIL_ADDRESS setting to Name
5.3. 1.6.0 – 2017-06-06 29 Zulip Documentation, Release 1.7.0
5.4 1.5.2 – 2017-06-01
• CVE-2017-0896: Restricting inviting new users to admins was broken. • CVE-2015-8861: Insecure old version of handlebars templating engine.
5.5 1.5.1 – 2017-02-07
• Fix exception trying to copy node_modules during upgrade process. • Improved styling of /stats page to remove useless login/register links.
5.6 1.5.0 – 2017-02-06
Highlights: • Completely redesigned the Manage streams interface. • Added support for emoji reactions to messages. • Added a lightbox for viewing images and videos. • Added an extensive user documentation site at /help/. • Added admin setting to auto-linkify certain strings (useful for issue numbers and Git commit IDs). • Upgraded how the main application runs from FastCGI on Django 1.8 to uwsgi and Django 1.10. • Added preliminary support for open graph previews of links (the setting, INLINE_URL_EMBED_PREVIEW, is disabled by default in this release). Full feature changelog: • Added an emoji picker/browser to the compose box. • Added markdown preview support to the compose box. • Added a new analytics system to track interesting usage statistics. • Added a /stats page with graphs of the analytics data. • Added display of subscriber counts in Manage streams. • Added support for filtering streams in Manage streams. • Added support for setting a stream description on creation. • Added support for copying subscribers from existing streams on creation. • Added several new search/filtering UI elements. • Added UI for deactivating your own Zulip account. • Added support for viewing the raw markdown content of a message. • Added support for deploying Zulip with subdomains for each realm. This entailed numerous changes to ensure a consistent experience. • Added support for (optionally) using PGRoonga to support full-text search in all languages (not just English). • Added AppFollow, GitLab, Google Calendar, GoSquared, HelloSign, Heroku, Librato, MailChimp, Mention, Papertrail, Sentry, Solano Labs, Stripe and Zapier integrations.
30 Chapter 5. Version History Zulip Documentation, Release 1.7.0
• Added a webhook integration for GitHub, replacing the deprecated github-services hook. • Normalized the message formatting for all the Zulip Git integrations. • Added support for VMWare Fusion Vagrant provider for faster OSX development. • Added a shields.io style badge for joining a Zulip server. • Added admin setting for which email domains can join a realm. • Added admin setting for controlling who can create streams. • Added admin setting to limit stream creation to older users. • Added a notification when you muted a topic. • Added a new hotkey for muting/unmuting topics. • Added support for testing websockets to the Nagios plugins. • Added a configuration option to disable websockets. • Added support for removing one’s own Zulip account. • Added support for realm admins which auth backends are supported. • Added new organization type concept. This will be used to control whether Zulip is optimized around protecting user privacy vs. administrative control. • Added #streamName syntax for linking to a stream. • Added support for viewing markdown source of messages. • Added setting to always send push notifications. • Added setting to hide private message content in desktop notifications. • Added buttons to download .zuliprc files. • Added italics and strikethrough support in markdown implementation. • Added errors for common installations mistakes (e.g. too little RAM). • Added a new /authors page showing the contributors to the current Zulip version. • Added illustrations to the 404 and 500 pages. • Upgraded all Python dependencies to modern versions, including Django 1.10 (all of Zulip’s patches have been merged into mainline). • Increased backend test coverage of Python codebase to 90%. • Increased mypy static type coverage of Python code to 100%. • Added several new linters (eslint, pep8) and cleaned the codebase. • Optimized the speed of the Zulip upgrade process, especially with Git. • Have peer_add events send user_id, not email. • Fixed problems with rabbitmq when installing Zulip. • Fixed JavaScript not being gzip-compressed properly. • Fixed a major performance bug in the Tornado service. • Fixed a frontend performance bug creating streams in very large realms. • Fixed numerous bugs where strings were not properly tagged for translation.
5.6. 1.5.0 – 2017-02-06 31 Zulip Documentation, Release 1.7.0
• Fixed several real-time sync bugs, and removed several AJAX calls. Zulip should be more performant than ever before. • Fixed Zulip Tornado service not working with http_proxy set in environment. • Fixed text overflow in stream subscriptions. • Fixed CSS issues with message topic editing. • Fixed several transactionality bugs (e.g. in Huddle creation). • Fixed missed-message email configuration error handling. • Fixed annoying @-mentions in Jira integration. • Fixed various mismatches between frontend and backend markdown implementations. • Fixed various popover-related UI bugs. • Fixed duplicate notifications with multiple open Zulip tabs. • Fixed support for emailing the server administrator about backend exceptions. • Cleaned up the "edit message" form. • Eliminated most of the legacy API endpoints. • Improved typeahead and autocomplete across the application. Highlights include much better handling of many users with similar names. • Improved the color scheme for code blocks. • Improved the message editing UI in several ways. • Improved how dates are displayed in the UI. • Improved default settings for zxcvbn password strength checker. • Upgraded jQuery to the latest 1.12 release. • Made numerous improvements to the development tooling. • Made extensive improvements to code organization. • Restyled all the registration pages to look nicer and be responsive. • Extensively refactored views to use common functions for fetching stream and message objects. • Suppressed @-all mentions being treated as mentions on muted streams. • Documented preliminary design for interactive bot system.
5.7 1.4.3 - 2017-01-29
• CVE-2017-0881: Users could subscribe to invite-only streams.
5.8 1.4.2 - 2016-09-27
• Upgraded Django to version 1.8.15 (with the Zulip patches applied), fixing a CSRF vulnerability in Django (see https://www.djangoproject.com/weblog/2016/sep/26/security-releases/), and a number of other Django bugs from past Django stable releases that largely affects parts of Django that are not used by Zulip. • Fixed buggy logrotate configuration.
32 Chapter 5. Version History Zulip Documentation, Release 1.7.0
5.9 1.4.1 - 2016-09-03
• Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0. • Fixed local file uploads integration being broken for new 1.4.0 installations.
5.10 1.4.0 - 2016-08-25
• Migrated Zulip’s python dependencies to be installed via a virtualenv, instead of the via apt. This is a major change to how Zulip is installed that we expect will simplify upgrades in the future. • Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncached network transfer for loading Zulip. • Added support for using Ubuntu Xenial in production. • Added a powerful and complete realm import/export tool. • Added nice UI for selecting a default language to display settings. • Added UI for searching streams in left sidebar with hotkeys. • Added Semaphore, Bitbucket, and HelloWorld (example) integrations. • Added new webhook-based integration for Trello. • Added management command for creating realms through web UI. • Added management command to send password reset emails. • Added endpoint for mobile apps to query available auth backends. • Added LetsEncrypt documentation for getting SSL certificates. • Added nice rendering of unicode emoji. • Added support for pinning streams to the top of the left sidebar. • Added search box for filtering user list when creating a new stream. • Added realm setting to disable message editing. • Added realm setting to time-limit message editing. Default is 10m. • Added realm setting for default language. • Added year to timestamps in message interstitials for old messages. • Added GitHub authentication (and integrated python-social-auth, so it’s easy to add additional social authenti- cation methods). • Added TERMS_OF_SERVICE setting using markdown formatting to configure the terms of service for a Zulip server. • Added numerous hooks to puppet modules to enable more configurations. • Moved several useful puppet components into the main puppet manifests (setting a redis password, etc.). • Added automatic configuration of postgres/memcached settings based on the server’s available RAM. • Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo. • Added preliminary support for Python 3. All of Zulip’s test suites now pass using Python 3.4. • Added support for Name
5.9. 1.4.1 - 2016-09-03 33 Zulip Documentation, Release 1.7.0
• Added numerous special-purpose settings options. • Added a hex input field in color picker. • Documented new Electron beta app and mobile apps in the /apps/ page. • Enabled Android Google authentication support. • Enhanced logic for tracking origin of user uploads. • Improved error messages for various empty narrows. • Improved missed message emails to better support directly replying. • Increased backend test coverage of Python code to 85.5%. • Increased mypy static type coverage of Python code to 95%. Also fixed many string annotations to properly handle unicode. • Fixed major i18n-related frontend performance regression on /#subscriptions page. Saves several seconds of load time with 1k streams. • Fixed Jinja2 migration bug when trying to register an email that already has an account. • Fixed narrowing to a stream from other pages. • Fixed various frontend strings that weren’t marked for translation. • Fixed several bugs around editing status (/me) messages. • Fixed queue workers not restarting after changes in development. • Fixed Casper tests hanging while development server is running. • Fixed browser autocomplete issue when adding new stream members. • Fixed broken create_stream and rename_stream management commands. • Fixed zulip-puppet-apply exit code when puppet throws errors. • Fixed EPMD restart being attempted on every puppet apply. • Fixed message cache filling; should improve perf after server restart. • Fixed caching race condition when changing user objects. • Fixed buggy puppet configuration for supervisord restarts. • Fixed some error handling race conditions when editing messages. • Fixed fastcgi_params to protect against the httpoxy attack. • Fixed bug preventing users with mit.edu emails from registering accounts. • Fixed incorrect settings docs for the email mirror. • Fixed APNS push notification support (had been broken by Apple changing the APNS API). • Fixed some logic bugs in how attachments are tracked. • Fixed unnecessarily resource-intensive rabbitmq cron checks. • Fixed old deployment directories leaking indefinitely. • Fixed need to manually add localhost in ALLOWED_HOSTS. • Fixed display positioning for the color picker on subscriptions page. • Fixed escaping of Zulip extensions to markdown. • Fixed requiring a reload to see newly uploaded avatars.
34 Chapter 5. Version History Zulip Documentation, Release 1.7.0
• Fixed @all warning firing even for @all. • Restyled password reset form to look nice. • Improved formatting in reset password links. • Improved alert words UI to match style of other settings. • Improved error experience when sending to nonexistent users. • Portions of integrations documentation are now automatically generated. • Restructured the URLs files to be more readable. • Upgraded almost all Python dependencies to current versions. • Substantially expanded and reorganized developer documentation. • Reorganized production documentation and moved to ReadTheDocs. • Reorganized .gitignore type files to be written under var/ • Refactored substantial portions of templates to support subdomains. • Renamed local_settings.py symlink to prod_settings.py for clarity. • Renamed email-mirror management command to email_mirror. • Changed HTTP verb for create_user_backend to PUT. • Eliminated all remaining settings hardcoded for zulip.com. • Eliminated essentially all remaining hardcoding of mit.edu. • Optimized the performance of all the test suites. • Optimized Django memcached configuration. • Removed old prototype data export tool. • Disabled insecure RC4 cipher in nginx configuration. • Enabled shared SSL session cache in nginx configuration. • Updated header for Zulip static assets to reflect Zulip being open source.
5.11 1.3.13 - 2016-06-21
• Added nearly complete internationalization of the Zulip UI. • Added warning when using @all/@everyone. • Added button offering to subscribe at bottom of narrows to streams the user is not subscribed to. • Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT, Transifex, and Updown.io. • Added menu option to mark all messages in a stream or topic as read. • Added new Attachment model to keep track of uploaded files. • Added caching of virtualenvs in development. • Added mypy static type annotations to about 85% of the Zulip Python codebase. • Added automated test of backend templates to test for regressions. • Added lots of detailed documentation on the Zulip development environment.
5.11. 1.3.13 - 2016-06-21 35 Zulip Documentation, Release 1.7.0
• Added setting allowing only administrators to create new streams. • Added button to exit the Zulip tutorial early. • Added web UI for configuring default streams. • Added new OPEN_REALM_CREATION setting (default off), providing a UI for creating additional realms on a Zulip server. • Fixed email_gateway_password secret not working properly. • Fixed missing helper scripts for RabbitMQ Nagios plugins. • Fixed skipping forward to latest messages ("More messages below" button). • Fixed netcat issue causing Zulip installation to hang on Scaleway machines. • Fixed rendering of /me status messages after message editing. • Fixed case sensitivity of right sidebar fading when compose is open. • Fixed error messages when composing to invalid PM recipients. • Fixed LDAP auth backend not working with Zulip mobile apps. • Fixed erroneous WWW-Authenticate headers with expired sessions. • Changed "coworkers" to "users" in the Zulip UI. • Changed add_default_stream REST API to correctly use PUT rather than PATCH. • Updated the Zulip emoji set (the Android Emoji) to a modern version. • Made numerous small improvements to the Zulip development experience. • Migrated backend templates to the faster Jinja2 templating system. • Migrated development environment setup scripts to tools/setup/. • Expanded test coverage for several areas of the product. • Simplified the API for writing new webhook integrations. • Removed most of the remaining JavaScript global variables.
5.12 1.3.12 - 2016-05-10
• CVE-2016-4426: Bot API keys were accessible to other users in the same realm. • CVE-2016-4427: Deactivated users could access messages if SSO was enabled. • Fixed a RabbitMQ configuration bug that resulted in reordered messages. • Added expansive test suite for authentication backends and decorators. • Added an option to logout_all_users to delete only sessions for deactivated users.
5.13 1.3.11 - 2016-05-02
• Moved email digest support into the default Zulip production configuration. • Added options for configuring Postgres, RabbitMQ, Redis, and memcached in settings.py.
36 Chapter 5. Version History Zulip Documentation, Release 1.7.0
• Added documentation on using Hubot to integrate with useful services not yet integrated with Zulip directly (e.g. Google Hangouts). • Added new management command to test sending email from Zulip. • Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations. • Added Nagios plugins to the main distribution. • Added ability for realm administrators to manage custom emoji. • Added guide to writing new integrations. • Enabled camo image proxy to fix mixed-content warnings for http images. • Refactored the Zulip puppet modules to be more modular. • Refactored the Tornado event system, fixing old memory leaks. • Removed many old-style /json API endpoints • Implemented running queue processors multithreaded in development, decreasing RAM requirements for a Zulip development environment from ~1GB to ~300MB. • Fixed rerendering the complete buddy list whenever a user came back from idle, which was a significant perfor- mance issue in larger realms. • Fixed the disabling of desktop notifications from 1.3.7 for new users. • Fixed the (admin) create_user API enforcing restricted_to_domain, even if that setting was disabled for the realm. • Fixed bugs changing certain settings in administration pages. • Fixed collapsing messages in narrowed views. • Fixed 500 errors when uploading a non-image file as an avatar. • Fixed Jira integration incorrectly not @-mentioning assignee.
5.14 1.3.10 - 2016-01-21
• Added new integration for Travis CI. • Added settings option to control maximum file upload size. • Added support for running Zulip development environment in Docker. • Added easy configuration support for a remote postgres database. • Added extensive documentation on scalability, backups, and security. • Recent private message threads are now displayed expanded similar to the pre-existing recent topics feature. • Made it possible to set LDAP and EMAIL_HOST passwords in /etc/zulip/secrets.conf. • Improved the styling for the Administration page and added tabs. • Substantially improved loading performance on slow networks by enabling GZIP compression on more assets. • Changed the page title in narrowed views to include the current narrow. • Fixed several backend performance issues affecting very large realms. • Fixed bugs where draft compose content might be lost when reloading site. • Fixed support for disabling the "zulip" notifications stream.
5.14. 1.3.10 - 2016-01-21 37 Zulip Documentation, Release 1.7.0
• Fixed missing step in postfix_localmail installation instructions. • Fixed several bugs/inconveniences in the production upgrade process. • Fixed realm restrictions for servers with a unique, open realm. • Substantially cleaned up console logging from run-dev.py.
5.15 1.3.9 - 2015-11-16
• Fixed buggy #! lines in upgrade scripts.
5.16 1.3.8 - 2015-11-15
• Added options to the Python api for working with untrusted server certificates. • Added a lot of documentation on the development environment and testing. • Added partial support for translating the Zulip UI. • Migrated installing Node dependencies to use npm. • Fixed LDAP integration breaking autocomplete of @-mentions. • Fixed admin panel reactivation/deactivation of bots. • Fixed inaccurate documentation for downloading the desktop apps. • Fixed various minor bugs in production installation process. • Fixed security issue where recent history on private streams might be visible to new users (to the Zulip team) who were invited with that private stream as one of their initial streams (https://github.com/zulip/zulip/issues/230). • Major preliminary progress towards supporting Python 3.
5.17 1.3.7 - 2015-10-19
• Turn off desktop and audible notifications for streams by default. • Added support for the LDAP authentication integration creating new users. • Added new endpoint to support Google auth on mobile. • Fixed desktop notifications in modern Firefox. • Fixed several installation issues for both production and development environments. • Improved documentation for outgoing SMTP and the email mirror integration.
38 Chapter 5. Version History CHAPTER 6
Zulip in production
To play around with Zulip and see what it looks like, you can install a dev environment or check out the Zulip development community server. If you like what you see, you can set up Zulip for your team by installing a production Zulip server. These pages will walk you through how.
6.1 Requirements
You’ll need a few things to run a production Zulip server. Key requirements include: • a dedicated server or VM, running Ubuntu, with at least 2GB of RAM (or 4GB for a large site); • a valid DNS name and SSL certificates; • a way to send outgoing email. See the requirements page for more details, including free options for SSL and for outgoing email if you don’t have those already.
6.2 Install
Follow the install instructions. You’ll download the built release tarball, run the Zulip install script, and configure a handful of required settings; then create your Zulip organization through your new server’s web interface.
6.3 Running
You now have a running Zulip install! • Read our advice on helping your community make the most of Zulip. • Customize Zulip to your needs.
39 Zulip Documentation, Release 1.7.0
• Read about Zulip’s support for backups, monitoring, and other important production considerations.
40 Chapter 6. Zulip in production CHAPTER 7
Requirements
Note that if you just want to play around with Zulip and see what it looks like, it is easier to install it in a development environment following these instructions, since then you don’t need to worry about setting up SSL certificates and an authentication mechanism. Or, you can check out the Zulip development community server.
7.1 Server
7.1.1 General
The installer expects Zulip to be the only thing running on the system; it will install system packages with apt (like nginx, postgresql, and redis) and configure them for its own use. We strongly recommend using either a fresh machine instance in a cloud provider, a fresh VM, or a dedicated machine. If you decide to disregard our advice and use a server that hosts other services, we can’t support you, but we do have some notes on issues you’ll encounter.
7.1.2 Operating System
Ubuntu 16.04 Xenial and Ubuntu 14.04 Trusty are supported for running Zulip in production. 64-bit is recommended. We recommend Xenial if you have a choice, since 14.04 is approaching end-of-life and you’ll save yourself the work of upgrading in a few months.
7.1.3 Hardware Specifications
• CPU and Memory: For installations with 100+ users you’ll need a minimum of 2 CPUs and 4GB RAM. For installations with fewer users, 1 CPU and 2GB RAM is sufficient. We strongly recommend against installing with less than 2GB of RAM, as you will likely experience out of memory issues installing dependencies. We recommend against using highly CPU-limited servers like the AWS t2 style instances for organizations with a hundreds of users (active or no). See our documentation on scalability for advice on hardware requirements for larger organizations.
41 Zulip Documentation, Release 1.7.0
• Disk space: You’ll need at least 10GB of free disk space for a server with dozens of users. If you intend to store uploaded files locally rather than on S3 you will likely need more, depending how often your users upload large files. You’ll eventually need 100GB or more if you have thousands of active users or millions of total messages sent.
7.1.4 Network and Security Specifications
• Incoming HTTPS access (usually port 443, though this is configurable) from the networks where your users are (usually, the public Internet). If you also open port 80, Zulip will redirect users to HTTPS rather than not working when users type e.g. http://zulip.example.com in their browser. If you are using Zulip’s incoming email integration you may also need incoming port 25 open. • Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so that Zulip can properly manage inline image previews. You’ll also need outgoing SMTP access to your SMTP server (the standard port for this is 587) so that Zulip can send email.
7.1.5 Domain name
You should already have a domain name available for your Zulip production instance. In order to generate valid SSL certificates with Let’s Encrypt, and to enable other services such as Google Authentication, you’ll need to update the domain’s A record to point to your production server.
7.2 Credentials needed
7.2.1 SSL Certificate
• An SSL certificate for the host you’re putting this on (e.g., zulip.example.com). If you don’t have an SSL solution already, read about getting an SSL certificate for free using Let’s Encrypt.
7.2.2 Outgoing email
• Outgoing email (SMTP) credentials that Zulip can use to send outgoing emails to users (e.g. email address confirmation emails during the signup process, missed message notifications, password reset, etc.). If you don’t have an existing outgoing SMTP solution, read about free outgoing SMTP options and options for prototyping. Once you have met these requirements, see full instructions for installing Zulip in production.
42 Chapter 7. Requirements CHAPTER 8
Production Installation
Make sure you want to install a Zulip production server. If you’d instead like to test or develop a new feature, we recommend the Zulip server development environment instead. You will need an Ubuntu system that satisfies the installation requirements. In short, you need: • Either a dedicated machine, or a fresh VM on an existing machine. • Ubuntu 16.04 Xenial or Ubuntu 14.04 Trusty, 64-bit. If you have a choice, install on Xenial, since Trusty is approaching its end-of-life and you’ll save yourself the work of upgrading a production installation. • At least 2GB RAM and 10 GB disk space (4GB and 2 CPUs recommended for 100+ users). • A DNS name, an SSL certificate, and credentials for sending email.
8.1 Step 1: Install SSL Certificates
Zulip runs over https only, and requires SSL certificates in order to work. It looks for the certificates in /etc/ ssl/private/zulip.key and /etc/ssl/certs/zulip.combined-chain.crt. Note that Zulip uses nginx as its webserver and thus expects a chained certificate bundle. If you need an SSL certificate, see our SSL certificate documentation. If you already have an SSL certificate, just install (or symlink) it into place at the above paths, and move on to the next step.
8.2 Step 2: Download and install latest release
Download and unpack the latest built server tarball with the following commands: cd $(mktemp -d) wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz tar -xf zulip-server-latest.tar.gz
43 Zulip Documentation, Release 1.7.0
If you’d like to verify the download, we publish the sha256sums of our release tarballs. Then, run the Zulip install script:
sudo-s # If not already root ./zulip-server-*/scripts/setup/install
This may take a while to run, since it will install a large number of dependencies. The Zulip install script is designed to be idempotent, so if it fails, you can just rerun it after correcting the issue that caused it to fail. Also note that it automatically logs a transcript to /var/log/zulip/install.log; please include a copy of that file in any bug reports. The install script does several things: • Creates zulip user, which the various Zulip servers will run as, • Creates /home/zulip/deployments/, which the Zulip code for this deployment (and future deployments when you upgrade) go into. At the very end of the install process, the script moves the Zulip code tree it’s running from (which you unpacked from a tarball above) to a directory there, and makes /home/zulip/ deployments/current as a symbolic link to it. • Installs Zulip’s various dependencies. • Configures the various third-party services Zulip uses, including Postgres, RabbitMQ, Memcached and Redis.
8.3 Step 3: Configure Zulip
Configure the Zulip server instance by editing /etc/zulip/settings.py and providing values for the manda- tory settings, which are all found under the heading ### MANDATORY SETTINGS. These settings include: • EXTERNAL_HOST: the user-accessible Zulip domain name for your Zulip installation (i.e., what users will type in their web browser). This should of course match the DNS name you configured to point to your server and for which you configured SSL certificates. If you plan to use multiple domains, add the others to ALLOWED_HOSTS. • ZULIP_ADMINISTRATOR: the email address of the person or team maintaining this installation and who will get support and error emails. • EMAIL_HOST, EMAIL_HOST_USER: credentials for an outgoing email (aka "SMTP") server that Zulip can use to send emails. See our guide for outgoing email for help configuring this. • If desired, you can also configure additional authentication backends while you’re editing /etc/zulip/settings.py. Note, however, that the default (email) backend must be enabled when you complete Step 6 (creating an organi- zation) below.
8.4 Step 4: Test email configuration
Test your outgoing email configuration. This is important to test now, because email configuration errors are common, and your outgoing email configuration needs to be working in order for you to complete the installation.
8.5 Step 5: Initialize Zulip database
At this point, you are done doing things as root. The remaining commands are run as the zulip user. Change to the zulip user and initialize the Zulip database for your production install:
44 Chapter 8. Production Installation Zulip Documentation, Release 1.7.0
su zulip # If you weren't already the zulip user /home/zulip/deployments/current/scripts/setup/initialize-database
The initialize-database script will report an error if you did not fill in all the mandatory settings from /etc/ zulip/settings.py. It is safe to rerun it after correcting the problem if that happens. This completes the process of installing Zulip on your server. However, in order to use Zulip, you’ll need to create an organization in your Zulip installation.
8.6 Step 6: Create a Zulip organization and login
• Run the organization (realm) creation management command:
su zulip # If you weren't already the zulip user /home/zulip/deployments/current/manage.py generate_realm_creation_link
This will print out a secure one-time-use link that allows creation of a new Zulip organization on your server. • Open the generated link with your web browser. You’ll see the "Create organization" page (screenshot here). Enter your email address and click Create organization. • Check your email to find the confirmation email and click the link. You’ll be prompted to finish setting up your organization and initial administrator user (screenshot here). Complete this form and log in! Congratulations! You are logged in as an organization administrator for your new Zulip organization.
8.7 Step 7: Next steps
• Subscribe to the extremely low-traffic Zulip announcements email list to get important announcements for Zulip server administrators about new releases, security issues, etc. • Follow Zulip on Twitter to get Zulip news. • Learn how to setup your new Zulip organization. • Learn how further configure your Zulip server. • Learn about maintaining a production Zulip server.
8.8 Troubleshooting
• The zulip user’s password. By default, the zulip user doesn’t have a password, and is intended to be accessed by su zulip from the root user (or via SSH keys or a password, if you want to set those up, but that’s up to you as the system administrator). Most people who are prompted for a password when running su zulip turn out to already have switched to the zulip user earlier in their session, and can just skip that step. • If you get an error after scripts/setup/install completes, check the bottom of /var/log/zulip/ errors.log for a traceback, and consult the troubleshooting section for advice on how to debug. • If that doesn’t help, please visit #production help in the Zulip development community server for realtime help or email [email protected] with the full traceback, and we’ll try to help you out! Please provide details like the full traceback from the bottom of /var/log/zulip/errors.log in your report.
8.6. Step 6: Create a Zulip organization and login 45 Zulip Documentation, Release 1.7.0
46 Chapter 8. Production Installation CHAPTER 9
Troubleshooting
Zulip uses Supervisor to monitor and control its many Python services. Read the next section, Using supervisorctl, to learn how to use the Supervisor client to monitor and manage services. If you haven’t already, now might be a good time to read Zulip’s architectural overview, particularly the Components section. This will help you understand the many services Zulip uses. If you encounter issues while running Zulip, take a look at Zulip’s logs, which are located in /var/log/zulip/. That directory contains one log file for each service, plus errors.log (has all errors), server.log (has logs from the Django and Tornado servers), and workers.log (has combined logs from the queue workers). The section troubleshooting services on this page includes details about how to fix common issues with Zulip services. If you run into additional problems, please report them so that we can update this page! The Zulip installation scripts logs its full output to /var/log/zulip/install.log, so please include the context for any tracebacks from that log.
9.1 Using supervisorctl
To see what Zulip-related services are configured to use Supervisor, look at /etc/supervisor/conf.d/ zulip.conf and /etc/supervisor/conf.d/zulip-db.conf. Use the supervisor client supervisorctl to list the status of, stop, start, and restart various services.
9.1.1 Checking status with supervisorctl status
You can check if the zulip application is running using: supervisorctl status
When everything is running as expected, you will see something like this:
47 Zulip Documentation, Release 1.7.0
process-fts-updates RUNNING pid 2194,
˓→uptime1:13:11 zulip-django RUNNING pid 2192,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-0 RUNNING pid 2209,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-1 RUNNING pid 2210,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-2 RUNNING pid 2211,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-3 RUNNING pid 2212,
˓→uptime1:13:11 zulip-senders:zulip-events-message_sender-4 RUNNING pid 2208,
˓→uptime1:13:11 zulip-tornado RUNNING pid 2193,
˓→uptime1:13:11 zulip-workers:zulip-deliver-enqueued-emails STARTING zulip-workers:zulip-events-confirmation-emails RUNNING pid 2199,
˓→uptime1:13:11 zulip-workers:zulip-events-digest_emails RUNNING pid 2205,
˓→uptime1:13:11 zulip-workers:zulip-events-email_mirror RUNNING pid 2203,
˓→uptime1:13:11 zulip-workers:zulip-events-error_reports RUNNING pid 2200,
˓→uptime1:13:11 zulip-workers:zulip-events-feedback_messages RUNNING pid 2207,
˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_mobile_notifications RUNNING pid 2204,
˓→uptime1:13:11 zulip-workers:zulip-events-missedmessage_reminders RUNNING pid 2206,
˓→uptime1:13:11 zulip-workers:zulip-events-signups RUNNING pid 2198,
˓→uptime1:13:11 zulip-workers:zulip-events-slowqueries RUNNING pid 2202,
˓→uptime1:13:11 zulip-workers:zulip-events-user-activity RUNNING pid 2197,
˓→uptime1:13:11 zulip-workers:zulip-events-user-activity-interval RUNNING pid 2196,
˓→uptime1:13:11 zulip-workers:zulip-events-user-presence RUNNING pid 2195,
˓→uptime1:13:11
9.1.2 Restarting services with supervisorctl restart all
After you change configuration in /etc/zulip/settings.py or fix a misconfiguration, you will often want to restart the Zulip application. You can restart Zulip using: supervisorctl restart all
9.1.3 Stopping services with supervisorctl stop all
Similarly, you can stop Zulip using:
48 Chapter 9. Troubleshooting Zulip Documentation, Release 1.7.0
supervisorctl stop all
9.2 Troubleshooting services
The Zulip application uses several major open source services to store and cache data, queue messages, and otherwise support the Zulip application: • postgresql • rabbitmq-server • nginx • redis • memcached If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some common configuration problems and how to resolve them: • An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running; to fix this, try:
service rabbitmq-server restart
If RabbitMQ fails to start, the problem is often that you are using a virtual machine with broken DNS configu- ration; you can often correct this by configuring /etc/hosts properly. • If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus failed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. To fix this, configure them properly and then run:
service nginx restart
• If your host is being port scanned by unauthorized users, you may see messages in /var/log/zulip/ server.log like
2017-02-22 14:11:33,537 ERROR Invalid HTTP_HOST header:'10.2.3.4'. You may need
˓→to addu'10.2.3.4' to ALLOWED_HOSTS.
Django uses the hostnames configured in ALLOWED_HOSTS to identify legitimate requests and block others. When an incoming request does not have the correct HTTP Host header, Django rejects it and logs the attempt. For more on this issue, see the Django release notes on Host header poisoning Next: Making your Zulip instance awesome.
9.2. Troubleshooting services 49 Zulip Documentation, Release 1.7.0
50 Chapter 9. Troubleshooting CHAPTER 10
Customize Zulip
Once you’ve got Zulip setup, you’ll likely want to configure it the way you like.
10.1 Making changes
Most configuration can be done by a realm administrator, on the web. For those settings, see the documentation for realm administrators. This page discusses additional configuration that a system administrator can do. To change any of the following set- tings, edit the /etc/zulip/settings.py file on your Zulip server, and then restart the server with the following command:
su zulip-c/home/zulip/deployments/current/scripts/restart-server
10.2 Specific settings
10.2.1 Authentication Backends
AUTHENTICATION_BACKENDS is a list of enabled authentication mechanisms. By default the email backend is enabled. If you want an additional or different authentication backend, you will need to uncomment one or more and then do any additional configuration required for that backend as documented in the settings.py file. See the section on Authentication for more detail on the available authentication backends and how to configure them.
10.2.2 Mobile and desktop apps
The Zulip apps expect to be talking to to servers with a properly signed SSL certificate, in most cases and will not accept a self-signed certificate. You should get a proper SSL certificate before testing the apps.
51 Zulip Documentation, Release 1.7.0
Because of how Google and Apple have architected the security model of their push notification protocols, the Zulip mobile apps for iOS and Android can only receive push notifications from a single Zulip server. We have configured that server to be push.zulipchat.com, and offer a push notification forwarding service that forwards push noti- fications through our servers to mobile devices. Read the linked documentation for instructions on how to register for and configure this service. By the end of summer 2017, all of the Zulip apps will have full support for multiple accounts, potentially on different Zulip servers, with a convenient UI for switching between them.
10.2.3 Terms of Service and Privacy policy
Zulip allows you to configure your server’s Terms of Service and Privacy Policy pages (/terms and /privacy, respectively). You can use the TERMS_OF_SERVICE and PRIVACY_POLICY settings to configure the path to your server’s policies. The syntax is Markdown (with support for included HTML). A good approach is to use paths like /etc/zulip/terms.md, so that it’s easy to back up your policy configuration along with your other Zulip server configuration.
10.2.4 Miscellaneous server settings
Zulip has dozens of settings documented in the comments in /etc/zulip/settings.py; you can review the latest version of the settings.py template if you’ve deleted the comments or want to check if new settings have been added in more recent versions of Zulip. Since Zulip’s settings file is a Python script, there are a number of other things that one can configure that are not documented; ask on chat.zulip.org if there’s something you’d like to do but can’t figure out how to. Some popular settings in /etc/zulip/settings.py include: • The Twitter integration, which provides pretty inline previews of tweets. • The email gateway, which lets users send emails into Zulip. • INLINE_URL_EMBED_PREVIEW, which controls our experimental feature providing inline previews of links pasted into Zulip.
10.3 Zulip announcement list
If you haven’t already, subscribe to the zulip-announce list so that you can receive important announces like new Zulip releases or major changes to the app ecosystem..
10.4 Enjoy your Zulip installation!
If you discover things that you wish had been documented, please contribute documentation suggestions either via a GitHub issue or pull request; we love even small contributions, and we’d love to make the Zulip documentation cover everything anyone might want to know about running Zulip in production. Next: Maintaining and upgrading Zulip in production.
52 Chapter 10. Customize Zulip CHAPTER 11
Mobile push notification service
Zulip’s iOS and Android mobile apps support receiving push notifications from Zulip servers to let users know when new messages have arrived. This is an important feature to having a great experience using the Zulip mobile apps. For technical reasons (explained below), in order to deliver mobile push notifications in the app store versions of our mobile apps, you will need to register your Zulip server with the Zulip mobile push notification service. This service will forward push notifications generated by your server to the Zulip mobile app automatically.
11.1 How to sign up
Starting with Zulip 1.6 for both Android and iOS, Zulip servers support forwarding push notifications to a central push notification forwarding service. You can enable this for your Zulip server as follows: 1. First, contact [email protected] with the zulip_org_id and zulip_org_key values from your / etc/zulip/zulip-secrets.conf file, as well as a hostname and contact email address you’d like us to use in case of any issues (we hope to have a nice web flow available for this soon). 2. We’ll enable push notifications for your server on our end. Look for a reply from Zulipchat support within 24 hours. 3. Uncomment the PUSH_NOTIFICATION_BOUNCER_URL = "https://push.zulipchat.com" line in your /etc/zulip/settings.py file, and restart your Zulip server. Note that if you installed Zulip older than 1.6, you’ll need to add the line (it won’t be there to uncomment). That should be all you need to do! If you’d like to verify the full pipeline, you can do the following. Please follow the instructions carefully: • Configure mobile push notifications to always be sent (normally they’re only sent if you’re idle, which isn’t ideal for this sort of testing). • On an Android device, download and login to the Zulip Android app. If you were already logged in before configuring the server, you’ll need to logout first, since the app only registers for push notifications on login.
53 Zulip Documentation, Release 1.7.0
• Hit the home button, so Zulip is running in the background, and then have another user send you a private message (By default, Zulip only sends push notifications for private messages sent by other users and messages mentioning you). A push notification should appear in the Android notification area. Note that use of the push notification bouncer is subject to the Zulipchat Terms of Service. By using push notifications, you agree to those terms.
11.2 Why this is necessary
Both Google’s and Apple’s push notification services have a security model that does not support mutually untrusted self-hosted servers sending push notifications to the same app. In particular, when an app is published to their respec- tive app stores, one must compile into the app a secret corresponding to the server that will be able to publish push notifications for the app. This means that it is impossible for a single app in their stores to receive push notifications from multiple, mutually untrusted, servers. Zulip’s solution to this problem is to provide a central push notification forwarding service, which allows registered Zulip servers to send push notifications to the Zulip app indirectly (through the forwarding service).
11.3 Security and privacy implications
We’ve designed this push notification bouncer service with security and privacy in mind: • All of the network requests (both from Zulip servers to the Push Notification Service and from the Push Notifi- cation Service to the relevant Google and Apple services) are encrypted over the wire with SSL/TLS. • The code for the push notification forwarding service is 100% open source and available as part of the Zulip server project on GitHub. The Push Notification Service is designed to avoid any message content being stored or logged, even in error cases. • The push notification forwarding servers are professionally managed by a small team of security experts. • There’s a PUSH_NOTIFICATION_REDACT_CONTENT setting available to disable any message content being sent via the push notification bouncer (i.e. message content will be replaced with ***REDACTED***). Note that this setting makes push notifications significantly less usable. We plan to replace this feature with end-to- end encryption which would eliminate that usability tradeoff. If you have any questions about the security model, contact [email protected].
54 Chapter 11. Mobile push notification service CHAPTER 12
Secure, maintain, and upgrade
This page covers topics that will help you maintain a healthy, up-to-date, and secure Zulip installation, including: • Upgrading • Upgrading from a git repository • Backups • Monitoring • Scalability • Management commands You may also want to read this related content: • Security Model
12.1 Upgrading
We recommend reading this entire section before doing your first upgrade. To upgrade to a new version of the zulip server, download the appropriate release tarball from https://www.zulip.org/ dist/releases/. You also have the option of creating your own release tarballs from a copy of the zulip.git repository using tools/ build-release-tarball or upgrade Zulip to a version in a Git repository directly. Next, run as root:
/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
The upgrade process will shut down the Zulip service and then run apt-get upgrade, a puppet apply, any database migrations, and then bring the Zulip service back up. Upgrading will result in some brief downtime for the service, which should be under 30 seconds unless there is an expensive transition involved. Unless you have tested the upgrade in advance, we recommend doing upgrades at off hours.
55 Zulip Documentation, Release 1.7.0
Note that upgrading an existing Zulip production server from Ubuntu 14.04 Trusty to Ubuntu 16.04 Xenial will require significant manual intervention on your part to migrate the data in the database from Postgres 9.3 to Postgres 9.5. Contributions on testing and documenting this process are welcome!
12.1.1 Preserving local changes to configuration files
Warning: If you have modified configuration files installed by Zulip (e.g. the nginx configuration), the Zulip upgrade process will overwrite your configuration when it does the puppet apply. You can test whether this will happen assuming no upstream changes to the configuration using scripts/ zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changes it would make. Using this list, you can save a copy of any files that you’ve modified, do the upgrade, and then restore your configuration. If you need to do this, please report the issue so that we can make the Zulip puppet configuration flexible enough to handle your setup.
12.1.2 Troubleshooting with the upgrade log
The Zulip upgrade script automatically logs output to /var/log/zulip/upgrade.log. Please use those logs to include output that shows all errors in any bug reports. After the upgrade, we recommend checking /var/log/zulip/errors.log to confirm that your users are not experiencing errors after the upgrade.
12.1.3 Rolling back to a prior version
The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/ contain- ing a complete copy of the Zulip server code, and then moving the symlinks at /home/zulip/deployments/ {current,last,next} as part of the upgrade process. This means that if the new version isn’t working, you can quickly downgrade to the old version by running /home/ zulip/deployments/last/scripts/restart-server, or to an earlier previous version by running / home/zulip/deployments/DATE/scripts/restart-server. The restart-server script stops any running Zulip server, and starts the version corresponding to the restart-server path you call.
12.1.4 Updating settings
If required, you can update your settings by editing /etc/zulip/settings.py and then run /home/zulip/ deployments/current/scripts/restart-server to restart the server.
12.1.5 Applying Ubuntu system updates
The Zulip upgrade script will automatically run apt-get update and then apt-get upgrade. If you’d like to minimize downtime, you should do an apt upgrade before running the Zulip upgrade script (and then restart the server and check everything is working) before running the upgrade script. Also note that you are responsible for running this on your system on a regular basis between Zulip upgrades to ensure that your system is up to date with the latest upstream security patches.
56 Chapter 12. Secure, maintain, and upgrade Zulip Documentation, Release 1.7.0
12.1.6 API and your Zulip URL
To use the Zulip API with your Zulip server, you will need to use the API endpoint of e.g. https:// zulip.example.com/api. Our Python API example scripts support this via the --site=https://zulip. example.com argument. The API bindings support it via putting site=https://zulip.example.com in your .zuliprc. Every Zulip integration supports this sort of argument (or e.g. a ZULIP_SITE variable in a zuliprc file or the en- vironment), but this is not yet documented for some of the integrations (the included integration documentation on /integrations will properly document how to do this for most integrations). We welcome pull requests for integrations that don’t discuss this! Similarly, you will need to instruct your users to specify the URL for your Zulip server when using the Zulip desktop and mobile apps.
12.1.7 Memory leak mitigation
As a measure to mitigate the impact of potential memory leaks in one of the Zulip daemons, the service automatically restarts itself every Sunday early morning. See /etc/cron.d/restart-zulip for the precise configuration.
12.2 Upgrading from a git repository
Starting with version 1.4, the Zulip server supports upgrading to a commit in Git. You can configure the git repository that you’d like to use by adding a section like this to /etc/zulip/zulip.conf; by default it uses the main zulip repository (shown below).
[deployment] git_repo_url= https://github.com/zulip/zulip.git
Once that is done (and assuming the currently installed version of Zulip is 1.7 or newer), you can do deployments by running as root:
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git
12.2.1 Upgrading from Zulip 1.6 and older
If you’re currently using Zulip older than 1.7, you will need to add zulip::static_asset_compiler to your /etc/zulip/zulip.conf file’s puppet_classes entry, like this: puppet_classes= zulip::voyager, zulip::static_asset_compiler
Then, run scripts/zulip-puppet-apply to install the dependencies for building Zulip’s static assets. Once you’ve upgraded to Zulip 1.7 or above, you can safely remove zulip::static_asset_compiler from puppet_classes to clean it up; in Zulip 1.7 and above, it is a dependency of zulip::voyager.
12.2. Upgrading from a git repository 57 Zulip Documentation, Release 1.7.0
12.3 Backups
There are several pieces of data that you might want to back up: • The postgres database. That you can back up like any postgres database; we have some example tool- ing for doing that incrementally into S3 using wal-e in puppet/zulip_internal/manifests/ postgres_common.pp (that’s what we use for zulip.com’s database backups). Note that this module isn’t part of the Zulip server releases since it’s part of the zulip.com configuration (see https://github.com/zulip/zulip/ issues/293 for a ticket about fixing this to make life easier for running backups). • Any user-uploaded files. If you’re using S3 as storage for file uploads, this is backed up in S3, but if you have instead set LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars) will be stored in that directory and you’ll want to back it up. • Your Zulip configuration including secrets from /etc/zulip/. E.g. if you lose the value of secret_key, all users will need to login again when you setup a replacement server since you won’t be able to verify their cookies; if you lose avatar_salt, any user-uploaded avatars will need to be re-uploaded (since avatar file- names are computed using a hash of avatar_salt and user’s email), etc. • The logs under /var/log/zulip can be handy to have backed up, but they do get large on a busy server, and it’s definitely lower-priority. If you are interested in backups because you are moving from one Zulip server to another server and can’t transfer a full postgres dump (which is definitely the simplest approach), our draft conversion and export design document may help. The tool is well designed and was tested carefully with dozens of realms as of mid-2016 but is not integrated into Zulip’s regular testing process, and thus it is worth asking on the Zulip developers mailing list whether it needs any minor updates to do things like export newly added tables.
12.3.1 Restore from backups
To restore from backups, the process is basically the reverse of the above: • Install new server as normal by downloading a Zulip release tarball and then using scripts/setup/ install, you don’t need to run the initialize-database second stage which puts default data into the database. • Unpack to /etc/zulip the settings.py and zulip-secrets.conf files from your backups. • Restore your database from the backup using wal-e; if you ran initialize-database anyway above, you’ll want to first scripts/setup/postgres-init-db to drop the initial database first. • Reconfigure rabbitmq to use the password from secrets.conf by running, as root, scripts/setup/ configure-rabbitmq. • If you’re using local file uploads, restore those files to the path specified by settings. LOCAL_UPLOADS_DIR and (if appropriate) any logs. • Start the server using scripts/restart-server. This restoration process can also be used to migrate a Zulip installation from one server to another. We recommend running a disaster recovery after you setup backups to confirm that your backups are working; you may also want to monitor that they are up to date using the Nagios plugin at: puppet/zulip_internal/files/ nagios_plugins/check_postgres_backup. Contributions to more fully automate this process or make this section of the guide much more explicit and detailed are very welcome!
58 Chapter 12. Secure, maintain, and upgrade Zulip Documentation, Release 1.7.0
12.3.2 Postgres streaming replication
Zulip has database configuration for using Postgres streaming replication; you can see the configuration in these files: • puppet/zulip_internal/manifests/postgres_slave.pp • puppet/zulip_internal/manifests/postgres_master.pp • puppet/zulip_internal/files/postgresql/* Contribution of a step-by-step guide for setting this up (and moving this configuration to be available in the main puppet/zulip/ tree) would be very welcome!
12.4 Monitoring
The complete Nagios configuration (sans secret keys) used to monitor zulip.com is available under puppet/ zulip_internal in the Zulip Git repository (those files are not installed in the release tarballs). The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirec- tories under /usr/lib/nagios/plugins/. The following is a summary of the various Nagios plugins included with Zulip and what they check: Application server and queue worker monitoring: • check_send_receive_time (sends a test message through the system between two bot users to check that end-to-end message sending works) • check_rabbitmq_consumers and check_rabbitmq_queues (checks for rabbitmq being down or the queue workers being behind) • check_queue_worker_errors (checks for errors reported by the queue workers) • check_worker_memory (monitors for memory leaks in queue workers) • check_email_deliverer_backlog and check_email_deliverer_process (monitors for whether outgoing emails are being sent) Database monitoring: • check_postgres_replication_lag (checks streaming replication is up to date). • check_postgres (checks the health of the postgres database) • check_postgres_backup (checks backups are up to date; see above) • check_fts_update_log (monitors for whether full-text search updates are being processed) Standard server monitoring: • check_website_response.sh (standard HTTP check) • check_debian_packages (checks apt repository is up to date) If you’re using these plugins, bug reports and pull requests to make it easier to monitor Zulip and maintain it in production are encouraged!
12.5 Scalability
This section attempts to address the considerations involved with running Zulip with larger teams (especially >1000 users).
12.4. Monitoring 59 Zulip Documentation, Release 1.7.0
• For an organization with 100+ users, it’s important to have more than 4GB of RAM on the system. Zulip will install on a system with 2GB of RAM, but with less than 3.5GB of RAM, it will run its queue processors multithreaded to conserve memory; this creates a significant performance bottleneck. • chat.zulip.org, with thousands of user accounts and thousands of messages sent every week, has 8GB of RAM, 4 cores, and 80GB of disk. The CPUs are essentially always idle, but the 8GB of RAM is important. • We recommend using a remote postgres database for isolation, though it is not required. In the following, we discuss a relatively simple configuration with two types of servers: application servers (running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and database servers. • You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) with just a single reasonably large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB of RAM) sitting mostly idle (<10% CPU used and only 4GB of the 16GB RAM actively in use). You can probably get away with half that (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale. Beyond a 1000 active users, you will eventually want to increase the memory cap in memcached.conf from the default 512MB to avoid high rates of memcached misses. • For the database server, we highly recommend SSD disks, and RAM is the primary resource limitation. We have not aggressively tested for the minimum resources required, but 8 cores with 30GB of RAM (e.g. AWS’s m3.2xlarge) should suffice; you may be able to get away with less especially on the CPU side. The database load per user is pretty optimized as long as memcached is working correctly. This has not been tested, but from extrapolating the load profile, it should be possible to scale a Zulip installation to 10,000s of active users using a single large database server without doing anything complicated like sharding the database. • For reasonably high availability, it’s easy to run a hot spare application server and a hot spare database (using Postgres streaming replication; see the section on configuring this). Be sure to check out the section on backups if you’re hoping to run a spare application server; in particular you probably want to use the S3 backend for storing user-uploaded files and avatars and will want to make sure secrets are available on the hot spare. • Zulip does not support dividing traffic for a given Zulip realm between multiple application servers. There are two issues: you need to share the memcached/Redis/RabbitMQ instance (these should can be moved to a network service shared by multiple servers with a bit of configuration) and the Tornado event system for pushing to browsers currently has no mechanism for multiple frontend servers (or event processes) talking to each other. One can probably get a factor of 10 in a single server’s scalability by supporting multiple tornado processes on a single server, which is also likely the first part of any project to support exchanging events amongst multiple servers. Questions, concerns, and bug reports about this area of Zulip are very welcome! This is an area we are hoping to improve.
12.6 Securing your Zulip server
Zulip’s security model is discussed in a separate document.
12.7 Management commands
Zulip has a large library of Django management commands. To use them, you will want to be logged in as the zulip user and for the purposes of this documentation, we assume the current working directory is /home/zulip/ deployments/current. Below, we show several useful examples, but there are more than 100 in total. We recommend skimming the usage docs (or if there are none, the code) of a management command before using it, since they are generally less polished and more designed for expert use than the rest of the Zulip system.
60 Chapter 12. Secure, maintain, and upgrade Zulip Documentation, Release 1.7.0
12.7.1 manage.py shell
You can get an iPython shell with full access to code within the Zulip project using manage.py shell, e.g., you can do the following to change an email address:
$ /home/zulip/deployments/current/manage.py shell In [1]: user_profile = get_user_profile_by_email("[email protected]") In [2]: do_change_user_email(user_profile, "[email protected]")
manage.py dbshell
This will start a postgres shell connected to the Zulip database.
12.7.2 Grant administrator access
You can make any user a realm administrator on the command line with the knight management command:
./manage.py knight [email protected]
Creating API super users with manage.py
If you need to manage the IRC, Jabber, or Zephyr mirrors, you will need to create API super users. To do this, use ./manage.py knight with the --permission=api_super_user argument. See the respective integration scripts for these mirrors (under zulip/integrations/ in the Zulip Python API repo) for further detail on these.
Exporting users and realms with manage.py export
If you need to do an export of a single user or of an entire realm, we have tools in management/ that essentially export Zulip data to the file system. export_single_user.py exports the message history and realm-public metadata for a single Zulip user (includ- ing that user’s received messages as well as their sent messages). A good overview of the process for exporting a single realm when moving a realm to a new server (without moving a full database dump) is in management/export.py. We recommend you read the comment there for words of wisdom on speed, what is and is not exported, what will break upon a move to a new server, and suggested procedure.
12.7.3 Other useful manage.py commands
There are a large number of useful management commands under zerver/management/commands/; you can also see them listed using ./manage.py with no arguments.
12.8 Hosting multiple Zulip organizations
This is explained in detail on its own page.
12.8. Hosting multiple Zulip organizations 61 Zulip Documentation, Release 1.7.0
62 Chapter 12. Secure, maintain, and upgrade CHAPTER 13
Security Model
This section attempts to document the Zulip security model. Since this is new documentation, it likely does not cover every issue; if there are details you’re curious about, please feel free to ask questions on the Zulip development mailing list (or if you think you’ve found a security bug, please report it to [email protected] so we can do a responsible security announcement).
13.1 Secure your Zulip server like your email server
• It’s reasonable to think about security for a Zulip server like you do security for a team email server – only trusted administrators within an organization should have shell access to the server. In particular, anyone with root access to a Zulip application server or Zulip database server, or with access to the zulip user on a Zulip application server, has complete control over the Zulip installation and all of its data (so they can read messages, modify history, etc.). It would be difficult or impossible to avoid this, because the server needs access to the data to support features expected of a group chat system like the ability to search the entire message history, and thus someone with control over the server has access to that data as well.
13.2 Encryption and Authentication
• Traffic between clients (web, desktop and mobile) and the Zulip is encrypted using HTTPS. By default, all Zulip services talk to each other either via a localhost connection or using an encrypted SSL connection. • Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks. • The preferred way to login to Zulip is using an SSO solution like Google Auth, LDAP, or similar, but Zulip also supports password authentication. See the authentication methods documentation for details on Zulip’s available authentication methods.
13.2.1 Passwords
Zulip stores user passwords using the standard PBKDF2 algorithm.
63 Zulip Documentation, Release 1.7.0
When the user is choosing a password, Zulip checks the password’s strength using the popular zxcvbn library. Weak passwords are rejected, and strong passwords encouraged. The minimum password strength allowed is controlled by two settings in /etc/zulip/settings.py: • PASSWORD_MIN_LENGTH: The minimum acceptable length, in characters. Shorter passwords are rejected even if they pass the zxcvbn test controlled by PASSWORD_MIN_GUESSES. • PASSWORD_MIN_GUESSES: The minimum acceptable strength of the password, in terms of the estimated number of passwords an attacker is likely to guess before trying this one. If the user attempts to set a pass- word that zxcvbn estimates to be guessable in less than PASSWORD_MIN_GUESSES, then Zulip rejects the password. By default, PASSWORD_MIN_GUESSES is 10000. This provides significant protection against online attacks, while limiting the burden imposed on users choosing a password. Estimating the guessability of a password is a complex problem and impossible to efficiently do perfectly. For background or when considering an alternate value for this setting, the article "Passwords and the Evolution of Imperfect Authentication" is recommended. The 2016 zxcvbn paper adds useful information about the perfor- mance of zxcvbn, and a large 2012 study of Yahoo users is informative about the strength of the passwords users choose.
13.3 Messages and History
• Zulip message content is rendered using a specialized Markdown parser which escapes content to protect against cross-site scripting attacks. • Zulip supports both public streams and private ("invite-only") streams. Any Zulip user can join any public stream in the realm, and can view the complete message history of any public stream without joining the stream. • A private ("invite-only") stream is hidden from users who are not subscribed to the stream. Users who are not members of a private stream cannot read messages on the stream, send messages to the stream, or join the stream, even if they are a Zulip realm administrator. Users can join private streams only when they are invited. However, any member of a private stream can invite other users to the stream. When a new user joins a private stream, they can see future messages sent to the stream, but they do not receive access to the stream’s message history. • Zulip supports editing the content and topics of messages that have already been sent. As a general philosophy, our policies provide hard limits on the ways in which message content can be changed or undone. In contrast, our policies around message topics favor usefulness (e.g. for conversational organization) over faithfulness to the original. The message editing policy can be configured on the /#organization page. There are three configurations pro- vided out of the box: (i) users cannot edit messages at all, (ii) users can edit any message they have sent, and (iii) users can edit the content of any message they have sent in the last N minutes, and the topic of any message they have sent. In (ii) and (iii), topic edits can also be propagated to other messages with the same original topic, even if those messages were sent by other users. The default setting is (iii), with N = 10. In addition, and regardless of the configuration above, messages with no topic can always be edited to have a topic, by anyone in the organization, and the topic of any message can also always be edited by a realm administrator. Also note that while edited messages are synced immediately to open browser windows, editing messages is not a safe way to redact secret content (e.g. a password) shared unintentionally. Other users may have seen and saved the content of the original message, or have an integration (e.g. push notifications) forwarding all messages they receive to another service. Zulip stores the edit history of messages, but it may or may not be available to clients, depending on an organization-level setting.
64 Chapter 13. Security Model Zulip Documentation, Release 1.7.0
13.4 Users and Bots
• There are three types of users in a Zulip realm: Administrators, normal users, and bots. Administrators have the ability to deactivate and reactivate other human and bot users, delete streams, add/remove administrator privileges, as well as change configuration for the overall realm (e.g. whether an invitation is required to join the realm). Being a Zulip administrator does not provide the ability to interact with other users’ private messages or the messages sent to private streams to which the administrator is not subscribed. However, a Zulip administrator subscribed to a stream can toggle whether that stream is public or private. Also, Zulip realm administrators have administrative access to the API keys of all bots in the realm, so a Zulip administrator may be able to access messages sent to private streams that have bots subscribed, by using the bot’s credentials. In the future, Zulip’s security model may change to allow realm administrators to access private messages (e.g. to support auditing functionality). • Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially everything the user can do; for that reason, users should keep their API key safe. Users can rotate their own API key if it is accidentally compromised. • To properly remove a user’s access to a Zulip team, it does not suffice to change their password or deactivate their account in the SSO system, since neither of those prevents authenticating with the user’s API key or those of bots the user has created. Instead, you should deactivate the user’s account in the "Organization settings" interface (/#organization); this will automatically also deactivate any bots the user had created. • The Zulip mobile apps authenticate to the server by sending the user’s password and retrieving the user’s API key; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user’s phone is lost, in addition to changing passwords, you should rotate the user’s Zulip API key. • Zulip bots are used for integrations. A Zulip bot can do everything a normal user in the realm can do including reading other, with a few exceptions (e.g. a bot cannot login to the web application or create other bots). In particular, with the API key for a Zulip bot, one can read any message sent to a public stream in that bot’s realm. A likely future feature for Zulip is limited bots that can only send messages. • Certain Zulip bots can be marked as "API super users"; these special bots have the ability to send messages that appear to have been sent by another user (an important feature for implementing integrations like the Jabber, IRC, and Zephyr mirrors). They also have the ability to see the names of all streams (including private streams). They can only be created on the command line (with manage.py knight --permission=api_super_user).
13.5 User-uploaded content
• Zulip supports user-uploaded files; ideally they should be hosted from a separate domain from the main Zulip server to protect against various same-domain attacks (e.g. zulip-user-content.example.com) using the S3 inte- gration. The URLs of user-uploaded files are secret; if you are using the "local file upload" integration, anyone with the URL of an uploaded file can access the file. This means the local uploads integration is vulnerable to a subtle attack where if a user clicks on a link in a secret .PDF or .HTML file that had been uploaded to Zulip, access to the file might be leaked to the other server via the Referrer header (see the "Uploads world readable" issue on GitHub). The Zulip S3 file upload integration is relatively safe against that attack, because the URLs of files presented to users don’t host the content. Instead, the S3 integration checks the user has a valid Zulip session in the relevant realm, and if so then redirects the browser to a one-time S3 URL that expires a short time later. Keeping the URL secret is still important to avoid other users in the Zulip realm from being able to access the file.
13.4. Users and Bots 65 Zulip Documentation, Release 1.7.0
• Zulip supports using the Camo image proxy to proxy content like inline image previews that can be inserted into the Zulip message feed by other users over HTTPS. • By default, Zulip will provide image previews inline in the body of messages when a message contains a link to an image. You can control this using the INLINE_IMAGE_PREVIEW setting.
13.6 Final notes and security response
If you find some aspect of Zulip that seems inconsistent with this security model, please report it to zulip- [email protected] so that we can investigate and coordinate an appropriate security release if needed. Zulip security announcements will be sent to [email protected], so you should subscribe if you are running Zulip in production.
66 Chapter 13. Security Model CHAPTER 14
Authentication methods
Zulip supports several different authentications methods: • EmailAuthBackend - Email/password authentication. • ZulipLDAPAuthBackend - LDAP username/password authentication. • GoogleMobileOauth2Backend - Google authentication. • GitHubAuthBackend - GitHub authentication. • ZulipRemoteUserBackend - Authentication using an existing Single-Sign-On (SSO) system that can set REMOTE_USER in Apache. • DevAuthBackend - Only for development, passwordless login as any user. It’s easy to add more; see the docs on python-social-auth below. The setup documentation for most of these is simple enough that we’ve included it inline in /etc/zulip/ settings.py, right above to the settings used to configure them. The remote user authentication backend is more complex since it requires interfacing with a generic third-party authentication system, and so we’ve documented it in detail below.
14.1 Adding additional methods using python-social-auth
The implementation for GitHubAuthBackend is a small wrapper around the popular python-social-auth library. So if you’d like to integrate Zulip with another authentication provider (e.g. Facebook, Twitter, etc.), you can do this by writing a class similar to GitHubAuthBackend in zproject/backends.py and adding a few settings. Pull requests to add new backends are welcome; they should be tested using the framework in test_auth_backends. py.
67 Zulip Documentation, Release 1.7.0
14.2 Remote User SSO Authentication
Zulip supports integrating with a Single-Sign-On solution. There are a few ways to do it, but this section documents how to configure Zulip to use an SSO solution that best supports Apache and will set the REMOTE_USER variable: (0) Check that /etc/zulip/settings.py has zproject.backends.ZulipRemoteUserBackend as the only enabled value in the AUTHENTICATION_BACKENDS list, and that SSO_APPEND_DOMAIN is correct set depending on whether your SSO system uses email addresses or just usernames in REMOTE_USER. Make sure that you’ve restarted the Zulip server since making this configuration change. (1) Edit /etc/zulip/zulip.conf and change the puppet_classes line to read:
puppet_classes= zulip::voyager, zulip::apache_sso
(2) As root, run /home/zulip/deployments/current/scripts/zulip-puppet-apply to install our SSO integration. (3) To configure our SSO integration, edit /etc/apache2/sites-available/zulip-sso.example and fill in the configuration required for your SSO service to set REMOTE_USER and place your completed configuration file at /etc/apache2/sites-available/zulip-sso.conf zulip-sso.example is correct configuration for using an htpasswd file for REMOTE_USER authentication, which is useful for testing quickly. You can set it up by doing the following:
/home/zulip/deployments/current/scripts/restart-server cd/etc/apache2/sites-available/ cp zulip-sso.example zulip-sso.conf htpasswd-c/home/zulip/zpasswd username @example.com # prompts for a password
and then continuing with the steps below. (4) Run a2ensite zulip-sso to enable the Apache integration site. (5) Run service apache2 reload to use your new configuration. If Apache isn’t already running, you may need to run service apache2 start instead. Now you should be able to visit https://zulip.example.com/ and login via the SSO solution.
14.2.1 Troubleshooting Remote User SSO
This system is a little finicky to networking setup (e.g. common issues have to do with /etc/hosts not map- ping settings.EXTERNAL_HOST to the Apache listening on 127.0.0.1/localhost, for example). It can often help while debugging to temporarily change the Apache config in /etc/apache2/sites-available/ zulip-sso to listen on all interfaces rather than just 127.0.0.1 as you debug this. It can also be helpful to change /etc/nginx/zulip-include/app.d/external-sso.conf to proxy_pass to a more explicit URL possibly not over HTTPS when debugging. The following log files can be helpful when debugging this setup: • /var/log/zulip/{errors.log,server.log} (the usual places) • /var/log/nginx/access.log (nginx access logs) • /var/log/apache2/zulip_auth_access.log (you may want to change LogLevel to "debug" in the Apache config file to make this more verbose) Here’s a summary of how the remote user SSO system works assuming you’re using HTTP basic auth; this summary should help with understanding what’s going on as you try to debug:
68 Chapter 14. Authentication methods Zulip Documentation, Release 1.7.0
• Since you’ve configured /etc/zulip/settings.py to only define the zproject.backends. ZulipRemoteUserBackend, zproject/settings.py configures /accounts/login/sso as HOME_NOT_LOGGED_IN, which makes https://zulip.example.com/ (a.k.a. the homepage for the main Zulip Django app running behind nginx) redirect to /accounts/login/sso if you’re not logged in. • nginx proxies requests to /accounts/login/sso/ to an Apache instance listening on localhost:8888 apache via the config in /etc/nginx/zulip-include/app.d/external-sso.conf (using the up- stream localhost:8888 defined in /etc/nginx/zulip-include/upstreams). • The Apache zulip-sso site which you’ve enabled listens on localhost:8888 and presents the htpasswd dialogue; you provide correct login information and the request reaches a second Zulip Django app instance that is running behind Apache with with REMOTE_USER set. That request is served by zerver. views.remote_user_sso, which just checks the REMOTE_USER variable and either logs in (sets a cookie) or registers the new user (depending whether they have an account). • After succeeding, that redirects the user back to / on port 443 (hosted by nginx); the main Zulip Django app sees the cookie and proceeds to load the site homepage with them logged in (just as if they’d logged in normally via username/password). Again, most issues with this setup tend to be subtle issues with the hostname/DNS side of the configuration. Sugges- tions for how to improve this SSO setup documentation are very welcome!
14.2. Remote User SSO Authentication 69 Zulip Documentation, Release 1.7.0
70 Chapter 14. Authentication methods CHAPTER 15
Postgres database details
15.1 Remote Postgres database
This is a bit annoying to setup, but you can configure Zulip to use a dedicated postgres server by setting the REMOTE_POSTGRES_HOST variable in /etc/zulip/settings.py, and configuring Postgres certificate authentication (see http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for documentation on how to set this up and deploy the certificates) to make the DATABASES configuration in zproject/settings.py work (or override that configuration). If you want to use a remote Postgresql database, you should configure the information about the connection with the server. You need a user called "zulip" in your database server. You can configure these options in /etc/zulip/settings.py (the below descriptions are from the Postgresql documentation): • REMOTE_POSTGRES_HOST: Name or IP address of the remote host • REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are: – disable: I don’t care about security, and I don’t want to pay the overhead of encryption. – allow: I don’t care about security, but I will pay the overhead of encryption if the server insists on it. – prefer: I don’t care about encryption, but I wish to pay the overhead of encryption if the server supports it. – require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want. – verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust. – verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it’s the one I specify. Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf: postgres_password= xxxx
71 Zulip Documentation, Release 1.7.0
Finally, you can stop your database on the Zulip server via: sudo service postgresql stop sudo update-rc.d postgresql disable
In future versions of this feature, we’d like to implement and document how to the remote postgres database server itself automatically by using the Zulip install script with a different set of puppet manifests than the all-in-one feature; if you’re interested in working on this, post to the Zulip development mailing list and we can give you some tips.
15.2 Debugging postgres database issues
When debugging postgres issues, in addition to the standard pg_top tool, often it can be useful to use this query:
SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY
˓→procpid; which shows the currently running backends and their activity. This is similar to the pg_top output, with the added advantage of showing the complete query, which can be valuable in debugging. To stop a runaway query, you can run SELECT pg_cancel_backend(pid int) or SELECT pg_terminate_backend(pid int) as the ’postgres’ user. The former cancels the backend’s current query and the latter terminates the backend process. They are implemented by sending SIGINT and SIGTERM to the processes, respectively. We recommend against sending a Postgres process SIGKILL. Doing so will cause the database to kill all current connections, roll back any pending transactions, and enter recovery mode.
15.3 Stopping the Zulip postgres database
To start or stop postgres manually, use the pg_ctlcluster command: pg_ctlcluster 9.1[--force] main {start|stop|restart|reload}
By default, using stop uses "smart" mode, which waits for all clients to disconnect before shutting down the database. This can take prohibitively long. If you use the –force option with stop, pg_ctlcluster will try to use the "fast" mode for shutting down. "Fast" mode is described by the manpage thusly: With the –force option the "fast" mode is used which rolls back all active transactions, disconnects clients immediately and thus shuts down cleanly. If that does not work, shutdown is attempted again in "immediate" mode, which can leave the cluster in an inconsistent state and thus will lead to a recovery run at the next start. If this still does not help, the postmaster process is killed. Exits with 0 on success, with 2 if the server is not running, and with 1 on other failure conditions. This mode should only be used when the machine is about to be shut down. Many database parameters can be adjusted while the database is running. Just modify /etc/postgresql/9.1/main/postgresql.conf and issue a reload. The logs will note the change.
15.4 Debugging issues starting postgres pg_ctlcluster often doesn’t give you any information on why the database failed to start. It may tell you to check the logs, but you won’t find any information there. pg_ctlcluster runs the following command underneath when it actually goes to start Postgres:
/usr/lib/postgresql/9.1/bin/pg_ctl start-D/var/lib/postgresql/9.1/main-s-o \ '-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
72 Chapter 15. Postgres database details Zulip Documentation, Release 1.7.0
Since pg_ctl doesn’t redirect stdout or stderr, running the above can give you better diagnostic information. However, you might want to stop Postgres and restart it using pg_ctlcluster after you’ve debugged with this approach, since it does bypass some of the work that pg_ctlcluster does.
15.5 Postgres Vacuuming alerts
The autovac_freeze postgres alert from check_postgres is particularly important. This alert indicates that the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to the autovacuum_freeze_max_age setting. When the oldest XID hits that age, Postgres will force a VACUUM operation, which can often lead to sudden downtime until the operation finishes. If it did not do this and the age of the oldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagios alert, perform a VACUUM in each indicated database as a database superuser (postgres). See http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND for more details on postgres vacuuming.
15.5. Postgres Vacuuming alerts 73 Zulip Documentation, Release 1.7.0
74 Chapter 15. Postgres database details CHAPTER 16
Development environment installation
16.1 Requirements
The Zulip development environment can be installed on macOS, Windows, and Linux (Ubuntu recommended). You’ll need at least 2GB of available RAM. Installing the Zulip development environment requires downloading several hundred megabytes of dependencies, so you will need an active, reasonably fast, internet connection throughout the entire installation processes. You can configure a proxy if you need one.
16.2 Recommended setup (Vagrant)
For first-time contributors on macOS, Windows, and Ubuntu, we recommend using the Vagrant development environment. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Vagrant adds a bit of overhead to using the Zulip development server, but provides an isolated environment that is easy to install, update, and uninstall. It has been well-tested and performs well.
16.3 Advanced setup (non-Vagrant)
For more experienced contributors, or for first-time contributors who don’t want to or can’t use Vagrant, Zulip supports a wide range of ways to install the Zulip development environment on macOS and Linux (Ubuntu recommended): • On Ubuntu 16.04 Xenial and 14.04 Trusty, you can easily install without using Vagrant. • On other Linux distributions, you’ll need to follow slightly different instructions to install manually. • On macOS and Linux (Ubuntu recommended), you can install using Docker, though support for this remains experimental.
75 Zulip Documentation, Release 1.7.0
Unfortunately, the only supported method to install on Windows is the Vagrant method.
16.4 Slow internet connections
If you have a very slow network connection, however, you may want to avoid using Vagrant (which involves down- loading an Ubuntu virtual machine or Linux Container) and either install directly on Ubuntu 16.04 Xenial or 14.04 Trust, or use the manual install process instead. These options only support Linux. An alternative option if you have poor network connectivity is to rent a cloud server and install the Zulip development environment for remote use. See next section for details.
16.5 Installing remotely
The Zulip development environment works well on remote virtual machines. This can be a good alternative for those with poor network connectivity or who have limited storage/memory on their local machines. We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04, with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to. If you want to run a non-Ubuntu distribution, follow the generic Linux directions.
16.6 Next steps
Once you’ve installed the Zulip development environment, you’ll want to read these documents to learn how to use it: • Using the Development Environment • Testing (and Configuring Travis CI) And if you’ve setup the Zulip development environment on a remote machine, take a look at our tips for developing remotely.
76 Chapter 16. Development environment installation CHAPTER 17
Vagrant environment setup tutorial
This section guides first-time contributors through installing the Zulip development environment on Windows, macOS, and Ubuntu. The recommended method for installing the Zulip development environment is to use Vagrant with VirtualBox on Windows and macOS, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run. Contents: • Requirements • Step 0: Set up Git & GitHub • Step 1: Install Prerequisites • Step 2: Get Zulip code • Step 3: Start the development environment • Step 4: Developing • Troubleshooting and Common Errors • Specifying a proxy If you encounter errors installing the Zulip development environment, check Troubleshooting and Common Er- rors. If that doesn’t help, please visit #provision help in the Zulip development community server for real-time help, send a note to the Zulip-devel Google group or file an issue. When reporting your issue, please include the following information: • host operating system • installation method (Vagrant or direct) • whether or not you are using a proxy • a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual machine
77 Zulip Documentation, Release 1.7.0
17.1 Requirements
Installing the Zulip development environment requires downloading several hundred megabytes of dependencies. You will need an active internet connection throughout the entire installation processes. (See Specifying a proxy if you need a proxy to access the internet.) • All: 2GB available RAM, Active broadband internet connection, GitHub account. • macOS: macOS (10.11 El Capitan or 10.12 Sierra recommended), Git, VirtualBox (version 5.1.8 recommended – we find it’s more stable than more recent versions), Vagrant. • Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc. – or Debian: 9.0 "stretch" 64-bit • Windows: Windows 64-bit (Win 10 recommended), hardware virtualization enabled (VT-X or AMD-V), ad- ministrator access, Git for Windows (which installs Git BASH), VirtualBox, Vagrant. Don’t see your system listed above? See Advanced setup for details about installing for other Linux and UNIX platforms.
17.2 Step 0: Set up Git & GitHub
You can skip this step if you already have Git, GitHub, and SSH access to GitHub working on your machine. Follow our Git Guide in order to install Git, set up a GitHub account, create an SSH key to access code on GitHub efficiently, etc. Be sure to create an ssh key and add it to your GitHub account using these instructions.
17.3 Step 1: Install Prerequisites
Jump to: • macOS • Ubuntu • Debian • Windows
17.3.1 macOS
1. Install Vagrant (1.8.4-1.8.6, do not use 1.8.7). 2. Install VirtualBox (5.1.8). (For a non-free option, but better performance, you can also use VMWare Fusion with the VMWare Fusion Vagrant plugin.) Now you are ready for Step 2: Get Zulip Code..
78 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
17.3.2 Ubuntu
The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same. If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code: sudo apt-get-y purge vagrant&&\ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb&&\ sudo dpkg-i vagrant *.deb&&\ sudo apt-get-y install build-essential git ruby lxc lxc-templates cgroup-lite redir&
˓→&\ vagrant plugin install vagrant-lxc&&\ vagrant lxc sudoers
For a step-by-step explanation, read on.
1. Install Vagrant
For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in the official Ubuntu repositories. First uninstall any vagrant package you may have installed from the Ubuntu repository: christie@ubuntu-desktop:~ $ sudo apt-get purge vagrant
Now download and install the .deb package for Vagrant 1.8.6: christie@ubuntu-desktop:~ $ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb christie@ubuntu-desktop:~ $ sudo dpkg -i vagrant*.deb
2. Install remaining dependencies
Now install git and lxc-related packages: christie@ubuntu-desktop:~ $ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
3. Install the vagrant lxc plugin: christie@ubuntu-desktop:~ $ vagrant plugin install vagrant-lxc Installing the 'vagrant-lxc' plugin. This can take a few minutes... Installed the plugin 'vagrant-lxc (1.2.1)'!
If you encounter an error when trying to install the vagrant-lxc plugin, see this.
17.3. Step 1: Install Prerequisites 79 Zulip Documentation, Release 1.7.0
4. Configure sudo to be passwordless
Finally, configure sudo to be passwordless when using Vagrant LXC:
christie@ubuntu-desktop:~ $ vagrant lxc sudoers [sudo] password for christie:
If you encounter an error running vagrant lxc sudoers, see this. Now you are ready for Step 2: Get Zulip Code.
17.3.3 Debian
The setup for Debian 9.0 "stretch" is just like for Ubuntu 16.04, with one difference. If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code:
sudo apt-get-y purge vagrant&&\ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb&&\ sudo dpkg-i vagrant *.deb&&\ sudo apt-get-y install build-essential git ruby lxc redir&&\ vagrant plugin install vagrant-lxc&&\ vagrant lxc sudoers
For a step-by-step explanation, follow the Ubuntu instructions above, with the following difference: in "2. Install remaining dependencies", the command is
sudo apt-get install build-essential git ruby lxc redir
17.3.4 Windows 10
1. Install Git for Windows, which installs Git BASH. 2. Install VirtualBox (version >= 5.1.6). 3. Install Vagrant (version 1.8.4-1.8.6, do not use 1.8.7). (Note: While Git BASH is recommended, you may also use Cygwin. If you do, make sure to install default required packages along with git, curl, openssh, and rsync binaries.) Also, you must have hardware virtualization enabled (VT-X or AMD-V) in your computer’s BIOS.
17.3.5 Running Git BASH as an administrator
It is important that you always run Git BASH with administrator privileges when working on Zulip code, as not doing so will cause errors in the development environment (such as symlink creation). You might wish to configure your Git BASH shortcut to always run with these privileges enabled (see this guide for how to set this up).
Enable native symlinks
The Zulip code includes symbolic links (symlinks). By default, native Windows symlinks are not enabled in either Git BASH or Cygwin, so you need to do a bit of configuration. You must do this before you clone the Zulip code.
80 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
In Git for BASH: Open Git BASH as an administrator and run:
$ git config --global core.symlinks true
Now confirm the setting:
$ git config core.symlinks true
If you see true, you are ready for Step 2: Get Zulip Code. Otherwise, if the above command prints false or nothing at all, then symlinks have not been enabled. In Cygwin: Open a Cygwin window as an administrator and do this:
christie@win10 ~ $ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
Next, close that Cygwin window and open another. If you echo $CYGWIN you should see:
christie@win10 ~ $ echo $CYGWIN winsymlinks:native
Now you are ready for Step 2: Get Zulip Code.
17.4 Step 2: Get Zulip Code
1. In your browser, visit https://github.com/zulip/zulip and click the fork button. You will need to be logged in to GitHub to do this. 2. Open Terminal (macOS/Ubuntu) or Git BASH (Windows; must run as an Administrator). 3. In Terminal/Git BASH, clone your fork:
git clone [email protected]:YOURUSERNAME/zulip.git
This will create a ’zulip’ directory and download the Zulip code into it. Don’t forget to replace YOURUSERNAME with your git username. You will see something like:
christie@win10 ~ $ git clone [email protected]:YOURUSERNAME/zulip.git Cloning into 'zulip'... remote: Counting objects: 73571, done. remote: Compressing objects: 100% (2/2), done. remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569 Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done. Resolving deltas: 100% (51448/51448), done. Checking connectivity... done. Checking out files: 100% (1912/1912), done.`
Now you are ready for Step 3: Start the development environment.
17.4. Step 2: Get Zulip Code 81 Zulip Documentation, Release 1.7.0
17.5 Step 3: Start the development environment
Change into the zulip directory and tell vagrant to start the Zulip development environment with vagrant up. christie@win10 ~ $ cd zulip christie@win10 ~/zulip $ vagrant up
The first time you run this command it will take some time because vagrant does the following: • downloads the base Ubuntu 14.04 virtual machine image (for macOS and Windows) or container (for Ubuntu) • configures this virtual machine/container for use with Zulip, • creates a shared directory mapping your clone of the Zulip code inside the virtual machine/container at ~/ zulip • runs the tools/provision script inside the virtual machine/container, which downloads all required depen- dencies, sets up the python environment for the Zulip development server, and initializes a default test database. We call this process "provisioning", and it is documented in some detail in our dependencies documentation. You will need an active internet connection during the entire process. (See Specifying a proxy if you need a proxy to access the internet.) vagrant up can fail while provisioning if your Internet connection is unreliable. To retry, you can use vagrant provision (vagrant up will just boot the guest without provisioning after the first time). Other common issues are documented in the Troubleshooting and Common Errors section. If that doesn’t help, please visit #provision help in the Zulip development community server for real-time help. On Windows, you will see The system cannot find the path specified. message several times. This is expected behavior and is not an error. Once vagrant up has completed, connect to the development environment with vagrant ssh: christie@win10 ~/zulip $ vagrant ssh
You should see something like this on Windows and macOS:
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Wed May4 21:45:43 UTC 2016
System load: 0.61 Processes: 88 Usage of/: 3.5% of 39.34GB Users logged in:0 Memory usage:7% IP address for eth0: 10.0.2.15 Swap usage:0%
Graph this data and manage this system at: https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud
0 packages can be updated. 0 updates are security updates.
82 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
Or something as brief as this in the case of Ubuntu:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64)
* Documentation: https://help.ubuntu.com/
Congrats, you’re now inside the Zulip development environment! You can confirm this by looking at the command prompt, which starts with (zulip-py3-venv)vagrant@. If it just starts with vagrant@, your provisioning failed and you should look at the troubleshooting section. Next, start the Zulip server:
(zulip-py3-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip $ ./tools/run-dev.py
You will see several lines of output starting with something like:
2016-05-04 22:20:33,895 INFO: process_fts_updates starting Recompiling templates 2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates done Validating Django models.py... System check identified no issues (0 silenced).
Django version 1.8 Tornado server is running at http://localhost:9993/ Quit the server with CTRL-C. 2016-05-04 18:20:40,716 INFO Tornado loaded0 event queues in 0.001s 2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 seconds Performing system checks...
And ending with something similar to: http://localhost:9994/webpack-dev-server/ webpack result is served from http://localhost:9991/webpack/ content is served from /srv/zulip webpack: bundle is now VALID. 2016-05-06 21:43:29,553 INFO Tornado 31.6% busy over the past 10.6 seconds 2016-05-06 21:43:35,007 INFO Tornado 23.9% busy over the past 16.0 seconds
Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in the browser on your main machine. You should see something like this:
17.5. Step 3: Start the development environment 83 Zulip Documentation, Release 1.7.0
The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in your browser, check your terminal and you should see something like:
2016-05-04 18:21:57,547 INFO 127.0.0.1 GET 302 582ms (+start: 417ms) /
˓→(unauth via ?) [04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0 2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth via
˓→?) [04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0 2016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /
˓→login/ (unauth via ?)
Now you’re ready for Step 4: Developing.
84 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
17.6 Step 4: Developing
17.6.1 Where to edit files
You’ll work by editing files on your host machine, in the directory where you cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++, etc.). When you save changes they will be synced automatically to the Zulip development environment on the virtual ma- chine/container. Each component of the Zulip development server will automatically restart itself or reload data appropriately when you make changes. So, to see your changes, all you usually have to do is reload your browser. More details on how this works are available below. Zulip’s whitespace rules are all enforced by linters, so be sure to run tools/lint often to make sure you’re follow- ing our coding style (or use tools/setup-git-repo to run it on just the changed files automatically whenever you commit).
17.6.2 Understanding run-dev.py debugging output
It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along with every backend request will be printed there. See Logging for further details on the run-dev.py console output.
17.6.3 Committing and pushing changes with git
When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (ma- cOS/Ubuntu) or Git BASH (Windows) in the directory where you cloned Zulip on your main machine. If you’re new to working with Git/GitHub, check out our Git & GitHub Guide.
17.6.4 Maintaining the development environment
If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server or running tests, this is probably not because Zulip’s master branch is broken. Instead, this is likely because we’ve recently merged changes to the development environment provisioning process that you need to apply to your devel- opment environment. To update your environment, you’ll need to re-provision your vagrant machine using vagrant provision (this just runs tools/provision from your Zulip checkout inside the Vagrant guest); this should complete in about a minute. After provisioning, you’ll want to (re)start the Zulip development server. If you run into any trouble, the #provision help in the Zulip development community server for is a great place to ask for help.
17.6.5 Rebuilding the development environment
If you ever want to recreate your development environment again from scratch (e.g. to test as change you’ve made to the provisioning process, or because you think something is broken), you can do so using vagrant destroy and then vagrant up. This will usually be much faster than the original vagrant up since the base image is already cached on your machine (it takes about 5 minutes to run with a fast Internet connection).
17.6. Step 4: Developing 85 Zulip Documentation, Release 1.7.0
Any additional programs (e.g. Zsh, emacs, etc.) or configuration that you may have installed in the develop- ment environment will be lost when you recreate it. To address this, you can create a script called tools/ custom_provision in your Zulip Git checkout; and place any extra setup commands there. Vagrant will run tools/custom_provision every time you run vagrant provision (or create a Vagrant guest via vagrant up).
17.6.6 Shutting down the development environment for use later
To shut down but preserve the development environment so you can use it again later use vagrant halt or vagrant suspend. You can do this from the same Terminal/Git BASH window that is running run-dev.py by pressing ^C to halt the server and then typing exit. Or you can halt vagrant from another Terminal/Git BASH window. From the window where run-dev.py is running:
2016-05-04 18:33:13,330 INFO 127.0.0.1 GET 200 92ms /register/ (unauth
˓→via ?) ^C KeyboardInterrupt (zulip-py3-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ exit logout Connection to 127.0.0.1 closed. christie@win10 ~/zulip
Now you can suspend the development environment:
christie@win10 ~/zulip $ vagrant suspend ==> default: Saving VM state and suspending execution...
If vagrant suspend doesn’t work, try vagrant halt:
christie@win10 ~/zulip $ vagrant halt ==> default: Attempting graceful shutdown of VM...
Check out the Vagrant documentation to learn more about suspend and halt.
17.6.7 Resuming the development environment
When you’re ready to work on Zulip again, run vagrant up. You will also need to connect to the virtual machine with vagrant ssh and re-start the Zulip server:
christie@win10 ~/zulip $ vagrant up $ vagrant ssh
(zulip-py3-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip $ ./tools/run-dev.py
17.7 Next Steps
Next, read the following to learn more about developing for Zulip:
86 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
• Git & GitHub Guide • Using the Development Environment • Testing (and Configuring Travis CI to run the full test suite against any branches you push to your fork, which can help you optimize your development workflow).
17.8 Troubleshooting and Common Errors
Below you’ll find a list of common errors and their solutions. Most issues are resolved by just provisioning again (by running ./tools/provision inside the Vagrant guest or equivalently vagrant provision from outside). If these solutions aren’t working for you or you encounter an issue not documented below, there are a few ways to get further help: • Ask in #provision help in the Zulip development community server, • send a note to the Zulip-devel Google group, or • File an issue. When reporting your issue, please include the following information: • host operating system • installation method (Vagrant or direct) • whether or not you are using a proxy • a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual machine. If you choose to post just the error output, please include the beginning of the error output, not just the last few lines. The output of tools/diagnose run inside the Vagrant guest is also usually helpful.
17.8.1 Vagrant guest doesn’t show (zulip-py3-venv) at start of prompt
This is caused by provisioning failing to complete successfully. You can see the errors in var/log/provision. log; it should end with something like this:
ESC[94mZulip development environment setup succeeded!ESC[0m
The ESC stuff are the terminal color codes that make it show as a nice blue in the terminal, which unfortunately looks ugly in the logs. If you encounter an incomplete /var/log/provision.log file, you need to update your environment. Re- provision your vagrant machine; if the problem persists, please come chat with us (see instructions above) for help. After you provision successfully, you’ll need to exit your vagrant ssh shell and run vagrant ssh again to get the virtualenv setup properly.
17.8.2 The box ’ubuntu/trusty64’ could not be found
If you see the following error when you run vagrant up:
17.8. Troubleshooting and Common Errors 87 Zulip Documentation, Release 1.7.0
The box 'ubuntu/trusty64' could not be found or could not be accessed in the remote catalog. If this is a private box on HashiCorp's Atlas, please verify you're logged in via `vagrant login`. Also, please double-check the name. The expanded URL and error message are shown below: URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]
Then the version of curl that ships with Vagrant is not working on your machine. You are most likely to encounter this error on Windows/Cygwin and macOS. On macOS this error is most likely to occur with Vagrant version 1.8.7 and is a known issue. The solution is to downgrade Vagrant to version 1.8.6 (available here), or to use your system’s version of curl instead of the one that ships with Vagrant: sudo ln-nsf/usr/bin/curl/opt/vagrant/embedded/bin/curl
On Windows/Cygwin, the fix is simple: replace it with the version from Cygwin. First, determine the location of Cygwin’s curl with which curl: christie@win10 ~/zulip $ which curl /usr/bin/curl
Now determine the location of Vagrant with which vagrant: christie@win10 ~/zulip $ which vagrant /cygdrive/c/HashiCorp/Vagrant/bin/vagrant
The path up until /bin/vagrant is what you need to know. In the example above it’s /cygdrive/c/ HashiCorp/Vagrant. Finally, copy Cygwin’s curl to Vagrant embedded/bin directory: christie@win10 ~/zulip $ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/
Now re-run vagrant up and vagrant should be able to fetch the required box file.
17.8.3 Vagrant was unable to mount VirtualBox shared folders
For the following error:
Vagrant was unable to mount VirtualBox shared folders. This is usually because the filesystem"vboxsf" is not available. This filesystem is made available via the VirtualBox Guest Additions and kernel module. Please verify that these guest additions are properly installed in the guest. This is not a bug in Vagrant and is usually caused by a faulty Vagrant box. For context, the command attempted was:
mount-t vboxsf-o uid=1000,gid=1000 keys/keys
If this error starts happening unexpectedly, then just run:
88 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
vagrant reload
This is equivalent of running a halt followed by an up (aka rebooting the guest). After this, you can do vagrant provision and vagrant ssh.
17.8.4 ssh connection closed by remote host
On running vagrant ssh, if you see the following error:
ssh_exchange_identification: Connection closed by remote host
It usually means the Vagrant guest is not running, which is usually solved by rebooting the Vagrant guest via vagrant reload. See Vagrant was unable to communicate with the guest machine for more details.
17.8.5 os.symlink error
If you receive the following error while running vagrant up:
==> default: Traceback (most recent call last): ==> default: File"./emoji_dump.py", line 75, in
˓→format(name)) ==> default: OSError ==> default: : ==> default: [Errno 71] Protocol error
Then Vagrant was not able to create a symbolic link. First, if you are using Windows, make sure you have run Git BASH (or Cygwin) as an administrator. By default, only administrators can create symbolic links on Windows. Second, VirtualBox does not enable symbolic links by default. Vagrant starting with version 1.6.0 enables symbolic links for VirtualBox shared folder. You can check to see that this is enabled for your virtual machine with vboxmanage command. Get the name of your virtual machine by running vboxmanage list vms and then print out the custom settings for this virtual machine with vboxmanage getextradata YOURVMNAME enumerate:
christie@win10 ~/zulip $ vboxmanage list vms "zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}
christie@win10 ~/zulip $ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1 Key: supported, Value: false
If you see "command not found" when you try to run VBoxManage, you need to add the VirtualBox directory to your path. On Windows this is mostly likely C:\Program Files\Oracle\VirtualBox\. If vboxmanage enumerate prints nothing, or shows a value of 0 for VBoxInter- nal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable symbolic links by running this command in Terminal/Git BASH/Cygwin:
17.8. Troubleshooting and Common Errors 89 Zulip Documentation, Release 1.7.0
vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/
˓→srv_zulip1
The virtual machine needs to be shut down when you run this command.
17.8.6 Connection timeout on vagrant up
If you see the following error after running vagrant up: default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying...
A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’s BIOS settings. Look for a setting called VT-x (Intel) or (AMD-V). If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system. For further information about troubleshooting vagrant timeout errors see this post.
17.8.7 Vagrant was unable to communicate with the guest machine
If you see the following error when you run vagrant up:
Timed out while waiting for the machine to boot. This means that Vagrant was unable to communicate with the guest machine within the configured ("config.vm.boot_timeout" value) time period.
If you look above, you should be able to see the error(s) that Vagrant had when attempting to connect to the machine. These errors are usually good hints as to what may be wrong.
If you're using a custom box, make sure that networking is properly working and you're able to connect to the machine. It is a common problem that networking isn't setup properly in these boxes. Verify that authentication configurations are also setup properly, as well.
If the box appears to be booting properly, you may want to increase the timeout ("config.vm.boot_timeout") value.
This has a range of possible causes, that usually amount to a bug in Virtualbox or Vagrant. If you see this error, you usually can fix it by rebooting the guest via vagrant reload (or equivalently, vagrant halt followed by vagrant up):
17.8.8 Vagrant up fails with subprocess.CalledProcessError
The vagrant up command basically does the following: • Downloads an Ubuntu image and starts it using a Vagrant provider.
90 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
• Uses vagrant ssh to connect to that Ubuntu guest, and then runs tools/provision, which has a lot of subcommands that are executed via Python’s subprocess module. These errors mean that one of those subcommands failed. To debug such errors, you can log in to the Vagrant guest machine by running vagrant ssh, which should present you with a standard shell prompt. You can debug interactively by using e.g. cd zulip && ./tools/ provision, and then running the individual subcommands that failed. Once you’ve resolved the problem, you can rerun tools/provision to proceed; the provisioning system is designed to recover well from failures. The zulip provisioning system is generally highly reliable; the most common cause of issues here is a poor network connection (or one where you need a proxy to access the Internet and haven’t configured the development environment to use it. Once you’ve provisioned successfully, you’ll get output like this:
Zulip development environment setup succeeded! (zulip-py3-venv) vagrant@vagrant-base-trusty-amd64:~/zulip$
If the (zulip-py3-venv) part is missing, this is because your installation failed the first time before the Zulip virtualenv was created. You can fix this by just closing the shell and running vagrant ssh again, or using source /srv/zulip-py3-venv/bin/activate. Finally, if you encounter any issues that weren’t caused by your Internet connection, please report them! We try hard to keep Zulip development environment provisioning free of bugs. pip install fails during vagrant up on Ubuntu
Likely causes are: 1. Networking issues 2. Insufficient RAM. Check whether you’ve allotted at least two gigabytes of RAM, which is the minimum Zulip requires. If not, go to your VM settings and increase the RAM, then restart the VM. yarn install warnings
$ yarn install yarn install v0.24.5 [1/4] Resolving packages... [2/4] Fetching packages... warning [email protected]: The platform "linux" is incompatible with this module. info "[email protected]" is an optional dependency and failed compatibility check.
˓→Excluding it from installation. [3/4] Linking dependencies... [4/4] Building fresh packages... $ browserify node_modules/sockjs-client/lib/entry.js --standalone SockJS > node_
˓→modules/sockjs-client/sockjs.js Done in 23.50s.
These are warnings produced by spammy third party JavaScript packages. It is okay to proceed and start the Zulip server.
17.8. Troubleshooting and Common Errors 91 Zulip Documentation, Release 1.7.0
17.8.9 vagrant-lxc errors
Permissions errors
When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.
NoMethodError
If you see the following error when you try to install the vagrant-lxc plugin:
/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_
˓→by' for nil:NilClass (NoMethodError) from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_
˓→plugin' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin' from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run' from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run' from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in
˓→execute' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute' from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute' from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute' from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli' from /usr/bin/vagrant:173:in `
And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of the issue, which should be fixed when Vagrant 1.8.2 is released. In the meantime, read this post for how to create and apply the patch. It will look something like this: christie@xenial:~ $ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch patching file bundler.rb
17.8.10 Permissions errors when running the test suite in LXC
See "Possible testing issues".
92 Chapter 17. Vagrant environment setup tutorial Zulip Documentation, Release 1.7.0
17.9 Specifying a proxy
If you need to use a proxy server to access the Internet, you will need to specify the proxy settings before running Vagrant up. First, install the Vagrant plugin vagrant-proxyconf: vagrant plugin install vagrant-proxyconf
Then create ~/.zulip-vagrant-config and add the following lines to it (with the appropriate values in it for your proxy):
HTTP_PROXY http://proxy_host:port HTTPS_PROXY http://proxy_host:port NO_PROXY localhost,127.0.0.1,.example.com
Now run vagrant up in your terminal to install the development server. If you ran vagrant up before and failed, you’ll need to run vagrant destroy first to clean up the failed installation. If you no longer want to use proxy with Vagrant, set values of HTTP_PROXY and HTTPS_PROXY to "" in ~/.zulip-vagrant-config file and restart Vagrant. You can also change the port on the host machine that Vagrant uses by adding to your ~/. zulip-vagrant-config file. E.g. if you set:
HOST_PORT 9971
(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your development server. If you’d like to be able to connect to your development environment from other machines than the VM host, you can manually set the host IP address in the ’~/.zulip-vagrant-config’ file as well. For example, if you set:
HOST_IP_ADDR 0.0.0.0
(and restart the Vagrant guest), your host IP would be 0.0.0.0, a special value for the IP address that means any IP address can connect to your development server.
17.9. Specifying a proxy 93 Zulip Documentation, Release 1.7.0
94 Chapter 17. Vagrant environment setup tutorial CHAPTER 18
Zulip development environment setup without Vagrant
Contents: • Installing directly on Ubuntu • Installing manually on Linux • Using Docker (experimental)
18.1 Installing directly on Ubuntu
Start by cloning this repository: git clone https://github.com/zulip/zulip.git If you’d like to install a Zulip development environment on a computer that’s already running Ubuntu 16.04 Xenial or Ubuntu 14.04 Trusty, you can do that by just running:
# From a clone of zulip.git ./tools/provision source/srv/zulip-py3-venv/bin/activate ./tools/run-dev.py # starts the development server
Note that there is no supported uninstallation process without Vagrant (with Vagrant, you can just do vagrant destroy to clean up the development environment). Once you’ve done the above setup, you can pick up the documentation on using the Zulip development environment, ignoring the parts about vagrant (since you’re not using it).
18.2 Installing manually on Linux
• Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental)
95 Zulip Documentation, Release 1.7.0
• OpenBSD 5.8 (experimental) • Fedora/CentOS common steps • Steps for all systems If you really want to install everything manually, the below instructions should work. Install the following non-Python dependencies: • libffi-dev — needed for some Python extensions • postgresql 9.1 or later — our database (client, server, headers) • nodejs 0.10 (and yarn) • memcached (and headers) • rabbitmq-server • libldap2-dev • python3-dev • python-dev • python-virtualenv • redis-server — rate limiting • tsearch-extras — better text search • libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs
18.2.1 On Debian or Ubuntu systems:
Using the official Ubuntu repositories, PGroonga PPA and tsearch-extras deb package:
Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python3-dev \ python-dev python-virtualenv hunspell-en-us nodejs \ nodejs-legacy git yui-compressor puppet gettext postgresql
# If using Ubuntu, install PGroonga from its PPA sudo add-apt-repository-ys ppa:groonga/ppa sudo apt-get update # On 14.04 sudo apt-get install postgresql-9.3-pgroonga # On 16.04 sudo apt-get install postgresql-9.5-pgroonga
# If using Debian, follow the instructions here: http://pgroonga.github.io/install/
˓→debian.html
# Next, install Zulip's tsearch-extras postgresql extension # If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras: cd zulip sudo apt-add-repository-ys ppa:tabbott/zulip sudo apt-get update # On 14.04
96 Chapter 18. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.7.0
sudo apt-get install postgresql-9.3-tsearch-extras # On 16.04 sudo apt-get install postgresql-9.5-tsearch-extras
# Otherwise, you can download a .deb directly # If on 12.04 or wheezy: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch-
˓→extras_0.1.2_amd64.deb sudo dpkg-i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
# If on 14.04: wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3-
˓→tsearch-extras_0.1.3_amd64.deb sudo dpkg-i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb
# If on 15.04 or jessie: wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch-
˓→extras_0.1_amd64.deb sudo dpkg-i postgresql-9.4-tsearch-extras_0.1_amd64.deb
# If on 16.04 or stretch wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5-
˓→tsearch-extras_0.2_amd64.deb sudo dpkg-i postgresql-9.5-tsearch-extras_0.2_amd64.deb
Alternatively, you can always build the package from tsearch-extras git. Now continue with the All Systems instructions below.
Using the official Zulip PPA (for 14.04 Trusty or 16.04 Xenial):
Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo add-apt-repository ppa:tabbott/zulip sudo apt-get update sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \ memcached rabbitmq-server libldap2-dev redis-server \ postgresql-server-dev-all libmemcached-dev python3-dev python-dev \ hunspell-en-us nodejs nodejs-legacy git yui-compressor \ puppet gettext tsearch-extras
Now continue with the All Systems instructions below.
18.2.2 On Fedora 22 (experimental):
These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git sudo dnf install libffi-devel memcached rabbitmq-server \ openldap-devel python-devel redis postgresql-server \ postgresql-devel postgresql libmemcached-devel freetype-devel \ nodejs yuicompressor closure-compiler gettext
Now continue with the Common to Fedora/CentOS instructions below.
18.2. Installing manually on Linux 97 Zulip Documentation, Release 1.7.0
18.2.3 On CentOS 7 Core (experimental):
These instructions are experimental and may have bugs; patches welcome! Start by cloning this repository: git clone https://github.com/zulip/zulip.git
# Add user zulip to the system (not necessary if you configured zulip # as the administrator user during the install process of CentOS 7). useradd zulip
# Create a password for zulip user passwd zulip
# Allow zulip to sudo visudo # Add this line after line `root ALL=(ALL) ALL` zulip ALL=(ALL) ALL
# Switch to zulip user su zulip
# Enable EPEL 7 repo so we can install rabbitmq-server, redis and # other dependencies sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.
˓→rpm
# Install dependencies sudo yum install libffi-devel memcached rabbitmq-server openldap-devel \ python-devel redis postgresql-server postgresql-devel postgresql \ libmemcached-devel wget python-pip openssl-devel freetype-devel \ libjpeg-turbo-devel zlib-devel nodejs yuicompressor \ closure-compiler gettext
# We need these packages to compile tsearch-extras sudo yum groupinstall"Development Tools"
# clone Zulip's git repo and cd into it cd&& git clone https://github.com/zulip/zulip&& cd zulip/
## NEEDS TESTING: The next few DB setup items may not be required at all. # Initialize the postgres db sudo postgresql-setup initdb
# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf
# Change these lines: host all all 127.0.0.1/32 ident host all all::1/128 ident # to this: host all all 127.0.0.1/32 md5 host all all::1/128 md5
Now continue with the Common to Fedora/CentOS instructions below.
18.2.4 On OpenBSD 5.8 (experimental):
These instructions are experimental and may have bugs; patches welcome!
98 Chapter 18. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.7.0
Start by cloning this repository: git clone https://github.com/zulip/zulip.git doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \ memcached node libmemcached py-Pillow py-cryptography py-cffi
# Get tsearch_extras and build it (using a modified version which # aliases int4 on OpenBSD): git clone https://github.com/blablacio/tsearch_extras cd tsearch_extras gmake&& sudo gmake install
# Point environment to custom include locations and use newer GCC # (needed for Node modules): export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl" export CXX=eg++
# Create tsearch_data directory: sudo mkdir/usr/local/share/postgresql/tsearch_data
# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/local/share/postgresql/tsearch_data/english.stop sudo touch/usr/local/share/postgresql/tsearch_data/en_us.dict sudo touch/usr/local/share/postgresql/tsearch_data/en_us.affix
Finally continue with the All Systems instructions below.
18.2.5 Common to Fedora/CentOS instructions
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
# Build and install postgres tsearch-extras module wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.
˓→tar.gz tar xvzf tsearch-extras_0.1.3.tar.gz cd ts2 make sudo make install
# Hack around missing dictionary files -- need to fix this to get the # proper dictionaries from what in debian is the hunspell-en-us # package. sudo touch/usr/share/pgsql/tsearch_data/english.stop sudo touch/usr/share/pgsql/tsearch_data/en_us.dict sudo touch/usr/share/pgsql/tsearch_data/en_us.affix
# Edit the postgres settings: sudo vi/var/lib/pgsql/data/pg_hba.conf
# Add this line before the first uncommented line to enable password # auth: host all all 127.0.0.1/32 md5
# Start the services sudo systemctl start redis memcached rabbitmq-server postgresql
18.2. Installing manually on Linux 99 Zulip Documentation, Release 1.7.0
# Enable automatic service startup after the system startup sudo systemctl enable redis rabbitmq-server memcached postgresql
Finally continue with the All Systems instructions below.
18.2.6 All Systems:
Make sure you have followed the steps specific for your platform: • Debian or Ubuntu systems • Fedora 22 (experimental) • CentOS 7 Core (experimental) • OpenBSD 5.8 (experimental) • Fedora/CentOS For managing Zulip’s python dependencies, we recommend using virtualenvs. You must create a Python 3 virtualenv. You must also install appropriate python packages in it. You should either install the virtualenv in /srv, or put a symlink to it in /srv. If you don’t do that, some scripts might not work correctly. You can run python3 tools/setup/setup_venvs.py. This script will create a virtualenv /srv/ zulip-py3-venv. If you want to do it manually, here are the steps: sudo virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenv sudo chown -R `whoami`:`whoami` /srv/zulip-py3-venv source /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv pip install --upgrade pip # upgrade pip itself because older versions have known
˓→issues pip install --no-deps -r requirements/dev_lock.txt # install python packages required
˓→for development
Now run these commands: sudo ./scripts/lib/install-node yarn install sudo mkdir /srv/zulip-emoji-cache sudo chown -R `whoami`:`whoami` /srv/zulip-emoji-cache ./tools/setup/emoji/build_emoji ./tools/inline-email-css ./tools/setup/build_pygments_data ./tools/setup/generate_zulip_bots_static_files ./scripts/setup/generate_secrets.py --development if [ $(uname) = "OpenBSD" ]; then sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/
˓→tsearch_data/ else sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/ ˓→9.*/tsearch_data/ fi ./scripts/setup/configure-rabbitmq ./tools/setup/postgres-init-dev-db
100 Chapter 18. Zulip development environment setup without Vagrant Zulip Documentation, Release 1.7.0
./tools/do-destroy-rebuild-database ./tools/setup/postgres-init-test-db ./tools/do-destroy-rebuild-test-database ./manage.py compilemessages
To start the development server:
./tools/run-dev.py
. . . and visit http://localhost:9991/.
Proxy setup for by-hand installation
If you are building the development environment on a network where a proxy is required to access the Internet, you will need to set the proxy in the environment as follows: • On Ubuntu, set the proxy environment variables using: export https_proxy=http://proxy_host:port export http_proxy=http://proxy_host:port
• And set the yarn proxy and https-proxy using: yarn config set proxy http://proxy_host:port yarn config set https-proxy http://proxy_host:port
18.3 Using Docker (experimental)
Start by cloning this repository: git clone https://github.com/zulip/zulip.git The docker instructions for development are experimental, so they may have bugs. If you try them and run into any issues, please report them! You can also use Docker to run a Zulip development environment. First, you need to install Docker in your develop- ment machine following the instructions. Some other interesting links for somebody new in Docker are: • Get Started • Understand the architecture • Docker run reference • Dockerfile reference Then you should create the Docker image based on Ubuntu Linux, first go to the directory with the Zulip source code: docker build-t user/zulipdev-f Dockerfile-dev.
Commit and tag the provisioned images. The below will install Zulip’s dependencies: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash $ /bin/bash sudo chown -R zulip:zulip /srv/zulip $ /bin/bash /srv/zulip/tools/provision --docker docker ps -af ancestor=user/zulipdev docker commit -m "Zulip installed"
18.3. Using Docker (experimental) 101 Zulip Documentation, Release 1.7.0
Now you can run the docker server with: docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \ /srv/zulip/tools/start-dockers
You’ll want to read the guide for Zulip development to understand how to use the Zulip development. Note that start-dockers automatically runs tools/run-dev.py inside the container; you can then visit http://localhost:9991 to connect to your new Zulip Docker container. To view the container’s run-dev.py console logs to get important debugging information (and e.g. outgoing emails) printed by the Zulip development environment, you can use: docker logs--follow
To restart the server use: docker ps docker restart
To stop the server use: docker ps docker kill
If you want to connect to the Docker instance to run commands (e.g. build a release tarball), you can use: docker ps docker exec -it
If you want to run all the tests you need to start the servers first, you can do it with: docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash $ tools/test-all-docker
You can modify the source code in your development machine and review the results in your browser. Currently, the Docker workflow is substantially less convenient than the Vagrant workflow and less documented; please contribute to this guide and the Docker tooling if you are using Docker to develop Zulip!
102 Chapter 18. Zulip development environment setup without Vagrant CHAPTER 19
Using the Development Environment
Once the development environment is running, you can visit http://localhost:9991/ in your browser. By default, the development server homepage just shows a list of the users that exist on the server and you can login as any of them by just clicking on a user. This setup saves time for the common case where you want to test something other than the login process; to test the login process you’ll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTION case of zproject/settings.py from zproject.backends.DevAuthBackend to use the auth method(s) you’d like to test. While developing, it’s helpful to watch the run-dev.py console output, which will show any errors your Zulip development server encounters. To manually query the Postgres database, run psql zulip for an interactive console. When you make a change, here’s a guide for what you need to do in order to see your change take effect in Develop- ment: • If you change JavaScript, CSS, or Jinja2 backend templates (under templates/), you’ll just need to reload the browser window to see changes take effect. The Handlebars frontend HTML templates (static/ templates) are automatically recompiled by the tools/compile-handlebars-templates job, which runs as part of tools/run-dev.py. • If you change Python code used by the main Django/Tornado server processes, these services are run on top of Django’s manage.py runserver which will automatically restart the Zulip Django and Tornado servers whenever you save changes to Python code. You can watch this happen in the run-dev.py console to make sure the backend has reloaded. • The Python queue workers will also automatically restart when you save changes. However, you may need to ctrl-C and then restart run-dev.py manually if a queue worker has crashed. • If you change the database schema, you’ll need to use the standard Django migrations process to create and then run your migrations; see the new feature tutorial for an example. Additionally you should check out the detailed testing docs for how to run the tests properly after doing a migration. (In production, everything runs under supervisord and thus will restart if it crashes, and upgrade-zulip will take care of running migrations and then cleanly restaring the server for you).
103 Zulip Documentation, Release 1.7.0
104 Chapter 19. Using the Development Environment CHAPTER 20
Developing on a remote machine
The Zulip developer environment works well on remote virtual machines. This can be a good alternative for those with poor network connectivity or who have limited storage/memory on their local machines. We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04, with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.
20.1 Connecting to the remote environment
The best way to connect to your server is using the command line tool ssh. • On macOS and Linux/UNIX, ssh is a part of Terminal. • On Windows, ssh comes with Bash for Git. Open Terminal or Bash for Git, and connect with the following:
$ ssh username@host
If you have poor internet connectivity, we recommend using Mosh as it is more reliable over slow or unreliable networks.
20.2 Setting up the development environment
After you have connected to your remote server, you need to install the development environment. If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.
105 Zulip Documentation, Release 1.7.0
20.3 Running the development server
Once you have set up the development environment, you can start up the development instance of Zulip with the following command in the directory where you cloned Zulip:
./tools/run-dev.py--interface=''
This will start up the Zulip server on port 9991. You can then navigate to http://
You can port forward using ssh instead of running the development environment on an exposed interface. For more information, see Using the development environment.
106 Chapter 20. Developing on a remote machine Zulip Documentation, Release 1.7.0
20.4 Making changes to code on your remote development server
To see changes on your remote development server, you need to do one of the following: • Edit locally: Clone Zulip code to your computer and then use your favorite editor to make changes. When you want to see changes on your remote Zulip development instance, sync with Git. • Edit remotely: Edit code directly on your remote Zulip development instance using a Web-based IDE (recom- mended for beginners) or a command line editor.
20.4.1 Editing locally
If you want to edit code locally install your favorite text editor. If you don’t have a favorite, here are some suggestions: • atom • emacs • vim • spacemacs • sublime Next, follow our Git and GitHub Guide to clone and configure your fork of zulip on your local computer. Once you have cloned your code locally, you can get to work.
Syncing changes
The easiest way to see your changes on your remote development server is to push them to GitHub and them fetch and merge them from the remote server. For more detailed instructions about how to do this, see our Git & GitHub Guide. In brief, the steps are as follows. On your local computer: 1. Open Terminal (macOS/Linux) or Git for BASH. 2. Change directory to where you cloned Zulip (e.g. cd zulip). 3. Use git add and git commit to stage and commit your changes (if you haven’t already). 4. Push your commits to GitHub with git push origin branchname. Be sure to replace branchname with the name of your actual feature branch. Once git push has completed successfully, you are ready to fetch the commits from your remote development instance: 1. In Terminal or Git BASH, connect to your remote development instance with ssh user@host. 2. Change to the zulip directory (e.g., cd zulip). 3. Fetch new commits from GitHub with git fetch origin. 4. Change to the branch you want to work on with git checkout branchname. 5. Merge the new commits into your branch with git merge origin/branchname.
20.4. Making changes to code on your remote development server 107 Zulip Documentation, Release 1.7.0
20.4.2 Editing remotely
Web-based IDE
If you are relatively new to working on the command line, or just want to get started working quickly, we recommend web-based IDE Codeanywhere. To setup Codeanywhere for Zulip: 1. Create a Codeanywhere account and log in. 2. Create a new SFTP-SSH project. Use Public key for authentication. 3. Click GET YOUR PUBLIC KEY to get the new public key that Codeanywhere generates when you create a new project. Add this public key to ~/.ssh/authorized_keys on your remote development instance. 4. Once you’ve added the new public key to your remote development instance, click CONNECT.
Now your workspace should look similar this:
Command line editors
Another way to edit directly on the remote development server is with a command line text editor on the remote machine. Two editors often available by default on Linux systems are: • Nano: A very simple, beginner-friendly editor. However, it lacks a lot of features useful for programming, such as syntax highlighting, so we only recommended it for quick edits to things like configuration files. Launch by running command nano
108 Chapter 20. Developing on a remote machine Zulip Documentation, Release 1.7.0
• Vim: A very powerful editor that can take a while to learn. Launch by running vim
20.4.3 Next steps
Next, read the following to learn more about developing for Zulip: • Git & GitHub Guide • Using the Development Environment • Testing
20.4. Making changes to code on your remote development server 109 Zulip Documentation, Release 1.7.0
110 Chapter 20. Developing on a remote machine CHAPTER 21
Writing a new integration
Integrations are one of the most important parts of a group chat tool like Zulip, and we are committed to making integrating with Zulip and getting you integration merged upstream so everyone else can benefit from it as easy as possible while maintaining the high quality of the Zulip integrations library. On this page you’ll find: • An overview of the different types of integrations possible with Zulip. • General advice for writing integrations. • Details about writing webhook integrations. • Details about writing Python script and plugin integrations. • A guide to documenting your integration. A detailed walkthrough of a simple "Hello World" integration can be found in the webhook walkthrough. Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up with any tips or tools that help writing integration, please email [email protected], open an issue, or submit a pull request to share your ideas!
21.1 Types of integrations
We have several different ways that we integrate with 3rd party products, ordered here by which types we prefer to write: 1. Webhook integrations (examples: Freshdesk, GitHub), where the third-party service supports posting content to a particular URI on our site with data about the event. For these, you usually just need to create a new python package in the zerver/webhooks/ directory. You can easily find recent commits adding new integrations to crib from via git log zerver/webhooks/. 2. Python script integrations (examples: SVN, Git), where we can get the service to call our integration (by shelling out or otherwise), passing in the required data. Our preferred model for these is to ship these integrations in the Zulip Python API distribution, within the integrations directory there.
111 Zulip Documentation, Release 1.7.0
3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user needs to install a plugin into their existing software. These are often more work, but for some products are the only way to integrate with the product at all.
21.2 General advice
• Consider using our Zulip markup to make the output from your integration especially attractive or useful (e.g. emoji, markdown emphasis, @-mentions, or !avatar(email)). • Use topics effectively to ensure sequential messages about the same thing are threaded together; this makes for much better consumption by users. E.g. for a bug tracker integration, put the bug number in the topic for all messages; for an integration like Nagios, put the service in the topic. • Integrations that don’t match a team’s workflow can often be uselessly spammy. Give careful thought to provid- ing options for triggering Zulip messages only for certain message types, certain projects, or sending different messages to different streams/topics, to make it easy for teams to configure the integration to support their workflow. • Consistently capitalize the name of the integration in the documentation and the Client name the way the vendor does. It’s OK to use all-lower-case in the implementation. • Sometimes it can be helpful to contact the vendor if it appears they don’t have an API or webhook we can use – sometimes the right API is just not properly documented. • A helpful tool for testing your integration is UltraHook, which allows you to receive webhook calls via your local Zulip development environment. This enables you to do end-to-end testing with live data from the service you’re integrating and can help you spot why something isn’t working or if the service is using custom HTTP headers.
21.3 Webhook integrations
A webhook allows a third-party service to push data to you when something happens. It’s different from making a REST API call, where you send a request to the service’s API and wait for a response. With a webhook, the third- party service sends you an HTTP POST when it has something for you. Your webhook integration defines the URI the service uses to communicate with Zulip, and handles that incoming data. New Zulip webhook integrations can take just a few hours to write, including tests and documentation, if you use the right process. For detailed instructions, check out the "Hello World" webhook walkthrough. For a quick guide, read on. • First, use http://requestb.in/ or a similar site to capture an example webhook payload from the service you’re integrating. You can use these captured payloads to create a set of test fixtures for your integration under zerver/webhooks/mywebhook/fixtures/. • Then write a draft webhook handler under zerver/webhooks/; there are a lot of examples in that directory. We recommend templating off a short one (like stash or zendesk), since the longer ones usually just have more complex parsing which can obscure what’s common to all webhook integrations. In addition to writing the integration itself, you’ll need to create Integration object and add it to WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py; search for webhook in that file to find the existing ones (and please add yours in the alphabetically correct place). • Then write a test for your fixture in the tests.py file in the zerver/webhooks/mywebhook directory. You can now iterate on debugging the tests and webhooks handler until they work, all without ever needing to
112 Chapter 21. Writing a new integration Zulip Documentation, Release 1.7.0
post directly from the service you’re integrating with to your Zulip development machine. You can run just the tests for one integration like this:
test-backend zerver/webhooks/pagerduty/
See this guide for more details on the Zulip test runner. • Once you’ve gotten your webhook working and passing a test, capture payloads for the other common types of posts the service’s webhook will make, and add tests for them; usually this part of the process is pretty fast. Webhook integration tests should all use fixtures (as opposed to contacting the service), since otherwise the tests can’t run without Internet access and some sort of credentials for the service. • Finally, write documentation for the integration; there’s a detailed guide below.
21.3.1 Files that need to be created
Select a name for your webhook and use it consistently. The examples below are for a webhook named ’MyWebHook’. • static/images/integrations/logos/mywebhook.png: An image to represent your integration in the user interface. Generally this Should be the logo of the platform/server/product you are integrating. See Documenting your integration for details. • static/images/integrations/mywebbook/001.png: A screen capture of your integration for use in the user interface. You can add as many images as needed to effectively document your webhook integration. See Documenting your integration for details. • zerver/webhooks/mywebhook/fixtures/messagetype.json: Sample json payload data used by tests. Add one fixture file per type of message supported by your integration. See Testing and writing tests for details. • zerver/webhooks/mywebhook/__init__.py: Empty file that is obligatory part of every python pack- age. Remember to git add it. • zerver/webhooks/mywebhook/view.py: Includes the main webhook integration function including any needed helper functions. • zerver/webhooks/mywebhook/tests.py: Add tests for your webbook. See Testing and writing tests for details. • zerver/webhooks/mywebhook/doc.html: Add end-user documentation. See Documenting your inte- gration for details.
21.3.2 Files that need to be updated
• zerver/lib/integrations.py: Add your integration to WEBHOOK_INTEGRATIONS to register it. This will automatically register a url for the webhook of the form api/v1/external/mywebhook and associate with the function called api_mywebhook_webhook in zerver/webhooks/mywebhook/ view.py.
21.4 Python script and plugin integrations
For plugin integrations, usually you will need to consult the documentation for the third party software in order to learn how to write the integration. But we have a few notes on how to do these: • You should always send messages by POSTing to URLs of the form https://zulip.example.com/v1/ messages/.
21.4. Python script and plugin integrations 113 Zulip Documentation, Release 1.7.0
• We usually build Python script integration with (at least) 2 files: zulip_foo_config.py containing the configuration for the integration including the bots’ API keys, plus a script that reads from this configuration to actually do the work (that way, it’s possible to update the script without breaking users’ configurations). • Be sure to test your integration carefully and document how to install it (see notes on documentation below). • You should specify a clear HTTP User-Agent for your integration. The user agent should at a minimum identify the integration and version number, separated by a slash. If possible, you should collect platform information and include that in ()s after the version number. Some examples of ideal UAs are:
ZulipDesktop/0.7.0 (Ubuntu; 14.04) ZulipJenkins/0.1.0 (Windows; 7.2) ZulipMobile/0.5.4 (Android; 4.2; maguro)
21.5 Documenting your integration
See our guide for writing integration documentation.
114 Chapter 21. Writing a new integration CHAPTER 22
Documenting an integration
Every Zulip integration must be documented in zerver/webhooks/mywebhook/doc.md (or templates/ zerver/integrations/
./manage.py send_webhook_fixture_message \ --fixture=zerver/webhooks/pingdom/fixtures/imap_down_to_up.json \ '--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'
When generating the screenshot of a sample message, give your test bot a nice name like "GitHub Bot", use the project’s logo as the bot’s avatar, and take the screenshots showing the stream/topic bar for the message, not just the message body.
115 Zulip Documentation, Release 1.7.0
22.1 Markdown macros
Macros are elements in the format of {!macro.md!} that insert common phrases and steps at the location of the macros. Macros help eliminate repeated content in our documentation. The source for macros is the Markdown files under templates/zerver/help/include in the main Zulip server repository. If you find multiple instances of particular content in the documentation, you can always create a new macro by adding a new file to that folder. Here are a few common macros used to document Zulip’s integrations:
22.1.1 {!create-stream.md!} macro
• About: Recommends that users create a dedicated stream for a given integration. Usually the first step in setting up an integration or webhook. • Contents: See source. Note: {{ integration_display_name }} is replaced by Integra- tion.display_name and {{ recommended_stream_name }} is replaced by Integration.stream_name. • Example usage:
{!create-stream.md!}
• Example rendering:
First, create the stream you would like to use for GitLab notifications, and subscribe all interested parties to this stream. We recommend the name gitlab.
The integration will use the default stream gitlab if no stream is supplied in the URL; you still need to create the stream even if you are using this default.
22.1.2 {!create-bot-construct-url.md!} macro
• About: Instructs users to create a bot for a given integration and construct a webhook URL using the bot API key and stream name. The URL is generated automatically for every webhook by using attributes in the WebhookIntegration class. • Contents: See source. Note: If special configuration is required to set up the URL and you can’t use this macro, be sure to use the {{ external_api_uri }} template variable, so that your integration documentation will provide the correct URL for whatever server it is deployed on. If special configuration is required to set the SITE variable, you should document that too, inside an {% if api_site_required %} check. • Example usage:
{!create-bot-construct-url.md!}
Usually used right after {!create-stream!}. • Example rendering:
Next, on your Zulip settings page, create a bot for GitLab. Construct the URL for the GitLab bot using the bot API key and stream name:
https://yourZulipDomain.zulipchat.com/api/v1/external/gitlab?api_key=abcdefgh&
˓→stream=gitlab
116 Chapter 22. Documenting an integration Zulip Documentation, Release 1.7.0
Modify the parameters of the URL above, where api_key is the API key of your Zulip bot, and stream is the stream name you want the notifications sent to.
22.1.3 {!append-stream-name.md!} macro
• About: Recommends appending &stream=stream_name to a URL in cases where supplying a stream name in the URL is optional. • Contents: See source. • Example usage: Usually used right after {!create-bot-construct-url.md!}.
{!append-stream-name.md!}
• Example rendering:
To specify the stream, you must explicitly append `&stream=stream_name` to the end of the above URL, where `stream_name` is the stream you want the notifications sent to.
22.1.4 {!append-topic.md!} macro
• About: Recommends appending &topic=my_topic to a URL to supply a custom topic for webhook notifi- cation messages. • Contents: See source. • Example usage: Usually used right after {!create-bot-construct-url.md!}.
{!append-topic.md!}
• Example rendering:
To change the topic used by the bot, simply append `&topic=name` to the end of the above URL, where `name` is your topic.
22.1.5 {!congrats.md!} macro
• About: Inserts congratulatory lines signifying the successful setup of a given integration. • Contents: See source. • Example usage: Usually used at the end of the documentation, right before the sample message screenshot.
{!congrats.md!}
• Example rendering:
**Congratulations! You're done!**
Your WebhookName notifications may look like:
22.1. Markdown macros 117 Zulip Documentation, Release 1.7.0
22.1.6 {!download-python-bindings.md!} macro
• About: Links to Zulip’s API page to download and install Zulip’s API bindings. • Contents: See source. • Example usage: Currently mostly used in non-webhook integrations docs under templates/zerver/ integrations/
{!download-python-bindings.md!}
• Example rendering:
Download and install our [Python bindings and example scripts](/api) on the server where the IntegrationName bot will live.
22.1.7 {!change-zulip-config-file.md!} macro
• About: Instructs users to create a bot and specify said bot’s credentials in the config file for a given non-webhook integration. • Contents: See source. • Example usage: Usually used in non-webhook integration docs under templates/zerver/ integrations/
{!change-zulip-config-file.md!}
• Example rendering:
On your Zulip settings page, create a bot for Codebase.
Next, open `integrations/codebase/zulip_codebase_config.py` with your favorite editor, and change the following lines to specify the email address and API key for your Codebase bot:
ZULIP_USER = "[email protected]" ZULIP_API_KEY = "0123456789abcdef0123456789abcdef" ZULIP_SITE = "http://localhost:9991/api"
22.1.8 {!git-append-branches.md!} and {!git-webhook-url-with-branches. md!}
• About: These two macros explain how to specify a list of branches in the webhook URL to filter notifications in our Git-related webhooks. • Contents: See git-append-branches and git-webhook-url-with-branches. • Example usage: Used exclusively in Git integrations.
{!git-append-branches.md!} {!git-webhook-url-with-branches.md!}
118 Chapter 22. Documenting an integration Zulip Documentation, Release 1.7.0
22.1.9 Other useful macros
• {!webhook-url.md!} - Used internally by {!create-bot-construct-url.md!} to generate the webhook URL. See source. • {!zulip-config.md!} - Used internally by {!change-zulip-config-file.md!} to specify the lines in the config file for a non-webhook integration. See source. • {!webhook-url-with-bot-email.md!} - Used in certain non-webhook integrations to generate URLs of the form (see source):
https://bot_email:[email protected]/api/v1/external/
˓→beanstalk
A good example is Zulip’s Beanstalk integration
22.1. Markdown macros 119 Zulip Documentation, Release 1.7.0
120 Chapter 22. Documenting an integration CHAPTER 23
Webhook walkthrough
Below explains each part of a simple webhook integration, called Hello World. This webhook sends a "hello" message to the test stream and includes a link to the Wikipedia article of the day, which it formats from json data it receives in the http request. Use this walkthrough to learn how to write your first webhook integration.
23.1 Step 0: Create fixtures
The first step in creating a webhook is to examine the data that the service you want to integrate will be sending to Zulip. You can use http://requestb.in or a similar tool to capture webhook payload(s) from the service you are integrating. Examining this data allows you to do two things: 1. Determine how you will need to structure your webhook code, including what message types your integration should support and how; and, 2. Create fixtures for your webhook tests. A test fixture is a small file containing test data, one for each test. Fixtures enable the testing of webhook integration code without the need to actually contact the service being integrated. Because Hello World is a very simple webhook that does one thing, it requires only one fixture, zerver/ webhooks/helloworld/fixtures/hello.json:
{ "featured_title":"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe", }
When writing your own webhook integration, you’ll want to write a test function for each distinct message condition your webhook supports. You’ll also need a corresponding fixture for each of these tests. Depending on the type of data the 3rd party service sends, your fixture may contain JSON, URL encoded text, or some other kind of data. See Step 4: Create tests or Testing for further details.
121 Zulip Documentation, Release 1.7.0
23.2 Step 1: Initialize your webhook python package
In the zerver/webhooks/ directory, create new subdirectory that will contain all of corresponding code. In our example it will be helloworld. The new directory will be a python package, so you have to create an empty __init__.py file in that directory via e.g. touch zerver/webhooks/helloworld/__init__.py.
23.3 Step 2: Create main webhook code
The majority of the code for your webhook integration will be in a single python file, zerver/webhooks/ mywebhook/view.py. The Hello World integration is in zerver/webhooks/helloworld/view.py: from django.utils.translation import ugettext as _ from zerver.lib.actions import check_send_stream_message from zerver.lib.response import json_success, json_error from zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view from zerver.lib.validator import check_dict, check_string from zerver.models import Client, UserProfile from django.http import HttpRequest, HttpResponse from typing import Dict, Any, Iterable, Optional, Text
@api_key_only_webhook_view('HelloWorld') @has_request_variables def api_helloworld_webhook(request, user_profile, payload=REQ(argument_type='body'), stream=REQ(default='test'), topic=REQ(default='Hello World')): # type: (HttpRequest, UserProfile, Dict[str, Iterable[Dict[str, Any]]], Text,
˓→Optional[Text]) -> HttpResponse
# construct the body of the message body='Hello! I am happy to be here! :smile:'
# try to add the Wikipedia article of the day body_template=' \nThe Wikipedia featured article for today is **[{featured_title} ˓→]({featured_url})**' body+= body_template.format( **payload)
# send the message check_send_stream_message(user_profile, request.client, stream, topic, body)
# return json result return json_success()
The above code imports the required functions and defines the main webhook function api_helloworld_webhook, decorating it with api_key_only_webhook_view and has_request_variables. The has_request_variables decorator allows you to access request variables with REQ(). You can find more about REQ and request variables in Writing views. You must pass the name of your webhook to the api_key_only_webhook_view decorator so your webhook can access the user_profile and request.client (Zulip’s analogue of UserAgent) fields from the request. Here
122 Chapter 23. Webhook walkthrough Zulip Documentation, Release 1.7.0 we have used HelloWorld. To be consistent with Zulip code style, use the name of the product you are integrating in camel case, spelled as the product spells its own name (except always first letter upper-case). The api_key_only_webhook_view decorator indicates that the 3rd party service will send the authorization as an API key in the query parameters. If your service uses HTTP Basic authentication, you would instead use the authenticated_rest_api_view decorator. You should name your webhook function as such api_webhookname_webhook where webhookname is the name of your webhook and is always lower-case. At minimum, the webhook function must accept request (Django HttpRequest object), and user_profile (Zulip’s user object). You may also want to define additional parameters using the REQ object. In the example above, we have defined payload which is populated from the body of the http request, stream with a default of test (available by default in the Zulip development environment), and topic with a default of Hello World. If your webhook uses a custom stream, it must exist before a message can be created in it. (See Step 4: Create tests for how to handle this in tests.) The line that begins # type is a mypy type annotation. See this page for details about how to properly annotate your webhook functions. In the body of the function we define the body of the message as Hello! I am happy to be here! :smile:. The :smile: indicates an emoji. Then we append a link to the Wikipedia article of the day as pro- vided by the json payload. • Sometimes, it might occur that a json payload does not contain all required keys your integration checks for. In such a case, any KeyError thrown is handled by the server backend and will create an appropriate response. Then we send a public (stream) message with check_send_stream_message which will validate the message and then send it. Finally, we return a 200 http status with a JSON format success message via json_success().
23.4 Step 3: Create an api endpoint for the webhook
In order for a webhook to be externally available, it must be mapped to a url. This is done in zerver/lib/ integrations.py. Look for the lines beginning with:
WEBHOOK_INTEGRATIONS=[
And you’ll find the entry for Hello World:
WebhookIntegration('helloworld',['misc'], display_name='Hello World'),
This tells the Zulip api to call the api_helloworld_webhook function in zerver/webhooks/ helloworld/view.py when it receives a request at /api/v1/external/helloworld. This line also tells Zulip to generate an entry for Hello World on the Zulip integrations page using static/images/ integrations/logos/helloworld.png as its icon. The second positional argument defines a list of cate- gories for the integration. At this point, if you’re following along and/or writing your own Hello World webhook, you have written enough code to test your integration. First, get an API key from the Your bots section of your Zulip user’s Settings page. If you haven’t created a bot already, you can do that there. Then copy its API key and replace the placeholder "
23.4. Step 3: Create an api endpoint for the webhook 123 Zulip Documentation, Release 1.7.0
Now you can test using Zulip itself, or curl on the command line. Using manage.py from within the Zulip development environment:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/webhooks/helloworld/fixtures/hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key=
After which you should see something similar to:
2016-07-07 15:06:59,187 INFO 127.0.0.1 POST 200 143ms (mem:6ms/13) (md:
˓→43ms/1) (db: 20ms/9q) (+start: 147ms)/api/v1/external/helloworld (helloworld-
˓→[email protected] via ZulipHelloWorldWebhook)
Using curl: curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn
˓→Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://
˓→localhost:9991/api/v1/external/helloworld\?api_key\=
After which you should see:
{"msg":"","result":"success"}
Using either method will create a message in Zulip:
23.5 Step 4: Create tests
Every webhook integration should have a corresponding test file: zerver/webhooks/mywebhook/tests.py. The Hello World integration’s tests are in zerver/webhooks/helloworld/tests.py You should name the class
# Note: Include a test function per each distinct message condition your
˓→integration supports def test_hello_message(self): # type: () -> None expected_subject=u"Hello World"; expected_message=u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/ ˓→Marilyn_Monroe)**";
# use fixture named helloworld_hello
124 Chapter 23. Webhook walkthrough Zulip Documentation, Release 1.7.0
self.send_and_test_stream_message('hello', expected_subject, expected_message, content_type="application/x-www-form-
˓→urlencoded")
def get_body(self, fixture_name): # type: (Text) -> Text return self.fixture_data("helloworld", fixture_name, file_type="json")
In the above example, STREAM_NAME, URL_TEMPLATE, and FIXTURE_DIR_NAME refer to class attributes from the base class, WebhookTestCase. These are needed by the helper function send_and_test_stream_message to determine how to execute your test. STREAM_NAME should be set to your default stream. If it doesn’t exist, send_and_test_stream_message will create it while executing your test. If your test expects a stream name from a test fixture, the value in the fixture and the value you set for STREAM_NAME must match. The test helpers use STREAM_NAME to create the destination stream, and then create the message to send using the value from the fixture. If these don’t match, the test will fail. URL_TEMPLATE defines how the test runner will call your webhook, in the same way you would provide a webhook URL to the 3rd party service. api_key={api_key} says that an API key is expected. In get_body, the first argument in the call to self.fixture_data specifies the prefix of your fixture file names, and file_type their type. Common types are json and txt. When writing tests for your webhook, you’ll want to include one test function (and corresponding fixture) per each distinct message condition that your integration supports. If, for example, we added support for sending a goodbye message to our Hello World webhook, we would add another test function to HelloWorldHookTests class called something like test_goodbye_message:
def test_goodbye_message(self): # type: () -> None expected_subject=u"Hello World"; expected_message=u"Hello! I am happy to be here! :smile: \nThe Wikipedia ˓→featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";
# use fixture named helloworld_goodbye self.send_and_test_stream_message('goodbye', expected_subject, expected_
˓→message, content_type="application/x-www-form-
˓→urlencoded")
As well as a new fixture goodbye.json in zerver/webhooks/helloworld/fixtures/:
{ "featured_title":"Goodbye", "featured_url":"https://en.wikipedia.org/wiki/Goodbye", }
Also consider if your integration should have negative tests, a test where the data from the test fixture should result in an error. For details see Negative tests, below. Once you have written some tests, you can run just these new tests from within the Zulip development environment with this command:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./tools/test-backend zerver/webhooks/helloworld
23.5. Step 4: Create tests 125 Zulip Documentation, Release 1.7.0
(Note: You must run the tests from the top level of your development directory. The standard location in a Vagrant environment is /srv/zulip. If you are not using Vagrant, use the directory where you have your development environment.) You will see some script output and if all the tests have passed, you will see:
Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_goodbye_message Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_hello_message DONE!
23.6 Step 5: Create documentation
Next, we add end-user documentation for our webhook integration. You can see the existing examples at https: //zulipchat.com/integrations or by accessing /integrations in your Zulip development environment. There are two parts to the end-user documentation on this page. The first is the lozenge in the grid of integrations, showing your integration logo and name, which links to the full documentation. This is generated automatically once you’ve registered the integration in WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py, and supports some customization via options to the WebhookIntegration class. Second, you need to write the actual documentation content in zerver/webhooks/mywebhook/doc.md.
Learn how Zulip integrations work with this simple Hello World example!
The Hello World webhook will use the `test` stream, which is created by default in the Zulip dev environment. If you are running Zulip in production, you should make sure that this stream exists.
Next, on your {{ settings_html|safe }}, create a Hello World bot. Construct the URL for the Hello World bot using the API key and stream name:
`{{ external_api_uri_subdomain }}/v1/external/helloworld?api_key=abcdefgh&stream=test`
To trigger a notification using this webhook, use `send_webhook_fixture_message` from the Zulip command line:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$ ./manage.py send_webhook_fixture_message \ > --fixture=zerver/fixtures/helloworld/hello.json \ > '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'
Or, use curl:
curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn
˓→Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://
˓→localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>
{!congrats.md!}
{!congrats.md!} is an example of a Markdown macro. Zulip has a macro-based Markdown/Jinja2 framework that includes macros for common instructions in Zulip’s webhooks/integrations documentation.
126 Chapter 23. Webhook walkthrough Zulip Documentation, Release 1.7.0
See Documenting your integration for further details, including how to easily create the message screenshot.
23.7 Step 5: Preparing a pull request to zulip/zulip
When you have finished your webhook integration and are ready for it to be available in the Zulip product, follow these steps to prepare your pull request: 1. Run tests including linters and ensure you have addressed any issues they report. See Testing and Linters for details. 2. Read through Code styles and conventions and take a look through your code to double-check that you’ve followed Zulip’s guidelines. 3. Take a look at your git history to ensure your commits have been clear and logical (see Version Control for tips). If not, consider revising them with git rebase --interactive. For most webhooks, you’ll want to squash your changes into a single commit and include a good, clear commit message. 4. Push code to your fork. 5. Submit a pull request to zulip/zulip. If you would like feedback on your integration as you go, feel free to post a message on the public Zulip instance. You can also create a [WIP] pull request while you are still working on your integration. See the Git guide for more on Zulip’s pull request process.
23.8 Advanced topics
More complex implementation or testing needs may require additional code, beyond what the standard helper functions provide. This section discusses some of these situations.
23.8.1 Negative tests
A negative test is one that should result in an error, such as incorrect data. The helper functions may interpret this as a test failure, when it should instead be a successful test of an error condition. To correctly test these cases, you must explicitly code your test’s execution (using other helpers, as needed) rather than call the usual helper function. Here is an example from the WordPress webhook:
def test_unknown_action_no_data(self): # type: () -> None
# Mimic send_and_test_stream_message() to manually execute a negative test. # Otherwise its call to send_json_payload() would assert on the non-success # we are testing. The value of result is the error message the webhook should # return if no params are sent. The fixture for this test is an empty file.
# subscribe to the target stream self.subscribe(self.test_user, self.STREAM_NAME)
# post to the webhook url post_params={'stream_name': self.STREAM_NAME, 'content_type':'application/x-www-form-urlencoded'} result= self.client_post(self.url,'unknown_action', **post_params)
23.7. Step 5: Preparing a pull request to zulip/zulip 127 Zulip Documentation, Release 1.7.0
# check that we got the expected error message self.assert_json_error(result,"Unknown WordPress webhook action: WordPress Action
˓→")
In a normal test, send_and_test_stream_message would handle all the setup and then check that the web- hook’s response matches the expected result. If the webhook returns an error, the test fails. Instead, explicitly do the setup it would have done, and check the result yourself. Here, subscribe_to_stream is a test helper that uses TEST_USER_EMAIL and STREAM_NAME (attributes from the base class) to register the user to receive messages in the given stream. If the stream doesn’t exist, it creates it. client_post, another helper, performs the HTTP POST that calls the webhook. As long as self.url is correct, you don’t need to construct the webhook URL yourself. (In most cases, it is.) assert_json_error then checks if the result matches the expected error. If you had used send_and_test_stream_message, it would have called send_json_payload, which checks the result with assert_json_success.
23.8.2 Custom query parameters
Custom arguments passed in URL query parameters work as expected in the webhook code, but require special han- dling in tests. For example, here is the definition of a webhook function that gets both stream and topic from the query parame- ters:
def api_querytest_webhook(request, user_profile, client, payload=REQ(argument_type='body'), stream=REQ(default='test
˓→'), topic=REQ(default='Default Alert')):
In actual use, you might configure the 3rd party service to call your Zulip integration with a URL like this:
http://myhost/api/v1/external/querytest?api_key=abcdefgh&stream=alerts&topic=queries
It provides values for stream and topic, and the webhook can get those using REQ without any special handling. How does this work in a test? The new attribute TOPIC exists only in our class so far. In order to construct a URL with a query parameter for topic, you can pass the attribute TOPIC as a keyword argument to build_webhook_url, like so:
class QuerytestHookTests(WebhookTestCase):
STREAM_NAME='querytest' TOPIC="Default Topic" URL_TEMPLATE="/api/v1/external/querytest?api_key= {api_key}&stream={stream}" FIXTURE_DIR_NAME='querytest'
def test_querytest_test_one(self): # type: () -> None
# construct the URL used for this test self.TOPIC=u"Query Test" self.url= self.build_webhook_url(topic=self.TOPIC)
# define the expected message contents
128 Chapter 23. Webhook walkthrough Zulip Documentation, Release 1.7.0
expected_subject=u"Query Test" expected_message=u"This is a test of custom query parameters."
self.send_and_test_stream_message('test_one', expected_subject, expected_
˓→message, content_type="application/x-www-form-
˓→urlencoded")
def get_body(self, fixture_name): # type: (Text) -> Text return self.fixture_data("querytest", fixture_name, file_type="json")
You can also override get_body if your test data needs to be constructed in an unusual way. For more, see the definition for the base class, WebhookTestCase in zerver/lib/test_classes.py.
23.8. Advanced topics 129 Zulip Documentation, Release 1.7.0
130 Chapter 23. Webhook walkthrough CHAPTER 24
Interactive bots
Zulip’s API has a powerful framework for interactive bots that react the messages in Zulip. This page documents how to run a bot implemented using that framework, both on your laptop for quick testing as well in a production server environment. On this page you’ll find: • A step-by-step tutorial on how to run a bot. •A guide on running a Zulip botserver. • Common problems when developing/running bots and their solutions.
24.1 Installing the zulip_bots package
24.2 Running a bot
This guide will show you how to run a bot on a running Zulip server. It assumes you want to use one of the existing bots found in zulip_bots/bots in your Zulip organization. Hint: Looking for an easy way to test a bot’s output? Check out this guide. You need: • An account in an organization on a Zulip server (e.g. chat.zulip.org or yourSubdomain.zulipchat.com, or your own development server). Within that Zulip organization, users will be able to interact with your bot. • A computer where you’re running the bot from. Note: Please be considerate when testing experimental bots on public servers such as chat.zulip.org. 1. Run pip install zulip_bots to install the package. Hint: Do you want to install the latest development version? Check out this guide. 1. Register a new bot user on the Zulip server’s web interface.
131 Zulip Documentation, Release 1.7.0
• Log in to the Zulip server. • Navigate to Settings () -> Your bots -> Add a new bot. Select Generic bot for bot type, fill out the form and click on Create bot. • A new bot user should appear in the Active bots panel. 2. Download the bot’s .zuliprc configuration file to your computer. • In the Active bots panel, click on the little green download icon to download its configuration file .zuliprc (the structure of this file is explained here). • Copy the file to a destination of your choice, e.g. to ~/.zuliprc. 3. Run the bot. • Run
zulip-run-bot
(using the path to the .zuliprc file from step 3). • Check the output of the command. It should start with the text the usage function returns, followed by logging output similar to this:
INFO:root:starting message handling... INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
˓→(1): localhost
• Congrats! Your bot is running. You can now play around with the bot and get it configured the way you like. Eventually, you’ll probably want to run it in a production environment where it’ll stay up, by deploying it on a server using the Zulip Botserver.
24.3 Zulip Botserver
The Zulip Botserver is for people who want to • run bots in production. • run multiple bots at once. The Zulip Botserver is a Python (Flask) server that implements Zulip’s Outgoing Webhooks API. You can of course write your own servers using the Outgoing Webhooks API, but the Botserver is designed to make it easy for a novice Python programmer to write a new bot and deploy it in production.
24.3.1 Installing the Zulip Botserver
Install the zulip_botserver PyPI package using pip: pip install zulip_botserver
24.3.2 Running bots using the Zulip Botserver
1. Register new bot users on the Zulip server’s web interface. • Log in to the Zulip server.
132 Chapter 24. Interactive bots Zulip Documentation, Release 1.7.0
• Navigate to Settings () -> Your bots -> Add a new bot. Select Outgoing webhook for bot type, fill out the form and click on Create bot. • A new bot user should appear in the Active bots panel. 2. Download the flaskbotrc from the your-bots settings page. It contains the configuration details for all the active outgoing webhook bots. It’s structure is very similar to that of .zuliprc. 3. Run the Zulip Botserver by passing the flaskbotrc to it. The command format is:
zulip-bot-server--config-file
˓→
If omitted, hostname defaults to 127.0.0.1 and port to 5002. 4. Now set up the outgoing webhook service which will interact with the server: Create an Outgoing webhook bot with its base url of the form:
http://
bot_name refers to the name in the email address you specified for the bot. It can be obtained by removing -bot@*.* from the bot email: For example, the bot name of a bot with an email followup-bot@zulip. com is followup. In the development environment, an outgoing webhook bot and corresponding service already exist, with the email [email protected]. This can be used for interacting with flask server bots. 5. Congrats, everything is set up! Test your botserver like you would test a normal bot.
24.3.3 Running Zulip Botserver with supervisord
supervisord is a popular tool for running services in production. It helps ensure the service starts on boot, manages log files, restarts the service if it crashes, etc. This section documents how to run the Zulip Botserver using supervisord. Running the Zulip Botserver with supervisord works almost like running it manually. 1. Install supervisord via your package manager; e.g. on Debian/Ubuntu:
sudo apt-get install supervisor
2. Configure supervisord. supervisord stores its configuration in /etc/supervisor/conf.d. • Do one of the following: – Download the sample config file and store it in /etc/supervisor/conf.d/ zulip-botserver.conf. – Copy the following section into your existing supervisord config file.
[program:zulip-bot-server] command=zulip-bot-server--config-file=
˓→hostname
--port˓→botserver will be logged here redirect_stderr=true
• Edit the <> sections according to your preferences. 3. Update supervisord to read the configuration file:
24.3. Zulip Botserver 133 Zulip Documentation, Release 1.7.0
supervisorctl reread supervisorctl update
(or you can use /etc/init.d/supervisord restart, but this is less disruptive if you’re using super- visord for other services as well). 4. Test if your setup is successful:
supervisorctl status
The output should include a line similar to this: zulip-bot-server RUNNING pid 28154, uptime 0:00:27 The standard output of the bot server will be logged to the path in your supervisord configuration.
24.4 Common problems
• My bot won’t start – Ensure that your API config file is correct (download the config file from the server). – Ensure that you bot script is located in zulip_bots/bots/
134 Chapter 24. Interactive bots CHAPTER 25
Writing interactive bots
Zulip’s API supports a few different ways of integrating with a third-party service. • Incoming webhook integrations, for when you just want notifications from a tool to be sent into Zulip. See the integrations guide. • Interactive bots, for when you want the tool to react to messages in Zulip. • This guide is about writing and testing interactive bots. We assume familiarity with our guide for running bots. On this page you’ll find: • A step-by-step guide on how to set up a development environment for writing bots with all of our nice tooling to make it easy to write and test your work. •A guide on writing a bot. •A guide on adding a bot to Zulip. •A guide on testing a bot’s output. •A documentation of the bot API. • Common problems when developing/running bots and their solutions.
25.1 Installing a development version of the zulip_bots package
1. git clone https://github.com/zulip/python-zulip-api.git - clone the python-zulip-api repository. 2. cd python-zulip-api - navigate into your cloned repository. 3. ./tools/provision - install all requirements in a Python virtualenv. 4. Run the source
135 Zulip Documentation, Release 1.7.0
Hint: ./tools/provision installs zulip, zulip_bots, and zulip_botserver in developer mode. This enables you to make changes to the code after the packages are installed.
25.2 Writing a bot
The tutorial below explains the structure of a bot
class MyBotHandler(object): ''' A docstring documenting this bot. '''
def usage(self): return '''Your description of the bot'''
def handle_message(self, message, bot_handler, state_handler): # add your code here
handler_class= MyBotHandler
• The class name (in this case MyBotHandler) can be defined by you and should match the name of your bot. To register your bot’s class, adjust the last line handler_class = MyBotHandler to match your class name. • Every bot needs to implement the functions – usage(self) – handle_message(self, message, bot_handler) • These functions are documented in the next section.
25.3 Adding a bot to Zulip
Zulip’s bot system resides in the python-zulip-api repository. The structure of the bots ecosystem looks like the following:
zulip_bots --zulip_bots --bots | --bot1 | --bot2 | | | --bot2.py | --bot2.conf | --doc.md | --test_bot2.py | --assets | | | | | --pic.png | --fixtures | | | | | --test1.json
136 Chapter 25. Writing interactive bots Zulip Documentation, Release 1.7.0
| --libraries | | | --lib1.py -- lib.py -- test_lib.py -- run.py -- provision.py
Each subdirectory in bots contains a bot. When writing bots, try to use the structure outlined above as an orientation.
25.4 Testing a bot’s output
If you just want to see how a bot reacts to a message, but don’t want to set it up on a server, we have a little tool to help you out: zulip-bot-output • Install all requirements. • Run zulip-bot-output
25.5 Bot API
This section documents functions available to the bot and the structure of the bot’s config file. With this API, you can • intercept, view, and process messages sent by users on Zulip. • send out new messages as replies to the processed messages. With this API, you cannot • modify an intercepted message (you have to send a new message). • send messages on behalf of or impersonate other users. • intercept private messages (except for PMs with the bot as an explicit recipient).
25.5.1 usage usage(self) is called to retrieve information about the bot.
25.4. Testing a bot’s output 137 Zulip Documentation, Release 1.7.0
Arguments
• self - the instance the method is called on.
Return values
• A string describing the bot’s functionality
Example implementation def usage(self): return ''' This plugin will allow users to flag messages as being follow-up items. Users should preface messages with"@followup". Before running this, make sure to create a stream called"followup" that your API user can send to. '''
25.5.2 handle_message handle_message(self, message, bot_handler) handles user message.
Arguments
• self - the instance the method is called on. • message - a dictionary describing a Zulip message • bot_handler - used to interact with the server, e.g. to send a message • state_handler - used to save states/information of the bot beta – use state_handler.set_state(state) to set a state (any object) – use state_handler.get_state() to retrieve the state set; returns a NoneType object if no state is set
Return values
None.
Example implementation
def handle_message(self, message, bot_handler, state_handler): original_content= message['content'] original_sender= message['sender_email'] new_content= original_content.replace('@followup', 'from %s:'% (original_sender,))
138 Chapter 25. Writing interactive bots Zulip Documentation, Release 1.7.0
bot_handler.send_message(dict( type='stream', to='followup', subject=message['sender_email'], content=new_content, ))
25.5.3 bot_handler.send_message bot_handler.send_message(message) will send a message as the bot user. Generally, this is less convenient than send_reply, but it offers additional flexibility about where the message is sent to.
Arguments
• message - a dictionary describing the message to be sent by the bot
Example implementation bot_handler.send_message(dict( type='stream', # can be 'stream' or 'private' to=stream_name, # either the stream name or user's email subject=subject, # message subject content=message, # content of the sent message ))
25.5.4 bot_handler.send_reply bot_handler.send_reply(message, response) will reply to the triggering message to the same place the original message was sent to, with the content of the reply being response.
Arguments
• message - Dictionary containing information on message to respond to (provided by handle_message). • response - Response message from the bot (string).
25.5.5 bot_handler.update_message bot_handler.update_message(message) will edit the content of a previously sent message.
Arguments
• message - dictionary defining what message to edit and the new content
25.5. Bot API 139 Zulip Documentation, Release 1.7.0
Example
From zulip_bots/bots/incrementor/incrementor.py:
bot_handler.update_message(dict( message_id=self.message_id, # id of message to be updated content=str(self.number), # string with which to update message with ))
25.5.6 Configuration file
[api] key=
• key - the API key you created for the bot; this is how Zulip knows the request is from an authorized user. • email - the email address of the bot, e.g. [email protected] • site - your development environment URL; if you are working on a development environment hosted on your computer, use localhost:9991
25.6 Writing tests for bots
Bots, like most software that you want to work, should have unit tests. In this section, we detail our framework for writing unit tests for bots. We require that bots in the main python-zulip-api repository include a reasonable set of unit tests, so that future developers can easily refactor them. Unit tests for bots make heavy use of mocking. If you want to get comfortable with mocking, mocking strategies, etc. you should check out our mocking guide.
25.6.1 A simple example
Let’s have a look at a simple test suite for the helloworld bot (the actual test is written slightly more compact).
from __future__ import absolute_import
from zulip_bots.test_lib import BotTestCase # The test system library class TestHelloWorldBot(BotTestCase): bot_name="helloworld" # The bot's name (should be the name of the bot module
˓→to test).
def test_bot(self): # A test case (must start with `test`) # Messages we want to test and the expected bot responses. message_response_pairs={"":"beep boop", "foo":"beep boop", "Hi, my name is abc":"beep boop"} self.check_expected_responses(message_response_pairs) # Test the bot with
˓→our message_response_pair dict.
140 Chapter 25. Writing interactive bots Zulip Documentation, Release 1.7.0
The helloworld bot replies with "beep boop" to every message @-mentioning it. Note that our helper method check_expected_responses adds the @-mention for us - the only thing we need to do is to specify the rest of the message and the expected response. In this case, we want to assert that the bot always replies with "beep boop". To do so, we specify several test messages ("", "foo", "Hi, my name is abc") and assert that the response is always correct, which for this simple bot, means always sending a reply with the content "beep boop".
25.6.2 Testing your test
Once you have written a test suite, you want to verify that everything works as expected. • To test a bot in Zulip’s bot directory: tools/test-bots
25.6.3 Advanced testing
This section shows advanced testing techniques for more complicated bots that have configuration files or interact with third-party APIs. The code for the bot testing library can be found here.
Asserting individual messages
self.assert_bot_response( message={'content':'foo'}, response={'content':'bar'}, expected_method='send_reply' )
Use assert_bot_response() to test individual messages. Specify additional message settings, such as the stream or subject, in the message and response dicts.
Testing bots with config files
Some bots, such as Giphy, support or require user configuration options to control how the bot works. To test such a bot, you can use the following helper method:
with self.mock_config_info({'entry':'value'}): # self.assert_bot_response(...)
mock_config_info() mocks a bot’s config file. All config files are specified in the .ini format, with one default section. The dict passed to mock_config_info() specifies the keys and values of that section.
Testing bots with internet access
Some bots, such as Giphy, depend on a third-party we service, such as the Giphy webapp, in order to work. Because we want our test suite to be reliable and not add load to these third-party APIs, tests for these services need to have "test fixtures": sample HTTP request/response pairs to be used by the tests. You can specify which one to use in your test code using the following helper method:
with self.mock_http_conversation('test_fixture_name'): # self.assert_bot_response(...)
25.6. Writing tests for bots 141 Zulip Documentation, Release 1.7.0
mock_http_conversation(fixture_name) patches requests.get and returns the data specified in the file fixtures/
Testing bots that specify initialize()
Some bots, such as Giphy, implement an initialize() method, which is executed on the startup of the bot. To test such a bot, you can call its initialize() method with the following helper method:
self.initialize_bot()
Calling initialize_bot() invokes the initialize() method specified by the bot.
Examples
Check out our bots to see examples of bot tests.
25.7 Common problems
• I modified my bot’s code, yet the changes don’t seem to have an effect. – Ensure that you restarted the zulip-run-bot script. • My bot won’t start – Ensure that your API config file is correct (download the config file from the server). – Ensure that you bot script is located in zulip_bots/bots/
25.8 Future direction
The long-term plan for this bot system is to allow the same ExternalBotHandler code to eventually be usable in several contexts: • Run directly using the Zulip call_on_each_message API, which is how the implementation above works. This is great for quick development with minimal setup. • Run in a simple Python webserver server, processing messages received from Zulip’s outgoing webhooks inte- gration. • For bots merged into the mainline Zulip codebase, enabled via a button in the Zulip web UI, with no code deployment effort required.
142 Chapter 25. Writing interactive bots CHAPTER 26
Writing a new application feature
The changes needed to add a new feature will vary, of course, but this document provides a general outline of what you may need to do, as well as an example of the specific steps needed to add a new feature: adding a new option to the application that is dynamically synced through the data system in real-time to all browsers the user may have open. As you read this, you may find you need to learn about Zulip’s real-time push system; the real-time push and events documentation has a detailed explanation of how everything works.
26.1 General Process
26.1.1 Files impacted
This tutorial will walk through adding a new feature to a Realm (an organization in Zulip). The following files are involved in the process: Backend • zerver/model.py: Defines the database model. • zerver/views/realm.py: The view function that implements the API endpoint for editing realm objects. • zerver/lib/actions.py: Contains code for updating and interacting with the database. • zerver/lib/events.py: Ensures that the state Zulip sends to clients is always consistent and correct. Frontend • static/templates/settings/organization-permissions-admin.handlebars: defines the structure of the admin permissions page (checkboxes for each organization permission setting). • static/js/settings_org.js: handles organization setting form submission. • static/js/server_events_dispatch.js: handles events coming from the server (ex: pushing an organization change to other open browsers and updating the application’s state). Backend Testing
143 Zulip Documentation, Release 1.7.0
• zerver/tests/test_realm.py: end-to-end API tests for updating realm settings. • zerver/tests/test_events.py: tests for possible race bugs in the zerver/lib/events.py implementation. Frontend Testing • frontend_tests/casper_tests/10-admin.js: end-to-end tests for the organization admin settings pages. • frontend_tests/node_tests/dispatch.js
26.1.2 Adding a field to the database
Update the model: The server accesses the underlying database in zerver/models.py. Add a new field in the appropriate class. Create and run the migration: To create and apply a migration, run:
./manage.py makemigrations ./manage.py migrate
Test your changes: Once you’ve run the migration, flush memcached on your development server (./scripts/ setup/flush-memcached) and then restart the development server to avoid interacting with cached objects.
26.1.3 Backend changes
We have a framework that automatically handles many of the steps for the most common types of UserProfile and Realm settings. We refer to this as the property_types framework. However, it is valuable to understand the flow of events even if the property_types framework means you don’t have to write much code for a new setting. Database interaction: Add any necessary code for updating and interacting with the database in zerver/lib/ actions.py. It should update the database and send an event announcing the change. Application state: Modify the fetch_initial_state_data and apply_event functions in zerver/lib/ events.py to update the state based on the event you just created. Backend implementation: Make any other modifications to the backend required for your feature to do what it’s supposed to do (this will be unique to the feature you’re implementing). New views: Add any new application views to zproject/urls.py, or update the appropriate existing view in zerver/views/. This includes both views that serve HTML (new pages on Zulip) as well as new API endpoints that serve JSON-formatted data. Testing: At the very least, add a test of your event data flowing through the system in test_events.py and an API test (e.g. for a Realm setting, in test_realm.py).
26.1.4 Frontend changes
JavaScript: Zulip’s JavaScript is located in the directory static/js/. The exact files you may need to change depend on your feature. If you’ve added a new event that is sent to clients, be sure to add a handler for it in static/ js/server_events_dispatch.js. CSS: The primary CSS file is static/styles/zulip.css. If your new feature requires UI changes, you may need to add additional CSS to this file. Templates: The initial page structure is rendered via Jinja2 templates located in templates/zerver. For JavaScript, Zulip uses Handlebars templates located in static/templates. Templates are precompiled as part of the build/deploy process.
144 Chapter 26. Writing a new application feature Zulip Documentation, Release 1.7.0
Zulip is fully internationalized, so when writing both HTML templates or JavaScript code that generates user-facing strings, be sure to tag those strings for translation. Testing: There are two types of frontend tests: node-based unit tests and blackbox end-to-end tests. The blackbox tests are run in a headless browser using Casper.js and are located in frontend_tests/casper_tests/. The unit tests use Node’s assert module are located in frontend_tests/node_tests/. For more information on writing and running tests, see the testing documentation.
26.1.5 Documentation changes
After implementing the new feature, you should document it and update any existing documentation that might be relevant to the new feature. For more information on the kinds of documentation Zulip has, see Documentation.
26.2 Example Feature
This example describes the process of adding a new setting to Zulip: a flag that allows an admin to require topics on stream messages (the default behavior is that topics can have no subject). This flag is an actual Zulip feature. You can review the original commit in the Zulip repo. (This commit displays the work of setting up a checkbox for the feature on the admin settings page, communicating and saving updates to the setting to the database, and updating the state of the application after the setting is updated. For the code that accomplishes the underlying task of requiring messages to have a topic, you can view this commit.)
26.2.1 Update the model
First, update the database and model to store the new setting. Add a new boolean field, mandatory_topics, to the Realm model in zerver/models.py.
# zerver/models.py
class Realm(ModelReprMixin, models.Model): # ... restricted_to_domain = models.BooleanField(default=True) # type: bool invite_required = models.BooleanField(default=False) # type: bool + mandatory_topics = models.BooleanField(default=False) # type: bool
The Realm model also contains an attribute, property_types, which other backend functions use to handle most realm settings without any custom code for the setting (more on this process below). The attribute is a dictionary, where the key is the name of the realm field and the value is the field’s type. Add the new field to the property_types dictionary.
# zerver/models.py
class Realm(ModelReprMixin, models.Model) # ... # Define the types of the various automatically managed properties property_types = dict( add_emoji_by_admins_only=bool, allow_edit_history=bool, # ... + mandatory_topics=bool, # ...
26.2. Example Feature 145 Zulip Documentation, Release 1.7.0
The majority of realm settings can be included in property_types. However, there are some properties that need custom logic and thus cannot use this framework. For example: • The realm authentication_methods attribute is a bitfield and needs additional code for validation and updating. • The allow_message_editing and message_content_edit_limit_seconds fields depend on one another, so they are also handled separately and not included in property_types. When creating a realm property that is not a boolean, Text or integer field, or when adding a field that is dependent on other fields, do not add the field to the property_types dictionary. The steps below will point out where to write additional code for these cases.
26.2.2 Create the migration
Create the migration file: ./manage.py makemigrations. Make sure to commit the generated file to git: git add zerver/migrations/NNNN_realm_mandatory_topics.py (NNNN is a number that is equal to the number of migrations.) If you run into problems, the Django migration documentation is helpful.
26.2.3 Test your migration changes
Apply the migration: ./manage.py migrate Output:
shell $ ./manage.py migrate Operations to perform: Synchronize unmigrated apps: staticfiles, analytics, pipeline Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites,
˓→auth, contenttypes Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying zerver.NNNN_realm_mandatory_topics... OK
Once you’ve run the migration, restart memcached on your development server (/etc/init.d/memcached restart) and then restart the development server to avoid interacting with cached objects.
26.2.4 Handle database interactions
Next, we will implement the backend part of this feature. Like typical apps, we will need our backend to update the database and send some response to the client that made the request. Beyond that, we need to orchestrate notifications about the setting change to other clients (or other users, if you will). Clients find out about settings through two closely related code paths. When a client first contacts the server, the server sends the client its initial state. Subsequently, clients subscribe to "events," which can (among other things) indicate that settings have changed. For the backend piece, we will need our action to make a call to send_event to send the event to clients that are active. We will also need to modify fetch_initial_state_data so that the new field is passed to clients. See our event system docs for all the gory details.
146 Chapter 26. Writing a new application feature Zulip Documentation, Release 1.7.0
Anyway, getting back to implementation details... If you are working on a feature that is in the realm property_types dictionary, you will not need to add code to zerver/lib/actions.py, but we will describe what the process in that file does: In zerver/lib/actions.py, the function do_set_realm_property takes in the name of a realm property to update and the value it should have. This function updates the database and triggers an event to notify clients about the change. It uses the field’s type, specified in the Realm.property_types dictionary, to validate the type of the value before updating the property; this is primarily an assertion to help catch coding mistakes, not to check for bad user input. After updating the given realm field, do_set_realm_property creates an ’update’ event with the name of the property and the new value. It then calls send_event, passing the event and the list of users whose browser sessions should be notified as the second argument. The latter argument can be a single user (if the setting is a personal one, like time display format), members in a particular stream only or all active users in a realm.
# zerver/lib/actions.py
def do_set_realm_property(realm, name, value): # type: (Realm, str, Union[Text, bool, int]) -> None """Takes in a realm object, the name of an attribute to update, and the value to update. """ property_type= Realm.property_types[name] assert isinstance(value, property_type), ( 'Cannot update %s: %s is not an instance of %s'%( name, value, property_type,))
setattr(realm, name, value) realm.save(update_fields=[name]) event= dict( type='realm', op='update', property=name, value=value, ) send_event(event, active_user_ids(realm))
If the new realm property being added does not fit into the property_types framework (such as the authentication_methods field), you’ll need to create a new function to explicitly update this field and send an event. For example:
# zerver/lib/actions.py
def do_set_realm_authentication_methods(realm, authentication_methods): # type: (Realm, Dict[str, bool]) -> None for key, value in list(authentication_methods.items()): index= getattr(realm.authentication_methods, key).number realm.authentication_methods.set_bit(index, int(value)) realm.save(update_fields=['authentication_methods']) event= dict( type="realm", op="update_dict", property='default', data=dict(authentication_methods=realm.authentication_methods_dict()) ) send_event(event, active_user_ids(realm))
26.2. Example Feature 147 Zulip Documentation, Release 1.7.0
26.2.5 Update application state
zerver/lib/events.py contains code to ensure that your new setting is included in the data sent down to clients: both when a new client is loaded and when changes happen. This file also automatically handles realm settings in the property_types dictionary, so you would not need to change this file if your setting fits that framework. The fetch_initial_state_data function is responsible for sending data when a client is loaded (data added to the state here will be available both in page_params in the browser, as well as to API clients like the mobile apps). The apply_event function in zerver/lib/events.py is important for making sure the state is always correct, even in the event of rare race conditions.
# zerver/lib/events.py
def fetch_initial_state_data(user_profile, event_types, queue_id, include_
˓→subscribers=True): # ... if want('realm'): for property_name in Realm.property_types: state['realm_'+ property_name]= getattr(user_profile.realm, property_name) state['realm_authentication_methods']= user_profile.realm.authentication_methods_
˓→dict() state['realm_allow_message_editing']= user_profile.realm.allow_message_editing # ...
def apply_event(state, events, user_profile, include_subscribers): for event in events: # ... elif event['type'] =='realm': field='realm_'+ event['property'] state[field]= event['value'] # ...
If your new realm property fits the property_types framework, you don’t need to change fetch_initial_state_data or apply_event. However, if you are adding a property that is handled sepa- rately, you will need to explicitly add the property to the state dictionary in the fetch_initial_state_data function. E.g., for authentication_methods:
# zerver/lib/events.py
def fetch_initial_state_data(user_profile, event_types, queue_id, include_
˓→subscribers=True): # ... if want('realm'): # ... state['realm_authentication_methods']= user_profile.realm.authentication_
˓→methods_dict() # ...
For this setting, one won’t need to change apply_event since its default code for realm event types handles this case correctly, but for a totally new type of feature, a few lines in that function may be needed.
26.2.6 Add a new view
You will need to add a view for clients to access that will call the actions.py code to update the database. This example feature adds a new parameter that will be sent to clients when the application loads and should be accessible via JavaScript. There is already a view that does this for related flags: update_realm in zerver/views/ realm.py. So in this case, we can add our code to the existing view instead of creating a new one.
148 Chapter 26. Writing a new application feature Zulip Documentation, Release 1.7.0
You’ll need to add a parameter for the new field to the update_realm function in zerver/views/realm.py (and add the appropriate mypy type annotation).
# zerver/views/realm.py def update_realm(request, user_profile, name=REQ(validator=check_string,
˓→default=None), # ..., + mandatory_topics=REQ(validator=check_bool, default=None), # ...): + # type: (HttpRequest, UserProfile, ..., Optional[bool], ... # ...
If this feature fits the property_types framework and does not require additional validation, this is the only change to make to zerver/views/realm.py. Text fields or other realm properties that need additional validation can be handled at the beginning of update_realm.
# zerver/views/realm.py
# Additional validation/error checking beyond types go here, so # the entire request can succeed or fail atomically. if default_language is not None and default_language not in get_available_language_
˓→codes(): raise JsonableError(_("Invalid language' %s'"% (default_language,))) if description is not None and len(description)> 100: return json_error(_("Realm description cannot exceed 100 characters.")) # ...
The code in update_realm loops through the property_types dictionary and calls do_set_realm_property on any property to be updated from the request. If the new feature is not in property_types, you will need to write code to call the function you wrote in actions.py that updates the database with the new value. E.g., for authentication_methods, we created do_set_realm_authentication_methods, which we will call here:
# zerver/views/realm.py
# import do_set_realm_authentication_methods from actions.py from zerver.lib.actions import ( do_set_realm_message_editing, do_set_realm_authentication_methods, # ... ) # ... # ... if authentication_methods is not None and realm.authentication_methods_dict() !=
˓→authentication_methods: do_set_realm_authentication_methods(realm, authentication_methods) data['authentication_methods']= authentication_methods # ...
This completes the backend implementation. A great next step is to write the backend tests.
26.2. Example Feature 149 Zulip Documentation, Release 1.7.0
26.2.7 Backend Tests
To test the new setting syncs correctly with the property_types framework, one usually just needs to add a line in each of test_events.py and test_realm.py with a list of values to switch between in the test. In the case of a boolean field, no action is required, because those tests will correctly assume that the only values to test are True and False. In test_events.py, the function that runs tests for the property_types framework is do_set_realm_property_test, and in test_realm.py, it is do_test_realm_update_api. One still needs to add a test for whether the setting actually controls the feature it is supposed to control, however.
26.2.8 Update the front end
After completing the process of adding a new feature on the back end, you should make the required front end changes: in this case, a checkbox needs to be added to the admin page (and its value added to the data sent back to server when a realm is updated) and the change event needs to be handled on the client. To add the checkbox to the admin page, modify the relevant template, static/templates/settings/ organization-permissions-admin.handlebars (omitted here since it is relatively straightforward). Then add the new form control in static/js/admin.js.
// static/js/admin.js
function _setup_page() { var options = { realm_name: page_params.realm_name, realm_description: page_params.realm_description, realm_restricted_to_domain: page_params.realm_restricted_to_domain, realm_invite_required: page_params.realm_invite_required, // ... + realm_mandatory_topics: page_params.mandatory_topics, // ...
The JavaScript code for organization settings and permissions can be found in static/js/settings_org.js. There is a front-end version of property_types, which reduces the code needed on the front end for a new feature. Add the new feature to the property_types object in settings_org.js. The key should be the setting name and the value should be an object with the following keys: • type • checked_msg (what message the user sees when they enable the setting) • unchecked_msg (what message the user sees when they disable the setting)
// static/js/settings_org.js
var property_types = { settings: { // ... }, permissions: { // ... + mandatory_topics: { + type: 'bool', + checked_msg: i18n.t("Topics are required in messages to streams"), + unchecked_msg: i18n.t("Topics are not required in messages to streams"), },
150 Chapter 26. Writing a new application feature Zulip Documentation, Release 1.7.0
}, };
Additionally, any code needed to update the UI when the setting is changed should be written in a function inside settings_org.js. For example, when a realm description is updated, that value change should occur in other windows where the description field is visible:
# static/js/settings_org.js exports.update_realm_description = function () { if (!meta.loaded) { return; }
$('#id_realm_description').val(page_params.realm_description); };
This ensures the appropriate code will run even if the changes are made in another browser window. In the example of updating a mandatory_topics setting, most of the changes are on the backend, so no UI updates are required. Finally, update server_events_dispatch.js to handle related events coming from the server. There is an object, realm_settings, in the function dispatch_normal_event. The keys in this object are setting names and the values are the UI updating functions to run when an event has occurred. If there is no relevant UI change to make, the value should be noop (this is the case for mandatory_topics). However, if you had written a function in settings_org.js to update UI, that function should be the value in the realm_settings object.
// static/js/server_events_dispatch.js function dispatch_normal_event(event) { switch (event.type) { // ... case 'realm': var realm_settings = { add_emoji_by_admins_only: settings_emoji.update_custom_emoji_ui, allow_edit_history: noop, // ... + mandatory_topics: noop, // ... };
The rest of the dispatch_normal_events function updates the state of the application if an update event has occurred on a realm property and runs the associated function to update the application’s UI, if necessary.
26.2.9 Front End Tests
A great next step is to write front end tests. There are two types of frontend tests: node-based unit tests and Casper end-to-end tests. At the minimum, if you created a new function to update UI in settings_org.js, you will need to mock that function in frontend_tests/node_tests/dispatch.js. Add the name of the UI function you created to the following object with noop as the value:
26.2. Example Feature 151 Zulip Documentation, Release 1.7.0
# frontend_tests/node_tests/dispatch.js set_global('settings_org',{ reset_realm_default_language: noop, toggle_email_change_display: noop, toggle_name_change_display: noop, update_message_retention_days: noop, update_realm_description: noop, });
Beyond that, you should add any applicable tests that verify the behavior of the setting you just created.
26.2.10 Update documentation
After you add a new view, you should document your feature. This feature adds new functionality that requires messages to have topics if the setting is enabled. A recommended way to document this feature would be to update and/or augment Zulip’s user documentation to reflect your changes and additions. At the very least, this will involve adding (or modifying) a Markdown file documenting the feature to templates/ zerver/help/ in the main Zulip server repository, where the source for Zulip’s user documentation is stored. For information on writing user documentation, see Zulip’s general user guide documentation. For a more concrete example of writing documentation for a new feature, see an example commit in the Zulip repo that documented a new realm feature, the current source, and the final rendered documentation.
152 Chapter 26. Writing a new application feature CHAPTER 27
Writing views in Zulip
27.1 What this covers
This page documents how views work in Zulip. You may want to read the new feature tutorial or the integration guide, and treat this as a reference. If you have experience with Django, much of this will be familiar, but you may want to read about how REST requests are dispatched, and how request authentication works. This document supplements the new feature tutorial and the testing documentation.
27.2 What is a view?
A view in Zulip is everything that helps implement a server endpoint. Every path that the Zulip server supports (doesn’t show a 404 page for) is a view. The obvious ones are those you can visit in your browser, for example /integrations, which shows the integration documentation. These paths show up in the address bar of the browser. There are other views that are only seen by software, namely the API views. They are used to build the various clients that Zulip has, namely the web client (which is also used by the desktop client) and the mobile clients.
27.3 Modifying urls.py
A view is anything with an entry in the appropriate urls.py, usually zproject/urls.py. Zulip views either serve HTML (pages for browsers) or JSON (data for Zulip clients on all platforms, custom bots, and integrations). The format of the URL patterns in Django is documented here, and the Zulip specific details for these are discussed in detail in the life of a request doc. We have two Zulip-specific conventions we use for internationalization and for our REST API, respectively.
153 Zulip Documentation, Release 1.7.0
27.4 Writing human-readable views
If you’re writing a new page for the website, make sure to add it to i18n_urls in zproject/urls.py
i18n_urls = [ ... + url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html
˓→')), + url(r'^postcards/$', 'zerver.views.postcards'), ]
As an example, if a request comes in for Spanish, language code es, the server path will be something like: es/ features/.
27.4.1 Decorators used for webpage views
This section documents a few simple decorators that we use for webpage views, as an introduction to view decorators. require_post:
@require_post def accounts_register(request): # type: (HttpRequest) -> HttpResponse
This decorator ensures that the requst was a POST–here, we’re checking that the registration submission page is requested with a post, and inside the function, we’ll check the form data. If you request this page with GET, you’ll get a HTTP 405 METHOD NOT ALLOWED error. zulip_login_required: This decorator verifies that the browser is logged in (i.e. has a valid session cookie) before providing the view for this route, or redirects the browser to a login page. This is used in the root path (/) of the website for the web client. If a request comes from a browser without a valid session cookie, they are redirected to a login page. It is a small fork of Django’s login_required, adding a few extra checks specific to Zulip.
@zulip_login_required def home(request): # type: (HttpRequest) -> HttpResponse
27.4.2 Writing a template
Templates for the main website are found in templates/zerver.
27.5 Writing API REST endpoints
These are code-parseable views that take x-www-form-urlencoded or JSON request bodies, and return JSON-string responses. Almost all Zulip view code is in the implementations of API REST endpoints. The REST API does authentication of the user through rest_dispatch, which is documented in detail at zerver/lib/rest.py. This method will authenticate the user either through a session token from a cookie on the browser, or from a base64 encoded email:api-key string given via HTTP Basic Auth for API clients.
154 Chapter 27. Writing views in Zulip Zulip Documentation, Release 1.7.0
>>> import requests >>> r= requests.get('https://api.github.com/user', auth=('[email protected]',
˓→'0123456789abcdeFGHIJKLmnopQRSTUV')) >>> r.status_code -> 200
27.5.1 Request variables
Most API views will have some arguments that are passed as part of the request to control the behavior of the view. In any well-engineered view, you need to write code to parse and validate that the arguments exist and have the correct form. For many applications, this leads to one of several bad outcomes: • The code isn’t written, so arguments aren’t validated, leading to bugs and confusing error messages for users of the API. • Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent error messages. • Every view function comes with another function that does the validation that has the problems from the last bullet point. In Zulip, we solve this problem with a the special decorator called has_request_variables which allows a developer to declare the arguments a view function takes and validate their types all within the def line of the function. We like this framework because we have found it makes the validation code compact, readable, and conveniently located in the same place as the method it is validating arguments for. Here’s an example:
from zerver.decorator import has_request_variables, REQ, JsonableError, \ require_realm_admin
@require_realm_admin @has_request_variables def create_user_backend(request, user_profile, email=REQ(), password=REQ(), full_name=REQ(), short_name=REQ()): # ... code here
You will notice the special REQ() in the keyword arguments to create_user_backend. has_request_variables parses the declared keyword arguments of the decorated function, and for each that has an instance of REQ as the default value, it extracts the HTTP parameter with that name from the request, parses it as JSON, and passes it to the function. It will return an nicely JSON formatted HTTP 400 error in the event that an argument is missing, doesn’t parse as JSON, or otherwise is invalid. require_realm_admin is another decorator which checks the authorization of the given user_profile to make sure it belongs to a realm administrator (and thus has permission to create a user); we show it here primarily to show how has_request_variables should be the inner decorator. The implementation of has_request_variables is documented in detail in zerver/lib/request.py) REQ also helps us with request variable validation. For example: • msg_ids = REQ(validator=check_list(check_int)) will check that the msg_ids HTTP pa- rameter is a list of integers, marshalled as JSON, and pass it into the function as the msg_ids Python keyword argument. • streams_raw = REQ("subscriptions", validator=check_list(check_string)) will check that the "subscriptions" HTTP parameter is a list of strings, marshalled as JSON, and pass it into the function with the Python keyword argument streams_raw.
27.5. Writing API REST endpoints 155 Zulip Documentation, Release 1.7.0
• message_id=REQ(converter=to_non_negative_int) will check that the message_id HTTP parameter is a string containing a non-negative integer (converter differs from validator in that it does not automatically marshall the input from JSON). See zerver/lib/validator.py for more validators and their documentation.
27.5.2 Deciding which HTTP verb to use
When writing a new API view, you should writing a view to do just one type of thing. Usually that’s either a read or write operation. If you’re reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if a resource is available to be read with GET, without the expense of the full GET. OPTIONS is also read-only, and used by clients to determine which HTTP verbs are available for a given path. This isn’t something you need to write, as it happens automatically in the implementation of rest_dispatch–see zerver/lib/rest.py for more. If you’re creating new data, try to figure out if the thing you are creating is uniquely identifiable. For example, if you’re creating a user, there’s only one user per email. If you can find a unique ID, you should use PUT for the view. If you want to create the data multiple times for multiple requests (for example, requesting the send_message view multiple times with the same content should send multiple messages), you should use POST. If you’re updating existing data, use PATCH. If you’re removing data, use DELETE.
27.5.3 Idempotency
When writing a new API endpoint, with the exception of things like sending messages, requests should be safe to repeat, without impacting the state of the server. This is idempotency. You will often want to return an error if a request to change something would do nothing because the state is already as desired, to make debugging Zulip clients easier. This means that the response for repeated requests may not be the same, but the repeated requests won’t change the server more than once or cause unwanted side effects.
27.5.4 Making changes to the database
If the view does any modification to the database, that change is done in a helper function in zerver/lib/ actions.py. Those functions are responsible for doing a complete update to the state of the server, which often entails both updating the database and sending any events to notify clients about the state change. When possible, we prefer to design a clean boundary between the view function and the actions function is such that all user input validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsible for atomically executing the change (this is usually signalled by having the actions function have a name starting with do_. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connection to the database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have a name starting with check_. For example, in zerver/views/realm.py:
@require_realm_admin @has_request_variables def update_realm(request, user_profile, name=REQ(validator=check_string,
˓→default=None),...)): # type: (HttpRequest, UserProfile, ...) -> HttpResponse realm= user_profile.realm data={} # type: Dict[str, Any]
156 Chapter 27. Writing views in Zulip Zulip Documentation, Release 1.7.0
if name is not None and realm.name != name: do_set_realm_name(realm, name) data['name']='updated' and in zerver/lib/actions.py: def do_set_realm_name(realm, name): # type: (Realm, Text) -> None realm.name= name realm.save(update_fields=['name']) event= dict( type="realm", op="update", property='name', value=name, ) send_event(event, active_user_ids(realm)) realm.save() actually saves the changes to the realm to the database, and send_event sends the event to active clients belonging to the provided list of users (in this case, all altive users in the Zulip realm).
27.5.5 Calling from the web application
You should always use channel. to make an HTTP
27.5.6 Calling from an API client
Here’s how you might manually make a call from python: payload={'name': new_name}
# email and API key api_auth=('[email protected]','0123456789abcdeFGHIJKLmnopQRSTUV') r= requests.patch(SERVER_URL+'api/v1/realm', data=json.dumps(payload), auth=api_auth, )
This is simply an illustration; we recommend making use of the Zulip Python API bindings since they provide a nice interface for accessing the API.
27.5. Writing API REST endpoints 157 Zulip Documentation, Release 1.7.0
27.6 Legacy endpoints used by the web client
New features should conform the REST API style. The legacy, web-only endpoints can’t effectively enforce usage of a browser, so they aren’t preferable from a security perspective, and it is generally a good idea to make your feature available to other clients, especially the mobile clients. These endpoints make use of some older authentication decorators, authenticated_json_api_view, authenticated_json_post_view, and authenticated_json_view, so you may see them in the code.
27.7 Webhook integration endpoints
Webhooks are called by other services, often to send a message as part of those services’ integrations. They are most often POST requests, and often there is very little you can customize about them. Usually you can expect that the webhook for a service will allow specification for the target server for the webhook, and an API key. If the webhook does not have an option to provide a bot email, use the api_key_only_webhook_view decorator, to fill in the user_profile and request.client fields of a request:
@api_key_only_webhook_view('PagerDuty') @has_request_variables def api_pagerduty_webhook(request, user_profile, payload=REQ(argument_type='body'), stream=REQ(default='pagerduty'), topic=REQ(default=None)): request.client will be the result of get_client("ZulipPagerDutyWebhook") in this example and it will be passed to check_send_stream_message. For more information, see Clients in Zulip.
158 Chapter 27. Writing views in Zulip CHAPTER 28
Life of a Request
It can sometimes be confusing to figure out how to write a new feature, or debug an existing one. Let us try to follow a request through the Zulip codebase, and dive deep into how each part works. We will use as our example the creation of users through the API, but we will also highlight how alternative requests are handled.
28.1 A request is sent to the server, and handled by Nginx
When Zulip is deployed in production, all requests go through nginx. For the most part we don’t need to know how this works, except for when it isn’t working. Nginx does the first level of routing–deciding which application will serve the request (or deciding to serve the request itself for static content). In development, tools/run-dev.py fills the role of nginx. Static files are in your git checkout under static, and are served unminified.
28.2 Nginx secures traffic with SSL
If you visit your Zulip server in your browser and discover that your traffic isn’t being properly encrypted, an nginx misconfiguration is the likely culprit.
28.3 Static files are served directly by Nginx
Static files include JavaScript, css, static assets (like emoji, avatars), and user uploads (if stored locally and not on S3). File not found errors (404) are served using a Django URL, so that we can use configuration variables (like whether the user is logged in) in the 404 error page.
159 Zulip Documentation, Release 1.7.0
location/static/{ alias/home/zulip/prod-static/; # Set a nonexistent path, so we just serve the nice Django 404 page. error_page 404/django_static_404.html; }
28.4 Nginx routes other requests between django and tornado
All our connected clients hold open long-polling connections so that they can receive events (messages, presence notifications, and so on) in real-time. Events are served by Zulip’s tornado application. Nearly every other kind of request is served by the zerver Django application. Here is the relevant nginx routing configuration.
28.5 Django routes the request to a view in urls.py files
There are various urls.py files throughout the server codebase, which are covered in more detail in the directory structure doc. The main Zulip Django app is zerver. The routes are found in
zproject/urls.py zproject/legacy_urls.py
There are HTML-serving, REST API, legacy, and webhook url patterns. We will look at how each of these types of requests are handled, and focus on how the REST API handles our user creation example.
28.6 Views serving HTML are internationalized by server path
If we look in zproject/urls.py, we can see something called i18n_urls. These urls show up in the address bar of the browser, and serve HTML. For example, the /features page (preview here) gets translated in Chinese at zh-hans/features/ (preview here). Note the zh-hans prefix–that url pattern gets added by i18n_patterns.
28.7 API endpoints use REST
Our example is a REST API endpoint. It’s a PUT to /users. With the exception of Webhooks (which we do not usually control the format of), legacy endpoints, and logged-out endpoints, Zulip uses REST for its API. This means that we use: • POST for creating something new where we don’t have a unique ID. Also used as a catch-all if no other verb is appropriate. • PUT for creating something for which we have a unique ID. • DELETE for deleting something
160 Chapter 28. Life of a Request Zulip Documentation, Release 1.7.0
• PATCH for updating or editing attributes of something. • GET to get something (read-only) • HEAD to check the existence of something to GET, without getting it; useful to check a link without download- ing a potentially large link • OPTIONS (handled automatically, see more below) Of these, PUT, DELETE, HEAD, OPTIONS, and GET are idempotent, which means that we can send the request multiple times and get the same state on the server. You might get a different response after the first request, as we like to give our clients an error so they know that no new change was made by the extra requests. POST is not idempotent–if I send a message multiple times, Zulip will show my message multiple times. PATCH is special–it can be idempotent, and we like to write API endpoints in an idempotent fashion, as much as possible. This cookbook and tutorial can be helpful if you are new to REST web applications.
28.7.1 PUT is only for creating new things
If you’re used to using PUT to update or modify resources, you might find our convention a little strange. We use PUT to create resources with unique identifiers, POST to create resources without unique identifiers (like sending a message with the same content multiple times), and PATCH to modify resources. In our example, create_user_backend uses PUT, because there’s a unique identifier, the user’s email.
28.7.2 OPTIONS
The OPTIONS method will yield the allowed methods. This request: OPTIONS https://chat.zulip.org/api/v1/users yields a response with this HTTP header: Allow: PUT, GET We can see this reflected in zproject/urls.py:
url(r'^users$','zerver.lib.rest.rest_dispatch', {'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'}),
In this way, the API is partially self-documenting.
28.7.3 Legacy endpoints are used by the web client
The endpoints from the legacy JSON API are written without REST in mind. They are used extensively by the web client, and use POST. You can see them in zproject/legacy_urls.py.
28.7.4 Webhook integrations may not be RESTful
Zulip endpoints that are called by other services for integrations have to conform to the service’s request format. They are likely to use only POST.
28.7. API endpoints use REST 161 Zulip Documentation, Release 1.7.0
28.8 Django calls rest_dispatch for REST endpoints, and authenti- cates
For requests that correspond to a REST url pattern, Zulip configures its url patterns (see zerver/lib/rest.py) so that the action called is rest_dispatch. This method will authenticate the user, either through a session token from a cookie, or from an email:api-key string given via HTTP Basic Auth for API clients. It will then look up what HTTP verb was used (GET, POST, etc) to make the request, and then figure out which view to show from that. In our example,
{'GET':'zerver.views.users.get_members_backend', 'PUT':'zerver.views.users.create_user_backend'} is supplied as an argument to rest_dispatch, along with the HTTPRequest. The request has the HTTP verb PUT, which rest_dispatch can use to find the correct view to show: zerver.views.users. create_user_backend.
28.9 The view will authorize the user, extract request variables, and validate them
This is covered in good detail in the writing views doc.
28.10 Results are given as JSON
Our API works on JSON requests and responses. Every API endpoint should return json_error in the case of an error, which gives a JSON string: {'result': 'error', 'msg':
162 Chapter 28. Life of a Request CHAPTER 29
Reading list
Here you can find a list of relevant resources that you may find useful for learning new skills, or polishing the ones you already have. The topics cover a wide variety of topics, from basic Python coding to general developing guidelines. Feel free to create a pull request in Zulip’s GitHub repository with any interesting books, articles or videos you would like to see in this list. Some titles have been shortened for organizational purposes.
29.1 General programming/IT
Book - Clean Code: A Handbook of Agile Software Craftsmanship (Not free!) Books - Free programming books list Blog - Free Code Camp blog Blog - Idle Words talks transcripts Tutorial - HTTP Can Do That?!, by Sumana Harihareswara (PyCon 2016) Video - Minimum Viable Documentation, by Matthew Lyon (WriteTheDocs 2014) Video - NoOps, by Kelsey Hightower (DepOpsDays 2016) Video - The mind behind Linux (TED interview) Tutorial - Learn code the hard way Tutorial - What happens when... Article - An Interview With Linus Torvalds Article - Effective Learning Strategies for Programmers Article - Readme Driven Development
163 Zulip Documentation, Release 1.7.0
Article - Systematic Debugging Paper - Floating-Point Arithmetic
29.2 Python
Video - Intro to Python for beginners, by Jessica McKellar (PyCon 2013) Video - Breaking the rules, by Jessica McKellar (PyCon Sweden) Video - Build & break a Python sandbox, by Jessica McKellar (PyCon 2014) Video - Cache me if you can, by Guillaume Ardaud (PyCon 2014) Video - Loop like a native, by Ned Batchelder (PyCon 2013) Video - Modern Dictionaries, by Raymond Hettinger (SF Python) Video - Python Language, by Guido van Rossum (PyCon 2016) Video - The Mighty Dictionary, by Brandon Rhodes (PyCon 2010) Tutorial - Code Like a Pythonista: Idiomatic Python Article - Static types in Python, oh my(py)! Guide - The Hitchhiker’s Guide to Python!
29.3 Java/Android
Course - Android Development for Beginners Blog - Java Tutorials for Beginners
29.4 JavaScript/ECMAScript
Tutorial - clean-code-javascript Software engineering principles Course - React native and redux course( Not free!) Slides - TypeScript vs. CoffeeScript vs. ES6
29.5 Git/Version Control Systems (VCS)
You may want to take a look first at our Git and GitHub guide. Article - Git tips
29.6 Computer Science/Algorithms
Blog - GeeksforGeeks Book Introduction to Algorithms( Not free!)
164 Chapter 29. Reading list Zulip Documentation, Release 1.7.0
Blog - Setosa data visualization and visual explanations Course - Algorithms, Part I Course - Open Source Society University Course - MIT CSAIL 6.828: Operative Systems Engineering
29.7 Community experience
Book - Producing Open Source Software Article - Advice on Starting And Running A New Open Source Project Article - How to ask good questions Article - Notes for New FLOSS Contributors Article - To be mentored Article - To mentor List of good projects for new contributors
29.8 Competitions/Camps
CodeForces Free Code Camp
29.9 Massive Open Online Courses (MOOC) Platforms
Coursera edX MIT OpenCourseWare Udacity
29.7. Community experience 165 Zulip Documentation, Release 1.7.0
166 Chapter 29. Reading list CHAPTER 30
Screenshot and GIF software
The following list documents different screenshoting & GIF-making techniques and free software. We encourage you to make use of these when making front-end pull requests, as other contributors can see the changes you have made without having to checkout your branch.
30.1 Screenshot tools by platform
30.1.1 Browser
• LightShot Screenshot (Chrome, Firefox, IE & Opera)
30.1.2 macOS
• Command-Shift-3 to capture all of the screen • Command-Shift-4 and drag cursor to select a specific area • LightShot Screenshot • Gyazo
30.1.3 Windows
• Snipping Tool (inbuilt) • LightShot Screenshot • Gyazo
167 Zulip Documentation, Release 1.7.0
30.1.4 Linux
• gnome-screenshot (inbuilt, you can use Ctrl-Shift-PrtScn as a shortcut for its “select an area to grab” feature)
30.2 GIF tools by platform
30.2.1 Browser
• GIPHY • Chrome Capture (Tip: Use Alt+i to interact with the page while recording)
30.2.2 macOS
• GIPHY • GIF Brewery • Gyazo GIF
30.2.3 Windows
• ScreenToGif • Gyazo GIF
30.2.4 Linux
• SilentCast • Peek
168 Chapter 30. Screenshot and GIF software CHAPTER 31
Fixing Commits
This is mostly from here.
31.1 Fixing the last commit
31.1.1 Changing the last commit message
1. git commit --amend -m "New Message"
31.1.2 Changing the last commit
1. Make your changes to the files 2. Run git add
31.2 Fixing older commits
31.2.1 Changing commit messages
1. git rebase -i HEAD~5 (if, for example, you are editing some of the last five commits) 2. For each commit that you want to change the message, change pick to reword, and save 3. Change the commit messages
169 Zulip Documentation, Release 1.7.0
31.2.2 Deleting old commits
1. git rebase -i HEAD~n where n is the number of commits you are looking at 2. For each commit that you want to delete, change pick to drop, and save
31.3 Squashing commits
Sometimes, you want to make one commit out of a bunch of commits. To do this, 1. git rebase -i HEAD~n where n is the number of commits you are interested in 2. Change pick to squash on the lines containing the commits you want to squash and save
31.4 Reordering commits
1. git rebase -i HEAD~n where n is the number of commits you are interested in 2. Reorder the lines containing the commits and save
31.5 Pushing commits after tidying them
1. git push origin +my-feature-branch (Note the + there and substitute your actual branch name.)
170 Chapter 31. Fixing Commits CHAPTER 32
Git Cheat Sheet (Detailed)
See also fixing commits Commands: • add – git add foo.py: add foo.py to the staging area – git add foo.py bar.py: add foo.py AND bar.py to the staging area • checkout – git checkout -b new-branch-name: create branch new-branch-name and switch/checkout to that new branch – git checkout master: switch to your master branch – git checkout old-branch-name: switch to an existing branch old-branch-name • commit – git commit --amend: changing the last commit message. Read more here • config – git config --global core.editor nano: set core editor to nano (you can set this to vim or others) – git config --global core.symlinks true: allow symbolic links • diff – git diff: display the changes you have made to all files – git diff --cached: display the changes you have made to staged files – git diff HEAD~2..: display the 2 most recent changes you have made to files • fetch – git fetch origin: fetch origin repository
171 Zulip Documentation, Release 1.7.0
– git fetch upstream: fetch upstream repository • grep – git grep update_unread_counts -- '*.js': search all files (ending in .js) for update_unread_counts • log – git log: show commit logs • pull – do not use for Zulip • push – git push origin +branch-name: push your commits to your origin repository • rebase – git rebase -i HEAD~3: interactive rebasing current branch with first three items on HEAD – git rebase -i master: interactive rebasing current branch with master branch – git rebase upstream/master: rebasing current branch with master branch from upstream repos- itory • reflog – git reflog | head -10: manage reference logs for the past 10 commits • remote – git remote -v: display your origin and upstream repositories • reset – git reset HEAD~2: reset two most recent commits • rm – git rm oops.txt: remove oops.txt • show – git show HEAD: display most recent commit – git show HEAD~~~: display third most recent commit – git show master: display most recent commit on master • status – git status: show the working tree status, unstaged and staged files
172 Chapter 32. Git Cheat Sheet (Detailed) CHAPTER 33
Git Cheat Sheet
See also fixing commits Commands: • add – git add foo.py • checkout – git checkout -b new-branch-name – git checkout master – git checkout old-branch-name • commit – git commit --amend • config – git config --global core.editor nano – git config --global core.symlinks true • diff – git diff – git diff --cached – git diff HEAD~2.. • fetch – git fetch origin – git fetch upstream • grep – git grep update_unread_counts -- '*.js'
173 Zulip Documentation, Release 1.7.0
• log – git log • pull – do not use for Zulip • push – git push origin +branch-name • rebase – git rebase -i HEAD~3 – git rebase -i master – git rebase upstream/master • reflog – git reflog | head -10 • remote – git remote -v • reset – git reset HEAD~2 • rm – git rm oops.txt • show – git show HEAD – git show HEAD~~~ – git show master • status – git status
174 Chapter 33. Git Cheat Sheet CHAPTER 34
Shell tips
The shell is a command line interpreter. To use it you can open a terminal (sometimes called a console). This is how most terminal windows look like:
If you haven’t used it before, you should probably take a look at this tutorial. If you’re using Windows, these videos may be useful too, but keep in mind that the following tips only apply to Linux/macOS environments (Unix shells). You can also use a tool, for example Cygwin, to have a Unix-like shell on Windows.
175 Zulip Documentation, Release 1.7.0
34.1 The prompt ($)
When searching Google, or Zulip’s docs, you’ll find commands that begin with a dollar sign $ or a dollar sign preceded by some text (e.g. (venv)john@laptop:~$). This is called the prompt, and it’s only an indicator that the shell is awaiting new orders. The prompt can contain useful information, let’s look at (venv)john@laptop:~$: • (venv) informs the user that they’re currently in a virtual environment (more on Python virtual environments) • the john before @ is the username • the laptop is the host machine name • the ~ after the colon informs the user they’re currently in the home folder of the user john You shouldn’t type the prompt or the text preceding it, since it isn’t a part of the commands.
34.2 Tilde character (~)
It’s very frequent to see the tilde (~) in paths. The tilde is an abbreviation for your home directory (/home/ YOUR_USERNAME most of the times). That’s why the following is exactly the same, if the user running it is john:
$ cd ~ $ cd /home/john
34.3 Change directory (cd)
When you’re using the shell, you work inside a directory (the one specified in the prompt). This way you can point to files relative to your current directory, instead of writing the whole path. Imagine you have a file called ideas.txt inside /home/john/notes/, and you want to edit it using nano. You could use:
$ nano /home/john/notes/ideas.txt
However, that isn’t very practical, especially if you are working with longer paths. That’s why it’s very useful to change the path where you are currently located (usually known as working directory). To do that, you use cd (change directory):
$ cd /home/john/notes/ ~/notes$ nano ideas.txt
Or, if you’re the user john: cd ~/notes. You can now access to all the files inside /home/john/notes directly, without needing to type the whole path. Relative paths make it much easier to move through files and directories, too.
176 Chapter 34. Shell tips Zulip Documentation, Release 1.7.0
34.4 Running commands as root (sudo)
You may have noticed that many commands begin with sudo. This informs the shell that the following command must be run as the root - a user that by default has access to all commands and files on a Unix operating system (i.e. a user with administrator privileges). That’s why you may be asked for a password in those cases: the system verifies you have permission to act as the root user. In case you were wondering, the name sudo comes from super user do.
34.5 Escaping characters
Some characters cannot be used directly in the shell, because they have a special meaning. Consider the following example:
$ echo "He said hello" He said hello
What if you wanted to display double quotes? You can’t use echo "He said "hello"", because in that case you’re using the double quotes for two different purposes: • Delimiting the string you want to use, from He to "hello". • Quoting something, by literally printing ". You have to specify which double quotes are used in each case. When you want one of those "special characters" to be literally printed, that’s called character escaping. To escape a character, simply add a backslash (\) before it. Returning to our example:
$ echo "He said \"hello\"" He said "hello"
As you can see, the double quotes with the backslash are shown, but the ones without it are used as string delimiters. Double quotes aren’t the only case of special characters. Some others are $, #, { or }, but there are many more. The backslash itself can be escaped as well, using the same procedure: \\.
34.6 Sequencing commands
It’s also possible to run multiple commands in a single line. For that purpose, the shell provides two different separa- tors: • Semicolon ;: runs a command, and once it has finished, runs the next one:
$ echo "Hello"; echo "World!" Hello World!
• Double ampersand &&: runs a command, and only if it finished without errors, it proceeds with the next one:
$ qwfvijwe && echo "Hello" qwfvijwe: command not found
34.4. Running commands as root (sudo) 177 Zulip Documentation, Release 1.7.0
Notice that it doesn’t print Hello at the end, because the previous command (qwfvijwe) returned an error. When using an incorrect command with a semicolon, the Hello will still be printed:
$ qwfvijwe; echo "Hello" qwfvijwe: command not found Hello
34.7 Splitting commands into multiple lines
Sometimes you end up with a very long command, that is hard to read and may be unclear. This is a problem, especially if you want to share that command, e.g. in a documentation file. In those cases, you can use a backslash at the end of each line, to inform the shell "wait, there’s more on the next line". This is an example, taken from the docs on how to install the Zulip development environment: sudo apt-get-y purge vagrant&&\ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb&&\ sudo dpkg-i vagrant *.deb&&\ sudo apt-get-y install build-essential git ruby lxc lxc-templates cgroup-lite redir&
˓→&\ vagrant plugin install vagrant-lxc&&\ vagrant lxc sudoers
It’s all a single command, joined using the double ampersand explained in Sequencing commands. If you’re typing it manually, you don’t need to include the backslashes, just write it all on the same line, and hit ENTER/RETURN at the end. If you think about it, what is happening here is actually another case of character escaping. The newline character (the one that appears when you hit ENTER) usually means "read this command". However, here we want to literally have the newline character, and thus the \
34.8 Arguments
Most commands need additional data to work, like a path or a file. That extra information is called an argument, and it’s specified after the name of the command, like this:
$ cd /home/john/notes
Here, the command is cd, and the first (and only) argument is /home/john/notes: • cd - command: changes your current directory. • /home/john/notes - argument: the directory where you want to go. In each command the arguments are specified in different ways, and have different meanings. Sometimes, a command can accept arguments indicated with dashes. Here’s another example of arguments usage:
$ nano -C /home/john/backups --mouse todo.txt
As you can see, some arguments imply that more information has to be specified, while others don’t.
178 Chapter 34. Shell tips Zulip Documentation, Release 1.7.0
In this case, we’re saying: "Bash, use the app nano to open the file todo.txt, enabling mouse support, and saving the backup files to /home/john/backups". The different parts are: • nano - command: program that allows editing text easily. • -C - argument: needs you to indicate where the backups should be stored, and thus you have to add an additional argument after it, to specify the directory (/home/john/backups in the example). • --mouse - argument: is just an option you set, nano doesn’t need anything else to make it work. Thus, there isn’t any extra argument for that. Note that the todo.txt is the file we want to open! It has nothing to do with the previous argument. This will probably clarify it (taken from nano’s help):
Usage: nano [OPTIONS] [FILE]...
So, in the options you indicate the arguments, and FILE is... well, the file. Don’t worry, you don’t have to memorize the meaning of all the arguments for every single command. There are tools that help you with that :wink:.
34.9 Shebang
You can run some files directly, without specifying a program to interpret them. That’s why you may have seen cases, in the Zulip codebase or elsewhere, when some Python scripts are called with python:
$ python my_program.py
While other times, python isn’t used:
$ ./my_program.py
In the latter case, the operating system will look at the beginning of the script to identify what interpreter (like python in this example) to run it with. The note telling the OS how to interpret the file goes on the file’s very first line, and it’s called a shebang. In our Python scripts, it looks like this:
#!/usr/bin/env python3
With this, you’re telling the operating system: "if I tell you to run this, ask /usr/bin/env python3 how to understand it". The result is that the operating system is asked to run the script, it runs the command specified in the shebang, with the script filename added as a command-line argument. So, returning to our example with my_program.py, when you run ./my_program.py, what happens under the hood is equivalent to:
$ /usr/bin/env python3 ./my_program.py
The purpose of /usr/bin/env in our shebangs is as a way to locate the python3 program in your current en- vironment, the same one the shell would use if you ran python3 my_program.py. You may see Python scripts outside of Zulip with a shebang like #!/usr/bin/python3; but because of the way Python virtualenvs work, this has the effect of running the script outside of any currently activated virtualenv. We use /usr/bin/env to keep our scripts running inside the virtualenv where we’ve installed all our dependencies. Another example of a shebang is the one used in Bash scripts. In those cases, #!/bin/bash or #!/bin/sh is used.
34.9. Shebang 179 Zulip Documentation, Release 1.7.0
34.10 Understanding commands
Frequently, you may find commands that you don’t understand, or don’t know what they do. You can use man
34.11 Cheatsheet
There are many more commands in the shell, besides the ones explained in this file. Here you can find a simple yet useful cheatsheet, created by Tower, that could help you understand and remember what other common commands do (e.g. ls).
34.12 Git
Probably at this point you’ve heard about Git. It’s basically a tool that most developers use to manage all the changes in their code. At first it seems like magic, but once you get the basic concepts you find it extremely useful and even easy to use (at least the 99% of the time). To learn more about how to use it, read our docs on Git and Github. This cheatsheet will be useful in your journey, as well.
180 Chapter 34. Shell tips Zulip Documentation, Release 1.7.0
34.12. Git 181 Zulip Documentation, Release 1.7.0
182 Chapter 34. Shell tips CHAPTER 35
Working copies
When you work on Zulip code, there are three working copies of the Zulip git repo that you are generally concerned with: • local copy: This lives on your laptop or your remove dev instance. • forked copy: This lives on GitHub, and it’s tied to your account. • official Zulip repo: https://github.com/zulip/zulip We sometimes call the forked copy the origin remote. We sometimes call the official repo the upstream remote. When you work on Zulip code, you will end up moving code between the various working copies.
35.1 Workflows
Sometimes you need to get commits. Here are some scenarios: • You may fork the official Zulip repo to your GitHub fork. • You may fetch commits from the offical Zulip repo to your local copy. • You occasionally may fetch commits from your forked copy. Sometimes you want to publish commits. Here are scenarios: • You push code from your local copy to your GitHub fork. (You usually want to put the commit on a feature branch.) • You submit a PR to the official Zulip repo. Finally, the Zulip core team will occasionally want your changes! • The Zulip core team can accept your changes and add them to the official repo, usually on the master branch.
183 Zulip Documentation, Release 1.7.0
35.2 Names
We call remote working copies of the repository by these short names. • origin: This is your fork. • upstream: This is the official Zulip repo.
35.3 Relevant git commands
The following commands are useful for moving commits between working copies: • git fetch: This grabs code from another repo to your local copy. • git push: This pushes code from your local repo to one of the remotes. • git remote: This helps you configure short names for remotes. • git pull: Do not use this, please!
184 Chapter 35. Working copies CHAPTER 36
Git & GitHub Guide
36.1 Quick start: How Zulip uses Git and GitHub
This quick start provides a brief overview of how Zulip uses Git and GitHub. Those who are familiar with Git and GitHub should be able to start contributing with these details in mind: • We use GitHub for source control and code review. To contribute, fork zulip/zulip (or the appropriate repos- itory, if you are working on something else besides Zulip server) to your own account and then create fea- ture/issue branches. When you’re ready to get feedback, submit a work-in-progress (WIP) pull request. We encourage you to submit WIP pull requests early and often. • We use a rebase-oriented workflow. We do not use merge commits. This means you should use git fetch followed by git rebase rather than git pull (or you can use git pull --rebase). Also, to prevent pull requests from becoming out of date with the main line of development, you should rebase your feature branch prior to submitting a pull request, and as needed thereafter. If you’re unfamiliar with how to rebase a pull request, read this excellent guide. We use this strategy in order to avoid the extra commits that appear when another branch is merged, that clutter the commit history (it’s popular with other large projects such as Django). This makes Zulip’s commit history more readable, but a side effect is that many pull requests we merge will be reported by GitHub’s UI as closed instead of merged, since GitHub has poor support for rebase-oriented workflows. • We have a code style guide, a commit message guide, and strive for each commit to be a minimal coherent idea (see commit discipline for details). • We provide many tools to help you submit quality code. These include linters, tests, continuous integration with Travis CI, and mypy. • We use zulipbot to manage our issues and pull requests to create a better GitHub workflow for contributors. Finally, take a quick look at Zulip-specific Git scripts, install the Zulip developer environment, and then configure your fork for use with Travis CI.
The following sections will help you be awesome with Zulip and Git/GitHub in a rebased-based workflow. Read through it if you’re new to git, to a rebase-based git workflow, or if you’d like a git refresher.
185 Zulip Documentation, Release 1.7.0
36.2 Set up Git
If you’re already using Git, have a client you like, and a GitHub account, you can skip this section. Otherwise, read on!
36.2.1 Install and configure Git, join GitHub
If you’re not already using Git, you might need to install and configure it. If you are using Windows 10, make sure you are running Git BASH as an administrator at all times. You’ll also need a GitHub account, which you can sign up for here. We highly recommend you create an ssh key if you don’t already have one and add it to your GitHub account. If you don’t, you’ll have to type your GitHub username and password every time you interact with GitHub, which is usually several times a day. We also highly recommend the following: • Configure Git with your name and email and aliases for commands you’ll use often. We recommend using your full name (not just your first name), since that’s what we’ll use to give credit to your work in places like the Zulip release notes. • Install the command auto-completion and/or git-prompt plugins available for Bash and Zsh.
36.2.2 Get a graphical client
Even if you’re comfortable using git on the command line, having a graphic client can be useful for viewing your repository. This is especially when doing a complicated rebases and similar operations because you can check the state of your repository after each command to see what changed. If something goes wrong, this helps you figure out when and why. If you don’t already have one installed, here are some suggestions: • macOS: GitX-dev • Ubuntu/Linux: git-cola, gitg, gitk • Windows: SourceTree If you like working on the command line, but want better visualization and navigation of your git repo, try Tig, a cross-platform ncurses-based text-mode interface to Git. And, if none of the above are to your liking, try one of these.
36.3 How Git is different
Whether you’re new to Git or have experience with another version control system (VCS), it’s a good idea to learn a bit about how Git works. We recommend this excellent presentation Understanding Git from Nelson Elhage and Anders Kaseorg and the Git Basics chapter from Pro Git by Scott Chacon and Ben Straub. Here are the top things to know: • Git works on snapshots: Unlike other version control systems (e.g., Subversion, Perforce, Bazaar), which track files and changes to those files made over time, Git tracks snapshots of your project. Each time you commit or otherwise make a change to your repository, Git takes a snapshot of your project and stores a reference to that snapshot. If a file hasn’t changed, Git creates a link to the identical file rather than storing it again.
186 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
• Most Git operations are local: Git is a distributed version control system, so once you’ve cloned a repository, you have a complete copy of that repository’s entire history. Staging, committing, branching, and browsing history are all things you can do locally without network access and without immediately affecting any remote repositories. To make or receive changes from remote repositories, you need to git fetch, git pull, or git push. • Nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to make Git perform actions that you can’t undo. However, Git can’t undo what it doesn’t know about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote repository. • Git is designed for lightweight branching and merging. Branches are simply references to snapshots. It’s okay and expected to make a lot of branches, even throwaway and experimental ones. • Git stores all data as objects, of which there are four types: blob (file), tree (directory), commit (revision), and tag. Each of these objects is named by a unique hash, the SHA-1 has of its contents. Most of the time you’ll refer to objects by their truncated hash or more human-readable reference like HEAD (the current branch). Blobs and trees represent files and directories. Tags are named references to other objects. A commit object includes: tree id, zero or more parents as commit ids, an author (name, email, date), a committer (name, email, date), and a log message. A Git repository is a collection of mutable pointers to these objects called refs. • Cloning a repository creates a working copy. Every working copy has a .git subdirectory, which contains its own Git repository. The .git subdirectory also tracks the index, a staging area for changes that will become part of the next commit. All files outside of .git is the working tree. • Files tracked with Git have possible three states: committed, modified, and staged. Committed files are those safely stored in your local .git repository/database. Staged files have changes and have been marked for inclusion in the next commit; they are part of the index. Modified files have changes but have not yet been marked for inclusion in the next commit; they have not been added to the index. • Git commit workflow is as follows: Edit files in your working tree. Add to the index (that is stage) with git add. Commit to the HEAD of the current branch with git commit.
36.4 Important Git terms
When you install Git, it adds a manual entry for gitglossary. You can view this glossary by running man gitglossary. Below we’ve included the git terms you’ll encounter most often along with their definitions from gitglossary.
36.4.1 branch
A "branch" is an active line of development. The most recent commit on a branch is referred to as the tip of that branch. The tip of the branch is referenced by a branch head, which moves forward as additional development is done on the branch. A single Git repository can track an arbitrary number of branches, but your working tree is associated with just one of them (the "current" or "checked out" branch), and HEAD points to that branch.
36.4.2 cache
Obsolete for: index
36.4.3 checkout
The action of updating all or part of the working tree with a tree object or blob from the object database, and updating the index and HEAD if the whole working tree has been pointed at a new branch.
36.4. Important Git terms 187 Zulip Documentation, Release 1.7.0
36.4.4 commit
As a noun: A single point in the Git history; the entire history of a project is represented as a set of interrelated commits. The word "commit" is often used by Git in the same places other revision control systems use the words "revision" or "version". Also used as a short hand for commit object. As a verb: The action of storing a new snapshot of the project’s state in the Git history, by creating a new commit representing the current state of the index and advancing HEAD to point at the new
36.4.5 fast-forward
A fast-forward is a special type of merge where you have a revision and you are "merging" another branch’s changes that happen to be a descendant of what you have. In such these cases, you do not make a new mergecommit but instead just update to their revision. This will happen frequently on a remote-tracking branch of a remote repository.
36.4.6 fetch
Fetching a branch means to get the branch’s head ref from a remote repository, to find out which objects are missing from the local object database, and to get them, too. See also git-fetch(1).
36.4.7 hash
In Git’s context, synonym for object name.
36.4.8 head
A named reference to the commit at the tip of a branch. Heads are stored in a file in $GIT_DIR/refs/heads/ directory, except when using packed refs. (See git-pack-refs(1).)
36.4.9 HEAD
The current branch. In more detail: Your working tree is normally derived from the state of the tree referred to by HEAD. HEAD is a reference to one of the heads in your repository, except when using a detached HEAD, in which case it directly references an arbitrary commit.
36.4.10 index
A collection of files with stat information, whose contents are stored as objects. The index is a stored version of your working tree. Truth be told, it can also contain a second, and even a third version of a working tree, which are used when merging.
36.4.11 pull
Pulling a branch means to fetch it and merge it. See also git- pull(1).
188 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
36.4.12 push
Pushing a branch means to get the branch’s head ref from a remote repository, find out if it is a direct ancestor to the branch’s local head ref, and in that case, putting all objects, which are reachable from the local head ref, and which are missing from the remote repository, into the remote object database, and updating the remote head ref. If the remote head is not an ancestor to the local head, the push fails.
36.4.13 rebase
To reapply a series of changes from a branch to a different base, and reset the head of that branch to the result.
36.5 Get Zulip code
Zulip uses a forked-repo and rebase-oriented workflow.. This means that all contributors create a fork of the Zulip repository they want to contribute to and then submit pull requests to the upstream repository to have their contributions reviewed and accepted. We also recommend you work on feature branches.
36.5.1 Step 1a: Create your fork
The following steps you’ll only need to do the first time you setup a machine for contributing to a given Zulip project. You’ll need to repeat the steps for any additional Zulip projects (list) that you work on. The first thing you’ll want to do to contribute to Zulip is fork (see how) the appropriate Zulip repository. For the main server app, this is zulip/zulip.
36.5.2 Step 1b: Clone to your machine
Next, clone your fork to your local machine:
$ git clone [email protected]:christi3k/zulip.git Cloning into 'zulip' remote: Counting objects: 86768, done. remote: Compressing objects: 100% (15/15), done. remote: Total 86768 (delta 5), reused 1 (delta 1), pack-reused 86752 Receiving objects: 100% (86768/86768), 112.96 MiB | 523.00 KiB/s, done. Resolving deltas: 100% (61106/61106), done. Checking connectivity... done.
Note: If you receive an error while cloning, you may not have added your ssh key to GitHub.
36.5.3 Step 1c: Connect your fork to Zulip upstream
Next you’ll want to configure an upstream remote repository for your fork of Zulip. This will allow you to sync changes from the main project back into your fork. First, show the currently configured remote repository:
$ git remote -v origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push)
36.5. Get Zulip code 189 Zulip Documentation, Release 1.7.0
Note: If you’ve cloned the repository using a graphical client, you may already have the upstream remote repository configured. For example, when you clone zulip/zulip with the GitHub desktop client it configures the remote repository zulip and you see the following output from git remote -v: origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push) zulip https://github.com/zulip/zulip.git (fetch) zulip https://github.com/zulip/zulip.git (push)
If your client hasn’t automatically configured a remote for zulip/zulip, you’ll need to with:
$ git remote add upstream https://github.com/zulip/zulip.git
Finally, confirm that the new remote repository, upstream, has been configured:
$ git remote -v origin [email protected]:YOUR_USERNAME/zulip.git (fetch) origin [email protected]:YOUR_USERNAME/zulip.git (push) upstream https://github.com/zulip/zulip.git (fetch) upstream https://github.com/zulip/zulip.git (push)
36.5.4 Step 2: Set up the Zulip development environment
If you haven’t already, now is a good time to install the Zulip development environment (overview). If you’re new to working on Zulip or open source projects in general, we recommend following our detailed guide for first-time contributors.
36.5.5 Step 3: Configure Travis CI (continuous integration)
This step is optional, but recommended. The Zulip Server project is configured to use Travis CI to test and create builds upon each new commit and pull request. Travis CI is free for open source projects and it’s easy to configure for your own fork of Zulip. After doing so, Travis CI will run tests for new refs you push to GitHub and email you the outcome (you can also view the results in the web interface). Running Travis CI against your fork can help save both you and the Zulip maintainers time by making it easy to test a change fully before submitting a pull request. We generally recommend a worfklow where as you make changes, you use a fast edit-refresh cycle running individual tests locally until your changes work. But then once you’ve gotten the tests you’d expect to be relevant to your changes working, push a branch to Travis CI to run the full test suite before you create a pull request. While you wait for Travis CI to run, you can start working on your next task. When the tests finish, you can create a pull request that you already know passes the tests. First, sign in to Travis CI with your GitHub account and authorize Travis CI to access your GitHub account and repositories. Once you’ve done this, Travis CI will fetch your repository information and display it on your profile page. From there you can enable integration with Zulip. (See screen cast.)
36.6 Using Git as you work
36.6.1 Know what branch you’re working on
When using Git, it’s important to know which branch you currently have checked out because most git commands implicitly operate on the current branch. You can determine the currently checked out branch several ways.
190 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
One way is with git status:
$ git status On branch issue-demo nothing to commit, working directory clean
Another is with git branch which will display all local branches, with a star next to the current branch:
$ git branch * issue-demo master
To see even more information about your branches, including remote branches, use git branch -vva:
$ git branch -vva * issue-123 517468b troubleshooting tip about provisioning master f0eaee6 [origin/master] bug: Fix traceback in get_missed_
˓→message_token_from_address(). remotes/origin/HEAD -> origin/master remotes/origin/issue-1234 4aeccb7 Another test commit, with longer message. remotes/origin/master f0eaee6 bug: Fix traceback in get_missed_message_token_
˓→from_address(). remotes/upstream/master dbeab6a Optimize checks of test database state by moving
˓→into Python.
You can also configure Bash and Zsh to display the current branch in your prompt.
36.6.2 Keep your fork up to date
You’ll want to keep your fork up-to-date with changes from Zulip’s main repositories. Note about git pull: You might be used to using git pull on other projects. With Zulip, because we don’t use merge commits, you’ll want to avoid it. Rather that using git pull, which by default is a shortcut for git fetch && git merge FETCH_HEAD (docs), you should use git fetch and then git rebase. First, fetch changes from Zulip’s upstream repository you configured in the step above:
$ git fetch upstream
Next, checkout your master branch and rebase it on top of upstream/master:
$ git checkout master Switched to branch 'master'
$ git rebase upstream/master
This will rollback any changes you’ve made to master, update it from upstream/master, and then re-apply your changes. Rebasing keeps the commit history clean and readable. When you’re ready, push your changes to your remote fork. Make sure you’re in branch master and the run git push:
$ git checkout master $ git push origin master
You can keep any branch up to date using this method. If you’re working on a feature branch (see next section), which we recommend, you would change the command slightly, using the name of your feature-branch rather than master:
36.6. Using Git as you work 191 Zulip Documentation, Release 1.7.0
$ git checkout feature-branch Switched to branch 'feature-branch'
$ git rebase upstream/master
$ git push origin feature-branch
36.6.3 Work on a feature branch
One way to keep your work organized is to create a branch for each issue or feature. Recall from how Git is different that Git is designed for lightweight branching and merging. You can and should create as many branches as you’d like. First, make sure your master branch is up-to-date with Zulip upstream (see how). Next, from your master branch, create a new tracking branch, providing a descriptive name for your feature branch:
$ git checkout master Switched to branch 'master'
$ git checkout -b issue-1755-fail2ban Switched to a new branch 'issue-1755-fail2ban'
Alternatively, you can create a new branch explicitly based off upstream/master:
$ git checkout -b issue-1755-fail2ban upstream/master Switched to a new branch 'issue-1755-fail2ban'
Now you’re ready to work on the issue or feature.
36.6.4 Run linters and tests locally
In addition to having Travis run tests and linters each time you push a new commit, you can also run them locally. See testing for details.
36.6.5 Stage changes
Recall that files tracked with Git have possible three states: committed, modified, and staged. To prepare a commit, first add the files with changes that you want to include in your commit to your staging area. You add both new files and existing ones. You can also remove files from staging when necessary.
Get status of working directory
To see what files in the working directory have changes that have not been staged, use git status. If you have no changes in the working directory, you’ll see something like this:
$ git status On branch issue-123 nothing to commit, working directory clean
If you have unstaged changes, you’ll see something like this:
192 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
On branch issue-123 Untracked files: (use"git add
newfile.py
nothing added to commit but untracked files present (use"git add" to track)
Stage additions with git add
To add changes to your staging area, use git add
On branch issue-123 Changes to be committed: (use"git reset HEAD
new file: newfile.py
You can view the changes in files you have staged with git diff --cached. To view changes to files you haven’t yet staged, just use git diff. If you want to add all changes in the working directory, use git add -A (documentation). You can also stage changes using your graphical Git client. If you stage a file, you can undo it with git reset HEAD
$ git add test3.txt On branch issue-1234 Changes to be committed: (use "git reset HEAD
new file: test3.txt
$ git reset HEAD test3.txt $ git status On branch issue-1234 Untracked files: (use "git add
test3.txt
nothing added to commit but untracked files present (use "git add" to track)
Stage deletions with git rm
To remove existing files from your repository, use git rm (documentation). This command can either stage the file for removal from your repository AND delete it from your working directory or just stage the file for deletion and leave it in your working directory. To stage a file for deletion and remove it from your working directory, use git rm
36.6. Using Git as you work 193 Zulip Documentation, Release 1.7.0
$ git rm test.txt rm 'test.txt'
$ git status On branch issue-1234 Changes to be committed: (use "git reset HEAD
deleted: test.txt
$ ls test.txt ls: No such file or directory
To stage a file for deletion and keep it in your working directory, use git rm --cached
$ git rm --cached test2.txt rm 'test2.txt'
$ git status On branch issue-1234 Changes to be committed: (use "git reset HEAD
deleted: test2.txt
$ ls test2.txt test2.txt
If you stage a file for deletion with the --cached option, and haven’t yet run git commit, you can undo it with git reset HEAD
$ git reset HEAD test2.txt
Unfortunately, you can’t restore a file deleted with git rm if you didn’t use the --cache option. However, git rm only deletes files it knows about. Files you have never added to git won’t be deleted.
36.6.6 Commit changes
When you’ve staged all your changes, you’re ready to commit. You can do this with git commit -m "My commit message." to include a commit message. Here’s an example of committing with the -m for a one-line commit message:
$ git commit -m "Add a test commit for docs." [issue-123 173e17a] Add a test commit for docs. 1 file changed, 1 insertion(+) create mode 100644 newfile.py
You can also use git commit without the -m option and your editor to open, allowing you to easily draft a multi-line commit message. How long your commit message should be depends on where you are in your work. Using short, one-line messages for commits related to in-progress work makes sense. For a commit that you intend to be final or that encompasses a significant amount or complex work, you should include a longer message. Keep in mind that your commit should contain a ’minimal coherent idea’ and have a quality commit message. See Zulip docs Commit Discipline and Commit messages for details.
194 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
Here’s an example of a longer commit message that will be used for a pull request:
Integrate Fail2Ban.
Updates Zulip logging to put an unambiguous entry into the logs such that fail2ban can be configured to look for these entries.
Tested on my local Ubuntu development server, but would appreciate someone testing on a production install with more users.
Fixes #1755.
The first line is the summary. It’s a complete sentence, ending in a period. It uses a present-tense action verb, "Integrate", rather than "Integrates" or "Integrating". The following paragraphs are full prose and explain why and how the change was made. It explains what testing was done and asks specifically for further testing in a more production-like environment. The final paragraph indicates that this commit addresses and fixes issue #1755. When you submit your pull request, GitHub will detect and link this reference to the appropriate issue. Once your commit is merged into zulip/master, GitHub will automatically close the referenced issue. See Closing issues via commit messages for details. Make as many commits as you need to to address the issue or implement your feature.
36.6.7 Push your commits to GitHub
As you’re working, it’s a good idea to frequently push your changes to GitHub. This ensures your work is backed up should something happen to your local machine and allows others to follow your progress. It also allows you to work from multiple computers without losing work. Pushing to a feature branch is just like pushing to master:
$ git push origin
If you want to see what git will do without actually performing the push, add the -n (dry-run) option: git push -n origin
36.6.8 Examine and tidy your commit history
Examining your commit history prior to submitting your pull request is a good idea. Is it tidy such that each commit represents a minimally coherent idea (see commit discipline)? Do your commit messages follow Zulip’s style? Will the person reviewing your commit history be able to clearly understand your progression of work? On the command line, you can use the git log command to display an easy to read list of your commits:
36.6. Using Git as you work 195 Zulip Documentation, Release 1.7.0
$ git log --all --graph --oneline --decorate
* 4f8d75d (HEAD -> 1754-docs-add-git-workflow) docs: Add details about configuring ˓→Travis CI. * bfb2433 (origin/1754-docs-add-git-workflow) docs: Add section for keeping fork up- ˓→to-date to Git Guide. * 4fe10f8 docs: Add sections for creating and configuring fork to Git Guide. * 985116b docs: Add graphic client recs to Git Guide. * 3c40103 docs: Add stubs for remaining Git Guide sections. * fc2c01e docs: Add git guide quickstart. | * f0eaee6 (upstream/master) bug: Fix traceback in get_missed_message_token_from_ ˓→address().
Alternatively, use your graphical client to view the history for your feature branch. If you need to update any of your commits, you can do so with an interactive rebase. Common reasons to use an interactive rebase include: • squashing several commits into fewer commits • splitting a single commit into two or more • rewriting one or more commit messages There is ample documentation on how to rebase, so we won’t go into details here. We recommend starting with GitHub’s help article on rebasing and then consulting Git’s documentation for git-rebase if you need more details. If all you need to do is edit the commit message for your last commit, you can do that with git commit --amend. See Git Basics - Undoing Things for details on this and other useful commands.
36.6.9 Force-push changes to GitHub after you’ve altered your history
Any time you alter history for commits you have already pushed to GitHub, you’ll need to prefix the name of your branch with a +. Without this, your updates will be rejected with a message such as:
$ git push origin 1754-docs-add-git-workflow To [email protected]:christi3k/zulip.git ! [rejected] 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (non-fast-
˓→forward) error: failed to push some refs to '[email protected]:christi3k/zulip.git' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Re-running the command with +
$ git push origin +1754-docs-add-git-workflow Counting objects: 12, done. Delta compression using up to 4 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (12/12), 3.71 KiB | 0 bytes/s, done. Total 12 (delta 8), reused 0 (delta 0) remote: Resolving deltas: 100% (8/8), completed with 2 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (forced
˓→update)
196 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the branch. If others are working along with you, they might run into complications when they retrieve your changes because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.
36.7 Create a pull request
When you’re ready for feedback, submit a pull request. Pull requests are a feature specific to GitHub. They provide a simple, web-based way to submit your work (often called "patches") to a project. It’s called a pull request because you’re asking the project to pull changes from your fork. If you’re unfamiliar with how to create a pull request, you can check out GitHub’s documentation on creating a pull request from a fork. You might also find GitHub’s article about pull requests helpful. That all said, the tutorial below will walk you through the process.
36.7.1 Work in progress pull requests
In the Zulip project, we encourage submitting work-in-progress pull requests early and often. This allows you to share your code to make it easier to get feedback and help with your changes. Prefix the titles of work-in-progress pull requests with [WIP], which in our project means that you don’t think your pull request is ready to be merged (e.g. it might not work or pass tests). This sets expectations correctly for any feedback from other developers, and prevents your work from being merged before you’re confident in it.
36.7.2 Step 1: Update your branch with git rebase
The best way to update your branch is with git fetch and git rebase. Do not use git pull or git merge as this will create merge commits. See keep your fork up to date for details. Here’s an example (you would replace issue-123 with the name of your feature branch):
$ git checkout issue123 Switched to branch 'issue-123'
$ git fetch upstream remote: Counting objects: 69, done. remote: Compressing objects: 100% (23/23), done. remote: Total 69 (delta 49), reused 39 (delta 39), pack-reused 7 Unpacking objects: 100% (69/69), done. From https://github.com/zulip/zulip 69fa600..43e21f6 master -> upstream/master
$ git rebase upstream/master
First, rewinding head to replay your work on top of it... Applying: troubleshooting tip about provisioning
36.7.3 Step 2: Push your updated branch to your remote fork
Once you’ve updated your local feature branch, push the changes to GitHub:
$ git push origin issue-123 Counting objects: 6, done. Delta compression using up to 4 threads.
36.7. Create a pull request 197 Zulip Documentation, Release 1.7.0
Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 issue-123 -> issue-123
If your push is rejected with error failed to push some refs then you need to prefix the name of your branch with a +:
$ git push origin +issue-123 Counting objects: 6, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done. Total 6 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), completed with 1 local objects. To [email protected]:christi3k/zulip.git + 2d49e2d...bfb2433 issue-123 -> issue-123 (forced update)
This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the branch. If others are working along with you, they might run into complications when they retrieve your changes because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.
36.7.4 Step 3: Open the pull request
If you’ve never created a pull request or need a refresher, take a look at GitHub’s article creating a pull request from a fork. We’ll briefly review the process here. The first step in creating a pull request is to use your web browser to navigate to your fork of Zulip. Sign in to GitHub if you haven’t already. Next, navigate to the branch you’ve been working on. Do this by clicking on the Branch button and selecting the relevant branch. Finally, click the New pull request button. Alternatively, if you’ve recently pushed to your fork, you will see a green Compare & pull request button. You’ll see the Open a pull request page:
198 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
Provide a title and first comment for your pull request. Remember to prefix your pull request title with [WIP] if it is a work-in-progress. If your pull request has an effect on the visuals of a component, you might want to include a screenshot of this change or a GIF of the interaction in your first comment. This will allow reviewers to comment on your changes without having to checkout your branch; you can find a list of tools you can use for this over here. When ready, click the green Create pull request to submit the pull request. Note: Pull request titles are different from commit messages. Commit messages can be edited with git commit --amend, git rebase -i, etc., while the title of a pull request can only be edited via GitHub.
36.8 Update a pull request
As you get make progress on your feature or bugfix, your pull request, once submitted, will be updated each time you push commits to your remote branch. This means you can keep your pull request open as long as you need, rather than closing and opening new ones for the same feature or bugfix. It’s a good idea to keep your pull request mergeable with Zulip upstream by frequently fetching, rebasing, and pushing changes. See keep your fork up to date for details. You might also find this excellent article How to Rebase a Pull Request helpful. And, as you address review comments others have made, we recommend posting a follow-up comment in which you: a) ask for any clarifications you need, b) explain to the reviewer how you solved any problems they mentioned, and c) ask for another review.
36.8. Update a pull request 199 Zulip Documentation, Release 1.7.0
36.9 Collaborate
36.9.1 Fetch another contributor’s branch
What happens when you would like to collaborate with another contributor and they have work-in-progress on their own fork of Zulip? No problem! Just add their fork as a remote and pull their changes.
$ git remote add
Now you can checkout their branch just like you would any other. You can name the branch anything you want, but using both the username and branch name will help you keep things organized.
$ git checkout -b
You can choose to rename the branch if you prefer:
git checkout-b
36.9.2 Checkout a pull request locally
Just as you can checkout any user’s branch locally, you can also checkout any pull request locally. GitHub provides a special syntax (details) for this since pull requests are specific to GitHub rather than Git. First, fetch and create a branch for the pull request, replacing ID and BRANCHNAME with the ID of the pull request and your desired branch name:
$ git fetch upstream pull/ID/head:BRANCHNAME
Now switch to the branch:
$ git checkout BRANCHNAME
Now you work on this branch as you would any other. Note: you can use the scripts provided in the tools/ directory to fetch pull requests. You can read more about what they do here.
tools/fetch-rebase-pull-request
36.10 Review changes
36.10.1 Changes on (local) working tree
Display changes between index and working tree (what is not yet staged for commit):
$ git diff
Display changes between index and last commit (what you have staged for commit):
200 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
$ git diff --cached
Display changes in working tree since last commit (changes that are staged as well as ones that are not):
$ git diff HEAD
36.10.2 Changes within branches
Use any git-ref to compare changes between two commits on the current branch. Display changes between commit before last and last commit:
$ git diff HEAD^ HEAD
Display changes between two commits using their hashes:
$ git diff e2f404c 7977169
36.10.3 Changes between branches
Display changes between tip of topic branch and tip of master branch:
$ git diff topic master
Display changes that have occurred on master branch since topic branch was created:
$ git diff topic...master
Display changes you’ve committed so far since creating a branch from upstream/master:
$ git diff upstream/master...HEAD
36.11 Get and stay out of trouble
Git is a powerful yet complex version control system. Even for contributors experienced at using version control, it can be confusing. The good news is that nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to make Git perform actions that you can’t undo. However, git can’t undo what it doesn’t know about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote repository.
36.11.1 Undo a merge commit
A merge commit is a special type of commit that has two parent commits. It’s created by Git when you merge one branch into another and the last commit on your current branch is not a direct ancestor of the branch you are trying to merge in. This happens quite often in a busy project like Zulip where there are many contributors because upstream/zulip will have new commits while you’re working on a feature or bugfix. In order for Git to merge your changes and the changes that have occurred on zulip/upstream since you first started your work, it must perform a three-way merge and create a merge commit. Merge commits aren’t bad, however, Zulip doesn’t use them. Instead Zulip uses a forked-repo, rebase-oriented work- flow.
36.11. Get and stay out of trouble 201 Zulip Documentation, Release 1.7.0
A merge commit is usually created when you’ve run git pull or git merge. You’ll know you’re creating a merge commit if you’re prompted for a commit message and the default is something like this:
Merge branch'master' of https://github.com/zulip/zulip
# Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.
And the first entry for git log will show something like:
commit e5f8211a565a5a5448b93e98ed56415255546f94 Merge: 13bea0e e0c10ed Author: Christie Koehler
Merge branch'master' of https://github.com/zulip/zulip
Some graphical Git clients may also create merge commits. To undo a merge commit, first run git reflog to identify the commit you want to roll back to:
$ git reflog
e5f8211 HEAD@{0}: pull upstream master: Merge made by the 'recursive' strategy. 13bea0e HEAD@{1}: commit: test commit for docs.
Reflog output will be long. The most recent git refs will be listed at the top. In the example above e5f8211 HEAD@{0}: is the merge commit made automatically by git pull and 13bea0e HEAD@{1}: is the last com- mit I made before running git pull, the commit that I want to rollback to. Once you’d identified the ref you want to revert to, you can do so with git reset:
$ git reset --hard 13bea0e HEAD is now at 13bea0e test commit for docs.
Important: git reset --hard
$ git reflog
13bea0e HEAD@{2}: reset: moving to HEAD@{1} e5f8211 HEAD@{3}: pull upstream master: Merge made by the 'recursive' strategy. 13bea0e HEAD@{4}: commit: test commit for docs.
And the first entry git log shows is this:
commit 13bea0e40197b1670e927a9eb05aaf50df9e8277 Author: Christie Koehler
202 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
test commit for docs.
36.11.2 Restore a lost commit
We’ve mentioned you can use git reset --hard to rollback to a previous commit. What if you run git reset --hard and then realize you actually need one or more of the commits you just discarded? No problem, you can restore them with git cherry-pick (docs). For example, let’s say you just committed "some work" and your git log looks like this:
* 67aea58 (HEAD-> master) some work * 13bea0e test commit for docs.
You then mistakenly run git reset --hard 13bea0e:
$ git reset --hard 13bea0e HEAD is now at 13bea0e test commit for docs.
$ git log * 13bea0e (HEAD -> master) test commit for docs.
And then realize you actually needed to keep commit 67aea58. First, use git reflog to confirm that commit you want to restore and then run git cherry-pick
$ git reflog 13bea0e HEAD@{0}: reset: moving to 13bea0e 67aea58 HEAD@{1}: commit: some work
$ git cherry-pick 67aea58 [master 67aea58] some work Date: Thu Oct 13 11:51:19 2016 -0700 1 file changed, 1 insertion(+) create mode 100644 test4.txt
36.11.3 Recover from a git rebase failure
One situation in which git rebase will fail and require you to intervene is when your change, which git will try to re-apply on top of new commits from which ever branch you are rebasing on top of, is to code that has been changed by those new commits. For example, while I’m working on a file, another contributor makes a change to that file, submits a pull request and has their code merged into master. Usually this is not a problem, but in this case the other contributor made a change to a part of the file I also want to change. When I try to bring my branch up to date with git fetch and then git rebase upstream/master, I see the following:
First, rewinding head to replay your work on top of it... Applying: test change for docs Using index info to reconstruct a base tree... M README.md Falling back to patching base and 3-way merge... Auto-merging README.md CONFLICT (content): Merge conflict in README.md error: Failed to merge in the changes.
36.11. Get and stay out of trouble 203 Zulip Documentation, Release 1.7.0
Patch failed at 0001 test change for docs The copy of the patch that failed is found in:.git/rebase-apply/patch
When you have resolved this problem, run"git rebase --continue". If you prefer to skip this patch, run"git rebase --skip" instead. To check out the original branch and stop rebasing, run"git rebase --abort".
This message tells me that Git was not able to apply my changes to README.md after bringing in the new commits from upstream/master. Running git status also gives me some information:
rebase in progress; onto5ae56e6 You are currently rebasing branch'docs-test' on'5ae56e6'. (fix conflicts and then run"git rebase --continue") (use"git rebase --skip" to skip this patch) (use"git rebase --abort" to check out the original branch)
Unmerged paths: (use"git reset HEAD
both modified: README.md
no changes added to commit (use"git add" and/or "git commit -a")
To fix, open all the files with conflicts in your editor and decide which edits should be applied. Git uses standard conflict-resolution (<<<<<<<, ======, and >>>>>>>) markers to indicate where in files there are conflicts. Tip: You can see recent changes made to a file by running the following commands:
git fetch upstream git log-p upstream/master--/path/to/file
You can use this to compare the changes that you have made to a file with the ones in upstream, helping you avoid undoing changes from a previous commit when you are rebasing. Once you’ve done that, save the file(s), stage them with git add and then continue the rebase with git rebase --continue:
$ git add README.md
$ git rebase --continue Applying: test change for docs
For help resolving merge conflicts, see basic merge conflicts, advanced merging, and/or GitHub’s help on how to resolve a merge conflict.
36.11.4 Working from multiple computers
Working from multiple computers with Zulip and Git is fine, but you’ll need to pay attention and do a bit of work to ensure all of your work is readily available. Recall that most Git operations are local. When you commit your changes with git commit they are safely stored in your local Git database only. That is, until you push the commits to GitHub, they are only available on the computer where you committed them.
204 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
So, before you stop working for the day, or before you switch computers, push all of your commits to GitHub with git push:
$ git push origin
When you first start working on a new computer, you’ll clone the Zulip repository and connect it to Zulip upstream. A clone retrieves all current commits, including the ones you pushed to GitHub from your other computer. But if you’re switching to another computer on which you have already cloned Zulip, you need to update your local Git database with new refs from your GitHub fork. You do this with git fetch:
$ git fetch
Ideally you should do this before you have made any commits on the same branch on the second computer. Then you can git merge on whichever branch you need to update:
$ git checkout
$ git merge origin/master
If you have already made commits on the second computer that you need to keep, you’ll need to use git log FETCH_HEAD to identify that hashes of the commits you want to keep and then git cherry-pick
36.12 Zulip-specific tools
This section will document the zulip-specific git tools contributors will find helpful.
36.12.1 Set up git repo script
In the tools directory of zulip/zulip you’ll find a bash script setup-git-repo. This script installs the Zulip pre-commit hook. This hook will run each time you git commit to automatically run linters, etc. The hook passes no matter the result of the linter, but you should still pay attention to any notices or warnings it displays. It’s simple to use. Make sure you’re in the clone of zulip and run the following:
$ ./tools/setup-git-repo
The script doesn’t produce any output if successful. To check that the hook has been installed, print a directory listing for .git/hooks and you should see something similar to:
$ ls -l .git/hooks pre-commit -> ../../tools/pre-commit
36.12.2 Set up Travis CI integration
You might also wish to configure your fork for use with Travis CI.
36.12. Zulip-specific tools 205 Zulip Documentation, Release 1.7.0
36.12.3 Reset to pull request tools/reset-to-pull-request is a short-cut for checking out a pull request locally. It works slightly differ- ently from the method described above in that it does not create a branch for the pull request checkout. This tool checks for uncommitted changes, but it will move the current branch using git reset --hard. Use with caution. First, make sure you are working in a branch you want to move (in this example, we’ll use the local master branch). Then run the script with the ID number of the pull request as the first argument.
$ git checkout master Switched to branch 'master' Your branch is up-to-date with 'origin/master'.
$ ./tools/reset-to-pull-request 1900 + request_id=1900 + git fetch upstream pull/1900/head remote: Counting objects: 159, done. remote: Compressing objects: 100% (17/17), done. remote: Total 159 (delta 94), reused 91 (delta 91), pack-reused 51 Receiving objects: 100% (159/159), 55.57 KiB | 0 bytes/s, done. Resolving deltas: 100% (113/113), completed with 54 local objects. From https://github.com/zulip/zulip * branch refs/pull/1900/head -> FETCH_HEAD + git reset --hard FETCH_HEAD HEAD is now at 2bcd1d8 troubleshooting tip about provisioning
36.12.4 Fetch a pull request and rebase tools/fetch-rebase-pull-request is a short-cut for checking out a pull request locally in its own branch and then updating it with any changes from upstream/master with git rebase. Run the script with the ID number of the pull request as the first argument.
$ tools/fetch-rebase-pull-request 1913 + request_id=1913 + git fetch upstream pull/1913/head remote: Counting objects: 4, done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (4/4), done. From https://github.com/zulip/zulip * branch refs/pull/1913/head -> FETCH_HEAD + git checkout upstream/master -b review-1913 Branch review-1913 set up to track remote branch master from upstream. Switched to a new branch 'review-1913' + git reset --hard FETCH_HEAD HEAD is now at 99aa2bf Add provision.py fails issue in common erros + git pull --rebase Current branch review-1913 is up to date.
36.12.5 Fetch a pull request without rebasing tools/fetch-pull-request is a similar to tools/fetch-rebase-pull-request, but it does not re- base the pull request against upstream/master, thereby getting exactly the same repository state as the commit author
206 Chapter 36. Git & GitHub Guide Zulip Documentation, Release 1.7.0
had. Run the script with the ID number of the pull request as the first argument.
$ tools/fetch-pull-request 5156 + git diff-index --quiet HEAD + request_id=5156 + remote=upstream + git fetch upstream pull/5156/head From https://github.com/zulip/zulip * branch refs/pull/5156/head -> FETCH_HEAD + git checkout -B review-original-5156 Switched to a new branch 'review-original-5156' + git reset --hard FETCH_HEAD HEAD is now at 5a1e982 tools: Update clean-branches to clean review branches.
36.12.6 Delete unimportant branches
tools/clean-branches is a shell script that removes branches that are either: 1. Local branches that are ancestors of origin/master. 2. Branches in origin that are ancestors of origin/master and named like $USER-*. 3. Review branches created by tools/fetch-rebase-pull-request and tools/ fetch-pull-request. First, make sure you are working in branch master. Then run the script without any arguments for default behavior. Since removing review branches can inadvertently remove any feature branches whose names are like review-*, it is not done by default. To use it, run tools/clean-branches --reviews.
$ tools/clean-branches --reviews Deleting local branch review-original-5156 (was 5a1e982)
36.12.7 Merge conflict on yarn.lock file
If there is a merge conflict on yarn.lock, yarn should be run to regenerate the file. Important don’t delete the yarn.lock file. Checkout the latest one from origin/master so that yarn knows the previous asset versions. Run the following commands
git checkout origin/master-- yarn.lock yarn install git add yarn.lock git rebase-- continue
36.12. Zulip-specific tools 207 Zulip Documentation, Release 1.7.0
208 Chapter 36. Git & GitHub Guide CHAPTER 37
Version control
37.1 Commit Discipline
We follow the Git project’s own commit discipline practice of "Each commit is a minimal coherent idea". This discipline takes a bit of work, but it makes it much easier for code reviewers to spot bugs, and makes the commit history a much more useful resource for developers trying to understand why the code works the way it does, which also helps a lot in preventing bugs. Commits must be coherent: • It should pass tests (so test updates needed by a change should be in the same commit as the original change, not a separate "fix the tests that were broken by the last commit" commit). • It should be safe to deploy individually, or explain in detail in the commit message as to why it isn’t (maybe with a [manual] tag). So implementing a new API endpoint in one commit and then adding the security checks in a future commit should be avoided – the security checks should be there from the beginning. • Error handling should generally be included along with the code that might trigger the error. • TODO comments should be in the commit that introduces the issue or the functionality with further work required. Commits should generally be minimal: • Significant refactorings should be done in a separate commit from functional changes. • Moving code from one file to another should be done in a separate commits from functional changes or even refactoring within a file. • 2 different refactorings should be done in different commits. • 2 different features should be done in different commits. • If you find yourself writing a commit message that reads like a list of somewhat dissimilar things that you did, you probably should have just done multiple commits. When not to be overly minimal:
209 Zulip Documentation, Release 1.7.0
• For completely new features, you don’t necessarily need to split out new commits for each little subfeature of the new feature. E.g., if you’re writing a new tool from scratch, it’s fine to have the initial tool have plenty of options/features without doing separate commits for each one. That said, reviewing a 2000-line giant blob of new code isn’t fun, so please be thoughtful about submitting things in reviewable units. • Don’t bother to split backend commits from frontend commits, even though the backend can often be coherent on its own. Other considerations: • Overly fine commits are easy to squash later, but not vice versa. So err toward small commits, and the code reviewer can advise on squashing. • If a commit you write doesn’t pass tests, you should usually fix that by amending the commit to fix the bug, not writing a new "fix tests" commit on top of it. Zulip expects you to structure the commits in your pull requests to form a clean history before we will merge them. It’s best to write your commits following these guidelines in the first place, but if you don’t, you can always fix your history using git rebase -i. Never mix multiple changes together in a single commit, but it’s great to include several related changes, each in their own commit, in a single pull request. If you notice an issue that is only somewhat related to what you were working on, but you feel that it’s too minor to create a dedicated pull request, feel free to append it as an additional commit in the pull request for your main project (that commit should have a clear explanation of the bug in its commit message). This way, the bug gets fixed, but this independent change is highlighted for reviewers. Or just create a dedicated pull request for it. Whatever you do, don’t squash unrelated changes together in a single commit; the reviewer will ask you to split the changes out into their own commits. It can take some practice to get used to writing your commits with a clean history so that you don’t spend much time doing interactive rebases. For example, often you’ll start adding a feature, and discover you need to do a refactor- ing partway through writing the feature. When that happens, we recommend you stash your partial feature, do the refactoring, commit it, and then unstash and finish implementing your feature.
37.2 Commit Messages
First, check out these examples of commits with good commit messages. The first line of the commit message is the summary. The summary: • is written in the imperative (e.g., "Fix ...", "Add ...") • is kept short, while concisely explaining what the commit does • is clear about what part of the code is affected – often by prefixing with the name of the subsystem and a colon, like "zjsunit: ..." or "docs: ..." • is a complete sentence, ending with a period. Good summaries: zjsunit: Fix running stream_data and node tests individually. gather_subscriptions: Fix exception handling bad input. Add GitLab integration. Compare "gather_subscriptions: Fix exception handling bad input." with: •" gather_subscriptions was broken", which doesn’t explain how it was broken (and isn’t in the imperative) •" Fix exception when given bad input", in which it’s impossible to tell from the summary what part of the code is affected
210 Chapter 37. Version control Zulip Documentation, Release 1.7.0
•" gather_subscriptions: Fixing exception when given bad input.", not in the imperative •" gather_subscriptions: Fixed exception when given bad input.", not in the imperative The summary is followed by a blank line, and then the body of the commit message. • The body is written in prose, with full paragraphs. • The body explains: – why and how the change was made – any manual testing you did in addition to running the automated tests – any aspects of the commit that you think are questionable and you’d like special attention applied to. • If the commit makes performance improvements, you should generally include some rough benchmarks showing that it actually improves the performance. • When you fix a GitHub issue, mark that you’ve fixed the issue in your commit message so that the issue is automatically closed when your code is merged. Zulip’s preferred style for this is to have the final paragraph of the commit message read e.g. "Fixes: #123." • Any paragraph content in the commit message should be line-wrapped to less than 76 characters per line, so that your commit message will be reasonably readable in git log in a normal terminal.
37.2. Commit Messages 211 Zulip Documentation, Release 1.7.0
212 Chapter 37. Version control CHAPTER 38
Code style and conventions
38.1 Be consistent!
Look at the surrounding code, or a similar part of the project, and try to do the same thing. If you think the other code has actively bad style, fix it (in a separate commit). When in doubt, ask in chat.zulip.org.
38.2 Lint tools
You can run them all at once with
./tools/lint
You can set this up as a local Git commit hook with
``tools/setup-git-repo``
The Vagrant setup process runs this for you. lint runs many lint checks in parallel, including • JavaScript (ESLint) • Python (Pyflakes) • templates • Puppet configuration • custom checks (e.g. trailing whitespace and spaces-not-tabs)
213 Zulip Documentation, Release 1.7.0
38.3 Secrets
Please don’t put any passwords, secret access keys, etc. inline in the code. Instead, use the get_secret function in zproject/settings.py to read secrets from /etc/zulip/secrets.conf.
38.4 Dangerous constructs
38.4.1 Misuse of database queries
Look out for Django code like this:
[Foo.objects.get(id=bar.x.id) for bar in Bar.objects.filter(...) if bar.baz<7]
This will make one database query for each Bar, which is slow in production (but not in local testing!). Instead of a list comprehension, write a single query using Django’s QuerySet API. If you can’t rewrite it as a single query, that’s a sign that something is wrong with the database schema. So don’t defer this optimization when performing schema changes, or else you may later find that it’s impossible.
38.4.2 UserProfile.objects.get() / Client.objects.get / etc.
In our Django code, never do direct UserProfile.objects.get(email=foo) database queries. Instead always use get_user_profile_by_{email,id}. There are 3 reasons for this: 1. It’s guaranteed to correctly do a case-inexact lookup 2. It fetches the user object from remote cache, which is faster 3. It always fetches a UserProfile object which has been queried using .select_related(), and thus will perform well when one later accesses related models like the Realm. Similarly we have get_client and get_stream functions to fetch those commonly accessed objects via remote cache.
38.4.3 Using Django model objects as keys in sets/dicts
Don’t use Django model objects as keys in sets/dictionaries – you will get unexpected behavior when dealing with objects obtained from different database queries: For example, UserProfile.objects.only("id").get(id=17) in set([UserProfile. objects.get(id=17)]) is False You should work with the IDs instead.
38.4.4 user_profile.save()
You should always pass the update_fields keyword argument to .save() when modifying an existing Django model object. By default, .save() will overwrite every value in the column, which results in lots of race conditions where unrelated changes made by one thread can be accidentally overwritten by another thread that fetched its UserProfile object before the first thread wrote out its change.
214 Chapter 38. Code style and conventions Zulip Documentation, Release 1.7.0
38.4.5 Using raw saves to update important model objects
In most cases, we already have a function in zerver/lib/actions.py with a name like do_activate_user that will correctly handle lookups, caching, and notifying running browsers via the event system about your change. So please check whether such a function exists before writing new code to modify a model object, since your new code has a good chance of getting at least one of these things wrong.
38.4.6 Naive datetime objects
Python allows datetime objects to not have an associated timezone, which can cause time-related bugs that are hard to catch with a test suite, or bugs that only show up during daylight savings time. Good ways to make timezone-aware datetimes are below. We import timezone function as from django. utils.timezone import now as timezone_now and from django.utils.timezone import utc as timezone_utc. When Django is not available, timezone_utc should be replaced with pytz.utc below. • timezone_now() when Django is available, such as in zerver/. • datetime.now(tz=pytz.utc) when Django is not available, such as for bots and scripts. • datetime.fromtimestamp(timestamp, tz=timezone_utc) if creating a datetime from a times- tamp. This is also available as zerver.lib.timestamp.timestamp_to_datetime. • datetime.strptime(date_string, format).replace(tzinfo=timezone_utc) if creating a datetime from a formatted string that is in UTC. Idioms that result in timezone-naive datetimes, and should be avoided, are datetime.now() and datetime. fromtimestamp(timestamp) without a tz parameter, datetime.utcnow() and datetime. utcfromtimestamp(), and datetime.strptime(date_string, format) without replacing the tzinfo at the end. Additional notes: • Especially in scripts and puppet configuration where Django is not available, using time.time() to get timestamps can be cleaner than dealing with datetimes. • All datetimes on the backend should be in UTC, unless there is a good reason to do otherwise.
38.4.7 x.attr('zid') vs. rows.id(x)
Our message row DOM elements have a custom attribute zid which contains the numerical message ID. Don’t access this directly as x.attr('zid') ! The result will be a string and comparisons (e.g. with <=) will give the wrong result, occasionally, just enough to make a bug that’s impossible to track down. You should instead use the id function from the rows module, as in rows.id(x). This returns a number. Even in cases where you do want a string, use the id function, as it will simplify future code changes. In most contexts in JavaScript where a string is needed, you can pass a number without any explicit conversion.
38.4.8 JavaScript var
Always declare JavaScript variables using var. JavaScript has function scope only, not block scope. This means that a var declaration inside a for or if acts the same as a var declaration at the beginning of the surrounding function. To avoid confusion, declare all variables at the top of a function.
38.4. Dangerous constructs 215 Zulip Documentation, Release 1.7.0
38.4.9 JavaScript for (i in myArray)
Don’t use it: [1], [2], [3]
38.4.10 Translation tags
Remember to tag all user-facing strings for translation, whether they are in HTML templates or JavaScript editing the HTML (e.g. error messages).
38.4.11 State and logs files
When writing out state of log files, always declare the path in ZULIP_PATHS in zproject/settings.py, which will do the right thing in both development and production.
38.5 JS array/object manipulation
For generic functions that operate on arrays or JavaScript objects, you should generally use Underscore. We used to use jQuery’s utility functions, but the Underscore equivalents are more consistent, better-behaved and offer more choices. A quick conversion table:
$.each → _.each (parameters to the callback reversed) $.inArray → _.indexOf (parameters reversed) $.grep → _.filter $.map → _.map $.extend → _.extend
There’s a subtle difference in the case of _.extend; it will replace attributes with undefined, whereas jQuery won’t:
$.extend({foo: 2}, {foo: undefined}); // yields {foo: 2}, BUT... _.extend({foo: 2}, {foo: undefined}); // yields {foo: undefined}!
Also, _.each does not let you break out of the iteration early by returning false, the way jQuery’s version does. If you’re doing this, you probably want _.find, _.every, or _.any, rather than ’each’. Some Underscore functions have multiple names. You should always use the canonical name (given in large print in the Underscore documentation), with the exception of _.any, which we prefer over the less clear ’some’.
38.6 More arbitrary style things
38.6.1 Line length
We have an absolute hard limit on line length only for some files, but we should still avoid extremely long lines. A general guideline is: refactor stuff to get it under 85 characters, unless that makes the code a lot uglier, in which case it’s fine to go up to 120 or so.
216 Chapter 38. Code style and conventions Zulip Documentation, Release 1.7.0
38.6.2 JavaScript
When calling a function with an anonymous function as an argument, use this style:
$.get('foo', function (data) { var x = ...; // ... });
The inner function body is indented one level from the outer function call. The closing brace for the inner function and the closing parenthesis for the outer call are together on the same line. This style isn’t necessarily appropriate for calls with multiple anonymous functions or other arguments following them. Combine adjacent on-ready functions, if they are logically related. The best way to build complicated DOM elements is a Mustache template like static/templates/ message_reactions.handlebars. For simpler things you can use jQuery DOM building APIs like so: var new_tr = $('
Passing a HTML string to jQuery is fine for simple hardcoded things that don’t need internationalization: foo.append('
/
'); but avoid programmatically building complicated strings. We used to favor attaching behaviors in templates like so:but there are some reasons to prefer attaching events using jQuery code: • Potential huge performance gains by using delegated events where possible • When calling a function from an onclick attribute, this is not bound to the element like you might think • jQuery does event normalization Either way, avoid complicated JavaScript code inside HTML attributes; call a helper function instead.
38.6.3 HTML / CSS
Avoid using the style= attribute unless the styling is actually dynamic. Instead, define logical classes and put your styles in external CSS files such as zulip.css. Don’t use the tag name in a selector unless you have to. In other words, use .foo instead of span.foo. We shouldn’t have to care if the tag type changes in the future.
38.6.4 Python
• Don’t put a shebang line on a Python file unless it’s meaningful to run it as a script. (Some libraries can also be run as scripts, e.g. to run a test suite.) • Scripts should be executed directly (./script.py), so that the interpreter is implicitly found from the shebang line, rather than explicitly overridden (python script.py). • Put all imports together at the top of the file, absent a compelling reason to do otherwise. • Unpacking sequences doesn’t require list brackets:
38.6. More arbitrary style things 217 Zulip Documentation, Release 1.7.0
[x, y]= xs # unnecessary x, y= xs # better
• For string formatting, use x % (y,) rather than x % y, to avoid ambiguity if y happens to be a tuple.
38.6.5 Tests
All significant new features should come with tests. See testing.
38.6.6 Third party code
See our docs on dependencies for discussion of rules about integrating third-party projects.
218 Chapter 38. Code style and conventions CHAPTER 39
Python static type checker (mypy)
mypy is a compile-time static type checker for Python, allowing optional, gradual typing of Python code. Zulip is using mypy’s Python 2 compatible syntax for type annotations, which means that type annotations are written inside comments that start with # type:. Here’s a brief example of the mypy syntax we’re using in Zulip: user_dict={} # type: Dict[str, UserProfile] def get_user(email, realm): # type: (str, Realm) -> UserProfile ... # Actual code of the function here
You can learn more about it at: • The mypy cheat sheet is the best resource for quickly understanding how to write the PEP 484 type annotations used by mypy correctly. • The Python 2 type annotation syntax spec in PEP 484 • Using mypy with Python 2 code The mypy type checker is run automatically as part of Zulip’s Travis CI testing process in the backend build.
39.1 type_debug.py zerver/lib/type_debug.py has a useful decorator print_types. It prints the types of the parameters of the decorated function and the return type whenever that function is called. This can help find out what parameter types a function is supposed to accept, or if parameters with the wrong types are being passed to a function. Here is an example using the interactive console:
>>> from zerver.lib.type_debug import print_types >>> >>> @print_types ... def func(x, y): ... return x+y
219 Zulip Documentation, Release 1.7.0
... >>> func(1.0,2) func(float, int) -> float 3.0 >>> func('a','b') func(str, str) -> str 'ab' >>> func((1,2), (3,)) func((int, int), (int,)) -> (int, int, int) (1, 2, 3) >>> func([1,2,3], [4,5,6,7]) func([int, ...], [int, ...]) -> [int, ...] [1, 2, 3, 4, 5, 6, 7] print_all prints the type of the first item of lists. So [int, ...] represents a list whose first element’s type is int. Types of all items are not printed because a list can have many elements, which would make the output too large. Similarly in dicts, one key’s type and the corresponding value’s type are printed. So {1: 'a', 2: 'b', 3: 'c'} will be printed as {int: str, ...}.
39.2 Zulip goals
Zulip is hoping to reach 100% of the codebase annotated with mypy static types, and then enforce that it stays that way. Our current coverage is shown in Codecov.
39.3 Installing mypy
If you installed Zulip’s development environment correctly, mypy should already be installed inside the Python 3 virtualenv at zulip-py3-venv (mypy only supports Python 3). If you’d like to install just the version of mypy that we’re using (useful if e.g. you want mypy installed on your laptop outside the Vagrant guest), you can do that with pip install -r requirements/mypy.txt.
39.4 Running mypy on Zulip’s code locally
To run mypy on Zulip’s python code, run the command: tools/run-mypy
It will output errors in the same style as a compiler would. For example, if your code has a type error like this: foo=1 foo='1' you’ll get an error like this: test.py: note: In function"test": test.py:200: error: Incompatible types in assignment (expression has type"str",
˓→variable has type"int")
220 Chapter 39. Python static type checker (mypy) Zulip Documentation, Release 1.7.0
If you need help interpreting or debugging mypy errors, please feel free to mention @sharmaeklavya2 or @timabbott on your pull request (or ask in chat.zulip.org) to get help; we’d love to both build a great troubleshooting guide in this doc and also help contribute improvements to error messages upstream. Since mypy is a new tool under rapid development and occasionally makes breaking changes, Zulip is using a pinned version of mypy from its git repository rather than tracking the (older) latest mypy release on PyPI.
39.5 Excluded files
Since several Python files in Zulip’s code don’t pass mypy’s checks (even for unannotated code) right now, a list of files to be excluded from the check for CI is present in tools/run-mypy. To run mypy on all Python files, ignoring the exclude list, you can pass the --all option to tools/run-mypy. tools/run-mypy--all
If you type annotate some of those files so that they pass without errors, please remove them from the exclude list.
39.6 Mypy is there to find bugs in Zulip before they impact users
For the purposes of Zulip development, you can treat mypy like a much more powerful linter that can catch a wide range of bugs. If, after running tools/run-mypy on your Zulip branch, you get mypy errors, it’s important to get to the bottom of the issue, not just do something quick to silence the warnings. Possible explanations include: • A bug in any new type annotations you added. • A bug in the existing type annotations. • A bug in Zulip! • Some Zulip code is correct but confusingly reuses variables with different types. • A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project). Each explanation has its own solution, but in every case the result should be solving the mypy warning in a way that makes the Zulip codebase better. If you need help understanding an issue, please feel free to mention @shar- maeklavya2 or @timabbott on the relevant pull request or issue on GitHub. If you think you have found a bug in Zulip or mypy, inform the zulip developers by opening an issue on Zulip’s GitHub repository or posting on zulip-devel. If it’s indeed a mypy bug, we can help with reporting it upstream.
39.7 Annotating strings
In Python 3, strings can have non-ASCII characters without any problems. Such characters are required to support languages which use non-latin scripts like Japanese and Hindi. They are also needed to support special characters like mathematical symbols, musical symbols, etc. In Python 2, however, str generally doesn’t work well with non-ASCII characters. That’s why unicode was introduced in Python 2. But there are problems with the unicode and str system. Implicit conversions between str and unicode use the ascii codec, which fails on strings containing non-ASCII characters. Such errors are hard to detect by people who always write in English. To minimize such implicit conversions, we should have a strict separation between str and unicode in Python 2. It might seem that using unicode everywhere will solve all problems, but unfortunately it doesn’t. This is because some parts of the standard library and the Python language (like keyword argument unpacking) insist that parameters passed to them are str.
39.5. Excluded files 221 Zulip Documentation, Release 1.7.0
To make our code work correctly in Python 2, we have to identify strings which contain data which could come from non-ASCII sources like stream names, people’s names, domain names, content of messages, emails, etc. These strings should be unicode. We also have to identify strings which should be str like Exception names, attribute names, parameter names, etc. Mypy can help with this. We just have to annotate each string as either str or unicode and mypy’s static type checking will tell us if we are incorrectly mixing the two. However, unicode is not defined in Python 3. We want our code to be Python 3 compatible in the future. This can be achieved using ’typing.Text’, a Python 2 and 3 compatibility type. typing.Text is defined as str in Python 3 and as unicode in Python 2. We’ll be using Text (instead of unicode) and str to annotate strings in Zulip’s code. We follow the style of doing from typing import Text and using Text for annotation instead of doing import typing and using typing.Text for annotation, because Text is used so extensively for type annotations that we don’t need to be that verbose. Sometimes you’ll find that you have to convert strings from one type to another. zerver/lib/str_utils.py has utility functions to help with that. It also has documentation (in docstrings) which explains the right way to use them.
222 Chapter 39. Python static type checker (mypy) CHAPTER 40
Reviewing Zulip code
Code review is a key part of how Zulip does development! If you’ve been contributing to Zulip’s code, we’d love for you to do reviews. This is a guide to how. (With some thoughts for writing code too.)
40.1 Principles of code review
40.1.1 Anyone can review
Anyone can do a code review – you don’t have to have a ton of experience, and you don’t have to have the power to ultimately merge the PR. If you • read the code, see if you understand what the change is doing and why, and ask questions if you don’t; or • fetch the code (for Zulip server code, tools/fetch-rebase-pull-request is super handy), play around with it in your dev environment, and say what you think about how the feature works those are really helpful contributions.
40.1.2 Please do reviews
Doing code reviews is an important part of making the project go. It’s also an important skill to develop for partici- pating in open-source projects and working in the industry in general. If you’re contributing to Zulip and have been working in our code for a little while, we would love for some of your time contributing to come in the form of doing code reviews! For students participating in Google Summer of Code or a similar program, we expect you to spend a chunk of your time each week (after the first couple of weeks as you’re getting going) doing code reviews.
40.1.3 Fast replies are key
For the author of a PR, getting feedback quickly is really important for making progress quickly and staying productive. That means that if you get @-mentioned on a PR with a request for you to review it, it helps the author a lot if you
223 Zulip Documentation, Release 1.7.0 reply promptly. A reply doesn’t even have to be a full review; if a PR is big or if you’re pressed for time, then just getting some kind of reply in quickly – initial thoughts, feedback on the general direction, or just saying you’re busy and when you’ll have time to look harder – is still really valuable for the author and for anyone else who might review the PR. People in the Zulip project live and work in many timezones, and code reviewers also need focused chunks of time to write code and do other things, so an immediate reply isn’t always possible. But a good benchmark is to try to always reply within one workday, at least with a short initial reply, if you’re working regularly on Zulip. And sooner is better.
40.1.4 Protocol for authors
When you send a PR, try to think of a good person to review it – outside of the handful of people who do a ton of reviews – and @-mention them with something like "@person, would you review this?". Good choices include • someone based in your timezone or a nearby timezone • people working on similar things, or in a loosely related area
40.2 Things to look for
• The Travis CI build. The tests need to pass. One can investigate any failures and figure out what to fix by clicking on a red X next to the commit hash or the Detail links on a pull request. (Example: in #1219, click the red X next to f1f474e to see the build jobs for that commit, at least one of which has failed. Click on the link for Travis continuous integrations details to see the tests Travis ran on that commit, at least one of which failed, and go to one of the failing tests to see the error.) • Technical design. There are a lot of considerations here: security, migration paths/backwards compatibility, cost of new dependencies, interactions with features, speed of performance, API changes. Security is especially important and worth thinking about carefully with any changes to security-sensitive code like views. • User interface and visual design. If frontend changes are involved, the reviewer will check out the code, play with the new UI, and verify it for both quality and consistency with the rest of the Zulip UI. We highly encourage posting screenshots to save reviewers time in getting a feel for what the feature looks like – you’ll get a quicker response that way. • Error handling. The code should always check for invalid user input. User-facing error messages should be clear and when possible be actionable (it should be obvious to the user what they need to do in order to correct the problem). • Testing. The tests should validate that the feature works correctly, and specifically test for common error con- ditions, bad user input, and potential bugs that are likely for the type of change being made. Tests that exclude whole classes of potential bugs are preferred when possible (e.g., the common test suite test_bugdown. py between the Zulip server’s frontend and backend Markdown processors, or the GetEventsTest test for buggy race condition handling). • Translation. Make sure that the strings are marked for translation. • Clear function, argument, variable, and test names. Every new piece of Zulip code will be read many times by other developers, and future developers will grep for relevant terms when researching a problem, so it’s important that variable names communicate clearly the purpose of each piece of the codebase. • Duplicated code. Code duplication is a huge source of bugs in large projects and makes the codebase difficult to understand, so we avoid significant code duplication wherever possible. Sometimes avoiding code duplication involves some refactoring of existing code; if so, that should usually be done as its own series of commits (not squashed into other changes or left as a thing to do later). That series of commits can be in the same pull request
224 Chapter 40. Reviewing Zulip code Zulip Documentation, Release 1.7.0
as the feature that they support, and we recommend ordering the history of commits so that the refactoring comes before the feature. That way, it’s easy to merge the refactoring (and minimize risk of merge conflicts) if there are still user experience issues under discussion for the feature itself. • Completeness. For refactorings, verify that the changes are complete. Usually one can check that efficiently using git grep, and it’s worth it, as we very frequently find issues by doing so. • Documentation updates. If this changes how something works, does it update the documentation in a corre- sponding way? If it’s a new feature, is it documented, and documented in the right place? • Good comments. It’s often worth thinking about whether explanation in a commit message or pull request discussion should be included in a comment, /docs, or other documentation. But it’s better yet if verbose explanation isn’t needed. We prefer writing code that is readable without explanation over a heavily commented codebase using lots of clever tricks. • Coding style. See the Zulip code-style documentation for details. Our goal is to have as much of this as possible verified via the linters and tests, but there’s always going to be unusual forms of Python/JavaScript style that our tools don’t check for. • Clear commit messages. See the Zulip version control documentation for details on what we look for.
40.2.1 Zulip server
Some points specific to the Zulip server codebase: • Testing – Backend. We are trying to maintain ~100% test coverage on the backend, so backend changes should have negative tests for the various error conditions. • Testing – Frontend. If the feature involves frontend changes, there should be frontend tests. See the test writing documentation for more details. • mypy annotations. New functions should be annotated using mypy and existing annotations should be updated. Use of Any, ignore, and unparameterized containser should be limited to cases where a more precise type cannot be specified.
40.3 Tooling
To make it easier to review pull requests, if you’re working in the Zulip server codebase, use our git tool tools/ fetch-rebase-pull-request to check out a pull request locally and rebase it against master. If a pull request just needs a little fixing to make it mergeable, feel free to do that in a new commit, then push your branch to GitHub and mention the branch in a comment on the pull request. That’ll save the maintainer time and get the PR merged quicker.
40.4 Additional Resources
We also strongly recommend reviewers to go through the following resources. • The Gentle Art of Patch Review article by Sarah Sharp • Zulip & Good Code Review article by Sumana Harihareswara • Code Review - A consolidation of advice and stuff from the sinternet article by James J. Porter • Zulip Code of Conduct
40.3. Tooling 225 Zulip Documentation, Release 1.7.0
226 Chapter 40. Reviewing Zulip code CHAPTER 41
The chat.zulip.org community
chat.zulip.org is the primary communication forum for the Zulip community. It is a Zulip server that you can connect to from any modern web browser. You can go through the simple signup process at that link, and then you will soon be talking to core Zulip developers and other users. To get help in real time, you will have the best luck finding core developers roughly between 17:00 UTC and 2:00 UTC or during office hours and sprints, but the sun never sleeps on the Zulip community. Most questions get a reply within minutes to a few hours, depending on the time of day.
41.1 This is a bleeding edge development server
The chat.zulip.org server is frequently deployed off of master from the Zulip Git repository, so please point out anything you notice that seems wrong! We catch many bugs that escape code review this way. The chat.zulip.org server is a development and testing server, not a production service, so don’t use it for anything mission-critical, secret/embarrassing, etc.
41.2 Community conventions
• Send any test messages to #test here or as a PM to yourself to avoid disrupting others. • When asking for help, provide the details needed for others to help you. E.g. include the full traceback in a code block (not a screenshot), a link to the code or a WIP PR you’re having trouble debugging, etc. • Ask questions on streams rather than PMing core contributors. You’ll get answers faster since other people can help, and it makes it possible for other developers to learn from reading the discussion. • Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it’s usually easy to see which message you’re replying to, so you don’t need to mention your conversation partner in every reply. Mentioning other users is great for timely questions or making sure someone who is not online sees your message. • Converse informally; there’s no need to use titles like "Sir" or "Madam".
227 Zulip Documentation, Release 1.7.0
• Use gender-neutral language. • Follow the community code of conduct. • Participate! Zulip is a friendly and welcoming community, and we love meeting new people, hearing about what brought them to Zulip, and getting their feedback. If you’re not sure where to start, introduce yourself and your interests in #new members, using your name as the topic.
41.3 High traffic community
The chat.zulip.org community sends several thousand messages every single week, about half of them to streams that we have included in the default streams for new users for discoverability. Keeping up with everything happening in the Zulip project is both difficult and rarely a useful goal. To make the best use of your time, we highly recommend that you unsubscribe from streams that you aren’t interested in, and mute streams that are only of occasional interest.
41.4 Streams
There are a few streams worth highlighting that are relevant for everyone, even non-developers: • #announce is for announcements and discussions thereof; we try to keep traffic there to a minimum. • #feedback is for posting feedback on Zulip. • #design is where we discuss the UI design and collect feedback on potential design changes. We love feedback, so don’t hesitate to speak up! • #user community is for Zulip users to discuss their experiences using and adopting Zulip. • #production help is for production environment related discussions. • #test here is for sending test messages without inconveniencing other users :). We recommend muting this stream when not using it. There are dozens of streams for development discussions in the Zulip community (e.g. one for each app, etc.); check out the Streams page to see the descriptions for all of them. Relevant to almost everyone are these: • #checkins is for progress updates on what you’re working on and its status; usually folks post with their name as the topic. Everyone is welcome to participate! • #development help is for asking for help with any Zulip server/webapp development work (use the app streams for help working on one of the apps). • #code review is for getting feedback on your work. We encourage all developers to comment on work posted here, even if you’re new to the Zulip project; reviewing other PRs is a great way to develop experience, and even just manually testing a proposed new feature and posting feedback is super helpful. • #documentation is where we discuss improving Zulip’s user, sysadmin, and developer documentation. • #translation is for discussing Zulip’s translations. • #learning is for posting great learning resources one comes across.
228 Chapter 41. The chat.zulip.org community Zulip Documentation, Release 1.7.0
41.5 Chat meetings
We have regular chat meetings on Zulip to coordinate work on various parts of the Zulip project. While most developer discussions happen asynchonrously, these meetings are used mainly to coordinate work within a major area of Zulip. These meetings are usually scheduled in Pacific time mornings, since that seems to be the best time for our global contributor base (the part of the world where it’s the deep middle of the night is the Pacific Ocean). Anyone is welcome to attend and contribute to the discussions in these meetings, and they’re a great time to stop by and introduce yourself if you’d like to get involved (though really, any time is, so). Here are the regular meetings that exist today along with their usual times (actual times are listed in the linked agenda documents): • Mobile team on #mobile, generally Wednesdays at 10AM Pacific time. Agendas. • Backend/infrastructure team on #backend, generally Fridays at 10AM Pacific time. Agendas. • Bots and integrations team on #integrations, generally Fridays at 9AM Pacific time. Agendas.
41.5.1 Office hours and sprints
We also do project-wide ad-hoc "office hours" and remote sprints irregularly, about once a month. Anyone can schedule one: announce it in #announce and on the zulip-devel mailing list a few days ahead of time, and ideally, tell Sumana so she can put it on the public Zulip meetings calendar. Office hours are simply times for us to informally discuss current global project priorities, find out what questions people have, and so on. We set them up so people know there’ll be more people around at a particular time to chat. You don’t need to RSVP and you don’t need to show up on time or stop conversations when the "hour" stops. They start in #general and conversations move into other streams and topics as they come up. Sprints are times when Zulip developers get together in chat, and sometimes in person, to work on related issues at the same time.
41.5. Chat meetings 229 Zulip Documentation, Release 1.7.0
230 Chapter 41. The chat.zulip.org community CHAPTER 42
Using zulipbot
Zulip uses @zulipbot, a GitHub workflow bot deployed on all Zulip repositories, to handle issues and pull requests in our repositories in order to create a better workflow for Zulip contributors. Its purpose is to work around various limitations in GitHub’s permissions and notifications systems to make it possible to have a much more democractic workflow for our contributors. It allows anyone to self-assign or label an issue, not just the core contributors trusted with full write access to the repository (which is the only model GitHub supports).
42.1 Usage
• Claim an issue — Comment @zulipbot claim on the issue you want to claim; @zulipbot will assign you to the issue and label the issue as in progress. – If you’re a new contributor, @zulipbot will give you read-only collaborator access to the repository and leave a welcome message on the issue you claimed. – You can also claim an issue that you’ve opened by including @zulipbot claim in the body of your issue. – If you accidentally claim an issue you didn’t want to claim, comment @zulipbot abandon to abandon an issue. • Label your issues — Add appropriate labels to issues that you opened by including @zulipbot label in an issue comment or the body of your issue followed by the desired labels enclosed within double quotes (""). – For example, to add the bug and help wanted labels to your issue, comment or include @zulipbot label "bug" "help wanted" in the issue body. – You’ll receive an error message if you try to add any labels to your issue that don’t exist in your repository. – If you accidentally added the wrong labels, you can remove them by commenting @zulipbot remove followed by the desired labels enclosed with double quotes (""). • Find unclaimed issues — Use the GitHub search feature to find unclaimed issues by adding one of the following filters to your search:
231 Zulip Documentation, Release 1.7.0
– -label: "in progress" (excludes issues labeled with the in progress label) – no:assignee (shows issues without assignees) Issues labeled with the in progress label and/or assigned to other users have already been claimed. • Collaborate in area label teams — Receive notifications on issues and pull requests within your fields of expertise on the Zulip server repository by joining the Zulip server area label teams (Note: this link only works for members of the Zulip organization; we’ll happily add you if you’re interested). These teams correspond to the repository’s area labels, although some teams are associated with multiple labels; for example, the area: message-editing and area: message view labels are both related to the Server message view team. Feel free to join as many area label teams as as you’d like! After your request to join an area label team is approved, you’ll receive notifications for any issues labeled with the team’s corresponding area label as well as any pull requests that reference issues labeled with your team’s area label. • Track inactive claimed issues — If a claimed issue has not been updated for a week, @zulipbot will post a comment on the inactive issue to ask the assignee(s) if they are still working on the issue. If you see this comment on an issue you claimed, you should post a comment on the issue to notify @zulipbot that you’re still working on it. If @zulipbot does not receive a response from the assignee within 3 days of an inactive issue prompt, @zulipbot will automatically remove the issue’s current assignee(s) and the "in progress" label to allow others to work on an inactive issue. • Receive Travis build status notifications — If you would like to receive a notification whenever the build status of your pull request is updated, label your pull request with the "travis updates" label using the command @zulipbot label "travis updates", and @zulipbot will let you know the build status (e.g. passed, failed, errored) of your pull request once all tests finish.
42.1.1 Contributing
If you wish to help develop and contribute to @zulipbot, check out the zulip/zulipbot repository on GitHub and read the project’s contributing guidelines for more information.
232 Chapter 42. Using zulipbot CHAPTER 43
Accessibility
43.1 Guidelines
In order to accommodate all users, Zulip strives to implement accessibility best practices in its user interface. There are many aspects to accessibility; here are some of the more important ones to keep in mind. • All images should have alternative text attributes for the benefit of users who cannot see them (this includes users who are utilizing a voice interface to free up their eyes to look at something else instead). • The entire application should be usable via a keyboard (many users are unable to use a mouse, and many accessibility aids emulate a keyboard). • Text should have good enough contrast against the background to enable even users with moderate visual im- pairment to be able to read it. • ARIA (Accessible Rich Internet Application) attributes should be used appropriately to enable screen readers and other alternative interfaces to navigate the application effectively. There are many different standards for accessibility, but the most relevant one for Zulip is the W3C’s WCAG (Web Content Accessibility Guidelines), currently at version 2.0. Whenever practical, we should strive for compliance with the AA level of this specification. (The W3C itself recommends not trying to comply with the highest AAA level for an entire web site or application, as it is not possible for some content.)
43.2 Tools
There are tools available to automatically audit a web page for compliance with many of the WCAG guidelines. Here are some of the more useful ones: • Accessibility Developer Tools This open source Chrome extension from Google adds an accessibility audit to the "Audits" tab of the Chrome Developer Tools. The audit is performed against the page’s DOM via JavaScript, allowing it to identify some issues that a static HTML inspector would miss. • aXe An open source Chrome and Firefox extension which runs a somewhat different set of checks than Google’s Chrome extension.
233 Zulip Documentation, Release 1.7.0
• Wave This web application takes a URL and loads it in a frame, reporting on all the issues it finds with links to more information. Has the advantage of not requiring installation, but requires a URL which can be directly accessed by an external site. • Web Developer This browser extension has many useful features, including a convenient link for opening the current URL in Wave to get an accessibility report. Note that these tools cannot catch all possible accessibility problems, and sometimes report false positives as well. They are a useful aid in quickly identifying potential problems and checking for regressions, but their recommendations should not be blindly obeyed.
43.3 GitHub Issues
Problems with Zulip’s accessibility should be reported as GitHub issues with the "accessibility" label. This label can be added by entering the following text in a separate comment on the issue:
@zulipbot label"accessibility"
If you want to help make Zulip more accessible, here is a list of the currently open accessibility issues.
43.4 Additional Resources
For more information about making Zulip accessible to as many users as possible, the following resources may be useful. • Font Awesome accessibility guide, which is especially helpful since Zulip uses Font Awesome for its icons. • Web Content Accessibility Guidelines (WCAG) 2.0 • WAI-ARIA - Web Accessibility Initiative Accessible Rich Internet Application Suite • WebAIM - Web Accessibility in Mind • The MDN page on accessibility • The Open edX Accessibility Guidelines for developers
234 Chapter 43. Accessibility CHAPTER 44
Bug report guidelines
Please include these elements in your bug report to make it easier for us to help you. • A brief title • An explanation of what you were expecting vs. the actual result • Steps to take in order to reproduce the buggy behavior • Whether you are using Zulip in production or in the development environment, and whether these are old versions • Whether you are using the web app, a desktop app or a mobile device to access Zulip • Any additional information that would help: screenshots, GIFs, a pastebin of the error log Further reading: • How to write a bug report that will make your engineers love you • How to Report Bugs Effectively
235 Zulip Documentation, Release 1.7.0
236 Chapter 44. Bug report guidelines CHAPTER 45
Testing and writing tests
45.1 Overview
Zulip has a full test suite that includes many components. The most important components are documented in depth in their own sections: • Django: backend Python tests • Casper: end-to-end UI tests • Node: unit tests for JS front end code • Linters: Our parallel linter suite • Travis CI details: How all of these run in Travis CI This document covers more general testing issues, such as how to run the entire test suite, how to troubleshoot database issues, how to manually test the front end, and how to plan for the future upgrade to Python3. We also document how to manually test the app.
45.2 Running tests
Zulip tests must be run inside a Zulip development environment; if you’re using Vagrant, you will need to enter the Vagrant environment before running the tests: vagrant ssh cd/srv/zulip
Then, to run the full Zulip test suite, do this:
./tools/test-all
237 Zulip Documentation, Release 1.7.0
This runs the linter (tools/lint) plus all of our test suites; they can all be run separately (just read tools/ test-all to see them). You can also run individual tests which can save you a lot of time debugging a test failure, e.g.:
./tools/lint # Runs all the linters in parallel ./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube ./tools/test-backend BugdownTest # Run `test-backend --help` for more options ./tools/test-js-with-casper 09-navigation.js ./tools/test-js-with-node utils.js
The above setup instructions include the first-time setup of test databases, but you may need to rebuild the test database occasionally if you’re working on new database migrations. To do this, run:
./tools/do-destroy-rebuild-test-database
45.2.1 Possible testing issues
• When running the test suite, if you get an error like this:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_
˓→array(unknown, text, tsquery) does not exist LINE2:...ECT message_id, flags, subject, rendered_content, ts_match_l... ^
. . . then you need to install tsearch-extras, described above. Afterwards, re-run the init*-db and the do-destroy-rebuild*-database scripts. • When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run id username on the host where username is the user running Vagrant and the output is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder permissions.
45.2.2 Internet access inside test suites
As a policy matter, the Zulip test suites should never make outgoing HTTP or other network requests. This is important for 2 major reasons: • Tests that make outgoing Internet requests will fail when the user isn’t on the Internet. • Tests that make outgoing Internet requests often have a hidden dependency on the uptime of a third-party service, and will fail nondeterministically if that service has a temporary outage. Nondeterministically failing tests can be a big waste of developer time, and we try to avoid them wherever possible. As a result, Zulip’s major test suites should never access the Internet directly. Since code in Zulip does need to access the Internet (e.g. to access various third-party APIs), this means that the Zulip tests use mocking to basically hardcode (for the purposes of the test) what responses should be used for any outgoing Internet requests that Zulip would make in the code path being tested. This is easy to do using test fixtures (a fancy word for fixed data used in tests) and the mock.patch function to specify what HTTP response should be used by the tests for every outgoing HTTP (or other network) request. Consult our guide on mocking to learn how to mock network requests easily; there are also a number of examples throughout the codebase.
238 Chapter 45. Testing and writing tests Zulip Documentation, Release 1.7.0
We partially enforce this policy in the main Django/backend test suite by overriding certain library functions that are used in outgoing HTTP code paths (httplib2.Http().request, requests.request, etc.) to throw an exception in the backend tests. While this is enforcement is not complete (there a lot of other ways to use the Internet from Python), it is easy to do and catches most common cases of new code dependning on Internet access. This enforcement code results in the following exception:
File"tools/test-backend", line 120, in internet_guard raise Exception("Outgoing network requests are not allowed in the Zulip tests." Exception: Outgoing network requests are not allowed in the Zulip tests. ...
Documentation tests
The one exception to this policy is our documentation tests, which will attempt to verify that the links included in our documentation aren’t broken. Those tests end up failing nondeterministically fairly often, which is unfortunate, but there’s simply no other correct way to verify links other than attempting to access them.
45.3 Schema and initial data changes
If you change the database schema or change the initial test data, you have to regenerate the pristine test database by running tools/do-destroy-rebuild-test-database.
45.4 Wiping the test databases
You should first try running: tools/do-destroy-rebuild-test-database If that fails you should try to do: sudo-u postgres psql > DROP DATABASE zulip_test; > DROP DATABASE zulip_test_template; and then run tools/do-destroy-rebuild-test-database
45.4.1 Recreating the postgres cluster
warning This is irreversible, so do it with care, and never do this anywhere in production. If your postgres cluster (collection of databases) gets totally trashed permissions-wise, and you can’t otherwise repair it, you can recreate it. On Ubuntu: sudo pg_dropcluster--stop 9.1 main sudo pg_createcluster--locale=en_US.utf8--start 9.1 main
45.3. Schema and initial data changes 239 Zulip Documentation, Release 1.7.0
45.5 Local browser testing (local app + web browser)
This section is about troubleshooting your local development environment. There is a separate manual testing doc that enumerates things you can test as part of manual QA.
45.5.1 Clearing the development database
You can use:
./tools/do-destroy-rebuild-database to drop the database on your development environment and repopulate your it with the Shakespeare characters and some test messages between them. This is run automatically as part of the development environment setup process, but is occasionally useful when you want to return to a clean state for testing.
45.5.2 JavaScript manual testing debug.js has some tools for profiling JavaScript code, including: • ‘print_elapsed_time‘: Wrap a function with it to print the time that function takes to the JavaScript console. • ‘IterationProfiler‘: Profile part of looping constructs (like a for loop or $.each). You mark sections of the iteration body and the IterationProfiler will sum the costs of those sections over all iterations. Chrome has a very good debugger and inspector in its developer tools. Firebug for Firefox is also pretty good. They both have profilers, but Chrome’s is a sampling profiler while Firebug’s is an instrumenting profiler. Using them both can be helpful because they provide different information.
240 Chapter 45. Testing and writing tests CHAPTER 46
Linters
46.1 Overview
Zulip does extensive linting of much of its source code, including Python/JavaScript files, HTML templates (Django/handlebars), CSS files, JSON fixtures, Markdown documents, puppet manifests, and shell scripts. For some files we simply check for small things like trailing whitespace, but for other files, we are quite thorough about checking semantic correctness. Obviously, a large reason for linting code is to enforce the Zulip coding standards. But we also use the linters to prevent common coding errors. We borrow some open source tools for much of our linting, and the links below will direct you to the official docu- mentation for these projects. • eslint • mypy • puppet (puppet provides its own mechanism for validating manifests) • pyflakes Zulip also uses some home-grown code to perform tasks like validating indentation in template files, enforcing coding standards that are unique to Zulip, allowing certain errors from third party linters to pass through, and exempting legacy files from lint checks.
46.2 Running the linters
If you run ./tools/test-all, it will automatically run the linters (with one small exception: it does not run mypy against scripts). You can also run them individually:
241 Zulip Documentation, Release 1.7.0
./tools/lint ./tools/run-mypy ./tools/run-mypy--scripts-only
Finally, you can rely on our Travis CI setup to run linters for you, but it is good practice to run lint checks locally. Note: The linters only check files that git tracks. Remember to git add new files before running lint checks. Our linting tools generally support the ability to lint files individually–with some caveats–and those options will be described later in this document. We may eventually bundle run-mypy into lint, but mypy is pretty resource intensive compared to the rest of the linters, because it does static code analysis. So we keep mypy separate to allow folks to quickly run the other lint checks.
46.3 General considerations
Once you have read the Zulip coding guidelines, you can be pretty confident that 99% of the code that you write will pass through the linters fine, as long as you are thorough about keeping your code clean. And, of course, for minor oversights, lint is your friend, not your foe. Occasionally, our linters will complain about things that are more of an artifact of the linter limitations than any actual problem with your code. There is usually a mechanism where you can bypass the linter in extreme cases, but often it can be a simple matter of writing your code in a slightly different style to appease the linter. If you have problems getting something to lint, you can submit an unfinished PR and ask the reviewer to help you work through the lint problem, or you can find other people in the Zulip Community to help you. Also, bear in mind that 100% of the lint code is open source, so if you find limitations in either the Zulip home-grown stuff or our third party tools, feedback will be highly appreciated. Finally, one way to clean up your code is to thoroughly exercise it with tests. The Zulip test documentation describes our test system in detail.
46.4 Lint checks
Most of our lint checks get performed by ./tools/lint. These include the following checks: • Check Python code with pyflakes. • Check JavaScript code with eslint. • Check Python code for custom Zulip rules. • Check non-Python code for custom Zulip rules. • Check puppet manifests with the puppet validator. • Check HTML templates for matching tags and indentations. • Check CSS for parsability. • Check JavaScript code for addClass calls. The remaining lint checks occur in ./tools/run-mypy. It is probably somewhat of an understatement to call "mypy" a "linter," as it performs static code analysis of Python type annotations throughout our Python codebase. Our documentation on using mypy covers mypy in more detail. The rest of this document pertains to the checks that occur in ./tools/lint.
242 Chapter 46. Linters Zulip Documentation, Release 1.7.0
46.5 lint
Zulip has a script called lint that lives in our "tools" directory. It is the workhorse of our linting system, although in some cases it dispatches the heavy lifting to other components such as pyflakes, eslint, and other home grown tools. You can find the source code here. In order for our entire lint suite to run in a timely fashion, the lint script performs several lint checks in parallel by forking out subprocesses. This mechanism is still evolving, but you can look at the method run_parallel to get the gist of how it works. Note that our project does custom regex-based checks on the code, and we also customize how we call pyflakes and pycodestyle (pep8). The code for these types of checks mostly lives here.
46.5.1 Special options
You can use the -h option for lint to see its usage. One particular flag to take note of is the --modified flag, which enables you to only run lint checks against files that are modified in your git repo. Most of the "sub-linters" respect this flag, but some will continue to process all the files. Generally, a good workflow is to run with --modified when you are iterating on the code, and then run without that option right before commiting new code. If you need to troubleshoot the linters, there is a --verbose option that can give you clues about which linters may be running slow, for example.
46.5.2 Lint checks
The next part of this document describes the lint checks that we apply to various file types.
Generic source code checks
We check almost our entire codebase for trailing whitespace. Also, we disallow tab (\t) characters in all but two files. We also have custom regex-based checks that apply to specific file types. For relatively minor files like Markdown files and JSON fixtures, this is the extent of our checking. Finally, we’re checking line length in Python code (and hope to extend this to other parts of the codebase soon). You can use #ignorelinelength for special cases where a very long line makes sense (e.g. a link in a comment to an extremely long URL).
Python code
The bulk of our Python linting gets outsourced to the "pyflakes" tool. We call "pyflakes" in a fairly vanilla fashion, and then we post-process its output to exclude certain types of errors that Zulip is comfortable ignoring. (One notable class of error that Zulip currently tolerates is unused imports–because of the way mypy type annotations work in Python 2, it would be inconvenient to enforce this too strictly.) Zulip also has custom regex-based rules that it applies to Python code. Look for python_rules in the source code for lint. Note that we provide a mechanism to exclude certain lines of codes from these checks. Often, it is simply the case that our regex approach is too crude to correctly exonerate certain valid constructs. In other cases, the code that we exempt may be deemed not worthwhile to fix.
46.5. lint 243 Zulip Documentation, Release 1.7.0
JavaScript code
We check our JavaScript code in a few different ways: • We run eslint. • We perform custom Zulip regex checks on the code. • We verify that all addClass calls, with a few exceptions, explicitly contain a CSS class. The last check happens via a call to ./tools/find-add-class. This particular check is a work in progress, as we are trying to evolve a more rigorous system for weeding out legacy CSS styles, and the ability to quickly introspect our JS code for addClass calls is part of our vision.
Puppet manifests
We use Puppet as our tool to manage configuration files, using puppet "manifests." To lint puppet manifests, we use the "parser validate" option of puppet.
HTML Templates
Zulip uses two HTML templating systems: • Django templates • handlebars Zulip has an internal tool that validates both types of templates for correct indentation and matching tags. You can find the code here: • driver: check-templates • engine: lib/template_parser.py We exempt some legacy files from indentation checks, but we are hoping to clean those files up eventually.
CSS
Zulip does not currently lint its CSS for any kind of semantic correctness, but that is definitely a goal moving forward. We do ensure that our home-grown CSS parser can at least parse the CSS code. This is a slightly more strict check than checking that the CSS is compliant to the official spec, as our parser will choke on unusual constructs that we probably want to avoid in our code, anyway. (When the parser chokes, the lint check will fail.) You can find the code here: • driver: check-css • engine: lib/css_parser.py
Markdown, shell scripts, JSON fixtures
We mostly validate miscellaneous source files like .sh, .json, and .md files for whitespace issues.
46.6 Philosophy
If you want to help improve Zulip’s system for linting, here are some considerations.
244 Chapter 46. Linters Zulip Documentation, Release 1.7.0
46.6.1 Speed
We want our linters to be fast enough that most developers will feel comfortable running them in a pre-commit hook, so we run our linters in parallel and support incremental checks.
46.6.2 Accuracy
We try to catch as many common mistakes as possible, either via a linter or an automated test.
46.6.3 Completeness
Our goal is to have most common style issues by caught by the linters, so new contributors to the codebase can efficiently fix produce code with correct style without needing to go back-and-forth with a reviewer.
46.6. Philosophy 245 Zulip Documentation, Release 1.7.0
246 Chapter 46. Linters CHAPTER 47
Backend Django tests
47.1 Overview
Zulip uses the Django framework for its Python back end. We use the testing framework from django.test to test our code. We have over a thousand automated tests that verify that our backend works as expected. All changes to the Zulip backend code should be supported by tests. We enforce our testing culture during code review, and we also use coverage tools to measure how well we test our code. We mostly use tests to prevent regressions in our code, but the tests can have ancillary benefits such as documenting interfaces and influencing the design of our software. If you have worked on other Django projects that use unit testing, you will probably find familiar patterns in Zulip’s code. This document describes how to write tests for the Zulip back end, with a particular emphasis on areas where we have either wrapped Django’s test framework or just done things that are kind of unique in Zulip.
47.2 Running tests
Our tests live in zerver/tests/. You can run them with ./tools/test-backend. The tests run in parallel using multiple threads in your development environment, and can finish in under 30s on a fast machine. When you are in iterative mode, you can run individual tests or individual modules, following the dotted.test.name convention below:
cd/srv/zulip ./tools/test-backend zerver.tests.test_queue_worker.WorkerTest
There are many command line options for running Zulip tests, such as a --verbose option. The best way to learn the options is to use the online help:
./tools/test-backend-h
We also have ways to instrument our tests for finding code coverage, URL coverage, and slow tests. Use the -h option to discover these features. We also have a --profile option to facilitate profiling tests.
247 Zulip Documentation, Release 1.7.0
Another thing to note is that our tests generally "fail fast," i.e. they stop at the first sign of trouble. This is generally a good thing for iterative development, but you can override this behavior with the --nonfatal-errors option. A useful option to combine with that is the --rerun option, which will rerun just the tests that failed in the last test run.
47.3 Writing tests
Before you write your first tests of Zulip, it is worthwhile to read the rest of this document, and you can also read some of the existing tests in zerver/tests to get a feel for the patterns we use. A good practice is to get a "failing test" before you start to implement your feature. First, it is a useful exercise to understand what needs to happen in your tests before you write the code, as it can help drive out simple design or help you make incremental progress on a large feature. Second, you want to avoid introducing tests that give false positives. Ensuring that a test fails before you implement the feature ensures that if somebody accidentally regresses the feature in the future, the test will catch the regression. Another important files to skim are zerver/lib/test_helpers.py, which contains test helpers. zerver/lib/test_classes.py, which contains our ZulipTestCase and WebhookTestCase classes.
47.3.1 Setting up data for tests
All tests start with the same fixture data. (The tests themselves update the database, but they do so inside a transaction that gets rolled back after each of the tests complete. For more details on how the fixture data gets set up, refer to tools/setup/generate-fixtures.) The fixture data includes a few users that are named after Shakesepeare characters, and they are part of the "zulip.com" realm. Generally, you will also do some explicit data setup of your own. Here are a couple useful methods in ZulipTestCase: • common_subscribe_to_streams • send_message • make_stream • subscribe_to_stream More typically, you will use methods directly from the backend code. (This ensures more end-to-end testing, and avoids false positives from tests that might not consider ancillary parts of data setup that could influence tests results.) Here are some example action methods that tests may use for data setup: • check_send_message • do_change_is_admin • do_create_user • do_make_stream_private
47.3.2 Testing with mocks
This section is a beginner’s guide to mocking with Python’s unittest.mock library. It will give you answers to the most common questions around mocking, and a selection of commonly used mocking techniques.
248 Chapter 47. Backend Django tests Zulip Documentation, Release 1.7.0
What is mocking?
When writing tests, mocks allow you to replace methods or objects with fake entities suiting your testing requirements. Once an object is mocked, its original code does not get executed anymore. Rather, you can think of a mocked object as an initially empty shell: Calling it won’t do anything, but you can fill your shell with custom code, return values, etc. Additionally, you can observe any calls made to your mocked object.
Why is mocking useful?
When writing tests, it often occurs that you make calls to functions taking complex arguments. Creating a real instance of such an argument would require the use of various different libraries, a lot of boilerplate code, etc. Another scenario is that the tested code accesses files or objects that don’t exist at testing time. Finally, it is good practice to keep tests independent from others. Mocks help you to isolate test cases by simulating objects and methods irrelevant to a test’s goal. In all of these cases, you can "mock out" the function calls / objects and replace them with fake instances that only implement a limited interface. On top of that, these fake instances can be easily analyzed. Say you have a method greet(name_key) defined as follows:
def greet(name_key: str)-> str: name= fetch_database(name_key) return "Hello"+ name
• You want to test greet(). • In your test, you want to call greet("Mario") and verify that it returns the correct greeting:
def test_greet()-> str: greeting= greet("Mario") assert greeting =="Hello Mr. Mario Mario"
-> You have a problem: greet() calls fetch_database(). fetch_database() does some look-ups in a database. You haven’t created that database for your tests, so your test would fail, even though the code is correct. • Luckily, you know that fetch_database("Mario") should return "Mr. Mario Mario". – Hint: Sometimes, you might not know the exact return value, but one that is equally valid and works with the rest of the code. In that case, just use this one. -> Solution: You mock fetch_database(). This is also referred to as "mocking out" fetch_database().
from unittest.mock import MagickMock # Our mocking class that will replace `fetch_
˓→database()`
def test_greet()-> None: # Mock `fetch_database()` with an object that acts like a shell: It still accepts
˓→calls like `fetch_database()`, # but doesn't do any database lookup. We "fill" the shell with a return value;
˓→This value will be returned on every # call to `fetch_database()`. fetch_database= MagicMock(return_value="Mr. Mario Mario") greeting= greet("Mario") assert greeting =="Hello Mr. Mario Mario"
That’s all. Note that this mock is suitable for testing greet(), but not for testing fetch_database(). More generally, you should only mock those functions you explicitly don’t want to test.
47.3. Writing tests 249 Zulip Documentation, Release 1.7.0
How does mocking work under the hood?
Since Python 3.3, the standard mocking library is unittest.mock. unittest.mock implements the basic mocking class Mock. It also implements MagickMock, which is the same as Mock, but contains many default magic methods (in Python, those are the ones starting with with a dunder __). From the docs: In most of these examples the Mock and MagicMock classes are interchangeable. As the MagicMock is the more capable class it makes a sensible one to use by default. Mock itself is a class that principally accepts and records any and all calls. A piece of code like from unittest import mock foo= mock.Mock() foo.bar('quux') foo.baz foo.qux= 42 is not going to throw any errors. Our mock silently accepts all these calls and records them. Mock also implements methods for us to access and assert its records, e.g. foo.bar.assert_called_with('quux')
Finally, unittest.mock also provides a method to mock objects only within a scope: patch(). We can use patch() either as a decorator or as a context manager. In both cases, the mock created by patch() will apply for the scope of the decorator / context manager. patch() takes only one required argument target. target is a string in dot notation that refers to the name of the object you want to mock. It will then assign a MagickMock() to that object. As an example, look at the following code: from unittest import mock from os import urandom with mock.patch('__main__.urandom', return_value=42): print(urandom(1)) print(urandom(1)) # No matter what value we plug in for urandom, it will always
˓→return 42. print(urandom(1)) # We exited the context manager, so the mock doesn't apply anymore.
˓→Will return a random byte.
Note that calling mock.patch('os.urandom', return_value=42) wouldn’t work here: os.urandom would be the name of our patched object. However, we imported urandom with from os import urandom; hence, we bound the urandom name to our current module __main__. On the other hand, if we had used import os.urandom, we would need to call mock.patch('os. urandom', return_value=42) instead.
Boilerplate code
• Including the Python mocking library:
from unittest import mock
• Mocking a class with a context manager:
with mock.patch('module.ClassName', foo=42, return_value='I am a mock') as my_
˓→mock: # In here, 'module.ClassName' is mocked with a MagicMock() object my_mock.
250 Chapter 47. Backend Django tests Zulip Documentation, Release 1.7.0
# my_mock has an attribute named foo with the value 42. # var = module.ClassName() will assign 'I am a mock' to var.
• Mocking a class with a decorator:
@mock.patch('module.ClassName', foo=42, return_value='I am a mock') def my_function(my_mock): # ... # In here, 'module.ClassName' will behave as in the previous example.
• Mocking a class attribute:
with mock.patch.object(module.ClassName,'class_method', return_value=42) # In here, 'module.ClassName' has the same properties as before, except for
˓→'class_method' # Calling module.ClassName.class_method() will now return 42.
Note the missing quotes around module.ClassName in the patch.object() call.
Zulip mocking practices
For mocking we generally use the "mock" library and use mock.patch as a context manager or decorator. We also take advantage of some context managers from Django as well as our own custom helpers. Here is an example: with self.settings(RATE_LIMITING=True): with mock.patch('zerver.decorator.rate_limit_user') as rate_limit_mock: api_result= my_webhook(request) self.assertTrue(rate_limit_mock.called)
Follow this link for more information on the "settings" context manager. A common use is to prevent a call to a third-party service from using the Internet; git grep mock.patch | grep requests is a good way to find several examples of doing this.
47.4 Zulip Testing Philosophy
If there is one word to describe Zulip’s philosophy for writing tests, it is probably "flexible." (Hopefully "thorough" goes without saying.) When in doubt, unless speed concerns are prohibitive, you usually want your tests to be somewhat end-to-end, partic- ularly for testing endpoints. These are some of the testing strategies that you will see in the Zulip test suite...
47.4.1 Endpoint tests
We strive to test all of our URL endpoints. The vast majority of Zulip endpoints support a JSON interface. Regardless of the interface, an endpoint test generally follows this pattern: • Set up the data. • Login with self.login() or set up an API key. • Use a Zulip test helper to hit the endpoint.
47.4. Zulip Testing Philosophy 251 Zulip Documentation, Release 1.7.0
• Assert that the result was either a success or failure. • Check the data that comes back from the endpoint. Generally, if you are doing endpoint tests, you will want to create a test class that is a subclass of ZulipTestCase, which will provide you helper methods like the following: • api_auth • assert_json_error • assert_json_success • client_get • client_post • get_api_key • get_streams • login • send_message
47.4.2 Library tests
For certain Zulip library functions, especially the ones that are not intrinsically tied to Django, we use a classic unit testing approach of calling the function and inspecting the results. For these types of tests, you will often use methods like self.assertEqual(), self.assertTrue(), etc., which come with unittest via Django.
47.4.3 Fixture-driven tests
Particularly for testing Zulip’s integrations with third party systems, we strive to have a highly data-driven approach to testing. To give a specific example, when we test our GitHub integration, the test code reads a bunch of sample inputs from a JSON fixture file, feeds them to our GitHub integration code, and then verifies the output against expected values from the same JSON fixture file. Our fixtures live in zerver/fixtures.
47.4.4 Mocks and stubs
We use mocks and stubs for all the typical reasons: • to more precisely test the target code • to stub out calls to third-party services • to make it so that you can run the Zulip tests on the airplane without wifi A detailed description of mocks, along with useful coded snippets, can be found in the section Testing with mocks.
47.4.5 Template tests
In zerver/tests/test_templates.py we have a test that renders all of our back end templates with a "dummy" context, to make sure the templates don’t have obvious errors. (These tests won’t catch all types of errors; they are just a first line of defense.)
252 Chapter 47. Backend Django tests Zulip Documentation, Release 1.7.0
47.4.6 SQL performance tests
A common class of bug with Django systems is to handle bulk data in an inefficient way, where the back end populates objects for join tables with a series of individual queries that give O(N) latency. (The remedy is often just to call select_related(), but sometimes it requires a more subtle restructuring of the code.) We try to prevent these bugs in our tests by using a context manager called queries_captured() that captures the SQL queries used by the back end during a particular operation. We make assertions about those queries, often simply asserting that the number of queries is below some threshold.
47.4.7 Event-based tests
The Zulip back end has a mechanism where it will fetch initial data for a client from the database, and then it will subsequently apply some queued up events to that data to the data structure before notifying the client. The EventsRegisterTest.do_test() helper helps tests verify that the application of those events via ap- ply_events() produces the same data structure as performing an action that generates said event. This is a bit esoteric, but if you read the tests, you will see some of the patterns. You can also learn more about our event system in the new feature tutorial.
47.4.8 Negative tests
It is important to verify error handling paths for endpoints, particularly situations where we need to ensure that we don’t return results to clients with improper authentication or with limited authorization. A typical test will call the endpoint with either a non-logged in client, an invalid API key, or missing input fields. Then the test will call assert_json_error() to verify that the endpoint is properly failing.
47.5 Testing considerations
Here are some things to consider when writing new tests: • Duplication We try to avoid excessive duplication in tests. If you have several tests repeating the same type of test setup, consider making a setUp() method or a test helper. • Network independence Our tests should still work if you don’t have an internet connection. For third party clients, you can simulate their behavior using fixture data. For third party servers, you can typically simulate their behavior using mocks. • Coverage We have 100% line coverage on several of our backend modules. You can use the --coverage option to generate coverage reports, and new code should have 100% coverage, which generally requires testing not only the "happy path" but also error handling code and edge cases. It will generate a nice HTML report that you can view right from your browser (the tool prints the URL where the report is exposed in your development environment). Note that test-backend --coverage will assert that various specific files in the project have 100% test cover- age and throw an error if their coverage has fallen. One of our project goals is to expand that checking to ever-larger parts of the codebase.
47.5. Testing considerations 253 Zulip Documentation, Release 1.7.0
254 Chapter 47. Backend Django tests CHAPTER 48
JavaScript unit tests
As an alternative to the black-box whole-app testing, you can unit test individual JavaScript files. If you are writing JavaScript code that manipulates data (as opposed to coordinating UI changes), then you probably modify existing unit test modules to ensure the quality of your code and prevent regressions. The JS unit tests are written to work with node. You can find them in frontend_tests/node_tests. Here is an example test from frontend_tests/node_tests/stream_data.js:
(function test_get_by_id() { stream_data.clear_subscriptions(); var id= 42; var sub={ name:'Denmark', subscribed: true, color:'red', stream_id: id }; stream_data.add_sub('Denmark', sub); sub= stream_data.get_sub('Denmark'); assert.equal(sub.color,'red'); sub= stream_data.get_sub_by_id(id); assert.equal(sub.color,'red'); }());
The names of the node tests generally align with the names of the modules they test. If you modify a JS module in static/js you should see if there are corresponding test in frontend_tests/node_tests. If there are, you should strive to follow the patterns of the existing tests and add your own tests.
48.1 HTML output
The JavaScript unit tests can generate output to be viewed in the browser. The best examples of this are in frontend_tests/node_tests/templates.js.
255 Zulip Documentation, Release 1.7.0
The main use case for this mechanism is to be able to unit test templates and see how they are rendered without the complications of the surrounding app. (Obviously, you still need to test the app itself!) The HTML output can also help to debug the unit tests. Each test calls a method named write_handlebars_output after it renders a template with similar data. This API is still evolving, but you should be able to look at existing code for patterns. When you run tools/test-js-with-node, it will present you with a message like "To see more output, open var/test-js-with-node/index.html." Basically, you just need to open the file in the browser. (If you are running a VM, this might require switching to another terminal window to launch the open command.)
48.2 Coverage reports
You can automatically generate coverage reports for the JavaScript unit tests like this:
tools/test-js-with-node--coverage
Then open coverage/lcov-report/js/index.html in your browser. Modules we don’t test at all aren’t listed in the report, so this tends to overstate how good our overall coverage is, but it’s accurate for individual files. You can also click a filename to see the specific statements and branches not tested. 100% branch coverage isn’t necessarily possible, but getting to at least 80% branch coverage is a good goal.
48.3 Handling dependencies in unit tests
The following scheme helps avoid tests leaking globals between each other. First, if you can avoid globals, do it, and the code that is directly under test can simply be handled like this:
var search= require('js/search_suggestion.js');
For deeper dependencies, you want to categorize each module as follows: • Exercise the module’s real code for deeper, more realistic testing? • Stub out the module’s interface for more control, speed, and isolation? • Do some combination of the above? For all the modules where you want to run actual code, add statements like the following toward the top of your test file:
zrequire('util'); zrequire('stream_data'); zrequire('Filter','js/filter');
(Deprecation note: you may see code where we use add_dependencies or direct require statements. We should use zrequire instead.) For modules that you want to completely stub out, please use a pattern like this:
set_global('page_params',{ email:'[email protected]' });
// then maybe further down global.page_params.email='[email protected]';
256 Chapter 48. JavaScript unit tests Zulip Documentation, Release 1.7.0
Finally, there’s the hybrid situation, where you want to borrow some of a module’s real functionality but stub out other pieces. Obviously, this is a pretty strong smell that the other module might be lacking in cohesion, but that code might be outside your jurisdiction. The pattern here is this:
// Use real versions of parse/unparse var narrow= require('js/narrow.js'); set_global('narrow',{ parse: narrow.parse, unparse: narrow.unparse });
// But later, I want to stub the stream without having to call super-
˓→expensive // real code like narrow.activate(). global.narrow.stream= function () { return 'office'; };
48.4 Creating new test modules
The nodes tests rely on JS files that use the module pattern. For example, to test the foobar.js file, you would first add the following to the bottom of foobar.js:
if (typeof module !=='undefined'){ module.exports= foobar; }
This makes foobar.js follow the CommonJS module pattern, so it can be required in Node.js, which runs our tests. Now create frontend_tests/node_tests/foobar.js. At the top, require the Node.js assert module, and the module you’re testing, like so:
var assert = require('assert'); var foobar= require('js/foobar.js');
And of course, if the module you’re testing depends on other modules, or modifies global state, you may need to review the section on handling dependencies above. Define and call some tests using the assert module. Note that for "equal" asserts, the actual value comes first, the expected value second.
(function test_somefeature() { assert.strictEqual(foobar.somefeature('baz'),'quux'); assert.throws(foobar.somefeature('Invalid Input')); }());
The test runner (index.js) automatically runs all .js files in the frontend_tests/node directory.
48.4. Creating new test modules 257 Zulip Documentation, Release 1.7.0
258 Chapter 48. JavaScript unit tests CHAPTER 49
Web frontend black-box casperjs tests
These live in frontend_tests/casper_tests/. This is a "black box" integration test; we load the frontend in a real (headless) browser, from a real (development) server, and simulate UI interactions like sending messages, narrowing, etc., by actually clicking around the UI and waiting for things to change before doing the next step. These tasks are fantastic for ensuring the overall health of the project, but are also costly to maintain and keep free of nondeterministic failures, so we usually prefer to write a Node test instead when possible. Since the Casper tests interact with a real dev server, they can often catch backend bugs as well. You can run the casper tests with ./tools/test-js-with-casper or as ./tools/ test-js-with-casper 06-settings.js to run a single test file from frontend_tests/ casper_tests/.
49.1 Debugging Casper.JS
Casper.js (via PhantomJS) has support for remote debugging. However, it is not perfect. Here are some steps for using it and gotchas you might want to know; you’ll likely also want to read the section on writing tests (below) if you get stuck, since the advice on how to write correct Casper selectors will likely be relevant. The first thing to do when debugging Casper tests is to check the additional debug output that our framework provides: • You can check the screenshots of what the UI looked like at the time of failures at var/casper/ casper-failure*.png. • If it’s possible there’s a backend exception involved, var/casper/server.log will contain the server logs from the casper run; it’s worth looking there for tracebacks if you get stuck.
49.1.1 Print debugging
If you need to use print debugging in casper, you can do using casper.log; see http://docs.casperjs.org/en/latest/ logging.html for details. An additional debugging technique is to enable verbose mode in the Casper tests; you can do this by adding to the top of the relevant test file the following:
259 Zulip Documentation, Release 1.7.0
var casper= require('casper').create({ verbose: true, logLevel:"debug" });
This can sometimes give insight into exactly what’s happening.
49.1.2 Remote debugging
This is a pain to setup with Vagrant because port 7777 and 9981 aren’t forwarded to the host by default, but can be pretty useful in rare difficult cases. To turn on remote debugging, pass --remote-debug to the ./frontend_tests/run-casper script. This will run the tests with port 7777 open for remote debugging. You can now connect to localhost:7777 in a Webkit browser. Somewhat recent versions of Chrome or Safari might be required. • When connecting to the remote debugger, you will see a list of pages, probably 2. One page called about:blank is the headless page in which the CasperJS test itself is actually running in. This is where your test code is. • The other page, probably localhost:9981, is the Zulip page that the test is testing—that is, the page running our app that our test is exercising. Since the tests are now running, you can open the about:blank page, switch to the Scripts tab, and open the running 0x-foo.js test. If you set a breakpoint and it is hit, the inspector will pause and you can do your normal JS debugging. You can also put breakpoints in the Zulip webpage itself if you wish to inspect the state of the Zulip frontend.
49.1.3 Reproducing races only seen in Travis CI
We’ve sometimes found it useful for reproducing Casper race conditions in Casper tests that mostly only happen in Travis CI with really cheap VPS servers (e.g. Scaleway’s 2GB x86). This works because an ultra slow machine is more likely to have things happen in an order similar to what happens in Travis CI’s very slow containers.
49.2 Writing Casper tests
Probably the easiest way to learn how to write Casper tests is to study some of the existing test files. There are a few tips that can be useful for writing Casper tests in addition to the debugging notes below: • Run just the file containing your new tests as described above to have a fast debugging cycle. • With frontend tests in general, it’s very important to write your code to wait for the right events. Before essen- tially every action you take on the page, you’ll want to use waitUntilVisible, waitWhileVisible, or a similar function to make sure the page or elemant is ready before you interact with it. For instance, if you want to click a button that you can select via #btn-submit, and then check that it causes success-elt to appear, you’ll want to write something like:
casper.waitUntilVisible("#btn-submit", function () { casper.click('#btn-submit') casper.test.assertExists("#success-elt"); });
260 Chapter 49. Web frontend black-box casperjs tests Zulip Documentation, Release 1.7.0
In many cases, you will actually need to wait for the UI to update clicking the button before doing asserts or the next step. This will ensure that the UI has finished updating from the previous step before Casper attempts to next step. The various wait functions supported in Casper are documented in the Casper here: http://docs.casperjs. org/en/latest/modules/casper.html#waitforselector and the various assert statements available are documented here: http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype • The casper.wait style functions (waitWhileVisible, waitUntilVisible, etc.) cannot be chained together in certain conditions without creating race conditions where the test may fail nondeterministically. For example, don’t do this:
casper.waitUntilVisible('tag 1'); casper.click('button'); casper.waitUntilVisible('tag 2');
Instead, if you want to avoid race condition, wrap the second waitFor in a then function like this:
casper.then(function () { casper.waitUntilVisible('tag 1', function () { casper.click('#btn-submit'); }); }); casper.then(function () { casper.waitUntilVisible('tag 2', function () { casper.test.assertExists('#success-elt'); }); });
(You’ll also want to use selectors that are as explicit as possible, to avoid accidentally clicking multiple buttons or the wrong button in your test, which can cause nondeterministic failures) • Generally casper.waitUntilVisible is preferable to e.g. casper.waitForSelector, since the former will confirm the thing is actually on screen. E.g. if you’re waiting to switch from one panel of the the settings overlay to another by waiting for a particular widget to appear, casper.waitForSelector may not actually wait (since the widget is probably in the DOM, just not visible), but casper.waitUntilVisible will wait until it’s actually shown. • The selectors (i.e. things you put inside casper.waitUntilVisible() and friends) appearing in Casper tests are CSS3 selectors, which is a slightly different syntax from the jQuery selectors used in the rest of the Zulip codebase; in particular, some expressions that work with jQuery (and thus normal Zulip JavaScript code) won’t work with CSS3. It’s often helpful to debug selectors interactively, which you can do in the Chrome JavaScript console. The way to do it is $$("#settings-dropdown"); that queries CSS3 selectors, so you can debug your selector in the console and then paste it into your Casper test once it’s working. For other browsers like Firefox, you can use querySelectorAll("#settings-dropdown"), syntax which is only available in the browser’s JavaScript console. You can learn more about these selectors and other JavaScript console tools here. • The test suite uses a smaller set of default user accounts and other data initialized in the database than the development environment; to see what differs check out the section related to options["test_suite"] in zilencer/management/commands/populate_db.py. • Casper effectively runs your test file in two phases – first it runs the code in the test file, which for most test files will just collect a series of steps (each being a casper.then or casper.wait... call). Then, usually at the end of the test file, you’ll have a casper.run call which actually runs that series of steps. This means that if you write code in your test file outside a casper.then or casper.wait... method, it will actually run be- fore all the Casper test steps that are declared in the file, which can lead to confusing failures where the new code you write in between two casper.then blocks actually runs before either of them. See this for more details about how Casper works: http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work
49.2. Writing Casper tests 261 Zulip Documentation, Release 1.7.0
262 Chapter 49. Web frontend black-box casperjs tests CHAPTER 50
Travis CI
The Zulip server uses Travis CI for its continuous integration. This page documents useful tools and tips to know about when using Travis CI and debugging issues with it.
50.1 Goals
The overall goal of our Travis CI setup is to avoid regressions and minimize the total time spent debugging Zulip. We do that by trying to catch as many possible future bugs as possible, while minimizing both latency and false positives, both of which can waste a lot of developer time. There are a few implications of this overall goal: • If a test is failing nondeterministically in Travis CI, we consider that to be an urgent problem. • If the tests become a lot slower, that is also an urgent problem. • Everything we do in CI should also have a way to run it quickly (under 1 minute, preferably under 3 seconds), in order to iterate fast in development. Except when working on the Travis CI configuration itself, a developer should never have to repeatedly wait 10 minutes for a full Travis run to iteratively debug something.
50.2 Configuration
The main Travis configuration file is .travis.yml. The specific test suites we have are listed in the matrix section, which has a matrix of Python versions and test suites ($TEST_SUITE). We’ve configured it to use a few helper scripts for each job: • tools/travis/setup-$TEST_SUITE: The script that sets up the test environment for that suite (E.g., installing dependencies). – For the backend and frontend suites, this is a thin wrapper around tools/provision, aka the develop- ment environment provision script. – For the production suite, this is a more complicated process because of all the packages Travis installs. See the comments in tools/travis/setup-production for details.
263 Zulip Documentation, Release 1.7.0
• tools/travis/$TEST_SUITE: The script that runs the actual test suite. The main purpose of the distinction between the two is that if the setup-backend job fails, Travis CI will report it as the suite having "Errored" (grey in their emails), whereas if the backend job fails, it’ll be reported as "Failed" failure (red in their emails). Note that Travis CI’s web UI seems to make no visual distinction between these. An important detail is that Travis CI will by default hide most phases other than the actual test; you can see this easily by looking at the line numbers in the Travis CI output. There are actually a bunch of phases (e.g. the project’s setup job, downloading caches near the beginning, uploading caches at the end, etc.), and if you’re debugging our configuration, you’ll want to look at these closely.
50.3 Useful debugging tips and tools
• Zulip uses the ts tool to log the current time on every line of the output in our Travis CI scripts. You can use this output to determine which steps are actually consuming a lot of time. • For performance issues, this statistics tool can give you test runtime history data that can help with determining when a performance issue was introduced and whether it was fixed. Note you need to click the "Run" button for it to do anything. • You can sign up your personal repo for Travis CI so that every remote branch you push will be tested, which can be helpful when debugging something complicated.
50.4 Performance optimizations
50.4.1 Caching
An important element of making Travis CI perform effectively is caching the provisioning of a Zulip development environment. In particular, we cache the following across jobs: • Python virtualenvs • node_modules directories • Built/downloaded emoji sprite sheets and data This has a huge impact on the performance of running tests in Travis CI; without these caches, the average test time would be several times longer. We have designed these caches carefully (they are also used in production and the Zulip development environment) to ensure that each is named by a hash of its dependencies, so Zulip should always be using the same version of dependencies it would have used had the cache not existed. In practice, bugs are always possible, so be mindful of this possibility. A consequence of this caching is that test jobs for branches which modify package.json, requirements/, and other key dependencies will be significantly slower than normal, because they won’t get to benefit from the cache.
50.4.2 Uninstalling packages
In the production suite, we run apt-get upgrade at some point (effectively, because the Zulip installer does). This carries a huge performance cost in Travis CI, because (1) they don’t keep their test systems up to date and (2) literally everything is installed in their build workers (e.g. several copies of Postgres, Java, MySQL, etc.). In order to make Zulip’s tests performance reasonably well, we uninstall (or mark with apt-mark hold) many of these dependencies that are irrelevant to Zulip in tools/travis/setup-production.
264 Chapter 50. Travis CI CHAPTER 51
Manual testing
As a general rule, we like to have automated tests for everything that can be practically tested. However, there are certain types of bugs that are best caught with old fashioned manual testing (also called manual QA). Manual testing not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics of a feature they’re working on. This doc assumes you know how to set up a local development server and open the Zulip app in the browser. It also assumes a basic knowledge of how to use Zulip.
51.1 Basic Stuff
When testing Zulip manually, here are things to focus on: • The best bugs to catch are security/permissions bugs. • Don’t rush manual testing. Look for small details like display glitches. • Always test with multiple users (you can use incognito windows to facilitate this). • Always keep the inspector console open and watch for warnings or errors. • Be methodical about collecting information on bugs. (You will eventually want to create tickets, but you may want to consolidate your own notes before filing tickets.) You generally want to test with Cordelia as the primary user, and use Hamlet as her primary conversation partner. Use Iago when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things such as Cordelia not being able to receive messages not intended for her. The rest of this document groups tasks into basic areas of functionality of the system. If you have multiple people testing at once, you can divvy up QA tasks by these sections in the doc.
51.1.1 Message view
We mostly test the message view as part of testing everything else, but there are few things to specially test here.
265 Zulip Documentation, Release 1.7.0
Try using all the navigation hotkeys: • Up/k • Down/j • PgUp/K • PgDn/J/Spacebar • End (or fn-right-arrow on OSX) • also try scrolling aggressively with the mouse Try narrowing from the message view: • Hotkeys – use Esc to go to home – use s to narrow to a stream (select message first and verify in sidebar) – use S to narrow to the topic (and verify in sidebar) – use v to navigate to private messages • Click on the recipient bar – narrow to a stream – narrow to a topic – narrow to PMs with one user – narrow to a group PM • Click on the Zulip logo – narrow to a topic – click on the Zulip logo (and verify you’re in the home view)
51.1.2 Message editing
With message editing we mainly want to exercise topic changes. Here are some tasks: • Do lots of editing – send a message to the topic "original" – edit the message content – send two messages to the "original" stream – start to edit a message but then cancel – change the topic for the first message to "change1" (just this message) – narrow back to "original" – send one more message to the stream – change the topic for the last two messages to "change2" – narrow back to "original" – send two more messages to the stream
266 Chapter 51. Manual testing Zulip Documentation, Release 1.7.0
– edit the 2nd message on topic and change all messages to "change3" • Test UI entry points – hit "i" then down arrow to edit with the popup – use the popup using the mouse – enter edit mode using the pencil icon
51.1.3 Narrowing
Zulip uses the term "narrowing" to refer to opening different views of your messages, whether by clicking on sidebar options, recipient bars, or by using search. The main focus of these tasks should be watching unread counts. Of course, you also want to see messages show up in the message pane. And, finally, you should make sure that no messages outside the narrow show up in Cordelia’s view. Important: Make sure that Cordelia is subscribed to Verona but not subscribed to Denmark; if not, you should use different streams for your testing. When testing narrows, you want to have Hamlet send the same message several times in a row, while cycling Cordelia through various narrows. Here are the main tasks for Hamlet (and each message gets sent several times): • Send Cordelia/Othello a PM. • Send Cordelia a PM. • Send Othello a PM. • Post to Verona/foo. • Post to Verona/bar. • Post to Denmark/foo. • Post to Denmark/foo and mention Cordelia. For each of the above types of messages, you will want to cycle through the following views for Cordelia (and have Hamlet send new messages after each narrow): • Go to Home view. • Go to Private Messages view. • Go to Private Messages w/Hamlet. • Go to Private Messages w/Hamlet and Othello. • Go to Verona view. • Go to Verona/bar view. • Go to Verona/foo view. • Go to Denmark view. • Go to Denmark/foo view. There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then the whole exercise is about 30 minutes, assuming no bugs.
51.1. Basic Stuff 267 Zulip Documentation, Release 1.7.0
51.1.4 Composing messages
We have pretty good automated tests for our markdown processor, so manual testing is targeted more to other interac- tions. For composing a message, pay attention to details like what is automatically populated and where the focus is placed. • Hotkeys – use r to reply to a stream message – use r to reply to a PM – use R to reply to the author of a PM – use R to reply to the author of a PM stream – use c to compose a stream message – use C to compose a new PM • Buttons – Narrow to a stream and click on "New topic" – Narrow "Private Messages" and click on "New topic" – Narrow to a stream and click on "New private message" – Narrow "Private Messages" and click on "New private message" • Topics – Compose/send a message to a stream with no topic. – Compose/send a message to a stream with a new topic. – Compose/send a message to a stream with autocomplete. – Compose/send a message to a stream manually typing an existing topic. • Formatting stuff – Use the "A" icon to get markdown help. – Use the eyeball icon to show a preview and send from preview mode. – Toggle in and out of preview before sending a message. – Use @-mention to mention Hamlet (and send him a message). – Use #**devel** syntax and send to Hamlet, then follow the link. – Create a bulleted list. – Use the emoji icon to find an emoji in the picker. • Attachments – Send a message with an attachment using the paperclip icon. – Send a message with multiple attachments. – Copy an image from the clipboard. – Use drag/drop from the desktop to upload an image. • Drafts – Start composing a message then click outside the compose box.
268 Chapter 51. Manual testing Zulip Documentation, Release 1.7.0
– Use "restore drafts" to restore the draft. – Start composing then use "Esc" to abort the message. – Use "restore drafts" to restore the draft. – Start composing a stream message and then abort using the little "x" icon in the compose box. – Click on "New private message" and restore the draft. (You should now be sending to a stream.) • Click to send – Turn off enter-to-send.
* Send a two-paragraph message using tab and enter. * Send a two-paragraph message using control-enter or command-enter. – Turn on enter-to-send.
* Hit enter to send.
51.1.5 Popover menus
For this task you just want to go through all of our popover menus and exercise them. The main nuance here is that you occasionally want to click somewhere on the UI outside of an existing popover to see if the popover menu is "too sticky." Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view, then the message will disappear from the view. Here are the things to test: • Stream sidebar menus – Stream settings (just make sure it goes there) – Narrow (and then have Hamlet send a message) – Pin/unpin (do both) – Compose (send a message to the stream) – Mark as read (scroll back and then have Hamlet send you a message) – Mute/unmute (do both) – Unsubscribe (and then go to Stream settings in the gear menu to resubscribe) – Choose custom color (play around with this) • Topic sidebar menus – Narrow (and then have Hamlet send a message) – Mute/unmute (try both) – Mark as read (scroll back and then have Hamlet send you a message) • Left-message-pane menus (click on person’s name) – Verify email – Verify date message sent – Send a PM (make sure compose box is filled out ok) – Narrow to PMs with – Narrow to PMs sent by
51.1. Basic Stuff 269 Zulip Documentation, Release 1.7.0
• Right-pane-pane menus (click on chevron when hovering) – use "i" hotkey to open the menu – Edit a message you sent (using the down-arrow key to navigate the popup) – View Source for somebody else’s message (make sure it’s not editable) – Reply (send a message) – Collapse/uncollapse (try both) – Mute/unmute (try both, watch left sidebar) – Link to this conversation • Buddy list chevron menus – Narrow to PMs with – Narrow to message sent by – Compose a message to
51.1.6 Sidebar filtering
This is a fairly quick task where we test the search filters on the left sidebar and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to that stream. • Streams filtering – Use "w" hotkey to open the search. – Filter on "d". – Pin/unpin Denmark. – Clear filter. – Use "A" and "D" hotkeys to cycle through the streams. – Filter again and then click somewhere else. • Buddy list filtering – Use "q" hotkey to open the search. – Filter for Hamlet, Prospero, Othello, etc. – Log on Hamlet and log off Hamlet while filtering for Hamlet. – Log on/log off Hamlet while filtering for Othello. – Log on/log off Hamlet while not filtering at all. – Filter again and then click somewhere else.
51.1.7 Stream permissions
This is an important category to test, because we obviously do not want to have bugs where people can read messages on streams they should not have access to. The general flow here is for Hamlet to create the streams and verify that Cordelia has the correct visibility to them. First, we start off with "positive" tests.
270 Chapter 51. Manual testing Zulip Documentation, Release 1.7.0
• Positive tests – Have Hamlet create a public stream w/Cordelia subscribed and have him post a message to the stream. – Have Hamlet create a public stream without Cordelia and then...
* Have Hamlet post to the stream. * Have Cordelia subscribe to the stream. * Verify Cordelia can see the previous message. * Have Cordelia post a message to the stream. – Have Hamlet create an invite-only stream with Cordelia invited and test a two-way conversation between the two users. For negative tests, we want to dig a little deeper to find back doors for Cordelia to access the stream. Here are some techniques to try: • Try to have her compose a message to the stream by circumventing autocomplete. • Try to have her narrow to the stream using stream:foo in search. • Go to stream settings and see if the stream shows up. For public streams, it’s ok for Cordelia to know the stream exists, and she can subsequently subscribe. For private streams, she should not even know they exist (until she’s invited, of course). • Negative tests – Have Hamlet create a public stream without inviting Cordelia.
* Verify Cordelia can see the stream in her settings. * Verify Cordelia can’t compose a message to the stream. * Verify that Cordelia sees nothing when Hamlet posts to the stream. – Have Hamlet create a public stream with Cordelia, but then have Iago revoke her subscription using the admin page.
* Verify that the stream appears in Cordelia’s left sidebar and then goes away. * Try to have Cordelia view the stream using a sneaky search along the lines of stream:foo. – Have Hamlet create a private stream without inviting Cordelia.
* Verify Cordelia can’t compose a message to the stream.
51.1.8 Search
The main task for testing search is to play around with search suggestions (autocomplete). Once you select an option, verify the message view is consistent with the search and that the left sidebar reflects the current narrow. If a search comes up legitimately empty, have Hamlet send a message that matches the search. Here are searches you should be able to do with autocomplete: • stream:design • stream:Verona topic:Verona1 • stream:Verona keyword • sent by me • @-mentions
51.1. Basic Stuff 271 Zulip Documentation, Release 1.7.0
• starred messages • messages sent by Hamlet • PMs with Hamlet • PMs with Hamlet matching keyword "foo" There are some things you can try that don’t come up in autocomplete: • -stream:Verona (exclude Verona) • stream:Verona stream:devel (should return no results) Miscellaneous: • Use the "/" hotkey to start a search. • Use the "x" icon to clear a search. • Use the "Esc" hotkey to clear a search.
51.1.9 Stream settings
Test various UI entry points into stream settings: • Use small gear menu in left sidebar, then filter to "devel". • Use popover menu in left sidebar next to "devel". • Use gear menu above buddy list and filter to "devel". • Use gear menu and click on "devel." • Use gear menu and then click on chevron menu next to "devel." (I’m not sure why we still have the chevron at this writing.) Create new public stream "public1" and add Hamlet: • Type "public1" in the text box and then click "Create new stream." • Select "People must be invited" and then verify you can’t select "Announce stream". • Select "Anyone can join" again to make it be public. • Check the checkbox for Hamlet. • Hit the "Create" button. Test subscribe/unsubscribe: • Log in as Hamlet and go to his stream settings. • As Cordelia, unsubscribe from "public1" using the checkmark in the streams settings page. • Verify that Hamlet sees that Cordelia has unsubscribed (and the subscriber count should decrement). • As Cordelia, resubscribe to "public1." • Verify Hamlet sees that change. As Cordelia, exercise different options in Create Stream dialog by creating streams s1, s2, s3, etc.: • s1: anyone can join, announce it, and add Hamlet using filter feature • s2: people must be invited • s3: anyone can join, don’t announce
272 Chapter 51. Manual testing Zulip Documentation, Release 1.7.0
• s4: check all, then uncheck all, then invite only Hamlet • s5: invite everybody but Hamlet • s6: – create the stream as public, but don’t subscribe anybody initially – then click on stream options to add Hamlet using "Add" button Test per-stream options: • Use "devel" stream and send a message to it • Do mute and unmute, have Hamlet send messages • Test notifications on/off, have Hamlet send messages • Test pin and unpin, view left sidebar • Change stream color, and then view the left sidebar and the Home message view • Verify stream subscriber counts in the main stream view
51.1.10 User Settings
You can modify per-user settings by choosing "Settings" in the gear menu. Do these tasks as Cordelia. • Your account – Change full name (Hamlet should see the name change) – Customize avatar – Deactivate account (and then log in as Iago to re-activate Cordelia) • Display settings – Right now, these unfortunately require reloads to take effect. – Default language (change to Spanish) – User list on left sidebar in narrow windows (verify by making window thinner) – 24-hour time (and then test going back to AM/PM) • Notifications – Stream Message
* turn off notifications at user level · create a new stream · have Hamlet send a message
* turn on notifications at user level · create a new stream · have Hamlet send a message · then turn off notifications for that stream · have Hamlet send another message – Private Messages and @-mentions
* Test Desktop/Audible options
51.1. Basic Stuff 273 Zulip Documentation, Release 1.7.0
* You can ignore other stuff for now • Bots/API key – Create a bot with a generic avatar and send it a PM – Create a bot with a custom avatar and send it a PM – Change your API key • Alert words – Create an alert word – Have Hamlet send you a message that includes the alert word • Zulip labs – Turn on auto-scroll to new messages (and have Hamlet send you one) – Turn on/off "Enable desktop notifications for new streams" and test. (We may eliminate this option soon.)
51.1.11 Keyboard Shorcuts
We mostly test keyboard shortcuts as part of other tasks. Here are the tasks for this section: • Use the "?" hotkey to open the keyboard help • Proofread the dialog for typos. • Close the dialog. • Re-open the keyboard help using the gear menu. • Find a hotkey that you don’t frequently use and experiment with its usage.
51.1.12 Miscellaneous menu options
Make sure that these options launch appropriate help screens: • Proofread and try a couple random options: – Message formatting – Search operators • Make sure help launches in a separate browser tab: – Desktop and mobile apps – Integrations – API documentation
51.1.13 Inviting users/tutorial
Here are the tasks: • Invite [email protected] using the link beneath the buddy list but then don’t take further action. • Fully invite [email protected] using the gear menu.
274 Chapter 51. Manual testing Zulip Documentation, Release 1.7.0
• Go to the development console to get the login link for [email protected]. • Go through the signup flow. • Follow the tutorial. • Use the gear menu to log out. • Log back in as Cordelia (admittedly, this step doesn’t really QA much of our production code, since the login flow is customized for the development environment).
51.1.14 To be continued...
This document does not cover settings/admin options yet. The main things to do when testing the settings system are: • Verify that changes are synced to other users. • Verify error messages appear if you do something wrong and look right. • For organization settings, verify that they look right in read-only mode (i.e. when not logged into an adminis- trator account).
51.1. Basic Stuff 275 Zulip Documentation, Release 1.7.0
276 Chapter 51. Manual testing CHAPTER 52
Provisioning and third-party dependencies
Zulip is a large project, with well over 100 third-party dependencies, and managing them well is essential to the quality of the project. In this document, we discuss the various classes of dependencies that Zulip has, and how we manage them. Zulip’s dependency management has some really nice properties: • Fast provisioning. When switching to a different commit in the Zulip project with the same dependencies, it takes under 10 seconds to re-provision a working Zulip development environment after switching. If there are new dependencies, one only needs to wait to download the new ones, not all the pre-existing dependencies. • Consistent provisioning. Every time a Zulip development or production environment is provisioned/installed, it should end up using the exactly correct versions of all major dependencies. • Low maintenance burden. To the extent possible, we want to avoid manual work and keeping track of things that could be automated. This makes it easy to keep running the latest versions of our various dependencies. The purpose of this document is to detail all of Zulip’s third-party dependencies and how we manage their versions.
52.1 Provisioning
We refer to "provisioning" as the process of installing and configuring the dependencies of a Zulip development envi- ronment. It’s done using tools/provision, and the output is conveniently logged by var/log/provision. log to help with debugging. Provisioning makes use of a lot of caching. Some of those caches are not immune to being corrupted if you mess around with files in your repository a lot. We have tools/provision --force to (still fairly quickly) rerun most steps that would otherwise have been skipped due to caching. In the Vagrant development environment, vagrant provision will run the provision script; vagrant up will boot the machine, and will also run an initial provision the first time only.
52.2 Philosophy on adding third-party dependencies
In the Zulip project, we take a pragmatic approach to third-party dependencies. Overall, if a third-party project does something well that Zulip needs to do (and has an appropriate license), we’d love to use it rather than reinventing the
277 Zulip Documentation, Release 1.7.0 wheel. If the third-party project needs some small changes to work, we prefer to make those changes and contribute them upstream. When the upstream maintainer is slow to respond, we may use a fork of the dependency until the code is merged upstream; as a result, we usually have a few packages in requirements.txt that are installed from a GitHub URL. What we look for in choosing dependencies is whether the project is well-maintained. Usually one can tell fairly quickly from looking at a project’s issue tracker how well-managed it is: a quick look at how the issue tracker is man- aged (or not) and the test suite is usually enough to decide if a project is going to be a high-maintenance dependency or not. That said, we do still take on some smaller dependencies that don’t have a well-managed project, if we feel that using the project will still be a better investment than writing our own implementation of that project’s functionality. We’ve adopted a few projects in the past that had a good codebase but whose maintainer no longer had time for them. One case where we apply added scrutiny to third-party dependencies is JS libraries. They are a particularly important concern because we want to keep the Zulip webapp’s JS bundle small, so that Zulip continues to load quickly on systems with low network bandwidth. We’ll look at large JS libraries with much greater scrutiny for whether their functionality justifies their size than Python dependencies, since an extra 50KB of code usually doesn’t matter in the backend, but does in JavaScript.
52.3 System packages
For the third-party services like Postgres, Redis, Nginx, and RabbitMQ that are documented in the architecture overview, we rely on the versions of those packages provided alongside the Linux distribution on which Zulip is deployed. Because Zulip only supports Ubuntu in production, this usually means apt, though we do support other platforms in development. Since we don’t control the versions of these dependencies, we avoid relying on specific versions of these packages wherever possible. The exact lists of apt packages needed by Zulip are maintained in a few places: • For production, in our puppet configuration, puppet/zulip/, using the Package and SafePackage directives. • For development, in APT_DEPENDENCIES in tools/lib/provision.py. • The packages needed to build a Zulip virtualenv, in VENV_DEPENDENCIES in scripts/lib/ setup_venv.py. These are separate from the rest because (1) we may need to install a virtualenv before running the more complex scripts that, in turn, install other dependencies, and (2) because that list is shared between development and production. We maintain a PPA (personal package archive) with some packages unique to Zulip (e.g the tsearch_extras postgres extension) and backported versions of other dependencies (e.g. camo, to fix a buggy init script). Our goal is to shrink or eliminate this PPA where possible by getting issues addressed in the upstream distributions. We also rely on the pgroonga PPA for the pgroonga postgres extension, used by our full-text search.
52.4 Python packages
We manage Python packages via the Python-standard requirements.txt system and virtualenvs, but there’s a number of interesting details about how Zulip makes this system work well for us that are worth highlighting. The system is largely managed by the code in scripts/lib/setup_venv.py • Using pip to manage dependencies. This is standard in the Python ecosystem, and means we only need to record a list of versions in a requirements.txt file to declare what we’re using. Since we have a few differ- ent installation targets, we maintain several requirements.txt format files in the requirements/ di- rectory (e.g. dev.txt for development, prod.txt for production, docs.txt for ReadTheDocs, common.
278 Chapter 52. Provisioning and third-party dependencies Zulip Documentation, Release 1.7.0
txt for the vast majority of packages common to prod and development, etc.). We use pip install --no-deps to ensure we only install the packages we explicitly declare as dependencies. • virtualenv with pinned versions. For a large application like Zulip, it is important to ensure that we’re always using consistent, predictable versions of all of our Python dependencies. To ensure this, we install our depen- dencies in a virtualenv that contains only the packages and versions that Zulip needs, and we always pin exact versions of our dependencies in our requirements.txt files. We pin exact versions, not minimum ver- sions, so that installing Zulip won’t break if a dependency makes a buggy release. A side effect is that it’s easy to debug problems caused by dependency upgrades, since we’re always doing those upgrades with an explicit commit updating the requirements/ directory. • Caching of virtualenvs and packages. To make updating the dependencies of a Zulip installation efficient, we maintain a cache of virtualenvs named by the hash of the relevant requirements.txt file (scripts/ lib/hash_reqs.py). These caches live under /srv/zulip-venv-cache/
52.5 JavaScript and other frontend packages
We use the same set of strategies described for Python dependencies for most of our JavaScript dependencies, so we won’t repeat the reasoning here. • In a fashion very analogous to the Python codebase, scripts/lib/node_cache.py manages cached node_modules directories in /srv/zulip-npm-cache. Each is named by its hash, computed by the generate_sha1sum_node_modules function. scripts/lib/clean-npm-cache handles garbage-collection. • We use yarn, a pip-like tool for JavaScript, to download most JavaScript dependencies. Yarn talks to standard the [npm][] repository. We use the standard package.json file to declare our direct dependencies, with sections for development and production. Yarn takes care of pinning the versions of indirect dependencies in the yarn.lock file; yarn upgrade updates the yarn.lock files.
52.5. JavaScript and other frontend packages 279 Zulip Documentation, Release 1.7.0
• tools/update-prod-static. This process is discussed in detail in the static asset pipeline article, but we don’t use the node_modules directories directly in production. Instead, static assets are compiled using our static asset pipeline and it is the compiled assets that are served directly to users. As a result, we don’t ship the node_modules directory in a Zulip production release tarball, which is a good thing, because doing so would more than double the size of a Zulip release tarball. • Checked-in packages. In contrast with Python, we have a few JavaScript dependencies that we have copied into the main Zulip repository under static/third, often with patches. These date from an era before npm existed. It is a project goal to eliminate these checked-in versions of dependencies and instead use versions managed by the npm repositories.
52.6 Node and Yarn
These are installed by scripts/lib/install-node (which in turn uses the standard third-party nvm installer to download node and pin its version) and scripts/lib/third/install-yarn.sh (the standard installer for yarn, modified to support installing to a path that is not the current user’s home directory). • nvm has its own system for installing each version of node at its own path, which we use, though we install a /usr/local/bin/node wrapper to access the desired version conveniently and efficiently (nvm has a lot of startup overhead). • install-yarn.sh is configured to install yarn at /srv/zulip-yarn. We don’t do anything special to try to manage multiple versions of yarn.
52.7 Other third-party and generated files
In this section, we discuss the other third-party dependencies, generated code, and other files whose original primary source is not the Zulip server repository, and how we provision and otherwise maintain them.
52.7.1 Emoji
Zulip uses the iamcal emoji data package for its emoji data and sprite sheets. We download this dependency us- ing npm, and then have a tool, tools/setup/build_emoji, which reformats the emoji data into the files under static/generated/emoji. Those files are in turn used by our markdown processor and tools/ update-prod-static to make Zulip’s emoji work in the various environments where they need to be displayed. Since processing emoji is a relatively expensive operation, as part of optimizing provisioning, we use the same caching strategy for the compiled emoji data as we use for virtualenvs and node_modules directories, with scripts/ lib/clean-emoji-cache responsible for garbage-collection. This caching and garbage-collection is required because a correct emoji implementation involves over 1000 small image files and a few large ones. There is a more extended article on our emoji infrastructure.
52.7.2 Translations data
Zulip’s translations infrastructure generates several files from the source data, which we manage similar to our emoji, but without the caching (and thus without the garbage-collection). New translations data is downloaded from Tran- sifex and then compiled to generate both the production locale files and also language data in static/locale/ language*.json using manage.py compilemessages, which extends the default Django implementation of that tool.
280 Chapter 52. Provisioning and third-party dependencies Zulip Documentation, Release 1.7.0
52.7.3 Pygments data
The list of languages supported by our markdown syntax highlighting comes from the pygments package. tools/ setup/build_pygments_data is responsible for generating static/generated/pygments_data.js so that our JavaScript markdown processor has access to the supported list.
52.7.4 Authors data
Zulip maintains data on the developers who have contributed the most to the current version of Zulip in the /about page. These data are fetched using the GitHub API with tools/update-authors-json. In development, it just returns some basic test data to avoid adding load to GitHub’s APIs unnecessarily; it’s primarily run as part of building a release tarball.
52.8 Modifying provisioning
When making changes to Zulip’s provisioning process or dependencies, usually one needs to think about making changes in 3 places: • tools/lib/provision.py. This is the main provisioning script, used by most developers to maintain their development environment. • docs/dev-setup-non-vagrant.md. This is our "manual installation" documentation. Strategically, we’d like to move the support for more versions of Linux from here into tools/lib/provision.py. • Production. Our tools for compiling/generating static assets need to be called from tools/ update-prod-static, which is called by tools/build-release-tarball (for doing Zulip re- leases) as well as tools/upgrade-zulip-from-git (for deploying a Zulip server off of master).
52.8. Modifying provisioning 281 Zulip Documentation, Release 1.7.0
282 Chapter 52. Provisioning and third-party dependencies CHAPTER 53
Settings system
The page documents the Zulip settings system, and hopefully should help you decide how to correctly implement new settings you’re adding to Zulip. We have two types of administrative settings in Zulip: • Server settings are set via configuration files, and apply to the whole Zulip installation. • Realm settings (or organization settings) are usually set via the /#organization page in the Zulip web applica- tion, and apply to a single Zulip realm/organization. (Which, for most Zulip servers, is the only realm on the server). Philosophically, the goals of the settings system are to make it convenient for: • Zulip server administrators to configure Zulip’s featureset for their server without needing to patch Zulip • Realm administrators to configure settings for their organization independently without needing to talk with the server administrator. • Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable configuration.
53.1 Server settings
Zulip uses the Django settings system, which means that the settings files are Python programs that set a lot of variables with all-capital names like EMAIL_GATEWAY_PATTERN. You can access these anywhere in the Zulip Django code using e.g.: from django.conf import settings print(settings.EMAIL_GATEWAY_PATTERN)
Additionally, if you need to access a Django setting in a shell script (or just on the command line for debugging), you can use e.g.:
$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN %s@localhost:9991
283 Zulip Documentation, Release 1.7.0
Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the ~1000 lines of settings needed by the Zulip Django app. As a result, there are a few files involved in the Zulip settings for server administrators. In a production environment, we have: • /etc/zulip/settings.py (the template is in the Zulip repo at zproject/ prod_settings_template.py) is the main system administrator-facing settings file for Zulip. It contains all the server-specific settings, such as how to send outgoing email, the hostname of the Postgres database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys, etc.). The way we generally do settings that can be controlled with shell access to a Zulip server is to put a default in zproject/settings.py, and then override it here. • /etc/zulip/zulip-secrets.conf (generated by scripts/setup/generate_secrets.py as part of installation) contains secrets used by the Zulip installation. These are read using the standard Python ConfigParser, and accessed in zproject/settings.py by the get_secret function. All secrets/API keys/etc. used by the Zulip Django application should be stored here, and read using the get_secret function in zproject/settings.py. • zproject/settings.py is the main Django settings file for Zulip. It contains all the settings that are constant for all Zulip installations (e.g. configuration for logging, static assets, middleware, etc.), as well as default values for the settings the user would set in /etc/zulip/settings.py (you can look at the DEFAULT_SETTINGS dictionary to easily review the settings available). zproject/settings.py has a line from prod_settings import *, which in a prod environment has the effect of importing /etc/ zulip/settings.py (via a symlink). In a development environment, we have zproject/settings.py, and additionally: • zproject/dev_settings.py has the settings for the Zulip development environment; it mostly just im- ports prod_settings_template.py. • zproject/dev-secrets.conf replaces /etc/zulip/zulip-secrets.conf. • zproject/test_settings.py has the (default) settings used for the Zulip tests (both backend and Casper), which are applied on top of the development environment settings. When adding a new server setting to Zulip, you will typically add it in two or three places: • In DEFAULT_SETTINGS in zproject/settings.py, with a default value for production environments. If the settings has a secret key, you’ll add a get_secret call in zproject/settings.py (and the user will add the value when they configure the feature). • In an appropriate section of zproject/prod_settings_template.py, with documentation in the com- ments explaining the setting’s purpose and effect. • Possibly also zproject/dev_settings.py and/or zproject/test_settings.py, if the desired value of the setting for Zulip development and/or test environments is different from the default for production. Most settings should be enabled in the development environment, to maximize convenience of testing all of Zulip’s features; they should be enabled by default in production if we expect most Zulip sites to want those settings.
53.1.1 Testing Google & GitHub authentication
In order to set up Google or GitHub authentication in the development environment, you’ll have to go through the steps detailed in prod_settings_template.py with some changes. Here is the full procedure:
• Visit https://console.developers.google.com, click on Credentials on the left sidebar and create a Oauth2 client ID that allows redirects to https://localhost:9991/accounts/login/google/done/.
284 Chapter 53. Settings system Zulip Documentation, Release 1.7.0
• Go to the Library (left sidebar), then under "Social APIs" click on "Google+ API" and click the button to enable the API. • Uncomment 'zproject.backends.GoogleMobileOauth2Backend' in AUTHENTICATION_BACKENDS in dev_settings.py. • Uncomment GOOGLE_OAUTH2_CLIENT_ID in prod_settings_template.py & assign it the Client ID you got from Google. • Put the Client Secret you got from Google as google_oauth2_client_secret in dev-secrets. conf.
GitHub
• Register an OAuth2 application with GitHub at one of https://github.com/settings/developers or https://github.com/organizations/ORGNAME/settings/developers. Specify http://localhost:9991/ complete/github/ as the callback URL. • Uncomment 'zproject.backends.GitHubAuthBackend' in AUTHENTICATION_BACKENDS in dev_settings.py. • Uncomment SOCIAL_AUTH_GITHUB_KEY in prod_settings_template.py & assign it the Client ID you got from GitHub. • Put the Client Secret you got from GitHub as social_auth_github_secret in dev-secrets.conf.
53.1.2 Testing non-default settings
You can write tests for settings using e.g. with self.settings(GOOGLE_CLIENT_ID=None). However, this only works for settings which are checked at runtime, not settings which are only accessed in initialization of Django (or Zulip) internals (e.g. DATABASES). See the Django docs on overriding settings in tests for more details.
53.2 Realm settings
Realm settings are preferred for any configuration that is a matter of organizational policy (as opposed to technical capabilities of the server). As a result, configuration options for user-facing functionality is almost always added as a new realm setting, not a server setting. The new feature tutorial documents the process for adding a new realm setting to Zulip. So for example, the following server settings will eventually be replaced with realm settings: • NAME_CHANGES_DISABLED • INLINE_IMAGE_PREVIEW • ENABLE_GRAVATAR • Which authentication methods are allowed should probably appear in both places; in server settings indicating the capabilities of the server, and in the realm settings indicating which methods the realm administrator wants to allow users to login with.
53.2. Realm settings 285 Zulip Documentation, Release 1.7.0
286 Chapter 53. Settings system CHAPTER 54
Real-time Push and Events
Zulip’s "events system" is the server-to-client push system that powers our real-time sync. This document explains how it works; to read an example of how a complete feature using this system works, check out the new application feature tutorial. Any single-page web application like Zulip needs a story for how changes made by one client are synced to other clients, though having a good architecture for this is particularly important for a chat tool like Zulip, since the state is constantly changing. When we talk about clients, think a browser tab, mobile app, or API bot that needs to receive updates to the Zulip data. The simplest example is a new message being sent by one client; other clients must be notified in order to display the message. But a complete application like Zulip has dozens of different types of data that need to be synced to other clients, whether it be new streams, changes in a user’s name or avatar, settings changes, etc. In Zulip, we call these updates that need to be sent to other clients events. An important thing to understand when designing such a system is that events need to be synced to every client that has a copy of the old data if one wants to avoid clients displaying inaccurate data to users. So if a user has two browser windows open and sends a message, every client controlled by that user as well as any recipients of the message, including both of those two browser windows, will receive that event. (Technically, we don’t need to send events to the client that triggered the change, but this approach saves a bunch of unnecessary duplicate UI update code, since the client making the change can just use the same code as every other client, maybe plus a little notification that the operation succeeded). Architecturally, there are a few things needed to make a successful real-time sync system work: • Generation. Generating events when changes happen to data, and determining which users should receive each event. • Delivery. Efficiently delivering those events to interested clients, ideally in an exactly-once fashion. • UI updates. Updating the UI in the client once it has received events from the server. Reactive JavaScript libraries like React and Vue can help simplify the last piece, but there aren’t good standard systems for doing generation and delivery, so we have to build them ourselves. This document discusses how Zulip solves the generation and delivery problems in a scalable, correct, and predictable way.
287 Zulip Documentation, Release 1.7.0
54.1 Generation system
Zulip’s generation system is built around a Python function, send_event(event, users). It accepts an event data structure (just a Python dictionary with some keys and value; type is always one of the keys but the rest depends on the specific event) and a list of user IDs for the users whose clients should receive the event. In special cases such as message delivery, the list of users will instead be a list of dicts mapping user IDs to user-specific data like whether that user was mentioned in that message. The data passed to send_event are simply marshalled as JSON and placed in the notify_tornado RabbitMQ queue to be consumed by the delivery system. Usually, this list of users is one of 3 things: • A single user (e.g. for user-level settings changes). • Everyone in the realm (e.g. for organization-level settings changes, like new realm emoji). • Everyone who would receive a given message (for messages, emoji reactions, message editing, etc.); i.e. the subscribers to a stream or the people on a private message thread. It is the responsibility of the caller of send_event to choose the list of user IDs correctly. There can be security problems if e.g. an event containing private message content is sent to the entire organization. However, if an event isn’t sent to enough clients, there will likely be user-visible real-time sync bugs. Most of the hard work in event generation is about defining consistent event dictionaries that are clear, readable, will be useful to the wide range of possible clients, and make it easy for developers.
54.2 Delivery system
Zulip’s event delivery (real-time push) system is based on Tornado, which is ideal for handling a large number of open requests. Details on Tornado are available in the architecture overview, but in short it is good at holding open a large number of connections for a long time. The complete system is about 1500 lines of code in zerver/tornado/, primarily zerver/tornado/event_queue.py. Zulip’s event delivery system is based on "long-polling"; basically clients make GET /json/events calls to the server, and the server doesn’t respond to the request until it has an event to deliver to the client. This approach is reasonably efficient and works everywhere (unlike websockets, which have a decreasing but nonzero level of client compatibility problems). For each connected client, the Event Queue Server maintains an event queue, which contains any events that are to be delivered to that client which have not yet been acknowledged by that client. Ignoring the subtle details around error handling, the protocol is pretty simple; when a client does a GET /json/events call, the server checks if there are any events in the queue. If there are, it returns the events immediately. If there aren’t, it records that queue as having a waiting client (often called a handler in the code). When it pulls an event off the notify_tornado RabbitMQ queue, it simply delivers the event to each queue associated with one of the target users. If the queue has a waiting client, it breaks the long-poll connection by returning an HTTP response to the waiting client request. If there is no waiting client, it simply pushes the event onto the queue. When starting up, each client makes a POST /json/register to the server, which creates a new event queue for that client and returns the queue_id as well as an initial last_event_id to the client (it can also, optionally, fetch the initial data to save an RTT and avoid races; see the below section on initial data fetches for details on why this is useful). Once the event queue is registered, the client can just do an infinite loop calling GET /json/ events with those parameters, updating last_event_id each time to acknowledge any events it has received (see call_on_each_event in the Zulip Python API bindings for a complete example implementation). When handling each GET /json/events request, the queue server can safely delete any events have have an event ID less than or equal to the client’s last_event_id (event IDs are just a counter for the events a given queue has received.)
288 Chapter 54. Real-time Push and Events Zulip Documentation, Release 1.7.0
If network failures were impossible, the last_event_id parameter in the protocol would not be required, but it is important for enabling exactly-once delivery in the presence of potential failures. (Without it, the queue server would have to delete events from the queue as soon as it attempted to send them to the client; if that specific HTTP response didn’t reach the client due to a network TCP failure, then those events could be lost). The queue servers are a very high-traffic system, processing at a minimum a request for every message delivered to every Zulip client. Additionally, as a workaround for low-quality NAT servers that kill HTTP connections that are open without activity for more than 60s, the queue servers also send a heartbeat event to each queue at least once every 45s or so (if no other events have arrived in the meantime). To avoid a large memory and other resource leak, the queues are garbage collected after (by default) 10 minutes of inactivity from a client, under the theory that the client has likely gone off the Internet (or no longer exists) access; this happens constantly. If the client returns, it will receive a "queue not found" error when requesting events; it’s handler for this case should just restart the client / reload the browser so that it refetches initial data the same way it would on startup. Since clients have to implement their startup process anyway, this approach adds minimal technical complexity to clients. A nice side effect is that if the Event Queue Server server (which stores queues in memory) were to crash and lose its data, clients would recover, just as if they had lost Internet access briefly (there is some DoS risk to manage, though). (The Event Queue Server is designed to save any event queues to disk and reload them when the server is restarted, and catches exceptions carefully, so such incidents are very rare, but it’s nice to have a design that handles them without leaving broken out-of-date clients anyway).
54.3 The initial data fetch
When a client starts up, it usually wants to get 2 things from the server: • The "current state" of various pieces of data, e.g. the current settings, set of users in the organization (for typeahead), stream, messages, etc. (aka the "initial state"). • A subscription to receive updates to those data when they are changed by a client (aka an event queue). Ideally, one would get those two things atomically, i.e. if some other user changes their name, either the name change happens after the fetch (and thus the old name is in the initial state and there will be an event in the queue for the name change) or before (the new name is in the initial state, and there is no event for that name change in the queue). Achieving this atomicity goals means we save a huge amount of work that the N clients for Zulip don’t need to worry about a wide range of potential rare and hard to reproduce race conditions; we just have to implement things correctly once in the Zulip server. This is quite challenging to do technically, because fetching the initial state for a complex web application like Zulip might involve dozens of queries to the database, caches, etc. over the course of 100ms or more, and it is thus nearly impossible to do all of those things together atomically. So instead, we use a more complicated algorithm that can produce the atomic result from non-atomic subroutines. Here’s how it works when you make a register API request; the logic is in zerver/views/events_register.py and zerver/lib/events.py. The request is directly handled by Django: • Django makes an HTTP request to Tornado, requesting that a new event queue be created, and records its queue ID. • Django does all the various database/cache/etc. queries to fetch the data, non-atomically, from the various data sources (see the fetch_initial_state_data function). • Django makes a second HTTP request to Tornado, requesting any events that had been added to the Tornado event queue since it was created.
54.3. The initial data fetch 289 Zulip Documentation, Release 1.7.0
• Finally, Django "applies" the events (see the apply_events function) to the initial state that it fetched. E.g. for a name change event, it finds the user data in the realm_user data struture, and updates it to have the new name. This achieves everything we desire, at the cost that we need to write the apply_events function. This is a difficult function to implement correctly, because the situations that it tests for almost never happen (being race conditions). So we have a special test class, EventsRegisterTest, that is specifically designed to test this function by ensuring the possible race condition always happens. In particular, it does the following: • Call fetch_initial_state_data to get the current state. • Call a state change function that issues an event, e.g. do_change_full_name, and capture any events that are generated. • Call apply_events(state, events), and compare the result to calling fetch_initial_state_data again now. The apply_events code is correct if those two results are identical. The final detail we need to ensure that apply_events always works correctly is to make sure that we have EventsRegisterTest tests for every event type that can be generated by Zulip. This can be tested manually using test-backend --coverage EventsRegisterTest and then checking that all the calls to send_event are covered. Someday we’ll add automation that verifies this directly by inspecting the coverage data. In the Zulip webapp, the data returned by the register API is available via the page_params parameter.
54.3.1 Messages
One exception to the protocol described in the last section is the actual messages. Because Zulip clients usually fetch them in a separate AJAX call after the rest of the site is loaded, we don’t need them to be included in the initial state data. To handle those correctly, clients are responsible for discarding events related to messages that the client has not yet fetched.
290 Chapter 54. Real-time Push and Events CHAPTER 55
Queue processors
Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes: • Asynchronously doing expensive operations like sending email notifications which can take seconds per email and thus would otherwise timeout when 100s are triggered at once (E.g. inviting a lot of new users to a realm). • Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g. UserActivityInternal) which don’t have any immediate runtime effect. • Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the Tornado-based events system. Example events might be that a new message was sent, a user has changed their subscriptions, etc. • Processing mobile push notifications and email mirroring system messages. • Processing various errors, frontend tracebacks, and slow database queries in a batched fashion. • Doing markdown rendering for messages delivered to the Tornado via websockets. Needless to say, the RabbitMQ-based queuing system is an important part of the overall Zulip architecture, since it’s in critical code paths for everything from signing up for account, to rendering messages, to delivering updates to clients. We use the pika library to interface with RabbitMQ, using a simple custom integration defined in zerver/lib/ queue.py.
55.1 Adding a new queue processor
To add a new queue processor: • Define the processor in zerver/worker/queue_processors.py using the @assign_queue decora- tor; it’s pretty easy to get the template for an existing similar queue processor. This suffices to test your queue worker in the Zulip development environment (tools/run-dev.py will automatically restart the queue pro- cessors and start running your new queue processor code). You can also run a single queue processor manually using e.g. ./manage.py process_queue --queue=user_activity.
291 Zulip Documentation, Release 1.7.0
• So that supervisord will known to run the queue processor in production, you will need to add to to normal_queues in puppet/zulip/manifests/base.pp; the list there is used to generate /etc/ supervisor/conf.d/zulip.conf via a puppet template in app_frontend.pp. The queue will automatically be added to the list of queues tracked by scripts/nagios/ check-rabbitmq-consumers, so Nagios can properly check whether a queue processor is running for your queue. You still need to update the sample Nagios configuration in puppet/zulip_ops manually.
55.2 Publishing events into a queue
You can publish events to a RabbitMQ queue using the queue_json_publish function defined in zerver/ lib/queue.py.
55.3 Clearing a RabbitMQ queue
If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue
./manage.py purge_queue user_activity
You can also use the amqp tools directly. Install amqp-tools from apt and then run: amqp-delete-queue--username=zulip--password='...'--server=localhost \ --queue=user_presence with the RabbitMQ password from /etc/zulip/zulip-secrets.conf.
292 Chapter 55. Queue processors CHAPTER 56
Custom Apps
56.1 Definition
Zulip defines a "custom app" to be a piece of code that runs in the Zulip ecosystem, but which is not part of the core Zulip codebase. Custom apps are mostly synonymous with "bots" and "integrations" in the Zulip ecosystem. We currently do not support any kind of browser plugin model.
56.2 Problem statement
Zulip wants to enable people in the world to author custom apps with the following goals in mind: • Simple custom apps should be simple to write and deploy. • Custom app authors should be able to easily distribute their work. • Zulip should provide deployment support for mature, general-purpose bots, ideally either within organizations (a Zulip admin can vet their own custom apps and easily deploy them across upgrade cycles) or across organizations (custom apps get distributed with the Zulip tarball). This document describes Zulip’s current infrastructure, as well as laying out a roadmap for some future features.
56.3 A quick note on bots/integrations
As noted earlier, a custom app is just a generic term for what we often call bots or integrations. We recognize that bots and integrations can have different connotations. A bot typically spends most of its time responding to Zulip messages. An integration usually represents an app that interacts with some large third party system like an issue tracker. We will use both terms in this document in an informal sense, but from an architecture standpoint, we treat bots and integrations as essentially two shades of the same color. Many integrations are implemented as "bots." Likewise, any bot that does stuff outside of Zulip acts as an "integration."
293 Zulip Documentation, Release 1.7.0
Since the line between what a "bot" is and what an "integration" is can get very blurry, we try to be informal about "bots/integrations" and more formal about how "custom apps" actually function within the system.
56.4 Categories of custom apps
56.4.1 Stimulus/response and read/write
At the end of the day, most useful apps respond to some stimulus and produce a response. In the Zulip universe, Zulip can be the source of the stimulus, or the target of the response, or both. Along those lines, we divide custom apps into these three types: •A Zulip Reader uses activity on Zulip to stimulate an external response. An example here would be a follow-up bot that sees messages with the alert word "@todo" on a stream and then adds a task to a third party todo-list tool. •A Zulip Writer reacts to external stimuli and generates Zulip responses. An example here might be a build bot that gets triggered by an automated code build finishing and then writes "build finished" to a Zulip stream. •A Zulip Read/Writer reacts to a stimulus from Zulip by responding to Zulip. An example here would be a math bot that sees a message saying "compute 2+2" and responds with "2+2=4" on the same stream or back to the user in a PM. The above three classifications represent kind of a Zulip-centric view of the universe, but we should put ourselves in the shoes of somebody "out in the world." •A World Reader is an app that gets some stimulus from the outside world and produces a response in Zulip. (So, a world reader is a Zulip writer.) •A World Writer is an app that gets some stimulus from Zulip and produces a response in the outside world. (So, a world writer is a Zulip reader.) Some things are a little outside of the scope of this document. We could plausibly extend Zulip some day to host World Reader/Writer apps that don’t even write Zulip messages but simply use Zulip as a kind of middleware platform. More in the short term, we will have custom apps that may read/write from multiple sources. For example, a meeting bot may take input from both a cron job and a Zulip stream, and it may write to both a Zulip stream and a third party calendar tool. For the scope of this document, we won’t spend a lot of time talking about how to build these types of apps, but we are aware that any solution needs to accommodate multiple sources and targets.
56.4.2 World Reader/Zulip Reader
Finally, we set the stage for how we talk about custom apps in terms of these two broad categories: •A World Reader responds to stimuli from the outside world (and typically produces a response in Zulip). •A Zulip Reader responds to stimuli from Zulip conversations (and typically produces a response in the outside world). Again, we recognize that there can be overlap between those two categories for complex custom apps, but we mostly leave it as an exercise for the reader how to implement those apps.
56.4.3 Other classifications
We discussed one dimension for classifying custom apps, which is whether they are world-readers or Zulip-readers. Here we cover a few other classification schemes briefly:
294 Chapter 56. Custom Apps Zulip Documentation, Release 1.7.0
• Generality Does the custom app have a specific use case or a general one? The spectrum here could run from a bot that Alice runs to update a text file on her laptop (specific) to a Twitter Bot that is optionally deployed on all Zulip realms (general). • Authorship Who wrote the custom app? Was it written by contributors to the Zulip project? • Maturity How well tested is the custom app? Is it just a prototype? Has it been sanctioned by an open source community? Has it been vetted by Zulip developers? • Deployment Where does the custom app run? Does it run on Alice’s laptop? Does it run on a Zulip server? Does it run as a plugin on third party infrastructure? • Authorization What streams are the custom app allowed to read and write from? Which users can the custom app interact with? • Identity How does the custom app identify itself on Zulip? How does it identify itself to the outside world? • Third party We call the non-Zulip target or source of a custom app the "world." The "world" could be almost anything, ranging from an electronic device or text file to a large third-party system like Twitter or GitHub. A lot of the classification schemes are interrelated. Here are some examples: • For specific-purpose custom apps, authors may be happy to just deploy them on their own hardware. For general- use custom apps, authors may want to have them deployed on the Zulip server with super-user capabilities. • As a custom app becomes more well-tested and well-vetted, the author will likely upgrade its deployment over time. At first the author may run the custom app on their laptop, then they may find dedicated hardware, and then finally they contribute the app to the Zulip project so that Zulip admins can deploy the app on Zulip servers. • The nature of the third party will influence the deployment strategy. If I have a little home-grown gadget that can turn off the lights in my kitchen, I may run a custom app on my laptop that reads my PMs for "turn-off-the-light" messages. If I write a generic custom app that needs to update a third party corporate system based on Zulip events, I may want to deploy code to a public webserver or try to get my code to be part of the Zulip project itself.
56.5 World Reader
A World Reader custom app is an app that responds to stimuli from the world outside of Zulip. It typically functions as a Zulip Writer and posts some kind of message to a Zulip stream or user to alert people of world events. Here are some example stimuli: • A Travis build finishes. • Somebody tweets on Twitter. • A hardware sensor notices a temperature increase. • A pull request is submitted to GitHub. • A cron job gets started on your laptop to send a reminder. • Nagios detects a system anomaly. Setting aside issues of how a custom app is constructed or deployed, you basically have to solve these problems: • Detect events. • Translate events into Zulip messages. • Post the messages to Zulip.
56.5. World Reader 295 Zulip Documentation, Release 1.7.0
56.5.1 Zulip integrations
Zulip actually supports a bunch of integrations out-of-the-box that perform as World Readers. The three different integration models basically differ in where they perform the main functions of a World Reader.
Webhook integrations
In a webhook integration, the deployment model is usually this:: 3rd party hardware: • detect event • send data to Zulip webhook Zulip: • support webhook endpoint • translate event to messages • internally post messages One current limitation of our system is that we don’t have a great way to deploy prototypes of webhook-based custom apps before Zulip has vetted the translation and added an official endpoint. Maybe we could set up some kind of webserver that can run translation code outside of Zulip and externally post the messages, and we could think about how to structure the code so that it is easy to eventually turn it into a Zulip-hosted integration.
Python scripts
In script integrations, the deployment model is usually this: Custom app author’s hardware: • detect event by polling a third party system • translate event in the script • externally post messages These type of integrations are typically easy to prototype, but they can be harder to deploy in production settings, since we rely on the authors to run their own scripts. In some cases authors might want to at least move the translation/posting code to live on Zulip, by contributing that code to Zulip as a server-side integration. Then, there would still be the challenge of detecting events in the third party system, where maybe the user submits a patch to the third party as well.
Plugin integrations
In plugin integrations, the deployment model is usually this: Third party system (driver): • detect event Third party system (plugin): • further detect/triage event • translate event
296 Chapter 56. Custom Apps Zulip Documentation, Release 1.7.0
• externally post to Zulip For third parties that have a plugin model, there are often other issues at play, like the plugins may need to be written in a non-Python language like Ruby. There are probably still some scenarios, however, where a lot of the logic for translation could be moved to a Zulip-side integration, and then we supply very thin client code for the plugin.
56.6 Zulip Reader
A Zulip Reader custom app gets stimuli from Zerver itself. Most Zulip Reader apps are packaged/advertised more as what people commonly call "bots" than as "integrations." (But sometimes what is currently a "bot" should really be deployed more like an "integration" in an ideal Zulip universe.) Example custom Zulip Reader apps can be serious or whimsical. Serious • A user tags a message with an alert word like @followup or @ticket. • A user needs help computing something, like a simple math expression or a timezone conversion. •A World Reader custom app posts something to a Zulip stream that we want to cross-post to another external system. • A user wants the custom app to query the outside world, like look up the weather or search Wikipedia. • A bot collects RSVPs for an event. • A bot conducts a user survey. Whimsical • A user wants to see a random quote of the day or a random cat fact. • A user wants to tell the office telepresence robot to "turn left." Setting aside whether a custom app is performing a serious or whimsical function, there are a few different types of Zulip Readers: • Some readers will do simple local computations and post right back to Zulip. • Some readers will do more expensive/web-related computations like searching Wikipedia, but then post right back to Zulip. • Some readers will mutate the outside world in some way, like posting messages to third party APIs or controlling hardware. • Some readers will do some combination of the prior bullets.
56.7 Deployment issues
Zulip currently provides only minimal deployment support for Zulip Reader custom apps: • It ships with a few native server-side bots like the welcome bot and the notifications bot. (These are nice to have, but they are so tightly integrated into the Zulip core that they don’t act as great examples for future app authors, and they are not easy to extend/customize.) • Zulip does ship an API client that can conveniently read a .zuliprc file, poll for incoming messages/events, and post new messages to the Zulip server.
56.6. Zulip Reader 297 Zulip Documentation, Release 1.7.0
56.7.1 Local deployment
If you download the API client and write a bot that reads from Zulip, you face the following challenges if you deploy your code on your own devices: • It can be difficult to keep the app running 24/7. • You may have latency issues connecting to the server. • If you want super-user permissions, you have to secure the API key. • Without integration to the Zulip server, the app may spin needlessly during upgrades. • If you’ve written a personal-use bot, it can be difficult to distribute code to your friends and have them be able to deploy it. • If you’ve written a general-use bot, it may be difficult to persuade your admin to give you a superuser account. We want to make it easier to deploy Zulip Readers on Zulip hardware. The following document talks about how we want to enable this from a code structuring standpoint: Writing contrib bots This document, on the other hand, is more about designing the Zulip backend system to support eventual deployment of reader apps on the Zulip server. Before we talk about server-side apps, we should consider an intermediate solution.
56.7.2 Non-Zulip dedicated hardware
There are some scenarios, mostly with general-purpose "serious" custom apps, where an app author might use the following development process: • Create a prototype and deploy it locally. • Publicize the app and deploy it on non-Zulip hardware. • Contribute the app to the Zulip distribution, so that admins can run it Zulip-side. To give a concrete example, let’s say that I work for a company that is building an issue tracker, and we want to offer Zulip support. I would start by writing a Zulip Reader that scans for the alert word @ticket on certain public Zulip streams, and part of that app would have logic to post to my company’s issue-tracking API. Once I’m confident in my prototype, I will probably run it on dedicated company hardware that might already have tight physical security, 24/7 IT monitoring, etc. But what if I don’t have this kind of infrastructure available to me? Typically what I will do instead is rent time on some kind of hosting service. Some hosting platforms are basically just remote Unix systems, but others are more oriented toward hosting web apps. Zulip’s current roadmap assumes that authors will likely gravitate toward web-based solutions (even if it’s just running a web server on their own Unix host in the cloud). Zulip intends to offer support for "outgoing webhooks." The term "outgoing webhook" can be confusing, depending on your perspective, but it simply means that an HTTP request is outgoing from Zulip, so that it will hit a web endpoint that runs a third-party custom app. Zulip will allow the custom app author, probably with the help of a Zulip admin, to configure Zulip to send a subset of Zulip messages to the author’s web endpoint, and then the protocol for the custom app will to read the HTTP request and send some kind of HTTP response that optionally results in a message being written to Zulip. Meanwhile, the custom app can mutate the "world" as it sees fit.
298 Chapter 56. Custom Apps Zulip Documentation, Release 1.7.0
56.7.3 Zulip-side support for reader apps
Even for app authors that have access to dedicated hardware, there would be several advantages to running Zulip Readers under the same umbrella as the core Zulip system. • Your app will automatically inherit the uptime of the Zulip server itself (in terms of hardware availability). • There will be no network latency between the app and the server. • Securing apps to have superuser permissions will be less problematic. • Keeping your app in sync with Zulip upgrades could become more automatic. • Allowing multiple users in your realm to run their own copies of personal-use bots would be easier to administer. The only problem with the above bullets is that we haven’t built out any of that infrastructure yet. We do have pending PR #1393, which addresses some of the issues that might come up. In order to run apps inside the Zulip server, we basically need to solve the problems below. (One assumption is that we don’t run apps truly in-process.) • Contributions: We need a process for users to contribute code. • Configuration/Discovery: We need Zulip to be able to find which apps are allowed to run for a particular deployment. (The admin may choose to run only a subset of contributed apps.) • Queuing: We need to queue up events for readers, with some possible optimizations to scan for alert words during the in-process part of the call. • Drivers: We need a generic driver that can pull events off of a queue and hand them off to our specific reader objects. • Nannying: We need to launch readers with some kind of supervisord-like nannying. • Pausing: We probably need a way to pause/stop readers without stopping the Zulip main processes. (At first this may just be part of solving the nanny problem.) • Identity: We need to identify reader instances as specific Zulip users (non-owned bot, human-owned bot, or human). • Superusers: We may need some readers to have users with special privileges like being auto-subscribed to all public streams. • Read-only: We may need some readers at the other end of the spectrum to be highly locked down, e.g. enforce that they truly only have read access to Zulip messages. • UI: We will want to provide some UI features that give admins and/or regular users visibility into which server- side apps are running.
56.7. Deployment issues 299 Zulip Documentation, Release 1.7.0
300 Chapter 56. Custom Apps CHAPTER 57
Unread counts and the pointer
When you’re using Zulip and you reload, or narrow to a stream, how does Zulip decide where to place you? Conceptually, Zulip takes you to the place where you left off (e.g. the first unread message), not the most recent messages, to facilitate reviewing all the discussions that happened while you were away from your computer. The scroll position is then set to keep that message in view and away from both the top and bottom of the visible section of messages. But there a lot of details around doing this right, and around counting unread messages. Here’s how Zulip currently decides which message to select, along with some notes on improvements we’d like to make to the model. First a bit of terminology: • "Narrowing" is the process of filtering to a particular subset of the messages the user has access to. • The blue cursor box (the "pointer") is around is called the "selected" message. Zulip ensures that the currently selected message is always in-view.
57.1 Pointer logic
57.1.1 Recipient bar: message you clicked
If you enter a narrow by clicking on a message group’s recipient bar (stream/topic or private message recipient list at the top of a group of messages), Zulip will select the message you clicked on. This provides a nice user experience where you get to see the stuff near what you clicked on, and in fact the message you clicked on stays at exactly the same scroll position in the window after the narrowing as it was at before.
57.1.2 Search or sidebar click: unread/recent matching narrow
If you instead narrow by clicking on something in the left sidebar or typing some terms into the search box, Zulip will instead select the first unread message matching that narrow, or if there are none, the most recent messages matching that narrow. This provides the nice user experience of taking you to the start of the new stuff (with enough messages you’ev seen before still in view at the top to provide you with context), which is usually what you want.
301 Zulip Documentation, Release 1.7.0
(When finding the "first unread message", Zulip ignores unread messages in muted streams or in muted topics within non-muted streams.)
57.1.3 Unnarrow: previous sequence
When you unnarrow using e.g. the escape key, you will automatically be taken to the same message that was selected in the home view before you narrowed, unless in the narrow you read new messages, in which case you will be jumped forward to the first unread and non-muted message in the home view (or the bottom of the feed if there is none). This makes for a nice experience reading threads via the home view in sequence.
57.1.4 New home view: "high watermark"
When you open a new browser window or tab to the home view (a.k.a. the interleaved view you get if you visit /), Zulip will select the furthest down that your cursor has ever reached in the home view. Because of the logic around unnarrowing in the last bullet, this is usually just before the first unread message in the home view, but if you never go to the home view, or you leave messages unread on some streams in your home view, this can lag. We plan to change this to automatically advance the pointer in a way similar to the unnarrow logic.
57.1.5 Narrow in a new tab: closest to pointer
When you load a new browser tab or window to a narrowed view, Zulip will select the message closest to your pointer, which is what you would have got had you loaded the browser window to your home view and then clicked on the nearest message matching your narrow (which might have been offscreen). We plan to change this to match the Search/sidebar behavior.
57.1.6 Forced reload: state preservation
When the server forces a reload of a browser that’s otherwise caught up (which happens within 30 minutes when a new version of the server is deployed, usually at a type when the user isn’t looking at the browser), Zulip will preserve the state – what (if any) narrow the user was in, the selected message, and even exact scroll position! For more on the user experience philosophy guiding these decisions, see the architectural overview.
57.2 Unread count logic
How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range of ways people might use the product. The algorithm is as follows: • Any message which is selected or above a message which is selected is marked as read. So messages are marked as read as you scroll down the keyboard when the pointer passes over them. • If the whitespace at the very bottom of the feed is in view, all messages in view are marked as read. These two simple rules, combined with the pointer logic above, end up matching user expectations well for whether the product should treat them as having read a set of messages (or not).
302 Chapter 57. Unread counts and the pointer Zulip Documentation, Release 1.7.0
57.3 Testing and development
In a Zulip development environment, you can use manage.py mark_all_messages_unread to set every user’s pointer to 0 and all messages as unread, for convenience in testing unread count related logic. It can be useful to combine this with manage.py populate_db -n 3000 (which rebuilds the database with 3000 initial messages) to ensure a large number of messages are present.
57.3. Testing and development 303 Zulip Documentation, Release 1.7.0
304 Chapter 57. Unread counts and the pointer CHAPTER 58
Markdown implementation
Zulip has a special flavor of Markdown, currently called ’bugdown’ after Zulip’s original name of "humbug". End users are using Bugdown within the client, not original Markdown. Zulip has two implementations of Bugdown. The backend implementation at zerver/lib/bugdown/ is based on Python-Markdown and is used to authoritatively render messages to HTML (and implements slow/expensive/complex features like querying the Twitter API to render tweets nicely). The frontend implementation is in JavaScript, based on marked.js( static/js/echo.js), and is used to preview and locally echo messages the moment the sender hits enter, without waiting for round trip from the server. Those frontend renderings are only shown to the sender of a message, and they are (ideally) identical to the backend rendering. The JavaScript markdown implementation has a function, markdown.contains_backend_only_syntax, that is used to check whether a message contains any syntax that needs to be rendered to HTML on the back- end. If markdown.contains_backend_only_syntax returns true, the frontend simply won’t echo the mes- sage for the sender until it receives the rendered HTML from the backend. If there is a bug where markdown. contains_backend_only_syntax returns false incorrectly, the frontend will discover this when the backend returns the newly sent message, and will update the HTML based on the authoritative backend rendering (which would cause a change in the rendering that is visible only to the sender shortly after a message is sent). As a result, we try to make sure that markdown.contains_backend_only_syntax is always correct.
58.1 Testing
The Python-Markdown implementation is tested by zerver/tests/test_bugdown.py, and the marked.js implementation and markdown.contains_backend_only_syntax are tested by frontend_tests/ node_tests/markdown.js. A shared set of fixed test data ("test fixtures") is present in zerver/fixtures/markdown_test_cases. json, and is automatically used by both test suites; as a result, it is the preferred place to add new tests for Zulip’s markdown system. Some important notes on reading this file: • expected_output is the expected output for the backend markdown processor. • When the frontend processor doesn’t support a feature and it should just be rendered on the backend, we set backend_only_rendering to true in the fixtures; this will automatically verify that markdown.
305 Zulip Documentation, Release 1.7.0
contains_backend_only_syntax rejects the syntax, ensuring it will be rendered only by the backend processor. • When the two processors disagree, we set expected_marked_output in the fixtures; this will ensure that the syntax stays that way. If the differenes are important (i.e. not just whitespace), we should also open an issue on GitHub to track the problem. • When those above settings are not in use, we set bugdown_matches_marked to false. bugdown_matches_marked is the predescessor to the more descriptive backend_only_rendering and expected_marked_output fields, and when false, should be replaced be one of those. We plan to eliminate it once we’re out of cases where it is false. • For mobile push notifications, we need a text version of the rendered content, since the APNS and GCM push notification systems don’t support richer markup. Mostly, this involves stripping HTML, but there’s some syntax we take special care with. Tests for what this plain-text version of content should be are stored in the text_content field. If you’re going to manually test some changes in the frontend Markdown implementation, the easiest way to do this is as follows: 1. Login to your development server. 2. Stop your Zulip server with ctrl-C, leaving the browser open. 3. Compose and send the messages you’d like to test. They will be locally echoed using the frontend rendering. This procedure prevents any server-side rendering. If you don’t do this, backend will likely render the Markdown you’re testing and swap it in before you can see the frontend’s rendering.
58.2 Changing Zulip’s markdown processor
When changing Zulip’s markdown syntax, you need to update several places: • The backend markdown processor (zerver/lib/bugdown/__init__.py). • The frontend markdown processor (static/js/markdown.js and sometimes static/third/ marked/lib/marked.js), or markdown.contains_backend_only_syntax if your changes won’t be supported in the frontend processor. • If desired, the typeahead logic in static/js/composebox_typeahead.js. • The test suite, probably via adding entries to zerver/fixtures/markdown_test_cases.json. • The in-app markdown documentation (templates/zerver/markdown_help.html). • The list of changes to markdown at the end of this document. Important considerations for any changes are: • Security: A bug in the markdown processor can lead to XSS issues. For example, we should not insert unsani- tized HTML from a third-party web application into a Zulip message. • Uniqueness: We want to avoid users having a bad experience due to accidentally triggering markdown syntax or typeahead that isn’t related to what they are trying to express. • Performance: Zulip can render a lot of messages very quickly, and we’d like to keep it that way. New regular expressions similar to the ones already present are unlikely to be a problem, but we need to be thoughtful about expensive computations or third-party API requests. • Database: The backend markdown processor runs inside a Python thread (as part of how we implement timeouts for third-party API queries), and for that reason we currently should avoid making database queries inside the
306 Chapter 58. Markdown implementation Zulip Documentation, Release 1.7.0
markdown processor. This is a technical implementation detail that could be changed with a few days of work, but is important detail to know about until we do that work. • Testing: Every new feature should have both positive and negative tests; they’re easy to write and give us the flexibility to refactor frequently.
58.3 Zulip’s Markdown philosophy
Note that this discussion is based on a comparison with the original Markdown, not newer Markdown variants like CommonMark. Markdown is great for group chat for the same reason it’s been successful in products ranging from blogs to wikis to bug trackers: it’s close enough to how people try to express themselves when writing plain text (e.g. emails) that it helps more than getting in the way. The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot of wikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to fix the formatting after typing it the first time. While that’s basically fine when writing a blog, it gets annoying very fast in a chat product; even though you can edit messages to fix formatting mistakes, you don’t want to be doing that often. There are basically 2 types of error rates that are important for a product like Zulip: • What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through your Markdown implementation, would you need to change the text of your email for it to render in a reasonable way? This is the "accidental Markdown syntax" problem, common with Markdown syntax like the italics syntax interacting with talking about char *s. • What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doing so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this figure substantially. Both of these are minor issues for most products using Markdown, but they are major problems in the instant messaging context, because one can’t edit a message that has already been sent and users are generally writing quickly. Zulip’s Markdown strategy is based on the principles of giving users the power they need to express complicated ideas in a chat context while minimizing those two error rates.
58.4 Zulip’s Changes to Markdown
Below, we document the changes that Zulip has against stock Python-Markdown; some of the features we modify / disable may already be non-standard.
58.4.1 Basic syntax
• Enable nl2br extension: this means one newline creates a line break (not paragraph break). • Allow only * syntax for italics, not _. This resolves an issue where people were using _ and hitting it by mistake too often. Asterisks surrounded by spaces won’t trigger italics, either (e.g. with stock Markdown You should use char * instead of void * there would produce undesired results). • Allow only ** syntax for bold, not __ (easy to hit by mistake if discussing Python __init__ or something). • Add ~~ syntax for strikethrough. • Disable special use of \ to escape other syntax. Rendering \\ as \ was hugely controversial, but having no escape syntax is also controversial. We may revisit this. For now you can always put things in code blocks.
58.3. Zulip’s Markdown philosophy 307 Zulip Documentation, Release 1.7.0
58.4.2 Lists
• Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it. • Allow only * for bulleted lists, not + or - (previously created confusion with diff-style text sloppily not included in a code block). • Disable ordered list syntax: stock Markdown automatically renumbers, which can be really confusing when sending a numbered list across multiple messages.
58.4.3 Links
• Enable auto-linkification, both for http://... and guessing at things like t.co/foo. • Force links to be absolute. [foo](google.com) will go to http://google.com, and not http:// zulip.com/google.com which is the default behavior. • Set target="_blank" and title=(the url) on every link tag so clicking always opens a new window. • Disable link-by-reference syntax, [foo][bar] ... [bar]: http://google.com. • Enable linking to other streams using #**streamName**.
58.4.4 Code
• Enable fenced code block extension, with syntax highlighting. • Disable line-numbering within fenced code blocks – the